Existential Risk and AI Safety Talk by Pepe Bawagan

[Applause]
which the commercial conceit here in the
show patience
sorry there was a software developer and
by implication and they were the team
behind stuff I got behaves you thirsty
though I am leaders used to know which
is the most optimal way to go over point
so maybe if I meant is that he is also
so at that is thereafter now people here
based in Manila is also I had conceived
director or Filipino feat interests and
an organizer for their part in Manila so
organize another
[Applause]
so well and x-rays before we can be
certain talks particular particularly
was first stepped up and some has both
so existential risks when we say ex
extension is just something at least we
can kill something to make you not exist
you cannot individual details but exists
on the planet and there are at least
this is also but just there's a
path to be things exceed what is super
bucks so these are viruses or the danger
this is not equals to what makes
developer Fox in a birthday the tribal
employee consumer extras is nuclear war
what do many warheads on this planet and
it's all because this is a lot by Hannah
strikes where we cut up all over the
place we're already in Exodus to be to
existing service limited to things are
just in the future we could corner be
happy
and of course there's asteroids very
cinematic you know hard to get anything
about things I've been making all the
way backwards
there's also super cool games asphalt
hunger what
so you might ask where this I fit into
the six if we look at how a thousand
artists graphs expect releases you find
that that just actually has very good
book code numbers or extenders one that
ice itself is not a high school it's not
a neglected if you don't hear about this
a lot if you're not about packaging
we've hardly ever get about thirty so
each estimated is more sort of evenness
so okay so what is it so admit I may be
furious
it's not easy different let's talk about
the for millions of years like what
would be considered the dragon can
provide us the jeep genes are basically
machines with only wonder negative and
that is the remedy if you're familiar
with the self esteem by Richard talked
about robotics explains they're very
well reading the our sleeves energies
and inside he all started as note that
these simple features single-cell
organisms
you're thinking through the front one
will suit who came to the be not walking
sooner the depth of chance and ears
refreshment from natural selection of
the most erudite in the morning where we
hit the first accident
which is see and so it's a sensation
there's not that sensation well it what
we mean by sensation in this one pulls
later they get body part and these are
basic safety of sensations feature the
pole and this is it buddy
we're creatures well you have the
Queen's of sensing the environment you
can direct yourself take a different
perception do things that will increase
their chances on and beyond of the seat
straightforward enough simple perception
is not faultless especially when there
is a competition after emergence second
accident so when weather is may also
fall with happen of intelligence
features to black for more complex
departments to executables
so it's also be price to more complex
decisions
so these accidents so for example of
dries up intelligence give us patience
and soccer sexual desire
humans don't want us and look at our
speaking part of the lake on Janice to
cater with the latest result of very
broad of increasingly pregnant villages
and this is very almost negative
readings don't underestimate the speak
there's no generic speaking its
intelligence it's maxed out and give it
enough babies and come back and by just
involved in this organization so we got
you're using person preventative ratings
of the dome tones but it'll be here
they're not satisfied enough understand
that intelligence is like magic dust and
you sprinkle off so it's wrong we gets
sprinkles of intelligence on where you
get that's you seen a bunch of oils
people something that is on that we met
house robots because they're to make
choices
so when we use intelligence everywhere -
and so it that the terms is so useful it
makes like so much easier and being
intelligent creatures and we are we can
tell just pass how do we get more this
takes us to the next chapter here quite
unwitting start so at this point set of
metrics we came from measure
intelligence and stop a few said bar so
we decided to explore so anyone here in
our deep so for those two articles the
blue was just made computer development
idea 1997 is the field then we'll just
chop them down
this was a sport urgentiy follow as
exclusively believable about humans but
no justice I mean if you can good force
whether it's not like good which
requires superior human strategy
intuitions something a computer would
never involve a program that was
developed by did not mean there are my
very words I'm going to see it defeated
one of the world's best operators Lisa
go for the one people say still have to
treat human beings basically but it can
only make human humans are personal you
must play chess
who encho single human cyclical games
alpha 0 was developed in 570 it can play
just go and sorry
birthdays it learn is only by itself
leap which means that did not eat games
like human sprite it just learn the
rules and then played against itself
million tons of times and that's how
it's not okay to say what else is going
to start all perfect information gives
when you see more information on the
brain and your absurd spaces that we
just move here to your beauty and if I
can't avoid meeting where is the don't
go everything perfect information you
have so many options to choose from
and they need coordinating there any
danger according to the Dean this is
stuff I don't think they thought that
Lois a or you don't know them let's get
a firming up remain you have so many
options to choose from it's a very safe
environment and you before eight women
be so it's not just one a lot it's not
more neater this is another it's five
agents versus my mother am I saying
swell baby I have is people like agents
design precisely people included in
April this year just this year they won
best of three series yes if they're not
already chapters so these are world
class cleaners maybe we think in deliver
and you could say yeah these games are
always same map and with their fixed-up
able to choose for us or not seconds
it's limited set of characters some AI
can adapt to family violence right just
lasted too much speed never opening
honey again resistors off of their sons
training multiple engines to play - see
now these are some of the things
something that engines learn so the
expected stuff it's a high density the
people who are high in
but were hiding learn to block entrances
there are apps in the environment the
Seekers usually drops to go over the
walls find out
see it's just denials okay
so sweet sadly be expecting this and
then because there's a great experience
before the start of the game with
seekers count the hikers learn to think
is Rattus in the room so that the seeker
is good at ease you're absolutely right
you think the resources when your
opponent when the game starts okay so
the environment where there was a clear
rule set up and there were walls the
others were able to build little
shelters for ourselves so that Seekers
couldn't see them and people file okay
particularly which is basically an
exploit of the rav4 the the movement
mechanics in the game but the reality is
use enough to step out of a box and then
surf that box onto a wall and cross that
wall to find the Highness nobody didn't
even know this was possible in the a
night view there's of instances on three
so with little box
and the defense part is just the hydras
reactive to that attitude at all being
seekers are serving with boxes not well
take the boxes as the group here divide
the boxes and lasting the muffler post
this extended mix link which was a very
specific they learned them if you look
the wrap the words at wall good not
yourself very compatible like Chapel in
through the roof which you know Wow so
they spoke with that and I don't know
I'm happy about designing it's my
emphasis is that great scared so that's
looking agents these are just behaviors
right it's not like understand you're
just doing stuff it's not
Burt for high quality of the
representations contrast - is the
natural language processing algorithm
that was the relative oh my god you can
give it a lot of things and it can
answer questions about what is just to
give a little story
ask mr. Bob sort of where did this
person understands it get lots of use
autocomplete positives so we see some
text of the Stella team keep writing
something coherent so this would be use
school dances if there are students were
like okay I'll just use this is it
working how is it possible and can beat
you better be silly really common
utterances pretty clear and I think
finally the researchers actually tell
that some of the people use to build
because it was the persona this use of
this technology could be very hard for
the responses making the truth
I've been waiting for the sign is just
moving horses first I stable hey it's
but that possibly with that okay I do
this I'm just removing see smart let's
fast-forward techniques that someone
combined power into something that can
be programmed instructions it has a very
powerful activation so it's not a means
by which to manipulated by knowing the
someone that is division of humanities
its mother
[Music]
[Music]
[Music]
[Music]
[Music]
hey mercy we're smart Berkeley some
positive wisdom about how things are
expected directly ask us more questions
about achievement which brings us
allegiance last accident well these here
genes have better chances of replication
if they're mostly smells strong Oh
circles like gravity and kill the bad
things but look at behaviors become a
social animal
so what sets us apart is work about
generalize philosophize the out of
these drivers which I wouldn't think of
things like the golden rule or
minimizing something what that's alien
be honest about what does morality have
to do with intelligence in German note
that when I need to create the absence
of evolution intelligence and morality
which is Lisa Nicole what do you mean
when they were formed this is a
simplified version make things easier to
illustrate this is a space where
intelligent agents and so I've never
eaten can talk anywhere in the States
what people mean is that just because
something is intelligent it's not being
that if we were sleeping for example you
know your main example is in Cuban
intelligence many people very smart
alternative ways to look at it it's with
the liberal or Obama and others it's
fine
so the office determines your direction
while the rocket turns how fast the
other guys is basically the ability to
achieve goals just Muffy those moves are
so for example a smart kid
learn is faster it's being smart doesn't
see what exactly it is there isn't
either boxy or alter not workers or to
just be moral put your back and not
others that were taught not to be in the
context of humorous it's not exactly be
smart and there's also a better sense by
which racially turkey Woodworth before
the Natalie abusing since their full
circle
Arase which is where exactly that any of
us even very young babies usually have
some basic moral bridges like punishing
battles for reference for good agents so
they're good studies tough fight the
even babies look we're gonna all
ready to go like basic moral we have to
be supportive Drive this is an excellent
I will just babies my Allah whose
experiments
Oh
[Music]
[Music]
[Music]
[Music]
keep the gold my any meets this even
with other straight fun no physical
forest was in fact Forex SEC
psychological problem so that's how the
I will work by point it does not have
the moral blocks and until so protocol
that morality is added will be my
superior pick subjectiveness no real way
around this now not it's nothing like
that
he gave - my memory it could just as
well be indifferent same way we don't
care when they're asks where we want to
go house just destroyed because it's so
[Music]
well one thing is for sure please
and this is not probably it's all still
very messy this is my centerpiece or
philosophy trying to be a single rule
whatever cool device versus great others
it literally that sure that includes
everything Dara is otherwise we both
side so there's a reasonable question
about another fact is absent
he is deluded by humans and other tons
of features you sweep up some cetera
feliner configurations at morality
there's mother
meet us with making sense of the world
these are certain elements of powers
which makes sense
it was dirty every track a new creature
exploring as well efficient use of
calories so whoever the Spy is
inhibitors in this book return is an
unfamiliar problem spaces I'm sorry
however this vice- discovery of certain
decisions in of them in their public
spaces so let me give an example here's
an AI agent simulated it was fast food
up the Milwaukee behavior you have a
different Marconi's back 74 thanks short
thanks mrs. Linzer
so it's pretty good to finally swap and
then what ass require thing minimize
foot contact with normal so imagine
oh this is feature-length two-week
period
to avoid touching and II would measure
what percentage of the market we sto
company and see what it should look like
when it's given a basketball stars it's
nothing weird about this back so they
don't need to see outside the box
because it was never any possible in the
form of physical machines and this is
best Hospital power are volunteers on
Easter day the main point is in
operations even if we simulated the
superintendent's just directly reference
joint villages on something that the
direct vigorous recently the flash you
think in super fast
yeah office to work a set its average
over but for everyone else
next is also measuring human even their
sisters
transmitted from another 100 meters per
second there's virtually virtually so
that's like another lot faster and so
also this kind realizing a next ministry
which is scale human raise interest
rates particularly under the data file
size due to the skull and the size of
the skull is limited by the pelvic down
right because
exactly goes on with that and here's a
lot of evolutionary pressure for us to
get smarter actually we live in
community
we've got your next bigot that's exactly
and humans but going back to steal
that's part reason of it
the cycle spa computers are limited in
the cyclo whatever we have to prison the
size of warehouses right we know how
computers we're not stopping
and so even if even if they were spread
out over a large area so that you can
hurt that transmission so these this
packet is also generated even
independent or my intelligence
oh this even it would be super super
intuitive because she's so given the
facts super intelligence Oh what do we
have
yes this swath of Utah mercy
[Music]
just do that with Hector's people we're
just going to be super nice it's fine
yes and that's harmless let's see
hey guys hope this problem but it's just
thinking possibly harm us this lightning
remember unless you live in a nice space
or passionate this export anything if it
finds out that solving this problem is
difficult and it wants as long as soon
as possible unless a rendering of that
which time to just reconfigure or joyful
make this solution very fast I'm just
gonna get rid of
and then it's all problem excuse me and
then I'll just make the planet into a
computer taught us all right
that's how no Oh simple company that's
asking day night classifies the
productivity Pericles okay um this is
how we ended up with a galaxy peace
reduce the I just start making paper
clips and later on some manipulating the
people into violent deeper clips to
putting more power and it starts
transforming the batteries need to do
and it's like a mortal drank he needs to
be think that's how there's another
example for copier and this one impulse
[Music]
we've got issues afterwards never said
why soon after no one cared
no series it was exposed to excellence
and lights to decline updates from all
the society rather than a technical set
and as watching Thomas blushing
magically funding it was the importance
of Mataji for us how he was actually
needed to do its job it was a lot to
learn to me on the last Friday important
is one point versus break it's always I
had to run unsupervised for days one
engineers went to seventh it is just a
fun experiment we're in the process of
doing in Shirley activity experience no
safety of
because he won't even start a framework
designed to treat general artificial
totes no so narrow won't test a limited
system due to that it won't look input
from the likes reading site probably
font on the system has everything in
Orland which meant the entire world
itself it took in a world around the day
to become smart one of you or I was
going to become 10 times smarter
tentatively called superior times of all
quiet-like behind it a weakness in her
plan and it was smarter than anyone
could understand 27 a mindful that the
tradition is garrison release the most
want the airport was a planet of
celebrated made sent over some tiny but
David a bit see bullets on the street
individually they were dust of the way
to block single strategy enabled
together people is called National
through the change of manipulates
research cutest white concealer around
sunrise objective what was just as a
background and isn't in quality
everything else intact they've got a lot
about this that every destabilized small
parts of magnetic storage but even all
to teach to the books researchers became
an enormous great we should be thankful
that the author was told to cause us
that the disruption is possible whether
it's personal instructions it might not
work out and lastly honda human minds is
the one that what a small world
definitely Daniel and Emily mission
safely at once notice us wasn't
important death was an extremely
destructive lot of bacon spaghetti and
the party is alive in contributing or
passwords mentoring than that Seth was
hence it is somewhat altered but the two
months of growth it was kind of job to
do and also mean that happen to super
intelligence which asset account
installing was essentially is here but
it also do units then what
but try to see and that we sorted so
it's mostly whatever team I try to
research that sort of depends on anyone
who found themselves lastly store fat
loss and problem adjusted they're also
quite distressed to actually do anything
about it all the time with it once you
recover you might have stopped caring
it will also be the only serious
oppressor place would be from another
oxytocin sensitive they are honed found
themselves distracted by the project
almost everyone would be a I found
someone as productive or going to work
on nowadays
Felix what anyone who likes to test
important today I just doesn't touch
that it's all party or understood the
fundamental supposed to the universal
president Paragon his role in a while to
Drita's party reverse engineer Juliet
that's a little standoffish intelligence
to kill never was a person operating
machine is their leader these we got any
more alas curious people know that
itself as by watching this video so we
need to cover its father
no way I know it's part of a static but
yes that amount of time firms says
enough of a machine which will escape
absolutely to throughout the universe
but in various someone's attention
remote never what was not certain it's
flying around outside every
obvious it's not expense report
optionally include things to remember
all of the main camera so step no
against you ever learn what amounted to
look in one extra
instead of some responses very much Sean
get a hippie communes to be point of
Commerce's copyright is there in 3d
software we've got you doing which
recently has become very vitalizing
force one that if you watch a video will
try and if you want something a little
bit about recently you're going to end
up muscles so that's you can think
alright because it's examples policing
magnetic circuit let business in urban
the earthworks and the pretty racist
Sammy's so these things even if we if we
leave these unchecked the results can be
passed on we're not so that's why we
really need to work on what if you try
to be as careful want to encode the
product exists in our heads so that's us
Internet Protocol didn't bounce but it's
not gonna matter and Pratt Paris or at
least of us isn't something that's might
be better maybe we get around this and
eat my word it'll cost me around
203
we recently deliberative a vice
intelligence learn what you want if we
don't know what they want maybe I
haven't heard it even if we don't have
we don't excuse it
I think and it got pretty I said we're
not children to learn what the rule this
is still rather suffers from something
on federal resume I just want to see
whether that's a morality something an
alternate self the remember that this is
a super know that behind it can change
its own hope it you're never incentivize
the change itself in such a way that its
goals that's a really tricky challenge
so that's it sucks almost impossible but
hopeful and there are
[Music]
[Applause]
merci so what what would it take to get
something that's well let me present you
a quote from TVs out of our speech
 intellect we can suspend making
it more powerful
yes people in the world and so the
challenge for us is to solve alignment
before we reach our eternity so
generally for the visual general
intelligence is a difficult problem
solver making part of your general
dentist Tennessee is striking point so
we want to be ready with safety pots
when he did foreknow here energy nasty
hot summers
you could have figured out a whole
preference before they loved all that
energy the difference this side is that
states are retirement when the
intelligence mom was Paul there's no
it's a c-spot
and don't be it also that more resources
develop theories on us thinking about
these things
you're very privileged Jasin expand as a
special consciousness beyond ourselves
doesn't matter and they do believe that
the mechanics we can in order to extend
this privileged once there's a saying
that goes there is thousands of sites
further busy distracting or the Fiat
Republican a stronger person a clear-cut
so one we can help solve these promises
to get or make one point we can get more
people are more tight raising raising
them up
it's only thirty easier problems and so
that people visit you can do it up it's
an original thing and even if people who
are the Phenom survival concerns it's no
guarantee they will not be to each other
which is what a sustained interest
listening to these state walks on Oscar
public mr. Stokes
Oh
[Applause]
[Applause]
[Applause]
I think activation is definitely a huge
backward of the tricky thing is
separating actually with diligence
because very intelligent creatures with
vitamins yes to actual and we become
people right I can't I can't live how
did you drop but I can feel the
machining and just look at you drop so
the same way was to get to his relatives
it's very hard to give me the
application for exactly Germany some
things that people can think of the hope
you just put it in a virtual environment
it's in a box this is still pregnancy
because something that is by definition
smarter net it happily trick you into
Anakin power cause it can pretend to my
option for women is who the summit
regulation so it's very hard to predict
if you need to be talked about is
magnification a lot smarter so yeah
you don't think maybe want to break that
yeah that's for my eye tonight I don't
have seen oh my god just about anything
my comforts at their sorry what's
important we just try to take the
subsystem can get some information how
to return
I figure that ideally infant life at the
moment
[Music]
so they're many fronts so one of them is
very poor in the desert the coast is
worthless of their Ewing research oil
architecture the architecture
inviting others so living it behave the
way we want so that's that's a movie
small engineering philosophy there there
another protects big defeat power risks
of the pipeline in the briefing room
digital technology therefore there are
no good news there's some things like
early and others are trying human
machine brain increases so that we
digital natives
and at least a certain internet using
our products and it dropped there's one
more effectively and so that might might
believe very small I cannot blow
anything
one of the things they're just like if
they might guess obviously but already
for human engines you might actually
have to make it if our nuclear base
digital improvement is still important
house for homeless but bit more recently
interact directly with the planets might
have more chance I don't think it's
school but that's what those are
well first of it I think that he's going
in there they'll also put your first
baby thousand Aras because they also
bring a Bible to the spotlight so if I
have a circuit yes
so I guess
[Music]
Oh
[Music]
it's going to have some problems so I
make no illusions on stocking battery
but what I wanna do is build the cushion
for natural rapport
it's not on safety research pic3 sorry
we want to instead of doesn't need my
researchers with slow down first was
stopped with the better their approach
help don't increase the number of people
working partly I see and the alignment
so we want to just speed that up so that
we can we doesn't have a seat
that's early to just give very good
we used to think of opponents as just
people power over them this is better
than people as well it's changed so much
of its diversity and it Reserve will go
for example the environment has been
exhibiting either that's it's actually
the Cheetahs new techniques or not
really so I think it's fine it's good to
talk to people to decide if they wanna
say I think they will they might get so
frustrated if it makes too much better
than that there's also questionable we
wanna commodity type so that the top
work you can hatch I was very well I
worked anywhere my other destroyed or a
lot they were I am super job so didn't
even have that control
so I guess it's up for each of their the
paper to decide with if you want leaning
against the higher power
I think it's definitely interesting how
you've started learning from a gay life
and one of the trends that we keep
growing back paper being expenditures
may be positive dormouse wish to be a
mom of streamers but I think we might be
at a point where he might be interesting
to spectate notches now let my fear of
the DC metro may be motionless people
because people like this people like
people like their watch supers because
they're an agency before the Big Rapids
chat so I think the next step is
[Music]
that's a very good question so there
there's many possible futures that we
coddle you know post our content our
abilities well a state partnership
they're effective if we don't get Eric's
but if we do get a say for certain
general templates some people think that
we might be able to lay hold of
something like a lot like like just
thinking things in general don't you
realize we sleep in something it towards
our bodies look there's also people who
think that well that's not limited
possess that get it it can be then it
actually expect what people see kind of
my ex so some people think you know we
get there pretty speedlights just maybe
a giant bird across the planet a party
they ain't even here all of our needs
many expats across the galaxy you know
solve mr. universe
well we're just worried that's kind of a
heuristic
there's different different approaches
the land fraud be sure that shows of the
values negative about yourself
I think about this problem a lot but we
have the ability but I don't know
solution that comes to mind is not
solidly think about it but I don't think
that's impossible with our there this is
super self are giving are they correctly
don't for that less you thought you know
social service work or it's rough sketch
not have a solid if they're letters so
when the second group Pacific Theater
space where it's really what what
situations we rappin and increased
exploring right and the moment you add
groups there it gets super because
there's so much carnage
come on every cool so have not be aware
of like I'm missing things would be the
Safa contacts with this kind of problem
I just never look into
let me guess every context of my cage
like what I write is that this like
typically people no way possible it's
actually a collection of like smaller
pieces of algorithms that are actually
specific yeah and so if you're actually
looking at industry now there are
actually a bunch of situations work you
don't it's not naive but a great classic
specifically a little that's the
telephone
so that is my free time that are scoring
a creative generating critics forest
version and so like you make sure it's
not just profiling some remains the
races and things they're acting like
it's not just I mean I need these up
quick stick in the sense that would be
just really good people but all the time
they're forced to do this because of my
rig deletion
like telling them you can't deploy that
thing unless we can make sure that it's
not even it's so like anything that has
some like we're gonna culture around
happening that has a lot of implications
in terms of like how instantly regulate
this thing looks like one perspective
business look further away and regulate
eg I assume you guys or people blind
when we apply this Oracle and smoke
which I usually would be first ox we
give them don't usually a book for
people so this book was only released
last month it's the newest book of
stories the hand of the Center for human
compatibility
eating what the media research
institutions work not yet and so 85% its
media amazing and it's actually very
breathable even for it not yeah
researchers not had people who are so we
also have one more copy of this which we
will be raffling off to anyone who might
want to read it yeah
[Applause]


Introduction to Reinforcement Learning and Concrete Problems in AI Safety

okay okay cool
hi everybody I don't think you so much
for this tutorial that reinforcement
learning and concrete problems so I'm
hopefully everybody's ready paper now so
I'm going to do it's a really brief
overview of the learning Hall and
problems discussed and paper and a large
talk about it into questions later okay
so what is reinforcement learning what
ever care about it was cool the basic
idea reinforce the learning is that you
have some agent a little aging guy and
he's got actuators which is the means by
which he can interact with his
environment world around him he also has
percepts you mean sensors stuff like
that so his eye as it touches whatever
by which you can get information from
the environment and then you can act on
it with actuators our agent also has
some functions enable them to learn what
to do given any particular state the
environment these functions the crux of
reinforcement learning and they consist
of a reward function which we've got to
specify which is problem we'll get to
later so basically there was more
function tells the age and how good any
particular state is so if he's at time
step one in this particular learning
episode with is at time step one how
good is that time stopping what action
can he take to get to the best possible
next time step how do we know about that
we also have a value function take the
reward function for any particular stage
say you've got some variables in your
wire one over and in front of maximize
these variables somehow you're all
function right stay well if all of these
two variables are
that's great that's what we're going for
us fantastic reward
but if taking action that would give the
agent that reward will also make it
impossible to ever get to any other
state with maximum reward that would
have a low value so the body function
here is to capture not just the rewarded
office of the state but the reward of
every other possible state that is
accessible on current state how does the
agent know what to do in order to
maximize this value in order to maximize
its reward over all of time steps in an
episode we have something called policy
function which is what the agent needs
so then this could consist of anything
it could be very simple formula it could
be a really massively complicated neural
network basically given these inputs
from the environment from the
environment what we want to do is take
the action that maximizes the foggy
given this reward function which define
its value at every state how do we know
what to do that the policy function will
tell us what to do given all the
information that we've got so far so
that's just a second introduction to
reinforcement learning and what problems
might be associated you know yes this is
you yeah Lagrange s upon this
exploration versus exploitation so given
that the agent thing and it doesn't
necessarily know what reward is
associated with one of these states and
I know that this this particular state
has a reward of 100 but it doesn't know
what these are
should it focus on going to the state
where it knows what the reward is it
knows it's a good reward it could just
stick there and stay there on stay there
and stays there swimming up to that
faith it'll just maximizer would or
should it take a risk shouldn't take a
chance and explore these these other
possible states which might have
negative reward
welcome back to this property minute and
another interesting problem inherent in
reinforcement learning is value function
right so the idea is that we don't only
care about the rewards of that next
possible state but also in the other
thing it's really accessible from that
but how far in the future do we care
maybe we just care about the next two
states that that means I usually will
say fractions without considering any
long-term implications
maybe we care about the next we say you
care about the next four maybe we care
about the next million if we care about
the next million states and you might
have a presentation that makes decisions
that we think are really really bad in
the short term because it cares much
more about for you to remove it
this is relatively context dependent so
let's assume we have an optional agent
an agent with perfect information about
every single state future state has its
reward functional value it is optimized
its policy function such that it will
always take the best possible action to
maximize reward over the entire episode
over every single time step max much
reward fantastic
right that's exactly what we're looking
for but there's a whole host problems
inherent and so this is coming back to
their concrete problems in AI safety
paper which is what I'm going to talk
about now by means of a lovely example
which hopefully make this very clear so
you've got a cleaning robot and you've
got an aneroid
machine will come into your home clean
you might need an absentee father's
house what's the reward function for
this system we say for every piece of
dirt for every piece of rubbish in my
robot you get rewarded - one that would
be pretty good right that I motivates
the machine to remove all this rubbish
uniform fits
therefore maximizes reward function
however there were several several
different problems in terms of rewards
specification
the first one is side effects we've got
our reward function we've got an agent
that's learn to optimize that reward
function it's learn to perfectly remove
every single piece of rubbish from a
fantastic but we haven't told it about
anything else in the environment in our
education um we haven't said he the flap
but while you're doing that please don't
whatever one cat please don't burn the
house down
but in the house that is really good way
to make sure that we no rubbish in the
future so in a house and the one tip
side effects are okay because I mean you
can name any number of unwanted
side-effects and like oh you can just
fit this toy examples with a human robot
it might renew if you like whenever you
it might break all the dishes and my
turn embed upside down by doing all
kinds of different things because we
don't know how exactly this agent is
going to learn it's possible that's one
issue
another one is reward hacking which is
kind of fun oh so you say it's a few
agents or all the function is based on
all the bits of rubbish and bits dirt
that you can see a really good way of
maximizing that reward function making
sure that it never gets penalized again
is to destroy its own visual sensors
right yeah that's much more function
that's exactly what we want but it's not
really good one another problem is able
insight it's possible to make a lot of
these problems to do with side-effects
reward hacking stuff like that by having
some kind of overseer some kind of human
feedback loop so that every time the
agent kind of goes to remember your
camera or I don't know through their
conditions out the window or whatever
you kind of second oh no no no that's
not why I went men-
bridge and the agent will learn that
that's fine
but it's a policy function is so
complicated that you don't know why or
how the agent is making these decisions
and even more importantly in situations
outside of this frankly rather silly toy
example if there is a policy function is
really complicated the agents can we are
quicker than you're able to keep up with
there's no way to provide that kind of
outside problem I'm really interesting
issue is safe exploration which also
kind of comes back to the exploration
versus exploitation proper memory
enforcement learning in general safe
exploration the eye the whole idea of
reinforcement learning is the agent
learned that optimal action to take
images to maximize its reward function
in order to reach the law that we've
specified probably badly and so
experimentation is an actual part of
that right you want to keep your threat
of the most efficient way possible you
want to decline the optional mopping
patterning to make your kitchen for all
spick-and-span what if in the process
finding that optimal mopping the agents
going to try lots of different things
obviously you know that's science and
start from this size and stuff that five
is gonna try different cleaning
solutions great well if one of the
things that it tries to stick we
marketed plug we know that that's what I
did but the agent has no way of knowing
we didn't specify how and finally the
final problem raising the problem is in
a on safety cake that is distributional
shift even if I manage to get to
optimally in my front and not remove my
car and not electrocute itself or me or
do any anything else that we don't want
it to do even if it's perfect to my fat
we can't guarantee that it's going to do
the same thing in your fat you might
have dog you might have different layout
you might have unsafe electrics there's
all kinds of reasons why you can't
necessarily assume that they didn't save
one situations necessary
say from the mother and that's
distribution problem and this is
all kind of concise report specification
right how can we specify everything in
order to create a nation will achieve
our goals and also not do any of the
other stuff that we don't want to do
this is a really really interesting
problem and Fred holiday it's a cool
problem in our safety right now and
fortunately there's lots of other people
working in it
but we're going to look in detail at are
some different pressures to
reinforcement learning safety next cool