2019 09 19 Stuart Armstrong   Research Agenda Online Talk

in this talk I'll give a presentation of
the impossibility result in reward
function learning and by research agenda
to get around this result and still
learn what humans prefer the no
pre-lunch theorem for value learning is
quite simple we have the behavior of an
agent or a human and we wish to infer
what their preferences are from Matt
unfortunately we can't we cannot go
directly from the behavior to the
Preferences without making strong
assumptions now no free lunch theorems
are not normally about relevant because
one apply the simplicity prior or
regularization and then they go away
however here simplicity priors don't
help in fact they make it worse the
three simplest explanation of human
behaviors are that humans are fully
rational all the time humans are fully
anti rational or humans have flat zero
preferences and it doesn't matter too
much
what's form of simplicity one's uses I
tend to use comb-over of complexity but
one can also use things like code or
complexity plus program run time and one
gets sensibly the same results how does
this work well I'm gonna model an agent
as a pair P a planner and R a reward
function what is a planner well we're
trying to model rationality rationality
bias or all ways of acting in the most
general form possible so what the
planner is is a map from reward
functions to policies for instance the
fully rational planner is one such map
satisficing is another such map and so
on
what we can observe from a view of the
agent is only the policy PI so the image
of the planner and we can observe this
only in part however I'm going to assume
for simplicity that we can observe the
entirety of the policy which means we
observe not only behaviors with
counterfactual behaviors if we're
thinking about humans and we imagine
looking inside the brain looking inside
the gray matter this could give us a way
of observing the full policy by
estimating what the brain would do in
exotic circumstances now let us start
with the pair that is the simplest pair
compatible with PI simplest platter
reward function pair compatible with PI
means that well P maps are on to this PI
so these are possible interpretations of
the agents policy I'm going to do a
simple operation here in most computer
languages I'm going to apply the planner
to the reward function this gives us the
policy by definition and then I'm going
to construct three other pairs from this
also very simple operations I have the
greedy planner which always takes the
action that maximizes immediate reward
that's the PG this goes along with a
reward function defined by the policy
this reward function is one if the agent
takes the action a policy wouldn't would
make it take and zero otherwise
so obviously the greedy planner applied
to this reward function would always
choose the action that the policy would
say so this pair is also compatible with
the policy double negating this with the
anti Creedy planner and the negative of
that reward function also gives us a
compatible pair finally there is the
indifferent planner
hi which maps every reward function to
that policy add to that the zero reward
function and you have a compatible pair
since note that the complexity of these
pairs are the complexity of the policy
so in the top two the planners are very
simple and the roar'd functions very
complex for the indifferent planner the
planner itself is very complex and the
reward function is very similar however
since these operations of applying one
applying the planner to the reward
function to get the policy and then
either construction the reward function
from that policy or constructing the
indifference planner are both simple
operations this shows that these three
are in most computer languages very
simple pairs so they are floating around
the bottom of complexity now contrast
that with the real pair so a planner and
a reward function which are well what
we're calling real something that better
encompasses our understanding of human
values this pair contains bias
information I the difference in
expectation between the optimal policy
for this reward function and the actual
human policy these bias can be quite
complex contrast that with the other
three pairs they don't have any bias
information whatsoever
the greedy planner has no bias at all
the anti greedy planner is maximally
biased at every point and the
indifference planner has no concept of
bias because it doesn't even look at the
ward function and the ward function is
zero anyway what this means is that any
realistic plan reward pair must contain
more information than these three these
three pairs do
hence in terms of complexity it is of
higher complexity because it contains
more information than these three doom
now I want to emphasize that these three
planners these three degenerate planners
I'm not particularly afraid that we'll
stumble into them you can rule them out
with one bit of information however what
this shows is that simplicity does not
get us what we want and if we start
ruling out everything by hand we don't
know the one we have left there will be
what we want at this point there may be
an objection springing to mind it
certainly had sprung to my mind is that
we know what humans want from observing
them so what is supposedly impossible is
for us easy and you can predict that
behavior with that knowledge so this
isn't just internal certainty this has
demonstrably effects on the real world
for example if we see someone getting
very angry yeah nope not someone getting
very angry someone getting red in the
face shouting at us and throwing stuff
at us we can conclude that they are
green wish to harm us we've got done
here has gone from behavior to
preferences something supposedly
impossible so what is going on here well
humans have a theory of mind which I'm
gonna model here as an empathy module
and a human predictor
these have co-evolved for three humans
and the typical history h/h for a human
age the empathy module applied to this
history by both humans K and L are
roughly the same thing so the humans
theory of mind are quite similar from
one human to another
what's also the case is that our own
internal theory of mind when the
appliance to ourselves is quite similar
to other humans theory of mind about us
not identical but very similar in terms
of agents in possible mind space
moreover once we have the planner and
reward function that comes from these
empathy modules if we apply the
predictor to it and it's a pretty good
predictor so given the empathy module
ends the predictor the reward is simple
to use and typically predictive
limitations of this is that for it only
applies in typical circumstances when we
encounter very novel situations our
theory of mind may start to break down
and more importantly it's only given the
theory of mind so if we don't have an AI
that starts out with this theory of mind
it will not deduce the same thing about
the preferences of humans we use this
theory of mind a lot in practice though
it feels a bit like debugging and design
rather and error correcting rather than
using our theory of mind and injecting
our preferences so when we decide to
model something
or they use the candidate functions for
expressing human preferences and when we
throw out obviously wrong results and
change the hyper parameter until it
works what criteria of it works
I'll come from our empathy module and
our prediction module and this is okay
because the programmers are implicitly
using their theory of mind which is
somewhat shared in order to correct
debug or refine their programs or in
practice inject their own values into
the process which which since human
values and preferences are quite similar
is is actually a prat and practice a
fine thing to do now the limitations of
this are that implicit to explicit is
not easy in AI witness the failures of
good old-fashioned AI and the expert
systems so just because we can do it
implicitly doesn't mean that we can do
it explicitly evening the other thing is
that our empathy modules are not fully
shared just a small example here they
compared Indians and Americans and found
that Americans tended to assign
responsibility for certain actions more
to people's innate preferences whereas
Indians assigned it more to
circumstances so empathy modules or
theories of minds are not perfectly
aligned or not perfectly the same so
there is some limitation to using
empathy modules to deduce the values of
other agents the other problem is that
this does not generalize very well
reason being well the obvious is that if
we assign implicitly if we chew
our hyper parameters or tune-up programs
locally by our implicit module when we
extrapolate this we're likely to
encounter situations where our
extrapolated values go quite wild in
order to get around this and ground a
definition of human preferences we need
a good theory and the sketch of such a
theory is what I'm going to present in
the rest of this talk now taking Daniel
Dennett was famous for talking about the
intentional stance if we look at say
alphago we can look as a physical stance
this is a collection of atoms and
electrons and put in certain
configuration we could also look at it
the design stance this was designed to
win it off ago we could also look at it
as the intentional stance in which
alphago intends to win so if we play
alphago then we intend we don't intend
but we expect to lose notice the
different stances are useful in
different circumstances if I'm moving
alphago from one place to another the
physical stance is the most useful if I
am playing limit go the intentional
stance is the better way of modeling
them now then it's intentional stance
was for the purpose of predicting the
agent's behavior though our use of it is
to predict the agent's preferences but
because this is a stance it is possible
to have a super intelligent AI that we
could have all the world's video feeds
all of Wikipedia all social science
research perfect predictions of human
behavior be able to perfectly manipulate
humans so ideal at interacting and
sniffing humans in every way
and it could still conclude that humans
are fully rational and it would not be
wrong it would not necessarily be right
but it would not be wrong it is using a
different theory of mind so given that
how can we go about actually learning
human values well we can start with the
human source code or heart brain and
from that we can look at human mental
models the models in which we rank
different options like ooh that would be
embarrassing or oh I really want that or
I really I would like to go that way
I will not go that way I think I prefer
this option these preferences exist
within our mental models when we compare
one option with another and extract
these partial preferences as I call them
as a series of this is better than that
this is better than that in various
background circumstances some of these
will be meta preferences preferences
about what we ourselves would want to
prefer like we might prefer to be more
generous or more consistent or less
hypocritical to do this we need some
human symbol grounding we need to
understand what the symbols in our brain
mean when translated into the world
fortunately this question seems to have
empirical solutions then the assumption
that I am going to make the thing that
removes the no free lunch theorem by
making the assumption is that these
preferences are relevant these a greater
than B this altering greater than that
is preference relevance these are in a
sense the definition
of what the pieces of our preferences
are the preferences within our mental
models then using a complicated
alchemical process of merging and
waiting all these partial preferences
together we can get an actual reward
function or utility function that
summarizes the preference of this one
human this can then be given to the AI
in order that it can maximize that
human's preferences of course there is a
AI symbol grounding issue there
what does behavior have to do with this
well there is a lot of uncertainty
potentially about the humans internal
brain States about various versions of
symbol grounding and this would
translate into uncertainty about the
ultimate reward function that the AI
must maximize human behavior is then
taken as evidence of the human source
code of the human brain States and of
the human symbol grounding so in this
way once this theoretical construction
is complete the AI can look at human
behavior and infer what we would prefer
now since it is good to at least start
normalization here is how I am imagining
these things to go starting with the
source code having the preferences as
pre-orders then using using model theory
to ground these these human pre-orders
into preferences over worlds and then
assuming that these are preference
relevant and construct
the human utility function from this and
another round of symbol grounding for
the AI and the AI now has a utility
function defined in terms of its own
world theories to maximize
these are in a sense the key parts of
this research agenda the mental models
the human symbol grounding the meta
preferences and the synthesis process
the Sigma that puts all these pre-orders
together the meta preferences are
generally seen as preferences over
different Sigma's the theory of mind the
empathy modules the i9h wants in this
view it gives us privileged access to
the pre-orders to the mental models when
we say that we know what other people
want what we are getting is using our
own theory of mind to access their
internal models this method even if not
carried out fully can be used to improve
other methods for example one method is
stated preference ask humans what they
want or reveal preferences observe what
humans do and conclude that that is what
they genuinely want or some form of
idealized human thinker like mega
philosopher in a box for a thousand
years all of these have flaws there are
many cases where stated preferences are
wrong and where we know them to be wrong
revealed preferences have a lot of
problems with bounded rationality for
example I have not submitted a patent to
cure various cancers yesterday even
though I probably could if I was
unbounded Lee rational and looked
through Wikipedia and quantum mechanics
so you can't conclude from that that I
do not wish to cure cancer or do not
wish to be rich the problem is that in
most toy examples of revealed
preferences the actual options that the
humans have are very impressed rikta
dand are not in a realistic set
idealized human thinker has
the problem of value drift now in these
cases we can patch these things we say
okay yes in politics or in the court we
don't expect people to necessarily be
truthful in romance need either reveal
preferences yes there is this problem
there's that problem and then we can
patch and patch problem with this of
course is that we patch until we can no
longer think of any patches to apply
further so we basically go as far as we
can think and then give up that is
almost certainly still flaws that we
have not discovered where the method I'm
proposing could help is instead of just
doing these patches we can say that
method M fails in situation s so that's
a patch but we say that it is because
the internal models of the human differ
from what this method would predict so
once we have a reason for this patching
in this failure it seems a lot easier to
get the AI to generalize from that
finally there are a bunch of other
methods like courage ability the ability
to modify the heirs goals safely
low-impact distillation amplification
insular methods they all require some
portion of human preferences they cannot
be done in a value of agnostic way in a
fully evaluate Gnostic way however as
you can I like to say candy Hitler how
Thanos all agree on what those terms
mean like they would all agree on what
acreage abou a I is of what a low impact
area is so what this means is that we
don't have to fully define a human
utility function just some pieces of it
in order to apply it to these math
it's thank you for listening for more
details look at the Occam's razor paper
on the archive and my research agenda
unless wrong links to these will be
provided in the youtube comment area box