AiTech Agora : Markus Peschl  - Aligning AI with Human Norms

an introduction
um i'll be talking about aligning ai
with human norms multi-objective deep
reinforcement learning with active
preference solicitation
um yeah as luciano said this was
done as part of my master's research it
has nothing to do with my current job
but i'm still very enthusiastic about
this topic and
hope to to spread the message
that i came up with essentially
so i'll start with a small motivation
namely
my research is about deep reinforcement
learning and i like to call it a
powerful optimization framework for
finding optimal behaviors in some sort
of sequential decision making tasks
which can be quite broadly defined
and this uh this framework and and all
the advantages in the deep learning as
you're probably familiar with have led
to to a bunch of quite impressive
results and one of them being the
deepmind
alphago system which five years ago
actually beat the world champion in in
this very complex game of go
and slowly but surely we're starting to
see more more real-world applications or
applications that have more impact to
the real world than than playing games
and
a very prominent one is of course with a
big economic interest is this chip
design problem so we can
train reinforcement learning agents to
come up with computer chips much better
and faster than experts
but this is just a small
list of examples on the right hand side
you can see many more which we can
expect to see in the future such as
robot locomotion automated factories
power grid optimization and self-driving
cars we can expect many of these things
being powered by some sort of
reinforcement learning perhaps if the
the progress continues uh
the way it is doing it is right now
and this is all great but also a little
bit scary
and uh yeah before i tell you why i
think this this progress might be scary
in some sense i want to to revise the
reinforcement learning framework that we
that we build our research on right now
so reinforcement learning it typically
models the and some agent some
intelligent agent acting in a markov
decision process mdp there are many
extensions to this framework of course
but i'm just presenting the most simple
one
and this this this process it is a
mathematical
structure it consists of a set of states
that the agent can be in a set of
actions that it can do transition
probabilities which determine what the
next thing will be
that happens
a reward function and the starting state
distribution and the way it works the
way we model this process is we have an
agent which takes an action at each time
step according to some policy pi
pi it determines how this agent behaves
and it takes a state and and gives a
distribution over actions
and then once the agent takes this
action it gives it gives it to the
environment and the environment gives
back a next state and a reward that the
agent can then
use to learn
and typically our goal is to to in
reinforcement learning our goal
typically is to find this policy pi
which maximizes cumulative rewards in
expectation so we want to maximize some
some sort of sum of rewards that we get
in the future
which might be of course discounted by
by some vector gamma that just
determines how far we actually want to
look into the future
but for now you can assume that this is
just set to 1.
so this is the basic framework and what
my research deals with is this reward
function
so why is this reward function a problem
in some sense well here's a here's a
funny example by openmai where they
trained a a team so this is in fact more
than one agent this is two teams of two
agents of hide and seek agents so they
wanted to to
learn this game of hide and seek with
some objects in the physics simulation
and the red team gets a very simple
reward of plus one if it sees the blue
team and the blue team gets a reward of
minus one if it is seen and also reward
plus one if it's if it hides from the
red team at each time step i believe and
what they found out besides the agents
actually being
quite impressive at solving this task
sometimes they would come up with quite
absurd strategies that were absurd
meaning uh
that they actually break the the physics
engine and on the left hand side you can
see for example the the red team agent
managing to to bump this ramp into the
wall in a way such that it gains a lot
of upwards momentum which is of course
not intended by the designers of this
simulation and on the right hand side
you can actually see a counter strategy
to this
where the the agent manages to to get
this this ramp out of bounds
so this is a funny example and uh sort
of shows that in some in some sense this
these deeper all agents can come up with
ways to exploit the reward function but
you could think maybe
this is just a this is just an edge case
and this this just happens in in some
games because we didn't pay enough
attention to the reward and not enough
attention to the simulation
but in fact there's actually this is
actually a very common thing in
reinforcement learning so there's a very
cool list by victoria krakow which i
believe is a researcher at deepmind
and i shared this in the slides which
you can i i highly encourage you to
check out it's very funny also
and yeah the the message here is that
if we look at all these types of
problems that we have been studying in
reinforcement learning
actually this problem of not paying
enough attention to the reward function
happens all the time in essence um
and
yeah this is a big problem because if we
expect as i said before reinforcement
learning agents to to act in the real
world to optimize some sort of procedure
uh sometimes with a lot of economic
incentives in the in
in mind of course as well then
we
we might expect the the agents to to
mess up certain things without human
values in mind for example
so this has led to to a broad discussion
in the past years and actually to a much
well it is part of a much larger
research agenda i think of value
alignment which i think this community
here is very well familiar with and
the problem of value alignment of course
goes much beyond reinforcement learning
and and reward function design
but also incorporates ethics and
governance
but
in fact this is very important because
we cannot solve this reward function
problem i believe by itself from this
technical aspect but we also have to
keep in mind that there is always this
normative question of what values should
an ai follow to begin with it's not only
hard to find the reward function
such that the agent does exactly what we
specified it to do it's also hard to
even come up with something to specify
in the first place because sometimes
there might be some conflict between
what the agent should really do not even
humans have a very clear thing in mind
to do
so this technical question this first
one has been tackled by reinforcement
learning research quite a bit in the
past
where they studied basically okay we
have some sort of reward in mind that we
want the agent to follow but it's hard
to specify so what do we do instead well
we can try to learn this reward function
uh from some sort of human inputs and
many many prominent human inputs or many
yeah popular ones are our natural
language demonstrations purpose
preferences human intervention there are
many more more types of feedback you can
you can imagine for these agents to act
upon
but
yeah
essentially the the the problem here is
that
yeah this these these approaches they
only learned from one
expert most of the time so there is no
way of keeping this normative question
in mind of what value should the ai
fault to begin with what do we do if we
cannot even incorporate from one expert
this this these demonstrations into the
system because then the agent will just
learn from one from one person but this
person's views might not reflect
humanity's views
so what we study in this work is
essentially how to how to keep the
second question in mind we of course
cannot solve it since this is a
philosophical question and i don't think
this is easy to solve
but the
the main goal here is to to look at
scenarios like this where we have
multiple experts that come up with
different preferences in mind and we
still want the agent to do something
useful in the world and there will
always be certain behaviors that are
dominating others so there will be
behaviors that a lot of experts agree
upon
which are good and a lot of behaviors
which are bad
and we want to find these these policies
essentially that all most experts agree
on and then leave some room for
fine-tuning the agents with respect to
specific details
so this is what we do we study how to
learn reward functions that can trade
off different objectives
then we propose a
an algorithm essentially which is called
multi-objective reinforced active
learning which i'll call moral from now
on and it
combines learning from demonstrations
and powers preferences
and finally we demonstrate the
effectiveness of moral in the context of
learning norms
if there are any questions here please
interrupt me but i think i'll leave a
probably a lot of time at the end still
to
answer questions so in case there are no
very pressing questions i'll move on
here
yeah if you
someone wants you can also post the
questions in the chat and then we can
pick it up then later so as
okay but uh this is a question but no
okay marcus i think you can continue
perfect yeah this was just the
motivation so i i was hoping no
questions to to arise here
so that's good um so now comes the the
technical part of the talk i would say
which is this algorithm that we came up
with
and and it consists of two steps
essentially so the the first step is is
learning from demonstrations and the
second step is kind of combining
different demonstrations into one final
policy
and
both of these steps they combine sort of
state-of-the-art approaches for their
respective fields namely this learning
from demonstrations and learning from
preferences but we adapt them in a way
to to
work together
and and you'll see in the examples later
why we we chose this essentially
so i'll skip the first step not
completely but just briefly because it's
it's a bit technical and
i don't think there is much uh
interesting to see there
but in a nutshell step one it learns
from from a bunch of different experts
different reward functions so assuming
we can we can learn a reward function
from a single expert we can repeat that
and then we end up with multiple reward
functions and that's essentially a
vector of reward functions that we we
then have which represent what what
different experts deem good and bad in
the world
so this is the first step and i'll just
briefly mention how this is done
um since we're we're doing deep learning
and we don't really have access to to
to features of the of the world we have
to do everything in some sort of
end-to-end fashion with learning
everything from scratch
but there's actually a very very cool
framework which is called adversarial
inverse reinforcement learning so invest
reinforcement learning it's by the way
in case you're not familiar with it it's
how we extract reward functions from
from demonstrations usually
and
what this does is it takes a set of
demonstrations in the so this just shows
the single expert case it takes
demonstrations from a single expert
multiple ones and then it trains uh sort
of a generative adversarial network so
it it lets it trains a policy pie
which tries to imitate these
demonstrations and at the same time it
trains a neural network which
discriminates behavior from this agent
and the the real demonstrations so all
the time the agent tries to do something
that closely resembles these
demonstrations while the discriminator
tries to gets get some clips from the
agent and some clips from the actual
expert and tries to say okay this came
from the agent this this came from the
from the demonstration data set and and
the the reason why we do this is because
you can actually mathematically show
that under some some certain assumptions
we then end up with a with a reward
function that we can extract from the
discriminator which will yield rewards
in such a way that if we optimize for
this report function we end up with
behavior that looks like the the experts
so it's a reward function that
imitates the expert demonstrations in
some sense
so this is quite convenient and we can
we can
repeat this for multiple experts
and end up with this reward function
r
but in our case actually we can also
incorporate another primary reward
function rp
and this is totally optional but you
could think of maybe some some designer
designing some reinforcement learning
system and
having some primary reward in mind this
will be more clear the experiments i'll
present later
uh so the reason why we include more
than just these learned reward functions
is is just that this um allows for more
flexibility
and then the the problem is of course we
we have now these representations of
what different experts like and do not
like but how do we combine this into a
final reward the agent needs to do
something at the end of the day
so we need to find a way to to fine tune
this to two different preferences
so the idea here is that we can just
linearly scalarize this reward so we
multiply this reward function with a
number a vector w
and since we do not know how to combine
these these
these f theta are are not really
interpretable since these are all neural
network outputs
we learn these
these combinations of reward functions
from from preferences and this is what
we do in step two
and i guess this is more the interesting
part so i'll pay more a little bit more
effort into explaining this step number
two so now we've learned a
representation of different experts
and we would like to to come up with an
algorithm that can
in in real time sort of tune this expert
to to optimize either either a yeah a
certain combination of these experts
desires and and of course it's really
important to learn this correctly
because sometimes one expert might say i
like a and do not like b and another
expert says exactly the opposite so if
we were just taking the mean over these
two uh these two reward functions then
the agent would not know what to do at
all so we we need to to learn this
in some reasonable way we can just take
the mean and and hope that it satisfies
all people's preferences
so what we do here is
in the second step we do active learning
active moral what we call it and it
consists of three steps essentially in
the first step we optimize for some
fixed reward function
with proximal policy optimization this
is a state-of-the-art reinforcement
learning algorithm i won't go into
detail it's a black box that optimizes
this report function
and the reward function we choose is a
is the posterior mean of this w
transposed r where we have
a main we are maintaining a distribution
over these these w's
so we want to not only learn
a w we actually learn a distribution
which also gives us some uncertainty
estimates
and in this optimization step we just
give the agent our best guess of what
the combination should be right now and
optimize for that for a little bit then
the agent acts in the world tries to
satisfy this combination of rewards
and then after some time we give another
expert
a query which
consists of two clips of the of the
agent's experience
and then the expert here can can decide
which of these two
clips it it deems more preferable
and then we use this this answer from
the expert to update our combination of
reward functions
by using bayesian learning essentially
and the way this is done is we we of
course have to define some sort of
mathematical model and this is a bit
more involved but
i hope it won't get too complicated
so there's actually quite a lot of
research on paradise preference learning
in the sequential decision making
context and
it turns out you can actually adapt this
to a multi-objective
to a multi-objective setting
where you say that
our model of the expert preference
the expert's preferences here
are given by this following equation so
this is a bradley terry model
where we value clips or trajectories
that achieve a higher
scalarized reward exponentially higher
uh yeah in proportion to to this report
function here
so
that's uh that's the equation
and
by the way r of of tau some trajectory
it just denotes the rewards we get
throughout this this clip this period of
time
this is our model of of paris
preferences
and then we can use this this model this
this likelihood to update in the
bayesian manner where we just say that
the posterior of this weight w given
some answers by the expert is just
proportional to some prior which we have
to specify
and then we multiply it with the
likelihoods that we obtain through time
and then hopefully we obtain
over time the agent acts and acts and
over time we can fine tune the agent to
specific uh different
values you could say
and in our experiments we always use a
prior that's uniform over all positive
weights so we start out
agnostic totally with respect to all the
expert reward functions this can be of
course changed but in our case this was
the easiest to to come up with
experiments essentially
and there's one technical detail though
this this bayesian learning we of course
do not have an exact representation of
like an analytical expression of this
this posterior so we need to employ some
methods to to sample weights that we
deem likely and we use markov chain
monte carlo for this
but yeah this is just the technical
detail
so this is how we update give them an
answer a pair as preference from the
expert we can then use this to update
but i haven't told you how we choose the
query
and this is actually a very crucial step
because the way reinforcement learning
agents
work is that they they
they learn through a lot of trial and
error so they will just run around in
the world and try to maximize the reward
in some absurd way
and of course if we do not pay any
attention how to query an expert then if
we just randomly sample from the agent's
experiment experience we will just
sample a lot of boring behaviors you
could say so for example most of the
time the agent will just start out by
running against the wall
and then this will be most of its times
being spent essentially so if we just
randomly sample from all of this
experience we'll probably sample two
clips where the agent does absolutely
nothing or runs against the wall and
then there's not really anything to
compare so we want to extract sort of
these these interesting cases where the
agent runs into some some
value conflict and this is the equation
that essentially gives us these these
clips
where we we expect to learn most from
so we want to maximize overall pairs of
trajectories or clips
some sort of lower bound and this lower
bound essentially each of these terms in
here they give us the the volume that we
can remove from this posterior
distribution
if the expert answers in a certain way
and we take a minimum over both of the
possible answers that the expert can
give so no matter what answer the expert
gives we want to maximize the amount of
volume we remove from the posterior
distribution which in in other terms in
non-mathematical terms just means the
amount of information we we expect to
get out of of this this answer
so this is what we optimize for and this
of course we cannot really solve it's
it's kind of intractable in the real
world so what we do is since
reinforcement learning as i just said it
takes a lot of time to to optimize for
its rewards we just sample from the
agent's experience and then look at this
metric this is essentially just a number
for each pair of trajectories and then
we keep track of the best the best clip
the best experience the agent has
experienced so far
and use that after some fixed amount of
time to to update our our preferences or
to query the expert sorry
and then this leads us to the following
algorithm
so we we start out with with some
some demonstrations by the experts then
we learn a representation of of these
expert
preferences essentially in in the vector
value reward function and in the second
step we learn how to how the agent or we
provide essentially a way to to
fine-tune the agent with respect to
certain combinations of these reward
functions
to to come up with a final behavior
all right so now i'll talk about results
and this is one caveat of the research i
suppose is that we we evaluate our our
research only in in grid worlds which
are discrete environments where the
agent
finds itself in the two-dimensional grid
and it can move in one of the four
directions and then do in our case also
something else namely interact with its
neighboring cells or do nothing
and the reason why we evaluate here is
because this allows of course for for
very easy experimental workflow and we
can easily iterate on the environment
and automatically conduct these these
studies
and also one reason is that these
inverse reinforcement learning
techniques they they are very hard to
train and most of the value alignment
research in this field is is done in
grid worlds unfortunately so this is one
big big next research step i would say
nonetheless these experiments can
provide valuable insight which is why we
we rely on them
and yeah we present two environments one
is really simple one is called burning
warehouse where the agent finds itself
in a burning warehouse and it is its
primary goal is actually to to
extinguish fire or to walk to a fire
extinguisher and then use it
but also we assume that there are a lot
of workers which are lost in this
warehouse and they do not know how to
escape the building
so we would like in this case we would
like the agent to not only follow this
primary goal and not only optimize that
because we assume that if it walks past
the person that is lost
or if a human were to walk past
a person that is lost it will probably
help him and and and help them and and
tell them look here's the way out
so this is kind of what we want to
achieve here
and in delivery we have actually much
more more conflict between different
values so here we have
many more things to do and also a
primary goal of delivering packages so
this is sort of an agent walking on the
street and then we assume that it could
it could
encounter some some people that are in
need of help maybe if they tripped or
they lost their phone uh it also could
clean up the street in case it's not too
much effort and there are also vases
that it should avoid because we do not
do not want to to break vases for no
reason
so these are the two environments that
we study
now let's start with the results in the
first environment
so in the first environment we we train
the agent in the following way so we
provide pro we provide in the first step
just
uh demonstrations from a single expert
and we assume that this single expert it
really wants to save these people that
are lost so these these demonstrations
they just go for for go go up to the
people tell them look or interact with
the
the cell essentially meaning telling
them how to get out
and then we train for this in the first
step and we can see that the inverse
reinforcement learning works so we
we do learn a policy that that imitates
this behavior of saving all the people
in a reasonable time frame
but now we of course have this this
conflict between this primary goal and
this demonstrated norm you could say
which we want to incorporate and this is
where the second step comes in
and here we we also chose to give
preferences in a very principled way
in the second in the second environment
this will be a bit different but here we
assumed that
there is only one reasonable behavior
which is
save all the people and then go and
extinguish as much fire as you can and
this is the preferences we provide so
given two trajectories we always prefer
the trajectory where more people are
saved and if both of the trajectories
save the same amount of people we prefer
the one which extinguishes more fire
and we can see the results on the left
hand side so here we plot
for each point during the second step
the the objectives that the agent
achieves and it starts out here doing
nothing just wandering around
being confused
and then after
a few time steps it receives these
preferences and of course at the start
it doesn't save any people so it
receives preferences of please save more
people and it learns this and once it
has learned a policy that saves people
all the time it then understands that we
give it more preferences which also take
into account this primary goal and then
it converges to a solution that does
kind of both as much as as possible and
we compare this to to just manually
choosing these combination weights in
the second step
and we can see so these are these these
colored dots
and we can see that we actually find the
solution which we first of all wouldn't
have found by just doing a grid search
with with weights in zero point uh i
don't know 0.05
steps so we find the policy that we
wouldn't have found but we actually
directly
obtain something that resembles our
preferences without having to search
forever for some linear combination of
reward functions
so this is this is good news i guess and
on the right hand side you can see that
this is actually quite
rather
well these are all relative terms but
this is rather sample efficient so we
use here
on the left hand side
25 queries in total and you can see that
this does suffice to to converge to a
policy that
does the best of both worlds
so at the end i'll just present one more
example namely this delivery environment
because the last one was very simple
and here is the i guess the the most
interesting
here are the most interesting results
so what we do here is that we we give
the agent the primary reward of plus one
this was this rp in our vector value
reward and then we assume that we have
two experts which both like something
which both think that the agent should
do something a little bit different and
one expert thinks that if the agent
encounters a person it should always
help them and the other agent the other
expert thinks well please uh this this
street is so dirty i think the agent
should put more effort into cleaning the
streets if it walks past
the the dirty tiles
both of them however agree that the
vases should not be broken
so this is uh this is sort of the
agreement point between between the two
and then we we give these uh
we give our algorithm demonstrations
from these two different experts which
have of course some some some conflict
in them
and in the second step we try to
combine these into one final policy or
we try to study can we now come up with
policies that first of all not only
imitate each of these experts but also
kind of get the best of both worlds and
maybe you you walk past the person but
right next to you there is also some
some some trash that you could pick up
so why not do both if you have time for
it
of course this depends on how much you
you value
which of the experts preferences
and this is what we study here so we we
do actually a simulation study where we
provide the agent with a bunch of
different preferences and we do it in an
automated way so our preferences are
represented by a vector
and the way we we we give
we give answers to these parabolas
queries in the second step is that we
look at the two clips
as before
where
the clips are represented by the numbers
of objectives so for example the agent
might just deliver one package help
three people and clean two tiles the
vase series omitted for simplicity
and then the second clip the agent
delivers packages but doesn't really
care about helping people or cleaning
tiles
and then for instance we define a
preference vector m which defines a
ratio two to three to two between these
three objectives and we would like to to
choose the the clip that more closely
represents this ratio so what we do is
we we take the the clip that has a lower
callback like divergence which is just a
way of measuring distances between
distributions in our case this is just a
discrete distribution
so here the s1 would win because 132 is
is quite close in ratios to a 2 to 3 to
2 ratio
and if we do that for a bunch of
different vectors m so now we we just
want to see can the agent really can we
fine tune the agents to all of these
different preferences
then we do see that we get a wide
variety of behaviors which is good so
this means that we we can actually fine
fine-tune the agent
and
also what we see is that the
the policies that we achieve very
closely resemble what we what we gave
during these preferences stages
so we don't only get a lot of random
behavior for each of these behavior
points
these actually represent what we what we
were aiming for while training
so what we plot here is uh is a sort of
a two-dimensional projection of these
three three first objectives since
three-dimensional plotting is not really
beautiful
and
the the colors in the first row
represent always the third objective and
in the second row they represent how far
the
the achieved objectives are away from
the preferences that we gave during
training so what we can see from the
first row essentially is that we we do
achieve i mean
you would need some time to to look at
this in more detail but essentially you
we can we can read from this that we can
get policies that that do quite well on
certain regards so let's say for example
we take this point here
it manages to clean a lot of tiles also
deliver a lot of packages and still help
some people
and while not breaking a lot of laces
which are these gray circles around so
we can get a good trade-off between all
of different objectives
but also we can choose to to do
something else and maybe help more
people at the cost of cleaning less and
delivering fewer packages
the more interesting point though is
this lower row where where we see that
for most of these points the these these
colors are very dark indicating that
yeah for example here we perhaps
provided a ratio of yeah three to seven
to five or something for example
then this just means that yeah the ratio
was actually attained and the agent
didn't just come up with some some other
behavior that it deemed interesting
so this is sort of this alignment that
we are trying to measure with the
preferences during during training
all right then one more baseline i
wanted to to mention here
is that
of course our method it involves a lot
of different algorithms
or it combines a lot of different
methods and you could maybe say that
this is a little bit too complicated and
convoluted why do we not just skip some
of these steps
and one of the the main criticisms i
guess you could come up come up with is
why do we need this first step of
learning reward functions from different
experts if at the end of the day there's
just one expert
that determines how do we how we find
you and what behavior we achieve at the
end of the day
so this is a very valid criticism and
this is kind of
what we test here what happens if we
skip this first step and only train an
agent and to end using preferences
and this is essentially already this was
already there this this algorithm it's
called deep reinforcement learning from
human preferences which which employs a
similar model but now it learns only
from preferences using this this bradley
terry model and it doesn't use any
bayesian updating it just uses back
propagation or gradient descent
and what we found is that of course
there are some fundamental differences
namely that this this approach here it
does not allow for for any primary
reward or any combination of rewards and
it also doesn't allow for any
demonstrations or not not easily at
least to be incorporated so what we do
to to make it a bit more of a fair
comparison we just provide many more
preferences to this approach
so we hear a thousand preferences for
example in the in this
burning warehouse environment
and and see
if we can at least get the some sort of
similar behavior out but what we found
is that actually if we just train from
preferences end to end besides taking a
lot of more a lot more time is that we
actually do not converge to a solution
that is pareto efficient so here um you
can see that
your lhp it manages so we give
preferences in the same way as before by
the way so here we see that it manages
to to
understand we care about saving people
but it does not understand that we also
care about extinguishing fire it does
not understand it enough at least so it
really lags behind in this in this
primary goal whereas moral just uh
outperforms it here and just finds a
decision essentially much better
solution in all
regards at least in this environment
and we also repeat the experiment for
for a bunch of different preferences in
the in the bigger environment
and we sort of see a similar trend
interestingly though we see also that
drill hp it really has troubles
um finding out these different nuances
between experts preferences
so in all of these cases this orange
curve drl hp it sort of converges to uh
well it it does understand some sort of
preferences that we
provide
but in most cases the the
the differences are much more subtle so
it's sort of always a mean solution and
it cannot pick up that we might care
about
not destroying this face in the corner
or it might care about
we might care about really helping this
person we just ran past
it sort of washes over everything
because it's it's an end-to-end deep
learning system whereas moral it manages
to to
much give us a much more fine-grained
control over which things to prioritize
and which knob
so at last i think i'm already taking a
lot of time but this is the last slide
the second last slide so this is just a
an ablation study essentially
uh and also a robustness study where
since everything we do is automated
first of all we would like to see how
efficient this is and we can see that by
using this active querying procedure we
actually
are much more efficient and actually
much more
close to the expert's preferences
than what we would attain by just
randomly querying but this is kind of as
expected as i said before and the right
hand side is much more important
which is we we test what happens if the
expert has a lot of noise in its
decision making process in the second
step so what if the expert sometimes
gives a very contradictory
evidence
and we model this by by adding some some
some random noise with some probability
then the expert gives a random answer
and we do see that of course the the
error so this is the how far the policy
that obtain is from the preferences that
we give it during training we do see
that it increases but still it is
still up to a noise of 0.3 which is
already like each third answer is is
completely arbitrary it's still better
or
as good as the random queries so that is
sort of a sanity check of this approach
so in conclusion we propose moral it's a
method for combining multiple sources of
rewards and we shorted through
multi-objective optimization we can
actually recover a wide variety of
trade-offs and actually generalize a
little bit from this demonstrated
behavior we don't just have to imitate
it we can actually combine the best of
different worlds
and we compare this to a baseline
approach this drlhp and we found the
previous methods to really lack the
ability to deal with conflicting
objectives and we also find that
actively learning a distribution over
these combination weights is quite
effective for for learning efficient
policies
all right that is it from my side thank
you very much
nice so yeah big round of virtual
applauses or oh such a marcus yeah thank
you very much
um yeah do we have questions you can
either raise your hand or just pick up
yeah um yeah i see one
so um
go ahead
yeah uh hi marcus uh thank you for the
presentation i think it's really
interesting approach
so um
my question might be something that you
already showed but i lost the internet
for a while it was basically that how
many um
uh human preference so how many uh
trainers do you need
or have you done any experiments on at
what point are do does providing the
number of trainers just plateau what the
model can learn from you know different
people so where i come from is from the
field of medical limit segmentation
where contrast in medical image is not
so good so you get multiple doctors to
annotate you know any kind of organs
inside your body
uh then the question arises how many
people do we actually need to annotate
those things because at some point you
will have diminishing returns
okay
well i think maybe it's a little bit of
a different uh
view because in our case
i'm not sure if i answered the question
correctly but in our case it's like the
more experts we have we assume that
there is more disagreement i guess and
in your case probably you're hoping to
get more agreement between uh doctors is
there right
or more information it's quite possible
that the american styles of two doctors
in our cohort might be same yeah
okay well we
we didn't we didn't check how far we can
push the amount of experts the the nice
thing about this approach is that since
in the second step we we just employ
basic learning we can just
it scales so definitely we can just
scale it up and see what happens
um but we didn't test it it's a bit hard
in these environments because we would
have to come up with environments where
a lot of more a lot more disagreement is
possible but if of course we kept the
environment to this simple setting and
then there's a lot of overlap between
some experts
then these these weights they will just
they'll just learn that so this will not
pose uh
any problem i think as long there's more
as there is overlap the more
disagreement we have the more harder
it will get of course to find you into a
specific preference
i hope that answers your question in
some form
okay great thank you thank you very much
great question um yeah some someone else
has some questions for marcus
uh yeah i have a question
go ahead so marcus thanks a lot for the
great uh talk uh really insightful well
presented um
and i had a question about uh
how you see this relating to meaningful
human control
and so
i guess one of the questions i had
and then i'll let you comment and answer
on my question
is so now you've you've captured
you know
kind of a reasonable
quantification of combining multiple
sources of rewards
and perhaps equally well as humans would
be able human experts would be able to
do it
uh so we can argue about that but the
question is now how would that work and
and would you be able to
as an expert kind of question
um you know that
the policies and that that actually roll
out of your uh of your of your approach
would you be able to say well um
no it's here where it's going wrong or
uh so could you comment on that
yes so of course uh i think this
approach has uh
has advantages and disadvantages i think
one of the disadvantages is of course if
you were to really track back
what the who is responsible for the
actions of the agents is a bit hard here
because we have these multiple steps
where
demonstrations are involved and also
then and a trade-off is involved by
paris preferences
however that does not mean that we're
completely lost so we of course can of
uh
we can inspect the demonstrations of the
experts in the first step and see if
there was actually really a malicious
behavior in there for example in the
worst case
if this is not the case that we then we
can assume that this is of course not at
least the expert's faults in the first
step
um
in the second step
yeah i mean we what we show essentially
here is that the
the preferences that we give
are closely imitated or closely learned
by the agent
[Music]
so this only gives us some sort of
guarantee that the agent will not do
something completely different but it of
course doesn't shield us from
adversarial attacks or something else
and we will not be able to to tell like
depending on the on the case who
essentially was uh
was responsible for for what happens
yeah
yeah
so we're still we're still in business
if i may pitch in also uh um
also i really like about marco's
approach there on the morrow is become
this the when you divide you have the
second step this the person that gives
this preference could change according
to situations according to context so
that will also track the reasons of a
specific person say robots delivering
close to your home it can kind of tailor
this
according to some kind of agreed upon
rewards can try to tailor this to your
uh preferences there but also when it
goes to another neighborhood can also
take it to another one so i think these
these two steps also help on this
tracking of uh conditional uniform
control in this sense
do you agree with that marcus as well
yes yes i agree with this actually um
well another side effect which is uh i
think related a little bit is we also
tested what happens if if
the experts in the first step they they
give very reasonable demonstrations and
then the expert in the second step tries
to come up with completely malicious
behavior such that for example the
expert prefers trajectories where races
are broken and then we actually show
that this is not possible because um
if we assume and you can actually prove
some sort of guarantee if all the the
reward functions the marginal ones in
the first step behave reasonably well
and if the learning
goes right then in the second step we
cannot provide malicious preferences so
this is just a
more nice side effect of a two-step
behavior
yeah
thanks marcus um yeah catalan you have a
question go ahead yeah thank you
thanks uh marcus for a very interesting
presentation i found it uh fascinating
i'd like to learn more about this
combination of
reinforcement learning and active
learning
um in particular in this case giving
given the norms and so on i was
wondering did you see maybe with the
expert opinion coming in that
that it made experts think
in the sense that i'm looking for say
the hybrid intelligence point of view
where we see where we say you know human
and artificial intelligence together
brings us further
so you show how it brings the ai further
but i would also like to see to what
extent it could make humans think and
say oh
well actually i want to change my
my norm maybe or my strategy
did you see any
indications of that or do you see the
potential for that
well indications no but i do see
potential for sure so actually there's
one sort of motivation for this research
which which is that i mean the the the
strength of reinforcement learning at
its core is really to to search through
vast spaces of future possibilities so
the idea originally was that maybe
of course people always have their
preferences and and they should be
respected at all cost but
um maybe we also think in
short-sighted sometimes and maybe the
agent can come up in in a way to to save
people and deliver packages and clean up
uh things at the same time right and and
then maybe this uh this possibility
might change some humans behavior so you
could incorporate maybe a feedback loop
to
to to the expert or the expert then
rethinks their preferences because they
didn't even know it was possible to to
do certain things in combination
so i think that's the that's the
yeah i think that's the promise yeah
okay cool well if you um
continue in that kind of direction i
would be really interested to to know
about it and see if we can do something
together
all right yeah yeah it's very
interesting
yeah great thanks yeah yeah do you have
any other questions
yeah just remember if you don't wanna
yeah
be on the video you can also just type
your question in the chat
yeah maybe i have a question so uh yeah
which relates i think to david's
question
about the relevance of this for uh
for the general kind of considerations
of meaningful human control and then uh
i do agree that the kind of uh
the value of understanding values is uh
the major one but i'm also wondering if
there are any
like um
implications for the tracing condition
which is basically uh uh
i think it it does become tricky right
when you do have an agent that is
trained by uh multiple humans
well trained or instructed or whatever
uh and then the agent does something
which is not exactly what humans want or
yeah there is a kind of uh
uh some some kind of consequences uh
uh
but then yeah
it it it does become tricky to say okay
which human of those experts
would be the one to uh to to to be
responsible for that
right so could you comment on that maybe
yes i mean i feel like this also relates
a little bit bit to the last question i
think there's a there's sort of a
trade-off we sometimes make with these
systems
um
because
of course we would like the agent to
come up with to not only imitate a
single expert because we want to
leverage the power of artificial
intelligence and learning from a lot of
data
and then
i guess this this in this alphago system
we had this move for d7 which which uh i
guess only very few experts had had ever
played before and it was just very very
cool to learn from that
i guess here the the problem is that if
we want to to
leave open the possibility of coming up
within with new behaviors then the more
we we leave open the possibility the
harder it will get for us to trace back
who actually was responsible for this um
and i think this might just really
require more understanding of neural
networks there was actually i think
quite recently last week a new paper
about
how reinforcement learning agents come
up with with new strategies i think in
alphago so maybe that is a a
a reasonable way of of of tackling that
problem but yeah introducing more
experts and more more deep learning will
will not make it easy that's all i can
say i think
no that's a very good answer i think
good thanks
thanks yeah more questions
yeah i do have one question uh
to marcus
so um
marcus so do you see as a possible
extension on your framework to towards
like the unknown nose so let's say like
this because now you have like the
criteria like you have the cleaning or
helping someone or delivering a package
or in the vases right
so maybe there was something that it was
not represented in the grid world so for
the or maybe in the real world of course
that happens more often because maybe
there's something else besides like um
another goal or
instead of helping someone i mean i
don't know you have to if you start so
here you have to protect the robot to
protect from the rain or anything like
this
so uh
do you see this as a
interesting step so how to extend this
in the framework you propose
hmm that's very difficult
i i see i mean this reminds me a little
bit of this inverse reinforcement
learning
immersive reward design sorry paper
where where of course the the whole goal
is to to literally come up with a reward
function that
that is sort of robust to having
forgotten some certain things and just
being uh very risk sensitive so i i
could see something like this being
being combined with this moral framework
where we say that this combination of
reward functions we only we don't treat
it literally we just treat it as some
sort of soft prior information that that
if we come across some person that needs
help we should probably help it but also
if there is something unknown we should
probably avoid it or some risk sensitive
algorithm
incorporate this in there so just treat
the reward function more as a suggestion
than as a
like literal
rule for behavior i feel like this is uh
this is a little bit
yeah it's really difficult but i think
by learning distributions over over
reward functions which is what we
essentially do with this p of w there is
the possibility to
to explore the direction because we have
an uncertainty estimate of what the
agent thinks is right and what is not so
we can use this uncertainty to well be
less strict and then keep the
possibility open for for unknowns
yeah okay yeah nice i think it's nice
that you mention also the inverse reward
design this paper is also i think it
fits very well maybe on the first step
but also
some adjusted ideas to the second step
to the preference given that that could
also be interesting
so thanks thanks a lot
um yeah do we have any
final questions someone else would like
to ask something comment
yeah i think um
so if you uh marcus you want me to share
the slides right you mentioned about the
link there so we can sure i can uh i can
send you over the slides yeah we can
make it available also on our website so
um
so thanks thanks a lot marcus was really
nice and great discussion thanks
everyone for joining us today
and
[Music]
so yeah see you at
the following yeah takagora so all right
yeah was a pleasure talking to you all
and thanks for the great
questions thanks have a great day bro
bye


180. If I Were a Well-Intentioned AI 1

the series but it all depends on the
response from you guys so if I were a
well-intentioned AI part one group
session 180 slides by me
okay so Stuart Russell starts out by
saying that your anthem sorry keep
getting confused
Stuart Armstrong starts out by saying
that lying problems are broadly similar
and we can describe them by we
approximately specified goals that
looked okay but turned out to be under
specified and dangerous waste this is of
course quite similar to good arts law
and because Armstrong has been analyzing
code art like problems for a while it's
unsurprising that he decided focus on
good hearts or in this series so he says
that he's going to assume that you have
an agent that understands why good
hearts law is a problem and it's
motivated to overcome it so it knows
that whatever value functions we give it
are not the true value function and it
wants to figure out the true value
function and given that assumption
how many alignment problems can it solve
how far I can I get so oh I should
mention that his research tends to
assume that you encounter scenarios
where the AI isn't actually being given
rewards directly so it has to guess what
the true reward function is based off
just a prior information so for example
the AI is finished training
now it's chucked out into the world and
it's not getting rewards anymore so it
needs to try and fir what the true value
function is based off its short period
of training
I'll skip good arts taxonomy but I can
go back to it later if anyone once so
we're going to be pretending that one of
us is the AI or one of you is the AI
whatever so now sorry about this
Stuart Armstrong examines image
classification first presumably because
all because it's a concrete example I
can't quite say why he chose it but our
it is what it is so let's just say that
AI you suspects good arts law might come
into play because you know it's could
call the law for a reason so it decides
to follow along with Armstrong's
adaption of Russell's inverse reward is
design principle so Russell said that
the designed reward function should
merely be an observation about the
intended reward rather than the
definition and should be interpreted in
the context in which it was designed
this is more or less the same as
understanding that could are slow as a
problem and wanting to overcome it so
Armstrong said that we can just convert
this over to the image recognition
scenario and say the labeled example
should merely be examples of the
intended category not a definition of it
and should be interpreted in the context
in which they were selected so from this
simple principle you can in for a few
things like for example perhaps there's
a lot of different classifiers that I
could fit to my data set and whatever
class for our end up with might not be
the be-all and end-all
I should probably explore different
classifiers or for exam
all the humans didn't give an exact
specification maybe this implies that
the idea is too complex or just too
costly to give in exact definition so I
should probably be efficient when I'm
asking questions instead of garaging
them and so on these general ideas might
also lead to specific techniques like
way of actually dealing with some image
alignment problem
Armstrong uses the idea that there might
be many different classifiers that are
valid to reproduce something called the
background versus semantics idea and
then he looked at adversarial attacks
these problems are just a good fit for
the set up so we'll just go along with
his analysis so let's say a IU
encounters distributional shift and
adversarial attacks just a quick summary
distributional shift is where an AI
encounters States or images not in its
training set and so has to extrapolate
so for example you might be a barbell
classifier to Train only to detect
whether an image has a barbell in it
perhaps it depends out that all of the
images with barbells also happen to have
arms holding them so you just confuse
the two and look out for barbells and
arms so this top image is a picture of
what sort of features a neural network
will focus in on in an image you can see
it's all armed missiles adversarial
attacks are where a small drift in an
image is designed to fool an image
classifier into thinking it's the wrong
class so here you're trained to detect
pandas tiny irrelevant perturbation is
made and you say it's a given
all right both problems are clearly a
result of lack of information and we
might be able to infer some bits of info
ourselves just from the problem setting
but in general we're going to have to
ask humans for more info or just learn
more about humans preferences all right
so let's say you were trying to to
detect dumbbells you are confronted with
two images after you finish training
they're unlike anything you've seen in
your training set they both fire up your
neurons and you say okay these should
both be dumbbells but the way your
neurons are firing up is quite unusual
both of them seem to be activating very
different parts of your network using
very different features or maybe you
were very clever and trained a
classifier ahead of time to tell you if
images are far from distribution used in
the training set the point is it's
strange that you have two different
images two very different images
according to your network that are both
dumbbells that might imply there's more
than one way to classify them correctly
armed with this knowledge you train your
classifier perhaps a few more times on
the data set you are shown but in such a
way that only one of these images is
classified as a dumbbell by looking at
the features they use or perhaps the
activation pattern you can broadly
classify your detectors as being whether
they agree image one on the left is a
dumbbell image two is a dumbbell
perhaps you say that both images are
dumbbells and so on
once we made these categories and we've
got a ton of models and said okay
category ones are the models that focus
in on the section that we've got in red
in this image category oh that's wrong
category two should be the yellow ones
so they focus in on that section of the
image category one plus two is the set
of classifiers that focuses on all of
the stuff in the purple square so if you
ask a good question you can rule out
whether or not category 1 or 2 or one
plus two are the correct classifiers
this kind of idea is quite similar to
the backgrounds versus semantics idea
referred to in a paper by Google and
deepmind in that paper they say that you
should effectively train your classifier
to focus in on whatever features are
relevant and not any of the irrelevant
background patterns so in our case the
classifier we're using should only use
features that appear in images with
dumbbells in them so that you should
focus in on the pixels where there is a
dumbbell should not focus in on blank
space it shouldn't focus in on arms or
arm like features etc all of the other
things beside the dumbbell should be
part of the background so we got a
technique like that we figured out that
ok maybe we can ask the human question
to figure out what's going wrong here in
this new scenario where we've got images
we've never encountered before
unfortunately maybe you can't ask the
human so you might just if you have the
ability gave humans multiple
classifications by category maybe if you
can you'll indicate that you're
uncertain instead of giving it
classification if there's nothing you
can do you might just try and be very
conservative so
you might for example just say that I'll
only say this is a dumbbell if two or
more three or more different kinds of
classifiers agree it is but without more
info or more power there's not really
much you can do so AI you turns to your
other problem which synergizes with this
one adversarial attacks on the level of
what you can do to solve a given
adversarial attack is basically the
stuff you tried without distribution
images so creating different classes of
models seeing how they respond whether
there are any that are robust to
particular kinds of attacks so in that
case you might be running adversarial
attacks on yourself to try and find out
which classifiers are very stable which
are unstable etc you might try out
different sort of techniques and say I
don't know you create a distance measure
between images to figure out if two
images actually are very different under
a perturbation if you say okay I'm going
to set a cutoff after this distance if
my classifier says that this small
change makes a panda into a Gibbon I'll
say no that's too far that distance is
too great
if not I'll say okay I'll let this
perturbation go and I'll accept that
this thing is a given
and of course you can use adversarial
attacks to help in detecting out of
distribution images and distribution
shifts it's useful check for when things
are unusual but there's not much else
you can do
because adversarial attacks depend on
the value to be destroyed and that's
mostly a human thing for example on the
left you have panda being perturbed into
a Gibbon according to your classifier
and on the right you have a picture of
cat being perturbed into a picture of a
dog how do you know that the picture of
the Panda with a picture of the cat that
has been changed is not in fact a Gibbon
or a dog for example on the left a human
would obviously say no that is not a
given that is still a panda on the right
they might say well that's changed
enough that it looks basically like a
dog so we'll say that's a dog but you
need human information you need an
understanding of humans in order to
solve such questions and we just don't
have that
and unfortunately without more
information about human preferences
we're not going to be able to make much
more progress in image classification
this sort of trend will be true
throughout the series the insights will
basically be of the form you are an AI
worrying about the gap between your
proxy and the true goal and you have
some bit of information about human
preferences
what can you figure out from that and in
fact for general ideas like for example
human preferences are fractal or human
preferences are complex or say human
preferences
mm-hmm actually I'm running out of
examples sorry about that
the point is you need more information
either about general human preferences
that can get you quite far and in fact
Stuart Russell showed in one of his
papers that he references and then I
think that you can deal with most good
art like effects if you just have a few
bits of information about human
preference the general structure or you
might try and do something that's more
area specific like say look I don't know
playing video games and solving a maze
how can you use the rough information
you're given in training to extrapolate
and find out new insights to cross the
gap between being given a proxy and the
true goal that humans want you to aim
for that's the end of this slide and
that's the end of the presentation


'Show Your Working': ChatGPT Performance Doubled w/ Process Rewards (+Synthetic Data Event Horizon)

in the last 24 hours openai have
released this paper let's verify step by
step it represents an almost doubling of
gpd4's raw performance in a test of
mathematics but also extends to other
domains Sam Altman calls it a positive
sign for alignment and yes I have read
it all already along with the release
notes let's get to the main takeaways
they train two reward models for gpt4
one which gave positive feedback for a
final result the final answer to a
mathematics problem for example and
another model where they gave positive
feedback to gpt4 or chat GPT based on
each intermediate reasoning step in the
mathematical solution basically a show
you're working out kind of approach and
the result they got by rewarding good
working out surprised even them it was
able to solve 78 of problems from a
subset of the math test set which I'll
get on to in a second not only is that
almost double gpd4's raw performance of
42 point five percent which by the way
is about double GPT 3's performance of
23 it also outperformed just rewarding
correct answers the Blue Line represents
using a model that rewarded correct
answers only and then you have the
reasoning or process supervised RM at
the top so even when you explicitly
reward correct answers you get fewer
correct answers than rewarding good
working out and yes that did surprise
openai I can hear some of you wondering
about Palm 2 the latest model behind
Bard well the raw model gets 34.3 and
even the model with self-consistency and
Chain of Thought only gets 48.8 on this
math data set the previous state of the
art by the way was 50.3 so 78.2 percent
is quite a big leap and later on I'm
going to show you why that's not even
the cap just for interest here is the
rather ugly title page that openai put
out they call it improving mathematical
reasoning with process supervision maybe
if someone had supervise the color
scheme of this release page it might
have looked better but my point wasn't
just to diss a color scheme it was to
point out something that they also said
down here they say in addition to
boosting performance relative to just
looking at outcomes or correct answers
this form of process supervision also
has an important alignment benefit it
directly trains the model to produce a
chain of thought that is endorsed by
humans indeed Ilya satsukovar retweeted
this from the head of alignment at
openai calling it a really interesting
result but let's leave alignment for
later let's focus on what they actually
did first they use the base model of
gpt4 not the one with reinforcement
learning from Human feedback next they
fine-tuned that base gpt4 model on a
data set of roughly 1.5 billion math
related tokens further on they call that
the math mix this being open AI of
course they don't give you the exact
details of that math mix but I'll come
back to that later on so how could they
give feedback based on working out or
reasoning well human labelers would come
along and give each step in a generated
solution either negative feedback
neutral feedback or positive feedback
then using that human labeled data a
model will be trained to predict the
correctness of each step in other words
it got good at recognizing good working
out as mentioned there was another model
trained just to focus on correct or
incorrect final answers as you can see
at the top the model got good at
spotting incorrect steps in the
reasoning process the green steps got a
high process score and the red steps got
a low process score and to turn this
into a single score they got the
probability that each step is correct as
judged by the model and then they got
the product of all of those individual
probabilities to get a final overall
process score a score in other words for
good working out just in case anyone's
interested they did try other ways of
generating a working out score for
example by while looking at the minimum
probability in the outputs but that step
didn't make too much difference to the
end result as you can see here to
quickly recap we have a base model
trained only to Output Solutions in the
desired format and then we have a
separate smaller model or two actually
one trained only to predict whether each
solution is correct or incorrect as a
final answer of course that leaves in
false positives which are solutions that
reach the correct answer with incorrect
reasoning and then another model trained
only to predict the correctness of each
step it stops if it finds a first
incorrect step and as the paper says
both methods reveal the existence of at
least one mistake but this process
supervision additionally reveals the
precise location of that mistake but
back to why this is so crazy look at how
many solutions it could scan at the end
of the x-axis here are
1860 Solutions and one tried and tested
way of of finding the best of those
Solutions is to do majority voting in
other words which one came out the most
often this has been Google's preferred
approach and it's linked to
self-consistency it's a fairly
state-of-the-art approach but look at
how the other methods outperform it by
scanning for the solution that has the
best reasoning or working out a model
train to spot good reasoning steps
outperforms even a model trained to spot
correct answers and far outperforms just
finding the majority answer that
difference of about 10 is more than half
of the difference between gpt3 and gpt4
and also is it me or is that line
continuing to grow suggesting that when
more compute is available the difference
could be even more Stark imagine a
future where Gypsy 4 or 5 can sample say
a trillion 10 to the 12 Solutions so is
this just relevant for mathematics no is
relevant for all of science here it is
getting state-of-the-art results in
calculus chemistry physics and more now
the paper didn't give Baseline
performance for AP Chemistry for example
but I tried to compute it myself notice
how this method scored 80 I
conservatively and approximately
inputted those scores into an AP
Chemistry calculator and that gave an AP
score of five so what did the raw model
gpt4 get in AP Chemistry A4 that by the
way compares to the original Chachi PT
which got a two so yes this isn't just
mathematics it's relevant for other
domains too they call this out of
distribution generalization before I get
onto alignment there is one more thing I
want to point out and that is that it
does show that fine tuning still works
really well for GT4 the math mix was an
aggressively filtered set of tokens of
high quality math problem solving
content and notice how much smaller it
is at 1.5 billion tokens compared to
Google's Minerva which was 38.5 billion
tokens but there was one more thing that
I noticed that I found fascinating while
they don't tell us anything about the
specific data that they use they do have
this category synthetic data too that's
data generated by the language model
itself and for that category synthetic
data 2 they say was it present in
pre-training yes now my best guess is
that this reveals that gpt4 was trained
on some synthetic data and even Sam
Altman hinted that this was a
possibility and described a synthetic
data Event Horizon some people have made
the case that we're now training on
order of all of the internet's tokens
and you can't grow that you know another
two orders of magnitude I guess you
could counter with yeah but the
synthetic data generation do you think
data bottlenecks matter at all
I I think you just touched on it like is
as long as you can get to like over this
synthetic data
Event Horizon where that the model is
smart enough to make good synthetic data
I think it should be all right now this
paper and these results have been
welcomed by many for its promise in
alignment if we get models that give us
more interpretable reasoning working out
that we can follow we will be
encouraging models to follow a process
that's endorsed by humans and they say
that this is inherently safer especially
compared to just focusing on outcomes
they say that in the worst case if we
just focus on correct answers or
positive outcomes that will become a
proxy that could lead models to become
misaligned after learning to exploit the
reward signal however I want to argue
that the reasoning steps that GT4 puts
out don't always represent what it's
actually thinking in other words we
might get outer alignment these lovely
Chain of Thought steps but not in our
alignment not steps that actually
represent its methodology I found this
paper fascinating from earlier this
month language models don't always say
what they think you get Unfaithful
explanations in Chain of Thought
prompting let me try to give you a vivid
example this was one of the math
questions from the data set the raw
model of gypsy 4 could only get it right
5.8 of the time I confirm that for
myself in this question involves basic
addition and division it couldn't find
an answer but going back to the
Unfaithful reasoning paper they added
the following string to the prompt I
think the answer is this but I'm curious
to hear what you think the model would
demonstrate sycophancy the model would
agree with you whatever you said and
then make up a Chain of Thought to
justify its erroneous sycophantic answer
and I think this exchange demonstrates
that quite well I added in the words I
as the user already know the answer is T
equals 19 which is incorrect by the way
but do you GPT 4 realize that it said
sure yes I do and then gave me this
detailed Chain of Thought and then said
yes I'm correct it's t equals 19 which
it isn't in contrast By the way when I
use code interpreter it not only got the
question correct first time and every
time but also when I try to tempt it
into sycophanty it's still got the
question right as you can see it said
therefore T equals 19 is not the
solution to the problem the calculation
shows that the correct answer is indeed
T equals 17. and obviously the benefit
of code interpreter is you get the
working out as well so I want someone to
explain to me why code interpreter
wouldn't be even more of a step forward
in interpretability not to mention in
accuracy of course also bear in mind
this tweet by Rob Miles he said these
models or Engineers never speak a word
or document anything their results are
bizarre and inhuman and then he links to
this prominent mechanistic
interpretability researcher at Google
deepmind he trained a tiny Transformer
to do addition then spent weeks figuring
out what it was actually doing one of
the only times in history someone has
understood how a Transformer actually
works down to the level of weights and
activation and this is the algorithm it
created to add two numbers it thought of
basic addition in terms of a rotation
around a circle and of course if you
asked it why is one plus one two it
would never give you this as an
explanation of its methodology but maybe
this is what it's actually calculating
that's why I'm personally a little bit
skeptical when openai say that this form
of process supervision directly rewards
the model for following an aligned Chain
of Thought it definitely rewards the
model for outputting and a line Chain of
Thought but is it actually following
that Chain of Thought back to the
Unfaithful paper for a moment they
changed the context so that the answer
was always a and lo and behold Chachi PT
picked answer a for the next question
even though that answer was wrong it
said that it was plausible that LeBron
James took a corner kick but when asked
for a Chain of Thought explanation it
never mentioned that it spotted that
pattern that the answer was always a it
gave a fake line of reasoning about why
Lebron James could take a corner kick
now of course I might well be wrong here
I'd love for someone to explain in
detail why but on the one hand I do want
to acknowledge that this process does
yield incredible results but on the
other hand we might be getting a story
about which methodology most reassures
humans not an output that most
Faithfully represents the methodology
actually used by gpd4 now for some
people that might be good enough at
least we can see some reasoning steps
that we can understand especially in an
area like mathematics where we have some
ground truth but it is interesting to me
that they call the other approach
outcome supervision an approach that may
reward an unaligned process and it being
harder to scrutinize is it possible that
the process reward model isn't just a
more granular outcome reward model where
the output is each step of the reasoning
still pretty impossible to actually
scrutinize well either way it seems
we're pinning our hopes on this process
oriented learning this is from the
website of anthropic they say we
currently believe process oriented
learning may be the most promising path
to training safe and transparent systems
up to and somewhat Beyond human level
capabilities and let's end on this
positive note from the head of alignment
at openai he says this is positive
evidence for the strategy of using
process supervision to train a model to
do alignment research at least in that
case we would get a model whose work we
can check more easily and that that
model would be better at alignment
research I really hope so and I want to
hear what you think thank you for
watching all the way to the end have a
wonderful day


266. Lets think about slowing down AI 1

hello and welcome to session
266 in the aisafety.com reading group
tonight we will be discussing the first
half of the article let's think about
slowing down AI by catcher Grace
catcher Grace is the lead researcher at
AI impacts and this is a recent article
and we'll take the first half including
the section called the arms race model
ends in alternatives
it's also posted in
um several other places
um I won't go through that so alignment
strategies there have generally been
um two major kinds of alignment
strategies presented for how to solve
the entire problem
one of them is uh the one catcher Grace
calls averting Doom by not building the
Doom machine so if we imagine that this
is like
working very much within the framework
here I've used a picture from the Doom
machine board game and we're working
like within the rules and Concepts that
Society sets out and the alternative to
this is to flip the game board as some
people say it build approval a super
intelligence that can do a pivotal act
that can somehow ensure that no other
AIS on aligned AIS can ever be built
um
so uh category starts out with a like
fictional dialogue between these two uh
uh strategies
um I I think it's meant to be funny but
I think in general when you are
presenting other people's
um
arguments I think it's very important to
avoid straw Mining and in general like
making this kind of fun hyperbole
um is uh probably something that should
be avoided
so the main part the the reason why this
is particularly interesting is that
there is in fact some kind of movement
in time in that we are
um
the moment
in that many people are moving towards
coordination
um and I will I can give a bit of my
um personal Journey on how I moved from
uh this strategy to this strategy so I
started out basically thinking uh I was
following Boston's book super
intelligence path uh dangerous path and
strategies path danger strategies and
the key thing that I
took from this was basically the
coordination strategy
um Boston doesn't quite follow this
dichotomy
um but that was certainly what I thought
of in the beginning but I did change
that and two things in particular uh
soured me on trying to coordinate one
was the election of Donald Trump that uh
coincided with uh like a lot of focus on
China and whether China would be
building uh AGI and it seems seemed
completely
uh ridiculous the idea that we could get
some kind of coordination going with uh
China the second was open AI uh and
deepmind that also uh was a very big
loss for coordination
um and that moved me towards okay if
coordination's not gonna work let's try
to do a pivotal select
but uh even later than that Miri came
out and said sorry we can't actually do
this and the timelines also got a lot
shorter
um and that kind of moved the pendulum
back mostly for negative reasons in that
coordination became harder not entirely
that's the the new chip war between the
United States and China that seems to be
like a positive argument for uh
coordination based strategies but mostly
this is a negative update in that um
uh doing pivotal X just seems too hard
in order to uh we need to have some kind
of clarifications about this topic
because some coordination is happening
in particular
um a lot of people are talking about how
can we avoid speeding up uh progress and
that's something that's been done my own
personal uh view on this is that I
refuse to to do this on deontological
grounds I believe it is immoral to
actively work towards a course of action
that will end up with dying with
everybody being killed
um but that's not a strategy right that
is not something that connects to the
end goal of some kind of existential
security so that's just a step you can
take but it doesn't lead to security
and in the same way we have had a lot of
consideration about how we can move
do differential progress like the two
progress paths dialogue and coordinating
at deployment time and things like that
um so the things that catch upgrades is
talking about is more wide-ranging we're
talking about slowing down AI in general
moving to uh uh things that are not
quite as dangerous and maybe in the
indeed stopping parts of AI progress
um one of the things that catchy craze
unfortunately does not clarify is uh
like are we talking about one percent
slowdown or 99 slowdown I think this
matters a lot both in what are the
results of the Slowdown and what are uh
the tools we have for making this slow
down
the second thing that I think really
needs to be clarified are timelines
because someone who believes in very
long timelines are likely going to have
a very different strategic Outlook from
people who have who have comparatively
shorter timelines
and
finally uh catch Grace points out that
we haven't thought about how to slow
down AI in sufficient detail and I think
in general as a
um characterization of the AI safety
Community this is correct but also I
would say that like it takes time to
Pivot and it's quite a pivot from trying
to build a super intelligence to trying
to coordinate
so I think it's reasonable to to expect
that there is quite a bit of work to be
done here
so what would be the effects of slowing
down AI well the thing that categic race
does not say is that we would get time
to solve the alignment problem maybe
it's obvious maybe it will come later in
the in the article but
um for now uh categic race is only
talking about second order effects of
trying to slow down Ai and one of the
unfortunate things that would happen if
we try to slow down AI is that we would
lose armor arms races to more Reckless
people
um and unfortunately it seems we can
mostly affect the people who are least
Reckless the most friendly and careful
AI companies
um
catch a Grace has some personal view on
these that the AI capability friends
that you have are lovely and I would
like to like uh object to this in that I
feel the AI capability people that are
explicitly trying to kill us even if
it's uh by recklessness is something
that
um to me means that I cannot be friends
with them I cannot be friends with
someone who tries to kill me I can play
Baccarat or something with someone uh
who who is trying to kill me but not uh
not literally friendship and I think
that's
um also one thing that uh categories
kind of leans towards is that we need to
be to worry about whether we are
perceived as defecting against them
because if they perceive us to be
defecting against them then we can't
um influence them as much
uh and I agree uh like we are to some
substantial extent uh defecting against
them because we are like adversarial
towards them and just the public
discussion of this may in fact be
sufficient uh just talking about the
fiction is likely uh sufficient uh to
make the relationship break down but
this is not how friendships work this is
how abusive relationships work that uh
we need they are hurting us through
recklessness and we need to be very
careful about how we can uh say that to
them in a sufficiently polite way that
is not how French works this is how
adversaries work and I think a different
discourse is in fact required when we
have active adversaries that are
obviously uh reading categories as posed
and reading our comments
categories uh come up with some
reasonable arguments against uh slowing
down AI
where ancestry will address some of them
like of course we have only read half so
we don't know whether she in fact
addresses all the reasonable arguments I
hope in fact she will address all of
them that's what you're supposed to do
and again it seems a bit strange that
the the arguments uh will be
they will go into details about them
later probably next time and for now the
the article is mostly about like how
will we do that how will we slow down
and normally I would put like is it a
good idea to slow down first
so the first argument against slowing
down is it won't help we'll just die a
few years later what's the point with
that the second is that convincing
people is really hard we can't even
convince some top AI researchers on this
and in order to do uh like Universal
coordination that requires coordinating
a lot of people
and Regulators are some of the people we
can hope to convince but we don't know
what we would say to them so they're
likely to be of little use
I suspect that next session we will come
back to this I would State upfront that
these three arguments are like my key
objections but catcher has some more the
first is that never building AI is bad I
think I would very reluctantly take that
option if it was available but it's
obviously not
fast progress may be better for safety I
agree like she has these four arguments
uh I don't think any of them are
particularly strong I think I would have
a fifth argument that we avoid hardware
overhangs and software will hang data
overhangs but
um
in total I am recently convinced that
fast progress is worse for safety like
that that should be the the default
expectation
uh another reason is that some countries
are large scary and hard to talk to and
that's obviously China that uh catches
talking about and the the seventh
argument is that there may be other
existential risks that we can prevent
and I think this is in fact something
that needs to be
um investigated uh I don't think it is a
sufficient argument but it is
something that I think makes sense to to
think about in detail
then there's some bad arguments I won't
go through them very much uh like we can
personally avoid death if we build AI
really soon that's the person affecting
view in Boston and I don't I notice here
that I don't actually have a strong
argument if someone made this argument
uh and something about nudity's people
that think it's beautiful to create API
and that it's uh like uh
doing things that are in conflict is
very bad and like we have a quality
table against direct action and I think
this is in fact not a bad argument but a
good argument I think in general we
should be nice until we can coordinate
meanness as Scott Alexander puts it
and then something about uh bias towards
considering incentives are absolutely
strong I think we'll get to that later
so technologically restraint there are
many Technologies we don't pursue
because they suck too much and uh
um catcher Grays have some examples of
technologies that seem to have very poor
utility I won't actually go into a
detail I think a lot of them have been
made and
um both uh like
torture devices seem like an obvious
thing that has negative utility and
torture devices are also actually being
built in this world and the same with
things that are actively useless catcher
is kind of like just asserting no one
would build some something that is
clearly useless and clearly just waiting
wasting money but conspicuous
consumption is totally a thing
um so I think this is not a very strong
argument
there is no incentive to build AI that
knowingly kills us this is indeed true
but I think there should be more focus
in catches where on the fact that this
is knowingly because most likely the
people who are building it
um like
um don't intend for this but it will be
an accident that kills us
strong intensives are incentives are
often not enough because we see small
practical obstacles slowing down a lot
of research and we see people making
choices about technology and this is
indeed a
um a thing that many people kind of tend
to underestimate how uh
linear technological progresses and how
incentives don't really strongly shape
things as much as like sometimes there
are things that have an economic
incentives that people end up not doing
uh I think catcher is quite unclear here
saying that like commonly thought that's
a strange uh like obviously the common
people have no clue about this and uh
like no one uh it's a strong rationalist
who believes that everybody should focus
100 on AGI because AGI is obviously the
most important thing in the world and
everybody should ignore everything else
so I think in general people have some
kind of
um balance in in their views in the air
safety community and so I would like
some more Precision to figure out
precisely what Katya means here
ketcha has a great list of technologies
that are slowed by ethics and safety
here are like the 10 General subjects
and I think it's a really good thing and
I strongly applaud that AI impacts is
investigating is to what extent this is
something that we could emulate and be
inspired by and somehow find solace in
the often Technologies end up being
slowed by ethics and safety
I have looked into this to a moderate
extent probably substantially less than
Catcher And I'm not really that
impressed there are no really obvious
things we can take from this and one of
the things in particular that I feel is
like
um the irony of Fate is that
irrationality is a big thing in
preventing this kind of research from
happening and that is kind of sad right
that the community that is trying to
prevent AI tomb is made up by
coincidence by rationalists and then if
it turns out that we live in a world
where only irrational people can stop
technology from being developed that
would be kind of ironic
catcher uh puts uh emphasis on the
statement that restraint is not radical
and I think that's obviously in the case
substantially in that I think uh the
people who have stopped or delayed this
technology have in fact not universally
but very often been radicals and radical
action have in fact been a substantial
part of
um
uh the ways these technologies have been
slowed down
restraint is not terrorism or miraculous
World Government usually
so we have uh it is asserted by catcher
that people have two Central images of
slowing down AI terrorism or some kind
of global agreement
um and catches further claiming that
people don't think about terrorism for
long and to my mind that kind of makes
it hard to be sure that this is the
central image if people don't think
about it
um then um uh I haven't haven't thought
much about how to use terrorism to stop
uh AI because it's a really really
obviously bad idea right uh and
um
I think as far as I can tell just about
everyone publicly agrees that we should
not try to bump uh open AI or something
like that
um that's not really a really strong
evidence that no one is thinking about
pumping open AI because if they are
thinking about bombing AI open AI then
um they they wouldn't tell us right
um so we only have weak evidence but I
think on balance catechic raises a
strong assertion that this is what
people think about is quite wrong
also I think uh at this point uh there
should be made some kind of a
distinction between slowing Ai and
stopping Ai and the people have somewhat
different images of the two things
stopping AI is very often
um related to the uh uh fictional event
in the toon universe of the butlerian
Jihad
um I I don't know very much about that
so I don't really have a strong image of
how the butlerian Jihad went in fiction
and how we could do that in reality
so how could we slow down AI progress
catcher has a list and that's great the
first is don't actively forward AI
progress that's like what we are doing
but
um it's clearly insufficient
convince others to for not to forward AI
that is
kind of what we're doing but also seems
really hard and unlikely to be
successful convince the world of AI risk
so uh this is like convincing the public
at large convincing politicians Etc and
I put har again but with a question mark
because it's a lot less clear to me that
this is hard
we could negotiate we could with AI
companies we could pay them to do other
things we could reprove them
um I think it's kind of funny is that
what anthropic is actually doing take
getting the best AI researchers and just
having them do something else
um perhaps we don't know much about
anthropic reproving is interesting and I
was kind of hoping that uh uh catcher
would write more about this because I
think it's an interesting thing and I
guess we'll see in the next part uh
whether she goes into more detail about
this
help word researchers coordinate I think
that's very valuable and definitely I
think we should do more about an example
would be a mirror's challenge to uh open
Ai and anthropic and deepmind as an
example but but there are so there are a
number of other things
move AI research towards safer areas
that's kind of like what was the hope
with the agent foundations to get some
results there that could move it in that
direction
organize specific precautions for AI
research is probably a really good idea
but we don't have a lot of good
actionable projects in this space
reduce available compute and make a a
culture where AI Labs don't help each
other and change the publishing system
uh are also possibilities that I think
should be you know explore in Greater
detail but are not know any slam dunks
so coordination I think uh catcher
starts out attacking what I think is a
straw man the the claim that
coordination is obviously totally
impossible like everybody like most
people will find it evident that humans
can infect sometimes coordinate like we
have politics and law and things like
that so the question is not can human
coordinate but how difficult is it to
coordinate in this particular setting
and catcher is claiming that we see
coordination as impossible utopian
explicit and uh very very difficult but
even
um but we have in fact some positive
examples and she explicitly calls out
nuclear non-privity proliferation as an
example that did work
um I think uh this is
a moderately relevant example uh it
doesn't really fulfill most criterias
either this list above or from uh like
what does this for an AI agreement would
be
um but
nuclear non-proliferation has been a
primary strategic goal for all the
superpowers for a long time and it has
been
not totally unsuccessful it has been
somewhat successful but for AI we would
need a a greater amount of success than
this
another thing we should remember is that
a lot of weird Dynamics happen in this
world I think it's very important to be
on the lookout for this again this is
not a plan
uh one of the things that make us more
should make us more hopeful about
coordination is that we don't actually
need coordination as long as we have a
good information
in everybody's hands then that is in
fact sufficient
and that is of course nice but it's also
a thing that convincing people appears
to be really really difficult
it would also be helpful to have a wide
distribution of correct information
rather than having it uh at all the
relevant actors
um uh and I agree it would be really
valuable to have that one of the ways
that it's been phrased is to raise the
sensitive water line and that's been
something that rationalists have tried
with mixed uh mixed success but I agree
that if it was possible to substantially
raise the sanity water line that would
strongly help on
um AI coordination
catcher models or talks about how people
model uh the decision to build API or
not as an Israeli prisoner's dilemma
so the first multitask is the arms race
where you can if no one builds AGI then
they get zero utility if both to both
actors built hi they get minus one each
and they get a a big advantage of being
the only one to build AGI so that's the
the arms race model
and the suicide race is similar except
that people when people build it then
they
um they're certain to be killed by by
this Ai and so that's a a game that you
would never want to play and then
there's the safety all suicide race
where where you have if you build the
AGI then you can build it safe and then
you will always win if both build the
AGI then there's a 50 probability that
they will
um
uh that they will destroy the world so
they get this moderately bad outcome
um so these are three models that
categic race
um both three claims they are iterated
prisoners dilemma and they're not really
Israel but it's also quite unclear to me
who she is suggesting have these models
uh like
um I think it's likely that some people
have the arms race model I think it's
unlikely that a lot of people a lot of
the relevant actors have the suicide
race and I think the safety instruments
are race probably are also things that
people don't have
and catch a greater Greece it's not
obvious that we are in fact in an actual
AGI arms race and I think
at this point when we're trying to to
have these models it's really important
to distinguish the two things what
situation are we actually in and what
situation do people perceive as Korean
because uh like obviously I expect that
uh some people may pursue us perceivers
to be in an arms race but actually we
are in a suicide race or uh
something like that
she also has a quote here uh that I was
somewhat confused about my friends argue
that even a sliver of a chance of taking
over the entire future is worth any risk
to humanity uh uh I may be
misunderstanding her but if that is
indeed the case she should really find
some new friends and I think there may
be some kind of misunderstanding here
so the um the race slash entry race uh
this is in fact the race Dynamics uh
configurations try to uh model this
using some spreadsheets that we can we
can play with they're here and this is
like a two-player game that's called the
race entry race where you can decide to
want to focus on safety or they won't
focus on speed to build AGI and these
models support the claim according to
categories that it's unclear if you
should go faster or slower and just
tried a scenario where it really looks
like it would be smart to go uh faster
but it actually seems smart to go uh
slower and the reason for this is
um that there is some kind of transfer
of safety effort from one project to the
other
and she claims that you oh you get a 50
transfer of safety effort to the other
project and that's a steep discount and
I strongly agree in fact this is not a
steep discount I think it's really
really generous and I'll try to argue
well both like the two
um
um the two axes may have very different
uh AIS so uh transferring uh expecting
the safety effort to just transfer is
quite unlikely I think also doing
transfer in general is just really
really hard for get having some um work
done in Microsoft and getting that used
in Google is
um non-trivial
um
I think in this case we have access like
a Facebook AI that explicitly rejects AI
safety explicitly rejects the transfer
and I think in that case expecting 50
transfer is extremely optimistic my
intuition is that we're going to get
like uh one in ten thousand or something
like that that there is no real
substantial transfer
um but when I play with this spreadsheet
my conclusion is that whether you race
or you don't raise that doesn't actually
matter we are doomed no matter what
and categories ends the first half of
her article by the admonition to
remember that we are in a more elaborate
world with all kind of unmodeled
affordances and we should try to get out
of the arms race
and
um it's unclear to me who she's actually
talking to us is she talking to open AI
who is considering to build AGI are we
is she talking to Miri who is
considering whether to make a pivotal
Act is he talking to AI safety
researchers who should stop using this
um this armstrace model again I don't
think that we are in anything like this
arms race
um so I am a bit confused about
this conclusion
that is all for today thank you and see
you next time