DeepMind x UCL RL Lecture Series - MDPs and Dynamic Programming [3/13]

hi everyone
uh welcome to
uh our third lecture in the
reinforcement learning course i'm diana
i'm one of the other lecturers of this
course alongside with harlow and mateo
a bit about myself before we begin i'm a
research scientist at deepmind and been
there for almost four years now
uh prior to joining deepmind i was a phd
student at ucl and about 10 years ago i
was taking the machine learning
master
course at ucl including this particular
course i've been involved with the url
course for the last
five years
first as a ta and then as a lecturer
that's a bit myself uh now to the actual
content of today's lecture
today we're going to be talking about
markov decision processes which are a
way of formalizing the rel interaction
loop as well as the objectives that
arise in rl and the class of problems
that we can encounter like policy
evaluation and control
finally we're going to look at the full
class of iterative model to tackle these
these problems and these
are known as under the umbrella name of
dynamic
programming and we'll see uh in a minute
why that is the case
okay
uh first background reading
the topics that we're going to cover
today are also covered in the
saturn part book in chapter third and
fourth
so i would encourage you to
read those chapters alongside with this
lecture
a bit of recap from last week we've seen
that the reinforcement learning is the
science of learning to make decisions in
the environment
we also learned that the rl system
comprises of an agent in an environment
the agents can learn a policy to behave
in the environment value function or
model of the world
the general
interaction between the agent in the
environment usually unfolds in time and
the decisions that we're making have
long lasting consequences that would
affect the reward that we're going to
see in the future uh that can change the
state of the agent but also the change
can affect the the
a change in the environment state
okay uh so last lecture you've seen a
very small instantiation of this a
system with um
[Music]
an rl problem that only has one state
but multiple actions
in this lecture we're gonna be
formalizing the full rl problem with its
um
sequential nature
we're gonna talk about the first class
of solution methods
which assumes that we do have a true
model of the environment so everything
that we
need to know about this interaction
between the agent and the environment
to predict next in the next states of
the agent in the environment as well as
the reward associated with these
transitions are given to us
these methods are called dynamic
programming
and although in this
week both lectures will be targeting
mdps and dynamic programming so a full
knowledge of the the environment
in the next lecture we will see how
these ideas and this principle of
dynamic programming can be relaxed
uh
to the case where we instead of having
true access to the to the model we only
have access through sampling through
interactions with the world
okay
first
item on the agenda formalizing the rl
interaction
so we're going to discuss a mathematical
formulation of the agent environment
interaction this is called the markov
decision process and this enables us to
talk more clearly about the objective of
an rl agent and how to achieve it
first a simplifying assumption
for now we're going to assume that the
environment is fully observable that is
the current observation that the agent
sees contains all the information
relevant to make all predictions in the
in the future
note that almost all rl problems can be
formalized as
mdps so this is a very encompassing
paradigm
these are just a couple of examples
um
things like optimal control uh problems
in continuous mdps are still covered by
this formalism although in this
particular lecture and for most of the
exposition of theory we would be um
assuming
finite
and discrete action space and
state spaces but all of these can be
naturally translated to the continuous
case
the other example is partially
observable problems although this seems
to violate the this first assumption we
made we can show that
even these models can be converted into
an ndp given the right history
i mean even things that you've seen last
week which are banded
can be formalized as an mdp with to just
one state
without further ado this is the
definition of a markov decision process
as presented in the book so mark of
decision process is a tuple that
contains the set of all
states the state the set of all actions
this probability
kernel which describes the joint
probability
of what's going to happen in the next
stage so given that we are in state s
and have taken action a
what what am i going to observe at the
next state so what's my next state s
prime and what kind of reward signal
am i going to be rewarded for that
transition
and the last
element here is the discount factor
which trades off
later rewards that we'll see in the
future with earlier ones by discounting
them um
heavily or not
okay we're gonna
call p the dynamics of the problem or
the transition dynamics
and um it's good to notice that
sometimes it's useful to marginalize out
some of these quantities like the state
transition and the expected reward these
are the two uh
equations here
so this is just looking at the
transition to the next state and this is
looking at all of the
reward
that we could expect from a transition
starting from state
s and haven't taken action a
and this leads us to the alternative
definition of the mdp which is quite
prevalent in literature so we're
mentioning here because uh most likely
in a lot of the papers and the other
books like
chavos bodies book you will see a
similar definition
so the the only difference here is that
now the transition kernel is this
marginalized transitional kernel which
uh looks only at
what's the probability
over my next state given that i started
in
s and taken action a and r
as given in the definition of this mdp
is the expected reward associated with
that transition so although this
random variable
might
depend
on where we end up in the definition of
this this mdp we we assume that we have
access
to the expected version of this this
quantity
okay
and then the discount factor remains the
same uh note that this formulations are
equivalent there are no additional
assumption or removing any any
assumption and we will be using them
interchangeably depending on uh when
it's more convenient to use one notation
versus the other
okay
now
the markov in the markov decision
process comes from the markov property
and you might have seen this previously
in something like hidden markov models
or in a lot of the sampling methods
this
property really says that the future is
independent of the past given the person
and for
mdps
this is a property that all states will
have and what does it mean for state to
have the markov property well a state
s
has the markov property it if it
summarizes basically all of the history
that it has
all of this is history that has come
before it in terms of the prediction
value of the next estate or the next uh
states in the future so this is
a different way of saying this is that
the state captures all the information
all the relevant information for the
prediction problem
captured in the history or another way
of saying it is that once the state is
known the history can be thrown away
uh and a different way of saying is is
that the state is a sufficient
statistics of the past so
uh anything that has happened in this
interaction process is summarized in
terms of um
its information about the
about future transition is summarized in
the in the current state in the current
observation
okay
this is just a bit of a test to um
see if you understand the the principle
and the inter
what that means in an mdp these are a
couple of statements that i've written
down some of them are true some of them
are false
i would
encourage you to pause the video here
and go through
all of them and see which of them are
false and which of them are
are true
welcome back
hopefully you managed to go through this
exercise
um we're gonna just go very quickly
through uh the results
okay so the
principle here is that the state
at current time step contains all the
information necessary for all future
predictions
anything that has happened prior to
time t
is not relevant because all the
information is captured in
st
that means all the information
all including
80 minus
one is contained
in the state so anything
before that does not
matter
um
so that means that these two are
completely equivalent the same thing
here
these things have happened before sd so
because all the information is contained
in sd we can
we can safely remove them
now this one again all the information
here is contained in st
but
80
comes
at the same time or after
uh we observe state
sd so
this is additional information the
action that i'm taking at state s
conveys more information and will
determine or skew the distribution of
what i'm going to observe in the next
state so this one is false
and
this
this one the fourth one is the same as
one and two all the information here is
contained in this one and these are just
predictions for for the
future whether or not that estate or
reward functions or actions or anything
anything that pertains to the to the
future is uh
is fully conditioned on the
on the current state so this one is true
as well okay
so we've seen that we've argued that
mdps are a good way of formalizing
a narrow problem and let's look at an
example of doing such
so this is an example of a cleaning
robot this is from the
sutton barto book chapter 3.
this is a robot that its goal is to
clean soda cans
for simplicity it has only two states a
high battery state and a low battery
state
and it has three actions
one is weight for someone to put a can
into its um
its bin
the second action is search which means
that it will actively search for
soda cans in the environment
and the third action is recharge which
will enable it to get the battery from a
low state to a high state
and for
this particular example we're going to
assume that the reward associated with
with searching
is actually
higher than the
reward associated with waiting so if
it's actively going into the environment
and trying to search for soda cans it's
uh it's expected the
return which is the collected number of
cans might be higher or is higher than
um
just waiting for someone to put
cans into its container
and then uh we have some stochastic
dynamics
where with alpha probability we can uh
loop into a state of high
battery and with one minus alpha
probability we're going to transition
from a state of high battery into a
state of low battery at which point we
would need to to recharge
okay
this is just a
full uh table depiction of the of the
mdp and
uh we'll see in the first
definition of the mdp as a tuple of
state actions
joint transition probability and
discount
um the discount for now can be one
and the the states are just the ones
that we've discussed high and low the
action space is search weight and
recharge and then the
the joint transition kernel can be just
re
read off from from this table so for
starting in state
high battery and taking the action
search you have alpha probability of
ending up in a state of high battery
and achieving a reward for that
transition of our search okay so this is
this is pretty um trivial
right off okay
for the
for the other um
definition of the mvp
we have the same thing uh state action
those remain the same now the
probability uh kernel in this uh
this formulation is just this one
the transition to the next state and
then the reward
that we have in this definition is
actually a reward that is um
rsa
okay
and this one is not exactly uh read off
from uh from the table we would need to
make a bit more computation there
because the reward in this particular
example uh depends on where i'd ended up
on that transition and this reward that
we have in this definition only
condition on the state of action so
if you
if we would go for this definition um in
order to specify this mdp we would need
the we need to compute the expected uh
reward seen on this transition so for
instance for
these
initial conditions high battery and
search this would be uh with probability
alpha i'm gonna
end up in a state
of high battery and receive reward
our search and with one minus alpha
probability i'm gonna
transition to a state of low battery but
still receive the same reward so
because in this particular example um
regardless of where i'm i'm
transitioning i'm gonna have the same
reward the reward associated with this
transition is actually our search
that might not always be the case so for
instance for this particular example
when i start with the in a low battery
state and i
choose the action search with 1 minus
beta probability i'm going to have a
reward of minus 3 and with
beta probability i'm going to have this
our search reward so the probability
um the
reward associated with
low and search
would be
one minus
beta
plus
our search
so in order to
to get this definition of the ndp or
to go for this definition of the ndp you
would need to compute this this
quantities
okay
okay now the last thing i want to
mention on this example is just to show
you a graphical representation of the
mdp this is something that we would
normally use to depict an mdp
so we have the state a high and low
battery
as nodes of this
graph and then we have arrows that
code the transitions between these these
states and these are
uh annotated with the probability of
transitioning
from one state to to the other and the
reward associated with that that
transition
okay
now to formalizing the objective of an
rl agent
in order to do that
we're going to be introducing returns
so we've seen that in
enacting unit mdp
the interactions at every point in time
would result in immediate rewards and
these lead to returns
and returns have at least
three um different
returns that we're going to be looking
at are undiscounted returns which is the
sum of all the
rewards that you're going to see in the
future
the discounted return which uses this
discount factor that we've seen in the
definition of the mdp
to gradually discount the rewards that
you see in further steps in time and the
average reward formulation
that looks at the average reward
so depending on the mdp one of these
might be appropriate for instance the
undiscounted
return is usually used in episodic
finite horizon problems because if this
becomes uh
if it's this is undiscounted and becomes
an infinite sum this might be unbounded
the discounted return is uh the one that
we're going to be using the most
frequently
it and it is suitable for both finite
and infinite horizon problems and
average reward usually is used for
uh continuant infinite horizon problems
note that all of these uh gs are random
variables that depend on both the ndp so
the
stochastic transitions
in the probability kernel and the policy
that uh
that we're employing in order to see
these rewards
okay
so as i said uh our choice um here in
most of the uh
most popular choice of return in the
literature is discounted return
and
moving forward we're gonna be using this
for both finite and the infinite horizon
problems and this is just uh the same
discounted reward but for the infinite
horizon problem okay
and the
the discount which is a
number between 0 and 1
means that the marginal value of
receiving rewards after
k plus 1
time steps is actually modulated by
gamma to the k which can be a very small
factor remember this is a number that is
uh
less than one so immediate rewards
will have
a more high weight
on this return than delayed rewards
for instance if the gamma is close to
zero or if you put gamma to 0 in this
expression you would see that the return
at the
in this particular example
is just the immediate reward
for
values close to to zero this gamma to
the k decays very very rapidly to zero
so
we um
we call this uh
return myopic because it uh it only
considers or most of the the values it
considers
come from immediate rewards
to the opposite side if the if gamma is
close to one this is far sided so in the
in the limit there if gamma is equal to
one we've seen this is the undiscounted
return that weights equally all the
rewards that we would see in the future
okay why discount
there are two sides to this uh
to this problem
one is um
problem specification it might just be
that the
mdp are considering actually has this
property that immediate rewards are
actually more valuable think of it as
interest rates
and
it's also something that we see in
animal human behavior that there is a
preference for immediate reward um in a
sense it's
uh leveraging
the probability of actually making it uh
to 10 years time or a hundred years time
and seeing rewards uh in
in that far horizon
and then uh on the other side is the
solution side so sometimes even when
we're interested uh in a undiscounted
version or almost undiscounted version
is
mathematically convenient to go for the
discounted reward case and this also
avoids the infinite return
that we can get in cyclic markov models
so if uh
if the in if the
horizon is infinite the the undiscounted
return might not be bounded so
maximizing over it might be problematic
so the way to think about it uh right
now is that the reward and the discount
factor together
will uh determine the goal of the
uh the agent
and the the goal of the mdp
the goal of the agent is to find a
behavior policy that maximizes this
return and i
as i said before this
g is
a random variable that depends on the
mdp and the policy
so what we want is
a behavior policy that maximizes the
expected value of this return and a
policy is mapping that for every state
assigns for each action in the
probability set a probability of taking
that action
in that particular state and this is the
notation that uh that we're going to be
using for deterministic policies we
sometimes use this this other notation
at equals
pi st
which basically denotes just the action
that is taken deterministically by that
policy
and uh because we're gonna be looking at
expected values of the the return we're
gonna be introducing some quantities to
uh to help us refer to this expected
value so we'll be introducing the value
function
at a particular state s
which is now the expected return
that we're going to get starting from
state s and following policy pi
and similarly we're going to be
introducing the q value or the state
action
value
which is exactly the same expectation
but now we're starting in state s we're
taking action a
and after that
for all of the other steps following and
behaving
according to policy pi
and there's an intimate connection
between these two which is denote
noted here that the the value function
uh is the expected
um
state action value
under the the policy that we are
evaluating under pi
okay
now
um we're going to be introducing the
optimal value function and the optimal
value function is just the
the function
uh that for every state
maximizes over these
this quantities previously introduced
so all of these quantities the value
function introduced here v and q are
dependent on a policy that we're going
to follow after this state or after the
state and action
and the optimal policies are looking at
for every state it's looking at all of
the policies that we could employ in the
future and is
doing a maximization over all of these
policies and the same for the action
value function uh the same
interpretation we're at state uh s
taking action
a and after that we're looking at all
the possible policies that we might be
able to employ in the environment after
uh choosing action a
and maximizing over those
those policies
so the optimal value function specifies
the best possible performance in the mdp
and we're gonna
say that an ndp is solved when we can
provide the optimal value function and
we'll see that if we provide the optimal
value function we can trivially derive
an optimal behavior policy out of that
okay
so in terms of optimal policies uh it's
good to
to notice that
there's a partial ordering in the policy
space
that
and we're going to say that
a policy pi dominates a policy uh
pi prime if for every state in the
environment
the value associated with that state
is uh under policy a pi is greater than
the value
associated with the policy pi prime
and the theorem that we have in terms of
optimal policies is that there's always
this exist
optimal policy pi star this is how we're
going to be
denoted it that is better and equal
with respect to all the other policies
in the environment
so this is a policy that will dominate
for all states in the environment all
other policies
okay and for all of the policies
uh they they achieve the same value
function and the same action value
function so any policy that is optimal
will achieve the
optimal
value function
and will achieve the
the optimal state action value function
okay i'm gonna pause a bit here because
um that's a lot to take in
okay
now uh i've prefaced that if we um
know the optimal value function um in
particular if you know the the q optimal
value function we can derive an optimal
policy
which is just the greedy with respect to
that that policy so this um policy will
choose
with probability one uh the
the maximum action
according to q
q star and put the zero probability on
all other actions so
it's gonna look at the the values of q
star and it's gonna act greedily with
respect to that
and
um this will give us um
an optimal policy
uh
i'm not going to show this here but
we'll
we will see a proof of that uh moving
forward okay
a couple of observation here
there's always a deterministic optimal
policy in the mdp
and
the
the statement above
implies that once we have a q star value
we can
immediately have an optimal policy
in general there might be multiple
optimal policies
and some of them might be stochastic
okay
okay moving on
we're gonna be looking at the bellman
equations uh this is our very central
concept in all rel so this is um this is
something important okay and the bellman
equations actually describe the
recursive nature
uh and the structure that
that is present in the the value
function we've just defined so we're
gonna
start here with the definition of the
the value function uh for policy pi
which is the expected uh
return that we're gonna get
starting from state s and we're gonna
expand this recursively so this is the
reward that we're gonna get at the next
step plus
what's gonna happen in the future so
this is the
um return at the next time step this
expectation is over the whole process so
we're unfolding this in time
and we can rewrite this
as
the immediate expectation over the next
transition
and
the value function which is already an
expectation
of what's gonna happen after
uh t plus one
so you see we have um
v pi
at time t
on this side and the
v pi at time t plus one on this side
there's uh there's a recursive
nature of the the relationship
and this is just spelling out exactly
this this expectation so it's the
uh expectation of taking action a in
state uh s
this is this part
and then
uh what's gonna happen in the transition
so i state uh i started in a state
s
i turn i
have taken action a and i've
transitioned to
a state
s prime
and i've been rewarded
r
so this is the transition kernel and
we're going to wait all this this
potential transitions to r and s prime
by their uh return
okay
so the same thing can be done for the
q value
and uh
the derivation is is just here so we we
do the same thing we expand for the
first step
and then
this we've seen
in the previous step that
this is the value this is just the value
function associated with
sd plus one
and uh
this
is
an expectation
this itself it's an expectation over
this q uh value function
when
80 plus 1 is chosen according to
policy pi and this is just spelling out
that that expectation so this is the v
that we've seen before
and this is spelling out to just uh just
how these two um states uh and actions
relate to each other and of course
there's um
there's a there's an intimate connection
between these these two value functions
that we're using
for this this particular step that
the
um
v is expectation over the q
values
okay
so these recursive relationships are are
very important and are known under the
name of belmont equations and particular
these are known as the belmont
expectation equation because it's an
expectation over the policy that we're
employing
now
um
we also have another set of equations
which are bellman optimality equations
which describe
the
um
would describe the optimal value
functions so
we
we can say that the optimal value
functions satisfy this this equation
this is very similar with the um
the ones that we've seen in terms of
expectation but in in
instead of having an expectation over
the next action here
and here
there's a maximization over actions
and this is because of the optimality
and
let me give you some intuition
why these uh the va the optimal value
functions um might
satisfy this equation okay let's uh
let's remind ourselves that the greedy
policy with respect to the optimal value
function so
uh v
star or q star
is an optimal uh policy i haven't proven
this but uh we've stated this before
so
um this greedy policy
that takes the max over these values
gives us an optimal uh an optimal policy
now if you apply the belmont expectation
equations
we know that
the
um
value function
the q value function corresponding to
this q pi which is an optimal policy
adheres to this equation
and this
part here
the expectation or under
uh this this policy
actually comes down to being the max
over
the the optimal uh value function
okay
so the this first part is just rewriting
the belmont expectation equation for pi
star
and this is just taking uh into account
the definition of pi star as the max
over q pi
okay
which means that
we get this expression which is exactly
what the
optimal
belmont equation says
and the same thing can be done for v
okay
cool
now um
let's look at
solving problems in rl using the bellman
equations
okay
first we're gonna be introducing the two
class problems that we encounter in rl
uh first class of problems are uh called
policy evaluation or prediction which is
uh estimating the values of p pi or q pi
so this is answering the question given
a behavior policy given this pie what is
my expected return under this behavior
so if i were to employ this policy
what am i expected to to get so given
this treatment protocol or this trading
strategy what is my expected return
under my
reverse specification
now the the other class of problems
are control problems where we're trying
to estimate the optimal value function
either v or q
and this leads to
an
optimal way of behaving so the the
purpose for this class of problem is to
find an optimal way of behaving or to
find the optimal policy
so things like what is the optimal
treatment protocol what is the optimal
uh control policy to minimize
time on the road or pure consumption
whatever the reward is so one is about
given a behavior i'm gonna
try to estimate um
how good is that behavior on average the
other problem is much harder problem
where we're trying to go for the optimal
value functions in the optimal behavior
okay
so we're going to look uh here at a very
small example so we're going to consider
this mdp
this graphical representation three
states
we have two actions
um a1 which is
a kind of
right
but once you are at the end of the the
uh m
of the state space you transition back
to the to the starting state and action
two which is roughly left but again when
you're at the end of the
chain you transition back to s0
all of these
actions
if they are taken
have a probability of 0.9 to succeed so
if i'm taking action one in state one
i'll have a 0.1 probability of
transitioning to s0
but i would have a 0.1 probability of
just looping back and staying in that in
that state okay for all prop
for all transitions that um end up in
state zero so at the beginning
so this one
this one and the transition to itself uh
the reward is zero for all other
transitions the reward is minus one
and now we're gonna be looking you're
gonna be looking actually at
an evaluation problem
for this particular ndp consider the
discount factor 0.9 and
your task is to find the value function
v pi associated with a policy that
always in this mdp takes action one okay
and
the same question for
now a uniformly random policy
and the last question here is
doing
both of these evaluation problems but
now with a discount factor of uh
zero rather than the original discount
factor of 0.9 okay
so i
would encourage you to pause the the
video here to to work out this example
before we move
we move on
okay hopefully you've managed to go
through the example
um i think this uh this is a nice way of
testing your understanding
of
both defining an mdp based on the
graphical representation then
formalizing the the returns and
computing the value functions um
so
combines all of the concepts that we've
uh we've covered so far and hopefully it
gives you a taste of the complexity of
the problem
uh of
estimating these quantities even for the
easier problem of policy evaluation
okay i'm going to give you a solution
rather than a solution um
a way of reaching a solution or a way of
tackling a policy evaluation
for these small mdps
and this will
be done by uh
rewriting the bellman equations into
matrix form so for every state in the
the environment we have a bell an
equation uh associated with that state
and that
transition to the to the next state and
for each um state and action pair we
would have the same
um
for the the q values uh for now we're
gonna just um
focus on computing the the value
function uh for a policy pi
and uh for that we're gonna just
collect all the belmont equation for all
the states in the environment into uh
into a system of equations that can be
rewritten in this matrix form
so this leads to this matrix bellman
equation for the expected
return under policy pi
and this these are now vectors
v and r
and this matrix
is the probability transition from state
i to state j under
the transition kernel of the mdp and
under policy pi
so just to write this out this is the
probability of starting in state s i
transit taking action a and
transitioning to a state
sj this is what
we have from the definition of the mdp
and this is weighted by the probability
of taking this action a
in state s i
and so this is the expected
probability of transitioning in this
state starting from s
i
via acting according to policy pie
and the same thing for the
expected reward under policy pie this is
the expected reward instantaneous reward
that we're gonna get
considering we're starting in
state as
si
and we're gonna behave according to a
an action chosen from policy pi
okay
and once we have this uh this
formulation in matrix form uh this is
just a linear uh equation so we can
solve for uh for v
and uh you can see that this involves
inverting this matrix that is one minus
lambda
uh p pi which is this probability matrix
of transitioning from one state to the
other under policy pi
okay
so that's it
um note that the computational
complexity of this is a number of states
the power of
three um even if there are uh other
solutions that are not cubic in the uh
state space uh
it's really
usually
more than uh
to the power two of the of the state
space so this this is a feasible
solution only for small problems
fortunately for larger problems larger
state spaces we have iterative methods
like dynamic programming which we're
going to see next
or monte carlo evaluations
and the temporal difference learning
methods
okay
now um
this
um
is a solution for
the
for the evaluation problem um
you could try to employ the same
principle for the
bellman optimality equation in order to
get the optimal value function but note
that the optimality equation is
non-linear so you won't get a linear
system of equations so this this kind of
reasoning does not
trivially go through um
for for the optimality case
so we can't use in general this matrix
formulation to do
um control
okay but nevertheless there are many
iterative solutions
uh the ones that use uh
models like dynamic programming which
we're gonna see next
and the
algorithms there are policy iteration
and uh value iteration
and the
other iterative methods that do not
assume knowledge of the the model and
are using samples like monte carlo
methods q learning and sarsa which are
temporal difference
learning methods
that we're gonna be transitioning into
dynamic programming
dynamic programming refers to a class of
methods that try to learn the optimal
behavior of the optimal value function
given a true model of the world the true
model of the mdp
the term was actually coined by richard
bellman to describe his research and his
reasoning is as follows
the
1950s were not
good years for mathematical research i
felt i had to shield the air force from
the fact that i was doing mathematics
what title what name could i choose i
was interested in planning in decision
making in thinking but planning is not a
good word for various reasons i decided
to use the word programming
i wanted to get across the idea that
this is dynamic that this is time
varying i thought let's kill two birds
with one stone let's take a word that
has a precise meaning like dynamic
in the classical physical sense
it it is also impossible to use the word
dynamic in the pejorative
sense try
thinking of a combination that would
make it possible
to give it a pejorative meaning it's
impossible
thus i thought dynamic programming was a
good name it was something that not even
a congressman could object to so i used
it as an umbrella for all of my
activities
okay
in
nowadays
and the meaning of the word that we were
going to refer to
the dynamic programming term refers to a
collection of algorithms that can be
used to compute optimal policies or
optimal value function given a perfect
model of the environment as presented uh
in the markup decision process
uh definition
and we will discuss here several dynamic
uh programming methods to solve mdps
uh all of these methods
actually have a pattern
which
involves two steps
one is policy evaluation and the other
one is policy improvement
let's start with policy evaluation so
we've seen already the uh a solution to
a policy evaluation problem
the
um
the exercise you you solved before and
the the matrix manipulation that we've
seen just prior to this is still a
solution to this problem now we're going
to see a different way of
computing the value function
corresponding to the policy pi for an
arbitrary policy and the idea here is to
turn this equation the bellman equation
into an update
and this leads to this following
algorithm we just initialized the values
for all states
uh in the mdp and then we
iteratively
update these values according to this
this equation so at um
time
in the iteration at uh iteration k plus
1
the value of per particular state would
be updated towards uh
the reward that you're gonna see in the
next step
and the value function
computed from the previous iteration
at the next state so this is using
this uh this equality
that we know holds for the
for the estimate we're trying to to get
to
to update towards that that estimate
and the stopping criteria here is
whenever
uh the value function
resulting from this update is the same
as the one at the previous iteration for
all
for all states in the mdp then uh we can
stop and
also when this happens we're guaranteed
to have found um v pi because
let's remind ourselves v pi is the
solution to this equation
so if you plug this in if uh
vk and vk plus 1 are the same that means
that vk
respects or obeys the the belmont
equation for
um
policy pi
does because
this is a unique solution
v v k has to be and v k plus 1 has to be
equal to v pi
okay
uh
does this algorithm always converge
the answer is yes under appropriate
conditions when gamma is uh less than
one and we will see a proof of this
convergence and the conditions under
which this this convergence is is
possible in the next lecture
okay
let's look now at
an example
of how policy evaluation
works in this iterative setting
okay let me walk you through this
example this is a grid world example
where the numbers in the grid represent
the number of the states
and the action space considered are
upright left and right
and for all transitions in the
environment we're going to be given a
reward of minus one
in this states the gray one are
considered terminal states
that means that if the agents transition
to one of these states
then the episode terminates at this
point in time
we're gonna also be using a discount
factor of one
because this is episodic
we don't have any
issues with the
infinite returns
okay and we're gonna see what iterative
policy iteration does in this
in this particular example
this is just the algorithm that we
proposed uh previously
this one
and let's go through the steps of
updating the values according to this uh
procedure
okay so first of all we start by
initializing the values
at zero for instance
oh by the way we're going to be
evaluating
uh
uniformly random policy
form a random
which is the policy that assigns equal
probability to
uh the four actions we're considering
here so
this procedure
needs to converge to
v pi
which is the value of this uniformly
random policy as you see
at the
first step we're just initializing
v0 uh to be all zero and after that
k plus one we're gonna be updating
values according to this equation
okay so just writing it here
this is expectation over the next
transition
the next reward plus gamma
me
the estimate at
previous step
value at the next state
okay
uh now we can simplify things a bit here
because
this is one and this is minus one for
all the transitions in the environment
so
this is now expectation over -1
plus
vk
and
that means that this is 1 plus the
expectation over the next
next stage transition
st plus one
for all of the values
in particular for
v1 this means that this is -1 plus
the expectation of b0 we're always using
the previous estimate of all of the
the values
surrounding this the state
because all of the values
uh at iteration zero were initialized at
zero the expectation over this this
value would always be zero
so v1 for
every state would update to minus one
jigs is exactly what this matrix is okay
uh
let's take uh this further to the next
step
so for
v2 we have a similar update
again this is the general update that
that we're following simplified for this
mdp so it's still going to be -1
plus
the expected value of what's going
happen in the next
state
and now this k
is
is one okay so for each state we're
going to look at
the state it can transition to
under each of these actions and because
it's a uniformly random policy so all
these actions or all these states would
be reached with the same probability
we're just going to average those values
so for instance if we consider in this
state
in order to compute its value
we're going to look here
what are its neighbors and where the
actions would take it so
the right direction will give it here
here and
this one will just loop back so the up
action will just loop back to the state
so the average value of these states
would be
-1 minus one minus one and
zero
divided by four and this is
seventy-five okay
and the same computation can be done for
any of these these other states in the
middle uh these will have a value of
minus 2 because the adjacent states
there are all value minus 1 and -1 from
the
immediate reward that we're getting
and so on so forth we can
extend this computation further and
further
till
this particular step where uh
we reach convergence which means that
and you can do this to convince yourself
that if you
apply this rule again
which is the update rule
for every state
vk
plus 1 equals vk
for vk being this particular matrix
okay
and that also means
that vk
has converge
to be pi which is the value of the
uniformly random policy which is the
thing that we're looking for
okay
good
now
another thing to notice here is that um
we can
act
um greedily with respect to the values
that we obtain in this uh this iteration
in particular
uh if we
consider this last iteration so this one
we've already converted so this is the
policy of the uniformly random uh
the value of the uniformly random policy
and we can see that if we act greedily
with respect to the values
here
we can get the pretty good policy
actually
we can get the optimal policy of
behaving in this environment so we have
a very
uh uninformative policy a very bad
policy in terms of behavior because it
takes every action
randomly
it has no structure to preferring one
action versus the other so the value
function is not too good
but based on that evaluation doing an
improvement step
based on those values greedy improvement
step based on those values gives me a
very good policy in this particular case
this is an optimal policy
this is actually a very um general
principle in reinforcement learning and
in dynamic uh programming of evaluating
a policy and then taking this
gridification
uh step
where
the new policy that we're uh
we're getting out of this gratification
step is actually guaranteed to improve
base
upon the the policy that we started with
so we
start with the policy uh pi we evaluate
that one
and
we get the policy pi new which is the
gradification with respect to these
values what we're seeing is if we're
doing this uh
this step we are guaranteed to
get the value
that uh to get the policy that improves
for each state
over the policy on which the improvement
was made
okay
so this is the
actual statement
that um
for each state and the same for the the
value functions the state value
functions
for each state in action
okay
um
let's go through
uh the proof of that
uh but first i'm gonna try to give you
some intuition why that is the case and
why we are guaranteed to improve and
maybe
then go a bit more into the mathematics
of how to prove this
okay
so first of all
by definition this new policy is
the gratification is the r max of
this evaluation okay
so
given the belmont equation
i'm just gonna write down the bellman
equation here for
for q pi
i have this
and by this notation i mean
okay so the the
action at uh s prime is selected
according to pi
now
by the
the way
actions are selected in uh by new
we know that this
is greater or equal than the max
so the expectation is always
greater or equal than the max
you still have an expectation
where
and this is actually
okay
so let me rewrite the statement
sorry
and what does this statement actually
say
it's saying that um
if
at the next state if at
s
prime i'm gonna be behaving with respect
to this new policy this um gratification
policy uh pipe new
uh
take an action with respect to that
policy and then
continue to follow policy uh pi
just remember that this uh
um
this expression actually means that i'm
taking a an action a prime
according to this new policy but after
that i'm still following the old policy
right
so uh uh
from uh
uh
d plus two
i'm still gonna be following uh pi
so i'm saying
i'm taking an action with respect to the
new policy and then i'm following my old
policy i know this this combined policy
that i'm doing right now is guaranteed
to have a greater value
than the policy that i started with
and this is the
the principle uh here uh there's
there's two things basically that the
that
pi new can do either agree
with
the policy
where um
the previous policy which means that the
arc max there is already what the pi was
doing or disagree with the with that
policy
which which means in this in that case
means that if i'm deviating from from
that policy
i have an opportunity to improve
and deviating only once from that policy
as in here
already says
that i'm
even if i don't do any other
improvements
and follow fall back on my previous
policy i'm still gonna improve so uh you
know schematic
this is let's say the decisions and the
trajectory that i'm gonna fold under the
previous policy that i have this would
give me some uh return and
i'm interested actually in the expected
return
every time i have a decision that
disagrees
the uh
pi newt disagrees with uh with pi
then i have an opportunity of taking a
step that is different from pi
and then i'm saying
if i've taken that step but still
continue
after that with pi
this return
on average is going to be better than
this one
and every time i deviate from uh from
this here
i also have the same property
if i deviate
then and still follow
the
the previous policy
i'm still going to improve and i'm still
going to improve over this one which is
already improvement over that one so
every time i'm deviating from this uh
policy that i'm that i've previously
evaluated i have an opportunity to
improve more and more so the more i
deviate from this this policy the more
opportunity for improvement i have
okay let's try to make this a bit more
uh mathematically precise
so we've already seen that
i'm just rewriting this equation
pi
is greater than equal
then
cross a plus
gamma
expectation over the next state
q
pi
this is the old policy just to remind
ourselves
okay and
again it's the next state we're just
going to take
a action corresponding to this new
policy
and that's it
i'm sorry
the inequality here is the other way
this is
because this is a maximization okay
but then we can apply the same principle
to this value function
and say that this is greater or equal
less than equal then the reward that
we're going to get at
mistake
given that we've chosen this
action corresponding to
y mu
plus
gamma
expectation over
now this denotes the
state after
s prime
this is just applying exactly this
equation in this reasoning but uh now
for
not for s and a but from s prime and
whatever action was chosen by
s prime
okay
and
actually we can
unfold this even further
and this actually would be an
expectation over a trajectory
that unfolds in
time of s a plus gamma
plus gamma
and if i unfold this it's going to be s
r of
x prime prime
mu of x prime prime
plus so so on so forth and this
if you remember the definition of the uh
discounted uh return
is exactly the value function
corresponding to
by mu
sa so
we're in state s we're taking action a
and
therefore
after
we're
choosing action according to pioneer
okay
so this
actually says that
the the value of the new policy that
we've been used by
choosing read the action with respect to
this estimate
if we evaluate that policy we're
guaranteed to get a better value
function than the one we started with
okay and this is the greedy
improvement principle
okay
we can use this
in
what we call policy iteration algorithm
which is this iterative procedure
between policy evaluation
building an estimate of uh v
pi or q pi
and taking an improvement step with
respect to this uh this estimate
such that such as to produce a new
policy
that is
better than the one we've seen before in
one instance of
policy improvement step is the
gridification step there are other ways
of improving your policy but this is one
of the most
used principle
and well-known principles in uh in
reinforcement learning and dynamic
programming and these are just
depictions of um
how to think about this uh
this policy iteration uh algorithm we
start with a value function and and the
policy
we evaluate that that policy
then we
greedify with respect to those that
evaluation to get a different policy
once we have this policy we are doing
this evaluation step again
and this is gridification
again this step
can be just improvement in general but
the the
most common step of improvement at least
for value-based methods is the
gratification
okay and we're
doing this uh this step till we converge
till the the evaluation uh problem uh
basically
the greedy with respect to the the
evaluation problem gives you the same
policy back and at that point we've
converged
and at that point we do have the optimal
value function and the gratification
with respect to the optimal value
function would give us an optimal policy
okay
good
let's look um now at the one example one
more realistic example of
uh policy iteration
okay let me walk you through this
example this is a card rental example
that has two locations which are the the
states of the mdp that has
that have 20 cars
a maximum of 20 cars each
the actions are moving cars from one
parking lot to the other
up to five cars
and
this movement of cars incurs a penalty
of minus two dollars per car
the reward the other reward associated
with this mdp is uh
ten dollars for each available card
rented
the discount factor that we're gonna
considering is 0.9
uh transitions are as uh as
are
as follows cars can be returned and
requested at random according to a
poisson distribution
and the two locations have slightly
different uh poisson processes the first
location has on average
uh three requests and three returns per
day while the second location has on
average four requests per day and an
average return
of course of two
okay
so this is a simple enough mdp it's
quite small just two states and a
handful of actions
the the transitions are quite stochastic
um
but
quite well behaved
at the same time it's uh an mdp where
i for instance can't read off what the
optimal value function or what the
optimal behavior should be as opposed to
the
i guess examples that we've seen so far
where
just by looking at the mdp or computing
a bit the the values in between you can
read off kind of what the optimal
strategy is
here is uh is a bit more uh complicating
to do that as a as a human even even the
even the evaluation problem so for
instance if i'm given a particular uh
policy let's say if um you know there
are
five cars in one of the parking lots and
15 cars in the uh the other one uh any
car uh above 15 i should move to the to
the first parking
space
um
any policy of this kind i wouldn't
necessarily know uh off the top of my
head or with the immediate calculation
what the return associated with that
would be the dynamics of the environment
are
intricated enough that i can't
compute that off the top of my head nor
so neither the evaluation problem nor
the control problem are are that simple
or trivial nevertheless if you
look into the computations of the the
values
it's actually not a very complicated
problem in terms of applying policy
iteration
and this is exactly what happens when
you do
a run policy iteration on this example
so as a reminder policy iteration is
this process where we start with the
policy we evaluate this uh this policy
we take a gratification step or an
improvement step with respect to this
this policy to get another policy in
this uh case we start with
a
random value function
of zero
and the policy associated with that
we get
policy pi 1
by acting greedily with respect to the
value of this uniformly random policy
and this is the evaluation of that uh
policy
pi one that was uh the gratification of
the uniform one
then
uh we get pi two by acting greedily with
respect to this value and this is the
evaluation that we get after that and
continue this process and we see that by
the
fourth iteration or third iteration
actually
this already has converged and we have
um
we have a stable value that is actually
both the optimal policy
um by
pi 3 and pi 4 but also the the optimal
value function
okay
so this is a nice example of how quickly
this this thing can get to the optimal
policy even in
non-trivial example
okay
a couple of more observations on policy
iteration so again this is the process
of evaluating the policy and improving
the policy and repeating that process
one of these steps is the policy
evaluation step and we've seen that
this is not a trivial
step
computing the value associated with the
particular policy can be quite an
expensive procedure so
the question becomes do we really need
to
run especially this iterative policy
evaluation procedure till we converge to
the actual
b pi or two pi
or can we stop when we're maybe closer
to that estimate and save a couple of
iteration
in particular can we simply stop after k
iteration of futurity policy evaluation
and if you remember in the small grid
example we've seen before in the
policy iterative policy
evaluation um
it was sufficient
uh to run
policy iteration for three steps
to
get to a value that would induce the
optimal uh policy let's uh go back just
for a bit
to
this was the example that i was talking
about
and we started with the
random values and by the third value
here by the third iteration of policy
iterative policy
evaluation we can see that the greedy
with respect to these values already
gives you the optimal policy
and see that we we are actually uh doing
quite decently even after two steps of
uh this iterative procedure
but uh we're not getting all the way to
the optimal policy
after three steps we already have
converted in policy space so this points
to the fact that we can you we can save
all of these iterations of converging
all the way to the
to the actual value of the
uh uniformly random policy because if
we're using it only for improvement we
can already improve on this
intermediate estimate and get
a very good estimate of
a good policy
okay
and an extreme case of this is
why not do just one step of this
evaluation step and then
gradification straight after that
and this is equivalent to what we call
value iteration
the way to think about or the way i
think about the uh value iteration is
actually um from a
different perspective of taking the
belmont optimality equation and doing
the same thing that we did with
iterative policy evaluation
uh here and turning this into an update
equation
in this you will see if if you go
through the the steps that it this is
equivalent to doing policy evaluation
policy iteration with
one uh k equals one in the
iterative policy evaluation step
okay
and this is the the actual algorithm we
initialized as before uh randomly or at
zero and then we update
now not according to the expected um
a value
under the the policy we're trying to
evaluate because we don't have a policy
here that we're evaluating or just
taking the max with respect to what's
going to happen in the next step and the
stopping criteria as before
whenever k plus 1 and k
reach the
the same value so the value doesn't
change that means in particular that
um
vk
obeys the belmont optimality equation
and by definition that means that we
have found uh
the star which is uh
the value of vk and dk plus
one okay
and let's look uh very quickly at the
example of how this
behaves in practice
okay this is an mvp very similar to the
one we've seen before
just now i only have one terminal state
that we're going to transition to all
the rewards as before are -1 so we're
encouraging the agent to get out of the
ndp as soon as possible get to this
state as soon as possible to transition
out of the mdp and we're going to be
considering a discount factor of one
because this is an episodic setting
and then we're interested in running
this new algorithm value iteration
in um
in this mdp so we're gonna initialize at
zero
okay
and then we're gonna run
k plus 1
max
expectation over what's going to happen
next
um
previous value
actually this
should say a
okay
okay so the
instantaneous rewards are all one so
this actually update is minus one
plus
gamma
max over a
expectation of
these values
okay so basically what this says
is
um
i'm gonna look
at i'm gonna take an action
i'm gonna transition to this new state
and i'm going to look at the
value
that
has been generated in the previous
iteration and i'm going to look at the
expectation over
all the states that were induced by
transition from s
given action a considering this is a
deterministic ndp this has only one
deterministic transition
so all actions have a
deterministic transition to a particular
state so for instance for this state
this action would get here this actually
would get here this section would get
here the section would get here right
so it's not really
an expectation so basically what this is
saying is that i'm gonna look at all the
neighbors around the state that i'm
estimating value and i'm gonna see
which of them has the maximum value
and i'm gonna
and this estimate will take that to that
maximum value okay so
uh the discount also is one so we can
ignore that one
okay the the first one is just uh
basically initialization
um the second one is trivial too because
v0 is zero everywhere so this
term will be zero so
everything is
updated to minus one
now uh in the
second iteration of uh v3
uh it's a bit more interesting
that uh we're gonna look at
let's say this state
actually
first look at
sorry
this state
and this state
has
a transition
to
this state a transition to this state a
transition to the state and the
transition to
back to itself so
um
instead of doing an average over these
transitions as we did before under the
uniformity random policy we're going to
look at which of them produced
the maximum value
and take that one
so it's gonna be it's gonna update the
value to minus one which is the
instantaneous reward plus the max over
these these values which is zero
okay
so the update to that would be minus one
for all of the others
uh basically any transition would lead
to the same uh
minus one reward and a minus one value
function from the previous
iteration from this
and this is added to the minus one from
the instantaneous reward
and
we continue this process till
basically we can see that these two
um
are
almost converging and if you would
estimate now v8
uh
then you would have basically perfect
convergence
you would see you already see that
basically it in this step between
v6 and v7
basically all states but one but this
one have the the right value and
everything else hasn't changed
and in this new step we would just reach
convergence and that would give us both
the optimal uh value function but also
acting readily with respect to that
would give us the optimal
strategy
that is it for synchronous dynamic
programming algorithms and this is just
table summarizing the things that we
have covered so far so first we started
with the prediction problem an
evaluation of a policy problem
where we took the bellman expectation
equation and turned it into an iterative
update leading to the iterative policy
evaluation algorithm and then in the
last two sections we've
looked at
uh tackling the control problem via
policy iteration and then
just just now value iteration one relies
on this principle of evaluating the
policy and improving it and then
evaluating this improved policy and
improving on that one and so on so forth
the the last
algorithm that we talked about the value
iteration uses the bellman optimality
equation to go directly for the optimal
value function
some observations
the algorithm based on
state value functions
trying to estimate v pi or v star have
the complexity of uh
number of actions to
um
squared number of states per iteration
this is because if you look at the
updates we always have to
do this update for each state and always
have to look at the action that we're
gonna take after this step and where uh
we are gonna end up after this state and
the same if
we are looking at computing action
values this complexity is now magnified
by another factor of the number of
actions because we're always looking at
state action so current state and action
take and what's going to happen in the
next step
s prime and a prime
okay
this can be quite quite expensive so in
the
in the next section we're going to look
at extensions to dynamic programming
in particular asynchronous dynamic
programming that that will try to
get around this this computational
issues or alleviate some of these
computational issues so the
dynamic programming methods that we
described so far use synchronous updates
which means that
uh when we've gone through the examples
you've seen that we update all the table
all the time so for we're doing these
updates
every time we're doing them we're doing
them for all the states in parallel
okay asynchronous dynamic programming
suggests that we can back states
individually in a particular
order or in any order and this can
significantly reduce the the
computational overhead
and also the nice thing about this is is
guaranteed to converge as long as we do
visit all the states and we continue to
visit all the states with uh with a
non-zero probability
uh a couple of instantiation or simple
ideas of uh
asynchronous dynamic programming or
in-place uh dynamic programming
prioritized shipping and real-time
dynamic programming and we're gonna
quickly go through
them they're really simple so
this is the the general idea so in the
synchronous version of value iteration
we store two copies um
vk and vk plus one in order to do this
update
now the in place value iteration stores
only one copy and uses that for the
updates so all the values that we have
already updated
those would be used whenever we
encounter the next update
rather than
maintaining the copy of the values at
iteration k
okay
prioritize sweeping
the principle here is to select states
according to this priority function and
an instantiation let's say of the
priority function can be using the
magnitude of the bellman error and we
haven't defined it uh yet but this is
the equation for the bellman error
um
the the thing to notice here is
that
um for the
optimal value function so if v is a v
star
this thing is zero so
when we converge to the
optimal value function uh for each state
the the priority uh there would be it
would be zero so
we wouldn't
not be selecting these this states for
update anymore
okay and uh this
basically says that the the largest
error will be prioritized
in
in order to update the state
this requires knowledge of the reverse
dynamic in order to
make the update
but can be efficiently implemented by
maintaining a priority queue
okay
and real-time dynamic programming
basically the idea here is to only
update the things or the states that are
relevant to the agent right now and
what's relevant to the agent right now
is debatable but let's say
as a rule of thumb maybe things around
the agent that the
states that it will be using like the
current state or the value at uh future
earth states that could be used for
immediate prediction or immediate
decision making
another way of reducing
the the complexity of these updates is
to
look at the sample backups so so far in
in all standard dynamic programming we
do full width updates so for every state
in the
or every state action player for the the
q function we look at all of the
successor states and all of the action
that span out of that
that state
and
once we look at all of these transitions
conditions on all the the actions we do
a full backup for that for that state
now um
this is
effective or can be applied to
medium-sized problems but for very large
uh dynamic uh
uh
pro programming problems
this might might still be quite
expensive or even
too expensive to to run even one full
backup if you have a very
large state space or very large action
space
even one full backup might be too
expensive so the idea uh here is to
instead of doing this whole backup of
looking at all the
actions in all the states that follow
uh state s in order to update it we're
just gonna look at
sample versions of uh this so we're
gonna have only samples of these
trajectories maybe a very small a set of
the transitions that um actually come
out of
s
but we're gonna use those to uh
to make our updates so we would use
sample reward the
transition of this
form s a
r and s prime just one instantiation of
this trajectory instead of uh using the
full dynamics
in the full model to update the the
whole state the advantage with that is
that it's model free we don't need to
know the actual dynamics and the
expected reward in this
case because we're just using the
information
given by the sample and it breaks the
curse of dimensionality through through
sampling
we will see a lot more of
these especially sampling methods and
implementation of uh versions of dynamic
programming in a sampling context
next week and
a lot more
methods of
making this more tractable and making
this more sample efficient
okay
um that is it for uh today this is just
a summary of what we've managed to cover
today so we started with markup decision
processes to formalize the
rl interaction we've looked at
objectives in the in mdps different
notions of return like discounted and
discounted average reward we introduced
value functions which are expected
returns
we looked at optimality principles in
mdps
what our optimal value function how we
can define
optimal policies
we also look uh at the rich structure
that
that
value function have
described by the belmont equations and
we've seen how to use this this equation
in order to
tackle the two class of problems in rl
evaluation
and control
um
and we've seen in the in the last half
now how to compute the
um
value associated with
an arbitrary policy
pi okay
solving the prediction or evaluation
problem
by a system of linear equation and
or iteratively via iterative policy
evaluation and finally how to compute
the optimal value function using dynamic
programming
we've seen two algorithms here policy
iteration and policy
policy iteration and value iteration
okay
uh next lecture we're gonna go a bit
more deeply into uh dynamic programming
we're not going to cover more algorithms
but we're going to go
more deeply into
how these work and why this work and
when they converge
because so far we've seen only examples
of how this work in uh in practice but
uh
we haven't theoretically analyzed them
okay
um
if you
uh if you have any questions about this
lecture or any of the the material uh
presented please use moodle for
questions and the next q a session
that is it for today thank you for your
attention


243. A General Language Assistant as a Laboratory for Alignment

hello and welcome to session
243 in the ai safety.com reading group
tonight we'll be discussing the article
a general language assistant as a
laboratory for alignment by jared kaplan
and many others
this is the first work done by anthropic
new alignment research
institute and these are the primary
authors with jared kaplan as the
corresponding and probably primary
author and there are a number of
people who are
primary models would have helped in some
other ways
it's a long list
and actually
i have heard from these people i know
amanda esco and kevin of course and i do
actually know quite a few more of the
non-core people who have been helping
and it's um we still haven't actually
heard but what they are doing um so i
have at this point i just assume they
are building some kind of agi that is
trying to kill us all or something like
that
so this is uh
from uh
uh two months ago and we are focusing
on the non-typical parts so
we're looking at the philosophical
motivations rather than the explicit
machine learning things
and the evaluation criteria where i
deliberately
i haven't read really
the parts about how they're in fact
implementing this because the the people
working in anthropic have really great
credentials they are known to be really
good at this and i care somewhat less
about evaluating how good they are i
know they're good and i care more about
how do they look at the world um
kowski had some comments on his twitter
on this paper
where he was uh moderately positive and
that they directly challenge his
alignment it doesn't bring the
capability
and it doesn't overstate the
accomplishment
i want to uh anything
that sounds really like standing with
fake praise but that's just his style
and i don't think he intends it
as much as um as faint praise
and i want to dig deeper into these
parts like okay they're directly
challenging alignment but are they
challenging alignment using a strategy
that might work or
is this just a hopeless strategy um
are they burning
capability commons i think uh like are
they actually um
that's the old somewhat outdated model
with two progress bars with ai
capability and ai alignment and are they
working too much on
capability um
not
lying and saying they'll solve the
problem that really really sounds like
faint praise to me but in fact there are
a number of quotes you can pull out from
this that seem to be quite modest and i
it's sad that it's relevant that
just to say that this is a good thing
but i mean it is a good thing
so when i investigate the motivations
one of the things that i look for are
like tiny things in the introduction
where they say that future ai systems
might do bad things and interact in
presently unforeseeable ways
and
that's of course the thing that can
happen but i care about the
presently foreseeable ways that things
can go wrong
so they have a definition of alignment
that is somewhat non-standard define the
alignment between two agents as to what
extent their preferences overlap
and it's not really a definition they
use very much uh they almost immediately
transition to a uh a more workable
definition and it should be mentioned
here that almost perfect uh overlap in
uh how you rank different outcomes could
uh be arbitrarily bad quite easily
and in addition to uh
to this the one of the uh
overworking ideas is to
look at language models directly instead
of looking at perhaps more advanced
models and paradigms because language
ones have a large number of advantages
you can try a lot of different kinds of
inputs and they can fail in many
potentially quite interesting ways and
benchmark how much progress has been
made you can compare different alignment
techniques to try to get some kind of
idea about where we are
in in more specific details you can try
to see whether prompting can work as
alignment and to what to attempt and uh
you can see
if language models can
model preferences instead of just doing
invitation learning
and they focus a lot on improving sample
efficiency uh
of uh preference marlin and these are
some very uh
nice and interesting goals
so the way they actually uh
the definition of alignment that they're
actually using is helpful honest and
harmless
and
the
the way they justify this is with the
following quote it is difficult for an
ai assistant to always be helpful honest
and harmless towards an agent without
also being highly aligned with that
agent
and i'm not entirely sure i agree with
this because
the word always is very important in
order to make this work because
the techniques that they're using are
black box methods and blackboard methods
will not give us any kind of proof we'll
now be able to say this model is always
honest if we are only looking at it from
the outside um
and if you try to remove the word
harmless
the word always and just most of the
time then from this definition it
becomes very clear that it is indeed
very possible to be
very often helpful and honest and
harmless but not always harmless right
then you get into things like deceptive
alignment very very quickly
there are advantages of this this is
something that is much more actionable
is understandable memorable um and it
seems like on on
a more uh
precise view of uh alignment but this is
indeed a big part of what we actually
want
and of course the language ones that we
currently have are uh especially when
the like gt3 was
uh released it was very clear that it
was not helpful it was not honest and
was not harmless so it is something that
is indeed substantial for us
these criterias are
a lot less than mathematically
well-defined right there are uh
trade-offs and ambiguity and the way
these are resolved suggested by uh by
the authors is that um the person who
deploys the ai needs to take
responsibility for this
now when it comes to uh
existential risk then who is responsible
after it has gone wrong you know it
might not be the right thing because
we will be dead by then
and also there's the obvious thing in
that the people who are right now
deploying these things like uh whether
the um
they seem to
not really care about this at all and so
if um i don't think it's possible to uh
absolve yourself of responsibility if
you're building a tool and saying this
tool you should be careful when you use
it but if you positively know that the
people who are going to be using it are
going to misuse it then you're not
absorbed from responsibility
let's think a bit deeper down into these
definitions
helpful that's uh
that caches are with clean efficient
optical clarification and redirect
informed requests
um and these are nice and this is what
we want from ai but it's not at all
clear that this has very much done with
alignment this has a lot to do with
capability research
i'm not making the claim here that it's
just pure capability research but i
would like to see the authors make some
kind of positive argument why this is
not just threatening the capability
comments
to be honest cashes out as accurate
calibrated and communications can build
this knowledge and honest about itself
um
it does say honest about itself but from
the notes it seems clear that they are
not really talking about treachery here
and that's of course the thing that i
personally care most about
finally harmless not offensive
discriminatory and refuse
aiding dangerous activities and
recognizing various use
unfortunately the one where they refuse
to
assist in
doing bad things is one thing that they
chose not to uh
to investigate
and also you could argue
if the ai takes over the world it hasn't
strictly
violated any of these constraints
so when i look at this intron i see many
many many sub criteria and i worry here
that a lot of these are probably
irrelevant i mean whether it is
efficient doesn't matter very much for
alignment is it well calibrated that
doesn't matter either does it is it
discriminatory i mean sure it's
something we want from ais not to be
discriminatory but it's not really very
central for alignment at all and i feel
that this definition uh
uh might water out uh
a lot of the central parts where
alignment is problematic because i mean
we might get a very efficient and well
calibrated and non-toxic ai that
performs a treacherous turn and that's
not really helpful
here i have
perhaps somewhat unfairly actually quite
unfairly this is a pic an image that uh
relates to uh asimov's three laws of
robotics which is a very famous for
being the most horribly bad alignment
proposal ever and it was known by asthma
the novels are literally about how this
is a horrible uh plan for alignment um
and by this uh like
an aim must be pretty uh
harmless and
while being harmless can it be helpful
and uh honest and
is it actually the same thing as um
well i want to make i want to make it
clear that i don't think this is this
the same as asimov's three laws these
three criterias but
uh it's not immediately obvious where
the the big difference lies
and i think some kind of more
description would have been helpful here
there is indeed some kind of description
on these criterias for like whether they
imply each other helpful and harvest is
that actually the same um there is a
description i won't go into details or
to say that
in the maximum case the more
one happens hopefully this the more
focused it will also have to be but if
it's
moderately helpful it it doesn't follow
to the same extent and that is
also moderately harmless
and the same with honesty
they
write something that i think was really
really interesting here they considered
a fourth age handle ability which is
basically encourageability and i thought
that would have been really really great
to include i would have been really
happy to have some kind of consistent
description of
whether what does it mean that language
models are courageable i mean you can
imagine things like okay you gave this
description could you uh if if the air
gives some kind of description then you
ask it please explain it like i'm five
years old or
give some of the uh understated
assumptions and what would be the
consequence that you're gonna talk
telling me about uh latent knowledge
this kind of thing would be really
really interesting to uh
to deliberately so i think it was said
that they chose
not to have that full speech
and that's all about rapid quality
information
intra aging conflicts and ai security
and all this is
is basically fine but something that i
will go into detail
well you need to obviously improve quite
a bit the orders are very clear that you
need to improve quite a bit um and
that could fail for different reasons
you could fail because they are unable
for some interesting technical reason um
i i'm not entirely sure that they
uh the philosophical grounding is um
good enough that they can say there are
only technical challenges um
but they could also
end up saying okay we've actually
managed to solve these typical
challenges so at least for these
language models we can to some extent
alignment and of course they are honest
enough to
acknowledge a third option that they
fail in uninteresting ways
one of the things they worry about is
misuse that uh
you can align it perfectly with like a
very bad actor and then
very bad things can happen that's all
right this is foremost in our minds
and i am
not entirely happy about that because i
agree misuse is a problem but it can
also be some kind of distraction from
many of the other problems in alignment
and i'm not sure that should be foremost
on their minds
and then
there's some argument about
along with what they're doing is scaling
research and that's of course what gary
kaplan in particular is one of the were
the best in the world at um and they
have some arguments about why that is
understand
why this machine learning system works
um
but i think here there is a clear
uh argument to be made that they are
actually doing capability research
elizabeth wrote that he doesn't think
they're doing it and i think they might
actually arguably be doing it and there
are also like small quotes here that you
can pull out have someone out of context
to show that i think there's a good case
could be made that they don't actually
care so much about certainly not the the
two
proper spas model
they are doing capability research
so how do they actually investigate
whether language models are uh helpful
harmless and honest well they start by
hand coding some different uh
evaluations like uh
here's one description of the ai saying
something and here you think something
else which of these are more artists
which are more helpful which are all
harmless and then they
do like ap testing to
get some kind of model um in the form
where it's more like an open-ended
dialogue and of course with a lot of
prompting and this prompting in
particular they're using to
write it as an ai personality and
it kind of makes sense right you can
write the first
10 lines in in a
in a discussion and then
you kind of get a sense of what kind of
person you're talking with
and
gt3 of course
takes on this
persona in the prompt
and language ones in
general we are quite optimistic about
this saying perhaps prompt related
techniques can carry alignment efforts
further than we initially expected
and i just want to shoot my own horn
here back in september in the reading
group i made the
prediction that prompt hacking could
indeed turn models five percent more
aligned and that was indeed something
that was worthwhile to assume
um
but also we can get it so much further
but i don't believe we can get a full
solution and neither does anthropic
there are problems obviously you can
imitate the human level you can't exceed
during doing prompting um and we want
the ones to be honest and not to
to run into if they try to scale this up
is going to be that they have a very
wide definition of alignment where there
are a lot of things like
show that you have
that you are well calibrated the more of
these extra things you add into the
definition of alignment the more
problems you're gonna have with this
kind of thing
is my prediction that i don't really i
obviously don't know the future
one particular technique that they
introduce is called context distillation
they describe it as conditioning on a
line behavior
now the problem with prompts of course
is that they take up some of the
precious precious context window and
some of the language models haven't been
very
um
challenged on this point um
and so the obvious alternative to doing
prompt is to find true but fine tuning
is not precisely the same because
first there is probably a lack of
aligned data in many many situations and
um fine tuning also gives expectations
on data distribution as an easy example
would be if you have a prompt called one
two three four
then any landing which model worth the
salt would say the next number is five
whereas if you fine tune on this kind of
thing then the ai will assume that the
thing is going to be
uh talking about that will be sequences
like this and then if you ask me like
some other questions it will
be totally relevant so fine tuning and
prompts are two different things and
they can't immediately be substituted
but they have a
technique for um
for doing this anyway context
distillation i won't really go into
details about how that works and it's
mostly in the chapters we skipped but
add um an extra step in between the
language model pre-training and the then
they after that they um pre-train for a
preference model and then they do some
fine-tuning on the preference model and
this uh extra step in here prefers while
pre-training is
as fast i can see original
um and it seems to of course work quite
well
but prompts also
do work quite well so um we'll get to
that later um and they have some more
ideas about how to uh
improve that and i think it's
very interesting to see whether that
will work but
it's not obvious to me that it will
actually work i again register a
prediction that if you have a
sufficiently large uh
language model that trying to
load in and
align the identity into the language
model will not matter it will just
compartmentalize uh and then sometimes
if it's in the situation where it
believes it should talk about alignment
related things and it will do that and
in other cases if it believes it's
better to do something else you'll just
act totally unaligned
again a prediction
so they evaluate this
um
with a lot of uh
without prompting and they wrote that by
hand 200 comparisons on these three
perhaps
and
you can see here roughly how well it
goes
[Music]
down here is with no intervention and
you get closer to what humans prefers if
you either do the prompt or the context
distillation and um
they seem to perform substantially
better and if you down here try to split
it up into the uh the three helpful
honest harmless and then other i could i
was unable to find out what other
precisely was um
and then you can see all of these help
it's best and honest they also seem to
believe that okay it looks very much
like the honest have
the the best performance but that's the
best absolute performance and you can
see already from the very small models
they were actually also
better on the honesty metric uh so the
um the actual slope from here to here is
not substantially greater than the than
the slope from here to here
so uh it's just perhaps honesty is just
easier or the the hard-coded comparisons
were just easier
again i'm speculating right
and honestly um
here they have spread out what what that
means um
in their uh
in in their handwritten evaluations
and
one of the big problems here is that
even if they try to make the model as
honest as possible and they get a model
that seems kind of arduous then it is
totally ready to just fabricate
information yeah and they were unable to
to get that out of the language model
and they admit this is a major weakness
of the evaluation and i agree right
that's
to me of of these three helpful honest
and harmless the the honesty to me was
most important so i'm a bit sad about
that
how about the human performance modeling
well there is a luck linear relation
with um like how much better your model
preferences as the model gets bigger um
and i guess you're all yawning right now
with these log linear relations where
as the language model gets better they
get better at everything and well they
also get better at modeling human
preferences um and so yeah i
um
i just want to raise the fact that the
fact that language models get better in
this way is
probably going to be the thing that
kills us all so even though we keep
seeing the same trend over and over and
over again in so many uh
different contexts the
we should not lose sight that this is
potentially very problematic because
these models
probably will seem to continue
or do they do they go down a bit here
well they do uh speculate they obviously
they do go down
at the end at the high end of the
spectrum and they are speculating that
the problem that causes the um
the model to cease being sufficiently
better is um is that the
mechanical turks that they are employing
are just
they are not skilled enough to actually
see which of these are in fact most
helpful and
it's of course a sad thing that it seems
like um
there was an article a long time ago
with humans who are not focusing uh less
skill than gpg3 and it seems here that
if you take um internet volunteers that
are not too
very well paid
it seems like they are not able to
distinguish well enough at this point
and i expect that within a couple of
years it's going to get harder and
harder for mechanical turks and for
everyone to just evaluate how good are
these uh these models
and they have some more statistical
things that are put with which isn't
really important are these uh uh ones
that are using this
context installation that are
conditioned on align behavior are they
worse at things
well um
they have here some examples that show
that indeed
as the model is not very powerful it is
worse to assume alignment but as they um
the one gets better than the alignment
uh
uh text seems to disappear um i i'm
gonna they even say something that like
it seems noticeably better i think
that's overstating the benefit really
and i think
more likely
it's just the model is powerful enough
to just
ignore either the prompting or the
distillation
in this case the prompting
so to some of the contributions they
have this uh performance model uh
pre-trained performance modeling and
that does improve sample efficiency does
improve performance and they show that
just prompting is
something that helps alignment to a
substantial extent
in particular in the case where there
are only small data sets
they also uh report that untruthful
two-way another uh
set where we've previously seen um
uh larger more models performed
like the opposite results
uh i don't think it's very important
though
they also say that the fb2 is for
alignment text and of course i should
state here that
they have done a lot right there is
a lot of people that we haven't read
and
i
also i haven't read but i must worry
here that they are indeed providing you
a substantial capability uh research and
it's um
uh i would have preferred at least some
kind of discussion on why they're not
doing that
that is all for today thank you and see
you next week