DeepMind x UCL RL Lecture Series - Policy-Gradient and Actor-Critic methods [9/13]

hello and welcome to this ninth lecture
in this course marine enforcement
learning
my name is heart of the hustles and
today we will be talking about policy
gradients and extra critics
in terms of background material i
recommend that you have a look at
chapter 13 from the book by rich sutton
and andy bartow
and in some sense the motivation for
today's talk can be captured in a quote
by vladimir putnik who
famously wrote in a book about learning
theory
that one should not solve a more general
problem as an intermediate step
so what does he mean with this well if
you are going to solve a more general
problem and sometimes this is tempting
then this is going to almost necessarily
be harder
not always but typically especially if
it's truly more general but that means
that you're actually spending maybe data
compute resources in whatever way on
something that is harder than you need
and maybe it's better to solve the thing
that you actually care about directly
it's a rule of thumb it's not always the
case that this is generally true but
it's a good thing to keep in mind and if
we apply this reinforcement burning
there's a question that we can ask
ourselves
if we care about optimal behavior in the
end why do we not learn the policy
directly
so in this lecture we will be talking
about this
but to dig in a little bit further into
this high-level view let's first compare
to other approaches to ai
where we've talked about model-based
reinforcement learning and this has some
benefits including the fact that it's
relatively easy to learn a model in the
sense
that this is a well-understood process
it's supervised learning
so when you learn a model typically we
learn
either an observation to observation or
a state-state model and we learn a
reward model and at least the mechanisms
which we could use to learn these are
fairly well understood because this is
basically just supervised learning of
course the model itself could be very
complicated and this can be a problem
but the other benefit from learning a
model is that in some sense you are
going to extract all that you can from
the data you could imagine that if you
see a transition in which nothing too
exciting happens in terms of rewards
and maybe there's not much to learn
about your policy directly it could
still be useful to condense some
information from that transition into
some structures in your head some
knowledge and if you learn a model about
if you're trying to learn about
everything um
then at least you're extracting this
information
but of course there's also downsides
including the fact that you might spend
quite a bit of computation and capacity
on irrelevant details
classic example of this could be
what if say you're playing a game let's
say pac-man and
now consider learning a model for the
game of pac-man so you could imagine
having a frame and maybe we're literally
just learning a model from observations
to observations so you have a frame
you're trying to predict the next frame
now that might already be quite
difficult but now let's
extend the example and now let's imagine
that instead of the normal black
background you have in the game of
pac-man let's assume that there's an
irrelevant video playing
maybe just some television program is
playing in the background
and let's assume it's not too
distracting so you
as a human playing pac-man could still
just play the game and you would
basically fairly quickly learn to kind
of tune out the video in the background
but if you are training a model and
you're not telling the model which parts
are important because maybe you don't
know in advance you're just supervised
learning from frame to frame most of the
capacity of the model might be focused
on trying to learn
the pixels associated with the
background video rather than the pixels
that are important for us to play the
game this is what we mean
when we said it might
spend compute and capacity on irrelevant
details things that do not matter for
your policy do not matter for the reward
but if that's not known to the model
learning then it might still focus on
that
in addition
even if you have a very good model and
you were able to learn maybe even a
perfect model from the environment you
would still have to compute a policy
this we could call planning and this is
typically non-trivial and can be quite
expensive in terms of computation
because especially if you want to have a
very accurate model of a very
complicated world you could imagine that
this internally also will be quite a
complicated thing and therefore even
computing one step into the future one
imagine step can be quite
computationally heavy
we talked a lot about value based in
this course so let's also list some
properties of this pros and cons
so first it's a lot easier to generate a
policy if you have a value function in
particular if we've learned an action
value function
and
in the case where we have a discrete set
of actions
then picking the
greedy action with respect to these
action values is very easy and that's
this is a valid policy a greedy policy
we could also of course consider a soft
greedy policy or other things but
generally it tends to be relatively easy
to generate a policy we don't need to
have a very slow planning process to
extract the policy from the values
in addition this is fairly close to the
true objective closer than the model
because at least when we learn values
well that's what we were wanting to
optimize with our policy right so
learning values is
less likely to capture all sorts of
irrelevant details
and maybe it's closer aligned with the
true objective
it's also very well understood because
there's been lots of research in
value-based reinforcement learning and
very good algorithms do exist although
sometimes there's caveats or things that
are a little bit less well understood or
even understood not to work very well
and
then sometimes solutions for this are
proposed as well
but it's still fairly well understood
it's maybe a little bit harder than
supervised learning in general but we do
have good algorithms
however it's still not the true
objective and you might still focus on
capacity only relevant details for
instance
if you're trying to learn a value
function you might see quite a bit of
function approximation capacity in
learning the accurate values for one
action versus another action
whereas maybe the difference in values
could be huge and so for the optimal
policy it's
clear much earlier that one action is
strictly better than the other one
and then syncing in more
data and compute in and function
approximation capacity in figuring out
exactly how much the difference is might
be irrelevant in terms of which policy
you're going to follow
in addition to this because the
objectives are
although they're aligned they're not
truly aligned fully aligned a small
value error can sometimes lead to a
larger policy error this is particularly
the case when you have a value function
approximation
errors for instance because your
capacity is limited or because your data
is limited so that means what if we
can't actually accurately
model the policy values for all actions
in that case you're going to have to
have some trade-offs and these
trade-offs these function approximation
errors might sometimes lead to a
different action seem seeming to have a
higher value
but in fact it might be that that is
really not a good action and it's just
because of generalization or function
approximation error that you think that
has a good value and this might lead to
a fairly large policy error
so in this lecture we're going to talk
about policy-based reinforcement
learning which at least is the right
objective let's if we're interested in
finding the best policy maybe you can
optimize this exactly
but we'll talk about more properties
pros and cons of this approach on later
slides
in general it's good to keep in mind
that all of these different approaches
generalize in different ways if you
learn a model it might generalize in
some ways but maybe worse so in others
as in the video in the background
example value-based reinforcement
learning again will generalize in some
ways good sometimes bad in other ways
and policy-based reinforcement printing
as well will generalize in different
ways
sometimes learning a model is easier for
instance it's dynamics or particularly
simple you could think for instance of
let's say a very simple game in which
you just
move in a grid and it's very predictable
what will happen if you pick a certain
action it might not take you a lot of
data to figure out oh in this grid if i
move right if my action is to move right
i will literally move right a step
unless if there is a wall then i'll just
stay where i am
that might be a model that is relatively
easy to learn and it could be very local
in some sense where you only need to
know a little bit about the immediate
vicinity of where you are in order to
accurately model what will happen next
and then maybe you could learn a model
that is easy to learn and you could use
that to plan very deep into the future
but of course sometimes learning a
policy is easier like you could imagine
instead if we consider the real world
well that's hard to model right we we
find it very um
hard to predict exactly what will happen
and it's a very messy stream of
observations that you get for through
your eyes say or a robot could get
through its camera and there will be
some
noise perhaps and in general it's just
very hard to model the whole
real world obviously but the policy
could still be very very easy in some
cases at least it could be that you just
always need to go forward you're a robot
you just need to go forward and maybe
that's the optimal thing to be doing and
maybe that you could learn that quite
easily without having to worry about all
of the intricate details of the world
so now formalizing these things a little
bit farther
in previous lectures we've approximated
value functions parametrically so v pi
and q pi here to know the true values of
a policy pi
and v w and q w then
denote the approximations
so
in this case we can then generate a
policy for instance by picking it
greatly but now we're going to do
something different we're going to have
a different set of parameters theta
and these will be used to directly
parameterize the policy so the previous
lectures we still had a policy because
you always have to pick your actions in
some way but the policy was inferred
from your value function and now we're
just going to basically have some sort
of a function that will output the
policy parameters
for instance this could be
a neural network or a linear function or
something as a form where these that
could then be the weights of the neural
network or the parameters of your linear
function
we will focus on a model free
reinforcement learning policy direct
policy search or learning policies can
of course be combined with models as
well but we'll just
not go into that that topic in this
lecture
so in terms of terminology it's good to
be aware
it's a little bit of a so you see a venn
diagram here on the side and there's a
little bit of
terminology
that people tend to use for these things
quite obviously value-based
reinforcement learning will use values
but typically when people say value
based they mean that the policy is
implicit instead if you just have a
policy of course you could then call
that policy-based reinforcement learning
but then if both of those are there
there's and the value function and the
policy people typically use the
terminology of actor critic
and this is somewhat older terminology
where the active then refers to the
policy and critic refers to the value
function the reason to use that word is
that the value function is then used to
update the policy to critique the policy
in a sense so this is where those terms
come from
so we will touch upon actor critics in
this lecture as well
so
enumerating the advantages and
disadvantages of poly based
reinforcement learning i already
mentioned that one of the prime
advantages is that it's the true
objective if we really try to optimize
the policy why not tackle it head-on
it's also turns out relatively easy to
extend the algorithms that we'll talk
about to high-dimensional or even
continuous action spaces
so for instance if
if you think of a robot a robot doesn't
pick between say three different actions
or five different actions no instead it
could send electrical signals to its
motors and maybe in an almost continuous
fashion right so you could
exert a certain amount of power to
something but maybe the amount of power
that you can exert is really a real
valued number it's not just one discrete
choice you cannot just pick one or two
or three but maybe you could also pick
1.2 or 3.7 or something like that
in addition another benefit from
parameterizing the policy directly is
that we can parameterize this in such a
way that the policy is stochastic which
means that it's a random policy rather
than a fully deterministic one
we saw the stochastic policies before
the greedy policy is an example of a
deterministic policy
epsilon greed is an example of a
stochastic policy where there's some
randomness in which action you pick
and in addition i already mentioned that
sometimes policies are very simple while
values and models are complicated
and this has multiple benefits one is
sometimes it just means that learning
the policy can be a lot easier and in
addition to this it's sometimes also
much easier to represent the policy you
could imagine that a value function
could be really complicated
because maybe it matters a lot how close
or far you are from a certain goal and
then even if you have a lot of data and
you learn really well you have really
good algorithms it could be that if you
pick a certain function class it just
doesn't fit you can't actually represent
the whole value function but it could be
that in exactly the same problem that
the policy is relatively simple
and that you can pick a function class
that is relatively simple and still fit
it exactly the optimal policy
that's not always the case but it can be
the case and then it's good
to basically tackle it hello as i
mentioned before
so examples include when the dynamics
are very complicated but the optimal
policy is for instance always to move
forward or maybe just always to spin
around or something like that
there's also disadvantages
for one if you we do policy search well
there's different ways to do that but
typically what we do is we do some sort
of hill climbing we use gradient
algorithms or even when we use something
else like for instance evolutionary
algorithms if you don't know what those
are that's not that's not a big big
problem but also those tend to be local
in the sense that you're you're
considering certain policies and you're
searching around those policies for
better ones
and then it turns out you could get
stuck in a local optimum
which means that then you find a policy
that is relatively good
but there could have been much better
policies but they're too different
for you to incrementally move towards
from your current policy
in addition to this the obtained
knowledge can be very specific which
means it doesn't always generalize well
this is related to the point i made
earlier about model-based reinforcement
in capturing everything that you
possibly can about the environment
policy-based reinforcement planning
doesn't do that in fact it tries to
reduce
the data to the most
narrow thing that you could need for
optimizing your behavior which is the
behavior itself
but that means that if say up until now
in all of the situations you were it was
always optimal to move forward
and then suddenly it isn't because you
maybe move into a new room and now you
have to do something else it could be
that by that time the policy has
basically learned to completely ignore
its observations and just move forward
and then it can be hard to unlearn this
it might not generalize well in that
sense
and this is related to this property
that basically at a high level if you
learn a policy directly you're not
extracting all the useful information
from the data
and that means that it might be harder
to adapt to new situations
okay and now we're going to talk a
little bit about stochastic policies in
particular and why we might care about
learning those
so why would we need stochastic policies
um you might
recall or you might know that in markov
decision process there is always an
optimal deterministic policy there it
always exists such a thing
but turns out most problems are not
fully observable i talked about this
even in the very first lecture where we
uh talked about for instance like
consider a robot with a camera rolls
looking in front of it you can see what
is in front of it but it cannot see
what's behind it that would be a not
fully observable markov decision process
or a partially observable markov
decision process especially if it
matters what's behind it it might see
something oh that might matter for what
i'm gonna do now turn around you don't
see it you don't see it anymore
but the best thing to do might still
depend on this
and this is the common case and in fact
even if you have a markov decision
process if you could say well the world
is the mark of decision process it's
just a really really complicated one
sure but if you're using function
approximation then the agent can still
maybe not distinguish between states
which basically means you're still in a
partially observable setting even if the
world itself is markovian doesn't mean
that the learning algorithm can
fully exploit
this and if that is the case then the
optimal policy itself might actually be
stochastic
in addition to this the search space
can be smoother for stochastic policies
because
you could imagine for a deterministic
policy in every state you basically have
a choice do you think this one or maybe
you pick that one and that might be hard
to optimize space because you basically
have this discrete choice of either
doing this or either doing that so
there's a big combinatorial search
problem there if you're only searching
within deterministic policies if however
you're thinking about smoothly changing
the probability of selecting one action
above the other that could be a much
smoother surface and that turns out to
be important for optimization in
particular we can then use gradients
which are very successful at optimizing
deep neural networks and similar
structures and these tend to lead to
successful and impactful learning
algorithms
and then finally
having a stochastic policy can be
beneficial because you automatically get
some expiration i put expiration between
quotes here because maybe this is not
exactly the right type of explanation
just picking actions a little bit
randomly might not actually give you
enough coverage it might not seek out
important information about the world
but still in many cases it's better than
nothing and it might still lead you to
reasonably exploit especially if your
policy is stochastic in some states and
less stochastic in others this might be
reflective of the fact that you don't
really know yet which actions are
correct in this one state but in the
other state you've seen enough data that
you now know oh now in this state i need
to do this
and then that might sometimes lead to an
appropriate amount of expiration
okay now i'm going to show you an
example of an aliased grid world to show
you that the
stochastic policy can be
optimal in a partial observable setting
so consider this
let's say that you're in this tiny
little grid world where
we start somewhere in the top corridor
and the gray states look the same for
instance it could be the case that you
have features that represent the states
and these features only look at where
are the walls
now
in the top left for instance the feature
for the
wall
above you and a wall to your
left would both be on and the features
for a wall to your right or a wall below
you would be off
and
if we would have this feature
representation note that these two gray
states would indeed have exactly the
same features and therefore would be
fully aliased they would look alike
now in the top corner you can just move
back and forth you can move left and
right and you can move up but you would
just bump into the wall and you could
move down if there's a wall there you
would bump into it you would stay in the
same place but if in if you're above one
of these three other states
you would go into a terminal state and
either you
die or you get the money for simplicity
we could assume that in either case the
episode ends so yeah you either die or
you get the money and you still maybe
die the episode still ends but maybe
then you were happier
now in this setting we could imagine
doing learning but we don't even have to
think about a specific learning
algorithm here we're just going to
consider what the policies
uh
should be
now and particularly we're going to
compare deterministic and stochastic
policies
now here's an example of a deterministic
policy
if you're above the money you go down if
you're above the skull and bones you
basically can see by looking in which
corner you are in which direction you
should go you should always move away
from the wall obviously if you're in a
corner never go down because that's bad
you don't need to go up because that
doesn't help you also don't want to bump
into the wall so instead you move the
other direction
if you're in the top right
in this specific case this would be the
optimal thing to be doing and in fact
let me point with the mouse you would
maybe start here then you would move
over here and then you could move over
over here and then move down so this
would be an optimal path right if you
would start at the top right corner
you'd move left left down and you win
the game
however
let's say that you either begin in this
corner or in the other corner randomly
every episode
if this would be your deterministic
policy because it's a deterministic
policy and because these two states look
exactly alike as far as the policy is
concerned if this is all that it knows
it cannot distinguish between these two
states so the policy must be the same in
both cases
so this means that if we start over here
in the top left corner you would move
right but then you immediately move left
again under the same policy
it could also of course be the case that
in this gray state you would move right
instead of left but that means that in
the other gray states you'd also also
move right and you'd have the same
problem over here
so you can see here that essentially
under state aliasing and as i mentioned
this is the common case right if you do
function approximation and if the world
is complicated you cannot assume to have
a fully observable representation
and then we can see that an optimal
deterministic policy
will either move left in both grey's
tastes or will move rise in both gray
states and neither policy is optimal
because there are episodes in which you
get stuck indefinitely
and then you would reach the money and
your average reward wouldn't be very
good
so what else could we do well
what if we would have a stochastic
policy that randomly moves left or right
with equal probability in each of these
gray states
then what would happen is let's say that
again we started the top left corner and
we move right
it might be that here you move left
again
but then you can just try again you
could move right again and eventually
you'd pop out at the other end and you
just go and grab the money
so with this stochastic policy
if you start in
one corner or in the other it doesn't
matter in expectation it will take you
the same number of steps to get to the
goal and reliably every episode ends by
getting you to the to the gold in the
middle
and you would never end enter any of the
bad states but you would also never get
stuck indefinitely and you could
calculate you could assign
you could actually assign rewards to
this for instance you could just assign
a plus one to when you grab the money
and maybe or maybe a minus one per step
and then you could calculate that this
stochastic policy has a much higher
average return
than the other one
so this is just an example to show that
sometimes having a stochastic policy can
be beneficial because even the optimal
policy could be stochastic
so importantly the last two points here
is that the stochastic policy can be
learned if we run the policy parameters
directly instead of just learning value
functions
and in addition to this
note that this this is just an example
in which we happen to have equal
probability across actions in some of
these states but the example can be
extended to having non-equal probability
so i'm not saying you need uniform
policies in some states necessarily
that's just in this example where you
uniformly move left or right but know
that even in this example the policy is
actually not completely uniform because
you move up or down with zero
probability in those states
you could also have situations problems
in which it's optimal to pick one action
with say 75
a different one with say 12 or whatnot
and uh it could be that the stochastic
policies arbitrarily uh
stochastic where it could be almost
greedy or it could be very uniform or
anything in between
so this is important to know because
this is different from random tie
breaking with values if certain actions
would just have exactly the same value
you could still break ties randomly then
and have a stochastic policy this is
different because you don't in that case
you could only pick between
deterministic or random
randomly picking uniformly randomly
picking between actions here you could
have different trade-offs
okay and now we're going to formalize
the policy learning objective which will
allow us to then derive concrete
algorithms that can help us solve these
problems
so the goal
at high level of course is just to find
a policy
and if we parameterize the policy with
just some parameters thetas then of
course this translates into the goal of
finding these parameters theta
but how do we measure the quality of the
policy
well in episodic environments we can use
the average total return per episode we
can basically just look at all of the
episodes we've ever seen right average
all of those returns and say well how
good the policy was was
um the average return of those episodes
if we would have kept policy fixed
in continuing environments that do not
have terminations we could use the
average reward per step and that seems a
reasonable choice where this is a
well-defined quantity
in a continuing environment note that
the
episodic value could be infinite because
you basically are in one very long
episode that might never end but the
average reward is still well defined
so let's formalize that and let's first
start with the episodic return
so we're going to introduce some
function
j which we subscripted here with g
where g
corresponds to the return
and that's why there's the letter g
there
and the definition then of the objective
could be this expectation
where the expectation is taken over a
potentially random distribution on the
start states so we sample some s0 from
some random distribution d
d0
and then from that state onwards we take
actions according to our policy
so the actions are random maybe because
the policy is random and of course in
addition to that but implicit in the
notation there will be some randomness
due to the mark of decision process so
the transitions themselves might be
random
the expectation just folds all of that
into one so note that the expectation is
not conditioned on anything else it's
conditioned on the distribution of star
status start states implicitly on the
mvp and then on the policy
and then our goal is just to maximize
the discounted return
from the starting state right so we
basically only look at this initial
state and then we are going to roll
forward throughout the whole episode
until it ends there's a summation there
to infinity
because we're basically just going to
assume that whenever you hit a terminal
state all of the rewards from there on n
are zero
or equivalently that your discounts
might be time varying so here we have a
constant discount which is just to the
power t
alternatively you could write this down
with a time varying discount and then
the discount on termination could be
taken to be zero
and those would be equivalent so the
summation is in tune to infinity but a
lot of that time is spent in an
absorbing terminal state where the
rewards are just zero
so alternatively you could also just
think of this as a finite sum but
for mathematical convenience you're
writing it as an infinite sum
this can of course be rewritten as the
expectation of the return from time step
zero where we
consider every episode to start at time
zero for convenience
now we can um
write this up maybe a little bit ex more
explicitly by
splitting out the expectation so let's
now as an outer expectation just
consider the expectation which is due to
speaking the starting state
and then the inner expectation is
already conditioned on that starting
state
and so basically we're writing here that
s0 is now a random quantity
and we're saying well if we're just
going to imagine a different random
quantity st that is equal to s0 when
that is the case the return from that
time step onwards so from st onwards
which is gt
won't be due to the distribution for
starting state anymore because that
one's already is already conditioning on
that state
and instead it's just a return that is
due to your policy and the markov
decision process underpins it but this
quantity is a familiar one that is
actually the value of state zero
so that is the definition of the value
that if you consider a an arbitrary time
step t
and you consider the expectation of the
return conditioned on
uh st being whatever state you're
interested in and then the return gt
that would be exactly the definition of
the value function so this is the value
of this random state s0
where again d0 is the star state
distribution
so we can see that we can write this
objective in multiple different ways but
effectively what we're just doing is
saying hey we want to optimize our
parameters theta in such a way that the
actual value v
of the policy that is parameterized for
theta
will be
maximized under the distribution that
generates the starting state
distribution for simplicity we could
consider a special case here where the
starting state distribution is a dirac
it's a deterministic process which
basically always picks the same state
then our objective is just the value of
your starting state
feel free of course as always to pause
the video whenever you want to reflect
on this a little bit and i'm gonna move
on
note um
that if we want to write down the
average reward objective
this at first glance looks a little bit
simpler but there's some subtleties here
so we're going to
expand this a little bit as well so the
expectation here is no longer over a
start state distribution because we're
considering a continuing setting now
where we're just going to take actions
indefinitely long
and it's never going to stop you're
never going to start a new episode and
we're just interested in the average
reward
long-term average reward
this can be considered an expectation
where the state is now
drawn from a different distribution
which is the distribution that you're in
a state under this policy and implicitly
again in the markov decision process
this is the long-term probability of
being in a state so basically think of
it this way even in a continuing setting
you might still start in some states
right according to some distribution or
maybe deterministic in some state
but if it's a continuing setting and
you're going to be in there indefinitely
long
under some mild assumptions there's
going to be some frequency in which you
visit states and this frequency in the
long term won't depend on where you
started it will just depend on your
policy and the dynamics of the markov
decision process
often
so people will assume things such as
that the markov decision process is
mixing or
ergodic as it is called which
essentially means that this distribution
exists
and that you basically always can
recover states that you visited before
that the mdp is in some sense connected
and if that is the case then this
distribution is well defined and it will
basically just be the frequency of how
often you are in each state
so we can consider this expected reward
this average expected reward we can
consider that to be
an expectation where the starting state
is drawn or the state you're in is drawn
according to this what they call
steady-state distribution
and then in that state condition on that
state we're going to draw an action
according to our policy and observe the
immediate reward and that's the thing
that we're interested in now
we can write this differently with a
explicit summation where we are summing
over states and we're looking at the
probability of being in each state
uh under this stationary distribution of
course if the state space is continuous
we could just write an integral there
and everything would be similar
and we're summing over action so in each
state we're looking at how likely are we
to be in that state
then we're going to look at how likely
are we going to pick each of these
actions given that we're in this state
and then we're going to look at the
probability of picking uh if after
picking this action of getting each
reward
and then you basically just multiply
that with the reward
so this is just writing out this out
this expectation very explicitly in a
summation
and now we're going to talk about how to
optimize either of these objectives and
to do so we're going to use
gradient-based algorithms which are
therefore called policy gradients
so policy-based reinforcement learning
is an optimization problem we want to
optimize something we want to find the
theta that maximizes this j theta where
j theta is one of the two objectives
that we just defined before or maybe you
could come up with other variations that
you might like
we will focus on stochastic gradient
descent because this is a powerful
method which is often quite efficient
and it's easy to use with deep neural
networks which are also a very good tool
in this context
there are approaches that do not use
gradients
like hill climbing or simulated
annealing or genetic algorithms or
evolutionary strategies but we won't
consider them now we're going to
consider gradients
and the policy gradient is then simply
defined as
updating your parameters theta
in a way that corresponds to the
gradient so we have this gradient here
of j
and i'll talk more about how how that
looks where do you get this what does it
look like and we're basically going to
update theta so delta theta here refers
to the change that we're going to do to
theta
with some small step size times this
gradient you could use more advanced
mechanisms i mentioned this in some
earlier lectures as well you could
imagine using newer optimizers like
rms prop or adam or aidagrad
instead of using vanilla stochastic
gradient ascent in this case not decent
but ascent
but it's very similar and we won't talk
about those specific things these are
choices you could always do whenever you
have a gradient you can always transform
this gradient in a way to make the
optimization more efficient but for
simplicity we're just going to focus
only on the uh
the pure gradient-based algorithm
here on the right you see some pictures
of how the lost landscape might look and
how then the gradient algorithm might
work it traverses these lost landscapes
which are typically implicit so you
basically get some local information of
what the gradient looks like and then
you're going to move up or down
according to that the gradient will
always point in the direction of
steepest descent locally where you are
right now
now this gradient is just a vector
which takes the partial derivative of j
with respect to each of the components
within theta so theta are the parameters
of our policy right so the weights of a
neural network that represents a policy
for instance and this will just be a
vector with partial derivatives with
respect to each of these individual
weights
and alpha is a step size parameter
typically a small number so that we make
small incremental steps but we will take
many of those and eventually we'll get
to
in this case higher values of j
again i mentioned this before but here
this becomes important stochastic
policies can help ensure that j theta is
actually smooth
this is the case because the way j
depends on your parameters we want this
to be smooth because then the gradient
will point
reliably in a good direction maybe more
reliably so than if it's discontinuous
and if the policy itself is
parameterized with uh probabilities that
means that a small change to your
parameters will actually also mean a
small change to the value
because we're not switching all the way
from one action to the other
instead we're just switching slightly
with the probabilities
of selecting one action rather than the
other
but how are we going to compute this we
didn't really answer this question yet
so now we're going to go into there
so we're going to assume first this is
important that the policy itself is
differentiable almost everywhere for
instance it's a neural network i say
almost everywhere because oftentimes
these days people use neural networks
with slight discontinuities like
rectifier linear units and that's not
too bad that doesn't really matter too
much but you basically want a
differential function something that
itself is smooth
and for the average reward then we want
this gradient
but this raises the question so how does
the expectation of r actually depend on
theta
it's not immediately obvious and we'll
dig into this a little bit more in this
lecture
so we're going to start
for simplicity in the contextual bandit
case
so consider now a one step
episode
such that the average reward is well
defined
and we are talking about the average
reward but we're basically going to only
be interested in the
reward because we can assume now that
the distribution of states does not
depend on your policy
this is why we go for the contextual
bandit it makes some things a little bit
easier because of course
normally our distributional states will
depend on our policy
but if we're uh in a contextual bandit
and if in addition the state itself is a
pure function of the observation so it's
not
it's not a parameterized agent update
function but for instance the state
could just be the observation
then uh this the distribution of those
states does not depend on your policy
that's
that's a property of the contextual
bandit so it's a more limited case and
we're starting there because it's
simpler to reason about
so the expectation here is over actions
and over states but the distribution d
does not depend on then depend on pi and
that's important
later we will consider the case where it
does depend on pi so this is just a
temporary assumption to make it a little
bit easier to understand
so what will happen is we'll see some
context s
this will be out of our control but then
we want to pick an action and then we'll
see a reward r which depends on the
state and the action and then we want to
optimize the policy such that the
rewards become higher
we can't just sample the reward and then
take the gradients because the reward is
just a number doesn't depend on theta
and we saw this before in the second
lecture but we're just going to step
through this again to make sure that we
fully understand this case
so we're going to use this following
identity which we've derived and i'll
drive again on the next slide where the
gradient of the expected reward turns
out to be equal to the expectation of
the reward times the gradient of the
logarithm of the policy
i'll prove this on the next slide and
the importance of this equality is that
so this is a true equality right is that
the thing on the right hand side can be
sampled whereas the thing on the left
hand side you can't just sample the
reward as i mentioned and then take the
gradient because the gradient of a
number is just zero so that doesn't work
because we're not taking into content
how the expectation depends on the
parameters
but if we can rewrite this as an
expectation of a gradient then we can
just sample this expectation and get an
unbiased estimate for the gradient that
we're actually interested in
and this will give us concrete
algorithms
this idea was introduced in the context
of reinforcement learning by ron
williams and he called the algorithm
reinforce
okay so now let's re-prove again
introduce a little bit of notation let's
let's uh
call little rsa
the expected reward given that you've
taken action a in state s i see there's
a slight typo on the slide there big a
equals small s should be big a equals
small a
but
otherwise it should hopefully be clear
and then we're just going to write out
this expectation first to derive that
this equality on the previous slide is
true
so the gradient of the expected reward
will just be the gradient of the sum
over all states
let me point
and then the probability of him being in
that stage which i mentioned again we're
in the contextual banded case here this
probability does not depend on our
policy at all
times the summation of actions the
probability of taking each action and
then the expected reward given that
you're in that state and you've taken
that action
now we can just push this gradient in
through the summations all the way until
it hits the thing that does depend on
their parameters which is
our policy
and i've just rewritten it in a way to
push it all the way to the right hand
side to make clear the gradient only
applies to this last bit
and then we're going to do the score
function trick
or the log likelihood trick as it is
also known we're going to multiply
by the probability of picking the action
according to our policy and also divide
by this so we're effectively just
multiplying by one right so this is
exactly equal
there's no approximations happening here
we're just multiplying by one but we're
writing out this one
so that we can write out this as an
expectation again because now we have
the probability of selecting the action
back which means we can rewrite this
again let's reshuffle it first and we
see something very similar to what we
had before here a summation of our
states with the probability of being in
that state and then the summation over
actions with the probability of picking
then that action and then some term
behind there which depends on the state
in action this means we can now rewrite
this as an expectation
and it'll be the expectation of the
reward rsa times the gradient of the
logarithm of the policy
we have derived this before in lecture
number two and we're just rederiving it
here for clarity because it's an
important step and it's important to
understand where this comes from
so we've proven this equality in the
previous slide so i'm just putting it
here on the top of the slide and now we
have something we can sample and then
our stochastic policy gradient update
can be this update where we update our
parameter theta by adding a small step
size times the reward times the gradient
of the logarithm of the action that we
selected
80.
in expectation this is an unbiased
algorithm
and therefore this is pure unbiased
stochastic gradient ascent we're going
up we're not going down right we want to
increase our values we don't want to
decrease them but it's very similar to
stochastic gradient descent we're just
going in the other direction
the intuition if you look at the update
is if the reward is high
you will change the parameters such that
the policy
goes up or actually more specifically so
that the logarithm of the policy goes up
but the logarithm is
is itself
an increasing function it's a
monotonically increasing function so
increasing the logarithm of
the possible probability of selecting an
action is equal to
increasing the probability of selecting
that action itself
so it's good to stop there for a moment
and to think that through whether you
see why that is the case um i mentioned
before in lecture number two as well
if all of your rewards are positive this
means that whatever action you select
you will make that action more likely to
be selected
whenever you actually perform this
update it actually turned out that most
of the time increasing the probability
of one action means decreasing the
probability of the other actions right
so if the
rewards are all positive if you perform
this specific update
you would always push up the probability
of selecting the action that you
selected
however if the rewards are not equal for
all actions you would push up the
probabilities of actions with high
rewards more so than the probabilities
of actions with low rewards and in the
limit then you would still find the
actions with the highest rewards
however that maybe is a little bit
unintuitive and now let's introduce a
little trick to reduce the variance and
this will also make intuitive sense in a
moment but let's first define it
mathematically we can pick any b which
doesn't depend on your actions and then
we can note that if you multiply b with
the gradient of the logarithm of your
policy
we could go through these steps
where we can first write out
that this is an expectation over
actions and states so let's first pull
out the expectation of the actions the
the state is still random here
but the action is now just written out
this this this part of the explanation
is now written out explicitly
and then we just notice then
that we can
do the inverse of the log-likelihood
trick
and basically note that the gradient of
the logarithm of something is via the
chain rule the same as
one divided by that something times the
gradient of that something so that means
there's basically a way to rewrite this
as the gradient of pi divided by pi
which will cancel out with this first pi
and then that means we get this we just
get a gradient of each pie
feel free to step through that more
carefully
by yourself on paper but it's exactly
the inverse step of the thing we did
before with the score function trick
we're just using that same trick in the
opposite direction it's good to convince
yourself that this is true this step
from this one to this one
but then we notice that putting the
gradient outside of the summation now
that the summation is by definition
equal to one because this is a policy
it's a well-defined probability over
actions but that means that we're just
taking the gradient of one
the gradient of a constant is zero so
whatever b is doesn't matter we're going
to multiply with zero
this means that this whole expectation
is zero
it's good to convince yourself that this
is true
and we're going to use this fact now in
later slides and this is true when b
does not depend on the action but it can
actually depend on the state so in the
derivation above here i just had any b
but it doesn't depend on state
necessarily but actually if you make b
depend on state you can still do the
exact same steps everything goes through
so b is allowed to depend on state it
just is not allowed to depend on actions
for this to be true
this implies that we can subtract the
baseline to reduce the variance so what
would this mean well effectively what
we're going to do is we're going to
allow
something to be part of the update which
won't take change the expected value of
the update but it's allowed to vary per
state and that's important it's kind of
a covariate i'll talk more about this in
the next lecture but by picking this
smartly you can pick something that
actually reduces the variance of the
updates and we've already derived above
here on this slide that it won't change
the expectation so we're still doing a
valid stochastic gradient ascent
algorithm
the only difference is that the variance
might be lower
which is of course a benefit
now intuitively this also makes sense
because as i said before all your
rewards might be positive right
or let's do a different example let's
say that the reward is plus one
if you win a game and it's zero
otherwise
the algorithm on the previous slide let
me put it up
would actually only update when you win
because if the reward is zero your
parameters will not be updated
this is kind of okay but it means that
if you if you lose a lot you will
basically not learn anything from those
losing games right it's only whenever
you win that you learn to change your
policy in the direction that will help
you
uh improve
what is effectively happening there is
that this is a very high variance thing
if if you only win very occasionally
like one percent of the times you're
only ever going to update your policy
parameters one percent of the times
so this is this is um an example of a
high variance update because most of the
time you're not doing anything and
everyone so often you're doing an update
if you would introduce a baseline you
could do something else where for
instance you could pick the baseline to
be equal to say a half if it's equal to
a half that means that whenever you win
you up table whatever you lose you also
update but in the opposite direction
and this means you can now also learn
from the games that you lose
we haven't actually changed the expected
value right in expectation we're doing
the exact same thing the only thing that
we've changed is the variance of the
updates but it makes a real practical
difference
we will use this fact about the baseline
more often in proofs below it's a
generic fact it's useful to to be aware
of
and now to make this a little bit more
concrete let's consider an example we
consider the softmax policy on some
action preferences where similar to the
book
by
rich suss and andy barto i'm going to
use h to refer to some
preference of an action it's just a
number
i'm using h rather than q to make it
clear that these are not predictions
those don't correspond necessarily to a
prediction of some return it's just a
preference
and then we can define a policy we can
parameterize this policy by basically
parameterizing h
although i've suppressed this from the
notation in his updates and a
parameterized policy could just be the
exponentiation
of each of these preferences and then
taking the one corresponding to the
action and dividing it by the normalized
by the normalization term
so note that the division here by the
summation of the exponentiated action
values implies that the total sum of
this thing over all actions will be
equal to one this is a well-defined
policy the thing that we're dividing by
is simply a normalization term to make
sure that that we sums to one
then if we take the gradient and i
encourage you to go through this
yourself and to check that this is true
it turns out that the gradient of the
logarithm of this quantity will look
like this where we will have the
gradient of the preference for the
action that we've selected so this is
for a gradient of logarithm of the
policy of a t
in st
and this will be equal to the gradient
of the preference for st80
minus
basically the expected gradient under
the policy so this is the gradient of
all the other actions including the 180
but also all the other ones
and this just turns out to be the grad
log pi term for the softmax
okay and now we're going to um
go into the sequential case and we're
going to look at the policy gradient
theorem which is a generic theorem that
proves
what policy gradients can look like and
how you could use them as an update
so basically we're going to go now to
the model markov sorry the multi-step
markov decision setting and the
difference is now that the state
distribution of the states we actually
end up in will now also start depending
on our policy this was different from
the contextual bandit case
and we're basically not going to
consider the immediate reward anymore um
in the contextual bandit case only the
immediate reward depends on your policy
but the next state doesn't so you can
basically simplify things a little bit
but now we're going to go back to the
full case in which not just the
immediate reward but also your next
state depends on your uh
action and therefore in your policy
and this will make things slightly more
complicated i'm reminding you that
there's these two different objectives
the average reward return per episode
and the average reward per step
the average return per episode is
applicable to episodic problems so
whenever you have an episodic problem
that's basically the one you should be
using but if you have a non-episodic
problem in which there are no terminal
states there are no terminations then
the more appropriate objective is the
average reward per step
because then the return per episode is
simply undefined
you shouldn't use the average reward per
step for an episodic task and let me
give you a very simple example for why
that might be the case let's consider
you have a maze and you get a minus one
reward on every step
now your goal is to exit the maze as
quickly as possible the minus one reward
is basically a penalty for every step
and that means that the optimal policy
according to the average return per
episode would be to exit the maze as
quickly as possible so a policy would be
better if it exits the maze faster
because the episodic return would then
be
higher
so if you can exit the maze in say three
steps that will be better than exiting
in five steps because the return in the
first case is minus three and the return
in the second case would be minus five
however if in this setting we would
consider the average reward per step
nothing matters none of your policies
will differ you will always just have a
minus one per step because we've
literally defined the reward to be minus
one on every step
so that is an example to show that you
shouldn't use the average reward as an
objective for an episodic task the
inverse is also true if you have a
continuing task with no episode
terminations then you shouldn't use the
average return per episode because you
will only ever be in one episode it's
not a well-defined objective for that
case and instead then you should use the
average reward for step
we're going to start in the episodic
case and here is the policy gradient
theorem for the episodic case so now
we're in the full mdp setting
and the theorem states the following so
we're going to have some differentiable
policy pi with parameters theta
we're also going to have some initial
starting state distribution d so every
episode starts somewhere where the
episodes start starts does not depend on
your policy right
the trajectory dirty episode depends on
your policy actions you select along the
way depending your policy but when you
terminate you transition back to some
starting state distribution or maybe a
deterministic starting state
and that state does not depend on your
policy because you haven't yet taken any
actions in that in that episode
so do we have this d0 which needs to be
given
and our objective will be
as written here the expected return
where the expectation is hidden from the
notation but the expectation depends on
the mdp dynamics and on your policy
and its conditions on the starting stage
being samples from the starting state
distribution
now if we have this objective
then it turns out that its gradient can
be written as follows
where we're going to slowly unpack this
so we have an expectation here
under your policy
and it's also conditioned on the
starting state being sampled from the
starting state distribution
and then we have a summation over the
whole episode so t here big t is the
last
step in the episode so we have an
episode here that lasted big t plus one
steps because we started at zero and we
ended big t
and then we
sum over these terms in the episode
where there's a gradient to the power t
i'll get back sorry another gradient a
discount to the power priority gamma to
the power t i'll get back to that in a
moment you can ignore that for now and
what we see here is the value of the
action that you took on that time step t
times the gradient of the logarithm of
your policy so this looks familiar this
looks similar to the contextual case but
we're summing now over the whole episode
and there's this weird gamma to the
power t thing which we'll i'll get back
to in a moment
the value here q
is just defined as usual it's your
discounted return
from
that point in time
so this is the policy gradient theorem
it basically says that if you would walk
through an episode and you would
accumulate all of these terms within
this sum and then at the end of the
episode you apply these to your
parameters this would give you an
unbiased estimate to the policy gradient
to the actual policy gradient
now
you might think okay but it's a long
episode maybe i don't want to accumulate
everything and wait all the way until
the end of the episode and only then
apply it so what people often do is
instead of summing it over the whole
episode and then applying this thing at
the end you could also just look at
every single step within the episode at
this term and just use that to update
your parameters
then you get a slight bias to your
gradients
because it might be that you end up in a
certain state because your policy was a
certain way but then you update your
policy in such a way that you would
actually never run up in that state in
the first place
and then you continue if you continue
updating from that point in the policy
you would have a slightly biased
gradient with respect to your current
policy but it's kind of okay people do
this all the time and it's quite common
to update during the
journey episode i just want you to be
aware that in your policy grading
estimate will be slightly biased
similarly this term here this discount
to the power t
this basically means that the further
you are in the episode the less you're
going to update your policy and this
makes sense because if you're an
episodic task but you also still have a
discount in some sense the farther you
are from the starting point according to
this objective the less it matters
because we are considering a discounted
objective
so one could argue that maybe in the
episodic case the most natural thing to
do is not to discount at all because
your episode is going to end in finite
steps anyway
and then this term would also disappear
in practice people do discounts because
the algorithms they tend to work a
little bit better if you have a discount
factor it's easier to estimate the
values and things like that but people
often drop this term discount to the
power t
and turns out the algorithms typically
still work well but i do want you to be
aware that that will give you a biased
gradient and then sometimes it can
actually point in the wrong direction in
some uh in some educate cases
i am going to prove this that this is uh
true this statement
but we first first before we do we're
going to point out that actually the
policy gradient does not need to know
the mark of decision process dynamics
you don't need to know the transition
dynamics and that's actually a little
bit surprising should we know how the
policy influences the states and
actually you should
but it's captured and it's captured
implicitly here in this value estimate
so this value estimate does capture how
your policy influences states but i will
now also prove this statement so we can
go through this and we can see how this
drops out why we don't need the dynamics
so before we do let's introduce a little
bit of notation we're going to introduce
tau
to be a random variable which captures
your whole trajectory so tau is defined
as is just notation as the initial state
and then the action in that state the
reward then the next state and so on and
so on and then we can write the return
as a sum sorry as a function of this
random
trajectory now i will pause it without
proof but it's fairly easy to prove this
that the
gradient of your objective which is
defined as this term the gradient of the
expected return
will be equal to the expectation of the
return times the gradient of the
logarithm of the probability of the full
trajectory this is just using the score
function trick feel free to write it out
step by step for yourself
but we're basically just considering the
whole trajectory in one go and we're
just saying okay so how what how can we
write this expectation here we could
write it as an integral over all
possible trajectories or sum over all
possible trajectories
and then
in that sum we could have the
probability of each trajectory times the
return if you saw that trajectory and
then we can just use the score function
trick as we did before or the log
likelihood trick if you want to call it
that
and we get this term on the right hand
side
but we're not done yet because now we
have this complicated element there this
probability of the object of the
trajectory so what is that let's unpack
that a little bit
so the gradient of the logarithm of the
probability for trajectory will be equal
to the gradient of the logarithm of
this probability written out so what
we've done here is basically just taking
the probability of the trajectory and
we're going to look at what this means
so the probability of a specific
trajectory happening
is equal to the probability of the
initial state in that trajectory
happening times the probability of the
action that you took in that state
times the probability of then
transitioning to the next state that you
actually saw in this trajectory tau
given that you took that action in that
state
and so on and so on and so on
so note that these are all probabilities
that are all values between 0 and 1 this
means that this total multiplication of
things is probably a very small number
but that makes intuitive sense because
the probability of any specific
trajectory is also probably going to be
very very low there's many different
trajectories that could have happened
you happen to see one specific one this
is the probability of that specific
trajectory happening
now we notice that we have a logarithm
of a product
we know from the law from the rules of
how the logarithm works from what the
definition essentially of the logarithm
that a logarithm of a multiplication of
things is equal
to the summation of the logarithms so we
can basically push this logarithm inside
this multiplication and turn the
multiplications into summations
and now we're inspecting this thing and
we can see that it's basically the
gradient of a big sum
so the gradient of logarithm of a
product is equal to the gradient of a
sum of logarithms
but now some of these terms
interestingly do not depend on the
parameters of our policy the very first
term is just the probability of starting
in the state s0 which we called d0
before and that does not depend on our
policy parameters so the gradient of
that will just be zero
so the gradient of this thing will be
relevant this one this one stays around
because pi does depend on our policy
parameters but then next the probability
of transitioning to a state s1 given
that we were in state zero
and to action zero does not depend on
our policy parameters because we're
already conditioning on the action
so interestingly there's a couple of
these terms like the initial start
distribution and each of the transition
terms that do not depend directly on our
policy parameters so the gradient with
respect to those will be zero
the grading with respect to the next
policy step so the probability of
selecting action a1 in s1 does depend on
our policy parameters again so that one
sticks but we can
get rid of all the other terms of which
the gradient is zero anyway and just
write it like this gradient of a
summation of the policy parameters sorry
gradient of the summation of the
logarithm of the policy
so we just
plug that in we had this equation up
here which was the expectation of the
return
for that trajectory
and we just replaced this the gradient
of the logarithm of the probability of
the trajectory with this summation of
the logarithm of the probability of
taking each action along the way so we
see that the dynamics of the mvp do not
have to be estimated here directly
because they drop out we don't need them
for the policy gradient
we're going to continue a little bit
farther so this is the same equation as
at the bottom of this slide
and now we're going to write it out even
farther by first pushing the the
summation to the left hand side so
outside of the whole term
and we're going to plug in the
definition of this return what is this
return well the return is just
a discounted sum of rewards right this
is just by definition the return of the
trajectory is just the discounted sum of
all of the rewards within that
trajectory
now we notice that we basically have a
nested sum
we have a summation from t 0 to big t
and the summation inside or from k zero
to big t
and we have this grad log pi term
but now i'm going to recall we talked
about base lines earlier and we said
that if something doesn't depend on your
actions
then actually
this thing times grad log pi
the expectation of that will be zero
this means
that all of the rewards that happened
before
for each of these time steps all of the
rewards that happened before that time
step are uncorrelated with this
probability of taking that action
the action cannot influence
those
rewards because they happened
in the past essentially and it turns out
the expectation of all of the rewards on
all of these previous time steps will
just be zero according to a very similar
derivation as we did for the baseline
so you can write this out step by step
if you wish and convince yourself that
this is true but it basically means that
the inner sum we can start this at t
rather than zero and the expectation
will be exactly the same
and now we note that we can then pull
out maybe the discount factor
because
there's a there's something here that
looks a lot like the return from time
step t but there's too much discounting
happening
so instead we can start basically this
summation at time step t but only start
discounting at that time step so note we
start now the summation is now starting
at time step little t right k is little
t
and the first reward we're not gonna
discount and the second reward we're
gonna discount once and so on and so on
in order to do that we need to pull a
term out which is equal to gamma to the
power t
and then we can finally rewrite this
this summation is just equal to the
return gt
because we pulled out the discount to
the power t
this term inside is just your reward
from time step t plus one
plus the discounted reward at t plus two
plus the twice discounted reward at t
plus three and so on and so on which is
by definition the return so by doing
these steps we can rewrite the thing
that we had at the top
do something that has this discount to
the power t and the return at time step
t
and the reason to do this why would we
do it like this why didn't we just stick
to the original thing which was the
return at time step zero
is that this is um
this is also possible but it could be
higher variance because we've basically
gotten rid of some terms on this step
that would just maybe just add to the
variance but don't not necessarily help
us
get a better estimate
so it's just to rewrite the thing at the
top
and then we get basically the equation
back that we had before except that we
now have gt rather than
q
but
because we're within the expectation we
can just replace the random return gt
with the expected return
q pi
and this is going to be in expectation
equal of course if we would have q pi we
should just use that because it's much
lower variance than g
in practice we could estimate g
q pi and use that instead of g but then
you do run the risk that you are going
to bias your policy gradient because
your estimation might be a little bit
off and then you're not guaranteed
anymore to follow the true
policy gradient so instead of using q pi
in practice you might actually prefer to
use g
because then at least you get an
unbiased estimate for a policy gradient
as was proven here
so this brings us back to the statement
in the theorem
and now we can sample this if we have a
whole episode
and turn that into an algorithm but as i
mentioned people typically pull out the
summation and you split it up in
separate gradients for every time step
so you basically get some some term on
every time step and if you would add all
of those together
like this then you'd get an unbiased
estimate for the policy gradient if
instead of adding them all together for
the whole episode and then applying them
if instead you apply them on every step
you don't get an unbiased estimate for
the policy gradient but it's typically
still okay and it allows you to start
learning during the episode already
and as i mentioned people typically
ignore this gamma to the power t term
similarly this will bias your policy
gradient if you just scrap that term you
can prove you can come up with counter
examples in which this then does the
wrong thing
but in practice discounts are typically
quite close to one anyway and it turns
out that this is also kind of okay you
can interpret it as an algorithm that
does something a little bit weird
but it's not fully unreasonable
and in practice this is nice because if
you have very long episodes
you actually do also want to learn about
the later part of the episode you don't
want to eventually zero that out
especially because we typically
generalize so for instance you might
have a very long episode
and you might occasionally bump into
states that are extremely similar to the
starting state or maybe even equal to
the starting state if you would have
this
gamma to the power t at some point you
would just stop updating even though
there might be very useful information
later in the episode that you should
just maybe be using and this might be a
motivation to drop the disc contour to
priority
in some sense it would be much cleaner
if we would just drop the discount all
together and consider as the objective
the undiscounted
returns but unfortunately in practice
that doesn't work that well because the
variance is so high
so view the discount here as in some
sense just biasing
the objective and then um doing
something slightly simpler
but it's easier to optimize and maybe
it's not quite what you want because
maybe you're actually interested in the
undiscounted episodic return but it's
easy to optimize in the algorithm tends
to then work a little bit better
so basically this is okay we just
partially pretend on every step that we
could have started the episode there
instead as well that's one one way to
interpret it or you could indeed just
view it as a slightly biased gradient
without interpreting it otherwise
now there's a different policy gradient
theorem which looks quite similar but
it's actually slightly different in
several ways for the average reward case
so again we're going to assume that
there's a differential policy pi
and the policy gradient of the reward
given pi
and this is the long-term reward where
we are assuming that our policy does
also change our state
distribution this can then be written as
follows
as the expectation of the q value
but let me get back to that the normal q
value turns out times the gradient of
the logarithm of pi so it looks quite
similar there's no summation here we do
just do it on step by step basis
but there's a difference in the
definition of this q value
it's the average reward value so what is
the average reward value it's
undiscounted
and it basically
adds rewards together and subtracts the
average reward this is a little bit
weird so it's good to just stop the
moment and think about this so what is
this
well
row here is just the average reward
under your policy
q here is defined as the summation of
rewards
over time but on every step also
subtracting the average reward
so you might think well doesn't that
mean that this is just zero because
aren't we just subtracting the expected
reward from the expectation of the
reward so isn't this just zero no it's
not zero because in the q value we're
conditioning on a specific state in
action and row is across all states and
actions under the steady state
distribution
so the difference here is that basically
your q value captures if i'm in the
specific state or action
is my average reward conditioned on
being in that state in action is it
going to be a little bit lower or a
little bit higher than the overall
average for a little bit so your action
value can actually differ first for
different states and actions because for
instance you might just be just in front
of like a big reward given that you're
taking that action or we might be
considering an action that actually is
has low probability under the policy and
this might have a different average
reward than the total average
um
slight technical note that you can
forget if you want after i've said it
but the slight technical note is that
these equations together do not
necessarily actually define a specific
value
but they do define
relative values so the relative values
are well defined but the system of
equations actually has
um a line of solutions rather than a
point
that's okay it's it's okay for the
updates it's okay for the learning
algorithms but it's a slight technical
point that the lack of discount factor
actually means in this case that the the
value is not 100 well defined
you can easily make it well defined if
you want
but that's kind of like out of scope for
this for this lecture
okay
so
alternatively we can state the same
theorem that we have here in a different
way and this might be slightly more
intuitive in some cases
so the
this is exactly the same objective we're
still considering a differentiable
policy and the policy grade will still
be defined as the expected reward given
that policy and
all of the consequences that his policy
has including on the state visitation
distribution
now if you have the same objective then
you can rewrite the thing that we had
before as the instantaneous reward times
the summation into the past
of the gradient of the logarithm of your
policy
where the expectation is again over
states and actions
the difference here is that previously
we had this value which is essentially a
summation into the future
so we have a summation of rewards into
the future times the gradient of the
logarithm of your policy here we have a
single reward and a summation into the
past
if you remember the eligibility traces
which we discussed earlier when we were
talking about value learning this might
look a little bit familiar and indeed
ron williams called this term
the characteristic eligibility trace
where the characteristic eligibility is
just a name he gave to this gradient of
the logarithm of pi
and then the fact that we're summing
into the past makes it a trace
so this is just an equivalent different
way to write down uh the average ward
case
and um before going to activate it let
me give you one intuition of why this is
the case in some sense this trace
captures the steady state distribution
this trace of these policies going into
the past basically captures how does my
state visitation depend on my policy
parameters
so you can view this as similar to this
probability of the trajectory we saw
essentially we have here
the gradient of the probability of the
trajectory up to that point in time
and that turns out to be via a similar
reasoning as we had before you can write
this as the summation into the past
okay so that brings us to the end of
talking about these policy gradient
theorems and now we're going to talk a
little bit about ectocritic algorithms
so what is an exocritic let's first
recall what the term meant right an
actor critic
is just an agent that has an actor a
policy but it also has a valid estimate
a critic and we're going to talk about
some concrete reasons for why you might
want that and what that could look like
so first we're going to basically reduce
variance
and
we're recalling this property that i
mentioned before that if you have any
functional state b
we're calling it b for baseline
then the probability sorry the
expectation of this b
of function of state s
times the gradient of the logarithm of
pi will be zero for any b that does not
depend on the action
and now we're just going to reduce that
and use that
to reduce variance and a very common
choice for b is just the value of your
policy so we can estimate the value of
the policy we can just subtract that in
the equation
and the expectation will remain
unchanged
this is useful because it means that you
can reduce the variance of the updates
by picking this smartly because it will
vary with states but it doesn't vary
with actions and this allows you to
co-vary with q
in such a way that you can actually
reduce some of the variance in the
updates
so typically we just estimate this
explicitly
and then we can sample
the q value as i mentioned before as
just your monte carlo return
but of course since we have v now anyway
we can also minimize variance further by
instead picking a target that itself
bootstraps
so instead of filling in g sorry so
sorry instead of replacing q pi with the
full return from that moment onwards we
can also do the normal thing where we
take one step or multiple steps and then
bootstrap with a value this will bias
our gradient slightly potentially
especially if you bootstrap after only
one step but it does reduce variance
quite a bit and it can still be a win
we'll talk more about techniques to
reduce variance in the next lecture more
generally and also especially when doing
off policy learning which is going to be
important
for policy gradient algorithms in
practice
for now just keep in mind that this is a
normal thing that people do all the time
they estimate value functions and these
serve a double purpose
first you can use them as a baseline
second you can use them to bootstrap
so a critic is just a value function
learned via policy evaluation which
we've covered at length
and
we've considered monte carlo policy
evaluation temporal difference learning
and sub td you could use any of those in
combination with the policy gradient
algorithms to obtain an extra critic so
then the extra critic is quite simply
you update the parameters w of your
value function with td
or monte carlo and your update your
policy parameters theta with policy
gradients
but then bootstrapping potentially so
here's an example of a concrete
algorithm which is a one-step ectocritic
sorry this first line should have also
initialized w rather than just theta and
the state
so we are in some state s and then we're
going to step through through time we're
going to sample an action according to
our policy we're going to sample a
reward and a next state we can then
compute a one step td error doesn't have
to be one step right this is just a
concrete instance of the algorithm this
which is a one step actor critic we
could also have a multi-step temporal
difference error here or there's
something else
um
sometimes this one step temporal
difference error is called an advantage
because you can argue that this is in
some sense the advantage of taking the
action or a random
instantiation of the advantage of taking
the action that you took compared to all
other actions because this temporal
difference error does depend on the
action that we took
then to update our critic our value
function we can update the parameters
thereof by adding a step size times your
td error times the gradient of your
value function
and quite similarly we can update the
policy parameters by adding a different
step size which we here call alpha times
the same temporal difference error times
the gradient of the logarithm of your
policy
and this is a valuable c gradient update
except for the fact that we're updating
during the episode and we're ignoring
this gamma to the power t term which as
i mentioned before are two ways that you
will slightly buy your policy gradient
updates but they tend to work okay in
practice and they're just a little bit
easier to use you don't have to keep
track of this gamma to the power t term
and in addition by just updating online
you can always update your policy
already during the episode which can be
beneficial if the episodes are really
really long
so this is a concrete instance of an
actor critic algorithm where we have the
actor which is our policy with explicit
parameters theta and we have our critic
which is the value function with
parameters w which are both learned at
the same time
there are many variations of these
algorithms and there's many ways to
extend them in various ways but actually
this
main gist of this algorithm in some
sense underpins a lot of the current day
algorithms that people use so a lot of
deep reinforcement learning in terms of
applications uses algorithms that are
remarkably similar to this one but just
extended in various different ways
so as i said many extensions and
variations exist
and there's one thing to be very
particularly careful about which is that
bad policies might lead to bad data
reinforcement learning is an active
endeavor the actions we take don't just
influence our rewards but they actually
also influence the deity that you get
and that's a little bit different from
supervised learning where we typically
consider the data to be given you can
make bad classification mistakes or bad
regression mistakes but this won't
influence the quality of the data that
you can use to learn later
in policy gradients this can happen
where maybe you make a bad decision on
how you update your parameters and all
of a sudden all of the data you get is
garbage
so we have to be a little bit careful
and one way to do that is to increase
stability by regularization
and a popular method to do so is to
limit the difference between subsequent
policies we basically want to make sure
that we don't update the policy too much
because then we might break it and all
of a sudden your agent is just in the
corner bumping his head against the wall
and the data is not good enough to learn
to do us anything else anymore
so a popular thing here is to use uh
basically this is a difference between
policies
and one popular choice is to cover a
library divergence but you could use
other other things here as well if
you're unfamiliar with this divergence
is not too important the main thing to
keep in mind that it's basically in some
sense a distance
between the old policy and the new
policy and the nice thing about using a
callback library divergence or something
similar
often just called kl divergence is that
it's differentiable
it's an expectation over states of
basically a summation
or in this case i'm written as an
integral because this applies to
continuous actions as well but just
think of it as a summation over actions
of the old policy so this is the one
that you used to have
times the logarithm of the new policy
according to your parameters divided by
the old so what is the old policy it
could just be your current policy but in
terms of computing the gradient of this
regularization term we're going to
ignore that the old policy is also a
function of your parameters
so that means if you're going to move
your theta too much then this code like
library divergence will grow and it will
basically
if you use this as a regularizer it will
try to keep you close it will try to
avoid changing your policy too much
the main purpose of this slide is not so
much to understand the exact
technicalities here but more to get the
gist of the idea here the idea is to
keep your policy from moving too much
and this can avoid bad policy updates
a divergence in general is just like a
distance between the distributions and
we could pick a different one and the
idea is then to simply define our
objective
with a regularization term where where
we have some hyper parameter eta which
determines how careful do we want to be
if we set eta to zero we're back at the
normal thing that we had before if we
set eta really really high your policy
will not want to change at all and for
anything in between you're just changing
your policy as normal but you're
regularizing yourself not to change it
too quickly
there's a lot of algorithms modern
algorithms that use this or variations
of this including trpo which stands for
trust region policy optimization by john
schulman and others
ppo which is a variation thereof mpo and
there's a bunch of other algorithms in
this space that use similar ideas
sometimes directly based on the kl
diversion sometimes on variations
thereof
but the idea is basically just oh if you
regularize yourself maybe you'll be a
little bit more careful with the policy
updates and this might help you get
enough data so that when you do make
your policy updates you are confident
that they are moving in the right
direction
okay now we're going to switch gears a
little bit and i'm going to talk about
continuous action spaces and we can see
how these algorithms that we talked are
actually quite natural to extend to
continuous action spaces
so
pure value-based rl which we talked
about in the previous lectures a lot can
be a little bit non-trivial to extend to
continuous action spaces because how do
we approximate the action value if the
state space and the action space can
both be continuous if they can just be
real valued numbers or maybe even
vectors
what if what if your action is a bunch
of motor controls for a row ball which
can have real valued numbers
and how do we then compute if we even if
we could approximate this how do we
compute this maximization how do we
maximize our actions if we can't just
select from a limited set so there's a
couple of practical problems if you
would consider continuous actions but
actually when we directly update the
policy parameters they're somewhat
easier to deal with
and most algorithms we discussed today
these policy gradient and exocritic
algorithms can actually be used for both
discrete and continuous actions the only
difference is how do we parameterize pi
how do i parameterize our policy so
let's look at an example in a moment
but first before i do i do want to note
that expiration in high-dimensional
continuous spaces can just be
challenging this has nothing to do with
specific algorithms in some sense but if
you have a very high dimensional space
that you're searching in searching in
the high dimensional space in general is
just hard but note if your high
dimensional space corresponds to actions
that you're taking exploration can be
quite tricky how do you pick an action
from this high dimensional space that's
just something that i want to mention
we're not going to go into a lot of
depth here but it is an interesting
research problem and an interesting
issue that we're going to have to deal
with when we want to apply these
algorithms at scale
okay so as an example let's consider a
concrete instance
of a continuous action algorithm and to
do so we're going to parameterize our
policy as a gaussian policy
which means we're going to define some
function of state which will represent
the mean of our gaussian
and for simplicity for now we can keep
it at that and we can say there's some
fixed variance
so for if this is a single dimensional
uh
policy so we have a real valued number
and that will be our action the mean
will just be a number but it depends on
state and it depends on our policy
parameters theta and the variance will
just be a number but we're going to
consider it fixed i'm just noting we
could parameterize this as well we could
have policy parameters that are not just
the mean of the gaussian but also the
variance of the gaussian and then you
could just update them with policy
gradients as well
if we do that the policy will now just
pick an action according to the gaussian
distribution centered around that mean
with the variance that we gave
and notice i'm using mu here for the
mean which is conventional but in
previous lectures sometimes we use mu
for the behavior policy so just be aware
that i'm overloading notation here
unfortunately at some point we always
run out of greek letters
but me mu in this case means the mean
and that's just uh uh what the actor is
is
explicitly representing but note that
the policy itself is random because of
this fixed variance which can be larger
than zero so the action you pick is a
random random quantity
then what does this grad log pi look
like what is the gradient of the
logarithm of our policy well we can just
calculate that for the gaussian policy
this is not too difficult
and if you do that turns out it will
look like this where we basically see
the action that we picked
and the difference between that action
that we picked and the mean
divided by our variance times the
gradient of the mean
so what does this mean well if we
multiply this later on in a positive
variant algorithm with the return for
instance if the return was positive
then this would update your mean towards
the action that you actually took
if the return was negative it would move
away from the mean as the first from the
action you actually took so we're going
to sample this random action and then
depending on whether our our our uh
signal from the critic or from the
sampling the return
depending on whether that's positive or
negative you would either move towards
or away from the action you've taken
and we can just plug this in we can plug
this into a policy gradient algorithm
into reinforce or into an actor critic
and that looks like this where for the
gaussian policy let's say we have a
monte carlo return g we're using that
we're not bootstrapping we could also
bootstrap but let's just say we use
multi-color return we have a baseline v
because you could also always use that
that doesn't change
and then we just have this specific term
which for the gaussian policy it becomes
this as i showed on the last slide
and then we can see that basically now
instead of just looking at whether the
return is high we're basically just
looking did we have a good surprise
right because we're subtracting this
baseline
again the baseline does not change the
expectation of the update but it maybe
makes it in this case a little bit more
interpretable or easier to interpret
um that if your return happens to be
higher than you expected the order
according to your value function then
you move the mean towards the action
that's what this would be doing
now
policy gradient algorithms they work
really well and like i said they
underpin many of the current algorithms
that people use in practice but they
don't actually exploit the critic all
that strongly
and if you have a good critic so if your
value function is very accurate can we
maybe rely on it more can we do
something else
so we're in the continuous action space
right remember
so
we can estimate our action values we can
still do that with for instance sarsa
but we can also then define the
deterministic actions so the action here
is either a real valued number or it
could even be a vector so it could be a
multi-dimensional output of this
function so our policy is now a
deterministic policy
you're just like basically you're just
plugging in your state and out comes
your actions that you're going to select
now there's a thing you could do which
is if you can estimate the value of each
action in each state which we can
basically do by using sarsa or something
like that
we can do policy improvements by moving
the
policy parameters in the direction of
gradient ascent on the value
quite directly
because now we don't have to estimate
this j function that we had before which
was the expectation over all states and
so on and so on now we're just saying
within this state
can we improve the value
by taking the gradient with respect of
this critic with respect to our policy
parameters
but how does this critic depend on our
policy parameters well
you can do the chain rule and we can
just look at how this value depends on
our policy and then how the policy
depends on your parameters
and this algorithm
basically performs grading ascent on the
value and it's known under various
different names
maybe the oldest name name for this
algorithm is
um perhaps slightly awkwardly named
action-dependent heuristic dynamic
programming it's quite descriptive but
it's a mouthful
because the
the idea is that you're doing policy
improvements so in some sense we're
doing dynamic programming but we're not
doing dynamic programming exactly so
maybe we could call that heuristic
dynamic programming and this is the
action dependent version where we have
an action value function that we're
estimating this algorithm is maybe as
old as the 1970s
and it's been described and maybe
invented by paul variables um there's a
nice paper by poholf and wundch from
1997 which also talks about this
algorithm in many variations one little
bit of warning if you're going to look
at that paper the notation there is
quite different from the notation that
we currently use so it might take a
little bit of effort to parse the paper
but it is a really nice paper
i personally also investigated this
algorithm in the context of other
algorithms at some point and i there i
just called it gradient and sent on the
value in 2007
these days most people call this
algorithm deterministic policy gradient
which is also quite descriptive and that
term comes from a paper by dave silver
from 2014.
and note that this is a form of policy
iteration so going back to dynamic
programming it's kind of an apt name in
some sense because we do have this this
notion here of doing policy evaluation
and then using that to do policy
improvement but instead of doing a
gridification step we do a gradient step
we can do this gradient step because our
policy is
in this case just outputting an action
which is directly an input to this
action value function and that means
that we can just pass the gradient all
the way through the action value
function into the policy parameters
of course in practice if we're going to
estimate this quantity if this is
actually q w rather than q pi which
we're going to have to use because we
don't know q pi
if we plug in qw here then this gradient
could be biased and indeed that is an
important thing if you want to make
these policy gradient algorithms work
these deterministic policy graded
algorithms you have to take some care
you make sure that your critic estimates
the values well
and that this gradient is well behaved
because otherwise it sometimes might
just update your action into some weird
situation because the critic just thinks
oh if i make my action higher and higher
and harder i get more and more value
which is not actually true but it might
be what the critic thinks
so you have to be a little bit careful
and maybe think about some stabilizing
steps regularization steps
um
if you want to use these algorithms but
then they can work quite well
now i want to talk about yet another
algorithm so why am i talking about
different algorithms here well partially
just to give you an intuition
and also to show you that there's
actually many ways you can implement
these algorithms and many ways you can
use them and there's not one right way
or one only way you could define an
algorithm
so here's an algorithm called continuous
active critical learning automaton or
kaklaf for short
and in this case we're going to do
something very similar to what we're
doing before but instead of defining the
ac error in
parameter space we're going to define it
in action space so how does the
algorithm work we're going to pick an
action this is just the output of an
actor which is again deterministic
but now we're going to explicitly take
into account expiration
and we're going to sample for instance
uh from say a gaussian policy around the
action
so this is similar to before
but here this is just purely considered
basically an exploration step
where we in some sense add a little bit
of noise but we could also add very
deliberate noise it doesn't have to be a
gaussian it could be anything else you
could just change your action in some
way to deliberately pick a different
action
then we can look at our temporal
difference error similar to what we did
for the actor critic
and then we can update our value
function using this temporal different
error or maybe we have a multi-step
error maybe we're doing monte carlo
something else we just define our
temporal difference error in one way or
the other and we update our value
function as in normal hexacritics
but then to update the actor we're going
to do something slightly different
if the action value was positive
we update the action actor towards the
action that we took
so this is quite similar to what we saw
before with the gaussian policy
there is just a slight difference that
we're not dividing by the variance of
the expiration
in a sense
and
the other difference is that the update
does not depend on our values
right there's no delta here there's no
td error here
and the intuition behind this is that
maybe
in order to decide how much you want to
update your action you don't want to
look at how big the value is because the
value will be in completely different
units from your actions
and if you're going to scale up
your values or scale down your values or
maybe in some states the values are just
higher than others then maybe your actor
will be updated less fast or faster in
some of those states than in others here
instead
we're just going to look at was the was
the uh the action a happy surprise was
it a good thing was my temporal
difference error positive and if so just
update towards that that action
so it's a slightly different algorithm
and if the temporal difference error is
not positive you simply don't update
that's another difference
and why do we do that
well the intuition here is that if you
saw let's say that you you're close to
optimal and you have an actor that
outputs very good actions but you
explore around that then actually most
of the things you could explore around
that would be bad right a lot of the
actions apart from the one all the way
at this top we've done gradient ascent
we're at the top of some some mountain
in value space
and then we're considering an action
that is a little bit away from that
action this will most likely be down
this will most likely be less good than
the action that you're currently uh
than your current proposal from the
actor but that doesn't mean you should
move in the other direction because then
you'll just walk off of the mountain in
the other way so instead of doing
a gradient in some sense here we're
doing hill climbing we're just looking
at which actions work and if they work
we move towards them if they don't work
we don't know that moving away from them
is a good idea right we don't actually
know that that will be up we're just
saying well if they're not good we're
not going to move
and the update of then therefore doesn't
depend on the magnitude of the values
okay now i'm going to quickly show you a
video of this specific algorithm that's
another reason why i wanted to explain
it that shows you can use this algorithm
to do interesting things
so let's go over here
so here you can see also the uh the
author source of paper that is
associated with um the algorithm is not
exactly the cache algorithm they've
extended in various ways um
and you can see that you can then train
these simulated
animals essentially running around
look at
they visualize some things here you
could see it could deal with several
types of terrain let's skip ahead a
little bit
and you can see it jump jumps over
things
there's slopes goes up or down
it can even deal with
gaps and such so this has been trained
this has been learned to do that it's
reminiscent of the video that i showed
earlier as well which was a different
video with a different algorithm but we
can see these extra critical algorithms
can be used and various different
algorithms can be used to do things like
this and to learn them and it's
completely obvious how to do this
yourself right if you would have to
write a policy
that does this in continuous actions
that's actually very tricky
and because it's a learning algorithm it
can also generalize to different body
shapes different terrains so that's
quite cool this is the benefit of
learning right that you can generalize
in this way to things you haven't seen
before
okay it's a longer video but i'll stop
stop it short there
the last slide that's the end of this
lecture
so as always if you have any questions
please do direct them to moodle
and i will see you at the next lecture
which will be about
reducing variance and off policy
learning and multi-step learning which
will turn out also to be important for
policy gradient algorithms in practice
thanks for your attention