RL Course by David Silver - Lecture 7: Policy Gradient Methods

okay so a couple of announcements just
forget about that it's a very
detrimental projector so first of all
the final lecture lecture 10 I sent an
announcement to the mailing list but
just in case you didn't catch that that
will take place the morning of the 1st
of April so this isn't an April Fool so
if you don't turn to the list in your
thinking I don't have to wake up that
day there really is going to be a
lecture
I appreciate that that is outside time
where classes normally occur as a result
this is an optional lecture so it's
going to be a lecture on games case
study on classic board games with
reinforcement learning I think it's
interesting it'll pull together a lot of
elements of the course but it's optional
so if you've already booked your time
away and you can't attend any don't
worry it's not going to affect your your
exam but if you do have a chance to come
along it should be interesting and I'll
also mention that all of these videos
will be made available online very
shortly so if you do miss anything you
can catch up online second announcement
is the deadline for the assignment so
that will be the following day April 2nd
so hopefully you guys have made a start
on that if you do have questions please
send them to the mailing list because if
you're wondering about something
probably means 10 other people have the
same question so it's quite useful just
to clean those backwards and forwards on
the mailing list to just make sure
everything is crystal ok so let's get
going so today's lecture is on policy
gradient methods this is a very popular
widely used class of reinforcement
learning methods again we're in the part
of the class where we talk about things
which are actually used in practice to
solve interesting challenging
reinforcement learning problems so just
to give a an outline of how this is
going to pan out we're going to start
off with an introduction where will
understand roughly what's going on and
then we'll talk about we're focusing on
policy gradient method so what's going
on in this class is we're really talking
about methods that optimize the policy
directly so instead of working in value
functions as we've considered so far
this is now about working with the
policy how can I see some experience and
from that experience figure out how to
change my policy in a direction that
makes that policy
better and what we'll do is we'll see a
variety of different methods for doing
this which are all based on the simplest
principle if you like for improving a
policy which is policy gradients so we
adjust the policy in the direction that
makes it better we follow the gradient
will consider three different ways to do
that and we'll start off with a sort of
de nieve but sometimes effective
numerical approach by using finite
differences and then we'll move on to
see that actually for many of our
reinforcement learning methods there is
an analytic formulation for the policy
gradient so despite what you might think
you can actually just have an agent
which follows its trajectory sees how
much reward it gets along the way but
also knows which direction to adjust its
policy so as to move it directly in the
direction that makes it is positive data
and finally the most practical set of
methods which we'll consider the act
critic policy gradient methods so these
are methods which combine everything
from this class with everything from the
previous class so they have they work
with both value functions and with
policies and try to get the best of both
where else
okay so introduction so last lecture we
had a lecture where we talked about
value function approximation we consider
two different types of value function
approximation we considered
approximating the stake value function
so this is the true state value function
that tells us how much reward
can I really get in this NTP in
expectation if I follow this policy PI
SOPA in some state s what's the true
expected reward I'll get from that point
onwards all we have the key function
which tells us for some state s and some
action a what's the true cumulative
accumulated reward along some trajectory
that I can get from that point onwards
taking that action under this policy so
those are the ground truths and what we
did in the last lecture was we
approximated those things using a
function approximation like a linear
combination of features or a neural
network we tried to estimate this thing
we had some parameterised function which
had parameters W we basically tried to
fit this thing using ideas which are
familiar from supervised learning we
tried to
to function to this ground truth so we
sorted starts to understand the shape of
how much reward can I get from different
parts of the state debase in different
parts of the actual space respectively I
put in and in that in that lecture the
last lecture the policy which we then
use when we actually wants to do control
we didn't represent the policy
explicitly instead we just use the value
function to pick actions so once we got
our our function approximated for cube
that was enough to pick actions because
we could just act greedily or perhaps
Epsilon greedily to pick the the action
which gets the most Q and so that was a
very effective approach to control using
function approximation but it's only one
approach to function approximation and
in some sense more direct or sometimes
more natural approach is to directly
parameterize the policy not the value
function so now we can have a different
function approximator we're going to
have some parameters you so in the last
lecture we had W so the difference
between last lecture and this lecture is
like can whether it's W or you and we're
going to use these parameters U and
we're going to parameterize the policy
now so we're going to parameterize this
distribution our policy PI is now going
to be something we directly manipulate
so we control these parameters which
affects the distribution by which we're
picking actions so think of this if this
was deterministic which will consider
later on this could just be a function
the policy could just be a functioning
at which action to be picking in a given
state and then we learn the parameters
which tell us you know for each state
which function should act which amp
which action should I pick and so again
you could think of this as some
arbitrary function approximator like a
neural network where you plug in your
state and it tells you which action to
pick or distribution their actions to
pick and so formally this policy is
going to be now probability distribution
over actions that's condition both on
the state and our parameters so we're
actually defining probability
distribution and we're tweaking we're
learning how these parameters should
change the probabilities by which we
pick different actions in different
states and what we're going to do is try
and understand how to modify this
parameter vector U so as to solve the
reinforcement learning problem so to
maximize the amount of reward that we
can get in this environment
and again what we're going to do in this
lecture is we're going to focus on model
free reinforcement learning so we're
really going to focus on how to throw
down your agent throw your robot into
some maze and this thing just wanders
around and just directly from experience
without anyone telling at the dynamics
of the environment it should be able to
figure out how to adjust its parameters
of its policy so as to get more reward
from the environment and in case it's
not clear
the main motivation behind both of these
methods is that we want to be able to
scale we want to able to scale to large
interesting MVPs large complicated
environments in which it's not possible
to separately for each distinct state
say oh I'm going to take this action
here in this state and this action here
in this state so we need a function
approximate so we need some parameters
that sort of specify how across this
state space we're going to modify these
parameters apparently I spent to you on
the left side this was a pelican
so while we're waiting for my computer
to come back to life any questions
hopefully a some the motivation should
be clear so
so what we'll start to do now is to
understand how we can actually take this
parametrized policy with its parameter
vector U and start to adjust those
parameters so as to get more reward and
the main mechanism that we're going to
consider is gradient ascent in other
words how can we compute the gradient to
follow so as to make this policy better
and if we can follow that gradient will
strictly be moving uphill in a way that
improves our policy if we know the
gradient with respect to the total
reward that we're going to get in this
MPP we're just going to follow that
gradient in the direction that gets us
more reward what we're going to do over
the next few slides is formalize what
that means my computer comes back to
life so that we can start to understand
how to maximize your ward and we'll
consider a few different ways that we
can formalize this objective function so
before we do that I'm going to ask
something to you guys while we're
waiting which is can you think of any
advantages or disadvantages of using
policy gradient competitive value based
methods like why work with policy based
methods you'll think why it might be a
better idea to work with policy based
methods than value based methods
yes you need to store few things
if you're welcome
right so you might need to storm off a
few of things so why why do you think
you might need to top your things right
so so there are some situations where
it's more efficient to represent the
values of the policy than the value
function there might be situations where
imagine you're playing one of these
Atari games like breakout or something
and it might be that the value function
is quite complicated it might be that
actually precisely working out how much
score you might get from some point
onwards might be tricky but to represent
the fact that if the ball is coming down
over here
you should always move left might be a
very simple policy to represent whereas
actually working out precisely that
you're going to get 1714 points if you
move left might be much more complicated
function to a proper to to approximate
so there are cases where a policy can be
more compact so now we've got the slides
back I'm going to come back to this yet
in this question here so so some other
reasons why it might be a good idea to
use policy based reinforcement learning
so maybe the main one you'll see in the
literature is that it has better
convergence properties that we saw that
there are some cases where value based
methods can oscillate or chatter or even
diverge if you do things in the wrong
way whereas if you just follow the
gradient a good policy
these methods tend to be more stable
you're smoothly updating your policy you
don't get these dramatic swings in what
you're what action you're going to take
because you just make a little
incremental change to your policy and
you move in the direction that makes
your policy better so they tend to have
better convergence properties and indeed
if you just directly follow the policy
gradient you've guaranteed to converge
at least to a local optimum another big
one is that these methods are effective
in continuous action spaces or high
dimensional action spaces with a barely
based method you have to work out how to
compute the max and I would say this is
probably the number one reason to use
policy gradient methods is they
circumvent the requirement to have this
max but so if we're doing something that
q-learning or indeed we're doing
something like sosser there's always a
maximization I've always been arguing
that well once we've got the queue once
we've got queue all you need to do to
pick actions is to maximize over a and
you
you pick the action with the highest Q
and you're done you know you found the
the best way to behave but what if that
maximization is itself prohibitively
expensive what if you've got you know a
trillion actions or a continuous action
space what do you do this face this is
actually quite commonly if you've got a
robot to operates in a continuous action
space of torques and so forth if you've
got you're trying to I don't know silver
an internet-based problem where you're
trying to create the perfect adverb to
show to your to your users actually
creating that advert might have many
many different components that you'd
like to move around and you'd like to
make a sequence of events that you show
that actually maximize your income with
that user this is a vast action space so
what do you do well you can solve this
complicated maximization every step or
you can use policy based methods where
you just adjust the parameters of your
policy directly and you never have to
solve for this map where you're
incrementally starting to understand
what the max will be rather than
estimating the max directly every step
and the third reason that you might want
to consider these policy based methods
is that they can learn stochastic
policies in the next couple of slides
we'll see why that can sometimes be a
good idea
so disadvantages so the main
disadvantage is the naive policy based
reinforcement learning can be slower and
higher variance and less efficient than
value based methods I think the way to
think of this is that value based
methods when you use the max are
extremely aggressive it's like this max
in one step is trying to improve your
policy in the direction that absolutely
pushes you all the way to what you
currently think is the best policy
whereas policy gradient methods just
take a little step in that direction
they smoothly update in that direction
which makes them more stable but also
sometimes less efficient and sometimes
actually computing that step to take
that gradient if you do it naively it
can be high variance and slow down the
whole algorithm but we'll see you later
how to get the best of both worlds by
including a value function and in
practice you may want a value function
as well which can add to the complexity
and just the flipside of the point which
was made earlier sometimes there are
examples like we discussed and breakout
where a policy can be more compact but
it's also conceivable that there are
cases where a value function gives you a
very nice compact representation and
gives you a more direct way to learn
your solution to your MVP okay so so
let's consider how could it be the case
why would we ever want to learn this to
cast a policy you know ultimately aren't
we just maximizing don't we just want
the thing which maximizes don't we want
to maximize our award and if we're
always maximizing reward intuitively you
might think the deterministic policy
should be good enough because
deterministic policy is always something
you can always find us maximizing
something can be a deterministic process
but there are cases where it's actually
desirable to other stochastic policy
we'll see two of those now so the first
example is rock paper scissors or
Rochambeau or has lots of different
names depending where you're from and
how you play this but it's just the game
where you show paper scissors or stone
and scissors beats paper rock beats
scissors paper beats Rock and so you
probably know this but if you were to
just play this deterministically any
deterministic strategy is very easily
exploited if I always play rock and I
start to play against you you'll very
quickly figure out that you should just
play paper and I'll lose every single
time and in fact the only optimal policy
in this domain in terms of a Nash
equilibrium which is the game theoretic
notion of optimality the only optimal
policy is to play actually uniformly at
random anything else can be exploited
because if you play one choice more
often your opponent will learn to
exploit that so this is an example where
the optimal behavior is actually
stochastic a stochastic policy is
necessary a second example is when we
have state aliasing so this comes up
when the markov property doesn't hold so
we might be in an MDP but it might be
that we we lose the markov property we
get partially observed environment so if
you remember this distinction between
fully observed and partially observed
what's going to happen now is we're only
going to see certain things about this
environment or aquiver
we were only going to use features which
give us incomplete information about
where we are in this environment so
consider this this little problem here
so as you might guess getting to this
treasure is good
getting to the skull and crossbones is
bad and this is just a little grid world
you get to move around it going north
east south or west and the question is
what's the best that you can achieve in
this domain except we're going to
restrict the way that we can create our
value functional policy to use the
following features so we're going to
consider features which basically tell
us their features of both the state an
action so they can be features that say
things like is there a wall to the north
of me is there a wall to the south of me
in between these squares by the way if
they're adjacent there's no wall between
them so this is like is there an
exterior wall and the other part of the
feature is is there an exterior wall to
the north and I'm moving to the east and
that would be a feature of my state I'm
action pack ok so we're going to
consider features like this is there a
wall to the north when I'm going east
I'm going to consider all such features
that is there a wall to the west when
I'm going south and all kinds of things
like this and we want to know you know
what's what's the best I can do using
either value based or policy based
reinforcement learning so in each case
we're going to approximate either the
value function using these features and
it doesn't matter what the function box
maker is this could be linear it could
be a complicated neural network it could
be a favorite thing the question is
what's the best that could be done in
principle and we're going to compare it
to policy based reinforcement learning
where we parameterize the policy using
some function of these features and the
question is which will do better so does
anyone have any intuition into this I'll
show it over the next couple of slides
but just have a quick think if you can
figure out what's going on
yeah you're not going to be able to tell
the difference between those two words
with the features right so so the point
was that these two gray squares are
aliased there's no way to tell the
difference between these two gray
squares using the features that you have
like if you're here and going west or if
you're here and going west
those two things look exactly the same
to you because you've got a wall to the
north you've got a wall to the south and
you're going west so this your feature
vector will be identical and you can't
tell the difference between those two
states so as a result if you use a
deterministic policy then you have to
pick the same action in those two great
states a deterministic policy which is
parameterized in terms of those two
things and and so imagine that you learn
a value function now the value function
for these two states has to be the same
and if you act greedily with respect to
that value function you'll either go
west all the time or you'll go east all
the time
okay and so this greedy idea of
generating a deterministic policy or
even in a deterministic policy would
basically lead to the following behavior
which is if you're on this side of the
of the map you would oscillate backwards
and forwards forever a fuse ingredie or
for quite a long time if it's epsilon
greedy and you can never reach the
treasure or take a very long time to
reach the treasure if you use a
stochastic policy then things are much
better because we can we can choose we
can optimize our policy so as to choose
a policy that goes east or west with
50/50 probability here so now if we're
in this great state well we'll just
randomly pick a direction and if we find
ourselves over here well sometimes we'll
go left but sometimes we'll go back
right and then we'll be pushed down to
the treasure so in just a couple of
steps we're likely to reach the treasure
compared to a value based method where
if we were to act greedy with respect to
our value function we would end up
taking in certain cases a very very long
time to reach the treasure
possibly
okay
so the summary of these couple of slides
is to say that whenever state aliasing
occurs a stochastic policy can do better
than the deterministic policy so we had
a theorem early on in the course which
was that there's always a deterministic
optimal policy for any MVP but that's
for a Markov decision process where you
have that was in the table lookup case
where we have the perfect state
representation the moment you have state
aliasing so you're partially observed so
if you're in a partially observed Markov
decision process or if your function
approximates the features that you use
limit your view of this world which is
equivalent of being in a partially
observed MVP then it can be optimal to
use a stochastic policy in which case
policy search methods can do better than
these value based methods which just
that really okay so let's talk about how
to optimize a policy so what does it
mean to optimize a policy well we need
to know what the objective should be so
here's three different objective
functions we might choose we want to
know how good is this policy the first
choice is we're going to consider some
parametrize policy with these parameters
you and want to know essentially what's
the best you answer the first case we're
going to consider is if we're in an
episodic environment we're going to
consider the start value this basically
says if I always start in some state s1
or if I have some distribution over
start state s1 what's the total reward
that I'll get from that start state
onwards okay so this only works if
there's this notion of a start state so
if there's a start state now you might
say really the thing I really care about
is I want to find the policy which when
I dumped my agent down and start state I
start my game of Atari and I play it all
the way to the end it gets the most
score or we could use the average value
and this is something we could do in a
continuing environment so now an
environment which just goes on forever
there might not be a start state so now
we might want to say well let's consider
the probability that we're in any state
this is the stationary distribution this
is the probability that we actually
end up in any state under our policy pi
times the value from that state onwards
so it's like averaging over the values
of all states so it might be in state
one and gets ten units of reward from
that point onwards or it might be in
state two and get 20 units of reward
from that point onwards and so we might
a ver Ajay well the objective is maybe
fifteen or we might consider another
formulation which is the average reward
pair time step which is similar but we
might say well what we really care about
is just let's continue some continuing
environment I'm going round and round
around and all I care about is to make
sure that pair time step I'm getting the
most reward so we're not summing these
rewards over time we're just saying
we're going to take the average of my
immediate rewards over the entire
distribution of states that I visit so
this is basically saying the objective
is there's some probability that I'm in
a state for some probability I'll take
an action from that state under my
policy and this is the immediate reward
that I'll get at that step what we care
about is getting the most reward pair
time step that's another way of saying -
extracting the most juice from my
environment I get the most reward at
time step this relates to the average
reward formulation that was like an
extra lecture - that you find in the
notes okay so it turns out that
fortunately for us that exactly the same
methods apply to all three of these they
turn out to the policy gradient is
essentially the same for all of them
the only thing which differs is the
definition of this distribution term
here that was a probability that I need
some state s and and they're just
rescaling x' of each other so they turn
out to basically follow the same
gradient direction so we don't really
need to worry which of these we're
following because we end up with the
same policy gradient so now what we're
going to consider is trying to optimize
one of these objectives so we're going
to pick one of those three objectives
I'm going to say what we want to do is
to find the you find the parameters that
maximize this objective function to find
the U such that when you drop your your
I Bo robot down your I bow will walk as
fast as possible and so how can we do
that well there's lots of familiar
approaches to optimization hopefully
you've come across some of these
optimization methods are classically
divided up into gradient based methods
and gradient free methods for
optimization so if you don't have access
to the gradient there's some algorithms
you could use for optimization like hill
climbing or simplex method we've kind of
got a triangle that you flip over and
over and over as it folds its way down
the hill sometimes called vme build
Nelda Mead genetic algorithms these are
all optimization methods that you could
throw at a a policy search problem and
try and find the best policy which
maximizes one of these objective
functions but typically the reason
people consider gradient based methods
is if you have access to the gradient
you can almost always get greater
efficiency out of your optimization
method and some of the familiar methods
for doing this are things like gradient
descent conjugate gradient quasi Newton
methods these are there's another large
family of methods that make use of the
gradient information to gain greater
efficiency because you know this
gradient points you in the direction of
greater reward
Morton web point to the direction of how
to do better in this MVP rather than
having to blindly try things by trial
and error and figure out
which of those is the best so we will
focus on gradient % the simplest
instance of this of course once you
understand how to do gradient descent
you should also understand that this can
be extended to other methods for example
second-order methods that quasi Newton
all of these things are possible but I
think the ideas are best illustrated and
often work most effectively with
simplest case which with very little
computation you can get a great deal of
effective reinforcement learning out of
this we're also going to focus on
methods that exploit the sequential
structure by which I mean we're not just
going to do blind optimization like a
genetic algorithm where you have to run
you have to throw your robot down wait
until the end of its lifetime
to get one number that pops out of it
that says you know I achieved X in my
lifetime we would rather break open the
trajectory see the part way through this
trajectory and achieve some sequence of
states and rewards and make use of that
sequence of states and rewards to do
better than waiting all the way until
the end until this thing dies before you
can actually optimize it so we'd like to
learn within the lifetime of an agent
okay so we'll start with a very simplest
version which is finite different policy
gradient methods so so let's begin by
this our familiar idea of what policy
gradient is so in the last lecture we
considered policy and we considered
gradient descent because we had
something like a mean squared error and
we're trying to minimize the error in
this lecture we're going to consider
gradient ascent where we have an
objective function which is something
like how much reward can I get out of
this system and we want to make this
thing higher we want to basically ascend
our objective function so we can imagine
that there's some surface that says you
know for any policy parameter so imagine
that these are two perhaps two different
policy parameters and there's some
surface defined by those two different
policy parameters and they might be
parameters of how my OBO walks around
and we're going to adjust those policy
parameters in the direction that gets us
a faster walk for example and and this
surface has some shape and the gradient
basically points us uphill in the
direction of steepest
send up hell and so all that says is
mathematically what we're going to do is
adjust our policy parameters a little
bit in the direction of the gradient of
this objective function and the gradient
of this objective function is just the
vector of partial derivatives so it's
the vector that says you know along this
direction here how much progress can I
make by adjusting this parameter along
this other axis here how much progress
can I make and we just move things in
the direction that maximizes our
progress and makes the most possible
difference in a Infant estimate radius
around where I am now so we're just in
the krauser's to get more juice out of
our environment okay so the simplest way
this is the naive approach the numerical
approach if you have no way to
understand what the gradient was you
could just do the following you could
basically look at your objective
function and estimate this thing
numerically by saying well if I just
prepare my parameters a little bit I can
protect them in each dimension I could
say well what would happen if I change
my policy parameter along the first axis
a little bit and I look at the
difference between the objective after I
perturbed it and rejected before I
perturb dead there'll be an estimate of
the gradient in that direction and then
we do that for each dimension separately
and that gives us a numerical estimate
of the gradient this is a little bit
naive because you need an awful lot if
you were working in high dimensional
space you know imagine your policy has
is a neural network with a million
parameters now you need to do a million
different evaluations of your policy you
need to preserve it in all those
different directions to get an estimate
of what the gradient is in that
direction there are techniques that use
random directions like SPSA which reduce
the number of samples you need to take
but they still are very very noisy and
so it takes an awful lot of computation
using these methods to actually even get
one estimate of the correct gradient but
nevertheless it's simple there's no you
know this is if you really can just
return your policy and try again
multiple times
you will get something which points you
in the correct direct direction
eventually and so it can work for
arbitrary policies and want big
advantages this works even if you've got
some non-differentiable policy like
someone exposes your gives you the
program for your robot and tells you you
know here's some parameters for your
program but we don't know how to
differentiate them you know you can
still you can still adjust it by
following this process so I've been
talking about I bow so let's make that
concrete so it turns out that in Robocop
soccer things have got a little bit
better but I think it's roughly still
true that with the ivo league in
particular this is one of the league's
of RoboCup soccer they have these little
dog robots the AI bows running around
trying to score goals against each other
so it's quite fun competition but the
thing which determines which team wins
it's basically how fast you can get your
your AI botom to run if it can walk
faster you basically able to just take
the ball and move it faster down down
the field and so so the team with the
fastest I post historically as always
one
and so it's become an interesting
machine learning challenge to make the
AIBO gate more efficient and make it run
faster and so the AI Bogut is basically
controlled by twelve different real
numbers which are sort of shown in this
diagram here doesn't really matter what
they are but the goal is to adapt these
parameters now and and the first people
to do this Peter stone Mattel they did
it by finite difference policy gradient
just like we just described in the last
slide and what they did was they
basically just ran these robots
backwards and forwards along this field
and measured how long it took them with
each gate and gave them a few walks up
and down and measured how long it took
did that for each of these twelve
dimensions separately and then so you
have to run up and down to get one
dimension run it up and down again
twelve times to get just one gradient
update you adjust your parameters in
that direction and iterate this process
until your grad students collapse of
exhaustion and and so they did this
let me see if I can pull up some some
videos
okay
so this is an early stage before it's
really been trained and I think what the
main thing to notice is that it's uh it
slips a lot that this thing is really
dependent on their on the ground which
is on and it tends to slip an awful lot
and over time it starts to learn
something a little bit better this one
okay let me try that again I think that
was just a
right but you can see it's there this
much better to gate that deals with the
the sleepiness of the surface much
better and it can move much much faster
so this was just by fault adjusting the
parameters in the direction of the
gradient in a very naive straightforward
way and you know some limited number of
iterations were sufficient to make
progress in this domain and actually
figure out a reasonable gate now if you
move to higher dimensions these
numerical methods tend to collapse and
so what will consider for the rest of
the class is how to analytically compute
the gradient so you don't have to
literally compute this gradient
separately for each of your dimensions
okay so we'll start with the simplest
approaches no value functions yet that's
the Monte Carlo approaches and then
we'll come back and we'll bring value
functions back into the picture okay so
we're going to use the following idea
which is we want to compute the policy
gradient analytically and what we're
going to assume is that our policy is
differentiable doesn't actually
technically have to be differentiable
everywhere only has to be differentiable
when it's actually picking actions so if
you have some non differentiability but
you're not actually ever picking that
action that's okay we're also going to
assume that we know the gradient of our
policy that we're in control of our
policy not something like a again it
could be a Gaussian policy or softmax
policy we'll see some examples or a
neural network and we know the gradient
of this thing because we created it and
what we're going to use is a trick
called likelihood ratios and likelihood
ratio trick I'm just going to put it up
in its in isolation because what we're
going to see is pretty much for the rest
of the course
this magic gradient of the log policy
term is going to appear and I think it's
useful to understand why this gradient
of the log appears because otherwise
you'll just be scratching your head
saying you know why why we're suddenly
looking at the log of the policy not the
policy itself and it basically comes up
just from this following likelihood
ratio trick well what we're going to
want to do is we want to take the
gradient of our policy we want to
understand policy gradient and we're
going to want to take expectations of
that thing and so the way we do that to
basically note that we can multiply and
divide by our policy without changing it
and that this term over here on the
right the gradient the policy divided by
the policy is equal to the gradient of
the log of the policy okay so that's
just straightforward calculus and this
term over here is a very familiar
quantity from statistics and machine
learning is sometimes called the score
function but if you were doing something
like maximum likelihood learning this is
the term which would come up if you were
just trying to maximize the likelihood
of some action a which someone gave you
you would follow this gradient this is
the thing which maximizes the log
likelihood and so so the way to think of
this term is this is the thing which
tells you how to adjust your policy in a
direction
gets more of something so if you were to
tell it the right action and you were
doing supervised learning you would plug
in exactly this term for some particular
supervisor action here and that would
tell you how to get more of that
particular action so this tells you how
to adjust your policy in a particular
direction to achieve more of something
that feels good and so we're going to
use that and what we see now is that by
basically rewriting this gradient in
this way we're able to take expectations
like completing the expectation of this
thing is hard but computing the
expectation of this thing is easy
because we have this policy here and
this is actually the policy we're
following and so we can take an
expectation of this term because this is
something we can actually sample from
this is the policy that were actually
following ourselves so we'll see that in
a minute but just before we do that
let's consider this score function and
understand what it looks like for two
common examples so perhaps the simplest
example you should have in your mind
which we use for discrete actions is the
softmax policy so the softmax policy is
basically something where we want to
have some smoothly parameterised policy
that tells us how frequently we should
choose an action for each of our
discrete set of actions so there's an
alternative to something like Epsilon
greedy or and and the idea is to
basically say what we're going to do is
we're going to take some features we're
going to form some linear combination of
features for example and we're going to
consider that as like there's some kind
of value that tells us how much we'd
like to take an action so I've got
features of our action from our state
we've got some parameters for those
features and then we weight those
features by by some weight and we add it
all together that tells us how much we
desire that particular action a and then
what we do is to actually turn that into
a probability we just exponentiate it
and we normalize so the idea is that the
probability that we actually pick an
action is proportional to the
exponentiating value that we get when we
take a linear combination of these
features so this is called the linear
softmax policy
so softmax in general is a policy that's
proportional to some exponentiated value
and you can choose those values to be
anything you want and here we're just
parameterizing those values in the
simplest possible way by using a linear
combination of some parameters with our
features so this is a very
straightforward way to to prioritize
policy in a discrete domain so imagine
we're doing Atari and we want to know
you actually go left or should I go
right while we'd have some features of
going left or we have some features
correspondence going right we would
weight each of those and whichever one
scores more highly when we make this
weighted sum would get higher
probability when we actually come to
pick these actions and now what we can
do is find the gradient of this thing we
want to know well how do we adjust this
thing so as to get more of a good things
so we need to know the score function
and the school function is very
intuitive for this softmax the score
function the gradient of this lock
policy is just the feature for the
action that we actually took minus the
average feature for all the actions we
might have taken so so it's basically
saying how much more of this feature do
I have the neutral that's the score
function and so what we'll see is when
we start to do these kind of policy
gradient algorithms and what will
actually end up doing is saying you know
if a feature occurs more than usual
that's what this thing is telling us if
it occurs more than usual and it gets a
good reward then we want to adjust the
policy to do more of that thing
okay so this is the softmax policy we're
also I just want to give one more
example of policy which is the the
Gaussian policy so in a continuous
action space like the I Bo example the
communist policy to use is a Gaussian
policy and what do we do with the
Gaussian policy well we basically just
parameterize the mean of this Gaussian
and then you have some randomness around
that mean some variance around that mean
that says most of you know most of the
time I'm going to take this mean action
and it's given by say a linear
combination of some features but
sometimes I might take some deviation
from that mean and there'll be some
variance like Sigma squared which could
also be parametrized you could have a
variable variance but the simplest case
is to say now the action is selected
according some normal distribution with
mean mu of s where this might be a
linear combination of our features like
before we've got features of our state
and some weights for those features and
some very X parameter so basically we
get to pick the mean and then we just
add some noise to that thing to make it
stochastic and so if we were using a
Gaussian policy what would this core
function look like well the score
function is basically again this tells
us you know how to get more of a
particular action and and here again we
have something very similar to the last
slide where this tells us this is the
action we actually took this is our our
action a and this is the mean action
this is the mean so we basically want to
know you know let's take the action we
actually took minus the mean and that
tells us how much more than usual we're
moving with doing a particular action
multiplied by the feature and then we
just scale that by the varix so in both
of these cases the score function takes
this form of sort of how much more than
usual am i doing something and and
that's quite intuitive when we start to
look at these policy gradient updates
okay so that's just to give some
intuition for what these score functions
look like now let's talk about the
actual policy gradient this analytic
public mr. president and to do this I'm
just going to derive it for the simplest
possible case and you know homework if
you like could be to extend this to the
full MDP case and we'll talk about you
know what it looks like but I won't
arrive before you and the special case
we're going to consider a one-step MDPs
so what do I mean by one-step MVP I mean
that you start in some start state s you
get to take one step and you get one
reward it depends on where you were and
what action you took but then the
episode terminates immediately
there's no sequence in this case it's
just a one step and then you're done and
you get some reward in the goal is to
pick your actions so that you maximize
that reward and the right thing to do
might depend on the state and the action
so we'll see later this is also called
in subsequent lectures as a type of
contextual bandit and so now if we were
to try and solve this we want to find
the policy gradient that we want to
understand how to adjust our policy
parameters so that no matter what
staggering we pick an action that's
that's good we want to remove our adjust
our way of picking action so as to get
more Awards the way we're going to do
that is to start off by picking our
objective function so in this one step
case all three of those objective
functions we talked about become the
same essentially yeah and and so our
objective function is just to the
expected reward on drug policy so we
want to find the parameters which give
us the most expected reward so we want
to find the gradient of this thing and
ascend the gradient of this thing which
will give us more expected reward so if
we're thrown into any state and we pick
an action according to our policy we
want an expectation that to give us the
most reward and so the way we're going
to do that let's just expand this out so
this is the expectation over the state
that we start in and this is the
expectation over the actions that we
pick and our own policy so this is the
expected reward
and now we're just going to plug in the
likelihood ratio trick so we're going to
take the gradient of this whole thing
you want to know the gradient that gets
us more reward so the gradient of this
whole thing basically the only term
which depends on you is our policy so
the gradient of this whole thing we
could push the gradient right in to the
gradient just of this term here and
expand that using our likelihood ratio
trick so this whole thing the gradient
becomes now the policy multiplied by the
gradient of the log policy multiplied by
the reward that you get and so this is
again now an expectation that's the
whole point of using the likelihood
ratio trick we start with something
which is an expectation you take the
gradient and you recover something which
is still an expectation so this thing is
an expectation under our policy so this
is the expectation under the stuffs
start state an expectation under the
action that we pick of the gradient of
the log policy multiplied by the reward
that's just the expectation of the score
times the reward so after all that we
basically come back to something very
simple tells us that if we want to
increase our our objective function if
we want to get more reward we just need
to move in the direction that's
determined by the score times the reward
okay and so again this tells us this
tells us how to adjust our policy so to
get more or less of something and this
tells us whether it was good or bad so
if you have a really big positive reward
you want to move more in the direction
that gets more of that thing we have a
really low negative reward a really
large negative reward you want to move
in the opposite directions your gradient
okay
any questions stunned silence okay so
yes I just asked little three right yeah
and we have the capital R okay so so
notation this capital R is the random
reward the actual reward you experience
this curly R is the reward function for
the MDP so we've expanded this in terms
of something which actually is described
in terms of the dynamics of the true MVP
but we brought it back again to an
expectation where we only need to look
at the sample reward that the agent
actually experiences this is completely
modern Frank
so model 3 what this means is if we want
to estimate this expectation we just
take an action from a given state we
compute the gradient of this log policy
which we know because it's our own
policy and we multiply it by the rewards
that we actually experience and that
gives us a gradient step and we follow
that gradient step since model free
Nicholas
okay so we want to do the same thing in
MVPs multi-step MVPs not just these
one-step things and to do that we
actually just need to do one trick which
is to replace this instantaneous reward
that we had so we were multiplying the
score by the immediate reward and to do
a multi-step MVP all we need to do is
actually replace this immediate reward
with the value function the long-term
reward and that actually turns out to be
the true gradient of the policy and the
policy gradient theorem tells us
precisely what that means this is just
to formalize that just to say that if
you start with some policy there for any
of those objective functions we
considered early at the start state the
average reward the average value the
policy gradient is basically given by
this thing here which is some
expectation over the score function
multiplied by the action value function
Q so it basically tells you you know
again how to adjust the policy so to get
more or less of that particular action
multiplied by how good that particular
action was for that particular action is
in expectation so you want to adjust the
policy in the direction that does more
of the good things and less of the bad
things
and if you were just doing soup
supervised learning and maximum
likelihood learning you'd have something
which look very very similar but you
just wouldn't have this value function
term here you'll just be adjusting
things in the direction that major
policy achieve more of the things that
your teacher tells you so what we have
in reinforcement learning is instead of
that we adjust it for all actions we
just have to try those actions and see
well are they good or bad in our
according to our value function okay so
that's the policy gradient theorem let's
consider the simplest way to actually
use that now to make an algorithm and we
call this the Monte Carlo policy
gradient which is roughly the same as a
really old well-known algorithm called
reinforce and the idea is to update the
parameters by stochastic gradient
descent so we're going to get rid of
that expectation now and just do
something very practical and we're going
to use the pulse of gradient theorem
what going to do is just sample that
expectation and the way we're going to
sample that expectation is just to say
what we wanted to the term which
appeared in the policy gradient theorem
was this Q so let's just sample Q let's
just estimate Q by using the return as
an unbiased sample of this Q so we're
going to be in a state we're going to
start by taking this action we're going
to see what return we've got and we can
use that as an estimate Q we're just
going to plug that in into our policy
gradient to give us a direction an
estimate for direction to move and so
every single episode now what we're
going to do is we're going to say you
know each step we're going to adjust our
parameters a little bit in the direction
of this score multiplied by the return
return we actually got from that point
onwards and so what would that look like
in an algorithm well you start off you
pick some arbitrary parameters you run
your episode and for that episode so
first of all you run forward you
generate all of your rewards this is
like a forward view algorithm so you
have to go all the way to the end of the
episode generate all of your awards at
the end of that you know what the return
was from each of your steps now so now
you go over each of your steps again and
you say for each of those steps we want
to adjust our parameters a little bit in
the direction of this gradient the
stochastic gradient which is the score
how to adjust up our parameters in the
direction that gets us more or less of
something multiplied by this sample
return from that step so that it's as
simple as that it's a very simple
algorithm so this is a the most
straightforward approach to policy
gradient which our sample returns and
you adjust your policy and direction
that gets you more of those returns
these cases are GT GT is the sum of the
rewards it's the return so it's the sum
of the rewards from the beginning view
from time step T onwards until the end
of the episode so it's the same
definition of return we've used before
so GT is the returned accumulated
rewards from that time step on it okay
so here's a little example this is using
that same Monte Carlo policy gradient
with a particular policy
parameterization and the idea is that
you start off with this this puck in
this world and this puck is very sliding
along there's very low friction in this
domain and you just get to exert a small
force on it every time step which can
just slightly perturb its direction so
if you're not careful it can really
overshoot it's its target and then the
target is given by this Pentagon which
moves around every now and then and it
has to try and reach that target and so
if it gets a reward and it gets close to
the target and there's reset every 30
seconds and then you just go you just
run trajectories and you use this Monte
Carlo policy gradient and you update to
the direction of those returns so you
always you measure your return you go
back over and you adjust your policy a
little bit in the direction that gets
you more of those returns and then you
see how how this works and so there's
two things I want you to take away from
this the first is that you get this very
nice smooth learning curve the
value-based reinforcement learning tends
to be much more Jaggi you end up with
sometimes collapses and sometimes
chatters but you can be candy very well
the second thing to take away so it's
set themselves this problem very nicely
but the second thing to take away is
that the scale here that I mean we're
talking
you know 100 million iterations here to
solve this problem so it's slow Monte
Carlo policy gradient methods tend to be
slow they're very high variance they're
looking at this complete return of what
happened estimating the gradient from
that return and moving in the direction
of estimated gradient and so the rest of
this class is going to be about using
similar ideas but making them more
efficient and how to take this very nice
idea where you get this smooth learning
where you adjust the parameters of your
policy to solve a problem and make it
more efficient reduce the variance make
it work better okay so that's going to
bring us to the final family of
algorithms actor critic methods so just
a quick recap so the main idea is that
we've got this high variance estimators
of the of the gradient now so we had
this policy gradient we plugged in the
return to estimate the depth of the
gradient by take a sample of our Q
values but that thing's very high
variance like you know it might be that
I'm playing you note re again and on one
particular episode I might get a score
of 1,000 and next episode I might get a
score of 0 and that's just in the
randomness of what happens because you
know over the course of 10,000 steps
there are many many different random
events which might occur this thing's
very noisy and the main idea about to
critic methods is that instead of using
the return to estimate the action value
function we're going to explicitly
estimate the action value function using
a critic using a value function
approximator we're going to bring back
the ideas from the last lecture combined
value function approximation with our
policy methods so again we're going to
have our true action value function and
we're going to estimate this thing using
a function approximator and then we're
going to plug this in to our policy
gradient as a substitute for q pi
so what this means is that we now have
two sets of parameters and that's why
these are called after critic methods
we've got a critic and an actor and so
intuitively the actor is the thing which
is doing things in the world and it
contains the policy it's picking actions
it's actually making the decisions of
what to do in the world is called that's
why it's called the actor whereas the
critic doesn't actually take any
decisions it's just watching what the
actor does and seeing what that's good
or bad evaluating that thing saying oh
you know those decisions were good they
got a score of a thousand or they got a
score of minus a thousand so we're going
to combine those two things together the
actor and the critic and the main idea
is now to use an approximate policy
gradient instead of the true policy
gradient we're going to adjust the actor
we can adjust the policy in the
direction which according to the critic
will get more reward so the critics
going to say hey I think that if you go
in this direction you can actually do
better and then the actor is going to
move in the direction of that gradient
and so the way we're going to do that if
we're going to take our original policy
gradient algorithm which was an
expectation of the score multiplied by
the value function that was our policy
gradient theorem but we've replaced the
true action value function with this
estimated approximate value function
where we grow our neural network or
whatever to estimate this thing and what
we're going to do then is that each step
we're just going to move a little bit
using stochastic gradient descent every
step we're going to move a little bit in
the direction of the score multiplied by
a sample from our our own function
approximator so the critic is saying hey
you know I think this thing is good or I
think this thing is bad and then we move
a little bit in the direction that gets
more or less of the things that the
critic says are good or bad okay
so how do we estimate the the action
value function how do we estimate Q what
do we do for the critic well that part
that should be familiar that's what we
did last lecture we started off with
this problem of policy evaluation we
want to know you know if I'm following
some policy PI if I'm following this
policy PI we're trying to estimate this
Q PI we're trying to say how good is
that policy we're not trying to optimize
it we're not trying to build a critic
that estimates Q star we're just trying
to say according to this current
behavior how good is it how much reward
will I get under this current behavior
and then we're going to use that to
point us in the direction that will
improve our policy so a way to
understand this is we would say how good
is this current policy pi PI you for our
current parameters you want to know how
good is that thing and so we can use
everything we've learned so far
Monte Carlo policy evaluation temporal
difference learning TD lambda pick your
favorite policy evaluation algorithm
least squares temporal difference
learning anything you like throw it into
the pot get an estimate of your of your
action value function and use that to
adjust your actor
so here's a canonical example just to
make these concrete so let's say we were
using a linear value function
approximator for our critic so we can
estimate Q by using some features of our
state and action multiplied by some
critic wait delete so in this class we
can use V for the critic parameters U
for the reactor parameters to keep those
separate and distinct from what we did
in the last lecture so we can estimate
our critic just by using a linear
combination of features and then the
critic we could just update in the usual
way using linear TV 0 and the actor we
can update the actor parameters you by
policy gradient score multiplied by what
the critic says so that would look
something like this this is the Q actor
critic so basically what this is saying
is that every step of the algorithm now
we don't need to wait until the end of
the episode this is now an online
algorithm which we can every single step
perform an update because we're using T
D in our critic we don't need to wait
all the end like we do with Monte Carlo
so every step of our episode or every
step of the Asian experiences we're
going to sample the transition we're
going to see what the reward was we're
going to see where we ended up after
that step pick an action according to
our own policy we're going to get the TD
error between the value before that step
and value after that step just like we
did in the previous lectures now we're
going to update our critic in proportion
critic we can update a little bit in the
direction of the TD error multiplied by
the features so that was a linear TD 0
so that's from last lecture that's how
to update our critic in the direction
that minimizes the error between what we
thought was happening before what we
thought the value was and what the value
ended up being after one step which is
like correcting our errors and making
ourselves self consistent we're also
going to adjust the actor we can adjust
the actor in the direction that gets us
more of the things that the critic says
are good
that's the actor critic idea using
canonical case using linear TD okay any
questions so far I'll just pause there
before we're gonna start adding you know
more tricks to reduce variance to make
these algorithms more efficient so so if
you're if you're feeling uncomfortable
now in trying to say because we're just
going to layer more more pieces on from
from here on in yes okay so you have a
policy you get a estimate and then from
that estimate you try to get another
policy that's right yeah so so we can
think of this is another form of
generalized policy iteration where we
start off with a policy we evaluate that
policy using the critic and then instead
of doing greedy policy improvement we're
moving a gradient step in some direction
to get a better policy so it's just like
we are familiar machinery where we're
making the policy better and better and
every step we're evaluating that policy
moving a little bit up the gradient to
find a better policy choosing the first
policy you can pick an arbitrary policy
so you initialize your policy parameters
however you want it could be it could be
typically now we've just got parameters
so we're not using ingredient just got
some parameters and those parameters to
share my apology I'm scaring it away
okay and we're making those policies
we're following the gradient so as to
make those policy parameters better so
that our policy gets more and more
reward down to the MVP okay so we start
off with some policy parameters that
determines our behavior we want to make
that behavior better and better and
better
there's no greedy anymore no epsilon
really the policy itself determines how
we how we move around to this
environment these policy parameters were
always picking always picking actions
according to the policy that's
determined by this these parameters you
so yeah one more question but if you
live your policy was really do you don't
fall back to cubone um let me go back to
slide that I kind of skipped in the
beginning because this projector was
down this one okay we had this in the
introduction I think it's worth
reiterating so there's classifier
algorithms into either whether we have a
value function that we represent whether
we have a policy that we represent or
whether we have both that's what this
Venn diagram represents if we have a
value function if we represent that
value in fact we call it value based if
we have a policy that we represent we
call that policy based and if we have
both of these things in our algorithm
we'll call it an actor critic okay
and Epsilon greedy epsilon greedy is
something we do when we when we live in
this side of the diagram epsilon really
we don't actually explicitly represent
the policy instead we use a simple trick
which as we say if we've got the value
function if we've got Q we're just going
to act greedy with respect to Q and
sometimes up randomly what we're
considering now in this lecture is an
alternative approach which is to
actually learn directly how to pick
actions can parametrize the policy so
instead of doing greedy or epsilon
greedy
we're directly parameterizing the policy
we're deciding how to pick those actions
and we want to figure out well if I pick
my actions in particular way will I get
more award will I get less reward and
and so there were actually a different
space here altogether where we've
directly parameterizing things so
there's no greedy there's no Epsilon
greedy anymore you parameterize your
policy to be like greedy
if you parameterize it so that's like an
implicit parameterization which I would
say is what we call value-based yeah so
say if we were basically saying our
policy is the thing which is acting
greedy with respect to my value function
that's purely value-based you're not
explicitly represented at all because
they're algorithms the work is there a
way to recover the losses accurate value
basic mechanism there's a result which
says that basically for a particular
class of policy gradient algorithms that
we'll see shortly it turns out that if
you take the limiting case where you
make your step size bigger so for all of
these policies gradient algorithms
there's some step that you take in the
direction of the gradient and you could
ask the question what happens if you
take an infinite step in the direction
of that gradient so if we just get back
to where we are so in all of these don't
algorithms there's some there's some
gradient update where we update our
parameters a little bit in the direction
of our score and for a certain class of
policy gradient algorithms if you make
this thing infinitely large then it
actually takes you to a greedy step so
that's true for certainly for natural
gradient algorithms which we'll see
later
so the step size is what kind of
controls how greedy you are so think
about the softmax policy okay this thing
where we were exponentiating some score
if you take it if you see that one of
these actions appears to be a little bit
better than the other so so you had you
had some policy where you're taking this
left more often than right and now you
see something where left actually did
even better than right so you can soar a
good score for this guy what are you
going to do well you're going to push
yourself a little bit in the direction
that makes left even more probable and
right less probable if you make your
step size infinite will push the
probability of going right all the way
down to zero and the probability of
going left all the way up to 1 which is
equivalent to acting greedily so there
is a limiting case where you recover
kind of greedy reading us and the step
size is what controls the smoothness so
so it's certainly true for some
algorithms yeah I'll take one more
question then we go
when you value yeah yeah do you have the
same thing here because okay this is a
really great question so the question
was you know if we're using policy
gradient methods do we still have the
same guarantees that we'll find a unique
local optimum or could we get trapped in
some local optima so it turns out that
at least in the case where we're using
so you could consider a table lookup
representations of policy so let's
consider the table lookup case the table
lookup case you can represent your value
function by having one value for each
state and if you use the bellman
equation you get contraction you
guarantee that you get the global
optimum with policy based methods if you
just follow the gradient for example
with a soft max it's knowing that for
the soft max that you also find the
global optimum in the table lookup case
so if you have a soft Max which is like
a separate soft max parameters for each
state you also achieve a local optimum
now for the case where you've got some
more general function approximator it's
clear that if you've got something like
and you're on there
neither method will guarantee that you
find a global optimum you can always get
trapped in some kind of local optimum
for certain special cases in between
it's unclear and I think it's an open
research question actually right okay
so so we're still in this space where
we've got like this basic principle
we've got this family of actor critic
methods we've got this family where the
actor is basically picking the actions
we've got some policy we're using that
to pick the actions we've got some
critic which is basically evaluating
those things and saying you know whether
those actions are good or bad and then
actually the actor moves in the
direction suggested by the critic that's
the basic idea of all of these but what
we want to do is to make these things
better so we're considering now now some
some tricks to make make this thing
better so the first check and the
perhaps the easiest and best no trick is
is to reduce parents using what's called
a baseline so the idea is to subtract
some baseline function from the policy
gradient and this can actually be done
in a way that doesn't change the
direction of percent in other words it
changes the it changes the variance of
our estimator changes that we can reduce
the variance of this thing without
changing the expectation so another way
to say that is that what we're going to
do is we're going to subtract off some
term which looks like this from our
policy gradient so a policy gradient was
the score multiplied by the value
function and we're going to subtract off
that subtract off the score multiplied
by some baseline function and now we're
going to choose that baseline function
just to reduce the variance to kind of
make this thing lean zero to make it
roughly about the right scale so that so
that are we basically you don't want to
end up in situations where when one
moment you see a reward of a million and
the next moment you see a reward of
999,000
and you have to kind of adjust in the
direction where you're always moving of
policies parameters up but it's just the
relative difference that determines how
much up you should make them be much
better to have like plus one or minus
one and so this gives us a way to
rescale things and so very very briefly
what we see is that if we just expand
this out so our expectation is just this
again is our expectation over States
multiplied by the gradient of our policy
multiplied by the baseline we can pull
the baseline outside at this Sun because
it doesn't depend on the action we can
pull the gradient outside of this sum
and then we know that our policy
sums up to 1 this is a probability
distribution so the probability
distribution must sum to 1 so now we
have the gradient of 1 gradient of a
constant is always zero and so we see
that this whole term here is actually 0
this whole term that we're adding or
subtracting is is 0 meaning 0
expectation okay so it's completely
legitimate to add or subtract any term
of this form so what that means is that
we can basically whenever we we have our
value function which we so if I go back
whenever we had our value function here
this was our policy gradient theorem and
our policy gradient theorem told us that
the direction we really want to move is
this score function multiplied by Q but
now what this result tells us is that we
can add or subtract anything we want
from this Q as long as that thing is
just a function of state and not a
function of action we can add up
subtract anything we want from this so
as to control the variance of this whole
term here we want this expectation to be
low variance but we don't want to change
the direction of ascent so we can do
that and there's a particularly nice
choice that we can pick for this thing
which can have some nice consequences
for us and the particular base levels
we're going to choose is the state value
function okay so what we're going to do
is we're going to start off with our Q
values our action value function and
what going to do is subtract off the
state value function and what we're left
with is something called the advantage
function this is something which tells
us how much better than usual is it to
take action a so basically we're going
to look at the Q value of action Abe
tells us how good is it to go left if
I'm in a particular state compared to
how good is it to be in that state in
general once that tells us how much
better than usual
how much more reward than usual will I
get if I take a particular action and so
intuitively that's just the right
quantity to use in our in our update can
we call this thing D like the difference
the advantage function and so what we're
going to do is now rewrite our policy
gradient theorem in the following way so
this is still the policy gradient
theorem this is still just ground truth
we've just rewritten it by subtracting
off of a particular baseline function
and so our policy gradient theorem we
can rewrite as the expectation of the
score multiplied by the advantage
function so again let's let's see if we
can understand this intuitively it's
basically telling us this tells us how
much better than usual is a particular
action a and this tells us how to adjust
our policy so as to achieve that action
a
so if this thing is ever positive we
have it to better than usual using a
particular action a this will tell us
how to move in the direction that
achieves that game for this thing is
ever negative this will move us in the
opposite direction away from the
direction that achieves that particular
action so this is always pushing the
policy parameters towards situations
where you do better than usual okay so
that's the advantage function so again
we're just rewriting the same policy
gradient theorem so how do we estimate
this advantage function now how do we do
this in our critic well there's a lot
there's a lot of different ways to do
this actually and I'm just going to
suggest a couple of them so this slide
is just to say well one way to do this
would be to learn both Q and V so our
critic would basically learn Q we could
also learn V using yet another set of
parameters and we will just take the
difference between those things as our
estimate of the advantage function so we
could basically do that this becomes
more complex in the sense we have more
parameters but it gives us a literal
estimate of the advantage function which
we've then plugged back in here so it
can plug in to this policy gradient
theorem here we plug in to this
advantage function the difference
between our state action value function
and our action value function
there's an easier way and probably a
better way and that's what the second
slide is about so this is probably the
most commonly used variant of the actor
critic although it depends who you talk
to and many people use many different
variants and it uses the following idea
which is that the TD error is a sample
of the advantage function so consider
the following TD error this D pipe this
Delta spy thing here so if we knew the
true value function V PI then this
basically tells us that the TD error is
equal to the reward plus the discounted
true value of the next state minus the
true value in the state I mean that's
just the definition of this TD error now
in expectation we can take the
expectation of this whole thing we can
say well what's the expected TD error
the expected TD error is the expected
reward plus discounted value at the next
date minus the value of the state I was
in the expectation doesn't affect this
thing because there's no random variable
in here s was s is what we're
conditioning on this term here the
expected reward plus state value at the
next state given s comma a is precisely
the definition of Q this is just like
unrolling our bellman equation one half
step so Q is equal to the expected state
value of the next step and so this whole
thing here basically tells us that the
TD error is an unbiased sample we've
basically got our Q minus V now this is
the advantage function this whole thing
tells us that the TD error is an
unbiased sample of the advantage
function so if we want to move in the
direction of the advantage function all
we need to do is to measure the TD error
and move a little bit in the direction
of the TDR in other words if we ended up
I'm in a state where I think I'm winning
again you know I make an action I find
myself in a situation where I'm losing
the game so that generates a TD error
these things were inconsistent I now
have to think oh well I was probably
losing the game all along so you end up
with a strong negative TDR
that tells you need to reduce your your
value estimate and so now what we can do
is plug in that TD error as an estimate
of the advantage to say that whatever I
did on that step that generated that TD
error whatever move I made you know it
was probably a you know bad idea it was
something which I was in a situation
where I took an action that took me to a
situation where I started off the girls
winning and I took some move and now
discovered that I'm in a situation where
I'm losing so probably that was a bad
move and so that's the intuition here
we're now going to plug in that TD error
as an unbiased sample estimate of the of
the advantage function and the nice
thing about this is that we only need to
estimate V in our critic we don't need
to estimate Q we just need to estimate
the state value function there's no
actions that come into it and and what
we do is we just plug in so this would
actually be another way to rewrite our
policy gradient theorem again this is
just a rewriting of the policy gradient
theorem we haven't changed anything
there's no approximation yet but if we
use a critic to estimate V if we use a V
hat now instead of the true V PI and we
use the TDR based on our our approximate
TD on based on our approximate value
function and we just plug in that that
estimate of the TD error there then we
end up with something which is a good
approximation to this policy gradient
here and that's a practical algorithm
estimate V we generate TD errors using
our V and we move in the direction of
those TD hours and we only need one set
of critic parameters V
okay
so just a couple more ideas to throw in
for the this family of different things
that you can try and then I'll summarize
and just bring them all back together
but one slide at the end that hopefully
is just like a summary where you can
come back and say you know these are the
different things that you can do with
actor critic so this one is really just
a reminder of the previous lectures
which is to say you know what about what
about this idea of eligibility traces
and different time scales so if you
remember from the previous lectures
we've often reused this idea that you
don't always want to go all the way to
the end of the episode and estimate the
return and neither do you necessarily
just want to take one step and bootstrap
from the value function after one step
because that's biased you often want to
trade off the bias and variance between
these two things using something like TD
lambda so now we can do the same thing
with actor critic algorithms it turns
out there's quite straightforward to do
that so if you remember from last
lecture we had a choice of things that
we could plug in so this is last lecture
you could plug in when you were updating
your value function approximator
you could plug in the return you could
plug in your one-step TD target you can
plug in your lambda return you can make
any of these the target for your for
your updates and then we also have this
idea of eligibility traces which
basically gave a backward view that was
equivalent to plugging in this lambda
return in the forward view so hopefully
this is starting to become a little bit
familiar if it's not then go back to the
notes and because it's a it's useful
stuff a key idea so what we're going to
do now is plug in exactly the same idea
to our actors so I mean first of all we
should say that you can you can do all
this with with the critic and sorry I
think some of these should say B rather
than W but you can do all of this with
that with the critic where you can
basically you know as I mentioned before
the critic we can evaluate our policy by
any means we like we can plug in any
algorithm we want for our policy
evaluator could be TD 0 could be Monte
Carlo could be TD lambda least squares
policy evaluation any of these are valid
and and then and so we can introduce
different time scales on to our critic
by choosing a TD line
on eligibility trace that doesn't affect
the actor yet it doesn't affect the
actor and the question is can we also
get these different time scales on the
actor can we get information to flow
back more efficiently in the actor so
the actors not just based on the current
TV era but based on something which
bootstraps from all different time steps
and the answer is yes and so very
briefly all we're going to do is
basically we're going to plug in so this
is again our original this is the true
policy gradient theorem the score
multiplied by the value function or the
advantage function equivalently and what
we're going to see is there's different
things which we can plug in so far we've
seen two different examples we've seen
that we can do the monte carlo policy
gradient which looks something like this
where we plugged in the return so we use
the return x the score as our estimate
and we subtract off some baseline here
so if we always just subtract off this
baseline value function here we end up
with a return minus this value function
multiplied by the score when we used the
T the error we ended up with the one
step target the TD target the rewards
plus the discounted value of the next
step - our baseline okay so these are
different targets when we use the Monte
Carlo target we end up plugging in the
return when we use this TD error version
where we're estimating the advantage
function using the TD error we plug in
the TD target and subtract off the value
function as a baseline so these are
equivalent these are different so these
are different ways to estimate our
original policy gradient theorem these
are different approximations and
sometimes it's a good idea to do this
this is a high variance low bias
estimator of the policy gradient and
sometimes it's a good idea to VIP do
this where we introduce bias by
bootstrapping from a value function but
we dramatically reduce the variance and
again the goal is to to get the best of
both worlds where we want to come up
with a gradient estimate which
bootstraps from our value from our
critic all kinds of different time steps
we don't just want to rely on a critic
of this time step to estimate the Grady
we want to use the critic so we want to
be able to say well the critic at this
time step says I should go this way the
critic at the next time step says I
should go this way so what we really
want to do is to combine these together
and actually move at some combined
direction so that's the idea of using
eligibility traces and it's exactly
analogous to the way we do it in
value-based reinforcement learning
before we compute our TD error we build
up an eligibility trace and the
eligibility trace now basically is an
eligibility over our scores so basically
remember all of the scores that we've
seen recently and so if I had a high
score for this particular action in this
particular state we remember that we
bump up our eligibility Trace and then
later on when we come to make an update
we move in the direction whether the
scores have been largest most recently
and most frequently so the way to
understand this is the best way to
understand this is by analogy with the
eligibility traces in the value based
case where you can literally look at the
update and you can see that wherever you
have an eligibility trace in the value
based case the updates were identical
like if we were doing value-based
reinforcement learning we look at say
our lambda return - our current value
and we'd have just multiplied directly
by the gradient of that value function
not by the score so all we need to do is
like a search replace wherever we had
the gradient before we search replace
with the score and that will actually
work and give us a legitimate
eligibility trace instead okay
so summary of this idea so we're
building up our toolkit of ideas that
help us to make a critic effective and
practice so now we've got a new piece to
our toolkit which is element eligibility
traces how to make our actor make use of
critics from many different steps all
the way into the future and the nice
thing about this approach as well is
that unlike monte-carlo policy gradient
we can apply this online we can
basically use critics that build up
things going far far into the future but
we can do it online and we can apply
this in incomplete sequences or in
situations where
where the environment is continuing
never stops or in situations where we're
working off policy okay
so I think what I'm going to do is
I'll talk very briefly about this so
there's a really important question in
actor critic algorithms which is in some
sense it should be the first question
that you asked about actor critic
algorithms which is that we've used this
critic and we just kind of said let's
replace the true critic value with the
some approximation to that critic value
and we just plugged in that thing and
hoped that the gradient that we follow
is still correct in other words we we
started off by deriving the policy
gradient theorem we said this is the
true gradient to follow if you really
want to improve your policy and then we
substituted in something it substituted
in a critic we said okay well you should
follow what the critic says instead of a
true value function but how do we know
that that's about it how do we know that
this really is pushing us in the correct
gradient direction and we're not
actually following some bogus
gradient and the answer is that
amazingly if you choose the value
function approximation carefully that
you use it's possible to pick a value
function approximator that doesn't
introduce any bias at all in other words
despite the fact that we don't have a
true value function they were
approximating the value function we can
still follow the true gradient we can be
guaranteed to follow the true gradient
with our policy updates and that
approach is called compatible function
approximation and compatible function
approximation I won't go into the
details of it but the main idea is to
say that the features that we use to
have a compatible function approximator
the features that we use are themselves
the score function we basically build up
features for our critic where the
features are the score of our policy and
if we use that particular type of
feature and we use linear combination of
those features then we actually
guarantee according to the theory on
this slide and the following slide which
I'll need to look at afterwards we can
actually guarantee that we don't affect
the policy direction we actually still
follow the true gradient direction if we
follow these compatible features
okay now I know that I'm throwing a lot
of ideas at you and there's a big super
possible things to follow I want to
throw in one last idea this is a recent
idea I think it's a useful idea to know
about and it's just this one slide which
is to say that so far we've considered
these stochastic policies we've
considered these policies like a
Gaussian policy where we say now what
the mean if my policy to be this thing
and I'm sometimes going to take actions
which are like Ramin sometimes I'm going
to put a bit a little bit by some noise
but this thing can be very very noisy
like we're actually so far we've been
estimating our policy gradients by
sampling our own noise we've basically
got a noisy policy and we want to take
an expectation over that noise it turns
out that clicking expectations of the
gradient of our own noise can be a
really bad idea that this thing is
really really hard to estimate and in
fact if you start off with like a
Gaussian policy and you start to improve
this thing over time and you'd hope it
would start to narrow down as you start
to figure out how to solve the problem
but as this Gaussian becomes narrower
and narrower and narrower the variance
of your estimates actually starts to
blow up to infinity like it turns out
that estimating the policy gradient
becomes harder and harder and harder the
noise basically hurts you more and more
and more as you would get better and
better at better at finding the right
policy and that's sort of an unfortunate
property of the policy gradient
algorithms we've seen so far turns out
there's an alternative and this is
something we did last year which is to
work directly with the limiting case so
instead of adding noise into our policy
and then trying to narrow this noise
down to something which is roughly 2
termina stick what if we just start off
with deterministic policies so we can
start off with some deterministic policy
and we're going to adjust the parameters
of this policy is deterministic policy
so as to get us more objective we're
going to follow the same objective
functions we did before and it turns out
that if you just take a limiting case of
the policy gradient theorem that we've
seen so far you end up with this very
very simple update which has a slightly
different form to what we've seen so far
but again you should think of this as
just in
the rewrite this again is exactly
equivalent it's just a limiting case
when we actually consider a
deterministic function where we've
narrowed our noise down to zero
basically just got a deterministic
function now we're just picking the mean
always we're never adding any noise in
and it has this particularly nice form
where we say all we need to do to follow
the gradient to adjust the parameters of
this policy this should having you there
as well if we just want to adjust the
parameters of this policy to get more
objective all we need to do is to take
the gradient of our own Q function so we
look at our critic a critic basically
tells us hey look you know if you were
it gives us the gradient that says this
is the way that will give you better
actions it says actually look this is
how you this only works for continuous
action spaces by the way but it
basically says here's the gradient of
your value function with respect your
actions this is how if you were just to
make this action here a little bit
higher you would get more reward do to
make this other action a little bit
lower you'd get less reward so the
critical ready contains all that
information about how to adjust the
actions so to get more or less reward so
that's all there and the gradient of Q
that tells you how to get more reward
like just just move left a little bit
more and you'll get a lot more reward
that's that's this gradient of Q and
then all you need to do is to work out
how to adjust your parameters so as to
get more of those actions which are
which the critic is suggesting and so
that's the gradient of the policy so
this is just the chain the chain rule
says adjust the policy in the direction
that gives you more Q so that's the
deterministic policy gradient theorem
it's very simple intuitive and in
practice it seems to work a lot better
than scattered policy gradient in
situations where we've got continuous
actions basis scales up much better to
high dimensions
I'll skip the natural to critic so
there's one more approach which is in
the notes which is the natural actor
critic the natural actor critic is an
alternative descent direction or ascent
direction which has some nice properties
and it turns out it falls out very
nicely in the actor critic framework
that's there in the notes feel free to
refer to it I wants to put up this
summary slide before we finish and this
is like a summary of all the different
manipulations that we've done to our to
our different actor critic algorithms so
we started off with basically our policy
gradient theorem what we did was we
plugged in different approximations to
this thing we said okay the gradient of
our of our policy we can get it by
taking the score function and
multiplying it by by the return we can
multiply it by the value function we can
multiply it by the advantage function so
that should be a B we can multiply it by
the TV error this was another way to
estimate the advantage function we take
a sample of the advantages in the TDR we
can use an eligibility trace where we
accumulate us course instead of just
using the current score we accumulate an
eligibility over all of our scores over
many time steps and we use that
eligibility trace instead of just that
one current score we can use the
deterministic actor critic where we
basically move in the direction that
gets us more Q okay all of these are
essentially different variants at the
same idea they're all different ways
they're different manipulations of
trying to say move the polity in the
direction that gets you more value and
then the way that we estimate values in
the critic is what varies here and the
way that we make use of this how we
update our policy is what varies between
here and here in here so the critic can
vary the actor can vary essentially
they're all doing roughly the same thing
for each of these cases we can make us
to cast a gradient descent algorithm
where essentially all we do is we just
drop this expectation by sampling the
thing which
inside and in each case you end up with
a very very straightforward simple
algorithm where all you do is you take a
step or an episode if you're doing the
top one and you look at what happened in
that one step only we now have the
gradient of the score for that step we
took an action from some state we know
for that today how to have adjusted our
policy so it so as to have taken the
action more often and we look at whether
that step was good or bad according to
our critic if it's good we push our
policy parameters to do that thing more
often so all of these are very simple
stochastic updates and then the critic
we basically use our favorite algorithm
Monte Carlo of T DT D lambda that's the
whole family of different approaches
that we've considered before in previous
lectures okay
that's it any last questions before we
in the last 30 seconds okay thanks
everyone
you