RL Course by David Silver - Lecture 5: Model Free Control

some sense you know everything in the
course up to this point has been leading
to this lecture okay we're going to
finally find out how if you drop your
robot or agent into some unknown
environment and you don't tell anything
about how that environment works how can
it figure out the right thing to do how
can it maximize its reward in that
environment how can it figure out the
actions that lead to the most possible
future reward
without telling anything in advance
about how this environment works
so that's the goal of this class
model 3 control and after we've seen
this class in the subsequent lectures
we'll look at how to scale up how to get
to more realistic large-scale domains
but the core techniques are essentially
building on what we saw in the last
lecture and bringing those into a
control setting now
so roughly the proceedings today we'll
have a brief introduction and then what
we're going to look at is two different
paradigms for doing control on policy
and off policy
which will understand the distinction
between those essentially on policy
means
learning sort of on the job
while you're following the behavior that
you're learning about and off policy
means learning whilst following someone
else's behavior
and so we'll start with on policy which
is a simpler case and again just like
last lecture we're going to follow the
format of starting with monte carlo
control before we move on to temporal
difference learning to get more
efficient lower variance techniques
so how are we going to do that well
last lecture we basically built up the
toolkit that we're going to need last
lecture we saw how to do model 3
prediction how to evaluate a given
policy if you drop your robot into a
given mdp and you ask it how much reward
will it get if it follows a particular
behavior
last time we saw how to do that using
monte carlo evaluation and temporal
difference learning
and that just helped us to estimate the
value function of some unknown mdp
and now that basic toolkit is what we're
going to use to now do control methods
where we want to not just estimate the
value function but actually optimize it
find v star find q star find the best
possible
behavior find the most reward you can
extract from this mvp but we're going to
use the same tools and we're just going
to iterate them and essentially use what
we had before kind of as an inner loop
for getting to the best possible
behavior
so just to get you back in the flavor i
know we've had a week off reading week
so so i think it's helpful just to get
um
a reminder of why this is interesting
you know why should we care why should
we care about this particular flavor why
this problem of unknown mdp
reinforcement learning and the answer is
that in most interesting problems and i
put up just a few interesting problems
here that you know all fall under this
umbrella so you know controlling an
elevator getting some car to park itself
um trying to automatically
steer some ship or control a bioreactor
or
navigate a helicopter or do the
logistics for an airplane or play
robocup soccer or control a bot in quake
or manage a financial portfolio or learn
how to fold proteins or figure out how
to make a robot walk by itself or to
play a complicated game
all of these problems
have mdps there are mdp descriptions of
these problems in each case there's some
complicated environment in which this
problem can be described where there are
some underlying dynamics in some
environment that has some state that
tells you how it's going to move around
in the state
but either
that thing is
unknown to us we don't know what that
environment is and so we have to resort
to model-free control to figure out how
to do this
or
the mdp might be known but it just might
be so complicated
that it's actually too big to use except
to sample it
and if you can only sample it you're
essentially back to the same case as
model 3 control where where you really
the only way you can access this model
is by just trying things trial and error
and see what happens because you can
never do even one step of integrating
over some vastly complicated model might
be too expensive
you know the real world's an example
where you know the
underlying dynamics of the real world
are extremely complicated even if we
knew them you wouldn't really want to
work at the atomic level and you know
integrate over the dynamics of every
atom moving around you would just take
samples see what happens and work with
that
so model free control helps us to solve
problems like these so that's why this
lecture is interesting
i hope
okay so we're going to look at this
basic um dichotomy between on policy
learning and off policy learning so
let's just understand what that means
now so on policy learning i like to call
it learning on the job so it's like
you've got some policy you follow that
policy and while you're following that
policy you learn about it
um so you basically use it actually a
sampling experience by sampling actions
from some policy pie and at the same
time you're trying to evaluate that same
policy that's what we mean by on policy
learning that the actions that you take
uh determine the policy that you're
trying to evaluate
but that's not the only choice we can
also do off-policy learning which you
can kind of think of as looking over
someone's shoulder so you might have
you know some robot which is watching
some other robot walk around and it's
trying to figure out the optimal
behavior by just observing some other
robot or maybe even a human behave so
maybe a human gives some example
demonstrations um maybe some
tele-operated arm and the human says
this is how you should move this arm
around um and now just from the
trajectories that have been sampled from
the human behavior we could actually do
off-policy learning and we could learn
to evaluate um the robot could start to
learn for itself well what would happen
if i did something else
you know maybe there's some better
behavior and the robot can learn about
that better behavior from samples of
experience drawn from say the human
behavior
you don't have to learn about the
behavior that you sample and learn about
other behaviors that's called off policy
learning
okay so we'll start with a simpler case
on policy learning
and the main idea that we're going to
use throughout this course is the same
idea that we saw a couple of uh let's go
when we're doing dynamic programming
and it's the basic framework of
generalized policy iteration so just to
quick refresh the slide so the main idea
in
policy iteration is to alternate between
two different processes where first of
all we evaluate a policy
and then we improve that policy and we
have this iterative process which i kind
of have sketched up on this diagram here
where you start with some value function
and some policy and every step
every time you go up
you basically evaluate your current
policy to get your new value function
and every time you go down you act say
greedily um with respect to that value
function
you know just say greedy actually
and so
this alternating process we saw when we
were doing dynamic programming actually
does converge on
the optimal policy and the optimal value
function
um and it's just this cycle that can go
round and round and round and the
question is well what can we slot in for
these two processes and what we're going
to do today is we're going to vary the
choices of what slots into these two
pieces here so to enable us to do model
free control
rather than in dynamic programming where
everything was using full knowledge of
the of the mdp model
um so so far we've only seen in the
dynamic programming examples of these
where we are up arrows where some way to
evaluate the policy and we saw things
like iterative policy evaluation and the
down arrows were things like greedy
policy improvement and now we're going
to vary those two components and try
other choices
so so let's make a first try okay let's
say
you know we want to do the simplest case
let's start with monte carlo learning um
so we're doing basically
monte carlo control that's this section
we're in that chapter of the
of the lectures
and and so we're going to start with
monte carlo control later we'll move on
to td control
now if we're doing monte carlo control
is it possible to just slip that in can
we just
swap in monte carlo policy evaluation
instead of our dynamic programming that
would be a way to evaluate the policy so
remember monte carlo evaluation is just
you just execute some trajectories and
take the mean
and that's your value of your of
your value function
so
what about doing that as our process for
estimating the value of the policy what
would happen if we just ran some
trajectories use those trajectories to
estimate v
and then
apply greedy policy improvement would
that work
any thoughts any problems with this idea
do people understand what the suggestion
is that we can take this general policy
iteration framework and plug in a
model-free step for policy evaluation
to just run some trajectories in the
unknown environment see how well we do
use that to estimate the value of the
policy and then iterate by improving our
policy
uh evaluate the new policy using by
running some more trajectories evaluate
our new policy
act greedy with respect to that
run some more trajectories etc um and
according to our um our knowledge of
generalized policy
iteration this has to converge on the
optimal policy
so what's wrong can anyone think there's
actually two problems with this okay
good
so if you're if when you're running your
trajectories you follow your your policy
then we're gonna explore very well okay
great so that's one of the two problems
uh we'll actually come to that one
second so um but the point was which is
correct is that there's an exploration
issue
which is that if you act greedily all
the time you don't guarantee that the
trajectories that you're following
explore the entire state space and so it
might be that there's parts of the state
space which are really interesting and
actually have better potential that you
just never see
okay there's one more problem with this
any thoughts so that's actually the
problem with the second step um any
problems with this first step with
evaluating v in this way
yeah you need to run a whole load of
trials to get a good measure of what the
best
uh best policy would be
okay so so the point was that maybe it's
slow to do this that it might be um you
might need a lot of episodes to actually
make this work that's true and we'll
improve on that by using temporal
difference learning later um but let me
move on so so so the issue is that we're
trying to be model free actually
we want to be model free
but
how can you be model free
when you're using the value function v
if you use the state value function you
only have the state value function
and you want to act greedy with respect
to your state value function you still
need
a model of the mdp to figure out how to
act greedily with respect to that value
function you only know the value of each
state and so if i'm in some state and i
want to know which action is best i have
to imagine what would happen for each
possible action i have to roll forward
the dynamics one step to look at the
value function of my successor state
we don't know those dynamics we're
trying to be model free
okay so if you work with the state value
function v you always need a model in
order in order to do your policy
improvement because you need to have
some one step of look ahead to get your
next state
so the alternative is to use q
the alternative is to use action value
functions action value functions enable
us to
do control in a model-free setting
and the reason is that we can do policy
improvement in an entirely model-free
way like if we have q
and we want to do
greedy policy improvement
all we need to do is maximize over our q
values
so we've got the q values the q values
tells us from each state how good is it
to take each different action
now if we want to make a better policy
what do we do well we just pick the
action that maximizes our q values
that's the best action it's immediate
you don't need a model to roll anything
forward it's already there in our in the
structure that we've created
so by caching the values of our actions
we would um kind of reduce the burden of
what we require from our from our model
we don't need to look at the model
anymore we don't need to roll it forward
is that clear
okay
so so let's let's drop that in so what
would that look like now so the proposal
then is that we have now an algorithm
which looks like this
um where we start it with our q value
function now an action value function
and some policy and every step we're
going to do a monty
monte carlo policy evaluation run a
bunch of episodes
um
at the end of those we've got some
estimate of the
of our policy of q pi so we run a bunch
of episodes we take the mean um from
each state action pair we look at the
mean return to estimate qsa
and that tells us how good that
particular state action pair is taking
that action from that state what was the
mean return that you got do that for all
states and actions that gives us a new
value function
we then act greedily with respect to our
q that's a media we just
take this arg max over our q values that
gives us a new policy
we run that policy for a bunch of more
episodes
um take the mean of all our state action
pairs again and iterate
and again this results in
q star and pi star
or does it
and the issue has already been raised by
someone in the audience
which is we still have one issue which
is that we're acting greedy as our
policy improvement step and if you act
greedily you can actually get stuck
you can actually not see
the states which you need in order to
get a correct estimate of the
of the value function
so we're basically estimating the value
of all state action pairs by trial and
error by running episodes but if we
don't see certain states and actions
then we won't evaluate them correctly
and we won't select them because we
haven't seen how good they are
so that's the distinction now that we're
running things we're not doing these
full sweeps that we were seeing in
dynamic programming
we're running things by actually
interacting with the environment so we
need to make sure that we continue to
see everything
so here's an example to make that a bit
clearer
okay
so this is something from the academic
world where you can imagine there's some
situation in life where you've got two
doors and you're trying to pick the best
door so in this cartoon one of them
behind that door is tenure or one of
them has tenure and one of them is
flipping burgers at mcdonald's
so so now we have to basically figure
out by trial and error which door to go
through so let's see say you get
repeated trials and let's say the
outcome of these doors is actually
stochastic
okay so you're just going to get some
immediate reward this is called a bandit
problem we'll see more examples of this
in a subsequent lecture when we focus
really on exploration it's a bandit
problem we've got two choices um
door one or door two
um so what happens now if you open the
left door
and you see a reward of zero
okay
so you get this reward of zero we're
going through the left door um so now
i think okay zero wasn't so great maybe
i'll try the right door so you open the
right door
and you get a reward of plus one
so if you're acting greedily there's
only one logical thing to do which is to
open the right door again
let's say it gets a
reward of plus three
so now your mean um after these steps
you've seen one plus one and one plus
three so your monte carlo estimate of
the value of that door is now plus two
the mean of those two
um episodes that you've seen
so plus two is clearly better than zero
so if you're acting greedily you would
pick it again
maybe you get a reward of um plus two
again
um and now you've got a mean of two so
you'll open it again
um and then maybe your mean is just
going to fluctuate a little bit around
two let's say you always get a reward
between day one and three
um you'll keep on opening this door
forever
and the problem is that actually we
don't really know what the value of this
store is we've only tried it once
and so you need to carry on exploring
everything to make sure that you
understand the value the true value of
all of your options
if you stop exploring certain actions
you can end up making incorrect
decisions getting stuck in some local
incorrect optimum your beliefs can be
incorrect and you need to carry on
exploring to make sure that you really
understand the values correctly
okay so do people see how you can get
stuck in this situation and just um keep
on mis-evaluating
so how do we do that how do we make sure
that we carry on um well there's gonna
be a whole lecture on this but we're
gonna start with the simplest possible
way to guarantee
um that you uh continue to visit all
states and all actions infinitely often
um and so this is the simplest idea for
ensuring continual exploration it's
actually
surprisingly hard to do better than this
simple idea although there are lots of
more very much more sophisticated
algorithms
and the simplest idea is what's called
epsilon greedy exploration
and it's very simple all it says is that
you flip a coin
with probability epsilon
um
you choose a random action
completely random a uniform random
action with probability one minus
epsilon you choose the greedy action
okay so you flip a coin if it comes up
heads you act completely randomly to
make sure you see everything if it comes
up tails you choose the best thing that
you you you know so far you take your
your arg max over your q damps
so it's just like our greedy
policy improvement but we've softened it
a little bit to make sure that with some
small probability you try all the other
actions
so this might seem naive but it has some
nice properties it guarantees that you
continue to explore everything and it
guarantees that you actually improve
your policy as well as we'll see shortly
so this is just a fancy way of saying
precisely that you know this is saying
the probability is
if you flip the coin
so this is
this epsilon over m is the probability
of what happens if it comes up tails and
you you explore um and this one minus
epsilon
is the probability of actually if it
comes up heads then you pick the greedy
thing but you might also pick the greedy
action just by random as well if this
tail thing comes up that's why i have a
bit more probability mass on this one
but the idea is very simple you just
either take the best action or you
explore at random
okay
so one of the reasons that epsilon
greedy is a nice idea is that it
actually guarantees that we get a step
of policy improvement so remember for
this generalized policy iteration idea
we want to alternate between steps of
improving our policy in steps of
evaluating our policy
and so what we can see is that epsilon
greedy actually is a policy improvement
like you start off with one epsilon
greedy policy
and now
the question is if you evaluate that
policy so you start off with pi
that's your previous epsilon greedy
policy if you evaluate that to get you v
pi the question is can we be sure
that by taking epsilon greedy step by
acting epsilon greedily with respect to
v pi can we be sure that we've really
made a better policy
and the answer is yes
so the simple proof here which basically
says that
if you look at
so we're just going to do the same idea
we saw in the dynamic programming
lecture well we're just going to prove
this over one step and then we're going
to argue that that whole thing
telescopes by unrolling it using the
bellman equation
so over one step we can see that the
value of our normal of our previous
policy of our original electron greedy
policy if we take one step of our new
policy
so we want to show that this thing here
taking one step by new policy is
basically
uh better than
than our original policy than the value
of our original policy
so this is just saying well this is just
an expectation
um so this is our epsilon greedy policy
i'm going to say for one step there's
some probability that we're going to
take each action
multiplied by the value of that action
this is just unrolling the expectation
and we can split that into
the probability of taking the greedy
action
plus the probability of taking all other
or every action
so this is what happens if you choose to
explore and this is what happens if you
choose to act readily
and now what we can say is that well
this thing of where you choose to act
greedily if you act really if you choose
the max
the max of all your q values has to be
at least as as good as any weighted sum
of all of your actions right the max is
better than any weighted sum of your of
your actions your q values
um so we're going to choose one
particular weighting
that sums to one here
and we're just going to say that the max
has to be better than that weighted sum
of your q values
and now you can just collect terms
together and we see that we've got one
minus epsilon on one minus epsilon so
this term over here now just cancels
with this term over here
and you're just left with
an expectation of your q values
which is your original value function
okay you can look at this afterwards but
the main thing i want you to take away
is this idea that epsilon greedy
very simple idea but it really does
guarantee that you're making progress
so you really do make progress you start
off with some apps on greedy policy
you evaluate it you make a new epson
greedy policy and it's definitely better
than what you started with or at least
as good as what you started with
okay and then for the final to telescope
it um refer back to
the dynamic programming lecture that
explained how this telescoping argument
works it's the same idea
okay
so
that's the one slide of math go on
questions
um preview to show those doesn't really
tell us anything about the way to begin
exploration really because i think if
you did like uh
um
completely greedy policy then that would
actually you know prove that it's better
than anything that you're not taking
into account
so i'm not sure i understood the
question so it's the question can we say
anything about how
how frequently you need to
like what
like the yeah how epsilon affected is it
anything about um in our real problem uh
when we execute our real problem how
much we should be exploring
um no this doesn't really in itself tell
us anything about how rapidly we should
be exploring so
um but in general um
we'll see a little bit about that in a
second that you need to choose a
schedule for your epsilon and you need
to make sure that that
decays to zero roughly but we'll see
um so so let's let's see how to plug
that in now so let's let's we've got
we've got two ideas now we started with
this generalized policy iteration
framework and we basically we've changed
each of these steps now we have these
two processes
so for the policy evaluation we're now
plugging in monte carlo as our as our
way to evaluate our policies using q
using the action values so we've got
this one process here where we just run
some episodes using our current policy
we estimate the value of all states all
actions from all states
just from what we've seen so for every
state action pair we look at the mean
return we've seen that's our evaluation
procedure
as someone's pointed out that might be a
little bit inefficient and we'll see how
to improve that shortly but now we've
also changed our policy improvement
procedure to use this epsilon greedy
policy improvement where we basically
have this soft
softening of the greedy idea so every
time we go down we're not getting to the
completely greedy policy we've got a
little bit of exploration left and so
the policy is always stochastic along
here now and that stochasticity and the
policy ensures that we at some rate at
least explore everything in the
environment now as pointed out by the
question there
the rate at which you actually see it
might be very very slow still there
might be some
states that are way off down some
strange trajectory and you have to
explore and explore again and explore
again to be able to even see those
states
so this doesn't actually give you very
strong guarantees of how long it takes
to explore all of those states it just
says asymptotically at least this thing
this thing now really will
find the optimal policy at the end which
will take your stuff
it should all
okay so let's try and make this a little
bit more efficient
so the first thing to note is um again
we saw in the dynamic programming
lecture that it's not necessary in these
kind of policy iteration frameworks it's
not necessary to go all the way um to
the top of this line every time it's not
necessary to fully evaluate your policy
sometimes you can just spend a few steps
to evaluate your policy and you've
already got enough information there to
guide you to a much better policy
without wasting many many more
iterations on gathering more information
um so what would that look like in the
context of monty carlo well let's take
it to its extreme and say why not do
this every single episode
so we're gonna run one episode gonna
make the robot do one episode collect
all the steps along that episode
update the q values just for those steps
so we basically get one new return
and so for every state and action that
we've taken along that return we're
going to update the mean value just of
those visited states and tried actions
along that episode
so
one episode one sequence of updates to
our returns
um and then improve our policy straight
away why wait why wait to get more
episodes of information when you can
already improve the policy
so the idea is
always to act greatly with respect to
the freshest most recent estimate of the
value function like if after one episode
you can already update the value
function to something slightly better
then why continue using your old
estimate of the value function to
generate behavior you may as well use
your new updated value estimates to
continue generating behavior
and so basically we change the sort of
um the
the time that the rate at which we um
operate in this diagram becomes more
frequent the frequency increases we
don't go as high up we just run one
episode
and then we immediately approve the
policy one episode immediately improve
the policy
run one more episode
improve the policy
is that clear
so
this question's already come up and it's
a natural question which is
how can we really
guarantee
um
that we find
the best possible policy like what we
really desire
is pi star we really want to know the
best possible behavior in this
environment
so to do that we have to kind of balance
two different things we need to make
sure that we continue exploring to make
sure that we don't exclude things which
are better but we also want to make sure
that asymptotically
that we get to a policy where we're not
exploring at all anymore because we want
the best possible policy and the best
possible policy will not include this
random behavior
that is extremely unlikely to be optimal
um so how do we balance those two
factors um and so
one idea for balancing those two factors
is called
this um
glee idea greedy in the limit with
infinite exploration
the idea of glee is basically to come up
with any
schedule if you like for exploration
such that two conditions are met the
first condition is that you continue to
explore everything
that you basically make sure that all
states and all actions are visited
infinitely often
so that's like guaranteeing that you
just um that you never miss out on
anything that just to make sure that
every little part of the state space
will be seen
every action from every part of the
state space will be tried
so for example epsilon greedy has that
property that eventually if you behave
randomly and you you try all possible
actions then every um every reachable
state in the state space and every
action from those reachable states will
be tried
okay um
and the second property is that we want
to make sure that the policy eventually
becomes greedy
and it needs to become greedy because we
need this thing to satisfy the bellman
equation and the bellman equation is
basically the optimality equation
basically requires a max in there not
just some soft thing and we want to make
sure that eventually we're acting greedy
with respect to q we're really taking
the max or the arg max of our q values
eventually
and so one way to achieve this by no
means the best way but but certainly one
simple idea
is to choose an epsilon 3d policy and
just decay epsilon slowly towards zero
so for example if you choose a
hyperbolic
schedule so each episode you basically
on the kth episode you set your epsilon
to say one over k
um that satisfies these two properties
that you will see everything infinitely
often
um
but asymptotically you become closer and
closer to to optimum
closer and closer to the greedy policy
okay so that's one way to balance things
so what does that look like in practice
um well we can make an algorithm now um
let's call this glee monte carlo control
um so this algorithm basically we start
off by sampling episodes um
so
again we've just got our robot we throw
it down we run one episode um that
generates a trajectory of states actions
rewards that state action award until we
get some terminal state
um we sample that from our current
policy pi
and then for each state in action that i
visited so i got to this state and i
picked this particular action but each
of those states and actions we update um
our action value the way that we do that
is just by counting how many times we've
seen that state action pair
and doing an incremental update to the
mean
so this is just our incremental update
to the mean which says that if we
consider that particular state action
pair this one over here this state and
this action
um how many times have i tried that um
well let's just
adjust our previous mean
which was this we're going to update a
little bit in the direction
of the return that we just got
and the amount that we have to adjust it
to get a correct mean estimate is this
one over n
what's interesting about this is we're
not taking a mean
in the way that you think about it's not
like a statistical mean
over
some iid quantity now that the policy is
actually changing over time we're
iterating over our policy we're
improving our policy but we're basically
taking returns from better and better
policies into account so as we improve
our policy we're gathering more and more
statistics our policy starts to get
better and we collect all of those
statistics together to get us one
overall mean of how good it is
and the glee property basically ensures
that over time that the statistics that
we're collecting really sort of converge
on getting the mean return for
um
these policies sort of get more and more
like the greedy policy and so we
basically find out more and more
information about the policy that we
really care about that the past kind of
gets dominated eventually by
by these new policies
um and so what we're going to do now is
we're going to iterate over this whole
process so we're going to sample the
case episode in this way
update our q values
and then improve our policy so this was
the policy evaluation step now the
policy improvement step just looks like
this where we can for example set
epsilon to its new value one over k and
now we're just going to act epsilon
greedily with respect to these new q
values
and those q values are going to be the
same everywhere apart from the states
and actions that i just visited
so we only update those q values along
one trajectory but that's already enough
to change the behavior that we we've
actually seen there
so in practice if you're actually
writing this you know you don't need to
explicitly store pi you just store q but
when you actually come to select actions
you always just look at the freshest
most recent queue
and you flip a coin and you either act
you either pick the maximum q or you
pick a random action
so this pie is kind of implicit and all
you need to remember is your one over k
schedule that you're using to determine
the bias of this coin that you're using
and this algorithm actually finds the
optimal action value function so this is
our this is really you know this is our
first full solution this is something
which you throw into any mdp
and you just let loose with this thing
and it will find the right solution and
it's considerably
more efficient
to run it in this way
where you actually update the key values
every single episode you update your q
values and then immediately improve your
policy rather than having to generate a
whole batch of episodes thousands of
episodes to just get one evaluation
queue
so this actually works much better in
practice
okay are people clear about this
algorithm
like the idea
it's a good time to ask questions if
you're not so we're just gonna keep on
adding to these ideas
okay good i'll take silence to mean
comprehension
um
okay question i can't remember if you
addressed this in an earlier lecture but
is there any particular
idea behind the initialization of the
values for q or do you do random
initialization okay so the question is
does it matter how you initialize q
and
in terms of the theory you can start
with anything it will still converge in
practice of course it helps to have a
good estimate of q the closer you are to
the optimal key values the faster this
thing will well um will converge
um
it's a bootstrapping
sorry it's not bootstrapping like with
td but still you um we're basically
starting from these q values and we're
updating incrementally from um
actually
sorry so that's not true with this
particular schedule let me let me back
off that we'll see that for later
versions where we have an incremental
update um
so for this particular algorithm it
makes no difference because we're just
taking the mean
um and
and so the very first time you see a new
state action pair you'll replace any q
value that you started with
completely
because n will be one
you completely replace your old q value
with it with that return that you saw
so for this particular algorithm it
makes absolutely no difference what your
initial q values are in practice
and in the sub the rest of this lecture
we'll see algorithms where you don't
have a like this perfect mean instead
you have some non-stationary estimator
that has something like a constant alpha
here and for those algorithms it matters
much more what q values start with
because you're incrementally moving on
from that initial start value and then
your start value the closer you are to
optimal the better
so yeah sorry correct myself man
okay question
for every episode yes but um
so that's a great point
so i guess yeah i'm just focusing on
serial computation for now um there are
many possibilities for exploiting
parallel computation and you're right
that you would want to that parallel
computation introduces constraints on
what you can and can't do and um and you
may want to use
it's almost
i think the same is roughly still true
with parallel computation that you still
want to use the freshest possible q
values that you can
but with parallel computation
those might necessarily be a little more
stale than the very latest episode to
come in because you simply don't have
access immediately to all of that
parallel computation
okay so let's just do a quick example so
we had this monte carlo example in a
previous lecture
in
last lecture we just looked at
evaluation we basically said okay if i
send you into a casino and i tell you
that you have to
stick
and
basically unless
you're always going to ask for a new
card unless you're on 19 or 20 or 21 i
think was the policy we had before
and then i want to know how much money
would you win or lose in that casino so
that's what we looked at before and we
had a particular plot of the value
function um and it had a particular
shape to it and now what we're doing is
we're actually running this
algorithm we saw on the previous slide
to generate the optimal value function
so now this is the question you probably
really want to know which is you know i
walk into a casino what policy should i
play to get the maximum possible amount
of reward um and again real casinos use
slightly different rules so don't try
this
but
i'm not responsible if you lose all your
money
and
and so this basically tells you
now we've run our monte carlo we've
iterated through several processes where
each episode we update our value
function a little bit we actually store
q internally but just to make the plots
easy to look at we're now basically
summing over all of the actions to
estimate v star from our queue so you
basically have some q
that generates some epsilon greedy
policy and then you alternate backwards
and pause by running these episodes just
like we saw in the last algorithm and
you end up with a policy which looks
something like this
which says that if you don't have an ace
in your hand
you should actually have this quite
interesting looking policy is the
optimal policy to follow
where depending on what the dealer has
here and what cards you've got is this
threshold
somewhere between like 16 and 17 where
if you've got 16 or less it's better to
ask for a new card
but there's some situations where a
dealer has some kind of um
quite difficult card to play with where
there's a high probability the deal will
go fast in which case you should
probably just stick anyway and just wait
for the dealer to go bust okay so you
kind of vary your policy according to
that if you've got an ace in your hand
um of course you can afford to
to ask for more cards more often because
that ace can either be a 1 or 11. and
then this is what the value function
looks like
so it's just a continuation of that last
example to show you that you know this
this process really does end up with
usable information in the sense of you
know you get these nice
shapes
out of these value functions that really
characterize exactly the optimal
behavior in this in this domain and you
really end up with something which is
the optimal policy there's no way to do
better than this
in this particular game that we've
described this is the optimal way to
behave okay
why
why would you have a stick at a level
because you can't go bust
you wouldn't it always says to
hit on the left
when dealer is showing between four and
six so this line here says if you're on
11 you always hit those sorry so the
line is between 11 and 12.
so so it's like a
a grid and each cell of this grid is
indicating what to do so this shaded
this this could be shaded in if you see
for 11 you should always
um you should always hit
but if you've got 12 and the dealer has
a bad card there's some chance the
dealer might go bust and so you should
just stick on that 12 sometimes
okay
so
we're now going to move on to
temporal difference learning methods
so just like the last lecture we kind of
split things between monte carlo
learning and then we moved on to td
learning and then we saw there was a
spectrum in between these two things
we're going to follow the same process
now but with control methods rather than
evaluation methods we're really going to
try and understand how can we actually
gain efficiency by bootstrapping by
using the same tricks that we saw last
time and gain a lot of the efficiency we
want to basically use td why because it
can be lower variance
because it's
can be run online
in continuing domains even if there's no
termination you can still run this it
can be run from incomplete sequences
and what we'll see actually in this
lecture is that there's an additional
benefit to using td which is when you
use off policy learning it becomes
particularly important to use tv
and so the natural idea well let's try
the obvious thing which is use the same
generalized policy iteration strategy
um but
we we know that to do model 3 policy
iteration we need to use q
we need to use q so that we don't have
to do any look ahead in our um to do our
greedy or epsilon greedy policy
improvement
and the idea is basically to use
now td learning to estimate q so let's
just slot in td learning
in place of monte carlo policy
evaluation and continue to use epsilon
greedy policy improvement
and that's immediately going to give us
an algorithm
that we can apply which is probably the
best known algorithm in reinforcement
learning um and
the only difference we're going to do is
because we're tds can operate at this
higher time scale so td learning is
something where you don't need to wait
until the end of the episode to update
your value function
in fact you can update your value
function after just one step that's one
of the advantages it's online like you'd
see one step of data you bootstrap you
update your value function immediately
and so again if we follow this general
principle of always using the freshest
most recent value function to to
pick your actions what we're going to do
is we're going to increase the frequency
of our policy improvement to be every
single time step we're going to improve
our policy
okay
so
the general idea is called sasa
um so
why is it called sarsa well this diagram
should illustrate it we're basically
starting off
in some
state action pair that's this black
dot's a decision node we're basically
choosing an action we're going to
randomly sample our policy
we're going to see a reward r we'll end
up in some new state s prime and we're
going to sample our policy again to
generate a prime so there's two steps of
sampling sorry we're not sampling our
policy so we start off with state here
we're asking a question about this state
s and this particular action a
we're going to sample from the
environment to see what reward we end up
in and what state we end up in here
and then we sample our own policy at
that next state
so we end up with s a r s prime a prime
or sarsa algorithm
okay
so
sarsa basically indicates a particular
update pattern that we're going to use
based on sata here
and so what do we do we're basically
going to start with our q values we're
going to move our q value a little bit
in the direction
of our td target
reward plus our discounted q value of
the next state minus
the q value of where we started so again
it's like i was in some state and i'm
considering taking some action and what
i want to know is
if i actually take that action
look at the reward i got and then the
value of the next action which i would
take
that gives me an estimate of the value
of this policy
and i'm going to use that
estimate to update the value of the
state action pair i started in
okay that's the general idea this comes
from the bellman equation for queue
we've seen this hopefully before
becoming familiar with these type of
ideas
so that's a sasser update
okay
so now we're going to take these sarsa
updates and we're going to plug them in
to
our generalized policy iteration
framework
so every single time step we're going to
move up and down in this diagram
every single time step
so each time we take a step we're going
to
update our value function by applying
one step of salsa so for the state in
action that we were just in we're only
updating the value function for that one
state action pair of that time step by
applying this update so i was in this
state action pair i ended up in some new
state action pair and do one update to
my q value for that single
entry of my table
um
and now i've already changed my value
function like something's different if i
end up back in that state action pair
again which i might in some loopy
environment
i want to make sure that i use the
latest most interesting uh most
up-to-date best information um that i've
got from my my policy evaluation
so every single time step we're going to
improve our policy as well
by acting epsilon greedily with respect
to the latest value function
so again what does this mean in practice
it means that you know you're just
storing q in your memory you've just got
this big table of q values for all
states and all actions
and every step when you actually come to
take an action you just flip a coin you
look at your q values
and
if it comes up heads then you look at
your q values and pick the best one if
it comes up tails you explore randomly
so
the policy is
implicitly represented by your cue
values
and every single step we're going to
we're going to evaluate the latest
action i took and update my policy just
to that one state action pair
okay
so we've really got this very rapid
process of evaluating by sarsa and
improving the policy
okay
let's try and
get to know this a bit better so what
would an algorithm look like i think
it's actually useful in this case to
just maybe step through some pseudo code
it's really straightforward
um so you can arbitrarily initialize
this lookup table q is just a lookup
table and subsequent next lecture
actually we'll see how to generalize
beyond these naive lookup table
representations but for now it's just a
lookup table every state every action
has its own entry in this table
we initialize this arbitrarily just say
zero okay and now we're just going to
repeat and every single step
is out to loop is just over episodes but
really you know the inner loop is just
every single step
we're going to
take an action
observe the reward observe the next
state that we end up in
um we're going to choose our action
using our epsilon greedy policy
and we're just going to apply this
sarsa update
to that one step so update a little bit
in the direction of my td target the
reward plus the discounted value of the
next date action pair
and then repeat so the next time we come
around we'll already have a different
policy
because we're acting epsilon greedily
with respect to our q values if i end up
back in that same state again for
example in a loop i will already behave
differently using the latest and
freshest information stored in queue
okay yeah question
s-dash is chosen according to e
as well
s dash is chosen by the environment s
dash is the next state i'm so state
yes
so a prime
so the question was is a prime selected
according to the same policy um so
so yeah so a prime is selected using the
your current policy
and then you basically remember that
and now when you come in you remember
what your last a prime was that becomes
your a at the next step
you just kind of um
remember your last a prime and that
becomes your new ah
choosing that i mean there's no
reworking from that last stage right
the reward the r that you're using there
is from that
yeah
um so the question is you know why does
this even make sense as an update i
guess um and so let me try and give some
intuition and i think the right
intuition to have is comes from from the
bellman equation that really what we're
trying to do this is just a sample of
the bellman equation like this this
right hand side that we're moving
towards this this thing here this target
um is a sample of the bellman equation
we're basically saying what we want to
do is we want to consider an expectation
over everything which happens in the
environment over one step
um which basically is the reward
plus
whatever state you end up in next so
that gives you your expectation of the
environment
and then
this
um and now we want to know what would
happen under our own policy we want to
know what would happen under our own
policy after that point we want the
expectation under our own policy
from that state onwards and that
expectation under our own policy is
precisely what we're trying to evaluate
with q so that's why it has to be
evaluated we have to select actions
using the policy that we're trying to
evaluate this is an on-policy algorithm
fundamentally on policy
we're selecting actions and we're
evaluating that policy
so sarsa is a non-policy algorithm
okay
so
you should be wondering well does this
work um and just like
glee
monte carlo
this
version of sarsa actually will convert
to the optimal policy and the only thing
we require is again glee policy so again
you could choose your epsilon greedy
with this decaying schedule that would
be a valid choice just to make sure that
you continue to explore everything and
that you eventually end up greedy
but you also need to be careful about
the step size that you use so you have
to make sure your step size if you
really want convergence you need to look
at stochastic approximation theory and
you need to make sure that your step
sizes basically follow these two
conditions
and all this tells you is that the step
sizes are sufficiently large
that you can move your q value as far as
you want
so you might need to move your q value
from whatever your initial estimate was
to some very very large or very very low
value if you're wrong
and this thing
basically tells you that
um eventually the changes to your q
values become smaller and smaller and
smaller the changes to your q values
have to vanish and become zero
eventually otherwise you'll still have
noise and jumping around in your policy
so that's just stochastic approximation
theory but really when you do all that
if you do all the machinery correctly
this thing really converges
in practice
we don't worry about this and we
sometimes don't worry about this either
and sarsa typically works anyway
that's an empirical result um
but this is the theory okay
so let's have an example so this example
actually is doing precisely that is
throwing out this theory and just using
a fixed step size
and a fixed exploration rate let's see
what happens
um
so this is called the windy grid world
we're just trying to wander around from
this start point this goal
we can take
any of these king's move operations
so all the diagonal moves as well as
north east south west
and the only special thing about this
grid world is that every time we take a
step the wind pushes up up pushes
upwards
and pushes the agent upwards some number
of grid cells which are indicated by
this number at the bottom the column
so there's no wind over these first
three columns then you get pushed up one
step when you move here
two steps when you move up here one step
here and zero here
um so the optimal behavior is basically
it has to take account of the fact you
want to make as much progress as you can
when there's no wind and and um i think
you end up kind of taking this route
you'll see on the next slide something
like that as the optimal behavior you
have to come back around to get back to
the goal
and then be blown up to the
um
that's not optimal um if you try to go
down
then what happens is you end up being
blown up one blown up one again blown up
one again blown up two and you end up
overshooting the goal
okay so um so you can try to do that but
you don't hit the goal so you're better
to get pushed up against the ceiling
um come back around from the ceiling
after that point and come back back to
the goal again
um but if you can go diagonally
downwards take me to the right then
those first three you can just go for it
the one
sorry the way you're blowing up one
um
so maybe this is a version that just
uses the standard moves then i'll have
to check that i think this might just
use standard moves and that the wind
blows you into a diagonal direction
you're right that would be two in that
case
so here is the diagram the option policy
so this plot basically shows us what
happens if we use the naive version of
sarsa so it's just something to show you
that you know this is the policy that it
finds
which i claim is the optimal policy you
can check that offline
and and this plot here basically shows
the the learning curve for running salsa
um and so this is exactly the algorithm
we just looked at
and what this is showing is actually
the number of time steps so this axis
here is literally time step by time step
what we're looking at is how many
episodes were actually completed in
those number of time steps so you can
kind of see here that the first episode
took 2 000 time steps to complete it
started off with no knowledge about how
to do this and so it's basically just
doing a random walk and that random walk
is biased upwards so it kind of it's not
even as good as a random walk so it
takes a long time to actually stumble on
the goal
but when it does and this is really
typical of reinforcement learning and
these generalized policy iteration
algorithms when it does
the next very next episode it does much
much faster and you get these
exponential increases because once it
learns some knowledge it can bootstrap
from that knowledge and do better and
better and better and better as it
starts to learn things so immediately
learns faster
and starts to complete episodes at much
faster rate now
until up here you can essentially look
at the gradient of this and the gradient
of this should tell you the the path
length which is taking taking which i
think is like a 14 or something the
ratio between this and this
which is the optimal policy so most of
the time it's behaving optimally but
this is still because we're not decaying
epsilon it won't be completely optimal
because it will still be throwing in the
odd exploratory action
okay
and that's why it's got this bobbly look
to it
okay
so
now we're going to take the step that we
took in the last class which is we're
going to consider the spectrum between
monte carlo and td learning algorithms
we're going to consider these
eligibility traces and lambda variants
of these algorithms to see if we can
again get the best of both worlds get
the best of
the unbiased behavior that you get from
monte carlo and also get the best or at
least control the bias have a knob that
controls the bias variance trade-off
and so the way we're going to do that is
by start off by considering these n-step
returns again
and so what we're going to do is
consider these
returns now which are basically
the one step return looks at one step of
real reward and then our q value after
one step
but we could also use the two-step
return we could have looked at one step
of reward here a second step of reward
here um and then look at a few values
after two steps or we could do that for
any number of steps all the way down to
infinity which is essentially monte
carlo where we look at the reward after
one step or after two steps all the way
until the end of the episode and we
never bootstrap from our value function
okay so that was the idea of these
n-step returns
um
and so this q return we can define in
the following way we can define the q
return
to basically be
the generalization of this to any n
and
n step sasa just does the obvious thing
where we say for each state action pair
what we're going to do is instead of
updating towards the one step
estimate of our value function using
sarsa we're going to use
this n step return as our target so
again this means we're going to take our
estimate of the value of taking action a
in state s we're going to update that a
little bit in the direction of our
n-step target
okay this is this sort of idea of the
incremental mean that's non-stationary
updates again
um and this in-step target now is you
know essentially saying well i'm going
to look at you know one two three steps
and then bootstrap from my q value which
represents the value of the remainder of
the trajectory you can use that as my
estimate of the overall value of the
trajectory and use that to update my
value of the state i was in and the
action i took from this starting point
and we could do this for any n um and so
again
just like in the last lecture we want to
be able to
consider
algorithms which are robust to the
choice of n and which average over many
different ends
we want algorithms which kind of get the
best of all possible n
yeah so i want to ask just for reminder
so the queue function represents like
the estimate of the whole projector from
sd in the
yeah that's right so so all value
functions so so let me be clear about
that so all value functions including v
and s
are long-term estimates of the amount of
reward that you get so the definition of
q is the expected total reward that you
get
um if you start in state st and then you
take action 80 and then you follow your
policy for all subsequent actions
that's the definition of q
um so so specifically as a long-term
estimate um we never consider um just
these like even when we talk about these
n-step returns
that's just our our target it's still an
estimate of the complete return
but we're estimating that by looking at
end steps of immediate reward plus
something which summarizes the
remainder all the way out to infinity or
the end of the episode
of the remaining reward
all discounted by the appropriate
discount for each day
okay
people good with that
right
so let's make a lambda version of this
algorithm
so this algorithm is called sarsa lambda
uh very well known now regularly
so what we're going to do is we can
consider first of all just the update
like what does this look like when we
talk about the full algorithm
and the idea is we're going to basically
average over all of our n-step
um
returns
so we're going to consider starting in
this state action pair
the environment pushes me to this state
i take some action
and then i estimate my q value here
and we're going to weight this
particular return by one minus lambda
we're also going to say well what if i
start in this state action pair
environment picks a state i pick an
action environment pick the state i pick
an action we're going to look at the q
value add up the rewards along here plus
the q value at the end of this
weight this by 1 minus lambda times
lambda so each time we're going to
multiply by another factor of lambda
because 1 minus lambda is just like to
normalize the whole sum to add up to 1.
so we get this weighted sum of all of
our n-step returns where we basically
are looking um
we take more account of of
the trajectories which are of the
shorter end
and then we
progressively discount by another value
of lambda per as we increase in
so
so the main idea is to now make this
average return so we had this uh these
returns which were n-step returns and
now we're going to average over all end
and the way we're going to average over
them is just by defining this weighting
just like over here we're going to
weight each n-step return by a factor of
lambda to the n minus 1.
just normalizing factor there there
and so when we make this lambda return
this is just a way to average over all
of our n-step returns in some way that's
a computationally convenient
and
b takes account of all different m
okay
and it has this lambda parameter that we
can tweak which can make us either more
or less far sighted in terms of how how
much we prefer the larger n or the
shorter
okay
and so this gives us now we can plug
this right into our um into our update
um and so again what we can do is we can
say now i'm going to update my q value
you know i start over here and now i'm
going to consider a weighted sum for
this q value i want to know how good is
it to take this action and i'm going to
consider some weighted sum that takes
account of my estimate based on my q
value my reward plus my q value from
here but also based on my reward plus my
reward again plus my q value from here
onwards we're going to wait all of these
things together
all the way up to n equals infinity um
that's going to give us our q lambda
and we'll make that our target and we're
going to update our original q value a
little bit in the direction
of that target
okay
so that's the idea of sarsa lambda
um we'll see an illustration in
two slides
um just before we do that
i want to spend one slide just saying
that this was the forward view if you
remember from the last lecture there was
a forward view and a backward view of
how to use td lambda that with these
eligibility traces um we can actually
come up with a back so so what's the
problem with this this approach so far
so this is great we've got something
which lets us
build a spectrum between monte carlo
algorithms and and td algorithms so
we've got now a variant of sarsa that
can look
um all the way out to the future if we
set lambda to one it turns into monte
carlo learning if we set lambda to zero
it turns into our familiar sarsa that we
saw in the previous steps so now we've
got this this control where we can
choose you know our bias variance
trade-off in between and average over
all of these things
the only problem is that we're kind of
looking forward in time that um this
isn't an online algorithm it's not
something where we can take one step
update our q value and um immediately
improve our policy because we have to
wait until the end of the episode
to be able to compute our q lambda
and we'd like to be able to run things
online
and be able to get the freshest possible
updates update immediately
and have this kind of clock ticking over
every step of getting the maximum amount
of our policy improvement by making an
adjustment every single step running
online
we don't want to have to wait until the
end of the episode there might not even
be an end of the episode this might go
on forever
so what we're going to do is come up
with an equivalent just like in the last
class
by building up eligibility traces such
that when we sum up over those
eligibility traces we're going to end up
with an update which is exactly
equivalent to this one
and so the idea is very similar to the
last class
we're going to build an eligibility
trace now
but the eligibility trace
is going to be for all state action
payers so now we've got a new table a
table of our eligibility for each state
action pair
think of this as the thing which is
telling you how much credit or blame you
should assign to every action you took
from every state
like you end up at the end of your
episode you know you get a big carrot at
the end of your episode
so which of the states and actions are
responsible for that current and your
eligibility trace is your best estimate
of who was responsible for receiving
that carrot and so what we do again is
we say that the states and actions that
i took that were most recent
before i got the carrot
and the states and actions i took most
frequently along the trajectory are the
ones which should be blamed or or
credited the most for their negative or
positive reward that i receive at the
end of the trajectory
and so the way we do that then is we can
do it online by just increasing this
state action pair so every time we
actually visit a state action pair
so if we actually visit it
we increment it so this is just
something that says this is an indicator
saying if i'm actually in that state s
and take that state at action a um then
i'm going to increase this eligibility
trace by one
okay so if i actually see that
particular state action pair increase my
trace and then every step
for all state action pairs even the ones
we don't visit we're going to decay them
a little bit
over time we kind of have this process
where you know if i'm going around and
this is the state
that we're really considering here if i
keep going around and every time i
revisit this state i'm going to bump up
my eligibility and over time it's going
to decay until i get back there again
and it's going to bump up again and then
if i receive my carrot whatever the
latest eligibility is that's how much
blame i'll put on that particular state
and so what do we do then well now we've
got these eligibility traces um we just
update our q values um for every state
and every action we update in proportion
to the td error that's the the event
that happens of getting the carrot or
the surprising events where you've got
more or less reward than you expected
and the credit or blame that you're
assigned is proportional to your
eligibility trace so now the algorithm
is to say let's update our q values
in proportion to the td era the
difference between what i thought was
going to happen and what actually
happened
multiplied by the this credit assignment
trace this eligibility trace
yeah question
um is there a way to apply this to very
large state spaces where you don't
effectively ever visit the same space
twice where you're using function
approximation for your value functions
so the question is can we do this in
very large state spaces where you can't
store this table next class so next
class we deal with function
approximation
for all of the algorithms we've seen so
far we'll generalize to function
approximation in the next class
and so the answer is yes you can you can
do that and should do that because
table table lookup is typically naive
and you can't solve large problems
until next class and then you'll be able
to
so sarsa lambda
it looks something like this
so we're going to
basically initialize our eligibility
traces to zero at the beginning of each
episode you want to say well i can't
blame anything yet i haven't seen
anything
and then for each step of the episode
we again we take our actions using our
policy we're again on policy here so we
pick our action from our policy for
example acting epsilon greedily
we compute our td error so we look at
the difference between the rewards plus
the
queue
the value of the state in action i ended
up in
minus
compared to my previous estimate so this
is just the one step error between what
i thought the value was going to be
before
and what i think the value is going to
be now
we increment our eligibility trace
and we decay all our eligibility traces
down here so here we're just
incrementing the state action pair we
actually visited but here we're decaying
it for all states and actions
and for all states and actions not just
the one that we visited now we do an
update because
we need to update everything
in proportion to its eligibility not
just the thing which we visit at this
step everything might be blamed now or
credited for what happens so we need to
update all state action pairs
and we update them all in proportion to
the tdrf and the eligibility trace
and we just iterate every single step
so what does that look like
so i think this
hopefully will help to get some flavor
of what the difference is between sarsa
lambda and standard salsa
so imagine you're in some grid world
where you're trying to get from here
um to this gold state
this could be like the windy grid world
we just looked at you do some random
walk to begin with and eventually you
end up there
okay and now let's say you start off
initializing all of your state action
values to zero
okay now what we're going to do is
indicate the value of those state action
pairs by an arrow in the direction of
that state action pair and the size of
that arrow indicates how big our q value
is for that particular action
so what would the updates look like
well if you just did one step salsa
you get to the end here
you started off thinking you had a value
function of zero everywhere you get to
the end and like aha i actually got a
reward of one once i reached there
so surprise
now you need to make some updates to
adjust your the values of your
trajectory towards what actually
happened
but in salsa
if we look at the updates which are made
at every step
every other step apart from the final
one like here
we thought the value of zero
after this step we also thought the
value of zero
because we initialized all our q values
to zero so nothing changes here
over here we thought the value of going
right was zero we ended up in this in a
state where if you go right you get a
value of zero so again nothing changed
here the only place you get a change was
in this very final state step here where
you thought you were in a situation
where if you go north you got a value of
zero but then you ended up in a
situation where you got your your carat
and so now this guy will be updated and
you'll end up adjusting this one to say
okay now you're going to have a very
nice positive increment you're going to
increase this value of this action going
north to some higher value and you'll
think that this was a good action
now if you run another trajectory
and you end up coming back through that
same action
that will now propagate backwards one
further step so if you if you're over
here now
you start in a situation where you
thought the value of going left was zero
you ended up in a value where you're now
estimating the value of going north to
be something high
um
so now you would increase the value of
this guy for the value of going west
from here
you would increase by some large amount
and the next step that would propagate
backwards and the next episode it
propagate backwards again we need a lot
of episodes
you're only propagating the information
back by one step per episode in sarsa
zero
okay
in sarsa lambda it looks quite different
so the lambda parameter determines
how quickly how far
that information should propagate back
through your trajectory
so again if we consider the same
trajectory
you would build up your eligibility
all the way along this trajectory so
each of these state action pairs that
you visit would have their eligibility
trace increased
um
the first ones would have decayed quite
a bit down towards zero the ones more
recent would have decayed less
and now when you actually see this
reward of one at the end
you would increase all of those state
action pairs in proportion to your
eligibility trace
so that means all of them when you see
this surprising positive benefit all of
them get updated in the direction of
what actually happened so this
information flows backwards in just one
episode most of the way back towards the
beginning of the episode
so you get a much faster flow of
information backwards through time by
choosing this lambda parameter so the
idea of lambda
is it
so we can say that it basically defeats
the tyranny of the time step but if you
if you have many many many time steps
sarsa zero is always susceptible to the
fact that you need more and more steps
to actually flow this information back
whereas using a lambda parameter you can
always just pick your lambda so that
information flows back at the right rate
regardless of the granularity that you
make your timesteps operator
so lambda basically overcomes this
tyranny of the time step
okay any questions there before we move
on to the final section yeah you said
that the lgbt traces would decay
no or they so if we go back to the
um to the updates and the algorithm the
eligibility trace is incremented
in all the states and actions you visit
but it's also decayed
every time step is decayed by a factor
of lambda
and also by your discount factor gamma
so that's why
this lambda parameter
sets the rate at which you how far back
in time you look because now you've got
this decay rate
which basically tells you
how steeply um you want your edge
ability trace to decay go back along
this trajectory so if you set lambda to
one
this these will be of equal magnitude
like these arrows would still be thick
like the updates that you make all the
way along the trajectory would still be
large and that's what happens in monte
carlo because in monte carlo you run
your trajectory you see that you get
some eventual reward and everyone gets
updated in the direction of that reward
because everyone says hey you know i
took this action i ended up getting
something good i should be updated
so by tuning your lambda you basically
determine how far back along this
trajectory you look it controls your
bias variance trade-off the further back
you look the more variance you have
because there are all these random
things which happens along this
trajectory
but you reduce your bias you're less
susceptible to
less susceptible to the effects of
bootstrapping
okay
um question back first
you don't need so okay the great
question so so the question was you know
why wait until um so for the back review
the advantage is supposed to be you
don't need to wait until the end of the
episode
um
so we're actually running an update
every single step um now the fact that
the reward only occurs at the end of the
episode um means that you don't actually
gain any information until you reach
there
so these arrows are basically showing
you um
by the time you get to the end of the
episode what's the effective update that
you see
but the nice thing about imagine that
you pass through this reward and then
you carry on and then you get another
award and then you get another award and
then you get another award
so what would happen with sarsa lambda
is that you nothing would happen nothing
would happen nothing would happen
until you hit your first reward
and then immediately all of these guys
would be updated even though you haven't
finished your episode you would carry on
you carry on and you carry on until you
hit your next informative event
then everything would be updated
because of that next informative event
and you would carry on and you'd carry
on
so the fact that nothing happens isn't
because we had to wait until the end is
because actually that we gained no
information until this point like until
this point we thought the value of zero
everywhere and we got zero
so we are making updates every single
step it's just those updates contain
zero information until the first
informative event
um but you're right i think this example
is it's not clear from this example that
that
is really a property of
the online algorithm
okay there's one more question
yeah so uh these are the action values
um and over many episodes we're
basically going to be taking the media
um
so we in in sasa we never take an
explicit mean we just use these sarsa
updates
um so the the updates that we do are
precisely the updates that we saw in
there in the algorithm which are
basically updating the q values a little
bit
um in the direction
of
um
the
so if we see some reward where we
thought there wasn't going to be a
reward that generates an error and we
update every value a little bit in the
direction of these um
of our tdrm multiplied you know
our unexpected reward that we got
multiplied by the trace by the credit
that we're assigning to each of those
states and actions that's it that is the
process by which we estimate the mean
so there's no additional step of taking
the mean
it's an incremental way to estimate the
the mean value if you like the expected
value
but we're not we're not taking we're not
forming explicit meaning and furthermore
we're forming this non-stationary
estimate
and actually that's the point i should
make more explicitly which is if we look
at the updates that we that we make
so we use this fixed step size alpha so
could you just take the mean by using
say a one over n like could we do the
same thing that we did in monte carlo
where you use where you count the number
of times you visited that state action
pair and plug in one over n
um
as your step size
that will give you like a mean
an incremental way to estimate the mean
but the thing that you're plugging into
that mean is changing all the time
because you're bootstrapping
so
so it's not really the same as forming a
true mean you're forming like a
non-stationary mean of that you're
throwing in there all of these different
targets that you've seen at all these
different times
so
so i think the right answer to your
question is that we are forming an
estimate of the expected reward expected
future award all the way to the end of
the episode
one way to estimate that is using monte
carlo learning which explicitly
estimates the mean another mechanism for
estimating that expectation is by
bootstrapping
and that's sort of instead of computing
the mean we now we now update
a little bit in the direction of our
of the errors that we see so we
bootstrap we look at the error and we
use that error to inform what we think
the previous value should have been
and that is the process by which we
estimate the expectation
okay
and
and again this
the equivalence between the backward and
forward is not obvious but it is
possible to prove there was a proof in
the last lecture and that proof follows
through for the control case as well
roughly um and
but i think a useful way to understand
what's going on is to understand that
really what we're doing when we do these
traces is to implement this algorithm
here
so we're implementing an algorithm
um in an incremental way which is
basically updating our q values a little
bit in the direction that mixes over
our estimates at all these different
time scales where we bootstrap from all
from n steps from all different end step
returns so we're just you start in some
state action pair
um so if we go back to this view here
no another way to look at this what
would the forward view look like so
forget eligibility traces the forward
view would say you start at each point
in this diagram
now you really do wait until the end of
the episode that's the forward view
at the end of the episode
from each of these states we would
consider the one step return which will
give you no information we consider the
two-step return which will give you no
information consider the three-step
return which will give you no
information
and the only one which will give you the
information is that the longer returns
that go all the way to here or beyond
okay and the amount that you take
account of those things um the weights
that we put on that final
um
m on that largest end the weight that we
put on our final one
is precisely the amount by which we
would update that guy here so if you're
very close to the end here all of your
weight goes on your
um on your return on your on your final
return because the episode stopped here
whereas if you're way back at the
beginning here a lot of your weight
is going on these shorter returns that
don't contain any information yet
and so that gives you the same effective
result
you can see just intuitively that just
looking at these diagrams you get the
same picture by either this forward view
computation or the back review
computation
um but the backup view is
computationally much nicer we do that
online you just keep one step in memory
you just do that thing you move on you
don't need to look forward in time or
wait until the end of your episodes it
works in situations where you don't even
have episodes
one question about the size of the it's
just storing the eligibility trace yeah
in the computer
so is that a kind of matrix which is has
got two dimensions one is the number of
states
and the other dimension is the number of
actions yes
so in in this lecture we're building
tables for both q which is a matrix
number of states times number of actions
and
eligibility trace
and the count when we were doing monte
carlo this next comma a all of these
quantities were tables and the tables
were number of states times number of
actions in in size
you can store that as a matrix
if that's a helpful way to think of it
in the next class we're going to look at
ways to approximate that potentially
very large matrix by
an arbitrary function approximator with
a smaller number of parameters
for both
q and also for eligibility traces
okay
let's um change gear a bit
um now we're going to talk about of
policy learning
so everything so far
has
considered the case of on policy
learning where the policy which i'm
following is the policy that i'm
learning about
um but there are many cases where
we want to consider
how to
evaluate
some other policy
while we're following so let's say we're
going to follow some behavior policy
that we're going to call mu
so this is the behavior policy that we
actually pick actions from in the
environment so mu
and what we want to do is to evaluate
some other policy pi
so as to compute say v pi or q pi or
ultimately figure out maybe the optimal
way to behave in that environment but
the behavior that we see is not going to
be drawn from this thing that we're that
we're wondering about
so why would we want to do that why
would why do we care about off-policy
learning
um
so
so here's four possible reasons that you
might care about this a lot
okay
um the first reason is that you might
want to learn from observation of other
agents like humans you might want to see
how humans behave
and not just do supervised learning to
copy what the human did but look at the
traces of behavior look at the episodes
of behavior that the human executed
and actually just from that experience
figure out how to do better figure out
how to solve the mdp just from watching
some human you know steer some boat
around and then look at what they did
and say huh i know how you could have
done better you could actually have got
more reward by following this other path
around um that's the goal
okay we want to be able to observe some
other policy
and maybe figure out how to do better or
learn from that behavior learn um
so learn from one behavior how what the
value of another behavior would be
a specific case um we used this for
example in the atari work in
i mentioned in the first lecture
is to
reuse experience from old policies
so this is a case where
imagine now that we're doing our
generalized policy iteration so we move
through a whole sequence of policies
when we do that we start with some
initial random policy pi one and then we
make it a little bit better to give us
pi two um eventually we end up getting
better and better policies um but what
if we want to go back over the data that
was generated using those old policies
and use it more than once instead of
just discarding this data which is very
data inefficient what if we want to go
back over it and look again and say oh i
remember i did this thing i've seen this
state before when i was last here i did
this other action and i got this other
award reuse that data do something a bit
more like you know batch methods where
you kind of um you just treat this as a
big data set and and figure out how to
do the best thing from that data set
well what's the problem there the
problem is that
we're trying to evaluate our current
policy when we're doing policy iteration
we want to know this is our current
policy we want to know how good that is
but we've generated behavior using all
kinds of different policies
so is it possible
to make use of this additional data to
get a better estimate of our the value
of this final policy and the answer is
yes if you use off policy learning you
can basically just treat this as
other behaviors that generated that data
just like the human you know this could
be think of this as the human and this
is some other robot
and you kind of just want to look at all
of that data and figure out from all
those other sources of experience how to
do better
um
probably the best known example of why
where
of policy learning is used
is this third point which is to say
we know that there's this big
issue in reinforcement learning which is
making sure that you explore the state
space effectively
at the same time
we want to learn about the optimal
behavior which doesn't explore at all
so why not make use of off policy
learning to generate
now we can have an arbitrary exploratory
policy that just explores around with as
much exploration as we want
and at the same time learn about the
optimal policy
that requires us to be off policy we
want to learn about
how to behave optimally something which
is ultimately going to be deterministic
whilst following something completely
random
okay
that's another motivation for our party
learning
and perhaps looking forward you know to
grand divisions the most exciting reason
to care about off policy learning so we
might want to learn about multiple
policies while following one
so there might be many different
behaviors we want to figure out i might
want to know what will happen if i leave
this room
how much reward will i get you know will
you guys be pleased or or disappointed i
don't know you know maybe i should
figure that out whilst carrying on
talking to you um but i might also be
wondering what would happen if i changed
the topic and went back to three slides
ago you know there's all kinds of
questions i might be asking about the
future which are conditioned on
following different policies
and i want to know what would happen if
i followed all of those policies and i
want to learn about them all from one
stream i only get this one stream of
data that we call life and i want to
learn about all of these other policies
and figure out the answer to all kinds
of different conditional questions
whilst following that one stream of
experience
okay these are all reasons to care about
off policy learning
so it's a it's a significant problem and
it's actually a thorny problem for
reinforcement learnings we'll see next
class
um
but
we're going to look at two mechanisms
for dealing with this um the first of
which is important sampling oh there's a
lot of time advertising it so
let me talk about this
so the first mechanism is to
basically do important sampling and the
main idea there is to take this
expectation
there's some expectation like your
expected future award and all we do with
important sampling
is to say
an expectation over say your future
award is just to sum over some
probabilities times say how much reward
you got
and now what we can do is just multiply
and divide by some other distribution
and
now this ratio here that we've got you
can basically say this is an expectation
over our other distribution
of this quantity here
where all we had to do was kind of
divide out our old probability
basically you divide you multiply and
divide by the ratio between your old and
your new distributions and that corrects
for the change between your
distributions
so we can apply important sampling to
monte carlo learning
um by basically doing important sampling
along the entire trajectory so when we
import and sample acro in monte carlo we
have to look at the trajectory we have
to multiply these important sampling
ratios across the entire trajectory like
every single step there was some action
i took according to
my behavior policy and there's some
probability that action would have been
taken under the behavior i'm actually
trying to learn about
and so it kind of you have to multiply
these ratios together and in general it
becomes a vanishingly small probability
that the return that you saw under your
behavior policy actually matched gives
you any information at all about what
would have happened if you followed some
completely different behavior
um so
you can use this idea
but
it's extremely high variance and in
practice it's just useless okay
so monte carlo learning is a really bad
idea of policy
it just doesn't work because you're over
many steps your
your your target policy and your
behavior policy just never match enough
for it to be useful
and so you have to use td learning when
you're working off policy it becomes
imperative to bootstrap
and
and so the simplest idea to use td
learning is just to important sample but
you only need to important sample over
one step now because we're bootstrapping
after one step
so all you have to do is basically
update your value function
a little bit in the direction
of your td target
um and the td target now all we're going
to do is take that td target like what
happened over one step of this action
we're just going to correct for our
distribution over that one step now i'm
going to say you know over that one step
i had some behavior policy that
generated some action and there's some
probability that i would have taken that
same action
under my actual
target policy i'm trying to learn about
we just consider the ratio between those
two things and we multiply our target by
those things so that we weight how much
we trust this target that we look at by
how much
our policy really matched what was
actually taken in the environment if
these don't match at all we can't we
can't consider this you know essentially
this basically leads us to
intentionally reject something or to
take huge account of something so you
can still get high variance we still
increase the variance by important
sampling
and it can be
can still blow up
but it's much much better than monte
carlo
now the idea which works best with off
policy learning um is known as q
learning this is specific to td 0 now um
i'm tessa zero for example um and what
we're going to do is consider a specific
case where we're going to make use of
our q values we're going to make use of
the action values to help us do off
policy learning in a particularly
efficient way that doesn't require
important sampling
and so the idea is very simple the idea
is we're going to select our next action
using our behavior policy
okay
but we're also going to consider
some alternative successor action that
we might have taken um had we been
following up our target policy
so we sample this is our random variable
representing the real thing we took in
the in the world that's time step t plus
one and this a prime is like our
alternate thing like we're just
imagining if i'd taken this thing under
my target policy you know what what what
might the world look like
and now all we're going to do is we're
going to update our q value for the
state we started in and the action that
we actually took we're going to say well
the value of that state and that action
we actually took we're going to update
towards the value of our alternate
action
when we bootstrap we bootstrap from the
value of our alternative action because
that's the thing that tells us how much
value would actually have got under our
target policy
when we sample from our
from this guy that tells you something
about what really would have happened
under our target policy
so it looks a bit like this we now
replace our sarsa update where we have
our q value for s and a being updated a
little bit in the direction of the
reward that i actually uh saw
so i was in this state i took this
action for real even though it wasn't
the one that i would do now it doesn't
matter we're just asking how much value
would i get if i took that action
i'm going to update it a little bit in
the direction
of the reward
plus the discounted value of the next
state
of my alternate action under my target
policy
this is what would have happened if i
was really following the policy i care
about
so this is a valid update this is the
bellman equation again it's completely
valid we've the bellman equation for q
doesn't require us to import some sample
we're sampling that kind of bellman
equation
that's the idea of queue landing yeah
you just be clear like which parts are
from your behavior that you stored
which is online in that last question
um
nothing is stored i'm not sure so with
the keyboard something a t plus one from
so you're sampling a t plus front one
from your behavior policy and you're
sampling a prime from your target policy
so each time you get to a state you're
considering the thing you actually took
um and you're also that's generating a
real behavior the thing you actually
took
and that's the trajectory you're
actually following but at every step
you're also considering some alternate
action you might have taken and that's
the thing you bootstrap from you
bootstrap from that q value
now there's a special case of this which
is the well-known q learning algorithm
so if you hear about the q learning
algorithm
essentially is what we're going to talk
about in this slide
and it's a special case where the target
policy um is a greedy policy so this is
the special case where we're trying to
learn about greedy behavior while we're
following some exploratory behavior
um
so
and the special thing about hue learning
is that both the behavior and the target
policies can improve we allow
improvement steps to both of them
but what we're going to do is basically
at every step we're going to make the
target policy greedy with respect to our
value function so we're going to ask
about this deterministic thing which
really acts you know as aggressive as
possible we're trying to learn
aggressively about what we consider to
be the best behavior given our q values
but our behavior policy is going to be
epsilon grievance so behavior policy is
roughly going to follow sensible things
to get us to good parts of the state
space but it's going to explore a little
bit as well to make sure we get to new
things as well
and when we plug this in to the previous
slide
we basically end up seeing that so this
is our the target we had in the last
slide our r plus
q
the q value of our alternate action our
alternate action
um
is this r max the greedy policy so when
we plug in this arg max
if you
basically want to know what's the q
value of the arg max of your q values
that's the same as the max of your q
values
so it's basically what q learning does
is it updates a little bit in the
direction of the maximum q value that
you could take
so that looks like this
so we're just going to have one
diagram to display this this is the
well-known q-learning algorithm i call
the salsa max updates because you're
looking at s a r s prime and then the
max of your a primes
and what we're doing is we're maxing
over these guys we're updating our q
values a little bit in the direction of
the best possible next q value could
have after one step
that's the idea so it should be familiar
from the bellman optimality equation
this is basically
just like bell and optimality equation
and this algorithm again converges to
the optimal action value function
so it's a standard idea for
reinforcement learning
um so just to wrap up
i just want to
bring out
these relationships between these full
width updates that we had in dynamic
programming and what goes on in temporal
difference learning
um
so
in this column here what we've got is a
dynamic programming algorithm that we
looked at in the previous lecture on the
right hand column are these are sample
based algorithms td and algorithms that
we've looked at in the last lecture and
this one
okay
and so if you use the bellman
expectation equation so this is which
belmont equation is if you use the
bellman expectation equation for the
value function v for the state value
function
you can use dynamic programming to
evaluate that your current policy
or you can just sample this thing and
take just one sample of this diagram by
sampling from the environment and
sampling from your policy and that gave
us td learning
so these guys are just a sample of these
guys
wherever you've got an expectation here
you sample it to get your td learning
algorithm here and that gives you your
target so the target for your td
learning algorithms are samples of the
right hand side of the bellman equation
when we do the bellman equation for q pi
um
we can actually do
sasa now
so so when we use the bellman equation
the belmont expectation equation for q
pi we can use that to evaluate our q
values and then plug that into our
generalized policy iteration to make our
policies better and better and better
so we can either do that using a policy
iteration framework for dynamic
programming or that gave us the sas
around it
and then finally we can use the bellman
optimality equation the q star
and when you use the bellman optimality
equation
you can either plug that into value
iteration
which is the dynamic programming method
or you can just sample this sample your
expectations to give you the q learning
algorithm
okay so td we can think of as samples of
the bellman expectation equations or the
verbal optimality equations they're kind
of doing a one-step sample of what would
happen if you were doing dynamic
programming
so that's just kind of to tie things
together
okay thanks everyone next week we'll
talk about function approximation how to
scale up how to make these things work
in larger and more practical examples
cheers