DeepMind x UCL RL Lecture Series - Model-free Control [6/13]

hi and welcome to this sixth lecture in
this course on reinforcement learning
my name is advanced and today i will be
talking to you about model free control
the background material for this lecture
is chapter six from susan embardo you
might recall from the previous lecture
that i said
there were a bunch of uh chapters
background material 5 6 7 9 12.
but this lecture specifically will be
focusing mostly on the contents of this
one chapter six
just to recap again here's the
interaction loop of an age interacting
with the environment and then
reinforcement learning can be taken to
be the science of learning how to make
decisions
and the agent can internally learn a
policy a value function and role model
this lecture will be focusing on value
functions again like last lecture in the
last lecture we were focusing on
predicting value functions and now we're
going to focus on additionally using
those value functions to then come up
with a policy we won't yet talk about
models
this general problem
involves taking into account time and
consequences so we have to make
predictions about long-term futures
because actions cannot just influence
just the reward but also the asian state
and the environment state and this means
that actions can have long-term
consequences
now in the previous lecture as i
mentioned we covered model 3 prediction
which is all about estimating a value
function in an unknown markov decision
process
so we imagine there's some reward
for instances could be taken part of the
observation or this could be taken as
part of the agent where the agent has a
certain preference over different
observations
and then value functions are defined as
the expectation of the accumulation of
these rewards and that's what we try to
optimize
but in previous lecture we didn't talk
about optimizing those we just talked
about predicting them and then this
lecture is about model free control in
which we will try to optimize these
value functions
the first topic that we'll touch upon is
monte carlo control
so before introducing monte carlo
algorithms i'm first going to
recap policy iteration so policy
iteration was described in the lectures
on dynamic programming taught by lyanna
and just to remind you policy iteration
refers to interleaving two separate
steps
which are called policy evaluation and
policy improvement and that's depicted
here on the left where we start with
some arbitrary initial value function
for instance they could all be zeros and
some arbitrary policy for instance
uniformly random
and then the first step could be to
evaluate the this policy
so we depicted here with this arrow
going up and then when we reach this
line that means that the value function
is now fully accurate for the value of
that policy the actual high of our
policy
that would be the policy evaluation
phase
then when we've reached this value
function we can do a policy improvement
step for instance we can make the policy
greedy
and this policy improvements that will
change the policy
in such a way that we get a new policy
pi prime such that the value of that
policy in every state is at least as
high as the value of the policy that we
were just evaluating and picking a
greedy policy is one way to achieve this
but because we're changing the policy
that means that our value function will
become inaccurate because it's an
evaluation of the previous policy but
we've now changed our policy which means
we can do the evaluation phase again for
this new policy and again find the true
values for that policy
and then because the value function has
changed
greedifying the policy with respect to
that new value might also change the
policy and so on and so on
we see that depicted here as well where
these
arrows mean that we're basically
overwriting so first we start with some
policy let's say random we evaluate that
policy because it's a value function
then we consider a new policy which for
instance picks the greedy action in
every state
and that then we can do the evaluation
again and so on and so on when this
ceases to change anything that means
that by definition we must have found
the optimal value function because it
means that the true value of the policy
is equal to when you then consider the
greedy value and that by definition will
be the optimal value function
so this has all been discussed in
lectures three and four
we're going to use this as an intuition
to build model free algorithms because
before with dynamic programming for
instance the policy evaluation phase we
consider dynamic programming algorithms
which can draw upon knowledge of the
precise markov decision process but we
are not wanting to make that assumption
anymore so instead we'll come up with
approximate methods that sample and then
do this evaluations phase approximately
and turns out this will still be okay
this can still lead to converging
algorithms
so to recap for the policy evaluation in
the previous lecture we talked about
ways to do this model-free
so here at the top we see a an update
which could be taken to be a monte carlo
value update where for instance we're in
some episode maybe depicted n and at the
end of that episode there was a time
step t inside that episode in which we
saw a state st
and from that state on what we saw a
return gt and then we just incrementally
update the value of that state towards
that this is showing the update of one
state of course we normally
when using monte carlo algorithms we
would have to wait until the end of the
episode to update all states so instead
of just updating one we would actually
maybe update all of the states in one go
but now i'm going to draw your attention
to this target gt and i'm going to make
it explicit that monte carlo is one
choice this was all discussed in the
previous lecture where monte carlo would
use a full return so the reward plus the
discounted next reward busted plus the
doobly discounted reward after that and
so on which we can write on recursively
as well where the return
is
the first reward plus the discounted
rest of the return that would be our
monte carlo target and we discussed that
this is a fine target to use although it
might be quite high variance in some
cases
we also talked about alternatives which
might have lower variance like td0 td0 i
remind you is exactly the same algorithm
which we could also call one step td and
this refers to the fact that we're
taking one step in the world we see one
reward and then we use bootstrapping
which refers to the usage of the value
function estimate itself to take the
place of the random return so if we
compare it on monte carlo we saw monte
carlo we were using the random rest of
the return whereas in temporal
difference learning we replaced that
with our current estimate of the
expected return
this tends to be much lower variance
it can suffer from bias in some cases
so we also discuss intermediate
approaches that one can use
for instance
n-step approaches where we just take a
fixed number of steps in this case n
different steps which means we look at n
different rewards and then we bootstrap
the state n steps in the future
which can be written down
semi-recursively as well where the end
step return takes one step and then
looks at the n minus one step return
and additionally we also looked at well
if you're doing that that kind of
n-steps kind of means you're you're
either continuing or you're not
continuing and then we said we could
also do that partially so we could also
on every step decide to bootstrap a
little bit with weight one minus labda
and then continue to the rest
recursively with labda because this is
defined recursively this means a little
next step if you roll this further out
you would again bootstrap a little bit
and continue
and then bootstrap a little bit and
continue and so on
for lab.0 this is exactly the one-step
td algorithm for lab1 this is exactly
the monte carlo algorithm and that's
also why the one step approach is
sometimes called td0
so this is just to recap that there's
different mechanisms you could use to do
model-free policy evaluation
and in each of these cases the goal was
as in the previous lecture to estimate
the value of the current policy so
there's some policy pi this is used to
generate these transitions to generate
the behavior and we're just estimating
the value of that policy
now
we can consider then
improving the policy we didn't consider
that yet last lecture but in this
lecture we want to improve the policy
and one way to do that as i mentioned is
to make it greedy
however greedy policy improvement over
state value functions does require a
model of the markup decision process
because the greedy policy would then be
something like this where we take the
arc max over this expectation of the
transition
that's a bit cumbersome because we are
not model free anymore so now the policy
improvement step seems to require a
model
but fortunately there's a way around
that we can just instead of estimating
state values we could also choose to
estimate state action values
and then
we can much much more easily find the
greedy policy by picking just the
highest valued action in every state
this would give us a new policy so q
here could be an estimate of q pi of the
true value of the policy
and then pi prime could be
a greedy policy with respect to these
estimates
then of course having a new policy we
could try to estimate that one instead
so this is uh why we're considering
action values essentially because it
makes it so convenient to then inspect
the values of different actions and then
in infrared policy from this
so now basically the whole idea is to
plug in these model three estimates into
the policy evaluation phase of policy
iteration
so essentially what we're doing here is
we're depicting the same picture as
before but we replace the v's with q's
so that's a small change right we're
just considering state action values
rather than state values
but for now we're going to do the exact
same thing as before we're going to
estimate the value of the policy and
then go greedy the greedy step is
facilitated by using these action values
it's very easy to then if you have these
action values construct a greedy policy
and this can be done model-free
now in order to be to make the full
algorithm model free we also want this
evaluation step to be model free and one
way to do that is to use monte carlo
policy evaluation
so
we could just consider following a
certain policy and maybe do that
indefinitely long many many samples and
we all the way conversations through
action values
in a moment i will relax this assumption
greatly and we'll see we can even learn
from single samples and still do some
version of generalized policy iteration
but for now conceptually just keep in
mind oh we're just going to do maybe
exhaustive like evaluation of that
policy and then we just consider the
greedy policy so everyone so often after
a lot of learning we change our policy
become become greedy
however
there is a problem if we go fully model
free which is that if we really pick the
greedy policy
then we wouldn't explore
which means we wouldn't actually sample
all state action pairs when learning by
interacting
so why is this a problem well it's a
problem because it
makes the policy evaluation
susceptible to problems
essentially if you have a fully greedy
policy and you're trying to evaluate
that with a monte carlo algorithm that
means that you should be following that
greedy policy
but if you're following the greedy
policy it might simply not select
certain actions in certain states and
that means that we don't have a reliable
estimate of this of the state action
value from those actions
which means that then if we consider the
evaluation phase this will be
maybe severely severely
hampered and that means that then the
improvement step could be wrong
because we don't have accurate state
action pairs state action values for
certain state action pairs this means
that the improvement set could
wrongfully pick a different action
so that is a problem but it's also not
that hard to fix fortunately so instead
of doing full
greedification we could also consider an
epsilon gradient policy where we still
allow with some probability epsilon to
pick any action
turns out this is okay this is why this
procedure is called generalized policy
iteration recall that for the policy
improvement step we just needed well
just needed that the policy new policy
was better but we don't need this to be
fully greedy so it turns out epsilon
greedy can be plugged in here as well
and we depict it now in the figure that
we're also not doing a fully evaluation
phase
so this new line at the bottom is no
longer greedy but it's epsilon greedy
and in addition we're only going to move
partly in the way of the evaluation so
we're no longer going to assume we're
going to fully evaluate
every policy that we have instead we're
just going to do some evaluation let's
say a couple of episodes and then we
consider the epsilon greedy policy with
respect to these new action values
so now we have a full model free
algorithm that can be used for policy
iteration
and now you might question well sure but
you're not doing the full evaluation
anymore you're not doing full
gratification is this algorithm still
sound does this still converge to the
actual optimal value function and the
optimal policy
so first we're going to just make it
very clear what we're doing in terms of
the algorithm
so we're going to sample episodes
according to our current policy then
we're going to just do policy evaluation
so we're going to model free learn the
value of that policy
and for instance we could pick a step
size that decays over time
because otherwise we might have
irreducible variance right which might
make our action values to be quite
inaccurate
although you could also use this
algorithm with a fixed step size and in
some cases it might still be fine but
maybe it's better to consider a decaying
step size for instance that decays with
one over the number of episodes that
you've seen so far
and then in addition to that it might
also be useful to decay the greediness
sorry the uniformness of the policy in
the epsilon greedy policy so this will
make it
make sure that eventually you're
selecting the greedy policy
so this is basically a small change the
epsilon greedy policy where we're just
allowing the epsilon to be dependent on
the number of episodes you've seen so
far or on the time steps if you prefer
so we see that the evaluation and the
expiration are both
time dependent here but on every time
step we're doing this approximate
version where we have
some policy right now we're evaluating
that with some step size and some
samples and then we're going to go
partly greedy
and then the reason to pick this epsilon
is that it actually um
has a certain property which is
desirable for these
theoretical properties
and we call that property greedy in the
limit with infinite expiration
so what does that mean well greeting the
limit means that eventually you will
become fully greedy and this is
important to actually converse to the
optimal policy eventually you should
just pick all always pick the right
action and you shouldn't have this
epsilon probability of picking a wrong
action indefinitely
so what we want essentially is
eventually should become fully greedy
but you should also
explore infinitely often so what does
the second thing mean well essentially
we can formalize that by saying the
count for all state action pairs should
go infinity as time goes to infinity
this ensures that we are going to
evaluate everything properly so if you
continue to select every state action
pair indefinitely often that means you
eventually have enough data to have a
good evaluation of the value of the
current policy
and then this means that the
greedification step can be trusted
the
greedy and the limits can be formalized
quite simply as this where we just say
the policy our stochastic policy for
instance epsilon greedy as time goes to
infinity should eventually become fully
greedy with respect to whatever the
current action values are then
so note that these two conditions they
kind of put in opposite directions
greedy in the limit can for instance be
achieved by being immediately greedy
from the beginning
but then you wouldn't explore infinitely
often
likewise you could explore infinitely
often if you just kept the fixed epsilon
but you wouldn't be greedy in the limit
so are there exploration policies that
do both of these at the same time and
the answer to that is yes
for instance if you have an epson greedy
policy uh that decays the epsilon with
one over the number of episodes or one
of the number of time steps in total
this would actually have both these
properties
if you're careful about decaying the
epsilon properly
so you should indicate it too fast
because then you won't be exploring
infinitely often
but you should also decay eventually
and this can actually be seen by just
summing overall episodes 1 over k would
be
proportional
the summation of that over time would be
proportional to the logarithm of k which
is a growing function so if k sub k goes
to infinity this will also go to
infinity albeit very very slowly you
could also decay your epsilon a little
bit slower than this and that would also
be fine
so there's a theorem and i'm not going
to prove that here i'm just going to
state that basically to reassure you
that this is fine so if you have a
really in the limit
exploration policy which also explores
infinitely often so a glee policy
then more free control with monte carlo
estimates converges to the optimal
action value function in the limit
we'll see a variation of this shortly
which has also been proven in which you
can do this even with shorter evaluation
phases
so that's reassuring we've kept the
optimality that we had from our dynamic
programming algorithms but we can do
this now in a fully model-free approach
of course that was for the monte carlo
case and we've talked a lot in previous
lecture about
benefits of doing temporal difference
learning rather than monte carlo so now
maybe somewhat obviously we're going to
turn to that same question and we're
going to
see if we can use temporal difference
learning for control
so to recap temporal difference learning
has several advantage over monte carlo
learning
it has lower variance in the updates it
can be used online and it can also learn
from incomplete sequences which also
means that for instance if the episodes
are really long or if you only have
exactly one episode in your whole
lifetime temple difference learning can
still learn whereas monte carlo would
not be able to learn during the episode
so an actual idea would be to use
temporal different method temporal
difference methods instead of monte
carlo methods to do control so basically
we wanted to apply temporal difference
learning to the action value functions
to have the same convenience of being
able to easily pick a greedy or an epson
greedy policy
and then for instance we could use epsom
greedy policy improvement as before
but we could also continue to update
every time step because temporal
difference learning can learn from
individual transitions it doesn't need
complete episodes
so one way to do that is the sarsa
algorithm i briefly talked about this in
the previous lecture
sarsa is just temporal difference
learning for state action values and
one way to depict this is to look at
this diagram where we started an action
which corresponds to some action a taken
from some state s which is not in the
diagram and then we see some reward r
and we reach a new state s prime
and then we consider already what the
next action will be under our policy a
prime
and then we use this to construct an
update so our target here will be the
immediate reward as always plus a
discounted next state action value of st
plus 1 80 plus 1. and we're just going
to compare this target to our current
estimate as usual
have a small step size to get rid of the
variance or the noise in this update and
then we just update our action value
functions incrementally using this
update
sarso comes from the name sarso comes
from the usage of a state action reward
state and next action so that's state
action rewards action sarsa
now this is a very natural extension of
the temporal difference learning
algorithms and we briefly touched upon
it last lecture but then maybe somewhat
obviously you can use this in place of
monte carlo learning
so this is the proposal we're just going
to do generalized policy iteration again
but now we're actually going to consider
instead of taking full episodes or maybe
even multiple episodes we're actually
just going to do this time step by time
step so each evaluation phase will be
exactly one time set long
and then we immediately consider the
epsilon gradient policy with respect to
our new action values
now
if you think about this when you do this
epsilon greedy policy
the greedy action might not have changed
on a single update so it might be that
your policy actually remains fixed for a
couple of time steps which means that
the policy evaluation will also last for
a couple of time steps but occasionally
or sometimes maybe fairly often it might
be that the
identity of the greedy action does
change
and that means that the policy
evaluation step can last as short as a
single step
before you immediately apply this
greedification step this epsilon
gradification step
but then the next time step you'll
evaluate this new policy and we are
still doing a version of generalized
policy iteration it might not be
immediately obvious that this algorithm
is sound and will converge but actually
this has been proven this is not trivial
to prove
and i can give you pointers to papers in
which this is proven but i for now i
will just assure you that this is a
sound algorithm
so this is how that looks if you want to
write it down a little bit more
explicitly in pseudocode this is from
sutton ambarta and this is for the
tabular algorithm so we're going to use
a capital q to refer to a specific cell
in the table corresponding to some state
and action and now for each state action
pair we're just going to arbitrarily
initialize this for instance at zero
and then we have this double loop where
the outer loop is on episodes and the
inner loop is on steps within the
episode so each episode starts by
initializing the state
here think of state as the initial state
of your episode which is given to you by
the environments
for instance these states could simply
be the observation
and in the simplest case the observation
could be the full environment state
if you're in a fully observable problem
that would be the case and then as
agents did you could use just use that
observation
more generally when we say states this
should be the agent states
so the initial agent state could also be
due from the first observation in the
episode
somehow inferred for that from that then
we immediately pick an action according
to our policy and the policy is derived
somehow from the action value typically
although it doesn't have to be for this
algorithm we could also be using sarsa
for pure prediction as we talked about
in the previous lecture but in this case
if we're wanting to do a control for
instance the policy could derive from
our action values and could be epsilon
greedy
and then in the episode we just repeat
uh the following steps where we actually
execute that action in the environment
which means we can then observe a reward
in the next state or next observation
that helps us construct the next state
and then we immediately pick the action
that we're going to execute in that next
state
and then again this is from a policy for
instance derived from the action values
and then we use that action to update
because we're going to update the state
action value by adding this temporal
difference error which is the reward
plus the discounted state action value
at the next time step minus our current
estimate
and then we will just replace s with s
prime and a with a prime and we start to
loop again so on the next step we'll
actually execute this a prime that we've
just chosen so we picked the action here
and then on the next loop we'll actually
execute it in the world
and then this just continues until the
state is terminal so refresh the final
step of the episode and then we repeat
the next episode we start again in a new
state
so this would be the sarsa algorithm
it's not too complicated to implement
although you should be careful to get
the details correct because otherwise
there will be no learning
and
this algorithm as i promised there is a
theorem that this also converges and
specifically we need the same condition
that we needed for monte carlo control
which is that the policy should be
reading the limit with infinite
expiration
the gradient limit part is to ensure
that the action at the next time step
will eventually become the greedy action
so that we're actually evaluating the
greedy policy
instead of evaluating whatever policy we
currently happen to follow that would be
a different thing
and the infinite expiration is just to
make sure that we actually update all
state action values indefinitely often
because otherwise we might have
irreducible
uncertainty about some of these eight
action values which might lead us to
draw the wrong conclusions about which
policies are preferred
okay
so that's one way to go monte carlo
learning and social learning and in both
cases what we're doing is we're
interleaving this policy evo evaluation
and
this policy improvement step now we're
going to turn to a new topic which is
off policy learning which is about
learning about a policy different from
the one that we're following and there's
a specific very popular algorithm that
is an off policy algorithm called q
learning which we'll also talk about
here
before we're going to explain what
off-policy td and q-learning are i'm
first going to recap briefly from the
dynamic programming lecture
that there's different ways to find the
value of the
optimal policy
and specifically we talked about these
four different algorithms essentially
that are on the slide
where there's two algorithms here for
policy evaluation
and we've just discussed
how we could use policy evaluation in
policy iteration if we interleave the
pulse evaluation with a policy
improvement step
there are also algorithms in dynamic
programming which we can use to
immediately estimate the value of the
optimal policy in a more direct way and
one way to think about this
is that we're basically interleaving the
greedification very tightly with the
evaluation so this is what value
iteration does and if we look at this
equation here the second equation what
we see is we basically immediately
consider a greedy action
with respect to one step
uh look ahead according to our model
this is dynamic programming we'll go for
the we'll turn to the model three
approaches in a second so here we're
just assuming that we have a model so
we're just going to consider the maximum
value if we roll the model forward one
step and then bootstrap on our current
estimates that we have
so basically you can view this as
immediately doing a gratification of
this estimate
in one tight loop
and value iteration as we have discussed
or diana has discussed will also
converge and it will converge to the
optimal value function and hence you can
infer the obstacle policy
and then for both these categories
policy evaluation and value iteration
there are two variants one for state
action
sorry for state value learning and one
for state action value learning so we
see these v state values and q state
action values
we noticed that the policy evaluation
for
state action values looks a lot like
sarsa and indeed there's a strong
relationship to those
and indeed all of these algorithms can
be well not all of them many of them can
be turned into model 3 algorithms which
is what we're going to do here
so we already talked about temporal
difference learning in the previous
lecture and sarsa in the previous
lecture and also just now
which can be viewed as sampling the
policy evaluation algorithms going back
to the previous slide where there's an
expectation here which we can sample
according to our current policy
and then we can use this as a target to
update slightly towards
in doing so we introduce a step size to
only make a small step because we will
have a noisy target unlike in dynamic
programming
and hence we want to somehow average out
the noise and for this we use a small
step size to make sure that we don't
jump too far and that the updates become
too noisy and then we can see that td
corresponds to sampling this first one
where assassin corresponds to sampling
this third one
but there's a third algorithm and that
algorithm is called q learning and it
corresponds to sampling
this last equation where there's also an
expectation outside but there's a value
iteration equation here which has a
maximization over the actions inside
i've talked a little bit about value
iteration here on the second line where
we immediately go for greedy
in this last line because we're
considering a state an action we're
actually already conditioning on that
first action so that one can't be taken
to be greedy that's just
for any action you can do this
evaluation
but then we can still roll forward one
step and consider in the next states
the greedy action the one that maximizes
your action values
this is still a valid value iteration
algorithm and it's one that we can
sample which we've done here
so we see that q learning takes our
state action value and updates it very
analogously to sarsa where the only
difference is how we bootstrap instead
of considering the next action in the
next state we consider the best action
that we could possibly take
according to our current estimates and
then use that as our estimate for the
greedy value of our current
value function
and back that up
note
that there's no trivial analogous
sampled version of this algorithm which
is the value iteration algorithm for
state values and i want to encourage you
to briefly think about that why would
that be the case why are there four
dynamic programming algorithms and only
three model free algorithms on these
slides
so feel free to pause the video whenever
you want to take a second to think about
these things i will now immediately give
the answer to that
which is that it's
in some sense a convenience thing where
out of these dynamic programming
equations three of them have the
expectation on the outside so it's very
convenient to sample that
but the value iteration for state value
functions has a maximization outside the
expectation and that means that it's
normal trivial to sample this it's not
immediately obvious what the model 3
algorithm should look like if you wanted
to do this
so that's why we don't have an analogous
model free algorithm here
okay
now i want to make the distinction
between on policy and off policy
learning in some generality and i will
connect this to this new q-learning
algorithm
on-policy learning is about learning
from a behavior
about a behavior policy from experienced
samples from that policy that's what
we've been discussing so far for the
monte carlo control algorithm and for
sarsa and also in the previous lecture
on policy evaluation we always consider
there was just one policy and that
policy would be used to generate your
behavior because you want to make
predictions about the value of that
policy
we call that on policy learning because
we're learning about the policy that
we're following
we could also consider off policy
learning which refers to everything else
so on policy learning is in some sense a
specific case and off policy learning is
everything else which means that we're
learning about a target policy pi
but the experience sample is from a
different policy which we could call mu
sometimes in the literature the
terminology is slightly different
sometimes people use b instead of mu
that's kind of not too relevant we'll
try to consistently use mu for the
behavior in the upcoming slide and then
pi for the target policy so please keep
that in mind that mu is also just a
policy to select actions it's a
distribution of actions in each state it
could be deterministic if you wish
and then we want to learn about a
potentially different policy a target
policy pie
this refers essentially to learning
counterfactual about what ifs about
things you could be doing
so to give you some examples for
instance you could have a question this
is a valid predictive question
what if i would turn left maybe you're
not turning left all the time or maybe
you're turning left occasionally in some
cases but not in others like
consistently
it could also be you have a random
policy that turns left uh
stochastically everyone so often rather
than occasionally
and then we could have a question that
is but what if i would turn left all the
time
so that could lead to new observations
new rewards or we could have a question
when we're playing a game what if i
would have played more defensively would
this change my wind probability
or we could just for instance ask what
if i would continue to go forward let's
say you're a robot and you may
continually make predictions about how
many steps until i bump into a wall that
could be a value function where the
reward could just be the number of steps
in the episode termination could be when
you bump into the wall that's a valid
predictive question
but if we don't follow that behavior all
the time then we might still want to
make this prediction
so i haven't told you how to make those
predictions how to use your data to
learn these things but i just want you
to realize and think about these as
valid predictive questions that could be
helpful for
an agent in the world and maybe even
want to ask multiple of these questions
these are off policy questions because
they refer to asking questions about
what if behavior that i'm not currently
maybe
doing all the time
so in general we want to evaluate a
target policy to compute the estimated
value of that policy either the sales
value or state action value
and we're using a different policy mu to
actually generate the actions so why is
this important well
one reason why you might want to do this
is you might want to learn from a
database let's say you have some stored
experience for instance from observed
from human behavior or from other agents
from for instance data that was logged
somewhere
the behavior there is not under your
control so there is just some frequency
of actions being selected in the data
and maybe you have a what-if question
what if we would have instead followed
this strategy
of course the strategy should be
supported by the data there should be
some samples in the data that will
support and give you data about that
question otherwise you can't learn but
this is a valid question that you could
have where the experience might differ
slightly for instance all actions might
be selected selected under the database
but not on the same frequencies as you
want to consider them
and then learning of policy is kind of a
natural thing to be doing
somewhat similarly you might want to
reuse your own old experience
so far we've considered algorithms that
learn online so they basically consume
data and they update themselves or they
wait until the end of the episode like
monte carlo does and they update
everything at the end of the episode but
then the data is thrown away and you
move on to the new episode and you only
use the information from the new episode
to update your policy from that point
onwards
now arguably this is potentially a
little bit wasteful and indeed it's a
fairly popular approach to store the
data that you've seen
and try to reuse that to learn more but
that means if we're doing control your
policy may have changed in the meantime
when you were collecting the data you
may have had a worse policy for instance
or just a different policy or you could
have been exploring more at the
beginning
so that means that in general the
behavior policy the probabilities of
selecting certain actions might be quite
different in your old old data than it
is currently but you might still want to
learn about it you might still want to
use it
separately in addition to these other
cases you could also want to learn about
multiple things at the same time so this
is related to these what-if questions
that i showed on the previous slide
where even if we're following one policy
and we're not trying to learn from past
data or from other sources of data we
might still want to learn about many
things many what-if questions because
this is a way to accumulate knowledge
about the world
so that refers to learning about
multiple policies and then therefore all
but one of these policies should be of
policy at most one can be exactly your
behavior policy so any other policies
would be you would require off-policy
learning to learn about them
and finally now we're going to turn
towards this one a little bit more in
detail you might want to learn about the
greedy policy while following an
exploratory policy
this one is maybe obvious in the context
of policy iteration where we're always
interested in this policy improvement
step which can be implemented by making
your value functions or your policy
greedy with respect to your value
functions
so it's a valid question to ask what is
the value of the greedy policy because
that's the policy that we're going to
consider next
but we might not just want a sample from
that greedy policy because of reasons i
mentioned before that you might not
explore sufficiently often
this brings us back to this q learning
algorithm q learning estimates the value
of the greedy policy
it's conditioned on the state action
pair
oh sorry this x s and a should be st and
18.
so we're considering a specific state
and action pair and then we're taking
one step the reward and then in the next
state we consider acting greedily
so we are learning about the value if
you take this action but then in the
next step you would be ready
and that's a valid target to update and
it would be
learning about the greedy policy
but you don't want to necessarily act
all the time according to that policy
because that wouldn't explore
sufficiently
now fortunately it turns out
if you do this q-learning algorithm if
you do the tabular version in specif
specifically
then you will convert to the optimal
action value function as long as we take
each action in each state infinitely
often
now this might sound similar to what i
said about sarsap but there's an
important difference here which is that
you don't actually need your behavior to
become greedy in fact
q learning can learn the optimal value
function eventually
even when you explore uniformly at
random throughout your lifetime
of course to have actual conversions
because this is a random process and
you'll have random data this is only
true in the limit so you do need
to get infinite data in the limit but
you don't actually have to make your
policy greedy anymore that's a big
difference between queue learning and
sarsa
so this works for any policy that
eventually selects all actions
sufficiently often the uniform policy is
one example epsilon greedy will be a
different example but we no longer have
to decay the epsilon
you do require appropriately decaying
step sizes this is true for all of these
algorithms including sarsap but also for
policy evaluation with temporal
difference learning
and for instance the robin's monroe
conditions as they are called suffice to
have that and these conditions are that
your step size
is chosen to be such that if you sum
over all time steps until infinity
that in total your step size will be
infinitely large this ensures that from
any state action value estimates that
you currently have you can always move
far enough to find new estimates that
are more
more optimal
and then in addition to that we want
that the sum over all time steps of the
squared step size
is less than infinite it's a finite
number this essentially ensures that
eventually your step size becomes small
enough that eventually you cancel out
all the noise that you eventually start
averaging enough data in that you can
get rid of all of your uncertainties
these two conditions together ensure
convergence of the q learning algorithm
if you also explore indefinitely and for
instance a specific choice that you can
consider here is
one divided by the time step to the
power omega where omega is somewhere
between half and one
so the extremes here are one over t or
one over square root of t on the other
hand if you would pick
anywhere between those you would have
these conditions so that would be a
suitable step size parameter
and we can see that basically the step
size would then decay over time and
eventually you make smaller and smaller
updates
so that kind of tells us that q-learning
is a good algorithm that we can use um
and now i'm going to step through an
example a little bit in more detail to
show you the differences between
q-learning and sarsa
so for the example we'll go to the
blackboard
essentially
and let me just depict some state
which you could call s
and maybe there's some action you could
take and there's some other actions you
could take
and let's say for instance that we've
taken this action below here let's call
that a
and then from this we transition to a
next state
maybe for argument's sake let's say we
transition deterministically there could
be multiple states of course that we end
up
and then we consider
another step
and there's also some reward along the
way here let's say that the reward was
plus one or something of the form
and let's consider this one transition
here where we were in a state we took an
action we saw a reward and we saw next
state
now let's arbitrarily say that here
initially our action value function
was
let's say for this arrow pointing
downwards was say
one
then we could consider what happens if
we use sarsa or q learning to update
this action value function
now in order to through the update we
have to establish a little bit more so
let's say that discount factor is a half
for concreteness of the argument and
let's say that we have some estimates so
here
let's say that our q
estimate of s prime for the up action
might be for instance
let's say plus 4.
and then we also need some sort of an
estimate for the down action
i haven't yet told you which action
we're going to take
we'll get to that in a moment and let's
say the um
down action is only worth two according
to our current estimates now these are
not the true values these are just our
current estimates
then we could consider what the targets
are that key learning sarsa uh judo
mercer would use
now let's say that we've actually
selected
this action to go down then q learning
doesn't actually care too much about
which action you've actually taken
so the target there would be reward plus
discounted
maximum action in the next state
which in this case would be plus one for
the reward
plus
a half which is our discount factor
and then we look at this max action in
the next states which in this case would
be plus four
so in total this was give would give us
a target of
three
so depending on our step size we had an
estimate of one to start with
and then depending on our estimate
we are depending on our step size we
would update this estimate
uh some portion upwards but it would be
updated in any case upwards towards
three
phrase differently our temporal
difference error will be plus two and
then we add whatever our step size is
times that temporal difference error to
our estimates
so for instance to make a very concrete
if our step size would be 0.1
our new estimate
would be
1.2
you can you can calculate that yourself
if you want wish
now sarsa
needs the actual action that we selected
so let's say for argument's sake
so first let's write down the sarsa
update it takes the next state and the
next section into account
let's say for argument's sake that it
selected the action to go
down
then
everything else is the same the reward
is the same the discount is the same but
the next value would be plus two it's
the value of the action of taking down
which would mean that sarsa would only
give you a plus two
or
alternatively
if the up action was taken it could also
have generated a target of plus three
so
if
a prime
is
down
this is what happens
and if a prime is up
this is all the same
but the update is slightly different
so we can see that sars would perform
the exact same update if it would happen
to pick the greedy action but there's
some probability that it would select a
different action and then the update
would be different
that's okay it doesn't mean that one is
necessarily better than the other and in
fact in both cases here the target would
be higher than our current estimate so
we would update our estimate value a
little bit upwards but of course if we
see a plus two as a target we would
update it upwards less than if we would
see a plus three
one could also say that if if in fact
our policy was uniformly at random let's
say it would pick both of these actions
with the same probability
then in expectation the structure target
would be two and a half
instead of two or three
okay so i want to conclude the example
for now there
and we'll go back
to the slides
so now i actually want to show you
another example that would illustrate
differences between sarsa and q learning
and now we're going to use both these
algorithms for control so we're trying
to optimize something
and this is called the cliff world or
it's just a cliff walking example as it
says on the slide and i'm going to
explain the exam example to you and then
i'm going to explain the plots that you
can see below
so the example is quite simple it's kind
of a great world where you start in
state s
and in every step you get a reward of -1
there's no discounting so your total
reward in an episode will simply be the
number of steps you've taken and then
there's a goal all the way on the right
hand side
and if you reach that goal you end your
episode
this is beneficial because it means that
the minus ones stop so you're actually
encouraged to find the goal as quickly
as possible because the maximum value
will be attained if you're minimizing
the number of steps you take in an
episode
now there's an additional important
thing happening here which is that
there's this region here at the bottom
called the cliff and if you step into
that you get a reward of minus a hundred
and you transition back to the starting
state
um i believe this also terminates your
episode so the episode would then end
with a minus 100.
so what would be the optimal thing to do
here well the optimal policy
is shown in the figure
and it would
actually start here in the start set as
always would go up one and then walk
right past the cliff
and then it would go down onto the goal
you could also consider a safe path you
could go a couple of steps away from the
cliff and then walk according to the
safer path
now
we want to learn
the optimal value function in this
problem perhaps and we want to have the
optimal behavior but we know we can't
not explore because we don't we are
assuming here that the agent doesn't
know the structure of the problem
otherwise we could use dynamic
programming so instead we're just going
to learn from experience and we're not
going to assume that the agent knows
everything so it has to try each action
in each state
and we'll try these two different
algorithms sarsa or q learning to learn
in this example
and we're going to use an epsilon greedy
policy here
now what you see here at the bottom
bottom quite interestingly is the actual
rewards first per episode sorry that
these different algorithms attain
throughout learning and we can see that
after less than 100 episodes maybe after
even 50 episodes or so the algorithms
have both stabilized
to a certain apparently to a certain
policy
which gives us them a certain amount of
reward per episode
interestingly sarsa has stabilized
around -25 which means that it takes
roughly 25 steps to reach the goal
whereas q learning has saved lives
around minus 50.
does that mean that q learning also
takes minus like roughly around 50 steps
no actually it means that q learning
regularly falls into the cliff
so why does this happen why does q
learning perform worse here than sarsa
didn't i explain that both of them are
optimal in the sense that they converge
to the obstacle policy
and in fact i didn't say that we're
decaying the epsilon here which we're
not
and didn't we say that q learning can
convert to the optimal value function
when we don't decay the epsilon whereas
cersei cannot so what gives here why is
sarsa performing better than q learning
the reason is that sarsa is an on policy
algorithm and being an on-policy
algorithm this means that sarsa will
evaluate the policy that we're actually
following
and will give us valid estimates
accurate estimates for that policy
so at first the policy is changing right
but at some points it will kind of um
stabilize
even though we still have an epsilon so
there will be some randomness to the
policy but the policy will be um stable
because the epsilon's not changing but
also because the action values won't
change that much anymore
and that means that even though the
epson green policy depends on your
action values if your action values
don't change that much anymore the
policy can become quite stable
but this policy does explore and that
means that there's some non-zero
probability that the policy will
actually fall into the cliff and that's
captured here in these average rewards
per episode
um
so what does sarsa do sarsa being on
policy algorithm
accurately estimates that if you have
this epsilon if there's a probability to
explore then if you would be very close
to the cliff there's a pretty high
probability that you'll fall in
let's say the epsilon is 0.1 that means
that 10 of the time you pick a random
action from this state so this could
mean that you either go back or you go
up or you go right or you follow to the
cliff
so if you try to walk along this optimal
path there are many opportunities for
the algorithm to step into the cliff
and get this reward of minus a hundred
sar says estimating these values so
basically he starts to say well if i
have a choice
maybe i shouldn't walk so close to the
cliff because i know i will explore i
know that there will be some randomness
in my policy i know what the value is of
that policy and turns out maybe then if
i'm very close to the cliff
the
higher value action is to actually step
away from the cliff first
to this safer path which is a couple of
steps away from the cliff in which you
can reliably get to the goal
because if if on the safe path you pick
a random action you could step down but
then you have an opportunity to step up
again on the next step
of course occasionally it can happen
that you explore twice in a row it could
be that you step down randomly and then
step down again
but then if you're on the safe path at
least you can recover again immediately
whereas if you're immediately next to
the cliff just one exploratory action
will toss you in so why then is q
learning performing worse well what
we're evaluating here is the actual
rewards per episodes which does include
this expiration this means that
q-learning is actually has estimated
maybe by now accurately the values of
the optimal policy and maybe the values
of the optimal policies will tell you or
actually quite likely the values of the
optimal policy will tell you to walk
along this optimal path
but then if you execute an absolute
greedy algorithm around this the epson
greedy algorithm will try to walk along
the path but occasionally will be bumped
back or up
but then location will also fall into
the cliff
so on average because q learning is not
taking into account that it's actually
exploring it will do worse
if however you would take both these
algorithms let's say here at the end of
learning and then you would look at the
action values and then execute the
greedy policy according to both these
algorithms then q learning would
actually have found a shorter path and
it would actually walk very reliably
next to the cliff without any
exploration towards the goal
so it really depends what you're after
here sarsa
does better because it takes into
account the exploration that we're doing
but q learning has actually found a more
optimal policy if we would only execute
that policy instead of the exploratory
one
this is more meant to illustrate
differences between on and off policy
learning than it is to say sarsa is
better than q-learning or vice versa but
it's important to appreciate these
algorithms have different properties and
this is meant to illustrate one of these
properties
okay now we're going to turn towards a
different topic which is a different
failure mode for q learning because it
is known that two learning can
overestimate the action values
so this is about classical q learning as
discussed before
and the issue that we're going to talk
about is related to the following so
first we're just going to write out this
bootstrap target that is used in q
learning as mentioned before we're just
looking at the action values we're
picking the highest one we can actually
write this on differently as done on the
slide where we consider the
uh
arg max which means the maximum elements
of our estimate at st plus one and we
could consider that basically to be a
policy right we're picking a policy and
specifically we're picking the policy
that is greedy with respect to these
action values and we can write it down a
little bit verbosely a little bit like
more explicitly over here by saying
there's a policy that picks the highest
valued action maybe breaking ties
equally or something like that and then
we consider evaluating the value of that
action
this is equivalent to just taking the
maximum action value but writing it out
like this we can notice that we're
actually using the same value function
to select this action and then to
evaluate the action
but these action values are approximate
this in particular means they could be
wrong
and what this means in practice is that
you're actually more likely to select an
action value that happens to be high
than you are to select one that happens
to be low
so
what this means is if these values have
some noise they're wrong because you
only have finite samples or because
you're doing function approximation or
all sorts of reasons why your action
values might be wrong
then in fact some will be estimated
slightly too high by accident others
will be estimated slightly too low by
accident but this
this phase here where we're selecting
the action
will
very likely pick an action that is
overestimated much more likely than to
pick one that is underestimated
especially if the true values are fairly
close together let's say our true action
values in this state are actually all
zero
but so far we've seen limited data and
we don't have accurate estimates for all
of them yet you don't know yet that
they're actually all zeros so you think
some of them are positive some of them
are negative
but then picking exactly according to
the same function to evaluate it
we are picking an action that has a high
value and then we're saying oh yeah i
picked that because it has a high value
and now the value is high that means we
have an upward bias
so let's illustrate this with an example
so consider the game of roulette this is
a gambling game played in casinos and
i'll briefly explain the game details
are not too important
but we're going to formalize the game as
follows where we have
one state which is that we're at the
table and we're playing and in this
state we're going to consider
171 different actions
this is because there's actually 170
different
options of plays you can do on the
redhat table
roulette is a game in which a number is
drawn at random by placing a little ball
on the spinning table and then the ball
will fall into a slot and the slot will
correspond to some number
the number will also have a color so the
number could be a red number or a black
number and i believe the numbers go from
1 to 36 or something of the form
in addition to just being able to pick a
number or a color you could also pick
whether the number is odd or even
or whether it's in the first half of the
number so 1 to 18 or in the second half
19 to 36 so there's all sorts of
different bets you can place
and then in addition to all of these
there's a number zero which doesn't
actually have a color so it's not
covered in any of those and it's not
considered odd or even
so the number zero is green and you can
bet on zero but you it won't actually
pay out if you have chosen either red or
black as a bet
then these bets they will pay out and
what they will pay out is associated to
the likelihood of it happening so if you
bet one dollar on a red and a red number
comes up you get two dollars
if you bet one dollar on specifically
the number 17 and then 17 comes up you
get 36 dollars
so that means that
the win probabilities are random
obviously and um
on average you'll actually lose on every
action a little bit
because this zero could come up
now our actions here as we formalized it
not to make it too complicated is we're
always just going to bet exactly one
dollar
this is to avoid any issues with um
also having to pick betting amount which
would blow up the action space uh
potentially enormously
in addition to this we're not going to
take into account your current uh funds
so we're not going to consider you being
able to go bankrupt so instead the
action will just you will just be you
pick a
play to make you bet one dollar and you
get your reward associated with that and
then you can
transition back to exactly the same
state again so then you can play again
and then there's one additional action
which is to stop to play which will pays
out zero
so you either play a betting action and
then you can play again and again and
again
or on every of these
time steps you could always choose to
walk away
so all the other actions that don't stop
the play will have high variance and it
will have a negative expected value
because the existence of this zero on
the rilet board
betting actions do not end the episode
so you can continue to play indefinitely
okay i hope that's somewhat clear so the
idea here is that this is a fairly high
noise environment
which means that our action values will
be
fairly inaccurate for a while at the
beginning
but it's a valid markov decision process
and we can consider doing q learning and
q learning is known to convert to the
optimal values even eventually so it's
an interesting question to see how well
it pairs here
and what we find is that it actually
performs rather poorly
so what we see here is the expected
profit according to the q learning
algorithm so we're basically just
looking at the values that it's
estimating so the action value estimates
and what we see here is um
[Music]
there's some discount factor here at
play as well i think it's probably 0.99
but i should look that up um but we can
see here that g-learning essentially
thinks that even after updating all of
the actions a hundred thousand times so
we've had we've seen 70 million samples
in total for a tabular q learning
algorithm with just one state
we see that chief learning is nowhere
near the actual values it still thinks
it can actually win more than 20 dollars
as long as it just keeps on playing
so that's rather bad because it means
that q learning is in some sense some
something of a gambling addict here
and the actual values is much much
slower and also q learning doesn't seem
to be getting much closer
so that's in the picture a little bit
misleading it seems cooling has fully
stabilized it hasn't fully stabilized
this is actually using a decaying step
size and humidity is actually guaranteed
to converge
but the convergence will be extremely
slow that's what's happening here so
eventually clearly will get there but
much slower than is practical
we've avoided any issues with
exploration just for this example just
to make sure that we're not conflating
anything
and therefore we've just selected every
action
simultaneously on every time step
so instead of just picking one action
we're just picking all of them at once
sampling the reward for each of these
actions at once and then updating the
action value function for each of the
actions at once
and then we just do that hundred
thousand times
so that's pretty bad and this you could
imagine this showing up in larger
problems as well of course this is a
small contrived problem of specific
betting problem but you can consider
this to be part of a much larger mdp
let's say you're literally
trying to learn a robot how to act in
the world and maybe the role has some
sort of a uh money amount which is maybe
somehow part of its reward function
and then there's a casino in this world
so at some point the robot could walk
into the casino could start playing and
could find itself in a little sub-slice
of the real big problem which
corresponds to the smaller problem then
if that robot will be doing q learning
it would still stay at the table and
think it would win if only you could
play longer and longer and longer
so that seems problematic so there's a
question here can we fix this
unfortunately there are ways we can do
we can go around that and we can go back
to the main source of the problem here
so q learning essentially overestimates
because it uses the same value function
to select an action to evaluate the
gradient action and then to use the same
value function to actually evaluate that
action
so the solution here
is to decouple those two so in roulette
to make it concrete it's quite likely
that some of the betting actions that
you've done so far will have one on
average
maybe slightly more often an even number
has come up than an odd number or maybe
the number
14 has popped up slightly more often
than would be expected on average
there's always going to be some actions
that will have one more than they will
will do in the long run
but that means that on every step cue
learning transitioning back to that same
state thinks i and now i played an
action that didn't quite win
but on the next step i could play an
action that does win there's always an
action left that does look good or at
least for a very long time
so q learning will then update as if
this statement transition back to
actually has a high value because it
literally picks the maximum that it
could
and therefore maximizes over all of the
noise in the estimates
and the solution as i said is to
decouple the selection from evaluation
which we can do as follows
and this algorithm is called double q
learning
so the double here refers to the fact
that we're going to actually maintain
two different state action value
functions
and we're just going to basically
introduce a new action value function q
prime
so in equation one here we see that we
have our standard thing with the one
step reward and then discounted value of
the next state but the value of the next
state is defined by picking an action
according to our normal action value
function q
but then evaluating the value of that
action with q prime
we could also do the inverse we could
pick according to q prime and then
evaluate according to q
what we can use do then is we can use
each of these as targets for these two
different action value functions so on
every time step what we'll do is we'll
pick one of the action value functions
to be updated so either q or q prime
and then if we pick q we use equation
one to update it and if we pick q prime
we use equation two but we never update
both at the same time because this would
correlate them and then we would just be
doing q learning again but if instead we
pick either one or the other this d
correlates the action values they see
similar data about different sub slices
of the data and this means that we can
get a somewhat unbiased estimate
of the greedy action according to one of
them so we're still picking greedy this
greedification step this selection
process is still likely to pick an
action with a high value
but in as much as that value is purely
high because of for instance random
noise then having a separate evaluation
learned on different data might help us
to get an unbiased estimate for the
value of that policy
so that's the whole algorithm
essentially but we don't have to
actually um throw away data so we don't
have to only consider one of these
action values when we actually wanted an
inferior policy let's say we want to act
epsilon greedily or eventually we want
to select a greedy policy with respect
to this data
then we can use both action values so
the policy itself the
the way we select our actions can be
based for instance on the average of the
two values value estimates so we're not
losing any data in the sense that we
might need double the amount of data or
something like that no that's not
necessarily true we can still merge them
together to get our
estimates as we use it for the policy
we can also actually prove that q
learning double q learning does converge
to the same optimal policy under the
same conditions as q learning which is
nice this tells us that we can trust
this algorithm that asymptotically in
the tabular case at least it does
converge under the same properties that
q learning does
so that means that the algorithm is
sound but this is defective does it do
what we want it to do so obviously we
can apply the security sorry to the
roulette's example that we've seen
before and we find that indeed it does
much much better by decoupling the
selection and evaluation
we see that it's much more accurate in
its value estimates and indeed this also
means that the optimal
thing to do according to the estimates
learned by double q learning is to walk
away from the table
actually turns out this is not specific
to this tiny example and we've seen uh
previously a video of an agent playing
atari games
i mentioned briefly that this algorithm
was using something called dqm
dqn is short for deep q network
essentially what's happening there is
we're using the q learning update with
deep neural networks
and
that's very effective and it was
actually also using epsilon greedy so
you now know most that you need to know
to implement a dqn algorithm but we'll
go into more detail in a later lecture
and then we can apply this idea of
double queue learning to this algorithm
because it's using q learning and
therefore it might be susceptible to
these overestimations
so double dqm
does something very similar to the
double q learning algorithm as i
explained it just now with a slight
difference because something i haven't
explained yet is that dqn uses something
that is called a target network which is
already a separate network so dqn
already used a separate network to
bootstrap on but this was used more as a
stability thing where this separate
network is just a slow moving copy of
your
queue function that you're updating as
you go
and then because we already have a
separate network this makes
implementation of double queue learning
quite simple because we can just use
that separate network to evaluate the
actions but we could still pick the
actions according to the online network
that we're updating all the time
these details might be a little bit
fuzzy
for now because we haven't fully
explained dqn yet that's okay the main
purpose for this slide is to show that
if you apply something like double q
learning at skill to these atari games
then indeed we also see a large gain in
performance so the red here on the on
the right hand side would be q learning
how well it performed across all of
these different atari games and then the
blue is double q dq which is
basically like a one-line change in
terms of the full algorithm
so what does this mean this means that
double
queue learning might be important in
practice and indeed it's quite a common
component these days of many dprel
algorithms
and the reason why it's important is
because these are estimations they do
actually happen they don't just happen
because of random noise as in the
roulette example they also happen
because of function approximation
inaccuracies so there's various reasons
why your value estimates might be
inaccurate and then double learning can
be quite useful to
help
mitigate that
it's more general than double queue
learning and it can be generalized to
other updates
for instance if we're doing sarsa you
might think you're safe because there's
no max in the sarsa updates we're just
looking at the actual high of the next
action but turns out that's not quite
sufficient to be safe because we're
still actually picking that action
because it has a high value if we're
doing something like greedy or epsilon
greedy
so that means that sarsa can also
overestimate and then therefore we can
also think about using the same solution
this could lead to something we could
call double sarsa where we still use two
action value estimates
so now let's go back to the example that
we've seen before and look look at what
double learning would do in that case
so let's go back to the example
and let's maybe pick a different color
let's say yellow
and what we need for double learning is
we need a separate value function
estimate so let's say that these
estimates here at the next state
they were just random and we could have
maintained a separate estimate in that
state
for both these actions
and for instance maybe this plus four
was actually a little bit randomly
chosen and maybe it could also have been
zero
and maybe in this
other action for going down
maybe we could have had an estimate
it's maybe also zero let's see just
randomly we pick two values here so what
we're alluding to here is that this
estimate of plus four was actually quite
random and it could have been anything
else and similarly this estimate of plus
two that we had could also have been
anything else
and maybe we could also have had an
estimate of zero in both these cases i'm
not saying one is necessarily more
accurate than the other or anything like
that i'm just saying okay this is
another estimate that could have been
the case just to go through the
the update a little bit
so if this is the case then we can look
at the double q learning update and we
can just look at what it does so let's
write down double
queue learning here
and let's remind us of the update let's
consider that we're also still
interested in updating the action value
for q so we've randomly picked between q
and q prime we randomly picked q now
and then
we would have the same thing where we
have the reward plus the discounts but
then in the next states we would have
the arg
max
of
over action values and in fact
let me use the
eraser and let me give this a different
color
let's just pick that one
we're going to use q prime
to estimate so that's the difference
between q learning and double quoting
that we're using q prime there
and this would then
be equal to plus 1 again that's the same
everywhere plus the discount is the same
everywhere okay so what does this look
like this is the going to evaluate a
greedy action according to our online
queue
so the greedy action will be up
because it has a higher value according
to that online value
that we're updating but we're going to
evaluate this other value and that means
that we're going to actually evaluate it
at zero
which means that our update would tell
us
go to plus one actually that means that
our estimate won't change
so we can clearly see here the double q
ring
constructs the target which is not just
different from pure loading in this case
but it's actually also lower that's a
little bit by chance because it could
have been of course that these
evaluations were different
like this zero over here could have also
been seven or something else
but on by and large it is true that we
learning tends to pick lower values
because
q learning does is selection so for
instance it could also also have been if
both these actions actually have the
same value it could also have been that
this one was plus two and this one was
plus four so the other way around from q
learning in that case w learning here
would update towards plus two rather
than towards plus four
let me get rid of those for simplicity
now we could also consider updating q
prime i didn't say what q prime was on
the uh
starting state so we could also pick
something random here
[Music]
sorry
so we could also say q prime of this
states of going down
maybe this was let's say for simplicity
that is also one
and then let's consider what happens if
we're updating q prime so this would be
if
updating
q
and then if updating
q prime
double q learning would do r plus gamma
q
s prime
and then we take the arg max
of and now let me change colors again
q prime
reward is the same
discount is the same
but now we're going to pick greedy
according to q prime
well they both actually have the same
value so maybe we could also consider
the policy to be uniform across these
two
we could either break ties um
randomly if we consider uh the values
would be to be exactly the same but we
could also just say let's do it 50 50.
so let's just average both these values
so this would then be
plus three
which means we would get something that
is one
plus one and a half would be two point
five
so if we happen to choose to update q we
would actually keep it the same the
target will be the same as a current
estimate
but if we happen to update q prime
q prime is estimated at one and seems to
be a little bit low compared to the
estimation according to q in the next
state
again i'm stressing here that we're
actually not um
saying that necessarily one of these
updates in this example is better than
the other this is just to show you the
mechanics of the update this can be
quite useful
and in fact if you implement these
algorithms it can also be quite useful
to actually
just go through a couple of steps
and print a lot of stuff so just look at
what the actual updates are that you're
doing like how are the values changing
does it make sense how they're changing
you can do this in the long term you
could plot all sorts of statistics but
it can also be helpful to actually
really go into the small
little details and look at this step by
step
okay so this is just to show you the
mechanics of the updates and now let's
go back to the slides
we're going to move on to more
generically of policy learning we talked
before about different reasons to off
policy learning and then we went into
some detail to talk about
off policy learning
for
the greedy policy
because this is useful for value
iteration and policy iteration style
algorithms now we're going to talk more
generally about off policy learning and
specifically we're going to talk about
how to use important sampling
corrections to do this
so just to recap to recall of policy
learning means learning about one policy
pi
and we call it the target policy from
some experience generated according to a
different policy mu q learning is one
example where pi is the greedy policy
according to your current action value
estimates and mu can be any exploratory
policy
but there are our options as well and
fortunately there are some general tools
that can help us with this
but there's one important caveat you
just basically keep in mind throughout
which is you can never expect to learn
about things that you never do right you
can't expect to learn about a policy if
this policy contains actions that you
would literally never take according to
your behavior policy then you simply
don't have the data to support this
so we're first going to say this a
little bit more in general so the goal
is to find some function f with random
inputs x and the distribution d prime
sorry given some function f with random
inputs x and distribution d prime we
want to actually estimate the
expectation of f of x under a different
distribution
and the solution to this is to re-weight
the function so the idea is that this
function is going to shortly be the
update to our weight so we have some
sort of an update to our weights under
some sort of distribution and we want to
re-weigh this as if we're doing the
update under a different distribution so
let's very briefly step through these
steps they're not complicated we have
some expectation of this function of x
under some data distribution d in this
case
and what we this is what we want right
we want to actually sample fx under some
distribution d
but we don't have d we don't have that
distribution we're generating
the data under a different distribution
so let's first write out what this thing
means it's just a summation over all x
or the integral if it's a continuous
space but we're just going to
merge all of that into this one notation
of all x of the probability of x and
then times the function of x
what we can do is we can rewrite this in
a form that looks a lot like an
expectation under d prime by just simply
multiplying by d prime and also dividing
by d prime
and this we can write again as an
expectation and we notice that the thing
inside is now
d divided by d prime for every x times f
of x
what does this mean this means if we
sample this final thing we would
actually get an unbiased estimate of the
thing that we were looking for on the
left hand side these are all just
equations so these are all equalities
sorry
so that means that we can actually do
this we can sample this thing and get
actually an unbiased estimate for this
thing that we started off with
there is an assumption here of course
which is that the um
the data should support as i mentioned
before which we can also see here in the
ratio you never want to actually divide
by zero right this means that whenever d
is non-zero d prime also needs to be
non-zero otherwise this is not well
defined and you can't actually do this
and this is related to this caveat which
sounds maybe much more intuitive but
here's the mathematical side of this
and the intuitive explanation was well
if you never actually do anything if
you've never actually sampled a certain
x under d prime you can't expect to
learn from it
so this is the generic strategy that
we're going to use
and the intuition is that we're going to
scale up events that are rare under d
prime but common under d because then d
prime will be small but d will be large
and that's appropriate because we're
going to basically
have d prime right that's our sampling
distribution
and if things only very rarely happen
they're only very rarely sampled in your
data but they should they they would be
very common under the distribution that
you care about then obviously you should
scale them up in some sense right
and of course conversely if they're very
common on your data distribution but
they would be very uncommon
or maybe even would never occur d cap d
can be zero that's okay
then you would scale them down
so basically we're just going to average
all of the events that we see these uh
these random events and we're going to
re-weigh the averaging so that the
distribution that results from the
re-rating is appropriate
that's the generic idea
and now
we're going to apply this to
reinforcement learning and we're going
to start simple so we're first going to
apply this to one-step rewards let's say
you are doing policy evaluation for a
bandits but you're sampling according to
a different policy than the one you're
interested in
so our behavior will be mu as before and
we're just going to consider the
expected reward and the expected reward
only we'll get to value functions in a
moment
so we can again write this out as an
explicit expectation by
summing over all the possible actions
and writing down the reward as an
explicit function of the state action
pair
i'm using this notation here r s a
is the expected reward given that you
both have to stage and the action it's
just shorthand
if you have both saved in the action the
expected reward itself doesn't depend on
your policy anymore
then we're going to do the same trick
again we're going to multiply with the
behavior probabilities and divide by
them
which means we can write this as an
expectation under our behavior policy mu
which means if we sample this quantity
we get an unbiased estimate for this
quantity that we're actually after the
expected reward under policy pi
so
in short when following a policy pi we
can use this re-weighted sample as an
unbiased sample for the expected reward
under policy pi so when following mu
we can sample according to the policy pi
note that we're not changing the
sampling distribution so important
sampling sometimes refers to actually
picking mu in a specific way here we're
just using the corrections mechanism so
we're using this re-weighting of the
samples to make sure that the resulting
estimates are
unbiased
samples and now we can of course apply
this to estimating a value function
estimate
so we're going to generate some
trajectory we're going to use tau here
to refer to this trajectory generated
with our behavior policy mu
now we want to use the return
that we generated because that's all
that we have we have our behavior we
have some return and a return
is a function of the trajectory so we
can write it down quite explicitly so
note here we define we overload notation
slightly by saying ge could also be
interpreted as a random function of this
random trajectory
and specifically the function that it is
is just picks out the rewards and adds
them together with the appropriate
discounting
now we can use the exact same trick that
we just used for the banded case for the
one step reward case and apply this to
the monte carlo case
and in order to do so
we should have the probability of the
trajectory of that fuel trajectory
under our policy pi
divided by the probability of the
trajectory under our policy mu this
would be the appropriate re-weighting
in order to get an unbiased estimate of
the return under policy pi
for instance pi could be our greedy
policy mu could be our behavior policy
and we want to estimate the value of the
greedy policy
now this is somewhat of an unwieldy
thing so i've started writing it out
here on the slide so feel free to look
at this in a little bit of detail
what is this probability of a trajectory
under a policy pi
well i didn't
maybe make it very explicit in this
notation but the trajectory is meant to
start at some state st right so first
we're already conditioned on being an st
so the very first step is to consider
the probability of the specific action
that we've seen
given that states and this policy
we could also initially consider the
probability of being in that state that
doesn't really matter too much
um for instance we could think of this
at the beginning of an episode
but for now let's just consider s to be
given and then we consider probability
of selecting action a this will be
different under our target policy than
it is for our behavior policy so we're
doing the same process on both sides
here so we're expanding this probability
of the trajectory into its components
and then we also consider
given that we now have selected this
action so under the probability of
selecting this action there will be a
probability of the reward and the next
state that we've seen
and then there will be the probability
of selecting the next action and so on
and so on so the probability of this one
trajectory will be maybe somewhat
obviously fairly small because there's a
probability of selecting specifically
that action then transitioning
specifically to that state selecting
specifically the next section and so on
and so on
this is normal the probability of a full
trajectory will generally be quite small
because the trajectory could be quite
long and random
and then we do the exact same thing
for the behavior policy as well as we do
for the target policy
now we notice that there are some terms
that overlap and we can cancel those
particularly the transition dynamics are
the same for both the targets policy and
the behavior policy
given that we've selected an action the
probability of then transitioning to
next state
matches below and above
the dividing bar
that means that the ratio of this this
sub quantity is just one anyway and we
can just cancel them out against each
other
that means that we're left with this
with just the probabilities of selecting
the actions so the probability of
selecting the action or under that
policy and then the next action another
policy and so on and so on for both
these policies but these are these are
just by definition the policies
themselves so what we see here is that
we get these ratios of policies on every
time step
and then we just divide them with each
other so we divide the target policy by
the behavior policy we multiply all of
those together and then we can multiply
those with the return and this will give
us an unbiased estimate
of the value
on under pi
it can be a noisy estimate and again we
do need this condition that the behavior
policy will have some probability of
selecting all of the actions that the
target policy does because we never want
to divide by zero
but if that if we have that then we can
use this as an estimate so for instance
if we want to estimate the greedy policy
and we have a uniformly random policy
that we're acting with
we could re-weigh the samples that we
get and then we could use those as an
unbiased estimate for our
target policy our greedy policy in fact
if we consider a greedy target policy we
can see that if any of the actions
selected along the way
were not greedy then this total sum
would just be zero basically we're just
going to ignore
returns that don't correspond to the
greedy policy in that case
and then additionally any any one where
all of the target policies are won so in
in the case where we are considering
estimating the greedy policy the value
of degree policy and we have actually
found a trajectory where we always
select that that action corresponding to
the greedy policy then we would upscale
this
return so that on average compared to
all those zeros that we also accumulate
on average we get an unbiased estimate
so this is a generic technique for off
policy learning we can generate data
under some policy and then evaluate any
other policy
now this might be a bit unreal b as i
mentioned for the greedy policy you
would actually have to select all of the
actions exactly right in order to learn
a valid degree policy and that might not
happen that often a different way to say
that is that this is a very high
variance estimate
so what do we do when we have high
variance we turn to temporal difference
learning and we can use this to
construct temporal difference targets
that will also do the same thing so what
we're going to do is we're going to use
our temporal difference target
where v is now going to be an estimate
of the value of policy pi so the one
that we're interested in
and we're going to rewrite this target
with important sampling but then we only
need one step we only need one ratio of
the probability of the target policy
divided by the probability of the
behavior policy
this of course has much lower variance
on monte carlo important sampling this
is even more so the case when we're
doing important sampling than it is for
the standard difference between on
policy temporal difference learning
versus monte carlo so this is even maybe
more in favor of central difference
learning
than if we do the on policy
case
because we can learn now even if the
policies only match a single step so
whenever you happen to select a greedy
action
this td algorithm can learn about that
can use this to update its value
functions
whereas the monte carlo algorithm needed
a full trajectory to be greedy
for that special case when we're
interested in a greedy target policy
now let's step through that this is
accurate so we've already done this for
the one step reward case and we've done
it for uh
the more generic case with a generic f
let's just verify that everything's
still fine so what we're interested in
we have this estimate where we've
re-weighted it and we've
sampled according to this behavior
policy mu
and now we're just going to check that
everything works out so we're first
going to write out this expectation
explicitly
we'll keep the expectation on the inside
which is the expectation given now that
we have a state and an action and we're
just expanding that the first action so
the expectation we're just expanding in
terms of the action selection which is
according to this behavior policy mu
and we're in some state s so we're just
going to replace a
and s now with this a that we're summing
over and that s that we were
conditioning on this means that v of st
is just v s and additionally these
probabilities are now just for a and s
rather than a g and st
and then we have the summation on the
outside which is just the probabilities
of selecting each of these actions
and then we notice that we can push the
summation inside this bracket so inside
the tv error and this gives us the
following where the mu now cancels with
the other mu
and therefore we get a summation over pi
of this first bit and we've also pushed
this sum into the minus part where it
remains for now
but then we notice this is just a
summation over actions but the stage
value doesn't depend on actions hence
this summation is just something that
sums to one then we can replace with
anything else that sums to one
so we would just replace pi here at the
end with mu because this is if this is a
valid policy both these sums just sum to
one and they're multiplied with the
state value so these are exactly equal
now we know that we have a summation
over
the probability under the target policy
pi in both these cases so we can merge
them together again pull the pi outside
of the temporal difference error
and then we notice that we're actually
all the way back into a form that we can
write as an expectation again
so we do that and we notice that we now
have an expectation under policy pi of
the unweighted temporal difference error
so feel free to step through this
yourself carefully to convince yourself
that this is true so just doing this one
correction can give us an unbiased
estimate for the temporal difference
error under our target policy hence
using this for learning means that v
would start to approximate v pi
now that is useful but we can actually
do something interesting which is a
little bit different we can also
consider off policy learning for action
value functions and interestingly no
important sampling corrections will be
necessary
this is because we can directly we
already conditioning on the first action
so we don't need anything to do for that
but on the next step we can just choose
not under the behavior policy but we can
choose under a target policy that we're
considering so the actual action we pick
might be on our behavior policy but we
can use bootstrap under the
probabilities under the target policy
and that will look like this
so we have our state estimate state of
action value estimate q of st80 and
we're going to update that by
considering a temporal difference error
which takes the first reward
and then discounts
the next state value under our target
policy pi
and then turns out it doesn't matter how
we select a t it doesn't matter how we
select a t plus one which doesn't even
show up in this equation
and we can learn about this target
policy
for whatever behavioral policy we have
obviously the same caveat as always
applies if you don't ever actually
select certain actions under your
behavior policy you won't actually learn
their values
which means that if you don't use them
in bootstrapping these values might be
inaccurate but if your behavior policy
is for instance some exploratory policy
which does eventually select all of the
actions
then we could still learn about a
different target policy so for instance
again the target policy could be greedy
and the behavior policy could be
uniformly at random and this would still
be a valid algorithm
this algorithm is called expected sarsa
in such an embargo
when expected sarso was first introduced
in the paper a couple of years ago
well it was actually first introduced in
the first edition of southern nevada but
it wasn't named
and then in the paper it was called
expected sarsa that refers to the own
policy algorithm and a more generic
version where pi could be any policy is
sometimes also called general q learning
because this turns out cutening is a
special case of this algorithm
but the point here of the expectation is
that we don't necessarily have to pick
the behavior policy but we can pick any
target policy
however a special case is when you do
pick pi to be equal to your behavior
policy mu you could still do expected
structure for on policy learning and
then turns out this update is exactly
the same as the sarsa update except it
has lower variance so it is unbiased
compared to sarsa but it has lower
variance and therefore in some sense is
strictly preferable to sarsa
queue learning happens to be a special
case and we can just consider that by
considering pi to be the greedy policy
with respect to our action value
functions
and if that is the case then this
summation over policies will just
basically pick out the max and then we
can see this this full update becomes
exactly q learning so we can see that
this algorithm is actually a slightly
more general version of q learning
okay this brings us to the summary
of this lecture
now we're just going to briefly recap
basically the whole lecture and we're
going to start with a model-free policy
iteration
and recall that the point there was to
learn action values to predict the
current policy because that's how policy
evaluation works and policy evaluation
is a sub-component of policy iteration
and then we can do the other part which
is policy improvement for instance make
the policy footy greedy or epsilon
grading
and we talked about how we could extend
this idea of policy iteration to be
model free where both the evaluation
step and the improvement step can be
done model-free
then we talked about q learning and
noted that this is akin to value
iteration
where instead of having a separate phase
where we maybe evaluate a policy over
some time and then make it make it
greedy or improve it we could also do
this in a very tight loop where we
immediately make the policy greedy on
every step so when we update our action
values maybe the policy can change
immediately
and then q-learning is a model-free
algorithm that is very akin to this
value iteration idea
we've also talked about sarsa and just
now about expected sarsa and these can
be used more similarly to policy
iteration where we could also keep the
policy that we're evaluating fixed for a
while and on policy estimate that that
behavior and then do an improvement step
or we could also immediately update the
the policy as we go so sarsa could also
be used online with an epsilon greedy
policy where every time step you're just
re-evaluating what are the actions that
should be taken and we talked about the
convergence properties of these
algorithms as well
but sometimes we're also just interested
in maybe estimating a completely
different policy this in general is
called off policy learning
where expected sarsa could for instance
be used to pick any target policy that
we're interested in and then trying to
learn about the values of that learning
about the greedy policies then a special
case of policy learning which is the one
that is used by q learning
in general we want the behavior and the
target policy and policies to improve
that's what it means when we say that we
want to do control
and if the target policy is greedy
the behavior policy could still be
epsilon greedy or some other type of
behavior policy and then we could
construct this target for q-learning as
we saw earlier where we just take the
first reward as always this is a
temporal difference method and then we
bootstrap on the state value which is
defined by taking the greedy policy in
that state
written out here quite verbosely and
this is equivalent to just considering
taking the maximum action value in that
state
so conceptually what q loading is doing
is it's imagining that in the next state
it would pick greedy according to its
current estimates and then trying to use
that as an estimate for the value and
backing this up
in sarsa similarly but slightly
different leader target and behavioral
policies are the same and we're actually
considering the actual action that was
taken on the next time step before we
update our state action values at time
step t
then if we want this to converge because
it's an on policy algorithm we need the
additional requirement that the policy
becomes greedy so q learning is an off
policy control algorithm that doesn't
need this requirement but sarsa's own
policy so for this to actually converge
the policy itself the behavior itself
should eventually become greedy
one way to do that is to use epsilon
greedy or a different type of softmax
which is a generic term referring to
algorithms that are not quite greedy but
they're close to greedy so we're taking
a soft maximization of
the action values and then using that to
infer a policy
and then if you decrease the expiration
you get this property which we called
glee which is greedy in the limits but
it does explore infinitely often if you
decay slowly enough
so in summary q learning uses a greedy
target policy to learn about sarso uses
a stochastic sample from the behavior as
a target policy which means its own
policy because the behavior is used as a
target
and in expected structure we can relax
this assumption and we can use any
target policy
so both sarsa and keurig can be seen in
special cases of expected sarsa where
for sarsa the actual sample is
stochastic so the actual policy will be
stochastic and focusing is greedy
um but you can pick anything you want
and then we also talked about double
learning
where we use a separate value function
to evaluate the policy
for any policy
now double learning is actually not
necessary if there's no correlation
between the target policy and the value
function let's say we're doing pure
prediction you could use expected sarsa
for instance to learn about a target
policy which we're keeping fixed for
instance you could have a policy that
says
what would happen if i always walk
forward an unexpected search i could use
that to construct a bootstrap target
which would in the next state for any
transition in the next stage would
consider only the action that moves
forward
this action wouldn't depend on the
action values and therefore therefore
there will be no correlation between
selecting the action for the policy and
then evaluating it
then you will not need double learning
there won't be any bias but of course in
general there can be correlations and
specifically when using a greedy policy
like in q-learning there are quite
strong correlations the whole reason to
pick an action is because it has a high
value
and then we evaluate that using the same
values and this creates an upward bias
then double learning can be quite useful
more generally if there's correlations
double learning can be quite useful and
it can be quite useful to track these
action value estimates
just to see whether you might be
suffering from overestimations or
unrealistic action value estimates
and then if you are you could consider
doing something like double learning to
mitigate that bias
that brings us to the end of this
lecture please continue to use moodle to
ask questions thanks everybody who has
been asking questions there and we will
try to always
get back to and answering these on a
regular basis
and for now then i want to thank you for
your attention and i'll see you in the
next lecture