DeepMind x UCL RL Lecture Series - Multi-step & Off Policy [11/13]

my name is hardivan husselt and today i
will be talking to you about off policy
and multi-step learning and if these
intros are starting to sound a little
bit repetitive um
that's okay as long as the contents of
some of the material is hopefully more
engaging to you
in terms of background material um i
would advise that you have a look at
susan ambartos chapter 5 7 and 11.
um if you forgot anything about those
because you likely looked at these
before already
but there will also be some material in
this lecture that is not contained in
any of those chapters so
just be mindful of that
but we're going to start before we dive
into any new material in recapping and
also motivating what we're going to do
today just at a very high level we've
seen this over and over again now the
point is to take actions in the world
where we observe
certain observations which might include
some partial information about the
environment state but typically not the
food environment state and we want to
figure out how to take actions so to
improve our rewards
where the agent internally might have a
policy as we discussed last lecture in
our policy gradients and ectocritic
discussions it might include the value
function which is predictions about the
future also potentially a model or
multiple models
and then the general problem involves
taking into account time and
consequences because the decisions don't
immediate just impact the immediate
reward but also the agent's internal
state and potentially the state of the
environment which means certain actions
might have long-term consequences which
might also not be immediately observable
now in previous lectures we talked about
how to do model 3 prediction and control
in this setting we talked about using
multi-step updates rather than one-step
updates and eligibility traces because
sometimes this provides a better
trade-off in the learning efficiency
we also talked about more theoretically
about understanding dynamic programming
operators because a lot of reinforcement
learning algorithms can be understood as
approximating dynamic programming
operators so it's really helpful to
understand how these work
in principle because then the algorithms
that we actually deploy might be
considered approximations of this and
therefore might be better understood if
we understand the fundamentals
in addition we talked about what happens
if you then include function
approximation
and how to do this and typically we use
gradient based algorithms when we do
that
we talked about model based algorithms a
little bit and algorithms like dinah
and more recently we talked about policy
gradients and ectocritic algorithms
which as i said before
keep an explicit representation of the
policy and try to update that directly
now in this lecture we're going to talk
about off policy learning again because
we touched upon this before but we're
going to focus a little bit more on this
specific topic because it's important
and then especially when combined with
multi-step updates and with function
approximation this will be both in the
setting of model 3 prediction and
control but also in the setting of
policy gradients
we won't be talking about models so much
in this lecture
now in terms of motivation why do we
care about this topic why is this
important so there's a number of reasons
actually
so first off what is of policy learning
well of policy learning is learning
about a different policy than that is
used to generate the data
and there's a number of reasons why you
might want to do that but in general it
means you're basically posing what if
questions hypothetical counterfactual
what-if questions what if i would have
do done this rather than this other
thing that i actually did
now use cases in which this is useful
include you might want to learn about
the greedy policy but you might not want
to follow degree policy all the time
because that doesn't explore
sufficiently q-learning is an example of
an algorithm that does this and it's
often named as a canonical example of an
off-policy learning algorithm and this
is true but this is not the only uh of
policy learning you could do
in addition or instead you could also
learn about many other policies this
could be useful because you for instance
care about these policies but also
because you might just want to learn a
lot about the world and condense all of
that information into the agent somehow
to reuse this later
the greedy policy is typically a
tempting thing to approximate because
then you can do something like policy
iteration but it doesn't mean it's the
most effective thing to only learn about
the greedy policy all the time
in addition to that you might have
certain other constraints for instance
you might want to learn from observed
data for instance logs that you've
stored or from information gathered from
other agents
and learn maybe about the greedy policy
there or about a different policy and
this could just be because you're
interested in a certain application oh i
have this data store for instance
collected by watching humans interact
with the system and seeing which actions
the humans took and you might actually
want to then ask what if questions what
what if they have done this other thing
what would have been the value or what
have been what would have then been my
predictions
and of course this also includes
learning from your own past policies
because that's also just a data store so
you could imagine doing something called
replay
where
you collect a lot of data but you keep
on changing your policy right because
you want to keep on getting better but
that means that the old data that you've
collected doesn't actually correspond to
your current policy anymore and then you
might want to do off-policy learning
from that because the behavior that
generated the old data doesn't match the
current behavior that you're interested
in
so that might be a different reason to
want to do off policy learning because
you want to do extensive replay
for instance in order to be very data
efficient because you can then reuse all
the data that you've generated so far
now there's a different reason why
policy learning is
important rather than just trying to
learn about many things or trying to
learn
about degree policy or something like
that in a value-based method
it's also very important in policy
gradients to correct for mismatching
data distributions and i'll go into that
a little bit more in a second
so first let's just
remind ourselves what we could do so if
we want to do
off-policy learning about any
different policy pi so we're following
some policy mu which is not given on the
slide here but it might be different
from
the policy pi that we're actually
interested in
now one thing you could do is you could
use this general q learning or sometimes
called expected sarsa update this is
what rich
and andy called it in the book
which is basically straightforward
application of bootstrapping where we
take a certain action a or we consider
taking a certain action aat in a state
st
and then we just see what happens if you
take that action but in the next state
we consider following this policy pi and
pi does not have to match the behavior
it could match the behavior in that case
you're just doing on policy expected
sarsa but pi could for instance also be
greedy in which case you're doing q
learning but it could also just be
something else so in expected tarsa
in the original expected saksa it was
meant to be the current behavior policy
in the book expected tarsa is
used to refer to this more general
algorithm where it can be anything
in sarsa this is also interesting to
point out this still corresponds to this
update but then pi is a stochastic
policy which assigns all probability
mass on the action that you actually
pick in the next stage that is an open
policy algorithm
but in general you could just plug in
anything there
and learn about whatever policies of
interest to you
okay so now moving on this is all
recapping things that you've already
learned about before
we might want to do multi-step updates
and then we can't use this trick of
bootstrapping that we had on the
previous slide because now we have to
consider what happens on a longer
trajectory so just bootstrapping is not
enough anymore
and then we introduced important
sampling corrections
so basically if we have a trajectory
tau
which spans from st to as big t maybe
that's the end of the episode
um then we can correct
for the distribution um that use was
used to generate this by basically
multiplying the observed return g let me
point at it with the mouse
by multiplying this with the probability
of generating that trajectory that we
saw under the target policy that we're
interested in
and dividing that by the probability of
that same trajectory under the behavior
policy that we're interested in
if we do this then the expectation of
this return
will be equal to the expectation of the
of the return
that we're interested in under the
target policy so note here that gt is
the uncorrected return
and gt with a hat is the correct return
which is defined over here
and what this is saying is that if you
make this correction and then you take
the expectation under the behavior
policy mu
then with this correction you're
guaranteed the expectation of this whole
entity here which we call the g-hat will
be equal to the uncorrected return under
the target policy
so this provides us a method that allows
us to take any return under any
generated under any behavior and then
correct for the distribution of
generating that trajectory that led to
the return to something that has the
right expected value
so let me make that a little bit more
concrete what could happen is you had a
trajectory that was actually quite
unlikely under your behavior policy but
it just happened to happen but it's one
that would happen all the time under
your target policy
now what would happen here is that
effectively then you'd have a number
that is relatively high
divided by a number that is relatively
low so we would upweight that return and
that makes sense because this return
would be more common
this trajectory under the target policy
conversely we could also of course in
this might be the more
common case we could generate a
trajectory under the behavior that is
quite likely to happen but would be very
unlikely under the target policy maybe
it's even impossible maybe the
probability of that trajectory which
just happened under your behavior policy
would literally be impossible under the
target policy and the probability could
be zero in that case we just zero out
that return or we down weighs if the
probability is not quite zero but very
low
and this means that we're basically
re-weighting the samples in exactly such
a way that in in the end the expectation
is the same to the thing that we
actually want
now additionally as shown on the slide
the probability of a trajectory under a
policy divided by the probability of
that same trajectory under a different
policy can actually be written down as
just the multiplication of the
individual important sampling ratios for
each step
the transition probabilities which are
also contained in this probability of
the trajectory are the same
above and below the line so they cancel
out
and what we're left with is just the
probabilities of selecting each of the
actions in the trajectory
i notice there's a little typo on the
slide just to be clear the trajectory
here was ending in s big t
but here we're actually considering a t
as well as a big t so basically this
should have been t minus one because
this last action wasn't part of the
trajectory that's a minor typo
but the general principle here is
hopefully clear
this was discussed before in a previous
lecture when we talked about policy
corrections and we'll go back to it a
little bit more later in the lecture
but um so why do we consider these
multi-step updates we had a simple
algorithm let me go back to this slide
first where we would just bootstrap
immediately and make a correction and we
didn't need these awkward important
sampling ratios
so why do we consider them well because
this allows us to do multi-step updates
and we've seen before that this can be
beneficial to trade of buys and variants
and intuitively this is depicted in this
figure from
from the book by susan nobarto
where if you consider that a certain
path has been taken if you would just do
one step updating like in this case with
one step sarsa you would only ever
update the previous state before
reaching the goal and then you'd have to
wait all the way until the next episode
before we can propagate the information
from this newly updated action maybe
backwards to whatever action then led to
this state or maybe a different action
leads to the goal and then you update
that one so credit will propagate
correctly if you do one-step updates but
it might be quite slow
and then we argue that sometimes it's
better to do a multi-step update maybe
not full monte carlo maybe not all the
way until where we started but maybe
more than one step just to propagate the
information a little bit faster
and one way to interpret this is that
this is a bias variance trade-off where
doing multi-step returns increases the
variance but it decreases the bias of
your update because we're not relying on
estimated state values as much
and in addition to that you can just see
that the information will propagate
faster in an episode when you do this
typically the trade-off is that a
multi-step return tends to be more
effective but full monte carlo or full
td tend to be worse than doing a couple
of steps
so neither full monte carlo or one step
bootstrapping is typically the best
trade-off
now as i mentioned this is maybe also
quite important for policy gradients and
this turns out to be really the case in
practice
so recall for policy grady methods we
want to sample something similar to the
equation on the slide for details i
refer you back to the lecture on policy
gradients
but we basically get an expectation
where the expectation is over the
dynamics of the markov decision process
that we're in and over the policy that
we're currently following which is pi
and then we want to estimate this term
here which is the expectation of the
actual value of the policy q pi this is
not an estimate this is the actual value
times the gradient of the logarithm of
the policy at that state
and action
and the reason why was explained in the
policy gradient lecture so if you want
to remind yourself why this is the
equation that we're after
you could look at that
but this would give you
if we could sample this this would give
us an unbiased estimate a stochastic
estimate to the true policy gradient and
would allow us to make our policy better
by following that gradient
now
on policy we could sample a return a
multi-step return maybe even a monte
carlo return such that the expectation
of that return would be equal or
approximately equal to q pi
now if gt here is a monte carlo return
and we're on policy then this would just
be inequality even
if
where
if full monte carlo would be too high
variance and this is the common case we
might actually want a bootstrap which is
why i put an approximate equal here
because gt might be a multi-step return
instead of being a full monte carlo
return and in that case the expectation
is not quite exactly equal to q pi
but it's typically close and that's good
enough
but what if we're off policy
then this isn't necessarily the case
anymore if we're following mu then we
have to correct g somehow to get a good
estimate for q pi
so if the behavior is different we need
to do something to correct because
otherwise the gradient won't be pointing
in the right direction
and that's important for policy graduate
algorithms because if we get a wrong
estimate for the return here
um it might just mean that we're no
longer following any gradients or even
we're even following the gradient of
maybe the wrong objective and that means
that your policy for instance could get
worse rather than better that's not just
purely a hypothetical theoretical thing
it turns out to be the case that policy
gradients are quite sensitive to off
policy updates and you have to really be
careful that you correct for them
because if you're too off policy a
little bit of policy might be okay but
if you're true of policy with policy
grading updates
the gradient is quite likely to point in
a direction that's not going to make
your policy better and sometimes even
makes it worse
so that is another motivation for off
policy learning specific to policy
gradients where it's it may be even more
important to correct completely for the
of policiness
or more completely from your policies
than in value-based methods
okay and now we're going to uh basically
talk about a couple of issues in of
policy learning so that seems to present
so far a solution
but then
this isn't the full solution there are
still some problems that need to be
addressed and we're going to talk about
that for the remainder basically of this
lecture
so these issues can basically be
grouped into two main categories
so the first one is high variance and
this is especially true when using
multi-step updates and i'll show this
with a numerical example in a moment why
this is especially
problematic when we do off policy
corrections
or it can be and we'll talk about some
ways to mitigate this
and then a separate issue is that
sometimes you can get divergent and
inefficient learning because of a subtle
property that if you're going to go off
policy and you're going to correct for
that even if you're correct for that in
the targets there's still
maybe a mismatch in your state
visitation distribution
it's okay if you didn't understand that
sentence i will get back to that later
in this lecture and make clear what i
mean with that
so to start off
the first issue that we're going to talk
about is variance
and
this is specific to basically these
important sampling corrections in a
sense but it's also somewhat inherent to
off policy learning and one way to think
about this is that
we can consider a one-step reward
and then we can look at the
expectation of the important samples
reward so for simplicity here we're
first looking at the one-step case but
some of the conclusions here will
transfer to the multi-step case and this
is why this is an easier place to start
right
so we're first going to just verify that
this is true so we're going to sample an
action
under a behavior policy mu right
and then we're going to correct for that
with the in this case we just need to
look at one step because we're just
looking at the immediate reward so we're
looking at the probability of selecting
that one action
and
we're going to divide the probability
under the target policy that we're
interested in
whatever that is might be greedy might
be something else altogether
by the probability of actually selecting
it under behavior mu
and then we just write this out we say
oh this expectation can actually be
written because we're just sampling the
action mu we can just sum overall
actions or if it's a continuous action
space you could even have an integral
here
and we basically look at the probability
of each of these actions in the state of
interest that we're in state s in this
case
and then we look at the resulting
quantity here which is the random thing
over here is written here explicitly
where r s a is the expected reward when
taking action a in state s
and then we see that these muse they
cancel out by design so the probability
of selecting the action cancels out with
the mu that we used in this division
which means we're left with pi which
means we can actually rewrite this as an
expectation but now with the actions
selected according to the target policy
pi
so this is just to verify that indeed
the expectation is the same as the
import sampling corrections
but the variance is not necessarily the
same and we're going to look at that in
an example and sometimes the difference
can be quite large
so let's go to the blackboard for a sec
and
we'll show an example there
okay so
you can hopefully see
my writing
and i'm going to start by basically
um specking out what the situation is
that we're in we're going to consider a
one-step reward there's going to be two
actions so let me write them down we'll
have
um
one action to go
right and we'll have one action to go
left
and
we will also have a reward associated
with each of these actions we don't know
these rewards yet we're going to learn
right
but we
do have these rewards for the purposes
of our analysis so we know
just for the purpose of the example that
the reward of going uh to the right is
actually always just going to be plus 10
and the reward of going to the left is
better that's plus 20.
and then we have to pick a probability
so let's say the probability of
these actions
under the target policy that we're
interested in is to go right with
probability 0.9 so we're more likely to
go right and maybe we go left with
probability 0.1
now
to start us off right let's consider the
om policy case where um the behavior
policy
is the same as the target policy
this is what it means to be on policy
that those are the same
and then we can just reason through what
the
second
second moment is
for the
update so the update that we're gonna do
will depend somehow and we're just going
to consider the reward itself so we're
not going to actually consider the full
update we're going to have some
action 80
and we're going to correct it for the of
policiness
now because we're on policy this term
will actually just be one for now but in
a moment i'll go for the off policy case
as well and this is the term that we're
going to consider
we already showed that the expectations
this turn for any combination of pi and
mu will be the same
as the expectation that we wanted under
so let me just write that down so
the expectation of this under mu
we just show that this is equal to the
expectation of rt plus one
under pi and this actually holds under
any mu and pi
so in the example above in the table i
put the on policy case right and i'll do
a numeric example in a second but we
already knew this was the case in
general but now actually let me go
and not consider then the
first moment or expectation
but instead we're going to consider the
second moment
the second moment is enough to
understand the variance in this case
because the second moment
is equal to the variance
um
plus the squared expectation but because
the expectation is the same we can just
look at the second moment instead
now in this case because we're on policy
this will just be equal to um
let me just write it underneath
as i said that ratio is just 1 pi
divided by mu so this is just equal to
rt plus 1 squared
right
and because mu is in this case in this
example equal to pi
we can immediately just replace mu with
pi
and we get the
following uh expectation the reason i'm
doing this is because i'm going to
compute the second moment numerically
and then we're going to compare to when
we go off policy
so what will happen here well this
expectation is with probability 0.9
we're going to select the action to go
right
if we get that action we're going to get
a
reward of 10.
so the second moment will be
0.9 times 10 squared
plus with probability 0.1
we're going to select the action to go
left
and we're going to get a reward of 20.
so the second moment will be this in
total with 0.9 we get 10 squared with
0.1 we get 20 squared so we just write
this out
this will be 90.
this will be 40.
so we reach
a conclusion of our second moment in
this case turns out to be 130.
so just the the variance now let's make
that final step right the variance of um
this quantity
i'm not going to write it out again
um is going to be
as i mentioned in words just now but let
me just write it out
um let me actually
put a
random variable in there to make it a
little bit clearer
so let's call it x
this is a generic truth
expectation
minus the expectation squared
so we can see the
second moment we just calculated here
and we can see the
mean there
in this case the expected value
let's quickly do a back of the envelope
calculation of that
the expected value of this reward
is going to be
0.9
times 10
plus 0.1
times 20
which is just 9 plus 2 so that's 11.
so what would we have here so we know
that the variance is
130
minus
11 squared
11 squared is equal to 121 please check
my math
which means that the variance is
130 minus 1 to 1 which means it's 9.
or if you prefer the standard deviation
is going to be the square root
of the variance
is going to be 3.
okay so we did the policy case here
right so we basically are just going
through a numerical example
and now what i'm going to do next is i'm
going to change it to an off policy case
and we're going to compare what happens
to the variance in that case
okay
so maybe let me do that by introducing a
new
maybe i'll use green
and i'm just going to
change some things so we're going to
introduce
a change to the behavior policy
instead of having 0.9 and 0.8 let's
sample uniformly
so both actions now have a probability
of being selected of 0.5
okay
now
some things change because now
this is no longer an equality
so let me go back and erase this
we're going to write it out carefully
i'm going to show you that it's
something different
so this is all for the old case right
we're going to recompute that for the
new case
so what is this thing so this is let's
write it out
a summation over actions
probability selecting that action
times that thing that is in the
expectation
and notice that there's a square there
so what we get is pi
a
squared divided by mu
a
squared
r
s a
squared
now notice that something different
happens here from the uh expectation
case
the mu that we're multiplying because of
the probability that we're actually
taking that action
cancels only one of the
muse in the ratio so what we get here is
we can rewrite this as
let me pull out one of the target policy
probabilities
we cancel out one mu
so we can cancel this one with one of
the muse in the
ratio but there's one left so let me
pull out one probability there remains
one because we had a squared and there
remains one mu as well
so what we get is this new quantity
let me make a little bit of room let me
get rid of the
generic fact over here
and then i can continue and i can say
okay this is apparently
going to be equal to
the probability of selecting the action
on the retiree policy times this ratio
times the reward squared
so let's pick again the let's just write
it out for the explicit
calculations so what we have now is
again we pick the action under the
target policy we pick the action to go
to the right with probability 0.9
but instead of just multiplying this
with 10 squared there's a different
thing here which is 0.9 divided by 0.1
and that is times 10 squared
so this ratio here
is new to the on policy case that wasn't
there before
similarly if we pick the other action
to go to the left we will have
oh sorry i made a mistake here
let me go back
let me correct that mistake the behavior
policy here wasn't 0.1 it was 0.5 that
was just a little fluke
and the ratio here is we pick under the
target policy under pi
those are all target policies
we pick the other action and then the
ratio will be
this thing
20 squared
okay so now let's just write that out
what is that what are those numbers well
um
it's a little bit
less obvious perhaps so feel free to
check my math but maybe maybe let's just
write it out um we basically get 0.9
times
1.8 that's that ratio times 100
plus
0.1 times
0.2
times 400
and this is going to be equal to let me
put that on the next line
get rid of the
previous
if i did my math correctly this is 162.
it's
we can do this one first it's 90 and
then
plus 0.8 times 90
so that's 90 plus 72 so that seems
correct
plus this is 0.02 times 400
so that's
8.
so what we get is that in total the
second moment now is going to be 106
okay
so what is the variance then
the variance in this case is going to be
170 and we're still going to subtract
the expectation squared the expectation
did not change that's still 11.
this is still the expectation in the off
policy case because we've corrected
right the expectation by definition
because we're using this important
sampling correction did not change
so we're going to
compute the variance we're taking the
second moment and then subtracting the
mean squared
which will be 1
70 minus 1 2 1
will be
49.
so what is now the standard deviation in
the off policy case we find that the
standard deviation
is square root of 49 is 7.
substantially higher the squa the
standard deviation is more than double
from what it was in the on policy case
this is just a numeric example
but in general one can prove and
feel free to play with this a little bit
try more numerical examples or maybe
even try to prove in general that the
variance will increase if you do of
policy corrections
this makes intuitive sense
typically because under some like under
some conditions it will it will increase
but the intuition behind this is that
we're going to zero out some of the
returns that happen and we're going to
upgrade other returns that happen
especially
returns that are rare under the target
policy but they are very common under
the
behavior policy they will be down
weighted
but
the other one is what increases the
variance which are is the things that
are very
common under the target policy but rare
under behavior policy
so the example here is actually the
behavioral policy none of them are
particularly rare but we have this ratio
here
let me highlight that with a different
color so it's easy to see where i'm
writing
so this ratio here is the ratio that is
increasing the variance in some sense
it's where the target policy is fairly
high 0.9 but the behavior policy is
lower than that it's 0.5
okay
now let me go back to the
slides
now it actually turns out that in some
cases the variance of important weighted
returns can even be infinite there's an
example in the book i believe it's
example 5.5
where you have a very simple markov
decision process
and
the target pulsing behavior are also
quite simple there's a left action and a
right action
and the
behavior policy is similar to the
example i just gave is uniform in this
case the behavioral policy is depicted
with a b rather than mu but it's the
same thing
and the target policy always moves left
then turns out if you
take monte carlo estimates the variance
of this can be infinitely large and this
can exhibit itself
in the as shown in the graph where you
can get these spikes
and these spikes will continue to happen
and the reason why is that the variance
is literally unbounded in this example
which means that there's always a
probability of getting something that is
so high that even your average can get
polluted by it
infinite variance is typically a bad
thing now of course there might be ways
to limit the variance
by making it non-infinite right you
could for instance just have a hard cap
on something you could like
put some heuristic in there to make sure
that the variance is at least finite but
that doesn't really solve the underlying
problem which is the which is that the
variance is just very high
but it's interesting to know that the
variance can be actually so high that it
can be infinitely large when just doing
naive important sampling corrections
so we might want to mitigate this
variance right we want to reduce the
variance and we're going to talk about
multiple different ways in which we can
reduce the variance
so we will discuss specifically three
different strategies one is per decision
importance weighting
which is an
older idea it's some 20 years old now
but it's quite
useful in reinforcement learning it gets
rid of some unnecessary variants
completely completely unnecessary
variance in our updates
we also talk about control variants
um i will explain what i mean with that
and we will also talk about
bootstrapping again and specifically
adaptive bootstrapping which will allow
us to kind of pick
in some sense automatically by looking
at
the important sampling ratios when to
bootstrap when it's important to
bootstrap to minimize the variance
and we'll start with as i mentioned per
decision important something
okay so what does this term mean per
decision importance waiting
we're going to consider some states s
some arbitrary states and we're going to
look at some random variable
x
so think of x here as for instance a
reward
now
we're going to basically observe that
for any x that does not correlate with
the action
we don't actually need to important
sample
so
this is written here on the slide look
at the left hand side first we have the
expectation of this random variable x
for instance your reward under the
target policy pi this is the thing that
we want right we want this expectation
we noted before that this is actually
equal to the expectation of x
under the policy mu as long as we
multiply with this important sampling
ratio right
but now what we're claiming is that if x
is uncorrelated with the action that
this thing is actually equal to the
expectation of the uncorrected random
variable x
intuitively this makes sense if x does
not depend on the policy we don't
actually need to correct because the
expectation will be about everything
else that is random it will be about
your uh
mdp for instance right
but if it doesn't depend on the action
then the policy shouldn't matter we
shouldn't need to correct
now why is this a good thing well this
means that we might be able to
have lower variance than otherwise by
basically not adding this additional
variance that we do with these
corrections and just ignoring um the
random variables that don't correlate
with the actions under consideration
so let's first prove this fact and
we're basically just going to very
simple derivation so you can also kind
of check the assumptions that are being
made here right we start off with this
important sample
corrected random variable x under the
under sorry under the behavior policy mu
now because um
this ratio is a random variable
but it's only a random variable because
a is a random variable right
so if a does not correlate with x
this means that we can write this
expectation as
a multiplication of different
expectations
you can only do this when these random
variables x and this ratio do not
correlate with each other but that's the
assumption that we're making that are
uncorrelated now if that is the case we
can just unpack this last thing and this
is an important property to be aware of
in general the expectation of the
important sample sampling ratio itself
under the behavior policy is always
equal to one
we're going to prove that here by just
writing it out so again we're going to
sum over all of the actions look at the
probability of selecting the action
multiplying that with the ratio as usual
the behavior policies cancel out
apologies for the x
s k here that's just small s this is a
typo on the slide so these cancel out
with each other
what is left is just a summation over
actions for the target policy but
because the target policy is a proper
policy this will equal to one so we can
just
cancel it out it disappears
so this shows that the expectation of
the corrected x is equal to the
expectation of the uncorrected x if x
does not correlate with the action
this is a general statement as i said
before so the expectation of the ratio
itself
should always be equal to one
only of the ratio itself right it's not
if we multiply with something
then
this
ratio does not disappear as we saw
before you can actually change your
expectation by
adding this ratio but the ratio by
itself is always equal to 1. this can
also be useful if you use these things
for instance in experiments this could
be a useful check that on average your
important sampling ratio itself should
always be on an average one
now we're going to apply this to
reinforcement learning
but in order to do that we're going to
introduce a little bit of common
notation from the field um to be able to
condense the notation quite a bit and
hopefully make things quite clear
so the most important thing here to keep
in mind is that rho
is the new notation that we're going to
use for this import sampling ratio
because this term shows up all the time
right and it's quite a large term to be
to keep on writing down
so the important sampling ratio for the
different policies at some time step t
we will condense that notation into row
t so the t here refers to the states and
to the actions and implicitly raw
depends on the target policy and
behavior policy
now in addition to that we can introduce
this other notation where we can
consider rho from time step t to time
step t plus n
and this will simply be the
multiplication of all of the rows for
all of those time steps which by
definition of rho is just a
multiplication of the important sampling
ratios
so we will call rho the important
sampling ratio
rho could be thought of as a ratio right
so this might be a useful uh
point to remember
and if we do that then we can condense
notation quite a bit
so the thing that we're interested in is
the return
which i've written down here explicitly
for uh an
episode terminating with the final
reward at time step big t
so the return is just the summation of
the discounted rewards as usual starting
at time step t because we're considering
the return at time sub t
in order to correct this return if this
return was generated according to some
behavior policy mu but we're actually
interested in target policy pi
then we can multiply with this important
tempering ratio for the full duration of
the trajectory so in this case the
return spans a trajectory from time step
little t to times the big t and that
means that we're going to correct with
all of the actions in that trajectory
multiplied together all of the ratios
multiplied together which with the new
notation we could just write down as the
ratios from t to t minus one big t minus
one
so this gives us a very condensed nice
notation where this is the total ratio
which
captures the probability of the
trajectory under the target policy
divided by the probability of the
trajectory under the behavior policy
and we multiply the return in order to
make it an unbiased estimate for the
return under the target policy
however we can also note that this
return is actually a summation right a
summation of individual rewards
so in instead of
perceiving this ratio as correcting the
full return
we can also push it into this summation
and apply to each of the individual
rewards within that return
right the return is just a sum so we can
push this ratio through the summation
and multiply it with each of these
individual rewards but this raises a
really interesting question
so
is this the right or the best important
sampling ratio to be applied to each of
these individual rewards
and turns out maybe we can do a little
bit better
so what we're going to do here is we're
going to get rid of some terms that we
don't need so we're just going to remind
you rho t to big t minus 1 is the
important temperament ratio for the full
trajectory that spans
this return spans
and we could push it inside the
summation that this return actually is
by multiplying each of the individual
rewards
but now we're going to recall this fact
that i mentioned before and we're going
to note that earlier rewards cannot
depend on later actions when you've
received a reward you can't change that
reward in hindsight by picking actions
later on right this just violates basic
causality
so this means that the rewards that
happened earlier
if the policies are fixed which we're
considering here we have a fixed target
policy and a fixed behavior policy then
earlier rewards
cannot correlate with later actions
later actions have no influence on these
earlier
earlier rewards
this means
that for each of these individual
important sampling ratios multiplying
each of these rewards we can actually
truncate them
up until the action that was just before
the reward
we're basically cutting off all of the
actions that happened after we've
already received this specific reward
what does this mean well this means that
this is an equality if they don't
correlate right the actions after time
step k
cannot influence the reward we've
received at k plus 1.
this means that these expectations will
be equal even if we ignore those ratios
because of this earlier property that we
talked about where the ratio if it's
uncorrelated with the random variable
that you're multiplying
then you could just pull them apart and
the ratio will be equal the expected
ratio will be equal to one so it
disappears from the expectation
but this doesn't mean that these have
the same properties in fact this term
here might have much lower variance in
some cases because it's basically a
shorter
partial trajectory that we're
considering
right
now interestingly and maybe kind of like
aesthetically pleasing as well is that
this new thing can be written down
recursively quite nicely as shown here
so we're considering in this whole slide
we're considering monte carlo returns
right we're not considering
bootstrapping we'll get back to
bootstrapping later but for clarity
we're just considering monte carlo
returns for now
and then this term here can be written
down as follows where at time step t the
important sampled return denoted here
with this superscript row
will apply an important sampling ratio
but just one
just the important sampling ratio is
corresponding to that time step
to the reward
and to the next return at the next time
step
recall row here row t is simply the
division of the target policy at that
time step only with behavior policy at
that time step
so we're basically saying is oh yeah we
get a reward we need to correct for the
fact that we took this one action under
the behavior policy but it might have
been a different distribution behavior
policy than the target policy so we need
to correct for that
and we also need to correct the rest of
the return
but the subsequent corrections on all of
the future time steps t plus one t plus
two and so on and so on they're
recursively within this term right at
the next time step only we're going to
apply rho t plus
one and then row t plus two at the time
step after and so on
so instead of multiplying for instance
this first reward with the important
sampling ratio that would correspond to
the full trajectory we're only
multiplying it with this one
step
and this can reduce variance quite a bit
and especially if we later combine this
with bootstrapping
and this is called per decision
importance waiting because we're only
adding each important sampling ratio
essentially when we need it and no
earlier
and this means that if you then write
this out it will be equivalent to this
term where each of the rewards get
multiplied at least with the first
important sampling ratio but only the
reward the later rewards also get
multiplied with later import sampling
ratios rather than indiscriminately
multiplying all of the rewards with all
of the important sampling ratios
so we have this recursive definition
and now we know and we can prove that
the expectation of this return will be
an unbiased estimate for v pi even if so
rho uh corrects for this the mismatch
between the target policy pi and the
behavioral policy mu so the assumption
here is that we're generating the data
under this behavior policy mu but then
rho
sorry g row will be an unbiased estimate
for v pi so we can use this in a td like
update for instance right or monte carlo
like update
we could also do the same for action
values there's a slight subtlety there
because for action values we don't need
to
correct for the first action because we
will already be conditioning on having
taken that action if we want to learn
action values we typically say okay we
have some states say st we took some
action a t and we want to update the
value of that conditioned on the fact
that we were in that state and already
took that first action
what does this mean we need a slightly
different return
one that doesn't apply the first
important sampling ratio because the
first action is already given right when
we update our action values this means
that we can only we should only apply
the more temporary ratios a little bit
later
so again this return will imply apply
the important ratios on a step-by-step
basis on the prioritization basis
but it will start basically one step
later before it applies the first one
then on the next time step row t plus
two gets applied and so on and so on so
i just want you to basically take a
moment and reflect on this how and why
these are different and what that
actually means and convince yourself
that these are both
appropriate terms
so this one is unbiased for v pi is the
claim and this one will be unbiased for
q pi
for
the
state action that we already took
so for sd80
okay
now we're going to move on to the second
topic to reduce variance
which will be
control
variance
and to explain control variants i will
actually start by showing an example
so we're going to go back
to the
blackboard if you will
um
there we go
and we're going to do a very similar
example to before so let me just start
first by uh
make sure you let me use white
you want to start by writing the action
and let's just consider the immediate
reward case first we're going to go
uh discuss the sequential case in a
moment
but let's first do the immediate reward
case where again we're going to define
an action a reward
a probability of the action and a
probability under the behavior policy of
the action so again let's maybe consider
an action that goes to the right
or to the left let's make the rewards
very simple let's say to the right is
plus one
and the left is minus one
let's also consider a very simple target
policy we're only interested in
estimating what happens if you go to the
right
right of course by inspection you could
very easily like
determine already what the value is but
that's not the point we're going to
basically try to estimate this from data
we don't actually know the rewards
and maybe there are they might actually
also be random
you don't know that in advance so what
we're going to do is we're just going to
consider an update to our value function
and let's consider the tabular case just
for simplicity right
so our update to the value function
let's call it dv
t
and so the value of s t or
actually let's let's not even put the
state there um
so let's just call this v t
so the value is going to be
an estimate of the expected reward
under the target policy
so our update to the value is going to
be defined as uh some step size
as usual
times the reward
that we get
and we're going to multiply that reward
with
row
right we're going to correct the reward
so that we get the right expectation
and we're going to take our current
value and
update it towards it this is the tabular
update so there's no times gradient or
something just for simplicity
so what is this
um we can just write it out right um
there's two cases either we picked the
action to go left or we picked the
action to go right
and the behavior will actually pick each
of them with equal probability
so what happens here let's just write
out the cases
if we pick the action to go right
our update will be alph alpha
times
the important sampling ratio when we go
right what is the force information when
we go right it's going to be pi
divided by mu so what we get is
the in
pi
um
right divided by
mu right
is going to be 1 divided by half
is going to be 2.
okay
so if we picked the action to go right
we get
rho equal to 2
times the reward that we get which is
one
and we're going to subtract our current
value
so what is our current value it doesn't
really matter for the purpose of this
example so i'm just going to put vt
this is if
the action
was gonna
go right
and of course then the alternative is if
the actions to the left the important
sampling ratio then
has
zero divided by half so this will just
be zero
times minus one but the minus one
doesn't really matter of course because
we're multiplying by zero
if the action is
to go left
right so we basically get that we either
let's actually write that again so the
updates are either
basically update your value towards 2
or update your value towards zero
now you may have already done the math
uh it's not too hard our actual reward
ongoing right is always one and indeed
this is doing the right thing on average
right because we 50 50
we pick the action to go
right or left because the behavior
policy is 50 50.
and in one case we update towards two
and in the other case we update towards
zero so on average we're updating
towards one we're doing the right thing
we're updating towards the right return
but why would we why would we do it like
this why would we update towards this
arbitrary value of zero
when we take an action that the target
policy would never take
well it's because to get the right
expectation but is there something else
that we could be doing is there
something maybe a little bit better that
we could be doing
so let's change the update now let's
consider a separate different update and
what we're essentially going to do is
the following
we're going to
pull out and i'll explain in a moment
why we can do that
the row
outside of the brackets
so this first part is the same right we
are multiplying the row the import
sampling ratio with the reward
but it now also multiplies the value
the reason we can do that is because the
row and the value they don't correlate
with each other
and that means
that in expectation
rho times v is the same as v
now
let's go through the example what does
this mean for this specific example
let's just go through the cases again
we're going to have alpha times 2
because the row is now outside of the
brackets
plus 1 minus vt
if the action is
to go right
but if the action is to go left
we're going to multiply the whole error
term here
with zero
because this is our important sampling
ratio rho
so what does this mean well this means
that if we're going to
go
to the right
we actually can consider this to be
an update with a step size of two alpha
towards one
oh nice we're updating towards one which
is exactly the right value to update
towards right
if the action is to go right
and when the action is to go left
we just don't update at all
so you might notice here that this is
lower variance because we either update
towards one which is the value that we
actually want or we don't update at all
so when the value becomes equal
to one so when our value estimate
becomes accurate
this second update just stops updating
whereas the first update that we were
doing would continue to update with
updates that are on average they're
correct but they continue to bump around
right you continue to update towards two
and then to zero maybe it's a zero again
and it's 2 again and in the end it all
works out
but the second update is lower variance
because it will just update towards one
and when you've already reached one
you're done and it doesn't actually
update anymore the error will simply be
zero
so that seems intuitively to be a good
thing right
and indeed we can show that the
variance of this update
is going to be
alpha squared
steps i squared
if i did my math correctly
and the variance of this update
will be zero both of these are under the
assumption that are accurate that our
estimate is accurate so this is what i
just mentioned essentially let's
consider that your v is already one
so let's say that explicitly if v t is
already good if it's already one then
the variance here will be
alpha squared because we either update
with alpha up or with alpha down
you see that two minus one
is one so we either update up with alpha
and zero minus one is minus one so or we
update down with alpha so we're
continuing to update with whatever the
step size is right in the second case if
value is
already accurate vt is already one
then our variance is simply zero because
the error will be zero
so that looks good right we've lowered
the variance
now obviously this is somewhat of an
extreme example
because the estimate is already correct
but it turns out this
principle applies more generally and
what we've essentially done here is
we've added a control variance that's
how it's called and we've kind of done
the dimplicity but i'll i'll now talk
about this a little bit more explicitly
so we can better understand exactly
what's uh
what's going on
but let us first return to the slides
okay
so
continuing
we can extend this idea of using control
variants in a multi-step
case let's first do that
and in order to do that we're going to
consider a generic multi-step update
here we're going to consider the lab
update so instead of just assuming monte
carlo for simplicity we're actually
going to allow bootstrapping now and
recall that this lambda return
basically on every step so the
laboratory is defined to just take the
first reward
and then discount and then on the next
step it bootstraps with some weight one
minus labda or it continues oh sorry
there's an additional gamma here uh
there's already a gamma outside the
bracket so the second gamma should
disappear so it's either weights
the bootstrap target with one minus
labda or the rest of the return with
ladder
um here the gamma lab is correct because
now we've
pulled out these terms out of the
expectation and we can rewrite this
we've also mentioned this before in a
different
lecture that you can rewrite this as the
one step temporal difference error
plus
the discounted and additionally
down weighted with this labda parameter
recursive
laptop difference error
so the lab.temporal difference error
which is your lab the return monitor
current state estimates can be written
recursively as the immediate temporal
difference error plus the discounted
with additional labda
recursive itself
so
check just sort as a sanity check that
if labda is zero we get zero
we get one set td back td0
if lab is one this will turn out to be
equivalent to the monte carlo error
so this is just reiterating what the
multi-step case looks like and then this
idea of control variants can be applied
there
by basically weighting the errors rather
than the return
so basically we have this minus v part
inside here we're going to do exactly
the same trick as we did in the example
and this is going to
often reduce variance it doesn't
actually necessarily reduce variance but
it often reduces the variance
and this is sometimes also called error
weighting
now why does it reduce the variance so
let me in order to
basically
talk about that let me actually go back
to the
whiteboard and the idea is now that we
basically had let me do the one step
case first so we had this first thing
which was rho t
times rt
minus vt so why do we keep using this
term control variance
because we're going to replace this
with this second term right so this is 1
and then 2 is a different term where we
pull the row t outside
and we're going to note that this is of
course equal to rho t rt let me just
push it inside plus 1
minus rho t
v t right
and that's going to be equal to rho
t rt plus 1
minus vt
plus
1 minus rho t
vt
so we get
if we call this 1
that 2 is equal to 1
plus this additional term
so what is this additional term
well the expectation of this additional
term
one minus rho t
v t
i'm gonna claim actually let me be a
little bit more explicit and put the
behavior policy there
i'm going to claim the expectations this
is zero
well why is this zero because this thing
is going to be equal to the expectation
of
one minus rho t
times the expectation of uh vt if we
would be conditioning on state here uh
this fine expectation wouldn't even be
an expectation would just be the value
of that state
i'll just keep it in the expectation
just for clearness
but note that the value at that time
step doesn't correlate with the
important sampling ratio because we're
considering the important sampling ratio
of the the action after taking the state
so we can pull that out if we could
correctly condition on everything
and this term over here is going to be
equal to zero because the expectation of
rho under the behavioral is equal to one
so we get one minus one is zero so this
full thing is equal to zero
so what does this term do then this is
our control variant so implicitly by
weighting the error which we were doing
over here
rather than just the target which we
were doing
over here
then what we're doing is implicitly
we're adding this control variance which
we've
written over here
what this is doing is there's a
different term which has expectation
zero but it varies along with the thing
that we're interested in so the thing
we're interested in is this thing over
there number one if you will but we've
added a separate term at the end which
is expectation zero but because it
varies along with the thing that we're
interested in you can reduce variance
and that's what it's doing that's in
general called a covariate so we can
view this error weighting that we were
doing as essentially applying covariates
adding terms
that reduce your variance but do not
change your expectation
there's more discussion about this in
the book so feel free to
consult that if you still feel maybe
slightly confused about these things
i will now return back to the slides
where we've applied this idea
to the multi-step case by basically
waiting the errors rather than waiting
the
returns
and we've written that down recursively
here where we have this new notation
delta rho lambda where the row indicates
that we're important sampling and the
lab that just indicates that we're
partially bootstrapping along the way
and the rows now apply to the errors
rather than to the returns
so this by design includes those one
minus row v control barrier terms that
we've just seen
on the blackboard as well
and as i mentioned this is sometimes
called error rating as in contrast to
reward waiting or per decision
importance happening
but we can apply this like in the per
decision case of course as well which
you've already doing here with this
equation at the top right we're not
applying the full ratio for the full
trajectory we're just doing this on a
per step basis
and one can then show that the
expectation of this under the behavior
policy is equal to the expectation of if
we would only apply these things to the
return and then just subtract v as
normal
the expectations are equal but the
variance is not necessarily equal
and often times the variance on the
error-rated term is lower than that on
the return weighted term
this is by the way not necessarily the
case um
we run throughout the paper
where i was involved in as well with
rich sutton and
rupam i believe rupa mahmoud and doing
apricot
and
there we were we at some point thought
maybe we can prove that this actually
has lower variance in general turns out
it doesn't there are counter examples
but in general it it seems like more
generally speaking it often has a lower
variance which you just can't say it
always has lower variance it doesn't
necessarily have lower variance but as
the example showed that i showed you
before
um it makes intuitive sense to wait the
errors because essentially
it means that you don't update when you
get data that is irrelevant rather than
arbitrarily updating towards zero and
then updating towards higher numbers
later to correct for that
okay
so typically this is why i say that this
error rated term can have lower variance
um in practice it often has lower
variance you just can't say that
generally it always has lower variance
okay that brings us to the end of that
topic of control variants and now we're
going to go to the next topic to reduce
variance which will be
adaptive bootstrapping
so
[Music]
we already talked about bootstrapping
quite a lot obviously
and bootstrapping is typically a good
way to reduce your variance a little bit
and
we already
had these labda weighted
errors or returns where lab that could
be lower than one right and if lab is
lower than one then we're going to
reduce the variance
but we might incur some bias because we
might bootstrap
the bias is not because we are
estimating for the wrong distribution
necessarily the bias is there because
we're using estimates our value function
won't be completely accurate and
therefore there might be some bias due
to our function approximation error or
just our estimation error because we
have seen finite data so far
so for action values
we can use this important sample to
return
where you can see the normal thing
showing up here this is the expected
sarsa return
so we're doing the one step case here if
labda is equal to zero
but lab can be different than zero and
if lab is different than zero then we
basically recurse
and go into the next time step right so
this
quantity here is defined here
recursively
and we can see if labda is zero we can
see the expected sarsa thing reappearing
that we had previously which is shown
here on the slide as well
and there's no important sampling then
we already discussed this slightly
earlier in this lecture right where we
could just bootstrap and wait with our
target policy and then we are doing
correct of policy updates in some sense
but we are relying on these estimates
and these estimates might not be
accurate and therefore we might want to
do that
and we can see here in the equation if
lab is 1 then this bootstrapping term
just completely disappears and we find
the
important sampled
return for action values that we saw on
an earlier slide again so this smoothly
interpolates as per usual with lab
between zero and one
but there's a problem
if we bootstrap too much which is
specific to the off policy case so i
already said oh yeah maybe your
estimates are not great so you might
bootstrap and it might not be too too
great there's a completely different
reason why this bootstrapping might be
harmful it might open you up to what is
called the deadly triad so let's recap
what that is
the deadly trial refers to the
possibility of divergent learning when
we combine bootstrapping
with function approximation and of
policy learning
it only shows up when all three of these
are present
unfortunately they often are if you want
to do what is nowadays called dprl we're
clearly doing function approximation
because we have deep neural networks to
approximate our value functions
we're often doing bootstrapping because
just doing full monte carlo rollouts
would be very much too high variance in
many cases and therefore learning would
be quite slow or ineffective
and we're also often doing off-policy
learning for instance the dqn algorithm
the well-known dqn algorithm which was
used to play atari games quite
successfully and has sparked a lot of
follow-up work as well
it uses q-learning which means that we
are um in the in the
original dq and we're actually
bootstrapping after just one step it's
one step clear learning
in later iterations such as rainbow dq
to name one example
this has been extended to multi-step
updates but still
after say three or five steps or
something like that typically we
bootstrap because it just turns out to
be more effective that way the variance
will be lower and the updates will be
better but it does mean that we're
combining all these three things then
we're bootstrapping we're doing
deep neural networks so we're doing
function approximation and we're doing
off policy learning because we're using
something that's uh
q learning like we're bootstrapping with
a max that's the
state where we will trap it
so to recap so why does this happen
so we saw this example before so we have
two states here the rewards are always
zero but we're only going to consider
this one transition and we only have one
feature the feature value here is one in
this first state and the same feature
has a value of two in the second state
we only have one parameter
uh w and our value function will be
linear so it will be w times x which in
this case first state is just w and w
times x in the second state will be 2w
because the feature value has value 2.
the optimal value function is within
reach if we would set our weight
parameter w to 0 our estimates would be
perfect but if we use td learning one
september learning on this transition
then
something happens that we don't want so
our value function the the weights of
our value function will be updated
according to one step td learning as
shown on the slide
and if we fill in all the values so we
put r0 here
we the value of the next state will be
2w and the value of the current state
will just be w
the gradient will just be 1 because um
sorry it will just be w because the
gradient of our uh
um
no sorry i'm saying this wrong the
gradient will just be the the value of
the feature which is for this state one
right so the gradient here is because
it's linear is just x
and x in the state that we're interested
here is just one so we're multiplying
this whole term with one i got
distracted slightly with the w here but
the thing that happened here on this
step is that both of these values have a
w in there and we just pulled it outside
of the
parentheses
so this total update you can verify this
by carefully slowly stepping through it
will look like this
and then we can know that if the
discount factor is more than a half
which is of course quite a common case
then in this specific example if the
weights are already positive
then this term will also be positive
which means that the next weights will
be even more positive
similarly if the weights are negative
then the next term this term will be
within the brackets will be positive but
the weights are negative so the total
term will be negative which means that
the next weights will be more negative
so in both cases we're updating away
from zero and in fact will update away
faster from zero the larger your weight
becomes the larger the magnitude of the
weight becomes
this means you'll exponentially quickly
diverge to either plus infinity or minus
infinity depending on where your weight
started
so that's problematic and that's uh an
example of the deadly triad and it's it
it happens because we're updating of
policy which in this case refers to the
fact that we're never considering any
updates from this second state so
although we go there we never update by
going away from it again that is why
this example is off policy
so we can use multi-step returns as well
so let's expand the
example and let's not consider also the
transition that moves away from the
second state
and instead of just using one-step tv
let us consider a labor return we're
still only considering updating this
leftmost state so we're still going to
update off policy
where with off policy here i mean that
the state visitation distribution does
not correspond to our behavior our
behavior steps through these states as
follows we start here then we go here
and then we go there right and then we
terminate so we spend equal amounts of
time in the first state that's in the
second stage under our behavior
but the updates will conduct off policy
by basically only considering updates
from this first state
now if we consider the object to the
leftmost states we see if we just expand
everything there's this additional term
of one minus lab that appears compared
to a previous slide where we saw that
the update would be w plus alpha times
two gamma minus one times w
if you just write out this update where
the rewards are both zero
you get this different equation where
depending on la da this two times
gamma term gets down weighted
this actually means that this multiplier
to the weights will be negative whenever
this equation holds whenever this term
is smaller than one
this means that if lambda is larger than
one minus one over two gamma
then
whenever your weights are positive they
would go down rather than up and whether
whenever they're negative they will go
up rather than down
so if this condition holds if your
lambda is large enough
it becomes a convergent algorithm where
the weights will now be updated towards
one sorry towards zero rather than away
from zero
this is specific to this example right
i'm giving you specific equations
specific conditions to lab down
which basically
take into account the full example and
this two here for instance comes from
the fact that the feature vector the
feature value here is two
but the gen the
um general gist of it is true which
means that
if your lab is large enough you're good
to go you won't have any problem with
the deadly triad and this makes sense
because the deadly triad was about
bootstrapping function approximation and
of policy learning
the larger you set labda the lesser
bootstrapping the extreme case is when
you consider a lab equal to one in which
case we're using monte carlo returns
we're not bootstrapping at all if that
is the case you can update of policy you
can use function approximation you won't
diverge
but if
um you want to still bootstrap you can
use a lower labda it just needs to be
large enough so it's not actually a
binary thing it's not about all boot
trapping is bad
and if you do any bootstrapping you'll
be in the deadly try it no
this example shows that if you just
bootstrap less sometimes you can be fine
okay so let's for instance consider the
example like if your discount is as low
as a half then this is just saying oh my
lab needs to be larger than zero
and that's often the case if your
discount factor is say three quarters or
something like that then
this term would be
this whole term would be two-thirds so
that means that your ladder needs to be
larger than one third if i did my mental
math quickly correctly so basically the
larger your discount is the larger your
uh
lab that needs to be for this to still
be converging but we can see that there
are lab does that will make it converted
in this specific example
so concretely if your gamma is your
discount is 0.9 then it turns out it's
sufficient to have a lablet that is
larger than 0.45 that's not a hugely
large labla we often use lab less than
are say 0.8 or 0.9 or something like
that
so this turns out if you do that in this
example you're good to go you can update
off policy and you would still be fine
so the conclusion of all of this this is
just an example but the conclusion is if
we do not bootstrap too much
learning can be a lot better we might
mitigate if not fully remove the deadly
triad
so maybe we can reduce variance by
adaptively bootstrapping
because we don't want to bootstrap too
much because then we might get into the
deadly triad but we also don't want to
boost up too little because then the
variance will be very high especially if
we're doing these important sampling
updates right because important sampling
ratios tend to add variance to our
updates
so can we maybe adapt to the bootstrap
just enough
so that the variance is at least bounded
or is at least smaller than it would
otherwise be
i'm just going to present you one way to
do that there might be multiple ways you
could do that
and the idea would be here to bootstrap
adaptively only in as much as we go off
policy so if you don't go off policy you
don't need to bootstrap
you can just continue using the data but
when you grow policy this makes
intuitive sense in some sense right
you're following a certain policy you
reach a certain state at that point in
that state you choose a completely
different action that the target policy
would never take
maybe this is a good point to do to
bootstrap rather than trying to use this
very tenuous data that had nothing to do
with the thing you're trying to estimate
to somehow still update it that is
possible what tends to lead to very high
variance updates so there are some
intuitive uh
um
ideas behind this idea but we can
formalize it which we'll do next so
we're going to start with this important
sample corrected multi-step error right
recall delta t is just your one step td
error
we're correcting it to be a good off
policy error
and we're defining it recursively with
labda to make it a multi-step error
now
we might extend this example slightly by
allowing an initial bootstrap parameter
so basically we're down waiting this
whole thing with an initial lab that t
if labda t note that lambda is now time
dependent i'm allowing labda to differ
from one step to the other
so that's important because
now we can have a special case we've
basically just generalized the thing we
had before if we set lab.t to be equal
to one
right then we just get the thing back
that we had before
lambda t plus one doesn't also have to
be equal to one because i've now
subscripted by t
so this is a slight generalization from
before and this will just make some
things a little bit easier later on
and we can pick a different lablet on
every time step that's what i mean with
adaptive it's two things it allows us to
pick different labels on every time step
and we're going to pick the labla in a
data dependent way
now the idea is
if we define this thing recursively look
at the recursive definition first
we see that there's a lambda t
rho t
then there's a delta and then we go into
the recursion but that means in the
recursion there will be a ladder t plus
one times rho t plus one
they always go together they're always
next to each other
and then the idea is well this row what
is the problem with row so row is the
important sampling ratio
and we know that in expectation by
itself
rho will be equal to one
recall that that's just a property of
this ratio the ratio by itself will have
an expectation of one
but we also know that it can it can
increase variance quite a bit so what
happens why is it increasing variance
well sometimes it'll be lower than one
sometimes it will be larger than one
and indeed it can sometimes be very
large when is it very large it's very
large when we're dividing a large turret
policy probability with a very small
behavior probability that means we've
selected an action that was very rare
under our behavior but it's actually a
really common action under your target
policy occasionally that will happen and
this ratio can become very large when it
does happen
so intuitively the idea will be well
what if we pick labda in such a way that
if we multiply that with rho it's always
small or equal to one
then we know that this is bounded there
are no large ratios
anymore so what does this mean we just
rewrite this inequality and basically it
means that we pick lambda in such a way
that it's the minimum of one
or one
over
the important sampling ratio rho
why is minimum well we don't want labla
to go larger than one so basically what
is happening here if rho
is itself la um sorry if rho is itself
smaller than one
which is often the case right smaller
than one row basically just means that
your target policy
wasn't very likely to pick the action
but the behavior policy was that happens
right right you might pick an action and
the third policy is just not interested
in that action then this row can be very
small which means that one over row can
be large if that is the case we allow
ourselves to continue we allow labda to
be one
we continue to the next time step
however if row is very large if we
picked an action that was quite rare
under the behavior but it's really
common under the target policy then raw
can be very large and then we're saying
oh but then we're going to pick labda to
be smaller
so whenever we pick an action that was
very rare under the behavior but it's
very common under the desired policy
let's just bootstrap there
and that's the whole idea essentially
so when we're two of policy if raw is
very far from one we're going we're
going to truncate this multi-step error
the multi-step return if you will
and that's the same as just
bootstrapping there because
the closer lab is to one oh sorry the
closer level is to zero the more we're
bootstrapping right if it's all the way
equal to zero then we're just
bootstrapping fully there
if it's a little bit above zero we're
still partially bootstrapping there
the closer we are to one the lesser
bootstrapping so instead of thinking
about bootstrapping as a binary thing
where we either bootstrap or not we're
thinking about it smoothly
this idea is known as abtd adaptive
bootstrap td by rupam mahmud
or also as v-trace
from a paper called impala
by lasse espahol and others
and it's quite an effective algorithm
and
we're actually free to choose this right
we're to we're free to choose different
ways to bootstrap and in fact in the
tabular case all of these methods will
be updating towards some mixture of the
multi-step returns and therefore they're
all fine they will converge
in deep reinforcement playing this is an
extremely common thing there's a lot of
agents that are using v-trays and
v-trays essentially uses this idea
there's a
couple of more things to be said and i
refer to that paper if you want to know
more in depth but it basically boils
down to this idea of bootstrapping when
you go off policy
and this is an extremely common trick
especially for policy gradients
because policy grades as i mentioned
before really do not like biased return
estimates
because
if you have a biased return estimate
then
you're no longer necessarily following a
policy gradient and of course that
seemed that that might be harmful right
means your policy is not no longer
necessarily improving in fact it might
be getting worse
so v-trace is very common and this is a
very important trick and it really helps
reduce variance because we're basically
just truncating away parts of the
returns that would give you very high
variance but as long as the variance is
okay we're just allowing yourself to
continue and to use multi-step returns
okay now one final thing which is very
related but it's different
way of doing this is the algorithm
called tree backup
so i mentioned there's different ways
you could bootstrap and picking the
trace parameter adaptively as we were
just doing before is not the only way
a different thing we could do is to
first start with the bellman equation
and then we're going to go to the
multi-step case
and effectively we're going to do is
just just first just musing this
equation for a bit
this is the definition
of the action value for the policy that
we're interested in pi
and we know that from the bellman
equation that this is the expectation
of the reward given that you have
seen this state in action and then in
the next states
following that policy
in the next state and then picking the
action according to that this is also
where this expected sarsa update comes
from right it just samples this
note that the expectation does not
depend on the policy because we're
already conditioning on the action a
this is saying given that we have a
state and an action what would be the
value of the policy well we only need to
consider the policy at the next step we
don't need to consider it on the first
step because we're already conditioned
on the action
so to sample this
we can sample it as before we could
bootstrap immediately but a different
thing we could do is basically we can
pull out the action that we're going to
select on the next time step
because that's the action that we've
actually sampled and we can just replace
the estimated value that we had for all
of the actions with the return given
that action
so what are we doing here we're
replacing this summation over all of the
actions by simply pulling out
the
action that we're actually going to take
on the next time step we're going to
turn this into a multi-step update
and we are keeping the weighting right
because this is a weighted sum of action
values and what we're basically going to
do is we're going to use these as
expectations these are estimates of the
expected return right
and we're going to replace only the one
return that we actually had
so the way to think of this is think of
a tree at the root note you're in a
state
there's a whole bunch of actions
and the value of the root note for a
specific policy would be the policy
weighted
sum of the values of these actions right
but what we're going to do is we're
going to actually select one of these
actions and for that action we can
replace the estimate with the value the
the return that we've sampled
that's exactly what's going on here
except we're deferring it one step
because we're not conditioning just on
the state but also on an action so the
first days in action are already given
but then at the next step so basically
the next node in the tree
we're going to consider all of the
actions
and all of the actions that we haven't
selected we're going to use the
estimates that we have appropriately
weighted according to the policy but for
the action that we did select we can
actually use the sampled return it's
defined recursively so the next step you
might bootstrap again
on all of the actions that you didn't
select and again continue just with the
action that that you
did so we only removed the expectation
of q s t plus one a t plus one which is
the action we actually selected and we
replace that with the sampled return
this is unbiased and it's very low
variance because we don't have any
important sampling ratios there's never
a division by mu right
the
pi here
in fact plays a role that's quite
similar to lab of course you can add an
additional lab that and download it even
further
but there's a problem which is maybe
it's a practical problem sometimes this
works amazingly well but there is a bit
of a practical problem sometimes you
might bootstrap a little bit too early
and therefore you need to be careful of
deadly triads if pi is not your behavior
like if your behavior is something
different which is perfectly fine for
this method this is an off policy method
but if your behavior is different you
might be bootstrapping very much after a
first step let's consider that there's
many actions here each of which have a
quite low probability
then this is very similar to just
bootstrapping after one step
however if your policy is fairly greedy
then it might be that you actually take
a number of steps before your bootstrap
in fact i didn't put this on the slide
but if your policy is the greedy policy
then what this mechanism would be doing
is essentially it would take as many
steps as your policy agrees with the
greedy policy so as long as you're
picking greedy actions
and if the degree policy is the target
policy like if you're interested in a
greedy policy then this method would as
long as you're picking greedy actions
would continue to roll out a trajectory
but at the moment that you select a
non-greedy action it would terminate
there and then bootstrap with a max
q so a q learning like bootstrap target
that is actually an algorithm that was
already proposed by chris watkins when
he proposed a q learning algorithm
and if your policy is fairly greedy then
it can actually have long multi-step
returns but obviously if your policy is
quite uniform but you're interested in
the greedy policy then the returns might
be truncated quite a bit
this is more generically true this this
equation is not just for a greedy policy
ingredients hiring policy the target
policy could be greedy but doesn't have
to be
but it has the same issue that if you
if your behavior mismatches a lot with
this target policy then you might
bootstrap all the time and that might be
a little bit dangerous because of the
deadly triad
there are in fact other ways to deal
with the deadly triad there are
solutions for this
um
but no
um
basically we don't know yet what the
best solution is there's multiple
solutions to the deadly trials multiple
algorithmic innovations or different
ways to think about this some of which
are included in the book by sustenance
but we don't know for sure yet what the
best way is to deal with this so this is
still ongoing
research basically to figure out how
best to deal with these issues
okay that brings us to the end of this
lecture thank you very much for paying
attention
and if you have any questions please
post them on moodle
thanks