Reinforcement Learning 5: Function Approximation and Deep Reinforcement Learning

so I've alluded to the topic of today's
lecture quite a bit already in earlier
lectures this is quite natural as well
because we're doing both these parts of
the course for one part is focusing on
deep learning and the other part is
focusing on reinforced learning but all
the other hands may be before the course
you would have expected it may be more
tight integration of these two but turns
out there's a lot to be said about
reinforcement learning without even
talking about how to approximate the
functions so that's what we've been
doing so far we've been talking about
how to do these things maybe even when
you do them tabular ly and the
assignments also reflect that where you
haven't been doing much tensor flow or
deep learning in terms of the
reinforcement learning assignment but of
course many of you will know and have
known before this course started that
there is such a thing as deep
reinforcement learning and today I'll
touch upon what that means what that
means as a term but also what it means
in practice more generally we'll talk
about function proximation which is the
term that rich also uses in the region
and in the southern embargo book more
generally to talk about approximating
functions so again just to recap quickly
this is the setting reinforcement
learning and agents can learn a policy a
value function or a model and we'll
focus on that how to learn and
especially the value function in this
lecture and in general we're just in a
general problem including consequences
time and how to deal with that but
specifically I now want to highlight the
fact that all of these things are
functions I did that is in the first
lecture as well but I want to re
highlight that again so each each of
these a policy of value function or
model is basically a mapping from
Francis for the value function from a
state to a value or a state action pair
in the case of action values to a value
in the case of a model it's a mapping
from estates to for instance a next
state or maybe an expected next state
and in the case of a policy it's a
mapping from States to actions now what
we want to do essentially is learn these
from experience
and if there are too many states which
is often the case we will need to
approximate and in general this is
called reinforce including with for
approximation which is a bit of a long
term but when you use deep neural nets
people typically refer to this as deep
reinforcement learning to just basically
merge the terms deep learning and
reinforcement learning now this term is
fairly new it came to prominence
basically a few years ago but the idea
of combining neural networks worth
reinforcement learning is actually quite
old decades old
we just gotta may be better at it and of
course in addition we have the same
benefits that deep learning has
exploited quite successfully in recent
years which is the fact that we not only
do not only do we understand the methods
better we also now have the computer
resources to train these things for
fairly long amounts of time with fairly
large amounts of data and it turns out
if you do that then certain things just
work that we're not sure we were not
sure would work say in the 90s or
earlier that said many of the algorithms
the base algorithms that we use are
essentially that old q-learning for
instance is from 1989 by chris watkins
and it has it basically was immediately
extended to also be used with neural
networks but that doesn't mean that we
really understood what was going on back
then I say we but I wasn't researching
my them but we as a field that we really
understood what was going on back then
or that we were very effective at it but
it didn't work if you were a little bit
careful if you were a little bit of
aware of the properties of the function
approximator
of the neural network in that case but
all basically first I'll step up a bit a
little bit and not go immediately into
the deep Nets and just to be clear for
this lecture are we talking about
learning value functions in the next
lecture I'll talk about learning
policies explicitly so we'll return to
policy gradients which got covered in
the lecture and exploration and
exploitation when we were discussing the
single state immediate reward case and
next extra I'll talk about policy
gradients in the deep reinforced
building
where we work offering the fool
sequential problem and in addition we're
covering the case where we have
arbitrary nonlinear function
approximation but for this lecture we'll
stick to the value functions just for
clarity mostly now the motivation is
quite clear we want to use reinforcement
learning and turns out we can use
reinforcement learning to solve large
problems one of the earlier examples was
backgammon this is something that Jerry
Tercero did where he basically used TD
learning with neural network and then
use this in the game of backgammon which
is a tricky game but turns are actually
not that tricky for reinforced planning
because you can play many of these
things against yourself essentially
because the rules of the games are
fairly well-defined so you know exactly
you can you can program it program that
in into a computer you can have an agent
play against itself and it can learn to
become better and of course more
recently this was applied this
methodology not the exact algorithm but
the methodology of applying
reinforcement learning was applied to go
which is much much bigger than
backgammon and you could never hope to
learn this if you were going to try to
store every value that you could
possibly find for every state
individually in addition there are some
problems which are may be easier thought
of as having an infinite continuous
space so in example here is helicopter
helicopter control was done several
years ago go several years ago as well
and there's many similar control
problems that have been tackled with
reinforcement learning and here the
state space is basically infinite
although it's bounded typically and then
again the problem is to find a mapping
from that state space to either a policy
or a value function and of course again
it's good to keep in mind this example
of the robot just whenever you think
about reinforcing I find it quite
helpful when I think about algorithms or
what not to think about the case would
this learn or would this be able to
learn at all when it would be embodied
on the robot right if this robot has a
limited brain and it needs to do some
approximation because it has a limited
view of the world but the world is big
then somehow you need to be able to
cover that case so the main question
here is how can we scale up our methods
for prediction and control and so recap
not fully but which mostly considered
look-up tables so far at points i've
pointed towards hey this equation is for
the lookup table but you can extend it
to the function proximation case by just
replacing from some stuff but we haven't
really almost depth of what that means
or how that work and obviously whenever
we can apply this to a state value you
can typically also apply this to state
action value so I'll be trading these
these cases more or less indifferently
now there's a few problems with large
MVPs that we it's good to be explicit
about that we want to tackle so the one
is quite obviously that if there are so
many states you cannot even fit it in
memory in a computer you cannot build a
table that is that big to cover all the
states in a game like go for instance in
like in a typical big so especially the
robots are clear examples if the robot
is in the real world you might want to
put universe on the machine but that
doesn't fit so you need to do something
else there additionally and sometimes
even more importantly it's too slow to
try to learn the value for each state
individually and this is essentially the
problem of generalization when every
cease state we want to learn about that
state but we also want to learn about
all the similar states because if it's a
really big problem you're never going to
see the exact same state twice so that
means that if you would learn about each
state individually you would never learn
anything or you would never learn
anything that you can use in the later
States because they will all look like
brand-new states so obviously you want
to generalize this will come as no
surprise and another thing that often
pops up and I'll come back to is that
individual states are often not fully
observable again it's useful to think of
a robot which maybe has a camera to view
the world and it only sees whatever it
sees through its camera but it can't
look through walls it can't look maybe
behind itself the sensors are in that
sense limited and you want to somehow
deal with that
so the solution that we're proposing
here is just to estimate these things
with some function and notation that
I'll be using as I did before is that
there are some theta parameter I should
say here that the book these days uses W
for the parameters for the value
functions just to prevent confusion but
doesn't really matter it's just a name
variable name for the parameters and you
can think of this as all the weights of
your neural network and the idea is for
Institute in a prediction case to
approximate the true value of a policy
this also immediately kind of points
towards a means to do control because as
you may remember we've discussed policy
iteration at some point and generalized
policy iteration means you first
estimate the value of your current
policy and then you improve that policy
and you don't have to fully improve it
you can just improve it a little bit and
then you could do the estimation again
and the improvement and you can
interleave these steps quite fine-grain
e that's generalized policy iteration
which as an intermediate step has this
step of approximating the value of a
policy so if we can approximate the
values of policy this already gives us a
way to do control of course we can also
immediately try to estimate the optimal
values which is on the right there you
could or V star or Q star which
essentially means is we can do sarsa or
we can do Q learning we could do the on
policy thing of learning about the
current policy or we could try
immediately to learn about the greedy
policy which basically means we're
interleaving the steps of policy
evaluation and improvement very fine on
each step were kind of doing both at the
same time now the hope is if we have
such a function if it's a well defined
function on the full state space we can
plug in any state and we can get an
answer immediately and if your learning
algorithm is well designed in some sense
and if your function isn't too weird
you'll probably get a reasonable answer
if the state looks a little bit like
states you've seen before that's the
hope of course there's many ways to do
that and I'll talk about a few of them
in this lecture and the high-level idea
that we're talking about mostly here is
to update these parameters theta using
either Montecarlo or temporal difference
learning and in the end I'll talk about
how to unify these but the other thing
that I wanted to say which I won't
touch upon that much in this lecture but
I'll get to back back to a little bit
also in the next lecture is that the
environment States might not be fully
observable as I mentioned the robot
example when you just have your vision
from the camera which means you might
want to learn a state update function
and state up the functions is to remind
you is something that takes the previous
agent state and the current observation
and it outputs a new agent state so now
I'm using s here to do to refer to the
agent state which is essentially state
internal to the agents which might not
coincide with the environment State
which might be much much bigger so in
the slides that will come mostly I won't
talk about this too much and one way to
think about this is that whenever you
see a state you could just think about
the observation which is a simple way to
construct an agent state you just take
your observation you just ignore the
previous agent state and you're done
right this is an agent says but if you
only look at that your potential
solution might be limited the robot can
only ever do you can learn only ever
learn a function that is a direct
function from these observations which
might not always lead you to the optimal
policy so I just want you to keep this
in mind you can maybe forget about it
for most of the lecture but whenever you
then want to apply something like this
in the real world and I will get back to
this again this is something that is a
that is potentially important okay now
you might think I might merely go and
talk about deep neural networks now as
plugging those in but there's actually
many potential choices that I do want to
mention so the obvious first curve a
choice here says put into the artificial
neural network as a function parameter
alternatively one thing that has been
done in the past is to put a decision
tree in there which has maybe somewhat
harder boundaries in terms of
generalization but they're fairly well
understood and you can use them
effectively if you if your problem is
well suited for them you could also do
something nonparametric their nearest
neighbors just store a few samples find
the nearest one use that one and this is
actually something that is sometimes
pops back up again the people re
investigating turns out it works quite
well in certain problems or you
construct certain features I'll touch
upon that more so here it says free a
wavelet basis
one potential choice there's many ways
you can construct features course coding
is essentially a different way to
construct features don't worry if you
don't know what any of these two mean
I'll talk more about especially the
course coding in a moment but just think
of it as a way to construct features
which you then later use to maybe do a
simple linear function rather than to
try to learn a whole deep neural network
and these were very popular in the past
also because they're quite
computationally efficient and they tend
to be quite data efficient because it's
easier to learn a linear function then
it is to learn a deep neural network of
course you're limited in the type of
functions you can learn you need very
good features if all you're ever going
to do with it is construct a linear
function of your features now in
principle in terms of the generic
updates you can use any function
approximation but of course reinforced
learning has some specific properties
which might make it harder there are
some listed here this is not meant to be
an exhaustive list but one important
difference with the standard supervised
case is that the experience is not iid
typically successive time steps and
therefore successive updates if you
update your value function online will
be correlated which may or may not be a
problem that is good to be aware of also
as I mentioned in the first lecture and
after a couple of times the agent policy
affects the data it receives which
affects the nature of the function
you're learning so these things are
tightly integrated in basically highly
non-trivial ways and the learning
dynamics of the full system is something
that is not that well understood and
there's a related note the value
functions can be non stationary now the
policy iteration case is a very clear
example of that there we're not plugging
in one specific policy but we're going
to change the policy repeatedly because
we want to find something that works
really well which means each time we're
kind of trying to estimate a different
value function but turns out depending
on your learning algorithm the value
function might be non stationary or the
targets that you're trying to
approximate can be non stationary for
other reasons for instance when you're
doing bootstrapping as in TD learning
because then I'll show you the examples
later again on later slides but just to
remind you what does that mean to deal
earning it means that you're updating
the value of the states towards the
reward and then the value at the next
state add it together but the value at
the next state itself is part of the
function that you're updating which
means that in total your update is
non-stationary which may cause problems
for certain types of function
proximation or it may invalidate certain
algorithms that might assume that this
is not the case and it might actually
break them in practice as well another
property of reinforced learning is that
feedbacks delayed which means especially
in the online case you might do
something you might immediately update
using say TD learning but it's not
always clear that this is the best thing
to do sometimes you want to wait a few
updates as in Montecarlo learning but
then this it creates overhead you need
to do the bookkeeping if you if you
program this and it's not always true if
you'll to do this know that that part
might be easier than some of the others
so here's some some potential choices
like generic choices I would say for a
function proximation we started off with
the table right just store for every
state you can possibly see just think of
every observation you can possibly see
store these in a exact value in a table
somewhere which you can an update maybe
you still update it only slightly for
each observation because maybe sorry for
each transition because maybe your data
is noisy as we discussed in earlier
lectures but this is a fairly well
understood thing now an easy thing you
can do especially when you're sailors
say continuous spaces which are still
not that high dimensional you could
think of just cutting this up so if you
think about your state space as being
Sprint's as a two dimensional space it's
just a plane and its bounded well one
thing you could do is you could just cut
that up into pieces and then call each
of these pieces of state that's a valid
thing to do and then you're basically
back to the tabular version right we've
just manually created something that is
essentially a tabular MVP however one
thing that you should note then is that
we've actually aggregated States
together which means you might not be
able to observe exactly in which state
you are so you've made the problem
partially observable when you do that
slightly more generically or slightly
more generally you could do a linear
function approximation this basically
subsumes both the seller and the state
aggregation case
where we say we have some features to
cutting up the safe space was one
example but you could think of many
other features that you could construct
and then we hope that we can learn a
function that is accurate enough as a
linear combination of these features the
benefit of this is that it's very well
understood and we can say much more
about where does their duty algorithms
go when you run them for a long time do
they converge in it so where do they
converge it also tends to learn faster
than if you do arbitrary nonlinear
function approximation but of course
it's very much dependent on the nature
of the features and it might be very
tricky for a certain problem to define
good features but if you can this is
definitely something that you could
consider in the most general case and
one that we happen to like a lot these
days is to use differential nonlinear
functions such as neural networks the
benefit of this is that we can just toss
a raw signal in like pixels from a
camera and you can still hope to learn
and that is a big benefit because you no
longer have to create features you don't
longer have to actually understand your
problem that well for instance on these
Atari games I've run algorithms on Atari
games which I don't even understand I've
never played I don't know how to play
like the algorithms will be much better
at these games than I am and I can do
that because I don't need to understand
the game which is quite a nice benefit
so which you use I assume most of you
will be tempted to use the deep neural
networks which is fine and they tend to
give you best performance these days
depending on the setup but just to point
it out this list up there basically from
the top to the bottom starts off with
things that we understand really well
but might have a little bit more weak
performance and then they go to things
that we know work well because we've
seen it in practice but we can't really
say much with confidence about in in the
theory so we can't necessarily guarantee
you that it will work well it just
happens to be the case that we know when
we actually run this and you're a little
bit careful with how you do this that
you can make them work really well by
the way as always feel free to stop me
whenever when you have a question
because you're likely not to be the only
one with that question
okay so this is just this is very
generic you kept this might be a little
bit boring even considering you've seen
all this stuff in the other part of
course as well but just to be clear
we're going to define some
differentiable function denoted here J
just think of this as a loss which is a
function of your parameters and then
we're going to take its gradient which
is just a vector with all the partial
derivatives and then the goal is to say
find a minimum if you think of it as
loss actually in the policy gradient
case we might define things that we want
to optimize find the maximum rather than
the minimum but of course that's easy to
do just by putting a minus sign in front
but Denari we want to move the
parameters in the direction of the
negative gradients to do gradient
descent and this gives you one valid way
to update of course there's other
optimizers but this is a good one to
keep in mind and most of the concrete
algorithms are giving this lecture I'll
give you the stochastic gradient descent
version so actually maybe let me go to
the next slide which gives the
stochastic gradient descent for value
functions that doesn't mean that of
course you can only use that you can
apply different optimizers you can apply
rmsprop atom things you might other
optimized you might have heard of from
deep learning essentially anything you
can toss the gradient into and that
results into an update you can apply to
the deep reinforcement learning case but
for clarity I will only show you the
vanilla stochastic gradient descent
versions of the update so then to apply
this to value functions we define some
expectation and this is actually
important now something to touch upon a
little bit so I subscript this
expectation with Pi which is the current
policy and the thing that is random in
is expectation is actually the state
because note that we've put the true
value of that state on the left and then
the current estimate of that states V
theta on the right and then there's a
squared error but the only thing that's
random there is a state so what I mean
with the expectation under the policy of
this thing is I mean there are some
distribution over states which in this
case
is the thing that isn't used by the
policy that you're following so this is
a non policy loss function essentially
and this is important when we do
function approximation because we have
to somehow decide where to spend our
resources where you're going to fit a
function it's going to fit basically the
best it possibly can where you have the
most data so this ties into the question
of how do you then select your data
which is important for the control
problem and if you pick a certain policy
that never goes to a certain portion of
your state space you'll only get a
little bit of function approximation
resources allocated there and your
generalization there might not be that
good v theta yes sir good question V
theta is basically our current estimate
with any function approximation of the V
Pi in this case so this loss is when you
do the on policy policy evaluation case
I use VT so just generically as this is
some value function which is
parametrized by theta one way to think
about that is two it could be for
instance a neural network which takes
the agency this input which could just
be your observation and then the output
is the current estimate for the value of
that state under the policy so I'll give
you actual I'll give you the TD
algorithm in a moment so then you can
see exactly what the TD algorithm in
this case is so I should have preface
this by saying this is again me
basically just first defining things
because this loss is not something that
is actually available you will not have
access to V PI particularly so we need
somehow to instantiate this to turn this
into an algorithm we can actually run
and I'll do that in a moment this is
first just basically define what would
be an ideal updates if you want to do
stochastic gradient descent to learn
this value function yes
see the policies around attention for
today is gonna end up so the question is
why is so the random variable I said the
only random variable is the state and
the reason that is the case is because
I've actually used the true value for
that state so I'm not rolling out
anything - I'm don't have a Monte Carlo
return there in particular I will have
in a later slide because you have to
instantiate this and then that will also
be random in this particular formulation
the only thing that's random is the
state because I'm using I'm plugging in
the true value function which of course
is something you can't do in practice
but it's just for the for defining the
loss essentially and then of course we
can derive a stochastic sorry the
stochastic gradient descent update which
is on the bottom there I use this delta
theta notation here which basically
means this is the thing that I'm going
to add to theta to the parameters
instead of each time writing the next
theta is the previous theta plus
something
I'll just only write that letter that
letter thing the thing that we're going
to add which in this case is just a step
size times the sampled gradient of the
loss that we defined now in this case
this sampled gradient as it is given on
this slide is still not something that
is available because although we've
instantiated the states
I still haven't instantiated the true
value which we don't have I'll return to
this in a later slide where I will
instantiate it with both TD learning and
with Monte Carlo learning but this is
just first talking about what the goal
is and to make it concrete we can talk
about a specific function which is much
easier to show explicitly which will be
a linear function of some features
there's a question
yeah so the question is do we use while
do we assume we're out a policy we all
need assuming that it's fixed as you say
so this is the policy evaluation case so
this doesn't cover policy iteration
doesn't cover control this is just one
thing you might do and it might not be
the only thing you want to do or it
might not be enough for what you want to
do if you want to do control this is not
enough by itself it's a good question
Thanks oh sorry it's a very good
question so theta is the parameters of
the policy are not given here so the
policy is fixed we're assuming it's
basically something that might not even
be under your control there's just some
sequence of mapping from States to
actions theta here refers only to the
parameters of the value function so only
to that mapping from the states to the
value later on and especially in the
next lecture I will talk about
parametric policies where you also have
a function with its own parameters that
will output an action and then these
parameters we somehow have to think
about how to update but I'll defer that
to the next lecture it's good question
Thanks okay so to make this concrete
let's instantiate a function let's pick
a simple one a linear function to do
that first we have to define some
features because typically we don't want
to do a linear function of say the
pixels that come in so in this case
there's just some mapping from the
states to some features and one there's
a few examples here on the slide for
instance if you're if you have a robot
this could be the distance from certain
landmarks let's say sprinkle landmarks
all across some map and then this could
just be a vector with all the numbers
that correspond to the distances from
all these landmarks and if you have
enough of these landmarks you can
basically exactly determine where the
robot is from these features and you
could maybe hope to have some
predictions that also are linear
functions of these features it really
depends on what your prediction is
though really depends on the reward
signal whether that's enough whether
those features are rich enough in that
sense there's some other example things
that have
done in the past such as configuring
somehow picking out peace and palm
configurations and chests rather than
trying to map each configuration to a
specific state value to a specific
cassell in a table you could basically
aggregate many of them together and say
oh if it if your palms are roughly like
this this feature will be um it will be
equal to one and if they're not like
that it will be off it will be equal to
zero
oftentimes these features are by owner
Eenie don't have to be the distances for
instance iron bar in your binary but
many people have done things in the past
where they've basically defined binary
features which give you indicators of
whether certain things are true or not
and then your your way that some of
these things is just your estimate of
the value yeah it is how they used to do
chess ai so I think a lot of chess I'm
not actually that familiar with what was
done in a chess ai but I believe most of
it was actually based more on search and
nearest exert and there was very little
reinforcement learning applied to chess
so these days you can write apply
reinforcement learning to chess
obviously but it turns out especially in
these type of problems it's actually
quite easy to leverage a lot of human
knowledge if you have a lot of human
knowledge it's much easier to start
features that are quite informative but
you could also immediately construct
evaluation functions like oh if you have
this many palms and this many pieces
then you should be better off than the
other one who has fewer and then you can
just search through a big space do smart
search you don't search everywhere and
then you could use use these evaluations
directly so you don't have to construct
features and then learn a value of that
you can combine these things you can
learn these evaluations okay
so now the value function has a fairly
simple form the theta the parameters of
our value function is now just also a
vector with the same size or the same
number of elements as our features so we
have a number per feature and I already
said this a number a number of times on
the previous slide but essentially value
function will just be a weighted sum of
the features so we're multiplying each
of these features these Phi J at state s
with the corresponding number theta J
which is just a real number and this
will then constitute our value function
this is just a simple function in some
sense and this actually helps understand
it better as well so talk about this
function quite a bit in this lecture and
of course our loss function that we had
before we can now Stan she ate by just
replacing the V theta at a more generic
way of writing it down with this
specific choice where we multiply this
theta vector with the feature vector the
Thetas are global they're shared across
all states because we're learning this
one weight vector that you to define our
whole function and then the features
depend on the state now if we would if
we would have labels if we will have
these true values at each states then
this would just amount to regression
linear regression and we could find the
global optimum which means we could find
least square solution and the update
rule for the stochastic grade in the
same case is very simple what I'll do on
this slide and later often is a whenever
I take the feature of the state at s T
just to reduce to the notation a little
bit I'll just say features T so there's
just a Phi with the subscript T which is
the features you see at that point in
time so that means we don't have to
reason about States there's just a lot
of features coming at you and rewards
coming at you and this is where your
algorithm is done based on the update is
very simple it's a step size times the
prediction error where again I'm still
using the true value there so this is
the difference between the true value
and your current estimate
time's the feature vector in the case of
linear functions because it just happens
to be the case that the gradients of a
linear function with respect to your
parameter is just only the feature
vector that's not the case for deep
neural networks or more generally for
linear of or for nonlinear function
proximation a special case of this is we
can still do tabular using this exact
same formulation by considering a
feature vector that's basically one hot
which means that exactly one of the
values is 1 and each time and all the
other values are 0 of our feature vector
and then we just make sure that we have
one such feature for every state which
means that the linear function
proximation case that I just showed you
basically generalizes the tabular case
of course this also merely shows the
problem with this a Buehler case because
there will be many many states in those
in those cases when you have a large
problem that you want to solve which
means that this is a very large vector
potentially so this is just basically to
show you that these things are the same
in some sense or that the linear one
generalizes the tabular one now to give
a concrete example for how could you
potentially build features this is a
very simple example called course coding
and the idea is quite simple that you
have you overlay over it's let's say you
have a two-dimensional space over the
two-dimensional space you overlay some
basically location indicators and in
this case there's not a single feature
that will turn on
it doesn't say oh what is the nearest
thing but it will actually say when
you're near certain locations the
features will be on for those locations
which means you can actually in this
case there might be three features on
because we're near enough three of these
centers of these circles which means
that the combination of these three
features actually tells you fairly
precisely where you are and then we can
use that as a feature representation
just to do the way that some
and things like this have been tried
quite a bit in the past and it tends to
be quite effective but it's also quite
easy to consider failure mode so when
wood is fill well one clear way when it
would fail is when you have a very high
dimensional space if it's a continuous
space but it's not two dimensions but
it's like maybe 100 dimensions or maybe
if you think about pixels if you have a
hundred by 100 pixels that's already
10000 dimensions if each of these can
have a real value then maybe something
like this doesn't scale that well but
for a low dimensional space is sorry low
dimensional problems with low
dimensional state spaces this might be a
very appropriate way to first model it
and then try if you can find a simple
solution that that exploits these
features there's some generalization
here which means that whenever he
updates any of these weights associated
with the features we basically will
update the weight for the whole circle
for each of these circles so we're not
just updating the value within that
small almost triangle in the middle but
in fact we will be updating the value a
little bit for all of the other shaded
regions there as well because we're
updating essentially in this case three
different values corresponding to the
each of these three features and that
means that whenever we change one of
these values the whole value function
within that region gets updated a little
bit and this is nice because it means
that if you end up somewhere which is
close but not exactly at the same point
you will already get a well-defined
value there which is probably fairly
accurate if the true value is fairly
smooth there of course if there's large
discontinuities which sometimes happens
sometimes the true value really has a
sharp edge somewhere it depends how you
space these things whether you can
capture that and it could be that you
generalize over that cliff and that
maybe this might cause problems with
your approximation which is basically
just another way of saying if you have a
linear function of some features it's
probably going to be not that flexible
so you might not be able to capture all
of the rich functions that you might
need to solve a certain problem
[Applause]
also note and I mentioned this before
that when we do something like this
we're actually aggregating states so
even if you only consider like the small
little track triangle in the middle on
the left hand side there there's
multiple states that fall into this
triangle potentially you could end up in
this place near this place multiple
times but you can't actually distinguish
them if these are binary features
especially then all of them will have
the exact same feature representation
all of these all of these situations
where you're near enough which means
they cannot have different values what
this also means is that the problem
becomes non Markovian because the fact
that you cannot tell exactly where you
are means you could potentially also
move exactly not exactly determine what
the next reward distribution or state
distribution would be without taking
into account where you were before and
this is exactly violating the definite
definition of the Markov decision
process and in fact this is the common
case when you do function approximation
because we're going to generalize we're
going to want to generalize because
otherwise learning is in is incredibly
slow but when you generalize this does
mean that you lose the Markov property a
little bit which is something to take
into account to be aware of and you
might want to correct for that in
various ways one way that I'll just
mention here in passing but I'll get
back to in the next lecture is that you
might want to build up an agent state
that is very rich and it has a lot of
information you might want to think
about memory for instance yeah it's a
very good question so how what would
ease what would these features mean if
you for instance you do have a 2d input
but it's not spatial and essentially the
answer to that is it's unclear and it
might be fully inappropriate and this is
also why this slide actually shows three
examples one like on the Left we're
doing not that much generalization in
some sense the the circles are fairly
small which means we're fairly precise
and in that case you might still be okay
because near might still be well-defined
but but maybe you want to do more
general generalization
in the middle plot but maybe section
more appropriate to have something that
is very differently shaped and it might
be need fully inappropriate to
generalize in such a way across your
input space so this ties back to the
question of how do you define your
features and in general you really need
to understand your problem a little bit
in order to even define useful features
you cannot just hope to have a one
solution for all all problems in terms
of feature feature definitions it's a
very good question yeah yeah so why is
it non-markovian
I guess the the situation that may be
clearest is if you think about think
about a robot that can be in many
different rooms and let's say we have
features per room so this is kind of
like this where we have like a location
indicator right but well I'm just making
it very very large-scale in some sense
because rooms we can think of has
basically fairly large things now the
definition of the Markov property is
that you cannot essentially improve your
let me say let me say it more clearly
the distribution of the next reward and
States so the state stabilization
distribution if you will depends on your
current state but if you add previous
states you cannot make it it doesn't
depend of your previous state given your
current state so the shorthand that we
use for that was the the future
conditioned on the presence is
independent of the past so when you have
features saved per room for a robot
that's no longer the case because you
might have transitioned just into that
room on the previous step and if you
only look at the feature right now which
is in the room you might be anywhere in
the room but if you would take into
account that the previous feature was I
was just outside of the room in this
other room this will go if it gave you a
lot more information so the distribution
of your reward might might be quite
different when you add previous
observations in that case
it also has very concrete consequences
because it means that your value
function might be a lot more accurate if
you take into account a few previous
observations because it's much easy to
predict the actual value of being where
you are which might not be anywhere in
the room but it might actually be in the
northwest corner of the room or
something like that and taking previous
observations into a kind might allow you
to basically know that whereas just
looking at the current observation might
not now it's a good question so what is
the underlying dynamics this is saying
she did a question so you could imagine
doing something where where each time a
robot transition somewhere you reset it
to the center of that region say and
then it only then it can transition
somewhere else in that case it would be
Markovian again well that's not
typically the case the underlying
dynamics typically use the actual
location rather than the features that
the robot sees yeah oh yeah sorry this
is just this is not necessarily better
than a grid which is the question in
fact most people use grids this is
basically just to depict that it doesn't
have to be a grid it can be arbitrarily
shaped you could do arbitrary things
here this is actually also fairly often
done as I as I said with the example of
the robot with distances it's a little
bit like that and maybe you could have
instead of just using the distances
themselves as features you could have
thresholds on the distances and then you
get exactly this you could say if you're
within 10 meters of this locator then in
the feature ISM and then you basically
get exactly this and this has been used
as well in the past
okay so now we get back to that question
that was asked before very rightfully so
so we don't actually have the true value
function which we've been using as a
placeholder basically up till now so we
need to construct some targets some
valid target for that well one obvious
choice for that is just to use the full
mark of sorry full Montecarlo return
which actually is an unbiased estimate
for that true value we just run the
policy until termination of an episode
say and then we just take that return
which we've denoted G in the past just
to remind you GT is the reward of the
next step our t plus one plus the
discount reward after that plus the
discount it's double discounted reward
after that and so on all the way until
termination or indefinitely in the
continuing case so as mentioned when we
discuss differences between Montecarlo
and TD in previous lectures I've already
alluded to a problem with that a
potential problem with that is that they
could take very long before you actually
have this return which is one of the
reasons why you might want to prefer to
do something more like TV where we
essentially do the same thing we just
replace this true value with an estimate
and in this case that estimate is the
one step reward plus the discounted
approximated value at the next state so
now you can see V theta showing up in
more than one place it is the thing that
we're updating but it's also used as the
constructed targets for our update this
is very similar to the tabular update
the only difference is now that there's
this gradient there at the end
so we're basically multiplying the step
size with the temporal difference error
in this case or the Monte Carlo error in
the Monte Carlo case and then
multiplying that with your gradients of
the value function with respect to your
current parameters and this will then
give you an update
now to go a little bit more depth of the
Montecarlo version so the return as I
said is an unbiased noisy sample which
means we're almost exactly in the
supervise case we have something that
you could call a data set which is
inputs States say s 0 up to St say say
we have data up to s T right now and for
each of these say we have a Monte Carlo
return so let's assume we've actually
we're in a situation where we actually
have that final Montecarlo return as
well maybe the problem terminated after
time step T maybe it also terminated a
couple of times before them we don't
care but maybe this final a Monte Carlo
return is just as one step reward and s
T is the last state you've seen and then
you can use this training data to do the
normal supervised learning thing which
you've done many times I I assume and
this would be the linear case do you
know their first there's actually the
generic case and then the linear case is
given there where I've instantiated the
gradient of the value function with
respect to the parameters as the current
feature vector which is the case for the
linear function approximation which
gives us linear Montecarlo policy
evaluation this is just regression right
so in the bottom case it's a linear
regression in the top case it could
actually be nonlinear regression this
could still be a neural network if you
want but especially for the linear case
this converges to a local optimum
because it's a it's a complex loss
function and stochastic gradient descent
can find the minimum there and in the
general case it will find a local
optimum in the sorry nonlinear case you
can find a local optimum in the linear
case it can only find the global optimum
because there is only one optimum there
are no local optima but this does also
work for nonlinear functions so you
could plug in a neural network you could
just do regression with that you've
you've done that before
and this in that sense should just work
but we might want to do TD instead where
we instead of using a Monte Carlo return
we use this one-step return and we
bootstrap so we use the one step reward
and we have used this
counted value at the next states and we
plug that in into place of the true
value function which again allows you to
construct a training set and in this
case we construct this we can construct
this anytime immediately after taking a
step because it's immediately available
we don't have to wait indefinitely until
an episode ends and then we can
construction update looks very similar
to the Montecarlo case and this would be
in the the bottom equation it would be
linear because again I've instantiated
the the gradient there with the current
feature vector the top top one is more
generic where I still have the gradient
symbol in there which could also be
applied to neural networks and one thing
I did is the notational thing here of
replacing the TD error with this Delta
which is used often in literature so I
wanted to put it on a few slides as well
so you're familiar with that notation
which means we can write the update very
concisely this is again step size alpha
times the temporal difference error in
this case Delta times the feature Phi I
didn't subscribe subscript the step size
here with a wither T you could also have
a time varying step size in which case
all of these would be subscript it with
T and this gives you valid updates both
of these the Montecarlo case in the
temporal difference case and then we can
talk about where do these things go
where do they converge especially for
the linear case this is quite easy to
look at so this is why we focus on that
one a little bit and it turns out to
converge to this quantity there at the
top there's a small proofer on the slide
which is not as tricky it's something
you've probably seen many times before
because it's essentially just the normal
regression case but because we're using
the return G we can later replace that
again with the true value because it's
in an expectation
it's an unbiased sample so this is
normal regression you come you get the
normal least square solution essentially
that you would expect by doing so
category in the sense another
interesting question is this is
basically unsurprising because it's
normal regression again the question is
does TD find the same solution and it
turns out
does not the solution is slightly
different and the reason is essentially
the fact that we can't reuse this thing
this error all the way to zero and then
the question is so where shoot you
allocate your function approximation
resources and in the Montecarlo case
this only basically depends on your
state distribution so the error there at
the top that we're minimizing the
squared difference between the samples
return and the estimated value at each
of the states this is a random quantity
because of the value and the return but
how well we'll estimate each state value
basically depends on how often we visit
them and/or how often we update them so
if we assume your policy case I didn't
actually put a PI there so I'm not
saying anything about which policy but
let's assume we're doing the on policy
case where there's a whether it's a
policy that is the same one that was
used to generate the return it's also
the one that basically waits which
states we care about then this will
essentially weigh the importance of each
of the states by how often you are there
and it will make your function more
accurate in the states that you are
often compared to states where you are
less often you cannot hope to have a
function that is completely accurate
everywhere so there will be some
trade-off and depending on where you go
more the function will be better and in
the TD case something similar happens
but in addition there's this
bootstrapping that happens we're also
updating towards the guests so at some
point you'll stop updating because your
average gradient of the loss will become
zero at some point you've reached a
local optimum but this is not
necessarily at the same point as the
Montecarlo return which opens a question
which one should you use so typically
the asymptotic Montecarlo return is
preferred because it's essentially the
same thing as the true loss that we care
about for the policy evaluation case the
return is a noisy sample for the true
value but it doesn't actually matter for
the convergence because in the
convergence we can replace that because
it's in an expectation with the true
value which means it ends up in the same
place as if we would have used the true
value and regress towards that there's
just
the case 40d which ends up in a
different place and it can be quite a
bit different depending on the function
approximation that you use used
essentially because we're learning a
guess from a guess which means that the
value that we're bootstrapping on might
indefinitely be a little bit wrong for
the same reasons that the value that
we're loading with Montecarlo is a
little bit wrong we're doing function
proximation we cannot hope to have truly
accurate values everywhere but if we're
using these estimates indefinitely as in
our targets that means that we're going
to estimate something that is a little
bit different from the thing that we
actually care about
so asymptotically you might want to
prefer a monte carlo solution if you
have enough time you might just want to
run that indefinitely long and you'll
find the solution however temporal
difference methods typically converge
faster they learn faster in practice
which is why you might so prefer them
even if they'll go to exactly place you
want they still go to a well-defined
place which is still a good solution
typically and they might reach it apart
quite a bit faster so the trade-off
there might be quite quite good yeah
[Music]
yeah sorry I should I should clarify
that it's a very good question so
previously I discussed something where I
showed that TD found indeed basically
the solution to assuming there's a
Markov model and then solving that
whereas Monte Carlo found the solution
for basically the regression solution
which are difference that was in the
batch case where you basically learn
indefinitely on finite data so the Monte
Carlo sorry the the mark of problem that
TD soules is the one that best fits the
data up to that point and then if you
run over and over all the data again
it'll basically solve for that one in
the limits they go to the same place in
the limit of data all right so that
example was in basically the limit of
time of updates considering fixed data
so that's a need a little bit confusing
so thanks for helping to clarify and
this does mean this is
tights to the question of what why this
did he learn faster because it might
we'll learn something that is actually
more well suited to learn quickly um but
in the limit in this case it actually
ends up somewhere else altogether which
has nothing to do with that building the
Markov model in some sense it's just a
different solution
now I just wanted to there's another
potential source of confusion that I
want to preempt because if you're doing
the bootstrapping there's a lost at the
fact that just dependent now your
parameters and there's a fairly natural
thing you could do there which is just
to take it's in the middle of the slide
there to take a loss that is just your
square TD error and just to take the
gradient of that
so essentially what I've done here to
achieve that loss to get get that loss I
have replaced the true value with the
estimate that we're going to use the
reward plus discounted next state value
before I took the gradient previously I
first took the gradient I still had the
true value there I got my gradients and
then I replaced that true value with
something we can actually have for
instance the one-step TD target or the
Montecarlo return in this case I first
replace it with a target actually had
and then I take the gradients and turns
out that leads to a different algorithm
because if you don't take the gradient
you're also going to take the gradient
of the value at the next state which is
a little bit of an odd thing to do if
you think about it because what does
that mean it means we were in a certain
state st we did a transition we saw a
reward and we end up in a new state and
now we're going to update both of those
values so that the error on this
transition is lower but that kind of
violates causality because why would you
change the value of the next state which
the semantics of which are about the
future to make the updates for this
state better and indeed it turns out
this works a little bit less well in
practice this is called bellman
residuals for historical reasons there
there was a paper by Barrett who in
which he proposes this as maybe a
meaningful way to do things
the update is a little bit similar to
what we saw before but instead of having
the gradient of the sorry in the middle
on the right update the delta theta the
update to our parameters theta is now
step size times temporal difference
error that part is the same times the
gradients of the value function
at st- the discounted value function at
st plus one this is what you would get
if you define your square TD error to be
the loss and you just take the gradients
and this is also I felt it important to
mention this because it's a common
mistake I mean this isn't valid
algorithm it just happens to be
typically worse than TD but it's a
mistake people often make when they
implement these things especially in
current day auto differentiating
software packages like tensor flow so
what I propose to do instead is if you
implement something like this that you
put the targets the reward plus
discounted next eight value in a stop
gradients what that means is when you
define your loss what this means is
basically you're saying this is a proxy
for the real thing
but I don't want you to update this
proxy just to make the value at this
state better because it won't actually
make that value better necessarily in
fact you're changing your value at a
different state so it's kind of a
practical point but there's also a
fundamental point here that this
algorithm is actually doesn't work that
well and that was a little bit of a
surprise to some people because they
expect as well you can just define this
as a loss it seems to be a valid loss
and then we have a true gradient
algorithm on that loss and everything
seems fine but I think the reason for
that is less I just mentioned that
basically violating causality by the
weight a fixed point of these things is
in many cases the same so in the end
you'll find maybe the same solution but
it might take you long
so as I said I'd like to talk about
control and the easiest way to do that
like the small steps that we can take
from what we did just now is just to
replace the value the state value
estimates with state action estimates so
we're essentially just replacing a V
with a Q and then we can do policy
iteration where we do the generalized
policy iteration where we don't consider
fully evaluating policy and we don't
consider maybe fully improving policy so
for instance one thing we could do is we
could approximate the action value
function for the current policy with a
parametric function G sorry Q theta and
then maybe we will just follow an
epsilon greedy policy which is a form of
policy improvement of course there's a
difference here from the previous case
because we're not actually guaranteed if
we do these approximations with
functions that might depend a lot on how
you sample and where you sample and
there's noise and in addition there's
function approximation error
you're not actually guaranteed that your
value function has monotonically
increase or what sorry moment sonically
improved during an evaluation step
that's kind of okay if you in general it
improves you're still good but it's it
does mean that some theoretical results
get invalidated by just doing function
approximation in this case but we could
still probably method and in many cases
it's still completely valid which in
practice means for instance we could do
the linear function again for clarity we
could define state action features now
for instance we could just do the same
thing we did before we could split up a
state space by just discretizing into
two little cells or whatnot
and then maybe we have a feature for
each action which is one way to do this
this is especially useful if you have a
very small action space maybe you can
only move up down left right or
something like that and you're in a two
dimensional space then it's fairly easy
to find such features you could still
hope to have a good function if you have
a linear function of those features this
is so exactly the same as before all I
did is replace States with state action
pairs and V values with
values okay we can apply this
immediately for control and this is
linear salsa which is I I will remind
you the state action value version of
temporal difference learning so will
bootstrap on the value of the next state
and the next action as selected under
your current policy we'll use that as
the value to bootstrap on to construct
our target that's sarsa and then we just
apply that with a linear function in
this case using the course coding that
you seen before and mountain pukar is a
very simple problem that often shows up
in in examples and in older older papers
and it's nice for various reasons one is
that it's very easy to implement it's
fast to run and has some nice properties
for instance the state space can be
considered to be continuous and it
consists of a location where is this car
on the x-axis and the velocity of the
car
we're not encoding say the height
because it's a fixed setting so the
height is actually implicitly given by
the location of the car and then the
goal is to accelerate your car out of
this valley essentially and up to the
goal and the tricky thing about this
domain is that you can't actually if you
just push forward you won't reach it
what you have to do is first push back
so you climb the hill a little bit on
the left hand side and then you push
forward and you use the momentum from
first going down to go up and then reach
the goal and the reason that that's
interesting isn't because it's a
slightly tricky exploration problem
because you first have to go the wrong
way in order to go the right way you
just get a reward when you reach to go
and otherwise maybe you don't get any
rewards so it's also for that reason it
can be a hard exploration problem
because this may be sparse rewards so
you don't even know what you should be
doing but if you explore enough if you
have you have only two actions basic he
left and right sometimes there's three
sometimes there's also the coast action
which basically doesn't apply it in any
acceleration one way or the other but
you can solve this with just
accelerating or accelerating the other
direction which amounts to braking if
you're already
thing so then there would only be two
actions so if you can you can fairly
exhaustively explore if there's only two
actions even if it's a tricky
exploration problem and what these
figures then show is basically how the
the value function evolves if you do
something like course coding in this
two-dimensional space so here the axes
are the position and the velocity and
then the height of this function is the
value at that position and this must be
for one of the actions but I don't know
which one but what's something
interesting is happening around the
later episodes if you look at the
episodes down there a thousand and nine
thousand there's actually quite a bit of
structure there there's a ridge there's
a peak somewhere and the reason for that
is that this value function the optimal
value function actually has something
that is very it's basically
discontinuity because if you have a
certain velocity if you're say close to
the goal you will reach the goal if you
press accelerate further but if your
velocity is a little bit lower at some
point it's too low and there's like a
strict cutoff somewhere where your
velocity can be just a little bit lower
and then you might accelerate but you
won't reach it so then the optimal
policy all of a sudden is to go the
other way and then go back again which
means that there's this Ridge and it
shows up in the value function here and
the only reason this can show up is that
the course coding here is not to coarse
because otherwise we smoothing over this
and it might not learn to represent the
value there accurately whether that's a
problem really depends you could have a
value function that is fairly inaccurate
but it still tells you the right actions
to do because to pick your policy may be
only that you care about is the ordering
of the action values and not having each
action value to be completely accurate
that's another thing to keep in mind
sometimes it's okay if your function is
not that good as long as it represents
the right ordering of the actions or
roughly the right ordering of the
actions and if you have a smoother
function if you generalize more you do
have the opportunity potentially to
learn faster which is a bed of
now this has also been tried by the way
with other ways to discretize as I said
this is a a well well used toy problem
that's been used by many people in many
instances and also cutting this up into
rectangles or little squares rather than
Mugen chorus coding all of that just
works doing it more coarse less coarse
it typically just works but there's very
big differences in how quickly it learns
and how robust your solution is and how
optimally your solution is if you cut it
too coarse the car won't know whether it
can accelerate in a certain region or it
cannot and it might choose to do
something that is maybe safer and in
some cases where it could actually reach
to go it might actually decide to go
first the other way because then it's
sure that it will reach the goal if you
have this very rough function
approximation in this case it's
fine-grained enough to not have to do
that but that's something that also
again in general to keep in mind okay so
this is another view with radial basis
functions which is pretty much the same
thing as the coarse coding and here you
can see this ridge quite clearly which
is a it's an interesting structure to
value function not trivial not something
that I could have predicted immediately
okay
so some convergence considerations so
when do these incremental prediction
algorithms converge for instance when
using bootstrapping I already said it
confers you somewhere else when doing
the linear thing then the Montecarlo but
does it converge always when does it
converge what are the conditions doesn't
matter which function approximation we
use doesn't matter whether we use
function approximation in the tabular
case we know some of these things
converged do these results transfer and
what about off policy learning this
turns out to be an interesting and
important one because that's actually
quite tricky so ideally you want because
algorithms are converging all these
cases but turns out we we kind of happen
these days but it's not that trivial to
do that and this is one of the earlier
examples the example it's not the
simplest one you could think of but it's
for historical reasons I still use this
one here it's still a fairly simple MEP
-
MVP's in MRP and we're just doing
prediction we're trying to predict the
value of this process which is
completely defined by if you're in any
of the states at the top you will
deterministically go to that state at
the bottom if you're in that state at
the bottom you will go to that same
state 99% of the time and 1% of the time
you will terminate and which means
you'll go back to one of those states at
the top and a new episode starts you
always start in one of the states at the
top I think or maybe you start
uniforming in all six of them I don't
think that actually matters for the for
the purpose of this problem there's a
specific function defined here which is
a little bit old it's like it's manually
constructed to have a certain property
which is why it's a little bit old and
essentially there are seven features
here six states but seven features and
in the first state I there at the top
left what you see there is there
something feed at seven which is the
seventh feature sorry it says theta
seven which means that the seventh
feature is equal to one my value
function is a weighted sum of the
features and the weights are theta 1 up
to 7 but the features themselves in this
case the seventh feature is 1 and the
first feature is 2 in that state that's
what it means that's why the value
function is now defined as theta 7 plus
2 times theta 1 just means the feature
vector is 2 0 0 0 0 0 and N 1 I don't
know whether I said the right amount of
zeros there the second state is very
similarly defined but it has a number 2
on the second elements rather than the
first the seventh feature still equal to
1 now you don't have to fully understand
why specifically this but it turns out
that if you construct your function like
this and then you can start that bottom
state to have a 2x position 7 and a 1
position 6 then something weird happens
if you update all of these states
equally so what we'll do is well
essentially take all of these states and
we'll update all of them using this
function approximation at the same
time and then these parameters diverge I
didn't talk about rewards there are none
they're all zero so there's clearly a
solution here which is to put your
feature vector so your parameter vector
equal to zero but if it doesn't start
off at zero this is a log scale they can
oscillate out-of-control and this is
because we're doing the bootstrapping
we're doing TD and we're not updating
the state values in proportion to how
often they would be visited if you would
actually follow this MVP sorry MRP
marker for our process because if you
would actually step through this you
would spend a whole lot more time in the
bottom State then you were would in any
of the top states and turns out if you
would do that
then everything's fine for updating on
policy as it is called but if you change
your state distribution to be off policy
so you're predicting something else
you're predicting the value under
something that you're not following
right now then turns out you can get
these things these these things might
all say it out of control now there's
been lots of follow-up work by the way I
encourage you to try this out of you if
you want just to implement this it's
very simple to implement it's a simple
problem and then you could try it out
that these things actually do get out of
control the paper by bears which I'll
put a resource on Moodle it's point to
that paper he lists like specific
initializations that you could try out
and you should be able to replicate
exactly if you want to try that there
are other simpler examples of this as
well and there's been lots of follow-up
work later for people trying to fix this
trying to find algorithms that are not
linear temporal difference learning but
maybe slight extension slight variations
thereof that don't have this property
that do converge and they do exist there
are ways to do that to have guarantees
converge you with linear temporal
difference learning but it's already
quite tricky to get these things right
let alone if you do normally near
function proximation
that said this is a problem mostly I
would I would argue of theoretical
importance because it turns out if you
do linear time for different learning in
practice
it quite often works really well in
addition if you learn online using the
current policy this problem doesn't
occur it's only a problem because we
weren't updating proportional to how
often the policy would actually end up
there
but it does mean this so as I said
Montecarlo is basically regression which
means that we know what it does
roughly and the check mark here
especially so check mark means it's
converges for certain conditions so the
top row there means multi-car to
converge is when you do tell it's able
lookup well you do a linear function
proximation or when you do nonlinear
function approximation the load linear
function foxman comes first with the
caveat that it's only guaranteed to
converge to a local optimum as typically
the case when you do any nonlinear
regression temporal difference learning
on policy is not guaranteed to converge
the vanilla version of it when you do
nonlinear function proximation that said
it typically does work well in practice
when you go off policy and with this I
mean that you update the states not
proportional to how often the policy
that you're trying to estimate would
visit them then we also lose the
convergence property of temporal
difference learning although as I said
there are more advanced methods that
correct for this and that day they
basically turn that one cross into a
check as well nonlinear is still kind of
all bets are off
so basically in short tabular control
learning algorithms such as q-learning
and sarsa can be extended to function
proximation an example of that is DQ n
DQ Network I'll get back to that in a
moment the theory with function
proximation is not fully developed
although a lots a lots of work has been
done and we understand these things way
better than we did many years ago but it
is additionally another thing that I
wants to call out tracking is often
prefer refers to convergence anyway your
problem might be non stationary there
might be other agents in there that
might be changing there might be other
reasons why you want to prefer tracking
so it's actually unclear that you even
want to converge in the first place so
that's another thing to consider however
it's very hard to reason about what is
the appropriate tracking rate then or
it's a little bit less well-defined
which is why most Theory actually
focuses on the convergence rather than
the tracking which is still important
because what's convergence tells you
even if you don't necessarily care it
also tells you what the algorithm is
doing in the interim where is it headed
right if you know say for linear
temporal difference learning we know
what the fixed point is we know what the
eventual solution is that it would find
if it was run indefinitely long but this
also means that on any step it's roughly
going in that direction and this might
be informed if even if you want to track
and not not run anything to convergence
with tracking I mean for instance using
this fixed step size so that you never
actually hope to converge to a single
solution but instead you hope to track
this the data that comes at you this is
essentially related to the points I made
about the data are not being iid and not
being stationary necessarily in
reinforcement learning so for instance
if your policy changes you want to track
the values rather than trying to
converge to some value for the mixture
whole policies that you've ever seen you
would rather learn the value for the
policy that is right now that that's
what I mean when I say tracking thanks
okay so so far we considered mostly
online updates basically go through your
data and just update whenever you see
something you could also do something
that is a bit more deficient by batching
things as discussed before where we
talked about this more in a theoretical
sense of okay let's see there like a
small batch of data and train this all
the way until the end here I'm talking
about a little bit more general let's
say you just store the data you've seen
so far and you want to use that to be
more data efficient by updating on each
data point maybe more than once so one
way to do that is to have an approximate
value function as before right could be
a poor approximation to start off with
it's just a linear function or a
nonlinear function with some parameters
and we collect some data sets where I
use the generic notation here V hats of
Pi where this could be instantiated with
the TD targets reward plus discounted
next value or it could be instantiated
with the Monte Carlo targets whichever
you want and then the question is which
parameters would best fit this data in a
sense and now of course we could just
apply the same algorithm we did before
but just keep on applying it to this
data maybe we keep on collecting new
data store it into the replay sorry this
this big batch of data in reinforcement
learning terminology is typically called
a replay buffer because we're replaying
past experiences but you could also
consider for understanding just
basically collect this data once and
then keep on replaying on it and see
what you would learn if you do that now
it turns out if you do that if you keep
on learning on this batch of data you
will find the least square solution
which depends on your choice of V hats
and I'll go a little bit let's see yes
I'll go a little bit more concretely
into that but it's important to note so
the experience replay indefinitely like
if you run it indefinitely it's not
really really that surprising this case
is not specific to the linear version
but like the notational split the linear
version but
you just think about this as converging
in the linear case where it actually
finds the least square solution in
linear cases well-defined it's a single
thing right it's the only optimum and
then it will converge that that solution
of course this might take many
iterations not something to maybe do in
practice at least for a fixed data set
but it does make you a lot more simply
efficient because for any data you will
find the solution that best fits that
data up to now and especially if you're
in control this might be quite a
valuable trade-off because then you can
choose your next action really carefully
and if interaction with the world is
expensive and that's the expensive part
of your whole system then you want to be
very careful about which actually
selects so you might be better off
spending lots of compute replaying your
data many many times before you actually
choose a new action now using a linear
value function we can actually solve the
least square solution directly and oh
sorry one thing to point out let me go
back here I called the data there that
curly D which you could think of as a
distribution over data but it's in this
case an empirical distribution it's just
a fixed data set and then I also
subscript my expectations with that D
which basically means there's this
empirical distribution that I'm sampling
from the expectation is not a true
expectation across a continuous
distribution that might happen
it's just conditioned on the data for
this data and I'm using that notation
here again and yeah the expected when
you've converts this means that you're
expected update is zero this basically
just means you're you're there you're
done nothing happens anymore that's
quite simple to reason about but then we
can instantiate that by just thinking
about what that means in this case and
then we can find the solution here with
some simple algebraic manipulation which
looks of a great deal like a Monte Carlo
solution that I've shown before if you
just think of this V hats right now as
using a Monte Carlo return there the
difference is there's no expectations
we're using the actual data before I was
talking about where does Monte Carlo
converge in the long run and then there
were expectations which are under your
current policy where would it go and in
this case we're basically looking at
what
for the data you've seen so far what we
what would be the least square solution
now as you will probably know like
there's an inverse matrix here this is a
summation of outer products of your
feature vectors in this case for the
linear case let's assume that that thing
exists the inverse if you can if you can
compute that it would be cubic to
compute it on the fly right now but if
you build this thing up iteratively
which you can do using something that's
called the shermann morrison update then
you can do this in quadratic time
essentially it is just boils down to the
fact that we can add each of these outer
products one by one in critic time this
is still more expensive than central
difference learning algorithm and it
really depends on how big your feature
vector is whether this is a good thing
to do if your feature vector is saying
in say the hundreds or thousands then
this might be quite feasible if it's in
the millions it becomes quite unwieldy
and you might prefer to do the the TD
case instead of this but the nice thing
about this is that we can incrementally
build up these things both the inverse
matrix and the other parts for each new
data data a sample that comes in and we
basically get the batch solution so we
basically get the answer that you would
get if you would indefinitely keep an
updating on your data set so you don't
have to actually revisit the old data
you don't have to keep on replaying it
in the linear case you could also just
update these things immediately and you
get the exact same solution this only
applies for the linear case again so
this this is good to keep in mind but
then it's quite an efficient algorithm
so we don't know the true values so we
have these estimates V hat that I had on
the previous slide which we can now see
ate in both of these common ways we
could do a Monte Carlo where you could
do TD and we could solve directly for
the fixed point and we call this least
squares Monte Carlo or least squares
temporal difference learning and one
thing that happens is in the New York
Policy case the cross turns into a check
mark for LS TD least squares time for
different learning but there's little -
is there for the nonlinear case because
these things are simply just not defined
for nonlinear case at least in the
vanilla form
now let's move on so now we're going to
step away from the linear case the least
squares TV is efficient in terms of data
slightly more computation expensive but
it doesn't actually scale to nonlinear
functions which you might sometimes need
and the main reason you need them as I
said before us that sometimes is really
hard to construct good features and it
could be much more efficient and much
better not to do that just to rely on
the raw data and to have a function
during its own features essentially or
in other words to use normally near
function of the observations or off your
agent state so many of the ideas that we
talked about so far in this whole
process energy transfer basically
immediately to that case we just plug in
the nonlinear stochastic gradient update
for say temporal difference learning and
it just works essentially not completely
you have to tune things a little bit you
have to pick appropriate step sizes you
have to do the things you normally have
to do in deep learning construct a good
network find a good optimizer things
like that but then the idea is they
transfer so if you do these things
carefully you can get these things to
work so examples of that is temporal
difference earning monte carlo double Q
learning transfers experience replay
transfers very well actually
somebody's don't and I list two examples
here there are more but just to make you
where you see B which we discussed in
the second lecture is a means to explore
very efficiently in multi armed bandits
you could imagine trying to transfer
that algorithm and people have attempted
that to the sequential case one easy way
to do that is to do the Montecarlo
version where you just replace reward
with monte carlo and then you do UCB
with that there's other ways you can
transfer this to supplant your problems
there something called Monte Carlo tree
search where you could use ideas like
this but it's not a natural fit for a
nonlinear function approximation and the
reason is count
you see we kept track of counts of state
visits or state action visits how many
times have you selected a certain action
and if you do function approximation
it's actually hard to come up with these
counts and people have attempted this
and if there's been some progress on
that actually to basically preempt that
somehow Isildur in these counts but the
reason why it's hard is quite intuitive
because what we want is generalization
we want value functions that generalize
well so if you enter a say that you've
never seen before did you get a fairly
good estimate of what the value areas
but how uncertain you are about that
value depends on how often you've been
there according to do you see beeg
formulation which means you have to
count how often you've seen it but then
learning account across a stage space
you basically don't want to generalize
too much you want maybe you want to
generalize a little bit in as far as
these states are similar but it's very
hard to define what it means for them to
be similar and if you do this naively
turns out not to work that well and as a
similar thing so maybe you want to be
more data efficient maybe you want to do
something like an STD I'll remind you
Alice Lee basically souls the replay
setting where you could do you could
think about replaying your experience
indefinitely you can get the same
solution with LSD but you can get it
analytically the analytically part falls
apart when you do nonlinear function
approximation you can't solve this
analytically anymore you don't get the
exact solution immediately out of out of
the box you can still do it iteratively
using experience replay and that turns
out to be in practice much more often
used efficiently with deep reinforcement
learning so here is a concrete example
now for what what would that meant that
mean so one thing we can do is we could
try to do neural cue learning let's call
it so what does that mean we'll have a
network that maps observations to action
values this is sometimes called an
action out Network because we'll
basically just have a network that has
multiple outputs as many as you have
actions
so obviously this only works if you have
a discrete action set next lecture I'll
talk about what you can do when you have
a continuous action set and this
function can then just be a deep neural
network and this has been done for
instance or Atari
I put observation here you could also
put agent state there in this case maybe
you could say let's just consider the
case where the observations are your
agencies and these are the same we could
do an epsilon really exploration policy
where these Q values basically merely
turn it into a policy with Epsilon
greedy exploration so you take the
highest Q value current Q value with one
minus Epsilon and we take a random one
with Epsilon
and this can then lead to an action you
sample from that policy then we define a
loss
sorry I'll use slightly different
notation for the stop gradients here in
the previous slides they used double
brackets here are you single brackets
but it's again the brackets do you know
to stop gradients and it's on the slide
so I'm not that sad about that so we
define a loss and then we can just take
the gradient of that loss heating the
stop gradients which gives us this this
gradients which doesn't actually exactly
give us the update because you still
have to multiply it in with a step size
but instead what we typically do we toss
this gradient into an optimizer say if
you do tensor flow there's actually
predefined optimizers which take a step
size as an arguments or potential other
hyper parameters as arguments and you
could you could toss this gradient into
stochastic gradient descent with a
certain step size then we get exactly
the same thing we had before there's
just alpha times this thing is the
update to your weights but you can also
just take this gradient and plug it into
our mess prop or plug it into atom and
that the same she just works you can
just do that and it'll learn hopefully
if you tune your step sizes so here's a
little bit of pseudocode in tensor flow
there's some abstraction happening here
right there so magic there's this Q net
function that I haven't defined but
you've constructed some Network you
throw some observation and you get an
action value there's another magic
function or epsilon greedy that takes
these action action values and gives you
a sampled action these things are not
hard to implement but they're hidden
from the slide you don't index to get
your your action value for the action
that you've just selected hidden from
the notation it is Q there at the top is
of course for that current observation
so this is like a Q s but not yet for a
and then later on QA is also
when you select the action this is the
thing that we're going to update as it
says in the commentary we're going to
compute Q of s dat and we're going to
want to update that so to do that we
take the action we put it into the
environment we step through the
environment to get a reward discount in
the next observation and because we're
doing Q learning what we'll do is we
take the next observation we'll put it
through our approximation again over
here where we say Q net of the next
observation and then we do a max on that
which intensive first called TF not
reduce max this gives us the maximum
action value in the next state and then
we can build our target by saying
rewards plus discount times stop
gradients that thing max Q next and then
our temporal difference error which I've
here called Delta to construct that you
also have to deduct the current action
value this is our error
we're basically comparing this QA the
value of the state indicate the currents
action in the current state with this
target which is a one step Q learning
target and then we can define a loss in
this case squared loss on the central
difference error divided by two and this
is not a residual algorithm because we
had these stop gradients on the next
state value so this will implement TD
and then you can just toss this loss
into your optimizer it could be
stochastic gradient descent could be
Adam and then it should hopefully turn
now you can extend this slightly by by
using some tricks that were found to be
very useful in deep reinforcement
learning for instance you could make it
more like dqn which is essentially the
same algorithm but it has these two
additional components a replay buffer so
it stores all these transitions
observation action reward next
observation tuples it stores some
summary in a buffer and then instead of
updating on the online data that is
coming in DQ n samples mini-batches from
that buffer this also means that each
sample is typically looked at more than
once in a typical DQ anime
implementation you might look at each
transition four times or eight times or
something like that and then the
additional novelty of the
when is that there's also a target
Network which is typically denoted with
it's the same network but it has a
different parameter which is theta - and
the idea here is to keep the target that
we're going to bootstrap on fixed a
little bit more to make the updates more
stationary more like the supervised case
because we hope that would then improve
stability of the algorithm and it turns
out it does and the way this is
constructed is that we just take the
parameters that you're updating on each
after each mini batch and every one so
often say every 10,000 steps or
something like that you would copy them
over into the target Network but then
you keep them fixed in the interim and
this stabilizes the updates a little bit
and it's turned out to be important
important for performance back then not
all current algorithms use that anymore
for some algorithms it turns out to be
important for stability for other
algorithms turns out to be less
important so not that well understood
when you need is when you don't but it's
something that can help stability so it
can be useful to try which means the
loss function is not defined slightly
differently because now there's two
Thetas I still put to stop gradients
around the value at the next states but
maybe I didn't even have to anymore
because if you think of this thing and
then take up the Grady with respect to
theta there is actually no gradient with
respect to theta - if you just think of
this as an independent thing just for
symmetry and clarity I did put stop
Grady in there again and you could think
about theater - as a function of theta
in a slightly contrived way so maybe
it's a little bit safer to think about
it this way and then we can do exactly
the same as we did before we can toss
that into an optimizer we can minimize
this loss over time which is still non
stationary because the sargan our
network will still change your policy
will still change so it still has some
other benefits sorry some of the
disadvantages of reinforced learning if
you're used to supervised learning but
you can't apply this and it has been
successfully applied to things like
Atari where you couldn't basically just
take this algorithm run it and it can
learn these games of course there's a
lot of hidden magic now right there's
some pre-processing that was done there
some tuning of hyper parameters that is
importance certain discount factors
picks in SR it's typically point 99
which means you have a certain horizon
over which you're trying to act
optimally but given that there's this is
a fairly simple algorithm that can then
be applied to all of these different
video games without needing retuning per
game which was the interesting feat
there so there are some generality that
was achieved which is important the
replay and the target networks which are
the additional bits over what you might
call maybe more vanilla neural cue
learning algorithm they essentially and
one way to think about it is is that
they make the reinforced room setup look
a little bit more like supervised
learning the targets are a little bit
more stationary the sampling is a little
bit more iid if you sample uniformly
random from this replay buffer and that
may be for those reasons helped
stability it's unclear whether they're
vital as it says on the slide and I
mentioned already for the target network
but they helped in that case and one way
to think about this is this is deep
learning aware reinforcement learning
we've changed the update slightly we've
changed the things we're putting into
our updates the data in terms of replay
but also the actual update targets to
the algorithm we're applying we've
changed them a little bit to be to be
aware of the function approximation that
we're using and this helps you could
also do the opposite you could also have
reinforcement learning where deep
learning where maybe you construct a
network in such a way that it's well
geared to do reinforcement learning
which I won't talk about right now but
it's important to maybe think about
these things somewhat holistically think
about the whole system rather than just
trying to plug these things together and
hope for the best but that said if you
do plug them together you can actually
make them work yeah
so why why this is our Gannett we're
help essentially is the question or yeah
so so why do we need them it's unclear
that we do actually so more recent
algorithms sometimes don't use target
networks and they work just as well
without depending on what you do so the
the intuition behind it was that it just
makes the updates a little bit less
non-stationary because if you don't have
a target network each time you update
your network the values will change
everywhere and one observation that Vlad
who originally worked on the QM one
observation that he had was actually if
you make these networks bigger you
increase their capacity the target
network becomes less important and one
reason for that is probably or at least
this is one hypothesis if you have a
fairly small network then whenever you
update a state action pair value you
will basically update many values there
will be a lot of generalization which
means you're additionally changing the
targets which means you're tracking
something that is moving quite wildly
and this might be hard if you have a
bigger network you have more capacity
it's as if you're doing course course
coding or something but you've made it
more fine-grained which brings it closer
to the tabular case in a sense which
means that if you update this value that
you're trying to updates the target file
you might not change that much the value
at the next states because the network
has more capacity it generalizes maybe
less and that if you've generalized less
maybe this target networks become less
necessary and maybe you become less
important so that's one intuition but
the truth is we kind of really don't
know exactly why and how they help but
yeah sorry yeah typically what is done
is that these networks are exactly the
same except for the weights so the
weights in the target network are just
an old copy of the weights of your
online network and otherwise they're
exactly the same so every once in a
while they will be exactly the same
network because you've just copied the
parameters for your online networking
Zargon Network at which points they're
exactly the same but then you just keep
the target Network fixed and the online
network keeps updating and keeps
changing and then sometime later the
parameter values are not somewhere else
you just copy it over again and you make
them the same again
this is just one thing you could do not
necessarily the best or the only thing
but that's how it was implemented in dqn
okay so one thing I quickly wanted to
touch upon before we do run out of time
is to unify Monte Carlo in central
difference learning and this is
important as well when doing function
approximation for multiple regions
actually so the first one on the slide
is that when we bootstrap updates might
use old estimates which means that
information in a sense propagates slowly
I'll show you an example of that is even
holds in the tabular case in Monte Carlo
the information progresses propagates
much faster because the first time you
see reward if you just think of that
case the first time you ever see a
reward in a Monte Carlo case you might
be updating many state values that came
before all of them within the current
episode will be updated a little bit
towards that reward now when the episode
ends in a temporal difference learning
case if you do TD with a single step you
actually only update this value exactly
before getting that reward and the other
ones they were already updated they're
using the old values so they don't know
about this reward yet it's only when you
return back to a state and then that one
transitions to that say you've just
updated that the information can
propagate back further maybe let me
quickly show you the picture that
corresponds to that this is a trajectory
that you've taken on the left hand side
there where there was some winding path
that in the end ended up at the goal
let's say the agent didn't know where
the goal was so it just did some random
stuff and then the action values has
updated by one step Saoirse will only
update the action going up straight into
the goal the action before that that led
to that state has already been updated
using the previous estimates now if
instead you would propagate you would do
Monte Carlo you would propagate the
information all the way back to all the
is that you've did this might also not
be fully appropriate because you might
be assigning credits for this reward for
this reaching this goal to action say
that he stepped away now there's an
intermediate thing that you can do which
is n step returns and one way to depict
that is is this where temporal
difference learning just takes one step
then bootstraps Montecarlo takes all the
steps until the termination and uses
that but of course you could do
intermediate things where you just take
two steps and then your boot trip or you
take three and then you bootstrap and
you could do any of those so you could
make this a parameter and the updates
then look like this where and in this
one it's the on the top that's just CD
so we can now basically augment our
notation by putting a little one in
brackets over the return which means
we're doing a one step return in every
bootstrapping and then the infinite step
return is equal to Monte Carlo we can
just use this as a targets everything
else stays exactly the same
so it's in between Monte Carlo in
temporal difference learning where we're
just picking a certain number of how
many steps do we do and in this example
here on the right hand side this is the
update that you got for ten steps our
sir which doesn't propagate the
information all the way back to the
beginning but that might also be
inappropriate because some of these
actions earlier on they stepped up where
they shouldn't have and maybe you don't
want to propagate the information that
you've ended up in to go all the way
there because you don't want to
encourage these actions so this gives
you a way to go a little bit
intermediate and update the recent
actions to give them credit for the
reward
this is called the credit assignment
problem that this is referring to to
give them credit for the rewarded you
eventually obtained but the actions that
were too far in the past you don't give
them credit immediately if then they
later on turn out to go to good places
they will still learn right as TD did
but they won't immediately get credit
for everything this also makes the
update less noisy than for Monte Carlo
clearly the least variance you get for
TD and then the most varies you get from
Carlo these things are in between in
terms of bias and in terms of variance
this is an example where there's a
random walk you've seen the random walk
before the only difference is now
there's nineteen states instead of five
the picture depicts five but just
imagine this thing as being bigger
with 19 states and there's a reward of
one at one end and there's a reward of
zero at the other end the policy is just
moving randomly and we're just trying to
predict what the values for each of
these states and then there's these very
colorful plot down there what this shows
is you can basically just look at the
left hand side here you could look for
more diesel in the book this is on
umberto but what this shows here is
essentially four different step sizes it
shows you the curve for the error in
your prediction for each of these n
steps so the red line which is the one
over here is 4 td and is 1 and it shows
that you can use a fairly high step size
if you do that because the variance is
low but the best you can do in terms of
error for a certain amount of data this
is for the first 10 episodes of doing
this for a certain amount of data the
best you can do is not brilliant in
terms of error if you use a 2 step or a
4 step which you're down here you have
to tune your step size appropriately but
then you can get much lower error so
this shows there's some intermediate
gain here if you use two rollers they're
too far those are all the way bunched up
at the top there that's getting close to
Montecarlo again then you basically have
too much variance and you'll suffer from
that so the error will be fairly high
again just because of the variance yeah
yeah
this is prediction error sorry yeah this
is not control this is just prediction
error there isn't even control the
policy's always uniformly random in
terms of going left and right thanks ok
so it's almost 4 so we have to wrap up
this is just me stating again that these
have the benefits of both temporal
difference learning in Monte Carlo
I only have two more slices you can look
at at your leisure one I'll actually get
back to the next lecture so don't worry
too much about this I'll actually
basically somewhere in the beginning of
next lectures explain this one again I
just wanted to flash it up because it
uses multi-step returns and the other
thing is just basically saying there's
lots of research here this is an example
of combining many different advances
recent advances in
reinforcement learning and show that
these things can actually all help
together and we get much faster learning
these days than we did only a couple of
years ago so the research field is
moving quite fast and there's lots of
interesting stuff happening but there's
still many unsolved questions as well
and that was basically I had so thank
you