Reinforcement Learning 7: Planning and Models

today I wanted to talk
models but in the context of learning
maybe I should have put it in the title
but just this quickly to recap what
we've been doing last lecture we learned
policies directly from experience using
policy gradients methods and such before
that we talked a lot about learning
value function and in each of these
cases I said I'll not talk so much about
models in this lecture deliberately
because this is like a whole rich field
you know in and of itself and in
addition these things can look at in
isolation so then it's good to separate
them and to look at the components to
try to understand each of the components
separately before going all the way into
the complete breadth of what you could
imagine so in this lecture will
basically revisit models and
specifically we will talk about
basically two again kind of separable
things one is where you don't have a
model you have to learn this you have to
use this somehow and the other one is
where you want to plan with the model
and these are related but there are
difference in the sense that you could
also imagine just having a model and in
fact when you have a model you want to
do plan and we've already seen in the
case of that in the dynamic programming
lecture where we were doing dynamic
programming as a way to do planning
given a model and that's a valid way to
do planning of course other ways to do
planning and we'll discuss some today I
will not be able to cover the full width
of the field of planning because it's a
very rich field one thing to keep in
mind though is that a lot of the classic
planning algorithms they have basically
been built for the assumption that you
have true model and if you don't then
these algorithms might not be the right
choice and I'll discuss why that's the
case this is just refreshing the
terminology so we've talked a lot about
mobile free reinforcement learning this
covers both the value-based reinforce
building and the direct policy search
reinforcement learning and the extra
critical gur if we do where the idea is
to learn a value function or repose
from experience directly and the
contrast with that what is sometimes
called model-based reinforcement
learning is the setting where you you
don't have a model again but you're
going to learn it from experience or
you're given a model and then you plan a
value function or a policy from that
model now the terminology is a little
bit there's a little bit of a gray zone
in the literature if you search for say
model-based reinforcement learning and
you look at the papers that you find
some papers will strictly look at
basically the first case there where
there's no model but you have to learn
it and they'll say this is model-based
reinforcement learning by definition
this is motivation reinforced learning
whereas other papers might allow there
to be a model maybe even a true model or
an approximate model so the terminology
is a little bit ambiguous in that sense
but it's still useful distinction to
have a short shorthand that you can
refer to whether or not you're building
an explicit model and this is again
recapping what we've seen so far but
it's useful to have this picture in mind
so yes very good question so yeah when I
say here you learn a model or you're
given a model I essentially mean or the
best way to think about it for now is
that you're given an MDP that you can
Fuli inspect so you can look at the
state transition probabilities you can
look at the rewards that's the simplest
thing to keep in mind although in on
later slides you'll see exceptions to
that for instance you might not be able
to inspect the MVP you might just be
able to sample from it in that case it's
more like a black box that you can query
but you can't actually see the actual
probabilities it just gives you a random
sample we would still in some cases call
it a model in other cases we would just
call it a simulator and in yet other
cases we would just call that the
environment depending on whether you
care about that thing or whether you
care about something else that that
thing is trying to mimic so the word
model is a bit overloaded it's actually
very overloaded we're planning as well
but unless I say otherwise
when I say model you could think of some
MVP which is trying to model the problem
that you're actually interested in it's
a good question thanks yeah so yeah so
the question is if you would have a
through simulator inside say you're
doing Atari but you have the Atari
simulator inside the agents then that
would constitute in the sense of which
we're talking about here that would
constitute a model in that case it would
even be a true model it would map
exactly to the thing that you're trying
to solve in which case you never have to
collect real experience again you can
just do all the thinking in your head do
the panning if you will and this again
shows that there's a little bit of a
gray area here because then what is the
distinction well it depends what you use
for planning you could just query the
simulator inside your head and you could
still use a model free algorithm to
learn from that which means we're using
for instance Q learning to plan which is
a valid thing to do is that that model
based model free it kind of depends
where you draw the boundary between the
agent and the environment so some of
these things are definitely ambiguous
and that's okay right but it's good to
keep in mind that they might be so did
this thinking sort or aren't or was that
clear it's somewhat clearer when you
think about the case where there's an
external environment that you really
don't know and you're going to construct
a model inside and you're basically
going to accept that this thing is going
to be approximate it's going to be not
the true MVP but then you're going to
try to use this somehow so now there's
an explicit representation of the the
outside world of the the environment
that you're trying to solve and you're
trying to use that in some way to come
up with a good policy
okay so just this is just a general
overview of or at least one way to look
at reinforcement learning as a whole so
more or less where we started all the
way at the beginning was at the top left
corner where we talked about dynamic
programming as a way to solve Markov
decision processes
and specifically we talked about these
one-step updates where you basically Pam
one step into the future and if you do
this for the true value if you just
write down the true value of the states
as a function of the one step in your
model in the true MVP in that case into
the value at the next states then this
constitutes the bellman equation so
that's just the definition of what the
value is and then we talked about that
it's also possible that to define the
optimal value in that way and then we
talked about using that equation the
bellman equations to basically define
updates where are you instead of
plugging in the true value function
which you don't know you plug in an
estimate and n turns out if you keep on
doing these one-step rollouts and using
your estimate you'll converge to the
true values that was dynamic programming
then we talked about basically the
bottom left corridor temporal difference
learning which is basically a sample
based version of dynamic programming we
sample a one step and then turns out if
you do that sufficiently often you can
still get the same answers that dynamic
programming programming gives you but
you don't need a model we also talked
about Monte Carlo learning on the bottom
writes where you do something very
similar you sample but instead of using
the estimate the state value at the next
state you're going to sample a whole
trajectory say for instance for a given
policy and you use that to approximate
your value what we didn't talk about
that much is exhaustive search because
it basically is too big so that will be
the version where you use your true
model to basically plan through the
whole tree but you go all the way to the
end so you're not using any value
estimates along the way but of course
you go don't have to go all the way to
the end and this is a little bit of a
maybe a strawman and a lot of classical
planning algorithms actually work a
little bit closer to that space where we
are constructing at least part of the
tree maybe not just a single step and we
might go a couple of steps deep and then
we still might use a value function
somewhere along the way which means
we're maybe somewhere along the center
of this thing
especially if you also roll in sampling
and I'll talk about how to roll in
sampling when
do that so you could imagine combining a
model and a sampling based approach and
this can be actually very useful so just
to remind ourselves the difference
between top and bottom here is whether
you use the model or whether you sample
and one thing that I'm saying is you can
go in between you can use your model a
little bit a little bit and then still
sample the other dimension is whether
you use a one-step look-ahead or whether
you use full look ahead and we actually
talked about this quite a bit when I
said you could do multi-step updates
where you do a few steps and then you
bootstrap and you could even have
mixtures I didn't talk about that that
much but there's this lab the parameter
for their bottom which basically says
you might bootstrap a little bit you
might look a little bit at the e value
evaluates vation
at the certain states but you might also
still roll forward from the state and
then mix the sample based value from the
learned value at that state so this is
the full space and now we're going to
try to go a little bit a summary within
that space to find algorithms that might
usefully combine properties of these
extremes and so first to remind
ourselves this was the classic example
that I've shown many many times the
agent takes the action the environment
sends back to the observation there's a
reward somewhere which you could think
of as coming from the environment if you
please so the difference now is that
we're going to basically inject a model
in there so you're still going to act
but the experience now goes into the
model rather than into the agent
directly and then we use the model to
inform our value or a policy so there's
this indirection now you could still
have the direct loop so basically this
loop here on the top rat right it's
basically the same loop as you've seen
before but now there's this additional
loop which basically sits inside the
agents if you want where you build an
additional representation of the world
out there and you use that to
additionally augment your your value or
your policy now intuitively this already
sounds like it could be a good idea
because by trying to learn to model the
outside world we might just be learning
more about what's happening there if for
instance you're
if your values are fairly non
informative about everything that might
be happening you might be learning
specific values but it might be hard to
generalize when you go to a new
situation and you don't understand the
rules so to give give you a somewhat
abstract example you could imagine if
you go through a completely new
situation where your vision is
difference your value function might be
completely wrong but if you know the
laws of physics you might still be able
to figure out that if you toss a ball in
the air it will drop down even if you've
never seen a ball of that color of the
error of that size and so on so it does
make intuitive sense that you sometimes
want to learn a little bit more than
just the policy or just a value function
and maybe that more is then is then or
could then be a policy sorry a model
okay so that's the high level now I
wanted to make it more concrete and will
just be very explicit about what a model
is model is in this case a Markov
decision process it doesn't have to be
but this is how we're going to define it
for now and to keep things simple I'm
going to keep the state space in the
action space the same so the states and
the actions are the same as those from
the environments that's not necessarily
the case but it's just for the
simplicity of the exposition here that
I'm going to stick to that case which
means that the model is not fully
characterized by the transition function
and the rewards and these is that's what
we're going to learn this little P you
can see it in equation there this is the
same equation we've had before when we
were just talking about Markov decision
processes in general except that there's
now a little hats on the P and there's
this ETA subscript the each are the
parameters of your model because we're
going to typically consider parametric
models it doesn't have to be could also
be a non parametric model in a sense in
which case maybe the ETA is replaced by
data instead of parameters you have data
that you're that you're using but we
have some approximate model and the idea
is that this model somehow approximates
the true model I'm already going to call
out here that this is only one thing you
could do you could imagine that the true
model is too complex and sometimes it
might be useful to consider
that's not actually the true model but
something that is still useful but I
won't go into too much depth in that but
that's an interesting research direction
so we're going to assume that the states
and axis are the same which means that
we can basically just learn these things
and we could also think about learning
them separately which might be simpler
in some cases where you basically try to
learn for each state action what is the
expected reward and for similarly for
each state in action what is the next
state yeah so in a great world what will
be the approximation so yes we don't
know the probabilities so the question
is how do you then maybe maybe there's
multiple questions one is how do you
learn this and it's good to have a
concrete example so we could say for
instance in a great world what would
what would what would be the goal of
this learning this model let's take the
simplest example at a great world it's
completely deterministic which means
that when you for instance press up in a
certain cell in your great worlds you'll
deterministic we go to the cell above it
then what you're trying to learn here is
that the next States for any state where
you press up is the state above it and
maybe additionally you want to learn
that whenever you try to do that when
you're just below a wall then it's
actually the same state again you could
imagine learning this quite simply in a
simple small great world of course if
it's very complex domain you want to
generalize somehow so maybe this this
model is then something that generalized
as well so choose two deep neural
network so that you could also query it
for States and axis that you've never
actually seen and you hope to still get
a relatively reliably right reliable
answer through generalization and
actually here I'm going to make it a
little more concrete so we're just going
to collect some experience this could be
from multiple episodes so I'm not making
distinctions here between where the
episodes at end or in not end of course
you need to be careful about that if you
want to implement something
yes and then we're just going to note
that this experience can basically be
transformed into a dataset where you
have a state and an action as an input
and as the output you have the reward in
the next say that you've actually seen
in that situation on that time step and
one thing that you can do is then to
learn a function that basically tells
you what the expected reward and the
expected next state is for a given state
in action now you can pick a loss
function maybe you could have separate
loss functions for the reward or the
state or you could merge these somehow
you could also have separate functions
for the reward in the states each of
which has their own loss function and
for instance maybe you have something
like a mean squared error whether that
makes sense depends a lot on the
specifics of the of the domain for
instance if you think of about the Atari
games you can question whether it makes
a lot of sense to put a loss on the
pixels of one frame versus the pixels of
another frame whether you want to do
this pixel by pixel or whether it makes
more sense to maybe try to find some
features of the screen and then try to
define a loss in that space that's
non-trivial in general and I won't go
into much detail on how you should pick
that loss because it really does depend
but it's something to keep in mind when
you when you want to construct something
like this but whatever loss function you
pick for this model you then minimize
the parameters of the model so that you
you map you you find this function that
map's these inputs input States and
actions to these outputs so in this case
if you do it like that where for
instance you pick a mean square error
then the output would be an expectation
model that's fine in a sense but there
might be certain things that are that
are limitations of this before I go
there let me just quickly stop here to
see if everybody's on board if you have
any questions please PLEASE interrupt
okay otherwise I'll continue to the
expectation models because they might
have certain disadvantages and one way
to see that intuitively is you could
imagine that there's an action and it
randomly transitions to one of two
states in one state you went to the left
of a wall and the other you went to the
right of a wall so note I'm not talking
about the action going left or going
right there's this one action that you
do you you you give a certain decision
to say or your motors and it just so
happens to me the case and then fifty
percent of the time you end up to the
left if fifty percent of the time you
end up to a right of a wall maybe
because there's additional safety
controllers or whatnot that basically
prevent you from going into the wall
that you don't have control over that
are basically as far as the learning
algorithm is concerned they're part of
the environment but they are there and
they do do result in you're not hitting
the wall so you can imagine then if
you're depending on the loss you have in
your next state that the expected next
state might be exactly in the wall or
against the wall which is a state you
might not actually end up in so if you
think about the great world maybe that's
an easier example to think of if your
expectation if you have random
transitions on these grid on these grids
and let's say that if you press up you
don't actually go up ever but you go
diagonally left up or you go to Angley
right up you could imagine that the
dynamics work like that and let's say
that you're just below a grid which in
which there's a there's a hole then
pressing up would mean in all cases you
would never end up in the hole but if
your loss is such that you try to
interpolate between your position you
would end up exactly in the hole which
is a state that you never actually end
up in and the reason is that we're
discussing expectation models here and
the expected state might not actually be
a state that you could that would ever
happen in reality now so this might seem
that this is a problem for expectation
models and it could be but for linear
models and values this turns out not to
be so much of a problem and you could
take this in two ways one you could take
this as as meaning that we should be
doing linear values and models where
maybe the features are themselves
nonlinear functions of say your
observations but then you have a linear
model on top of that linear value or you
could still take this to be just a
limitation of the expected models and it
really depends on what you want to do in
terms of
creet algorithms but just to show you
that in a linear case is okay so what
I've done here is I've basically
replaced the states with features so we
assume for now we have some fixed
feature function that takes a state as
input and in turns that into a vector
Phi we've done that before as well so
there's a search and here's where we're
a 5 subscript T means this is your
features at States s T and then of
course 5 T plus 1 is the features at
States T plus 1 now if we would build an
expectation model on these features
where again the interpolation may or may
not make sense in terms of the actual
features you might see then we might
note if we have a linear model that this
is basically just a matrix
multiplication we take the features Phi
T we multiply a matrix P that we're
going to learn potentially and the
output of this to semantical said is
that this is an approximation to the
expectation of the features at the next
state for now we can even assume that
this is exact you've learned it's
modeler their orders model is given to
you you have this linear model and it
actually gives you the expected next
features just for simplicity and let's
assume that we also have a linear value
function which is parametrized with some
parameter vector theta which we might be
learning which means that the value at s
T is the dot product of this vector
theta with the features at SS T so
features subscript T then we can just
write it out and we could talk about
what is then the expectation of the
value of the state after a couple of
steps or even after one step but I did
it here for after n steps and we can
just write that out because we know
going n steps into the future with our
model would mean we just apply this
matrix P multiple times but you can go
through these steps yourself one by one
if you once but the main point that
we're using here is that the expectation
commutes with the linear operation of
the matrix so what essentially happens
all the way at the bottom is that we're
able to push the expectation all the way
through inside and in
stead of talking about the expected
value of some states in the future we
can talk about the value of the expected
state in a sense so that's what we've
done here actually the notation there at
the end is a little bit that should have
been expected states not the expected
feature now because I I gave States this
input to the value so for linear values
and linear models because the
expectation commutes with linear
operations basically expectation models
are fine but it's might not hold in
general when say the value function is
nonlinear you cannot push this
expectation through through the value in
which case these things are not
necessarily the same and typically we
are interested in the expectation of the
value of a state after multiple steps
but we don't have it because the only
thing their model can compute is the
expected state and then applying the
value to the expected state is no longer
necessarily the same as the expectation
of the value at that state so is there
an alternative yes first question sorry
so concrete example so that's a little
bit harder - harder to do on the fly I
mean the examples that I gave don't all
go through because in that case you
actually want to have something that's
nonlinear I brother had just think about
that a little bit and then give an
answer later for a concrete example
first let's discuss the alternative so
what what else could you do we might not
want to assume that everything is linear
and in that case the expectation model
might not be sufficient because it gives
you expected States that might not
actually be real states and then it
might not make sense to for instance
feed those expected states back into
your model which expects real states to
iterate on and it might not make sense
to feed is expected States into a value
function which also might be trained
only to give you valid answers for real
States so what else can we then do well
one potential possibility is to build in
a stochastic model something that is
trying to basically output rewards and
next states that are valid reward the
next states in a sense but it's not
committing to exactly giving you just
one it's it basically gives you the
whole distribution and one way to do
that is to use a stochastic model or a
generative model lists are also called
and how I noted that here is that
there's this predicted next reward and
next states which are our samples and
the input to the model is Omega which is
a noise term which means that if you
would query this model again you would
resample the noise term and you might
get a different sample for the reward in
the next state now these this reward in
this next state they will have variance
right because you'll Sam if you'll
sample them again you might get
different answers but the idea is that
each of these next days are actually
valid next states that you could put
into your model again and thereby you
could create a whole trajectory which is
basically very similar now to a Monte
Carlo rollout in the real environment
but instead of using the real
environment and carrying that to give
you random States you're carrying your
stochastic model yes in this case yes so
P hats now here is a function this is
that's a there's a good question and
it's can be a function doesn't have to
be a probability distribution because I
have this noise term that I also
explicitly give as an input and
therefore the output of the function can
still be noisy
it's not it's not a yeah it's not a
sample from its an equals but I shoot
probably subscripts Omega also with some
index because you're going to re sample
that thing every time you query this not
necessarily once for every time step you
could imagine querying it multiple times
at the same time step but you might put
in different or you would put in
different noise terms for every time you
sample it typically when you actually
implement this on a computer typically
you would just put uniform noise in
there and somehow internally this gets
transformed to the right noise that you
need to have your generative model yes
so what this could just be equivalent to
the linear model so this is already in
some sense even the types don't really
match because this one does take this
noise term and depends on the noise term
where the linear model just gives you
one answer every time you query it gives
you the same answer so in that sense
it's already not the same there of
course edge cases in which you can make
these things to be basically equivalent
one version would be where you basically
don't you just ignore the noise term and
maybe your loss function still makes it
so that then the output will be the
expected state and if your model itself
is linear then as well then you just
have your linear model back so
definitely in some case you could say
that linear expectation model is a
special case of this one but this one's
more general because it can use the
noise to basically try to match the
whole distribution of next States rather
than just trying to output one specific
next state the example I already
mentioned just was just sort of
reiterate that sorry the advantage is
this is that you can chain these the
outputs sampled States is a valid input
for your model again so you could create
a whole trajectory instead of just
having this single single output to
expect it next say that you don't then
cannot really put into your model again
except if it's linear however there is
noise introduced which might be a
downside in some cases because it might
mean that your samples have
have more variants so one consequence of
this is if this is a fairly expensive
function your your outputs might have
high variance if it's a cheap function
you might be able to mitigate that by
you're sampling multiple times and then
you create maybe a whole tree of outputs
yeah so the noise term there's a good
question so where does it come from how
is it used so this is basically just a
way to turn something that you want to
be stochastic into a proper function
it's it's just now a function of its
inputs it has nothing to do with
expiration or anything else it is just
trying to match the dynamics but it's
trying to match the dynamics in terms of
the full distribution rather than just
trying to match the expectation of the
next next state so if you would sample
this yeah so this map this this thing
does model indeed a probability
distribution yes and the idea would that
meet it if you sample this thing over
and over again and this is how also how
you would typically then construct your
loss to for train to train this thing if
you would sample it over and over you
would like the the outputs the rewards
and the next states that the outputs
outputs to match the distribution of
rewards the next states that you would
get from that state in action so the
simplest example is perhaps when you
stayed in the action are fixed let's
just take a very simple example where
you're basically in a one-armed bandit
where there's only one state one action
and there is no next state so we're just
looking at the reward now this is maybe
the simplest example you could think of
the reward itself might still be noisy
and now you could learn an expectation
model which would be the expected reward
but you could also try to learn it my
friends would be a Bernoulli distributed
as a reward the other thing you might do
is instead learn a function that
actually randomly samples that Bernoulli
with this
same probability as the actual reward
get sample in that case it might not be
particularly useful some actual the the
complete distribution compared to the
expected reward but especially if you
want to apply this is a sequential case
where you want to sample these next
states then it might become more useful
because then you can sample trajectories
that are plausible which were which is
not necessarily the case if you have
these expected states in the middle
because these expected states might not
actually be they might not correspond to
any real states that you might encounter
yeah so the question is whether the
noise term makes it easier to learn
function I would say we can that's a
separable question in a sense here I
just wrote it down with the noise term
because then you can write it down as an
explicit function if you have a
different way to build a function that
maps or something that maps through this
distribution so from a state and action
maps to a probability distribution and
then maybe you have a separate process
that that samples from a distribution
that's also fine and in fact these
things might be depending on how you
look at it might be equivalents
depending on how you draw the box which
is then labeled the P hat so this thing
now captures both the mapping a to
distribution and the sampling where the
sampling is regulated by this Omega term
but you could of course separate these
you could have a separate component
which tries just to Maps map to the
distribution and it doesn't have any
sampling it just tells you the complete
distribution maybe some statistics of a
parametric distribution like a Gaussian
or something like that and then you
could separately sample from that in
fact if you had a work in the software
typically is that the sampling itself
then still generates this Omega and
thereby constructs the actual sample
that you get there subdued a random
number generator and somewhere in there
which generates this Omega but here I
just made it explicit in the year inputs
okay now there is a third alternative
which is related to the previous one but
it's a little bit more explicit in the
sense that we're going to learn the full
transition dynamics including so casa
City this is different from the previous
one because in the previous one we
basically said we have this box which
takes a state and an action and
implicitly or explicitly takes a noise
term and it gives you a sample in this
case we're not going to take that next
take that second step of sampling we're
just going to stop basically at the
point where we have the probability
distribution so for any state in action
you need to supply an additional
argument which is the next state and
then it will tell you how likely let
next stasis or alternatively maybe it
could just give you the full
distribution like I said in some
parametric way but in any case we'll
have something which which you can write
down as in this top equation as the P
hats it is a function in the sense of
the state action and next States and it
this chute map to a a probability and in
addition this should sum to one over the
next state it should be valid
probability distribution of course if
you have an approximate model anyway
maybe that's actually something that you
could violate it's unclear whether you'd
want to but it's maybe something to keep
in mind but if you have that so in this
case I split the probability of the next
state from the reward model you don't
have to do that you could also try to
learn the joint model in one go but if
you have those you could basically plan
all the way ahead one way to do that is
to do something like dynamic programming
which is more akin to the first equation
here or if you want to know the value of
a state many steps in the future you can
actually unroll this thing again and
again and again which effectively builds
a big tree where the branching factor of
the tree depends on how many actions you
have but also how many next states you
have to consider so this could be a very
big tree so it can be hard to iterate
this explicitly but they can still be
useful and
indeed if your number of states and
action isn't too big you could at least
do something like dynamic programming
where maybe you each time you just take
one step in the tree and then you back
up using your approximate model so what
I'm saying is actually something that
it's maybe quite obvious in hindsight
what we've done here is we've basically
just approximated the transition
probabilities for the states and the
rewards and now here we're just assuming
that these are the true MVP and then we
use them with our standard dynamic
programming methods for instance to
approximate values so we had these three
let me just stop on this light we had
these three ways to do models now we
could do the expectation model we could
do the the the sampling model stochastic
model or also called generative model or
we could try to learn foam well fool
model and all of these cases are valid
cases in some sense and some in some
cases one might be more appropriate than
the other one might be easier to learn
than the other but then we still of
course need to use them somehow to plan
yeah yeah you do fool models for
continuous state spaces it's the
question yeah you could but of course
these sums who turn into turn into
integrals so then you bump into the
issue of having to evaluate these
integrals there's ways to do that like
MCMC or other methods you do have to
keep in mind then that these are going
to be approximations to the full
integral in general of course in certain
specific cases you might actually be
able to analytically evaluate the
integral which might be better but even
if you do that you should keep in mind
that you're still analytically and
therefore precisely evaluating something
that depends on an approximation so
whether that's then so much better is
unclear so then there's an alternative
which is quite obvious maybe in
hindsight which is okay you could just
sample this thing which is actually what
MCMC does to evaluate the integral and
in some sense then we're back to the
stochastic models again so we have this
probability distribution but then in
order to use it we're going to sample
from it which is a need a valid thing to
do
and this means that we've basically went
back and we learned the full model
internally but we did go back and do the
stochastic model where we've sampling
from this model and then sometimes it's
actually easier to sidestep this this
intermediate step of trying to learn
full model and just try to learn a
generative model immediately with just
having a loss on the samples rather than
having an internal explicit
representation of the probability
distribution depends similarly by the
way you can extend all of this to
continues actions in a similar way where
you could imagine that there's a
continuous action space but there might
still be a probability distribution that
might be relatively straightforward to
learn for instance there might be a
gaussian that you can put around maybe
your policies a gaussian which case you
only need a few parameters to
approximate this thing and that's of
course if your action is very
high-dimensional but that's that that's
typically just a hard problem in any
case okay now I want to talk a little
bit more more about how you can then
represent these well I've mentioned a
few of these already but let's look a
little bit more detail in in for
instance whether you use a table lookup
which is of course only useful or
possible in small problems where they
use linear models or whether you use
deep neural network and specifically
just to be also very concrete about what
we're doing let's look at the table
table lookup case so in this case the
model is an explicit finite Markov
decision process and one thing that you
can very easily do is to estimate these
transition probabilities and these
rewards by just tallying the actual
transitions that you've made so in this
case there's an indicator function here
which means that this is 1 whenever the
arguments are all true and it's 0
otherwise so the first equation here
basically says for each time you've been
in this specific state in action you're
going to add all all the times you were
in the next States s Prime on the left
which is bound on the right and you're
going to divide that by the number of
times you were in this state in action
in total
so this will be a number that is between
zero and one if you're always in the
same next date is this s Prime mr. state
you always end up in then this will
become one if you're never there and
this thing will be zero you're dividing
by something which is nonzero you're
dividing by the number of times you were
in the state in action but you never
went to this specific X Prime so then
this probability will be zero and in
general it will be between zero and one
and in addition if you sum over all the
possible S Prime's all the possible next
States this will sum to one so it's a
valid probability distribution and here
we separated out the the reward from the
transition again and then the rewards
could be similarly estimated where in
this case there should have been an
expectation here route over the reward
because of course could be random on a
sorry we're doing the sampling version
here so we basically is just looking at
the reward and whenever you were in that
state in action and you got a certain
reward you're going to average exactly
those so in this case we're doing an
expected reward but we're learning the
stochastic transition probabilities and
this is typically a relatively good idea
in the sense that you don't necessarily
have to keep the distribution of the
rewards around because we're not
iterating on that we don't need to
sample the rewards necessarily in our
model in fact you often get lower
variance if you can just plug in the
expected reward for each set in action
in general the next states and the
reward might be there might be a joint
distribution and it might be better to
model that one but this is a this is one
way to construct the tabular model
alternatively one thing that you could
also do is to use a nonparametric model
as a stochastic model which is something
that is very easy and intuitive I guess
in the in the tabular case so what we're
going to do then is keep around all of
our past data in the simplest case and
whenever you want to sample from your
model first state in action you're just
going to look into your
ask saison actions and you're going to
pick one that matches that stays in
action for simplicity let's just now
assume that you've done all actions in
all states at least once but you may
have them you may visit the certain
states multiple times and taking the
same action there so there might be
multiple copies of the same States in
action in your replay but if there's at
least one that you can do this you could
take you could query your replay buffer
you could look at the states in action
that you're in right now say and you
might just look at your replay buffer
for every time you were in let's say you
took that action you could look at the
reward and the next day that occurred
there now you could do two things you
could look at all the cases that this
happened and an average over them say in
which case you're doing expectation
model or you could just take the first
one that you find maybe randomly and
then in that case you would have a
stochastic model so this shows that
there's a some some link between
experience replay and model-based
reinforcement learning where you can
think of the experience replay buffer as
basically being a nonparametric model
that stores experience samples rather
than some explicit parametric
representation of course as you may have
realized when I was saying this when I
was explaining how this works it does
come with a limitation because if a
state and action pair is not in your
replay then you cannot really sample
meaningfully from it you cannot say what
the next reward the next state would be
which is of course very common let's say
you're playing Atari and you have these
pixel based states it might not be that
likely depending on the game that you're
playing it might not be that likely that
you see the same frame very often or
that you see is that you've seen certain
frames
sufficiently often that they will be in
your experience replay buffer and in
fact because we want to explore we would
also be tempted to actually pick new
states and states the new axis e states
that we've seen before which means that
there should be a lot of diversity in
your replay but you can still think of
the replay even in that case as a
nonparametric model that we're using
somehow and I'll get back to that in a
moment this is just an example this is
one we've seen before when we're talking
about the difference between temporal
difference learning in Monte Carlo
it's the same experience that we had in
that case we've been in States a once
and when we were in state a we saw a
reward of zero we transition to state B
and then we sell another reward of zero
then we had a bunch of episodes that all
started in state B and in six of these
seven episodes we saw a reward one and
then one we saw a reward of zero now if
you would apply these equations on the
previous slide we will build a model
that basically look look looks like this
we're in a hundred percent of the cases
we went from say A to B and then it's 75
percent of the cases we transitions with
a reward of 1 and terminated and in fact
25% of the cases we transitions with the
reward of zero and we terminate it note
that all the episodes ended with state B
so in this case from state B you always
terminate this might will be the real
MVP but it's the MVP that is consistent
with the data any questions about this
okay okay so now we've talked about what
type of mobile you might learn how you
might represent that model and of course
we still want to use that model so now
for now we're going to assume that this
P hats ETA it's it's it's built up from
somewhere you have it and one thing that
you could do is just sample from it and
then use model bit model free
reinforcement learning might be a little
bit surprising because we're going to do
model-based stuff we're going to plan
but this is a form of planning you're
you're trying to build let me say it
differently one definition that you
could of what you could call planning is
you take a model and you output a policy
you output a plan if you will you could
call that planning if that is planning
then applying something like Q learning
to a model is a form of planning because
when you apply it you will come up with
a new policy or a new plan if you
so you could call that planning the only
reason it's now called planning rather
than learning is because we're doing it
on the model rather not only in on the
real environments which of course is a
little bit of a subtle distinction so
let's assume a generative model for now
we're going to sample experience from
the model and then we're just going to
apply any model free RL algorithm that
we've discussed previously and going
back to the example that we just had
this could look like this where we've
constructed this this model and note
that we can do this in basically two
ways here one is we can construct a
parametric model using those equations
that I had where you basically just
average the experience that you've had
so far which you end up with something
that looks a lot like the thing here in
the middle looks exactly like the thing
here in the middle alternatively we
could just keep the experience around
and we could call it a nonparametric
model which in this case would be
equivalent and then we could sample from
that and the sampled experience might
look like what you see here on the right
which is basically in this case just
something that's fairly similar to a
permutation of the experience we already
sell on the left there's a small
distinction see if you can find the
difference but if you would iterate on
this long enough and we just use some
model free reinforce learning algorithm
you would come up with the exact answer
exact same answer
this is related to what we talked about
before we were looking at this example
I'll remind you what we were doing there
what we were discussing there we were
discussing the difference between
temporal difference learning and
Montecarlo learning and we were
specifically discussing it in the
context of batch learning where you
collect a certain amount of experience
and then you iterate over and over again
on that experience until convergence and
then we were talking about that the the
answers found by Monte Carlo differ from
the answers found by temporal difference
learning which is still the case by the
way but what we skimmed over in that
case was that using the batch in full is
basically a way to use a nonparametric
model essentially we've collected the
experience and we were using that we
were thinking about the experience more
and more without collecting new
experience and so this is what people
then often call planning
yes what yes now you touch upon
something this is very good you know so
what we're not modeling explicitly what
we should be modeling explicitly and
what would be models implicitly when we
just use the the experience is the start
state distribution so that's the need
something to keep in mind and it is
something that you should be modeling if
you just use the experience it's there
right you you it's in your experience so
it's already there if you construct an
explicit model you should also basically
construct that and I skimmed over that I
basically just didn't talk about it but
it is something that's part of your
model not just the transitions but also
where do you start when an episode ends
so there's a great question yeah yes so
how is batch learning equal to planning
in a sense it's a good question so the
way to think about it is is this so the
batch learning as we talked about this
is a specific type of batch learning
where we were just constructed sorry
we're collecting some data and then we
would learn over and over on that batch
of data so we're not talking about say
mini batches or something like that
we're talking about using the full batch
of data and learning over and over on it
and what I'm saying here is that you can
say you could look at the batch of data
you could call that a long parametric
model it's not going to be exact but
it's it is what you've seen so far and
then the going over and over on it you
could call that planning because you're
not collecting any new data so what
we're making we're making maybe a
somewhat arbitrary distinction here
between learning and planning where we
can we call it learning whenever
recollecting new experience and learning
from that and we call the planning
whenever it just happens in the agents
head innocence and if you're not
collecting you experience but you're
just reusing the old experience you're
just in your head in a sense so then we
might call that planning if you're using
model free reinforcement learning
algorithms to do the planning they might
not be aware of
this right the algorithm might not know
and might not care whether the data is
real data or generated by a model okay
so traditionally reinforced learning
algorithms do not store their
experiences the classic view is that
some experience piece of experience
comes in you somehow use that and then
it's discarded and then it becomes maybe
easier to talk about model free versus
model-based because some of the
experience is used for certain purposes
say to learn a value function or a
policy but you might not have an
explicit dynamics model but if you do
use it to construct an explicit dynamics
model then we might call it model-based
so this is again going back to that
terminology but as I've mentioned a
couple of times already it's actually a
little bit more gray than that and maybe
it's not so useful to make this sharp
distinction anymore
so so for the tabular case I've already
mentioned how Batchelor learning is
equal to say nonparametric model based
learning and there's other equivalences
i also mentioned the second point here
that basically a replay buffer is a
nonparametric model it's a stochastic
nonparametric model if you just sample a
certain state action and and then you
look at what the reward and next state
were in that case this will be a sample
so it's a it's a generative
nonparametric model if you want to call
it that and more generally you could
think about using your experience in a
variety of ways one is you could take a
sample you could put it somewhere in a
model maybe maybe your model is a
nonparametric experience buffer type
model maybe there's also an explicit
parametric model or one of these maybe
you're also using it up to the value and
the policy and then it's unclear whether
we want to call that model based or
model free so the distinction might
disappear a little bit in addition
there's this question about whether the
word model is the the most appropriate
one this goes back also to the question
of whether we want to construct a model
on the external state in actions or
maybe on some other representation
and in fact in statistics we might call
we might use a much more broader term
for the word model where he might know
almost anything that we're learning from
the data a model and in that sense you
could also call the value function a
model it's a fault model in a sense it
models the value so there's another
reason maybe not to go too much into
that terminology or at least when you
when you go into the terminology when
you want to talk about models and
planning it might be better to be a
little bit more explicit about what your
what you mean when you say model what
type of model what are you actually
learning and how are you actually using
it okay so why mobile base at all right
if we can just replay well there's
something nice about having a model an
explicit model and that's it you can
query it and I'll talk more about that
in this lecture but I just wanted to
make one maybe again going back to the
experience replay case somewhat clear
clear connection there if you can query
your model you might be able to ask for
station actions that are particularly
useful right now to improve your policy
for instance might be your current state
or action or you might have a policy or
a value function and that might induce a
policy and you might know it to be quite
wrong in certain cases and you want to
improve it there using your model this
is something that you can't easily do if
you don't construct any model because
then the experience that you learn from
is tightly coupled to the experience
that you gather and for instance you
might just be in a certain situation
which is nowhere near the states and
actions that you can meaningfully learn
about you might be in a situation that
you completely don't understand yet so
learning then is hard or you might be in
a situation that way of which you
basically have learned all that you
could learn and that's actually quite
common because say if you have to tab
sorry if you have an episodic problem
what often happens is that you start
with a certain start stage and then you
learn about the region around that start
state and at some point you've basically
learned enough that you can branch out a
little bit and you start learning about
the edge but that whenever you restart
an episode you're in the region where
you basically know
what to do it it's hard to learn more
there so first you have to go all the
way to the edge where the new things are
and then you can start learning again
but if you have a model now you can
query the model for the states and
actions that are in some sense
meaningful and one potential thing which
is sometimes called prioritize sweeping
proactive sweeping is a more general
term but what is often done in practice
there is that we sample certain States
from our model which have a high error
say a high bellmen error or a high
sample pulmonary so a high temporal
difference error and then we replay
those or require those from our model
instead of just randomly picking States
and actions to learn from and in a very
related idea this was applied to the
Atari suite of games as well and later
to other things as well
you could do the same thing with the
nonparametric model that is your
experience replay where you could replay
things to learn more effectively but you
could prioritize which ones you look at
for instance by looking at the the
estimation error you had in that state
action when you last in an update from
it so each time you get a new piece of
experience you might update from it but
you'll also look at how big the update
would be and you put it into a
prioritized experience replay where
there's basically a cue that now has
certain priorities on picking things
we're no long longer sampling uniformly
but we're sampling according to some sub
priority and this turns out to work
really well
it makes your value functions more
accurate faster because you can focus
your your learning algorithm on the
things that you can learn most about so
this is again a setting where we're
using a nonparametric model in this case
the experience replay and we're using a
model free algorithm in this case Q
learning or more specifically when
applied to Atari this was a very variant
of DQ n DQ n that was used to do the
planning within that yes
romantic good question yeah the question
makes sense let me let me rephrase it
again to see whether I say the same
thing so the question is this is mostly
applied in a nonparametric case but
because in order to even learn a
parametric model in certain states you
must have been there often enough in a
sense that your values and policy are
probably probably already quite accurate
there now why is that not the case
because it's not the case necessarily
and the reason is that sometimes it's
much easier to learn a model in a
certain certain situation than it is to
learn a value function because you have
two choices to learn a value function in
the extreme case you could either use
Monte Carlo samples which are multi-step
whereas your model could just be one
step and that might have very high
variance so the value function if you're
using Monte Carlo rollouts if you use
full trajectories from each state in
action might take very high variance
inputs because they're looking at the
long term value whereas the model might
just look at the one step which might be
very easy to learn now of course there's
an alternative which is use something
more like them for difference methods
which is the common case and this is
especially useful when you do planning
because you can more easily perhaps do
off policy learning like Q learning
where you're directly trying to optimize
sorry estimate the optimal value
function rather than for your current
policy but in that case you're also
learning a guess from a guess so your
bootstrapping on a value that might not
be accurate yet so another reason to
replay then even if your model is
completely accurate in that state you
might want to replay because the value
estimates keep on changing and keep on
updating so going back to the Monte
Carlo because I didn't say something
that might be important to say again say
as well even the Monte Carlo case is
non-stationary because you'll be
following a certain policy when creating
at Monte Carlo rollout and the policy
might change because we're doing control
we're trying to optimize the policy but
if you think about the one step model
which is conditioned on the stage and an
action that's the stationary thing so it
might be much easier to learn
that of course this depends on the
dynamics being easy to learn if you
think of a grit roll this is a nice
example because it it's often the case
that you could basically just look at it
like a 3x3 block around your agents and
you could construct a completely
accurate model which you could then use
to plan with whereas it's unclear that
you can that could as easily learn a
value function so it's a great question
thanks for asking okay so I think this
is a good time to break for five minutes
and then I'll continue so I I mentioned
this already but it's good to stop and
mention it a little bit more explicitly
again which is that there's a caveat you
might learn a model which is inaccurate
for various reasons one is you might
have limited experience so your model
might not cover all the states that you
might be interested in of course you
might also have a parametric model that
has a certain capacity which might not
be able to fully capture the real
dynamics in which case you will have
some what we might call function
approximation error error model and the
important thing to note here is that
your performance now if you're going to
plan with this crucially depends on your
model to give you a very concrete toys
example of that let's assume we've
learned some dynamics model of a robot
walking around in the real world but the
model has an error anything's there's a
hole in the wall and then if you plan
with this model and you plan with it
extensively the policy that you might
end up with will definitely go towards
that hole in the wall that doesn't
actually exist so the question here that
we also want to answer at least
partially is how do you deal with that
how can you still plan when you have an
inaccurate model so one way you could go
about this is just to not rely on it
whenever the model is wrong use model
free reinforcement learning
additionally or alternatively a second
approach might be that you could
potentially reason explicitly about the
uncertainty of your model for instance
you could reason about the uncertainty
of your parameters of the model that's a
very interesting and
potentially a powerful approach but it's
also hard in general if you can
meaningfully reason about the
uncertainty of your model this is a very
promising approach and it might actually
also help you an exploration because one
thing that you could then do is to take
into account the uncertainty of your
model and for instance be a little bit
optimistic with respect to the
uncertainty when you select when you
select your policy to create new data
but in general it's quite hard to do
this way especially when the domain is
very rich and the and the model might be
very complex and then the third approach
we shall talk about a bit more is to
combine model based on model free
methods and in order to do that we need
to consider both of these sources of
experience so one source of experience
is the standard thing we have the true
MVP we sample our rewards in our next
stage somehow from this through maybe by
just taking actions in the real world
and then we have the experience sample
from the model where we also sample a
reward in the next state but now we do
this from a parametric model and then
the model free approach just learns from
the first one the model-based approach
just learns from the second one and now
the maybe obvious thing to do is do both
and this is sometimes called dinah this
is how how it's called in the book this
is something that rich Sutton proposed
many years ago the idea is to learn
model as we discussed before from the
real experience but then we learn and
plan the value function from both the
real and the simulated experience so one
way to go about that is to treat the
real and the simulated experience
equivalently as far as the model free
algorithm is concerned so this just gets
samples it's a indiscriminate on where
the samples come from and then it just
tries to learn from that so this goes
back to this picture I showed before
where there's now experience that comes
from the real world and we're using it
to directly update our value function
but we're actually going to learn a
model and then use the data from that
model to update the exact same value
function and this is a concrete
algorithm so you might initialize some
let's for simplicity just consider the
tabular case there's some some finite
set of states and actions and we
initialize a table for action values Q
and we initialize a model where the
model might be either deterministic one
so maybe we get a stay next state and a
reward determine is to be from this
model whenever you give a state in
action in just think of a grid world for
for to have a concrete example in mind
and then how this algorithm works is
that you you you are in a certain state
and you might take a certain action this
is in the real States that you're
currently in physically say and then you
execute that action and you will observe
an actual next States and the reward we
just apply Q learning we're doing we're
trying to optimize here so we're just
going to learn an action value that
approximates the optimal action value
using Q learning but then in reading it
in addition we're going to sample all
sorry did he hear where it means we're
going to update the model using the next
state and the reward if it's a fully
deterministic model we can just replace
whatever was in the model for this state
in action with the next state and a
reward that you've seen right now if
they are already in there if it's fully
deterministic then these will be the
same if there wasn't an entry yet in
your model for these then you will just
put them in there of course in general
you could have any other model updated
your update your model towards this
sample of a next state and a reward and
then we have this additional step where
we repeat a number of times where this
number is basically just a parameter or
algorithm the following steps where you
randomly sample some previously observed
states and you randomly sample previous
observed action within that states you
query your model you see what it gives
us an output a next state and a reward
and then you use that sampled experience
to do a Q learning update on the exact
same value function now one simple
example of this is when you use a
nonparametric model a replay buffer and
this is exactly experience replay done
and in fact this is what dqn uses
the algorithm gal applied to the Atari
games except that in the case of vanilla
DQ and we might even skip the first step
we might not even use the data that you
get right now we just toss it into the
experience replay and then every once
often we sample a mini batch from the
experience replay of course you could in
addition do the duty step with the most
recent days and it might be better it's
unclear might depend on other parameters
as well but there this shows that
there's a nice link here to algorithms
that are used in practice as well but of
course this is a little bit more general
because the model doesn't have to be a
replay buffer it doesn't have to be a
nonparametric model it could be any
other model as well yeah yes so it's a
good good questions don't you run into
overestimation bias for instance you
could run an into into any of the normal
problems you might run into with
q-learning and indeed you might just
plug in double q-learning here instead
and that's perfectly fine we're just in
this case we're just using q-learning as
an example of a model free algorithm
that you might apply but you can use any
other you could also do multi-step
things you could also do policy
iteration maybe on a less fine grain
where are you doing this evaluation step
and then everyone's so often you improve
your policy all of these things are
possible so this is a concrete example
of what might be called the Dyna Q
algorithm dinah as a term refers to the
more general class of algorithms in this
case Dyna Q refers to the specific
combination of dinah with Q learning
inside just for the purposes of this is
also how it's defined in the book also
for just for the purposes of this slide
and the next one and so then does this
work does this help well here's a simple
example simple grid world where the
standard actions and what this graph
shows here at the bottom you can look at
it
it's in the book as well so you can look
at it in at your leisure if you want
down is better because what's on the
y-axis is the number of steps per
episode and you're only rewarded when
you enter the goal sand and you
basically want to minimize the number of
steps until you reach the goal so lower
is better when the top line there which
is where there's an arrow pointing with
no planning steps just uses q-learning
and then there's a line way further down
to the bottom which uses five planning
steps and there's one further that uses
50 planning steps in each of these cases
this is per normal q-learning update now
depending on how you look at this this
might seem a little bit an unfair
comparison because the 50 planning steps
does basically 50 times as much compute
but of course this depends on the
trade-off on how how expensive it is to
get gather real experience versus to Hux
how expensive how expensive it is to
query your internal model and to learn
from that and there's many cases in the
real world where gathering the real
experience is the bottleneck and it
might be quite expensive and slow in
some sense to gather new experience
whereas just carrying the internal model
or even the past sword experience might
be much faster so it depends on whether
you're interested in pure computes
versus sample complexity and this shows
that in terms of sample complexity so
the number of episodes that we've seen
it's much better to query your model and
this shows why this works yeah yes
so the algorithm set repeat n times and
then 0 5 or 50 is that n and in when you
set n to 0 it just becomes queue turning
because you're just not using the model
and this is the visualization of why
this helps on the left there we see
basically the policy that was learned
after the first episode by q-learning
and we're doing one step cue learning so
basically the only value that was
updated with anything that was not zero
with a non zero reward let's assume our
action values were initialized at zero
the only one that had a nonzero update
was the one leading indirectly into the
goal white a reward is a plus one so in
your next episode this is basically not
helpful at all until you reach that
stage just before the goal and then you
would update the state below that in
your second episode and then the third
episode would again be completely
blissfully unaware and it would continue
just random to explore the space until
it answers that said and slowly you
build up this trajectory going backwards
to the starting state over multiple
episodes now one way to go around that
is to use multi-step returns which
you've discussed in the past but in this
case we're just going to use the same
one step to learning algorithm but we're
going to plan and what this does is just
sample states that you've seen before
including once near to the goal and it
will update your action value function
from this past experience but now using
the new action value estimates to
bootstrap on because in each time when
we do one of these updates we're going
to do a max Q in the next state but
we're going to use the current action
value estimates there which means we can
propagate information back this is again
during the second episode so we've only
ever seen go once but already the model
can propagate a lot of information all
the way back and construct a policy that
from many states leads very quickly
towards the goal the agent here is
depicted as the black box so you can see
the agent in this right hand side is not
actually near where the model will tell
it where to go so it's basically just
roaming around the start state randomly
without going anywhere in particular but
while thinking about it it gets more and
more of a well-formed idea of how it
shoot acts when it enters one of these
states by the way this picture here on
the right hand side depending on how
many
actual updates we've done would look
similar to what you would get in the
left hand side if you just did many
episodes right this is how you would
construct your view function then sorry
the black mark is where the agent early
is physically because for each step of
the agents will do an update with that
step near the gold this won't actually
change anything because the reward is
error in your next state value is zero
but and then we do 50 planning steps
from arbitrary states from your past
experience so there's two things
happening here one is that we're doing
many more updates using the model and
the other one is that we can create a
model from any state so we can update
states that we're not actually
physically in because we've been there
before in this case of course the model
is really simple you only need to take
an action in a certain State once to
have an exactly accurate model what for
what the next state will be in that
state so this is a little bit of a toy
example in that sense but it does
highlight a get more generic maybe
property of being able to plan so
another thing that you might want to do
I mentioned that it might be hard to use
an inaccurate model if you rely on this
fully so now we're going to investigate
it a little bit are we going to look at
a situation where the model is going to
be inaccurate and we're going to enforce
that basically by changing in the
environment without telling the
algorithm so we start off with the
situation on the top left there where
it's a very simple great world you start
at a certain state at the bottom and the
goal is to get soo-ji the goal and of
course there's a clear path there and
you might find it at some point and the
graph here at the bottom basically shows
the learning progress for now just just
focus on this Dyna Q line and at the
beginning it's a little bit flat and
then at some point it goes up and that's
the cumulative reward that you've
received so this must go at some point
to a straight line that goes up because
you'll get a certain amount of reward
and the better you added are this task
the more rewards you get per time step
but now at time step 100 we're going to
be a little bit mean to the agents and
we're going to change the environment by
closing the gap that was going to go
through before and we're opening a gap
at the other side and this is a tricky
thing because if you just think about
the policy that was learned or in this
case if you would just learn a value
function without any model learning or
whatnot
the value function would all point from
the start state to the right and it will
take a very long time to correct for
that and to than explore especially if
you're doing something we absolutely
greedy you would have to just randomly
wiggle your policy and in addition if
you say from the start state would step
one step to the left the value function
that you've earned will tell you to
immediately go back go back right
because that was where you were should
be going so at first you'll go back up
to the place where the gap used to be
you might notice fairly quickly that
you'll be disappointed there that the
values will go down and at some point
you will be less likely to go there
perhaps but if you haven't explored much
to the left yet it'll take a very long
time for the agent even to randomly end
up there to maybe notice that now you
can go through the wall there
the model-based algorithms they have
kind of the same problem but because
they can queried our past experience and
they might randomly query all of the
states and actions that you were in and
keep on updating from them they might
more quickly learn this one thing that
they can very quickly learn is when they
enter these states where the situation
has changed in this case the model is
deterministic so only doing this once
they can notice that this is no longer a
valid transition which means they only
have to go towards the where the gap
used to be once bump into the wall and
our model is accurate again at least at
that part they still have to discover
the the opening at the other end so they
still have to explore but at least they
very quickly learned that this is no
longer the right place to go and it can
very quickly learn that all the values
near where the gap used to be are
actually much lower than the thing you
thought they were
that might be an interesting experiments
sit around normal q-learning here see
what happens it's only on I don't know
how would exactly look it's not only a
slide obviously but it's something
simple to call it up and it's quite
interesting to look at here's a
different situation so I'll say what
Dyna Q is sorry Dyna Q pluses in a
moments but I first want to talk about
this example we're going to ignore Dyna
AC which is also not in the new addition
I think a C stands for actor critic but
we're basically just going to ignore
that one this is just ignore that line
this is a different situation which
might be in some sense a little bit
harder the reason being that we didn't
close the gap and the gap here used to
be on the left rather than on the right
so you had to take this long detour
around the left to go all the way to the
goal and at some point later on a new
gap opens the shortcut appears and now
you can have a shorter path to the goal
now the norm will die in a queue agent
doesn't actually see that you can see
that when we change that at a time step
3,000 the line continues to go up at the
same gradients which means that the
agent still is taking the long way
around to the goal which is kind of fine
but it's not perfect of course if we
would close that gap then it might learn
to go first yard away but well we're not
closing the gap basically the algorithms
everything's fine I might get her a
little bit I might do a little bit of
explore Absalon exploration around my
trajectory but I'm virtually never going
to do enough exploration to take me all
the way against my current best
estimates towards this situation where I
think there's no gap just to find it
there's a gap now what is this Dyna Q
plus agents what this basically does it
gives a slight bonus in the model
updates for states and actions that you
haven't seen in a while so if you
haven't tried anything in a while
it'll basically try them again in the
planning steps it won't act to actually
go there yet but it will just assume
somewhere optimistically that things got
better over time this means that
everyone so often the agent will
actually go and try it again just to see
whether the model is still accurate so
what this bonus does it basically tells
you don't be too certain about your
model and in fact the specific bonus
tells you become more uncertain about
your model if you haven't been in a
situation for a while and then we're
basically going to be optimistic with
respect to that uncertainty we're going
to say I might just be able to do
something here that I can't I might just
get a higher reward than I thought I
would and this is why the Dyna Q plus
agent curves up after a little while
because this means in its planning it
has been optimistic about trying things
and then at some point it'll actually go
to the Wrights and collect new data
there because it's optimistic that that
that there might be something better
there the reward might be higher there
than it was before and if it wasn't the
case the model will quickly update again
and the uncertainty there will go down
and we go back to the normal thing but
in this case this is a good thing to do
and it will try this try to go to that
new gap and it finds that there's
actually a gap in and it goes to the
goal yeah it's a good question in this
case the bonus is a function of time so
it increases over time as well if you
haven't tried a search the state action
for a long time the bones will continue
to grow until you finally go back and
try it again
there's many choices in which way in how
you could do that and what's
particularly interesting but also
challenging and it's kind of an open
problem it's how to do this when you
don't have say a tabular situation in
which you can just keep track of counts
but now you have say a very messy state
space it's an open question how you
could best do these bonuses then because
it still seems like it's a good idea to
have some something of an optimism which
depends somehow here in certainty yes
why doesn't it underperform earlier
because it's xplory it is kind of if you
if you look very carefully you can see
that the Dyna Q plus line was
approaching the Dyna cube line at the
beginning it does a little bit better
because the exploration is more informed
it's more directions the Dyna Q agent
would randomly explore at the beginning
where's the diner q + agent who try to
seek out novel situations already at the
beginning which is why the curve goes up
for faster at the beginning it learns
more quickly but if you would keep on
running this the Dyna Q + agent would
keep and if nothing would change
the dining q + h it would keep selecting
gnome optimal situations in order to
keep on exploring depending on how you
set up these bonuses you can make that
go away in the limit so that it only
tries its for a limited amount but if
you assume that your environment might
keep on changing you don't actually want
to do that you want to keep on exploring
but it does mean that you incur a little
bit of a hit and these lines do actually
approach each other before they branch
off in this case how does it decide to
go all the way to that corner when you
just have to say bonuses on your rewards
and the reason is because we're doing
this planning step because let's just
take this one transition err you're on
the top right there where you didn't use
to be a gap but now there's a gap and
we're considering that action going up
there which is a bad idea at the
beginning so we haven't taken it in a
while and the reward that keeps growing
and growing let's ignore for now two
rewards in all the other states which
also play a role but let's ignore them
for now
and let's assume that your model
otherwise it's just the same old thing
even then because this reward keeps
growing the states that go into the
states you're going to update the value
in that state first of all for the
action going up you're going to update
it a little bit more optimistically
you're going to update it further up
than it used to be but then in your
planning steps even without going there
this the the states and actions that
lead to that state would also begin to
to grow so basically the agent itself
could keep on going to the left
going through this hole but at some
point the optimism bias of going into
the into the wall all the way at the top
rights would make its way all the way
back to the start state and you could
imagine the agent even if you're just
giving a bonus on that specific action
of walking into the wall there the agent
would would at some point decide to go
all the way there and go into it now as
I said the other rewards actually also
play a role and what the agent will more
likely do in practice is well it will go
a little bit in that direction
it will try one of the actions that it
hasn't tried for a while and then go
back and do the normal thing but that
means that you've tried this one now and
the exploration bonus went down which
means that maybe what it'll try first
it'll it'll bump its head against the
wall which is still they're a little bit
closer to the start state and then it
will go back to the normal thing but at
some point the values from these farther
states have been visited that
infrequently that the bonus starts to
make its way all the way back can you do
that again
okay now you can apply this with
function approximation as well what's
hard is perhaps to find the true model
in that case and what setting will slide
is it saying something I've said already
but it's it's it's basically just a hard
and more or less unsolved problem it's
also a very active area of research not
necessarily specifically for
reinforcement learning but trying to
create good generative models is a very
active and fairly large portion of deep
learning research right nowadays and I
think many of us it is my transfer to
the reinforcement learning case but it
would be good to explicitly look at them
and see how well these things don't
transfer but it remains in maybe in
essence just a hard problem to find good
probability distributions in high
dimensional feature spaces or input
spaces in addition it could be quite
computationally expensive which means
that it might take quite a bit of a
compute to go from say a frame into a
next frame especially if you want to
roll out multiple steps but it doesn't
mean it's not possible by the way just
call out again there's the alternative
of doing the replay where you basically
just do sample a nonparametric model
instead of having an expensive
parametric model which you could use
here but the idea is more general so
dinah the idea of dying even though
we've looked at the tabular example does
apply to the function próximas case as
well where your model could be
parametric a nonparametric and you could
still use that idea you could still
potentially see similar gains as we've
seen in these toys examples especially
if your environment can be non
stationary which is often the case when
for instance there might be other agents
in the environment or wet weather when
your policy just changes over time and
new situations and open up or your agent
just gains additional capabilities it
might be able to climb some steps that
it couldn't climb before so then it's
useful to go back to the steps to try
that again ok so now we're going to turn
towards so we've been learning a mobile
and planning with it and focus a lot
also on on how to do with the
inaccuracies now we're going to consider
the setting where the model is given and
we just want to use it
so normally so what we did in dine is
basically update a value function using
all our standard model free RL
algorithms that we might want to use and
specifically we looked at Q learning of
course we might also want to use a model
to increase improve our policy but then
in particular we're actually interested
in policy right now we want to select an
action we want to select the best
possible action that we can and for
simplicity it's good to keep in mind
that the the data collection might be
the most expensive part of your whole
system so it's very important to select
an action carefully you might be able to
spend quite a bit of compute and effort
in in doing that so it might be more
important as it says here to make a more
accurate local estimate of either your
policy or the value function then you
could feasibly get into your global
estimate this is also often what people
intuitively think of when you say
planning we think of you're in a
situation and you're going to look into
the future in order to decide what
you're going to do right now this is a
little bit different from the case of
Diana that we looked at just now where
we're basically doing this random
planning all the time we're just
updating our complete value function in
every state that we may have seen all
the time to just make it more accurate
but now we're going to focus our compute
so that might look a little bit like
this schematically so standard forward
search algorithm starts at a certain
state and then maybe plans through hole
three and these T's are may be terminal
states so you might create this whole
thing you might have your MVP you might
do the full exhaustive search and then
you back all that up and you find what
the best decision was in this starting
state so the close dots here the depict
actions in this case you have two axes
in each state and then you just try to
find that thing so you could use
classical search algorithms if the model
is a true model right then it's okay to
do that the the plan will be okay for
the for the true model in that case and
there's many heuristics especially
especially that can speed that up quite
a bit so you don't have to build the
full three
but you could also that could still be
too expensive the tree might be too big
in which case you could simulate so this
is very similar to the situation we were
in before except we're making the step
of calling out explicitly that there's a
model before we basically said yeah
that's the environment and we're just
going to sample from the environment but
even if you have the model it might be
useful to simulate and you might just
create a trajectory one difference
between the model and the environment is
that you could simulate multiple times
which you can't do in your environment
when you take an action you've changed
your stage you can't go back necessarily
but if you have a model you could be in
a state you could consider multiple
trajectories and you could use those to
update your value function before you
actually select an action so the idea is
now that you basically start off with a
certain state st and then are using k
here to denote the number of times you
sample a trajectory so we use our model
which might be an approximate or might
be the true model in this case we're
kind of indifferent about that and we're
going to sample multiple trajectories
and one thing you could for instance do
you could just look at all the total
returns across those trajectories and
you could average those so in this tree
this looks a little bit like we don't
just sample one of these but we sample
multiple and we just maybe average over
those alternatively of course you could
use this trajectory to do something else
like TD learning in this case for
instance our site if you want to learn
state action values the search tree
itself can be discrete even if your true
environment is very messy and continuous
or whatnot
if you especially if you think about the
sampling version it will actually have a
state and you'll actually have a next
state which is kind of a discrete
quantification Fontes quantization
what's the were there I don't know of
the continuous problem in which you can
then do search so this goes back to an
earlier question about can you do this
in continuous state spaces and continues
actions well yes you can especially if
you do the sampling approach you will
actually still build build a fine I
treat
so for mobile free reinforce improving
if you store everything in a table
that's a bit naive because this is not
always going to capture enough structure
also you're not going to generalize well
either a tables too small which means
that each state maps to too many things
in the real world or your table is too
big which means you don't generalize and
very similar states have separate values
that you'll eat in or separately and
everything will be slow but if you do a
simulation based search this is less
naive because you could actually build
this up these trajectories so you still
don't generalize so you might still want
to learn a value function you might want
to use that in your tree and I'll turn
to that in a moment but before we do we
just considered with maybe the simplest
case this is what I said just now we
will consider a simulation policy and
this could just be your real policy
right now but you could for some cases
you might want to consider something
else so we'll just call that the
simulation policy will start at a
current state we're just going to run
this simulation policy multiple times
and then we're just going to average the
returns this is a simple thing you could
do what this will give you in that case
is an estimate of the value of that
policy so we're not doing necessarily
control in this case we're just
estimating the value of the policy you
could still use that for control if you
do something like policy iteration in
some cases it's also just very useful to
know a value of a specific policy there
might be a certain set of policy that
you're considering am I just want to
estimate all of them and you stand still
for a while you just run all of them
through your model in your head and you
pick say the one with the highest value
before you actually take an action now
you could extend this very easily to
action values rather than state values
so in a previous slide we just had a V
and now I'm just going to replace the V
with a Q essentially and then we can
also select the real action very easily
by just looking at the for each of the
actions we did these unrolls say so what
we're going to do now is we're going to
pick each of the actions especially if
there's like a small amount of actions
actually you can do that and then we're
going to follow after this action or
simulation policy and then for each of
these actions we'll get an estimate
we'll do that multiple times assuming
the simulation is fairly cheap which
means we get these action
estimates for given this action and then
following your simulation policy this
will be your value or an approximation
thereof and then we can just pick the
best action according to these
approximated values note that these Q
values are not stored anywhere right
we're just computing them on the fly
from a certain state in action right now
in this specific setting so we're
constructing an estimate action value
but not to update a function right now
but just to use it immediately to select
an action using our model in the
background to plan ahead so this is a
fairly pure case of model-based
reinforcement learning where there's no
explicit value function we do need a
simulation policy which may or may not
be explicit it might just be a function
of something else might be random but
we're not explicitly constructing these
action value as estimates which is also
why a queue here is not parametrized
there's no parameter theta or anything
we're just constructing them we take
this one action in the end and then we
maybe recompute everything we tells
everything away and we recompute
everything in the next state this is of
course especially useful if planning is
cheap so you have a cheap simulator in
your head in a sense but the real
environment is expensive this arrow this
Wiggly thingy it means that if you
sample enough of these it goes to that
value so it's kind of like in the limit
of big K it'll go there it's a pseudo
notation it's the later command leads to
if you are yeah
yes so this is you can you can indeed
view this as an approximation of
computing fool 3 in in expectation and
each trajectory the expectation for any
of these trajectory will have the same
value as doing the complete research but
the variance will be very high
which is why we sample multiple of them
and then average over these but indeed
if you just write out the expectation of
any of these trajectories it'll be the
full three and then indeed you could
compute that explicitly but that's very
expensive so instead what we're going to
do we're just going to incur a little
bit of variance we're going to sample a
amount and then this K number might be
very important in to in practice because
you want a sample sufficient amount of
these to reduce the variance
sufficiently that you pick a meaningful
action however one thing that might be a
little bit interesting here is you can
do a little bit of say implicit
exploration here where you don't sample
too many and in that case you the action
that you actually select will have a
little bit of noise which in some cases
might be nice to induce a little bit of
exploration just using the variance of
the returns yeah so if you have the
wrong model here you'll get a wrong
estimation and in some sense if there's
multiple things you could do here right
you could use the data that we simulated
here still in a diner like sense to also
update a parametric value function
that's one but leave that one aside
that's what we discussed before the
other thing you could be doing is you
could be updating a learned model here
and that's actually actually completely
consistent with this view because we're
not assuming that you have the true
model here we're basically saying there
is a more
and we're going to use that the plan
head and in to pick an action and in
some sense if if you've learned this
model with the best of your capabilities
on the data you've seen it's going to be
a little bit wrong but it's also the
best you have but it's kind of okay
because you're going to select a new
action and you're just using this model
to the fullest to select the next action
and then indeed what's missing from this
slide is that you might still use the
actual experience that you didn't get to
update your model to continue to update
your model and that's of course then
very important this part is kind of
agnostic about whether you have an
approximate model or a true model you
could do that in both cases when you
have a true model the reason will be
what you set first just to avoid
computing fool three you might want a
sample instead okay and now this sets us
up to talk about something that's a
little bit more sophisticated which is
Monte Carlo tree search and it's a
fairly similar thing to what we had
before we're going to take we're happy
we have a model and this could be the
true model or it could be an approximate
model and we're going to simulate a
number of episodes from the starting
state but now we're actually going to
also explicitly build a tree containing
visited States and actions we're not
going to build the full tree of all
States and action so I'll be a little
bit more concrete about this in a moment
but then we're going to evaluate by
looking at the past basically experience
that we built and we're going to reuse
some of that experience it's maybe not
too important to think about this
equation right now we're also running a
little bit out of time but we're going
to reuse the experience explicitly but
we're also going to sample new
experience with a new rollout and then
you actually select is basically the
same we're basically going to do the
same principle but I'll be a little bit
more concrete about how this works then
but the principle is the same we're
going to sample stuff and we're going to
use that to select new action but
instead we now have two different parts
we have two different phases the first
phase sits within an explicit tree that
we'll be building and I'll show you that
on the next slide I think you know I'll
show that a little bit later I'll I'll
skip ahead to that slide
and we're going to store some
information in that so the easiest way
to think about that is that there's a
fixed start state and this is why the
next slide actually talks about going
which there is a fixed start state which
is the empty board and you can imagine
building a search tree from that start
States by simply slowly building up
things that you've seen we don't need to
toss away all the information every time
we restart
maybe we can reuse that information
sometime instead of just learning a
model and in fact ingo we might not
learn a model we might just use the
actual dynamics of the game as a model
and then in the second part is the same
as we've seen before at some point
you're going to reach the end of the
tree that you've built so far and from
there were just going to sample within
the tree we can use a tree policy which
is potentially better informed because
it can use all the passed information
that we had in addition it might also be
because the tree is smaller might be
more expensive because you take only a
number of steps in there and then the
rollout policy could maybe be fully
random this has been tried in the past
and it's actually maybe more successful
than you think it might be or it could
be some faster policy that allows you to
do many rollouts and then the Monte
Carlo tree search algorithm basically
has four phases for different parts of
it one is your in your search tree and
you're going to select actions according
to the tree policy which traverses your
tree this is a little bit similar to
exhaustive search where there's a
smaller tree now because we're not
considering the full tree but we're
going to consider a tree that we've
built ourselves and at some point we're
going to reach the edge of the tree
because the tree isn't full we can't
just go through the tree all the way
until the end at that point we might
expand the tree with one additional node
so this is an action you've taken in
this stage and we're going to add this
to the tree we don't have a value for
this action yet we haven't stored it yet
we've never made me visited it before so
what we'll do to get a value for that
action is we'll do going to do Monte
Carlo rollouts we're just going to roll
out this roll-up policy to all the way
towards the end say in the game of go
all the way until you win or you lose
you average maybe over multiple of these
and this gives you some estimates of
what the value of that new expanded
action is and we're going to store that
on the node so now we have an
approximate value for that action and
then we can back this up into the tree
using classical backups that you might
do in on a whenever you do it at
research you could think of this as like
a dynamic programming like procedure
where you back up this value all the way
to the roots maybe changing the values
of the actions that led to this specific
new node and then you update action
values along the way as well
so you update the nodes along the way
not just the not just value of the
action all the way Rupe of all the
access intermediates as well with this
new information that you have from the
new rollouts and then you could repeat
this over and over again note that we're
not adding all of the actions that we
took in the rollout to the tree just the
first one otherwise the tree would grow
very big very quickly which would defeat
the purpose
and so this was applied to go for which
filled building full trees just too big
branching factor is a hundreds of
potential moves so only a few ply is
deep and you're already completely lost
in terms of compute this is just some
standard gold stuff you can play it on
small boards or big boards there's two
players white sums and black stones not
too important for what we're talking
about the important thing is there's
only one reward function at the end you
either win or you lose all the rewards
along layer zero so it's fairly sparse
and then there's a policy that basically
tells you had one player moves and the
other player moves you might only have
control over one of these you might have
a fixed opponent and the value function
is very simp simple it's basically just
for the for a given policy it's
basically just the probability that you
win and the optimal value function is of
course trying to maximize that
probability that you win and this is a
simple example of then during rollouts
where maybe you've done four games from
a certain situation maybe you could
think of the empty border at the top as
the root and if you just simulate
rollouts you might win sometimes lose
sometimes in in this case one half of
the time so the value will be at half
but then the tree search would slowly
build up a tree and I'll just show you
schematically but I refer to the certain
'martha book in which it explained a
little bit more in detail so in this
case there's a root there at the top and
we only have one rollout which used the
default policy maybe random
happened to win so now the value at the
top there is one but now we expand this
this thing we consider one more node in
the tree from this top top node and from
there again we just roll a random roll
out to determine the value of this thing
and we happen to lose so the value of
this thing is now 0 0 divided by 1 is 0
we also update the value at the roots
which is now one behind divided by two
because we won once
and we lost once so next time we might
try something else
we try a different action this time we
pick something happily that one so we
update again the value of this state
with the single roller which is now one
because we've only one roll at any one
typically you do multiple rollers but in
this example we only do one we also
update the value at the root so now 2/3
because we've won twice lost once and
then of course you might be tempted to
try go from that same state again and
you expand from there not from the root
this time from the state there below you
lose once you update everything all the
way back up the tree maybe you try
something else you win once again you
update all the way back at the tree the
total number of rollouts for the top is
now five and three times we won so the
value of that sale is now point six for
the specific policies that we've been
following within the tree we can use a
more expensive and more elaborate policy
because the tree is relatively small the
rollers can be fairly long so there we
want something fast for instance a
random policy so typically the tree
policy and the default policy or rollout
policy as it's also sometimes called can
differ and a different practice ok this
was used in alphago with some additional
things but basically now you have all
the components we've covered all the
components in this course that you could
now build your own alphabet system if
you want so there's of course many
hidden details and many things that you
would need to know in order to be able
to do that but the main core components
are there and maybe I'll just start the
next lecture because we're running out
of time maybe I'll just start in the
next section and I could go in libname
or more detail how this works because
it's actually a very nice demonstration
of combining all these components
together so for now thank you
you