Reinforcement Learning 3: Markov Decision Processes and Dynamic Programming

this week we have a pretty intense
reinforcement learning schedule
especially should consider starting last
Thursday in a week we're going to cover
quite a but quite a lot of material I
highly encourage reading the background
material from satin Umberto because
there's only so much I can cover in a
single lecture and there's lots of
subtleties and details that are quite
good to have seen at least once and that
are described in the book also I'll
cover a few examples but I'll not be
able to cover them completely
step-by-step in all of the cases which
means that it's actually sometimes much
better to do go over them slowly and the
book also goes over some of these things
step by step which gives you a lot of
intuition if you do that so I highly
encourage you to do that do that
specifically the background for this
lecture will be chapter 3 and 4 3 was
also listed as background for the first
lecture and indeed I will be maybe back
basically backing up and recovering some
of the material I'll cover the first
lecture to place it appropriately into
context and also because these things
are important concepts that we need to
make sure that we're all on the same
page on so as always to recap we're
doing reinforcement learning the science
of learning how to make decisions agents
can learn a policy value function and/or
remodel in order to do so and the
general problem takes into account time
and consequences last lecture we didn't
we basically well we did a little bit at
the end but at first we just considered
the one-off setting where you've taken
actually you get some feedback there's
no consequences apart from the immediate
reward so you're able to redo many times
that was the bandit setting in general
decisions can affect the reward and the
agent state and the environment state in
this lecture I will not be talking so
much about agencies and environment
States I will be talking mostly about
states in general and that is because we
will be making the assumption that these
things are all the same for now will
violate that assumption later and I'll
come back to that
so we'll just simplify and just say
there is a state and this is both the
environment State and the agent States
more or less at the same time
so again last lecture basically covered
the multiple actions but only one state
case and in addition we basically
assumed there was no model we did cover
model-based reinforcement learning where
you learn a model but we weren't
assuming the model was given in this
lecture we'll go towards the full
problem by formalizing the problem of
the fools reinforcement learning setup
so the full sequential structure but
we'll first focus on a class of solution
methods in which we assume that a true
model is given and these methods are
called dynamic programming which your
memory no have heard before what we'll
cover what that means at least in the
reinforcement learning context and next
lectures will cover similar ideas again
but then we'll get rid of the assumption
that there's a true model so if you
don't like that assumption don't worry
it's only temporary
so we'll formalize the interface we did
a little bit of that in the first
lecture so some of this might feel
familiar which is good because this
should start to feel familiar and
essentially that means we'll discuss a
mathematical formulation of the agent
environment interaction we'll basically
just formalize what all these things
mean just to have both an intuitive and
the formal way to reason about these
things and we call that a Markov
decision process as discussed before and
then we can talk clearly about what the
objective is for the learning algorithm
or for the agent if you will and also
how to reach that objective how to
construct solution methods that's a part
that we didn't cover in the first
lecture so Markov decision processes
they formally describe basically an
environment mat for now as I said before
we're going to assume that this
environment is fully observable which
means that the current observation
contains basically all relevant
information to make a decision and also
to make predictions this is sometimes an
unrealistic assumption if you again
think of the robot case where there's a
robot driving around the corridor and it
maybe has a camera it maybe only has
this camera pointing one way and it
cannot see what happens behind it in
that case you could say that the
observation does not fully describe
anything you need to everything you need
to know about the environment states but
it's still useful to consider Markov
decision processes even if they don't
care for all of these cases because they
allow us to formalize everything and
indeed there are very general framework
because even in the case that I just
described you could say that the
observation to the agent might not be
the full environment state but the
problem that the agent is trying to
solve might still be a Markov decision
process underneath it just might be that
you have a limited view however in this
lecture we basically assume that you
don't have this limitation and it you
can basically see the environment state
it's indeed within the framework and it
just means that this is this is called a
continuous Markov decision process which
means that the the amount of states that
you have is basically in continuum and
also the amount of actions that you have
is basically continuum but the formalism
still applies and many of the results in
the formalism also still apply also even
when the interaction is partial partial
observability then still you can convert
this to a Markov decision process one
way was given in the first lecture where
if you think about the full history of
interactions that an agent has had with
an environment this is a Markov state
because you can never add more history
to it to make it more relevant that's
not a real solution in practice because
this history will grow unbounded but it
shows that the formalism at least
applies and may be there that means that
it more in general and indeed this is
true there are ways around partial
observable problems now also to tie it
back to the previous lecture
bandit problems are basically Markov
decision processes but they only have
one state so there's no sequentiality in
terms of the states they're still
rewards they're still actions they're
still basically the other parts of a
work of decision process but it's a
fairly degenerate one okay so to recap
the Markov property why are these things
called Markov decision processes a
process is Markov if essentially as it
says on the slide the future is
independent of the past given the
present this is just basically this type
this comes from the formal definition of
the Markov property it means that the
current state is a sufficient statistic
it contains all the information that you
need
no and for a Markov decision process
this basically means that the Joint
Distribution on the reward and the next
state when you condition only on the
current state is the same as if you
would conditional all previous states
this intuitively means that if you would
know all the previous states this gives
you no additional information about what
the next state will be and what the
reward will be which means you don't
have to look at them you can just look
at the current state so one the state is
unknown the history may be thrown away
which is very nice it's very convenient
and this is also there's algorithms that
make this assumption there are quite
efficient and we'll later talking later
lectures about what happens when you
can't make this as something how you can
still get a lot of good guarantees but
for now we'll just come to stick to the
Markov decision process which can then
be formalized as a tuple of set of
states a set of actions P is basically
the probability function that describes
your dynamics and gamma here is a
discount factor the dynamics are
sometimes described separately in the
literature or we have a separate reward
function and a state transition function
and indeed this is sometimes useful but
you can factor these out as it says on
the bottom of the slide you can
basically still describe these
separately if you have two joint
distribution and it turns out in some
cases is actually quite convenient to
have this joint distribution instead of
thinking about them separately the
reward and the discount together define
the goal of this Markov decision process
so today that they define what it means
to be optimal in this process so to give
an example concrete one this is from the
book so you can also look at the example
in the book for more detail or if you
just want to grow get more we're going
to consider a very simple set up which
kind of models a robot that cleans cans
and this is a toy version arrows in
which there are only two states that the
robot can see either the robot has a
high battery charge or it has a low
battery charge and these are the only
states the robe
can distinguish now in reality this
would be a partial service in here to
make it Markov is that actually the
probability of transitioning from one to
the other is fixed so doesn't depend in
your history this is an unrealistic
assumption but this is just to make it
simple and to have a simple example that
we can reason through in this case the
action sets are actually different per
state in the high battery States there's
two actions you can either wait or you
can search for cans and in the low
battery States there's an additional
action to recharge which basically means
that the road will go to the charger and
it will recharge and it will go back to
the high battery charge state you could
of course if you prefer think of the
interface between the agent and the
environment as fixed
that there's Princes certain motor
controls that you can do and it doesn't
matter in which state you are you have
the same controls available in this case
you could easily just expand the action
set for instance in one state where you
basically say okay there's this action
recharge but when I'm in a high battery
charge that doesn't actually do anything
and then the interface is fixed in
general you can allow these different
actions per state if you go through the
literature you'll notice that most
authors don't make that distinction and
they'll basically you say there's an
action set and it's kind of assumed that
this action set is the same in all of
the states now the dynamics themselves
might be stochastic and essentially
what's happening here instead of
modeling the full problem where if you
can only see whether you have a high or
a low charge that would actually depend
on your history whether you would
transition from one to the other instead
we're just going to assume that it's a
noisy transition where sometimes you go
from a high to a low charge and how
often this happens how quickly this
happens maybe depends on your action so
in this case the probability of
transitioning to a state with a high
charge if you are already in a state
with a high charge and you perform the
action of searching is a certain alpha
between 0 and 1 which means that the
transition to the low battery states
must be 1 minus that in this case 1
minus alpha because there's only two
states these things shoot together some
to 1 there's always a state User
internship transition to
in this case you could define a reward
function as well that could be for
instance the expected number of
collected cans if you want to have a
deterministic reward function or you
could make this thing stochastic as well
you could make it say the actual number
of collected cans which may be is again
you should assume then a Markov function
of your state but it can be stochastic
it can be a random number but in this
case in the book the assumptions made
that this is a fixed number and this is
the table from the book which basically
defines the whole MVP
these are simple functions so we can
just define the functions the transition
functions to reward functions by just
enumerated we go over all states all
actions all possible combinations and we
say what is the probability of ending up
in this next state what is the reward
you see along the way you see that
there's this minus minus three rewards
somewhere there close to the top this is
when you go from a low charge state and
you do a search action and then you go
to a high charge state essentially what
happened here is that the robot needed
to be rescued and this is a costly thing
so we're kind of abstracting away time
right we're not saying how long did this
take we're just saying okay we were in a
low battery States the robot tries to do
search it got stuck we needed to rescue
it we need to penalize it so it doesn't
get stuck that often anymore and then
the robot could itself learn not to do
that hopefully of course this is only
relevant if the probability of that
happening is large enough which in this
case is parametrized with this beta so
some of the things here are parametrized
so you could just plug in numbers and
you could then then solve this problem
we'll talk about how to solve later or
you could view this as a whole family of
Markov decision processes essentially
yes yeah so if you look at the the first
zero there there's no probability of
that happening but we're still
numerating it because it's one of the
combinations that you should be
considering and indeed this is typically
the case many states aren't actually
connected what we do sometimes assume
just because it's convenient for
Theoretical purposes
or for practical purposes we often
assume that all states are at least
eventually reachable from each other but
doesn't mean that there must be a
correct connection between any two
states so in this case whenever you're
it's a good a good question whenever
you're in the low battery states when
you take the recharge action there's no
probability that you'll stay in the low
battery States in this simple
formulation and of course this is one
way to look at it but sometimes it's
much easier to look at it more
graphically which is this is the same
problem and this is kind of lute using a
schematic notation that's used
throughout the certain umberto book
where open circles are used to depict
States and closed circles are used to
depict actions so what you see for
instance on the left hand side there
there's this high battery state and this
goes into a weight action which then
loops back into the high battery state
because apparently in this formulation
there is absolutely no charge on your
battery when you wait but maybe more
interesting on the right hand side if
you're in a low battery state if you do
a search action there's two things that
can happen so it goes deterministically
into the search action dot there but
then there's two branches and these
branches are there's a reward on each of
them and there's a probability on each
of them the one minus beta and the beta
are the probabilities of each of these
branches happening and the minus 3 and
the our search are the rewards
associated with that transition so here
you can see that this is a joint
distribution you never get the reward of
minus 3 when you go into one branch
because what this means is if you go
into the left branch this goes to the
high state this is the when you need it
to be rescue situation essentially you
were in a low battery charge you took a
risk you went for the search action you
failed you got stuck your batteries
deeply that somebody needs to rescue you
but the reward only happens when that
happens it might also happen that you do
get your in that you do collect some
cans you get your reward search for
whatever that is let's assume it's
positive right you collect some cans you
don't lose some but this only happens
when you are in the transition so to
join distribution
in this case transition accounts happen
aren't actually depicted we could of
course just make it the fully connected
graph but if there's a probability of
zero the the arrow is just missing even
though it was in the previous table
clear so in the first lecture we talked
about returns a little bit and acting in
the Markov decision process will in
general result in returns which is the
total discounted reward from each time
step onwards and this is a random
variable it's just all of the rewards
can be random because of the statements
issues it can be random because the
rewards themselves can be random and
because your policy can be stochastic
and therefore the actions can be random
so it depends in general on the MVP on
the dynamics thereof and on your policy
and the discount factor as mentioned is
an important part of this return it
basically defines the return along with
the reward function so the marginal
value of receiving a reward much later
is discounted in this case there's an
example if you wait k plus 1 time steps
that reward will be discounted with
discount to the power K in as is
compared to when it would happen
immediately if the reward is big enough
this will still be important and the
agent might still pursue it unless it's
too far away and at some point the agent
just stops caring at least when you just
kind of lower than 1 in that case
immediate rewards are more important
than delayed rewards is what that means
so what is also sometimes said is that
if the discount is close to zero or
right at zero then you have a fully
myopic or nearsighted agent and if it's
close to one the opposite then you have
far sighted agents a discount factor of
1 is typically only used in situations
where you know for certain that there
are terminations that are episodes in
the bandit case for instance you could
interpret this as being an MDP where
each episode lasts exactly one step
there are no discounts there because
there are is no sync when she a lady but
more in general if each it
if each episode is guaranteed not to
last longer than say ten steps then you
might still not need a discount factor
because everything's still well define
however if if your problem can continue
indefinitely maybe you want to trade off
the immediate versus the delayed rewards
so this is enumerated here reasons so
one actual reason for discount discount
rewards there's also mathematical
convenience we'll see an example that
later in this lecture where we can see
that the speed of learning can actually
depend on this discount factor in some
sense your problem becomes easier if you
have a lower discount factor however if
you change your discount factor it also
changes the nature of the problem and
your solution might change so there's a
certain trade-off here another thing so
this is related to the mathematical
convenience it avoids infinite returns
at least if the rewards themselves are
guaranteed to be finite but if your
rewards are finite then a discount of
some of them if you're just gonna just
guaranteed to be lower than one let's
say it's a constant lower than one then
it's guaranteed that the return itself
is also finite in an undiscovered
setting this is not guaranteed if you
just think of a very simple MVP where
you only if one state and an action that
loops to itself you get a reward of one
on each step if you would use no
discounting the value of let's say it
will be infinite this isn't the problem
maybe for one specific state but let's
say you want to do control you want to
compare one to the other it might
actually be the case if you don't
discount that all of the states look
equal because eventually you'll get
infinite rewards in all of them in which
case there's no need to act in any
different way in random yes it's a very
good question can you offset by
subtracting let's say let's say a
certain value the reason I make slightly
different formulation there from what
you were saying there are algorithms for
the average reward case and if you look
at the average rewards then your value
is guaranteed to be bounded even if you
don't discount if the rewards themselves
are bounded and
also algorithms have been proposed for
the average reward formulation of the
problem essentially but or they're out
of scope for this for this course I
would say also because we as a community
haven't really figured them out yet I
haven't really figured out how best to
do average reward reinforcement learning
there are algorithms we've talked a
little bit about an algorithm called Q
learning there's an algorithm called our
learning which does something very
similar to what you said it basically
subtracts the average reward and then
looks at the offsets compared to the
average reward but these are a little
bit less well understood in short and
less commonly used in practice as well
doesn't mean they're a bad idea it might
be a good idea okay so as I said it's
important to note that the reward and
discount together define the goal and
that when you change your discount you
might be changing the goal now we talked
a little bit about a value function as
well in the first lecture so let's just
recap that to be sure that we understand
the value function of a states is just
the expected return I say just but you
have to be careful here because the
expected return of course depends on the
MVP but also on the policy so we have to
make that explicit in notation I hope
you didn't forget it anywhere on the
slides the the return itself can be
defined recursively because the return
is just is a cumulation of rewards so
one thing you could do is you could take
the first reward and then you could look
at the rest of the return and just still
call that a return which was which which
is what I did there in the second
equation but if you have take the
expectation of that the expectation of
the next return is essentially the value
of the expected next state actually said
it wrong there it's not the value of the
expected next phase is the expected
value of the next state these things
might not commute so
one thing that happens there which might
be good to note is the first equation
conditions on being in state s t and on
policy pi when I just put a PI there I
mean any action that is ever selected is
selected according to this policy pi PI
OS denotes policy in the second equation
in the same block there in the middle of
the slide I more explicitly said a t is
selected according to the policy this is
because I plugged in the value function
so the other actions are no longer in
here they're basically abstracted away
within the definition of the value
function at the next time X the next
state one thing we can now do and
somebody asked this question as well in
the first lecture we can write it down
explicitly using sums because we could
basically just enumerate as we did
before in the table all the
possibilities starting with the actions
and we could look at the probability of
selecting each action that's what the
first part means there the summation
over a is over all the actions and pi is
the policy and pi a given s just means
what is the probability of selecting
that action in state s for each of these
actions then we also sum over all
possible rewards and over all possible
next States and we look at the joint
probability of these things happening
and then we just get the value of that
reward plus the discounted value of the
next state and by definition this is the
same as the the equation before where
you may be noted or maybe even object to
the fact that I sum here are things that
might be continuous so feel free
wherever you wherever you want to
replace each of these sums with an
integral for instance if you want to
assume that rewards don't come from a
finite set but they can take any say
real number now you might want to
replace it with an integral but the idea
behind it is the same
now this recursion is a very central and
important thing within reinforcement
learning so it's good to understand it
and we'll see versions of this
throughout this lecture sometimes it's
more convenient to write down the
explicit form with the summation
sometimes more convenient just to stick
with the expectations but whenever you
write the expectation it's good to be
aware what you're actually conditioning
on what you're not conditioning on yes
there was a question yes a good question
what is this first pie a given s this is
this is our policy which in this case
I'm assuming can be stochastic in the
simplest case our policy will
deterministically select an action in
each state like one specific action in
this case this sum basically just
becomes an indicator that only selects
that action the pie a of s in that case
will be one for that action and zero for
all the other actions the formulation
has a written wrote it down here is more
general and it allows you to have a
stochastic policy where there's actually
probability of selecting many different
actions an example that is the epsilon
greedy policy that we discussed last
time where there is an epsilon
probability of selecting uniformly
across all of the actions and there's a
1 minus epsilon probability of selecting
the highest-valued
action according to some estimates that
you have in this case there's no
estimates yet this is just the
definition yeah the summation yes it is
the question was does this sum over the
probabilities sum to 1 it does so this
is essentially a weighted sum it's a
weighted average of the things that come
after but note that the things that come
after do depend on this a through the
probability on the reward and the next
value next state so you can't pull it
out even though it sounds to one
it's a very good question
Thanks
so we can do the same trick using state
action values rather than state values
so this is more or less the same slide
again but there's a few little extra
nuggets so first note that the first
equation up there now conditions on both
the state and an action we're basically
just saying we're still going to locally
expected return but where we're going to
commit a little bit further on we're
going to commit after the action rather
than after the state and this means that
this should also appear on the left hand
side and this gives us a Q value or an
action value which we can again roll out
but the first step I roll out there I
actually stopped slightly short I only
look at the value of the next state I
don't have an action there yet this is
because of the definition of the return
this is perfectly valid and we could
just stop there but what we could also
do we could go on further room we could
also consider the action in the next
state now if this action is selecting
according to the policy which it would
be then these things are equal and
that's denoted there at the bottom where
the value of a state is equal to the
weighted sum of the state action values
by definition and this is maybe most
apparent if you just look at the first
sorry not the very first other
definition but the first equation where
there's a recursion of the Q on V if you
consider putting the summation over the
policy before this then you basically
get back this equation so this is by
definition true but that means that we
could also write this down recursively
only in action values which maybe is
quite obvious if you think about it
because essentially one way to do to
think about this is that we're not
considering the state through to be the
primary thing here but we're considering
just a combination of a state and an
actually to be the primary thing and
then you could do the same trick that we
did before
where the value of the previous state
action value is associated to the value
of the next state action value through a
recursive
equation note that now the summation
over the policy is all the way at the
end side rather than all the way the
outside and this turns out to have some
implications sometimes for some
algorithms making one easier to use than
the other now we can also just write
this down fairly condensed lis using
matrix notation if there's a finite set
of states especially so what this
equation here means is the same is what
we had before
there's nothing changed here we're just
writing it down differently and a way to
interpret this is that this V this bowl
that V is a vector that contains the
value for each state as its element so
there's a finite number of states let's
say ten then this is a vector of length
10 and this vector will be equal to the
reward vector these are the expected
rewards under the the policy that we're
considering so all of this is policy
conditional and these bold at P is
called the transition matrix it depends
on the policy and in fact this is a
definition so I probably should have
added the policy to V as well because V
is self also depends on the policy
similar as before and essentially what
this means is you can see now these
transition these transitions you could
see this as a as a mapping you map to
pre the sorry the values at the next day
you get those by mapping the values of
the previous state through the
transition matrix I'll just
I don't expect you if you if you're if
you're not like group believing that
immediately or seeing in a mealy that's
fine
that's okay if you just bear with me and
believe that this is true this is
exactly the same as the equation that we
had before but just written differently
the summations are just written down as
matrix notation rather than explicitly
so why are we going here well it gives
us our food first solution method
because essentially this is just a
linear system of equations this is
already true before if you have a finite
number of states we just had a an
equation there that related the value of
one state to the value of potentially
all of the other states but so if it's a
finite number of states it's a linear
system of equations that we can solve
and one way to solve that and this is
why you went to matrix notation is in
this case it becomes very clear how to
solve it because you can just manipulate
the matrices in such a way that you can
write down the solution basically in one
in one go and the way we do that is we
get all the values to one side and then
we just multiply with this matrix in
front of the value which is the identity
minus the discounter transition
probabilities we multiply both sides
with that and then we get something on
the left hand side which is just your
value and something on the right hand
side that doesn't depend on your value
anymore so this is a non recursive
definition of the value and if you know
your transition probabilities if you
know this matrix if you know your
discount factor and if you know the
reward vector then you can just compute
this I say just but this is not actually
that commonly used in practice because
it's it can be computationally
bothersome and it doesn't doesn't allow
you to focus your computer and we'll
talk about that later but one way to
think about that is that in an MDP there
might be certain states that you
basically never end up in states that
are basically irrelevant to any
prediction problem that you might want
to do in that in that Markov decision
process this doesn't care or doesn't
know about that and we'll try to solve
that which means that you're basically
spending compute on states that you
might not care about in some cases this
is fine if your system is small enough
right maybe this is a good way to just
solve it but if your system becomes
bigger if you have millions of state say
the use of states then this becomes
unwieldy of course or even smaller than
that typically does now there are other
methods iterative matter methods that
will discuss the one that we'll cover in
this lecture is dynamic program
and then a later lectures will we'll
cover a Montecarlo evaluation which uses
sampling and temporal difference
learning which uses ideas from both
dynamic programming and from Monte Carlo
sampling when they tend to be much more
efficient in very large problems and
especially Monte Carlo and temporal
difference methods also extend much more
easily to the function approximation
case where there might be infinite
possible states are almost infinite
let's think again after the robot with a
camera input so another thing that's
maybe missing here this is just for the
prediction case which means that we're
evaluating certain policy this was the
bellman equality equation or sorry
development expectation equation
sometimes called which was this one that
we discussed but this just defines what
the value is for a given policy but
that's not actually what we're typically
after we want to optimize now the
optimal value function can be defined in
terms of these values by maximizing over
the policy it's very easy to write that
down it's very easy to say it it's not
that trivial to actually do it because
how do you how do you even reason about
maximizing or for policies because
policies are functions they're functions
from States to actions and there might
be many of them even in small MVPs there
might be many ways to map your states
into actions there basically is a
combinatorial explosion of ways you can
pick different actions in different
states because each combination of
picking and a different action in one
States might lead to different values
all over in the Markov decision process
for all other states all these things
are interrelated however it's good just
to note it exists we can define it and
this is the optimal state value function
in the optimal action value function
which are very symmetric can be defined
as just maximizing or for these policies
these optimal values they give you what
the best possible performance is that
you can get in that problem they don't
tell you how yet but they do give you
what what that is
and then the Markov decision problem our
process can be considered Souls when you
find these values so just a little bit
of terminology estimating the value of a
certain policy either using state values
or action values doesn't matter we
typically we call this policy evaluation
and sometimes we call this prediction
because the sense you're making a
prediction and what about what happens
when you follow certain policy
prediction of course is maybe a more
general term that's used for other
things as well it's a bit overloaded but
you'll see it sometimes in the
literature estimating the optimal values
on the other hand it's sometimes called
control because these can be used for
policy optimization essentially if you
have these values it's very easy to cut
until you do the optimal control to find
the best possible policy of course both
of these are prediction in a sense one
is predicting the value of a given
policy one is predicting the value of
the optimal policy but there's just this
terminology to the use that is
associated to these things which is good
to be aware of now this means that
there's actually four bellman equations
or actually that's not implied by the
previous but there are four different
bellman equations if you just look at
first two the first one we covered
before this defines the value of a
certain policy for all states the second
one defines the value of the optimal
policy and this is done by maximizing
over the actions and then bootstrapping
using that same value this may not be
immediately obvious and I encourage you
to read the chapters of system umberto
where this is also more step-by-step
laid out but this is this is a known
equation that is known as the bellman
optimality equation all of these are
sometimes called bellman equations and
then the first one you could call may be
a bellman expectation equation and the
second one you could call bellman
optimality equation and then there's
those counterparts for the state action
values where similarly oh sorry there's
a typo
the final Q there behind the max a that
BQ star not Koopa I'll fix that before I
upload the slides so these are
recursively defined in terms of
themselves and you can show that there
is no there's no possible policy that
has a higher value than the value given
by this definition that's what you can
show that means that the policy
associated with these things is the
optimal policy especially for state
action values these then the policy is
very easy to get which I'll cover later
but essentially just pick the highest
valued action in every state now there's
equivalence between these state values
and these action values I already
covered the first one which says that
the state value for a certain policy is
actually equal to the weighted sum of
the action values where you weigh by the
policy essentially were just
conditioning we're conditioning either
honoring the only a state or state and
an action but when we do that and when
we do that we have to correct for how
likely it was to select each of these
actions the optimality equations or the
definitions of the values for the
optimal values are similarly related to
each other but the summation over policy
is replaced with a max we select the
highest valued action and this will give
you the optimal value from that state
this is maybe obvious in hindsight if
you believe that the state action values
are any the optimal state action values
then what's the highest possible value
could get in the state well just pick
the highest one and indeed follows also
from the definitions of these that is
this is true this means that especially
if you if you have a model as we'll
discuss in this lecture then these
things are somewhat interchangeable
because you have these options of first
estimating state action values in just
picking the highest valued action or you
have the option of estimating the
optimal state values and then just
planning one step ahead and then
selecting an action that requires you to
have to trim all oh but that's C
something that we were making here
anyway in practice a lot of times when
we want to do control we use these state
in values just because it's very
convenient to be able to pick according
to these values so another way to talk
about optimal policies is to define a
ordering over policy so that we can
simply write a policy is better than the
other policy well when is a policy
better than the other policy this is
true if the value of that policy is at
least as good as the value of the other
policy in every state so in both of
these cases it's it's kind of important
that this is a greater or equal then
these correspond to each other a policy
is considered greater or equal to
another policy if it's at least as good
as the other policy in all states and
then there's a theorem that for any mark
of decision process there exists such an
optimal policy that is better or equal
to all other policies according to this
ordering and all of these optimal
policies if there are multiple there
might just be one but there might be
multiple optimal policies achieve the
optimal value there's only ever one
optimal value this is uniquely defined
but there might be multiple policies
that attain that value one way to very
easily understand that is if you're if
you have an MDP in which all the rewards
are zero it obviously doesn't matter
what your policy is so all policies will
be optimal but there's still one optimal
value which in that case is zero so if
we write down the optimal policy if we
denote that by I star then we can
basically say okay what is the value of
that Pie star and this turns out to be
exactly that V star as defined by
development optimality equation these
things are the same so sometimes people
might also say that these stars just a
shorthand for a V of Pi star both of
these are true
so how do we find such an optimal policy
then I already mentioned this so you can
just maximize the optimal value function
state is a state action value function
if you had that there's a hard part then
becomes finding that there is always a
deterministic optimal policy and this is
the case because if there's multiple
actions that maximize you can pick
arbitrarily so you can pick randomly or
amongst the optimal actions or you could
just pick one make say the first but if
you pick the first this will become a
deterministic policy this is only true
for Markov decision processes if you
have a process that is partially
observable if my neck should be optimal
to behave randomly because you might not
be able to know exactly what the state
is and in that case it might be more
helpful to not commit fully to an action
but we'll talk about that again later so
the only point here is that it's easy to
get the optimal policy from the state
action values but we still haven't
talked about how to solve and turns out
this a little bit harder because so
previously I gave you one solution but
this was only for the prediction case
for the predicting value of a policy and
that's a linear system of equations but
the bellman optimality equation has this
max over the actions this means that
it's nonlinear and you can't do the same
trick it's not a linear system of
equations are nonlinear system of
equations and that's typically harder to
solve so what we typically do is we use
iterative methods rather than try and
write it down in one go rather than
trying to find an analytic solution
which may not exist instead we're just
going to iterate our way to the solution
and if we use models as we will do in
this lecture and this is called dynamic
programming and I'll talk about value
iteration and policy iteration which are
two ways to to get towards the optimal
policy and eventually find it and in
future lectures we'll talk about
the methods that the Muslim witch use
samples rather than the true model but
otherwise use similar ideas so I found
this to be a good quote in terms of what
this dynamic programming mean where does
it even come from I'll just I'll just
read it from the slide in the 1950s
where were not good years for
mathematical research I felt I had to
shield the airforce from the fact that I
was really doing mathematics what title
what name could I choose I was
interested in planning in
decision-making in thinking but planning
is not a good word for various reasons I
decided to use the word programming I
wanted to get across the idea that this
was dynamic this was time varying I
thought let's kill two birds with one
stone let's take a word that has a
precise meaning namely dynamic in the
classical physical sense it is also
impossible to use the word dynamic in
aperture pejorative sense try thinking
of some combination it will possibly
give it a pejorative meaning it's
impossible
thus I thought dynamic programming was a
good name it was something not even a
congressman could object to so I used it
as an umbrella for my activities this
was by Richard bellman I slightly
paraphrase just for conciseness the
point here is that it's a kind of a
weird name for a process that's actually
not that complex and some people they
get confused by this name especially
later on when programming more or less
started to meme what we also might call
coding whereas in the past and more
mathematical sense you'd had things like
linear programming which was more about
algorithms and how to solve things and
dynamic programming is programming more
in that sense but still dynamic
programming is not a very intuitive or
informative name for what the process is
so don't get thrown by that it's
basically by design it was meant to
obfuscate maybe more so we're just going
to stick to that there's a simpler
definition in certain embarrassin which
basically just says dynamic programming
refers to a collection of algorithms
that can be used to compute optimal
policies given a perfect model of
environments as a Markov decision
process and we'll discuss several
dynamic programming methods to solve
Markov decision processes all of which
consists of two important parts one is
the policy evaluation part
and when I suppose the improvement part
in some cases these parts are kind of
intertwined as you'll see but these are
two important things to keep in mind
which correspond to what we were talking
about before so this is a good break
point so I wanted to stop here for a
moment because this has basically been
more or less definitions up to now just
to make sure you're on board with the
one exception of the matrixes where we
solves one thing but otherwise we
haven't really talked about solution
methods so I wanted to take a five
minute break and then I'll talk about
dynamic programming to solve these MVPs
so first we'll talk about policy
evaluation which just means to estimate
that value that we're already talking
about before the states value to start
with I already gave you one solution
with the matrices but we now we're going
to talk about a different way to come to
the same conclusion and the benefit of
this way is that it will also apply
later on when we try to apply this to
finding optimal values but for now we're
just talking about estimating the value
of a given policy so PI you can see it
can consider given for now and we're
just estimating that value so the idea
is quite simple we have this definition
here the bellman equation and this is
something that is by definition true and
as I said it's actually a system of
linear equations that you could solve
but the idea of dynamic programming or
at least in this case of this policy
evaluation algorithm is to initialize a
first guess v-0 for instance you could
initialize the value for each and every
state to be 0 and then we're going to
iterate essentially this equation we're
going to assign values which we now
subscript with a K rather than a PI to
indicate that this is an estimate at
that time and the way we're going to
assign that is we're basically going to
take one step and expect it step using
the true model and we're going to
bootstrap that's what is this goal I'll
repeat that later on the value at
iteration K the previous iteration at
the next state and we're going to use
that to find new values k plus 1 now a
first thing to note here is that if this
operation this is a well-defined
algorithm right you can do this you can
each time update for
all of the states just think of it
happening for all of the states at the
same time you can just apply these
updates and get a new VK plus one now
the first thing to note is that if it
happens that your values didn't change
because of these updates then by
definition you have found the true
values that doesn't tell you that this
will happen but it tells you that if it
happens that you found the right values
it might be less clear that this
actually goes in the right direction
that you'll find me values eventually so
it's useful to step through that and
what I have here is essentially a short
proof sketch that this is a sound
algorithm so the question here is does
this converge when I say converge I mean
does it eventually find true values and
the answer is yes under appropriate
conditions and one such condition that
helps it converge is when your discount
factor is lower than one which is what I
use or in the proof sketch and I'll step
through this and the ideas are simply to
pick the maximum gap between the value
at the iteration k plus 1 and the true
values where I maximize overall States I
could have picked different things and I
think the book has a different version
of this this is one way you could you
could do that and then the first thing
that we do is we just expand the
definitions so the definition of V K
plus 1 as given on the previous slide is
essentially the expectation of the
immediate reward plus the discounted
value at the next stage according to the
values at iteration K the previous
iteration that's what the first line
says there and then there's a minus part
which is the true value that I've
expanded which is also defined as the
immediate reward plus the discounted
value at the next state according to the
true value that's my definition of the
bellman equation this is true but then
we can note that there is an expectation
there it has a reward but the
expectation is in both cases over the
same things its conditioned on the
states there's an expectation of the
reward can the
on the same policy because we're
considering a certain given policy that
means that these rewards cancel they
just disappear it's the same thing - the
same thing the other thing I did are at
the second equality is I also put both
of these things into the same
expectation which you can do because the
expectation is about the same things so
I've just got rid of the reward and just
wrote it down slightly shorter as one
expectation rather than two different
expectations - each other because the
expectations are all over the same
things now what's left here is the
discounted value at the next state
according to either the value function
at iteration K or the true value
function but the discount factor we've
assumed is smaller than one we can just
pull that out it's just a number we can
pull that outside of the expectation we
can also pull it outside the max because
it's just a number which means we now
have a discount it's max over all states
of the difference between the
expectation of the value at the next
state - the true value at the next state
and at the final step I basically get
rid of the expectation now why can I do
that that's because this expectation
inside at the previous step before the
inequality is basically a weighted sum
other states but the weighted sum other
states you could always just pick the
max and then you get this inequality
that the difference between that is
smaller or equal to there's a different
version that shows why these things
converge in the book I encourage you to
look at that one as well but what this
means is that all the way there at the
beginning we had something and all the
way there and at the end there we have
something that looks very similar but
it's the difference at the previous
iteration times this discount factor now
what happens if you apply this again and
again because the discount factor is
smaller than one this means that this
difference will go to zero and that
implies what's on the bullet point there
that in the limit if you iterate this
over and over again you find a true
value function yes
I hope it's true
why the last step is to my argument that
I just gave is that the expectation here
basically just implies there's a waiting
over states but if you look at the
waiting over set and the waiting is the
same in both cases so we have a way
that's weighted average of states on one
hand weights are of state values on one
end a weighted average with the same
weights of state values on the other end
and the claim here is that the
difference between those two is always
going to be smaller or equal to the
difference of the maximum gap there
which is what the final thing that
defines in worst case there's an
equality which is fine because we still
have the discount factor so we're still
converging even if that final step is in
equality in general it might even be a
little bit faster than that it might go
down a bit faster so this is to give you
an intuition it also shows you something
that I mentioned before is that the rate
at which these algorithms find the
optimal solutions might depend on your
discount factor this makes intuitive
sense as well because if your disk
infect or is high this means that you're
looking further into the future so in
some sense this should be a harder
problem to solve because you have to
take into account longer delays and
potential rewards that you might receive
and this is kind of in general true that
if we set the discount factor higher
when we model the problem it tends to be
a harder problem to solve so this
doesn't cover the final horizon case
this basically covers the episodic sorry
didn't the continuing case where we have
a discount factor maybe we have no
terminations you could make a similar
argument for the finite horizon episode
a case where there's always a limit to
how far you can go
maybe there's deterministically going to
be a team determination within a certain
amount of steps or maybe just
sarcastically you terminate sufficiently
often and then even if you don't
discount you could still show that these
things converge but the argument there
is slightly slightly more complex the
easiest one to show is with the discount
factor
okay so what does that look like let's
consider a very simple setup this is a
grit world and the axes are just up-down
left-right we've seen similar things
before and let's assume there's a minus
one on each transition and there's one
terminal state which is basically in two
places this is how rich likes to
formulate these things if you enter one
of these shaded corridors nearthe
process terminates that mean this means
there will not be a next state so what
you can do is you can apply this
iterative procedure which we're going to
apply on all of the states at the same
time we're going to update all the state
functions at the same time and the
policy we're conditioning on so the
policy we're trying to evaluate we've
picked the random policy which is
depicted here on the right hand side as
just having these arrows pointing
everywhere this means that you have are
equally equally likely to go in each of
these directions in addition we fix an
initial value function just arbitrarily
to be zero everywhere so at K a zero
before the first iteration if you will
all of the values are initialized at
zero what we then do we apply this
iterative procedure once we don't update
the values in the corners even though
they're in the in the picture there
because these are terminal states there
are no actual states that you want to
know the values off they're defined to
be zero in a sense that's the that's
another way to view that because they're
terminal but all the other states are
updated and at K one all of the state
values are minus one can someone explain
why
yeah yes
and in addition so the answer was we
took one step in all of them and in
addition the reward was defined to be
minus one so in all cases we saw exactly
one reward and it was minus one this is
exactly true and then we bootstrapped
we used the estimate at k0 to estimate
the value at the next state which was
defined to be zero so now let's repeat
that process let's do that again for all
of the states we're still just
considering your random policy going
anywhere and then we're going to do
another step of this policy evaluation
at K equals two
this gives us a new value function which
has a lot of two's so I could repeat the
question but the answer will be because
we took two steps and it will be two
because we're not discounting as well so
the second step like the second reward
is considered equally important as the
first reward but what we're essentially
doing here is we're bootstrapping on the
state value estimates so think of the
value for instance all the way in the
right upper right corner there first
it's zero we take one random step by the
way if you bump upwards here you just if
you try to go upwards in that state
you'll just bump back into the same
States by definition of the tree of
dynamics but if you take a random action
from the upper corner it doesn't really
matter you just get a minus one and
you'll bootstrap on the value of the
next state but the value of the next
state is zero everywhere
the first time you do that so the first
time the value of the upper right states
will become one after one update the
next time you do that same principle
applies you get a minus one reward and
you add the value the current estimate
of the value at the next state you end
up in it doesn't actually matter from
this top right corner it doesn't
actually matter which day you end up in
they all have the same value which is
now minus one so the new estimate will
be minus one for the reward minus 1 for
the next state is minus 2
but there are now also a couple of
states that don't have the value of -2
so can somebody explain why yeah yes yes
exactly true so the only reason this
says 1 minus 1.7 is because we rounded
but it's actually 1 point minus 75
because there's essentially three states
the random policy might end up in that
have an estimated value now minus 1
exactly like you said we do always get
the minus 1 reward but then the expected
next state value is minus point 75
because there's three that go to minus
one and they're still the terminal state
that had a value of zero which remains
at a value of zero always because it's
the terminal State
so now you see this discrepancy between
the states that are closer to the
terminal States and the states that are
further and we can repeat this process
because this is still just a very rough
estimate of the values but where you can
repeat that for case three the values
become a little bit different then we
repeat it more and eventually when we
have infinite iterations will get a
value function which has these minus 14
20 22 18 different values for different
states and these will be the true values
for this problem and the semantics in
this case of these values is this is the
negative amount of steps on average that
it takes before your random policy ends
up in one of the terminal states because
of the way it's set up there's no
discounting you get a minus one for each
step so this is just counting steps
until termination so an expectation from
the upper right corner apparently it
takes 22 two steps before you hit a
terminating state on average
now there's one thing that may be
interesting to note and I'll get back to
that as well is that on the right hand
side here we started up with just
depicting the random policy now what's
happening here on the right hand side is
basically completely separate from
what's happening on the left on the left
we were just doing this policy
evaluation but on the right what we're
doing is we're taking a one step look
ahead using the current estimates and
seeing what policy that gives us so we
notice after one step the states near
the terminal states
they already points towards the terminal
State can someone say why why would the
optimal policy be or why would why would
the policy condition on that value
function be to go to the terminal State
yeah yes so conditional doing a one step
look ahead so I did it
I did a fairly long sentence they are
pre facing the question conditioned on
doing the one step look ahead in those
states its optimal to go towards the
terminal state condition of your current
values in all the other states just
doing a one step look ahead isn't enough
to actually see the terminal state so in
those cases cases the policy still
doesn't care if you do another step you
see that more and more errors start
pointing towards the coroners and when
we repeat that actually the policy
doesn't change any more after the third
iteration in all of these cases they
basically point towards a shortest path
towards the corners we didn't ever
actually execute any policies we didn't
even consider evaluating this policy the
values there on the left bottom left are
the values for the random policy but you
could now consider using those values to
act and in fact if you would do that it
would already give you the optimal
policy in this case because it's such a
simple problem essentially
however this points towards what's
something that we can do in general
which is policy improvements and the
example saying she shows that we can
evaluate a random policy and then maybe
consider changing our policy and then in
that case it would already be optimal in
general this is not true but we can
still apply the principle we can iterate
by saying she's picking a new policy by
picking the greedy action according to
the action values according to that
policy so let's assume we've used the
the the process of policy evaluation to
find Q PI in a previous example we did
it for values but you could do the same
for state action values and you could
repeat the process find the actual
optimal values and then you could
consider a policy that does better using
those values now there's a hat
dangling let there D you should just
ignore but the idea is here to iterate
that and what we can do is take the new
policy we could just of course just stop
there we could say hey we now have a new
policy in this simple example it was
already put optimal but we could also
change the policy and then repeat we
could estimate the value of this new
policy the value of this new policy may
and in general is different from the
value had before and maybe if you are
greedy to the solution to the value of
this new policy again you can improve
yourself and you can actually show that
if you do this this way if you're greedy
with respect to your current value
estimate then the actual value of the
new policy will in fact be better than
or equal to the value of a previous
policy I don't give a proof here ignore
the dangling nuts again but this is this
is true so there is an intuition here
which is not a proof just an intuition
but let's assume that at some point we
did did an iteration of this we had a
certain policy we
evaluated this policy we have a certain
value function and then we pick a new
policy that is better according to that
value function this policy we did
depicts as PI new and now we consider
what is the value of that policy let's
say we redo the whole policy evaluation
to find the actual value of that policy
and now it turns out it's the same then
by definition this equation holds that
in each state's by the way we picked the
new policy by being greedy with the one
step look ahead you can write it down
similarly for the state action values
but we've picked the policy by being
greedy with the one step look ahead but
if the value didn't change by definition
this must be true which means we found
your optimal values this is the
intuition that if this process ends this
process of evaluating and improving and
and evaluating and improving then you've
ended up at the optimal value and
therefore also the optimal policy so the
intuition here is that if you improve by
picking a greedy policy with respect to
your current value function then either
your new policy must be an improvement
which I stated but didn't prove but this
is true or it must already be optimal so
that sometimes depicted a little bit
schematically like this you start off
with some random value function and a
random policy and maybe the first step
that we do is policy evaluation that's
depicted there on the left as this arrow
pointing upwards the line there at the
top you're basically at that line
whenever your value is exactly accurate
this is a very this is just an intuitive
picture there's no formal like semantics
to these things but just think of it
that way but if you do the evaluation it
basically means that your value will be
exactly accurate after some time now you
could then greedy Phi your policy you
pick a new policy that is greedy with
respect to the current estimates what
this will mean is you will not have
accurate values for this new policy by
definition you've only evaluated the
policy you had before but you've changed
your policy and the values may not be
the same that's depicted by the arrow
going down because right now you've
green
defy your policy but the value estimates
are no longer necessarily accurate but
then you could repeat that process go
back up again and evaluate this new
policy and the intuition here is that
these things actually
together they bounce towards a point
where nothing changes anymore but then
when nothing changes and that's what I
said on a previous slide by definition
we have found the optimal values we
found something that adheres to exactly
development optimality equation which
therefore means they must be the optimal
values another way to view that is like
here on the right it's basically the
same picture but we see this as a as a
cycle things where we have an evaluation
step and then we have an improvement
step and we just repeat and repeat
repeat until these things bounce back
without any change your policy is no
longer change your value is no longer
change this means they must be optimal
otherwise there could still be change
this process of evaluating and then
improving and evaluating and improving
again is called policy iteration and
it's a very central and important
concept within reinforcement learning
and there's many ways to do that so the
principle apply is more general than the
specific algorithm we discussed so it's
good to keep that in mind now to give
another example of that one a little
less trivial than the previous one let's
consider a fairly still simple setting
but more complex in the previous one
where there's two locations which can
each hold a maximum of 20 cars and
essentially the agent here is the is the
rental dealer or the person in charge
wants to make money by renting out cars
and the actions are to move cars from
one location to the other this costs you
a little bit in this case minus two your
reward is minus two dollars which
basically means it cost you two dollars
to move a single car and you can move at
most five so there's five different
actions or six if you don't move any and
the reward is otherwise apart from the
minus to ten dollars for each available
car at a location that is rented
we also have to define a discount factor
which we fairly arbitrarily set to point
nine which means you want to make money
now rather than later the transitions
are more interesting in a sense and it's
harder the reason through which shows
that you can use these things without
knowing the solution yourself because
the cars are returned and requesters
randomly we assume according to some
Poisson distribution in this case which
is not too important but the main thing
is that these distributions are
different for the two different
locations so an intuitive thing if they
wouldn't be different will be just
abandons the cars and then you should be
fine
but if these are different it's unclear
what the possible what the optimal
solution is so in the first location the
average number of requests is three and
the average number of returns is three
but the other location the average
number of requests is four and the
average number of returns is only two
because sometimes people pick up a car
there and then bring it up at the back
at the other location this sometimes
happens in real life as well so this
means that there's an imbalance and one
location will more quickly deplete its
cars than the other and this means that
you do want to sometimes ship cars from
one to the other now we could start off
with I'll have there's a there's a
depiction of this full problem let me
just grab that because I do have it it
should have been in the slide I thought
it was in the slide it's got a previous
iteration here we go
cheating a little bit doesn't want to
open in the preview so this is the
process of finding that solution using
policy iteration so we have a a policy
here that is depicted as a big matrix
where the axes are shown there at the
bottom there's a number of cars at one
location and there's a number of cars at
the other location this is basically for
you overnight because your decision is
overnight to move this many cars and in
all of these different situations you
might find yourself all these
combinations of numbers this shows how
many cars to move from one location to
the other if it's a positive number
you're basically moving from one
location to the other if it's a negative
number it means you're moving in the
other way the first policy basically
just says don't don't do anything and we
can estimate its value and when we do we
just estimate the value of that policy
and then we pick the greedy action
according to that first estimate and
that's when you get on there PI 1 there
and we see that an interesting policy
already emerged where we're very often
shipping cars from one location to the
other a maximum amount of cars that we
can ship which was five we are shipping
them except if there's already a lot of
cars there if we go further along the
axes at some point you stop shipping
them because there's already so many
cars at the other location this is just
after one sequence of suppose the
iteration but this is assuming that you
basically did the full policy evaluation
is an inner loop so you found the true
value of this first policy and then you
went greedy with respect to that if you
repeat that process you can see that the
policy basically becomes closer and
closer to what you get at the end which
in the
case is only after a few iterations very
quickly it approaches a certain form
which I couldn't have been able to
predict just by seeing the problem but
this turns out to be the optimal policy
and at some point it stops changing you
can already see that the difference
between the first and the second is
quite big and then the difference
becomes smaller and smaller each time
you go and then you can also look at
what the value is which is depicted
there as a three-dimensional plot for
each element of your state there's a
number
so we've depicted the state here as a
two dimensional thing the number of cars
in one location number of cars in the
other location and then the value just
becomes the height of that plot this is
the value associated with the policy
there at the end in the noted PI for so
the thing here to note is that this
thing also in this less trivial problem
converges to apparently an optimal
solution and it will be often you can
prove that it will be and that the
optimal solution is sometimes quite
non-trivial it's not something that you
can easily reason through without doing
the computer there was a question yeah I
forget the details so it might be so the
question is well what did that what do
the dynamics do when cars get returned
when the location is already at this
maximum yeah so you could basically
there's there's different versions of
this is you could define right there
this is basically a question about what
are the exact state dynamics and I must
say I forgot I don't know whether it
just caps a 20 or whether it goes beyond
20 but only when people return but you
can't bring any to give it a beyond 20
and these are any two different problems
which we'll need maybe have slightly
different solutions and I don't remember
which one this is exactly for but the
process is the same so you could solve
both of these with the same solution
method thank you yeah
why would their arrows pointing out the
box let's go back to sorry so Rebecca T
the oh yes so why are there arrows
pointing at the boundaries here that's
because you can't tell from your value
function that it's supposed to stay in
the same state so at the very first case
if you're in the right upper right
corner let's say each action that you
that you take will give you a reward of
zero now you might argue even then it's
maybe silly to say in stay in the same
situation because you might still know
you want to escape but actually what you
did well if you do assume that you made
an assumption about the problem which is
that the rewards are generally negative
in reality there might be a state where
you get a positive reward but only if
you go to the same state again and again
and in that case you would actually want
to stay in the same state so as far as
the algorithm is concerned are concerned
they're completely indifferent to these
things we're not putting that prior
knowledge in that in this case the
rewards are all negative if you would
put that prior knowledge in for instance
one thing you could do is you could say
actually you really never want to
transition to yourself because that's
just generally a bad idea but that's
actually true when your rewards are
generally negative on that transition if
they would be able to be positive then
it might actually be the optimal thing
to do this happens as well in real
problems in bigger problems because
sometimes it's much easier for an agent
say say an agent playing an Atari game
it might just learn to stand still
because it's safer to stand still then
to pursue a reward that is high but it
doesn't know it exists and it doesn't
know how to get it so this does indeed
sometimes happen you could call this a
suboptimal solution say in policy space
in dynamic programming you're guaranteed
not to find that right we can iterate
through everything and we can guarantee
you to find your policy but in the
general case where
you have partial observability and all
the other problems with approximations
then in the connection happen that you
get stuck it's a good question Thanks
so very well I'll put in the slide that
I went to the other presentation for in
this one as well before I upload so the
question was okay so beyond policy
iteration so far we've assumed that in
each of the evaluation steps of a policy
we went for the full value for the
policy we were considering at that time
a question is though you do you really
need to do that because it means we
basically have an inner loop we have the
outer loop of the policy iteration which
in each step greedy fires your policy
with respect to the current values but
it has this inner loop of policy
evaluation which itself might be
expensive so there's a question do we
need that and how much of it do we need
so for instance should we be able to
stop when we are close let's say your
values are updating and at some point
they seem to be updating less the
difference between each iteration
becomes a little bit smaller and you
might reason maybe I'm already getting
close maybe that's enough maybe you
should just stop there and not have the
small marginal returns of continuing so
for instance one way to implement that
is to have a threat threshold on the
change and to stop when the threshold
when your changed because it falls below
that threshold alternatively you could
also just say I'm going to do a certain
number of iterations I'm just going to
pick a K and I'm going to do K
iterations of evaluation before I then
do my policy improvement step in the
small grit world.we we looked at before
case three was already sufficient to
achieve an optimal policy this is
essentially because each of the states
was at most three steps away from the
terminal States but of course this was
only considering one step of fullest
iteration if you do multiple steps of
policy iteration maybe each of these
steps doesn't have to have that many
policy evaluation steps so for instance
why why not update the policy on every
iteration if an iteration of policy
evaluation truly move your value towards
the true values
can we not immediately use that to
immediately improve our policy and then
continue for the new policy rather than
waiting a long time before we finally
update our policy and finally start
considering his new thing turns out this
is required for into something called
value iteration but in order to get
there I'm actually going to do a
different thing I'm going to go back to
the same way we obtained policy
evaluation I'm just going to take a
bellman equation but instead of taking
the bellmen expectation equation I'm
going to take the bellman optimality
equation and I'm just going to turn that
into an update so to remind ourselves if
there wasn't an arrow there but an equal
sign then we could have the value on the
left hand side be V Star and evaluate
the right-hand side internally also be V
Star and this would just be your
definition of your optimal value so a
similar principle that we had for the
policy evaluation case applies here that
whenever this update stops doing
anything we must have found the optimal
values but interestingly we're not
reasoning about policies here explicitly
we're just finding the optimal values
basically immediately in some sense now
I haven't proven that this converges but
it actually does to the optimal values
through like a similar proof you could
do that as I showed before but now note
that this is actually equivalent to what
we were just calling policy iteration
where we just take one policy sorry one
value estimation step and then
immediately agreed if I we've just
written it down in one equation the
expectation there is essentially what
the policy Valiant if evaluation does
when you do exactly one step of it and
the max a there is the sense you were to
follow the improvement step does it
takes the greedy action with respect to
the new policy we're just doing it in
one in one step rather than in two so
these are equivalent and this is an
example of what happens if you do that
so this is similar to before this is
shortest path problem but in this case
we're not evaluating say a random policy
we're just going to IVA we're just going
to apply this value iteration idea sorry
this algorithm
called value iteration for historical
reasons it's just what it is so it's
sometimes these things are a little bit
hard to get in your head
sticky because again the name is not
that informative but this is an example
of applying that process that iterative
process so I'm sorry the indexing is
slightly unfair what we previously
called the zeroeth value here is called
the first value it's just in mixing by
run or indexing by zero so we just
initialized our value at v1 to be zero
everywhere this is an arbitrary choice
that we ourselves made the value of the
terminal States in the upper left corner
is pegged to be at zero by definition
terminal states have a value of zero
your life stops at the terminal states
you cannot get more rewards it has a
defined value of zero then we apply a
value iteration first we get a similar
thing that we had for the policy
evaluation case we just get a minus one
everywhere because we've defined the
reward to be minus one on each step this
is because we want to do shortest paths
we want to basically get our values to
be the negative amount of steps it takes
for your optimal policy to get to the
termination not at the next step the
value is nearly nearly termination don't
actually change any more they were at
minus 1 they're still at minus 1 this is
because we're considering the maximum
action that you can take from this state
we're no longer considering just a
random policy we're looking at all of
the next States and from this state next
to the thermal State the most promising
state is the thermal State which is a
value of 0 all the other states in
comparison look pretty horrible so you
don't go there so the value remains at
minus 1 and it stays actually that that
minus 1 throughout and if you do a
number of iterations here you find it
eventually you get these values and we
are here they won't change any more this
is undiscounted so again the semantics
now if these values are just a negative
number of steps until you reach the
termination but they're no longer the
negative expected steps under a random
policy these are now the the actual
steps you take when you take the
shortest path to the terminal State
because we're using value it's array
rather than policy evaluation this is
just an intuitive way to show you that
the process works and it works more in
general and then here you could
essentially stop because you would
notice if you would do it again you
would notice that your value function
hasn't changed and then by definition it
means that this is then the bellman
optimality equation because if k plus 1
the value k plus 1 is equal to the value
at k èk that means they must both be V
star which means you must have found
your phone values so it's fairly easy to
check if you just repeat this process
whenever it stops changing altogether
you found the optimal values in general
this can take a while for big MVPs so
you might also just stop a little bit
short of that and be ok with slightly
suboptimal values but still probably
much better than the arbitrary values
then you started off with so we've
discussed a number of methods now just
to recap that all of these are
synchronous dynamic programming
algorithms with which I mean that so far
we've just basically considered updating
all of the states on each step or at the
same time when we do policy evaluation
you will just update the values for all
of states and when we do the improvement
step we just make the policy greedy in
all of the states at the same time and
this is sometimes called synchronous
because we're doing a synchronous update
across states now for a prediction case
if we just want to do policy evaluation
the associated bellman equation is the
development expectation equation which
doesn't have a Max and the algorithm we
can use for that it's policy evaluation
and we can iterate so you could call an
iterative policy evaluation maybe the
problem you could also say is policy
evaluation rather than prediction and
then the algorithm is the dynamic
programming iterative policy evaluation
for the problem of control which is a
shorthand for saying the problem of
finding the optimal policy we can again
use the bellmen expectation equation to
do policy evaluation with or the
iterative version thereof and then we
could gree defy after each each time we
find found a better value
we've discussed that we can do that
waiting all the way till the end we
could fully evaluate a policy and then
greed if I or you could stop earlier and
then greed if I this process is called
policy iteration if we want you do
control we could also stop short after
exactly one step of policy evaluation
before we then improve our policy and
this process is called value iteration
and you associate the bellman equations
development optimality equation because
value iteration can also be considered
just turning that equation into an
update now these algorithms can be based
on state value functions as we as we
mostly discussed and then the complexity
is of order number of actions times the
number of states squared one way tune to
understand that is that there's this
transition matrix which goes from States
to States which is therefore number of
states by number of states so it's the
number states squared in order to do
each time one of these next steps
another way to say the same thing to
understand why this is order of states
squared order of number of states
squared is that for each state we have
to consider the possibilities of n
ending up in each of the other states
now of course in practice there might be
MVPs in which you don't necessarily have
to consider all of them so there might
be in practice cases where you can maybe
mitigate the compute do it slightly
cheaper but in general this is the
compute you need you can apply the same
principle to state action value
functions in which case the complexity
become slightly higher because you have
to do this for each action now rather
than in addition to the number of states
you're just all so essentially storing
more stuff you have estimates for states
and action pairs now rather than just
state values now the number of states is
more typically the bigger number so the
difference between those two things is
typically not that big of concern the
larger concern is that if your number of
states is big then this squared might be
too big to fit in memory if you have a
million states a million squared is a
really pretty weak number you don't want
to necessarily consider all of them
there's a solution or partial solution
to that which is to not do everything
this kind of sounds like a sensible
thing maybe you don't want to consider
all of the states all of all of them at
the same time and this can be done and
it's this is a sound idea and there's
different ways you can do this and I'll
discuss a couple so this in general is
called asynchronous dynamic programming
which is essentially just means we're
considering maybe one state at a time or
a couple of states at the time rather
than everything at the same time in
parallel always and this can
significantly reduce the extra compute
you use this is still guaranteed to
converge and the same conditions as the
synchronous dynamic programming is
essentially guaranteed if you do
continue to update everything so for
state value functions you do need to
consider all states basically often
enough in total at the end doesn't mean
you have to consider all of them at the
same time but you just have to
eventually get to them back and and and
do some updates now there's I'll just
discuss three simple ideas for
asynchronous dynamic programming these
are probably fairly intuitive first one
is in place which is a kind of a simple
idea that you could do the updates maybe
slightly faster than we were doing
before by avoiding storing two copies so
thank you what we did before we had a
new value which was defined in terms of
the old values and we would essentially
assign these things but note that the
state that were bootstrapping on the
state that we're evaluating is the old
value might be one for which we already
have a new value because we're already
computed that if we don't actually do
them in parallel but we just go through
all the states there's going to be one
state that you've done first so the idea
is very simple that whenever you
consider a state you just immediately
override that value and then you use
that it makes the notation slightly more
complex because before we had a V at
iteration K and then we considered
updating all of them to the viet eration
k plus 1 and we knew that for all of the
nember they were using the V at
narration K as their state estimate in
this case there would actually be all
these intermediate versions things
you've already updated things you
haven't yet updated but whenever you
start updating something you'll just use
the most recent value
and this should just be faster and it is
so it's a simple trick but it does speed
up your computer in this case we're
still doing a full sweep across the
states so this is still a little bit a
little bit synchronous in some sense
it's a little bit compute-intensive so
another thing we could do is we could
pick specific states to update and one
idea here is to use the magnitude of the
error we're essentially going to update
the states that were most wrong in the
past so you could you could essentially
keep the the thing on the left this max
a expected reward plus discounted next
value this is the target for a value
iteration updates so you could take that
from your previous iteration because we
might be doing the in-place stuff I
didn't actually index them with case or
K K minus ones but we're just you might
just consider what was the size of the
update I did in the past for this
specific state state s and then we just
store that somewhere and if the update
was big a sin she will return to this
quickly we'll try it again because we
think maybe there was a lot to learn if
for this specific state there was
basically no updates we might assume
maybe that values already quite accurate
maybe that's not considering that
quickly again and this is called
prioritized sweeping because we're still
sweeping across the state space but
we're doing it with a priority which in
this case is given by the previous
updates there's different ways to
prioritize you can come up come up with
your own ways to prioritize as well
another way which came up in the first
lecture as well someone you mentioned
why don't you just and then you start at
the end and then go backwards and that's
a need also a valid algorithm you could
start at the end and then go backwards
because then you're always bootstrapping
on things you've already updated which
is also a good idea this is in some
sense similar to the idea of
prioritizing except here we're giving a
different measure and this can be
implemented quite efficiently by the way
by using priority queue there's just
ways to do that and final finally one
thing that you could also consider is to
focus your compute even more
on the things that matter and this is
especially relevant if you have an
actual agent that is moving around in
his MVP and this agent will be in a
certain state and if that's the case
maybe you mostly care about getting this
the values at the next states that you
use to pick your action to have those be
accurate so if you're in a certain state
you have a little bit of time to think
maybe you want to spend that time to
update the values that are of interest
to you right now
rather than values that might eventually
be of interest to you because it might
be the case that there are states that
you're never going to return to for
instance and then why would you ever use
any compute to update those maybe you
don't just don't want to do that so this
is sometimes called real-time dynamic
programming the idea is again somewhat
similar to the prioritized sweeping
where we just basically picked a
different priority we are not
prioritizing in sometimes by closeness
to where we are rather than how big the
update was previously but otherwise you
could use similar ideas and this again
can very much limit the amount of states
that you actually update and therefore
very much limit your computer now one
way to think about dynamic programming
and this is especially useful to
contrast it later is to basically go
back just to the just think of the
synchronous case for simplicity but for
each state
dynamic programming will consider all of
the possible actions and then for each
state action pair it will consider all
of the possible outcomes this is
sometimes called a full-width backup
because we're using the full width of
the tree even if we're going to be using
one step in in terms of depth this this
works and it's it's convergent and it's
you can get it to be fairly efficient
but it only works up to say medium sized
problems where maybe you have millions
of states now obviously we come to the
synchronous dynamic programming with
millions of states and as you have a lot
of compute but if you do the
asynchronous things you can get pretty
good solutions for millions of states
and people have done this in practice
which is actually quite impressive
because with millions of states you can
model quite interesting and complex
problems some cases will still not be
enough
if your input say is a video stream I
also want to maybe remember a few
pictures from the past because otherwise
it's too non-markovian then the amounts
of ways you can cut that up are even if
you do fairly coarse cut off saying the
intensity of each pixel you easily get
more than millions of frames sorry
millions of states that you could
investigate so then we need different
solutions which we'll talk about in this
course as well but if you can write down
an MEP which is only say thousands or
tens of thousands of states then you can
apply modern modern dynamic programming
methods on modern machines quite easily
and you can fully solve them for larger
problems and maybe maybe yeah the best
way to say that is to say that dynamic
programming suffers the curse of
dimensionality this is a term that also
originated from Richard bellman which
essentially means if your problem is
more than one dimension the amounts of
states in your problem will very quickly
grow and one way to think about that is
that in the car dealership for instance
we've only considered how many cars
there were one situation at one location
and at the other but you could add a
dimension say to weather and if only if
the weather can only say take three
different states it already means that
we're multiplying the number of states
by three if there's yet another
dimension say time of year and let's say
month and there's yet another twelve
different possible situations that can
happen across this dimension we're again
multiplying the amount of states with
the number twelve so this means that
each time you add a dimension to your
problem the amount of states gets
multiplied by the number of elements in
that dimension in a sense there is a
continuous version of this as well of
course
and this just means that things very
quickly get unwieldy and it's very easy
to get systems that actually need more
than millions of states if you don't
have a lot of prior knowledge of how to
cut up and how to define your state
space in addition there's problems with
loads of actions as well that's another
thing to consider that we really didn't
touch much upon which can also make the
computer oh sorry
so one other thing that we'll touch upon
in depth but we can start talking about
a little bit is to apply the ideas we've
of course learned from the other side of
the course to use function approximation
and the idea is of course quite simple
there where we basically is going to say
we have a value function this is going
to be a parametric function let's say
it's a deep neural network and we're
going to use that instead of enumerating
all of these states there's many
benefits of this one benefit is they can
do maybe with larger state spaces but
another benefit is that it's much easier
maybe to generalize that's something we
haven't really considered so far we've
considered separate states but States
might be very similar and you might want
to exploit that deep networks and other
function proximities have the natural
tendency to generalize similar states
they already give you an estimate even
if you've never seen these specific
states just by generalizing things you
have seen which are kind of similar in
some sense well we can abstract away
which function approximation we're using
here what we're just going to say some
parametric function proximation and
we're going to denote the parameter
vector theta which you can think of as
being say for instance the weights in
your deep neural network and then we
have an estimate value function and how
do we then update this well we can use
the same ideas as before we could for
instance do asynchronous dynamic
programming so we just pick a couple of
states maybe we do have a full model
that we can reason through maybe there's
not too many actions so we can actually
consider all of the possible actions and
what we could then do is just pick a few
sample states this is the asynchronous
part and for each of these sample states
we can compute a targets using just for
instance value iteration in this case we
do the one step look at of course this
assumes that you can do the one step
look at with either a true model or
maybe a very good learn model but let's
assume the true model for now but then
maybe we can find a new parameter vector
by simply minimizing the difference
between our current estimates and this
target and one way to do that in
practice would be for instance you don't
fully minimize when you just take one
step other things people have done is to
consider a linear function approximation
in which case sometimes you can just
solve these things rather than doing a
gradient step and in general you there's
many ways
to minimize this loss slowly and this
turns out to also be a sound algorithm
you can use this on real problems you
could also use this as we'll see later
if you don't have a true model by
sampling and this turns out to work in
practice as well and in large problems
so I already mentioned this a couple of
times but it's good to be explicit about
this dynamic programming improves the
estimate of a value at a certain state
but in order to do that it uses the
estimate of the value at the next state
the same estimate essentially the same
value function that you're optimizing
you're also using this is sometimes
called learning a guess from a guess and
this is a very core idea to
reinforcement learning it's not
immediately clear that this is a sound
this is a good thing to do we've shown
firstly simple cases and we've just
mentioned for other cases that indeed in
dynamic programming this works but what
if your value functions are even more
approximate because they're just some
function approximation some deep neural
network that might be arbitrarily wrong
say in certain states does it still work
is it the sound idea well in some cases
we can really say it's sound we can say
this converges even if we don't have a
true model and we'll touch upon that
later in other cases we can't go that
far for instance if we have a very
highly nonlinear function we might not
be able to say exactly where it goes but
we can still see that in practice it
does work on big problems and we'll
discuss algorithms to use that there is
a theoretical danger of divergence which
means that your parameters go off into
basically lala land when combining the
bootstrapping
with general function approximation even
with linear functions but this happens
when you learn things what is sometimes
called off policy now we use that phrase
more often later in later lectures but
it means when you don't appreciate the
actual transitions in terms of where
they lead you in terms of your policy
these theoretical dangers rarely
encountered in practice but there's very
small toy problems and this gives one in
which you can show that this is a
problem now I don't have time to step
through this completely but just to give
you just to tell you what we're looking
at here there
two states and we have a function
approximator that only has one parameter
and we're basically saying the value of
the first states will be theta which is
just your parameter and the value of the
second state your estimate of the value
of the second state will be 2 theta
the rewards are zero everywhere so we
know that the true value is actually in
the span of this function you should
just set theta to zero and you're done
but turns out if your theta is not
initially 0 and the probability of this
loop happening is also nonzero then if
you update both of these states at the
same time you do the synchronous dynamic
programming like idea and you just use
this update where you fully minimize the
loss each time 4 theta turns out this
diverges theta just goes off into
infinity and the reason is that we're
not appreciating the fact that you're
actually if you would step through this
problem you would spend more time in the
second state and if you if you take care
of that if you are aware of that in your
update so you update that state more
often in the first state as well then
everything's fine but if you just update
both of them at the same rate
turns out theater goes off into lala
land
essentially ok so that's all the time
this is still relevant but maybe it
should go Moodle I don't know whether
people are kicking us out seems to be ok
if we do need to leave the room I'll be
available just outside if somebody still
has questions thank you