DeepMind x UCL RL Lecture Series - Theoretical Fund. of Dynamic Programming Algorithms [4/13]

hi everybody and welcome back to our
fourth lecture on reinforcement learning
today we're going to go deeper into the
fundamentals of dynamic programming and
revisit some of the algorithms we've
introduced the last time we see their
theoretical properties
as a reminder last lecture we've
introduced
the framework of markup decision
processes we also look at the two main
problems that arise in rl policy
evaluation and control and we've seen
iterative methods under the umbrella
term of dynamic programming
that can tackle some of these problems
in particular we've looked at value
iteration and policy iteration as a way
of doing control
this lecture we're going to
be deepening our
mathematical formalism and understanding
of the mdp and the dynamic programming
we're going to revisit the belmont
equation and introduce their operators
and
revisit the policy
iterative policy evaluation and value
iteration improve their conversion
properties based on these operators
in the next lectures where you're going
to see more
approximate versions of this dynamic
programming principle mainly in the
absence of perfect knowledge of the
environment okay
let's get started
before we get into the
actual material i want to cover a bit of
preliminary
functional analysis
that
just just the bare minimum to uh
to make sure that everybody is on the
same uh on the same page okay
so first recap is uh the concept of norm
vector spaces so hopefully you know what
the vector space is and
a norm is just a mapping
from a vector space to
r in this
case that has these properties it's
always positive
it has homogeneity and
it obeys the triangle in the quality
okay
you should have
seen this before but just as a reminder
okay
and uh
for this lecture most of the vector
spaces we're gonna be seeing are just uh
rd and the norms we're gonna be seeing
are uh max norm so the ellipfinity norm
and the l2 norms sometimes weighted by a
distribution
okay
a second concept here is a contraction
mapping and this is the definition of a
contraction mapping so we have a vector
space equipped with norm
as introduced before and a mapping theme
from this vector space 2
into this vector space is a
alpha contraction mapping so with
coefficient alpha if for any two points
in this space in this vector space if
you apply this operator this this
contraction mapping to this point
the distance between these two points
shrinks
at least by alpha
so this is the mathematical equation for
for that
we um
we say that t is non-expanding if alpha
is between 0 and 1 and it's a full
contraction if
alpha is less than 1.
just a
small observation every contraction
mapping as defined in this equation is
also by definition lipschitz which also
means it's continuous in particular for
us that would mean that for any sequence
x n
in the space that converges with respect
to this norm to a point x
we know that um
if we apply the operator
if we apply this mapping to each element
of this sequence the the resulting
sequence will also converge
in the same norm to t of x
okay so you take a sequence that is
convergent in that space you apply a
transformation t to it and you're
getting
another convergence sequence that
converges to
t of x which is the limit of that the
the mapping of the
limit of that sequence
okay
um another definition a fixed point so
um a point or vector in this
vector space x is a fixed point if any
operator or any mapping if you apply the
mapping to this operator to if you apply
this operator to this point you get the
point back
so it doesn't change the
if you apply the operator to it it it
just goes back
okay
and uh finally this was kind of all
building up to one theory
um in functional analysis which is the
bana fix point theorem
uh that says that if you have
um
x a complete norm space so we've saw
what the norm space is a complete norm
space basically means that every
sequence in that
space um
every convergence sequence in that space
its limit is still containing the the
space so another way of saying it is any
cauchy sequence will
converge within uh x
um
so we have a complete
norms vector space and we have
a contraction mapping with the
contraction coefficients
gamma
then we know that this contraction
mapping actually has one unique fixed
point
so there exists a fixed point where
if you apply this
operator to
a vector
x you get back x
and this point is unique in the space
also we know that if we
define this this sequence
as such x n
plus 1 equals t x n
this sequence
converges
to uh
x star which is the fixed point of this
uh this operator in a geometric fashion
and this is the expression that uh
basically says that
uh this says basically that the distance
between
an element xn in the sequence
and the fixed point
shrinks
with this rate
okay and you can see
uh that if you
let n go to infinity in this
equation you get that the upper bound
here actually goes to zero so that means
that this sequence is actually
convergent to this unique fixed point
okay
good that's that's the only math that
we're gonna we're gonna need
so now we're just gonna recap a couple
of
uh concepts that we've introduced the
last time just for those to be uh very
fresh in your mind okay
one uh is the definition of markov
decision process and in this particular
lecture we're going to use the second
definition
uh of the markov decision process where
we have states actions a transition
dynamic that only codes for the next
state r which is the expected reward on
under that transition
and a discount factor gamma
and we've seen
or argued last time that most rl
problems can be formalized by ndps and
we've seen several examples of how to do
that
okay
now uh we've also introduced value
functions and as a reminder the state
value function is the expected
return
starting from state s and following a
particular policy pi
uh in all in
in this lecture we're only going to be
using the discounted return
the same uh for the action value
uh function uh for policy pi this is the
expected return
knowing that we're gonna start in state
s taking the action a which might not be
according to pi but then following
policy pi
okay
and in terms of the the optimal value
functions we've defined the optimal
value function as the
um
the function that maximizes over all of
these policies
okay
and we've introduced
the the belmont equations uh first the
the belmont expectation equations
uh that says that for any policy pi the
the value function associated with them
the ones that we've just reviewed obey
this these equations
and these are just the form without the
expectation just spelled out exactly
what the the sums are
okay
and then um
for the optimal value functions we have
another set of equations called the
belmont optimality equations
and disobey these the following
equations
okay
well that's the end of the recap and
let's dig into the new material and
introducing the bellman operators
let's start with the bellman optimality
operator
given an mdp
we're going to define v or vs
the set of all
bounded real value functions over s so
this is the space of functions
that takes an argument in s
and returns a number
this is in a sense the space of value
functions or the space of value
functions that we're targeting is a
subspace of this
this be okay and we're gonna
define the belmont optimality operator
on the space
so v to v
according to the to the following
expression
so this
operator takes a function f
and this thing the whole thing returns
another function
that we're going to be evaluating at
state s
and this is the definition of that that
evaluation and this should look very
familiar because this is
um almost the bellman optimality
equation in particular if instead of f
here you would have
v star
this would be
the
right hand side of the bellman
optimality equation
and we know that this one if f equals v
would be equal to v star
in particular that means
that v star is
a
is a fixed point for this
this mapping we've just defined
okay
actually it turns out that this operator
that we've just introduced
has a lot of nice properties one of them
we've just discussed that the
optimal value function v is
um it's it's fixed point and it happens
to be an unique fixed point
also we know
that um
t star
the operator we just introduced is
actually an
gamma contraction mapping with the
uh contraction coefficient
the
discount factor of the mdp
with respect to the l infinity norm
and
the other property
is that um this
this operator is monotonic in a sense of
if
there are two value functions or two
functions in this space u and v
such that for every state
u
s is
smaller and equal than vfs so this is
not a partial
ordering it's uh
v dominates you in all of the states
then the same
relationship is maintained
if you apply the the operator to these
two vectors or to these two functions
okay
note that we as we've seen uh before
with value function and with policies
this might be a partial ordering it
might be for some states um
the value function is greater for some
states it might be lower than another
value function what we're saying here is
if there are these two points
for which the the relationship
regardless of the the state doesn't
change then the same is maintained for
um
for the the mapping through the operator
okay
so um it's okay if you don't uh see this
straight away
uh this one the first one should be um
quite evident because it really falls
out of the definition of the the
operator and the the bellman equations
we've introduced last time so i would
spend some time just convincing myself
of that
and
actually we're going to prove uh
property 2 and 3. okay
let's start with the the first property
there the contraction mapping
as a reminder the contraction property
basically says if you have two points in
the space in uh yield
applying the contraction mapping to this
point
distance between these points has to be
less than equal then gamma
the original distance between these two
points
and uh in this particular example we're
trying to
prove that
d star is a contraction mapping with
respect to the l infinity norm so this
would be with respect to the l infinity
norm
okay
so to start out we're just gonna expand
based on the definition of t star
what that means for a state s
and this is just the definitions
that we have okay
and once we have this we're going to use
the following
inequality
of
x
minus max of
x of g
this is
less than equal than the maximization
outside the back
the absolute value okay
and
the way to see this is
maybe not trivial but
we can prove it
sketch of a proof
we can start with x
this is less than equal then
f of x minus g of x
plus g
hopefully that's not too
controversial
uh if we take the max
we have max of
max of
the absolute value of the difference
plus
is our absolute value
plus
max of
g of x
which leads us to
max of x minus max of
g
less than equal then max of
f minus g
and the same thing can the the same
proof can can be done for
exchanging f and g
to get that
minus max of
max of the
okay
and these two now imply this one
okay
so we're going to use this inequality
in this
in this equation
and if we do that we get the following
the max is now out and we have the inner
parts
and these two
um
simplify and then we're left with max a
gamma
expectation over b
minus gamma expectation the same
expectation actually over u
which can be rewritten as gamma
p x
minus u of s prime
okay
and this
is less than equal than max
over
uh s prime of e of x prime
c
s prime
which is
u minus v
under the infinity node
okay so this
is true for any s
thus it's uh true for the maximization
over s
okay
which leads us to
b minus here
at
which is exactly what we wanted to prove
and this is just a type out version of
what we've just proven just for your
reference in case you don't understand
my handwriting
okay now we're going to move into
proving the other
property the third property
the monotonicity of the bellman operator
and as a reminder
this means that if we if we have two
points
b and u
uh such that
for all points in the state
for all states v
is dominated by u
then by applying
uh t star
the same ordering is maintained
for all states
okay
so um
the first uh
thing to to do is to pick an ordering
between these these two functions and
notice that this implies the following
inequality
actually it's v
and u
so we're just applying an expectation
and because this is an ordering for all
uh states s and s prime
this doesn't change the ordering and
then we're adding a constant to it
and that's it
now let's look at the difference between
tb
and tu
so the the mapping through the operator
and this is just expanding the
definition of the
of t star
and here we're going to use
uh the
a very similar inequality as the one
that we've had before with maxis
in particular that max of
f max of
g
is less than equal then max of the
difference
okay uh the proof is very similar to the
one we've seen before so we're just
going to apply this
this leads to the following uh equation
now the max is on the outside and these
inner values or on the on the inside
and
again
and we've already proven just from the
this uh
inequality that this is less than equal
than zero which implies that
is less than equal than zero which means
that
vf plus
t of
uf s
i'm sorry
so that uh t of v
of s is less than equal then t of u of s
for all s
which is exactly what we wanted to prove
so now we've proven all this uh nice
properties of the the belmont operator
that we've just introduced
why is this useful
well the the way this becomes useful is
that we can reinterpret the algorithm
when we introduced last time value
iteration through the lens of this
belmont operator in particular
what uh
what we can
say is that we can reinterpret the the
updates that these uh value iteration
algorithm was doing as just repeated
operations repeated applications of the
bellman operator so if you remember
um the value iteration algorithm would
start with an arbitrary uh value and
initialization value and at each
iteration
the
the value
at k plus 1 would be
updated towards
max of x a
plus
gamma
expectation over the next state
and value
at the next date is given by the
previous
iteration
of the value iteration
okay
so this
now
is just t star
of
bk
at s
okay
so
compactly
this is just written here for every
state
now the nice properties of this operator
that we've just proven
means that we can prove the convergence
of this algorithm in particular
as the number of iterations goes to
infinity we can show that this induced
sequence of value function converges in
l infinity
to the fixed point of this operator
which happens to be
v star so the fixed point of uh t star
is
the star
this is the first property that
that we've introduced and that falls out
of the definition of the
t star
to spell this out this is actually a
direct application of the banner fixed
point theorem
and we can
we can elaborate this so if we look at
the
distance between
v k and v star
under the l infinity norm this
can be rewritten as such
because v k is just the application of
p star
to v k minus one
this because it's uh it's the fixed
point of the operator
is
the same as t of
v star
this is the fixed point property and
then we can apply the contraction
property here that
applying the operator to two points in
the space
decreases the distance by at least
gamma
to the original distance and then if we
applied the property uh over and over
again till
we get to v0 we we get that the distance
between
vk
and the fixed point
is less than equal then
gamma to the k
the original distance
from the initialization to
the the fixed point and this as
k tends to uh infinity
at the limit of this
is zero
thus
vk
tends in this l infinity norm
to
the star
which means that we've just proven that
the value iteration algorithm is a sound
algorithm that will always converge as
long as gamma is less than one so it's a
full contraction if not is non-expansion
so we can't approve that
so for all gammas less than 1 this will
converge and will converge in an
exponential fashion
with the
geometric rate
gamma
so this is proving that value iteration
converges for v a similar argument can
be done for q
and moreover
we can
do a very similar procedure for
the
iterative policy evaluation algorithm
that we've defined the last time in
particular we're going to take the
belmont expectation equation now not the
optimality equation as as before and
were turned
into
an operator
this operator will be with respect to a
policy pi with respect to which the the
evaluation problem is
and we're going to define it as as such
the this is equation 13.
the
the thing to note is again here is that
if you plug in instead of f
v of pi
this is the fixed point of this operator
by definition this is
v pi is the fixed point of this operator
in particular we're gonna
see that this operator this new operator
again this is an operator that is um
indexed by pi because it depends on the
the policy we're trying to evaluate it's
not a max operator it's not the star
operator this depends on on pi
okay but it has the same properties as
t star
in particular it has only one unique
fixed point
t pi
a v pi
it is a contraction mapping
with the same gamma contraction
coefficient with respect to the l
infinity norm and it is monotonic
we're going to go
briefly through to the proof of
2 and three as before
so uh in particular we're gonna prove
that the v pi is a contraction mapping
with respect to the l infinity norm to
do that we're just gonna start again
with the definition of uh
these operators
and uh because the expectation here is
with respect to
or the summation here is with respect to
the same a this can be
this uh terms can be grouped together
in particular we can um simplify r in
this equation and we're left with just
the expectation over the
the action taken at the s and the
transition to
s prime
okay and
now what we're gonna do is just maximize
over these equations in particular
um we're gonna see that
uh
the expectation here
with respect to a and s prime this
expectation is less than equal so it's
uh upper bounded by the max of uh the
difference between u and b
so whatever the
whatever combination of states and
actions uh
here
will be upper bounded by them by this
max so the
um the whole expression here is upper
bounded by the mac the global max of uh
vnu
and uh
this is this is an upper bound for all
uh
states s
so
that gives us that uh if we maximize on
this side too
means that the l infinity norm of uh t
pi v minus t pi u
is uh upper bounded by gamma
the l infinity norm of
v minus u which is exactly the
contraction property
note that actually
uh property or inequality
14 equality 14
also gives us property through 3 which
is the monotonicity in particular if
there's a
global ordering on vmu let's say v is
dominated by you then this
um
this difference is always negative which
means this this this difference is
always negative
so the
the sign is maintained
okay so now we we've proven uh all three
properties
and then we can go back to iterative
policy uh evaluation the algorithm that
we've introduced last time to
the tertiary algorithm we introduced
last time for policy evaluation as a
reminder this would start with an
arbitrary value of v0 and apply
uh the bellman expectation equation as
an update to it and this
as as with the optimality
equation this is just a compact form of
of rewriting that that equation and
because of the same argument as before
with the optimality operator this
induces a sequence of value functions
and it is a convergence sequence because
this operator is a contraction mapping
and this
sequence will converge
to this norm
under which
this contraction
operator is a contraction
um
this uh the sequence will converge to
the unique fixed point of this operator
which is a v pi by definition
okay so um the the other way of saying
it this is a direct application of the
banach fixed point theorem that we've
introduced in the in the preliminary
version
okay
now just a summary of
what we've discussed and what we've
discussed last time in terms of dynamic
programming this is uh just the
rewriting of the algorithms introduced
last time for the control problem
through the lens of the
bellman operator so we've seen value
iteration as just um
a direct application of the bellman
optimality
operator and policy iteration the other
algorithms that that we've used the last
that would be introduced last time for
control uh would would have this um
alternative procedure between policy
evaluation and greedy improvement now
the the policy evaluation can be done
via the iterative policy iteration that
we've uh we've just seen and uh
this is just a repeating application of
the the policy that we're trying to
evaluate at this point in time
okay
um the last observation here is that
we've uh we've done all these proofs and
this introduce we've introduced these
operators with respect to
uh computations in the
valley function space but you we can do
the same thing and do exactly the same
reasoning for q values
the only difference is that these
the corresponding bellman operators both
for expectation and for
optimality
will be
on the space of q values so instead of
the definition being point wise on s
it's gonna be on sa
but the the same reasoning apply and the
uh the idea is the same that we are
gonna come up with these operators now
on the
q value space so our
space of bounded value
real value functions on sna
[Music]
that will have as fixed point either the
um
evaluation of a policy
queue evaluation of the the policy pi or
the
optimal value function q
and the same properties apply uh these
operators are
gamma contractions and they are
monotonic
okay so these these are
just spelling out the the definition of
these uh
other operators on the q space
we are now gonna talk a bit about uh
approximate dynamic programming this is
mainly an overview of what's to come in
the next three to four lectures and
for you to have a mindset or reference
point to map those methods into the
dynamic
programming principles that you've
learned so far
okay so so far we have
assumed the perfect knowledge of the mdp
in a perfect or exact representation of
the the value functions
realistically more often than not these
conditions would be violated okay we
won't have access to the
true underlying
mdp we won't know the true transition
dynamics nor the reward and we'll see in
the next two lectures how to deal with
that
but moreover we won't be able to
represent the value function exactly so
so far we had these tables where for
each state
uh we would have a value associated with
that or for each state and action we
would have a value associated with that
if the state space or the action space
is very
large we won't be able to represent that
or it might not be useful to represent
it like that because we won't get any
generalization
so more often than not we would be using
some kind of function approximation
now when we're violating this uh
this assumptions we are introducing some
uh errors in the first case where we
don't know the underlying ndp that means
that we won't have access to the true
backup operators that we could just
introduce either the expectation one nor
the optimality one because we we just
don't have the full model of what the
next transitions would be
thus
because we're going to make this step
approximately by a sampling we would be
introducing a sampling or an estimation
error here
moreover when we're using function
approximation we're gonna be introducing
uh an approximation error
due to the fact that the true uh value
functions might not be in uh in the
parametric function we decided to to use
nevertheless the objective still remains
the same under the above conditions
we want to come up with a policy that is
optimal or at least close to optimal
so let's see
what that means uh in practice for
the algorithms that we've seen so far so
policy value iteration and policy
iteration so we're going to start with
the
value iteration
and this is just a reminder of the value
iteration algorithm uh
in the operator notation and the only
thing that the um and remind ourselves
that this is guaranteed to converge to
the optimal value function now the
approximate part here is approximating
that step whenever we're doing this
updates from iteration k
k and k plus 1
instead of being able to back up the
whole uh operator the true operator here
we're gonna have some kind of
approximation denoted by a here
where
now v k plus one just approximates
either from uh from samples or uh and or
from uh
in a in a parametric form
the true value of the
uh optimality uh one step optimality
operator
applied to vk
now the question becomes
under this approximation or under which
approximation does
does this
sequence still converges to the optimal
policy
well uh it turns out that in general
without any assumptions about the nature
of this approximation we
this this thing might not converge at
all uh and the point that
of converging might not be the the
optimal value function
um
again i've uh i've said that this
approximation can come from two sources
one is the sample
and the other one is the the function
approximation class that we are using
considering you're gonna see a lot more
of the sample version
approximation here in the next two
lectures i'm going to spend a bit of
time uh giving you some intuition of
what happens when we're trying to do
this um
procedure with the functional
approximation we're still gonna um
we're still gonna have we're still gonna
assume that we have access to the true
uh mdp so that that is not the
approximation that we're looking at is
just a functional approximation
and
if
in this setting where we're trying to
uh approximate the
the value function with the function
approximator uh paramaterize here by
theta
means that the the value at each
iteration
previously denoted by v k is now v
theta k
and uh
[Music]
using dynamic programming we still can
compute the one step um
application of the the bellman operator
we're assuming now that we still have
access to the true mdp
so uh we we can still do this backup
but now in order to
still be in the same parametrization
family instead of backing this up to vk
a plus one we would need to find a point
in the parametrization class that is
closest to this so we've parameterized
our vector by
by theta so we need to find the set of
parameters uh theta k plus 1 such that v
theta k plus 1
approximates well this uh t star vk
for each state in the environment and
for instance
this can be done with uh
with respect to a square loss over the
whole state space and this is just
spelling out that that equation
okay
now um
the bad news here is
that
even
even in this particular scenario where
we're not introducing the other kind of
approximation via the samples
just for from pure
function
moving to a function approximation uh
setting we still could diverge
and this is just an example that was
introduced by cclis and
van roy
where um we have a linear uh
approximator
and
a very small
mdp and this in this example you can
you can show that the value function
that we're trying to uh to estimate can
actually diverge so let me
uh quickly
um go through this uh example with you
so this this mdp has only two states
this one and this one and the function
approximation that that we're using is
just one parameter and we're going to
parameterize the value function at the
the first state by theta and the value
function at
uh the second state by a two theta
okay
and uh this is uh
uh an mdp where all rewards are zero
this is a terminal uh state and there
are no decisions so this is a
deterministic transition to
the second state and then the second
state would loop on itself with
probability 1 minus
epsilon and with epsilon probability
with transition to this terminal state
okay
and if we initialize the value function
this approximation of the value function
away from the solution
we can show that there are cases where
this whole system
can diverge
by approximate dynamic programming
okay
to see this let's go quickly through the
computation
let's see what happens at the iteration
um k plus one direction
this one
okay so usually in pure dynamic
programming
we would have
v k plus 1 take the value of
t
v k
now in the approximate setting v k is
the
um
theta k
for some uh
theta
and then
uh
t bk is um now
the reward at the
uh at the state
state yes we don't have uh
uh any actions in this example so it's
just
the reward at that state plus gamma
expectation of what's going to happen in
the future
so v
s
prime
this is zero all of the rewards in the
uh in this uh
mdp are zero so this simplifies to
equals to
gamma
expectation of
basically what's going to happen at the
next state
okay
and because we are in an approximate
setting
we're trying to find
theta
k plus one
so v k plus one is actually now theta uh
v theta k plus one
such that
v k plus 1
approximates
t
okay and the the approximate
version of this would be uh with respect
to the uh squared loss so we actually
want that
um
theta key the k plus one is the art max
at art min
of
now we have a sum over the states
uh v theta
minus
expectation over
v
okay
sd plus one
keeping that
okay
this is just the vk so this is just
coming from this from this equation
okay and this is over the the whole
states in the in the environment
okay and let's spell this out
over
so for the first state
the value
v theta
has
value theta
minus
now um the
this expectation because uh
from uh the first state we're going to
transition deterministically to the the
second state so there's no
uh other possibility here
this is just gonna be the value at the
state um
at the the second state so this is b k
at
uh s2
which is
s
uh to
theta k
okay
that's
for the first state and then for the
second state the value
for the
uh second state the theta
uh s2 is
to
um theta this is the parametrization we
we've chosen and then
there's two things that can happen here
gamma and with probability
um
epsilon we transition to this terminal
state
and with 1 minus theta probability we
transition back
to
um s2
which is just
this value and this one again is two
theta k
okay so this is the arc min
of theta minus two theta k
plus 2 theta minus
1 minus
epsilon
okay
and uh note that this is a quadratic
equation so we can find it's uh it's a
minimizer by just
taking the gradient
so
let's call this f
so we
derive with respect to
and we get the following
to
the minus
plus
uh two
uh two
theta minus
uh gamma my uh one minus
epsilon
two uh theta k
and
this has to be
equal
equal to zero
for
this to be a minimizer so
we get that theta minus
two uh
gamma
theta k
plus
let's hope i don't mess this up
um
four theta
minus
four
uh gamma
1 minus
epsilon
equals 0
which means that 5 theta
equals
uh two gamma
theta k
one minus
one plus um
two
minus
okay
which means that
theta equals
theta equals
two gamma
three minus two
um
epsilon
over
five
okay
uh cool so this is
theta k plus one
now the
the interesting uh thing to to notice
here is that
if
this term
is less than one
then
theta plus one and this sequence
converges to zero which is the actual
solution
of the mdp because all the values are
zero so theta equals
zero would give us that um
that solution but
if
um this is
greater than one then this sequence
actually diverges
and you can uh show that there are
actual
values of
epsilon such that this is a divergent
sequence
uh for
instance epsilon i think equals 1 minus
1 over 8 is is such a value for which
this the sequence is a divergent one
now let's go back to the approximate
value iteration
uh formulations that we had before and
what we've seen so far is that this
approximation a
can lead to two problems even if
only one of these
sources of approximation the one coming
from the function approximation is in
place
things could go wrong
so
is this hopeless is this uh strategy of
using
value iteration or one of the principles
in
exact dynamic programming and trying to
approximate these updates is this
hopeless recipe
well not quite so it turns out that
sample version of these algorithms as
you will see in uh next week
uh can be shown to converge on the very
mild solution so this is without the
function approximation in the tabular
case
even for the function approximation uh
setting the theoretical danger of
divergence is rarely maturized in
practice so
even
though we don't have guarantees
uh anymore in most of the function
approximation cases whether or not this
would converge and to what uh value
this would converge we rarely see
divergence
but as the
the previous example showed this can
happen
okay and the
the last point i want to mention here is
that there are
there may be many functions that would
induce an optimal policy or a very good
policy so we don't need to nail down
exactly the
vstar or a very small
ball around the vstar in order to be
able to derive a good policy
so
just to exemplify this this is the
example from the the last lecture of
policy evaluation and you
see that the intermediate values here
that we've computed
as part of policy evaluation
already give us the optimal policy so
all of these values if you would um
act greedily with respect to these
values will give you the optimal policy
and note that none of these values are
actually the optimal policy which is the
one that
we are guaranteed to get the optimal
policy if you
acting readily with respect to to it so
there might be
many values
that we can compute
that
that are part of the evaluation problem
or
part of the iterative procedure in
value iteration though those
intermediate values to the convergence
that would still give rise to very good
policies so we don't need to
complete the process uh altogether in
order to
to get a good policy so or at least
there's hope
of getting a good policy uh before we
reach convergence
okay
but um
this is a bit hand wavy so
um we're gonna try to
formalize how much we can expect to get
out of a greedy policy
let's consider the following scenario
you have a
q value
q
this can be
an estimate you've arrived at
via
an intermediate step of value iteration
or one of the
evaluation of the
policies you had in policy iteration or
any other
value function we have no constraints
over that one and
from this value function
we're going to derive a greedy policy so
we have q and from that one we're going
to derive pi which is the greedy
policy with respect to these values
okay and we're interested in what we can
say about the quality of this policy so
in particular
what the
q
pi is
or how close is this to the optimal
value function
so this is what the
theorem says um basically it upper
bounds the
uh
how far you are how far this uh
policy is from the optimal value
function in terms of how good the
approximation
this q that you started with is so if
the skew
the approximation that you're using for
the optimal value function is very close
to the optimal value function then
this would be a small factor so
this would upper bound by a small
quantity the quality of the greedy
policy induced by this estimate
okay
that makes sense
let's prove this
this theorem
so let's um
uh let's start with
what we wanted to prove
so this is uh here and i'm just gonna
rewrite
so q
star
two pi
and i'm gonna first expand this into a
couple more factors
we will see
y
so i'm gonna
add and subtract
t pi q
okay
and this is by
triangle inequality less than equal
then
i'm just grouping the terms
if i
okay this is just um
as i said um
triangle inequality and now i'm gonna
rewrite a couple of things here
so this one
i can
rewrite it as
a t star q star
because
q star is
the fixed point of uh t star
so um these these are equivalent and
this can also be rewritten as p star
q
and this is because
um just as a reminder
pi is the greedy
with respect to q
so p pi
q
equals
uh you can you can
expand this expression to to convince
yourself of of that
and then
i'm gonna do the same trick here with
the the fixed point
of of uh t pi
so uh q pi is the fixed point of t pi
operator so i can i can uh
replace it with t pi q pi okay
so this this is actually inequality here
of minus
plus
okay
now i'm going to use the contraction
property of both of these operators
and the contraction property
says that this difference is less and
equal than gamma
q star minus q
and the same here this is less than
equal then gamma
q minus
q pi
okay
and the last thing i'm going to do here
is expand this
via the
triangle inequality into
q
q star
plus
q star minus
okay
this is just the triangle inequality
again
so this gives me that the the whole
expression here
is less than equal
then
two
gamma
so there's uh there's one from here and
one from here
plus
gamma
okay
and now
this term
i can
i can pull it on the other side with
this one
and i'm getting that one minus
gamma
is less than equal
than 2 gamma
q star
q
and if you rearrange these terms you get
just this inequality
again
so this is exactly what we wanted to
prove
let's go back to the statement of the
theorem now that we've proved it and
it's usually a good idea whenever we
have a theorem like that to
go through
a couple of edge cases to
to see what the the statement of the
theorem actually implies and
these are a couple of um observations or
edge cases that i have identified just
to guide your process through
so for instance for small values of
gamma
you
you get better or smaller upper bounds
so this this term uh here for small
values of
gamma is small
so um
another way of saying this is that
um you might get away with coarser
approximations of q so if you're
approximating the the value function the
the optimal value function q star
and q is an approximate uh version of
that you might get away with the corsair
approximation because this term is small
and still be guaranteed to get the
policy that is close to optimal as
opposed to the
uh case where gamma is close to one
then this term is quite large so the
potential loss in
the potential a loss
in
performance
induced by taking the greedy with
respect to this
this estimate might be quite large
another way of saying this
or interpreting this is that for
problems where
gamma is small
we can get away with
course estimates and still be very close
to the the optimal uh value function
on the other hand for
large
values of gamma this term is quite high
which means that even if we have a very
good approximation
of the optimal value function in terms
of pure values this still can induce
a greedy policy that can be quite far
away from the
optimal value function and the optimal
performance
okay
um
a couple of more of these questions uh
so we've seen that
smaller values of gamma makes the
problem easier what happens when gamma
is equal to zero
you can see that this term goes to zero
how do you explain that
and uh here's another one what if q the
the value function where deriving the
greedy policy with respect to is already
q star what does this bound
imply
moving on
the last point on our agenda today is to
revisit the
policy iteration paradigm in the lens of
approximation
so just a reminder the policy iteration
algorithm still computes the optimal
policy and the optimal value function
but it it does so by iterating this
process between policy evaluation and
improvement in our case greedy
improvement and this is
guaranteed as the number of iterations
go to infinity to converge to the
optimal value function and the policy to
converge to the
optimal
optimal
policy
now in the approximate setting
the policy evaluation set a policy
evaluation step is the one that would
incur the approximation
and that means that this greedy
improvement step will be done now
under an approximation not under the the
actual value of
our previous policy
the question that arises of course is
as uh as we iterate this process does um
this sequence of value functions still
converge
to q star
and
as you can imagine by now and as the
example
in the function approximation case has
shown that this might not be the case
so in general this is no but depends on
the the nature of the approximation here
for instance for example uh base methods
under mild conditions this will still go
through
the other uh question that arises is
even though the the value functions
might not converge to the optimal value
function to the
policies actually converge to the
optimal policy
and uh
we've seen previously that we don't need
to have the true optimal value function
in order to derive an optimal policy
from that so
um there might be a hint that
we can get a very good policy even from
approximate values
but in general we don't have any
guarantees on
you know the
if we don't make any assumption about
the approximate uh
approximation nature here and the
quality of the the policy
the quality of the approximation how
good is actually um
how far away is it from
uh it's actually a target of evaluating
the the policy and the previous
iteration
uh if we don't have that this greedy
improvement staff basically is not
guaranteed to improve
we've seen in the last lecture that if
we do have the true um
evaluation of
policy
pie i we are guaranteed to get
by this gridification step a policy
which value is greater on equal than the
one that we've seen before
this for an approximation value is not
necessarily the case
and uh i would refer you to the the
bound that we just proved the theorem on
the
the value of the greedy policy to uh
uh
to give you an idea of
why that might be the case so whenever
we have an approximation uh here the
problem is that this is not a strict
improvement step if um
somehow we can get a strict improvement
over the the policies that we've seen in
the past this is this whole procedure is
still
sound in the sense that at every point
in time we would still be improving our
policy
and uh
in finite cases we would still converge
to the optimal policy
okay so
uh even though this might not all be
good news
is it hopeless
in some cases no so it highly depends on
the nature of the approximation that
we're we're seeing here
and more about um
which approximations are are safe we'll
see in the next lectures
okay
so um
this is just a summary of the
approximate dynamic programming
uh paradigm and this is kind of what you
i want you to to have in mind moving
into the next lectures
um
try to picture those algorithms or try
to map those algorithms into exactly
these these two paradigms either value
iteration or policy iteration
and
think about the nature of the
approximation that that you're
inducing via samples or via a function
approximator whenever you you encounter
these these algorithms
okay
um that is all for today
if you have any questions please post
them on the moodle and
show up to the next q a session to to
discuss them
thank you for your time and see you next
time