Reinforcement Learning 8: Advanced Topics in Deep RL

let's get started
so we discussed a lot of different
topics so far item see if I can somehow
get this thing out of the way maybe not
oh well so what I thought I do today is
to go through some of the active
research topics right now I won't go
into that much breadth there's many more
research topics but I thought it would
be good to highlight a little bit what's
what is happening right now in in the
field of reinforcement learning deep
reinforcement learning and maybe also
give some pointers to what will be
interesting still to to do there are
some notable things missing here which
maybe might disappoint some of you for
instance I won't be talking about
multi-agent learning which is very
interesting in hard but it's also very
big topic and there's lots to be
discussed in that that respect and I
thought about doing it but I don't think
I could give it a fair enough
overview in the amount of time I still
have left but before I dive in I just
wanted to give a quick overview
basically to remind you of all the stuff
we've already discussed and it's quite a
lot actually so this is basically per
lecture I skipped over the introduction
lecture here which contains some
material but it's also was covered again
in later lectures which means there were
basically six lectures so far with
technical contents let's say the first
one was on learning how to make
decisions and we focus on banded
problems which there's a single States
but there's multiple actions and there's
already a trade of there between
exploration and exploitation also
because each action you make determines
what data you see if you take an action
you get data for that a criminal for the
other ones and we discussed greedy and
absent greedy algorithms who discuss
policy gradients which you can already
discuss in that context
and upper confidence bound algorithms
which are in sometimes optimal indoor
settings then of course removed in the
second exit to sequential decision
problems with discussed markov decision
processes how to planning these with
dynamic programming and also the genera
general framework of policy very
evaluation plus policy improvement which
then together is called policy iteration
and that's more general than dynamic
programming although these terms are
often used in the context of dynamic
programming the idea of doing policy
evaluation and then doing policy
improvement in order to get good
policies is very general and basically
underpins most of the reinforcement
learning algorithms for finding optimal
policies so then we wanted to get rid of
the assumption that you could have your
model so we went to model model free
prediction and control in which we
discussed Montecarlo returns where you
just follow a certain policy for a while
until a termination of an episode and
then you look at the return and you use
that basically as you could say
supervised target you're doing
regression towards these returns to
estimate the value of that current
policy and then you could separately
still do the policy improvements which
together again gives you policy
iteration and this allows you to learn
and then we discuss other ways to do
that for instance by using or maybe most
notably by using bootstrapping in
temporal difference learning and the
control variance of that q-learning
sarsen and double queue learning and
other related algorithms we also
discussed the distinction between on and
off policy learning where own policy
learning is what you might most
naturally do in a Monte Carlo setting
although you can do of policy learning
there but it's it means collecting data
according to your current policy thereby
estimating that current policy and then
maybe later deciding how to improve it
but you could also do off policy
learning for instance this is what
q-learning does which learns immediately
about the greedy policy with respect to
your current values which allows you to
more quickly optimize in many cases at
least what is the more general
distinction you could also just have
some data you're interested in a certain
policy but the the data was generated
with a different policy
for instance humans did some some some
things in a certain domain and you want
to learn from that but you want to see
what would happen if you would do
something else this would also be off
policy learning so the learning about a
greedy policy is a specific case of of
policy learning and I mentioned that
because sometimes people conflate these
and they say of policy learning but they
mean specifically to learn about the
greedy policy which is indeed off policy
but it's not the only way you could be
off policy
so then of course we discussed function
proximation and what these days is now
called deep are l often because this is
the combination of deep neural networks
as function approximation within a
reinforcement learning context and we
discussed the distinctions between
tabular representations linear
representations and nonlinear
representations also touched a little
bit of pop on convergence and divergence
in these settings we've seen very small
toy examples in which you can get these
algorithms to diverge if you basically
do it wrong if you're not careful there
are some fundamental issues there and
it's important to understand these
examples in practice however if you're
careful with how you set up these
systems they do work and they don't
don't go don't go off into weird
solutions that often we also discussed
least-squares prediction which is most
notably useful where you have a linear
function approximator because then you
can do basically you can exploit the
fact that it's a linear function
proximation to learn more efficiently
although it comes at the cost of some
compute these these squares methods if
you have any features that you're
learning from so your value function is
a linear function from some some feature
vector and this feature vector has n
components then these least square
methods they take N squared compute
whereas if you do a TD learning method
it would typically only take n compute
per update step that said these three
squares methods they tend to be more
data efficient so though so if your
feature vector is fairly small and you
can get away with using just a linear
function and the data is more expensive
than the computer to do the updates
which is quite often the case then you
should perhaps consider using these
least squares methods
we also discussed multi-step returns and
I'll return to that in this lecture as
well
neural q-learning and as a specific
instance of that DQ n which is short for
deep Q networks but it's also so it's a
generic term in the sense when you have
a deep network that represents the Q
function but DQ n has also come to refer
to this specific algorithm that was run
on the suite of Atari games and got good
performance there which used indeed it
even neural network to represent Q sorry
the action values but it had a couple of
other work components as well okay so
then we discussed how to learn policies
directly instead of going through a
value function or to use them in
addition to a value function mmm
in the Policy Gradius and extra critic
methods lecture so reinforce is an
algorithm you can use to just learn a
policy you could add a baseline or
bootstrapping or both to reinforce which
means you might still be using a value
function and then we typically call
these things extra critic methods there
are other ways to use value functions in
this context we call the policy the
actor and the value function to critic
which is just the terminology data Sui
has has been used for these things and
within this context we also discuss ways
to make these things more efficient for
instance by using trust region methods
which means that you don't update your
policy too much you stay somewhat close
to your current policy in a sense which
can lead to smooth or trajectories
better better learning and we discussed
how to use these methods to do
continuous actions because that turns
out not to be that hard when you're
already in the in the space of learning
these policies directly and we discuss
some specific methods and then last
lecture we focused on learning from
model we talked about the differences
between different types of models you
can have a full model which basically
means you're learning the whole MDP and
then you can just do dynamic programming
on that you can have an expectation
model which means you're only learning
the expected state not the full
transition dynamics you're not
interested in the distribution of the
direct transition dynamics but you're
interested in maybe predicting your next
state and we discussed this stochastic
or generative models in which you
implicitly learn distribution in a sense
which you can then say
from but you don't have the explicit
access to the digital distribution
perhaps and then you can just use this
as a simulator for instance you could
still chain the model together you could
do an update through your model you
could put a state and in stage comes out
and you could put that into your model
again and you get a next date you could
create a whole trajectory in that
fashion and maybe use that to learn from
and then we discussed the Dyna algorithm
which essentially means that you're
using a model to generate more data
whichever model you're using the full
model an expectation modelers or
stochastic model and then you're using
that data to learn from to learn your
value function form and in typical diner
you're also using the the raw data that
goes into your learning your model also
directly to update your value function
importantly we discussed this
distinction between parametric and
nonparametric models where maybe the
clearest example of a nonparametric
model is if you do if you store your
experience you store your transitions
that you've seen which you can then
later access again and we discussed how
this is a nonparametric model in the
sense that you can still query this you
can still ask for certain states and
action what was an x-ray or the next day
but it's just it will just return you
whatever you saw in that situation
rather than some prediction of what you
might see which also means that you've
corn of course also only query states
and a queue that you've actually seen
but then it does really work quite well
and I'll also return to that in this
lecture so that case we call this
experience replay if you sample from
this nonparametric model and at the end
there we discuss search and specifically
Montecarlo tree search a little bit but
I actually won't return to that that
much in this lecture so if you have any
questions about any of these topics of
course to interject so the main topic
today would be about what are some
advanced topics or some active research
to give you a flavor of what's going on
and what are the open questions as well
and the start it might be good to pop up
to the high level and to consider what
is the main question that we're trying
to answer
and for instance we could pick how do we
maximize future rewards I say for
instance because it's the question that
I'm personally most interested in but
you could actually imagine other
questions which are related to
reinforcement learning such as how can I
learn to predict certain things about
about different policies in in the
context where you have a lot of data so
you might do off policy evaluation as a
specific thing that you might might be
interested in not just for control not
just to maximize reward but maybe to
understand a certain problem but let's
say you are interested in maximizing so
we're interested in finding good
policies then you could imagine some sub
questions which are still fairly general
which include what do we learn do we
learn values models policies all of
these also how do we learn this for
instance do we use TD or Monte Carlo to
learn a value function and we've seen
some trade-offs it's not always
immediately clear how to do this most
efficiently and sometimes you just have
to try see what works there's also the
question how we represent the learned
knowledge a lot of people these days use
deep neural networks to represent these
but in some cases is actually more
appropriate to store things in a
nonparametric way just store some
samples you could in addition store
things in linear functions sometimes
which has the advantage that these
things are typically quite robust and
you can learn them very quickly but of
course they have limited capacity so it
really depends on what you're trying to
learn and whether this function class
that you pick is flexible enough to
learn what you want to learn and the
deep neural networks have definitely I
definitely have the benefit of being
quite flexible so they they are able to
represent many functions which is why
our use so so often and then of course
the last question which maybe is also
one of the first to ask is how do you
use the learned knowledge because if you
don't know how to use it why are you
learning it in the first place so maybe
you should think about this first before
you decide what you do and sometimes
it's more in in Reverse where people
focus a on value based methods without
even thinking about whether they they
are most interested in these values or
whether in the end they're more
interested in the policy say now there's
many specific open research topics this
is not intended to be a full list at all
but some important topics include expert
in the full sequential case because
we've discussed some advanced
exploration techniques in the first
lecture on bandits or second lecture
actually the first lecture with a
substantial technical competence perhaps
but a lot of these ideas don't naturally
or easily transfer to the full
sequential case where you're doing
function approximation for instance to
give a clear example of this the upper
confidence bound algorithm that works
really well in bandits requires you to
keep track of counts and these counts
they basically they basically count how
often you've selected a certain action
if you can do that that's very
powerfully you can basically use that to
get a measure of how uncertain you are
about the value of that action which
allows you to be optimistic in the face
of uncertainty and pick the actions that
either have a high value expected value
right now or you're very uncertain about
and especially actions that both have a
high value and you're uncertain about
you'll pick them and then either your
estimate will be correct and your
uncertainty will just decrease or you'll
find out if your estimate was too high
and maybe the the value will decrease
and maybe you're in certainty remains
high but in the end this all evens out
and it turns out if you do that you're
almost optimally exploring in the sense
that the long term regret that you
accumulate using that algorithm is
basically as low as you as you could get
it
however counts are fairly hard to do
when you're in a complex situation where
you're relying a function proximation
because in these settings we want to
generalize which is good we want to be
able to see in a new state we wants to
say immediately get a good estimate for
the value of that state but we don't
necessarily want these counts to
generalize because it might look like a
similar state but you're not 100% sure
it's a similar state so maybe you wants
to have these counts be a little bit
more specific and that turns out to be
relatively hard to do with deep neural
networks there has been some work which
I'll not touch upon too much I'll
mention one example but there's much
more out there in which people have been
able to get this to work better and also
to get more maybe advanced exploration
working but it remains an open topic how
best to do this another topic is credit
assignments we
maybe he's intuitive where you can have
problems when which takes a very long
time before you actually see the outcome
of an action you might take an action
that leads you into a certain corridor
but only the way way further do you
actually get the reward of going there
and how do you then de correctly assign
the credit for that decision of going
left or right all the way back through
time to that specific situation now the
long-term predictions already capture
this in a sense but it might mean that
your signal is very noisy and in
addition if you're exploring in the mean
time there might be lots of actions that
you took which actually we're not that
relevant for the reward but there's no
way for the algorithm to really know
this by just looking at the data so this
is also still a little open topic how
best to do this another thing related to
the previous lecture is how to plan with
inaccurate models we talked about this a
little bit if you have a full accurate
model you can just plan using say
dynamic programming if it's a small
enough model or maybe using something
like Monte Carlo tree search if it's if
the model is too big to go through the
full through the full model model but
padding has the tendency especially if
you use these classical planning
techniques which are very good for these
true models it has the it has the
tendency to find an optimal policy that
exploits the model in any way possible
so if they're actually an accuracy in
your model it might just exploit that
too much and it might find policies that
basically walk through walls weather
where you can't actually because the
model is slightly inaccurate anything's
there's a door there which might be do
the wrong thing to do and it's still
somewhat of an open question also or
maybe quite a big open question how to
best use models if you know you're these
models are either partial or inaccurate
now of course there's always more work
possible in simply efficient learning
because a lot of the algorithms that we
use these days still use quite a lot of
samples maybe this is related to the
previous points if we can learn these
models maybe we can be more data
efficient this used to be the case in
when we were doing smaller RL problems
but it hasn't really transferred yet to
the deeper L case maybe it's fairly
generic thing is how to appropriately
generalize this doesn't necessarily just
apply to the reinforced pruning
but one way to think about this is
especially if you have a long big
problem let's think of a a an agent with
a long lifetime this agent will find
itself in new situations again and again
if the world is sufficiently rich and
complex but the situations will resemble
each other in some way and specifically
the situations might be composed of
different parts that you've seen before
for instance if you're if you're taking
courses at UCL you might find yourself
into in a completely new room everyone
so often for for a lecture but you know
what a room is you know where to find
the seek you know how the hold of these
things work right because you can
generalize from these past experiences
it doesn't require you to be in exactly
the same place again in order to be able
to do something and this means that we
generalize appropriately and the way we
typically think about how we do that is
that we have something which maybe is a
little bit akin to a symbolic
representation where we can think about
a chair and maybe a laptop and we can
even think of some abstract things like
a course or a specific lecture as a
thing and we can reason about these and
we can combine these things together we
can talk about taking a tube to a
lecture where both these are actually
fairly high-level abstract concepts and
then the question is how do we learn
these how do we learn these things from
from raw data or is that even actually
what's happening at a low level say in
your brain or is is is the symbolic
nature of it is that something that is
our explanation of what's happening
there's something strong about symbolic
knowledge though which is that it's
quite easy to combine these things
compositionally as we also do for
instance with language where we can
combine different words together to form
new meanings and we don't have to
relearn the meaning of every little
every sentence that we might hear so
this is still an open question as well
how to best do this and also how best to
combine this with learning from very
low-level sensory motor inputs say
pixels and such and this is related to
the last point where we want to build a
useful general and information rich
agent state we haven't talked about that
that much but your agent state needs to
also incorporate for instance memory you
need to have some context of what you're
doing
we didn't discuss it we didn't really
talk about solution methods for this ok
so there's loads to still to be done so
we haven't finished like how to how to
solve the full reverse printing problem
that's basically what I'm what I'm
saying here but I think it might be
useful to go into a specific case study
and to discuss something that has been
done concretely which might give you a
flavor of how we're trying to approach
these problems and how we're trying to
just improve things and specifically
I'll talk about something that we yes by
the way sorry I should preface this by
saying I'll talk about things that I
know well so it's research that I've
worked on myself mostly just because I I
know it better doesn't mean that this
research is necessarily more important
or more interesting than other research
out there but I wanted to walk through
an agent that we recently built which
you call rainbow dqn this was by the way
in collaboration with Mateo Mateo hassel
who gave the tensorflow lecture at the
beginning of this course as well so
you've seen him and the starting point
here was to DQ an algorithm which I
think yeah I have in a slide or two so
I'll tell you again what's in there
which is basically q-learning with deep
neural networks including target African
experience replay but then includes
several additional components and these
were double Q learning prior to replay
dueling network architectures which
means we're splitting the values for
state values from the action advantages
I'll talk about all of these so explain
what that means
multi-step updates this is in yellow I
was aware that this probably wouldn't
show up well on the slide but I still
want us to keep the nice rainbow color
going there so apologies for that a
distributional reinforcement learning
and parameter noise for exploration
which is also sometimes called nosy
networks and then we combined all of
these components into an agents also to
see what happens but to also understand
the components better because each of
these was basically proposed in a
separate research paper
just looking at that specific components
and ensuring well this might be an
interesting thing to look at
but then combining them is not
necessarily as good or doesn't the the
different components don't necessarily
combine well so it was an open question
whether they would and so that's
basically what we did we compile compile
and we look at the performance we looked
at how the performance depends on all of
these components just to understand
better how all these things come into
play one thing I want to mention here is
that some of these you could think of as
changing the reinforcement learning to
be more aware of the fact that we're
using say deep learning techniques and
standards more or less standard deep
learning optimizers under the hood
sometimes it's better to change your
algorithm a little bit for instance to
change you lost a little bit to do to be
aware of that so that it works better
this was part of the motivation of using
experience replay and target networks in
the first place in dqn and in other
cases we might even change the deep
learning side a little bit you could
think of that as that so you could call
this deep learning where reinforcement
learning on the one hand and maybe
reinforcement learning aware deep
learning on the other hand where maybe
we want to think about this how do these
things combine and that's also still
somewhat of an open question because a
lot of the deep learning techniques that
we rely on were mostly proposed and
investigated at depth in say
classification tasks which are quite
different in nature than the especially
than the online reinforcement learning
setting I've mentioned this before
for instance one clear distinction is
that in reinforcement learning we're
actively collecting our data and we're
changing how we collect our data by
changing our policy which means with
everything is non-stationary so we're
violating one of the standard
assumptions that is made in supervised
learning so then of course we have to be
careful that we check that the methods
that were proposing that settings still
applying that they still work so I'll
step through each of these components
but first let me explain the benchmark
so I think a lot of you are familiar
with this I also mentioned this earlier
in the course but I just wanted to stop
here and be a bit more explicit about it
so the domain here is something called
the Arcade learning environments or da
le
which allows you to play with Atari
games and this has become quite a common
benchmark for instance is also available
within the open area gym and it's nice
because it has a diverse set of Atari
games which are fun and interesting for
humans that's why they were designed so
this this this means that there might be
an appropriate test domain if you want
to compare how these algorithms compare
to what say humans might do and they
might also be a good level of difficulty
the test algorithms I think we found
this in the past by doing research on
this that typically if you find that
things that work well across many
domains it's typically just a good idea
and it might apply more generally also
what's very nice about this is that it's
a simulator which is easier to work with
say than a real robots so this is good
to test ideas of course if you're
interested in doing things in say real
world robotics then you still have to
check whether these ideas still transfer
there but again we found that most of
the ideas that were that work really
well in simulated settings they Mari
also work pretty well in other complex
settings maybe one caveat here is that
the these Atari games a lot of them are
quite reactive which means that memory
isn't that big a component so we found
that agents that don't really have a
good memory component can still do quite
well on many of these games because you
basically can just look at the screen
and you know everything that you need to
know so that's maybe a limitation of the
benchmark there are some games in which
you might need memory more than others
but maybe for specific you want to look
at ages that have to use memory you want
might want to consider being careful
about which tasks you select the goal is
to build a general learning algorithm
without game specific knowledge so the
typical setup is here that we take a
learning algorithm and we train it on
each of these games separately it's the
same learning algorithm with the same
hyper parameters and everything and it
needs to be able to learn each of these
games this is different from another
thing that you could imagine which is to
take one learning algorithm and two
running them all of the games at the
same time this is something you could
also do which is maybe a harder task and
then you could also consider all of the
games together maybe to be
one thing that you don't do one task
rather than to consider each of these
games to be a separate task both of
these are valid things to do but this is
the one that we're doing where we're
training like from scratch from each of
these games and then we track checking
how well the algorithm does this is also
what was done for your original dqn work
we will allow some Atari specific
knowledge for instance the size of the
inputs are fixed across these games
which in a typical case is a downsampled
version of the game to 84 by 84 pixels
which is then fed to the agents in all
of these games it's exactly the same so
we're not considering how to deal with
say non-uniform in-ear observations but
it's fairly mild knowledge and we're not
putting any basically we're not putting
a lot of solution related knowledge in
there we're just putting in some
structure that allows us to play in all
of these games but we're not telling it
what even what the actions mean or even
what the agent is in each of these games
so the question is how could or can we
build an agent that plays well and the
starting point is the D queuing
algorithm to recap it includes a
convolutional neural network which takes
these pixels input it actually takes a
stack of a few frames of pixels this is
important because it doesn't really have
a memory component otherwise and Friends
is in the game of pong you have to hit a
ball from one side to the other and if
you don't have a couple of frames you
can't tell which way the ball is going
so that might make it harder to predict
an accurate value but if you just
stacked four frames say then you can
basically see which way weighting Bowl
is going so it's not a strong form of
memory but it's enough to detangle these
things this maps into a vector of the
same size of the number of actions that
you have so this is a discrete action
set these games have between 3 and 18
actions and we basically just output a
vector with a new property number of
elements which means for each state we
gave you all the action values and then
you can just grab the relevant one to
update this is combined with an epsilon
greedy policy which is quite maybe an
unsophisticated unsophisticated way to
explore but it works quite well we
garlis experience replay so we have a
replay buffer in which we store past
transitions this typically has some
windows so at some point you start
throwing away all transitions when you
add new ones and then use your sample
from that uniformly to update your
network it's not quite Dinah because in
this setting we're not actually using
the fresh data to update the network
which Dinah proposes that you should be
doing and maybe you should be doing it's
unclear this may be an easy thing to try
there's a target network which basically
basically means we have a copy of the
parameters which we keep fixed for a
while as it says on the bottom for say
10,000 steps or maybe 2,000 steps this
is the parameter you can set and then
every one so often you just copy in the
latest online parameters into this
parameter vector and this is used in the
bootstrapping so when you want to see
the value of the next state you use
those parameters instead of the online
parameters the idea of which being that
is keeps your target a little bit more
fixed and this might make the learning
easier and it was found in the original
work that just helped then we have a
loss this is one step cue learning in
this case using that target network and
we have an optimizer that minimizes that
loss there no there's to stop gradients
on the value at the next state which I
put here for completeness but if you
consider there's a loss of the online
parameters the next state value doesn't
even actually depend on those parameters
directly because it's using the target
network parameters but just for clarity
to stop grading it's still there and
then you just use some optimizer in the
originals you can work this was rmsprop
so that's the QN and then the first
components is basically wqm so it can be
very quick about this there's already a
target network so we already have two
networks which you need to do double Q
learning so what we'll do here is we'll
pick the maximum action according to the
online network and we will value it
value if you evaluate that according to
the target Network and this gives you a
form of double Q learning which you can
unplug in and this gives you then what
you could call W double D Q n and this
was shown to give you a healthy boost in
performance already because apparently
in some of these games the over
estimations were quite pronounced
which would hurt hurts performance I
assume this is roughly understandable
because we covered double curating
before but stop me if anything is
unclear okay
next components prioritize replay this
is related to our previous lecture where
one thing that we notice is is if you
have a model you might want to actively
query this model you might might want to
think about which things you grab from
the model this applies when you're doing
dinah when you want to generate some
data from your model to learn from then
it might be appropriate to think about
which data do you want to generate
generate and prioritize replay gives you
an answer to that where I used to
prioritize transitions on which you can
learn much now how do you know you can
learn a lot of certain transition well
one way is to look at the the magnitude
of the loss on the previous time you
looked at that transition because if
this magnitude is high that means that
when you would do an update with this
transition that the gradients would
sorry the gratings would also be high
and you would change your parameters in
your network quite a bit now before
implementing this or trying this you
might think maybe there's a caveat maybe
this is actually the wrong thing to do
because maybe the loss there is high
because it's just intrinsically very
hard to learn that and this might still
be true in certain cases but it's most
found at least in the in this setting
for DQ n that this is a very good signal
and if you prioritize your updates
according to this signal then you get
much better higher-quality updates you
get much faster learning and this might
be related to the fact that the deep
neural network isn't actually that big
for like commande deep learning
standards but it still has millions of
parameters which might be quite a lot
for these atari games so the network in
some sense might have sufficient
capacity that you should be able to
suppress the loss pretty much everywhere
at ease up to a degree and if that's the
case then it can't really hurt to try to
actually learn everything if you're in a
different setting where certain things
are just intrinsically Hardy you might
never learn them maybe it's the wrong I
need to focus too much on them but here
it's perfectly fine turns
there are some additional design choices
for instance I put like a bullet point
there that says sample according to the
priority so we're not actively like
picking the highest priority sample now
we're actually ranking basically the the
experience in the replay buffer
according to the priority but then we
were still sampling to get some
diversity in a typical case and there
are some parameters involved in that how
much do you sample how much do you care
about the priority compared to being a
little bit more uniform some design
choices there which might be important
to push performance up but the is the
main idea if you just implement a fairly
vanilla version of this it should
already help so the main idea is just a
prioritization and that's the important
bit perhaps okay so the next components
so again I'm going fairly quickly
through these components so just feel
free to stop me the net in the next
component we're going to do something
which you could call reinforcement
learning aware deep learning which is
maybe a simple idea if you think about
it in hindsight which is that you can
think of these action values as
decomposing into separate parts for
instance you could think of them as
decomposing into a state value part and
an advantage part where this advantage
now is basically the advantage of taking
that action in that state and one way
you could set it up is to basically
change your architecture a little bit
where you have a separate stream that
goes into a value and a separate stream
that goes into an advantage vector and
then you just add these together to give
you your action values there was an
additional bit here which is missing
from the slide which is that this gives
you an additional degree of freedom
normally let's say you have ten action
values you should then also have ten
advantages and you have one state value
so now you have eleven values to learn
that might not seem so much of a problem
but it actually means that the state
value can go up arbitrarily and then all
of the action advantages can go down
arbitrarily and this might cause some
instability or the other way of course
and it turned out to work better if you
subtract the mean advantage from this
which basically means that we're telling
the state value you
should basically consider the advantages
to be an on average zero and then you
should be estimating the state value
condition on that so you're really
trying to estimate the real state value
in some sense and then these advantages
can just learn to be the offset around
zero for specifically each action and
this turned out to work much better than
if you don't do that you could still
just put that into the architecture so
as far as the learning algorithm
otherwise is concerned we're still just
doing something on some action values so
this is in some sense hidden for the
rest of the algorithm that this is going
on but then you could still apply this
and you could still see if that helps
and let me show you a video I don't know
whether I'm going the right direction
with this one no wrong one
okay and I don't know how clear this is
on the screen I'm going to just quickly
tone down the lights I hope I don't know
which button does what so bear with me
okay this seems to be working before I
play the video I'll tell you what you're
going to see where you're going to see
is two visualizations of the agent
running on the same game which is a
racing game you're the car at the bottom
there and you can basically go left and
right to avoid bumping into things
superimposed on each of these screens
there will be some reddish blobs these
reddish blobs are the gradient with
respect to the input of either the value
side of your your action values or the
advantage side of your acumen tell us
what that means essentially is we're
looking at how much attention are you
paying to specific pixels of your screen
to predict your value and on the one
hand that's on the left or to predict
your advantages that's on the right and
what you can then see is that there's a
lot of attention it's in a sense where
you see these flashes you can kind of
see it now there so what you'll see on
the right hand side it's a little bit
less clear and it's
more sparse to signal but you'll
sometimes see flashes which are much
closer to the car especially when
there's other cars close because at that
points it really matters what you do
which means that the advantage is they
really care a lot about what happens
there at the bottom of the screen the
value on the left-hand side is then the
state value which we decomposed from the
advantage and the right-hand side is the
advantage and then together if you would
add these together you would get your
action value thanks that's good question
yeah yeah sorry I said that this could
happen if you don't do anything else but
then I then I said you what actually
happens in practice is that when we do
this when we create our action value we
subtract the mean advantage which means
that now they now they can no longer do
that because if the value would then go
up indefinitely the advance this could
go down but the subtraction of the
average means that that doesn't do
anything for your action values which
means that if the value now goes up your
action values will just become wrong so
now the value function is actually
basically pegged to the true value
instead of being able to to go up and
down and that's an important basically
implementation details or you could
think of it like that it's an important
part of the algorithm so this is the way
rich likes to depict these things where
these round circles depict states and
then the solid surfaces depict actions
so normal temporal difference earning
would start in the states we take one
action which bootstrap in the next state
and then you could consider doing this
for two steps or more steps in in the
end if you go all the way to a terminal
stage which is depicted here by a little
square let me turn the lights back on
sorry then then you have Monte Carlo now
we talked about how to then apply this
within the temporal difference learning
algorithm for prediction in which case
we're considering doing this basically
all of this is conditioned on a certain
policy there's a certain policy you're
following and we're trying to predict
the value of that policy and then you
could use TD learning which we which is
the one step return
we sometimes call it that and you could
consider Montecarlo and you can consider
everything in-between
so in general we'll talk about an N step
return or a multi step return which you
can use as a target and what we've seen
before in the lecture I discussed these
things is that there's a trade off and
typically you're best off not doing the
one step not doing the Montecarlo
but somewhere in between now we could do
something similar here but not bootstrap
on the state value but we're just just
going to bootstrap with in this case the
double Q bootstrap target this means
we're doing multi-step Q learning but
there's something a little bit maybe
partially weird about this because the
trajectory before we bootstrap would be
on policy but then we're bootstrapping
off policy with respect to whatever the
current greedy value course are greedy
action is according to your online
action values that's still okay this is
less greedy than a normal thing but
still policy improvement step in fact
even if you would be fully on policy
typically we take we do some exploration
with in this case epsilon greedy which
is already a way to be a little bit
greedy with respect in some sense with
respect to your current values which
means even in that case you'd already be
doing something which is akin to policy
improvements but especially here with
the bootstrap target which is off policy
you're doing policy improvements and
it's also still a well-defined
prediction target it's just a little bit
of a more unconventional one where the
prediction target is now sorry sort of
the semantics of the prediction target
is now what if I'm on policy for a
number of steps with in this case the
epsilon greedy policy and then I take
the greedy action which is maybe a bit
of a weird question to ask if you if you
would just be posing questions
prediction questions but it's a
perfectly valid one that you could learn
about and then you could still hope to
have the nice trade-offs between TD and
one end a multi-step of or Monte Carlo
earning in the other extreme on the
other hand okay so now I'm going
to go into depth into a little bit of
depth at least into something which is a
little bit more technical which is
distributional reinforcement learning
I mentioned this all the way in the
first lecture but I didn't explain how
it works so now explain a little bit how
this works or at least one instance of
this because there's now also other
instances and you could probably imagine
more this is a fairly recent thing the
citation up there says 2017 people have
investigated or considered similar
things in the past as well but this is
this is a very nice example where it
also showed like a nice performance
boost by doing this and the idea is to
basically go beyond expects its
cumulative rewards which is the thing
that we've been learning so far the
expected cumulative rewards could be on
or off policy but it was still this one
thing that we were trying to predict and
the realization is we could also try to
predict other things for instance we
could try to predict the distribution of
the returns instead of just this one
mean of it we could try to basically
somehow capture more of the structure of
this return knowing this might be
helpful for some things and I
deliberately kept it a little bit vague
for instance you could reason about
determine the probability of termination
let's say your actual reward function is
always one or maybe it's always zero and
it's one or minus one when you terminate
then you can maybe reason about okay
what's the actual distribution of these
things how likely am I to terminate when
I go here or there which might be useful
for instance if you're doing something
where you need a little bit of safe
exploration perhaps where you don't want
to necessarily go to places where
there's a probability that you might
terminate or your robot might break down
so sometimes these sort of things are
considered in that context you could
also even consider to having risk
seeking agents rather than risk-averse
agents one thing to note though is that
this is distribution of returns doesn't
give you your uncertainty it's actually
the distribution of the returns from
this state so it's not about how much do
you know about this this state value you
could have a lot of uncertainty about an
expected return but instead this is
basically trying to capture even if this
distribution is irreducibly high
variance
right there's just noise in the
environment and you can never collapse
this distribution to us to a single
point if you want to capture the full
distribution
it might just remain like that even if
your uncertainty about what the
distribution is goes down let's
distinction that I just want to make
make clear but it's still a lot of
things that you're trying to learn in
addition to the average things which
means for instance that your
representation might be first forced to
learn more
which might maybe not necessarily
immediately sound like a good thing but
it actually is especially if the normal
signals quite sparse or low information
which it often is in reinforcement
learning we just have this one scalar
signal that we're trying to predict and
then maybe trying to learn more learn
more about it just gives you more
learning signal to uh title to your
weights in your deep neural network so
this can speed up learning because
learning more about each sample
potentially means you need fewer samples
or if you need to be a little bit
careful about these things if you try to
learn things that are completely
unrelated to the thing you actually care
about then you might find that you're
using certain function proximation
capacity and you get inference and
things get worse on the thing you
actually care about but in this case
we're learning lots of things which are
quite related to the thing we care about
even if we're only interested in the
expected cumulative reward even if we're
not interested in risk seeking or
risk-averse agents then still these
things are fired fairly well aligned we
might hope that our network just learns
faster so an example of this is
categorical dqn or c-51 it's called in
the paper and this in this specific
instance there will be a fixed finite
support on which you are considering the
values to be able to lie so the support
here was picked explicitly for these
Atari games to be between minus 10 and
10 with increments of 0.1 I hope I did
that correctly probably not it must be
that 51 points in total so anyway
between minus 10 and 10 you sprinkle 51
points and then basically what what the
semantics of what we're trying to
predict is for each of these points how
likely is the value to be is the return
to be equal to that point of course it's
never going to be exactly equal to that
point and that's fine
so whatever basically dudes will
one way to interpret is that will
basically try to map the distribution on
this comb to be as close as possible to
the full distribution which we might be
more continuous between these different
values so for each of these points of
support we'll assign a certain
probability I put this between quotes
because you could also interpret this
just as a weight but let's call it a
probability and then this defines the
distribution we have the support we have
the probability for each point in the
support this together gives you your
fuel distribution and then we can of
course use that distribution to get your
for instance your mean action value and
in this case this is simply just doing
the dot product between your support and
this probability vector that you get and
then this thing should be approximately
equal to your action value that will be
one goal and then you can use that thing
to act for instance but the goal now is
to learn these probabilities rather than
just this reduction to the mean which
should mean we're catching more
structure and so how do we do that well
it turns out you can actually define a
bellman equation on these distributions
and specifically how that works in this
specific example is that we first
consider in the next States
we'll pick let's say we're doing Q
learning so we will just use the normal
online parameters you could use the
target params if you want to do double Q
but we'll pick the greedy action in the
next state according to the mean that's
same and then what we'll do is update
the support and this is depicted here on
in the picture on the right which means
we have a certain distribution which is
here depicted as a bar plot but it's
actually more more like a comb but this
is easier to visualize and then what we
basically do we first string that
according to our discount facts and then
we shifted according to the reward so
we're basically just moving this around
according to the sample that we got this
is for the one-step case then this new
distribution which is depicted in green
there at the bottom left it won't
actually map home to the support that we
had
we need to do an additional step there
which is basically to map it to the
closest distribution on the support that
we're allowing on these points that we
defined which is a projection step and
then we have a new distribution which is
defined on the same support that we
already had at the beginning so now we
can say okay the distribution of my
current state action value needs to be
closer in a sense to this distribution
you can consider this basically your
targets in the normal expected case this
would be say rewards plus discounted
next value but in this case it's now a
distribution that we're updating towards
that's the way to think about this and
then we just use that basically as a
target but in the normal case we can do
a square loss essentially we're update
our value towards this targets in the
distribution case it's maybe more
appropriate to use something like a
callback library divergence which is a
typical thing that is used to match
distributions onto each other this is
basically you can think of it as a loss
on distributions where you normally use
the square loss if you're not that
familiar with these so essentially this
part here which is in the picture which
is from the paper that I cited up there
shows you the first three steps and then
you still need to update your parameters
to have the distribution at your
previous taken action to more closely
map to this distribution at the next
after the one step that's a reward in
the next set so for details I would
definitely suggest you look at at either
the rainbow paper that I cited which
gives you the very short version of this
or of course for more depth at the paper
that that introduced this I understand
this is a bit fast but the idea is
hopefully a little bit clearer which
then brings us to the final component
which is noisy networks and the idea is
here to hopefully improve the
exploration as mentioned dqn use epsilon
greedy exploration which might not be
the most appropriate it basically just
picks greedy all the time and everyone
so often picks fully random which might
be a little bit of an uninformed way to
pick because you might know for instance
for sure that is one action is horrible
you should never take
absalom greedy doesn't know it would
just randomly take it everyone so often
so we learned that say UCB is better in
bandits but that's our with function
approximation as I explained because
it's hard to capture these counts
there's actually work on that trying to
do you see we like things in the deep
reinforce screen in case as well
but instead here I'm going to talk about
a difference proposed solution which is
to add noise and parameters essentially
so normally we have say a linear
function this vector Y is a linear
function of X where the weights W and by
is B are the things that we're wanting
to learn let's say so this is a linear
operation which also happens within
these deep neural networks specifically
in the dqn case we typically have three
convolutional layers which for this for
purpose of this thing you could consider
but we'll just skip over will not
consider and then there's two fully
connected or dense layers which are
basically linear operations like this
and there's no linearity in between but
we're not looking at that part we're
just looking at the linear part there
which basically means you're looking at
the features going into a layer and then
at whatever the output is of the linear
transformation of those features before
going into the next say non-linearity if
there is one and then what we'll do is
replace this operation with a different
one which is also linear but it we're
basically adding additional inputs in
some sense and those are these epsilon
epsilon W and epsilon epsilon B which in
the epsilon W there is a matrix of the
same size as your weight matrix and
epsilon B is a vector of the same size
as your bias vector so basically the
same size as as the vector of the linear
layer and then the idea is that we have
separate weights matrix W prime and bias
B prime that multiplied with these noisy
inputs in a sense component wise so you
could of course also imagine just having
one input that maybe goes into all of
these things but in this case it's done
component wise and what will happen then
if you train this thing well eventually
the network should learn that these
things are just noise
should ignore them so it should said
probably in the end it'll set these w
prime and B prime parameters all to zero
to ignore the noise yeah
you can add noise to the weights and
biases themselves as well yes but the
idea is say you then you have to tune
how much noise there is and you want to
reduce this noise appropriately somehow
which you might then want to fix and
what we're going to want to exploit here
is that actually learning takes care of
that it will set the weights to zero
appropriately over time but especially
for things you've seen often we're using
the same optimizer to update both our
normal weights and these additional
weights and the idea is basic to exploit
the fact that the things you've seen
often they will interact more heavily
with the noise which means that you'll
tune down the noise more quickly
which means that force a feature vectors
X that you've seen very often in effect
the noise will be quite small after a
number of updates and this might give
you propria separation because it means
that for inputs that you haven't seen a
lot you have more noisy outputs which
means you might do more random stuff but
if you've seen certain inputs a whole a
whole lot of times then maybe the noise
has disappeared and then you learn a lot
to explore there here will be greedy
there so one consequence of this is that
you're not necessarily equally greedy or
equally exploratory in all parts of your
stage space which maybe is a nice
property to have I'm not claiming this
is the only way or even the best way to
do this I'm just saying this is an
instance of something that you could do
and it has been proposed and it's so it
was proposed in a separate paper as all
of these components were and what we're
investigating here is how these things
can play together so what if we combine
is with all the other components in that
paper was actually a nice demonstration
they took a number of different
algorithms including dqn but also
including the actor critic
that we discussed in policy gradients
lecture and apply this idea to these
different algorithms and in all of these
cases they saw some gain okay so yeah
yeah so the question is is resemble
bayesian neural networks and indeed this
is very related to things that are in so
sometimes called Bayes buy back prop or
other Bayesian networks the idea is very
similar whether you want to view them as
a Bayesian thing or just as an
uncertainty thing maybe less well
grounded in the bay in part that's more
or less optional but it might very much
inspire where you go next right which
variants you might consider and there
might be ways to exploit a lot of the
knowledge from say Bayesian optimization
to be able to think of other ways to do
this and maybe improve but yes it's a
it's a good way to look at it it's nice
way to look at it okay so I think this
was a good good point for a break so
let's break for five minutes
okay so it'll come as absolutely no
surprise that we can successfully
combine all these methods so that's the
top line there rainbow-colored line
[Music]
it's a briefly explained where you're
looking at in this plot there is a line
at the bottom there which is maybe not
that easy to see but it's a gray line
that's the original dqn paper that's the
original DQ an algorithm I should say
and the x-axis here is millions of
frames in each of these games that
you've seen and y-axis is a median human
normalized score which means there is a
professional games tester that played
all of these games and what we're saying
is basically we're pegging whatever
scorer that person got on each of these
games we'll say that's a hundred percent
and then we'll play a random
agent we'll say this is zero percent and
this gives us a way to maybe
meaningfully combine the scores of these
different games because otherwise the
scores of the different games are
actually very variable some games have
scores in the millions
some games are scores in the single
digits and then it's hard to combine
these things so instead what we'll do is
roll up relying this human tester to
give us an appropriate scale so what
does means is if in a specific game you
get 100% that means you did as well as
that tester doesn't mean it's the best
possible or anything in some games it is
I mean on something like the game of
pong that's not a very hard game so
you're pretty much almost optimal if
you're if you played it for a while
other games are much harder and this
this tester also was the testing was
done under fairly constrained conditions
so he didn't get a limited time to play
every game before he was tested what a
hunk percentage then a good maybe
indication of what's a relatively
average human if they put a little bit
of effort in and a relatively good
obtain in these games
what score would they get and what we
see here is that the
formals of dqn by this rainbow algorithm
is attained in seven million frames
whereas dqn took 200 million which means
that learning is like roughly fourteen
times as data efficient and fast because
this algorithm isn't actually taking
that much more computer the algorithm is
actually slightly slower to run because
of the distributional updates although
there's ways to maybe speed those up
you're actually also just outputting
more stuff so there's a little bit of a
speed loss there but it's roughly on par
so the algorithm it doesn't run that
much slower and here we're basically
looking at the data efficiency which is
a huge huge gain of course more than
order of magnitude the other lines are
the yellow line is the A through C
algorithm which was notable because it
was were able to run without experience
replay on multiple processors at the
same time and by doing that it could
actually blaze through quite a lot of
data but it's not actually that data
efficient and one reason for that is
that it's not using experience replay
so a through C is barely more data
efficient and DQ and in fact the
learning curve goes up slower but then
it goes a little higher at the end there
but if you can think if you look at
Walker time it's fairly fairly efficient
though then the different algorithms
that were proposed that we discussed her
in there but I should also point out
here that's for instance the noise in DQ
n line there just adds noise in networks
to DQ n then above that we see the
double DQ n algorithm which just adds
double Q learning to DQ n but then a
couple of lines about that we see
prioritize double DQ n and Dueling
double DQ n and distribution DQ n and
these I'm not hung up to show the
distribution one actually but the
prioritizing duelling to use double DQ n
so they weren't using only this one
component prioritize or dueling but they
also added the double so these these are
not the specific components completely
detached from each other there are some
combinations there but we can see some
main thing here is basically compared to
the actual published algorithms in each
of these cases but what we could also do
is we can take that rainbow algorithm
and then we can
take away each of these components this
is called an ablation study where we
basically we have a bunch of new stuff
right we can't look at the full court
heating products of all the combinations
but what we can do is we can consider
roughly what this one is doing like I
said it's all exactly doing it but it's
considering if you start with the
baseline and you just add each of them
what happens then this plot is basically
saying what with what if you start from
the thing that has all of them and we
take one take each one away and what
then remains and that's also quite
interesting to look at and the way to
interpret this plot is to say is to look
at the lowest lines here basically means
that that component was really important
in this combination so the lowest three
lines are at the bottom are in yellow
it's the multi-step learning in blue
it's the prioritize replay and in orange
it's the distributional RL whereas
especially the blue and the and the
yellow one learning is also quite a bit
slower there at the beginning if you
don't have those components then they're
so noisy networks and somewhere more up
there we see the dueling and the double
components so one conclusion is that
these components were very well together
and that maybe in this specific
combination the most important
components were process replay and
multi-step returns and to a lesser
degree the distributional reinforcement
learning yes yes if there's no noisy we
do still use exploration but we're
basically swapping these out yes as a
good question thanks and in this setting
the least important were double and
duelling which was maybe a little bit
surprising because both of these gave a
huge boost in performance when they were
first proposed but one way to explain
that is to look at the full system what
is it actually doing and in this case
there's actually no no way to wildly
overestimate your values because we're
doing this distribution of RL which
means that we have a fixed support and
in this case the support was like I said
between minus 10 and 10 so you can
literally not represent values that are
higher than 10 so that sounds like it's
it's a way to combat our estimations but
it might be a bit arbitrary where it is
come from well this requires you to know
an appropriate range for these values
different games have very different
scores as I mentioned but what turns out
to be the case is that in basically all
of these algorithms that I just
discussed we have been clipping the
rewards which means that we're not
actually looking at the difference in
score you know actually optimizing the
actual score in these games but instead
of the rewards were clipped to minus 1 1
which means for instance that in a game
like pac-man where you eat pellets and
you can eat power pills and you can then
chase ghosts the algorithm has no way to
see the difference between eating
appellant and eating a ghost whereas for
the actual score there's actually quite
a big difference between those so this
was done to make learning easier but it
does change the objective and it might
lead to different performance so restore
a way not to do that
so this wasn't in the rainbow but there
is actually I mean there's always more
components that you can consider so we
couldn't consider all of them in the
rainbow but one way to do that is to
normalize targets before your do you do
an update so the thing to note here is
that an online reinforcement learning
you don't have access to the data set in
advance and why was this maybe an under
explore topic it I think it's because in
the supervised typical setting or
especially in classification you don't
have any issue of scale in
classification you know that
everything's between 1 and 1 0 & 1 say
all the outputs of your network but even
if you're doing regression people
typically just normalize the data and
then do the regression say which is an
appropriate way to do this if you have a
data set but if these things change over
time then it's much harder because in
these games for instance at the
beginning your scores might be quite low
but it might still be important to see
the difference between an extremely low
and a somewhat low score whereas much
later in the game your scores might be
thousands of times higher and certain
games and then it comes a little bit
harder to pick an optimizer and a
learning rate and things like that to be
able to learn across all of these
different skills because typically we
tune our learning rates and other
parameters to be appropriate for the
skill that we're at we can just
sometimes it's not even that we
normalize based on the data set we just
tuned our step size which is fine if you
have a fixed data set but in this case
it's less clear that that works well and
turns out he doesn't which is why we
were clipping the rewards so a proposed
solution here so that if you normalize
the updates and the specific algorithm
here is fairly simple I would say it has
more complex component maybe in the next
step also not that hard but so I thought
I'd walk through it explicitly it's more
general than the reinforced learning
setting but of course applied here in
reinforcement learning setting but it
may be it's more more generic applicable
tools to regression in general so we'll
just consider a target which I'll call T
and T Francis could just be your one
step Q learning bootstrapped
return and then the idea is simply to
have some normalization parameters for
instance you might keep track of the
first and the second moment from which
you can reconstruct invariance of course
you might need to be a little bit
careful here on this last step I didn't
put it on the slide but you have to of
course make sure that this estimate of
the variance is never below zero just
there's just numeric things that can
happen so you might want to do a little
bit of checking when you actually code
this up and then there's just some step
size on these statistics so typically
this is not that hard to tune because
you basically just want to have a rough
feel for where the mean of your target
is and where the standard deviation of
your target is and the idea is then to
basically consider an update for
instance proportional this would be for
the square at Los your update could be
proportional to where you take that
target you just subtract the mean you
divide by the standard deviation and
then you update the output of your
network towards that which means that
this is say roughly centered around zero
with roughly roughly a standard
deviation of one it doesn't really
matter what the targets are one little
thing to note here is that you can
actually do this and that's also what
happens there you can update your
statistics before you even normalize
your targets which means that the very
first time you see an extremely high
reward you can really correct for that
before you do your updates before
breaking your network before scrambling
your weights in a sense you can still
recover of course the unnormalized
target by simply just multiplying with
the standard deviation and and adding
the mean back in and that's also what's
used up there for the bootstrapping
because this is important because the
rewards
of a certain skill so to constructive
add a target with bootstrapping you need
to be able to recover the unnormalized
targets but that's fortunately very easy
to do now this is very simple you could
just try this but turns out maybe that
doesn't work that well because an even
permutation of this would change all of
the outputs of your network everywhere
whenever you update the normalization
and that might not be the right thing to
happen because you might be in a certain
state right now or especially in the
online case you might be in a certain
group of states where the rewards are
say fairly high and then you're updating
your standard deviation maybe your mean
things like that but this would
immediately also change your outputs for
the network in states you visit a long
time ago where you're not right now
because you're just multiplying these in
to get these on our normalized values so
there's an additional part of this
algorithm which basically says maybe we
should just change your network whenever
we do that to counteract the change to
the statistics so now we're only using
the statistics to update to change the
updates to the network but we're not
changing them to change the output of
the network and the way to do that is or
a way to do that is to basically realize
that this unnormalized output of your
network sorry do normalized output of
the network this q tilde which in the
previous slide I defined as the output
for the network network the thing you're
updating it's typically a linear
function in the dqn case we have a
couple of coneflowers and we have a
fully connected layer we have array lu
in between but then we have a linear
layer which goes into your action values
so you can ride that like this where
there's just some weight matrix
multiplies your state features adds a
bias and this gives you a vector of your
action values the action isn't there
because it's basically implicit in the
dimension of your vector that your
output and then there's basically a
simple idea here which is to change this
thing not with a gradient update but
literally just applying these
definitions here into a W Prime and a B
Prime in specifically this way because
whatever your Sigma T plus 1 is and
whatever your mu t plus 1 is this will
keep the output exactly the same as it
was before so if you change your weights
in one way and you basically change the
normalization up there in the other way
you basically change them in opposite
directions so that the total output is
unchanged
and you can exactly do that because all
of these operations are are just linear
so you can literally just if the outputs
it's exactly the same situation no
matter what happens to the normalization
that's nice
because again for the case where you get
a really big reward for the very first
time there might actually be quite a big
update to your normalization which might
otherwise still be harmful for your
network so why by doing this you
basically make sure the output don't
change but the gradient going into the
network is still properly scaled down
and then you update everything as normal
just using stochastic gradient descent
or atom or RMS book whatever optimizer
you prefer which means that basically
these weights now are updated twice once
to counteract the normalization and then
once with a gradient update now the
question is does this work so I call
this popper for preserve outputs
precisely which is that second step
while adaptively rescaling targets it's
a bit of a silly name but it's nice to
have a label to be able to refer to
these things so now we have a word we
can use to refer to it and what you see
here on the left is on these Atari games
essentially on the x axis you see
millions of frames and on the y axis
sorry there's a number of different
lines plotted their dotted lines and
y-axis shows you the norm of the
gradient on a specific game so there's a
number of dotted lines there which is
not really that easy to see but what
these filled in regions basically are
they're like the 50% 90% and 95% I think
a portion of games that fall within that
region and the y-axis there which is the
norm of the gradient is on a log scale
so what we can see if we don't clip the
rewards is that the gradient is going
into the network are basically six to
eight orders of magnitudes apart for
different games some games they're like
below one the normals are great in some
games are in the millions that makes
very hard for optimization
turns out we haven't really figured out
optimization that far yet that we can
handle these things very elegantly
so if you didn't clip the rewards which
is the middle part there on the left you
see that these norms are much smaller in
terms of range
they still span like maybe two to three
orders of magnitude but it's much more
constrained range than before and
especially their capped at the top you
don't get these extremely large gradient
norms which means it's easier to tune a
step size that works across all of these
now and this is what was done in say DQ
n and all of you follow what works of
that but if instead you apply this
adaptive normalization scheme the
gradient norms are even smaller in
smaller bands which means that maybe
it's easier than to do the optimization
now if you apply this in the full
setting so this was applied in this case
to double DQ n just add it on as a
component and turns out you get this
distribution of games what this shows is
basically the relative performance of
the algorithm with pop art and without
we're above zero means with dust better
and below zero means without this better
and surprisingly across these 57 games
it's basically on par in terms of the
median and the meaning performance but
there's huge differences per game some
games are way better and some games are
way worse now that might not sound like
an immediate win because why why have we
lost some games why are some games way
worse and my explanation to that is
basically we can look at the video now
is that the games have become harder in
some cases so this video again explains
basically the settings so the rewards
were clipped to plus 1 and minus 1 which
makes everything easier and this is what
happens if you run that on pac-man this
is the basically the original wqn and
what you see the agent doing is this
eating pellets it's also eating these
power pills and it's this turns the
ghosts blue which means you can eat them
for points but then it completely
ignores them it doesn't chase the ghosts
just try to eat them easily it's very
close to them well if we get better at
these things if we have better
reinforcement in algorithms obviously
doing the right thing going to the next
level shoot should be able to give you
more reward but in this specific case it
didn't and there's a number of other
games that are like that where basically
the it means that the optimizing the
score when it's unclipped is actually
harder to do and also
escorted its clipped and this might lead
you to worse performance so this is
popping up a level we want our agents to
understand and interact with the
environments and as mentioned the single
reward signal can be quite sparse
annoying information so we want to learn
many things this I mentioned is in the
context of the distributional RL where
we're learning basically distribution of
returns but there's more that you could
predict so the idea here this is from a
paper called hoard was this longer title
but the the idea is called maybe hoard
is that you have some you could you
could think of this learning function in
the middle as agents I just put learned
function into quite know what to put in
that box but the idea is that this is
your learning algorithm saying she'd RS
data coming in a stream of sensory motor
inputs on one end and there's a bunch of
predictions not coming out on the other
ends one example is the distribution or
L but you could think about many other
things as well there's a dotted line
here as well which means you might also
feed these predictions back in when
predict things about your predictions
which might seem like an old thing to do
but it's perfectly valid and it might
actually be quite a nice way to build up
knowledge where your turn now trying to
let's say some of these things that our
output are more like features and now
you're predicting the long-term features
so what are we then predicting well the
key idea is basically something that is
called a general value function so if
you discuss value functions a lot but
we've basically only been talking about
value functions over rewards the idea is
that a general value function is
something that is conditional more than
just the state in action and the
implicitly on the reward and the
discount factor now we're going to be
explicit about that we're going to
define something C which is accumulates
which is there in the subscript all the
way at the left a gamma a discount for
this cumulants and the policy that we
are considering now one thing that we're
already considered is if you just take
your cumulant to be a reward function in
your discount function to be the
standard discount maybe it's
undiscounted who knows we already
discussed you could learn about
different rewards you might use say
important Sam thing to do that in
practice it might be an interesting
question
would happen if I would follow this
reward well what this is saying you
could actually also consider other
signals to predict about for instance
you could predict what is the heat of my
motor when I'm doing certain things with
the robots how does that progress over
time how does that change over time or
you could predict how much rain is going
to fall over a certain period and you
could think of all these all these
questions that you could you could come
up with over certain horizons which is
then decided by this discount factor and
this could be useful knowledge in some
sense in fact there's this basically you
could call maybe the hypothesis or an
idea that rich doesn't likes to say
which is if you can predict everything
about the world
maybe that's all you need because you
it's hard to imagine other knowledge if
you can already predict everything so
how could other knowledge then still be
useful is there even other knowledge
that you can consider that is useful
apart from all the predictions of course
it's a bit of a there's a caveat there
because all the predictions is quite a
lot and hard to learn and hard to pick
and how to represent but maybe the
principle is sound so what we call this
we have this cumulants which is
basically standing for the reward
function the discount factor is
sometimes also called determination or
the continuation parameter which is now
also a function of states which can
differ from one state to the next and
then you buy your target policy it's an
open question how to pick these what
things to appropriately represents what
things to predict yeah the cumulants
is a very good question our cumulus part
of the states I need to probably be a
little bit more precise here the
cumulants are definitely part of the the
state as a whole so including the Asian
state and the environment state because
otherwise they're not there right they
need to be signals that you're
predicting they could just be part of
say your raw inputs you could be
predicting pixels they could be part of
your agent state you could be predicting
features all of these are sound but they
must indeed be real signals somewhere
so is this assumed with the model of the
environment that's not completely true
because as I said you could also predict
functions of state and in addition a
model of the environments but I actually
have a slide on that so I'll go back to
that modeling environment can be
considered one step prediction but here
we're considering maybe more generally
to also allow multi-step predictions and
also under different policies as far as
the cumulants go there's some truth in
what you say that these cumulus they
need to be somewhere in your input in a
sense and the input you could consider
the D input to the algorithm both the
raw observations and your current agent
state which is something that you build
up yourself but then if you allow for
that then yes the Kuna must be part of
that or a function of that so this is an
example recent example maybe the first
more general idea which is also the
title of this slide is universal value
function approximator x' which is
basically just we have this general
value function you might want to do
something with them so now can we maybe
learn these can we build function
approximation mission techniques or
can we build functions that actually
capture this and one key idea of the
universal value from Shiprock summations
is to basically feed a representation of
say a goal in this case I only took two
of them the cumulants and determination
and together you could call it a goal
you don't have to in some cases it's
appropriate you could feed a
representation of that as an input to
the network and what's nice about that
is that the network could then pretend
she learned to generalize over this
input which is a little bit different
from the picture that we had for the
hardware maybe they were kind of like
depicted separately but just you could
just think about these things I mean
here are the output side these
predictions are basically separate lines
here but you could also think about this
as being the the predictions are answers
to different questions that you're
asking and the question could also be an
input to your network and then the
network could maybe learn to generalize
over these inputs which means you could
also ask for new cumulants and
termination conditions on the fly and
hopefully get already a valid answer the
idea is that this allows generalization
across tasks and goals within an
environment one example of that last bit
there is there's a reason paper paper by
Denman kovitch and another's in which to
set up a situation where the actual goal
is to say get a chest but in order to do
that there's this sequence of things you
need to go through you're basically
first have to collect keys but because
with the keys you can then open a lock
when the lock is open you can open the
door or something like that you need to
lock for the door
in some sense and then you can go
through the door you can collect it test
that guess that's basically the D let's
say the abstract situation in reality
what was happening what the agent was
needing to solve was to collect certain
objects but for an object to be
rewarding it first had to collect the
different objects and a rubber sequence
is up to length for where basically for
the chest to be rewarding you first had
to first get the key and then the lock
and then the door and only then the
chest was a rewarding event if you would
grab it immediately it wouldn't be
rewarding if you learn this from
scratching your only focus on these
chests is very hard as an exploration
problem because there's basically no way
that your random exploration without any
feedback will learn to do exactly this
sequence of things unless you throw a
whole lot of data at it but then even
then in some caves you make it so hard
that it does basically never learns this
but you could also do is you could
optimize for all of these things
separately you could have a component
that tries to predict and optimize
grabbing just keys you could have a
separate component that tries to
optimize the locks and the doors and the
chests what happens is at first this
chest component will basically have give
you no direction at all but the keys
might be very quickly learned as a
rewarding thing in itself or I just go
and grab that I get reward don't know
why I just do that i granted reward but
then when you have the key all of a
sudden the locks become rewarding so
then all of a sudden you're a random
exploration Mises bump into a lock and
you might find hey that actually felt
good when I have the key
the lock is good and I can learn that
and now I can more directly go to the
lock and
and the next time it might have gone
through the lock to the door and so on
and that's basic what this picture here
on the year right shows that you get
more performance which is basically
different the magenta line here then if
you do a baseline which you doesn't do
this structured way of predicting
multiple things at the same time so this
is one way to use these multiple
predictions you could just be predicting
many things that could be useful maybe
act according to them occasionally and
see if this gives you better better
better exploration by switching between
different goals than just jittery me
randomly with your actions again this is
not necessary we find a word on this
it's probably not the final word on this
and there's many different variants you
could consider here but it's just the
idea a way to think about these things
and now going to the models as
appropriately asked already so a
transition model is a specific set of
general value functions where now the
cumulants are your components of the
state and determination / discount is
zero we're only looking typically at the
one step of course you could consider
multi step models but we've mostly
considered one step the models are often
action conditioning which means we don't
even have to care about a policy so the
policy is basically irrelevant because
conditioned on the one step and the fact
that you terminate immediately it
doesn't matter what your target policy
is so you don't have to care about that
similarly the unexpected reward model
would also would in this case use the
normal reward as a human but would also
put determination as immediate so I
wouldn't try to do the long-term things
and there's something nice about both of
these in in the sense that these
terminations are zero because it means
that the learning task is very
stationary I mentioned this when we were
talking about models this is basically
supervised learning now which we kind of
understand quite well a downside of it
is that if you change approximate
one-step models together the the errors
in these models might accumulate quite
quickly so it's unclear that that
actually works better than if you have
good multi-step models but then it's
unclear how to build good multi-step
models so that's a trade-off maybe the
formulation of general value function as
multiple multi-step models if we just
ignore the fact whether these are easy
or hard to learn is basically immediate
you just change this discount factor or
termination condition to something that
is not immediately zero and then you
have a multi-step model
you could also have to have things that
are kind of interesting but maybe a
little bit harder to reason about where
this discount factor might not be one
for a while and then zero but it might
actually be a slowly decaying one as we
normally do where the discount factors
say 0.9 or 0.99 which means we have a
multi-step model but it's a it's it
could be interpreted as one where you
can consider your model terminating at
each step softly or with a certain
probability which is maybe different
from the normal way we think about
models rolling forward a model me
basically is similar now and as using
predictions as input for other
predictions which is related to this
horrid picture that I have wherever we
have this dotted line going back as an
input you could change this you can do
this again and again and again and you
could also have multi-step predictions
on multi-step predictions if that makes
sense okay so we're almost done I
basically wanted to stop here at the
slide I had at the beginning as well and
if there's any more questions that I
wanted to leave a little bit of time for
that at the end although people have
asked questions along the way so so so
this is basically all the stuff that we
covered in addition to like the advanced
topics that we discuss now I in interest
of time I had to skip over a lot of
stuff I haven't talked about about
exploration that much apart from the
noisy networks and in the bandit setting
the other thing I didn't really talk
about that much was and I promise at the
last lecture that I might get back to
was alphago I just wanted to mention
that I deliberately didn't go back to
that here because for one I felt that
again it's something that a lot can be
said about and it's also something that
allows has been said about I'd be very
happy to point anybody towards resources
if they want so if you want to know more
about any topic specifically for
instance alphago but any other topic as
well feel free to let us know for
instance on Moodle and we can just point
you in that direction Moodle is
especially nice because there might be
other what other people are also
interested or might not even know that
are interested in that topic but if they
see it there they might think oh that's
kind of cool so we could have maybe a
collection of things there and also feel
free to share insights or resources that
you bump into yourself as well that
might be quite a useful resource in that
sense
okay so next week we'll have a guest
lecture at this timeslot by my flood who
did the original work on the QN and did
a whole lot of other work as well on
deep reinforcement learning and the week
after that we have Dave silver giving
you the guest lecture so so if you want
to bug somebody about alphago he's to
use the person to bug okay so that's it
for today thanks
[Applause]