Reinforcement Learning 9: A Brief Tour of Deep RL Agents

okay so let's start thank you all for
coming
I'm Vlad Nick from deep-lined and today
I'm going to talk about basically the
development of deep viral agents at deep
lines I'll talk about the various agents
we've used over the years and how things
changed so I guess it's good to start
with or start with a reminder for why
deep reinforcement learning is exciting
so I think the most exciting thing about
deep reinforcement learning is that it's
the right framework first start for
studying artificial intelligence or a
general artificial intelligence so
there's no perfect definition for what
that means and these days pretty much
any machine learning algorithm is called
AI but what I'm talking about is
basically building machines that are
good at sequential decision making
problems that humans care about and
specifically machines that are good at
many of these tasks at once so that's I
think a reasonable definition of
intelligence being good at many
sequential decision making problems
faced by humans so if we want to build
AI then it requires several components
sort of by definition we're looking at
sequential decision making problems so
we need something to deal with that I
think it's also not that controversial
to say that I think we will need
learning to do it so intelligence is in
static it requires being able to adapt
to the world so the machines will have
to learn from experience on the fly and
the last point I think also is in that
controversial and I think will need deep
computational graphs so pretty much all
examples of to working perception
systems and biology and in computer
science you
deep computational graphs like neural
networks but that's not really the only
reason for that it's not just about
perception so as soon as we start
thinking about time and sequential
decision-making requires dealing with
time than they think requires
accumulating information over time and
incorporating it into your belief about
the world so this is essentially a deep
computational graph so we'll need to be
able to deal with that so then what does
deep RL so it's the combination of
reinforcement learning and deep learning
and reinforcement learning comes in
because it's really I think the only
good framework for doing learning and
sequential decision making problems like
you know the current methods or the
methods that are used currently might
change we might end up using different
methods in the end but reinforcement
learning gives us the right problem
statement which is how do we learn in
sequential decision making tasks so
where does deep learning come in well
for one I think it's currently the best
way of making computers perceive the
world so if we want computers to see in
here currently deep neural networks are
the dominant approach but again we don't
know if the set the approach is being
used now we'll stick around but to me
what deep learning is it's really a
framework for learning and deep
computational graph so again the methods
might change but it's really the study
of learning in such models so putting
them together I think we basically have
the right framework for studying
artificial intelligence ok so how did
deep reinforcement learn and come about
well I think you've all been going to
the other class on deep learning and I
think it's interesting to look at the
parallels between the two fields so like
deep learning
think deeper all started in the 80s late
80s maybe early 90s and I think the
first notable example of what I would
call a deep RL model is the TD gammon
program from Jerry to sorry so this was
a program that learned to play
backgammon at essentially human champion
level by playing against itself and this
was done in I guess late eighties and
early nineties so this is kind of
amazing because this is you know several
more than twenty years ago and it was
already this impressive achievement and
I think when I was starting to study
machine learning this was one of the
most impressive applications of machine
learning period but somehow it didn't
lead to I guess systems similar systems
that worked well another task so not
much happened in the mid to late 90s so
things started picking up again maybe
five to ten years ago so people didn't
work on their own networks but around
2012 when neural networks were becoming
popular again people started looking at
it applying them to reinforcement
learning so what happened then like in
deep learning where it was a combination
of fast hardware namely GPUs good
datasets
specifically imagenet and a few tricks I
think essentially the same thing
happened in deep reinforcement learning
so in 2013 there cade learning
environment was released which was
basically an interface for playing Atari
games with reinforcement learning and
this was really the right platform for
showcasing deep aural some from from
there basically the work on deep cue
networks came out which showed the
convolutional neural nets can do quite
well on
reaching human level play on a number of
games and from there I think things
really took off and it's again this
combination of hardware GPUs and fast
CPUs they're having the right
environments Atari and say physics
simulators like mojo CO and a few new
methods for training neural networks and
tricks for designing them and around
2015 one alphago over 26 yeah 2015 when
alphago came out since then it's been
this explosion of interest in the field
so one thing that's worth wondering is
okay so this course is two parts so
there is a deep learning part and then
there is the reinforcement learning part
and is it just a matter of you know
taking the best off-the-shelf RL method
you learned about and the latest
greatest neural networks you learned
about I'm putting them together so I
guess as you've learned in the sports
already it's not that simple that the
reinforcement learning problem is quite
different from the standard supervised
and unsupervised learning setup that you
see in deep learning so one of the
biggest differences is of course a
feedback can be sparse and typically
delayed so it wasn't clear whether that
would be enough to train neural networks
as opposed to supervised learning where
you have the right labels for every
input what makes it even more difficult
is that the data distribution is
non-stationary and specifically in a way
where it depends on the agents actions
so it's one thing if it's just
non-stationary like maybe video or sound
but here it's basically the agent
determines which data it will see so you
have problems like exploration versus
exploitation if if your policy converges
to something deterministic
too early you will maybe not explore and
finds even more rewards and because of
this local minima do happen in
reinforcement learning so in
unsupervised learning I think there is
more theory and experimental evidence
that if you have a big data set and you
train your neural network with
stochastic gradient descent then there
aren't really bad local minima to worry
about for the right architectures so in
reinforcement learning this is not the
case you can definitely get stuck in a
local minimum by converging to something
deterministic here early and again any
kinds of non stationarity in the data is
a problem for neural network so the the
theory usually assumes ID data and once
the data is not iid convergence
guarantees are very hard to combine um
and yeah in the late nineties the RL
community was even a lot of people in
the RL community were saying that neural
networks are just inherently unstable
and people shouldn't use them with
reinforcement learning or at least the
methods that were around them so I think
in terms of how the two parts of the
course connect this tweet sums it up
nicely so Andrey Carpathia is a former
deep mind intern who's no director of AI
of Teslin and about a year ago he
tweeted ok everything I know about
design of confidence resonance
bigger-is-better batch norm is useless
and are all super basic four layer
confidence for Ignace so this there is a
lot of truth to this and I think
basically in the rest of the lecture I
want to talk about how how things
started how agents evolved and where we
are now and we'll get back to this tweet
at the end of the lecture ok so more
detail will
a closer look at dqn which you've
covered I think in some detail but this
will be a slightly different perspective
I think then we'll look at faster
training through parallel or
asynchronous agents and then we'll look
at more recent work on scaling up RL or
specifically scaling up distributed
training of reinforcement learning
methods and in the end I'll conclude
with basically some practical advice for
how to get reinforcement learning to
work with neural networks okay so what
is the basic idea behind the DQ n so
you've learned about value function so
really the key idea is if we want to do
RL in a rich perceptual domain like say
Atari games then let's use a
convolutional neural network to
represent the action value function Q
and let's take you learning a method
that we know works well and has nice
convergence guarantees and train this
end to end so how can we do this in a
stable way so probably it makes sense to
go back to just what key learning looks
like in the tabular setting so we start
with the guests for each action value
for a state as in action a and then the
agent starts interacting with an
environment by say epsilon greedy
exploration so you most of the time will
pick at the action with the highest
value but with permeability epsilon will
take a random action so the agent will
interact with the environment and
collect tuples of experience which are
basically state T action T reward Tian's
state T plus 1 so then the basic key
learning update looks like this which is
really turning the bellman equation and
turn it or a dive update so we move our
current estimates of the action value 4
State St in action 80 towards this
target which is the reward we got plus
the discounted value of the best action
at the next state according to our
estimate
Sargon and tabular environments under
reasonable assumptions this will
converge to the optimal action value
function and then you will have the
optimal policy if you are agreed or at
least an optimal policy since it's not
necessarily unique so what could go
wrong if you wanted to apply this to
training their own network so there are
two potential problems the first one is
that again typically in tabular
q-learning you're interacting with the
environments and you're applying these
updates online so successive updates
will be very correlated and neural
networks don't like this so a good
analogy is if you're training a
confident to classify them as digits
typically you shuffle the data if you
don't if you for example first train on
all the ones then train on all the twos
the network will not learn very well it
will be terrible actually unless you do
something to safeguard against them but
basically in the standard RL setting
where you're interacting with the
environment online applying the updates
as the data comes in gives you a similar
effect where the updates will be two
correlated and it can potentially harm
learning or basically prevent the
network from converging at all the
second potential issue is that we're
using our own estimates of the Q values
at the next state as the target for the
Q value at the current state so because
we're parameterizing q with the neural
network these two Q values
we'll also be quite correlated because
if you think about the inductive bias of
neural networks I think currently and
there is still this belief that it's the
inductive bias is basically smooth
functions so if you have two successive
inputs which are you know maybe
consecutive frames in the Atari game
then these will produce very similar Q
values because that's what the neural
network does so we have these two
sources of correlation which is one is
between successive updates and one is
between the targets and the source
values so what does DQ m do so the
high-level idea behind DQ n was okay and
2013 people had a pretty good idea of
how to do supervised learning with
neural network so can we make
reinforcement learning look as much as
possible like supervised learning and
there are really two key tricks that
help achieve this so the first one is
this idea of experience replay which
people were using in the early nineties
but McDonough was seen more as a
technique to improve the data efficiency
of algorithms so what does experience
replace instead of applying updates
online your agent just interacts with
the environment and as the experience
comes in you put it into a giant buffers
and frames where M could be something
like a million so now you have
essentially this big data set of
experience and to learn you just sample
from sample mini-batches from this
replay buffer so now as the agent is
exploring the environment the the many
batch samples you'll get from the buffer
will look a lot more like fixed data set
so now we will have less basically
non-stationarity but
between successive updates because the
data is changing very slowly so that
essentially addresses the first source
of correlation and again this is
something that's been around for a long
time but people saw it more as a way of
improving data efficiency because you
don't have to interact with the
environment as much so then the other
idea is basically looking at the other
source of correlation between our Q
value for the current state and the
target and the idea was to basically
just fix the set of weights used to
estimate the target values for periods
of time so how can you do this while
you're changing your neural network and
you will periodically take your weights
and copy them into the target network
and then use those weights to compute
the target so as you're getting more
experience or as you're updating the
estimates of your Q values that the
targets values will not change as much
so this again reduces the correlation
between these two Q values so to get a
little bit more intuition of why these
correlations are a problem and how this
fixes it we can look at basically you
say the game of space invaders where we
have here are two successive frames from
the game where there for Frank well
they're actually four frames apart
because they can use action repeats but
these are two successive frames that's
the algorithm see so it's trying to
predict the value of this state from the
value of this state on any rewarded gut
in between so these look very similar
and again because neural networks have
this smoothness as the inductive bias
they will tend to produce very similar
key values for these so if you use the
same weights to compute the targets and
the source
as you increase the value of this Q
value because you're getting a reward
for shooting down a space invader you
will also increase the key value of the
target and the next time you come back
to this couple of experience you will
have a higher target and then you will
increase again the value of this as
staying active or you will increase this
state action value which will again
increase the target so the network
essentially can end up chasing its own
tail and coming up with ever higher
estimates of the key values but if we
introduced this target network and we
use fixed weights to compute the target
values then the target will be the same
the first time and the second time you
see this topple until you copy in the
new weight so you essentially now have
these periods in which the targets
values don't change and now basically
dqn it starts to look like supervised
learning for while the target weights
are fixed you have a very slowly
changing data set and you have a fixed
source of targets so you're just fitting
the values to that and periodically you
have to update these with the new
information you've learned okay so
probably we don't need to go over the DQ
n training algorithm but it's basically
the interaction loop where you put data
into a replay buffer then you sample
mini-batches
and apply the DQ n update rule yeah
right so aliasing I guess I guess yeah
at the beginning of training the neural
network might not know anything about
its random weight so if you have two
consecutive states with the reward in
between
then the right if it fits the true
action value function that should
produce very different values for these
two states even though the inputs look
very similar so I guess I would say
aliasing is when a neural network Maps
different inputs to the same value which
happens at the beginning because it's
it's outputting smooth
I guess random values so yes yeah this
is particularly difficult difficult for
neural networks producing very different
outputs for very very similar inputs so
so it's related it's double key learning
goes a step further so it uses a target
network not just to compute the target Q
value but yeah so double Q learn or in
DQ on the target network is used to both
estimate the value of the target Q value
and to find the best action at the next
state yeah and in double Q learning you
basically use one of the your networks
to find the best action and the other
networks estimate the value okay any
more questions at this point
in the buffer yeah so we sample
I guess uniformly at random from the
replay buffer but the replay buffer
itself is essentially like a first in
first out queue so the oldest the oldest
frame gets removed as you add something
new so yeah so that's a good question in
dqn we did only the batch training but
there is some recent work showing that
if you learn on both the online
experience as you come in and on samples
from the replay buffer this ends up
being less sensitive to the size of the
replay buffer so in practice this I
think this tends to be what people do
now okay so one thing that's it's always
good to look back and one reasonable
question is okay if you go back and read
the RL literature people I've been
training neural networks with key
learning for a long time so along Jilin
who was who came up with experienced
replay he was doing key learning with
neural networks and more recently Martin
Reed Miller and many others have done a
lot of work on training neural networks
with key learning so what's the what's
the difference and why did we even need
in the algorithm so probably the most
similar method to dqn was neural fitted
cue iteration so neural fitted queue
duration takes the idea of turning RL
into supervised learning even one step
further so the way it works is if you
train a number of neural networks so you
start with the random one you collect
experience with this random policy
then every time you get a new episode
you will take all your data and compute
target Q values using your current
network then you will throw away your
current network weights and you will
train a new neural network from scratch
on all the data using the target values
you computed and here the he didn't he
wasn't using mini-batch gradient descent
but he used our proper resilient back
probe which is a full batch method so
now essentially what you're doing is
you're training many networks one after
the other throwing away everything
you've learned each time and only using
it to compute targets but during every
training iteration the targets are
completely fixed and you're doing this
big batch training so this method is
quite data efficient and work quite well
on basically problems with small inputs
sizes so maybe controlling a robotic arm
from not from perception but from sort
of positions of the joints and things
like that so when we wanted to start
applying narrow networks to Atari we
looked at neural fitted queue duration
and decided ok we don't want to Train
many confidence training a single
confidence is expensive enough so we
somehow wanted to break this dependency
or break this need for training a new
confident after each episode and really
nfq was the inspiration for the ideas
behind DQ n and DQ n was in some ways an
online online mini batch variant of
neural fitted Q iteration so we we don't
throw away the weights during training
at all and the targets sort of change
slowly but there is the still idea of
target Network
so NF q is probably more stable
than DQM but again if you can afford to
train many copies of your network in
sequence then this will be a good method
to use but if you're doing RL from
perceptual inputs it ends up being too
expensive so again going back even
further to long Dylan's PhD thesis which
was on reinforcement learning for robots
using their own networks he was again
training neural networks with key
learning um and again he was using
experience replay but I think what's
interesting is the network's he was
using so again in DQ n the input is your
current state or observations and you
have n outputs one for each action in
the environment and in linz neural
networks he was training at a separate
network for each action so for each
action you have a network which given
the state gives you the key value of
this action so this is a nice trick
because it's sidestep some of these
issues with state and action aliasing so
in DQ n if you update the Q value of
action one in a particular state you
will inadvertently change the Q values
of the other actions in this state and
also the all actions in similar States
so in linz architecture there wasn't
this problem so he didn't really need
target networks as much because if you
change the Q value of like action one
the Q value of action 2 stays the same
but again duuude probably if you're
training a convolutional neural network
you probably don't want to relearn
you're confident for each action and if
you end up sharing the weights and using
I guess a DQ n type of architecture then
you end up needing these tricks like
target Network
so really similar methods have been
around but to really make things work in
a stable way with neural networks on
high dimensional inputs we needed some
of these things like target networks so
yeah what did them decay on neural nets
actually look like so this is going back
to andreas tweet so the confidence in
the first DQ on paper had an 84 by 84
input where it was a stack of the four
previous frames and we had two
relatively small convolutional layers
followed by a rectifier linear units and
then one fully connected layer and then
one output per action so not only by
today's standards but even looking at
the networks people were training in
2013 this like this is a laughably small
confident anyone dode saw it said ok
well surely if you use a bigger one it
will work better and in the 2015 DQ on
paper there was an extra convolutional
layer and it wasn't that we didn't try
even deeper in that we did and they
simply didn't seem to help so this was
somewhat puzzling but these were the
networks that seemed to work best
so yeah I'll show this video which is
the original clip of the first version
of DQ n from 2013 playing some of these
Atari games and so these days the the
best RL agents I guess are far beyond
human performance on most of the games
but back then the exciting thing wasn't
that D Cohen was super human on any
particular game but the exciting thing
was that the system was fix hyper
parameters in the same network
architecture
was able to do reasonably well across
many different games and if we go back
to the definition of intelligence it is
being able to learn or do well across
many tasks so this was exciting that it
was doing okay on many different
problems using the same architecture and
hyper parameters of course here there
was the network was retrained on each
game separately so it's not a single
agent that does well on all these games
but we'll look at that again closer to
the end of the lecture so at the time we
thought that basically that Atari wasn't
rich enough that you don't need that
much perception to play these games well
so a lot of these games are involve very
reactive control like say space invaders
you know you should dodge lasers and
basically find where the alien ships are
and shoot lasers back and you don't
really need very powerful or precise
vision to do well there so this was this
was the intuition for a while yeah I was
convinced that the environments are just
not rich enough but yeah we'll see near
the end of the lecture that yeah
actually if you use a good method you
can do well with deeper networks so it
was I think partly a combination of the
method not being as good as it could
have been okay yes
yes so that's kind of I guess yet the
next few slides I had
yeah nice tears lecture I had some
slides showing the filters the DQ
unlearned and I guess they were
interesting because they looked really
bad so they're not in the in any of the
papers so if you look at say the image
nets Convenant paper from kirovsky and
company they show these beautiful like
edge detectors and gabor x' DQ n an
atari basically the filters were very
noisy and you could pick out maybe a
laser in space invaders but they didn't
look particularly good which again is
partly behind the intuition that okay
maybe you don't need that much structure
to do well but yet the next few slides
we'll look at i guess how the agents
what what the agents learned so this I
think is a good example on pong which is
really I think the simplest game in
Atari so we're showing four frames from
the game so the trajectory of the ball
is superimposed it's not in the input
and what we're showing at the bottom are
basically the Q values predicted by DK
and for that do nothing action and up
and down so we see at this frame the
okay DQ n is controlling the green
paddle here so the ball is coming back
from the opponent and all the action
values are positive and pretty similar
so this is saying okay I think I can
score points but at this point it
doesn't really matter what I do and the
action value is not one because the
network knows that before it scores a
points and how the ball has to travel
back and because of discounting the
value is less than 1 so going forward a
few frames now
the paddle is still at the bottom and
now we see the the network is predicting
okay now you better go up if you go up
you can still get a point but if you do
nothing or go down you'll miss this
point and you'll get a negative reward
and again going forward a few frames
it's essentially the same but now the
action values are even closer to the
extreme so it's saying okay if you go if
you do nothing or go down now then you
will get a negative reward essentially
right away and if you go up then it's a
sin it's basically saying the positive
reward is a little closer now if you go
up and yeah the network the agent
returns the ball and we see that again
at the last frame here it's about to
score points and it knows that no matter
what it does the predicted values are
one and you will get a point so this is
essentially how it uses the action value
function to act so in the game there is
some generalization between games
there's very little because if you
change the input in any way like maybe
even one pixel can can mess it up
because again the network was trained
only on the frames from the game so
anything that's out of the training
distribution yeah can give nonsensical
results so this is another example where
we're showing a live visualization of
the state value function on breakouts so
here you basically see as the ball goes
up the value estimate climbs and that as
it hits a block it falls back down and
then starts climbing again so this is
essentially what you would expect
our answer is I don't know if it will
break you won't break through here and
now
yes space invaders so this is an
interesting example where so the
different aliens are worth different
numbers of points I think from five to
twenty five or something like that or
maybe thirty and this purple mothership
is worth way more point so maybe 200 I
don't remember exactly so we see that as
the purple mothership appears the state
value estimate climbs high and as the
agent tried to shoot the mothership down
but it missed and you see the estimate
Falls because of that and as it
continues playing so this is another
episode so here we'll see a similar
thing where the mothership appears the
estimate climbs and this time the laser
hits it and the estimate falls back down
so this is again this is from the
original dqn and today's best methods I
think learn much better action value
function estimates okay and I think yeah
one last example so this is the game of
C quest which we were really excited
when dqn started working on C quest
because it had interesting examples of
trading off immediate rewards for future
rewards so here the submarine is
basically shooting the Sharks but it's
running out of oxygen so at some point
you need to go up and get more oxygen
but because there is no reward for
getting more oxygen all that does is let
you play longer so the agent has to
figure out that okay at this point the
best thing to do is stop shooting the
Sharks and give up the immediate rewards
to get oxygen
and get more rewards later and this is
really this was really to us to proof
that okay it's not just this a greedy
thing that goes after the closest reward
and key learning is doing its job in
some way okay so to summarize key
learning or DQ n so this yeah this graph
here shows the progression of the stator
there it results in Atari I think
starting from 2015 going until today so
what it's showing is the median human
normalized score over the 57 game so
it's essentially the percentage for all
the games you compute the essentially
the percentage of the human score that
the agent gets and then take the median
over the 57 games and dqn was somewhere
around 75% and we've seen something
close to 50% improvement every year
since then and now the best distributed
agents are way beyond this but yeah and
there are references to these methods at
the bottom but yeah basically what are
the pros and cons of DQM and other
mini-batch key learning methods so
they've been successful in Atari and I
think one of the most surprising things
is they've been very robust so deep are
L often gets a reputation of being very
unstable and difficult to Train but I
still think that it's impressive that
basically all of these papers that
extended the queuing they pretty much
still use the same hyper parameters and
the same learning rates from rear the
original paper and this is a across all
games so this is kind of nice it's not
like when you train an image net and
maybe you retune your learning rates for
every architecture so it was really
quite robust
in many ways so also it was GPU friendly
from the start because I think yeah
coming from supervised learning the
intuition was okay we know bigger
networks are gonna help so let's put it
on the GPU right away so it could
support larger networks in principle but
actually in practice the smaller
networks that worked well didn't make
the best use of GPUs and partly because
of this training was quite slow so we
would train for a week on a single GPU
and this is for every game and also
these methods are specifically one step
cue learning that DQ n type methods they
never really worked that well with
recurrent networks and because of this
they haven't worked well in 3d partially
partially observable environments so are
there so yeah the inst I think it has
again something to do with staid
aliasing so if your Q function is a
recurrent Network then I think it's even
harder to just to make very different
predictions for two consecutive states
because now not only do you have the
current input which is similar you also
condition on the iron and state which I
guess carries a lot of the information
so I think it yeah it's just even harder
to get off the ground and start
distinguishing between similar states
with different values and as we'll see
in later in the lecture yeah one way to
deal with this is to go from one step
key learning to n step key learning and
I'll mention that in a few slides
okay so that was DQ n
so after yeah after developing DQ n we
started thinking about okay we want to
try many different problems many
different models and Digimon is just too
slow so can we do something better and
can we get away from training on a GPU
for a week and maybe train for one day
on a single machine so maybe we could
have something that can learn simple
Atari games in an hour or several hours
and also we wanted to try and experiment
with methods that are on policy so DQ n
uses replay for stability so it requires
off policy learning and at the time
there was this whole class of policy
gradient methods which were mostly on
policy and it was interesting to see how
how well they could do and also we
wanted something that was flexible so
that discrete or continuous actions
there are continuous action versions of
DQ n so methods like deterministic
policy gradients but they require
changes to the framework and also we
really wanted to Train recurrent models
easily to look at partially observable
environments so the next section will be
a quick overview of basically this work
on asynchronous methods for deeper
neural from 2016 so the framework we
ended up developing which we called
async RL or asynchronous RL
basically look like this so now instead
of a single actor in a single learner on
one GPU like in DQ n we basically have
parallel actor learners running on the
same machine and different CPU threads
and each one has its own copy of the
environment and it's interacting
producing experience
and periodically it would send gradients
to the shared model which because
they're running on the same machine that
they have access to shared memory so
each actor learner would asynchronously
updates the shared model with its latest
gradients and get the updated parameters
so here the idea was okay we because we
want to be on policy we're not going to
do replay for stability um so how will
this be stable so the idea was that with
K actor learners interacting with the
same environment we could hope that they
would at any given time they would be in
different parts of the environment and
even if they're applying online updates
to this shared model they're coming from
different parts of the state space and
the overall updates being applied was
sort of averaged out and we wouldn't end
up getting stuck so that's essentially
the framework and now you can plug in
try and plug in any RL algorithm of your
choosing online where on policy around
policy so we looked at a basically a
bunch of standard RL myth it's a
one-step key learning of course I like
in DQ n we also looked at n step key
learning so here instead of
bootstrapping from the next state you
act for n steps getting n rewards and
you would bootstrap from the your
estimated key value from n steps in the
future so this is again a very old
method and this something like this can
propagate rewards faster through the
state space so the hope is that this
would be much faster then we also looked
at actor critical policy gradient
methods so here instead of learning a
queue function
we learn a model which predicts a policy
which is just a distribution over the
valid actions and a value state value
estimate so this is again a variance of
the reinforced algorithm which I believe
has been covered but it uses a few
tricks to reduce the variance ernst in
reinforce you just use your empirical
return to estimate to basically update
your policy gradient so here because we
don't want to necessarily wait until the
end of the game we truncates the return
of Suren steps and use our estimate of
the value at that state and also you
subtract the value at the current state
to reduce variance so this is
essentially this is called an advantage
actor critic because it's multiplying
the policy gradient by an estimate of
the advantage so this term here is an
estimate of the Q value of the current
state which is partly based on empirical
returns rewards and this term here is
again just the state value so the action
value at a state minus the value at that
stage is known as the advantage it tells
you basically how much worse this action
is than the best or the expected action
at this state so this is the update that
it was using to train the policy and the
value function was just trained with n
step TD learning minimizing this mean
squared error so I'll skip through the
details a little bit so when we trained
this on the single machine was 16 CPU
cores and 16 actors we compared this to
DQ n running on a GPU which is shown in
blue and these are basically training
curves
five Atari games where the x-axis is
time in hours so what was interesting is
that basically all the methods we tried
one-step couponing sarsa which is very
similar instead of key learning and
actor critic they all worked well so in
some sense yeah all these off-the-shelf
methods when plugged into this
asynchronous training framework
successfully trained their own that so
that was surprising and kinds of nice
also because the networks are reasonably
small training that these nuts in CPU
was fine and in terms of training time
all of these methods were essentially I
would performing DQ and at least
initially but somewhat surprisingly we
saw that yet the sector critic method
was much faster and better than the
others in practice so how well does this
scale with the number of parallel
electoral learners so here we looked at
seven Atari games and we looked at
different numbers of threads and what
we're showing is basically how much
faster you get to a certain reference
score when you use Catharines compared
to one thread so you would expect that
perfect scaling would be with eight
threads or eight times faster so what
was a bit surprising was that okay actor
critic scaled reasonably well but some
linearly so it was 16 threads it was
maybe like 12 13 times faster than was
one thread and stop key learning was
about linear but the one-step methods
were scaling super linearly which was
kind of weird like how can you be 20
times faster it was 16 threads we should
only be getting advantage from
essentially parallelizing
so when we dug in what was interesting
was when we looked at the data
efficiency it basically confirmed this
hypothesis that using parallel actor
learners does indeed stabilize training
of neural nets with these methods so
here if we look at break head here we're
showing the amount of data consumed by
all the parallel actor learners on the
x-axis so this is the total amount of
data consumed by all learners and here
we're showing the score and again blue
is at the end what you get 50 million
frames with a single actor learner you
get 250 points on break code but if you
do 50 million frames parallelized over
16 threads you get to you know almost
300 so what's going on so basically a
single actor learner is doing online
updates and it is basically less stable
and less robust you end up having to use
a really small learning rate not to harm
your network with these correlated
online updates and as you add more
parallel actor learners you get this
averaging effect over the state space
and with 16 threads you're basically
yeah getting a reasonable stability by
applying updates all over the state
space so that's where the super linear
speed-up for one step key learning comes
from so you're not only getting the
speed-up from parallel training you're
also getting improved data efficiency so
in practice this adds up to more than
linear but of course is you don't
actually want to use one step key
learning here because unstuck Methos an
extra critic was much faster so somewhat
surprisingly when we looked at a three C
at data efficiency these are again the
same plants where we have data consumed
on the x-axis
and different numbers of actor learners
in different colors so again this is the
game of break had a 3c basically doesn't
care how many actor learners you use the
data efficiency is the same so this
suggests that one of the reasons why
actor critic is so good is that somehow
there isn't this interference and there
isn't this harmful effect from
correlations between successive updates
because even with a single actor learner
applying online updates its able to
train a neural network in a stable way
so why is that
probably the current best guess is okay
you're learning a policy and a value
function and when you that's that
because we're learning a state value
function there is in this effect of
changing the value of one action when
you update the value of another action
it's only a state value function so it's
just one output and you're also learning
a policy which is normalized so when you
get a high reward for a specific action
you will essentially increase try to
increase its permeability and because
policies are normalized on my Q values
when you increase the probability of one
action you can't increase the
probabilities of all the other actions
because it all has to sum to one but
with the Q function when you try to
change one action value you will change
the values of the other actions so this
normalization of policies makes this
better behaved when in terms of
numerical optimization
well yeah so yeah I guess yeah if we
just jump back I see I guess the way it
works is each actor learner basically
say every K frames work a is like five
or twenty it asks for the latest weights
from the shared memories so it copies
these weights and then it runs these the
policy using these weights for say 20
frames then it computes an update and
applies it to the shared model and then
gets the latest parameters so yeah at
any given time all of these could be
running a slightly different version of
the policy oh yeah it's the same game
yeah any other questions
okay so results and you're not terribly
important here the probably the most
interesting thing that a three C on a
single machine and one day was able to
reach or surpass dqn performance on
Atari and if you train for four days
then you do even better and roughly on
par with what was state of the art at
the time but again the exciting thing
was that on one machine no GPU in one
day you could get good results and now
you can do research faster so that was
exciting also the nice thing
yes why didn't we use it or so I guess
two reasons one was practical so at the
time well and still now Google has more
CPUs than GPUs and the data centers so
GPUs are much more expensive to buy and
to run than CPUs in terms of like can we
run experiments there lots of idle CPUs
in a data center if you can use them you
can run lots of experiments and I guess
the other thing was yeah by then it was
clear okay on Atari these tiny
consonants were working best so we yeah
it's not enough to keep a GPU hungry so
we'll look and yeah in the rest of the
lecture we will look at okay how at some
point we will one bigger networks and
GPUs will be necessary so how can you do
actor critics and that set up um so yeah
another exciting thing was that this
method worked equally well with
recurrent and feed-forward methods and I
think this was really like the first
example of an agent learning from pixels
in like a 3d game to navigate random
mazes so this was exciting so again it
could do Atari it could do 3d games
recurrent feed-forward you could even do
continuous control by say instead of
your policy being a soft max over
discrete actions you could I would put
the mean and variance of a Gaussian
distribution and do continuous control
so it was a nice general method but
right so what are the main takeaways so
again somewhat surprising fact was that
okay actually all the basic RL
algorithms can trail and can train their
own that
works reasonably well as long as your
updates and not are not to on to online
so okay DQ n was using an replay buffer
to shuffle the data and async RL we used
the dislike parallelism and in some
sense wishful thinking that agents are
in different parts of the space but this
was enough so all of them actually work
reasonably well as long as you also use
an adaptive learning rate method by
Karma's prompt so so basically yeah
applying your updates online as the
experience comes in so yeah it can be
shuffled in different ways but it seems
to show that ok with key learning this
is kind of the problem that can lead to
collapse and if you deal with it
everything works and I guess he an actor
critic methods it's even fine to be
online so yeah right so right again CPU
training is fine because it these
networks were small even why I guess you
know standards of a few years ago and
another yeah takeaway was that ok and
step methods were clearly working better
than one step method which is not that
surprising because you're essentially
propagating rewards faster so with one
step key learning you use your estimate
of the value at the next state to update
the value of your current state so you
can think of it as like when you start
learning there are some states with
rewards and every time you apply
q-learning
they sort of diffuse from these states
to neighboring states and an update can
carry a reward only one step forward or
at least is one step update and step
updates basically speed this up
considerably because now a single update
can carry the reward and steps away
from the rewarding state so rewards
propagate much faster um another
somewhat surprising takeaway was once
they're doing these end step methods
target networks I mean they could still
sometimes help but there was really not
as much of a need for them and this is
because when you're bootstrapping from n
steps in the future those two value
estimates are much less correlated it's
simply because the inputs are going to
be that much difference being states
that are n steps away from each other as
opposed to one step so that makes it the
job of the neural network easier and
again with these instant methods
recurrent networks seem to just work I
think partly again because of this like
staid aliasing issue that yeah you
bootstrap both from an input that's n
steps in the future and an iron-on state
that was rolled forward n steps in the
future so these values are much less
correlated okay so so okay are we done
should we just use a 3c and again the
answer is no it depends what you want to
do so it worked well an Atari in 3d
environments it was faster to train so
faster to do research on say Atari and
we could train our Nan's a3c was a
little bit less stable than DQ and so
you could make it work with say the same
hyper parameters and all Atari games but
if you tuned the hyper parameters per
game and did much much better and I
guess it's kind of hard to compare with
dqn because these experiments were never
really done because it's much more
computationally expensive to say run a
hundred copies of DQ on in each game but
yeah it seemed like a three C was
slightly less stable
than tea queuing or slightly less robust
two different hyper parameter values now
it was also not that GPU friendly but
we'll see how we can deal with that and
there's a limit to scalability sorry
three C itself was using threads on a
single machine you can only have so many
threads you could try and distribute it
over different machines but as well see
there are issues with that so again a
three C was nice but once because it
started working on 3d games we were
actually back to the problem of training
time because your Atari could be done in
a day well once we started doing these
like 3d maze navigation problems and
working with robot simulators we were
back to training for like days or up to
a week to get good results so okay we're
kind of back to where we started so what
can we do how can we scale up these
architectures and specifically actor
critic approaches so the next section of
the lecture will be about this recent
paper called empower scalable
distributed RL with importance weighted
actor learning architectures and this is
work from lsas behold Hubert Sawyer and
Remy Munoz and other colleagues from
deep mind
so the question is okay can we scale up
a three-c or similar methods by simply
distributing at over many machines so we
can we could just put every actor
learner in a different machine and put
the weights on a shared parameter server
and now you can have as many machines
well not as many machines as you want
but you can have way more machines than
threads on a single machine so this does
this work yes and no it works in the
sense that this will go faster you will
get more
through the system simply because every
time you had an actor learner it will
increase the amount of data you go
through but you relatively quickly will
start hitting a ceiling of data
efficiency so why does this happen well
you end up running into the problem of
stale gradients so the way a3c works is
each actor learner or each machine or
thread is an actor learner so and gets
the parameters collects the data then
computes a gradient and this gradient is
with respect to the parameters you had
at the time and the parameters that you
ran the policy with and then you send it
back to the parameter server but because
other actor learners are updating those
ways by the time the gradient arrives
it's no longer a proper gradient because
the the way it's changed and this ends
up being quite harmful and what can you
do if the gradient is stale you you can
probably throw it away maybe you can
think of ways of correcting a gradient
but then the parameter server will have
to do much more work so it's not clear
what you you can do so again distributed
experience collection is clearly good
because that sort of that's very scales
embarrassing parallelism it scales
easily but communicating gradient seems
to be bad so what's the better thing to
do well it's better to use a single
centralized learner and to only
distribute the acting so now the
architecture looks similar but now each
of these machines is only an actor so it
gets the parameters from the learner
runs the policy and it returns the
observations to the learner instead of
sending the gradient so what's nice is
the actors can be CPUs which are cheaper
and the learner can be a GPU so it can
both now parallelize over the forward
and backwards gradient computations of
the network and because it has all the
data at once it can use big batches and
can go faster and also hypothetically
train larger networks um and again
because you only collect observations
you don't have the stale gradient
problem but yeah looking at a different
view of the architecture we now again
have these actors so each line here is
an actor and the green blocks show the
time it takes to execute environments
and the blue blocks show the time it
takes to run the forward pass of your
network so the environment time can be
highly variable if say we're rendering
in the 3d engine it can depend on the
number of objects in the scene or maybe
you're generating a new maze so it can
take longer and again in this
architecture the backward pass is
decoupled from the forward path so the
actors just send the experience and the
learner now does big batches of forward
and backwards and doing these updates so
I'm the learner again the gradients are
not stale anymore because you're always
computing them with respect to your
current weights but what you've done is
you've traded stale gradients for stale
experience so now again every actor gets
a copy of the way it's runs the policy
and by the time the experience arrives
on the learner the weights may have
changed so now the experience comes from
a slightly older version of the policy
so this can actually it doesn't seem
like it would be much of a problem
because it the policy will maybe like
several mini batch updates behind what's
on the parameter server
but it can actually harm learning
considerably but the good thing is that
stale experience we kind of know how to
deal with it's just off policy learning
because you know which policy you
executed on the actor you have the
probabilities of the actions so the
learner can hopefully correct for this
flag so the paper introduced this new
actor critic method called V trace which
is basically an extension for doing off
policy learning in an actor critic set
up probably we don't need to you know
look at go through the math basically
what it's doing is it is using important
sampling to correct for the change in
the probability of the actions between
what was executed on the actor which is
mu and what's the current policy on the
learner which is PI and yeah it looks a
bit messy but if the poll if Pyne me are
the same it ends up being exactly the a3
see update and if there is a difference
then basically it becomes biased but it
reduces variance so it's yeah
essentially just truncate an important
sample you know trying to speed up and
but it's okay what is empower it so
importance way to dr. learn in
architecture so it's the set up we
talked about where the learner is doing
V trace advantage actor critic and again
the learner is running on a GPU and the
actors are running on CPUs so yeah in
terms of throughput going distributed
makes things go a lot faster so there
are a lot Alliance here but III see
running on one machine with 64 CPUs and
32 threads on these two 3d environments
is getting six to nine thousand frames
per second distributed a3c was 200
workers gets up to around 50,000 frames
per second so you're going a lot faster
Impala basically with all the
optimizations can get to 200 to 250
thousand frames per second alright so
why because yet the actors are still
running on CPUs but they're only running
the forward pass and all the back
backward pass computation is happening
on the GPU which can parallelize really
well but of course the question isn't
the throughput it's can you actually
learn well so most of the experiments
were done on this deepmind lab 30 task
set which has been open sourced and is
basically a collection of 30 cognitive
tasks in this 3d environment um so the
paper used two different architectures
one is sort of the good old small
confidence LS TM and also used I don't
know a 12 layer resonance so something
that looks a lot more like what you see
in the supervised learning literature
today so here are results training on
five of these 3d levels so probably you
can ignore the light blue curve so the
green yeah the green curve is
distributed a 3c the orange curve a
single machine a 3c and dark blue is
Impala which is also distributed and so
the top line is showing score versus
training data and we see that okay even
though Impala is essentially running a
this
I'm actor critic algorithm with enough
policy correction given the same amount
of data it is often getting much higher
scores so this is suggesting that's the
stale gradient problem which again tends
to be worse for the distributed version
of a3c is harming your learning and
correcting for it does better so ok we
saw that impalas throughput was much
higher but here we're actually seeing
that given the same amount of data the
learning speed is much higher because
there is policy lag but you can correct
for that but in a 3 C there is gradient
lag and there isn't much we can do and
also the bottom line is showing
stability to hyper parameters were each
method was run with 25 hyper parameter
settings and it sort of it shows for
each method you sort the final
performance by decreasing score and
plotted and basically if a line is if
one line dominates another line it means
for any hyper parameter for all hyper
parameter settings you're essentially
doing better and again we see that
Impala it's not only faster when looking
at the best it's actually much more
robust to learning rates and things like
that
now ok the really interesting results I
think are the multitask learning results
in this paper so here we're taking a
single network and training it on all 30
levels at once so this is no different
from say all the Atari stuff we've seen
where it's one network per game so here
it's showing a three C trains a 3 C n
Impala trains on all 30 levels at once
so again this is showing data efficiency
or data versus score we see ok a3 C was
a deep
network gets to about 30% human
performance then we see the green curve
is showing impala the dark green curve
impala with the shallow network trained
on all the games so that does better
than a 3c with the deep network no the
light green curve is Impala trained some
so you trained it separately on all the
games with the deep network and looked
at the combined performance over 30
games so this is essentially cheating
this is one network per level and the
orange curve is showing impala trained
with the deep resonant on all 30 games
at one so what's nice here is that the
orange curve being higher than the light
green curve is showing that there is
positive transferred from training on
all the games at once so given the same
amount of data from all the games you do
better if you train one network on all
of them which is kind of nice and again
I think in terms of data efficiency the
difference how much empower is better
than a 3c here is it's pretty drastic
and if we look at training time versus
corner the difference is even more note
noticeable so this is now training time
in hours and what's the right comparison
here yes so deep impala and orange
essentially reaches the performance of a
3c and less than a day compare it to I
don't know that's something like a week
for a 3c so really I think the taker one
of the takeaways here this is the right
way to distribute these actor critic
methods and not only do you get better
throughput you get more stable learning
yes we can look at a few of the videos
quickly just to see I think the
difference between
yeah we're the first video of dqn was
and what's possible now so this is a
mushroom collecting task requiring
memory and this like I would door
naturalistic environment this will be
hard to see yeah this is hard to see but
these are language tasks so there is an
instruction at the top so like is there
anything runs you know more what most
things blue things like that and the
agent goes left for yes and right for no
and we see basically is able to acquire
basic grounded language skills and then
yeah more of these random A's navigation
tasks but we can skip over them and
again what's one of the exciting things
is that now
yeah the state-of-the-art deeper l
methods are able to do multitask
learning so learning many tasks at once
so to bringing things back a bit to the
beginning the paper also had results for
multitask Atari with both shallow and
deep network so the first surprising
result was that yeah for distributed a3c
these are expert means trains on every
game separately so it turns out actually
distributed a3 see was benefiting from
deeper nuts even on Atari so given not
too much training data shallow expert
gets 54% median and deep gets a hundred
and eighteen but we see that with
empower again training on each game
separately you do much better for both
network sizes and from the deep resonant
you're now getting
close to 200% median score over 57 games
so again with a better learning
algorithm we're now seeing there is a
benefit to these deep and networks even
on Atari and I think most surprisingly
the last line is the multitask result so
it's Impala trains on all 57 games at
once so a single network is getting
about 60% median score which is not too
far off the original dqn trained on all
the networks oh yeah I think this is a
really interesting result that you can
do all on Atari even with one network
and this is much harder than the 3d
domains because there the world is
essentially visually consistent but on
Atari there all the games are really and
diffic white a bit visually and on Atari
there is very little there's almost no
positive transfer because the games are
so different so what are the takeaways
so okay Impala's I think shows that many
badge training is the way to go Indy
pyro and gives both better data
efficiency and better robustness and I
think it explains why dqn is and was
more stable than a3 see we were
wondering okay is that the replay is a
because it's a value method compared to
a policy method and I think the answer
is okay it's many badge training that's
what you need for stability and
robustness um we also saw in the gap in
performance between a3 see an Impala
that being even slightly off policy can
harm you a lot an actor critic methods
and something like V trace mitigates
that up to a point if the experience is
from a totally different policy then the
importance weights will still either go
to zero or there will be clipped at one
and there won't be much learning but if
there is a slight change in the policy
you can correct for it
and I think yeah to bringing back to the
beginning a bit
so now deeper all is starting to look
like the rest of deep learning so many
batch training on GPUs the president's
analyst TMS Adam and rmsprop optimizers
so basically yeah if we think back to
the tweet and Andre should be happy now
because yeah the world makes sense again
these aren't these like totally
different things and I think the lesson
was yeah once the learning methods is
basically right or once you have the
right learning method then these things
aren't that difference and what you
learn and deep learning is much more
easily applied to deep our own um okay
so not much time left so the last little
bit is I think practical advice on
applying deep are all to problems so
it's always good to start with a simple
problem and it's good to I think make a
toy problem which you can solve in like
under minutes on your local machine and
you want to try and make sure that it
has some of the properties of the real
problem you want to solve like this is
of course hard if you want to I don't
know solve robot but manipulation it's
hard to come up with a reasonable time
problem but you can often learn a lot
for debugging purposes so and I think
ideally you want the toy problem to have
knobs for controlling different aspects
of the difficulty like how long the
credit assignment horizon is maybe the
size of the input space like this is
essentially how we started before moving
to Atari and I'll show the toy problem
we used for that so once you're training
it's good to plot and visualize
everything you can so it's good to plot
training curves it it kind of seems
obvious but to I don't know like I often
have the tendency to just print stuff
and you see the urchin
but plodding trends are much more
apparent when you applaud stuff compared
to just looking at numbers it's kind of
obvious but often easy to forget and
then again visualize everything look at
what the policy is doing look at what
the value function is predicting really
the the more things you can plot the
more you'll learn that it's always good
to keep that in mind so the specific
game we used yeah the video is from an
old paper but before starting Atari we
were we came up with this game of catch
which involve it kind of looks like
breakout so you have a pixel falling
from the top and the agent controls a
paddle and you get a reward of one if
you catch the pixel and a negative one
if you if you don't and now you can
start with something really easy words
say like five by ten pixels and the
paddle I don't know is two or three
pixels wide and you can vary
maybe the pixel can fall just
horizontally which is super easy and
should essentially be like a unit test
for your learning algorithm and then you
can start introducing angles and bounces
and making it bigger and changing the
size of the paddle and we basically got
an early version of dqn working and a
reasonably large version of this before
moving on to something like poem and the
jump wasn't wasn't that drastic and it
really helped sort of seeing something
that works in the small version then you
make it bigger and the algorithm breaks
and you quickly find you can find out
why and it this takes seconds to train
on often so maybe it's slightly
controversial points when it comes to
reinforcement learning especially in the
sutton and bar
book rewards and discounts are seen as
part of the problem definition and they
are but in practice when you're trying
to solve a problem you actually get to
tweak these things so for example the if
we're doing if we want to train an agent
to play Atari we measure the final score
but in practice most of the agents clip
their rewards at one so it's really
optimizing like expected reward
frequency or something like that so or
expected reward counts not their
magnitudes and this ends up being easier
to learn of course you want to be
careful if you change the reward
function too much or if you miss
specified in some way it could backfire
so I think one example I like is if
you're training or if you want to train
a robot to pick something up and put it
somewhere else a reasonable reward
scheme might be okay first the robot
needs to learn to reach the object then
it needs to learn to move the object and
then it needs to let it go so if you
specified the reward like that it will
learn to reach it might learn to pick it
up and move it but then it will
basically be in a local minimum or local
long optimum because it's learn to hold
on to the object for dear life because
that's sort of the most stable policy
while the reward is moving the object
you really don't want to let it go so it
will learn to it might learn a very
tight grip and then basically you will
not be exploring anymore and it becomes
really hard to learn to let go so this
is an example where okay the real reward
you care about is object ends up in the
place you want but sort of over
specifying it might mislead the agent
so I guess here you're sort of breaking
it down is like first you reach it then
you move it and then the last bit is
like letting go and yeah that's sort of
not it's it's easier to start learning
from but it ends up being hard to reach
the final policy so similarly discounts
discounts like if we care about the
final score then the disc discounting is
wrong but the scans reduce the variance
in your return estimates and actually
looking at the discount as a hyper
parameter is not an unreasonable thing
to do again because all our learning
methods have inductive biases and maybe
using a certain discount value makes
essentially your loss function have the
right shape for your optimization method
and ok with neural networks again
starting with small networks is good but
sometimes it can backfire so actually
first we were doing DQ n with really
small networks and it was really hard to
get them to train on say like a game
like C quest and this ended up being the
motivation for needing the target
network and later on when that came to
doing the experiments for the final
paper we went to show ok you need a
target network so by then we were using
much bigger networks and when we did the
comparison target networks were much
less important because actually a big
neural network does not in the alias
States as much as a small neural network
so there's less of a need for a target
network so in some sense
yeah the small neural network was faster
but it was making the problem harder so
it's always good to start small and then
periodically try bigger nuts to max out
performance
also yet be careful with the
initialization papers often don't talk
about this but when you initialize
policy you want to make sure that it
will not be deterministic and will
sample actions with some probability and
the RMS prop and Adam are crucial and
it's still good to be careful about the
latest tricks from supervised learning
like dropout and batch norm especially
bench norm is still not really used in
RL also yeah there is an excellent guide
from John Shulman that you can find on
his website was ya more detailed advice
so yeah thanks for your attention
[Applause]