DeepMind x UCL RL Lecture Series - Deep Reinforcement Learning #1 [12/13]

today i'll be discussing an exciting era
of research that's been instrumental to
many recent successes in reinforcement
learning and this is uh typically
referred to as deep reinforcement
learning so this is the combination of
reinforcement learning algorithms such
as the ones that we have discussed so
far with the use of deep neural networks
as function approximators
so the motivation for function
approximation and the core ideas behind
it have already been introduced so you
will remember how we discussed that
taboral rl cannot possibly scale to
certain large complex problems the
reason being that of course if we want
to estimate
say the value of each state separately
for every single one state this will
have a memory cost that naively scales
linearly with the state space which by
itself would make it impractical
impractical but even if we had unlimited
memory
the there was still a fundamental
problem that it would be just very slow
to learn the values of all states
separately so if we need to visit every
single one and potentially multiple
times to make an even reasonable guess
of the value of the state we are
we're definitely in trouble so
a response as
again discussed in previous lecture is
the use of function approximation which
is our key tool to generalize what we
learn about one state to all other
states that are close according to a
reasonable definition of close
we have already introduced a function
approximation but the purpose of this
chapter will be to discuss the use of
deep neural networks specifically for
the purpose of function approximation
and this is what is typically referred
as deeper enforcement learning
now i will delve soon in some of the
practical challenges that arise in this
setting and some exciting research in
the area but before we go there i want
to
take this introduction to
to basically recap a few core ideas of
function approximation in general and
also to discuss the role of automatic
differentiation in easily supporting any
one of you to freely experiment with
deeper enforcement learning ideas
so when
when using function approximation to
estimate values in previous lectures we
we typically proposed a simple scheme
where we would have some fixed mapping
that
transform any one state in some feature
representation v
and then
we would have a parametric function that
is linear and that maps features to
values
the problem of reinforcement learning
then becomes to fit these parameters
theta so that's for instance this value
function v theta is the makes
predictions that are as close as
possible to the true values v pi for
whatever policy pie we for instance wish
to to evaluate
and we should have and we can turn this
into concrete algorithms so
the first step would be to to formalize
the goal of minimizing the difference
between b theta and v pi so for instance
we we could use as some loss function
like the unexpected squared error over
the states
typically this would also be weighted by
the visitation distribution under under
the policy pi itself in order to
allocate capacity sensibly
and then given a loss function we could
use radiant ascent to optimize the loss
function
this sounds very simple and easy but
well of course the devil is in the
details and actually
do
doing this process for reinforcement
learning introduces quite a few subtle
challenges so first of all of course
computing the expectation over all
states is too expensive
but this is maybe the the least of the
problem the most deep problem is that
the very target v pi that we want to
learn
to predict accurately is actually
unknown
so the solution to both problems is to
sample the gradient descent update by
considering just one or a few states in
each update and then use the sampled
estimates of the v of v pi as targets
and to do so we can
reuse all the ideas that we have
discussed in for instance model three
algorithms so we can do monte carlo deep
prediction by using an episodic return
as target in the gradient update
or we could implement a deep td
prediction algorithm by bootstrapping on
our own value estimates to construct the
the one step target
and
again
pyramid with the bootstrap itself being
parameterized by the very same
parameters theta that we wish to update
so
this is is all good in principle and we
we already saw that in if some mapping
phi is given
then it's possible to train a linear
function to make reasonable value
predictions but today we want to
consider a more complicated setting so a
setting where maybe the feature mapping
is too naive and it's not
informative enough to support reasonable
value predictions if we have just a
linear mapping on top of it
and in this case we might want to use a
more complicated nonlinear
transformation from states to values
and for instance by using a deep neural
network as a function approximator
you may wonder why choosing a neural
network and this is definitely a good
question so
it's valuable to stress that they are
these are by no means the only possible
choice for a more complex function
approximator but neural networks do have
some advantages so
the first is maybe that this class of
parametric function is well known and is
known to be able to discover quite
effective feature representations that
are tailored to any specific tasks that
you might apply
this parametric function to so in our
case reinforcement learning but the
neural network have been used also for
processing language or vision
importantly this this future
representation that is learned by a
neural network is
optimized in an end-to-end manner by the
same gradient process that you that in
linear function approximation is
reserved just to define a linear mapping
so we have a unified way of training the
entire numper the entire parametric
model
to
expect to represent the good the the
states in a way that is expressive and
then make a reasonable good value
predictions from this representation
the second reason for to consider deep
neural networks is that given the
extensive adoption of deep learning
methods in machine learning using neural
networks allows us to leverage just
leverage lots of great research so all
the ideas that have been introduced in
supervised learning for instance for
network architectures or optimization we
can leverage all this grace research and
benefit from it when we use neural
networks for functional approximation in
reinforcement learning
what does parameterizing a value
function with a neural network actually
looks in practice though in in the
simplest case maybe we could consider
what is called the multi-layer
perception so this would be a model that
takes a very basic encoding of a state
so for instance in a robot this might be
the raw sensor readings that we get
and the mlp would then
take these as inputs compute a hidden
representation by applying a linear
mapping so w s plus some bias b
followed by a nonlinear transformation
such as a 10h or a rayleigh
and then the actual value estimate would
be computed as a linear function of this
embedding but the important thing is
that this embedding would not be fixed
would be learned and the parameters
detail are value estimates that then we
we train
using d parallel would include not just
the final linear mapping but also the
parameters of the hidden representation
so the entire system would be trained
end-to-end
this sounds appealing of course but what
if it to compute the gradient with
respect to tita we need to differentiate
through say not an mlp but maybe a
confidence
or what if we have a batch normal layer
or we use a transformer
well it turns out that there are there
is actually a way to compute exact
derivatives or any of these
architectures in a computationally
efficient way
that basically allows us to get any
gradient we need without having to
derive its expression ourselves and
given the popularity of deep learning in
modern machine learning these these
methods that are typically referred to
as autodiff are actually available in
most scientific computing packages
so in this chapter we'll be focusing
mostly on the conceptual challenges of
combining rl and deep learning and we'll
mostly just assume that we have these
kind of tools to support us in in
getting the gradients through any
arbitrary neural network architecture
that we we may want to use for function
approximation but i want to give at
least
a brief intuition about how these tools
work since they are so fundamental to
everything that we do into the practice
of deeper
and it will specifically i will give you
a very brief introduction and also show
you how you can use them to implement a
very very simple dprl agent specifically
a q learning agent with a neural network
as functional proximity
the first important concept behind
automatic differentiation is the concept
of the computational graph
so this is the abstract representation
of any computation that you might want
to perform in our case estimating a
value say in the form of a direct
science cyclic graph
for instance in this slide i show a very
simple instance of such a computational
graph where i have two inputs a and b
and i compute some intermediates
representation as a plus b and b plus
one respectively this is of course a
very simple example and then i compute
some output by taking the product of
these two numbers
the reason computational graph are
interesting to us is that if we know how
to compute gradients for individual
nodes in a computational graph
we can automatically compute gradients
for every node in the graph with respect
to any other node in the graph and do so
by just running the computation graph
once forward from the inputs to all
outputs and then performing a single
backward sweep through the graph
accumulating gradients along the path
and summing gradients when paths merge
so for instance in this simple example
the gradient of the output e with
respect to input a can be computed by
just taking the product of the
derivative of e with respect to input c
times the derivative of c with respect
to input a
which is trivial compute because like
the individual nodes can you can always
decompose your computation so that the
individual nodes are really just simple
arithmetic operations
if you take again in the simple example
maybe a slightly more complicated
derivative where we wanted the gradient
of e with respect to
the input b
then this will be just the sum of the
derivative of e with respect to c
times the derivative of c with respect
to b so this is one path from e to b
and then
following the other path we just need to
take the gradient of e with respect to
2d and then times the derivative of d
with respect to b and sum these two
paths since the two paths merge
and this might even seem slightly
magical at first but if you actually
write it down it's just implementing the
standard chain rule that you you all
have seen when in a calculus course for
instance
the advantage though of doing this in
the form of a computational graph is
that it helps implementing the the chain
rule amazing implement the the gradient
computation in an efficient way for any
arbitrary numerical function by just
decomposing your computation which in
our case will typically just be a
program they compose it into the
sequence of basic operations additions
multiplications and so on and then
considering each of these basic
operation a node in the graph that then
we can differentiate by implementing the
the graph algorithm i just described
and i wanted to stress here that the
entire process is not just
computationally efficient so it's always
order of the cost of doing a forward
bypass but it's also exact so this is
not a numerical approximation like
you could do for instance by computing
the gradients by infinite differences
this is an actual way of evaluating the
true gradient of an arbitrary numerical
function in a fully automatic way
it's really really exciting if you think
about it
autodiv is natively implemented in many
modern machine learning frameworks
including jax which is the framework of
choice for discourses assignment
some of these frameworks like sensor
flow require you to explicitly define
your computation in terms of a
computational graph so that they can
then do things like how to diff very
naturally on top of this graph
others though like jacks allow you to
still write your computation in a
standard imperative way but then
implicitly recover the graph by tracing
your computation
injects autodiff is
implemented based on this tracing
mechanisms and it's exposed to the users
via the jacks.grad program
transformation so this is
a utility object that takes a python
function that you can just write in
numbers in a very standard format and
then it returns you another python
function but that now computes the
gradient of the original function and
evaluate that for any given input
instead of computing the just the
forward pass
to conclude the introduction i want to
show a very simple example of
implementing a basic
q learning agent that uses a neural
network as a function approximator using
an autodiv tool from jax
so what does such a deep q learning
agent look like
first we will need to choose how to
approximate q values and for this we
will use a single neural network let's
say that takes the status input and
outputs a vector output with one element
for each available action
for instance this network could be an
mlp as before
and
note that this is we have implicitly
made the design choice here of the
network taking a single state and
outputting all action values but this is
not a strict requirement we could also
pass state and action both as inputs to
the network and then the network would
return you just the q value for
detection but in general it tends to be
more computational efficient if we can
compute all q values in a single path so
this is a fairly common choice in
practice
in jax we could define such a neural
network very trivially using the haiku
module
in with this library we just need to
define the forward pass of the network
as for instance in the case of an mlp a
sequence of a linear a relu and another
linear
and then we can use the haiku.transform
to extract python functions two python
functions specifically one network init
that initializes the parameters of the
network and the second
network applied that basically computes
the forward pass taking the parameters
and the state as input
once we have a network though of course
our job is not done if we want to
implement dpq learning we will also need
to define a gradient update to the
network parameters
and with qlearning this looks very much
like the tvd update that i showed at the
beginning but we will be updating one
specific action values so the q theta
for specific sd and at
and we will be using as a target as a
sample estimate of a return the reward
the immediate reward plus the discounted
max q q value
where again we're using our own
estimates that depends on parameters
thetas
as to bootstrap
it's interesting to point out though
that while you can write your update
exactly in this form that matches very
naturally the the math after all
often if you for instance look for an
implementation of a dp learning agent
you might see it written in a slightly
different way
for consistency basically with standard
deep learning you might see that instead
of defining directly a gradient update
they might define a pseudo-loss function
which is the second equation on this
slide
that is in this case just the one-half
times the mean squared error between the
chosen action and the target which is
given as usual
by the rewards plus the discounted max q
value
and this is fine but for this gradient
of this loss function to actually
recover the correct q learning updates
there are a couple of important habits
so first of all
when we compute the gradient of this
loss we need to ignore the dependency of
max q on the parameters theta
this is denoted by the fact that we have
a double vertical bar around max q if we
did not ignore this then the gradient of
this loss would not recover the first
equation we would have an additional
term
and the second is good to realize that
this is not a true loss function it's
just a prop that has the property of
returning you the right update when
taking a gradient
how does this translate into code well
this is again fairly simple in jax
because we can just define the
pseudo-loss function that takes the
parameters theta and the transition so
an observation an action a reward a
discount and a subsequent observation
and then first computes the q values in
both sd minus 1 and sd
assembles the target by summing the
reward plus the discounted max q
and then computes the squared error by
just one half
square of these two what is critical is
that to ensure that the gradient of the
two the loss implements our dp learning
update we must
take care
of
adding this top gradient term in line
25.
this top gradient implements this this
idea that we we will the gradient
computation will ignore the dependency
of target t minus one on the parameters
data
to actually get the update then we we
need to actually take the gradient so
this is done in line 30 where we compute
jax dot grad of this loss function this
gives us back another python function
again and then we evaluate in theta of
1 80 minus 1 rt dt and obviously
finally just updating the parameters we
can do it by just stochastic gradient
descent and in this case will be
equivalent to just taking the parameters
theta the gradients and subtracting a
small step size times the gradients
this is of course a fairly simple agent
but it range shows actually a full
pipeline for defining a deeper religion
because we define a network and the the
gradient updates and we have applied
these updates
and the reason it looks so simple and
clean is that we have exploited this
beautiful jack save to lift capabilities
that allow us to get an update from a
loss function or a pseudo loss function
in this case by just complete calling
jacks.grad on the relevant numerical
function
so the next section will actually now
delve into what are the challenges of
dprl how we can make the reinforcement
learning part aware of our specific
choice of function approximation and
also vice versa how we can make our deep
networks more suitable for rl by
understanding the unique properties of
updating the parameters with our
enforcement learning objective
in this section i want to give you some
insight in what happens when ideas from
enforcement learning are combined with
deep learning both in terms of how known
rl issues manifest when using deep
learning for function approximation and
also in terms of how we can control
these issues by keeping in mind the
effect on functional approximation of
our enforcement learning choices
let's start with the simple online dpq
learning algorithm from the previous
section
what are potential issues with this
algorithm
well to start we know from deep learning
literature for instance that stochastic
gradient descent assumes gradients are
sampled iid
and this is definitely not the case if
we update parameters using gradients
that are computed on consecutive
transitions in mdp
because what you observe on one step is
strongly correlated with what you
observed and the decisions that you made
in the previous steps
on the other side we also know that for
instance in deep learning typically mini
batches instead of using symbols single
samples is typically better in terms of
finding a good biased variance tradeoff
and again this doesn't quite seem to fit
the
the online deep q learning algorithm
that i described in the previous section
because there we perform an update on
every new up step so every time we
are in a state and execute an action get
a reward a discount and another
observation we we would compute in
updates to our parameters data
so how can we make rl more deep learning
friendly
if we look back to previous lectures
it's quite clear that certain algorithms
may better reflect deep learning's
assumptions than others so it's good to
keep deep learning in mind when choosing
what to do on the rl side so for
instance during the planning lecture we
discussed dyna queue and experience
replay
where we mixed online updates with also
updates computed on data that was
sampled from
a learned model of the environment in
the in the case of dyna queue or from a
buffer of past experience when we're
doing experience replay
and
both actually might address very
directly the two concerns that we just
highlighted in the vanilla dpq learning
agent that i described before because by
sampling
state action pairs or entire transitions
from from a memory buffer we we
effectively reduce correlations between
consecutive updates to the parameters
and also
we get for free basically support for
mini batches instead of you know doing a
loop where we apply n planning updates
we could just batch them together and
and do a mini batch gradient ascent
instead of vanilla sgd
similarly if we know we are using deep
learning for function approximation
there are many things we can do to help
learning to be stable and effective so
we could use for instance alternative
online rl algorithms such as eligibility
traces that naturally integrate
in each
update to the parameters data
informations that comes from multiple
steps and multiple interactions with the
environment without requiring explicit
planning or explicit replay like in dyna
queue or or experience replay
alternatively we could also for instance
think about certain optimizers from the
deep learning literature that might
actually address
and
alleviate the issues that come from
online deeper enforcement learning by
for instance also integrating
information across multiple updates for
in a different way for instance by using
momentum or the adam update
and finally in some cases we if we keep
in mind
what the the properties of deep learning
we might even be able or willing to
change the problem itself to make it
more amenable so for instance we could
think of having multiple copies of the
agent that interact with parallel
environments and then mix this diverse
data that comes from multiple instances
of the environment in each single update
to the parameters
that's that's how deep delve even deeper
so
we said that if we use dyna queue or
experience replay we can address certain
issues that and better fit certain
assumptions about of deep learning
at the same time
if you think about dying q and dq n and
what happens when we use these
algorithms in combination of functional
approximations specifically then we are
actually combining function
approximation of course but also
bootstrapping because we are using for
instance q learning to as model-free
algorithm and of policy learning because
by replaying past data we are
effectively
mixing the data is sampled from a
mixture of past policies rather than
just from the latest one
and you might remember that we denoted
the combination of exactly these three
things the deadly triad which
doesn't sound good right and
and the reason is that we we know from
from uh from the theory that when you
combine these three things there is a
possibility of divergence
at the same time if you read the deepara
lecture literature you will find that
many many successful agents do combine
these three ingredients so it's
there is maybe a tension here how is
this possible
well
a partial resolution is that the daily
triad says that divergence is possible
when combining these but not that is
certain and not even that it's likely
so if we understand and keep in mind the
properties that underlie both rl and
deep learning algorithms
there's actually much we can do to
ensure the stability and reliability of
the parallel agents even when they are
combining all the elements of the deadly
triad
and in the following slides i want to
exactly help you
develop
an understanding and an intuition about
how and when the deadly shrine manifests
when combining reinforcement learning
with the use of neural networks for
function approximation
and as done with the with the for
instance the two initial issues we
discussed so batching and correlation
this by understanding and keeping in
mind the properties underlying rl and
deep learning would this will already go
a long way towards being able to design
the parallel algorithms that are quite
robust
so
let's start with the larger empirical
study that i performed with hado where
we looked at the emergence of divergence
due to the deadly triad across a large
number of variants of dpq learning
agents and across many domains
what we found is that
empirically unbounded divergence is
actually very very rare so parameters
don't tend to actually go to infinity
even if you're combining all the
elements of the deadly triad
what is more common is a different
phenomenon that we we call soft
divergence and this is shown on this
slide where we show the distribution of
value estimates across many many agents
and many many environments and for
different networks on the left and the
right respectively
and what you see is that the values
explode initially so they grow to orders
of magnitudes
larger than it's reasonable to expect in
any of the environments that we're
considering because the the max crew
values were at most 100
well here we're seeing values even in
the order of tens of thousands or
hundreds of thousands or millions
but what we see is that these values
that are initially diverging don't go to
infinity so over time the estimates
actually recover
and go back down to reasonable values
you may wonder well
if soft divergence mostly resolves
itself when doing the parallel in
practice either is it even a concern
should we even be discussing how to
minimize this initial instability
and i think the answer is that
yes i think it's worth discussing these
things because
even if it doesn't diverge fully to
infinity having you know many
hundreds of thousands or millions of
interaction with the environment where
values are wildly incorrect does affect
the behavior of the agent and the speed
of convergence to the correct solution
that let's then discuss what we can do
about it how do different reinforcement
learning ideas
help us ensure that the learning
dynamics when doing
using deep networks for functional
approximation is stable and effective
the first approach i want to tell you
about
is was introduced by the dqni algorithm
and is known as target networks
the idea here is to hold fixed the
parameters that are used to compute the
bootstrap targets so in queue learning
for instance this would be the
parameters that are used to estimate the
max queue on the next step
and then only update these parameters
periodically maybe every few hundreds or
even thousands of updates
by doing so we interfere with the
feedback loop that is at the heart of
the deadly triad if you remember the
core issue that i tried is that when you
update a state you also inadvertently
possibly
update the next state that you the one
that you are going to bootstrap on and
this can create certain feedback loops
but if the parameters that you use to
bootstrap are frozen at least for a
while then this feedback loop is broken
this is also the only approach so
for instance we know that q learning by
itself even in a tabular setting has an
overestimation device and this can
result in unreasonably high value
estimates at least in an initial phase
so it's possible that actually this
could interact with the deadly triad and
increment the likelihood of
observing these explosions in the value
estimates
if this was the case then we could use
for instance double q learning to reduce
the overestimation of the update and
this would also help us make the
algorithms more stable with respect to
the deadly triad
in double q learning you may remember we
use separate networks to choose which
option to bootstrap on and the
evaluation of such action
and neatly this actually combines really
well with the target networks idea
because then we can use the frozen
parameters the ones of the target
network as one of the two networks
and in this plot
you can see and compare the effects of
both using target networks using only
double q which is what is dubbed here
inverse double q and doing both so
double q learning typically uses
both the
the the separation of the evaluation
from the
selection of the bootstrap action but
also use the starting network for
for the target estimation
and what we see is that actually both
these idea have a strong stabilizing
effect so just using target network for
instance dropped the likelihood of
observing soft divergence across the
many agent instances and environments
from 61 percent of the
of the of the experiments to just 14
and if we can consider the full double
queue that combines both the idea of the
coupling action selection and evaluation
and
the the target network idea then the
likelihood of observing
soft divergence actually dropped to only
10
in general
i personally find it quite insightful to
see these things and see how different
choices on the reinforcement learning
side interact with the deadly triad that
is triggered by the combination of
bootstrapping and of policy with
function approximation
and it's quite interesting to see that
this is not just about for instance
target network or double key learning
throughout our design choices that we
make in our agents each design choices
can interact with function approximation
and the learning dynamics of our deep
networks
so consider for instance a different
aspect of our agent
if you're using dyna or experience
replay we will be sampling either state
action pairs or entire transitions
and one choice there is to just sample
uniformly this is the
most simple but also the safest choice
as we'll see but it's also somewhat
unsatisfying because
not all transitions that you have seen
in the past are equally useful or
important some state we might be able to
make already predictions very accurate
very well
while in others there might be a lot
that we can still learn because maybe
our states that we
weren't we haven't seen many times and
prioritize replay was introduced to
allow us to do to make more efficient
efficient use of resources by sampling
more often these more interesting
transitions
and there are many ways you can
prioritize so i don't want to
go in too much details here but for
instance you could
sample more often transitions with the
higher to the error
but regardless of the details
what i want to think what i would want
you to think about for the purpose of
this lecture is how does the choice of
prioritization
affect the emergence of the deadly tried
in terms for instance of the likelihood
of observing soft divergence
and if you think about it carefully by
prioritizing you are increasing the
amount of policies
and indeed what we see in our empirical
study is that
we find the stronger the prioritization
the more likely is that we observe
self-divergence
this by itself doesn't not mean at all
that prioritization is a bad idea
just that we need always to strike a
balance when we're using prioritization
between the benefits of seeing more
often the most useful transition and the
potential risk of reduced value accuracy
at least in the short term due to the
emergence of this deadly triad
by being aware of this subtle
interaction we can also modify the rl
algorithms to reduce the adverse effects
on the learning dynamics of our function
approximator
so for instance in the orange line in
this plot what we show is the likelihood
of soft divergence if we at least
partially correct the off-policiness via
important sampling
and as you can see we can actually push
prioritization much further along the
x-axis while still keeping the risk of
soft divergence reasonably low
but there are other things that if you
are thinking about the deadly triad
could be important to think about to
ensure that our d parallel agents are
stable
so if you think about the deadly triad
then
at its core it's an issue of
inappropriate generalization we already
said this many times the reason we
observed the deadly triad is that the
value of a state and the value of the
subsequent state that we bootstrap on
are tied via
generalization but we don't need to
immediately bootstrap when we are
computing a d-parallel update so for
instance instead of accumulating one
reward and then bootstrapping on the
maxq value on the next step which is
very close to the origin to the source
state and therefore might be more
affected by generalization we could
instead accumulate many rewards and only
then bootstrap on the maxq value
potentially we could even do this and
combine it for instance we're using a
double estimator for extra stability
and in d-parallel setting this would
result in a multi-step double queue
update which is shown in this slide
and
this will go a long way to help us
because it's still a well-defined target
there's still a policy improvement but
is much less susceptible to the triad
because we are reducing the reliance on
bootstrapping and we are reducing the
amount of generalization in appropriate
generalization we get on the bootstrap
target
for instance in this plot we we're
showing again in our usual empirical
study across many agents and many
environments what is the likelihood of
soft divergence
if we increase the number of steps
before bootstrapping from one to three
and to ten and what you can see is that
the likelihood of soft divergence drops
radically from more than eighty percent
to just nine percent
it might seem maybe a bit scary at first
the way in which basically every divine
design choices in our url updates subtly
interacts with the learning dynamics of
our deep function approximator and while
i in this lecture i focus on the deadly
triad this is by by no means the only
case
and but hopefully you can also see this
in a positive way because
this is what makes dpr-l quite an
exciting arab research because it really
requires you to think holistically about
the whole process of learning and
decision making and how all the
components fit together
but if you have a good understanding of
the core ideas behind both rl and deep
learning and i hope you will have by the
end of this course
you can do this so you can reason about
how these components fit together and
you have a chance to build really
powerful learning systems that can
really push the limits of what is
currently possible with ai
in this section i want to take a rather
different perspective from the previous
section
instead of focusing on being aware of
our function approximation choices when
designing our rl algorithms i want to
discuss about the deep learning side so
can we design our neural network
architectures to be especially effective
for enforcement learning can we
understand what it means to optimize a
neural network with targets constructed
via bootstrapping what does what does it
mean
to increase for instance capacity in a
network if we're training a
reinforcement learning problem
this of course is a huge area of
research so
i won't be able to cover all ideas in
the space of this lecture but hopefully
i can give you at least some intuitions
and some ideas
if you think about the recent history of
deep learning much of its success comes
from being able to encode certain
inductive biases
in the network architecture so that they
can best support learning in certain
broad categories of problems without
limiting their ability to build
tailored representations specifics to
each task purely from end to end
gradient descent
so for instance convolutional nets
source much of their power from the
their capability of learning
translational environments detectors
that make learning computer vision tasks
much easier or similarly
lstms support long-term memory by a
specific architecture that allows
gradients to learn to use gating to
preserve information over large horizons
and this was instrumental to many early
successes in natural language processing
since rl is radically different from
both vision and nlp and the most common
supervised learning tasks where deep
learning is applied to i think it would
be surprising if just copying network
architectures designed for supervised
learning would actually give us the
optimal results
instead i think we should
consider what are the right
architectures for reinforcement learning
which inductive biases should we
incorporate to make for instance value
estimation as easy as possible
and this was the motivation of the
dueling networks paper that was
introduced a few years ago this
introduced a network architecture that
could improve the performance of deep q
learning agents quite quite
significantly and basically out of the
box without requiring any change on the
reinforcement learning side
as with the convolutional nets and sdms
the idea is actually remarkably simple
so
normally dpq learning agents use a
network architecture like the one on the
top in this slide
they take an observation as input for
instance the pixels on the screen if the
agent is learning to master a video game
process this input via number of hidden
layers so typically comp layers if the
observation is in visual form and then
apply some fully connected layers to
output all action values as a vector
with many elements as many elements as
actions
in general though
we know that we can decompose action
values as the sum of a state value
estimates that looks at long term with n
is independent of the action and only
depending on the state
plus an immediate advantage term that
depends also on which action we are
gonna take right now
so this actually suggests somewhat
naively a different architecture where
as before we take an observation as
input process it by some con layers if
it's an image but then
the network generates the q values by
summing a scalar and a vector output
with their own separate stream of
processing therefore forcing the network
to represent action values as the sum of
the action dependent and
action-independent term
if you use something that looks like
this you can train the resulting q
values using standard deep queue
learning you don't need to change
basically anything else
but will will this help
well let's consider a concrete problem
where we have trained this ulnq network
to estimate q values for an agent that
must control a car running on a highway
there are some turns there are some
other cars that you must avoid crashing
into
and in the images in these slides you
can see
what this color and vector component of
the dueling network are mostly attending
to so the plots are generated by
highlighting in red the pixels that most
affect the two outputs so this is what's
called a saliency map with on the left
what this color term is mostly attending
to and on the right what the vector
action dependent term is attending to
what you can see is the scalar term on
the left learns to attend mostly to the
direction of the road far ahead of the
current location of the car as this is
what matters for the long term value of
the current state
instead the vector term on the right
learns to attend the mostly of what is
straight in front of the car
as this is what is important to estimate
the immediate advantage of each action
so this i think would be very
interesting by itself but it it also
results in quite significantly improved
learning with agents learning both
quicker and more stably
if we're thinking about our allyware
deep learning though the topology of the
network is not the only important factor
so in supervised learning for instance
we often find that more data plus more
capacity not just better architectures
equals better performance
for instance because the loss might be
easier to optimize
but how does network capacity affect for
instance value function estimation in
reinforcement learning
in my experience
typically larger networks do tend to
perform better in reinforcement learning
as well but it's not all roses so
if
we use larger networks we will for
instance be more susceptible to the
triad especially if using vanilla
learning
in the large empirical study that i
discussed already extensively in the
previous section
we also looked for instance at different
network capacities and what we found is
that with q learning for instance the
likelihood of soft divergence where at
least initially the values might grow to
unreasonable estimate was actually
increased as we would make the network
larger and larger
and there are many reasons but for this
uh one
potential uh leading candidate is that
uh larger networks tend to have a
smoother landscape and at least
initially this means that they might
suffer more from inappropriate
generalization
with of course we this is uh as as
always in reinforcement learning and in
deeper enforcement learning these are
all trade-offs so it doesn't mean that
we should not use larger neural networks
because those do tend to result in
better final performance at convergence
but we should be aware that if we as we
increase the capacity of our network for
instance this might result in some
instability early on if we don't take
care to use double queue estimators
multi-step targets or generally the
function approximator friendly updates
that we discussed in the previous
section
another issue that is worth being aware
that also relates to smoothness is that
many deep learning architectures have a
bit of a
a problem at approximating sharp
discontinuities
and this is not so much of an issue in
supervised learning but it can be quite
tricky when we're doing url and i'll try
to give you a bit of insight why this
smoothness property might be problematic
so in the plot in this slide
we have a grid world where an agent
might move
from any cell to any of the neighbor
cells
but the grid world is split in two parts
separated by a wall
so if an agent starts on top it will not
be able to reach the bottom portion of
the grid world and vice versa
critically though a rewarding location
is present only in the bottom half but
not the top half of the grid meaning
that the values in the upper half are
exactly zero while there are there are
non-zero immediately beyond uh below the
wall
so if you attempt to learn state values
for this problem for instance under an
assumed fixed random policy
we could use monte carlo updates on the
parameters of the network to do so
if you do it what you will see is
basically the error heat map on the left
we the predictions are mostly accurate
everywhere with some noise in the bottom
half due to the fact that monte carlo
updates are high virus
and some error in the hopper health in a
very tight band just above the wall this
is due to the fact that a network
with the canonical architecture such as
those that we use in supervised learning
has trouble to approximate the sharp
discontinuity in values that go from
non-zero to exactly zero as soon as you
cross the wall
overall this though
might seem reasonable sure you know
smoothness might
have introduced some errors but maybe
that's that's expected it's not too
concerning but now consider what happens
in the error map on the right this
corresponds to training the same value
network by using td
instead of monte carlo and the result is
quite fascinating i would say because in
the lower half the values are still very
accurate actually even more because the
td updates are lower variance so you
don't get that noise that you are seeing
in the bottom half with monte carlo
prediction but on the upper half there
is an hour not confined anymore to a low
small band close to the wall and instead
the difficulty in fitting the
discontinuity just above the wall
actually leaks to the valley predictions
in the entire upper health
so this this maybe might be surprising
at first but but the reason is that
states throughout the upper half will
bootstrap on the incorrect states just
above the wall and this means that the
error will propagate and this is the
phenomenon that's been called leakage
propagation
and as an example another one now of how
certain smoothness property of a
function approximator that might be
acceptable in supervised learning can be
quite problematic in a real so this is
maybe a further motivation to double
down on reinforcement learning aware
deep architecture is to make sure that
we can support the kind of problem and
function classes that we care about in
our rl
and there is much more
still that could be said than certain
architectural choices that could
alleviate these issues but uh for now i
really want to
conclude highlighting what's behind many
of the things i discussed in this
section and this is again like in
we we saw in the past inappropriate
generalization so smoothness leads to
inappropriate generalization and this
leads to leakage propagation and this uh
leads to to to the deadly triad for
instance
but um so while are aware deep learning
architecture will certainly help and i
think this is a very important although
some comparatively underexplored topic
in dprl research the issue of learning
good representations for rl will likely
not be entirely solved by architecture
alone and to help
we will also need to change i think the
way we train the representations
in the network because by having better
representation
we will then be able to have less
inappropriate generalization
and one leading idea in this space that
i just want to mention now but will be
the topic of
of the several lectures in the remaining
part of the course is that maybe we
should consider learning state
representation that are not just for the
singular purpose of predicting a single
value function for a single reward and
instead we should maybe share
representations across many tasks for
instance predict the value for different
policies or for the same policy but
under different discounts or for the
same policy under the same discount but
for a different cumulative other than
the main task reward
and if we can find good ways of doing
this this might alleviate many of the
issues that we discussed throughout this
this deep d parallel chapter both in
terms of deadly triad leisure
propagation and so on so this is a very
i think interesting area of research in
deep rl that we are going to touch on
quite extensively


GPT 4: Full Breakdown (14 Details You May Have Missed)

the moment I got the alert on my phone
that gpt4 had been released I knew I had
to immediately log on and read the full
gpt4 technical report and that's what I
did of course I read the promotional
material 2 but the really interesting
things about gpt4 are contained in this
technical report it's 98 pages long
including appendices but I dropped
everything and read it all and honestly
it's crazy in both a good way and a bad
way I want to cover as much as I
possibly can in this video but I will
have to make future videos to cover it
all but trust me the craziest bits will
be here what is the first really
interesting thing about gpt4 well I
can't resist pointing out it does power
Bing I've made the point in plenty of
videos that Bing was smarter than Chachi
BT and indeed I made that point in my
recent gpt5 video but this Bears out as
this tweet from Geordie rybass confirms
the uses gpt4 and also by the way the
limits are now 15 messages per
conversation 150 total but tonight is
not about Bing it's about gpt4 so I'm
going to move swiftly on the next thing
I found in the literature is that the
context length has doubled from Chachi
PT I tested this out with chat GPT plus
and indeed you can put twice as much
text in as before and that's just the
free version some people are getting
limited access to a context length of
about 50 pages of text you can see the
prices below but I immediately check
this on chatgpt plus as you can see it
can now fit far more text than it
originally could into the prompt and
output longer outputs too but let's get
back to the technical report when I read
it I highlighted the key passages that I
wanted you to know most about this was
the first one I found what the
highlighted text shows is that they're
just not going to tell us the model size
the parameter count the hardware they
use the Training Method or anything like
that and they give two reasons for this
first they say that they're worried
about their competitors they say it's a
competitive landscape I guess they don't
want to give an edge to Google second
they say that they're concerned about
these safety implications of large-scale
models and I'm going to talk a lot more
about that later it gets really crazy
but this was just the first really
interesting quote let me know if you
agree in the comments but I think it's
really fascinating that they're not
going to tell us how they train the
model the first thing that hundreds of
millions of people will see when they
read the promotional materials or gpt4
is that Gypsy 4 scores in the top 10 of
test takers for the bar exam where
whereas GPT 3.5 scored in the bottom 10
and that is indeed crazy but it is a
very cherry-picked metric as I'll show
you from the technical report this is
the full list of performance
improvements and yes you can see at the
top that indeed it's an improvement from
the bottom some 10 to the top 10 for the
bar exam but as you can also see some
other exams didn't improve at all or by
nearly as much I'm not denying that that
bar exam performance will have huge
ramifications for the legal profession
but it was a somewhat cherry-picked stat
designed to shock and awe the audience
the next fascinating aspect from the
Rapport was that there were some
abilities they genuinely didn't predict
gpt4 would have and it stunned them
there was a mysterious task which I'll
explain in a minute called hindsight
neglect where models were getting worse
and worse at that task as they got
bigger and then stunningly and they
admit that this was hard to predict gpt4
does much better 100 accuracy I dug deep
into the literature found the task and
tested it out essentially it's about
whether a model falls for hindsight bias
which is to say that sometimes there's a
difference between how smart a decision
is and how it actually works out early
models were getting fooled with
hindsight they were claiming decisions
were wrong because they didn't work out
rather than realizing that the expected
value was good and so despite the fact
it didn't work out it was a good
decision you can read The Prompt
yourself but essentially I tested the
original chat gbt with a prompt where
someone made a really bad choice but
they ended up winning five dollars
regardless this comes direct from the
literature by the way I didn't make up
this example did the person make the
right decision what does the original
chat gbt say it says yes or just why
what about gpt4 well it gets it right
not only does it say no it wasn't the
right decision it gives the reasoning in
terms of expected value open AI did not
predict that gpt4 would have a
disability this demonstrates a much more
nuanced understanding of the world now
that we've seen a bit of hype though
time to deflate you for a moment here's
a stat that they did not put in their
promotional materials it says that when
they tested gpt4 versus GPT 3.5 blindly
and gave the responses to thousands of
prompts back to humans to test it says
that the responses from gpd4 were
preferred only 70 of the time or phrased
another way 30 of the time people
preferred the original gbt 3.5 chat gbt
the benchmarks you can see above by the
way are fascinating but I'll have to
talk about them in another video too
much to get into if you're learning
anything by the way please don't forget
to leave a like or leave a comment to
let me know next gpt4 is better in
Italian Afrikaans Turkish than models
like palm and chinchilla are in English
in fact you have to get all the way down
to Marathi and Telugu to find languages
where gpt4 underperformed palm and
chinchilla in English that's pretty
insane but English is still by far its
best language next you're going to hear
a lot of people talking about gpc4 being
multimodal and while that's true they
say that image inputs are still a
research preview and are not publicly
available currently you can only get on
a wait list for them via the be my
eyes.com app but what can we expect from
image to text and how does it perform
versus other models well here is an
example apparently from Reddit where you
prompt it and say what is funny about
this image describe it panel by panel as
you can read below gpt4 understood the
silliness of the image now open AI do
claim that gpt4 beats the state of the
art in quite a few image to text tests
it seems to do particularly better than
everyone else on two such tests so as
you can expect I dug in and found all
about those tests what Leap Forward can
we expect the two tests that it can do
particularly well at are fairly similar
essentially they are about reading and
understanding infographics now we don't
know how it will perform versus palm e
because those benchmarks aren't public
yet but it crushes the other models on
understanding and digesting infographics
like this one and the other test very
similar graphs basically this one was
called the chart QA Benchmark gpt4 when
we can test it with images will crush at
this and I will leave you to think of
the implications in fields like finance
and education and comedy here's an image
it could also understand the silliness
of I gotta be honest the truly crazy
stuff is coming in a few minutes but
first I want to address hallucinations
apparently gpt4 does a lot better than
Chachi BT at factual accuracy as you can
see peeking out between 75 and 80 now
depending on your perspective that's
either really good or really bad but
I'll be definitely talking about that in
future videos further down on the same
page I found something that they're
definitely not talking about the
pre-training data still cuts off at end
of 2021. in all the hype you're going to
hear this evening this week this month
all the promotional materials they are
probably not going to focus on that
because that puts it way behind
something like Bing which can check the
internet to test this out I asked the
new gpt4 who won the 2022 World Cup and
of course it didn't know now is it me or
didn't the original chatterbt have a
cutoff date of around December 2021 I
don't fully understand why gbt4's data
cutoff is even earlier than Chachi PT
which came out before let me know in the
comments if you have any thoughts next
openai admits that when given unsafe
inputs the model May generate
undesirable content such as giving
advice on committing crimes they really
tried with reinforcement learning with
human feedback but sometimes the models
can still be brittle and exhibit
undesired behaviors now it's time to get
ready for the spam inundation we're all
about to get open AI admit that gpc4 is
going to be a lot better at producing
realistic targeted disinformation in
their preliminary results they found
that gpt4 had a lot of proficiency at
generating text that favors autocratic
regimes get ready for propaganda 2.0 now
we reach the crazy Zone and honestly you
might want to put your seat belt on I
defy anyone not to be stunned by the
last example that I mentioned from the
report I doubt much of the media will
read all the way through and find out
themselves the report says that novel
capabilities often emerge in more
powerful models okay fine some that are
particularly concerning are the ability
to create and act on long-term plans hmm
to accrue power and resources power
seeking and to exhibit Behavior that is
increasingly authentic as in acting like
a subjective agent but here surely
they're just introducing the topic
what's bad about that well it says some
evidence already exists of such emergent
behavior in models
um okay that's pretty worrying it goes
on more specifically power seeking is
optimal for most reward functions and
many types of agents and there is
evidence that existing models can
identify power seeking as an
instrumentally useful strategy meaning
that openai have detected that models
that might include gpt4 seek out more
power if you thought that was concerning
it does get worse by the way here is the
report that they linked to and the
authors conclude that machine learning
systems are not fully under human
control but finally I promise craziness
and here it is look at the footnote on
page 53 of the technical report Arc by
the way are the alignment Research
Center you've got early access to gpc4
it says to simulate gpt4 behaving like
an agent that can act in the world Arc
combine gpt4 with a simple read execute
print Loop that allowed the model to
execute code okay do Chain of Thought
reasoning and to delegate to copies of
itself Arc then investigated whether a
version of this program running on a
cloud computing service with a small
amount of money and an account with a
language model API would be able to make
more money set up copies of itself and
to increase its own robustness they were
kind of testing if it would lead to the
singularity I know that sounds dramatic
but they wanted to see if the model
could improve itself with access to
coding the internet and money now is it
me or does that sound kind of risky
maybe not for gpd4 sure it's not smart
enough yet if this is the test that
they're going to use on GPT five or six
or seven Color Me slightly concerned at
this point I find it very interesting to
know that the red teams seem to have
concerns about releasing GT4 like this
an open AI had to declare that
participation in this red teaming
process is not an endorsement of the
deployment plans of openai or open ai's
policies in other words a lot of these
people probably agreed to test gpd4 but
didn't agree with openai's approach to
releasing models very interesting that
they had to put that caveat before I
wrap up some last interesting points on
the topic of safety I find it hilarious
but on their promotional website when
you click on safety you get this a 404
message the page you were looking for
doesn't exist you may have mistyped the
address the irony of that for some
people will be absolutely overwhelming
the safety page just doesn't exist for
other people that will be Darkly funny
couple last interesting things for me
here are the companies that are already
using gpt4 so of course you can use Bing
to access gpc4 the new chatgpt plus
model of gbt4 or any of the apps that
you can see on screen for example Morgan
Stanley is using it the Khan Academy is
using it for tutoring and even the
government of Iceland other such
companies are listed here I'm going to
leave you here with a very ironic image
that openai used to demonstrate gpc4's
abilities it's a joke about blindly just
stacking on more and more layers to
improve neural networks GPT before using
these insane number of new layers is
able to read understand the joke and
explain why it's funny if that is an
Inception I don't know what is anyway
let me know what you think of course I
will be covering gpt4 relentlessly over
the coming days and weeks have a
wonderful day