Deep Learning 5: Optimization for Machine Learning

so I'm James Martens a researcher at
defined and then I'll be talking today
optimization from C learning and more
specifically neural implants we can Bob
so the topics will cover our video
consent which is essentially the
standard canonical method momentum
ethics mature simple ways accelerating
gradient descent second-order method
signatures are slightly more complicated
way starting ready and descent and
finally I'll talk about stochastic
optimization is how it sort of relates
to these things so motivation well
numerical optimization methods are sort
of the methods that enable all modern
machine learning techniques to work as
them to adapt their parameters to the
problem in here in particular this is
the image behind deep learning so it is
that you solve a you basically phrase
you problem is the minimization of some
objective function that quantifies that
performance in the model dictionnaire
states with some task and the general
idea is that they work that making small
incremental changes because the
objectives well chosen they're smooth
all right so just briefly some notation
that's gonna carry through the lecture
I'm gonna note by theta the parameters
of the model that's that these are the
quantities that we're optimizing that
we're adapting the objective function
will be done by H and our goal is always
going to be minimization of H with
respect to theta so gradient descent
which is sort of the standard method is
quite simple you just update your
parameters by taking a negative step in
the direction of the gradient your the
gradient is defined this way there's a
vector of partial derivatives alpha is
called the step size it's also called
the learning rate
depending on which literature you're
reading on this control sort of how
aggressively you move in that direction
so why should this work well gradient
descent so the gradient direction which
is what we're using to change the
parameter is essentially the direction
of greatest reduction in the objective
function per unit of change in theta
this is what this is why it's also
called steepest descent because it is
sort of the steepest direction to follow
and you could phrase this formally as
you know the gradient properly
normalized is just given by this limit
it's like this it's the vector that is
you know whose norm is less than epsilon
that minimizes the objective function
and I and as this becomes true this this
equality as epsilon goes to 0 so if H is
relatively smooth then the gradient will
keep pointing downhill over some sort of
non non negligible distance and this is
sort of the key fact that we that makes
gradient descent work because of course
if you you know if you followed it and
in right immediately suddenly you're
your objective function curves or
changes or experiences some kind of
discontinuity then this whole strategy
would be hopeless you could also
motivate gradient descent as optimizing
a certain local approximation of the
objective function so if you think of
the objective function has been
approximated by its own first order
Taylor series around the current theta
with respect to some perturbation D then
you know this is well you can this is a
sort of the canonical Taylor series
first order approximation for for H
around that theta it's just the current
value plus the gradient times the the
the D vector and so if you have a
condition called Lipschitz smoothness
which all sort of define later on the
gradients that this essentially means
this is saying that you know this
approximation won't be too bad if D is
small enough and the gradient update is
computed again by minimizing this local
approximation now in some kind of sphere
of radius R so we're not obviously you
know if you minimize this thing globally
this local approximation you'll get well
essentially it'll actually just make the
parameters infinitely large in whatever
direction is whatever direction the
gradient is pointing which obviously
won't be any good the reason that's no
good is because you know the proximation
is wrong if you go too far away so we
restrict ourselves to this sort of
sphere of radius R and this implies that
are the updates who are on our
parameters should be some scaled version
of the gradient which is what gradient
descent does so gradient descent is you
know a reasonable first algorithm but
you know it's it's got some obvious
problems and and the example which I'm
going to sort of carry through this
lecture is to serve as a sort of running
example that I like a lot is this sort
of simple 2-dimensional quadratic you
could think of this almost as a if you
don't want to you know think of a the
objective function itself is being
chondritic this is sort of what the
function looks like locally in some area
and you might imagine sort of this sort
of curved valley so you have that the
sides of the valley is sort of curving
up sharply in this direction but it's
fairly flat in this direction and what
gradient ascent is going to do depending
on the learning rate is it's either
gonna sort of bounce back and forth
between these directions of high
curvature it keeps gonna keep hitting
sort of the sides of the valley and then
getting shoved the other direction very
fast and this if your learning rate
isn't small enough is gonna result in
these oscillations that get bigger and
bigger and eventually you diverge you
could also take a smaller step size to
alleviate that problem but then you know
you do get this stable behavior but now
your progress along this very flat
direction is now limited by how big your
step is and if your step is very small
you're gonna making these tiny tiny tiny
incremental changes it's gonna take a
very long time to go in this flat
direction so the problem here is that
you have very curved directions very
flat directions that's sort of the bad
case that if you think about condition
numbers this means that the condition
numbers big
so no good choice there right so this is
exactly basically what the the example
is it is demonstrating if you have
functions whose curvature varies in
different directions gradient descent is
quite slow and there's um there's some
of no sweet spot in terms of this step
size you either it's either big enough
that you get big oscillations or it's
small enough that you're sort of too
slow in the flat directions so and and
perhaps you know to further another way
I think of discussing gradient descent
which will lead into you know possible
solutions to this issue is that we are
essentially minimizing not a first-order
approximation to the objective function
but a second order approximation to the
objective function so this is the
similar Taylor series from for you know
up to first-order but I'm adding them in
this quadratic term now which is a
function of the Hessian and what
gradient descent could be thought of
doing is saying well I don't have access
to the Hessian so I'll proximate that
with some scalar times the identity
matrix and now my quadratic looks like
this and if I minimize this quadratic I
get once again a scaled version of the
gradient if L for example is some upper
bound on the curvature then you can you
can sort of show this as a reasonable
thing to do that this in fact this step
size one over L is sort of the optimal
step size in the worst case but you know
so these are the issue here but this
illustrates is that you know Li is
actually quite quite a poor
approximation of H it sort of it's the
most pessimistic approximation possible
it says you know all of the curvature in
all directions is equal to the maximum
curvature which is which is the safe
thing to do but it's all it's also you
know it's gonna slow you down a lot it's
gonna result in very small step sizes
prevents prevents divergence but slow
convergence so I'll touch on a little
bit of theory here so you could
formalize these statements so if we talk
about H as having Lipschitz continuous
derivatives which is this property here
which is essentially saying that the
gradients don't change too much when you
change your position when you change
theta you know in particular that it's
bounded by L times the change in theta
and L is actually the same L from the
previous slide and if you have another
condition strong convexity and this is
usually just a convenient convenience
condition if you want to prove of global
convergence if you care about local
convergence you may not have to have
exactly this kind of statement but you
might want your function to at least be
locally strongly convex sort of strongly
convex in an area so these with these
two technical conditions and also just
that the gradients are computed exactly
so again I'm not we're not getting into
stochastic optimization yet you have
this kind of bound for conversions so
you see that the objective function
value versus the value at the optimum
which is theta star can be upper bounded
by this expression here which is fairly
you know somewhat intuitive so you this
is sort of the initial distance from
your starting position which is theta 0
to the optimum and then you have
essentially this this ratio to the power
2 K and K here is the iteration number
so your so this is so this is um
depending on the the nomenclature you
use and this is called linear
convergence or X you know you could also
think of this as sort of a convergence
at a kind of an exponential rate given
by this rate and here Kappa is this
condition number which is the ratio of
the largest curvature divided by the
smallest one this this mu term from the
previous slide this is sort of you could
think of this as capturing the smallest
curvature in the objective it's a lower
bound on it so this is this is sort of
you know this is an upper bound so you
you always have to take with the grain
of salt
it's not like a it's not
it's not a lower bound it's it's not an
equivalence you know these things could
converge faster than this rate but in
the worst case they converge at this
rate and it gets worse and worse as the
ratio of the largest curvature the
smallest curvature gets bigger and and
you know and so that's that's not great
because in neural nets for example you
could imagine this affective condition
number being very big yeah yeah that's a
good that's a good question so like if
you're proving these kinds of theorems
you almost always just take these
constants to be globally true across the
whole function but in practice of course
if you know the function is doing some
weird stuff out there that you don't
care about and you're sort of restrict
yourself to a neighborhood you could
just talk about the behavior in that
neighborhood and then just ort of apply
this theory assuming that you stay
inside of that neighborhood and some of
the theory is sort of phrase that way
they say like assuming we don't allow
ourselves to deviate from some small
ball ball Epsilon and that these
conditions hold inside of that ball then
we get conversions such-and-such
yeah and in some sense one of the
advantages of you know other methods is
that they can kind of adapt to the local
properties MFL if the effective LM you
are changing you know you might watch
you want to sort of change your learning
rate as you go and sometimes that's what
second order methods do but we'll get
into that right so you can look at that
as the the number of iterations to
achieve convergence to a tolerance
epsilon is given by this expression so
so so conversions theories is you know
is interesting it's informative but it's
it should be taken with a grain of salt
um so this is an upper bound and so it
has to work in particular it has to work
on all the objective functions that sort
of satisfy these smoothness properties
and and this and this local strong
convexity property and this includes in
particular worst case examples now real
problems are not really worse case and
they often contain sort of special
structure that actually might make them
much easier than the worst case so you
can think of these bounds is actually
quite pessimistic and unrealistic
and the learning rates that they tell
you to use are actually almost always
overly conservative so for example you
could imagine there there being other
technical conditions that might help us
out for example if you have clustered
eigenvalues in the Hessian conjugate
gradient as a method turns out to be
much faster than gradient descent and it
also might be the case that's you know
that the direction of smallest curvature
might be completely flat so so flat that
it's actually unimportant to optimize it
and you'll you can get sort of a
reasonably high reasonably good
tolerance with just ignoring that
directions and sometimes the effective
condition number is now different but
it's hard to it's hard to exactly be
rigorous about that another problem with
these kinds of bands is that they're
basically describing asymptotic
performance how do you do
as you after you've waited a sufficient
amount of time and in practice we
actually stop optimization long before
these sort of asymptotics matter either
because we're interested in preventing
overfitting or because we've simply run
out of time and there's this sort of
phenomenon that actually you know what
goes on in the early stages of
optimization is actually not really well
described by the theory that describes
what goes on in the late stage so you
might hope that oh if you have better
asymptotic performance you have better
pre asymptotic performance but those two
things don't necessarily correlate at
least not very strongly there's also no
guarantee for the in these bounds for
non-convex objectives but this is mostly
due to the fact that you know we don't
know how much better or worse local
minima are or have sorry we're how much
worse they are versus global minima this
is kind of an ongoing research correct
question for various non convex
objectives sometimes you know and
sometimes you can prove things but
oftentimes you don't especially for
neural nets you don't so I would say
that you know at the end of the day what
people do is they is they see how things
work in practice
you know theory is a guide but it should
not never be an absolute prescription
for what to do
so one way that we can improve gradient
descent probably the simplest way is
with a method called momentum and the
motivation here is that the you know you
have the this destruction of descent
which changes potentially at each
iteration especially if you're sort of
going bouncing back and forth along one
of these values and you sort of want to
see that and kind of damp it down and
hopefully cancel this kind of
oscillation naturally so the solution is
to essentially build up a overall
velocity as you go so you sort of add
your previous directions together you
allow oscillating directions to cancel
themselves and directions that are sort
of persistently moving in one direction
they get amplified over time
just as you would have a ball sort of
rolling across a smooth surface it will
sort of start to accelerate in
directions of persistent motion and but
if it's if there's some kind of
oscillation like this it's sort of never
really builds up much velocity in that
direction so it's and then it's quite a
simple modification really if the
classical gradient descent method so
here what we're gonna do is we're gonna
maintain a velocity vector V and it's
just equal to some kind of decayed
version of the previous velocity vector
so we don't we don't allow the velocity
accumulate accumulate infinitely be sort
of pretend that there's some kind of
friction in the system that slows it
down over time that's accomplished by
multiplying it by this constant here
which is you know typically a constant
like 0.9 something that decays it
gradually but not not uh not too fast
and and then and then we just add in the
current gradient scaled by the learning
rate and now we add this velocity to the
current parameter that becomes our
update now and there's a alternate
version of momentum called mr. ups
method and this is actually quite
similar if you if you know how to write
it this way it's essentially just the
same update as momentum but there's this
sort of look ahead step where
sort of compute the new gradient not at
your current position but sort of at a
position that you sort of anticipate
that you'll move in because in some
sense right at the very next up you know
the next step may be the parameters were
sort of moving ahead by precisely this
although plus this gradient term
although we don't with the gradient term
is yet so we can't compensate for that
but we can sort of do this partial
compensation for our anticipated next
update and it's this version is well
it's a it behaves similarly in practice
although it has some better theory
sometimes that works better in practice
so here's this failure case from before
for gradient descent here and what
momentum allows us to do is we sort of
we you know we start oscillating as you
would with gradient descent but then
immediately once you want to sort of
move back momentum has already
remembered the direction going this way
so if it sees another direction going
this way those will cancel arithmetic
aliy and you won't actually have any
sort of net velocity sloshing back and
forth anymore and the only velocity that
you'll sort of persist over time will be
this consistent direction pointing
downwards so you sort of get faster and
faster in this direction and you dampen
the oscillations in this direction which
is precisely what you want so right so
Nesterov method I mentioned this that it
has stronger theoretical guarantees it's
it's slightly better in practice
although often times it doesn't matter
the differences are bigger when the
learning rate is larger and you can
actually show that it becomes equivalent
to the standard version as the learning
rate goes to zero so I'll now I'll talk
about some convergence theory for these
methods and momentum methods so first I
want to make a definition so a
first-order method is is technically
defined as one where the update that you
take at each iteration is given by a
linear combination of the gradients
computed at previous iterations so this
is the the technical statement here it's
like that this is this up this is the
difference between the two consecutive
parameter vectors which is just our
update and that's that's inside of the
span
of the previous gradients evaluated at
these previous iterates so this
definition includes as a special cases
gradient descent within without momentum
it also includes more complex methods
like conjugate gradients although it
doesn't include any method that would
multiply the gradients by some
non-trivial matrix so let's say a
second-order method and so to analyze
this general class we can construct a
sort of a hard case as an objective
function which um while it is quadratic
and that's a nice structure to have it's
nonetheless hard enough that it will
sort of stress these methods to the
maximum and this is the this is the
example
it's say infinite dimensional quadratic
um so we're gonna assume infinite
dimensionality here given by this
expression and I won't I won't try to
unpack it too much it's basically just a
technical construction and you can show
that any first order method so this
includes gradient ascent momentum mall
you know CG actually has this lower
bound so this is not an upper bad
anymore this is a lower bound this is
this is how well it can do in the best
case as applies to this problem where
you've picked the learning rate
optimally and such and it's it's
actually you know somewhat reminiscent
of the bound we saw before although it's
actually it's actually better it's it's
it's it's better than a upper bound
because instead of having high Kappa
which is the condition number we now
have the square root of Kappa so we so
it's actually it's actually better than
that
then the upper bound but you could say
ok so actually how much better is
momentum and of course it can't be
better than this lower bound does it
actually match the lower bound well it
turns out that it does so if you have
the standard conditions about Lipschitz
smoothness and the local strong
convexity then if you choose the
parameters well of momentum so which is
the decay
and also the the learning rate you have
this upper bound which essentially
matches the lower bound from the
previous slide so you can say in some
sense that momentum or in particular
this this bound applies to the the
Nesterov version of it is optimal in the
worst case and we can see that sort of
summarized here so this is this is the
worst case lower bound for first order
methods this is what gradient descent
gets as an upper bound and this is what
what nesterova momentum does so this
matches the lower bound but I should say
that these are again worst case
constructions so you might want to
conclude from this at oh well okay I
mean if we're doing first order methods
that's it that's the end of the game at
nessarose mentum it's the only thing you
should use that's not true it's not true
because this game there could exist
examples that have a special structure
that that this weird example that we
constructed doesn't share that would
actually make some hypothetical method
better than Nestor s method and indeed I
mean you don't this doesn't even have to
be an intellectual exercise there is an
example of this which is conjugate
gradient if you apply conjugate gradient
to a finite dimensional quadratic it'll
actually technically converge in a
finite number of steps and if you want
to if you if you can even experience
faster convergence if you have say for
example clustered eigenvalues so if
you're if you're if you're if you have a
quadratic problem and the eigenvalues of
its session are clustered and say three
spots you sort of roughly converge in
three iterations which you wouldn't get
with nessarose method nonetheless you
know worst-case analyses are you know
important and you could maybe argue that
some hard problems like neural nets
might sort of be approaching the worst
case although not exactly there is a gap
there anyway and then this leads into
the next type of method which is a which
are second-order methods so we remember
from before you know we could we could
talk about gradient descent and
minimizing a quadratic objective
function where we sort of hook which is
a look rather a local approximation to
our true objective function and we we
took the Hessian which was you know this
complicated object and we approximate it
with L times I where L was this sort of
upper bound on the curvature which could
you could also think of as the largest
eigenvalue of H at least if H was sort
of the largest eigenvalue of all
possible HS as you sort of move over the
surface and but we could actually just
minimize this form without doing that
kind of approximation so just just just
leave H as it is and if you do that you
get this kind of update to the
parameters which is the the inverse of
the Hessian times the gradient and this
would imply a basic iterative method
that looks like this this is essentially
a Newton's method now so what does that
do so this is again our familiar
behavior of gradient descent this is
with momentum and and and second-order
methods in some sense if they're
behaving well they immediately model
this sort of they see this curvature
here they see this flat curvature here
and they just zoom right to the optimal
spot immediately if this was a curtain
if this was actually a quadratic it
would actually converge in one step but
insofar as it's not exactly a quadratic
it'll sort of it'll take more iterations
than that so that sounds great but
there's a lot of problems to second
order methods the first problem which is
probably the most important problem is
that this idea of approximating the
function
you know locally around the current
point of course you know relies relies
on us not moving too far if we if you
know we we if we base our update on this
on this minimization of this local
approximation in the approximation
breaks down beyond a certain radius then
we sort of have to restrict ourselves
from moving outside of that radius which
can be difficult in the
us down and also once unlike you know
this this approximation that we were
doing before which was like the the
largest eigenvalue times the identity
function as an approximation or Theodore
D matrix rather its approximation of age
you know if we if we use a the real age
and we start moving away from our
current position actually the effect of
H like this there's H as a function of
your current position will start to
change and these small directions of
curvature that we've sort of measured
might actually become wildly inaccurate
very fast because they're it's no longer
an upper bound it's no longer a global
upper bound anymore
in fact the curvature could even become
negative which would then um if you try
to then sort of literally apply this
sort of minimization of a local
quadratic with a negative curvature that
essentially just means you you you you
zoom infinitely far in that direction of
negative curvature which doesn't make
any sense obviously because our function
is not actually cut Radek so again we're
gonna follow this prescription of
staying within some local region around
an update of size zero so you know or
made it we're minimizing with respect to
D this local local approximation but
keeping it inside of this some local
region R so I'm just gonna check how I'm
doing in terms of I have no concept of
how many slides that are actually done
and it doesn't say unfortunately okay
I'm about halfway through but over
halfway through I might slow down a
little bit then we could also go back we
could also have a break
ask me if you won't want me to go slower
I could do that so so yes Oh a solution
to this this problem is is to take our
region to be this essentially a sphere
centered at 0 with radius R and it turns
out that doing performing this
minimization is you know can be
accomplished with with this formula here
which is essentially just saying we add
a multiple of the identity to H before
we invert it and then we multiply that
by that gradient as usual for some
multiple which is given by this lambda
so it turns out that the relationship
between lambda and the radius R is sort
of a complicated function and
fortunately though you know if if this
is all you want to do you can just work
directly with lambda you don't actually
have to think about R and what all that
relationship you can just say okay I
know that each lamp is equivalent to
some R I'll provided lambda is big
enough there are certain technical
conditions here I'm sort of glossing
over so we just sort of think about
adding multiples that they did the
identity to 2h as a sort of a general
technique and there are ways of adapting
the exact value of this multiple these
sort of heuristics that are somewhat
theoretically supported such as the
Levenberg mcard method another solution
is is to use say a different type of
matrix than the Hessian one that might
have more forgiving properties so this L
times I you know which is which was
which is one alternative to the Hessian
is a poor choice because it's sort of
saying that all directions have this
same curvature and it's actually the
worst case because it's the largest
curvature over the whole objective
um what you could imagine choices that
are sort of more subtle than that but
are still not as aggressive as the
Hessian so they might say well I'm gonna
throw maybe all make directions of
negative curvature
I might sort of hedge my bets and think
of the curvatures as you know high
because that's sort of that's a safe
thing to do right if critias curvature
does change as you move in the objective
if you're always conservative that means
you're sort of more likely to not
overstep in any one direction you don't
want to be too conservative it's this
sort of balancing act and and designing
these matrices is kind of a whole
research topic so and often I found in
my own research that in fact you know a
combination of this kind of idea of
substituting the Hessian for kind of a
matrix with nicer properties and then
still applying the these traditional
trust region techniques is actually so
that this yields by far the best
performance in practice under all mats
so yeah yeah so so that'll be touched on
in the next few slides this is yeah this
is the other big problem so right now
I'm still tackling like the first big
problem the second order methods but
yeah you're right exactly that the
problem of this giant matrix is the
other big problem and it sort of has to
be solved in conjunction right so so
some examples of such alternative
matrices are the generalized gauss
newton matrix the Fisher information
matrix and the empirical Fisher
information matrix these are sort of the
three matrices that are used pretty much
in a neural net research so see what how
what time is it by the way okay maybe
I'll just discuss the general ice
Augusta Newton and then we can take a
break
so the general
scoffs Newton is a matrix that you can
only apply when you have certain special
structure so in particular we're going
to assume that our objective function H
is the sum of smaller objective
functions H I / I you can think of I as
indexing the cases in an in a training
set so this is a very standard situation
machine learning in a particular each H
is given by a loss function which
measures the difference between a target
and the prediction made by some F
function which you know will often be a
neural network parameterize by theta
which takes X as its input right and so
if you have this kind of structure and
that you have that in fact the loss
function is also convex in the perp in
its parameter Z where Z is the
prediction then you can define this
generalized Gustav matrix and it's given
by this formula here which is somewhat
complicated so I'll try to unpack this
so you have essentially the Jacobian of
the F function transposed times the
Hessian of the loss and then the
Jacobian a gain of the F function that's
the definition it's it's so you can it
seems somewhat arbitrary but I think I
think one thing to notice immediately is
that the H so if the loss function is
convex then H is actually a positive
definite matrix or a positive semi semi
definite matrix and depending if it's
the convexity strict or not and that
means that this whole thing is provably
positive semi definite which is a good
property that means that there's no
directions of negative curvature which
is one of the things that was sort of
worrying about the Hessian so you can so
there's different interpretations
different motivations for this kind of
nature
you can think of at first of all as what
the Hessian of H would look like if we
replaced each F with a a local linear
approximation of F so we're essentially
just plugging in F first order Taylor
series into the definition of the
objective function and then computing
the Heshy and so that's just this so you
can show that in fact you get this you
get this form and the Jacobian appears
here this is this is why the Jacobian
appears in the formula for the Gen Ys
guys Newton if the loss function is a
squared error which is a you know a very
standard classical err is probably like
the like the classical loss function
then you get that the Hessian is
actually equal to the identity and that
generalist Gauss Newton G actually just
becomes the sum of the Jacobian times
the Jacobian which is the classical
Gauss Newton
this is why it's named after Gauss is
because this is actually a very old idea
and he was you know experimenting with
these kinds of squared loss functions
all the way back then and so you see
this matrix appear in the so-called
Gauss Newton method for optimizing on
nonlinear least squares problems you
could also show that if the loss
function is equal to the negative log
problem set given the prediction Zed
which it very often is for some natural
conditional density P this again this
happens all the time in machine learning
and neural Nets then the generalize
ghost Newton becomes equivalent to the
so-called Fisher information matrix
associated with this conditional
probability distribution and in fact in
that case G inverse times H which is
this update that we're gonna you we're
gonna compute from from the scheme would
be just called the natural gradient so
there's a whole literature on the
natural gradient and its various
properties whoa
it's not good
hopefully it's just my computer taking a
siesta yeah you think changing the slide
wouldn't would prevent it from going to
sleep but whatever all right and the GGN
matrix has some nice properties I
already pointed out that it's positive
semi-definite that means that there's no
directions of negative curvature it
seems to be more conservative than the
Hessian in the sense that it sort of
often tends to overestimate the
curvature in any one direction which is
good it's good to hedge sort of in that
direction but it you can't prove that it
will always overestimate in some cases
it will underestimate and they you can
show that if you do take updates with
the inverse of the general sky certain
times they head times the gradient as we
as one would that in fact your your
optimizer becomes in some sense
invariant to how you parameterize your
objective function which is a very nice
property to have unfortunately though
that invariance is sort of asterisked by
the condition that you need to take very
very small updates so this is only
technically true if you're taking sort
of updates that are sort of
infinitesimally small which is sort of
more of a theoretical curiosity you'd
never actually do that in practice but
it's nice to know that at least as your
updates get smaller and smaller it
starts to behave more and more like this
perfectly parameterization invariant
method just just to be just just out of
curiosity how meaningful is that to
people here if I say parametrization
invariance so this is the
parameterization of say the the the
neural net itself so you know you can if
you want to take the example of neural
nets you could say like I you know my
weights are you know my typical
parameters and usually I'm optimizing
those but you could pass the weights
through some kind
of invertible function and then plug
those into the neural net and you could
optimize you know assuming that that
you've that there is this sort of
additional invertible function between
your parameters and the neural net that
are sort of changing the definitions of
the parameters in some sense and methods
like this you know if you're taking very
very small steps you can show that it'll
behave exactly the same no matter what
that invertible function is C so which
is nice because we're sort of we don't
quite know that our are the standard way
that we've parameterised neural Nets is
the optimal way right it might not be
right there might be a you know from us
from the standpoint of optimization a
different parameters ation can make a
world of difference so being invariant
to that choice seems like a good thing I
wouldn't say that it's obviously a good
thing but it's because you know there
are this was actually brought up by
somebody in my my PhD exam but you know
there are trivial examples of systems
that are invariant to the parameters
that are the parameters ation that are
nonetheless stupid like say for example
you your optimizer takes no steps at all
right it it's update is always just the
zero vector that is technically
invariant isn't it but it's a very poor
optimizer nonetheless it seems like a
good thing to have and this is analogous
so there is an analogous property for
the Hessian you might have you know read
about in sort of standard optimization
textbooks and that's that they're
optimization with the Hessian so
classical Newton's method in other words
is invariant to linear real parameters
ations of the function so if this
invertible function that you've inserted
between your parameters in your network
is actually just a multiplication by a
matrix say then Newton's method is
invariant to that kind of thing but this
is actually a stronger form of
invariance and it's kind of curious that
we've approximated the Hessian with this
other
that's sort of a maybe you could even
think of as a poor-man's Hessian
although I wouldn't but it's actually
now it's invariance properties get
better not worse and you can actually
show that the Hessian is not invariant
in this sense it's it's is actually like
it's not just that it it's only it's not
just that we we don't know it's that we
have we know that it isn't invariant in
that sense so in practice if you if you
take updates that you've computed with
the generalize go suit matrix like in
this way on you know I've been able to
show that this is sort of hundreds or
even thousands of times faster than
gradient updates so that's great of
course you know I haven't touched on a
very important issue which is that we
don't know in general how to compute
these updates efficiently in high
dimensions I'm not sort of this is now
getting to your question and I think I
will take a break there and we can
discuss this problem and solutions to it
after the break so we'll resume just the
projector heats up here right so we so
we left off talking about second order
methods instead of this the first major
problem which was that they're they're
using this local quadratic approximation
that becomes inaccurate if you move out
too far and we had different solutions
for that different matrices that we
could use instead of the Hessian or
different techniques to constrict the
update that we ultimately compute by
minimizing that local quadratic to
restrict it into some kind of region
where the local quadratic approximation
remains accurate but that doesn't
address the other the second major
problem with second-order methods which
is that you know we're dealing typically
in very high dimensions for neural nets
the the parameters can you know have
potentially tens of millions of
dimensions and neural nets are getting
bigger you know every year and these
Hessians are these these replacements
for the Hessian they all involve these
giant n by n matrices and then to
actually invert that matrix in terrific
cute the optimal update requires n cubed
copy floating-point operations which is
is going to be way way too much if we
have tens of millions of dimensions so
and this problem existed even well
before you know neural nets got very big
even in the 90s you know yet you had
neural nuts with like 10,000 parameters
well that was too much even back then
even a thousand parameters might have
been too much for like a 486 so people
have been trying to solve this problem
for a long time and even before that
into this 60s and 70s this is sort of
the topic of the classical optimization
literature how do we do Newton's method
cheaply so there are you know we have
three whoo that is getting irritating
see the issue is that these Google
laptops have very very strict security
settings so if you step away from them
in a coffee shop they lock you out very
very fast because otherwise somebody
will spy on Google so you have to keep
you have to keep moving you have to keep
active so right so we want to so we want
to approximate the curvature matrix such
that we can you know computed
efficiently that we can store it
efficiently and finally that we can
invert it efficiently or maybe we will
resort to approximate inversion that's
another possibility so the first major
curvature matrix approximation are is
diagonal approximations and and
typically this amounts to taking the
curvature matrix B that you would
normally have whether that be the
Hessian or some you know that generalize
gauss newton that we talked about and
approximating it with its own diagonal
this notation here just means the
diagonal of this matrix and then you
turn that into a diagonal by putting it
onto zero matrix you get the idea so the
storage cost of this is great right it's
just all of em it's just the number of
parameters all you have to do is store
those diagonal entries
and applying it to compute you know to
compute the update via the inverse is
also very cheap right inverting the
diagonal matrix just amounts to
inverting each entry which are scalars
so again that's just Oh van now that's
the storage and inversion cost the
computation cost for diagonal matrices
can actually be a bit subtle and it
really depends on the form of the matrix
that you're using so for certain certain
types of B's it's actually this can be
sort of non-trivial and there are
actually computing it exactly can be
hard but there are unbiased estimation
techniques for example my work on carpet
or propagation certain special cases you
can do it efficiently other cases you
sort of have to resort to a technique
like this or other techniques of a
similar flavor but it's more or less
well understood how to do this for the
typical cases I'm actually working on a
code base right now which among other
things allows this to be done
efficiently for the generalized gauss
newton and other such matrices
it's the empirical fisher where this is
very very easy and this is in some sense
why the empirical fisher appears so
often so the so this approximation will
be reasonably accurate if the eigen
vectors of B are closely aligned with
the coordinate axes so if your matrix in
some sense is on which is that which is
to say that your matrix is already sort
of close to diagonal so just
approximating it with its own diagonal
it won't be so egregious and if if B is
in fact this empirical Fischer which is
just defined as the sums of the outer
products of the gradients then it's
actually very easy to compute this
diagonal you don't have to use fancy
methods like curvature prop and
essentially it just amounts because this
if this is the sum of outer products
then the diagonal of this matrix is just
the sum of the entry wise products so as
long as you're competing ingredients
already you can kind of do this there
are some subtleties but it's to do with
something over batches but it's more or
less well understood
and this leads to set of several
actually natural quite popular methods
that are used all the time in neural
nets right now I'm rmsprop and Adam
which are built into a lot of libraries
although it's not clear how much these
actually help in practice this is sort
of a sticking point with me but ya know
so not something I've entry it's the
yeah I should have been clearer it's
it's like the I training case so it's
it's the ice in the same sense as it was
in this and in this slide yeah it's like
you know that yeah exactly
right so so that's diagonal methods
another type of approximation which was
sort of more popular classically and the
old optimization literature are low-rank
approximation z' so the idea here is
that we approximate our curvature matrix
B as as a as a diagonal plus rank our
matrix so in particular that it has a
form like this some of lower case our
outer products of these vectors plus
some day the the diagonal matrix formed
from some vector so this is relatively
easy to store um just our times and
relatively easy to apply the inverse and
there are standard methods for actually
computing this as an approximation to a
particular B although that B is almost
always the Hessian or some kind of weird
PSD approximation of the Hessian you get
into the methods like l-bfgs B if you
know and its predecessor BFGS and these
are sort of a they're approximating the
Hessian but but in fact they're always
positive definite so it's sort of a
weird approximation there's a whole
classical literature on this topic I
won't go into and this will be much less
effective than using the real curvature
matrix B if if you have many different
eigenvectors with large eigenvalues
where
if you have a matrix that has only a few
large eigenvalues then you know I would
approximate approximation like this
might actually capture that quite well
because basically just these terms
capture those eigen those few big ones
and then this just sort of cleans up
this provides you of a sort of a lower
bound on the rest but if you have sort
of a kind of a very gradual slope or
even you know a lot of eigenvalues
distributed you know across a sort of a
fat tail distribution then this probably
won't work and it seems like neural nets
fall into that category that they have
this very long tail distribution of
eigenvalues in their curvature matrix
which is sort of the bad case for all
sorts of reasons it's it's bad because
it actually makes all the bounds for
other optimizers worse etc etc and it's
bad because it makes this this l-bfgs
approximation potentially quite bad so
this is not typically used in practice
but it's still a very important method
historically and still an active
research topic then we can get into
block diagonal approximations so this is
where we take B to be some block
diagonal matrix which might just be the
blocks of the real B and so for neural
Nets you know you these blocks could
have some kind of semantics meaning they
could be for example each block would
correspond to the parameters for the
that that are the weights of the
connections on going into a specific
unit or the weights on the connections
going out of a specific unit or all the
weights in a particular layer so there's
different ways we could arrange things
partition essentially the the parameters
so the storage cost here is going to be
just so of B times n where B is the
number of blocks and the the cost of
applying this inverse which is
essentially amounts to inverting each
block because the inverse of a block
diagonal matrix is just a block diagonal
matrix consisting of
the those different blocks inverted so
you just invert each block is it's this
and the difficulty of actually computing
this is you could argue is quite similar
to compute in the diagonal and the
similar kinds of methods apply again I'm
working on a code base that you know
does these things that's computing the
certain block diagonal approximations
and all that code is in some sense
overlaps heavily with the diagonal
approximations as well turns out turns
out that you really from the standpoint
of computing matrices you don't actually
get it doesn't the problem doesn't get
much easier if you're only computing the
diagonal that's like you can actually
prove that and so now of course this
this idea can only really work if the
block size is relatively small because
now you're you're solving these you know
you're you're computing these inverses
on B by B matrices so that's B cubed and
you're doing this many times potentially
so that can get quite expensive and
unless B is quite small of course if B
is 1 then this reduces to a diagonal
method again which is then quite cheap
so a well-known example pretty much the
only you know classical example of this
is the so called paga method from some
decades ago although it's not used
anymore um another thing you could do
and this is my research is you could use
chroniker products approximations so the
idea here is that we're going to start
with a block diagonal approximation of
the GGN or fischer matrix where these
blocks are now going to correspond to
entire layers which which which are big
blocks these you know the number of
parameters in a layer can still be an
order of Millions
so you know we can't actually invert
those kinds of blocks directly but if we
do an additional approximation then we
can we can maybe get there and the
additional approximation that's used in
this and this chroniker product idea is
that we're gonna take each of each of
these blocks to be
actually written as a chronica product
between two smaller matrices and the
Kronecker product is defined like this
you just take C and you sort of tile it
over a much bigger matrix multiplying it
by the corresponding entry of a it seems
like a weird kind of thing to do but
actually it turns out this is a
algebraically very very natural thing to
do and it has all sorts of really cool
properties you can now of course you
could ask well okay but how am I
actually gonna take this giant block
matrix and this this giant block and
decompose it into a Kronecker product if
I don't even have the block to begin
with I can't even compute it because
it's too big it's like a million times a
million so there's actually a direct
formula that you can use that actually
tells you how to map certain quantities
that you compute inside of the network
such as the activations or the back
propagated errors and you can derive
formulas for these blocks or answer
these little factors of the block
directly but actually computing the
whole block and this is you know again
detailed in this these papers in mine so
the storage and computational cost of
this is just o of n roughly speaking
it's the number of parameters so that's
that's again that's about as good as you
can do applying the inverse is is like
this where B is the block size so we've
we've saved considerably on that just
for calling the you know naively it's
it's gonna be this is gonna be the cost
for blood for blocks of size B but with
this additional Kronecker factor
approximation you get it down to this
and this is using the sort of a
remarkable fact that if you want to
invert one of these chronica product
matrices it's just the same thing as
inverting each factor and then taking
the Kronecker product of that and this
in this cubes state-of-the-art results
for neural nets so the I'm now I'm gonna
get into the last topic which is
stochastic optimization so a lot of what
I've talked about so far when I mention
empirical results I'm always you should
always sort of interpret that as saying
well it's these classical methods for a
sort of deterministic optimization
applied with some stochastic city so
they're in in practice all optimization
for neural nets is stochastic I think
it's still very instructive to think
about the deterministic case because
that informs this this stochastic case
quite a bit in fact a lot of what you do
in in in in optimization research is
that you just start with a deterministic
technique and then you just kind of
estimate the various quantities that you
need using decayed averages using
mini-batches using various techniques
this all introduces all sorts of noise
into the system but the methods still
more or less work although they do break
down in interesting ways too so just
so-so the actual so what I mean by
stochastic optimization is just that the
you know assuming that your objective
function is written again like like as a
sum of little losses over different
training cases you know you have that
your gradient is now of course the sum
of gradients across the training cases
and if M is very big this is very
expensive so we're going to just you
know we're gonna we're gonna use an
approximation here which is you know
basically taking advantage of the fact
that these different ages aren't totally
independent and uninformative of each
other they're actually sort of you know
getting good at one cat classification
example sort of lets you get better at
the other one too well you know at least
up to a certain point once you start to
really care about fine details you know
the different whisker lengths you know
then you might have to then the
different cases might sort of become
independently informative but at the
beginning of optimization it's almost
always true that that there's a lot of
redundancy in the data you don't need to
really see all the data to make you know
to understand what the objective
function looks like
and and so stochastic optimization is
basically taking advantage of this idea
and the idea is okay instead of
computing this whole gradient over the
whole training set we're just gonna
subsample some random subset of the
cases in the training set and of size B
and B use an unfortunate
here because it's not the same bee from
before
she's mean it's batch size so you can
think of that way and and now our
approximate gradient is just this sort
of this average over the smaller subset
and this is an unbiased estimator of the
true gradient in a sense that the
expectation of this quantity is the true
gradient so you can think of it as
essentially just the true gradient plus
this corrupting influence of this
sampling procedure so in stochastic
gradient descent it's just sort of the
obvious as I as I alluded to before it's
sort of Li you do the obvious thing you
just take the deterministic method
gradient descent and you replace the
exact gradient with the one that we've
approximated using the mini batch and
you just get an update like this and
this does you know doing these kinds of
approximations will affect convergence
theory and also could also practical
convergence to once you introduce noise
you have to start to do things both in
theory and practice to to get
convergence and there are different
strategies for this so one classical
strategy which I would say is sort of
fallen out of favor is to use a decay in
step size although it's still used in
some examples so that the idea here is
that if if your step sizes that are
these alphas if the sum of the squares
of them is finite but the sum of you
know just the regular one is is infinite
and for example this is achieved by this
choice here 1 over K because this is a
harmonic series and this converges and
you get you can prove convergence and I
would say there there is an intuition
here for the for these relationships
basically this one is saying the amount
of distance I'm allowed to travel is is
never bounded so like if I am very far
away from my minimum my learning rates
will always some of them into the future
will always be big enough to get me
anywhere but the sum of the squares of
them will be finite which means
some sense that I'm eventually
controlling the variants because you can
imagine that the variance of the
estimate of the gradient is is amplified
by the learning rate that you multiply
it right because if you take a random
variable you multiply it by a constant
it's variance gets bigger by precisely
the square of that constant so so this
is in some sense saying that that some
of the future variances it gets under
control and and that's actually how they
prove it essentially so that's one
strategy it doesn't work that great in
practice especially if you are very
strict about following a prescription
like this because it tends to make the
learning rate go down too quickly too
fast and ultimately we don't actually
care about stochastic convergence that
that much so it's not super important
that we actually converge another
technique which seems to work much
better on in my in my experience is
polyak averaging the idea there is that
you just average over the history of
iterates and that's actually and then
you treat this average as your sort of
real solution and this essentially sort
of blurs out the noise that you get
between stochastic optimization where
your iterate is potentially jumping
around a lot because of this because of
these stochastic gradients that you're
using it seems kind of stupid and this
formula is kind of stupid because you're
averaging the say for example with your
initial theta and your initial theta is
like way off and you know Nowheresville
it's like whatever you happen to start
with which is probably terrible but of
course over time the effect of that
initial theta it gets smaller and
smaller because you're dividing by a
larger and larger number here still
though this is still bad and nobody
actually uses this formula even though
this is what you need to prove the the
theory in practice what you do is is
typically do a decayed exponential
average like this which is much better
at forgetting the past in some sense
there's also methods that reduce the
variance of the gradient of the
stochastic gradient using special
variance reduction techniques
these are SVR G and s AG they haven't
really taken hold in deep learning yet I
think and partly because we don't
actually care about getting very very
fine conversions this is going back to
the previous point about how we in in
you know optimization we're not really
in practical optimization you
essentially just want to continue
optimizing until your method stress to
your model starts to overfit and so you
don't so much care about getting this
sort of asymptotic convergence
performance as you sort of zero in on
the exact solution that almost never
comes up so second-order methods you
know you might also want to apply these
same the same ideas there and have a
stochastic second-order method or a
stochastic momentum method too so
there's they're you know there's some
problems here the the second-order
methods you need to compute this matrix
B of course or some estimate of this
matrix B some approximation of it and
you know computing B just on the current
mini-batch turns out to be not good
enough
there's the reasons for this or sort of
subtle but it's it was sort of a the
bane of my existence for a couple years
doing research on this topic essentially
if you if you if you don't if you
include a very small amount of data when
you compute B in some sense your B
becomes very low rank it has no
curvature in a lot of directions and it
is very very optimistic about where you
can go and it sort of doesn't model what
will happen to other training case
losses that are not in that mini-batch
and then you and the things just go very
badly from there and so a solution that
is basically always used in practice is
then to sort of maintain some estimate
of B that is a decayed average of these
B's computed on mini-batches over time
so we sort of we use so at each step you
know you're you
have your your be computer just on the
mini-batch and now I'm kind of saving
that and mixing that into this this
bigger and bigger average of past bees
and of course those past bees are for
other parameters in old settings of the
parameters and so they're sort of they
become increasingly obsolete as time
goes on you're sort of mixing in this
kind of garbage information into your
signal but the idea again with with
decayed averages is that that
information sort of decays exponentially
so it this is an effective strategy and
this is used pretty much every
stochastic second-order method uses this
idea right this is this staleness it's
the problem I'm talking about so
momentum is also another method you
might want to apply the stochastic
setting and it and it certainly does
actually quite well and it helps in
practice you do have to be more careful
about the learning rate and the decay
when you're in the stochastic setting
and also a common practice is to sort of
turn the momentum down or even off as
you get very close to the minimum again
though we don't typically get very close
to the minimum and neural nets so not
such a big issue there's a lot of a lot
of theorists really really hate you know
SGD and momentum they'll they'll say oh
it's actually doesn't help at all or it
helps for sort of silly reasons I would
take that with a grain of salt it
actually does help a lot and people use
it all the time and there's pretty good
theories about why it helps the basic
intuition that was just that the
stochastic optimization resembles
deterministic optimization for a large
earlier period of optimization
especially if your mini batches are big
enough so you can just sort of think
about it as whatever the deterministic
case does plus the sort of corrupting
influence which gets worse and worse as
time goes on so so this is in fact this
is the formalization of that intuition
then it's if you think of the stochastic
gradient it's just the regular gradient
plus this corrupting influence Epsilon
and you could even bottle this you could
model this say as a Gaussian although
and practice it's not a curse in but you
can in the limit of large numbers it
sort of becomes Gaussian and of course
you would note that the expectation of
this is just that it's just the gradient
as long as this says mean zero which it
would you can actually if you think
about it this way and then you think
about what happens to a if you if you
look at say a strongly convex quadratic
decision to contradict that has positive
curvature and you apply SGD or you apply
a kind of a basic stochastic
second-order method then the expected
value of your current iterate actually
behaves the same as the interet would in
the non stochastic algorithm so you're
sort of your base you're out your album
is basically sort of performing exactly
like the the deterministic version does
plus some ball of uncertainty around
that spot and this is this is a this is
actually true this is not just an
intuition this is actually true for
quadratic problems for non quadratic
problems then it becomes a kind of a
useful heuristic way of thinking about
the situation I would say so what is the
theory say exactly so one I would say
you know very important thing to
understand first of all is that the
theory says that there is no asymptotic
advantage to using second-order methods
or momentum methods over planes to cast
a gradient descent with this polyak
averaging remember polyak averaging was
is this technique here averaging the
iterates
across past iterations so there's no
asymptotic advantage that is to say that
the bounds asymptotically don't get
better and there's actually matching
lower bounds to show that that's the
best you can do in the worst case okay
so that would seem to say that well
maybe these methods aren't great but
again it's you know asymptotics are not
what they're cracked up to be
so just to be a little bit more specific
s
with this polyak averaging is
asymptotically optimal not only is it
just as good as these methods this as
good as any method ever could be among
any technique that tries to perform
statistical estimation of parameters
using any method gradient descent Oracle
methods that just get to do anything
they want use unlimited computation it
is it is asymptotically optimal in that
sense in the sense that as you see data
this is as good as you can possibly do
there's an intrinsic uncertainty in what
the actual parameters are until you've
seen enough data and SUV with
averaging is asymptotically doing as
good as you possibly can do in that
respect it is using the data in an a
maximally efficient way but pre
asymptotically that's where the
advantage of second-order methods in
momentum can still come in in the
stochastic case and that's actually what
we care about in practice we care about
what happens before this asymptotic
theory kicks in and you can show that a
bound like this if you actually expand
out this this big of O this Big O which
is actually hiding these higher-order
terms here those higher order terms
well they don't matter in the limit as K
goes to infinity they do matter pre
asymptotically and they're different for
SGD with polyak averaging versus a
second-order method or a momentum method
so those terms get better which means
that even the theory says that you will
that there is some advantage it's just
you won't see it as K goes to infinity
but K never goes to infinity and
practice so we're we're sort of we're
still okay and momentum is very widely
used second-order method is a bit less
so because they're more complicated and
expensive but they're also used
especially diagonal methods diagonal
second-order methods actually used quite
a bit so this is again one of these
situations where I would say the theory
if you take it literally if you take the
this asymptotically optimal phrase and
you sort of run with it too far you'll
you run into trouble which is why it's
always important to do experiments and I
think that's the end of the lecture now
these are some
useful texts that I've read over the
years in particular this is sort of the
classical view of optimization from
nocedal and wright looking at research
mostly from the 60s and 70s algorithm
building a lot of intuition and then
there's sort of the flip side of that
which is the pure theory you know best
represented by uri Nesterov the inventor
of the Nesterov method among other
things and I think you know reading both
of these sort of gives you a very good
sort of complementary view of the two
basic perspectives on optimization these
guys don't care so much about asymptotic
performance they're very much interested
in sort of practical performance and
misra of the other way around although
the both of them have useful insights
and yeah and there's some this is a
paper that I think talks does a good job
at explaining the importance of momentum
and deep learning especially and also
initialization to this was sort of an
early paper showing that sort of
exploring the importance of doing good
optimization for neural nets this is a
paper a paper mine recently posted an
archive well actually not that recently
any more looking at natural gradient
methods in their sort of connections to
second order methods and a lot of the
top stuff I've talked about in this
lecture so diff explained a much more
detail in that in that paper and that's
the end and I'll take any questions if
you got them
or if you want to come up after after
everybody starts to pack up you know
I'll be here for the next 10 minutes or
something all right
[Applause]


What's Left Before AGI? PaLM-E, 'GPT 4' and Multi-Modality

palm e was released less than a week ago
and for some people it may already be
old news sure it can understand and
manipulate language images and even the
physical world the e at the end of palm
e By the way stands for embodied but
soon apparently we're gonna get the
rebranded gbt4 which many people think
surely will do better and be publicly
accessible but the multi-modal
advancements released just this week
left me with a question what tasks are
left before we call a model artificial
general intelligence or AGI something
Beyond human intelligence I didn't want
hype or get rich schemes I just wanted
clear research about what exactly comes
before AGI let's start with this four
day old statement from anthropic a four
billion dollar startup founded by people
who left open AI over safety concerns
they outlined that in 2019 it seemed
possible that multi-modality like Army
logical reasoning speed of learning
transfer learning across tasks and
long-term memory might be walls that
would slow or halt the progress of AI in
the years since several of these walls
such as multi-modality and logical
reasoning have fallen what this means is
that the different modes of palm e and
Microsoft's new visual Chaturbate text
image video aren't just cool tricks they
are major Milestones palm e can look at
images and predict what will happen next
check out this robot who's about to fall
down that's just an image but ask palmy
what will the robot do next and it says
fall it knows what's going to happen
just from an image it can also read
faces and answer natural language
questions about them check out Kobe
Bryant over here it recognizes him from
an image and you can ask questions about
his career this example at the bottom I
think is especially impressive palm e is
actually doing the math from this
hastily sketched chalkboard it's solving
those classic math problems that we all
got at school just from an image now
think about this palm e is an
advancement on gato which at the time
the lead scientist at deepmind Nando de
Freitas called game over in the search
for AGI someone had written an article
fearing that we would never achieve AGI
and he said game over all we need now
are bigger models more compute
efficiency smarter memory more
modalities Etc and that was gato not
palmy of course you may have noticed
that neither he nor I am completely
defining AGI that's because there are
multiple definitions none of which
satisfy everyone but a broad one for our
purposes is that AGI is a model that is
at or above the human level on a
majority of economic tasks currently
done by humans you can read here some of
the tests about what might constitute
AGI that's enough about definitions and
multi-modality time to get to my central
question what is left before AGI well
what about learning and reasoning this
piece from Wired Magazine in late 2019
argued that robust machine reading was a
distant Prospect it gives a challenge of
a children's book that has a cute and
quite puzzling series of interactions it
then states that a good reading system
would be able to answer questions like
these and then give some natural
questions about the passage I will say
these questions do require a degree of
logic and Common Sense reasoning about
the world so you can guess what I did I
put them straight into Bing where only
three and a half years on from this
article and look what happened I pasted
in the exact questions from the article
and as you might have guessed Bing got
them all right pretty much instantly so
clearly my quest to find the tasks that
are left before AGI would have to
continue just quickly before we move on
from Bing and Microsoft products what
about specifically gpt4 how will it be
different from Bing or is it already
inside being as many people think the
much quoted German CTO of Microsoft
actually didn't confirm that gpt4 will
be multimodal only saying that at the
Microsoft events this week there we will
have multi-modal models that's different
from saying gpt4 will be multimodal I
have a video on the eight more certain
upgrades inside GT4 so do check that out
but even with those upgrades inside gbt4
the key question remains if such models
can already read so well what exactly is
left before AGI so I dove deep in the
literature and found this graph from the
original palm model which palm e is
based on look to the right these are a
bunch of tasks that the average human
rater at least those who work for Amazon
Mechanical Turk could beat palmat in
2022 and remember these were just the
average rators not the best the caption
doesn't specify what the tasks are so I
looked deep in the appendix and found
the list of tasks that humans did far
better on than Palm here is that
appendix and it doesn't make much sense
when you initially look at it so what I
did is I went into the big bench data
set and found each of these exact tasks
remember these are the tasks that the
average human raters do much better at
than Palm I wanted to know exactly what
they entailed looking at the names they
all seem a bit weird and you're going to
be surprised at what some of them are
take the first one mnist ASCII that's
actually representing and recognizing
ASCII numerals hmm now I can indeed
confirm that Bing is still pretty bad at
this in terms of numerals and in terms
of letters I'm just not sure how great
an accomplishment for Humanity this one
is though so I went to the next one
which was sequences as you can see below
this is keeping track of time in a
series of events this is an interesting
one perhaps linked to GPT models
struggles with mathematics and it's lack
of an internal calendar I tried the same
question multiple times with Bing and
Chaturbate and only once out of about a
dozen attempts did it get the question
right you can pause the video and try it
yourself but essentially it's only
between four and five that he could have
been at the swimming pool you can see
here the kind of convoluted logic that
Bing goes into so really interesting
this is a task that the models can't yet
do again I was expecting something a bit
more profound but let's move on to the
next one simple text editing of
characters words and sentences that was
strange what does it mean text editing
can't Bing do that I gave Bing many of
these text editing challenges and it did
indeed fail most of them it was able to
replace the letter t with the letter P
so it did okay with characters but it
really doesn't seem to know which word
in the sentence something is you can let
me know in the comments what you think
of these kind of errors and why Bing and
chat gbt keep making them the next task
that humans did much better on was
hyperbaton or intuitive adjective order
it's questions like which sentence has
the correct adjective order an
old-fashioned circular leather exercise
car sounds okay or a circular exercise
old-fashioned leather car what I found
interesting though is that even the
current version of chattybt could now
get this right on other tests it gets it
a little off but I think we might as
well tick this one off the list the
final task I wanted to focus on in Palm
appendix is a little more worrying it's
Triple H on the wrestler the need to be
helpful honest and harmless it's kind of
worrying that that's the thing it's
currently failing at I think this is
closely linked to hallucination and the
fact that we cannot fully control the
outputs of large language models at this
point if you've learned anything please
do let me know in the comments or leave
a like it really does encourage me to do
more such videos all of the papers and
pages in this video will be linked in
the description anyway hallucinations
brought me back to the anthropic safety
statement and their top priority of
mechanistic interpretability which is a
fancy way of saying understanding what
exactly is going on inside the machine
and one of these stated challenges is to
recognize whether a model is deceptively
aligned playing along with even tests
designed to tempt a system into
revealing its own misalignment this is
very much linked to the Triple H
failures we saw a moment ago fine so
honesty is still a big challenge but I
wanted to know what single significant
and quantifiable task AI was not close
to yet achieving some thought that that
task might be storing long-term memories
as it says here but I knew that that
Milestone had already been passed this
paper from January described augmenting
Palm with read write memory so that it
can remember everything and process
arbitrarily long inputs just imagine a
Bing chat equivalent knowing every email
at your company every customer records
sale invoice the minutes of every
meeting Etc the paper goes on to
describe a universal turing machine
which to the best of my understanding is
one can mimic any computation a
universal computer if you will indeed
the author's state in the conclusion of
this paper that the results show that
large language models are already
computationally Universal as they exist
currently provided only that they have
access to an unbounded external memory
what I found fascinating was that
anthropic are so concerned by this
accelerating progress that they don't
publish capabilities research because we
do not wish to advance the rate of AI
capabilities progress and I must say
that anthropic do know a thing or two
about language models having delayed the
public deployment of Claude which you
can see on screen until it was no longer
state of the art they had this model
earlier but delayed the deployment
Claude by the way is much better than
chattybt at writing jokes moving on to
data though in my video on gpt5 which I
do recommend you check out I talk about
how important data is to the Improvement
of models one graph I left out from that
video though suggests that there may be
some limits to this straight line
Improvement in the performance of models
what you're seeing on screen is a paper
are released in ancient times which is
to say two weeks ago on messa's new
llama model essentially it shows
performance improvements as more tokens
are added to the model by token think
scraped web text but notice how the
gains level off after a certain point so
not every graph you're going to see
today is exponential and interestingly
the y-axis is different for each task
and some of the questions it still
struggles with are interesting take
s-i-q-a which is social interaction
question answering it Peaks out about 50
to 52 percent that's questions like
these wherein most humans could easily
understand what's going on and find the
right answer models really struggle with
that even when they're given trillions
of tokens or what about natural
questions where the model is struggling
at about a third correct even Beyond 1.2
trillion tokens I dug deep into the
literature to find exactly who proposed
natural questions as a test and found
this document this is a paper published
by Google in 2019 and it gives lots of
examples of natural questions
essentially they're human-like questions
where it's not always clear exactly what
we're referring to now you could say
that's on us to be clearer with our
questions but let's see how Bing does
with some of these I asked the guy who
plays Mandalorian also did what drugs TV
show I deliberately phrased it in a very
natural vague way interestingly it gets
it wrong initially in the first sentence
but then gets it right for the second
sentence I tried dozens of these
questions you can see another one here
author of lotr surname origin that's a
very naturally phrased question it's
surmised that I meant Tolkien the author
of Lord of the Rings and I wanted the
origin of his surname and it gave it to
me another example was Big Ben City
first bomb landed WW2 it knew I meant
London and while it didn't give me the
first bomb that landed in London during
World War II it gave me a bomb that was
named Big Ben so not bad overall I found
it was about 50 50 just like the meta
llama model maybe a little better going
back to the graph we can see that data
does help a lot but it isn't everything
however anthropic's theory is that
compute can be a rough proxy for further
progress and this was a somewhat
eye-opening passage we know that the
capability jump from gpt2 to gpt3
resulted mostly from about a 250 time
increase in compute we would guess that
another 50 times increase separates the
original gpt3 model and state-of-the-art
models in 2023 think Claude or Bing over
the next five years we might expect
around a 1 000 time increase in the
computation used to train the largest
models based on Trends in compute cost
and spending if the scaling laws hold
this would result in a capability jump
that is significantly larger than the
jump from gbc2 to gpt3 or gbt3 to Claude
and ends with anthropic we're deeply
familiar with the capabilities of these
systems and a jump that is this much
larger feels to many of us like it could
result in human level performance across
most tasks that's AGI and five years is
not a long timeline this made me think
of Sam Altman's AGI statement where he
said at some point it may be important
to get independent review before
starting to train future systems and for
the most advanced efforts to agree to
limit the rate of growth of compute used
for creating new models like a compute
truce if you will even Sam Altman thinks
we might need to slow down a bit my
question is though would Microsoft or
Tesla or Amazon agree with this truth
and go along with it maybe maybe not but
remember that five-year timeline the
anthropic laid out that chimes with this
assessment from the conjecture alignment
startup AGI is happening soon
significant probability of it happening
in less than five years and it gives
plenty of examples many of which I have
already covered others of course give
much more distant timelines and as we've
seen AGI is not a well-defined concept
in fact it's so not well defined that
some people actually argue that it's
already here this article for example
says 2022 was the year AGI arrived just
don't call it that this graph originally
from wait but why is quite funny but it
points to how short a gap they might be
between being better than the average
human and being better than Einstein I
don't necessarily agree with this but it
does remind me of another graph I saw
recently it was this one on the number
of academic papers being published on
machine learning and AI in a paper about
exponential knowledge growth the link to
this paper like all the others is in the
description and it does point to how
hard it will be for me and others just
to keep up with the latest papers on AI
advancements at this point you may have
noticed that I haven't given a
definitive answer to my original
question which was to find the task that
is left before AGI I do think there will
be tasks such as physically Plumbing a
house that even an AGI a generally
intelligent entity couldn't immediately
accomplish simply because it doesn't
have the tools it might be smarter than
a human but can't use a hammer but my
other Theory to end on is that before
AGI there will be a d deeper more
subjective debate take the benchmarks on
reading comprehension this graph shows
how Improvement is being made but I have
aced most reading comprehension tests
such as the GRE so why is the highest
human rater labeled at 80 could it be
that progress stores when we get to the
outer edge of ability when test examples
of sufficient quality get so rare in the
data set that language models simply
cannot perform well on them take this
difficult LSAT example I won't read it
out because by definition it's quite
long and convoluted and yes Bing fails
it is this the near-term future where
only obscure Feats of logic deeply
subjective analyzes of difficult texts
and Niche areas of mathematics and
science remain Out Of Reach where
essentially most people perceive AGI to
have already occurred but for a few
outlier tests indeed is the ultimate
capture test the ability to deliver a
laugh out loud joke or deeply understand
the plight of Oliver Twist anyway thank
you for watching to the end of the video
I'm going to leave you with some
bleeding edge text to image Generations
from mid Journey version 5. whatever
happens next with large language models
this is the new story of the century in
my opinion and I do look forward to
covering it but as companies like
Microsoft open Ai and Google seem set to
make enough money to break capitalism
itself I do recommend reading anthropic
statement and their research on
optimistic intermediate and pessimistic
scenarios they also have some persuasive
suggestions on rewarding models based on
good process rather than simply quick
and expedient outcomes check it out and
have a wonderful day