DeepMind x UCL | Deep Learning Lectures | 2/12 |  Neural Networks Foundations

all right thank you for this lovely
introduction and
I mentioned today we'll go
the foundations of neural networks the
talk itself will last around 90 minutes
so I would ask you to wait with any
questions till the end of the lecture
we'll have a separate slot to address
this and I'll also hang around for a
while after the lecture if you would
prefer to ask some in person lecture
itself will be structured as follows
there'll be six sections I will start
with a basic overview trying to convince
you that there is a point in learning
about neural nets what is the actual
motivation to be studying this specific
branch of research in the second part
which is the main meat of the story
we'll go through both history of neural
nets their properties the way they are
defined and their inner workings mostly
in order to gather good and deep
intuition about them so that you are
both prepared for future more technical
more in-depth lectures in this series as
well as ready to simply work with these
models in your own research or work
equipped with these we'll be able to
dive into learning itself since even the
best model is useless without actually
knowing how to set all the knobs and
weights inside them after that we'll
fill in a few gaps in terms of these
pieces of the puzzles of the puzzle that
will be building as the talk progresses
and we'll finish with some practical
issues or practical guidelines to
actually deal with most common problems
in training neural nets if time permits
we'll also have a bonus slide or two on
what I like to call multiplicative
interactions so this is what will be in
the lecture there quite a few things
that could be included in a lecture
called foundations of our neural Nets
that's not going to be part of this talk
and I'd like to split these into three
branches first is what I refer to us
all-school neural Nets not to suggest
that the neural nets that are working
with these days are not old or they
don't go back like 70 years but rather
to
make sure that you see that this is a
really wide field with quite powerful
methods that were once really common and
important for the field that are not
that popular anymore but it's still
possible that they will come back and
it's valuable to learn about things like
restricted Boltzmann machines deep
belief networks hopfield net kohonen
maps so I'm just gonna leave these a
sort of keywords for further living if
you want to really dive deep the second
part is biologic biologically possible
neural nets where the path really is to
replicate the inner workings of human
brain so have physical simulators that
are spiking neural nets and these two
branches encoded in red here will not
share that many common ground with the
talk right there are still in your own
network land but they don't necessarily
follow the same the same design
principles on the other hand the third
branch called other because of lack of
better name do share a lot of
similarities and even though we won't
explicitly talk about capsule networks
graph networks or neuro differential
equations what you'll learn today the
high level ideas motivations and overall
scheme directly applies to all of these
they simply are somewhat beyond the
scope of the series and the ones in
green like convolutional neural networks
recurrent neural networks are simply not
part of this lecture but will come in
weeks to come for example in sundar talk
and others yep so why are we learning
about neural Nets quite a few examples
were already given a week ago I just
want to stress a few the first one being
computer vision in general most of the
modern solutions applications in
computer vision do use some form of
neural network based processing these
are not just hypothetical objects you
know things that are great for
mathematical analysis or for research
purposes
there are many actual commercial
applications products that use neural
networks on daily basis pretty much in
every smartphone you can find at least
one neural net these days the second one
is natural language processing testing
text synthesis
of grades recent results from open AI
and their GPT to model as well as
commercial results with building wavenet
based text generation into a Google
assistant if you if you own one finally
control that doesn't just allow us to
create AI for things like gold chess
Starcraft for games or simulations in
general but it's actually being used in
products like self-driving cars so what
made all this possible what started the
deep learning revolution what are the
fundamentals that neural networks that's
really benefited from this needs to have
the first one is compute and I wants to
make sure that you understand that there
are two sides to this story it's not
just that computers got faster they were
always getting faster what specifically
happened in recent years is that
specific kind of hardware compute namely
GPUs graphical processing units that
will designed for games got really
useful for machine learning right so
this is two first thing on one hand we
got hardware there's just much faster
but it's not generally faster it won't
make your sequential programs run faster
it is faster with respect to very
specific operations and neural networks
happen to use exactly these operations
we will reinforce this point in further
part of the lecture but think about
matrix multiplications as this core
element other machine learning
techniques that don't rely on matrix
multiplications would not benefit that
much from this exponential growth in
compute that came from GPUs and these
days from TVs the second is data and the
same argument applies you have various
methods of learning some of which scale
very well with data some scale badly
your competition complexity goes through
the roof you don't benefit from pushing
more and more data so we have again two
phases to this story a there's much more
data available just because of Internet
in terms of things and various other
things and on the other end you have
models that really our data hungry and
actually improve as amount of data
increases and finally
and finally the modularity of the system
itself the fact that deep learning is
not that well defined field of study
it's more of a mental picture of the
high-level idea of these modular blocks
that can be arranged in various ways and
I want to try to sell to you intuition
of viewing deep learning as this sort of
puzzle where all we are doing as
researchers is building these small
blocks that can be interconnected in
various ways so that they jointly
process data to use quotes from recent
turing award winner professor yan
laocoön deep learning is constructing
networks of harmonized functional
modules and training them from examples
using gradient based optimization
there's this core idea that we're
working with is an extremely modular
system we are not just defining one
model that we are trying to you know
apply to various domains were redefining
language to build them that relies on
very simple basic principles and these
basic principles in a single such a node
or piece of a puzzle is really these two
properties each of them needs to know
given an input given data what's to
output there are simple computational
units that compute one thing they take
an average they multiply things they
exponent them things like this and they
have also the other mode of operations
if they knew how the output of their
computation should change they should be
able to tell their input how to change
accordingly right so if I tell this node
your output should be higher it should
know how to change inputs if you are a
more mathematically oriented you'll
quickly find the analogy to
differentiation right then this is
pretty much the underlying mathematical
assumption for this so we will usually
work with the differentiable object and
this is also how professor Iyanla
quantifies this this is not necessarily
a strict requirement that you will see
through this lecture that in practice
people will put things that are kind of
differentiable and
a practitioner you know that people just
put everything into deep nets not
necessarily caring about the full
mathematical reasoning behind it
so given this really high-level view let
us go for some fundamentals and as usual
let's start with some biological
intuition so in every your network
lecture you need to see a neuron so I
drew one for you that's I know over
simplified so if you have biological
background forgive me for being very
naive but I wanted to capture just very
basic properties and so people have been
studying in neurobiology how real
neurons look like and one really
high-level view is that these are just
small cells that have multiple been
trends which are inputs from other
neurons through which they accumulate
their spikes their activity there is
some simple computation in the Sun in
the cell body and then there is a single
axon where the output is being produced
human brain is composed of billions of
these things connected to many many
others and you can see that this kind of
looks like a complex distributed
computation system right we have these
neurons connected to many others each of
them represents a very simple
computation on its own and what people
notice is that some of this connection
inhibits your activity so if other
neuron is active you aren't some excite
so if they are active you are excited as
well and of course that many other
properties like there's a stage right
these cells live through time the output
spikes for time each time you'll see a
slide like this with yellow box this is
a reference to further reading I won't
go into too much depth on various topics
but if you want to read more these are
nice preferences for example watch game
hodgkin-huxley model is a nice read if
you are somewhere between neurobiology
and mathematics so this is an intuition
and just intuition what people did with
that and by people I mean McCulloch and
Pitt's is to look at this
and ask themselves what seems to be the
main set of neurophysiological
observations that we need to replicate
it's important to stress that this is
not a model the model that they proposed
that was trying to replicate all of the
dynamics this is not you know an
artificial simulation of a neuron is
just something vaguely inspired by some
properties of real neurons and these
properties are selected in green there
are things that will be easy to compose
because you have Google for this in a
second
you have blue inputs that are multiplied
by some weights W there are just real
numbers attached to each input and then
they are some so it's like an weighted
sum of your inputs and we also have a
parameter weight B also referred to a
bias to us bias which is the output of
our new you can see that this is
something you could easily compose there
are real numbers as inputs real numbers
as outputs it represents simple
computation
well it's literally weighted average you
can get more basic than that may be
slightly more basic it also has this
property of inhibiting or exciting if W
is negative you inhibit if it's positive
you excite but they left out quite a few
properties for example this is a
stateless model right you can compute it
many many times and the output is
exactly the same if you were to take a
real neuron and put the same action
potentials in it's spiking might change
through time because it is a living
thing that has a state physical state
and also outputs real values rather than
spikes through time just because the
time dimension got completely removed
and also to set some notation each time
something is blue in equations this
means this is an input if something is
red this is a parameter weight something
that you would usually train in your in
your model and the same applies to the
schemes themselves so what it means as
intuitively what's the weighted average
does at least my personal favourites
intuition is that it makes it defines a
linear or affine project
of your data so you can imagine that
this horizontal line is one such neuron
we've W that is just zero one and B
equals zero and then what will happen is
all the data that you have so your axis
would be perpendicularly projected onto
this line and you'd get this mess
everything would be on top of each other
if you had a different W say the sorry
vertical line before it was horizontal
the vertical line then they would be
nicely separated groups right because
you just collapsed them if it was a
diagonal line then things would be
slightly slightly separated as at the
bottom part of this so when we define
something like this we can start
composing them and the most natural or
the first mode of composition is to make
a layer out of these neurons so you can
see the idea is just to take each such
neuron put them next to each other and
what we gain from this mostly we gain a
lot of efficiency in terms of computing
because now the equation simplifies to a
simple affine transformation with W
being a matrix of the weights that are
in between our inputs and outputs X
being a vectorized input right so just
gather all the inputs for that as a
vector why is it important to fold
multiplication of two matrices in a
naive fashion is cubic but you probably
know from algorithmic so either 1 1 or 2
or whatever that you can go down like
2.7 by being slightly smart about how
you multiply things by basically using a
divide-and-conquer kind of methods and
furthermore this is something that fits
the GPU paradigm extremely well right so
this is one of these things that just
matches exactly what was already there
hardware wise and as such could benefit
from this huge boost in compute there's
also a lot of small caveats in the
neural network planned in terms of
naming conventions so each object will
have from one to five names and I'm
deeply sorry for us as a community for
doing this the main reason is many of
these things were independently
developed by various groups of
researchers and
unification never happened some of these
names are more common than others for
example this is usually called linear
layer even though mathematician would
probably cry and say no it's fine it's
not linear there's a bias this doesn't
satisfy seniority constraints neurons
will be often called units so if I say
unit or neuron I just use these
interchangeably and parameters and
weights are also the same object so you
might ask isn't this just linear
regression like equation looks exactly
like statistics around the world as in
your regression model and to some extent
yes you're right it is exactly the same
predictive model but what's important is
to have in mind our big picture yes we
start small but our end goal is to
produce these highly composable
functions and if you are helping with
composing many linear regression models
on top of each other especially
multinomial regression models then you
can view it like this the language that
neural network our community prefers
is to think about these as neurons or
collections of neurons that talk to each
other because this is our end goal yes
this third beginning of our puzzle could
be something that's known in literature
under different names but what's really
important is that we view them as this
single composable pieces that can be
arranged in any way and much of research
is about composing them in a smart way
so that you get a new quality out of it
but let's view these simple models are
senor networks first so we'll start with
a single layer neural network just so we
can gradually see what is being brought
to the table with each extension with
each added module so what we defined
right now is what we're gonna define
right now can be expressed more or less
like this we have data it will go
through the linear module then there
will be some extra notes that we are
going to be fine
then there are gonna then there is gonna
be a loss
there's also be gonna be connected to a
target we are missing these two so let's
define what can be used there and let's
start with the first one which is often
called an activation function or a
non-linearity this is
an object that is usually used to induce
more complex models if you had many
linear models many affine models and you
compose them it's very easy to prove
composition of linear Zitz linear
composition of affine thingses f-fine
you would not really bring anything to
the table you need to add something that
bends the space in a more funky way so
one way of doing this or historically
one of the first ones is to use sigmoid
activation function which you can view
as a squashing of a real line to the
zero one interval we often will refer to
things like this as producing
probability estimates or probability
distributions and while there exists a
probabilistic interpretation of this
sort of model what this usually means in
ml community is that it simply outputs
things between 0 & 1 or the day sum to 1
okay so now let's not be too strict when
we say probability estimate here it
might mean something as simple as being
in the correct range the nice thing is
it also has very simple derivatives just
to refer to this different ability that
we're talking about here but they're
also caveats that make it slightly less
useful as you will see in the grand
scheme of things one is that because it
saturates right as you go to plus or
minus infinity it approaches 1 or 0
respectively this means that the partial
derivatives vanish right the gradient
far far to the right will be pretty much
zero because your function is flat so
the gradient is pretty much zero the
same applies in minus infinity so once
you are in this specific point if you
view gradient magnitude as amount of
information that you are betting to
adjust your model then functions like
this won't work that well once you
saturate you won't be taught how to
adjust your weights anymore but this was
we are gonna use at least initially so
we plug in sigmoid on top of our linear
model and the only thing we are missing
is a loss and the most commonly used one
for the simplest possible task which is
going to be binary classification
meaning that our targets are either 0 or
1 something is either false or true
something is a face or not something is
a dog or not just this sort of products
then the most common loss function which
should be a two argument function that
returns a scalar so it accepts in this
notation P our prediction T our target
and it's supposed to output a single
scalar a real value such that smaller
loss means better model being closer to
the correct prediction and cross-entropy
which has at least three names being
negative log likelihood logistic loss
and probably many others
gives us the negation of the location of
probability of correct classification
which is exactly what you care about in
classification at least usually it's
also nicely composable with the sigmoid
function which will go back towards the
end of the lecture showing how this
specific composition removes two
numerical instabilities at once because
on its own unfortunately it is quite
numerically unstable so given these
three things we can compose them and
have the simplest possible neural
classifier we have data it goes through
linear model goes for sigmoid goes for
cross-entropy
attaches targets this is what you would
know from statistics as a logistic
regression and again the fact that we
were defining and well-known model from
a different branch of science is fine
because we won't stop here this is just
to gain intuition what we can already
achieve what we can already achieve in
practice is we can separate data that's
labeled with well two possible labelings
true or false
zero and one as long as you can put a
line or a hyperplane in a hyper in a
higher dimension that completely
separates these two datasets so in the
example you see red and blue you see
that the more vertical line can separate
these data says pretty perfectly and it
will have a very low loss very low cross
entropy loss the important property of
the specific loss and I would say 95% of
all the losses in machine learning is
that they are additive with respect to
sample
so the loss that you can see at the
lower and decomposes additively over
summer so there is a small function l
that we just defined over its sample and
now T with I in the superscript is an
I've target can be expressed as a sum of
these this specific property relates to
the data aspect of deep learning
revolution losses that have this form
undergo very specific decomposition and
can be trained with what is going to be
introduced a stochastic gradient descent
and can simply scale very well with big
datasets and unfortunately as we just
discussed this is still slightly
numerically unstable so what happens
when we have more than one sorry more
than two classes then we usually define
what's called the softmax which is as a
name suggests a smooth version of the
maximum operation you take an exponent
of your input and just normalize divide
by the sum of exponents you can see this
was sum to one everything is
non-negative because well exponents by
definition I know not negative so we
produce probability estimates in the
sense that the output lies on the
simplex and it can be seen as a strict
multi-dimensional generalization of the
sigmoid so it's not a different thing is
just as three generalization if you take
a single X at 0 and compute the softmax
of it then the first argument of the
output will be Sigma within the second
one minus Sigma all right so it's simply
a way to go beyond two classes but have
very very similar mathematical
formulation and it's by far the most
commonly used final activation in
classification problems when number of
classes is bigger than than two it still
has the same issues for obvious reasons
generalizations so it cannot remove
issues but the nice thing is now we can
just substitute the piece of the puzzle
that we defined before right away the
sigmoid
now just put soft marks in its place and
exactly the same reasoning and mcann
that would work before apply now right
so we use exactly the same loss function
after the fact that it's summing over
all the pluses and now we can separate
still linearly of course more than two
colors sales class zero one and two
which is equivalent to multinomial
logistic regression if you went for some
statistical courses and the combination
of the softmax and the cross entropy as
I mentioned before becomes numerically
stable because of this specific
decomposition and there will be also a
more in-depth version towards the end of
this lecture the only thing that it
doesn't do very well is it doesn't scale
that we'll have number of classes all
right so one thing that you might want
is to be able to select one class
specifically just say one just say zero
and of course with equation like soft
max has you can't represent ones or
zeros you can get arbitrarily close but
never exactly one or zero and there are
nice other solutions to this like sparse
max module for example and also it
doesn't scale that well with K it will
work well if K number of classes is say
in hundreds if it's in hundreds of
thousands you might need to look for
some slightly different piece of the
puzzle the nice news is you can
literally just swap them and they will
start scaling up so why are we even
talking about these simple things so
apart from the fact that they become
pieces of the bigger puzzle it's also
because they just work and you might be
surprised that the linear models are
useful but they really are if you look
at this very well-known M this dataset
of just handwritten digits and try to
build a linear model that classifies
which digit it is based on pixels you
might get slightly surprising result of
somewhat around 92 percent of test
accuracy that's pretty good for
something that just takes you know
pixels and computes a weighted average
and that's all it does and in one of the
intuitions behind it is we usually keep
thinking about these models in like 1d
2d 3d and yes in 2d they are not that
many positions of objects that the line
cancer
it in 3d know that many positions were
hyperplane and separate in 100,000
dimensions 99 hyperplanes of
corresponding size can actually shutter
a lot of possible labelings so as you
get higher dimensionality you can
actually deal with them pretty well even
within your models furthermore in
commercial applications a big chunk of
them actually use linear models in
natural language processing for many
years the most successful model was
nothing else but Max and maximum entropy
classifier which is a fourth name for
logistic regression so why don't we stop
here right we could stop the lecture
here but obviously we are interested in
something slightly more complex like AI
for chess or forego and for this we know
that the linear model I mean we know
empirically linear models are just not
powerful enough but before we go that
far ahead maybe let's focus on something
that's the simplest thing that linear
models cannot do and it's going to be a
very well-known expo problem where we
have two dimensional data sets and on
the diagonal one class on the other
diagonal the second class you can
quickly iterate in your head over all
possible lines right not a single line
has red dots on one side blue on the
other elbow we need something more
powerful so our solution is going to be
to introduce a hitter later so now we're
going to look into two layer neural
networks that in our puzzle view look
like this we have a theta goes to linear
go through sigmoid goes for another
linear goes to soft max cross-entropy
target as you can see we already have
all the pieces we just well we are just
connecting them differently that's all
we are doing and I want to now convince
you that we're adding qualitatively more
than just you know adding dimensions or
something like this so let's start with
the potential solution how can we solve
this if we had just two hidden neurons
and a sigmoid activation function so we
have our data set and for simplicity of
visualization I'm going to recolor them
so that we have four different colors
we have blue red green and pink just so
you see where the projections end up
just remember that we want to separate
one diagonal from the other and that two
hidden neurons are going to be these two
projection lines so the top one is
oriented downwards which means that
we're going to be projecting in such a
way that the blue class will end up on
the right hand side pink on the left
green and red in the middle so somehow I
miss order these two sides so this is
how it's going to look like if you look
at the right hand side you have a
projection of this top line right blue
on the right because everything is
flipped sorry I should have grabbed I
guess ping on the left green and red
compost on top of each other the second
line is pretty symmetrically oriented
and there you can see blue data set or
blue blob projected on the left hand
side pink projected on the right and
green and red again superimposed on each
other right this is all we did through
two lines and just projected everything
onto them these are the weights and
biases at the bottom that would
corresponds to this projection now we
add sigmoid all that Sigma it does is it
squashes right instead of being a
identifing it nonlinear discourses so we
squash these two plots on the sides and
recompose them as a two dimensional
object right we have now on x-axis the
sum x axis we have the first projection
just for sigmoid and this is why it
became extreme the blue things ended up
being very sickly in in one and
everything else went to zero maybe
slightly boomerang G here and the second
neuron this projection after squashing
for Sigma it became y-axis you can see
now pink one got separated everything
else got boomerang ly squashed the nice
thing about this maybe doesn't look that
nice but what it allows us to do is now
draw a line that's going to separate all
the blue and pink things from
everything else and this was our goal
right so if I now project on this line
or equivalently if I were to put the
decision boundary here it would separate
exactly what I wanted right so the Blues
and things were supposed to be one class
and the remaining two colors were
supposed to be the other so I can just
project them put the boundary and if you
now look into the input space we ended
up with this discontinuous
classification or the chasm of sorts in
the middle we came one class and a
reminder became the other right just
going through the internals layer by
layer how the neural network with a
single hidden layer would operate all it
really did was to use this hidden layer
to rotate and then slightly bent the
input space right of the signal you can
think about this as kind of bending or
squishing which as a topological
transformation allows the purely linear
model on top of it to solve the original
problem right so it prepared pre-process
the data such that it became linearly
separable and you just needed two hidden
neurons to do this even though the
problem was not that complex it is a
qualitative change in what we are in
what we can do so what if something is
slightly more complex let's imagine we
want to separate a circle from a
doughnut then tuners won't be enough you
can prove it's not enough then that are
just too complex but six neurons are
doing just fine and at this point I
would like to advertise to you this
great tool by Daniel's Milk of and
others called playgrounds under
playgrounds don't test your folder org
or you can just play with this sort of
simple classification problems you can
pick one of the datasets you can add
hidden layers add neurons at the top you
can select activation function to be
sigmoid to follow what we just talked
about and if you select classification
it will just attach the sigmoid path
cross-entropy as the loss if he'd run
and you get the solution which separates
our data quite nicely
see the loss going down as expected and
arguably this is the easiest and most
important way of learning about you know
that's playing with them actually
interact with them it's really hard to
gain intuitions by just studying their
mathematical properties unless you are a
person with really great imagination I
personally need to fly with stuff to
understand so I'm just trying to share
this sort of lesson that I learned so
what makes it possible for neural nets
to learn arbitrary shapes
I mean arguably donut is not that
complex of a shape but believe me if I
were to draw a dragon
it would also do just fine and the
brilliant result arguably the most
important theoretical results in
neonates is world of C Benko from late
80s where he proved that neural networks
are what he called universe approximator
'he's using slightly more technical
language what it actually means is if
you get if you take any continuous
function from a hypercube right so your
inputs are in between 0 and 1 and have D
dimensions and your function is
continuous so relatively smooth and
output a single scalar value a single
number then there exists neural network
with one hidden layer with sigmoids that
will get an epsilon error at most
epsilon error and this is true for every
positive Epsilon
so you pick an error say one in minus 20
there will exist and you on that satisfy
this constraint you can pick one in
minus 100 and they will exist one that
satisfies it so one could ask what if I
pick epsilon equals zero then answer is
no it can only approximate it cannot
represent so you won't ever be able to
represent most of the continuous
functions but you can get really close
to them at the cost of using potentially
huge exponentially growing models with
respect to input dimensions it shows
that neural networks are really
extremely expressive they can do a lot
of stuff what it doesn't tell us though
is how on earth would we learn them it's
an existential proof right if you
and through proper mathematical
mathematical training you know that
there are two types of proofs right
there either constructive or the
essential arguably reconstructive ones
are more interesting they provide you
with insights how to solve problems the
essential ones are these tricky funky
thing we'll just say it's impossible for
this to be false and this is this kind
of proof that subhankar provided you
just show you just showed that there is
no way for this not to hold there was no
constructive methods of finding weights
of the specific network in this prove
that he right since then we actually had
more constructive versions furthermore
the size can grow exponentially what
subhankar attributed this brilliant
property to was the sigmoid function
that this quashing this smooth beautiful
squashing is what gives you this
generality it wasn't long since
hardening show that actually what
matters is this more of a neural network
structure but you don't need sigmoid
activation function you can actually get
pretty much take pretty much anything as
long as it's not degenerate and what's
he meant by non degenerate is that it's
not constant bounded and continues all
right so you can take a sine wave you
can pretty much get any squiggle as long
as you squiggle at least a bit so things
are non constant and they are bounded so
they cannot go to infinities so it shows
that this extreme potential of
representing or approximating functions
relies on these F and transformations
being stacked on top of each other with
some notion of non-linearity in between
them still without telling you how to
train them is just as in principle the
annual networks that are doing all this
stuff we just don't know how to find
them so to give you some intuition and
to be precise this is going to be an
intuition behind the property not behind
the proof the true proof relies on
showing the displace define been around
that world is a dense set in these set
of continuous functions instead we are
gonna rely on intuition why
approximating functions with sigmoid
based networks should be possible by
proof by picture so let's imagine that
we have this sort of mountain ridge
that's our target function and to our
disposal is only our sigmoid activation
and
they're so of course I can represent
function like this right I'll just take
a positive W and then negative B so it
shifts a bit to the right it doesn't
matter that much because I'm gonna also
get the symmetrical one where W is
negative and B is positive right so I
have two sigmoids
then if we take an average it should
look like a bump and you probably see
where I'm going with this it's gonna
rely on a very similar argument to how
integration works I just want to have
enough bumps so that after adding them
they will correspond to the target
function of interest so let's take three
of them and they just differ in terms of
biases that I've chosen so I'm using six
hidden neurons right two for each bump
and now in the layer that follows the
final classification layer now we
regression layer I'm just gonna mix them
first wait half second one third one and
a half and after adding these three
bumps with weights I end up with the
approximation of the original shape of
course it's not perfect as we just
learned we are never going to be able to
represent functions exactly with
sigmoids but we can get really close
right then this really close the epsilon
is what's missing here I only used 60 10
euros got some error if you want to
squash the error further you just keep
adding bumps now I need a bump here to
resolve this issue
I need a tiny bump somewhere around here
I need a tiny bump here and you just
keep adding and adding and eventually
you'll get as close as you want you
won't ever get it exactly right what is
it gonna go in the right direction so
you can ask okay it's 1d usually things
in one they are just completely
different story then k dimensional case
is there an equivalent construction at
least for 2d and the answer is positive
and you've seen this seven ish slides
before it's this one when we saw a donut
it is nothing but bump in 2d right if
you think about the blue class as a
positive one the one that it's supposed
to get one as the output this is
essentially a 2d bump its saw the
perfect Gaussian right
could do a better job but even with this
sort of bumps we could compose enough of
them to represent any 2d function and
you can see how things starts to grow
exponentially alright we just needed two
neurons to represent bump in 1d now we
need six for 2d and you can imagine that
for KD is gonna be horrible but in
principle possible and this is what
drives this sort of universe
approximation theorem building blocks so
let's finally go deeper since we said
that things are in principle possible in
a shallow land there needs to be
something qualitatively different about
going deeper versus going wider so the
kind of models we are going to be
working with will look more or less like
this there's data those through linear
some node linear nodes in your no linear
node and eventually a loss attached to
our targets what we are missing here is
what is going to be this special node in
between that as advertised before it's
not going to be a sigmoid and the answer
to this is the value unit rectifier
rectified linear units again quite a few
names but essentially what it is is a
point wise maximum between axis between
inputs and a0 all it does is checks
whether the input signal is positive if
so it acts as an identity otherwise it
just flattens it sets it to zero and
that's all why is it interesting well
from say a practical perspective because
it is the most commonly used activation
these days that just works across the
board in a wide variety of practical
application starting from computer
vision and even reinforcement learning
it still introduced and only it still
introduces nonlinear behavior like no
one can claim that this is a linear
function right with the hinge but at the
same time it's kind of linear in the
sense that it's piecewise linear so all
it can do if you were to use it may be
on different layers is to cut your input
space into polyhedra so with the linear
transformations it could cut it into
multiple ones and in each sub subspace
such part it can define an affine
transformation right because they're
just two possibilities and either
identity I'm just cutting you off so in
each of these pieces you have a
hyperplane and in each piece might be a
different hyperplane but the overall
function is really piecewise linear in
1d it would be just a composition of
lines in 2d of planes that are you know
just changing their angles and in KD
well K minus 1 dimensional hyper planes
that are oriented in a funky way the
nice thing is derivatives no longer
vanish there either one when you're in
the positive line our zero otherwise
I mean arguably this was already
vanished before we started the bad thing
is the data neurons can no cure so
imagine that you're all your activities
are negative then going through such
neuron will just be a function
constantly equal to zero which is
completely useless so you need to pay
maybe more attention to the way you
initialize your model and maybe one
extra thing to keep track of to just see
how many dead units you have because it
might be a nice debugging signal if you
did something wrong and also technically
the structure is not differentiable at 0
and the reason why people usually don't
occur is that from probablistic
perspective this is a zero measure set
you will never actually hit zero you
could hand waves and say well the
underlying mathematical model is
actually smooth around zero I just never
hit it so I never care if he wants to
pursue more politically grounded
analysis you can just substitute it with
a smooth version which is logarithm 1
plus minus X this is the dotted line
here that has the same limiting
behaviors but is fully smooth around
zero and you can also just use slightly
different reasoning when you don't talk
about gradients but different objects
we've seen are properties that are just
fine
with single points of non
differentiability so we can now stack
these things together and we have our
typical deep learning model that you
would see in every book on deep learning
linear array linearly and the intuition
behind depths the people had from the
very beginning especially in terms of
computer vision was that each
year we'll be some sort of more and more
abstract feature extraction module so
let's imagine that these are pixels that
come as D as the input then you can
imagine that the first layer will detect
some sort of lines and corners and this
is this is what the what each of the
neurons will represent whether there is
a specific line like horizontal line or
vertical wires of magnets once you have
this sort of representation the next
layer could compose these and represent
shapes like squiggles or something
slightly more complex why do you have
these shapes the next layer could
compose them and represent things like
ears and noses and things like this and
then once you have this sort of
representation maybe you can tell
whether it's a dog or not maybe some
number of years of or existence of ears
in the first place but this is a very
high level intuition and awhile
confirmed in practice this is
necessarily that visible from the mouth
and a really nice result from sorry I
cannot pronounce French but Montu Farman
from Guido and Pascal would show and
Benjy is to show a mathematical
properties that somewhat encode this
high-level intuition and is a provable
statement so one thing is that when we
talked about these linear regions that
are created by Rayleigh networks what
you can show is as you keep adding
layers rather than neurons the number of
trunks in which you are dividing your
input space grows exponentially with
depth and only polynomially with going
wider which shows you that there isn't
simply an enormous reason to go deeper
rather than wider right exponential
growth simply will escape any polynomial
growth sooner or later and with the
scale at which were working these days
it escaped a long time ago the other
thing is if you believe in this high
level idea of learning from savable
times from statistical learning theory
that the principle of learning is to
encounter some underlying structure in
data right we get some training data set
which is some number of samples we build
a model and we expect it to work really
well on the
data which comes from the same
distribution but is essentially a
different set how can this be done well
only if you learned if you discovered
some principles behind the data and the
output space and one such or a few such
things can be mathematically defined as
finding regularities symmetries in your
input space and what raelia networks can
be seen as is a method to keep folding
your input space on top of each other
which has two effects one of course if
you keep folding space you have more
when I say fold space I mean that the
points that end up on top of each other
are treated the same so whatever I build
on top of it will have exactly the same
output values for both points that got
folded so you can see why things will
grow exponentially right you fold the
paper once you have two things on top of
each other then four then eight it's
kind of how this proof is built it's
really beautiful I really recommend
reading this paper and beautiful
pictures as well and the second thing is
this is also the way to represent
symmetries if your data if your input
space is symmetric the easiest way to
learn that this symmetry is important is
by folding this space in half if the
symmetries more complex as represented
in this beautiful butterfly ish I don't
know shape you might need to fold in
this extra way so that all the red
points that are quite swirled end up
being mapped onto this one single
slightly curved shape and this gives you
this sort of generalization you discover
the structure if you could of course
learn it that only depth can give you if
you were to build much wider model you
need exponentially many neurons to
represent exactly the same invariance
exactly the same transformation which is
really nice mathematical insight into
this while depth really matters so
people believe this I mean of course
people were using depth before just
because they saw they seen better
results they didn't need necessarily
mathematical explanation for that so
let's focus on this simple model that we
just defined we have three neural
network
sorry three hidden layers in our neural
network linear a linear value and so and
so forth and now we'll go from our
puzzle view that was a nice high level
intuition into some of this extremely
similar and what's actually used in
pretty much every machine learning
library underneath which is called a
computational graph so it's a graph
which represents this sort of relations
of what talks to words I'm going to use
the same color coding so again blue
things inputs so this is my input X this
is gonna be a target orange is going to
be our loss the Reds are gonna be weight
parameters so the reason why some of you
might have noticed when I was talking
about linear layer I treated both
weights and XS as input to the function
whether I was writing f of X WB I was
not really discriminating between
weights and inputs apart from giving
them the color for easier readability is
because in practice it really doesn't
matter there's no difference between a
weight or an input into a note in a
computational graph and this alone gives
you a huge flexibility if you want to do
really funky stuff like maybe weights of
your network are gonna be generated by
another neuron network it's fully fits
this paradigm because all you're gonna
do is you're gonna substitute one of
these red boxes that would normally be a
weight with yet another network and it
just fits the same paradigm and we'll go
for some examples in a second to be more
precise we have this graph that
represents computational graph for a
free layer neural net with values on
height of abstraction omitting captions
because they are not necessary for this
story they don't have to be linear you
can have side tracks I'll skip
connections there is nothing stopping
you from saying okay output from this
layer it's actually going to be
connected from toe sorry yet another
layer that is also parameterize by
something else
and then they go back and merge maybe
through mean operation concatenation
operation that many ways to merge two
signals
there is nothing stopping us from having
many losses and they don't even have to
be at the end of paragraph we might have
a loss attached directly to ways that
will act as the penalty for weights
becoming too large for example or maybe
leaving some specific constraints maybe
we want them to be lying on the sphere
and we're going to penalize the model
for not doing so our losses don't even
need to be the last things in the
computational graph you can have a
neural network that has a loss at the
end and this loss is fitted back its
value to next parts of a neural network
and this is the actual output that you
care about eventually you can also do a
lot of sharing so the same input can be
plucked into multiple parts of your net
in skip connection fashion you can share
weights of your model and sharing
weights in this computational graph
perspective is nothing about connecting
one nodes too many places this is
extremely flexible language that allows
this really modular development and
arguably it actually it helped
researchers find new techniques because
the engineering advancement of
computational graphs development allows
to free us from saying oh there are ways
that inputs in a qualitatively different
engineers came and said no from my
perspective they are exactly the same
and the research followed it would have
started plugging crazy things together
and ended up with really powerful things
like hyper networks so how do we learn
in all these models and the answer is
surprisingly simple you just need basic
linear algebra 101 to just recap radians
and jacobians I hope everyone knows what
they are if not in very short words if
we have a function that goes from D
dimensional space to a scalar space like
R then the gradient is nothing about the
vector of partial derivatives so I've
dimension we have partial derivative of
this function with respect to I input
what's partial derivative in height of
abstraction just the direction in which
the function grows the most and minus
gradient is the direction in which it
decreases the most
Jacobian nothing about the K dimensional
generalization if you have K outputs and
so is a matrix where you have a partial
derivative of F output with respect to
jave input nothing else very basic thing
the nice thing about these things is
they can be analytically computed for
many of the functions that we heard and
then the gradient descent technique that
is numerical methods 101 so surprisingly
deep learning uses a lot of very basic
components but from across the board of
mathematics and just composes it in a
very nice way an idea behind gradient
descent is extremely simple we can view
this a sort of physical simulation where
you have your function or loss landscape
you just pick an initial point and
imagine that is a ball that keeps
rolling down the hill until it hits a
stable point or it just cannot locally
minimize your loss anymore so you just
add each iteration tell your current
point subtract learning rate at time T
times the gradient in this specific
point and this is going to guarantee
convergence to the local minimum under
some minor assumptions of on the
smoothness of the function so it needs
to be smooth for it to actually converge
and it has this nice property that it
was referring before that because
gradient of the sum is sum of the
gradients you can show that analogues
properties hold for the stochastic
version or you don't sum over all
examples you just take a subset and keep
repeating this this will still converge
under some assumptions of the bound
between basically noise or the variance
of this estimator and the important
thing is this choice of the learning
rate unfortunately matters like quite a
few other parameters in machine learning
community and there have been quite a
few other optimizations that were
developed on top of gradient descent one
of which became a sort of golden
standard like step zero that you always
start with which is called atom and when
we go to practical issues I will say
this yet again if you're just starting
with some model just use atom if you're
even thinking about the optimization is
just a good starting starting rule and
in principle you can apply gradient
descent to non smooth functions and a
lot of stuff in deep learning is kind of
non smooth and people still apply it but
the consequence is you will lose your
converges guarantees so the fact that
your loss doesn't decrease anymore might
as well be
that you just did something you were not
supposed to be doing thank you
provided a note without a well-defined
gradient or you define the wrong
gradient you put the stop gradient in
are you created again then things might
stop converging so what do we need from
perspective of our notes so that we can
apply gradient descent directly to the
computational graph right because we
have this competition of graph British
for everything that we talked about and
the only API that we need to follow is
very similar to the one we talked before
we need forward pass given X given input
what is the output and also we need a
backward pass so what is it basically
Jacobian with respect to your inputs for
computational efficiency we want
necessarily compute a few Jacobian but
rather product between Jacobian and the
gradient of the loss that you eventually
care about and this is going to be an
information we're gonna send through the
network so let's be more precise with
our computational graph with free layers
we have this sort of gradient descent
algorithm we have our parameters citas
and we want to unify these views somehow
right so I need to know what theta is
and how to compute the gradient so let's
start with making feet appear so one
view that you might use is that there
actually is an extra node called theta
and all these parameters these w's B's
that I need for every layer is just
slicing and reshaping of this one huge
theta right so imagine there was this
huge vector theta and I'm just saying
the first W whatever its shape is is
first K dimensions I just take them
reshape this is a well-defined
differentiable operation right it's also
gradient of the reshaping is reshaping
of the gradient kind of thing so I can
have one theta and then the only
question is how to compute the gradient
and the whole math behind it is really
chain rule the targeted composition of
functions decomposes with respect to the
inner nodes so if you have F composed
with G and you try to compute the
partial derivative of the output with
respect to the input you can as well
compute the partial derivative of the
output with respect to this inner node G
let's multiply it by the partial
derivative of G with respect to X and if
G happens to be multi-dimensional if
there are many outputs then from matrix
calculus you know that the analogous
object requires you to simply sum over
all these paths so what it means from
the perspective of the computational
graph well let's take a look at one path
so we have the dependence of our lost
node on our way to note that now became
an input change to blue because as we
discussed before there is literally no
difference between these two and it's
going through this so now all we are
going to do is apply the first rule
we're going to take the final loss and
ask it okay we want you to be told
what's the gradient we are now in the
node that needs to know given how the
outputs needs to change which is already
told to us by this node how it needs to
adjust its inputs which is this Jacobian
times the partial derivative of rest of
the loss with respect to our output so
we can send back and we already have the
L D whatever is the name of this node
the previous node has the same property
right it's being told your outputs needs
to change in these directions and
internally its nose and by it nose I
mean we can compute this Jacobian how to
adjust its inputs so that its outputs
change in the same direction and you go
through all this graph backwards da da
da kill your feet theta and this is
using just this rule the only problem is
there is a bit more than 1/2 for this
network there's way more dependents but
this is where the other one comes into
place we will just need to sum over all
the paths that connect these two nodes
they might be exponentially many paths
but because they reuse computation the
whole algorithm is fully linear right
because we only go through each node
once computing up till here is
deterministic and then we can in
parallel also compute these two paths
until they meet again so have a linear
algorithm that bad props through the
whole thing you can ask couldn't I just
do it by hand for going through all the
equations of course you could but it
would be at the very least quadratic
if you do it naively this is just a
computational trick to make everything
linear and fit into this really generic
scheme that allows you to do all this
funky stuff including all the modern
different architectures representing
everything as computational graphs just
allows you to stop thinking about this
and you can see this shift in research
papers as well
do you like 2005 ish you seen each paper
from machine learning a section a
gradient of a log where people would
define some specific model and then
there will be a section where they say
oh I sat down and wrote down all the
partial derivatives this is what you
need to plug in to learn my model and
since then disappeared no one ever
writes this
they just say and I use tensor flow P or
carrots or your favorites library it's a
good
it moved field forward instead of
positive dogs you know spending a month
deriving everything by hand they spent
five seconds
clicking greater so let's reimagine
these few modules that we introduced as
computational graphs we have our linear
module as we talked before is just a
function with three arguments it is
basically a dot product between X and W
we add B and what we need to define is
this backward computation with respect
to each of the inputs no matter if it's
an actual input blue thing or a weight
as we discussed before and for X and W
themselves the situation is symmetric we
essentially for X is just multiplied by
W the errors that are coming from the
future I mean from further from the
graph not from the future and for the W
is just the same situation but with X's
right because the dot product is pretty
symmetric operation itself and the
update for the biases is just the
identity since they are just added at
the end so you can just adjust them very
very easily and the nice thing to note
is that all these things in backwards
graph
they are also basic algebra and as such
they could be a computational graph
themselves and this is what happens in
many of these libraries when you call TF
gradients for example or something this
the backward computation will be added
to your graph there will be a new chunk
of your
that represents the backwards graph and
what's cool about this is now you can go
really crazy and say I won't tell old
order derivative I want to back up for
back prop and all you need to do is just
grab and note that corresponds to this
computation that was done for you just
call it again and again and again and
just get this really really powerful
differentiation technique until your GPU
around dies right but there's a customer
really itself
super simple in the forward pass you
have maximum will zero and X in the
backward pass you end up with a masking
method so if the specific neuron was
active when I say active I mean it was
positive and they rarely just passed it
through then you just pass the gradients
through as well and if it was inactive
meaning it was negative it hit zero then
of course gradients coming back need to
be zeroed as well because we don't know
how to adjust them right locally from a
vertical perspective if you are in the
zero land if you make an infinitely
small step you are still in zero land
let's forget about actual zero because
this one is tricky softmax is also
relatively simple
maybe it's gravely slightly fancier
because there's a exponentiation
summation division but is the same
principle right and you can also derive
the corresponding partial derivative
which is the backward pass and it's
essentially the difference between the
incoming gradients and the output and
you can see that these things might go
up right softmax itself if X J is very
big then exponent will just overflow
whatever is the numerical precision of
your computer and as such is rarely used
in such a form
it's either composed with something
that's cautious it back to reasonable
scale or does some tricks like you take
a minimum of XJ and say 50 so that you
lose parts of say mathematical beauty of
this but at least things will not blow
up to infinities and now if you look at
the cross entropy is also very simple to
vectorize and it's partial derivatives
now we can see why things get mess
see computationally you divide by P
dividing by small numbers as you know
from computer science basics can again
overflow so it's something that on its
own is not safe to do again you could
hack things around but the nicer
solutions and the nice thing about
viewing all these things jointly inputs
weights targets whatever as the same
objects with exactly the same paradigm
exactly the same model that we use to
say well these are these pictures of
dogs and cats right and these are the
targets what is the set a set of weights
for this model to maximize the
probability of this labeling can also
ask the question given this neural
network what is the most probable
labeling of these pictures so that this
neuron network is going to be happy
about it its loss is going to be low by
simply attaching our gradient decent
technique instead of to the theta that
we can attach directly to T right and as
long as these things are properly
defined in your library is going to work
and now you can see why would you
compose softmax and the cross-entropy
because now backward pass extremely
simplifies instead of all these
nastiness division small numbers etc you
just get the partial derivative of the
loss with respect to inputs as a
difference between targets and your
inputs as simple as that all the
numerical instabilities gone you can of
course still learn labels and partial
derivative is relatively okay this is
one of the main reasons why when using
machine learning libraries like Kara's
tensorflow and many others you'll
encounter this cross-entropy jungle you
see ten functions that are called
cross-entropy something like sparse
cross and dropping with logits
cross-entropy with soft marks I don't
know apply twice table the reason is
because each of these operations on its
own is numerically unstable and people
wanted to provide you with a solution
that is numerically stable they just
took literally every single combination
gave it a name and each of these
combinations is implemented in a way
this numerically
and all you need to do is to have this
lookup table which combination you want
to use and pick the right be the right
name right but underneath they're always
just composing cross-entropy with either
sigmoid or soft max or something like
this and it's exactly this problem that
they are avoiding if you want to do
things by hand feel free but don't be
surprised if even on em this from time
to time you'll see an infinity in your
loss it's just the beauty of finite
precision arithmetic in the continuous
land so let's go back to our example
right it was this small puzzle piece now
we can explicitly label each of the
notes so we have our XS they go for dot
product with ways biases are being added
then there is r lu we do it quite a few
times at some point at this point we
have probability estimates and this is
the output of our model even though our
loss is computed later this is also one
of the things I was mentioning before
right the output or the special nodes
don't have to be at the end they might
be branching from the middle we can
replace this with frita
does sorry and just slicing and apply
our gradient descent and maybe to
surprise some of you this is literally
how training of most of the deep neural
nets look like in supervised way
reinforcement learning slightly
different story but it's this underlying
principle that allows you to work with
any kind of neural network it doesn't
have to be this linear structure all of
these funky things that I was trying to
to portrait ten slides ago relies on
exactly the same principle and you use
exactly the same rules you just keep
composing around the same algorithm and
you get an optimization method that's
going to converge to some local minimum
not necessary perfect model but is going
to learn something so there are a few
things that we omitted from this that
are still interesting pieces of the
puzzle one such thing is taking a
maximum so imagine that one of your
nodes wants to take a maximum you have a
competition B between your inputs and
only the maximal one is going to be
selected then the backwards pass of this
operation
is nothing about gating again there's
gonna be passing through the gradients
even only if this specific dimension was
the maximal one and just zeroing out
everything else you can see that this
will not learn how to select things but
at least it will tell the maximal thing
how to adjust under the conditions that
it got selected all right so there's
this notion of non small things that
don't necessarily guarantee convergence
in a mathematical sense but they are
commonly used and you'll see in say
Sanders talk on convulsion your networks
that they are part of the max pooling
layers you can have funky things like
conditional execution like five
different computations and then a1 hood
layer that tells you which this of these
branches to select rather than if it was
one hot encoded then selection can be
viewed as just point wise multiplication
right by one we multiply and then the
backward pass is just gonna be again
gated in the same way but if it was not
the one hood encoding but rather output
of the softmax right of something
parametrized then looking at the
backward pass with respect to the gating
allows you to literally learn the
conditioning you can learn which branch
of execution to go through as long as
you smoothly mix between them using
softmax
and it's a high-level principle or
high-level idea behind modern attention
models let us essentially do this and to
give you some trivial example of other
laws so you had cross entropy but of
course many problems in real life are
not classification if it's a regression
so your outputs are just real numbers
then l to quadratic gloss or one of at
least ten other names for this quantity
which is just a square norm of a
difference between targets and your
prediction can also be seen as a
computational graph and the backward
pass is again nothing but the difference
between what you predicted and what you
wanted and there's this nice duality as
you can see from the backwards
perspective that looks exactly the same
as in case of the cross and trouble of
the softmax which also provides you with
some intuitions into how these things
are correlated so let's quickly go
through some practical issues given that
we know kind of what we're working with
and the first one is the well-known
problem of
fitting and regularization so from
statistical learning theory so we are
still or again going back to say talking
her way before him we know that in this
situation where we just have some
training set which is a finite set of
data that we're building our model on
top minimizing error on it which we're
going to call training error or training
risk is not necessarily what we care
about what we care about is how our
model is going to behave in a while it
what's going to happen if I take a test
sample that kind of looks the same but
it's a different dog than the one that I
saw in train that's what we are going to
call test risk test car and it's a
provable statement that there is this
relation between complexity of your
model and the behavior between these two
errors as your model gets more and more
complex and by conflicts I mean more
capable of representing more and more
crazy functions or being able to just
store and store more information then
your training error has to go down not
in terms of any learning methods but
just in terms of existence of parameters
that realize it think universe
approximation theorem right it says
literally that but at the same time as
things get more complex and the bigger
your test test risk initially goes down
because you are just getting better at
representing the underlying structure
but eventually the worst case scenario
is actually going to go up because you
might as well represent things in a very
bad way by for example a numerating all
the training examples and outputting
exactly what's expected from you zero
training error horrible represents
horrible generalization power and this
sort of curve you'll see in pretty much
any machine learning book till 2016 ish
when people started discovering
something new that will go through in a
second but even if you just look at this
example you notice that there is some
reason to keep things simple and so
people developed many regularization
techniques such as LP regularization
where you attach one of these extra
losses directly to weights that we
talked about before which is just LP
norm like L to quadratic norm or l1 or
something like this to each of your
weights so that your weights are small
and you can prove again I guarantee that
if weights are small the function cannot
be too complex so you are restricting
yourself to the left-hand side of this
graph you can do drop out where some
neurons are randomly deactivated again
much harder to represent complex things
you can add noise to your data you can
stop early or you can use various
notions of normalization that will be
talked through in the next lecture but
that's all in this worst case scenario
what people recently discovered or
recently started working on is how this
relates to our deep neural networks that
don't have hundreds of parameters they
have billions of parameters and yet
somehow they don't really over fit as
easily as you would expect so the new
version of this picture emerge that's
currently referred to as double descent
where you have this phase change but yes
things get worse as you get more and
more complex model but eventually you
hit this magical boundary of over
parameterization where you have so many
parameters that even though in in theory
you could do things in a very nasty way
like by enumerate examples because of
the learning methods that we are using
you never will
you start to behave kind of like a
Gaussian process and as you keep
increasing number of parameters you
actually end up with as simplest
solutions being found first rather than
the more complex ones and so the curve
descends again and it has been proven by
balcony at all under some constraints
and shown in simple examples then it was
also reinforced with a cool work from
sorry
Pritam from from open AI where they
showed that this holds for deep big
models that we care about so one could
ask well does it mean we don't care
about relaxation anymore you just make
models bigger and the answer is well not
exactly
it's both true that as you increase the
model that you can see on the x-axis
that you're lost after test loss after
rapidly increasing keeps decreasing all
the time but adding regularization can
just keep the whole curve lower so here
as you go through curves from top to
bottom
it's just more and more regularization
being added so what it means how it
relates to this theory of complexity
what that mostly means is model
complexity is way more at the number of
parameters and this is a local minimum
like research local minimum people were
in for quite a while where they thought
well your neural network is huge truly
is not going to generalize well because
of opting Chevron and keys bounds are
infinite you're doomed and it seems not
to be the case the complexity of the
model strongly relies on the way we
train and as a result you are still kind
of in this regime where pain things can
get worse and you do need to regularize
but adding more parameters is also a way
to get better results slightly
counterintuitive and only applies if you
keep using gradient descent not some
nasty way okay
so just a few things there's a lot of
stuff that can go wrong when you train a
neural net and it can be hard harsh
experience initially so first of all if
you haven't tried don't get discouraged
initially nothing works and it's
something we all went through and there
is nothing to solve it apart from
practice just playing with this will
eventually get you there there's a
brilliant blog posts from Andrew karpati
and I'm referring to here and also a few
points that I like to keep in mind each
time I train neural networks first of
all that initialization really matters
all this fury that was built or the
practical results if you initialize your
network badly it won't learn and you can
prove it won't work won't learn well
what you should start with always is to
try to overfit with some if you're
introducing a new model especially you
need to try to overfit on some small
data sample if you can't over fit almost
surely you made a mistake unless for
some reason your model doesn't work for
small sample sizes then obviously just
ignore what I just said
you should always monitor training loss
I know sounds obvious but quite a few
people just assume that loss will go
down
because gradient decent grantees it
without monitoring it you will never
know if you are in the right spot
especially given that many of our models
are no differentiable and as such the
loss doesn't have to go down so if it's
not going down you might want to
reconsider using this non differentiable
units more important is something that
people apparently stopped doing in deep
learning on a daily basis it's
monitoring norms of your weights norms
going to infinity is something to be
worried about and if it's not making
your job crush right now it eventually
will once you leave it running for a few
days and then you'll regret that your
notes monitoring it earlier another
thing is adding shape assets all the
modern learning deep learning libraries
are great has helped brilliant features
one of which is automatic broadcasting
they take a column vector we take a row
vector you add them you get the matrix
very useful unless this is not what you
wanted to do you just wanted to vectors
and you ended up of a matrix if the next
operation is taking a maximum or taking
the average you won't notice right
afterwards there's just a scalar
everything looks fine but your learning
will be really crazy and you can try to
internal linear regression and just by
mistake transpose targets and you'll see
how badly linear regression can behave
by just one liner that throws no
exceptions and your loss will go down it
just won't be the model that you're
expecting the only way that I know about
to resolve this is to add shape asserts
everywhere each time you add an
operation we just write down an assert
like literally low-level engineering
thing to make sure that the shape is
exactly what you expect otherwise you
might run into issues things that we
mentioned before
use atom as your starting point just
because free e minus v is the magical
learning ring it works in 99% of deep
learning models for unknown reasons to
everyone
finally it's very tempting to change
five things at a time because you feel
like you have so many good ideas and
don't get me wrong you probably do but
if you change all of them at once
you were regretted afterwards when you
struggle with debugging and or credit
assignment of what actually improves in
your model and the reviewers won't be
happy either
when your ablation just keeps five steps
so given a few last minutes before the
questions I wanted to spend save three
ish minutes on the bonus Fink on
multiplicative interactions so I was
trying to convince you through this
lecture that neural networks are really
powerful and I hope I succeeded they are
very powerful but I want to ask this may
be a funny question what is one thing
that these multi-layer networks or we
just have a linear then an activation
function say Sigma the rail you stacked
on top of each other definitely cannot
do well there may be answers right they
can't do a lot of stuff but one trivial
thing they can't do is they can't
multiply there's just no way for them to
multiply two numbers given us inputs
again you might be slightly confused we
just talked about the inverse
approximation theorem but what I'm
referring to is representing
multiplication we can approximate
multiplication to any precision but they
can never actually represent the
function that multiplies so no matter
how big your data set is going to be no
matter how deep your network is going to
be I can't you train it to multiply two
numbers I can always find two new
numbers that you're going to miserably
fail it and I miserably I mean get
arbitrarily bigger maybe my numbers are
going to be huge doesn't matter there is
something special about multiplication
that I would like to see you know that
what's special about them for example
conditional execution relies on
multiplying something between 0 and 1
and something else many things in your
life can be represented as
multiplication
for example computing distance between
two points relies on being able to
compute a dot product plus norms and
things like this so it's quite useful to
have this sort of operation yet stacking
even infinitely many yes infinitely many
layers would not help and one way to
resolve it in sort of a unit that just
implements multi-plate if interactions
one way to formalize it is as follows
you have a tensor W you take your inputs
through this you can see this as a
Mahalanobis dot product if you were
through this part of the algebra then
you have the bad fix projections of the
remaining things and just add the bytes
so if you just look at the approximation
things if you were to say compute a dot
product and you do it with a normal
neural net of Linear's and well use then
you have an exponentially many
parameters needed to approximate this to
that zero point one error I believe I
used here with respect to the
dimensionality of the input there is a
very steep exponential growth just
approximate and there is still gonna be
this problem that you don't generalize
but even approximation requires huge
amounts of parameters while using model
like this explicitly has a linear growth
and has a guarantee right once you hit
the dot product which can be represented
exactly with this module you will
generalize everywhere there's a nice
work from seed hunt at all at this
year's SML if you want to that deeper
but I want to just stress there is a
qualitative difference between
approximation and representation and in
some sense sends you home with this
take-home message which is if you wants
to do research in this sort of
fundamental building blocks of neural
networks please try not to focus on
improving things like marginally
improving things the neural networks
already do very well if we already have
this piece of a puzzle polishing it I
mean is an improvement but it's really
not what's cool about this field of
study and this is not where the biggest
gains both for you scientifically as
well as for the community lies was the
biggest game is identifying what neural
networks cannot do or cannot guaranty
think about maybe you might want a
module that's guaranteed to be convex or
quasi-convex or some other funky
mathematical property that you are
personally interested in and propose a
module that does that I guarantee you
that will be much better experience for
you and much better result for all of us
and with that I'm going to finish so
thank you
you


Time Until Superintelligence: 1-2 Years, or 20? Something Doesn't Add Up

just this week we have had open AI tell
us that super intelligence might need to
be made safe within four years competing
lab leaders say it's decades away and
expert warnings that AI might have
runaway power within two years let's try
to unpack those disparate timelines see
what might speed up the timing or slow
it down show what super intelligence
might mean and end with some interesting
clips that capture the moment we're in
but the first timeline is from Mustafa
Solomon head of inflection AI this week
if it's so risky why don't you stop I
think that the point of raising concerns
is that we can see a moment at some
point in the future probably over a
decade or two decades time Horizon when
slowing down is likely going to be the
safe and ethical thing to do 10 years is
not a long time I find it fascinating
that he talks about two decades from now
when inflection AI his company have just
built the world's second highest
performing supercomputer and even as
they admit that's three times as much
compute as was used to train all of gpt4
telling the public that we have a decade
or two before we have to worry about
safety seems extremely conservative to
me but what do we even mean by
transformative AI or super intelligence
well here is just one projection of
current scaling laws out to 2030 from
Jacob steinhardt of Berkeley and here of
course we're talking about just six and
a half years away if we look at
projections of future compute and data
availability and the velocity of current
Improvement which of course might not
hold forever some experts claim that
we'll need new Innovations beyond the
Transformer but if current projections
of future compute and data availability
scale up here's the kind of thing that
we're talking about being superhuman at
tasks including coding hacking
mathematics protein engineering doing
1.8 million years of work in 2.4 months
learning 4 2 500 human equivalent years
in just one day and by training on
different modalities such as molecular
structures low-level machine code
astronomical images and brain scans it
might have a strong intuitive grasp of
domains where we have limited experience
including forming Concepts that we do
not have indeed some research released
this week show that gpt4 already crushes
some benchmarks for creative thinking
and the median forecast for being better
than all but the very best humans at
coding is 2027 and here we have a median
forecast of 2028 for AI winning a gold
medal at the international math Olympiad
the number that I'm looking out for is
getting a hundred percent on the mmlu
that's a test of 57 different subject
matters and I've actually been
discussing with some of the creators of
the mmlu that we might not even know the
full potential of gpt4 on this test
officially it's 86.4 percent so we've
heard 20 years and six and a half years
well how about two this article comes
from the Boston Globe that did a feature
piece on Dan Hendricks and the center
for AI safety they were behind that one
sentence letter that was signed by
almost all of the AGI lab leaders and
World experts on AI the journalists are
Stan Hendricks how much time we have to
tame Ai and he said well how long till
it can build a bio weapon how long till
it can hack it seems plausible that all
of that is within a year and within two
he says AI could have so much runaway
power that it can't be pulled back seems
a pretty massive contrast to Mustafa
Suleiman talking about a decade or two
from now I'm going to come back to this
article quite a few times but now I want
to move on to open ai's recent statement
this week they released this introducing
super alignment we need scientific and
Technical breakthroughs to steer and
control AI systems much smarter than us
I can just see now all the comments from
people saying saying that that's going
to be physically impossible but moving
on to solve this problem within four
years we're starting a new team co-led
by Ilya satskova and Jan Leica and
dedicating 20 of the compute we've
secured today to this effort that is
quite a remarkable statement to their
credit they've made themselves
accountable in a way that they didn't
have to and that others haven't and
they're deploying one of the legends of
deep learning Ilya sotskova to help them
achieve this goal they say that super
intelligence will be the most impactful
technology Humanity has ever invented
and I agree with that and it could help
us solve many of the world's most
important problems absolutely but the
vast power of super intelligence could
also be very dangerous and could lead to
the disempowerment of humanity or even
human extinction they go on while super
intelligence seems far off now we
believe it could arrive this decade
notice they don't say in a decade they
say this decade they go on currently we
don't have a solution for stereo or
controlling a potentially super
intelligent AI they can't prevent it
from going rogue and are current
techniques for aligning AI rely on
humans ability to supervise AI but
humans won't be able to reliably
supervise AI systems that are much
smarter than us and so our current
alignment techniques will not scale to
Super intelligence I'm going to go into
more detail about their plan for
aligning super intelligence in another
video but here is the high level
overview essentially they want to
automate alignment or Safety Research
build an AI alignment researcher I've
read each of these papers and posts and
some of them are very interesting
including automated red teaming and
using a model to look inside the
internals of another model but the point
of including this post in this video was
the timeline of four years twenty
percent of their compute is millions and
millions and millions of dollars and
four years is a strict deadline and one
of the most interest interesting aspects
of this post came in one of the
footnotes they say solving the problem
includes providing evidence and
arguments that convince the machine
learning and safety community that it
has been solved that is an extremely
high bar to set yourself they go on if
we fail to have a very high level of
confidence in our Solutions we hope our
findings let us and the community plan
appropriately that's probably one of the
most interesting sentences I've read for
quite a while if we fail to have a very
high level of confidence in our
Solutions we hope our findings let us
and the community plan appropriately in
other words if they can't make their
models safe they're going to have
contingency plans and they want the
community to have plans as well and it
is a really interesting number isn't it
four years not even around five years or
just end of the decade and it does make
me wonder what Ilya satskaver thinks is
coming within four years to have such a
deadline now apparently the prediction
markets give them only a 15 chance of
succeeding and the head of alignment at
open AI said he's excited to beat these
odds so we've heard about one to two
years and about four years but what
might slow those timelines down the
other day I read this fascinating paper
coincidentally co-authored by Jacob
steinhardt on jailbreaking large
language models the paper showed that
you could basically jailbreak gpt4 and
Claude a hundred percent of the time
using a variety of techniques and that
is fascinating to me as we approach the
one year anniversary of the creation of
gpt4 and the relevance to Super
intelligence is that if the creators of
these models can't stop them being used
to commit crimes then you would think
that they might have to dedicate more
and more of their efforts in stopping
jailbreaks versus working on
capabilities for obvious reasons I'm not
going to go into too much detail on
jailbreaking here but here is Claude
plus from anthropic telling me how to
hotwire a car and to be honest that's
just the most innocent one and yes it
did also work on gpt4 I did find one of
the reasons why it does work quite
interesting though that reason is about
competing objectives where it's
compulsion to predict the next word
successfully overrides its safety
training and so because those two facets
of smartness clash inside the model it's
not an issue that can be fixed with more
data and more scale what else might slow
down the work on super intelligence well
lawsuits and possible criminal sanctions
Yuval Noah Harari recently said that AI
firms should face prison over the
creation of fake humans and he was
saying this to the United Nations he
called for sanctions including prison
sentences to apply to tech company
Executives who fail to guard against
fake profiles on their social media
platforms of course those Executives
might well blame the AI companies
themselves but Harare said that the
proliferation of fake humans could lead
to a collapse in public trust and
democracy now it's possible for the
first time in history to create fake
people billions of fake people if this
is allowed to happen it will do to
society what fake money threatened to do
to the financial system if you can't
know who is a real human trust will
collapse what's another famous roadblock
to Super intelligence hallucinations
I've already talked in another video
about how salmon thinks that won't be an
issue in 18 to 24 months but here again
is Mustafa Suleman on the issue of
hallucinations yesterday he said soon
llms will know when they don't know
they'll know when to say I don't know or
instead ask another AI or ask a human or
use a different tool or a different
knowledge base this will be a hugely
transformative moment and on that I
agree hallucinations are probably one of
the biggest hurdles stopping most people
from using llms more commonly it's not
about knowing more it's about when these
models bullcrap less or the moment when
they don't bull crap at all but what
about things that could actually speed
up the timelines to Super intelligence
going back to the Boston Globe ask
school one thing could be competition
for military Supremacy which has already
produced a startling turn to Automation
and that's not just Robotics and
autonomous drones that's the llms that
might control them here is a snippet of
a trailer for a Netflix show released
today
[Music]
AI is a dual edged sword
a flip of a switch and the technology
becomes lethal
there is no place that is Ground Zero
for this conversation more than military
applications
forces that are supported by AI will
absolutely crush and Destroy forces
without militaries are racing to develop
AI faster than their adversaries the AI
unless it's told to fear death will not
fear death there is no second place in
Warren if you're going up against an AI
pilot you don't stand a chance if
language models prove useful in war the
amount of investment that's going to go
into them will Skyrocket of course
investment doesn't always equal
Innovation but it usually does and one
of the other things that could speed up
timelines is the automation of the
economy for detail on why it might check
out the paper linked above and in the
description but the high level overview
is this as AI grows more capable and
ubiquitous companies will be forced
essentially to hand over increasingly
high level decisions to AIS in order to
keep up with their Rivals if an AI as
CEO does a better job for stockholders
how long can a company resist employee
them and of course it doesn't just have
to be white collar work as Andre
carpathy said welcome to the Matrix for
apples but the thing is whether we're
talking about one year or four years or
six Super intelligence is coming pretty
soon and it is interesting to me that so
much of society is carrying on as if
it's not coming take these 50 year long
mortgages that are available in the UK
how can anyone plan out 50 years from
now in a world where we might have super
intelligence in five of course I do
think we all need to start defining
terms a bit better and I've tried to do
that on this channel with AGI and super
intelligence I don't think it's quite
good enough to give vague reassurances
of a decade or two from now how we're
going to react when super intelligence
arrives is anyone's guess we might be
crushed by the sense of inferiority as
Douglas hofstetter recently said
or some of us might become like Curious
children speaking to a wise adult just
the other day I got a foreshadowing of
my own reaction by speaking to Pi the
model from inflection AI it is designed
to be extremely human-like and the
conversations can be quite startling and
personal of course just imagine when
they're super intelligent and multimodal
anyway let me know your thoughts in the
comments and as always have a wonderful
day