Deep Learning 3: Neural Networks Foundations

so I hope you enjoyed last week's
tutorial on tensor flow and this week we
again have something very special for
you Simon or Sendero here will give a
lecture about neural networks back
propagation how to train those networks
and so on and it's really quite special
to have Simon here he really is an
expert on the topic he also works a deep
mind in the deep learning group he is
educated locally to some degree at least
yeah so I'm a mother
Cambridge then PhD at UCL and then later
worked with geoff hinton in canada so
there couldn't be a better person to do
this before we start just a quick
announcement terry williams who attended
here last week he's running a reading
group on deep learning and the game of
Go I'll put this book cover and his card
here on the table in case anyone's
interested it's basically a new book
that came out that tries to explain deep
learning based on on the game of go in
the wake of alphago okay thank you very
much over to you Simon Hayter and get up
to noon everyone see can everyone hear
me okay you can hear me okay yes so sort
of saying
today's lecture is just me covering some
of the foundations of neural networks
and I'm guessing that some of you will
be quite familiar with the material that
we're going to go over today and I hope
that most of you have seen bits of it
before but nevertheless it's kind of
good to go back over the foundations to
make sure that they're very solid and
also one of things that I'm going to
hope to do as we go through is in
addition to kind of conveying some of
the mathematics also try and give you a
sense of the intuition to get a kind of
deeper and more visceral understanding
of what's going on and as we go through
there'll be a couple of natural section
breaks between the sections so that's
probably a good time to do questions
from the preceding section if there are
any and we'll also have an inci break in
the middle probably two-thirds of the
way through
and then the the last point is these
slides were all going to be available
online and in the slides I've added
quite a few hyperlinks out to additional
material which if one of the topics
we're talking about is particular
interesting to you you can kind of go
off and read more about that okay and so
this slide is in some sense a tldr of
what we're going to do today and at a
high level it's also kind of a tldr of
what we're going to do in this entire
course so deep learning good neural
networks is actually pretty simple as
it's more or less just the composition
of linear transforms and nonlinear
functions and it turns out that by
composing these quite simple building
blocks into large graphs we gained
massive powerful flexing flexible
modeling power and when I say massive
fight I do mean quite massive so these
days we routinely train neural networks
with hundreds of millions of parameters
and when I say training or learning what
does that mean well it basically means
optimizing a loss function that in some
sense describes a problem we're
interested in over some data set or in
the case of reinforcement learning with
respect to world experience with with
effective our parameters and we do that
using various gradient optimization
methods one of the most common of those
is SPD or stochastic gradient descent
and so from a thousand feet that's
that's kind of it it's pretty simple but
in this course what we're going to do is
look at the details of the different
building blocks when you might want to
make certain choices and also how to do
this well at a very large scale so
before we dive in let's step back a
little bit and ask why are we doing this
what what a neuron that's good for and
turns out they're actually useful for a
whole ton of things but these days you
know I think a better question is you
know if you can come up with the right
loss function and a quiet training later
what a neuron that's not good for so
just to kind of go over some examples in
recent years we've seen some very
impressive steps forward in computer
vision we can now recognize objects in
images with very high accuracy there's
all sorts of cool more esoteric
applications that the folks
so listen very nice work looking at
doing superhuman recognition of human
emotions by having a neural network that
can recognize micro expressions on folks
faces so essentially better or even
human emotions than humans are later in
this course there'll be a module on
sequence models with a recurrent neural
networks and there we've seen incredible
gains and speech recognition one of the
cool things again in recent years that
came up is this idea of using neural
networks for machine translation and
furthermore it turns out that you can
use neural networks for multilingual
machine translation so the echo is
hello-hello note them maybe the mics on
turn I'll turn raise my voice per dad
yeah please do raise your hand if
they're if you're having trouble hearing
me yes so one of the particularly cool
things that came out in the last year or
so is this idea of doing multilingual
translation through a common
representation so we can translate from
many languages into many other languages
heavily wide
okay
is that better for folks right yeah this
notion of a kind of interlink where so
if we have a common representation space
that is the bottleneck when we're
translating from one language to another
then in a very real sense you can think
of the representations in that space as
some kind of inter linguist so it's kind
of representing concept across many
different languages along similar lines
there's been some excellent work from
deep mind on speech synthesis so going
from text to speech and wavenet was a
something that was developed at D mind
starting back two years ago and now it's
in production so a lot of the voices
that you'll hear in say Google home or
Google assistant are now synthesized
with wavenet so a very fast turnaround
from research to large-scale deployment
other places where they've been enjoying
impressive uses in reinforcement
learning and you'll hear much more about
that in the other half of the course so
things like dqn or a3c and applying that
to aims headings like atari and then
also moving into moralistic games and 3d
environments also with reinforcement
learning you guys are all probably
familiar with alphago which was able to
beat the human world champion at go and
has now even superseded that by playing
just the games itself so not not even
using any any human data now the list
goes on and in all these cases what
we're dealing with is pretty simple and
there's just a couple of different
elements you see grab a laser pointer
yeah cool yes so we essentially have our
neural network so we defined some
architecture we have our inputs so it
could be images spectrograms
you name it we have parameters that
define the network and some outputs that
we want to predict and essentially all
we're doing is formulating a loss
function between our inputs and our
outputs and then optimizing that loss
function with respect to our parameters
and and again it's in a high level
everything we're doing is very simple
but the devil is in the details so
here's a road map
for most the rest of today so that the
the field of neural networks has been
around for a long time and there's a
fairly rich history so there's you know
not time to cover all that today what
would we are going to cover today in in
the course overall are the things that
are having the most impact right now but
I just wanted to begin by calling out
some of the topics that I think are
interesting but that we're not going to
cover and I'd also encourage you to kind
of delve into the history of the field
if there are particular topics that
you're interested in because there's a
lot of work dating back to the sort like
early 2000 and even the 80s and 90s that
is probably worth revisiting in the rest
of the course we'll begin by a treatment
of single layer networks and just seeing
ok what can we do with just one layer
weights and neurons we'll then move on
to talk about the advantages that we get
by adding just one hidden layer and then
we'll kind of switch gears and kind of
focus on what I call modern deep net
so here it's useful just to think in
terms of abstract compute graphs and
we'll see some very large networks and
also how to think about composing those
in software there'll be a session and
this is probably the most math heavy
part of today on learning and so there
will kind of recap some concepts from
calculus and vector algebra and then
we'll talk about modular backprop an
automatic differentiation and those are
tools that allow us to build these
extremely esoteric graphs without having
to think too much about how learning
operates I'll talk a bit about what I'm
calling a model Zoo so when we think
about these networks in terms of these
modules then what are the building
blocks that we can use to construct them
from and then toward the end I've
touched on some kind of practical topics
in terms of you want actually doing this
in practice what are things that you
might want to be aware of what a tricks
you can use to sort of diagnose if
things are going wrong and maybe we'll
talk about a research topic yes but as I
was saying it's a large field with many
branches dating back depending when you
count dating back to the 60s and then
there was another resurgence in the 80s
so a couple of things that I think are
interesting that won't be covered in
this lecture course are also machines
and hopfield networks
they were developed ran through the 80s
and for quite a while were extremely
popular and there was some interesting
early work I guess in the second wave of
neural networks that they're not in
favor as much now but I think they're
still useful so particularly for
situations were we're interested in
models of memory and in particular
associative memory so I think for me
that's that's one thing that's worth
revisiting another area that what's
property at one time that doesn't
receive as much attention now is models
that operate in the continuous time
domain so in particular spiking neural
networks and one of the reasons that
they're interesting that it's a
different learning paradigm but if you
have that kind of model it's possible to
do extremely efficient implementations
in hardware so you can have very
low-power
New York neural networks so I said yeah
there's lots of things to to look at I'd
encourage you to look at the history of
the field in addition to the stuff that
we cover in this course oh and one last
thing at a high level this small caveat
on terminology and this is a little bit
a function of the history of the field
we sometimes use different names to
refer to the same thing so I'll try and
be consistent but I'm sure I wouldn't
manage it fully so for instance people
interchangeably might use the word unit
or neuron to describe the activate
activity in a single element of a layer
similarly you might hear non-linearity
or activation function and they they
also mean the same thing slightly
trickier is that we sometimes use the
same name to refer to different things
so in the more traditional view of the
field folks would refer to the compound
of say a nonlinear transformation plus a
non-linearity as Leia
in more modern parlance particularly
when we're thinking about implementation
that things like tend to flow then we
kind of tend to describe as a layer
these more atomic operations so in this
case we'd call the linear transformation
as one layer and the
nonlinearity another layer and link to
that there's also slightly different
graphical conventions when we're
depicting models it should usually be
obvious from context but I just wanted
to call that out just in case that's
confusing okay so as I said we're gonna
start off with what can we do with a
single layer networks and to begin with
I'm gonna make a very short digression
on real neurons and describe some of the
kind of inspiration for the artifice in
your Andriy we use it's a very loose
connection and I won't dwell there too
much will then talk about what we can do
with a linear layer sigmoid activation
function and then we'll kind of recap
binary classification or logistic
regression which should have been in
either the last lecture or in their
business for that lecture and then we'll
move on from binary classification into
multi-class classification okay so in
the slide here in the bottom right this
is a cartoon depiction of a real neuron
so there's a couple things going on we
have a cell body the dendrites which is
where the inputs from other neurons are
received and then the axon with the
tunnel bulbs and that's kind of the
output from this neuron and more or less
the way this operates when a neuron is
active an electrical impulse travels
down the axon it reaches the terminal
bulb which causes vesicles of
neurotransmitter to be released those
kind of diffuse across the gap between
this neuron and the neuron that it's
communicating with when it's received in
the dendrites it causes a depolarization
that eventually makes its way back to
the cell body and B so some of the
depolarizations
from all these dendrites is what
determines whether or not the receiving
neuron is going to fire or not and in a
very very coarse way this process of
receiving inputs of different strengths
and integrating it in the cell body is
what this equation is describing so it's
just a weighted sum of inputs or an
affine transformation if you will so the
inputs X the the weights W and maybe
some bias B and so
is what we'd call a simple linear neuron
if we have a whole collection of them
then we can move into matrix vector
notation so this vector Y is a vector of
linear neuron States and we obtain that
by doing a matrix vector multiplication
between the inputs and our weight matrix
and some bias vector B and there's not
an awful lot we can do with that setup
but we are able to do linear regression
which I think you guys saw previously
but in practice we typically combine
these linear layers with some
non-linearity and particularly for a
stacking them in depth so let's let's
take a look at one of those
nonlinearities and this will kind of
complete the picture of our artificial
neuron so what I'm showing here is
something called the sigmoid function
you can think of it as a kind of
squashing function so this equation here
describes the input-output relationship
and so when we combine that with the
linear mapping from previously we have a
way to sum of inputs offset by a bias
and then we pass it through this
squashing function and this in a very
coarse way reproduces what happens in it
in a real neuron when it receives input
so there's some threshold below which
the neuron isn't going to fire at all
once it's above threshold then it
increases its fire and great but there's
only so fast that a real neuron can fire
and so it has upset rating and so at a
very high level that's what this
function is is performing for us it used
to be that this was the sort of
canonical choice in neural network so if
you look at papers particularly from the
90s or the early two-thousands you'll
see this kind of activation function
everywhere it's not that common anymore
and we'll go into some of the reasons
why but at a high level it doesn't have
as nice gradient properties as we'd like
when we're building these very deep
models however it is I still actively
use them a couple of places so in
particular for gating units if we want
to kind of have some kind of soft
differentiable switch and one of the
most common places that you'll see this
is in long short-term memory cells
which I'll hear a lot more about in the
class on recurrent networks so yeah as I
said even with just a simple linear so
signal neuron we can actually do useful
things so I just grabbed this purple box
here I grabbed some tourist slides so
there's a slight change in notation but
if you think back to logistic regression
what do we have we have a linear model a
linked function and then a cross and to
be loss and this linear model is exactly
what's going on in this linear layer and
the link function is what the sigmoid is
doing so there's an extremely tight
relationship between logistic regression
and bited classification and these
layers in in a neural network and so
with just a single neuron we can
actually build a binary classifier so in
this toy example I've got two classes 0
& 1 if I arrange to have my weight
vector pointing in this direction so
orthogonal to this red separating plane
and I adjust the strength of the weights
and the biases appropriately then I can
have a system where when I give it an
input from class 0 the output is 0 and
when I give it an input from class 1 the
output is 1 so that was binary
classification we're now going to move
on and discuss something called a soft
max layer and this essentially extends
binary classification into multi-class
classification so this type of layer is
a way to allow us to do either
multi-class classification another place
that you you might see this used is
internally in networks if you need to do
some kind of multi-way switching so if
say you have a junction in your network
and there's multiple different inputs
and one of them needs to be routed this
is something you can use as a kind of
multi way gating mechanism so what does
it actually do well if we first think
about the Arg max function so when we
apply that to some input vector X all
but the largest element is zero and the
largest element is one the softmax is
essentially just a soft version of the
Arg max so rather than
only the largest element being one and
everything else being zero the largest
element will be the one that's closest
to one the others will be close to zero
and the sum of activities across the app
vector what we want so it it also gives
us a probability distribution the
mathematical form is here so we have
these exponents and I don't know if the
resolution is high enough on this
monitor but what I'm showing in these
two bar plots here is two slightly
different scenarios so the red bars are
the inputs the blue bars are the outputs
and the scale of the red bars in the in
the lower plot is double that of the one
in the the upper plot so in this example
here the the output for the largest
input is the largest and you can't quite
see but it's about 0.6 so the closest to
one however if I increase the magnitude
of all the input so that the ratios are
still the same but now this is 0.9 so
it's much much close to 1 so as the
scale of the inputs gets larger and
larger this gets closer and closer to
doing a hard max operation and so what
can we use this for well as I said we
can use it to do multi way
classification so if you combine this
kind of unit with a cross entropy loss
we're able to Train something that will
do classification of inputs into one of
several different classes so let's take
a look at what this relationship looks
like so the output for the il iment
which you can think of it as the
probability that the input is assigned
to class I is given by in the numerator
we have an exponent that is a weighted
sum of inputs plus a bias and then this
is normalized by that same expression
over all the other possible outputs so
we have a probability distribution and
in a sense you can think of what's going
on in this exponent as being the amount
of evidence that we have for the
presence of the ID class and had we had
to retrain this had we learn we can just
do that by minimizing the negative log
likelihood or accordingly
the cross-entropy of the true labels
under our predictive distribution in
terms of notation how we represent that
something that you commonly see these
things called one-hot vectors to encode
the two plus label and what's that look
like well basically it's a vector that
is of the dimensionality of the output
space the element for the true class
like the the entry for the element of
the true class label is one and
everything else is zero so it's this
vector here in the example above these
digits so four-digit for the one hop
label vector would look like this so the
fourth element is one everything else is
zero if we plug this into our expression
for the negative log likelihood then we
see something like this so since the
only element that of T that is going to
be nonzero is the target we're
essentially asking this probability here
the log probability of this to be
maximized and then we just sum that
across our data cases so even just with
a linear layer if we were to optimize
this we could form a very simple linear
multi way classifier for say digits
it wouldn't work super well and we'll
talk about adding depth but that's
something that you can actually usefully
do with one of these layers now as I
said it's it used to be the case that
the the sigmoid was the dominant
non-linearity and that's fallen out of
favor and so in a lot of the neural
networks that you'll see nowadays a much
more common activation function is
something called the rectified linear
unit or so notice just shortened to a
ray Lu and it has a couple of nice
properties so it's a lot simpler and
computationally cheaper than the sigmoid
it's basically a function that
thresholds below by 0 or otherwise has a
pass through so we can write it down as
this so if the if the input to the
rayleigh function is below zero then the
output is just zero and then above zero
it's just a linear pass-through and it
has a couple of nice properties one of
which is in this region here the
gradient is constant
and generally in in your networks we
want to have gradients flowing so it's
maybe not so nice here that there's no
great information here but at least once
it's active the gradient is constant and
we don't have any saturation regions
once it was the you know is active so
you'll hear I think a lot more about the
details of the gradient properties of
this kind of stuff in James Martin's
lecture later on in optimization but
these are kind of some of the subtleties
that I was talking about they're
important to think about ok so we've now
seen just a very basic single layer now
let's move on one step and ask ok what
can we do if we have more than one layer
so what can we do with neural networks
with a hidden layer and to motivate this
we'll take a look at a very simple
example so what happens if we want to do
binary classification but the inputs are
not linearly separable and then in the
second part of this section I'll kind of
give a a visual proof for why we can see
that neural networks are universal
proper function approximate is so with
enough with a large enough network we
can approximate any function so when I
say a single hidden layer this is what I
mean so we have some inputs here a
linear module of weights some nonlinear
activations to give us this hidden
representation another linear mapping
and then either directly to the output
or some puppet non-linearity and
basically another way of thinking about
why this is useful is that the outputs
of one layer are the inputs to the next
and so it allows us to transform our
input through a series of intermediate
representations and the hope is that
rather than trying to solve the problem
we're interested in directly an input
space we can find this series of
transformations the render our problem
simpler in some transform representation
so again I think this was covered
towards the end of those previous
lecture but if you think back to what's
going on with basis functions it's a
similar kind of idea so this is probably
that the simplest example that can
exemplify that so it's kind of simple
XOR task so
let's imagine that I have four data
points living in 2d a B C and D and a
and B are members of class 0 C and D are
members of class 1 now if I just have a
single linear layer plus logistic
there's no way that I can correctly
classify these points there's no there's
no line I can draw that will put the
yellow B the yellow point to one side
and the blue points on the other now
let's think about what we can do with a
very simple Network as I've drawn here
so we're just gonna have two hidden
units and so let's imagine that the the
first in unit has a weight vector
pointing this direction so in terms of B
its outputs these will be 0 in this red
shaded region and one here and then the
second hidden unit will have a slightly
different decision boundary it'll be
this one so it'll be 0 here and one here
and now if we ask ourselves ok in this
space of hidden activities if I rewrite
the data fight if I plot it again which
I'm doing down here
what does my classification problem like
in this new space so let's go through
the steps of that so point a had one for
the first hidden unit and 0 for the
second so it would live here point B
same again 1 and 0 also lives there
Point C has 0 for the first in unit 0
for the second it lives here and then D
has 1 and 1 so it lives here so this is
the representation of these four data
points in the input space this is the
representation in this this first hidden
layer and so in this space the two
classes now are linearly separable and
so if I add an additional linear plus
sigmoid on top of this then I'm able to
classify these two point B this data set
correctly and so this is again it's a
very simple example but I think it's a
useful motivation for why having a
hidden layer gives us additional power
actually looks like there's a couple of
seats free I see a couple for extending
good
if you want to take a second to sit down
if that's easy for you there's a couple
down here at the front and the second or
so here's another problem of a similar
flavor but slightly less travel so if we
now have the setting here where the data
from different classes live in these
quadrants then just two hidden units on
their own won't cut it but it turns out
that with 16 units you can actually do a
pretty good job at carving up this input
space into the four quadrant and there's
a link from the slide out it's something
that if you guys are not aware of it
it's nice to look at there's a a
tensorflow web playground that basically
lets you take some of these very simple
problems in your browser and play around
with different numbers of Units
different nonlinearities and so on and
itself will typically train on these
problems in a few seconds in it even
looks very simple I think it's a really
nice thing to look at to refine your
intuition for what sorts of things these
models learn what the decision
boundaries look like and Academy to add
detail to your kind of mental picture of
what's going on so yeah when the slice
is shared I'd encourage you to take a
look at that and just kind of play with
some of these simple problems in the
browser to kind of refine your intuition
okay so we've seen that the power that
we can get for these toy problems I'm
now going to go through I guess I'd call
it a sort it's not quite a proof but a
visual intuition pump if you will for
why neural networks with just one hidden
layer can still be viewed as universal
function approximate is and this is one
of those ideas that was arrived at by
several people more or less concurrently
one the kind of well-known sort of
proposes a proof of this was a guy Chu
Benko from 89 and that the papers are
linked here there's also again in terms
of the hyperlinks there's again some
nice interactive web demos one of them
in Michael Nielsen's web become deep
learning that
I'd recommend you take a look at and
going a little beyond the scope of this
class it turns out there are interesting
links along these lines to be made
between neural networks and something
called Gaussian processes they're not
going to be covered today but again I'd
encourage you to take a look if you're
interested okay so what what is our
visual proof going to be the with enough
hidden units we can use a neural network
to approximate anything so let's begin
by just considering two of our linear
plus sigmoid units here and let's
imagine that we arranged for the weight
vectors to point in the same direction
or maybe we'll start off with just a
scalar case so the only difference
between unit 1 and unit 2 is the bias so
that's the kind of offset of where the
sigmoid kicks in and then let's imagine
okay what happens if we take this pair
of units and we we subtract them from
each other what does that difference
output look like and it turns out it
looks something a little like this this
kind of bump of activity Y well over to
the far left both these units is 0 so
the the difference is 0 over to the far
right the upper buddies answers 1 so
they cancel and then in the middle we
have this this little bump and so by
having this pair of units were able to
create this this bump here which is a
lot like a basis function right so let's
imagine that we want to use a neural
network with a hidden layer to model
this gray this arbitrary gray function
here one of the ways we could do it it's
probably not the best way but just as a
kind of proof to show it can be done is
you could imagine now that I've got
these little bumps of activity I can
arrange for that offset to light
different points along this line and I
can also scale the but a multiplicative
scale on this so the idea is through
pairs of units we can kind of come up
with these little bumps and if we think
of what the sum of all these bumps look
like if I have enough of them and
they're narrow enough then it starts to
look like this gray curve that we're
trying to fit so the Mobile's we have I
either the bigger the
the hidden layer the more accurate our
approximation and so that's the kind of
sketch proof for 1d in 2d this same
sorts of ideas apply except we now need
a pair of hidden units for each
dimension of the input so it's hard to
visualize in dimensions beyond two but a
similar sort of thing would apply in 2d
where we if we have four neurons we can
build these little towers of activity
that we can kind of shift around and
again the same idea would apply so
hopefully this is convinced to you that
with enough units we can approximate
everything although it doesn't sound
very efficient and you'd hope that
there's a much better way of doing that
and it turns out that there is so now
that we've seen what we can do mmm I
don't think so you're you're not taking
the area under each bump you're just
taking their kind of magnitude of the
function so there's dump your question I
think I may listen this is your question
okay okay I see any any more questions
before we move on okay so now we're
gonna start to think about deeper
networks so we've seen what we can do
with just a single hidden layer and we
do have this Universal approximation
property but we've also seen that it is
kind of a horrible way to do it it needs
many many units and it turns out that as
we add depth things get a lot more
powerful and we've become
a lot more efficient and again I'll give
a kind of a reference to a paper that
has the full proof but for the class
I'll try and give you a sort of more
visual motivation for how you can see
that that is something that happens and
again to kind of motivate what you were
what you get if you allow these very
deep transformations again coming back
this idea of rather than trying to kind
of go from inputs to outputs in one go
it allows us to potentially break it
down into into smaller steps so you know
cartoon from vision might be rather than
going straight from a vector of pixels
into some kind of scene level analysis
maybe it's easier if in the first stage
of transformation we can extract the
edges or into the edges from an image
from those you can start to think about
composing those edges into say junctions
and small shapes from there into part
there aren't objects and then there
enter into full scene so breaking down
these complicated computations into
smaller chunks in the in the second half
of the section will kind of flip to this
what I'm calling out a more modern
compute graph perspective and there will
kind of really start to see the creative
designs that you can do in these very
large networks and I'll also throw in
just a couple of examples of real-world
networks that you can see what I mean
when I when I say that the structure
these things can get very elaborate okay
so yeah what I'm gonna do for this slide
in the next one is just go over how we
can see the benefits of depth you can
ignore this is my slide from last year
when there was an exam but this era of
things cause what they say you know a
minute worried so here's the
construction so if we imagine taking the
rectified linear unit that we we saw
previously so one of these is just zero
if it's if neighbors blow zero zero it's
linear about that and imagine we take
another one of these rectifiers and
essentially flip the signs of the
weights and biases so it's kind of V
converse what this gives us oriented it
around the origin in this case is a full
rectifier and so in 1d this has the
property that anything we build on top
of this will have the same output for a
point of plus X as it will at minus X so
it's kind of it's mirroring where you
can imagine it as kind of folding a
space over so yet multiple points in the
input mapped at the same point in the
output and so this letters have multiple
regions of the input showing the same
functional mapping will kind of extend
that from 1d into 2d here so imagine
that I have two pairs of these full
rectifiers so that would causes you four
hidden units in this layer in total one
of the rectifiers
is arranged along the x axis and one
along the y axis and so what it means is
that any any function of the output of
these is replicated in each of these
quadrants and so one way you can think
about what these rectifiers are doing is
if I were to take that 2d plane and kind
of fold it over and then fold it back on
itself functions that I would map on
that folded representation if I unfold
it it kind of fall back into the
original input space so that's the kind
of underlying intuition you guys okay
yeah and so this is from this paper from
2014 by wonderful Pascal oho and Benjy
and what I just described is the sort of
basic operation they use to come up with
this interesting proof about the
representational power of deep networks
so I'll kind of step through this this
diagram fairly quickly again if you if
you're interested then it's a nice paper
and fairly easy to read but it's just
too too many details to go through today
so as I said we imagine by applying
these pairs of rectifiers what you end
up with is this folded space I can on
the outputs of that so
I can apply a new set of units on top of
that which would end up kind of folding
this space again and so what we end up
with any decision bound we have in the
final layer as we kind of backtrack so
going through this unfolding gets
replicated or distributed to different
parts of the input space so probably the
most helpful thing to look at is this
this figure here so if we have a network
arranged like this in this output layer
if we have a linear decision boundary
when we unfold that we end up with four
full boundaries one in each of the
quadrant represented here so we've gone
from two regions that we can separate
here to eight regions that we can
separate here if we were to unfold that
again then we end up with 32 regions so
the kind of the high-level take home
from this is the the number of regions
that we can assign different labels to
increases exponentially with depth and
it turns out it only increases
polynomial e with a number of units per
layer so so all that's being equal for a
fixed total number of neurons there's
potentially much more power by making a
narrow deep network than there is in
having a shallow wide Network you know
the details of that will depend on your
problem but that's one of the intuitions
for why adding depth is so helpful it's
guess so it's hot on to these questions
so I say the state of theory in deep
learning alone is is know in their world
we'd like it to be so there aren't of
good rigorous demonstration of that
empirically in a lot of problems what
you'll find is in a few
you try and tackle something with a
fixed budget of Units then in practice
often you will get better empirical
performance by adding a couple of hidden
layers rather than having one very wide
very wide one but it's also problem
dependent yeah I think there's another
question somewhere over there okay does
that answer your question
sure yeah don't worry but my pastor yeah
I just encourage you to read the paper
because it's it's really nicely written
in to the extent that yeah this works
for you as an intuition pump it's worth
taking the time to kind of go through
that argument and understand it okay so
now I said we're gonna switch gears a
bit and move from this what I would say
is a kind of more traditional style of
depicting and thinking about neural
networks and in this we sort of bundle
in our description of layers the
nonlinearities and move towards this
kind of more explicit compute graph
representation where we have separate
node for our weights and we separate out
separate out the linear transformation
from the nonlinearities and this is more
similar the kind of thing that you'll
see if you look at say visualizations in
tents aboard so these are kind of
isomorphic to each other and to these
equations here I'm just I just put
together an arbitrary graph just to kind
of highlight this so we have input to a
first and layer with a sigmoid the
outputs of this go to a secondhand layer
which I decided to pick a railing for
there's another pathway so that yeah
this one is really there's another
pathway coming through here and then
they combine at the app
that exactly the same thing here I'm
just kind of adding these additional
nodes and it seems like we've kind of
made this one looks more complicated
than this one but there's a reason for
kind of breaking it down like this which
will kind of move on to in the next
sections and that's the idea of kind of
looking at these systems just as kind of
compute graphs from modular building
blocks and the nice thing is if we if we
represent and think about our models in
this way then there's a nice link into
software implementation so we can kind
of take a very object-oriented approach
to composing these graphs and
implementing them and for most of what
we need to do there's a very small
minimal set of API functions that each
of these modules needs to be able to
carry out and you can basically have
anything as a module in your graph as
long as it can carry out these these
three functionalities so and well we'll
go through them and in the subsequent
slides but just to kind of signpost them
there's a forward path so Harry go from
inputs to outputs there's a backwards
pair so given some gradients of the loss
we care about how do we compute those
gradients all the way through the graph
and then how do we compute the prior
updates and this is just putting this up
here this is what the compute graph for
Inception before looks like and I just
wanted to kind of put this up the to
ground why it's important to have this
kind of modular framework because you
know for the for the small networks that
I was showing you initially it kind of
doesn't matter how you set up your code
you could you know you can drive
everything by hand you know maybe you
want to fuse some of the operations
yourself just to make things efficient
but once you have these massive massive
graphs then keeping track of that in
your head or by by hand is just not
really feasible and so you need to have
some automated way of plugging these
things together and being able to to
deal with them so this I think it's not
state-of-the-art anymore that's a kind
of sign of how the fields moving but as
of around this
last year this was a state-of-the-art
vision architecture it's still pretty
good this is another example this time
from deep reinforcement learning and
again and just kind of putting this up
there to give you a sense of what sorts
of architectures we end up using it in
real-world problems and the sorts of
somewhat arbitrary topologies that we
can have depending on on what we need to
do the details of this don't matter too
much but I I think towards the end of
the RL course Hado might cover some of
this stuff ok so the the next section
we're going to cover learning and it's
probably going to be one of the more
math heavy sections and I guess I'll
I'll cover up the material but I usually
find it's not super productive to be
very detailed with mathematics in a
lecture but you can kind of refer to the
slides for details afterwards so what is
what is learning as I said it's very
simple we have some loss function
defined with respect to our data and
model parameters and then learning is
just using optimization methods to find
a set of model parameters with minimize
this loss and typically we'll use some
form of gradient descent to do this and
there'll be a whole lecture that kind of
covers various ways of the optimization
I guess something else that I'll add
just cuz it's starting to become popular
in source is something that I'm working
on in my research of the moon so there
are great in free ways of doing
optimization so kind of 0th order
approximations to gradients or
evolutionary methods and again I guess
one of those things were you know these
things coming waves of fashion day they
were kind of popular in the early 2000s
they've fallen out of favor they're
actually appearing again particularly in
some reinforcement learning contexts
where you have the situation that sure
we can kind of deal with great in our
models but depending on the data that we
have available so in
wasn't learning the data you trained on
depends on how well you're exploring the
environment it might be that there just
isn't a very good gradient signal there
and so we won't cover it today I don't
know if James will touch on a bit on his
lecture but it's just useful pretty
aware of that there are these sort of
gradient free optimization methods as
well and depending on your problem that
might be something useful to think about
and at least be aware of so in this
section I'll start by doing a kind of a
recap of some calculus and linear
algebra will recap Green percent and
then we'll talk about how to put these
together on the compute graphs we were
just discussing with automatic
differentiation is something called
modular backprop and what I'll do at the
end of the section is we can kind of go
through a more detailed derivation of
how we do a set up if we wanted to say
do classification of endless digits with
a network with one hidden layer so just
a kind of very cruel example but once
you've got that it kind of generalizes
to all sorts of other things that you'd
want to do so there's two concepts that
it's useful to have in mind they're kind
of objects that allow us to write some
of the the equations more efficiently
and to kind of think about these things
in a slightly more compact way so one of
them is this notion of a gradient vector
so if I have some scalar function f a
vector argument then the elements of the
gradient vector which is denoted here
with respect to X are just the partial
derivatives of the scale output with
respect to the individual dimensions of
the vector the other concept that's
going to be useful in terms of writing
some of these things down concisely
is the Jacobian matrix and so there if
we have a vector function of vector
arguments then the Jacobian matrix the
NF element of that is just the partial
derivative of the nth element of the our
vector with respect to the F element or
the input vector
and in terms of gradient descent what
does that mean well if we have some lost
function that we want to minimize then
essentially we were just kind of
repeatedly doing these updates where we
take our previous parameter value we
compute the gradient and we can do this
either over our entire data set or which
would be kind of batch or a kind of
subset of the data which be mini batch
or something that we end up calling
online gradient descent which is if we
take one data point at a time we just
compute the gradient of our loss with
respect to that data and then take a
small step scale by this learning rate
eater in the direct descent direction
and then we end up repeating this in
what I'm gonna talk about the cone
slides I'm gonna operate in the
assumption that we're doing it online it
doesn't change it much if we do batch
methods it's just easier to represent if
we just have one data case I have to
think about and I'll cover this a couple
of times later as well but it's just
worth stressing that the choice of
learning rates are the step size
parameter ends up making a big
difference but how quickly you can find
solutions and in fact the quality of
solutions that you end up finding and so
that's something that will touch on when
we talk a bit about hyper parameter
optimization and moving beyond simple
gradient descent there's a lot more
sophisticated method so things like
momentum where you kind of keep around
gradient from previous iterations and
blend them wood grain from the current
iteration there's things like rmsprop or
atom which are adaptive ways of scaling
some of the step size as long different
directions and I think James is going to
go into a lot more detail about that in
a couple weeks time okay
so if you think that too kind of high
school calculus and in particular the
chain rule so let's start off with this
nested function so Y is f of G of X and
so if we ask okay what's the derivative
of Y with respect to X well we just plug
in the chain rule so it's the derivative
of F with respect to G considering
g-tube its argument and then the
derivative of G with respect to X so a
similar scalar case scalar output scalar
input if we make this multivariate so
now let's imagine that our function f is
a function of multiple arguments each of
which is a different function G 1
through m of X and again were interested
in the same question what's the the
derivative of Y with respect to X well
we sum over all these individual
functions and then for any one of them
it's again just the chain rule from
above so the partial of F with respect
to G I and then the partial of G IR with
respect to X so we basically for each
half of nesting we take a product along
a single path and then we sum over all
possible paths to get the total
derivative and well basically just gonna
take these concepts and scale them up so
that we can apply them to these compute
graphs and the only thing to be aware of
an hour I'll have mentioned this again
in a second there's a couple of
efficiency tricks that we should be
aware of so if there are junctions as we
traverse there's opportunities to
factorize these expressions and that
becomes particularly important if you
have a graph with a lot of branching in
its topology so let's let's take a some
arbitrary if you graph as an example
again so
it's a little dense when I write it out
but hopefully this will kind of like
carry over the point so so imagine we
have some function mapping from X to Y
and the way this is going to be composed
it's gonna be some G of F F is going to
be a function of its two inputs E and J
and then E is this kind of nested
sequence of functions or operations all
the way to X and similarly J so if I
take what I just set up here and ask
okay what's the derivative of Y with
respect to X then we take the product
along these two paths as I say so a
through G and then there's also this
path through here and so we get these
two expressions down here what I was
saying about kind of some of the
efficiency tricks is you'll notice
there's some common terms towards the
end of this expression and this
expression and so we could actually
group these together factor those out of
that sum in the scalar case it doesn't
matter too much but we'll move to the
the vector case and more elaborate
graphs you'll see why it's important
essentially if there's a lot of
branching and joining then we have to do
these sums over there kind of
combinatorially many paths through the
graph for the mapping that we're
interested in the other point that is is
worth mentioning is so if you look at
the literature on automatic
differentiation you might hear a couple
of different terms so there's something
called forwards mode automatic
differentiation and something called
reverse mode automatic differentiation
and that just that's really referring to
when were computing these expressions do
we compute the product starting from the
input working towards the output or do
we work in Reverse and the difference
between the two is to do with what sorts
of intermediate properties that we end
up with so if I work from the input
towards the output so if I can
this product see from the inputs to the
outputs then my intermediate terms are
things like da/dx if I then compute this
then I basically would end up with DB DX
DZ DX so in forwards mode we get the
partial derivatives of the internal
nodes with respect to the inputs which
is actually not super useful for what we
want to do it it's great if you want to
say do sensitivity analysis so if I want
to know how much changing a little bit
of the input would affect the output
this is exactly what we want to do and
that can be useful in deep learning if
you want to get a sense of how models
are representing functions or which bits
the input are important but it is not
useful for learning however if we
Traverse this in the opposite direction
so from outputs towards inputs then we
end up with two terms that are
derivatives of the output with respect
to the internal nodes and it turns out
that that's exactly what we need for for
learning so so it's interesting kind of
explaining this stuff because on the one
hand it's all kind of trivial it's you
know it's basically the chain rule you
know you'll have seen this in high
school so it's kind of one of these
simple ideas that actually had quite a
big impact so even though it's kind of
obvious when you look at it like this in
terms of the impact on efficiency when
you're computing gradient updates for
neural networks it makes a big
difference organizing the computation in
this efficient way and I think that's
one of the reasons why when backprop was
introduced it had such a big impact even
though at hardest a kind of
fundamentally simple method and also
what we'll see as we move on to kind of
the more vector calculus how to things
it all looks pretty trivial if we're
dealing with scalars but once we move
into large models then B again we'll see
why the ordering makes difference so
yeah essentially reverse mode or my
differentiation a clever application of
the chain rule back prop that all the
same thing
so basically in the backwoods pass
through the network what we're going to
want to do is compute the derivative of
the loss with respect to the inputs of
each module and if we have that then
that kind of goes into part of this
minimal API that I was describing those
three methods that if our modules
implement those then we can just plug
them together however we like and go
ahead and train the other thing that's
worth mentioning is interesting is that
this idea doesn't just apply to things
that you might consider to be simple
mathematical operations you can actually
apply this to the entire compute graph
including constructs like for loops or
conditionals and so on essentially we
just backtrack through the forward
execution path so if something has a
derivative we take it but if in the case
of an if Clause then we essentially
there's multiple execution branches that
we could have ended up following when we
work backwards we just need to remember
which branch we followed going forward
and that's the one that we we use when
we're going in the reverse direction so
essentially we can take an entire
computer program more or less and
everything we can apply this automatic
differentiation to and that's one of the
powerful things that tends to float does
for you it allows you to write these
I'll retreat in few graphs and then when
it comes time to learn it does the hard
work of doing all this backtracking for
you and kind of okie canoeing in terms
of how the gradients flow there's a
couple of things that you need to be
aware of so in most implementations of
this you need to store the variables
during the forward pass so in very big
models or sequence models over very long
sequence lengths this can lead to us
requiring a lot of memory but there are
also clever tricks to get around that so
there's a nice paper that I linked to
here which is one way of being memory
efficient and it's essentially boils
down to being smart about caching States
in the schema in the forward execution
so rather than remembering everything
you can think
it's like every few layers say we
checkpoint then in the back Pro pass
rather than having to remember
everything or the other thing would be
to kind of compute everything for
scrapped we can find the most recent or
the the closest cache state and then
just do a little forward computation
from that to get the states we need to
evaluate the gradients and yeah that
most of this is taken care of
automatically by things like tensor flow
and even I think this memory fish and
stuff is probably going to find its way
into the core tensor flow code probably
the next release or two so a lot of
these things you on a day to day basis
you don't need to worry about but again
I think it's always useful to kind of
know what's going on under the hood in
case you are doing something unusual or
if you are running into some of these
problems okay so in this cartoon here
what I'm showing is how those different
pieces fit together and the sorts of
things that looks like once we're in a
more realistic setting so we have vector
input SPECT outputs and as I said
there's these three API methods that as
long as we have some sort of
implementation of these then we can plug
together these arbitrary graphs of
modules and figure out the outputs given
inputs figure out the derivatives we
need to figure out the parameter update
so what are they the first one is what
I'm calling the forward pass so this is
just what's the output given the input
so through here and then there's two
methods that involve gradient so one
which I call the backward pass is we'd
like to know the gradient of the loss
with respect to the inputs given the
gradient of the loss with respect to the
output and so it turns out that what
does that look like well thinking back
to the chain rule slides from slides ago
if I want to think about this element
wise then
the gradient the lost with respect to
the I input is just the sum over all the
outputs of the gradient of the lost with
respect to each of those outputs and
then the gradient of those outputs with
respect to the input and if we want to
use our vector matrix notation then it's
the product of this gradient vector with
respect to the Jacobian of Y so this is
just a kind of compact way of
representing things similarly to get
parameter gradients or that's just the
derivative of the loss with respect to
the parameters which is then the sum of
all the outputs maduro to the loss with
respect those outputs the derivative
those outputs with recta parameters and
then these are obviously evaluated at
the state that it was 1 we're doing the
forward pass and that that's why I was
saying before that we need to keep these
states around because typically these
derivative terms will involve an
expression that involves what the
current state is so yeah these are kind
of compact ways of representing this in
practice we we actually don't if you
were to write these models yourself you
probably wouldn't want to form the full
Jacobian in these cases just because the
jacobians tend to be very sparse so if
there's there are many inputs that might
not have an influence on an output and
so many elements of the Jacobian are
often 0 but it's useful notationally
particularly if you kind of go back and
forth between this and the subscript
notation if you ever need to kind of
derive how to implement an R between new
module for yourself say if you have some
wid function there's and supported by
tons flow so yeah but that's more or
less what I just said so we have these
these methods that we we need to
implement and we chained the forward
passes together so how would we operate
this we'd we'd call the forward method
for the linear unit given the parameter
an input that would give us some output
the forward method of the relu the
forward method linear
method of the softmax and then we'd get
a loss and then we just call be
backwards method zombies to get our
derivatives of outputs with respect to
inputs and derivatives with respect to
parameters we apply the gradient that we
get from the parameters to take a small
descent step and then we just iterate
that so what I'm going to do in the next
couple of slides is go through what some
of those operations look like for these
building blocks and by the end of it
we'll have everything we need to do to
put together something like endless
classification with cross-entropy loss
and a single table later okay so the
forward pass for a linear module we're
calling for the Binnington class is just
given by this expression here so the
vector output is a matrix vector
operation plus a bias again I say in
these derivations is often useful to
kind of flip back and forth between
matrix vector notation and subscript
notation so this is just kind of
unpacking what the nth element of this
output vector is so we can compose the
relevant bits of the Jacobian that we
need so what do we need we want the the
partial of Y with the Spectras inputs
the partial of Y with respect to the
bias and the partial of Y with respect
to the weights and we get these
expressions this is what I was saying
before so this Kronecker Delta here most
of the elements of this Jacobian are
zero because if there isn't oh if
there's not a weight involved in this in
this particular but if a particular
weight isn't involved in producing a
particular output then there's
absolutely zero and so it's quite sparse
so armed with this we can come together
and get our backwards pass so what is
that it's just given by this expression
so we kind of plug in
these things that we've already derived
so if we have the as I said in the back
was possibly assume that were given the
gradient of the output with this of the
grading of the loss with respect the
output and so we just have this matrix
vector expression here similarly for the
parameter gradient if we kind of churn
through this this math then we we get
this outer product of the gradient
vector with the inputs and there's a
similarly simple thing for the biases so
armed with that we have everything we
need to do forward propagation backward
propagation and parameter updates for
the linear module the rally module is is
super simple so there's no parameters so
that the forward pass is just this max
of 0 and the input so it's our kind of
floor at zero and then the backward pass
is also simple is this kind of element
wise comparison so if the output was
above zero then the gradient with
respect to inputs is just 1 were in that
the linear pass through if the output
was below zero then there are no
gradients the softmax module is a little
trickier to derive from the omits for
but it's basically still simple calculus
so if we recall one that was the the
enth output is just this exponent of the
sum total and input normalized by that
same expression over all units we can
plug these in derive our Jacobian
element and then similarly we can plug
them in the backwards pass I've actually
skipped the derivation for this and I
think for the next one in the slides
just collecting that's going to come up
as something on your assignments but in
a later version of the slides I'll
update it with that the solution in
there okay
so second I don't think so so yeah good
question um I think I usually I usually
do a greater than zero so if it's equal
to zero then I treat the brain at zero
it's not it's not well-defined in
practice you can kind of assume it it
doesn't happen much but I would just set
the define the gradient at zero to be
zero but it actually doesn't matter too
much just because numerically or
extremely unlikely to hit something
that's exactly zero yeah so the final
part of this was the the loss the loss
itself and so again there's there's no
parameters in the forward pass we just
this is our definition of the loss when
we take derivatives we end up with this
expression and you might look at this
and be a little bit worried particularly
that with with this kind of expression
and X can vary a lot then you might
worry that if X is very small we might
run into numerical precision issues and
in fact actually that is a real concern
so what people typically do is use this
kind of compound module so it's softmax
plus cross-entropy and you'll see that
in terms of flow I think this
implementations for both but
you know unless you have your own
special reasons you probably should use
that the softmax plus cross-entropy so
it basically combines both the the
softmax operation and the cross and
pre-loss into a single operation and the
reason for that is if we do that and
look at the the grades that we get out
then it's this much more stable form
here so if we kind of go back what are
we done we had this graph that we wanted
to do learning in safer does it
classification we've gone through and
for each of these module types we
figured out what we need to do to kind
of propagate forwards what we need to do
to propagate backwards and what we need
to do to come up with the parameter
derivatives and armed with that we're
ready to go and we can plug together
things in whatever order we like so in
terms of learning we just kind of
iterate through getting an input and a
label running forward propagation
running backwards propagation getting
parameter updates applying the parameter
updates and cycling and the nice thing
is if we'd you know written this from
scratch ourselves and we wanted to try
adding in you know an extra hidden layer
then it'd be very simple we we we just
kind of put another one of these mod
modules here change the call sequence
and and we're good to go so once we have
those in place it's then very easy to
explore different topologies if I wanted
to California come up with some crazy
non-linearity instead of the rail ooh
then I am free to do so I would just
implement a module that has those three
API methods and and everything should
just work in this next section I'm going
to kind of do a quick tour of what I'm
calling a module Zoo so we've seen some
basic module types that are useful so
linear sigmoid relu softmax just gonna
go through some of the other operations
that you might see so there's actually
two main types of linear model the first
is the kind of simple matrix
multiplication that we've seen already
convolution and deconvolution all
laser also linear I'm not gonna talk
about those but Karen's going to cover
those in the next lecture on commnets
there's a couple of basic sort like
element wise operations so addition and
putting wise multiplication some group
operations and then a couple of other
nonlinearities that are worth knowing
about also in the slides I like how this
is sort of fairly inexhaustible writing
your possible activation functions you'd
wanna use typically the ones that we're
gonna cover today will will be in the
vast majority of thing you see but it's
also worth remembering but if you know
if you have a particular problem or if
you feel like you need to think
creatively about it your you have
license to kind of but pretty much
anything you want in these models as
long as they're differentiable you're
absolutely fine and even if they're not
perfectly differentiable you might still
be able to kind of come up with
something that's usable so yeah I'll go
through these relatively quickly so if
we want to do addition then the forward
pop method just obviously simple vector
addition the back pop method also
relatively straightforward there's no
parameters that there's no gradient
update
similarly for multiplication so element
wise multiplication this kind of thing
is kind of useful in as I saying into
like gating situations where depending
on some context say you might want to
propagate some parts of the state and
not others also comes up in modulation
or things like attention so if I want to
emphasize some parts of my
representation and relative to others
that's are elsewhere you'd see this kind
of operation there's a couple of kind of
group wise operations so summing for
instance so if we have a sum then the
gradient is kind of gets distributed the
back for grading gets distributed across
all the elements if we have a Mac so you
might see this in max pooling in
commerce for instance then basically
the for the back prop if the element was
not small then the gradient just passes
through otherwise there's no gradient if
we have a switch or a conditional one
way of representing it as I was saying
is with this kind of element-wise
multiplication and we basically just
need to remember which brand to which
switch was active that gets back propped
everything else gets set to zero here's
a couple of slight variance on
activation function we've seen already
so the tan H is basically just a kind of
scaled and shifted version of the
sigmoid so it's at 0 its 0 and it
saturates at 1 and minus 1 if you were
to build a feed-forward Network there's
some in potential in some cases there's
advantages to using 1080 over sigmoid in
that if you initialize with small
weights and small biases then you
basically get to initialize in this
linear region here and in practice it's
often nice if you can initialize your
network so that it it does a kind of
simple straightforward function rather
than kind of risking being in some of
these saturated regions where the
gradients are going to flow for similar
kind of gradient flow reasons rather
than using the rail ooh this would be
kind of zero here another thing that
people sometimes use is to have a very
small but nonzero slope in this negative
region and again it just kind of helps
with gradient propagation in the you no
longer lose all gradient if you're below
zero and that could also can be useful
I'd say that this actually on those
things were probably if you it's not a
default choice but maybe it should be in
that in my experience is often better to
use this than it is to use a rail ooh
that said I often don't use it just to
kind of keep as few moving parts as
possible because you know there are
design choices that you'd want to make
here so I if there was something that I
really really heard about getting the
best performance out of I probably start
to explore some of these variants but
day to day I tend to kind of stick with
the simple choices just because then
there's fewer few things to keep track
of in terms of mental overhead we've
already seen content to be lost and so
there's just another simple one so if
we're doing say regression problems then
squared error is a common choice yeah I
didn't have on the slides way I can add
it later just exciting again worth
noting
so square error is very common in
regression problems again in practice I
would probably try squared error if I
had this but I'd probably also try other
norms as well so in particular l1 one of
the problems with squared error is if
you have outliers or operations that for
whatever reason have to be way happen to
be way off mark that you can get
extremely large gradients and so
sometimes that can make learning
unstable so again in all these cases
there's sort like reasonable defaults
that are sensible to start with but it's
also useful to know kind of okay what
would the design choices that I might
want to revisit be if if things for
whatever is not working or if great
Nizar kind of blowing up and actually
that brings me on to this next section
where what I'll do is kind of go through
some sort of high-level practical tips
in terms of things that might be useful
for you when you're dealing with these
models and kind of good things to to
bear in mind this came up a bit in the
break as well it's sort of the the field
at the moment there's definitely a kind
of scarcity of strong theoretical
statements we can make and so
unfortunately that kind of means that a
lot of deep learning is still a bit more
of a dark art than it would be ideal so
there are some things that you can kind
of plug in and just rely on but there's
also a lot of trial and error and it's
some pieces where you kind of have to do
more of a an interrogated loop of okay
is this model working if so great if not
okay what might be going wrong and a lot
of getting good at this kind of stuff is
refining your intuition for if something
isn't working
what might the causes be
to quickly diagnose that and also what
sort of things you could do to fix that
so let's go through these so one problem
that you can run into is overfitting so
you get very good loss on your training
set but you don't generalize well so one
thing you can do there and this was kind
of probably in the early days is early
stopping so you basically just rather
than training to kind of push your loss
all the way to zero you kind of in
parallel or evaluating on some
validation set and you stop once say
that the loss in your validation step
starts to go up that's one method some
something else you can do and you know
you can do all these in combination
there's something else called weight
decay and it basically penalizes the
weights in your network from becoming
too big and one intuition for why this
might be helpful is if we think about
something like the sigmoid with small
weights we're going to tend to be in
this more often in this linear region so
our kind of functional mapping will be
closer to linear and so potentially
lower complexity what one thing to
mention actually about weight decay is
that it doesn't have as much an effect
on reloj units as it does on some of
these others so it may be a less useful
form of regularization for your relu
layers it'll still obviously have an
effect on the output but with Ray lose
you can get a new scale all the weights
down and you still have the same set of
decision boundaries so it doesn't quite
regularize where I lose in the same way
something else that you can do is I said
he add noise and this kind of brings us
on to things like drop out and there's a
couple of ways of interpreting what's
going on so you can add noise to your
your inputs which you could also think
of as a form of data augmentation you
could add noise to your activities you
can add noise to your parameters you can
kind of
mask out some of the activities of units
within layers and yet in terms of the
like what is this doing well you can
kind of think of it in a couple
different ways one is that it prevents
the Mott Network from being too reliant
on very precise conjunctions or features
so you can imagine that you know that'd
be one way to memorize your data set if
you kind of have very precise activities
that depend on the very precise pattern
that you see in a particular input you
can also view it as a kind of cheap way
of doing ensemble assay model multiple
times adding different amounts of noise
then that's some what you might spend
that to have somewhat similar effects to
if I had an ensemble of similar models
and so you can also kind of tie that
into some ideas from so phasing
statistics are rather than say have a
single model you have a posterior
distribution over parameters and adding
noise in a hand-wavy sense is a little
bit like looking at a local plastic
proximation and then probably the best
known of these is is dropout and so in
this you sort of randomly set a fraction
of activities in a given layer to 0 and
at testing time you kind of need to
rescale things by the proper fraction
because at test time you're gonna have
everything active so this would be
typical magnitude of the activities in a
given way are going to be higher it's
also worth noting that sort of drop out
it's one of those things that kind of
you know peaked in popularity I guess
around like 2012 or so it's not used as
much these days as it used to be I think
one of the reasons for that is the sort
of introduction of normalization so I'll
talk about that in a second but another
another factor that can be important in
terms of whether your models train well
or not is how will you initialize them
and yeah this can expect what I was
saying about you know the tonnage being
someone nice and that if you have small
weights then you can get to initialize
things in a more or less linear region
but
the beginning of training you want to
make sure that you have good gradients
flowing all the way through your network
so you don't them to be too big and you
don't them to be too small there's
various heuristics for kind of arranging
for this to be the case a link to a
couple of figures here so and for some
reason a lot of these are kind of named
after the first author of the proposed
these so there's something that protocol
Xavier initialization named after is a
burglar who's so deep mind I forget the
first name of hair but there's a
follow-on paper that the difference
between these two is that both trying to
say okay how should I scale my weights
and biases at initialization so that the
input to my normally era T's say have
some particular distribution so maybe 0
mean unit variance but differently in
these two is that the assumptions that
you might want to make if you're using
say a sigmoid unit are different from
those if you're using say a rectified
linear unit so yeah there's a couple of
papers here that you might want to take
a look at then there's this thing batch
norm which is used full extensively now
particularly in feed-forward networks
it's still not used as much in recurrent
models just because there's some
subtleties about how you'd actually go
about doing that and it's used
I'd say hardly at all in deep RL but
there's probably modifications to this
kind of idea that you could do a few if
you wanted to apply those approaches
there and it it kind of subsumes some of
the stuff in that you can think of it as
being similar to what we do in some of
these initialization methods but we also
continuously update to maintain these
properties so the idea is we'd like the
the inputs the some inputs to our units
to have a zero mean and unit variance
but for the reasons I described in terms
of initialization what batch norm does
is it kind
enforces that but it also introduces
some additional trainable correction
factors so that if it turned out in fact
I would rather have something that had
variance 10 and I've mean of one then
there's kind of scalings and offsets
that I can learn during training to help
that be the case but it all that's being
equal it kind of helps keep my
activities you know a reasonable regime
with respect to minorities and also with
respect to the kind of gradient scaling
that we get when we do back from another
nice benefit of Bachelor on that is I
think actually mentioned less often but
is is interesting and is perhaps part of
the reason why drop out isn't as favored
as much is that you you get a sort of
drop out like noise effect from batch
normalization and that in order to
enforce or to encourage these kind of 0
mean unit variance properties you look
at your local data batch and so just
because of randomization amongst the
cases that you get in a given batch from
the point of view of any one of those
data cases the contribution to the batch
normalization from the rest of the batch
members looks a lot like noise and so
that kind of gives you some some sort of
regularization effect anyway there'll be
a lot more about this in current lecture
on conducts another kind of area that's
important practice is how to pick good
hyper parameters so how do I know how do
I know what a good learning rate is if
I'm using dropout how do I know what
fraction units to dropout or how much
noise to add if I'm doing weight decay
and so on and we're still relatively
primitive and how we deal with this so
basically the idea is just to try many
combinations and kind of evaluate the
final results on some held out data set
and then pick the best but there are a
kind of a couple of kind of practical
tricks and some of these to it so if
there's lots and lots of hyper
parameters then the search space can be
huge so that's something that you might
worry about
for a long time people advocated grid
search so essentially for each hyper
parameter that you you care about maybe
kind of come up with some grid of things
to try and kind of systematically try
the cross base of all possibilities
turns out that in a lot of cases that's
actually not the best thing to do and
there's a nice paper by a box from
benzio which I've linked here and I've
taken this figure from it and this kind
of tries to illustrate why that might be
so depending on what the the sensitivity
of your model is to the hyper parameters
if you do grid sir you could very easily
miss these good regions just if your
grid happens to be poorly aligned with
respect to the reason is that useful so
they advocate and kind of empirically
demonstrated that this often gets better
results just doing random search so
rather than defining a grid for each
dimension you might define some sampling
distribution and then you essentially
just set a sample from that joint
probability space run run your models
and then a nice thing there is that you
can you get broader coverage of any
individual parameter value and there's a
better chance that you'll find a good
region that you can then explore more
carefully so I would say if you're if
you're faced with this kind of issue
then unless you have a good reason not
to don't do grid search to do do a
random search there's actually kind of a
lot of ongoing research in terms of ways
to get around some of these problems or
at least a kind of automate this search
process so there's some approaches from
kind of Bayesian modeling where the idea
there is if I could somehow form a model
of how well form a predictive model of
the performance of the models that I'm
training then I could be smarter about
figuring out which hydro parameter
values to try next there's also some
reinforcement learning approaches which
essentially there's some upfront cost
in terms of having to run training many
times but the hope is that I can
essentially learn how to dynamically
adjust these hyper parameters through
training so that if I then have another
instance of the same sort of learning
problem I can be much smarter about how
I treat that and then there's actually a
paper the I along with some other folks
would be mine published archived at the
end of last year which is this idea of
borrowing some tricks from evolutionary
optimization and a population of
simultaneous training models and
essentially the idea there is instead of
doing at a grid search or random search
let's say we initialize with random
search we're training everything all
together and periodically we look at the
training progress that each of the the
jobs know population has made and if
something seems to be doing particularly
poorly then we look for something that's
doing particularly well we copy its
parameters over and then do a small
adjustment to its hyper parameters and
then continue training and that lets us
do be kind of it's a nice combination of
Hydra parameter search and a little bit
of online model selection in that were
devoting more compute to the models that
seem to be doing better and also
exploring in regions of hyper parameter
space that seemed to be more promising
that she has another particularly nice
benefit in reinforcement learning so one
of the kind of hallmarks of many RL
problems is that the data distribution
that we deal with is is non-stationary
so you know if I'm a robot that's
letting to operate in the world that may
be you know the David distribution in
this room might be completely different
to the David distribution when I go into
the hallway and so it could well be the
case that throughout learning there it
the the hyper parameters that would
allow me to make the best learning
progress might be quite different and so
some of these methods like random search
just can't address that whereas the
population-based method that we propose
is actually kind of locally adaptive so
that's worth looking at it works super
well and a demine workhorse of like
using this
the vast majority of our experiments now
the downside is it's simple to implement
but it's a little resource-hungry in
terms of how much compete you're able to
access concurrently so if you're able to
run say 30 or 40 replicas of your
experiment in parallel then I this is I
think I said a clearly better way to do
hyper own search but yeah if you don't
have some Google's resources then it can
be trickier to kind of do that so you
might want to do these more sequential
methods so ya hit his just some kind of
rules of thumb but there's a much longer
list of this and exacting some of those
things that you just kind of build up
experience over time but a couple of
kind of easy things to do if you're not
getting the performance that you've
hoped for one is to sort check for dead
unit so you could say take a large
mini-batch and look at the histogram for
a given layer look at the histogram of
activities of units in that layer and
what you're looking for is basically you
know some units that maybe never turn on
so for whatever reason and maybe your
initialization was off or you went to a
weird letting regime but it might be the
case that say if you have rarely units
many of them are just never in that
linear region and so you have the
capacity there but it's actually not
useful for you and so I'm just getting
in the way
a similar diagnostic is it can be useful
to look at histograms of your gradient
say again visualized over a large mini
batch and again you're kind of looking
out for you know gradients that are
always zero in which case you're gonna
have from making any progress or very
heavy tailed grading distributions in
which case maybe there's some data cases
that are dominating or there's some kind
of numerical issues with your gratings
blowing up
something else is a really useful thing
to try is take a kind of a very small
subset of data or if it's an RL setting
if there's a kind of a simplified
version of your task I just try to try a
model on that simpler version of the
task and for a smaller subset you should
be able to get zero training error or
you know close to a depending on you
know
noisy labeling that kind of stuff but
the idea is if you're not seeing the
performance on the real world problem
you care about just as a kind of sanity
check scale back the size of your data
set and make sure that you can over fit
on a small amount of data and because we
could just get about ten minutes left
I'll go through this fairly quickly it's
a kind of research topic again from D
mind that relates to some of the stuff
we've talked about but I'm I'll leave
five minutes at the end for questions as
well so this is some work that was it's
from I guess a year and a half ago now
although what kind of stuff on going and
it was this idea that we called
decoupled neural interfaces using some
data gradients and basically the idea is
rather than running say our forward
propagation all the way to the end and
then back propagation although at the
end can we say midway through this chain
predict what the back propagated
gradients are gonna be before we
actually get them and it turns out that
you can do that you might ask why why
would I want to so there's two places I
think where it's useful one is if we
have is more a kind of I guess
infrastructure thing that we have
massive massive graphs and we want it we
need to do lots of most of computation
before we can do an update then if this
were model parallel say then essentially
the the machines holding this these
nodes would be waiting for the back prop
s to happen before they could do an
update after the forward pass so one way
is to kind of allow for potentially
better pipelining the other benefit and
that's partly why I kind of have this
graph here that's more of a sequence
model is there are some settings where
we actually don't want to have to wait
for the future to arrive before we
update our parameters so if I have a
sequence model over an extremely long
sequence or in the case of an and RL
agent you know it's kind of indefinite I
can't so I don't want to wait for an
extremely long time before I can run my
back
through time to get gradients and it
might not be might not be feasible right
now if what what people typically do is
they'll take a long sequence and they'll
chop it into chunks and they'll run
something called truncated back prop
through time and if you sit down and
think about what that's doing then it's
it's essentially assuming that outside
of the kind of truncation window the
gradient from the future are zero
because what we're just ignoring them
and so if you look at it like they're
the argument behind Cynthia gradients is
is kind of obvious you're basically
saying if I my default was to do
truncated back put through time which
implicitly makes the assumption that
gradients from outside the truncation
window are zero could I possibly do
better by predicting something other
than zero and the answer is probably yes
in most cases and so that's a kind of
good motivation for why it's interesting
there's a couple of papers that we
published on this now already and
there's a nice kind of interactive blog
post that you can you can look at here
if you if you want to hear some more so
you know that's it for today the next
lecture is going to be commnets with
Korean but yeah there's time some
questions now and if there's more
questions afterwards I'm happy to kind
of hand ran outside for a bit more than
we have time for yeah that's another
great question I said that's a kind of
another ongoing area research so the
sort of the fault of the moan is more
like you know kind of human driven you
briefed me optimization and that you
know I have some idea in my head of what
the kind of fitness of different
architectures would be and I kind of
prioritize trying those there's some
interesting work going on again using
some of these gradient free methods to
search over architectures so at a high
level this idea if I can start to build
a predictive model of how different
architectures might perform then I can
use that to automate the priority list
of what I should try next on the
population training side of things some
of the stuff that were
working on actually at the moment is
there are ways of adapting network
architectures online without having to
restart training so one example of that
there's a couple of papers on technique
service in a called net net or net
morphism and various other
transformations there I see so imagine a
mitem that I have some architecture and
I'm thinking would that architecture be
better if I were to interject an
additional hidden layer somewhere
I could just start training from scratch
but something else that I can do is take
something's thats been trained
originally and figure out a way to
inject an additional hidden layer in
there that doesn't change the function
that's been learned so far
but then after I've can added that
hidden layer I can then continue
training and potentially allowed the
model to make use of that additional
capacity and one one cartoon of how to
see I could do that is I could say
arrange to have an additional hidden
layer with say tonight's unit and
initialize them so that they're kind of
in Berlin ear region so it's it's more
or less a linear pass through so I could
take my previous model add in an
additional layer with the existing
weight matrix initialize the outgoing
wig matrix of that 10h layer to be some
kind of large values and that will that
will locally give me something that has
a very similar functional mapping as the
the network I start out with but now I
have the potential to learn additional
connections going from those ten each
unit so there's potentially ways of
doing this kind of architecture search
online and then there's model-based
approaches and then evolutionary methods
I'd say they're kind of the three main
ways of doing that
learners are you looking at kind of help
set performance are you looking at
convergence rates yeah it's a good
question
so I've mostly been thinking of this in
the context of reinforcement learning
and so they're sort of your test say is
your training set in a sentence yeah so
for kind of supervised problems then
yeah looking at it on a held out set
another thing that's worth mentioning
and again this is something that were
kind of actively working on at the
moment is you might not want to make
greedy decisions about that so a good
example is you know in supervised
learning it might be the so often it's
good to have a fairly high learning rate
initially and then to kind of drop it
down but one of things we noticed and
applying this to some of the supervised
problems is that you can if you kind of
look greedily you can appear to be doing
better by dropping their learning rate
earlier than you would in a nocturnal
setting because I kind of give you that
local boost and so something that again
this is appears to be less of a problem
in the RL settings we've looked at but
I'm saying that you probably want to do
as we extend these methods is think
about kind of performance metrics that
aren't just how well am i doing now but
kind of combining in some of that
model-based for looking things so not
how well am i doing now but given
everything I've seen about learning
progress so far how well could this run
or its descendants end up doing and kind
of use use a less greedy performance
metric way if there are no more
questions then thank you and yeah feel
free to ask because no salary
[Applause]