DeepMind x UCL | Deep Learning Lectures | 3/12 | Convolutional Neural Networks for Image Recognition

so convolutional neural network works
for image recognition so this is the
plan for this lecture I'm gonna give you
a little bit of background about
convolutional neural networks or as I'll
be referring to them henceforth conv
nets because that's a lot easier to say
so I'll give you a little bit of
background on continents and and sort of
the ideas behind them but crucially also
the history behind them because they're
really something that has developed a
lot over the past decade and then in the
second part we'll talk a little bit
about the building blocks of conv nets
we'll go into quite some detail about
the convolution operation and how it's
used in these neural networks and then
in the fourth part we'll sorry in the
third part will put these building
blocks together into convolutional
neural networks and I'll sort of show
you how they how they fit together and
the fourth part we'll look at some case
studies some some very successful
convolutional neural network
architectures that were developed in
recent years and that includes him some
more advanced building blocks as well
and then to wrap up the lecture I'll
hint at a few more advanced topics and
also talk a little bit about how
confidence might be used not just for
image recognition which is what we'll be
talking about today but maybe also other
applications other data modalities
things like that so let's start with
some background last week I don't know
who was here last week but so my
colleague Wojtek was here talking about
neural networks and I'm gonna recap very
briefly a diagram from his slide deck so
this is how we can visualize a neural
network so it's basically a sequence of
operations the sequence of computations
and data goes in at the one end and at
the other end we want to make a
prediction of some some variable that we
can extract from this data and we have
what's called the training data set
where we have a bunch of data points
with associated target values that we
would like the model to predict and then
we have a loss function indicated and
orange here which is going to measure
how well our network is predicting the
target values and we're gonna try and
change the parameters of our neural
network these are basically the weights
in the in the linear layers over here
and over here we're going to adapt those
to try and minimize the cross entropy
and we do that using gradient descent so
we do that using an algorithm called
back propagation which which Wojtek
talked about in detail last week so I'm
going to talk about image recognition
with the neural networks and so the
first question we need to ask ourselves
is how can we actually feed images into
a neural network because the neural
networks that Wojtek described last week
they take vectors as input thank you you
basically give it a series of numbers
and then it produces another series of
numbers at the output so it neural
networks operate on vectors so how
essentially can return images into
vectors this is an image that I'll use
as sort of a basis for all the the
examples that I'll be talking about so
we have a fairly simple image here with
you know some background and a tree in
the foreground is sort of one meaningful
object in this image so we want to feed
that image into the neural network how
do we turn it into a vector so that we
can do that an image is actually not a
vector it's a digital a digital image is
a two-dimensional grid of pixels so it
has a structure to it and it has a
topological structure and basically we
so we have this two-dimensional grid it
has a it has a height that has a width
and then for each discrete position in
the grid we record the intensity and the
color of the light essentially to create
a digital image and so for the purposes
of this lecture I'm going to use a
slightly stylized version of this tree
image where you can actually see the
individual pixels because the operations
that we'll be talking about the
convolutional operate the convolutional
operator and
the pooling layers and several other
layers that are being that will be used
in conclusion on neural networks they
will operate at this at this pixel level
so it's very important to understand
that this is this is what images look
like to a computer there are just a grid
of numbers corresponding to the colors
and intensities of at discrete positions
but so as I've already said a neural
network actually expects a vector of
numbers as input so we need to turn this
thing into a vector and so the simplest
thing we could do is just take the rows
of pixels in this image one by one and
just kind of string them together into
one long row and this is a vector right
this is so this image is nine by nine
pixels so this is 81 numbers
representing our image and that's a
valid thing to do so now you can just
take the network that you that you've
built with photic last week and and just
feed in these vectors and you can train
the model to try and predict for example
that there is a tree in this image like
you could do classification of objects
and images so that works but it's not
ideal because for example let's see what
happens if we just lightly change the
image by shifting the content so if we
if we look at this new image we'll just
go back and forth so you can see the
difference we've it's essentially the
same image but we've slightly moved the
tree to the top left corner of the image
so you can imagine that we're taking the
photograph from a slightly different
angle for example for the purposes of
image classification this is still the
same image right it's the same it's the
same type of object so we would expect
the output of our neural network to be
exactly the same thing now if we do the
same thing again as before and we turn
this into a vector by just taking the
rows of pixels and concatenating them
stringing them together they this will
end up looking very different than what
we had before like the image kind of
looks the same to us I mean it's shifted
a little bit where you can still see us
a tree but if you look at these vectors
they look very very different and to a
neural network they will look very very
different like the foliage of the tree
was mainly here before but now it's kind
of shifted all the way
to the left the trunk is somewhere else
and so this is kind of challenging for
the network because it will have to
learn many different they will have to
learn to detect many different patterns
to be able to say with confidence that
oh this is an image of a tree and so
clearly just that this flattening
operation is not the way to do it we
actually want to change the architecture
of the network that we're building to
take into account these this grid
structure that the image has and so two
key properties of natural images of
photographs our locality and translation
invariance so locality is the idea that
pixels that are nearby in the image that
are close together in the grid will tend
to be correlated so if you if you have a
photograph usually there'll be a few
objects in there and those objects tend
to take up actually quite a small
portion of the total number of pixels in
the image
and so those pixels corresponding to the
object they're gonna be very highly
correlated but a pixel in a top-left
corner of the image is not going to be
very correlated with a pixel in the
bottom right corner for example
so that's locality and related to that
another important property is
translation invariance which meet which
is that meaningful patterns in the image
can actually occur anywhere so I have
this example here with a photograph of a
bird in the sky and in these four
photographs the bird is in a slightly
different position again you could
imagine that you're taking the
photograph from a slightly different
angle but clearly it's the same bird and
it should be classified as you know a
bird regardless of where it is and so if
you if you think of this bird as a
pattern this pattern can actually occur
at a lot of different positions in the
image so I have a few examples of
photographs here that that exhibit these
characteristics sort of in an in an
extreme way and so you can you can see
that there's lots of patterns going on
here like the individual object sort of
give rise to patterns that are repeated
across the entire image
and that can also occur at different
scales so in the image on the right for
example there are there are interesting
patterns and in terms of the brake work
on the wall but there's also these
windows that this this window pattern
that occurs multiple times in the image
and there's also hints at the fact that
these images are compositional so there
are sort of objects and textures in the
image at many different scales and
smaller patterns sort of join to form
larger patterns and join to form objects
and that's that's a very important point
that will exploit in convolutional
neural networks I also want to point out
that images are not the only data
modality that have this that has to have
this property think about audio for
example if you if you record a sound or
more specifically if you record speech
someone someone speaking then the the
phonemes that the person is pronouncing
the sounds that the person is making
with their mouth they can occur anywhere
in the in the single right you don't
know a priori when they're gonna
pronounce which part of the word so
again that's this that's this
translation invariance but in only one
dimension this time in the time
dimension textual data exhibits this
property if you take a page from a book
a particular word could appear anywhere
on the page another interesting one I
think is graph structured data so if you
think about maybe molecules organic
molecules have a lot of patterns that
can occur at various positions in in in
the graph that represents their the
connectivity structure between the atoms
so how do we take advantage of this
topological structure of this grid
structure of images and of this
compositional nature of images so the
first thing we can do is something
called weight sharing when we when we
have a particular hidden unit and say
the first layer of our neural network
that detects a particular local pattern
for example this one then we might also
want to have units in the network that
can detect the same pattern at different
spatial positions in the network and we
can simply achieve that by looking at
the the weights corresponding to the
connections going into that unit and
then making copies of that unit all
across the image where we shift the
pattern across the image so that's
weight sharing and then a second thing
we can do is we can make our models
hierarchical because as I said these
images are sort of compositional and
combinations of patterns give rise to
more interesting more complex patterns
so we could incorporate that in our
model by stacking lots of layers that
extract progressively more abstract more
high-level features and so in this image
I've demonstrated that you have sort of
edges and textures that can combine to
form these object parts and object parts
then combine into entire objects that
you might want to detect if you're
trying to do image recognition so before
we go on with with the technical details
about convolutions I want to talk a
little bit about the history of these
models and how they came to be the way
they are today and the key story behind
that is that data drives research so the
availability of interesting data sets
has a massive impact on how much
innovation might happen in a particular
field and so for for the computer vision
field sort of the the thing that
kick-started this confident revolution
was actually the image net challenge
which was a competition that was run
from 2010 to 2017 so until a few years
ago that turned into a really major
computer vision benchmark that so a lot
of research was was done on this data
set and so every year - ran a
competition and the idea was that you
got a dataset of 1.4 million images so
quite a lot of images and in fact I
would say orders of magnitude larger
than what people have been using before
then
and these 1.4 millage --is were divided
into thousand different classes so
different types of objects household
objects animals lots of different things
there was actually an interesting
imbalance in the data set in that they
included lots and lots of different dog
breeds so about a hundred of those a
thousand classes are actually different
dog breeds and this is kind of
interesting because it forced people to
build models that could really pay
attention to the details on objects to
tell them apart because it's it's it's
quite easy to tell apart a cat and a dog
but if you have to tell apart certain
dog breeds that's a lot more difficult
you need a lot more local detail for
that so the goal here was image
classification and another challenge of
this data set was that the object that
one had to identify that one had to
classify wasn't always front and center
in the images so there were images that
might have multiple objects in them and
to to sort of compensate for that the
performance of the models for purposes
of a for the purpose of the competition
was measured using top five accuracy or
top five error rate so the idea is that
your model gets five guesses it can take
five classes out of a thousand and if
the correct class is among those five
guesses then then you get a point right
then then that's good so this is a
diagram of the top five classification
error rate of the competition winners
each year from 2010 to 2017 so in 2010
and 2011 people used traditional
computer vision techniques that were
state-of-the-art at the time the idea
there is that you try to do some you try
to extract some kind of feature
representation from your image that you
think will capture relevant properties
about the objects in the image but it's
it's entirely handcrafted so there's no
there's no learning involved or there's
very little learning involved in that
process and then once you have those
features you can you can do what we did
before actually you can turn them into a
vector and then you can feed them into a
simple classifier which could be a
neural network or an SVM and that kind
of used to be
thanks for done and so using that
strategy you can actually do reasonably
well there it is yeah you can do
reasonably well you can get you know
about two-thirds maybe three-quarters of
your answers right with this top five
accuracy metric but then in 2012
something interesting happened and this
was actually the year where Alex
Khrushchev ski Ilya sutskever and
Geoffrey Hinton submitted their Alex net
model to the competition and this was a
Convenant so this was actually one of
the first continents that was trained at
this scale on this larger dataset
continents had been around before you
know since since the 90s maybe even the
80s depending on who you ask
contacts have been around but they
hadn't really been applied at this scale
before and people didn't actually expect
them to work at this scale so that was
kind of the the most interesting aspect
of this is that suddenly these calm nets
were actually outperforming the existing
state of the art by a very large margin
so this was actually one of the first
major successes of deep learning
altogether I would say in 2012 and then
in 2013 people sort of noticed this and
immediately everyone switched over to
confidence so in 2013 basically all the
entries of the competition were
continents just and what they had done
was they had taken Alex net and they had
added some extra tricks added a few
layers that had a few modifications and
got the error rate down a little bit
further so you can see here from 16
percent down to 12 percent but then in
2014 another very interesting thing
happened and people sort of started
thinking commnets a bit further they
they looked at Alex net and they started
questioning fundamentally sort of the
design decisions in this model and asked
like how can we actually do even better
and so this gave rise to two models that
I'll talk about in more detail later
called vgg net and Google Annette
Google Annette is a reference to Lynette
which is one of the first incarnations
of continents from there from the early
90s so these models are much deeper they
have more layers but they also have a
lot more intricate architectures so
people thought more about the challenges
of training these deep models and tried
to figure out how to do that then in
2015 we had another major breakthrough
with rest net or residual networks the
idea there is the introduction of
residual connections where you you add
new connections in your network that
that allow it to skip a few layers and
this enabled training of proper deep
networks with hundreds of layers and
these residual connections are basically
an essential component of neural
networks today almost every network has
them so this was a very important
innovation that was again sort of driven
by by this competition and so we'll take
a closer look at this one later as well
in the section about case studies after
ResNet performance kind of saturated so
we see that there's still some
improvements in 2016 and 2017 but
there's no sort of major breakthroughs
anymore after this like people started
combining the predictions from lots of
models there are a few other building
blocks that were tried but nothing that
resulted in as dramatic and improvement
as we'd seen in the years before so
after 2017 the organizers decided this
was organized that at the by students at
a university of Stanford they said you
know what this is solved this problem is
essentially solved like anything we can
do that's better now like this might be
this was already a lot better than a
human could do even a trained human on
this data set so we're essentially
considering this problem solved and we
should move on to more challenging data
sets other problems so now let's look at
some of the building blocks of
convolutional neural networks so I'm
going to get my my three out again so
this is this is a
the tree image from before and I'm gonna
again use a stylized version with nine
by nine pixels and we're gonna look at
how we can go from fully connected to
locally connected so what do I mean by
that fully connected is like in a
traditional neural network where you
connect every input value in every every
element of the input vector to every
hidden unit in the first and there and
so on you always connect every unit to
every other unit and we're gonna go
we're going to move to locally connected
because as we said before we know that
objects tend to take up a small portion
of the image and so local correlations
between pixels are much more relevant
much more interesting than correlations
between pixels that are far away so so
this is a fully connected unit in a
hidden layer so I'm representing the
hidden layer here as so this is a vector
essentially of numbers that represents
the hidden layer and we have this we're
highlighting this particular unit here
and looking at its connectivity
structure so I didn't draw all the lines
because that would be tedious but
imagine that this thing is actually
connected to all the pixels in the input
image and then how do we compute the
output of this unit we basically
multiply the weights the parameters
associated with each of these
connections with the input pixel values
and then we optionally add a bias term
and then we get essentially a linear
function of the Pink's pixels and then
after that we can apply non-linearity to
that if we want to and that's
essentially what our neural network
layer does I should also mention that in
practice so I've used this image here I
say this has 81 connections in practice
this will actually have 243 connections
three times as many and that's because
an image a pixel and an image is not
represented by a single value it's
actually represented by three values
right red green and blue I'm not drawing
that here to keep things simple but keep
in mind that there are actually three
color channels in this image that we all
feed into the network so this is a fully
connected layer in
in a normal neural network how can we
make this locally connected so we can
basically connect each unit only to a
local region in the image so that's the
first thing we'll do so instead of
having you know 81 or 243 connections
here we'll only connect the 3x3 region
in the image to this particular unit and
then instead of having our units in a
vector representation here I also made
this two-dimensional because this will
also kind of preserve the topology the
input topology this grid structure will
also be in the output of our known
network so we're going to say this unit
connects two units up here so I'm going
to put this here and then this unit
connects two inputs up here so I'm down
here so I'm gonna connect put this here
so now we have locally connected units
with a 3x3 receptive field so that's a
word all I'll use more often later so
the receptive field is what we call the
part of the image that the unit can see
essentially so this formula doesn't
actually change much the only thing is
now that we that were no longer summing
over the entire image we're only summing
the contributions over a local region in
the image and so this will reduce the
number of parameters in the network
quite drastically because each unit just
has many fewer connections going into it
now how can we go from locally connected
to convolutional that's just the
introduction of weight sharing
essentially so all we're saying now is
that you have this locally connected
unit here and we have another one here
and we're just gonna make these weights
the same we're gonna say that the
parameters that they use will be the
same and so the result is a convolution
operation that's essentially all there
is to it and so we write that with with
this asterisk there are many notations
that are used in the literature for this
operation but the asterisk is the most
common so we have some some weight
vector that actually matches up with a
3x3 region in the image and we sort of
slide it across the image to compute the
the outputs of our hidden units
and and what what this means is that the
the resulting operation becomes equi
variant - translation so if we if we
translate the image of the tree like we
did before then this resulting output is
also going to be translated in the same
way and that's interesting because it
means that our networks kind of
preserves this this original structure
so as I already said so the region that
this connects to is called the receptive
field and and the output of a particular
unit that way sort of slide across the
entire image and then group the outputs
off in a 2d grid that's what we're going
to call a feature map the weights
associated with each unit we're gonna
call the kernel or the filter both terms
are used interchangeably and as I said
this operation will will preserve the
topology of the input so the feature map
is also grid structured so how can we
implement this operation in practice so
we we take this kernel and we
essentially slide it over the image and
this is basically a filtering operation
right we're applying a filter to the
image but the weights of our filter are
actually going to be learnt in this
instance so the kernel will slide across
the image and then we produce an output
value at each position and I'm
indicating these with different
grayscale values here and so once that's
done we have this new representation of
our image that's still two-dimensional
and that's basically contains detections
of that particular feature in the image
so if the if part of the image matched
the weights in the kernel very well then
we're gonna get a very high value at the
output so that means that the feature
was detected and if there's no match
then we're gonna get like a very low
value at the output and then the feature
wasn't detected so we can sort of
interpret this as a as a feature
detection map now in practice we will
have multiple kernels not just one
so I've given them different colors and
then each of these will be convolved
with the input image and give rise to a
different feature map so we get multiple
future Maps and we will refer to those
multiple feature Maps as the channels of
the output of the convolution as I
already said before the image is of
course an RGB image so it also has three
channels actually so what we're going to
do then is we're gonna each filter is
going to consist of a sub filter for
each color Channel and we're basically
just kind of sum their contributions of
the different input channels together so
what that means is that each output
Channel here is connected to all the
input channels in the in the input of
the convolution operation and so that
means that the inputs and the outputs of
the convolution operation are tensors
they're three-dimensional objects that
have with a height and a number of
channels so that's true for images as I
already said before red green blue but
that's also true for the output of our
convolution operation and all the output
channels of the convolution will be
connected to all the input channels as
well so let's take a look at a few
variants of the convolution operation
that have been used over the years so
the simplest one is a valid convolution
and in a valid convolution we're only
going to compute output values of the
feature map where we are able to fully
overlap the kernel and the image so
we're only going to compute a value
where where we can get this full overlap
and what this means is that the output
will be slightly smaller than the input
so for input images nine by nine and we
convolve it a three by three filter what
we're going to get out is actually a
seven by seven feature map because there
are only seven possible offsets for our
filter with respect to our image that
are that where we can compute a valid
output
so the output size is gonna be the input
size minus the kernel size plus one the
opposite of that is the full convolution
where we're actually going to try and
compute outputs wherever the kernel and
the image overlap by at least one pixel
and so that in practice what you what
you do is you actually just pad the
image with some zeros or whatever value
you like but typically people just pad
with zeros and then you do the same
thing as before so you could you can
think of a full convolution as a valid
convolution but with added padding and
the result is actually gonna be a
feature map that's larger than your
original image because there are more
valid offsets than there are actually
pixels in the original image so the
output size is going to be the input
size plus the kernel size minus one and
so if we stack a lot of valid
convolutions on top of each other the
effect that that's going to have is that
are the size of our feature Maps is
gradually going to shrink whereas if we
stack lots of full convolutions
the size of the feature Maps is
gradually going to grow and neither of
those are really desirable in in
convolutional neural networks so there's
a third variant that's actually the most
popular variant today which which is
called the same convolution where we try
to pad just enough zeros so that the
output size the feature maps will have
the same size as the image and so for a
3x3 kernel you just need a pad with one
row zeros all around the image and then
you get a nine by nine feature map at
the output note that this version
actually only makes sense if your kernel
has an odd size if your kernel is even
size then you would have to pad
asymmetrically you'd have to pad
slightly more on one side than on the
other
in practice this problem doesn't really
come up because everyone just uses odd
kernel sizes like I've seen very few
confidence where people actually use
even kernel sizes and so the nice thing
about this is that if we stock lots of
same convolutions on top of each other
you can do that as much as we want to
the size of the feature map will not
change of course what might happen is
that we get some edge artifacts right
because some of our filters might
end up detecting the edges of the image
if this is zero which which typically
corresponds to a black pixel it might
actually detect this this corner as
something meaningful whereas actually
that's just the corner of the image so
that's something to look out for there's
a few other variants that are
interesting so there's a so-called
striated convolution where instead of
computing the output of the conclusion
at every possible offset we're actually
going to skip some steps and the nice
thing about this is that it is obviously
a lot cheaper to compute because if you
if you use a stride of 2 for example you
reduce the computation by a factor of 4
because obviously you increase the step
size both in the height and the width
direction and so this gives you a way to
reduce computation it also gives you a
way to reduce the resolution of the
feature maps that you're operating on
and this will be useful when we stack
lots of layers together and we want to
create a feature hierarchy right because
you you would like the higher-level
features in the model to be operating at
a larger scale you want them to see more
of the image and so strided convolutions
can be very useful to sort of create
this hierarchy so if we if you move the
filter again you can see that in this
case I'm doing a valid convolution so
far I'm doing a valid convolution with a
stride of 2 and I'm getting a 4x4
feature map which is obviously a lot
smaller than the 9 by 9 image that we
started with another interesting variant
is a dilated convolution so here we're
not we're not striding so we're not
skipping offsets but we are skipping
values of the filter so if you want to
increase the receptive field of a
convolution the now you think to do
would be just to increase the size of
the kernel if you have a very large
kernel you have a large receptive field
that can get very expensive because
obviously the the cost the number of
parameters and the computational cost
will increase quadratically with with
the kernel size that you that you choose
and so dilation is a very cheap way to
do that where you basically say
typically the features in my image will
actually very slowly over space so it's
okay to subsample it's ok to skip a few
and not compute the feature value there
because it's it's not gonna be that
interesting anyways it's probably just
gonna be into interpolation between the
two values beside it anyway so we can
safely skip it and so in a dilated
convolution you basically oh sorry
you basically pretend that you have a
larger filter but you have a bunch of
zeros in there and this can be computed
more efficiently than then then the
naive way would then you would think
naively because you don't actually have
to Pat you don't actually have to put
those zeros in there and and then do
plots of multiplies with zero you can
actually do this efficiently with some
reshapes and and and other tricks of the
tensors and then a final variant that i
want to talk about is a depth wise
convolution because that one's really
come to the forefront
more recently so they said before in
convolution operations that we've talked
about every output channel will be
connected to so each output channel will
be connects to every input channel so we
kind of have dense connectivity between
the channels in a depth wise convolution
that's not the case in a depth wise
convolution we have one output channel
per input channel and there's no
interaction between the channels and so
that dramatically reduces the number of
parameters that this convolution has but
obviously it's also a lot less
expressive but it's been it's being used
more and more as a building block
together with with other types of
convolutions and then finally pooling is
another operation that's very common in
convolutional neural networks so this is
this is kind of an alternative destroyed
convolutions to reduce the resolution of
the feature Maps basically what you do
is you look at local windows of your
input and you just compute some
aggregation function of those inputs so
that will typically be the mean of the
values or the maximum of the values and
then you compute those for all positions
in the grid and then you get your output
feature map which will typically be a
lot smaller
so here I've done this directly on the
pixels and in practice you might want to
do this inside the network on your on
your feature maps so now let's talk
about convolutional neural networks and
how these building blocks actually fit
together in their own networks so so
I've already been referring to them as
continent so there's actually two
abbreviations that are that are in
common use today like CN NS and commnets
you'll see both used interchangeably
I like comments it's easier to say will
stack up to hundreds of these
convolution operations together in a
model and we'll alternate convolutions
and pooling or possibly strata
convolutions to create a feature
hierarchy where higher layers in the
model will extract more abstract
features from the image so a brief recap
about neural networks as computational
graphs so this is a slide a diagram
rather that I took from from vertex deck
last week so this is kind of a
computational graph representation of a
neural network where we have nodes
representing the input so that's both
the the image in our case the input
image but also the target the prediction
that we're trying to match and then we
have lots of computational nodes and
some of these nodes are linear layers or
convolutions which have learn about
parameters and so these are indicated in
pink here and then at the output side we
also have the the loss in orange so I'm
going to simplify this diagram I'm not
going to display the parameters anymore
they're implicit so they're considered
part of the computational nodes I'm also
not going to show the loss because
that's that's always there that's not
what we're focusing on right now so I'm
not going to draw that on the diagram so
I just have an input node and some
computation noise here straighten it out
a little bit as well and then I'm
actually going to differentiate the
computational nodes
because that's what we're interested in
here sort of the architecture of our
confidence so I'm going to distinguish
between fully connected layers which are
the layers that that Wojtek also talked
about so typical neural network layers
where every unit is densely connected to
all the units in the previous layer
those will be in pink and then
convolution operations will be until the
pooling operations will be in purple and
I've left the nonlinearities in dark
blue as before so now let's talk about
some more interesting more recent
convolutional neural network
architectures so the one that I actually
just showed you is an existing one
called Lynette five so this is one of
the earliest published content
architectures so this was a continent
for handwritten digit recognition so it
operated on fairly small images I think
28 by 28 or 32 by 32 grayscale images of
handwritten digits and tried to produce
a classification for which which digit
was in the image and this had kind of a
what was until then sort of the
canonical structure of a continent which
was you had the the input image and then
you had a few convolutions interspersed
with pooling so there was always this
structure of convolution non-linearity
pooling convolution on linearly pooling
and then at some point you would do the
vectorization operation that we talked
about before you would just take the
feature map at this point and just
flatten it into a vector and then from
from there on it would just be a regular
fully connected neural network so you
would have a few fully connected layers
interspersed with nonlinearities and
then maybe a softmax non-linearity at
the end to do the actual classification
so and then in 2012 obviously we had
with Alex net as I said so this is this
is actually an architecture diagram from
from the paper it's it's cut off at the
top and it sits like this in the paper I
don't to the victory this day we don't
know if that's intentional or not but
the reason for the sort of unusual
structure of this diagram is that this
was a very big model that was trained on
two GPUs so so the model was actually
kind of split over to different
processors that each contained half of
the parameters of each layer and so
that's why you have this kind of
separation running through in the middle
and you have very few connections going
across because that was that was
communication between two GPUs so that
was very costly especially at that time
so you have this kind of two two stream
network so the architecture of this
model is a bit more complex than than
Lynnette so now we have eight layers
that's that's eight layers with
parameters so that's five convolutional
layers and three fully connected layers
we have something else that was new here
was the value non-linearity so before
this people tended to use saturating
nonlinearities like the sigmoid function
or the tan h function which sort of are
limited in their output range and it was
actually really hard to train deeper
networks than say four or five layers
with this setup so with Lynette was okay
but if you added a few layers to you
Lynette it would be in trouble and
people found actually that you can just
use the value which is the function that
the value is defined as is literally
just a maximum of its input in zero so
basically if the input is negative you
set it to zero and this has a
discontinuity at zero and people thought
oh if we have discontinuities in our
nonlinearities then gradient based
optimization is no longer going to work
right because that uses the gradient so
clearly that's that's that's only going
to work if the gradient is defined
everywhere and it turns out that's just
not true it turns out that as soon as
someone tried this it turned out that
this was actually a very nice
non-linearity to use because it improve
the propagation of the of the gradient
information through throughout the model
and so it enabled deeper networks and so
this is actually kind of a key
innovation here to use these value
nonlinearities other important
innovations include regularization
strategies so as I said this model was
proposed for the image net data set
which is quite a large data set so you
would think that maybe you don't need to
regularize the model too much because
you have so much data but Alex
Khrushchev skis response was just to
make his model really really really
massive and have lots of millions of
parameters and so he still needed
regularization to make sure that the
model wouldn't over fit to this data set
and so weight decay was used which is
kind of a traditional regularizer where
you just make sure that the magnitude of
the parameters doesn't grow too much but
also drop out and that was also kind of
a new thing the idea that you can
regularized neural networks by randomly
removing units during training and the
idea is that this make the other units
more robust to potentially having inputs
that are absent or that are distorted
and that turns out to be an extremely
good regularizer so that was another
important innovation of alex net so yeah
as I said this was trained on two GPUs
and it actually took six days to train
one of these models back in a day
nowadays you can train it in minutes so
yeah if a user our color scheme then the
diagram of this network looks like this
I kind of have to wrap around here and
you can see that it's that is already
about twice as deep as the as Lynette
was so if we if we walk through this
from the input to the output so at the
input you have images coming in so three
channels and the images were scaled to
two 24 by 2 24 pixels which is a lot
larger than any confident had used
before then so typically comrades would
use inputs a 32 by 32 but people had
never really gone to come to that scale
and so the way they did this was
actually by having very
tried in the first convolutional layer
so only the first convolutional layer
was operating at this very high
resolution and then immediately the
resolution would be reduced by a factor
of four which meant that the actual the
the amount of computation was reduced by
a factor of sixteen from there on so it
had an 11 by 11 kernel again as I said
an odd odd sized kernel because that's
what people use 96 channels stride of
four and so it's output size was 56 by
56 by 96 so a lot smaller spatially but
obviously a lot more channels and then
we had the rail you nonlinearities and
then max pooling layer to reduce it even
further down to 28 by 28 by 96 which
means so this is essentially a pooling
operation with just where we just take
the maximum over 2 by 2 windows and so
that means that that means that the read
the rest of the network is actually
operating on things that are 28 by 28 or
smaller so not that different from the
networks that came before so it's only
really this first layer that's doing a
lot of hard work to to use this high
resolution information in the image and
that was new that was an innovation of
Alex I'm not going to go through all the
layers I'm gonna skip ahead to the last
fully connected layer which is going to
produce thousand outputs one for each
class in the image net data set and and
finally we have a soft max non-linearity
which which takes the output of the
fully connected layer and turns it into
a categorical probability distribution
where we can guarantee that the outputs
of the model will be probabilities that
sum to one so they will form a valid
distribution over the classes so here
are all the layers again and you can
sort of see that the the resolution
actually is reduced very rapidly at the
start and then more gradually throughout
the network
and so another thing actually that was
kind of new here was the realization
that we don't always have to pair a
convolutional layer with a pooling layer
so this is done here at the start twice
but then we have a few convolutions with
nonlinearities in between where there's
no pooling happening and and people just
didn't do this before Alex net it wasn't
considered to be a valid thing to do so
it's kind of interesting that these
things that we maybe now take for
granted we're just not done so by now I
think it's clear that the story is that
deeper models tend to perform better and
and to get some insight into that you
can consider each layer as acting is
kind of a linear classifier that's
detecting particular patterns in its
input and so that means that if you
stock more of these layers on top of
each other you actually get more
nonlinearities in your model you get a
more powerful parameterised function
that you can use to to fit to fit the
targets and so this this the question
arises like what what is actually
limiting the number of layers in
confidence like why was Alex and eight
players why wasn't it 80 layers and
obviously an obvious one is
computational complexity obviously if
you have more layers you have to do more
computation which you know we always
have a limited computational budget but
there were actually other issues as well
such as optimization so if we have a
deeper model how do we actually back
propagate through that entire model how
do we do a credit assignment if a model
makes a mistake how do we assign
responsibility for that mistake to
particular units in the network and that
gets harder and harder as you stack more
layers on top of each other so in 2014
we have vgg net and and there again we
see a doubling in depth essentially so
you see I now need four lines to fit
this model and so there the idea was
they kind of took this sequence of three
khans layers from Alex net to an extreme
and they said we can actually do this
all the way throughout the network we
can stack many many convolutional layers
on top of each other before we actually
do any pooling
and we can use same convolutions so with
with padding so that the output feature
maps are the same size as the input to
avoid resolution reduction where we
don't want it so if you're stacking
these convolutional layers we don't want
resolution reduction we want to be in
control of where the resolution is
reduced and that's going to be in the
pooling layers and so another idea from
PGD net is actually to to fix the kernel
size and did not treat this as a hyper
parameter of the architecture so unlike
Alix net which had different kernel
sizes for different convolutional layers
vgg net only uses 3x3 kernels throughout
so that simplifies the the search for
good architectures considerably because
what they realized was if we want a
larger receptive field we don't
necessarily need to make take take a
single layer and make its receptive
field larger by increasing its kernel
size you can actually just stock three
by three filters to create a larger
receptive field that spans multiple
layers so here if we have a stack of two
3x3 convolutions we can sort of see in
blue these are the receptive fields of
the of the first convolutional layer and
then in red I've superimposed the
receptive field of the second
convolutional layer with respect to the
to its input so with respect to the
outputs of the first layer but if we
look at these two layers is one block
and sort of look at the receptive fields
of the second layer with respect to the
input of the first we see that it's
actually five by five so it grows as we
stock more three by three convolution so
you can actually create something that
has an equivalent receptive field to a
single layer with a five by five kernel
but it will have fewer parameters and it
will be more flexible because we can
also insert an extra non-linearity there
so it'll be more it'll be able to model
more interesting functions so in terms
of architecture vgg net had up to 19
layers so again quite a bit more than
the eight layers of alex net it only
used three by three kernels with same
convolutions in terms of infrastructure
this was also a bit of an upgrade so
this was trained on four GPUs and it was
trained for two to three weeks so very
patient people had vgg in oxford and and
another thing here that was interesting
is they use data parallel
not model parallelism so for Alex net
the model is kind of split over these
two GPUs and you saw that this actually
affects the architecture like it affects
which which parts of the model we can
connect to each other so what what was
done for vgg net instead is data
parallelism where you just take your
batch of data that you're processing and
you just split it into four parts and
then you have the entire network on all
four processors on all four GPUs and you
just you just compute predictions on
smaller sub batches predictions and and
gradients obviously during training so
this is the error rate on sort of top
five error rate on image net for VG for
different versions of VG net with
different numbers of layers so they had
variance with 11 layers 13 layers 16
layers and 19 layers and what's
interesting here is that obviously up to
a point
deeper is better so 16 is better than 13
is better than 11 but it seems like
their performance saturates after 16
layers they tried one with 19 layers and
saw that it was actually slightly worse
so what's actually going on there and so
later models so at the time you didn't
know but later models actually use a lot
of tricks to to prevent this from
happening because what's happening here
is an optimization issue right now you
have these 19 layers of computation and
it's starting to get harder to do credit
assignment so yeah I've actually already
already mentioned both of these so the
challenges of deep neural networks are
our computational complexity more layers
is more computation that takes time and
energy and there are also optimization
difficulties that arise because
optimization of the parameters by
gradient descent becomes a lot harder
and so we'll look at some ways to
address these challenges next there will
be a future lecture in this series that
will cover optimization of very deep
models in detail so look out for that so
how I'll just I'll just give a quick
summary but my colleague I believe James
Martin's will will be doing that one
he'll go over this in detail so one
thing we can do is be very careful
with how we initialize the parameters of
our neural network if we if we just
randomly initialize these from say a
uniform distribution from minus 1 to 1
then that's not going to work because
the the activations the outputs of the
of the layers in our network are going
to grow as we go through the network and
and then if we try to optimize the
network we need to take the gradients
which means we need to do a backward
pass through the network and those
gradients there's intermediate gradients
that we can gonna compute are also going
to grow so we're actually going to get
exploding gradients if we do that you
might say ok just MIT just make them
really small you can't make them 0 oh by
the way because you have to do some
symmetry breaking like if you initialize
a neural network to zeros it has no way
to differentiate the different units so
you do have to do something random but
you could say like ok initialize all the
weights to very small values then what
you're gonna get is vanishing gradients
like the gradients are just gonna
collapse to 0 because your computer will
not have enough precision to represent
these really small values so you need to
be a little bit careful about how to
initialize these models and and people
have figured out various heuristics to
sort of ensure that the gradients have
the right scale at the start of training
and then luckily if you do that at the
start of training that tends to be
preserved throughout training another
thing you can do is use very
sophisticated optimizers so obviously
you can just use stochastic gradient
descent to train a neural network but
there are lots of interesting variants
of that algorithm that are you know
specifically tailored to neural networks
and and tend to do a better job
optimizing them more quickly again I'm
gonna I'm gonna leave this to my
colleague to to go into in detail an
architectural innovation that can help
with this is the introduction of
normalization layers so I haven't talked
about those yet but they're actually
today just as essential as the
convolutional layers and pooling layers
and the nonlinearities so we insert
normalization layers throughout the
network to sort of scale the activations
so that they're in the right range for
optimization to be easy
and I'm finally we can also just change
the network design to make gradient
propagation easier so we can we can
change the connectivity pattern and rest
nets that that we that I already briefly
mentioned before are an example of that
so adding these residual connections is
a good example of that so let's also
look at Google Annette because this was
actually the winner of the 2014
competition so vgd net came second but
was in retrospect just as influential
Google net was interesting because it
was a lot more intricate and a lot more
complicated than previous network
designs that beam so people hadn't
really considered that you could
actually kind of branch out and then and
then and then concatenate the the
resulting feature Maps and sort of have
these multiple convolutional operations
operating side-by-side so this is kind
of the the canonical diagram of Google
Annette this is a zoomed in version of
one of these what's called inception
blocks and so you can see they have
multiple convolutions with different
kernel sizes and an even pooling
operating in parallel this is the first
version of the inception module there
have been various iterations on top of
this in the meantime that I'm I'm not
going to go over all the different
variants in detail though so I
normalization layers so a key innovation
actually in the second version of the
inception module was the introduction of
batch normalization and the idea of
batch normalization is that you do
essentially you standardized activation
so you compute the the mean and the
variance and you estimate these across a
batch of data so you do this in every
layer you estimate the mean and variance
of the activations and then you just
normalize right here and then at the
output of the normalization what you
might want to do is have a trainable
scaling factor and
at this point so that the activations
aren't actually forced to be constrained
to be 0 mean unit variance thank you you
want to retain the ex Priscilla T of the
original neural network but you want to
have this normalization step in there
because it makes optimization easier and
so that's that's how you can do that and
so introducing batch normalization
layers throughout the network
dramatically reduces the sensitivity of
the model to initialization so even if
you kind of wing it in terms of how you
initialize the model with batch norm
you'll actually be able to still still
train it and it makes it more robust to
larger learning rates as well another
aspect of it which can be a downside or
an upside depending on what you're
trying to do is it introduces stock
assisity and it acts as a regularizer
because these statistics these these
music these mutant Sigma's they are
estimated on the current batch of data
and obviously the batch of data will be
relatively small so you're gonna get a
you're gonna get a rough estimate of
these statistics but it's not gonna be
exact
and that can actually be a good thing
because it introduces noise into the
network as we said before we dropped out
like introducing noise into the model
can actually make it more robust in
practice so this acts as a regularizer
now the downside of this is that at test
time when you actually want to use your
model and make predictions this will
introduce a dependency between the
different images in the batch so you
will get different predictions for a
particular image depending on which
other images are also in the batch and
that's not nice thank you you wouldn't
want your model to give deterministic
predictions so in practice what you need
to do is actually estimate these
statistics on a data set and just keep
keep track of them separately and then
use those at test time which is doable
but we've actually at least I found in
practice that this can be a source of a
lot of bugs if something is wrong with
my neural network latch norm is usually
the first thing that I suspect
so the original Google Annette did not
use bachelor but any all later versions
of it
use this and if you look at a comparison
between the original Google Annette
which is the dashed black line here and
and then a another version of this model
which uses batch norm you can see that
it actually converges a lot faster and
to a higher accuracy and this is because
Bachelor makes the model able to take
larger learning rates essentially
without without diverging so let's look
at ResNet which came in 2015 this is
actually one of my favorite sort of
continent innovations because it's
beautiful in its simplicity the idea
here is that oh if depth is what's
preventing us from trading deeper models
because it makes optimization harder why
don't we why don't we give it a skip
connection why don't we let it skip a
few layers if it needs to so that the
graded can back propagate more easily
and the way they achieve that is
essentially by adding this residual
connection which just means that you you
take your input from from the input of
this layer and you just kind of add it
back in later which means that when you
when you back propagate through this and
you take the gradient you can actually
go along this pathway and skip these
convolutional layers all together and so
residual connections facilitate training
much deeper networks and so the the rest
net that one the imagenet competition in
2015 was actually one hundred fifty two
layers deep so again a dramatic increase
an order of magnitude increase compared
to the previous year so there's a few
different versions of residual blocks
residual modules so the top one is the
one i just showed you which is the
original one from the from you from the
rest of paper in 2015 which was the one
used in the image net competition so
that looks that looks like like this one
here it has a three by three convolution
followed by a batch norm followed by a
nonlinear
and then one-by-one convolution for
their bachelor and then you have the
residual connection coming in and then
you have another non-linearity so this
block was kind of stocked many many
times to come to 152 layers there's a a
variant that they used to reduce the
number of parameters which is called the
bottleneck block and so that's this one
so as you can see instead of two
sequences of confident vaginal
non-linearity this one actually has
three and what's happening here is it
has convolutions with very small filter
sizes so one by one excite essentially
means that you do a convolution but
there's no actual spatial integration of
information happening you're just kind
of computing the same function at every
spatial position and so what they did
was they used one by once to reduce the
number of channels reduce the number of
feature Maps and then they do a three by
three on fewer feature Maps which means
you'll have fewer parameters in your 3x3
convolution which is doing the actual
work the actual spatial integration of
information and then you just go back
out and you increase the number of
channels again with another one by one
and so this is a nice way to sort of
have a large representation but still
have fairly cheap convolution operations
in the middle there and then the third
version at the bottom is it's called
resna b2 if you don't it's something not
clear if you don't mind I think that
will be easier thank you
so the the this one is resonant me too
and there they actually just moved the
operations around the bit so as you can
see that the bathroom is here and on the
air it is here and then a three by three
and I'm batch normally RT one by one and
then the residual connection comes at
the end here the advantage of this one
is that if you stack a lot of these on
top of each other there's actually a
direct connection from the output to the
input without any nonlinearities on the
path and so this allows you to go even
deeper and and even go to thousands of
layers if you want to
so this is a table describing the the
rest net architectures from the original
paper so these are a few different
versions of progressive depth and you
can kind of see the same pattern as
before they they start out with high
resolutions but they kind of very
quickly reduce them and and most of the
computation is actually done at these
lower resolutions you can see I actually
made a mistake earlier the the 152 layer
model on the right here you can see that
it's actually using the bottleneck
blocks not not the top one on the
previous slide and what's interesting
here is that this 152 layer model
because it uses bottleneck blocks
actually has requires fewer
computational operations we were
floating point operations then the 19
million video Ginette did so even though
it's an order of magnitude deeper it's
actually cheaper to compute which i
think is really nice and it actually
also obviously performs a lot better
because of this depth another variant of
this ideas dense net so here we don't
have residual connections in dense net
the authors decided to make
backpropagation easier just by
connecting every layer to every other
layer so whenever you stock a new layer
and a dense net you connect it to all
the previous layer all the previous
layers and not just a preceding one and
so you get this dense connection between
layers but obviously each layer is still
a convolution with batch norm with
rallies inside and then in this this
actually comes from the this was also
introduced as part of the image net
competitions so this was one one last
innovation in 2017 this the idea here is
to incorporate global context so
convolutions are actually obviously
great at capturing local patterns but
sometimes you will might you might want
to model eight modulate these patterns
based on the the global context of the
MHD the stuff that's happening elsewhere
in the image and so for that purpose
it's nice to
try and compress the entirety of the
image into just a feature vector and
then kind of broadcast that feature
vector to all spatial positions so that
at any spatial position in the image you
can actually get some extra information
about what's going on elsewhere in the
image and so you you can you can
incorporate global context into your
features and then another strand of
architecture design has become popular
more recently is neural architecture
search so up till now we've been talking
about these architectures that got
progressively more intricate and these
were all hand designed by humans right
so humans basically searched for the
optimal hyperplane meters the optimal
number of layers geofflove kernel sizes
in these models and so people started to
think like maybe we can actually
automate that process as well maybe we
can use a search algorithm or even
machine learning algorithm to find the
best architecture to use to then train
to do image recognition and so amoeba
net is a model that arose from such an
algorithm so it's an architecture that
was found by an evolutionary algorithm
that basically performed a search over a
cyclic graphs composed of a set of
predefined layers so that kind of said
the convolution operation is a
predefined layer and then we have a
pooling operation we have different
types of pooling and then basically
connect these up any way you want and
find the optimal connectivity pattern
that gives rise to a continent that
works really well and so that's
architecture search and then another
trend in recent years has been to try
and reduce the complexity computational
complexity in these models by by
parameterizing them in more efficient
way so I've already talked about depth
wise convolutions and the way that
they're they reduce the number of
parameters dramatically just by not
connecting all the input channels to
each output Channel but better
connecting channel by channel obviously
you pay a cost in terms of experts
civilly but sometimes this can be worth
it but people have used
depth-wise convolutions to build what's
called a separable convolution and a
separable convolution is essentially
just a combination of a depth-wise
convolution followed by a regular
convolution with a one by one filter
size so you're kind of dividing the work
here in the sense that a depth wise
convolution will do spatial integration
it will sort of capture information from
a local neighborhood and then the one by
one convolution that follows will do
redistribute the information across the
channels which the depth wise
convolution doesn't because it operates
within each channel and so if you
combine those two you have kind of a
separable version of a regular
convolution where one part of the
operation the spatial integration and
the other part does integration over two
channels this idea is also used in with
in another fairly modern building block
so I've talked about bottom like blocks
before and I talked about ResNet the new
cool thing is inverted bottlenecks where
instead of reducing the number of
channels and then applying a 3x3
convolution you actually increase the
number of channels inside the bottleneck
and then apply a 3x3 depth-wise
convolution the idea being that you sort
of do spatial integration in this really
high dimensional space and then collapse
it back to a more manageable feature
space for communication between the
different parts of the network so as
that's it as far as case studies is
concerned so to wrap up this lecture I'm
going to give a brief overview of some
more advanced topics and I'm also going
to talk a little bit about contents
beyond image recognition so one thing I
haven't mentioned which is actually a
crucial ingredient in in many modern
commnets as data augmentation so by
design Condit's are robust against
translation right if you translate an
image then the internal representations
inside the component will also translate
and eventually because of all the
pooling it comes quite easy for them all
to be invariant to that so to sort of
ignore that translation altogether and
to just classify the object that's in
the image but obviously translation is
not the only thing that can happen to an
image you could also rotate your camera
for example and take a photograph from a
different angle you could go closer and
away which changes the scale of the
object you could you know it could be a
bright day it could be a dark day that
will affect the lighting all these
things are nuisance factors essentially
nuisance factors a variation that will
not affect the classification output
that you want but obviously dramatically
affect the pixel value is that the that
the model will see and so the way to
make these models robust against these
variations is to just artificially apply
them during training so every time we
feed an image to our network during
training we randomly perturb it with
some of these transformations so I have
some examples down here of different
perturbations of this image and then we
basically say for all of these you
should always produce the output treat
right because this is a tree regardless
of how it is perturbed and that allows
the model to learn robustness against
these other transformations so the
robustness is not innate in a sense that
the architecture isn't designed to be
robust against these transformations but
we can still let the model learn that it
needs to be robust we can also visualize
the patterns and the filters that
accommodate is actually learning and so
a neat way to do that is to take a unit
in any layer of the continent and then
to try to maximize its activation by
changing the input image and so we can
just do that with gradient descent just
like we use gradient descent to Train
these models we can do great in ascend
with respect to the input pixels to try
and find images that maximally or
minimally activate different units in a
network and we can do this for different
layers and these are some figures from
paper by Matthew Zieler and his
colleagues which really nicely
demonstrate this idea of
compositionality and hierarchy in a
sense that these different layers are
learning different patterns and in layer
two you can kind of see that these are
fairly local patterns there are some
edge detectors some textures here in
layer three these are starting to get
aggregated and to more interesting
patterns and then if you go all the way
up to layer five you can actually see
that there's even layer four sorry you
can see that there's a dog head detector
here that arises
so it's kind of combined I think nicely
showing that these patterns are getting
combined into progressively more
interesting structures throughout the
network we can also run this procedure
with respect to the output layer of the
network so if we take an output unit
corresponding to a particular class so
one of our thousands and then we just
say you know find me the image that
maximizes the probability of this output
so we get sort of the canonical image
corresponding to a class this is what
you can get out so this is from word by
a Koran Simonian and as colleagues so
you can kind of see the objects in here
I mean obviously these don't look like
natural images but you can kind of see
that there are certain patterns that
arise here and these are the patterns
that the network will try to pick up on
to classify images if you do this with a
strong prior so you kind of add an
additional term in your in your loss
where you say this is what an image
looks like this is what a natural
natural image should look like then you
can get images like these out so this is
a this is from a different paper where
they essentially use the same procedure
but they have this extra prior that
tries to make the images look natural
and so now you can sort of see images
that would maximally activate
particularly units in the continent
there's a really nice interactive blog
post on the Stillwell Pub about this
idea feature visualization so it's it's
interactive so you can play with this
and you can kind of look at all the all
the different units in all the different
layers in a neural net I definitely
recommend checking this out that's
really cool to play with so some other
topics to explore that I don't have time
to go into today are pre training and
fine tuning so a lot of a lot of image
classification problems that are of
interest don't have large data sets
available so image net is obviously 1.4
million images is pretty good but for
many problems we may have
orders of magnitude fewer and so people
have sought for ways to sort of reuse
the model strained on imagenet for
different tasks and the way you can do
that for example is to take a train
model and imagenet and chop off the top
layer that does the actual
classification and then just fit another
layer on top of those features and that
turns out to work really quite well and
you can even fine-tune the rest of the
network a little bit to to work on your
new data set so that's a very effective
strategy to do transfer learning another
topic that I think is very interesting
is group echo variant confidence so
we've talked about how come nets are
invariant to translation and then all
other invariances kind of have to be
learned and so you can learn you can
learn them with data augmentation but
you can actually build confidence that
are intrinsically equivariance to
rotation and to scale and other things
like that and this is a line of research
that's kind of taking taken flight over
the past three years or so which i think
is worth exploring I also want to talk
briefly about recurrence and attention
so these are two other ways to
incorporate topological structure of the
input into our network architectures I'm
not going to talk about them now because
they'll be the subject of future
lectures but so just to say that
convolutions are not the only thing you
can do to exploit grid structures or
sequence structure in in your input and
so to wrap up let's talk about what else
we can use these models for so we've
talked about models for image
recognition so far but obviously there
are lots of other tasks involving images
that could benefit from these
architectural priors so what else can we
do with continents so the object
classification here is in the top left
so that's whatever that's what we've
been talking about so far you can also
do object detection where in addition to
identifying the class of each object in
the image you want to figure out where
in the image it is so you want to
produce a bounding box for each object
in the image another variant of this is
semantic segmentation or for each pixel
in the image you want to identify what
type of object it is part of and then
there's also instant segmentation where
you actually want to segment the
individual objects even if there's
multiple at the same class so that's all
in the image space we can also generate
images with confidence in many different
ways so there are various different
types of generative models generative
adversarial networks variational
autoencoders auto regressive models like
pixel CNN that all use the convolution
operation as a basic building block
because they also benefit from this
these priors of locality and translation
invariance so these are some images that
were generated by big Gann which is a
generative adversarial Network developed
by my colleagues at deep mind more with
commnets you can do representation
learning one thing that's getting very
popular right now is self supervised
learning so what do you do if you have a
very large collection of images but no
labels then you can do you can kind of
create labels yourself and and do self
supervised learning and hopefully get
features that might still be useful for
transfer learning later as I also
mentioned earlier you can use comp nets
for other data types like video audio
text graphs there's lots of options
there you can use commnets inside agents
inside intelligent agents trained with
reinforcement learning there's there's
lots of options many of these will be
talked about in future lectures as well
so to wrap up I want to leave you with
this sort of statement which is that
convolutional neural networks replaced
handcrafted features with handcrafted
architectures and the reason I want to
stress that is because people often used
to see confidence as kind of a a magical
thing that led us to no longer have to
sort of be clever about what we do with
images like how do we exploit structure
and images so we actually don't need to
do that anymore we just put a comment
off it we'll figure it out but that's
not how it ended up being because we've
actually used a lot of our prior
knowledge about about structure and
images to design these architectures to
work better so we're not we still have
to be intelligent we still have to do
research we still have to use the prior
knowledge that we have about the data
that we're working with we're just
incorporating it at a higher level of
abstraction than we were before
and we're using learning now so learning
is in the mix which is which is the main
differentiator I think so that is all I
have
I want to thank you very much for your
attention
[Applause]
[Music]