DeepMind x UCL | Deep Learning Lectures | 4/12 |  Advanced Models for Computer Vision

so thank you thank you for coming so
today we're going to talk about vision
vision beyond classification and I guess
like the first question is why do we
need to go beyond the classification so
you know how they say that a picture is
worth a thousand words and we say that
because like the the world that we live
in is very rich and very complex from a
visual point of view and most of the
information that we capture about our
surroundings comes from our eyes so 80%
of the information comes from from the
eyes
so however classification models they
can only learn few words about about an
image so if you take this particular
image which I think it's quite a nice
one and pass it through an image
classifier resonate 50 which is a very
popular image classifier it will tell
you something like bicycle and garden
and these are correct correct answers
however I think we are far from
understanding from correctly parsing the
scene that that we see here so first of
all we have two bicycles in the in the
scene and then we have a persons on on
those bicycles and at first you might
think that they're riding those bicycles
but then when you look closely you can
notice that okay one of the bicycles in
person they're upside down so that does
not really correspond with with the
biases that we have about the physics in
our world and so yeah there is there is
a really more to this scene that we need
to understand so first of all yeah we
need to understand the pose of these
objects with respect to the camera and
also the relative pose like the pose of
the objects with respect to each other
and then at the closer inspection we
notice that okay this exactly the
top-down view also the camera is
actually positioned at the top so this
image was actually taken with a drone at
the top of the of the garden so like the
holy grail in computer vision is we want
to train a system that achieves human
level scene understanding because if we
have that then we have really a large
number of applications that become
available so first of all think about
visually impaired people so such a
system could help them navigate
environments
and then we have all kind of
applications of virtual reality or
augmented reality that we could we could
have and then any kind of robotics or
autonomous vehicles and something that I
am really interested in I believe that
if we advance research in designing such
a system we will also achieve a better
understanding of our brain of how our
visual system works
however designing a system that really
achieves human level scene understanding
is a very challenging task because there
are so many factors to take into account
so for example dimension we have
different tasks that we need to think
about like object detection and pose
estimation and camera position and
others and that we need to think okay in
which order should we should we tackle
these tasks and then we can have
different inputs and different structure
in the outputs as we will see we will
see today and then we might have
different priors about the world that we
might be tempted to want to inject in
this models or we might want to let this
models learn everything and to end and
of course we want these models to be as
accurate as possible but this might
conflict somewhat with the efficiency of
of the models so the search space is
huge and yeah there's really a lot of
work to be done so today I am going to
guide you a bit through this to explore
this space and we're going to release
really take these constructions
supervised image classification and we
are going to try to replace each word in
this and see what we get so in the first
part we are going to replace
classification and see what other tasks
are there then we are going to replace
image like we will replace the single
image input and see what other inputs we
can use for for our models and then
we'll replace the supervised part and
try to go like beyond the strong
supervision so I will not be talking
here about the unsupervised learning
which is a very important topic because
this will this will be covered later in
the in the lecture series by one of my
colleagues I will talk about sub
supervised learning and then we will end
with with a few open questions in in the
field so in the previous lectures my
core
have introduced deep learning as this
this deep learning puzzle is being a
very very flexible framework where we
have this computational nose and then we
can use this computational nose to learn
a mapping between input nodes and output
nodes so and then we use we have this
loss function node we have the loss
function node that helps us train the
system so that it makes prediction that
are as close as possible to the desired
target so by the end of this lecture you
will know how to redefine these building
blocks so that we can perform different
visual tasks using different types of
inputs and different forms of of
supervision okay so let's start with the
tasks beyond classification so the
topics that that we will cover so we
will see different task definitions and
we will see models for these tasks and
how to train and evaluate them and then
some tricks of the trade
we will not go exactly in this order
over the topics because they are really
like intertwined so but these are the
topic that we will cover so other
important tasks for porcine
understanding but that we will not have
time to cover i'm listening here to for
example image captioning where you take
you give a given image to a system and
then the system has to generate a
language description of the image so for
example here the system could generate
two bit chairs under an umbrella on on
the beach and this problem could be
stated as a classification problem where
you give like let's say a thousand
different possible answers and the
system has to choose the correct one or
the system could generate the
description like in a free-form language
which is a more difficult task but may
be more meaningful actually and another
task that is also important for forcing
understanding for is pose estimation
where we actually define key points on
the human composed estimation where we
define key points on the human body and
then we want to detect those and to
track those so that we can learn about
the position of the people in in the
scene and this
example is very useful in any gaming
application that you imagine okay so
let's see the touch that we are really
going to cover and I've listed them here
like in an increasing increasing
granularity like we want to get more and
more details about the scene so with in
the previous lectures my colleagues
covered classification where you only
get like a sparse as far as description
of the image like just a few words and
then today we will see object detection
semantic segmentation and a note on
incident segmentation okay so task 1
object detection so first of all object
detection is a multi task problem
because we want to classify the objects
like to assign it to a category and we
want to localize the object like we want
to know the position of the object in
the image like what I'm showing here I
want for this object to know that it
belongs to the class sheep and then I
want to know the location and I'm
indicating this to this bounding box so
for this for this task the input to the
system is an RGB image with height and
width and three channels of color and
then the output it's a class label like
a one one hold one hot encoding for the
for the class label where we have zeros
everywhere for the index of the class
and then we want to have bounding box
around your object for the for the
location of the object and generally we
parameterize this through the
coordinates of the center and height and
width and we want to do this for every
object in the scene so for the for the
previous image that I've shown this
would be the desired output okay so to
Train such a system we would need a data
set where we have samples for training
and for testing and then in this data
set we would have samples and then each
sample would contain we'd contain an
image as I said an RGB image and then we
have a list of objects so for different
images we can have different numbers of
objects so then here we have this list
of objects and then for each object we
have the label as a one hot
and then we have the volume box with the
four coordinates so in the in the
previous lectures you've seen how to do
classification now the question is how
can we do how can we predict bounding
box coordinates which are real values so
like a quick quick recap in
classification we train a system we
train a mapping that assigns input data
points to two different classes so for
example if I have a capture a cat image
it put it in a cat category if I have a
car it's it assigns it to the car
category so the output in this case is
discrete
however for bounding box prediction we
we want to have a continuous output and
let's assume that we have a model that
has has an output module with four units
that can give me the four the four
coordinates of the box and then I would
have something like this so let's say my
ground through the bounding box is the
green one and then my system gives me a
prediction like the red one now I want
to train the system I need to give it
feedback on the the accuracy of this
bounding box I mean I want to say okay
is it you it needs to be smaller bigger
it needs to be more to the left or to
the right and with in the classification
setting we cannot do that like we cannot
give this kind of feedback because there
the data generally is not ordered when
we do classification so if I give you an
image of a cat and you mistake it for a
chair or do we take it for a car like
its it says worse like you cannot say oh
that was worse than the other whereas
for a bounding box you can say okay that
box should be bigger that box should be
moved to the left and we can achieve
this doing regression instead of
classification because in regression we
have continuous continuous outputs so
the way we generally do regression is we
minimize for example the quadratic loss
there are other losses but quadratic
losses is a simple one
so for example here I have he would be
my ground truth like the four ground
roots coordinates of the bounding box
and then X would be the prediction of
the of the network and then the goal is
to minimize the distance between now
between the two so we minimize the mean
squared error over the samples like a
quick quick recap classification versus
regression as I said in classification
we aim to map inputs to two predefined
classes
whereas in regression we want to map
four inputs to continuous continuous
values of course in output then the
output will be sorry in classification
the output will be discrete whereas in
regression it's a continuous value and
then we have this difference this
distinction that in classification there
is no order in the data whereas in
regression we have a notion of order in
the data and there are different
algorithms that can be used to perform
the tool now so it's all good we can do
regression to do bounding box prediction
now the problem is that generally we
will have more than one objects in an
image as I've shown and we want to
detect all of them so when I make a
prediction how do I know to which ground
through the bounding box and should I
compare it so one might say okay you
take the nearest bounding box and then
you assign it to that but this can get
like very messy very quickly and another
way to deal with this which is slightly
cleaner is to actually approach the
problem in different in two in two steps
first we do a classification such that
we roughly say okay to which to which
bounded to which ground fruit bounding
box your prediction belongs to and then
around that around that rough prediction
you will refine through true regression
so and with like the conversion between
regression tree from a regression to
classification we do it by discretizing
the output values and what I mean by
this is let's say that my ground truth
is like a value like that like 378 and
instead of predicting that which could
be very hard
I actually just been the output space
yeah and in equally-spaced bins like
here I have like nine nine minutes and
then I project my value and wherever
that hole that will become my one and
then this will give me the one hot label
that I can use in classification and
then if I have another value then I just
get this other one hot one hot label
like yeah like I don't like if you know
the game hot and cold
maybe that rings a bell what it means
like if I if you think of something and
I have to guess what you what you
thought of at first we would play it
like this right you would tell me hot
if I'm close to what you're thinking of
or cold if I'm far and very hot if I'm
getting very very close so first of all
I do this classification like I you and
once I'm I'm in the area where it's very
hot then there I can I can regress and I
can make the right prediction like in
that in that space so this is kind of
the approach that that we take here in
two steps so first classification and
then regression in that local local area
okay so let's see now
detection detectors that can actually
that actually apply exactly this
strategy like first the classification
and then and then refer refinement rule
regression and I will present two two
case studies for this and I think like
first of all there are many many papers
published on this topic but I've chosen
two that have very good accuracy and I
think they're very they're
representative for the like for the
class of hole detectors so faster are
CNN it's as two it's a two-stage
detector which in in which first you
identify good candidate bounding boxes
and then we refine truth regression so
the way this looks like so we just have
first okay weird I said yeah so first
you have your input image and you pass
this through a block of four
convolutional layers where we can all
like we have convolution rail you
fooling as you've seen like in the
classification models
well so we pass we pass these two blocks
of convolutional layers and then we
obtain here feature maps and then these
are passed through to this branch to get
to identify the most promising bounding
boxes and then once we have these
candidates then we passed then we we
collect the the features from here like
we collect the features corresponding to
these body muscles and then we do our
final final regression and
classification so first first step as I
said we discretize the output space so
discretizing the output space means
disguising the bounding box face so as I
said the bounding box has this four
coordinates so we have two first two
coordinates for the center and then
heightened with to discretize the space
of the center's we just choose this
anchor points and we distribute them
uniformly over over the image and then
for to cover to discretize the space for
height and width we just choose
candidate bomb candidate bounding boxes
with different scales in ratio and then
this will give us like nine candidates
and three and candidates Barranca anchor
and generally we choose like three
different scales and three different
ratios so that kind of in general we use
nine candidates per anchor and then we
train a classifier that predicts four
for every box predicts an objective
score so here we just say is there an
object or not in the box we don't we
don't care what the class of the object
is we just want to know if there is an
object or not and then we sort and keep
top top top candidates like the the
bounding boxes where the model is most
confident that there is a that there is
an object there and then we refine true
regression so imagine we would have an
MLP like if you feel fully connected
layers that they have at the end four
units to give me the four the four
coordinates of the bounding box so now
we know how to do that so and this gives
us a good good accuracy actually and
they tries to look at five frames per
second however something that I didn't
make explicit is the fact that the
operation that we have here here when we
do when we take the bounding boxes and
we keep the most promising ones actually
this operation is not differentiable so
if you've attended the previous lectures
my colleagues mentioned that okay deep
learning is very flexible and we can
replace those pieces in the puzzle but
we have to put their functions that are
differentiable so that we can backdrop
gradients to them to train the system
however the operations that we have here
they're not differentiable so we cannot
back drop gradients to here
so in particular we cannot back from
gradients with respect to the parameters
of the bounding box because we've just
chosen those we fixed those in advance
so and I just added their note there
there are ways to to make this operation
differentiable
however it would complicate things quite
quite a lot but I encourage you to check
this paper by other berg at all the
spatial transformer networks and there
they showed how you can do this
operation the cropping operation that we
have here in a differentiable way so
okay so as I said yeah I'll just add one
more note here that because this is not
differentiable here what this means is
that we have to train the system in two
steps so first we train this part here
and then we train and then once this one
is trained so this one to train it we we
just generate object nests ground fruits
labels and the object has grown through
labels so we just check if any of the if
any of the bounding boxes here if they
overlap significantly we reach one of
the brown fruit boxes yeah
and then this is how yeah we trained
this true through classification and
then once this part is trained we just
put it into the bigger into the bigger
model and this part does not get trained
after that right so this one is just
used to give me some some candidates but
the backpropagation happens and through
here right so that that is what we mean
by two stage detector right so first we
have to train this region proposal
Network if I probably didn't mention
them before the name like it's called
the region proposal Network and then we
we plug it into the bigger classifier
detector and then we we get the full
like the full object detector so this is
a and we get as I said we get good
accuracy with a good good speed but it
can be quite cumbersome to actually
train a system like this so then we
would really like to have a one stage
detector where we where we train
everything and Tran and we don't have
these issues of non-differentiable or
building blocks so I'm presenting here
another so the second case study this
retina retinal net which is a one stage
detector so I will just say to ignore
this part with the hierarchy of here
from here because I will explain a bit
later what what that exactly is but
let's here focus on the like on the
bigger picture so what it does it just
okay I'll just go back to this one to
show to say so here we just took
convolutional features that one scale
from here right and then we generated
our bounding boxes candidates and
predictions whereas in this one we have
this therapy your cure features at
different resolutions and I will explain
later how exactly we obtained this and
then from the different scales we just
generate predictions for bounding box
coordinates and for the object object
classes so as you can see like the
output here so here we have the output
for the for the classes of the object so
that's K k stands
for the number of classes that I have in
the dataset
and a stands for the number of anchors
of volume boxes and then for the second
head here we have four times a so four
stands for the four coordinates of the
bounding boxes so okay so why don't we
just train like this all the detectors
why why do we need to have the two stage
detector that I showed before so
actually it turns out that if we just do
this for all the all the bounding boxes
that we have in the image actually we
will end up with a very poor learning
signal and we will never get the very
good very good
accuracy and why this happens is the
following
so when when we want to identify the
promising candidates we use the cross
and two pillows so this is the loss for
classification now if we if we analyze
the graph of this of this loss so you
can observe that the loss penalizes
heavily like here when so this is like
the probability of the ground crews
class and the the loss is high when the
detector is not confident in the in the
correct class right however it goes down
very slowly
when we are starting when the detector
starts being correct right so even here
like when the when the probability is
higher than 0.6 so that means like the
detector is quite confident that in the
right class even there the loss is still
quite significant it's higher is higher
than zero and now when you think that we
have a ton of examples like this and
they most of the time belong to to the
background because in an image we will
have few objects that's most of the
bounding boxes we will come from the
background this will result in really
overwhelming the the useful examples
right so and this is why in the previous
detector we had this two-stage operation
right so the first stage was really
responsible to prune all those easy
negatives right that or that are just
from the background and they really
don't help much to train our system and
in general once a detectors to avoid
this problem that the edges from here
they employ hard- mining heuristics so
what what this means is that the way you
train a detector like this with hard-
mining is that when you build your
training set you start with the set of
positive examples wherever you have
objects and the bounding boxes
corresponding to those objects and then
you get a random subset of negative
examples so from the background and we
take a subset because as I said like the
full set would be would be too big so
and then you train the detector with
this lets more balanced more balanced
training set then we test it on unseen
images and then we check where the
detector make mistakes and it will make
mistakes because we've considered only a
subset of the negatives so which means
that we are probably not covering well
the distribution of the negative of the
negative examples so then we check where
the detector made mistakes so we check
for false positives so wherever the
detector said that there was an object
here I'm giving for a person and those
would be our hard negatives that we need
to add to the training set so that and
then we retrain it such that the
detector can learn to classify those as
negatives and we can iterate this
several times so this is an option and
I'm encouraging you to go to check this
reference which gives a really in-depth
presentation of this technique of heart-
mining with a nice formalism using the
Bayesian theory however it is a very
quite complicated
training procedure so instead of doing
this the the detector that I'm
presenting here the retinal net actually
comes with a much simpler and more
elegant solution so instead of being all
this hard
remaining heuristic it actually modifies
the the learning loss it modifies the
cross-entropy such that the the example
that are well classified that are in
this region they receive the their
downscale so we just add the weighting
factor so that they we put less energy
into those into those examples so and
you can simply do this by adding this
this term here so this is the focal loss
so this is the loss that they proposed
they call it the focal loss so simply
you take your process entropy and you
add this this weighting term so what
this means is that when you're you're
confident in this in your prediction
then this this term here will downscale
your own will don't scale the loss for
those zone for those examples and now
this leads to a really good accuracy and
this is also faster than the it's faster
than the faster are CNN that I've shown
before so this is considered now like
state of the art for for object
detection and it's a simpler detector as
I said it's a training in one one stage
so yeah it's it's much nicer okay so
that was for object detection let's see
now semantic segmentation why do we need
semantics annotation so actually for
certain types of object the bounding box
that we get from from object detectors
might not be a good representation so
for example here if you look at this
person just because a person has the arm
spread the bounding box is like very
very wide and then if you look inside
like most of the pixels in this bounding
box they do not correspond to to a
person and they actually belong to the
background or to other objects so we
want a more refined more refined
representation and we can get this
through semantic segmentation so here we
really go to the extreme and we say okay
I want to us
join a class label to every pixel in the
image and then this is a like a very
very refined refiner presentation so as
you see here I want I put a label on all
the pixels that belong on only the
pixels that belong to a person or I put
this label on all the pixels that belong
to the class sheep or plus dog and all
the rest is in background now we've seen
how we can do classification but now the
problem is okay how do we do this kind
of dance prediction because so far we've
done like sparse sparse prediction where
we only generated one label per image
now we want to generate an output that
has the same resolution as the input so
if you've attended previous lecture
you've seen this operation the pooling
operation and pooling operation is
responsible with reducing the resolution
of the future maps and we generally do
pulling cooperation such that as we go
deeper into into a model we want the
units to have an increased receptive
field and we want this because like when
you have a larger receptive field then
you can you can extract more abstract
features so pulling operation is where
you just compute min or max yeah like
you take a window and then you compute
min or max over that window and that is
the value that you copy in your output
feature maps so now to to up sample to
get a denser that's output from our
network we need the reverse operation so
we need an unfolding operation so the
simplest operation we can do is to do
like nearest neighbor up something so I
just take the value in my in my feature
map I copy it in the corresponding place
in the output in the up sampled output
and then I just I just copy in the
neighboring locations the same the same
value and I do this for all the for the
entire feature map and now I have enough
sample and up sample the sample
activations
so I just shown like the simplest of the
operations but there are other up
sampling methods so the thing with this
very simple operation is that you're the
result will be very blobby whatever we
get here is very blobby and this other
operations that I mention here for up
sampling they they want they try to
address this issue of blood - so for
example one thing we can do is to
improve with indices so we that mean
what that means is that when we do the
pulling operation we can preserve the
indices where the max pull like the arc
max of the max of of the max operation
in in the pulling window and then we can
use that indices to know where to copy
in the output and so that will give you
like an output feature map with with
holes but what we do after that we need
to apply like a convolution and then we
can smooth out the future maps another
another option is to do the convolutions
so basically there in the convolution
you just reverse the operation that you
have in conclusion and you can up sample
through that so I put here references
for for these other methods so okay now
we have a block that can do absently and
I will show here like as a case study a
network that that does semantic
segmentation so produces an output at
the same resolution as the input so this
is a unit and this was this was a
proposed for segmenting medical images
so especially a medical imaging
segmentation is a very important problem
and it really helps a lot like the
partitions in the in the field so in
this in this model so you have your
input your input image we go it goes
through different stages of convolution
conversion Rayleigh pooling as we know
we get here to the bottleneck so this
this kind of model
we call it like an encoder decoder model
so the part this part will be the
encoder and this is very similar to an
image classifier and then when we are
here we've reduced a lot the spatial
resolution now from here we need to
start up sampling so we go through the
decoder part to get an output that
really has the same resolution as as the
input and importantly as I said because
when we do the up sampling here we can
get like blobby blobby feature maps we
can we add this long skip connections
from the from the layers in the encoder
at the same level of resolution so by
adding this skip connections we can add
back in like high frequency details that
we might have lost when we did fooling
and on pooling so and these are very
important these keep connections like
you've probably seen them like in
resonance classifier and they're like
they are also very important for
training because like this makes a back
propagation of gradients easier like
because now the gradients will back
propagate through here but you will also
have register back propagate directly
through this skip connections okay so
and now to train this kind of system so
our output output will have the same
size as the input and times number of
classes because now for every pixel for
every location in the output I will have
a probability distribution over the
classes over the possible classes so
that's why so for every every location I
have the distribution over the classes
and now to train this we will use the
same crossed entropy that we've used
before for classification but now it is
average over all the all the locations
in in the input ok so now just a quick
recall of the retinal net the object
detector that I showed earlier where I
said that we don't need to understand
there how we get that here are your
features so maybe you recognize
is the same u-shape that we had before
just here it's a bit reversed yeah so
what they did there is like this is like
the encoder part yeah and this is like
the decoder part where they they did up
sampling here and then they added this
long skip connections from the
corresponding level Cinda
in the encoder yeah so this is like is
the same is the same idea that is used
in most in both tasks so this this just
shows again how flexible this models are
that we can really reuse the same kind
of ideas from one model to another okay
so now we know how to do how to do
semantic segmentation however this can
still be very confusing for especially
when we have overlapping objects from
the same class like here for example if
you just you if you have overlapping
objects in the same class you just get
like a big blob saying sheep but you
don't know where those sheep where one
sheep ends and where the second one
begins so what we can do is to avoid
this kind of problem is to do instant
segmentation so innocent segmentation we
actually combine object detection and
semantic segmentation so here in
semantic segmentation we just assign a
pixel to a class category here in
instant segmentation we also
differentiate the instances from the
same class so now we really have a less
confusing yeah a less confusing
confusing output and yeah we don't have
time to go in detail into this approach
but I encourage you to check the the
paper the masks are seen and paper for
example which is a good gives a good
solution for for this problem okay so we
have all these these detectors now how
do we evaluate them for the
direction or semantic segmentation so in
classification the to evaluate it's easy
we just take the accuracy so that's the
percentage of correct predictions out
out of the total number of predictions
and generally we use top one or top five
so top one you give plus one to the
model if the top prediction is
corresponds to the to the correct class
however in some dataset some classes can
be quite confusing like like even for
people it would be difficult to
distinguish between I don't know a chair
and an art chair or something like that
so then we can be more flexible in the
in the evaluation and we can use this
top five score so there you give a plus
one a correct correct score if the
correct class isn't the top five
predictions of of the model however for
object detection and semantic
segmentation the evaluation is a bit
different so we train object detection
for example in regression we train it
with a quadratic loss however we cannot
directly use that same measure to
evaluate the the accuracy the
performance of the detector because that
might be that might mislead mislead us
in how good that detector is and that
mainly because like quadratic loss is is
biased against small objects like if you
make like it considered it considers in
the same way a mistake let's say of ten
pixels that you make on a large object
or a mistake on a of ten pixels that you
make on a small object but for the large
object those 10 pixels might represent
like I don't know 2% of the whole of the
whole size of the object whereas for the
small object maybe it's more than half
of the of the object so we cannot use
reliably that measure to devaluate
performance so instead what we use to
get a better understanding its
intersection of a union so in in this
measure we just take the like the
intersection of the you take the
wrong truth and then the prediction and
then you take her in their intersection
and divide this by a by their union so
now actually we have like we train the
detectors with with quadratic laws but
then we evaluate them with this
intersection of our union this is not
recommended in general to use different
different measures between training and
testing because yeah you might get
surprises like your your detector might
be very good at raining but then when
you evaluate it you don't get the
expected result however we can not
directly train with iou because it has
some max operations when we compute ILU
and those are not differentiable so
that's why we cannot directly use that
however there are some personal written
papers here that I put there where they
they propose some approximations for
this for this measure for IU and then we
can actually train with that same
measure okay
so then in terms of of benchmarks so for
image classification we had image net
and really image net was kind of a
game-changer in the community because
having a benchmark for for a task can
really push progress like now things are
comparable people can compare their
works between them so it's important
than to have such benchmarks for
different tasks and here I just listed
to there several of them for object
detection and semantic segmentation
cityscapes is a benchmark for for
semantic segmentation they contain the
data set contains mainly like city city
city seems the filmed filmed in
different cities in Germany and and then
they gives ground truth for for images
and they also have some some videos and
then the second benchmark that I'm
showing here cocoa dataset
they have ground truths for for object
detection for semantic segmentation for
image captioning it's a rich rich
dataset so in general and in general
these these benchmarks they have like a
public platform where you can just
submit your model and evaluate it and
this really gives a good like for good
practices in the community because then
your model is evaluated by an objective
third party and then we can really
compare them compare compare results
okay so one of the tricks of the trade
so I've mentioned in the in the
presentation about the detectors I've
mentioned the the hard- mining which is
an important trick that appears in many
different places another trick that I
want to mention is transfer learning so
transfer learning is defined in terms of
a domain and a task so and a domain is a
set of features that follow some
probability distribution and then a task
is a pair like this pair so the task is
the pair Y and a function approximator
so Y is a set of ground truth labels and
then I have a function approximator that
takes in features beta points from my
from the feature set in my domain and
produces predictions and then I train
this function approximator
to to give predictions that are as close
as possible to the ground truth now when
we train all these kind of tasks visual
tasks like object detection or semantic
segmentation like we can notice that the
features must be shared because they're
so they are related like object
detection as I said is a classification
problem plus localization so the
intuition is that any features every
feature that were learnt in
classification they should be used like
useful for for object detection as well
so we should not
from scratch all the time when we when
we train a new model so we want to reuse
the knowledge and then we can reuse the
knowledge either across domains or
across tasks and we will see how to how
to do that so the simplest transfer
learning setting is to transfer across
tasks so let's say I have an image I
have trained an image classifier and now
I want to train I want to get an object
detector what I what I can do is to
start when I design my model for object
detection I just start from the image
classifier I might remove some of the
blocks which are not like because like
in object detection you have a different
structure in the output so what we need
to do is to remove the last layer so
let's say the object in the image
classifier and add new layers that would
adapt to the new structure that I need
in the output and then we just
initialize the weights of the layers
that I've kept I initialize them from
the from the image classifier and then
the extra modules that I've added those
will be trained from scratch whereas the
other ones that I've initialized from
the from the image classifier they can
be fine-tuned so I continue training on
that or they could even be frozen but
that it really depends on the problem
when when it's best to do what so this
is a very very useful trick to do that
you don't because training like takes
time and energy right to train a system
so we should not just throw away if we
have an image classifier like this is a
very common thing in in the community
almost every model that you see in a
paper they do start from an image free
train image classifier to like to reuse
the the feature that were learned in
classification and a paper that that
really studied this problem of transfer
learning between tasks is this the
economy does follow me paper
nice like a very nice analysis so they
consider like 2026 tasks visual tasks
like computing normals in the scene or
two key 2d key points let's see what do
we have like semantic segmentation what
we've seen object classification 3d Kim
point scholarization yeah so many
different tasks they they consider many
different tasks and the question that
they ask is okay in which order should
we learn this tasks so basically what
they do they consider a set of source
tasks and then a set of target tasks and
they train models for the source tasks
and then they they they check in which
how should they combine those source
networks to transfer knowledge to the
target networks but yeah have a look at
this paper it yeah it's a it's a very
nice and nice analysis okay and the
second case of transfer learning that I
want to mention is when we transfer
transfer across domains so this is a
project that a very cool project that
you might have heard of solving Rubik's
Cube from from opening eye so basically
here they use the robotic hand to
actually solve the Rubik's Cube and
doing this like in real real world using
like a real robotic hand would be
extremely difficult to train because it
really requires a lot of data and to do
yeah when you do this you're in a real
real world you can easily break that
robotic hand because at the beginning
when it's not trained it will do all
kind of weird weird movements so then
the important thing here is to actually
train first in simulation you train your
modeling simulation and then you
transfer this into the real into the
real world so here the source domain is
acceleration and then the the target
domain is the real world and like the
ingredient that makes this work this
transfer work is this automatic domain
randomization how they
call it so basically this has two like
two to two parts first is they do
extensive data augmentation so the
augmentation was mentioned in the
previous lecture is when let's say you
have a training set but of course a
training set very rarely will cover
exhaustively all the possible
transformations that you can have in
your inputs so what you can do is you
take your training set and then you
apply randomly transformations like
cropping or rotation or jittering yeah
this kind different kind of
transformations so that you can augment
your training set so that you cover
better than the space of transformation
and you can make your your model more
robust to such transformations so yeah
in this paper they use extensively this
kind of data augmentation techniques
plus they use again the hard negative
mining that I mentioned earlier such
that they can identify what are the most
useful transformations to use that would
generate the best training signal and
yeah after after that they were able to
actually run this than in real world and
to actually solve the Rubik's Cube which
I think it's quite impressive okay okay
so we've covered the first part with the
with the classification and we will go
now to see how we can what we can use
beyond the beyond just single a single
image input so I think for questions we
will take the questions at the end okay
let's see what other kinds of inputs we
can use for our network right like the
first question is why do we want to use
more than like we've seen we have very
nice models that can do object detection
semantic segmentation only with images
why do we need to use more than more
than just images and I'm going to show
here an experiment like a study that was
done on patients recovering from from
blindness so
they have undergone surgeries late in
life to recover sight so and they they
had this operations when they're already
able to speak so to communicate so they
could tell what they were seeing during
the during like their recovery period so
[Music]
so during like at one week I think or
two weeks after after the operation they
were shown this kind of this kind of
images and they were tested for their
visual abilities so yeah I guess these
are pretty simple images right it's not
there's no trick there there's no so
they were just asked okay for this for
the first image of how many objects do
you see yeah simple two objects
everybody agrees to object them and then
the same for the second and the same for
for all the images right and they were
shown like multiple trials different
instances of similar images to test for
robots and then yeah like this one
contains like a 3d object and they were
asked the same question how many objects
they can see and then this like to trace
what is the the longer curve in the in
the image and they were also shown like
natural images okay and they were asked
how many objects are in the image or if
they can recognize the images the the
objects in the image so important to
note to note is that these were like
children or adults who had already
interacted with this kind of objects but
through other senses not through true
sights so they knew what the triangle is
they knew what the square is they knew
what the cow is so and the results so
for the first test so in blue the light
blue we have the control group so like
people with normal sight yeah so they
did very well on all the on all the
tasks and then the these either three or
three patients like the results from
three patients and these are the they
got go in the first case
they were all three were correct on the
and then but they they were wrong in
both these cases here they were correct
here they were like correct but only
like 50% of the time here they were
so-so
and here they they were wrong all the
time now the simple and then okay for
the real image they did not recognize
the object they could not tell how many
objects are in the image and when they
were asked like to draw where where they
think the objects are in the image this
is what they drew so now when we look at
the results what what do we observe so
here they were correct because the
objects are completely separate right
but as soon as the objects start having
overlap they don't know how to separate
how to parse the image here they were
correct because the two different
objects they overlap but they have
different colors here for depth yeah
they were not very good
so like the conclusion is that at that
point in their learning in their
learning period after the operation they
were using contours and colored as main
cues to understand objects to understand
two separate objects in the scene so
here you see just following the contour
gave them the correct result but here if
you just start following the contour you
will not get two objects there right you
can say I don't know what number maybe
three or I don't know what number you
you might say so very interestingly
after that they were shown the same
images but of moving objects right so
the same type of images with overlapping
objects but now they are moving they are
moving in different directions which is
like relative relative motion and
actually now they did much much better
right so compared to before on this same
on just two types of images like before
they were really no no correct answer
now they really started to give correct
answers for that for those types of
images and another example where they
show the like this kind of a triangle
with some clutter in the scene they were
so so again at saying what is the object
there however when that object started
to move then they could immediately tell
what the correct object is so all this
is to say that motion really helps when
learning helps for object recognition
when we learn to see so it might not be
as obvious that motion is important once
you once you know how to see once you've
learned that but during the learning
process motion is very important and is
as important as contours and as
important as colors in distinguishing
objects and another another experiment
that was carried out on chicks and they
were shown videos of smoothly moving
smoothly moving objects or or frames
from video but taken not in order so
here you have temporally smooth object
moving object whereas here you have you
have frames from the same sequence but
they are not in order so you don't have
a temporal smoothness you know in your
input and what they've observed is that
the change that were raised using this
kind of input they've learned robust
representations for the objects like
representations that were robust to
transformation that they hadn't seen
during training whereas the the change
that were raised with this kind of
inputs they really over fit to the
transformation that they've seen and
they were not able to recognize the same
object under different different
viewpoints so again this is to confirm
that motion is really important when we
learn to see so the like the conclusion
is that using videos should be like the
the main direction in in training
also the video the they like the vision
models that that we want to design
because as I said motion provides
important cues for object recognition
and we have like a natural data
augmentation we have all these kind of
transformations that that occur in the
real world the translation scale 3d
rotation camera motion light changes and
everything so you get these in the video
and then you can train your model to to
learn representations that are robust to
this kind of changes okay so let's see
what this is what we will cover in this
in this part so we will see what we can
do when we have pairs of images as input
and when we have videos as input and
then we will see what kind of tasks
become available when we have more than
more than a single images input and we
will look at optical flow estimation and
actual recognition and then we will see
different types of models for for this
tasks and we will discuss a bit like the
challenges that we have that we
encounter when we use more than single
images okay so let's start with pairs of
images and we will look at optical flow
estimation so optical flow estimation is
looking at motion like tracking changes
from between them between images so
that's in this cartoonish picture let's
say we have two images and I know maybe
there's a wind there's wind blowing
there or something and in optical flow
we are given in optical flow estimation
we are given a pair of images and then
we want to say for each pixel in image 1
where did that pixel end up in the image
- right so the output of such a model
would be again a dance a dance output
like in semantic segmentation and each
so and we have real these are real
values like for each position and then
at each location we encode the true D -
D translation that that affected that
that pixel so we have the display
in X&Y yeah so maybe for this kind of
for this pair of images maybe it would
look something like this so these are
the pixels that moved in the x-direction
and these are the ones that moved in the
Y direction and we can see the like the
overlapping once they moved in both in
both directions and let's see a model
that can can do this very simply so flow
net this again is an encoder decoder
architecture so it's very similar to
what we've seen already like for
semantic segmentation and object
detection
I think the exception here is that we
have this encoder decoder but we don't
have the longship connections it would
have helped but probably they just
didn't think back then to use those long
connections so yeah we have so we take a
pair of images as input we pass through
this encoder and decoder and so we have
the decoder to up sample because the
output as I said we wanted to be as the
same at the same resolution as as the
input and then we we can train this in a
fully supervised regime and we can use
they use the Schloss the Euclidean
distance between the predictions of the
network and the ground truth and to
train this they use the flying jersey
that certainly data said that they've
invented so you might imagine that the
generating wrong truth for such a task
could be very difficult like if you just
show to a person two images and you ask
the person okay now tell me where did
each pixel from one image went to the
other image this is yeah it's impossible
to get this kind of labels from from
humans so what they did is they
generated automatically this this labels
so basically they just took they just
took real images and then they took 3d
models of objects like of familiar
objects like chairs yeah and then they
generated views from those those chairs
like the these are 3d 3d models like
meshes and then they just
and abuse from those from those models
and then they overlaid them over the
over the real images yeah and what I'm
from here like you have this is a pair
of images and we want to estimate the
optical flow between between those two
images and here I'm showing like the
wrong truth so as I said optical flow it
generally - we have two layers in the
feature map for displacement in X and y
here we are just using like a mapping
entry in three channels so that we can
display them as an RGB image but this is
like the gone through the optical flow
that that we want the system to learn to
generate and again this is this is an
example of transfer from sim to real
because we use this simulated
environments yeah to learn about motion
and then we test this in real world and
this actually works because so these are
completely non realistic images right
you don't see flying chairs anywhere
normally but what is real realistic is
the motion like because what we actually
want to capture is that pixels that move
together they belong to the same object
that's a does a the underlying
assumption and this the the system can
learn this from this kind of toys data
yeah and this actually this works well
okay now let's see what we can do and we
have a full video as input before we've
seen like pairs of sorts of images so
what kind of models can be used when we
have when we have videos as input so the
first like the obvious answer is to say
okay I will take my image model like a
segmentation semantic segmentation model
and I can just apply that on the on the
consecutive frames over video and that's
it that's my that's my video model and
we can do that and we can get decent
decent results so yeah here is just a
segmentation model that is wrong
over the consecutive frames you know in
a video and then we have like a
segmented video however the problem in
doing this is that we cannot take
advantage of the properties that we know
about about videos we know that the for
example that the videos are smooth that
so you don't have objects flying around
in any location so like let's see
so normally let's see a flickering
somewhere like here if you've seen it so
there are many kinds many parts and the
segmentation map that flicker from one
frame to the other so that actually
occurs because the frames are treated
independently
whereas ideally you would want when you
make a prediction for a new frame you
would want to take into account the
prediction from the previous frame
because things move smoothly you know in
in the real world right so this is the
like the the disadvantage of using this
type of models like image based image
based models so another let's say more
adapted model for videos is to use 3d
convolutions so you've seen 20
conversions in the in the previous
lectures for 4 images now we can
consider videos as a volume and we just
stack we just stack frames so then we
obtain volumes like this so I normally I
would have images that have height and
width in three channels now we stack
them along the time-dimension and we
obtain this this volume yeah big volume
and then we can apply 3d conclusions so
before like the kernel that we use for
2d convolutions we had a 2d shape and
now we have the kernels or also 3d 3d
kernels and you know in 2d you'd have to
you'd slide the you'd slide the kernel
over the the image over the spatial
dimensions you leave it to obtain the
feature the output the feature maps now
for for videos with 3d conclusions we
just need to slide in order in both in
space and in time to obtain the for
to obtain as output this volume of
spatial temporal and spatial temporal
features now some too important to note
all the properties from 2d conclusions
they apply to 3d convolutions as well so
all the notions about strided dilation
and padding they apply in 3d as well
something that it's maybe worth worth
noting is that 3d convolutions or
non-causal and what this means is that
when I compute the activation for this
unit here T for this unit at time T I
need the frame from time T plus 1 let's
assuming that I'm using a 3 cross 3 3
cross 3 convolutional filter so I need
to I need to have access to to the
future because the receptive field and
the for for 3d convolutions is symmetric
right you you look you look in the past
and you look in the future to compute
your activations and that's why because
you need to look into the future that's
why these are non causal operations and
this is fine for for offline processing
of videos right if I just want let's say
I just have a number of videos and I
just want to classify them into actions
different types of actions and this is
an offline processing so I can at any
point in time I have access to future
frames so that is fine we can we can use
that however in other applications like
real time applications in robotics or
any or any like cell for self-driving
cars or other applications that run in
real time you don't have access to to
the future frames you just receive a
frame at a time and then you have to
make computation to process that frame
and then you can move so in that case to
still use 3d convolution you would have
to use like a masked 3d convolution
where we
literally put like put a mask on the on
the weights of the filter that actually
need to look into the future and then in
that doing that we can use 3d
convolutions also for for for causal for
setting that required causal processing
okay so now that we have this powerful
3d convolution models let's see what we
can do for example we can do action
recognition so in action in action
recognition you receive you receive a
video as input so again we have the
volume T is the number of frames and
then the dimension the spatial dimension
of the frames and then three channels
optionally you could have a flow map
that was computed externally by like for
example the flow net that I've shown you
can so you can use this to your network
and then the output that we want is like
label for the type of action that we see
in the in the video and then yeah in
this case we would have like a cricket
rotor and very popular data set used for
this task was was collected by
colleagues and I in deep mind is
genetics genetics data set which is a
very large-scale data set so it tries to
get to the same scale as image net so
currently it is at 600,000 training
training videos but like the aim is to
have 1 million and we have 600 classes
and all the videos come from curated
YouTube YouTube videos and yeah it
should each video in the data set has
250 frames so it's about these are tips
of around ten ten seconds so 25 frames
per second like the current accuracy on
this data set it's around 82% so yeah
there's still room for improvement
all right so now just as a case I like
to see a model that can actually do
action recognition this is a very very
accurate model and efficient so actually
it uses like a two branch model so the
first branch processes processes the
video at a lower frame rate and then the
second branch looks at the video
processing the video at a higher higher
frame rate and this kind of two branch
model
the inspiration is from the human human
visual system like in the we know like
we know that the human humans process
have a to stream visual system so they
kind of borrowed the same intuition and
what they what they aim to do is so for
the lower frame rate stream they have
heavier feature Maps so they extract
more features from the lower lower frame
rate and then for the higher frame rate
here they extract only the future so
there their intuition here is to say
that this part is responsible with
extracting abstract abstract features
about the seal for example object
category right so if I have a chair in
the scene or if I have a person in the
scene I will still have that chair in
the scene even if I look only every
tenth frame right that the chair will
not evaporate or not or the person will
not disappear all of a sudden and so
this branch is responsible with
extracting this kind of abstract
information like the class class
category whereas this branch here that
looks at frames at a higher frame rate
is responsible with tracking changes
like things that can move very fast like
anything that is related to motion and
changes this is what but and they
hypothesize that this that you need less
features to do
to extract this kind of information and
then they merge the two streams like
different different points in the in the
branches and then they make the final
prediction by by concatenating the two
the two streams and this is actually a
very very good very good model so I like
just to to make a comparison we've seen
like the models that we we saw for
semantic segmentation and object
detection for images they also use like
a hierarchy key here your features like
where they looked at different
resolution resolutions in space and that
worked really well now this here this
model looks at different resolutions in
time yeah so they're kind of the same
the same principles of having this
hierarchy of features at different
resolutions okay so one more thing to
note again transfer learning the trick
of the trade that I mentioned in the
first part for the object detectors this
applies here as well so if we have an
image an image classifier already
trained we can use that to initialize
our weights for the for the 3d
convolutional model as well however as I
as I said in 20 convolutions your
filters are 2d in 3d for for 3d
convolutions your filters are now 3d so
the shape don't don't match but to to
deal with this we only need to tile
along the time-dimension so that we can
get like we just replicate we just
replicate the ways that we have here we
replicate a long time and then we can
usually initialize this and this makes
sense because like if you just tile an
image you will you will obtain something
that is a valid video it's a video of a
static scene that was filmed with a
fixed camera right so that's why it
really would make sense to do this kind
of operation and then of course this is
just to initialize your system then you
train the system to actually learn about
the motion and
all the changes in the in the videos
okay so working with videos is great and
I hope I convinced you that motion is
very important to learn about objects
why don't you why don't we all use use
videos well the reason is that there are
several challenges in doing this first
of all it's very difficult to obtain
labels so in it's difficult in general
to obtain labels even for images and I
will address this at this point in the
third part however the other challenges
that I I won't cover in detail the fact
that now we have we need a much larger
memory actually to do the processing so
imagine as I said like in kinetics
dataset one one training sample has 250
frames that is equivalent to training
like an image classifier with a mini
batch of 250 that's quite a lot like we
can't really fit that is Ayana on the
GPU another problem is that these models
can be quite quite slow because these 3d
convolutions require a lot of
computation and then of course all this
computation comes with a high energy
high energy consumption so like just to
give an idea like a current GPU that we
would use to run an object detector or
an action recognition model consumes
around hundred watts which is kind of
ten times more than the human brain and
the human brain really performs so many
tasks in parallel so yeah these models
are really energy hungry at least at
this point this this is how they are and
yeah well I think there there is hope to
improve on these models and actually
this is my main my main area of research
at this moment like improving the
efficiency of video models and we use a
lot inspiration from from biological
systems and from biological systems but
also like from general
Computers and like the first thing that
we are looking at is how to maximize
parallelism to to increase throughput
and reduce latency and we do this
because like there is strong evidence
from the neuroscience community that's
like the the performance of the human
brain is explained by the parallel
computation that it that it does so and
we have the same thing like in general
computer processors so all the
pipelining operations that you might be
familiar from computer computer
architectures so you have let's say a
larger larger instruction you break down
into a smaller smaller instructions that
can be executed by different units you
know in the your processor so then you
can actually pipeline the operation so
that all the units can work in parallel
so you don't need to do operations you
know in sequence and this is exactly the
same principle that we try to to bring
for for numeral network so that so that
we can maximize parallelism across the
depths of the of the network and another
thing that I'm looking at is how to
exploit the redundancies in the visual
data so when we have a high frame rate
video like all the like the in in local
neighborhoods the phrase will be very
very similar so we don't need to process
all the all the frames at the same
resolution all the time so basically
here we are training a model to blink so
humans blink a lot so it's it's thought
it was thought that we blink only to
clean the eye and that is true however
studies have shown that to actually
blink more more often than needed to
clean the eye so then the explanation is
that probably we blink so that we can
reduce the activity in the brain because
like the visual system really is the
part that consumes most of the energy in
the brain so if you can turn that down
like for now and then it's really very
useful so this is what I'm working on at
the moment
like how to make these models to learn
to exploit their analysis in the in the
data okay so okay I'll just briefly go
through the third third part where we
want to see what we can do beyond strong
supervision okay so we want to go beyond
strong supervision because as I said
labeling is very tedious and it's
tedious for images and it's even more
tedious for our videos like imagine it's
hard and like if you want to if you want
to uh if you ask a person to to create
ground truth for a segmentation task you
would have to take an image and then you
have to draw contours around the objects
and to say okay this is a car this is
the road and so on now imagine doing
this on a video so you have to multiply
that same task like 250 times so this is
yeah you can't really do that so like
labeling has become a research topic in
itself and how to make it easier so one
one direction for example is that you
ask humans to a human expert to only
label some frames some key frames and
then you can have models methods that
can propagate those those labels across
time exploiting the properties of videos
like the smoothness that that I
mentioned
however when we don't have labels what
can we do so we might have different
information about the data and not the
strong labels that that we know so in
the general setting when we use the
standard losses like cross entropy and
means further the goal there is to learn
the mapping between some inputs and some
output distribution or values like we've
seen for detection or or segmentation
but now if we have a weaker some some
weaker labels or some weaker form of
similarity in the data we can actually
use metric learning and now we can learn
about distances between the inputs so
for a very classic example or
this case is that we might have a data
set of images and we know which images
belong to the same person right and now
what we can do is to learn an embedding
where the images so as you can see the
images belong to the same person but
they are really very different like when
we look in the pixel space right and
this again they're all very different
but we know okay that this picture they
all come from the same person
and then this come from another person
and what we might want to do is to train
to learn and embedding where the
pictures that belong to the same person
so we just this is our label same person
or not or different person and we just
want to train a system that clusters
together in this embedding clusters
together the pictures from the same
person and then puts far apart images
that come from from different persons
and another so and then this can be used
like for for image retrieval like I'm
given a new a new image and then I want
to know to which person does this belong
then I can this just project this into
my embedding space and I can get the
nearest neighbor there and then I know
to which person this picture belongs to
another possible application for this is
for example when you have medical data
and maybe it's a high dimensional data
that you don't know how to interpret but
what you can do is to train a model that
learns this kind of embedding and then
you can learn okay what makes two people
different and if you know that okay
maybe one person has a disease then you
can you can try to trace where where are
the differences why what causes that
that disease so the things that you can
use in this kind of space for for metric
learning so there are different losses
that we can use so to replace like the
cross entropy and the mean squared error
that we've seen so we can use
contrastive loss or triplet loss and I
will just mention briefly like a paper
that was published like a week ago on
that is now the state of the art in in
this field so yeah
as I said there are different
applications that become available when
we do this kind of fun this kind of when
we train this time two types of four
models and yeah we don't have time to go
in detail but you can you can check
these references to get an idea
so for contrast applause as I said we
have pairs we just have let's say we
have a dataset where we have pairs of
data points and we just know if those
points belong to the same same person or
not same person and then what we want to
do like the label will be one if it's
same person or zero otherwise and then
we want to Train anybody to train
network that projects my data points
into an embedding where points from that
are saying they attract each other and
then they're different they reject each
other so and then we can use this
contrast loss to Train such a such a
system so you can easily observe that
when when when I have y equal to one so
it's the same person I want to minimize
the distance between the two points
whereas when I have y equal to zero so
these are different people I want to
maximize the difference in the distance
between the two however we don't want to
maximize like infinity the distance we
don't want to send them too far apart
because that would be like consuming
energy for no good reason so that's why
we put a margin and we say okay as soon
as those points are further than the
margin I'm happy enough all right so
this is exactly what we're plotting here
so this the blue curve corresponds to
two pairs that belong to the same person
so as the distance increases the loss
will increase as well whereas the red
curve corresponds to different points
that come from different people and then
this loss for a margin of this value
beyond this value this loss is zero so I
don't care anymore
to push those points apart as soon as
they are further than this margin M now
the problem with this is that how do we
choose this
margin because this would be like a
fixed margin for all the classes and
then some classes might be more more
distinct more varied like in the case of
the pictures that I've shown with the
faces is for some persons the pictures
might be more more similar for some they
might be very very different and then if
we impose a margin em like this for all
the people then I would force all all
the classes to be clustered you know in
a ball with the same ranges em and this
is not like this kind of can lead to is
unstable training what is a more robust
to do and better to do is to use a
triplet loss for example where we can
actually use like relative distances so
in the triplet loss I'm looking I'm
using like triplets with one anchor
point and then I have a positive point
and a negative point and then I want the
positive pair to attract and then the
negative third to reject but I want to
reject this only search that is further
away than my than my positive right and
now I also have a margin but now this
really allows to have classes that are
embedded in different and different in
in balls with different ranges so then
we can really allow for for some
distortions in the in the embedding
space
so again hard- mining returns that what
I mentioned earlier it's really
important how we select the triplets so
that we train the systems and yeah I
encourage you to have a look at this
paper sampling matters in our
independent learning to see in detail
how they do that and very quickly like
state of the art in representation
learning here so is the probably the
simplest type of metric learning that we
can have so before I showed examples
where you have same person so that's
your label it is the same person here we
have the label the same image right so I
just have an image and I apply different
date augmentations to the image
then I just train a system to say okay
all these different data orientations
they actually belong to this image and
then when I show another image of
another dog and I apply different to
date augmentations to that other dog the
system needs to learn okay this actually
comes from a different image right so
now like the label now is this same same
image and this is like really easy to
obtain because yeah you just generate
automatically all this data all the data
augmentations and you can turn the
system and actually this now achieves a
really really good accuracy which is
comparable so this is this new data
point and it's parable with the
supervised regime so the representations
that you get using this kind of metric
learning where you generate
automatically the labels these
representations are now as good as the
representations that you learn using the
full labels from from imagenet and this
is like really really important for it's
a really important result for the for
the community okay
and yeah I will just end with just few
words because I think we're all kind of
out of time on the open questions in the
in the field so first question so we've
seen so many so many cool applications
we can do object detection semantic
segmentation optical flow estimation and
actually recognition and more is visual
salt right however I think a better may
be another question to ask is what does
it really mean to solve vision and I've
said at the beginning that we really we
want human levels in understanding that
that's like the holy grail however how
do we benchmark that what is the right
task like okay we can have a data set
for object detection but data sets can
over fit like models can over fit or it
can work well on a data set but then
when you train it on another data set it
doesn't work as well so it's really hard
for at least at this moment it's hard to
say what do we need to achieve to say
that ok we've sold we've sold vision and
then another very important probe
is how to scale systems up so I've shown
models that do most of the models that I
shall do one task like they do object
detection or they do optical flow
estimation and so on
but we really need a system that can do
all of them like in the same way that we
as humans can we can do all that's what
we aim for so I really believe that to
scale system up systems up we really
need to look at model parallelism and
maybe we need a different kind of
hardware yeah a better hardware and for
sure we need to look at methods that
require less supervision and like a
question is okay what are good visual
representations for action because in
the end we want to have this dispersal
visual models and to put them into an
agent that can then act in the world
like what are the good representations
that is semantic segmentation a good
enough representation I think not but
yeah this is still really open for
discussion
and I just put here like a nice paper
that for example uses key points to
represent objects and they test this and
I can do control tasks so I guess the
takeaway message for for this for this
lecture would be that I really believe
that learning to see from static images
makes things things harder than they
should be and and unfortunately because
of different challenges this is the main
the main area of research at the moment
but I really hope that things will
change and we will start be using more
and more videos and I think we really
need to rethink vision models the way we
design them and how we train them from
the perspective of moving pictures and
really having the the angolan in mind
like to think about the intelligent
agents that would interact with the real
world and interact in real time yeah so
thank you
you


Do We Get the $100 Trillion AI Windfall? Sam Altman's Plans, Jobs & the Falling Cost of Intelligence

in the last few days Sam Altman the CEO
of openai has publicly stated how much
money he expects the company to make and
how he intends to distribute it many
people will assume he is bluffing but I
think GT4 shows that he's not this video
will cover his plans his predictions of
massive inequality and open ai's new
paper on job impacts together with just
released studies that back it all up but
let's start with money this week in the
New York Times he said that his Grand
idea is that openai will capture much of
the world's wealth through the creation
of AGI and then redistribute this wealth
to the people and yes he mentioned
several figures a hundred billion
dollars one trillion even a hundred
trillion dollars if openai make even a
fraction of these figures Sam Altman
will become one of the most important
people on the planet that's not to say
that he would become that rich The Wall
Street Journal this week reported that
he has no direct Financial stake in the
business but deciding where trillions of
dollars of wealth go does make you
incredibly powerful so where does he
want all the money to go well he seems
to have two main ideas plus a third one
that I'll touch on at the end his first
idea is Ubi or Universal basic income we
also have funded the largest and most
comprehensive Universal basic income
study as sponsored by open Ai and I
think it's like an area we should just
be be looking into how exactly would
that work well he laid out his theory in
this blog post and he began it with this
he says he's reminded every day about
the magnitude of socioeconomic change
that is coming sooner than most people
believe he said that the price of many
kinds of Labor which drives the costs of
goods and services will fall towards
zero once sufficiently powerful AI joins
the workforce he said that that was
great for people buying products but not
so much for those working to earn a wage
so where would their money come from he
proposed something called the American
Equity Fund it would be capitalized by
taxing companies that were above a
certain valuation 2.5 percent of their
market value each year and it would also
be funded by taxing 2.5 percent of the
value of all privately held Land by his
calculation that will be worth around 13
500 in about twenty Thirty and he said
that that money would have much greater
purchasing power than it does now
because technology would have greatly
reduced the cost of goods and services
it does raise the question for me though
about those countries that aren't the
home of massive AI companies where are
they going to get the wealth from on Lex
Friedman's podcast he admitted it wasn't
a full solution I think it is a
component of something we should pursue
it is not a full solution I think people
work for lots of reasons besides money
he thinks much more will be needed
because the cost of intelligence could
fall to almost zero my basic model of
the next decade is that the marginal
cost of intelligence and the marginal
cost of energy are going to Trend
rapidly towards zero like surprisingly
far so what is his other main idea
simply use the money to fund science are
you planning to take the proceeds that
presumably you're presuming you're going
to make some day and you're going to
give them back to society I mean is that
yeah whether we do that just by like
saying here's cash for everyone totally
possible or whether we do that by saying
like gonna like invest all of this in a
non-profit that does a bunch of science
because scientific progress is how we
all make progress unsure but yeah we
would like to operate for for the good
of society even with these two ideas he
admits there's still a big problem as he
put it recently he sees a lot of people
getting very rich in the short to medium
term but others might not fare as well
if it is as divergent as I think it
could be for like some people doing
incredibly well and others not I think
Society just won't tolerate it this time
samuelman isn't the only one making
predictions open AI itself released this
paper around 10 days ago it calculated
that with access to a large language
model about 15 of all work tasks in the
US could be completed significantly
faster at the same level of quality but
crucially when incorporating software
and tooling built on top of llms this
share increases to around 50 percent of
all tasks that is a colossal impact For
Better or Worse just with gpd4 plus
software on page 17 of the paper it had
this table which I think captures a lot
of the interesting analysis let me
briefly explain what it shows we have a
column of example occupations in the
middle and the the education that is
required for each of them and the job
preparation but the numbers on the right
are where it gets interesting these are
the percentages of exposure graded Alpha
Beta And Zeta the human assessment of
exposure is titled H and the M is for
the machine assessment they actually got
gpt4 to do an assessment too notice that
for the most part gbt4 agrees with the
human assessors so what are these three
grades Alpha is the proportion of tasks
in these occupations affected by current
language models alone without any
further advances or Integrations beta
represents the percentage of tasks
exposed in a realistic scenario of
language models plus a bit of software
integration and a few advances you could
think of it as their median prediction
finally Zeta is a bit like their most
extreme scenario with full adoption of
software plus advances of llms by the
way we're not talking gc5 here or text
video just basic software integration
like a longer context window or text to
image the trend that immediately stuck
out for me was how when you go up the
educational levels and these salary
ranges the effects of these large
language models on task exposure goes up
and up and up until you reach master's
degree or higher levels then it seems to
dip down a little maybe this is why Sam
Altman predicted inequality the people
on the very Cutting Edge of science
would still get paid well probably
better than ever but there may be a
further hollowing out of the middle
class with working class occupations
left largely untouched the paper also
touches on why so few people might be
currently focused on language models I
don't know about you but have you
noticed that feeling where it seems to
be us being super interested in this
technology with most people not being
that interested well here might be one
reason why currently only three percent
of U.S workers have over half of their
tasks exposed to llms but that's only
when considering existing language and
code capabilities without additional
software or modalities so not that many
people are seeing a massive change in
their work but it says that when we
account for other generative models and
complementary Technologies are human
estimates indicate that up to 49 of
workers could have half or more of their
tasks exposed to llms whether this means
doubling the amount of work done or
halving the number of workers doing it
I'll talk more about later in the video
but maybe this was the dramatic economic
impact that Ilya satsukver once
predicted on Lex Friedman what do you
think is the bar for impressing us do
you think that bar will continuously be
moved definitely I think when you start
to see really dramatic economic impact
that's when I think that's in some sense
the next barrier because right now if
you think about the work in AI it's
really confusing it's really hard to
know what to make of all these advances
the paper also points out that the
growing economic effect of llms is
expected to persist and increase even if
we hold the development of new
capabilities today they refer to recent
study bodies revealing the potential of
llms to program and control other
digital tools such as apis search
engines and even other generative AI
systems in my previous video on the
self-improvement in GT4 I mentioned
hugging GPT but I am doing a lot of
research on the new Microsoft Jarvis
model and auto gbt I'm hoping to bring
to you soon but interestingly there were
some tasks that neither Gypsy 4 nor the
human assessors could quite agree on in
terms of the impact that llms would have
even gpd4 couldn't quite figure out if
meetings and negotiations would carry on
or to what extent counseling or other
jobs that involve empathy would be
affected and the paper concludes with
this the power of relatively simple user
interface improvements on top of models
like Gypsy 4 was evident in the rollout
of chat GPT wherein versions of the
underlying language model had been
previously available via API usage
skyrocketed after the release of the
chat GPT internet
it's a great Point once these models are
made easy to use that could change
everything the paper then picks up on a
particular survey that shows worker
adoption of llms here is the survey with
the rather dramatic headline of one in
four companies have already replaced
workers with Chachi BT I don't think
that assertion is fully backed up by the
evidence but they did survey 1 000 U.S
Business Leaders and there were some
interesting findings on the question of
replacing workers it says that when
asked if Chaturbate will lead to any
workers being laid off by the end of
2023 33 of Business Leaders say
definitely while 26 say probably others
are a bit more optimistic Goldman Sachs
said this this economic analysis was
published only a few days ago and they
say about seven percent of workers will
be fully displaced over the next 10
years but that most are able to find new
employment in only slightly less
productive positions they also predicted
that generative AI will raise overall
labor productivity growth by around 1.5
percentage points per year which
effectively doubles the rate going back
to Sam Altman last week he was asked
about this augmentation versus
replacement question so in terms of
really replace jobs is that a worry for
you it is uh I'm trying to think of like
a big category that I believe can be
massively impacted I guess I would say
customer service is a category that I
could see there are just way fewer jobs
relatively soon I'm not even certain
about that but I could believe it
whatever call center employees are doing
now I found that last comment on call
Center's quite interesting given that
the gc4 technical report talked about
using language models for upskilling in
call centers so does this mean immense
productivity in the short term but
replacement in the long term a couple
days ago Sam Altman put it like this I
always try to be honest and say in the
very long term I don't know what's going
to happen here and no one does and I I'd
like to at least acknowledge that in the
short term it certainly seems like there
was a huge overhang of the amount of
output the world 1 and if people are way
more effective they're just doing way
more we've seen this first with codeine
and people that got Early Access to
co-pilot reported this and now that the
tools are much better people report it
even more yep but we're now in this sort
of gpt4 era seen it in all sorts of
other jobs as as well where you give
people better tools and they just do
more stuff better stuff the productivity
point is backed up by experiments like
this when developers were split into two
groups half that used openai's co-pilot
and half that didn't not only did more
of those who use copilot finish 78 to 70
they finished in less than half the time
this paper from a few weeks ago shows
that when white collar professionals
were given a language model like chatbt
the time they took to do writing tasks
dropped massively compared to the
control group you can see that they took
less than 20 minutes versus almost 30.
and when the assisted group and control
group were blindly graded you can see
that the mean grade was higher for those
who use the language models but surely
if productivity goes up that means
higher wages for those jobs well not
necessarily a couple of days ago Sam
Altman laid out how it might be more
efficient to use one worker to do the
tasks of two or three there's a huge
cost premium on work that has to be
split across two people there's the
communication overhead there's the the
miscommunication there's everything else
and if you can make one person twice as
productive you don't do as much as two
people could do maybe you do as much as
three and a half or four people could do
and for many kinds of tasks but is there
anything that might slow this economic
impact down I think there might be a few
things starting with politics this
survey from Youth Of America was
released only three days ago and while I
think it is a somewhat leading question
it does show that over 69 of Americans
would support a six-month pause on some
kinds of AI development and if we see
dramatic negative economic impact I
expect that figure would go higher
politicians would then be in
incentivized to slow down tax and or
regulate AI development indeed two days
ago President Biden tweeted this when it
comes to AI we must both support
responsible Innovation and ensure
appropriate guard rails and also don't
forget if you live in a country where
English is not the main spoken language
gpt4 isn't as good notice that in many
languages found in India GT4 is worse
performing than the previous model GPT
3.5 is in English this is just one
reason why Goldman Sachs predicted
different levels of Automation in
different countries the next Factor
could be cultural pushback when Levi's
wanted to test AI generated clothing
models and they said their reason was to
increase diversity that announcement was
met with backlash they then had to back
down slightly and say that they're not
replacing the job of any model if people
vote with their wallets for human-made
goods and services that could have a
massive impact and there is another big
factor people seem to intrinsic quickly
prefer human-made output to machine
generated output this piece came out
recently from wired and in it they test
the brain chemical reaction to
human-made Art and computer made art
these were the same pictures it's just
that sometimes people were told they
were made by humans and other times they
were told they were made by computers it
says a clear winner emerged people not
only claimed to prefer the identical
human made pictures their brain's
pleasure sensors actually lit up more
brightly so human goods and services may
have the edge simply by virtue of being
made by humans but I want to end the
video where I began it with samuelman's
piece in the New York Times some of you
may have noticed that I said Sam Altman
had a third idea of how to distribute
the wealth that I would mention at the
end well he admitted if AGI does create
all that wealth he is not sure how the
company will redistribute it money could
mean something very different in this
new world but what's the idea he said I
I feel like the AGI can help with that
maybe GPT 5 will decide where the money
made using Gypsy 5 will go thank you so
much for watching to the end and have a
wonderful day