DeepMind x UCL | Deep Learning Lectures | 9/12 |  Generative Adversarial Networks

hello everyone and welcome to the next
edition of the UCL deepmind lecture
series I Mahalo I'm a research engineer
at deep mind and a PhD student at UCL
and together with Jeff I'm gonna talk to
you today about generative adversarial
networks so let's start with an overview
why are we interested in generative
adversarial networks
well generative adversarial networks are
a type of genetic model and generative
models learn a model of the unknown
underlying data distribution from a set
of samples from our data set so imagine
this very simple one the example this is
our data set we have our points here and
we're trying to answer the question
what kind of distribution could have
generated in this data and we can answer
this question in two ways firstly we can
learn an explicit model of the data this
kind of probability distribution so here
and then we can answer questions using
this model we can ask well how likely is
it that this point comes from the
original distribution and the answer in
this case well would be not very likely
we haven't seen any samples here and our
model thus has no mass here but we can
also then sample from this model just
like we usually sample from probability
distributions and generate new types of
data in this type of model that models
the probability distribution directly is
what is called an explicit model on the
other hand we can also learn implicit
models in implicit models we don't model
the probability distribution explicitly
what we learn is a type of simulator
that is able to generate new type of
samples that have the same statistical
properties as our original data without
being able to model the distribution
explicitly so now we have some new data
points shown here in blue that match the
properties of the data and importantly
we've generalized we don't always see
the points in red now we were able to
chair it
only that we want these new points to
kind of capture the statistical
structure of our data and very likely
you've seen generative models before and
you've probably seen explicit likelihood
models so the kind of model that has
access to a probability distribution and
often these models are trained by what's
called maximum likelihood in maximum
likelihood we train a model to maximize
the probability distribution of the data
under our model and such models are
probabilistic PCA factor analysis
mixture models and so on and you can
also train URL based models using
maximum likelihood things like pixel CNN
pixel RNN wavenet water regressive
language models and so on
but when you want to Train latent
variable models with maximum likelihood
things get a bit more tricky and that's
when in practice we often use
approximate maximum likelihood and in
another lecture andrey has talked to you
about how to Train variational
autoencoders using approximate maximum
likelihood but today we're going to talk
about implicit models this kind of
simulator models that just generate new
samples without giving us access to like
leads and we're going to focus of one
type of implicit model specifically
generative adversarial networks and why
would we want to focus on generative
adversarial networks
well one practical reason is that
they're able to generate samples that
look like this so these are samples from
began a model that Jeff is going to talk
to you about later in which the model
trained on imagenet data said that has a
lot of variety it has images both birds
dogs
food and so on the model is able to
learn that statistical property present
in the data and generate samples that
match that and samples that look very
photorealistic so these are all
generated from began and this is
specifically remarkable if we think of
how this progress has been through the
last few years so the original again
paper in 2014
we can go from simple images of digits
to images of faces black and white small
resolution but from there a small
revolution has started canoeing faster
and faster and degenerating better and
better images so we go from and I can
wipe the colored images then we go to
higher and higher resolution of face
pictures of faces then we break the
image net barrier in 2018 these are the
first models that are trained on a
bigger set that contains such variety as
we've seen in image net then we start to
generate faces it's very very high
resolution with progressive Ganz this is
starting to look quite photorealistic
then began comes along and we're
generating image net samples not only on
a high diversity data set but also at
high resolution very high quality and
then we move on to style gun which was
published last year in which the authors
show that you can really generate very
high quality samples that look
indistinguishable from photos to the
human eyes so if you were to ask me
whether this person here exists and this
is a photo or is this a sample from
again I will not be able to tell the
difference this looks incredibly
incredibly realistic so this really
inspires us to think well how are Ganz
able to learn this probability
distribution so accurately that we're
able to generate this high quality data
and the answer is that they learn to
generate data through an implicit model
so our model doesn't have this explicit
likelihoods via two-player game and our
players are a discriminator that learns
to distinguish between real data from
our data set and generated data and chip
rented by a model in a generator and the
generator learns to generate data that
fools the discriminator into thinking
it's real so it has to generate really
quality good quality data such a
discriminator thinks well this looks as
good as real data so let's look at our
players in a little bit more detail so
our players are both are both going to
be
using deep neural networks so our
generator is going to have as input
latent poise so what do we mean by that
we need in some sense to model the
entropy and variety of our data
distribution and the way we do that is
that we have a distribution on the input
of our model because remember on the
output the generator now will not have
any distribution it will just produce
samples as the output so if you've seen
something like a variational or encoder
you're used to having a distribution on
the output of the model here we have
absolutely no distribution so in order
to model the entropy of the data we have
to have a distribution in the input and
often this is multivariate Gaussian
noise and interestingly here this noise
is often much lower dimensional then the
data that they are going to be a
high-resolution image while the noise is
going to be something like a hundred or
200 Gaussian datums we take a sample
from our latent noise distribution we
pass that through our deterministic deep
neural network that transforms that
distribution to generate a sample and
that sample can be images or text and so
on the discriminator on the other hand
has a different task the discriminator
has to answer the question given some
set of samples from our data and given
some set of samples from the our model
are these real or are these generated so
it has to answer the question of
distinguishing between these two
distribution the data distribution and
the model distribution and perhaps in a
less adversarial field we can think of
the discriminator as a teacher a teacher
that learns what you're doing well and
what you're doing not well and tells you
how to improve such that you get better
at better at generating real data from
the generators perspective and from this
perspective we can think of the
discriminator as some sort of learned
loss function because the discriminator
guides your training the training of our
model but while it guides it it also
improves itself
and in the original gam paper this was
done by a two-player game so now we have
a minimization with respect to our
generator this is our model in a
maximization problem with respect to our
discriminator of the same value function
and this value function says well make
sure that the discriminator is very good
at this thing wishing between real and
fake data in a classification sense so
we're trying to train a discriminator as
a classifier to maximize the log
probability than the real data is real
and to maximize the probability that da
predicts that the generated data is
generated so today so far is a
classifier once these trained so this is
what the min max game is telling us that
once the discriminator has been updated
we need to train the generator and the
goal of the generator is the opposite of
the discriminator it's a minimization
problem with the same objective as the
discriminator but with a different sign
and the call of the generator is to
minimize the prediction accuracy of D in
order to make sure that the data that is
generated it generates is classified as
real as opposed to fake and if we think
about this from an algorithmic
perspective how would we implement this
well we'll implement our discriminator
and our generator as deep neural
networks and we will taint we will train
them using stochastic gradient methods
so to do that we first have to train our
discriminator for a few steps in
practice this is 1 or 2 so remember that
the min max game said well I have to
maximize with respect to the
discriminator before training our
generator that would entail doing
multiple steps of optimization but in
practice we don't really have the
resources the computational resources to
do that to update the discriminator to
optimality every time you want to update
the generator so we only do a few steps
of gradient descent for the
discriminator and the way we do that is
well we sample a mini batch of data we
sample a mini batch of noise Leighton's
from our prior we pass that through the
generator now we also have a mini batch
of samples from the generator
and we update the discriminator by
stochastic gradient methods to make sure
that our loss is being maximized so we
want again to make sure that we maximize
the probability that real data is real
and maximize the probability that fake
data I generated by the generator is
classified as generated once we've done
this small inner loop of updating the
discriminator we can move on and update
the generator and now the generator aims
to make sure that the data that is now
generated so we sample a new batch of
noise samples we pass that through the
generator we have a new set of generated
data then this data is classified as
real by this new improved discriminator
that we keep with kept improving in our
last stage of training of the
discriminator so we have this game that
we alternate between improve the
discriminator at distinguishing between
real and generated data then use this
new discriminator to update the
generator such that theme generator
generates data that discriminate that
are the screw nature teams as being real
so the take-home message so far is that
gans are able to generate high-quality
samples through this implicit generative
model trained as a two-player game a
discriminator that learns to distinguish
between real and generated data and a
generator that learns to generate data
that looks so good that the
discriminator cannot longer distinguish
between real and generated data and
we've seen that this is done as a cero
sum game we have a minimization with
respect to G maximization with respect
to D of the same value function and this
has a lot of connections with game
theory literature we can think of Nash
equilibria we can think of strategies
that the two players might employ we can
use things such as fictitious play to
improve our game but in practice is
perhaps also interesting to think of
cans from the perspective of distance or
divergence minimization and that is
because we often think of generative
models as doing distance or divergence
minimization and very often explicitly
our
function is a distance or divergence so
we've already talked about maximum
likelihood maximum likelihood my
maximizes the likelihood of the data
under the model which is the same as
minimizing the kill divergence between
the data in the model and why would we
want to do divergence or distance
minimization well therefore chances and
distances give us some really nice
connections to optimality if the
distance between two distributions in
zero then we know that the two
distributions are the same so from the
perspective of learning if we train our
model to minimize this distance and our
distance is zero we know that our model
is a perfect fit of our data
distribution which gives us a very nice
guarantee and again if we look at
maximum likelihood the objective is not
of maximum likelihood is to minimize the
scale divergence which is the expected
value so this is this integral under the
data distribution of the log ratio
between the data distribution and the
model and because this is something that
we minimize so we minimize with respect
to the parameters of this model P of X
we want to make sure that this is as
high as possible because then this ratio
is as high as possible this ratio is as
low as possible because this P star is
fixed this is our data distribution so
though this expectation is as low as
possible
so we want to make sure that P of X is
giving high likelihood to our data which
is very intuitive we want the model that
is able to explain our data and yes the
kill divergence has the same property if
the kill divergence between two
distributions is zero then our model has
learned our data distribution but one
question that you might have here is
well if we are able to say this for a
lot of distances and divergences if
there is zero and our model has learned
a data distribution why are we concerned
with different divergences or distances
and the answer is that well in practice
our model might be miscible
and it might not be able to model the
true data distribution and this can even
be the case for very deep neural network
models because it might still be that
our data set for example image nets is
that complex that we're not able to
model the data distribution exactly and
in that case we might ant um I might
want to ask well what kind of different
trade-offs these different distributions
have so for example here our data is the
mixture of two gaussians and our model
is going to be a Gaussian distribution
and the Gaussian distribution cannot
model our full data distribution because
it's a misspecified model and one
question that we might have is well what
will happen if we train for example
using the maximum likelihood care so the
curl between the data in the model and
the reverse scale between the model and
the data because the KL divergence is
not symmetric and what we see here is
that the behavior is very different when
we use the maximum likelihood KL
the objective remember is to be able to
explain samples from our data all the
samples from our data and if we sample
from our original distribution from our
data distribution we'll have samples
here and samples here and for a Gaussian
distribution to explain both of these
Peaks it will have to put mass all
around them which means that yes it will
be able to explain the data but you're
also going to have a lot of mass here
where actually we don't have any mass
under the original distribution on the
other hand if we use the river scale
this is not what we will see what we
will see is that deme model is going to
focus only on one of the modes is going
to be able to explain it very well but
it's going to completely ignore the
second mode and if you then query your
model to say is it likely that data here
comes from the original data
distribution it's gonna wrongly answer
no because it's not able to capture
anything about this mode so even with
this very simple example of one
dimensional data we can see the
trade-offs of the kind of distribution
that we choose and that's gonna guide us
through as we go forward so one natural
question now might be well are you
during divergence minimization we talked
about this two-player game on
optimization between the discriminator
in the generator how is that connected
to doing divergence minimization and the
original paper showed that yes it is
connected if the discriminator P is
optimum so if we've trained a perfect
classifier to distinguish between
samples from the data and samples from
the model then the generator G is
minimizing the jet session and
divergence between the true and the
generator distributions and this is
great because it also gives us this
connection to optimality that we talked
about before
now if the chance of Shannon between two
distributions is zero then the two
distributions are the same and now we
want to understand a bit more about the
generation and divergence how does it
behave for example in the case of the
misspecified Gaussian when our original
distribution is a mixture of two
Gaussian and the answer is that well it
does a bit of maximum likelihood and a
bit of the repr scale because by
definition it is a mixture of the two
and in practice the answer depends on
how you initialize your model so if you
don't initialize your model too close of
your two Peaks then it's going to do the
maximum likelihood solution otherwise if
you initialize it very close then it
will revert to the reverse scale however
in practice the discriminator is not
optimal as we've seen from the
algorithmic perspective we often have
limited computational resources we can't
train the discriminator to optimality
every time we update the generator so
that at each step the generator is
minimizing the generational divergence
and even if we did even if we would work
to train these optimality given our data
we still still don't have access to the
true data distribution just a few
samples from it our data set so we will
still not have a truly perfect
discriminator and we're going to see why
that is important later on but let's
look at more properties of the kale and
the Jenson Shannon diverge
and here for simplicity I'm gonna focus
on explaining this on the KL divergence
but the same can be said about the
chance of Shannon as we've seen the
Johnson Shannon is a mixture of two
Kells and this property is important
because this has really sparked the
field to perhaps look beyond the Jensen
Shannon divergence look at other
divergences that we can use to train
ganz and why is that well we here our
example that we're gonna run throughout
is a case where we have two
distributions with no overlapping
support so what do I mean by that
here we have our data distribution in
red and our data distribution produces
samples here and it's PDF is given by
this truncated Gaussian here shown also
in red and we have our model and our
model is also truncated Gaussian and we
have a few samples from it here one
thing that we observe is that there is
no place in one-on-one D where both of
them assign nonzero probability so the
data only assigns them zero probability
here but here the model says well this
is not really likely under the model and
what happens in this case is that the
kill divergence in the Jensen Shannon
are going to be constant so the KL is
going to be infinity and the Jensen
Shannon is going to be locked and why is
that well remember the KL divergence
definition is the expected value under
the true data distribution of a log
ratio and this log ratio is the ratio
between the data distribution and the
model and if we look at the fat this
ratio under the data distribution
because this is our expectation we see
well we will have the probability of
this data in the sample here under the
data distribution which we can query is
something obtained from here divided by
the probability distribution of the data
under the model this is where the
problem comes from this probability
distribution is zero because the model
of science is zero mass so this ratio is
infinity so our kill divergence is going
to be infinity and this is especially a
problem from a learning perspective
because when we learn the model we want
to get rewarded if we do something good
right so imagine the case
I've moved my model a little bit from
here a bit closer to the data here so
this is good the model is doing
something good it's going closer to my
data distribution and we would want the
type of loss function that says he have
good job you're going in the right
direction you're doing well but the
kalman the chance of Shannon they can't
do that because this property that the
ratio is still infinitely here still
holds you still even though you've moved
your model closer to the data you're
still at a point where this ratio is
infinity because there's still no
overlapping support so this is why
people thought well perhaps we should
try to train guns that are inspired by a
different divergence so the question is
can we choose another V for our min max
game and will it correspond to a
different distributional divergence and
to do that we have to look at other
divergences and distances and see
whether we can somehow turn that into a
game that you can we can use for again
training and one very nice distance is
the Vassar Stein distance between two
distributions it looks slightly
different than the kale we already see
that there's no ratio we have a
difference of expectations here and a
maximization so just to estimate the
divergence we have to do a maximization
and this maximization has to be over one
Lipsius functions so one thing she's
functions have to be relatively well a
well behaved which means that the
difference from an absolute value of the
function at two points has to be smaller
or equal than the absolute value of the
two points so you can't grow too fast in
a particular region so this means that
the function has to be relatively smooth
and here when we maximize with respect
to the set of functions we're trying to
maximize the difference in expectation
of the expected value of the function
under the data distribution minus the
expected value after the function under
the norm so let's look at an example
here this is our example from before
only that here we're not going to use
the PDFs themselves but we're going to
use samples from tomorrow so these are
samples from our data distribution these
are samples from our model and we're
trying to find a function f that can
separate these expectations as much as
possible
so here we can see that we can put
positive mass under function f around
the data distribution then this
expectation is positive because we are
and we're sampling here we are
evaluating the function at all those
points all these points are positive so
this expected value is going to be
positive we do the same from for the
model but here the model under the model
the function is negative so when we take
the difference the difference is going
to be large is going to be something a
positive a positive number minus a
negative number and importantly the vast
search time distance goes down now if we
have a model that goes closer to the
data even when we don't have overlapping
support because remember this function
has to be 1 Lipschitz it can't grow too
fast in a small neighborhood so we're
moving closer to the data we have
restricted the amount of growth that
this function can have and thus the
difference in expectation is smaller so
we now have a distance that have this
property that if we're doing the right
thing we're getting rewarded for it
which is great now the question is how
do we turn this into again so we've
talked so far about estimating Basilan
distances and we've seen that this
itself involves an optimization over 1
Lipschitz functions but what we're
interested in ideally is in learning how
do we use this to learn probability
distribution or a model that can
generate data from our probability so we
have now our minimization with respect
to our generator again but now we want
to do with respect to the passive time
distance and if we just replace so we
keep the minimization in place and we'll
replace the definition of the passive
sighing distance that we've seen above
we have this form and this form already
looks very familiar we have a
minimization in a maximization so if we
think of our function now that learns to
distinguish between data samples and
model samples
from an expectation perspective rather
from a ratio like we've seen before then
this function can be thought of as our
discriminator so now our minimization
problem with respect to G stays the same
but we have a maximization problem with
respect to our discriminator subject to
the discriminator being well behaved and
this loss function this value function
that looks different because we're no
longer starting with a classification
and we're no longer getting to the
Jenson Shannon divergence but to the
Bathurst Island distance but it's
something that looks very similar right
so now we have something that learns to
distinguish between the data samples in
the model but in a certain sense and we
can use that to train again and this is
what's called the search time yeah and
we can look at other divergences and
distances one of them is mm V maximum
mean discrepancy and looks very similar
to the a certain case only that now the
optimization is with respect to a
different class of functions class
functions that are part of a repeat
using kernel Hilbert space and if we
look at the behavior of mnd on our
standard example we see that it does the
same the value of the function is
positive under the beta the value of the
function is negative under the model
only that the shape of the function
looks different because we're now
looking at a different family of
functions to estimate our model and just
like in the case of the best search time
distance we can try to turn this into
again we have a supreme over a class of
functions we turned that into a
maximization over our discriminator only
them now that the strong leader has to
be part of a reproducing kernel or space
and we have the loss function as an
expectation of the difference of
expectations and remember we started
talking about the KL divergence we
started with maximum likelihood as a
very common objective of training and
tail divergence is a type of F
divergence and if divergences look like
this there's an expected value on F
which is fixed so we know this function
for a kale for example
and a density ratio the problem here is
that if we want to train something like
again inspired by F divergences we will
encounter issues because we don't have
access to P of X we don't have access to
the probability distribution so how do
we get around this well we can't just
start training models using the F
divergences but we can find a
variational lower bound on our F
divergence objective and use that
instead so if you've seen bees before
variational autoencoders they're there
too we use a variational lower bound and
we replace that in our training
objective and in this case in the F
divergence case the variational lower
bound is telling us to optimize this
objective instead and this objective now
should look very similar we have a
suprema
over class of functions and we have a
difference in expectations only that now
we also have the complex conjugate of
the function f from here
the things are looking very very similar
to what we've seen before and the
optimality here is actually the density
ratio that we talked about before and
that we saw that can cause problems in
practice and we're going to go back to
the density ratio in a bit but
importantly now because we have the same
form than we've seen in the bus or sign
in in the case we can also turn this
into again just slightly different
objective now we still have the convex
cones you get a time here but we can use
this to pray normal so so far what we've
seen is that we can train Ganz using
multiple criteria which are inspired by
multiple divergences and distances we
started with the original again and
intergenerational divergence and we look
at the properties of the Jenson Shannon
divergence and based on that we looked
at other distances and divergences that
maybe have different properties
those were Vassar sign and MMD and at
the end we also asked the question well
ok but how about the KL divergence
something that's very used in practice
can we train again inspired by the KL
divergence
and the answer there was also yes now
one question that you might have is why
would I train again instead of doing
divergence minimization is if divergence
minimization gives me all this optimal
convergence properties and the answer is
well it depends in practice you might
not be able to do divergence and
minimization or you might not want to do
a divergence minimization because the
ends have some advantages and we're
going to talk about this now so firstly
remember how we mentioned just now that
the kill divergence requires knowledge
of this model P of X which we don't have
in the case of implicit models of models
like guns so if we want to train again
inspired by the kel-tec divergence we
have to use afghans
but now at least we can train models
that don't have an explicit likelihood
using the KL divergence which is
something that we couldn't do before
right so by using ganz we've expanded
the class of models that we can train
using KL divergence there's also the
computational intractability factor
we've talked about the passive time
distance and how just finding the value
for the master signed distance requires
an optimization problem over a class of
functions but that is intractable for
complex cases so you wouldn't be able to
do it this at each iteration step to
find the assertion distance and then use
that for training but if used a faster
stankin which now will have the same
type of algorithmic implementation as
we've seen for the original can update
the discriminator a few times two three
four or five times and then update the
generator then you can get around that
yeah you're not doing exact faster time
distance optimization anymore because
you haven't solved this optimization
problem but you're still doing something
inspired by the masters time distances
but you can now train a model bit and
remember our problem with the smooth
learning signal our problem
with the KL divergence and the Jensen
Shannon and how that inspired us to look
at other distances and divergences but
perhaps that's not as big of a problem
in the gang case as we originally
thought this idea that they will not
give you any signal to learn when
there's no overlapping support between
the data and the model and why is that
well remember our example the problem
that we have is that this density ratio
was infinity here and that meant that if
I remove my model closer to my data I'm
still not getting any useful signal but
in the case of Ganz I'm approximating
this ratio so perhaps we're not gonna
have the same problems so if we look in
peer eclis we can see that again stiller
so in this paper we show that if the
data is here and the model is here so at
an installation there's no overlapping
support and we train our can the model
after a bit of training still learns to
match the data distribution so why is
that well a simple way to think about
this is again inspired by the KL
divergence because that's a simple
divergence to look at but similarly we
can think about the Jenson Shannon so if
we look at the KL divergence we look at
its definition again we have this true
ratio here that's problematic right
because this is why we're getting these
problems with the KL divergence but in
the case where we train Ganz we actually
use this lower bound instead remember
when we talked about Afghan we use the
bound because we can't have access to P
of X but now we estimate this ratio
using our discriminator and we ask our
discriminator to be in a class of family
of functions because we have to
represent it somehow so that's either a
deep neural network or a function in the
reproducing kernel Hilbert space and so
on and these functions are relatively
smooth so we're approximating our true
ratio with sounding smooth and what
happens in practice is that these smooth
functions won't be able to jump from 0
to infinity or to represent infinity as
the
underlying ratio would so our standard
example again we have our data here our
model here the true ratio here goes to
infinity it's 0 everywhere else but our
MLP that is used to approximate our
ratio will not go to infinity
it starts low and then it starts growing
and growing and growing it needs it
knows that it needs to be higher here
but it won't be infinity and the nice
thing about this is that if I move my
model closer to my data it will know
because there's no jump of exactly here
you need to go to infinity and this is
similar if I use another function class
to represent our ratio so here if we
were using or producing kernel Hilbert
space we see the same type of behavior
around the data we're gonna have a high
ratio but it's not gonna be infinity and
again if I use my model closer to my
data then I'm gonna get a useful
learning signal that says yeah good job
you're going into the right direction
and this is why empirically we've seen
that again could learn even though we
initialize the models to no - don't do
not have overlapping support so the
crucial idea here is that the
discriminator is a smooth approximation
to the decision boundary of the
underlying divergence and we've seen
that with some experiments and with an
explanation of what happens in the case
of the KL divergence so in practice
games do not do divergence minimization
because the discriminator is not optimum
it doesn't really represent that true
density ratio for example but this also
means that gans do not fail in cases
where the underlying divergence were
like we've seen in the Jensen Shannon
case and perhaps another way to think of
discriminators is as learnt distances so
the discriminator is providing a loss
function to our generator but it's
something that itself is learned to
provide useful gradient store model and
this is the case both feet for the
original again the bass or sine gun and
so on they all have this form
minimization with respect to G and
maximization with respect to D of
we function but if we think of this bit
here this is the loss function for G but
it's just trained is trained D using the
discriminator parameters now the crucial
bit here is that we can use this to tell
the generator through our loss function
what we actually care about and the way
we do that is by putting the right
neural network features into the
discriminator so we know that if we're
training data on images we want to use
convolutional neural networks because
those are very good at distinguishing
between images and learning the right
features for that if we're using audio
we might want to use recurrent neural
networks and so on so the crucial bit
here is that we no longer just use
neural network features in our model but
we also use it in our loss and now the
loss in you provide additional signal to
the model to focus on the right aspects
of the data and this is something that a
true divergence not the learned
divergence that this is not a distance
or divergence in a mathematical sense
but this is able to provide you some
useful learning signal that you maybe
wouldn't get if you were using the KL
divergence or something else so to
answer the question of well why would I
want to do Ganz as opposed to divergence
minimization well we see that Ganz
provides very good samples and are you
using this learn loss function where you
can have this additional log to train up
to tell your model what to focus on but
they're hard to analyze in practice you
have to think of game theoretical
reasons and so on and in practice there
are no optimal convergence guarantees
because again the discriminator won't be
optimum however if you do divergence
minimization there are optimal
convergence guarantees and easy to
analyze loss properties but it's harder
to get good samples and the loss
functions don't usually correlate with
human evaluations because they focus on
aspects pertaining to the statistical
properties of the divergence rather in
the mortality of the data so the
take-home message
is that in practice Gant's to not to
divergence minimization and the
discriminator can be seen as a learn
distance it's something that is learned
to distinguish between the data and the
model samples and to provide useful
learning signal to the generator and one
question that you might have is well
which can should I use we've talked
about Bathurst again mmm began the
Jenson Shannon Gann that's the original
again and saw and empirically it has
been observed that the underlying loss
so the underlying divergence matters
less than the neural architectures the
training regime and the data and I think
if you're thinking of the importance the
importance of the features that the
discriminator is learning and the
convolutional or recurrent architectures
underlying them and the kind of
information that provides the generator
that's somewhat intuitive because now
you're focusing really on the features
that are useful and distinguishing
between data and samples and Jeff is
gonna tell you a lot more about this and
give you plenty of examples of neural
architectures that are used for programs
and so far we've talked about
unconditional generative models so far
we're asking our generator generator
please generate a sample I'm giving you
some later noise generate something out
of it but we might want to have a knob
to tune and we might want to tell the
generator generator please generate a
cat or generator please generate a dog
and so on and for that we have to change
our model a little bit so so far we've
talked about deterministic deep neural
networks that are able to transform
Gaussian noise into data but what we
want now is to provide additional input
to the generator to say well please
generate a dog or please generate a cat
and we often provide that in the form of
one hot vector if our conditioning
information is a label we're gonna say
one zero zero zero for dogs 0 1 1 for K
0 1 0 0 for cat and so on and this is
gonna tell the generator what it needs
to
and the reason you will listen to that
is because in practice we also change
how the discriminator strained and now
the discriminator also knows that the
generator should have generated at all
and if it generates a cat that the screw
the generator is not going to get a good
loss for that so now it has to listen to
the conditioning information as well
because the discriminator training
itself has also changed and this in
practice leads to better samples and the
bigger model for example that I've shown
it is able to generate very high quality
samples on imagenet is a class
conditional but sometimes when you train
dance even class conditional Gans you
might get something like this this is
what's called mode collapse so here in
the model instead of capturing the
diversity of the data it's now focused
only on a few examples a few phases and
it's generating them again and again and
what we would like is a way to
automatically know whether our model has
collapsed or not we want to evaluate our
samples without looking at them and
every iteration and so on and in
practice that's a bit hard because the
generator loss is not something very
interpretable so often when you train on
hollows where you store loss going down
smoothly but because we have this two
player game here where the generator
improves the school near improves and so
on the loss itself shown here doesn't
really tell tell us much so there's been
a lot of work of trying to answer the
question where how can we evaluate Gans
and this is a very difficult question
even answering the question broadly how
are we gonna evaluate generative models
is extremely hard so we have no metric
currently that is able to capture all
the desired properties that we want from
our model so some of these properties
are sample quality we want to be able to
generate high-quality samples but we
also want to be able to generalize we
don't just want our model to just give a
samples from the original
because for that we could have just used
a hash table and just say give me a
sample from the original data set and as
you Vina and I are going to talk in
another lecture
we're often also using this models for
representation learning and we might
want to answer the question how good is
this can at representation learning or
how good is this BA at representation
learning and so on and perhaps what we
actually want is to evaluate on the base
goal so what are we trying to do with
this generative model are we using it
for semi-supervised learning so are we
using the features for classification
then maybe we should use classification
accuracy are we using it for
reinforcement learning then maybe we
should use the agent reward and so on
but in practice because that is hard to
do and also more expensive and complex
and it makes it harder to compare models
what people often use are log like views
so you're asking your model to explain
validation data and that it hasn't seen
and based on that you're assessing how
good your model is baganz are implicit
so we're not able to use look
likelihoods to evaluate our Gantz
so people have come up with other
metrics to try to understand how good
our sample is it R and one such metric
is the inception score so in the
inception score what we're trying to say
is that the model is preserving the
class ratio that we've seen in the data
so imagine that we have a data set that
has 50% dogs and 50% cats then we want
that our model and practice is also
generating around 50% dogs and 50% hats
and notice here the inception score
doesn't care about the individual dogs
and the individual cats they can all be
the same as long as on average we get
50% cats and fifty percent dogs
inception score is happy so the way this
is done in practice is that we can use a
pre trained classifier of a known
imagenet to compare the distribution of
labels obtained from data with the
distribution of labels obtained from
samples in a KL divergence sense and
metric is able to capture sample quality
because if the model is generating
garbage you won't be able to get
anything useful out of the pre trained
classifier so the distribution of labels
coming from samples is going to be very
different than the distribution of
labels coming from data it's able to
know whether you're fully dropping a
class
so remember mode collapse or we've seen
that the model can focus on one or two
aspects of the data so if you're
dropping back classes for example you're
not generating any cats the exceptional
score is going to penalize you for this
and it's also going to penalize you if
you're generating a lot more dogs and
cats
for example if correlates well with
human evaluation it doesn't really
measure anything beyond class labels so
every as we've seen if you're carrying
your same dog again and again in such
the scores gonna be good I'm happy and
because if these people have looked at
other metrics for example fresh
inception distance and for station
inception distance is not happy if
you're generating the same look again
again now it's looking both at the
labels in terms of are we generating 50%
can cats and 50% dogs but also inside
the class and the way it does that is by
looking at features on the pre train
classifier rather just than the output
distribution of labels so if we're
comparing now instead of a Kail sense in
a fresher distribution appreciate this
sense of a sense the distribution of
their features obtained from the data
and the distribution of layer features
obtained from the model now we're
getting a more fine-tuned metric so
again you can see sample quality because
we're also using a pre training
classifier we're also able to see if
we're dropping classes altogether
because the feature is on average are
going to look very different they're
only generating dogs and forgetting
about cats but it also goes beyond that
and it captures higher level statistics
but there's a problem with this metric
it has been shown that it's biased for a
small number of samples and kie has been
proposed as a fix in practice and see
this paper from
I clear too
thousand 1844 to fix but we also want to
go beyond us we want to make sure that
our model has not over fitted and it's
not just memorize the data we want
generative models that are able to
capture the essence of the underlying
distribution and the statistical
properties of the distribution but
generalize beyond that and one way to
check this is to check for the closest
samples from our model sample in the
data but we don't want to do this in
pixel space because that's very noisy
and not really representative in a
semantic sense so again just like we've
seen with loss functions when we used
features in our training or just as we
do in our model we're going to use
neural networks features for evaluation
so again we're using a pre trained
classifier and we're going to search not
in pixel space but in the feature space
of this classifier for the closest
images in our data set to our sample so
here we have an example of a sample from
beacon and we're answering the question
well what are the most similar image net
samples from this sample and the answer
is that well they are there are data of
dogs in image net but this exact dog
does not exist in image net so we have
dogs of the same color different shapes
different sizes we have dogs and green
background but this exact same dog does
not exist in the data set so the model
has used training and the data to learn
how to generate talks but to generalize
beyond what it seemed and the take-home
message of this part is to remember that
we need multiple metrics to evaluate
game samples because we don't just care
about sample quality we also care about
overfitting and so on and with this I'm
going to hand it off to Jeff who's going
to talk to you about the Gansu hi
I'm Jeff Donohue I'm a researcher at
deep mind and I've been working on
developing and improving adversarial
networks at scale I'm particularly
interested in the application of Ganz
and other generative models for
representation learning a topic I'll be
discussing a little bit later in this
lecture so now that mihaela has given
you an overview of the theoretical
underpinnings of Ganz my goal for the
rest of the lecture is to take you on a
tour of the Ganz ooh to give you an idea
of the kinds of things that people have
been doing to improve these models from
where they started to the state of the
art now and all the different domains
and problem settings where these models
are being applied a lot of gand research
has focused on image synthesis so we'll
start by walking through the path that
is taking us from applying Ganz to small
data sets like Amnesty to large scale
image databases like image net and a
good place to start is the original gand
paper from Ian Goodfellow and his
collaborators in this paper they used
relatively simple data like the emne
stitch it's that are referred to in the
title of this part of the lecture and
other data sets like this faces data set
and the Seafarer data sets but they're
all pretty small images with resolutions
of about 32 by 32 or smaller in this
paper they used relatively simple models
in fact for these top two images that
you see here the mole the bottles were
multi-layer perceptrons or MLPs
so they weren't convolutional and they
treated the images as flat vectors
completely ignoring the spatial
structure of the images so there's
essentially no inductive biases in these
models and when you have data that's as
relatively simple as this that turns out
to work pretty well you can see that the
kind of digits that you get are
relatively convincing imitations of the
real digits you see highlighted in
yellow here these are the kind of digits
that you can generate with these kinds
of models so it worked reasonably well
but it was mostly just a proof of
concept that this sort of model could
work at all and it wasn't really meant
to be a demonstration of everything
these kinds of models were capable of
which we'll get to later so moving on
from that an extension that you can do
to these models as
Cayla mentioned in her part of the talk
is to make them conditional on a class
label this early work on Ganz called
conditional ganz generalizes Ganz to the
conditional setting we have some extra
information associated with each piece
of data such as a category ID in this
case instead of a category idea this
could be something as complicated as an
image in another domain although in this
work the conditioning was just a
category ID like cat or dog so when you
do this on a mist with the 10-digit
labels 0 1 2 3 4 5 6 7 8 9 you get
results like this where every row is a
different conditioning in this case a
digit label and it turns out when you
give it a 2 that's the label it produces
results that look like a 2 which so it's
great this works next we're going to
look at some early work that actually
managed to tackle some pretty high
resolution images with Ganz there's this
work called lap game by Emily Denton and
her collaborators and so this work was
really cool for a couple of reasons but
just to give you an idea of what it does
in terms of the generation process
basically they'll start from a tiny
image like 4 by 4 or 8 by 8 image and
they'll up sample it via Gaussian style
up sampling so that gives you a blurry
image at a twice as large resolution and
from there what you can do to get a
final image is you generate the
laplacian so basically so you can see if
you go from this image here this tiny
image point to this image all you have
to do is some trivial up sampling
operation but then to actually fill in
the details you have to produce the
laplacian which is the difference
between the blurry image and the final
higher resolution image so you can add
these up to get the final higher
resolution image the blurry image plus
laplacian
so the discriminators job is to take
both the blurry high resolution image
and the difference image either the real
one or the generated one and decide
whether that pair of images is real or
generated so this is a really
interesting formulation for a couple of
reasons in that it sort of decomposes
the problem down to a multi-step
generation processes with multiple
discriminators and generators each one
operating at a different resolution and
the discriminators and generators are
also conditional as you have the same
piece of conditioning information
display images that we're interested in
up sampling so what you have in the end
is this recursive way of going from a
small image to a high resolution image
um so this was pretty exciting at the
time especially because it was the first
scanned paper to produce relatively high
resolution and convincing images and one
of the other nice things was that it's
not a deterministic up sampling so you
can see on this slide it's not producing
the same high resolution image for each
solution input image on left it's
actually producing a full distribution
of high resolution images for each low
resolution image and so you have this
tiny starting image on the left and you
up settle up samples sample with again
until you get to 64 by 64 resolution or
whatever and because it's using random
noise at each stage as you have in any
standard again you wind up with a
slightly different high resolution
output whatever tiny input image you
started with every time you resample the
noise which is what you want if you have
a properly trained and generalizing
again another cool thing architectural
II is that this was a fully
convolutional generator so it's taking a
blurry say 32 by 32 images input and
maintaining that 32 by 32 resolution
throughout the network to produce a 32
by 32 laplacian is output and a nice
thing about that is it allows you to
play the generator to actually any
resolution although it's only going to
work really well at the resolution you
train it on so for example in this case
they only trained it on up to 32 by 32
images but you can keep reapplying this
recursive up sampling and laplacian
generation operation with the highest
resolution generator that you trained
and then in the end if you keep doing
this you get what looks like continues
to look like higher resolution images
although obviously it's a little bit
blurry and not necessarily the best
fidelity but you can't really expect too
much more when the models only have
received 32 by 32 images moving on to
this paper called deep convolutional
gans or DC games from Alec Bradford and
his collaborators um so this is another
really exciting paper at the time
because it was a very simple
architecture it was basically very
similar to the original game framework
but with deeper confidence
and it used batch normalization which
made this sort of notoriously difficult
Gann training process much smoother than
it was without batch normalization um
the two networks the generator and the
discriminator we were both confident so
the generator was a decom net or an up
sampling continent and the discriminator
was a down sampling continent and it's
basically a five ish layer Network not
too dissimilar from something like
Alec's net at the time so when you apply
DC Gans to a dataset of indoor scenes
you get results that look like this
which were at the time at least quite
impressive and exciting and one of the
cool things that you can do with a
network that's trained this way is you
can take to noise or to Z samples z1 and
z2 on the slide for example one of them
might produce an image of a desk that
looks like this and one of them might
produce an image of a bed that looks
like this and then you can interpolate
between these two Z's in z space and at
every point in between you get what
looks like a relatively realistic and
semantically meaningful result so of
course it's not perfect but one thing
that this shows is that the model is
able to properly generalize so it's able
to turn a data set of a hundred thousand
or ten thousand discrete examples into a
continuous distribution of images and
this also showed that the model isn't
simply memorizing the data set because
obviously in the data set you wouldn't
have an example of any interpolation for
any given pairs of images in the data
set and this is what happens if you do
that same kind of interpolation thing
for faces again obviously it's not
perfect and there's some kind of creepy
looking results in this case but still
interesting one really interesting
observation from this work is that there
appear to be some meaningful semantics
in the latent space so basically in this
sort of example they observed that if
you take a latent that produces a man
with glasses from a pre trained Gann
model and another Lathan that produces a
man without glasses have another
Leighton that produces a woman without
glasses and you do man with glasses -
man plus women you get women with
glasses and that might remind you a
little bit of the word Tyvek results for
language embeddings if you're familiar
with that work but what the shows for
ganz is that there are direction
in this DC game Layton space that
correspond to the presence or absence of
glasses as well as the gender of the
subject which is not something the model
has ever explicitly trained to do it's
just sort of learned to sort these
semantic properties and represent them
in the latent space in some way which is
really interesting we'll talk more about
that later jumping ahead a little bit
there was a paper in 2018 called
spectrally normalized gans from me otto
and collaborators and this was really
exciting - it was the first real crack
at using a single Gann a single
generator and a single discriminator to
model this imagenet dataset with a
thousand classes in 1.2 million images
the main trick in this paper was
intended to stabilize Gann training by
clamping the singular values of the
discriminator weights to one so that all
the weights of the network had a
singular value of 1 which basically
means that no matter what the input to a
layer is the output magnitude is not
increased and the way it's implemented
is every time you run the discriminators
forward pass you calculate an estimate
of the first singular value for each
layer and because this is a linear thing
you can just rescale the weight as shown
here by dividing by its singular value
to get a normalized version of the
weights with spectral norm 1 so this
regularizes the discriminator and
they're actually using here essentially
a linear loss function the hinge loss in
this case so if you didn't have this
regularization the discriminator could
basically improve its objective just by
increasing the magnitude of its weights
but because you do have the spectral
norm regularization the discriminator
has to improve its objective in ways
that actually meaningly improve the
gradient signals that it passes back to
the generator which is what we want that
of a discriminator
so when this is applied to imagenet you
get images that look like this which at
the time was particularly impressive
because nobody had successfully taken on
the full image in a dataset with a
single again before in some follow-up
work from the same group they added this
idea of a projection discriminator to
handle conditioning so previously they
used the kind of input conditioning we
saw before where you would feed in the
class label like pizza as the input to
the very first layer or other word this
is other variant called a CGI
auxiliary classifier against where you
would train the discriminator as a
classifier directly so what this paper
proposed to do is called a projection
discriminator so they're learning a
class embedding which is the same
dimension as the discriminators last
hidden layer and they project the class
embedding onto the hidden representation
or dot product it and that gives you a
class conditional realness score that
the discriminator outputs so basically
rather than feeding the label as an
input it becomes an output in this case
and there's a pretty interesting
theoretical justification based on the
underlying probabilistic model that
justifies doing it this way and it not
only makes sense theoretically but
performs very well empirically and you
see results that look like this which
was even more impressive than the
results we saw with SN Gann alone one
more pretty interesting innovation in
the Gann architectural space was what's
called self attention and self attention
is that this technique forgiving
networks the ability to do some sort of
global reasoning it's been applied a lot
of domains especially in language
modeling and machine translation in the
image domain it allows you to basically
learn to measure your global statistics
about the image so for example so this
was used in both the generator and the
discriminator and for example if you're
the discriminator you might want to be
able to ask questions like if the tail
of the dog is on the left side of the
image is the face of the dog on the
right side of the image which is
something you might want to know if you
want to tell whether the image is real
or fake
and you couldn't typically typically do
something like that with a single
convolutional layer because the kernels
are just too small to capture that much
of the image so this resulted in better
global coherence across the images that
the Gann would generate and they also
have these nice qualitative results to
visualize what the model ends up looking
at so for example in this case it looks
like the model decided to compare this
area around the head of the dog to this
area near the tail of the dog to make
sure you know that all the dog's body
parts are kind of in the right place
which you can imagine how that would
help they generate or learn to produce
images with better global coherence and
then at the end of the day you get
results that look like this on the image
net data set which again was another
advance both qualitatively and
quantitatively in terms of Inception
score compared to the previous results
that we've seen so finally we get this
project from our group at deep mind
called began led by Andy Brock the main
idea of this work which I think I'm
allowed to say because I was a co-author
on this paper was to make Gans really
really big and we wanted to do a big
empirical study and sort of digest all
of the image gain research that's been
done so far and scaled them up as much
as we could and just kind of see where
it would take us so yeah
begins we had big big batches big models
big datasets big high-resolution images
so the batch size that we used for our
main results was 2048 compared to batch
sizes of roughly 256 that were being
used before our work and this turned out
to be a particularly important hyper
parameter which is really critical to
making these models work as well as they
did and one hypothesis for why this
might have been so important is that the
imagenet dataset has a thousand classes
and if you're doing mini batch SGD
especially in a setting that's as
unstable as gand training still can be
you really want ideally each class to be
represented in each batch so that the
model doesn't end up sort of forgetting
about classes that it hasn't seen in a
while and so if you have a batch size of
2048 it's fairly likely that in any
given batch almost all of the thousand
classes will appear whereas obviously if
you have a batch size of 256 it's
obviously impossible for a thousand all
thousand classes to be in that batch so
we not only trained on imagenet but also
this internal Google data set called jft
which has three hundred million images
I'm sorry sort of used image that is our
development data set when designing
these models throughout the course of
the research and then we directly
applied the same models to jft and we
found that they worked pretty well there
even on our data set which was you know
two
200 or 300 times larger so you can see
on the right the type of images we get
from this kind of model and another few
of them are here and so overall this
paper was a really big empirical study
to build up a reliable kind of recipe
for large-scale scam training so we
inherited quite a few tricks from prior
work but we like to think that you can
be confident that each one was ablated
really well and turned out and really
turned out to be the best choice in
terms of the image fidelity and the
quantitative scores that you get so
among these tricks we had the hinge loss
which is basically a linear loss except
it's sort of clamps to a minimum value
when the discriminator is or a maximum
value when minimum value when the
discriminator is correct and
sufficiently confident in its
correctness and spectral norm which we
just discussed as well as self attention
and projection discriminators and
finally some tricks that we added to the
toolbox relative to previous work
included orthogonal regularization which
sort of enforces that each row of the
weights is orthogonal that they're kind
of doing different things and we used
skip connections from the noise so
basically there was a direct connection
from the noise z to every layer in the
generators convolution stack and
similarly for the class label embedding
in the generator we used we learned an
embedding that was shared across the
different layers each layer again having
a direct connection from the class
conditioning as well one interesting
trick that we introduced was with this
paper was what we called the truncation
trick
it's an inference time trick so it
doesn't affect training at all it's
something that you can do with any pre
trained generator at inference time when
you're want to go produce samples so
basically we can change the standard
deviation of the noise input to the
generator basically change the scale of
the noise distribution as you can see in
the figure here so it's sort of
shrinking closer and closer to zero so
if you watch the animation we start with
this you know Y distribution and the
resulting image is produced for each
class at the beginning of this animation
like now are quite different but as the
distribution gets skinnier the images
become more and more uniform
for a given class basically what this
does is when you make the distribution
really small near zero is it gives gonna
give you kind of a prototypical or a
modal example of each class and in this
case for the dot for these dogs it's
typically a very well centered and
camera facing example of each dog which
is sort of inherited from the biases of
the datasets because most people you
know will take pictures of their dogs
when they're facing the camera and
whereas if you keep the noise as it was
a training time as you can see here with
Sigma equals one for the for the
Gaussian input to the generator um you
get quite a bit more variety so the
truncation trick is really a way to
trade-off between the variety and the
fidelity of the samples that you can
generate with these models and yeah
here's just another example of what
happens with the truncation trick for
some bugs some butterflies kind of the
same thing as we saw for the dogs so as
I said the truncation trick is really a
way to trade off between variety and
fidelity so what you can do is compute
the inception score and the F ID at
every point along this curve of Sigma
values that you can produce via the
truncation trick so as Haley explained
earlier when she was talking about
evaluating ganz
um the inception score doesn't care
really about how diverse the samples you
produce are in each class it really just
cares how good samples are for each
class how confident it is in the
classifications for each class so if you
just want to maximize inception score
setting the scale to roughly zero is
really the best thing you can do and
when you do that you see that you end up
maximizing inception score down at this
point on the curve here at around two
hundred ten in this case but when you do
that you have relatively bad F ID of
thirty plus and higher is worse for F ID
on the other hand if you leave Sigma
equals one and the other end of the
curve here which is the default the Z
distribution as it was at training time
you get relatively bad inception scores
roughly 105 or 110 but very good F IDs
as you're capturing more of the inter
class distribution which F ID is a
little bit better at measuring
so as so kind of as an alternative and
more detailed way to evaluate Gansa can
look at this full truncation curve
whereas previous work had just looked at
individual points using the default
distribution those sort of gives you a
full frontier of the inception scores
and F ID scores across this entire curve
one more thing that we played with sort
of late in this work was this different
architecture called big and deep that
you see here so this is a deeper yet
more efficient architecture you can see
in a single block it has twice as many
convolutions in the main block so
there's four of them instead of two and
we had twice as many of these blocks in
the big and deep architecture so overall
it's four times as deep the key thing
that makes this even more efficient than
the original began is that we have these
it's not a new idea but we added these
one-by-one convolutions that go to a
lower channel count and then these 3x3
convolutions operate on this lower
channel count space so it all ends up at
the end of the day and the 3x3
convolutions are the most expensive part
so it all ends up being a little bit
more efficient than the original
architecture and the nice part is it
also performs better with inception
scores of over 240 at full truncation
down here and now heidi is around 6 with
the minimal truncation now this model is
definitely not perfect and a lot of
times the failures are kind of fun to
look at as well so for example this
image on the left that we sort of
affectionately refer to as dog ball and
this is an example of what we call class
leakage so according to began this image
is an example of a tennis ball so the
reason that we think this happens for
image net specifically is that there are
just so many dogs in the image net data
set there's roughly a hundred dog
classes so the model is sort of very
accustomed seeing dogs and it sees them
roughly a hundred times as often as
tennis balls so when it sees tennis ball
it says you know hey this that's fuzzy
it's probably a dog I'm gonna put some
eyes and a stout on it so this happens
at least some point in training and it's
not actually from the final converged
model but it's kind of fun to see what
happens as the models when you generate
better and better in images throughout
better and better images throughout
training
and other failure modes include classes
that are difficult particularly any
class that includes a human face now it
could be a little bit just that they
seem particularly bad because humans are
very sensitive to how good human faces
look or how realistic they look so
there's kind of this uncanny valley
effect although we're quite a bit off
here I think you'd probably agree and
classes with really complex structure
like the image of this band here are
also really hard with a lot of when they
have a lot of different objects in the
scene and classes that are
underrepresented in the dataset and have
also have complicated structure like
this image of a I think it might be a
two-bar French Horner and it's just
really hard for the model to capture
this sort of complex structure without
too many examples especially to
generalize to new instances of the class
as you're sort of asking began to do so
more recent follow-up work that we did
is this work called Logan or latent
optimization gains so latent
optimization is this idea intended to
improve the adversarial dynamics of the
gang game between the generator and the
discriminator and basically what it does
is it uses what's called the natural
gradient descent to optimize G's latent
inputs disease so it changes the Z's at
training time to make the discriminator
happier so does one natural gradient
descent step inside of the training loop
to change Z and it actually is going to
backprop through this entire process so
it's a little bit more expensive than a
standard gain it takes about roughly
twice as much computation time per step
but it results in really significant
improvements in begin in terms of the
variety and the fidelity that you can
get and it's particularly noticeable
when you compare along the truncation
curve so for example if we truncate such
that the inception score is roughly 259
you get much better F IDs when you train
using Logan than with a standard begin
deep so both quality so Logan is about F
id8 versus big and deep about 28 at the
same point and it's obvious also if you
just look at the samples at this point
in the truncation curve big and deep is
basically producing all
uniform samples per class whereas Logan
still has pretty diverse samples so a
parallel line of work to the began work
in all of the image network was this
line of work from Nvidia the first work
in the series was called progressive
ganz the idea of this was sort of
similar to what they did in lap game
although it's formulated quite a bit
differently so the idea here is both for
efficiency and to get the model to
converge dependably they start off
generating at a very low resolution like
a four by four resolution and then after
your tiny image generator has converged
you can add an extra upsampling layer
you like you see here and a few extra
convolutional refinement layers to get
an 8x8 image generator if you started
with four by four then you wait for that
one to converge you repeat for sixteen
by sixteen 32 by 32 and so on and so on
until you get up to the final resolution
that you would like to generate in their
case they went to very high resolutions
of up to 1024 by 1024 and in the end
this resulted in extremely compelling
images at least in this restricted
domain of celebrity human faces and you
get what looks like pretty much
photorealistic results of human faces at
this very high resolution of 1024 by
1024 you know at least for me it's very
hard to tell the difference between most
of these faces and real human faces the
follow-up work from this team was called
style Gans so style games were also
shown to be capable of generating
remarkably photorealistic face images
and in this case they used was probably
a more challenging data set than the
last one with a lot more variation in
the images the data set they used in the
previous work progressive games was
mostly images of celebrity as whereas
this data set was a lot of a lot more
diverse and can mostly consists of
consists of images of not so famous
people so the interesting thing about
the architecture that they used in this
work was that it had these structured
latent inputs so they had these the
usual global latent are the usual Z's
that you have this input to the
generator but they also had these
spatial noise inputs so you can see in
the image that each column has sort of
the same global
sure global semantics like this middle
column for example seems to be late in
corresponding to you know young children
and this column seems to correspond to
being centered on the right side of the
image and looking towards the center and
that's because each column uses the same
global latent or is the spatial latent
is the same in each row and it seems to
mainly control in this case the sort of
background of the image as well as the
skin tone so what the architecture looks
like um is on this slide so on the Left
we have the usual flat vector Z which
they explicitly called the latent and
it's passed through a sequence of eight
fully connected layers an MLP to get the
final latent vector down here and then
this latent is input into every hidden
layer of the generator but the
interesting new piece here is that they
also have these pixel noise inputs over
here so at every layer you have a single
channel of random noise of the
appropriate resolution so four by four
eight by eight and so on and so on and
that noise is going to get
reincorporated eight at each of these
layers and as we saw before it ends up
using this global latent to control the
overall global appearance of the image
while these pixel noise latencies are
used to control the local variation of
the image and another example of what
this looks like in action is on this
slide so if you freeze the global
Ayden's and the course-level pixel noise
if you freeze all those you can change
just the fine high-resolution pixel
noise to get stochastic variations you
know in this case controlling start of
the fine differences and how this
toddlers hairs look so I hope that what
you can take away from this part of the
talk is a couple of things first there's
been pretty rapid progress in the span
of about five years scanning scaling up
gans from the amnesty digit images that
we saw in the original game paper to
these pretty large scale databases of
high resolution images like image net
and the flickr faces HQ dataset and the
improvements occurred really in a
variety of different places it wasn't
just about changing the architecture or
changing the objective it was really all
of these things combined the G and D
architectures have gotten better and
deeper the conditioning structure has
changed
the normalization has improved we saw
that batch normalization and spectral
normalization were quite helpful the
parameterization of the discriminator
has changed
we started off taking the conditioning
vectors input and now with the
projection discriminator we project
class embedding onto the hidden
representation of the image the latent
space structure has changed for example
in the style game paper where we had the
pixel noise Leighton's to control local
appearance and the loss functions have
changed which we saw more in caelis part
of the lecture and the algorithms have
changed for example in Logan where we
have an inner optimization of the
Layton's but while we can produce some
pretty convincing image I'd say the
problem is still pretty far from solved
for example these state-of-the-art
methods take a good amount of time in
quite a bit of computation to converge
and even with begins you know we're
still not great at every single image
category so I hope this gives you a good
idea of how the research has taken shape
into what the state of the art is today
and you know maybe even inspires you to
try your own ideas and make these
methods work even better so next I want
to talk about an application of Ganz
that I'm particularly interested in
which is the use of Ganz for
representation learning you'll hear a
lot more about the topic of unsupervised
representation learning in the next
lecture from mahalia Irina but for now
I'm going to address a few of the
directions that people have been
thinking about in terms of using Ganz in
particular representation learning so
just to give a couple of motivating
examples for why it might be interesting
to you use Ganz for representation
learning this is a slide that we saw
before but just to remind you so in the
dcen work Alec Bradford and
collaborators notice that in the latent
space of a deep convolutional again or
DC Gann you can do these kind of
arithmetic operations in latent space
indicating that certain directions in
latent space correspond to high-level
semantic attributes in the observations
space in this case human faces such as
the presence or absence of glasses or
the gender of the subject and all of
this arises without began ever being
explicitly told without without ever
being explicitly told about these
concepts of
or gender as another motivating example
I took the big in architecture and I
added an extra latent variable to the
generator input so this is a categorical
latent variable with a thousand 24
outcomes and it's just fed into the
generator as a one hot variable in
conjunction with a regular continuous
latent variable the 120 D calcium and
the kind of things that you get out of
this are pretty interesting so I train
this without class information it's
unsupervised and unconditional but it
does have this use this categorical
latent variable in place of the usual
explicit class label that you'd get in
the conditional supervised setting so it
seems to learn to associate this
categorical variable with high level
semantic groupings that almost look like
image categories so in this slide you
see about eight sort of randomly chose
and outcomes of the 1000 way categorical
variable and for example in this one
value this categorical variable shown in
the first row corresponds to what looks
like sea anemones another one looks like
a certain breed of dog and a sort of
grassy green background another it looks
like these kind of mountainous
landscapes and so this is really cool
and you can imagine that in a sort of
idealized case the dream might be that
it learns a clustering all in its own
that looks exactly like say the 1,000
image net categories or at least each of
these categories might be represented by
some combination of these categorical
variable outcomes and if that were to
happen then training a model that can
predict this latent variable given an
image would be exactly like training a
fully supervised image that classifier
and of course all of this came for free
because it's unsupervised so it's not
like the image and a data set where we
had to manually label each of the images
with category ID or you know pay
somebody to do that so going towards
that dream there have been many attempts
to get models that fulfill this promise
of learning representations using Ganz
completely unsupervised and I'll discuss
just a couple of them here one of the
first interesting papers from a few
years ago was called info Ganz or
information maximizing Ganz and compared
to regular Ganz it adds this inference
inference network to recover the latent
code Z given the generator output G of Z
which
in this set of experiments that we're
looking at is an imminent image of a
menace digit and what this does is
forced the generator to use each of its
input latent variables meaningfully in
order to maximize the information
content about the variables and the
images that it outputs and when you
train it with these latent codes it
learns to associate each outcome of the
categorical latent variable with a
different digit value and use the
continuous valued variables to vary the
style and the size and the rotation of
the digit so basically is using the
discrete latent to capture the discrete
variation in the data set and the
continuous latent to represent the
continuous variation the data set so
that's pretty cool and so one sort of
disadvantage of this approach when it
comes to representation learnings that
you don't have a ground truth latent
associated with real images like you do
for generated images so the inference
network that you've added here is only
ever getting to see generated images or
you have where you do have the latent
and so that might be OK for
representation learning when you have a
very simple dataset like M nough store
the generator is able to capture it
almost perfectly like you can kind of
see on this slide but when you go to cut
something more complex like image that
if your generator isn't perfect and it
probably won't because image that is
still really hard your generator is it
perfect then when you go to apply the
learned representations trained on these
generated images there's going to be
kind of a domain shift between the
generated images that the inference
network has seen versus the real images
that you want to get feature
representation for so then comes this
other class of methods that was called
either adversarial elearn inference a li
or bi-directional gans or begins and
this is sort of an adversarial approach
to jointly learning to generate data and
learn representations from it so
compared to a regular Gann the setup
adds an encoder Network which we'll call
Eve for most of this which learns the
inverse mapping from the generator G so
whereas the generator maps from features
or latent to images
G of Z the encoder does the opposite it
matches from images or data X to latency
of X and the other difference from a
regular Gann is you have a joint
discriminator so it sees not only an
or data point X or G of Z but it also
sees the latent Z or e of X so these X Z
tuples can either come from taking a
data point X and passing it through the
encoder to get a predicted layton a of X
or it comes from sampling layton Z and
passing it through the generator to get
a image G of Z and then the
discriminators job here is to figure out
which of the two generating processes
each of its input tuples came from and
the generator and encoders job are to
fool the discriminator basically into
picking the wrong process and it might
be a little confusing when you first
look at this because it's not entirely
clear what the jet encoders job is like
why does it have to produce anything in
particular for a given X so well it
turns out that under this objective of
discriminating between these two
different types of tuples there's a
global optimum here where if you have a
perfect discriminator and the generator
and encoder are perfectly satisfying the
discriminator then it turns out that the
encoder and generator have to invert one
another so if you pass an x an image
through the encoder and get a predicted
latency of X and then you pass that back
through the generator it should
perfectly reconstruct the input X that's
the global optimum of this model and
unlike in say auto-encoders were you
explicitly training for this property by
minimizing a squared error in this case
the encoder and the generate communicate
a don't communicate at training time so
they they never see each other's outputs
it's all done through the discrete
discrete ER so the encoder never sees
the outputs of the generator and the
generator never sees the outputs of the
encoder so one thing that makes us
interesting for feature learning is that
the encoder never suffers from the
domain shift problem I mentioned before
of C having to see these kind of weird
bad or at least initially bad generated
images that the generator gives you it
only ever sees real data which is
exactly what we want for a presentation
lending because it means that there's no
domain shift when we go to apply the
encoder to real images so in practice
this inversion property that we proved
to be true at the global optimum doesn't
actually hold perfectly but what you see
is
the reconstructions that you get from
passing X through the encoder and the
result back through the generator often
capture quite interesting semantics of
the inputs so for example if we look at
the digits here often the digit identity
between the original data X and the
reconstruction G of X is the same so for
example you know 2 goes to 2 3 goes to 3
etc etc so what that tells you is that
the representation the encoder gives you
is capturing the digit identity at least
to some extent and this is all just from
looking at the data we never explicitly
tell it what a 5 looks like and so on um
so if you scale these models up because
the original work we just looked at was
sort of at the DCN scale if you apply
this in the big and setting where you
have the same generator and
discriminator architectures as in begin
and you add an encoder model which
something like a state of the art
recognition image recognition model like
a resident style model at least a few
years ago some very interesting things
happen and we call these resulting
models with a few other tweaks that you
can read about in the paper we call them
big begins naturally so for example if
you pass this dog through the big bag in
encoder and back through the generator
to get a reconstruction the
reconstruction that you get is what
looks like a pretty similar dog although
with its tongue stuck out and kind of
facing in a slightly different direction
this person in a red coat in the winter
becomes a slightly more zoomed in person
in a red coat in the winter so in
general what many of these semantic
properties of the input get maintained
in the reconstructions even though the
model is never told what semantic
properties are interesting and all this
is happening because the structure of
the discriminator is essentially shaping
an implicit reconstruction error metric
in semantic ways at least this is kind
of my intuition for what's going on um
so the discriminator is a convolutional
network and we know that convolutional
networks are good at predicting semantic
attributes of images so the resulting
implicit that reconstruction error that
we're minimizing implicitly if not
explicitly mind you but but this sort of
implicit reconstruction error emphasizes
the semantics remaining the same even if
the individual pixel value has changed
quite a lot
so for example the model isn't going to
remember exactly what kind of pizza you
gave it but a war will remember it was
some kind of pizza and it was roughly in
this part of the image so it's almost
kind of human-like in terms of what it
remembers about the input image it has a
sort of fuzzy semantic memory of what it
saw without for example having to
remember you know the exact position of
every single blade of grass and this is
in contrast to the standard pixel wise
reconstruction objectives where it's
basically forcing the model to remember
every single pixel value so this is in
some sense exactly what we want in a
reason Tatian learning objective which
is what at least you know in my opinion
makes this an interesting method and
when you evaluate this quantitatively
and this sort of standard setup where
you basically take the encoder and use
it as a feature representation and train
a linear classifier supervisor on top of
that you get something pretty close to
state-of-the-art results compared to all
of these self supervised methods that
are very popular these days and which
will think here about in the next
lecture and another way to see what
representations are being learned by
this method is by looking at nearest
neighbors in the data set so you can
take images from the validation set as
queries and this left showing this
left-hand column here and find the
training set images that are closest to
them in big bag and feature space so in
general you can see that the nearest
neighbors tend to be very semantically
relevant to the input image in fact you
know with this dog from the validation
set here its nearest neighbor and the
training set shown here I think based on
the background it's in fact exactly the
same dog even though it's obviously
facing a different direction and if you
just looked at the pixel values this
would be quite different so it's kind of
cool that out of 1.2 eight million
images in the training set that ended up
being the nearest neighbor that same dog
at a different angle although it's
probably a little bit lucky it's still
fun finally for the last part of the
talk I just want to give you a taste of
some of the other modalities and the
different problem settings that people
are trying to tackle using generative
adversarial networks so starting with a
couple of
in the image space one of the coolest
lines of work in my opinion started with
this paper called pix depicts by Phil
Isola and his collaborators and what
they did in this setting was train a
generator to translate between images
from two different domains so for
example if you had satellite images like
these and you wanted to be able to
automatically translate these images to
kind of roadmap type images like you see
here and the way that they do this in
pics depicts is you take all these
paired examples of images so the
satellite image view and the
corresponding map view of the same area
and you train a conditional gam that
takes the aerial view as an input and
produce that the map view is an output
so the way you train this thing is you
have a standard gain objective a
discriminator that says does the output
of the generator look like a map view
that I've seen before but you also have
this l1 reconstruction error so since
you have a ground truth for what this
aerial or this map view is supposed to
look like you you can use this kind of
l1 pixels reconstruction error to tell
the generator that this is exactly what
your output should look like for this
input so basically it's kind of like a
traditional supervised learning setup
and you can see that this works in a
number of domains as you can see on the
slide
labels - street scenes edges -
photographic images of purses for
example and yeah so it's quite cool but
in the more general setting you might
not actually have paired examples so for
example if you want to train again that
translates between images of horses and
to zebras or vice versa you're probably
not going to have paired images where
all the horses and all the zebras are in
the exact same positions in the image
like we assumed we had in the pics to
fix work that we just talked about and
so enter this method called cycle gam
where you want to be able to sort of
unsupervised be able to translate
between two different domains with it
but without paired-samples between these
domains and the high-level idea of how
this works is by enforcing this property
they call a cycle consistency in
addition to all the normal gain
objectives so it's still again so you
start with an image
in domains Z domain a say it's an image
of zebras and then you translate to
domain B say it's an image of horses and
then translate back to domain a so
translate back to zebras and the zebra
image that you get after that process
should look pretty much exactly like the
image zebra image that you started with
so that's gives you an idea of how the
method works and as a result you can
basically translate between any two
domains that have sort of reasonably
similar information content such as
going from summer scenes to winter
scenes horse scenes to zebra scenes
photographs to different artists so this
is a really cool approach it's almost
you know a little bit magical that it
works and it produces some really cool
compelling results now I'm going to
touch on a little bit of work using Gans
for audio synthesis so we've can on the
left here was one of the first attempts
to produce raw audio waveforms using
Gans and they showed that for example
you can train unconditional gans to
produce reasonable one second clips of
piano music or human speech Mel Gann was
work on text-to-speech that takes as
input Mel spectrograms and produces Ross
speech audio as output and then there
was this other text-to-speech work from
our team at deep mind called Gantt ETS
where we take the linguistic features
aligned in time as input and produce
also produce raw speech audio as output
and both of these text-to-speech methods
work reasonably well for speech
synthesis which is pretty exciting
because they're also quite efficient
relative to many of existing
state-of-the-art approaches to
text-to-speech so in addition to images
people have also used Gans to generate
videos and predict future frames of
videos so you can apply a lot of the
same tools and toolbox that we've used
for images to videos as well of course
since you know it since you have within
a frame the same two-dimensional
structure that we have for images a
frame is an image but you also have a
third dimension time and that turns out
to make this problem a bit different and
arguably quite a bit harder than it is
for images partially just because of
potentional resources it takes to store
and generate videos versus still images
but also because humans are quite
sensitive to unrealistic motion so it's
important to get that right in order to
have reasonably convincing results so in
all three of these methods on the slide
a lot of a lot of work has gone into
making that computationally feasible so
one thing that we did in DVD again for
example in the middle here and it was
further developed in Tribune again was
to decompose the discriminator into two
separate discriminators neither of which
are seeing all of the pixels in the
video so it ends up being
computationally feasible that way so
there's one discriminator that we called
the spatial discriminator it operates
only on a few individual full resolution
frames but it only sees a few of the
frames a subset of them so that inch but
that discriminator basically ensures
that each frame looks connect coherent
independently and then there's another
discriminator that temporal
discriminator that sees multiple frames
but they're spatially downsampled so
that also doesn't see all the pixels
because it sees downsampled versions of
the images but that one is going to
ensure fluidity over time so together
that makes the problem from almost
computationally infeasible to being
fairly feasible and finally just to give
you a final taste of the many domains in
which people are applying gans
there's a reinforcement learning and so
this work on using games for imitation
learning called the generative
adversarial imitation learning or Gale
and essentially it uses a game like
method to learn a generator which in
this case ends up being a policy which
learns to imitate expert demonstrations
by fooling a discriminator whose inputs
are state action pairs and it addresses
many of the typical problems that people
see with standard behavioral cloning
methods in reinforcement learning
there's work I'm using Ganz for image
editing so that amateur artists for
example could specify just the course
layout of a scene without having to
actually paint every single detail and
then the Gann can go in and fill in the
low-level details with some pretty
nice-looking results and they have a
pretty fun demo that you can try out
online if you're
if you're interested there's work on
using Gans for program synthesis there's
this work from deep mind called spiral
or you have a generator that instead of
specifying each pixel value has to
specify individual actions like the
brushstrokes in a painting program so it
has to produce these discrete
instructions and you can't directly
backprop through this generation process
like you can in sort of standard image
generation Gans so you end up having to
use a reinforcement learning approach to
do this and you can imagine that you
could apply this to all sorts of
different types of programs not just
drawing ones there was a really cool
piece of work recently called everybody
dance now which was used for motion
transfer so you could take photos of
somebody in different positions who's
not a very good dancer and map the
movements of a professional dancer onto
their body so it looks like they have
you know professional level dance skills
and if you haven't seen the video demo
of this already you really have to go
look it up and watch it because it's
pretty entertaining and super
entertaining Gans have also been applied
to domain adaptation so domain
adaptation if you don't know is this
problem or say you might have a bunch of
label images of things happening during
the day within the daylight and you want
to train a classifier on that data and
then apply it to images of things
happening at night and by default this
won't work very well is there's going to
be a domain shift between day scenes and
night scenes and there's different
methods of alleviating that problem some
of them are using games like this one
here and finally there's a number of
artists using Gans for different kinds
of human machine collaborative art work
kind of and they produce some really
compelling art this way this is just one
example of that called learning to see
from an artist memo Lockton whose work
you should definitely check out if
you're interested in cool so thank you I
hope this lecture has given you a good
idea of the broad array of things that
people are doing with Gans and I hope
this might even inspire you to look
further into some of these applications
or try some new applications of your own
thanks
you