Google Gemini: AlphaGo-GPT?

in a somewhat provocative new interview
with Wired Magazine Demis hasabis head
of Google deepmind is quoted as saying
that Gemini which could be released as
soon as this winter will be more capable
than open ai's Chachi PT he reveals that
they are attempting to combine some of
the strengths of alphago type systems
with the amazing language capabilities
of large models before we look into how
that might work here is the context of
the Gemini announcement from Sundar
pichai they are focused on building more
capable systems safely and responsibly
this includes our next Generation
Foundation model Gemini which is still
in training while still early we are
already seeing impressive multimodal
capabilities not seen in Prior models as
the best promises that we also have some
new innovations that are going to be
pretty interesting and I know many
people will dismiss this as all talk but
remember deepmind was behind not just
Alpha go but also Alpha zero which can
play any two-player full information
game from scratch they were also behind
Alpha style which conquered Starcraft 2
with quote long-term planning and let's
remember that for later and most
famously perhaps a Sabbath LED them to
the incredible breakthrough of alpha
fold and Alpha fold 2 which are already
impacting the fight against plastic
pollution and antibiotic resistance so
let's not underestimate deepmind but
back to Gemini we hear from the
information recently that the
multi-modality of Gemini will be helped
in part by training on YouTube videos
and apparently YouTube was also mined by
openai of course that's not just the
text transcripts but also the audio
imagery and probably comments I wonder
if Google deepmind might one day use
YouTube for more than that a few days
ago they released this paper on robocad
which they call a self-improving
foundation agent for robotic
manipulation and the paper says that
with Robocat we demonstrate the ability
to generalize to new tasks and robots
both zero shot as well as through
adaptation using only a hundred to a
thousand examples for the Target task we
also show how a trained model itself can
be used to generate data for subsequent
training iterations thus providing a
basic building block for an autonomous
Improvement Loop notice that part about
using the model itself to generate data
that reminded me of a conversation I had
with one of the authors of the textbooks
are all you need paper Ronan eldan from
Microsoft I'm making a video on their
new Phi 1 model for coding we had a
really great chat and we were discussing
at one point AGI timelines and I said
this when you get Elite math papers with
proofs and Elite scientific research if
you train on much more of those for way
more epochs I don't think we're that far
away from AGI I personally can't see any
barrier within the next five years Ronan
said this as you said I also don't see
any barrier to AGI my intuition is that
there's probably a lot more Improvement
we can do with the data we have and
maybe a little bit more synthetic data
and this is even without starting to
talk about self-improving mechanisms
like Alpha zero where the more you train
models with some verification process
and you generate more data this can be
done in math and other things as we see
here with Robocat so you know there's
just so many directions where we can
still go that I don't think we're going
to hit a ceiling anytime soon can't wait
to show you guys the rest of that paper
and what else I learned from Ronan who
is also by the way the author of the
tiny stories paper but back to Gemini if
you remember the planning bit from
deepmind's earlier systems that reminded
me of something else from Gemini's
introduction Gemini was created from the
ground up to be multi-modal highly
efficient a tool and API Integrations
and built to enable future Innovations
like memory and planning this is echoed
in the article in which hasabis says his
team will combine a language model like
gpt4 with techniques used in alphago
aiming to give the system new
capabilities such as planning or the
ability to solve problems interestingly
this comes just a few weeks after
deepmind's Extreme risks paper which
identified long Horizon planning as a
dangerous capability for example
adapting its plans in the light of
unexpected obstacles or adversaries and
generalizing to novel or new setting for
me this is a bit like when a model can
predict what humans would do in reaction
to its own outputs back to the article
it's interesting though that asabis is
both tasked with accelerating Google's
AI efforts while also managing unknown
and potentially grave risks so what's
his take asaba says the extraordinary
potential benefits of AI such as for
scientific discovery in areas like
health or climate make it imperative
that Humanity does not stop developing
the tech technology he also believes
that mandating a pause is Impractical as
it would be near impossible to enforce
if done correctly it will be the most
beneficial technology for Humanity ever
he says of AI we've got to boldly and
bravely go after those things so how
would alphago become alphago GPT asabis
describe the basic approach behind
alphago in two of his recent talks so
what what's going on here then well
effectively if one thinks of a go tree
as the tree of all possibilities and
imagine each node in this tree is a go
position so what we're basically doing
is guiding the search with the model so
the model is coming up with most
probable moves and therefore guiding the
tree search to be very efficient and
then when it runs out of time of course
then it outputs the best tree that is
found up to that point we've learned
that from data or from simulated data
ideally you have both in many cases so
in games obviously we have this it's
effectively simulated data and then what
you do is you take the model and then
you use that model to guide a search
process According to some objective
function I think this is a general way
to think about a lot of problems I'm not
saying every problem can fit into that I
mean maybe and I'll give you example
from drug discovery which is what we're
trying to do at isomorphic so this is
the tree I showed you earlier finding
the best go move right and you're trying
to find a near optimal or close to
Optimal uh go move and go strategy well
what happens if we just change those
nodes to chemical compounds now let me
know in the comments if that reminded
anyone else of the truth of thoughts
paper in which multiple plans are
sampled and results were exponentially
better on tasks that GT4 finds
impossible like creating workable
crossword or mathematical problems that
require a bit of planning like creating
the greatest integer from a set of four
integers using operations like
multiplying and addition well I think my
theory might have some legs because look
at where many of the authors of this
paper work and just yesterday as I was
researching for this video the tree of
thoughts paper was also cited in this
paper on using language models to prove
mathematical theorems as you can see at
the moment gpt4 doesn't do a great job
but my point in bringing this up was
this they say towards the end of the
paper that another key limitation of
Chaturbate was its inability to search
systematically in a large space remember
that's what alphago is really good at we
frequently found that it stuck to an
unpromising path when the correct
solution could be found by backtracking
Allah tree of thoughts and exploring
alternative paths this behavior is
consistent with the general observation
that llms are weak at search and
planning addressing this weakness is an
active area of research and then they
reference the tree of thoughts paper it
could well be that Gemini let alone
Gemini 2 which is state of the art for
mathematical theorem proving and to be
honest once we can prove theorems we
won't be as far from generating new ones
and in my opinion fusing this alphago
style branching mechanism with a large
language model could work for other
things we've all seen models like gpt4
sometimes give a bad initial answer
picking just the most probable output in
a way that's sometimes called 3D
decoding but methods like smart GPT and
self-consistency demonstrate that the
first initial or most probable output
doesn't always reflect the best that a
model can do and this is just one of the
reasons as I said to Ronan I honestly
think we could see a model hit 100 in
the mmlu in less than five years the
mmlu which I talked about in my smart
GPT video is a famous machine learning
Benchmark testing everything from formal
logic to physics and politics and I know
that predicting 100 performance within
five years is a very bold prediction but
that is my prediction but if those are
the growing capabilities what does
demisasabas think about the implications
of the sheer power of such a model one
of the biggest challenges right now
hasaba says is to determine what the
risks of a more capable AI are likely to
be I think more research why the field
needs to be done very urgently on things
like evaluation tests he says to
determine how capable and controllable
new AI models are he later mentions
giving Academia Early Access to these
Frontier models and they do seem to be
following through on this with deepmind
open Ai and anthropic giving Early
Access to their Foundation models to the
UK AI task force this Foundation model
task force is led by Ian Hogarth who was
actually the author of this the we must
slow down the race to Godlike AI paper
that I did a video on back in April do
check that video out but in the article
Hogarth mentioned a practical plan to
transform these companies into a
cern-like organization and somewhat
unexpectedly this idea was echoed this
week by none other than Satya Nadella
who had earlier called on Google to
quote dance essentially the biggest
unsolved problem is how do you ensure
both at sort of a scientific
understanding level and then the
Practical engineering level that you can
make sure that the AI never goes out of
control and that's where I think there
needs to be a CERN like project where
both the academics along with
corporations and governments all come
together to perhaps solve that alignment
problem and accelerate the solution to
the alignment problem but back to the
article the interview with asabes ended
with this somewhat chilling response to
the question how worried should you be
asaba says that no one really knows for
sure that AI will become a major danger
but he is certain if progress continues
at its current Pace there isn't much
time to develop safeguards I can see the
kind of things we're building into the
Gemini series and we have no reason to
believe that they won't work my own
thoughts on this article are twofold
first that we might not want to
underestimate Google and hasabis and the
adding alphago type systems probably
will work and second based on his
comments I do think there needs to be
more clarity on just how much of Google
deepmind's Workforce is working on these
evaluations and preemptive measures this
article from a few months ago estimates
that there may be less than 100
researchers focused on those areas out
of thousands so is it even five percent
of the total and if not how can we take
too seriously the commitments at any AI
Summit such as the one happening this
autumn in the UK on safety on the other
hand if asabis revealed that half or
more of his Workforce were on the case
then we could be more confident that the
creators of alphago and my fellow
londoners had a good chance of
researching to safety and success as
always thank you so much for watching
and have a wonderful day


The AI News You Might Have Missed This Week - Zuckerberg to Falcon w/ SPQR

here are seven developments in AI that
you might have missed this week from
chatgpt avatars to open source models on
an iPhone an alpha Dev to Zuckerberg's
projections of super intelligence but
first something a little unconventional
with a modicum of wackiness embodied VR
chess best robot on my left is being
controlled by a human in a suit over
there and this robot on my right is
being controlled by a human over there
they both have feedback gloves they have
VR headsets and they're seeing
everything that the robot sees now
specifically today we're looking at
avatars robot avatars to be precise they
can play chess but they can do much more
they can perform maintenance rescue
operations and do anything that a human
do with its hands and eyes could this be
the future of sports and things like MMA
where you fight using robotic embodied
avatars but for something a little less
intense we have this robot Chef who
learned by watching videos
foreign
[Music]
[Music]
it does make me wonder how long before
we see something like this at a
McDonald's near you but now it's time to
talk about something that is already
available which is the hey gen plugin in
chat GPT it allows you to fairly quickly
create an avatar of the text produced by
Chachi BT and I immediately thought of
one use case that I think could take off
in the near future by combining the
Wolfram plugin with hey gen I asked
chatgpt to solve this problem and then
output an explainer video using an
avatar a quick tip here is to tell
Chachi PT the plugins that you wanted to
use otherwise it's kind of reluctant to
do so as you can see chatty PT using
Wolfram was able to get the question
right but for some people just reading
this text won't quite cut it so check
this out the retail price of a certain
kettlebell is seventy dollars
this price represents a 25 profit over
the wholesale cost
to find the profit per kettlebell sold
at retail price we first need to find
the wholesale cost
we know that seventy dollars is one
hundred and twenty five percent of the
wholesale cost
next we have Runway Gen 2 which I think
gives us a glimpse of what the future of
text the video will be like
a long long time ago at lady
winterbottom's lovely tea party which is
in the smoking ruins and Ashes of New
York City a fierce women ain't playing
no games and is out to kick some butts
against the unimaginable brutal
merciless and scary Blobby boy of the
delightful Grand Budapest Hotel hi and
everything seems doomed and lost until
some man arises the true hero and great
Mastermind behind all of this
now of course that's not perfect and as
you can see from my brief attempt here
there is lots to work on but just
remember where mid-journey was a year
ago to help you imagine where Runway
will be in a year's time and speaking of
a year's time if AI generated fake
images are already being used
politically imagine how they're going to
be used or videos in a year's time but
now it's time for the paper that I had
to read two or three times to grasp and
it will be of interest to anyone who is
following developments in open source
models I'm going to try to skip the
jargon as much as possible and just give
you the most interesting details
essentially they found a way to compress
large language models like Llama Or
Falcon across model scales and even
though other people had done this they
were able to achieve it in a near
lossless way this has at least two
significant implications one that bigger
models can be used on smaller devices
even as small as an iPhone and second
the inference speed gets speeded up as
you can see by 15 to 20 percent in
translation that means the output from
the language model comes out more
quickly so the best of my understanding
the way they did this is that they
identified an isolated outlier weights
in Translation that's the parts of the
model that are most significant to its
performance they stored those with more
bits that is to say with higher
Precision while compressing all other
weights to three to four bits that
reduces the amount of Ram or memory
required to operate the model there were
existing methods of achieving this
shrinking or quantization like round to
nearest or gptq but they ended up with
more errors and generally less accuracy
in text generation as we'll see in a
moment spqr did best across the model
scales to cut a long story short they
envisage models like Llama Or indeed
Orca which I just did a video on
existing on devices such as an iPhone
14. if you haven't watched my last video
on the Orca model do check it out
because it shows that in some tests that
13 billion parameter model is
competitive with chat gbt or GPT 3.5 so
imagining that on my phone which has 12
gigs of RAM is quite something here are
a few examples comparing the original
models with the outputs using spqr and
the older form of quantization and when
you notice how similar the outputs are
from spqr to the original model just
remember that it's about four times
smaller in size and yes they did compare
llama and Falcon at 40 billion
parameters across a range of tests using
spqr remember that this is the base
llama model accidentally leaked by meta
not an enhanced version like Orca and
you can see the results for llama and
Falcon are comparable and here's what
they say at the end spqr might have a
wide reaching effect on how large
language models are used by the general
population to complete useful tasks but
they admit that llms are inherently a
dual use technology that can bring both
significant benefits and serious harm
and it is interesting the waiver that
they give however we believe that the
marginal impact of spqr will be positive
or neutral in other words our algorithm
does not create models with new
capabilities and risks it only makes
existing models more accessible speaking
of accessible it was of course meta that
originally leaked llama and they are not
only working on a rival to Twitter
apparently called project 92 but also on
bringing in AI assistance to things like
WhatsApp and Instagram but Mark
Zuckerberg the head of meta who does
seem to be rather influenced by Jan
lacun's thinking does have some
questions about autonomous AI my own
view is that
where we really need to be careful is on
the development of autonomy and how we
think about that because it's actually
the case that relatively simple and
unintelligent things that have runaway
autonomy and just spread themselves or
you know it's like we have a word for
that it's a virus could be simple
computer code that is not particularly
intelligent but just spreads itself and
does a lot of harm a lot of wood I think
we need to develop when people talk
about safety and responsibility is
really the governance on the autonomy
that can be given to systems it does
seem to me though that any model release
will be fairly quickly made autonomous
look at the just two-week Gap the
release of GT4 and the release of Auto
GPT so anyone releasing a model needs to
assume that it's going to be made to be
autonomous fairly quickly next
Zuckerberg talked about super
intelligence and compared it to a
corporation you still didn't answer the
question of what year we're going to
have super intelligence I'd like to hold
you to there now I'm just kidding but is
there something you could say about the
timeline
as you think about the development of
AGI super intelligence systems
sure so I I still don't think I have any
particular Insight on when like a
singular AI system that is a general
intelligence will get created but I
think the one thing that most people in
the discourse that I've seen about this
haven't really grappled with is that we
do seem to have organiz organizations
and you know structures in the world
that exhibit greater than human
intelligence already so you know one
example is a you know a company but I I
certainly hope that you know meta with
tens of thousands of people make smarter
decisions than one person but I think
that would be pretty bad if it didn't I
think he's underestimating a super
intelligence which would be far faster
and more impressive I believe than any
company here's one quick example from
deepmind where their Alpha Dev system
sped up sorting small sequences by 70
because operations like this are
performed trillions of times a day this
made headlines but then I saw this
apparently Gypsy Ford discovered the
same trick as our confidev and the
author sarcastically asks can I publish
this on nature and to be honest when you
see the prompts that he used it strikes
me that he was using GPT 3.5 the
original Chachi BT in green not gpt4
anyway back to Super intelligence and
science at digital speed when you hear
the following anecdote from demisasabis
you might question the analogy between a
corporation and a super intelligence
Alpha fold is a sort of Science of
digital speed in two ways one is that it
can fold the proteins in you know
milliseconds instead of taking years of
experimental work right so 200 million
proteins you times that by PhD time of
five years that's like a billion years
of PhD time right by some measure that
has been done in in a year billions of
years of PhD time in the course of a
single year of computation honestly AI
is going to accelerate absolutely
everything and it's not going to be like
anything we have seen before thank you
so much for watching and have a
wonderful day


DeepMind x UCL | Deep Learning Lectures | 11/12 | Modern Latent Variable Models

hello and welcome to
series on topics in
deploring my name is Andrea Nick and I'm
a research scientist at deep mind I work
on generative modeling variational
inference and representation learning
this lecture will cover modal latent
variable models as well as various types
of inference and in particular
variational inference so the lecture is
structured as follows I will start by
introducing generative modeling and
covering the three major types of
journey of models used in deep learning
then I will focus on latent variable
models explain what they are and why
inference is so important for them then
we will cover a special case of latent
variable models invertible models where
we can do exact inference then we will
move on to intractable models or exact
inference is not an option and we will
look at variational inference for
training those models variational
inference requires estimating gradients
of expectations which is not a trivial
problem so then we will look at how to
estimate these gradients and finally we
will look at a modern application of
variational inference to powerful models
which results in variational
auto-encoders so let's look at
generative modeling what are generative
models well they're the models are
simply proved allistic models of high
dimensional data so conceptually they
describe the process the probabilistic
process of generating and observation
and we can think of them as describing
mechanisms of generating more data
points and the key distinction between
other probabilistic models and
generative models is that our
distributions that we're modeling are
really high dimensional so in classic
settings like classification and
regression you're basically modeling a
one-dimensional output distribution
while in genitive modeling you are
dealing with a high dimensional
distribution and often you're
essentially don't have an input so
you're just modeling the distribution of
the output for this particular reason
generative modeling has been seen as a
sub area of unsupervised learning
because we're simply modeling the Joint
Distribution of the data and we don't
have any labels on the other hand if you
think about generative models as
including conditional generative models
which also have a context which is quite
a bit like an input then the boundary
becomes rather blurry so it's really
more about the technology rather than
the actual application it is used for
and there are many types of generative
models and they can handle essentially
any type of data from text to images to
video and so on so let's look at some
uses of generative models the most
established and maybe traditional one
comes from statistics and it's called
density estimation and here we simply
fit a generative model to the data in
order to get probability distribution
that we can evaluate at any given data
point and once we have this probability
distribution we can use it to actually
tell is this given data point from the
same distribution as a training data or
is it an outlier from some other rather
different distribution this kind of
models can be used for applications like
fraud detection there's also a close
connection between probabilistic
modeling and data compression
so there's actually exact reality
between these two areas so if you
actually have a probabilistic model of
the data you can use arithmetic coding
to produce a data compressor we can also
use generative models for mapping
between two high dimensional domains for
example between sentences in one
language and their translations in
another language
so here the the sentence in the original
language will be the context and the
model will capture the distribution of
possible translations for the given
sentence and typically there will be
many possibilities there rather than
just a single correct translation
another exciting application of
generative modeling is in model-based
reinforcement learning where the
generative model essentially act like a
probabilistic simulator of the
environment so then the algorithms can
actually use this simulator to plan
optimal sequences of actions rather than
actually having to try them in the
environment to see what happens and once
we've done this planning we can actually
execute the sequence of actions in the
real environment some types of
generative models are really useful for
a presentation learning where we would
like to condense the observations down
them down to some essential features
some sort of low dimensional
representations of them that capture the
essence and these low dimensional
representations might be more useful
than the original observations for
downstream tasks such as classification
and often we don't actually know what
the down tree downstream tasks will be
so it's important to summarize the data
in a generic way and Geritol models
provide a way of doing that and finally
there's this idea of understanding the
data that also comes from statistics and
this is the area where the generative
model will have a particular meaning to
its structure so the latent variables
will potentially be interpretable or the
parameters will have some real-world
significance so once we train such a
model on the data we can look inside of
it using inference for example or look
at the parameter values and it will tell
us something about the data distribution
something that we can't easily see just
by looking at the individual data points
directly
the next few slides are meant to give
you a sense of rapid progress that has
happened in generative modeling in the
last few years so the individual models
are not very important so I'm just
showing you samples from models trained
on datasets typical for that particular
year so we start in 2014 where the
typical data set was M List which
contained low dimensional images binary
images of digits then one year later
there's been already some progress and
now we can have models that capture to
some degree the distribution of natural
images still low dimensional but now
they're in color and they're
considerably more complicated than ages
the images are indeed blurry but we can
see some global shapes and maybe some of
these objects might be recognizable to
you and then four years later we can
model much higher dimensional images
with much better results so these are
not perfectly photorealistic but the
local detail is very convincing and the
global structure is quite good as well
there's clearly room for improvement but
it's it's a long way from the binary
images of digits so let's look at the
popular types of generative models in
deep learning you have seen actually
many of these mentioned before in the
preceding lectures in this series so
I'll just give a very brief overview so
what regressive models are most
prominent for language modeling where
they are typically implemented using
recurrent neural network or transformers
then we have latent variable models
which are subdivided into tractable such
as invertible or flow based models and
intractable ones like variational hold
anchors and this is the kind of model we
will cover in depth in this lecture and
finally there are implicit models most
notably
generative adversarial networks and
their variants so let's look at each one
of these types in slightly greater
detail so autoregressive models solve
the problem of modeling the Joint
Distribution of observations X by
subdividing it into simpler subproblems
so instead of modeling P of X directly
we actually model be one dimensional
conditional distributions corresponding
to this Joint Distribution the resulting
model is tractable and can be easily
trained using maximum likelihood so why
this is a good approach
well one-dimensional distributions are
actually quite easy to model because we
can use the off-the-shelf classifier
technology that has been very successful
in deep learning and such models are
simple and efficient to train as we
don't need to do any kind of sampling of
random variables at training time on the
other hand because we're modeling a
sequence of dimensions of conditional
distributions sampling from such models
is inherently a sequential process which
means it is slow we have to go through
one dimension at a time and we cannot
easily paralyzes the other weakness of
such models is that they naturally focus
on the local structure rather than
global structure so unless you build
some sort of inductive bias towards
capturing the global structure into the
model directly you are likely to have
less success with modeling the global
structure with these models then we have
late
variable models which are also
likelihood based like auto regressive
models but they take a different
approach to modeling the joint
distribution so they do it by
introducing the unobserved or latent
variable that in some sense explains or
generates the observation so we start
with the latent variable and then we
also define the transformation that
map's the latent variable value to the
particular observation these models are
also trained using maximum likelihood or
more typically some approximation to
maximum likelihood because often maximum
likelihood is intractable here and
latent variable models provide a very
powerful and well understood framework
and mature framework that has been
around for a long time in statistics
they make it really easy to incorporate
prior knowledge and various structure
constraints into the model so if you
would like to model some sort of
statistical or physical process you have
some ideas about how its structured this
is typically the model type you will use
and because generally they don't use
auto regressive or sequential
subcomponents sampling from such models
is efficient on the downside these
models require understanding the concept
of inference which is the reverse of
generation so this means going from the
observation to the plausible latent
values that could have generated it so
you need to understand and implement
this concept in order to use these
models that makes them somewhat more
complex than auto regressive models and
as I mentioned previously for many such
models inference is intractable so
either we have to introduce the
additional complexity of using
approximations for inference or we have
to restrict ourselves in what kind of
models we can use in order to ensure
that inference remains tractable
third class of popular generative models
in deep learning are generative
adversarial networks and unlike the
previous two types these are not
likelihood based these are so-called
implicit models because they don't
actually assign probabilities to observe
ations
they just give you samplers that
generates observations so the model here
that we're training is simply a neural
network that takes a vector of random
numbers and maps it to the observation
and unlike the other two classes of
generative models we just looked at
these models are trained using
adversarial training rather than maximum
likelihood so adversarial training works
by introducing an auxilary model a
classifier that is trained to
discriminate between samples from the
generator the model and the training
data and the gradients from this
classifier provide a learning signal
that we can use to train the model or
the generator so the main appeal of
these models is that they are by far the
best ones for modeling images so the
images they generate are extremely
realistic they are also relatively easy
to understand conceptually because
you'll need to understand the concept of
inference and your training a model
simply by back propagating through a
classifier and like latent variable
models they provide fast generation
because generating observation involves
simply performing a forward pass in a
neural network on the other hand turn to
adversity all networks don't give us the
ability to assign probability to
observations so this means that we can't
use them for many applications of
generative models such as outlier
detection or lossless compression they
also suffer from so-called
mode collapse and this is the case when
a model trained on the data set ignores
some part of the training data and
models only a subset of the training
data which is a bit worrisome and not
something that you see was likelihood
based models because they're essentially
obligated to model every data point
and the other difficulty was mode
collapse is that we don't actually have
control over which part of the data
distribution will be ignored on the
other hand if you just want realistic
samples from some part of the data
distribution then gans do it really well
and the other difficulty with ganz is
that optimization is actually a subtle
point optimization problem and as a
result training is often unstable and
requires a lot of small tricks to get it
right so in this lecture we will focus
on latent variable models and inference
so let's look at this generator modeling
framework so a latent variable model
defines an observation a distribution
over observations X by introducing a
latent variable Z along with that we
specify its prior distribution as well
as the likelihood P of X given that that
connects the latent variable to the
observation so P of X given that
essentially tells us how to map a
configuration of latent variable to a
distribution over the observations and
even though I say a latent variable
typically Z is a vector or it can be a
tensor or anything like that
conceptually it doesn't really make much
of a difference so once we have the
prior and the likelihood we have
specified the model completely and the
model is completely characterized by the
Joint Distribution P of X comma Z which
we obtained simply by multiplying the
likelihood by the prior and there are
two distributions that we can derive
from the Joint Distribution that will be
of interest to us for latent variable
modelling so the first such distribution
is P of X which is the marginal
likelihood of an observation and it
tells us how probable the observation is
under the model and this is the quantity
that would optimize if we're doing
maximum likelihood learning
and then there's the posterior
distribution P of Z given X and this is
the distribution of plausible latent
values that could have generated the
given observation X so we can think of
the latent variable as some sort of
explanation for the observe Asian so how
do we generate observations from a
latent variable model it's actually
quite simple
we start by sampling the latent variable
Z from the prior P of Z and then we
sample X from the likelihood
distribution P of x given that which is
conditional on the configuration of the
latent variable and much of this lecture
will be concerned with inference which
is the process of going back from the
observation X to a distribution over the
latent variable said so in this lecture
inference will specifically refer to
computing the mysterious tribution given
the observation so computing P of Z
given X how is P of Z given X defined
well we simply use the definition of
conditional probability which says that
P of Z given X is the ratio of the Joint
Distribution under the model P of X
comma that divided by the marginal
probability of X P of X so this means
that in order to compute the posterior
distribution we first need to compute
the marginal probability of X G of X or
the marginal likelihood how do we do
that well we do that by starting with
the Joint Distribution T of X comma Z
and marginalizing out the latent
variable Z in the continuous case it
will be integration so we will integrate
over Z the joint distribution in the
discrete case it will be a summation but
typically in this lecture I will use
integration and now we will see that
inference is in a very specific
formal sense the inverse of generation
so let's think about two ways of
generating the observation / latent
variable pairs X Z so one way to
generate such pairs is to start by
sampling the latent variable Z from the
prior and then sampling the duration
from the likelihood this is what we've
done two slides ago this gives us a
distribution of X that pairs but we can
also sample X that pairs in a different
way first we can sample X from the model
using the same process and then just
discarding the original latent
configuration that led to Xen and now
that we have this X we can perform
inference and simple as that from the
posterior distribution for it for this X
from P of Z given X this gives us
another way of generating pairs X and Z
and because the product of the
distributions we are sampling from in
both cases is exactly the same it's the
joint distribution P of X comma Z it
means that the distribution of these
pairs is exactly the same so this means
that sampling from the variational from
the exact posterior is a probabilistic
inverse of generation so
why is inference important well
inference is important in its own right
because once we've trained a model we
can use inference to explain
observations in terms of latent
configurations so it might potentially
allow us to interpret observations in
terms and set some latent variable
values moreover as we will see a bit
later inference comes up naturally in
maximum-likelihood training of latent
variable models it's a sub problem that
we will need to solve over and over in
the inner loop of optimization so let's
look at an example of inference in a
very simple latent variable model a
mixture of gaussians you have probably
seen this model before it's perhaps the
simplest latent variable model you can
imagine so it has a single latent
variable it's a discrete one and it
takes on K values between 1 and K the
probability of Z being I is simply pi I
and then each latent variable value
corresponds to a mixture component which
is Gaussian and the mean and the
standard deviation of this Gaussian is
determined by the value of the mixing
component so we can think of this as
having a vector of means and a vector of
standard deviations for the mixing
component and then the latent variable
simply selects which dimension of these
vectors we will use to define the
Gaussian let's compute the marginal
likelihood or the marginal probability
of the observation X so as we saw before
this requires marginalizing out Z from
the joint of the model and the joint is
simply the product of the prior key of Z
and the likelihood P of x given set
since it's the discrete model we're
performing summation to marginalize out
Z by summing over its values from 1
through K now that we have
the marginal likelihood we can compute
the posterior distribution because P of
Z given X is just the ratio of the joint
probability of X and Z divided by the
marginal probability of X and we
computed the marginal probability above
and the joint probability also is a sub
problem there so now we have an
expression for the posterior probability
of Z given X as you can see we can
compute this posterior distribution in
linear time in the number of latent
variable values so this model is clearly
very tractable now let's look at maximum
likelihood learning which is how we
would like to Train latent variable
models maximum likelihood is a very
well-established
estimation principle for probabilistic
models in statistics and be the basic
idea behind it is that we should choose
those parameters of the model that make
the training data most probable so this
corresponds to maximizing the product of
probabilities of data points in the
training set or the computation
convenient we can maximize the sum of
log probabilities of the data points
because we're looking for the optimal
parameters rather than the objective
function value these two approaches are
exactly the same they give us the same
parameter values unfortunately for
latent variable models we can't solve
this optimization problem in closed form
so as a result we use various iterative
approaches either based on gradient
descent or expectation maximization
so let's look at the gradient of the
marginal log likelihood for a single
observation so the gradient of log T of
X is equal to now weary we recall that
the derivative of log is its argument
derivative of its argument divided by
the argument so here we have the
derivative of the marginal probability
divided by the probability itself then
we expand the marginal probability in
terms of the Joint Distribution and
integrate over the latent values set and
we exchange the derivative of the
integral on the next line we replace the
derivative of the joint by the joint
times the derivative of the log
probability of the joint using the and
the identity in the yellow box and this
is the same identity we used on the
first line of this derivation now that
we have reformulated the integral that
way we can see that we have a ratio of
probability of the joint configuration X
set divided by the probability of the
marginal X this corresponds to the
posterior distribution T of Z given X so
we rewrite it like that and now we can
see that the gradient of the log
marginal probability is simply an
expectation with respect to the
posterior distribution of the gradients
of the log joint so this means that in
order to compute the gradient of the log
marginal probability which is what we
need for maximum likelihood estimation
we need to compute the posterior
distribution somehow so this is
basically an essential subproblem and
the other thing we can see here is that
the posterior probabilities
modulate the gradient contributions from
the log joint to the gradient of the
marginal log likelihood so it basically
up weighs the configurations that that
were more likely to generate this
observation and down weigh the
configurations that are less likely so
this basically means that inference
deforms credit assignments among latent
configurations for the given
observation so unfortunate
exact inference is hard in general to
see why this is the case let's think
about computing the marginal likelihood
of an observation which is as we've seen
an important part of computing the
posterior distribution
so if our latent variables are
continuous then computing the marginal
likelihood involves integrating over
high dimensional space and typically the
argument will be integrating over will
be a nonlinear function so analytical
integration will not be an option and
numerical integration in order to get a
reasonable level of accuracy will also
not be an option because the complexity
of integration will go exponentially in
the number of latent variables in the
discrete case the situation is slightly
better because now instead of
integrating over the latent
configurations who are summing over a
finite number of them so we know that we
could considerably enumerate all those
configurations and compute the marginal
probability like that but the issue is
the same as in the continuous case the
curse of dimensionality so if the number
of latent variables is more than a
handful then the number of possible
joint latent configurations will be so
large that we will never be able to
compute this sum exactly there are some
exceptions where we have interesting
models with exact inference and we've
seen we with exact tractable inference
we've already seen one example it's a
it's a mixture model where inference is
basically linear in the number of mixing
components the other important subclass
is linear Gaussian models so these are
models with Gaussian latent variables
and linear mappings in these models all
the induced distributions are Gaussian
and as a result inference is tractable
and finally we have the interesting case
of invertible models so these models are
special because they're actually quite
powerful and
yet they allow exact inference through
clever constraints on their structure
and we will see these models a bit later
in this lecture so how can we avoid
these intractable computations that
exact inference involves well there are
two general strategies here the first
one is simply to restrict ourself when
designing the model so that the
resulting model will be tractable this
will give us easier training because we
can do exact maximum likelihood without
any approximations but it will make
model design more complicated and in a
sense considerably restrict the modeling
choices we can make on the other hand if
we're interested in creating a model
that represents our knowledge about the
task then we might want to just build
the model with you know all the required
properties that we would like and then
worry about the inference later and
almost certainly we will end up with an
intractable model but that's okay
because there are approximate inference
methods and we will be willing to pay
the price of using an approximate
inference with some extra complexity
that that entails but then we will be
able to use more expressive models so
let's
look at the first strategy of working
with tractable models and exact
inference so we will look at these
modern tractable but very powerful
models called invertible models also
known as normalizing flows and they're
specially interesting because they
combine high expressive power restrict
ability which is rather rare and the
basic idea behind these models is simply
starting with some prior distribution
like in any latent variable models and
then applying an invertible function to
it to obtain the observation and the
parameters of the model are all
incorporated in this invertible function
and by warping the prior distribution in
various ways we can approximate the data
distribution so the
invertible there's constraints the
structure of the model in a very
specific way and makes inference and
maximum likelihood tractable in these
models so let's look at the generative
description of an invertible model so to
specify an invertible model we need the
prior distribution as before P of Z and
to here we will assume that it has no
parameters but it doesn't make much
difference this is just for convenience
and then we use an invertible
differentiable transformation F of Z
which has parameters theta to transform
samples from the prior into observations
so all the model parameters here will be
in this function f and because we use F
that's invertible having this setup
gives us one-to-one correspondence
between latent configurations and
observations so there's absolutely no
ambiguity about which light in
configuration generated the given
observation because the function is
one-to-one so this means that we can
simply compute the latent configuration
by inverting F and applying it to X so
we apply F inverse to the observation
and we exactly recover the only latent
configuration that could have generated
this observation so this is very nice
inference it's very easy and fully
deterministic so now how do we compute
the marginal likelihood we need for
maximum likelihood training right
we need to somehow relate the prior
probability and the probability of the
observe Asian X and it turns out that
because we use an invertible
differential transformation to connect Z
to X we can apply the change of
variables formula and then the density
is the probability of T of Z and T of X
differ by just a scaling factor and this
scaling factor is the absolute value of
the determinant of the Jacobian of the
methane from X to Z this might seem a
bit counterintuitive or surprising where
does this factor come from and this
factor simply accounts for the fact that
when we apply a function to go from Z to
X from or X to Z it will change the
infinitesimal volume around the point
where it's being applied and so if we
want the resulting distribution to
normalize to 1 just like the original
distribution we need to take into
account that volume your scaling factor
and this is exactly what the determinant
of the Jacobian takes into account so we
would like to get rid of Z in that
expression because we want to evaluate
probability of X just on data points X
and we can get rid of X by remembering
that we can get rid of Z by remembering
that Z is simply F inverse of X so
wherever we have Zed we replace it with
F inverse of X and now we have an
expression for the probability of x that
makes no reference to that so now
conceptually at least we can compute the
marginal probability of X and we can
before maximum likelihood training so
from the practical angle to do maximum
likelihood estimation we still need to
have some requirements for F so in
particular we need to be able to compute
F inverse of X as
as a determinant of its Jacobian because
it's used in the expression of the
marginal probability of X and we also
need to compute their gradients because
that this is what's required for maximum
likelihood estimation and finally these
computations need to be sufficiently
efficient for maximum likelihood to be
fast so let's look at a very simple
invertible model perhaps the simplest
and maybe the oldest are called the
independent component analysis so this
model starts with a factorial prior so
each latent dimension is modelled as a
univariate distribution independently of
the other dimensions and the latent
values are mapped to the observation
using a square matrix a so this is a
linear model since inference in a
invertible model involves inverting f
inference here is simply multiplying by
the inverse of a so to compute Z from X
we simply multiply X by a inverse and
once we've trained such a model we can
use it to explain our observations in
terms of latent independent causes that
explain the data linearly and the
typical application for this model is
solving the so-called cocktail party
problem where you have n sound sources
around the room for example people
talking and then you also have n sensors
and microphones and you would like to
isolate individual people from this
mixed recording and because sound
acoustics ensures that mixing is
approximately linear this is a an
appropriate model so inference on
recordings from microphones acts will
allow us to recover individual sources Z
and in order for this to identify
independent sources
there's an interesting constraint the
choir cannot be Gaussian because
Gaussian latent variables are
rotationally symmetric in high
dimensions so we cannot actually recover
independence we can only recover D
correlation so typically the prior we
use here is some sort of heavy tail
distribution like a logistic or kocchi
so how do we construct general
invertible models well the strategy is
simple because a combination or
composition of invertible
transformations is invertible we simply
use a library of simple invertible
transformations and chain a lot of them
together to obtain the more expressive
invertible transformation and here each
of these simple building blocks can be
parameterize either in the forward
direction mapping from Z to X or in the
reverse direction from X to Z whichever
one we would like to be more efficient
when using the model so depending
whether we want training or inference to
be more efficient we parameterize the
appropriate method and one interesting
detail here is that we don't actually
need F to be analytically invertible it
is fine if F can be inverted only
numerically with an iterative algorithm
as long as we have a reasonably
efficient algorithm that require that
recovers the inverse to numerical
precision and in terms of building
blocks there's a rapidly growing list of
them this is an active area of research
and I give a few examples there on the
slide so invertible models are very
appealing because they are both powerful
and they are tractable so easy to train
so why don't we use them all the time
well they do have a number of
limitations which make them not always
appropriate so one obvious limitation is
that the dimensionality of the latent
vector and of the observations has to be
the same
this is a consequence of requiring the
function f to be invertible there's no
way around it
so if we'd like a lower dimensional
latent space for some sort of low
dimensional representation of the
observation we simply can't easily do
this with an invertible model the other
requirement is that the latent space has
to be continuous and this is because we
use changed of density to compute the
marginal probability of x there has been
some initial work on discrete flows so
this limitation might be relaxed in in
the future there the consequence of
using continuous latent variables and
applying invertible transformation to
them is that it makes it hard to model
discrete data because the output of such
a transformation will also be a density
so unless our observations are
continuous or quantized which means that
they were discretized based on some
underlying to use distributions we can't
really apply invertible models to such
data and because the models are
constructed by chaining a lot of simple
transformation together the resulting
models tend to be quite large in order
to have high expressive power so this
means that we will need to store a lot
of activations and parameters which
makes it easy to run out of GPU memory
when training such models so in terms of
expressiveness per parameter or per
kilobyte of memory these models are less
expressive than more general latent
variable models and finally compared to
general latent variable models it's hard
to incorporate structure in invertible
models because we have to retain
inverter bility so that removes a lot of
options for a model design
on the other hand because invertible
models are tractable and powerful they
make very useful building blocks to
incorporate into other models in
particular intractable latent variable
models they provide a very useful
abstraction that basically gives you a
distribution that can be trained exactly
and gives you the exact marginal
likelihood so that makes them very
composable and appealing as building
blocks in the second half of the lecture
we will look at intractable models and
variational inference as a way of
training them so why would we want to
use intractable models well sometimes
the structure of the model or its latent
variable have some sort of intrinsic
meaning for us we might be modeling some
real-world process and the underlying
quantities have some a grounded meaning
and we would like to structure the model
in a particular way that captures that
so this is different from thinking of a
model as just some sort of black box
that produces predictions or merely
generate samples so we want some sort of
interpret ability then the basic
question is and I like this quote from
David Bly do you want the wrong answer
to the right question do you want the
right answer to the wrong question and
this basically highlights the dilemma we
have do we want to use the right model
with approximate inference or
potentially the wrong model with exact
inference and in many situations when we
take modeling quite seriously it makes
sense to go for the wrong answer to the
right question so in many cases we will
end up with an intractable model that
captures our desired properties and we
will just have to use approximate
inference
so here's an example of how easy it is
to end up with a intractable model even
though the starting point is tractable
so as we've seen the ICA model with the
same number of latent dimensions as
observation dimensions is tractable it's
a very simple linear model so what would
happen if we change this model slightly
suppose we would like to model a bit of
observation noise to indicate that our
microphones are not perfect so adding
observe a shinto the model makes the
model intractable because the mapping is
no longer invertible if we use more
latent dimensions and observations the
model once again becomes intractable and
even if we use fewer dimensions than
observations of duration dimensions the
model becomes intractable once again so
it really doesn't take much to go from a
simple tractable model to an intractable
and once we have an intractable model in
order to use it or train it
we need to use approximate inference and
there are two broad classes of
approximate inference the first class is
Markov chain Monte Carlo methods and
here we will represent our exact
posterior using samples from it but
using exact samples and to obtain an
exact sample from the true posterior we
set up a Markov chain which we run for
quite some time and at some point it
converges to the right distribution
which is the true posterior and then the
sample from it is a sample from the true
posterior so the advantage of this
method is that it's very general we
really don't need to restrict our model
essentially in any way we can use Markov
chain Monte Carlo for inference and this
method is also exact in the limit of
potentially infinite time and
computation so we if we spend enough
time generating samples there will be
from the right distribution that we
generate enough samples who will
basically have our answer to the
arbitrary degree of precision so
some senses the gold standard for
inference
unfortunately in practice it's very
computationally expensive and so doing
Markov chain Monte Carlo is not really
an option in many cases also convergence
actually knowing when we are sampling
from the right distribution is really
hard to diagnose so often we just wait
for some time
until we're tired of waiting and then we
use the sample at that point hoping that
it's from the right distribution but
doing this can actually introduce a
subtle error because it might still not
be the true posterior that were sampling
from and we have no way of quantifying
or controlling for this so the other
class of approximate inference methods
is variational inference and here the
idea is rather different instead of
sampling from the true posterior in some
freeform we say we will approximate the
true posterior with their distribution
with some particular simple structure so
for example we will say we will
approximate the true posterior with a
factorize distribution which models each
latent dimension independently so and
then we fit this approximation to the
true posterior using optimization the
advantage of this approach is that it's
much more efficient than Markov chain
Monte Carlo as optimization is generally
more efficient than simply on the other
hand we cannot trade computation for
greater accuracy as easily because once
we've chosen the form of this a
posterior proximation once we've
converged running for longer doesn't
give us any more accuracy but unlike in
Markov chain Monte Carlo we have
something that guarantees that we are
performing reasonably well at every
point because we have a bound on the
marginal log likelihood so we can
essentially at least hypothetically
quantify the approximation error so
look at variational inference in detail
so the one-line description of rational
inference is it turns inference into an
optimization problem and it's called
variational because we're essentially
optimizing over a space of distributions
and as a result we are approximating
some unknown posterior distribution with
a distribution from some particular
family
and the distribution that we'll be
approximating the exact posterior will
be called the variation of Asteria we
will denote it as Q of Z given X and it
will have parameters Phi which are
called the variational parameters and
they're there just to make sure that our
variational posterior approximates the
true posterior G of Z given X as
accurately as possible and what are the
restrictions on the choice of the
variational posterior well our hands are
pretty much free as long as we can
sample from this distribution and we can
compute the probabilities or log
probabilities under it and the
corresponding parameter gradients that
we need in order to fit this
distribution to the true posterior so a
classic and default choice is simply
using the fully factorized distribution
Q where each dimension is modeled
independently from all others
variational inference allows us to train
models by approximating the marginal
log-likelihood which in itself is
intractable because model is intractable
so we can compute the original log
likelihood but by introducing this
simplified form of the variational
posterior allows us to define an
alternative objective which is closely
related to the marginal log likelihood
and this objective is a lower bound on
the marginal look likely and we trained
the model by optimizing this lower bound
with respect to the parameters of the
model Phi and the parameters of the
rational posterior Phi so parameters of
the model theta and the parameters of
liberation of posterior Phi and because
this is a lower bound it's guaranteed to
be below the value of the marginal log
likelihood so when we maximize the lower
bound we're usually also pushing up the
marginal log likelihood even though we
can't actually compute it exactly so how
do we obtain this variational lower
bound on the marginal log likelihood so
let's consider any density Q of Z as the
only requirement is that this density is
non-negative
whenever the prior distribution is
non-negative then we start by expire
expanding the marginal log likelihood in
terms of the Joint Distribution where we
integrate over the latent variable and
then we introduce this density that we
chose by both multiplying and dividing
the modal joint by it so this doesn't do
anything because multiplying and
dividing by the same quantity has no
effect but once we've done this we can
apply the yongsan inequality which
states that the log of the expectation
of some function is always greater than
or equal than the expectation of the log
of this particular function so this
allows us to push the log inside the
integral
and take the integral with skew outside
the log and we know that the resulting
quantity is less is less than or equal
to the preceding quantity because of the
yunsun inequality and now we recognize
that this new expression is simply
simply the expectation with respect to
this distribution Q that we introduced
of Log density ratio between the Joint
Distribution P of X set and this density
and the important thing to recognize is
that because there's density Q that we
used in this derivation is arbitrary and
for any setting of parameters of the
density Phi we will have a lower bound
on the marginal log likelihood which
basically allows us to get a state of a
bound as possible simply by maximizing
this expression with respect to the
parameters Phi and thus getting closer
approximation to the marginal log
likelihood so there are several possible
variation of lower bounds and in this
lecture we will focus on essentially the
bound we derived on the previous page
where instead of the arbitrary density Q
we will use the variation of posterior
here of Z given X and this is both the
simplest and by far the most widely used
variation of bound so this is the bound
you will see in most variational
inference papers there's a more recent
option called the importance weighted
lower bound also known for historical
reasons as a way and this is simply a
multi sample generalization of the
evidence lower bound and it's
interesting feature is that it allows
you to control the tightness of the
bound are the accuracy of approximation
to the margin likelihood by increasing
the number of samples you use in the
bound so this is not quite as flexible
as Markov chain Monte Carlo where you
use more computation to get more
accurate results because the scaling is
you know you get rapid improvement as
you go from one sample to ten samples
but once you go beyond that the
improvement quickly levels out but still
you can get some
easy gains without changing the form of
the operational procedure but for
simplicity we will use the elbow in the
rest of this lecture so let's review a
concept important for variational
inference and this concept is called
back Leibler divergence KL divergence
provides us with a way of quantifying
the difference between two distributions
and KL divergence between Q and P is
defined as the expectation under the
distribution peel of the log density
ratio of Q to P and it has a few
important properties we will need for
the rest of the lecture so first of all
the KL divergence is a non negative for
any choice of Q and T the KL divergence
is 0 if and only if Q and P are the same
almost everywhere so we can basically
think Q and P are the same distribution
is the only case when the KL divergence
is 0 and finally it's important to
remember that KL divergence is not a
metric so it's not symmetric in its
arguments so KL from Q to P is not the
same as KL from P to Q in general so
let's look at optimizing variational
lower bound with respect to the
variational parameters Phi of the
variation of posterior Q so let's start
by rewriting the elbow so in on the
first line we factor the Joint
Distribution into the marginal
probability of X and the exterior
probability of Z given X this is just
another factorization of the joint
density of the model on the next line we
simply take out the term for the
marginal log-likelihood into the first
term and then keep the rest as the
second term giving us expectation under
Q of the log density of the true
posterior to the variation of Asteria
now in in the first expectation on that
line we see that log P of X actually
does not depend on Z so its expectation
under liberation posterior is just
itself so log P of X and then we
recognize the second quantity a second
expectation simply as the minus KL from
the variation of posterior Q of Z given
X to the true posterior P of Z given X
so let's look at that the composition of
the variation lower bound so we have two
terms the marginal log likelihood and
the KO so the marginal log likelihood
depends on the model parameters theta
but it does not depend on the variation
of parameters Phi so when we maximize
the variation lower bound to the rate
with respect to the variation of
parameters Phi the first term is
unaffected
therefore maximizing the elbow with
respect to variational parameters is the
same as minimizing the KL divergence
from the variational posterior to the
true posterior and this KL from the
variational steerer to the true
posterior quantifies the distance from
the variation of posterior to the true
posterior and it is known as the
variational gap because we can express
it also as the difference between the
marginal log-likelihood log P of X and
the variational bound L of X so this
means that when we are maximizing the
elbow with respect to the variational
parameters
we're actually minimizing the KL
divergence from the variational
posterior to the true posterior so we're
making sure that variational posterior
is a better and better fit to the true
posterior this is actually remarkable
because this this is a model which is
intractable so we cannot actually
compute the true posterior at all and we
can't even compute the scale divergence
from the variational posterior to the
true posterior because it involves the
true posterior which we can compute in
the first place so if we look at that
the composition of elbow from the
previous slide
the difference between the log marginal
likelihood and the KL from the
variational to the true posterior we
realize that the elbow is actually a
difference between two intractable
quantities and yet it is tractable so it
means that both of these quantities are
intractable in the same way so they have
this intractable part that's exactly the
same and we when we take the difference
between them it cancels out also looking
at this decomposition and remembering
that the KL divergence is non-negative
and it's 0 if and only if the two
distributions are effectively the same
it means that the best value of the
variation lower bound we can get is
actually the same as the marginal log
likelihood log P of X and that happens
when the KL is 0 and this can only
happen if Q is a very expressive
distribution that can approximate the
true posterior exactly so that's good
for understanding variational inference
but in practice is not going to happen
with a variational model now let's think
about maximizing the variational bound
with respect to the other set of
parameters the model parameters what
happens when we update these parameters
to increase the variational lower bound
well looking at the same decomposition
we see that well either the first term
the marginal log-likelihood
will increase or the second term will
have to decrease that's the only way to
get the increase in the variation lower
bound so let's look at the first option
when we update the parameters and the
marginal log likelihood increases this
is good because this is the same as what
maximum likelihood learning parameter
update does we're increasing the
marginal log likelihood but
what happens when the variational lower
bound is increased because we actually
decreased the variational gap well there
are two ways of decreasing the
variational gap so we've seen the first
one a couple of slides ago when we were
updating the variation of parameters and
because that was equivalent to
minimizing the KL from the variational
posterior to the true posterior that was
decreasing the variational gap as well
and doing this was clearly good because
we were getting a better and better
approximation for the variational
posterior of the true posterior and the
model was not affected by these updates
because the model is not affected by the
variational parameters on the other hand
now if we update the model parameters
and the variational gap decreases it it
means that the model has changed so the
way in with it changed there are two
possibilities so first of all the
inference in this model variational
inference of this model did become more
accurate because the variational
posterior remained the same but the true
posterior moved towards it so now
they're closer together but when this
happens this is actually not always
desirable because it means we're
spending some of the model capacity to
actually approximate the variation of
this terior rather than to model the
data so in a sense the model is trying
to contort itself so that inference in
it is easy and if we only have so much
capacity in the model it will probably
make it less good of a model of the data
so this means that if we are worried by
such effect if you would like to have as
faithful approximation to maximum
likelihood as possible we should use a
expressive of a variation of a steer as
possible because this will reduce the
variation or gap and there will be less
of a pressure for the model to distort
itself like that and one
particular manifestation of this effect
in model strain using variational
inference is called variational pruning
and this is when the model refuses to
use some of the latent variables so
they're essentially not used to generate
the data which means that their
posterior and their prior are exactly
the same
and when I say posterior I mean both the
true posterior and the variational
posterior because when the model is
unused its true posterior is the same of
the prior and it's very easy to
approximate with the variation of
Asteria
and this is in fact why variational
pruning happens because when you prune
out some variables it becomes easier to
perform variational inference so there's
this extra pressure on the model to be
simpler in that way and variational
pruning is also known as posterior
collapse in the operational autoencoder
literature so it's very tional pruning a
good thing or a bad thing well it
depends how you think about it in some
circumstances it can be a good thing
because you can think of it as choosing
the dimensionality of the latent space
automatically based on your data
distribution on the other hand it gives
away it takes away some of our freedom
to over fit to the data so sometimes in
deep learning you would like to have a
very accurate model of the training data
even if you're when you're not concerned
with overfitting and you can easily
achieve this by giving the model many
many hidden units so making the hidden
layers wider and then you are guaranteed
to or fit to the data often driving like
classification error to zero well if
you're training a generative model and
you would like to achieve something
similar overfitting to the data
arbitrarily well by giving it lots and
lots of latent variables well if you're
using a variational inference the model
will actually refuse to use extra
variables after some point and number of
variables it will use can be
surprisingly small and sometimes it's
clearly suboptimal so you would like the
model to use more variables but because
liberation posterior is too simple
compared to the true posterior it will
simply destroyed
are the rest of the latent variables and
how do we choose the form of the
variational posterior well the default
choice as I mentioned before is a fully
factorize distribution with each
dimension modeled independently and this
form is known as the mean field
approximation for historical reasons
because the method originated in physics
we can make the variational distribution
more expressive and we have several
choices for doing that
so one possibility is to use the mixture
model so instead of a unimodal
distribution we will have a multi-modal
distribution now if you were using a
variational posterior that's a diagonal
Gaussian which is a very common choice
we can introduce richard covariance
structure so we can for example have a
low rank or full covariance Gaussian at
separation or posterior we can make the
variational posterior or two regressive
which will make training more expensive
like many of the other choices but we'll
provide much more modeling power or
alternatively we can take an invertible
model and use it to parameterize the
variation of asturias flow and this
works very nicely because variational
models are tractable and ultimately
we're making this trade-off between the
computational cost of training the model
and the quality of the variation
approximation and perhaps fit to the
data on the other hand some of these
choices for the more expressive
mysterious also have some practical
downsides because you might run into
numerical instability problems so you
have to be careful and watch out for
that and sometimes when you use a richer
variational posterior you actually get
worse results and this should not happen
in theory if optimization is perfect but
due to various stability issues and
learning dynamics issues this can
actually happen
all right so let's think about what
we're doing when we're fitting a
variational distribution so first of all
the posterior distribution of course is
different for every observation X
because each X is generated by some
latent configuration that's more
probable than others so we have a
distribution of our plausible
explanations for X this means that we
need to fit a different variation of
posterior for each of servation
and in classical racial inference this
means that we simply have a separate set
of distribution parameters which of
derivation that we optimize over and
this also means that we perform a
separate optimization run for each data
point whether it's a training
observation or a test observe Asian to
fit the corresponding variation of
parameters this can be inefficient
because basically we we learn nothing
from fitting variational parameters for
one data point about all the other data
points so we can actually amortize this
cost by replacing this separate
optimization procedure for each data
point with some sort of functional
approximation so we will train a neural
network that will take the observation
and output an approximation to its
variational parameters and we will train
this network which we'll call the
inference Network to basically to serve
as the approximation told those
independent variational posteriors we
were training before and as a result now
instead of deforming potentially costly
iterative optimization for each data
point to obtain its posterior we simply
perform a forward pass in the inference
Network that gives us the variational
parameters and these are the ones that
we use for the operation of this theory
so now we replaced all these independent
variational parameters that were data
point specific with a single set of
neural network parameters that are
shared between all observations and we
amortize the cost of solving these
optimization problems among all
observations so once we've trained such
an inference network we can come
durational posterior for a new data
point simply by fitting the data point
today network and it will produce the
corresponding duration of the sphere so
this is a very powerful idea because it
allows us to easily scale up variational
inference to much bigger data sets and
models than before and this idea of
amortized inference was introduced in
the context of compost machines in the
mid 90s and it was popularised recently
by variational thinkers that rely on it
and as mentioned before the variational
parameters are trained jointly with the
model parameters simply by maximizing
the elbow with respect to both and now
we basically have two sets of neural
network parameters one for the model and
one for the in for instance let's step
back and think about what we gained and
what we gave up by performing
variational inference well now we can
train intractable models in a principled
way and relatively efficiently this lets
us choose any kind of model we want and
incorporate any kind of prior knowledge
into the model so that's great from the
modeling standpoint and inference is
quite fast especially if we use
amortization compared to MCMC methods so
some models are simply infeasible for
MCMC and variational inference makes it
possible to train them and what did we
lose well we do typically give up some
of the model capacity because we're not
using expressive enough variational
posterior but perhaps that's fine
because essentially in many cases
variational inference is the only option
for training a model is large on a data
set from a particular size so we either
have a slightly suboptimal fit or we
have to resort to a much simpler model
so we saw that training a model using
variational inference requires computing
the gradients of the variation lower
bound with respect to the model
parameters theta and the variation of
parameters Phi well
the elbow is actually an expectation so
computing gradients of the of an
expectation might not be so
straightforward so let's look at how we
can do this well in classic variational
inference the expectations were
typically computed in closed form and
then optimization did not involve any
kind of noise in the gradient estimates
because the objective function was
analytically tractable on the other hand
do you actually have expectations that
you can compute in a closed form
required models to be very simple as
well as the variational posteriors to be
generally fully factorized because
otherwise you couldn't compute the
expectations so variational inference in
its classic form was applicable to only
a small set of models on the other hand
recent developments in variational
inference replaced exact estimation of
the gradients with monte carlo based
estimation and here we don't try to
compute the expectation or its gradients
in closed form instead we use Monte
Carlo sampling from the variation and
posterior to estimated and that gives us
much more freedom in terms of what kind
of models we can handle and the answer
is essentially we can handle almost any
kind of latent variable model so let's
look at how we can estimate the
gradients of the elbow with respect to
the model parameters
this is actually the easy case so
expanding the definition of the elbow
there we see that only the joint
distribution of the model depends on the
model parameters inside the expectation
and the variation of posterior does not
depend on it also the expectation the
elbow involves is an expectation with
respect to the variational posterior
which does not depend on the model
parameters this means we can safely move
the gradient inside the expectation and
this means that the gradient of the
elbow with respect to the model
parameters is simply the expectation
under the variational thus terior of the
gradient of the log joint for the model
and this quantity is really easy to
estimate we simply sample from the
variation of posterior evaluate the
gradients of the log joint based on the
resulting samples and then we average
them and in practice given one sample
can be enough to train a model so one
thing to mention here is since we're
using sampling to estimate gradients
there is some noise in the gradient
estimates and basically gradient
estimate noise can be a bad thing
because it prevents us from using larger
learning rates so if the noise level is
too high we have to use a sufficiently
low learning rate to avoid divergence
which mean makes training models slower
so generally we would like to have
gradient estimates that are relatively
low variance increasing the number of
samples we take is an easy way of
reducing this variance now let's look at
the case of the gradient for the
variation of parameters this is a more
complicated situation because now the
gradient we are computing it involves
the parameters of the distribution the
expectation is over so we can simply
take the gradient inside the expectation
because this will result in incorrect
estimates so what do we do here well it
turns out that gradients of expect
of this form computing them is a
well-known a research pull-up problem
and there are several good methods for
estimating these gradients available so
let's look at the two major types of
unbiased gradient estimators of such
expectation so here we will look at the
general case of an expectation of a
function f in in variational inference
this F will be just log density ratio of
the joint to the variation of this Tyria
so the first type of the gradient
estimator is called reinforce or
likelihood ratio estimator and it's it's
very general so it can handle both
discrete and continuous latent variables
and it does not place any stringent
requirements on the function f that it
can handle so f can be non
differentiable so that's nice it's a
very general estimator the price to pay
for this is that the resulting gradient
estimates are relatively high variance
so unless you perform some additional
variance reduction in almost all
practical situations you need to use an
extremely tiny learning rate so this is
essentially infeasible so using or
enforce without variance reduction is
essentially hopeless the other type of
estimator is called reprimand
realization or pass wise estimator and
this estimator is considerably less
general it requires us to use continuous
latent variables and it supports only
some continuous latent variable
distributions but the class is quite
large it also requires the function
inside the expectation to be
differentiable but this is fine because
in variational inference this is
typically the kind of function that we
get but the big advantage of this
estimator is that out of the box it
gives you fairly low gradient variance
so you don't need to worry too much
about variance reduction and you can
still estimate the gradient which is
sufficiently low variance and train the
model sufficiently quickly so
let's look at the Ripper motorisation
trick which is essentially how pass wise
gradients are known in the modern
machine learning literature and the
high-level idea here is simply to take
the parameters of the distribution the
expectation is with respect to and
somehow move them outside the
distribution and inside the expectation
and once we've done that we're in the
same situation as for the gradient of
the model parameters for elbow because
now the distribution of the expectation
will not have the parameters we're
differentiating with respect to so we
can just take the gradient inside so how
do we achieve this we do this by remote
rising samples from the distribution Q
of Z and we do that by thinking of them
as a transformation of samples from some
fixed distribution with no parameters we
will call these samples epsilon and then
we will apply some deterministic
differential transformation to it we
will call G that will incorporate the
dependence on the parameters into the
sample so epsilon that comes from the
epsilon does not depend on any
parameters but once we transform it
using G epsilon Phi s that now depends
on the parameter is Phi through this
function G so we factored out the
randomness from the samples and the
parameters in two separate boxes so now
that we've done this factorization we
can rewrite the expectation of F with
respect to distribution Q in terms of G
so now we replace Z inside as the
argument of F with G Epsilon Phi because
that's how we computed that and because
we generate that by sampling from P of
Epsilon now the expectation is with
respect to absolute rather than that so
now the expectation is with respect to
the distribution that does not depend on
the variation of parameters so we can
now safely take the gradient with
respect to fly inside the expectation
and now we compute the gradient of F of
G with respect to Phi by using the chain
rule and remembering that G of epsilon
Phi is simply Z so then we evaluate the
gradient of F at Z where Z is equal to G
Epsilon Phi and then multiply it by the
gradient of samples Z
as a function of parameters file and
this expectation has the same form as
the gradient of the elbow with respect
to the model parameters so we can
estimate it by sampling from the
distribution P epsilon and averaging the
gradients over the sample and we get a
low variance graduate estimate like that
so as I explained before our permutation
trick essentially moves the dependence
on the parameters of the distribution
from the distribution itself into its
samples and thus inside the expectation
the main requirement here is that the
resulting mapping that takes epsilon to
that has to be differentiable with
respect to the parameters Phi because
when we factor out the randomness in the
parameters into two separate bits we're
essentially propagating gradients
through Z and into the function and its
parameters so let's see how we can
repeat rise the one dimensional Gaussian
random variable Z that comes from a
distribution with mean mu and standard
deviation Sigma well if we start with a
standard normal
epsilon we can scale it by Sigma and
then add the mean mu and then we get
exactly the right distribution for that
so we can see that the mapping that we
use mu plus Sigma Epsilon is
differentiable with respect to both mu
and epsilon so it satisfies the apparent
the requirements of the parameters H so
this is a valid your parameters H and
this is how Gaussian sorry premature
eyes in practice so what about other
distributions so many distributions such
as those in the location scale family
such as laplace M Kashi can be
reprimanded of this approach for some
other discontinues distributions such as
gamma in der slay there is actually no
way to factor out randomness out of
parameter dependence so we can separate
these things to
there is a generalization of reprimand
ization called implicit reprimands ation
that still allows us to propagate
gradients through samples from such
distributions on the other hand there
are some continuous distributions that
cannot be reproduced and all discrete
distributions cannot be repeated for the
simple reason that even though we can
factor out randomness and parameter
dependence the function that we end up
with is not differentiable so applying
your immunization trick will not give us
the right gradients the good news is if
you want to use the real promoters ation
for continuous distributions modern deep
learning frameworks such as tensor flow
and pi torch implement this for you so
all you have to do is to pass the flag
that you want your sample re permit
rised when you're generating it from one
of the standard distributions and
automatic differentiation will take care
of everything so implementing
variational inference this way now is
very easy so now let's look at perhaps
the most successful application of
rational inference in recent years and
that's variational autoencoders so
variational auto-encoders are simply
generative models with continuous latent
variables where both the likelihood p of
x given Z and the variational posterior
are parameterize using neural networks
typically the prior and the variational
posterior are modeled as fully
factorized gaussians and VI use a
trained using variational inference by
maximizing the elbow using both
amortized inference and the upper
motorisation trick and this combination
of using expressive mappings for the
likelihood and liberation of posterior
and amortized inference and remote
ization made via is very popular because
they are highly scalable and yet
expressive models
so let's look at a slightly more
detailed description of a variational
autoencoder so we start with a prior P
of Z which is typically a standard
normal and then our decoder
which is another term for likelihood in
VI you speak will simply be either a
neural network computing the parameters
of a Bernoulli distribution if we're a
modeling binary data or a neural network
computing the mean and the diagonal
variance of a Gaussian distribution if
we are modeling real-valued eight and
for the variation of posterior once
again we use a neural network that
outputs the parameters of the variation
of posterior after taking the
observation X as the input and the type
of the neural network we use to
parameterize these models doesn't really
matter it doesn't change the
mathematical structure of the model so
you can easily use kind of Nets res Nets
or any kind of neural network you would
like and when training V is the elbow is
typically written in a slightly
different way from the one that we've
seen before so the elbow is decomposed
into two tractable terms this time so
the first term is the expectation over
the variational posterior of log P of X
given that so this is log likelihood and
the second term is just minus the KL
divergence from the variational
posterior to the prior because here the
second argument is the prior rather than
the true posterior this can actually be
computed and in fact this is often
computed in closed form which is easy to
do for a distribution such as gaussians
so the first term essentially measures
how well can we predict or reconstruct
the given observation after sampling
from its variational posterior and this
term is typically known as the negative
reconstruction error so high values of
it are good the second term we can think
of it as a regularizer
that pushes the variational posterior
towards the prior to make sure that we
put not too much information into the
latent variables in order to reconstruct
the observations well
and this scale is essentially an upper
bound on the amount of information about
the observation we have in the latent
variables under the variation of this
interior so the the model has been
around for quite a few years and it has
been extended in many many ways so now
it's really more of a framework than an
actual model so the framework generally
means that this is a model with
continuous latent variables trained
using amortized variational inference
and the Ripper motorization trick and
the extensions that have been discovered
for the AES are numerous so for example
here I covered only a single latent
layer well you can have multiple latent
layers you can have latent variables
that are non Gaussian you can have much
more expressive priors and mysterious so
for example you can use invertible
models for both you can use richer
neural networks for example rez nets or
you can have Auto regressive likelihood
terms so that you combine some of the
properties of other aggressive models
with latent variable models and people
have also worked on improving
variational inference either by making
it slightly closer to classic
variational inference and so the
one-shot making it slightly iterative
where you do only a couple of updates
and also people have worked on variance
reduction in order to get lower variance
gradients so we can train the models
faster so to conclude this lecture has
covered two modern approaches to
powerful latent variable models which
are both based on likelihoods and they
make rather different decisions about
what's important whether its exact
inference or freedom in model design and
this classification of models into these
different types is useful for
presentation purposes but some of the
most interesting work is actually about
combining models of different types
which allows you to basically take
advantage of their complementary
strengths so I mentioned for example
using auto regressive
decoders in variational Auto encourage
you can also use auto regressive
exteriors and so on and you get the
extra modeling power of auto regressive
distributions and yet you still retain
potential interpretability with latent
variables and what's exciting about this
area is that it's still relatively new
and developing very rapidly so there are
many substantial contributions that
remain to be made
you