DeepMind x UCL | Deep Learning Lectures | 7/12 |  Deep Learning for Natural Language Processing

hello and welcome to the UCL and
deepmind lecture series
my name's Felix Hill and I'm going to be
talking to you about deep learning and
language understanding so here's an
overview of the structure of today's
talk it's going to be divided into four
sections so in the first section I'll
talk a little bit about neural
computation in general and language in
general and then give some idea of why
neural computation deep learning and
language might be an appropriate fit to
come together and produce the sort of
improvements and impressive language
processing performance that we've seen
over the last year in the second section
of focus in on one particular neural
language model which i think is quite
representative of a lot of the
principles that govern all new language
models and that model is the transformer
which was released in 2018 and then in
section 3 I'll go a bit deeper into a
particular application of the
transformer that's the well-known bird
model and bird in particular is an
impressive demonstration of unsupervised
learning and the ability of neural
language models to transfer knowledge
and from one training environment to
another and then in the final section
we'll take a bit more of a look towards
the future of language understanding and
deep learning and to do that we'll delve
into some work that's been done a deep
mind on grounded language learning where
we study the acquisition of language in
deep neural networks that have the
ability to interact and move around
simulated environments so that's the
overall structure it's important to add
that of course that natural language
processing is an enormous field and
there are many things that I'm not gonna
have the time to talk about during this
lecture so some of the most important
ones are things like sequence to
sequence models and specific
applications of deep learning to neural
machine translation
speech recognition and speech synthesis
are really important applications that I
won't have time to talk about I mean
then there's many NLP tasks which I also
won't get the chance to delve into for
machine comprehension and question
answering and dialogue and even in
grounded language learning in the last
section I won't get the chance to go
into things like visual question
answering video captioning so ensure
that the important thing to take away is
that I'm not gonna have a chance to
cover all aspects of natural language
processing I'm gonna just talk about a
few focused areas and that's because I
think they are quite representative and
they hopefully convey the key concepts
and it's not because I think they're
more important or more valid than any
other areas yeah cool so let's start off
with a bit of background about deep
learning and language and how they might
fit together of course where we are now
is that there's been a load of
impressive results relating deep
learning to natural language processing
in the last few years so you may have
heard of models like GPT - or Burt
wavenet which was developed in deep mind
and all of these models have done really
impressive things with respect to the
various aspects of language processing
that they focus on so GPT - as a
language model is now able to produce
long streams of text which look like
plausible stories and Berger has led to
very large improvements on many language
classification tasks and of course
wavenet has led to fantastic performance
in speech synthesis what we now able to
synthesize voices various speech
applications with much more fidelity
than was previously possible so it's
really like an exciting era of natural
language processing and we're moving at
a rate of progress which is possibly
unprecedented at least in recent years
so if you think about all that sort of
panorama of different things you might
be able to apply language models or
language processing technology to like
to a much greater extent than at any
point in the past neural computation and
deep learning plays a role in those
systems so on the Left we have systems
which are almost now entirely based on
new
networks from machine translation
systems to speech synthesis systems and
speech recognition and then on the right
here it's important to note that there
are still many applications which do
language processing but don't use deep
learning or neural networks for all of
their computation or even at all so
things like Homa systems which you might
have to provide specific pieces of
information from the internet we're
still a long way from her building
systems like that in an end-to-end
fashion in your networks having said
that the balance of this particular
scale has moved a lot over the last few
years and is certainly a trend towards
more applications of neural computation
and neural networks in language
processing applications and it's not
just in practical applications in the
slightly more focused world of research
we see a similar trend so this is data
from 2010 to 2016 and it covers
submissions to two of the main language
processing conferences ACL and NLP and
on the chart you just see the number of
papers published at those conferences
for which the word deep or neural is
found in the title and you can see that
back in 2010 there was close to or
effectively zero papers with those words
in the title but by the time we got to
2016 this number had scaled up rapidly
and of course there's a very good chance
that if we looked at the data up to 2020
we would just expect this trend to have
continued in that time and it's
obviously not just the numb
republication effective quality of
systems and models but seems to be
improving over this time so here's just
a snapshot in time and 2080 19 of how
well the best model was able to perform
on the glue benchmark so the glue mmmm
just a sort of intended to be a
represented language classification
challenges things like reading a couple
saying whether one of them entails
another one or maybe classifying them as
positive sentiment or negative sentiment
things like that so our ability to do
those sorts of things automatically is
rapidly increased according to this
benchmark just between 2018 and 19 you
can see the rate of improvement from
under 60% performance to over
approximately 85% promise and again on
this task on this benchmark and the
performance is just increased and
increased up to the present day so this
is a sort of taking together a bunch of
evidence that you know deep learning has
really been able to improve performance
on a bunch of language processing
applications and I think looking at that
evidence it raises the question of why
deep learning and models which have this
neural computation at the heart of their
processing have been able to be so
effective in language processing what is
it about deep learning and what is it
about language which has sort of allowed
this sort of effect to take place and of
course if we can answer that question if
we can understand that then that can
help us to think a little bit about ways
to improve things further but of course
in order to understand that we really
need to think a bit about language so in
the other lectures in this series I
think you've had a would have had a very
comprehensive introduction to deep
learning your networks and principles of
neural computation in this lecture I'd
like to just spend a bit of time to
think about language in itself so we can
start to think about why these two
paradigms fitball together so the first
thing about language it's often said
that language is a process of
relating symbols or the language
processing involves symbolic data in
operations on symbols so you know if we
had a sentence coming into a network and
a sentence coming out of a network then
one characterization of that problem is
from mapping symbols to symbols these
very discrete units but of course if
those who think a little bit more about
language specifically have many reasons
to believe that individual words that we
might be passing to these models don't
seem to behave like discrete symbols
exactly so let's just consider an
example the word face we think about the
word face we can find it in many
different contexts in language so in the
sentence did you see the look on her
face we could see the clock face from
below it could be time to face his
demons or there are a few new faces in
the office today and those we will as we
think about those uses of the word face
and we get some sense that they are
different in meaning or different in
usage and we call these differences word
senses but the important thing to note
about the different senses of the word
face is that they're not entirely
different so we it's not the case that
we should model these --is entirely
independent symbols which we would like
to past or model in actual fact what we
find when we when we delve into the
meaning is that these meanings have
certain aspects in common but they're
just not identical so if we think about
the first case of face we might think of
that as the most prototypical meaning
and of course that's just the face that
you can see in front of you now the face
which is the most important side of
somebody's head of sight for the eyes
and the nose so yeah as well as being
the most important side of some somebody
it's something that represents them and
it's something which is used to inform
or communicate other people so if we
think of all of those are sort of
features or properties of this sense of
face and then when we think about the
sense the clock face we see that it
actually shares
some of those properties but not all of
so it's also the most important side of
the clock clock face and it's also the
side that's used to communicate or
inform others by conveying some
information and then if we think about
this notion of faces of herb to face
your demons again there's this idea that
when you face somebody you point your
little face directly at them so it
conveys this same sense of the Poynting
aspect of face that's also conveyed in
the core prototype and then finally if
we think about the idea of new faces in
the office today
then it conveys this sense of identity
or self which is also potentially shared
by the core meaning of face so this
example shows and you will see these
effects if you look at many other words
but rather than discrete word sentences
which are orthogonal to each other we
might be better off modeling this
discrepancy in meaning within individual
words as operations that can interact
then but are not necessarily the same
okay now when we think about the fact
that each word could have many of these
different senses how could a process
have possibly make sense of a sentence
how could it possibly disambiguate the
possibilities to the different senses in
order to come to one coherent
understanding for the friend well one of
the ways in which we do that is of
course by using additional information
separate from a particular word so we
use the wider context and to give just
an explore example of how our language
processing really depends on context
consider this example it's actually goes
back to rumelhart in the mid to late 70s
so he noticed that if we had some
handwriting like this Jack and Jill went
up the hill we can read it very quickly
and in the bottom case the pole-vault
was the last event we can read back just
as easily and but if you look at the
areas highlighted in red here you'll see
that the there's actually a character
which is identical in both cases and
it's arguably midway between an e v and
a double
but when we read the sentence seamlessly
we just interpret this character which
could potentially be ambiguous in the
way which fits most naturally with the
whole of the wider sentence around it so
this sort of example intuitively gives
us some justification to thinking well
maybe it's the interactions between the
individual tokens that we're looking at
and all of the things around them which
actually allow us to solve the mystery
of which possible sort of sense of a
word we might be looking at in one time
and in particular what we probably do
according to this example at least is
think about the whole sentence and think
what's the most likely interpretation of
the whole sentence and that in itself
informs the individual interpretation of
the particular characters where
ambiguities might be another classic
example of this phenomenon can be simply
gained by reading the following symbol
on your screen the following image by
reading it across or down so obviously
as we read it down the character at the
very center of the image looks very much
like a 13 but as we read it across it
looks fairly like a B and this tells us
the extent to it even in our very early
perceptual processes the context is
informing the ways in which we map what
we're seeing into things further inside
our processor which might be our
memories of existing symbols like 13 or
B in this case so we've seen then that
words are not necessarily best modeled
as discrete symbols and we've also seen
that
in order to decisions that naturally fit
between these word like things we better
off be considering wider context in
order to modulate those computations
another very important fact about
language is that the important
interactions which we may well need to
model can very often be non-local so it
can be things that are not very close
together which we have to capture the
interactions between classic example is
sentences a bit like this the man who
ate the pepper sneezed so even though
the pepper sneezed those that part of
the sentence is contiguous we as we read
this sentence know that it's in fact the
man who sees we know that this sort of
image characterizes what happened when
we read this sentence and that there
isn't necessarily anything to do with
pepper to be seen part of the fact that
that's something that the man just ate
so this tells us that it can be things
at one end of a sentence and clings at
the very other end of the sentence which
must be considered to interact in order
for us to form the most satisfactory
meaning when we read sequences of words
however there are of course other
factors at play
so consider the sentence the cat who bit
the dog Bart now it's actually the case
the people are much slower to make sense
of this sentence than the man who ate
the pepper sneeze even though they have
exactly the same overall structure and
legs eventually upon thinking about it
we do realize that in the sentence the
cat who bit the dog barks it's actually
unusually the cat which does the barking
but our difficulty to process this also
tells us that many factors are at play
so in particular it seems to be that the
the three word phrase the dog barked
seems to capture our attention and we
sort of have an urge to consider that
it's actually the dog barking in a way
that's more strong than in the other
case where we don't have a such a strong
urge to consider that it's the pepper
that sneeze now where might those urges
come from and can we capture those in
our compensation
of models well these sorts of examples
seem to tell us that those odors can
come from our underlying understanding
of the world our understanding of the
meaning of dog and barking and the fact
that those are very likely to come
together and fall and describe a
particular situation whereas our
knowledge of Pepper's will tell us that
they don't typically sneeze and
therefore we don't think that the pepper
sneeze is very likely state of affairs
and we look for other ways to make sense
of the sentence and the correct way of
making center the sense of the sentence
in fact is more salient to us as we
process it so that's just a thought to
bear in moment we're thinking about
optimal processes of language and deep
learning models and then there's another
final point so lots of people who
consider and talk about language
particularly in the wider machine
learning community and consider language
to be compositional in the sense that
the meaning can be computed simply by
elegant operations on the individual
parts but when we actually consider how
meanings combine the picture is a little
bit less clear and it seems very likely
that whatever we do to combine meanings
very well ought to take into account
exactly what those meanings are and it
shouldn't operate arbitrarily on any
different set of inputs it should be a
function which really takes into account
the individual meanings in a particular
scenario before deciding the best way to
combine those me just to justify that
consider the following example here's a
characteristic image of something that's
red but if you look at red why none of
us would find that unusual but of course
the color of that wine is much darker it
could even be black in that particular
image and here's a red hand our
experience in pens tells us that even
red pens needn't be at all red when we
look at them from afar it's only the ink
that comes out of them that needs to be
read so even in something as simple as
combining a color adjective with a noun
there's all sorts of factors at play
telling us exactly
how those needles combined that don't
seem to be equivalent from one pair of
words to the next things get even more
wacky in certain cases so there's a
classic example about pets if we think
about a prototypical pet it's probably
black or white or brown because
obviously dogs and cats have those sort
of colors if you think about fish then
maybe in classical fish we think about
with the silver or gray slickery in that
way and when we think about pet fish
this sort of magic seems to happen where
our typical pet fish has lots of bright
colors it could be orange green or
purple yellow so something seems to have
happened in our mind to allow these
strong features to come into the
representation of pet fish which didn't
play a strong role in our
representations of either pet or fish
this doesn't always happen when we
combine words but it does sometimes
another example would be this
representation of plant which might be
typically looks something like that and
our representation of carnivore which
might be a bit like that but our
representation of carnivorous plant has
this additional feature about eating
insects so these are kind of wacky
effects of how meanings interact when
two words come together and it's not
necessarily easy to explain them in a
model which treated every pair of words
fed into that model with exactly the
same function to combine their meanings
it very much seems to me that what's
instead happening is that whatever
function is combining the meanings is
taking into account the individual
meanings of the components going into
that function and in additional
additionally that function may well need
to take into account our wider knowledge
of typical things we might encounter in
the world and how their properties might
fit together under the constraints of
the world as we know it so just to
summarize we've seen in this section
that words have many related senses and
that they're not necessarily
characterized
as two perfectly idealized discrete
symbols we've also seen that in order to
somehow find which of those senses is
most relevant in a particular scenario
many of some of the ways to settle that
problem might require us to look at the
wider context around that word and in
many cases we may need to look a long
way from a particular word
satisfactorily disambiguate the
uncertainty that we have at any
particular point and finally when we're
thinking about building models of how
word meanings might combine we've seen
that functions that combine meanings
will probably need to take into account
what the inputs to those functions are
in order to come up with the best
bespoke way of combining for those
particular words and we've even
suggested that they may well also need a
widened sense of how the world works and
how things can naturally fit together in
order to eventually arrive at the
optimal representation for the
combination of meanings in each
particular case so in the first part we
talked about particular aspects of
language and particular aspects of
neural computation that have seemed to
fit together in a particularly
appropriate way such that they define
certain ways in which a computational
model might need to behave in order to
capture the ways that meaning works in
language so in this section we're going
to talk much more concretely about a
specific model which was published just
a couple of years ago and has had an
incredible impact on a large number of
natural language processing problems
from machine translation to sentence
classification and essentially any
problem that requires a model to process
a sentence or a passage of multiple
sentences and compute some sort of
behavioral prediction based on that so
it's fair to say that for any of any
problem of that form transformer is
probably the state-of-the-art method or
some variant of a transformer is the
best way to for the model to learn
and to learn to extract the signal from
those sentences in order to make optimal
protections and in this section I'll
talk about the details of the
transformer and just refer back to those
aspects of language processing that we
saw in the first section in order to
give some intuition about why the
transformer might be so effective when
it processes language so just here's
credit to the authors of the transformer
from Google brain and collaborators and
the paper is obviously available for you
to find out fine details that are given
broad overview and starting in
particular with the first layer so the
transformer contains a distributed
representation of words in its first
layer which is something it has in
common with almost any euro language
models now what do I mean by a
distributed representation of words well
the first thing that we do when we
construct a neural language model is we
have to determine what is the vocabulary
on which the model is going to operate
so what I mean by that is we do need to
chop up the process of the input which
the model sees into some sort of units
in order to pass them to the model now
if you think about a large page of text
those units could be individual
characters in the extreme case they
could be individual pixels if we
consider the the text an actual image
but in general with language processing
applications because we have text all
stored in digital form we don't need to
go through that focus and subject to our
models having to learn to process pixels
so we have to make a decision about what
in what are the units that we actually
costs the model and in most applications
of neural language models these days
that can either be character level which
is where we pass each unit as an
individual letter or it can be word
level which is where we split the input
according to white space in the text and
then we pass each of the individual
words the model as discrete different
symbols but of course as we've talked
about in the last section and a model of
which just takes symbols and
treats them as symbols might not be
optimal for capturing all of the aspects
of meaning that we see in natural
language so instead of doing that the
developers of neural language models
have come up with a procedure which
allows the model to be more flexible in
which we represent in the ways in which
it represents words and that process is
something like the following so let's
say we do take the decision to chop up
our input text according to individual
words so what we should what we first do
is we consider all of the words that we
want our model to be aware of and we
lend that to find the boat the total
boat cabin with the water so to get such
a list we might scan hundreds of
thousands of pages of text and count all
the words that we find there and then we
can take some subset of the words which
appear the most frequently or
alternatively if we have lots of memory
and a really big model we can take all
of the words and allow all of those to
be in the vocabulary of what we
typically do in a neural language model
then is pass each of the words to an
input layer and that input there
contains a particular unit for each
corresponding to each of the words in
the vocabulary of the model but
importantly those units are then
connected to a set of ways and as always
each unit is connected to the same
number of weights and those weights
connect to a set of units of a
particular dimension now that dimension
we can think of as the word
representation dimension or the word
embedding dimension and when the model
sees a given word we turn on the unit
corresponding to that word and we leave
all of the other units is zero so we put
an activation of one on unit
corresponding to the word leave all of
the other weights is zero
and we've mark those weights in this
diagram here with yellow and light blue
shows the space occupied by the whole
layer of input base for the bottle so in
this case the model sees the word V we
turn on the weight corresponding to the
word V and of course because we activate
the unit with strength of one and we
activate each of the units at the output
of the next layer which is Korres
according to this black box around the
grave rectangle and we activate each of
the units there according to exactly
what the weight is that went from the
word the unit corresponding to d2 this
distributed layer so effectively we get
a representation in that in that layer
with the black box around the rectangle
we get a representation corresponding to
the word B but that representation is
actually a finite number of weights
floating point value Thwaites and if we
do this for all the words we get a
different representation for all of the
words so we can unroll the info and
actually do repeatedly do this and get a
sequence of vectors of floating point
values for each of the words in our
input and those vectors live in a space
and importantly that space has certain
geometric properties so we might find
that it representing words in a space
like that allows words to move together
in the space if it's useful for the
model to represent them as somewhat
similar and to move further away in that
space if it's useful for the model to
represent them as different because
remember with backpropagation all of the
weights in this first layer of the model
are going to be trained to optimize the
model to achieve its objective
so this gives the model the flexibility
to move its representation of individual
words around as it sees fit and the best
way to achieve its objective so just to
recap this is the first layer of many
neural language models including the
transformer and it contains quite a lot
of weights so if we have a total of
capital V words in our vocabulary and if
capital D is the dimension of the vector
that we're going to represent each of
these words with in a floating point
vector then the total number of weights
that we have in the first layer is V
multiplied by D and we end up with a
d-dimensional euclidean space with which
to represent these input units in the
model now this idea of representing
words or letters or whatever we take is
the input units to a model in some sort
of high dimensional floating value real
vector space is actually quite an old
idea if we go back to 1991 Mickie Lydon
and daya produced a language a neural
language model with much less
computational power than current models
have but it's still trying to execute
this principle of representing input
words in this distributed geometric
space and it was able to exhibit certain
types of interesting generalization when
trained on real text that a model which
represented words as individual discrete
symbols wouldn't be able to represent or
achieve and of course perhaps the most
famous example of this demonstration
came from a very famous paper in which
Jeff Hellman introduced the recurrent
neural network to the wider community
and in this paper
Elmen analyzed the distributive
representations corresponding to lots of
different words as he trained the model
on sequences of sort of
subject-verb-object star sequences of
natural language starts minutes the
objective of this model was just to
represent a sequence of words such that
the model was able to optimally predict
the next word with as much accuracy as
possible and what Ullman found when he
had analyzed the way that the model was
district was representing these words
internally was that of all the words in
his vocabulary they started to cluster
together in this geometric space such
that words with similar meanings cater
and importantly also words with similar
syntactic rods so things like verbs or
nouns subjects or objects also started
to clasp together in the specs this
tells us that neural language models as
they experience more and more text start
to slowly infer the underlying
structures in language which we might be
able to perceive as language users such
as subject object and how things fit
together like that as well as an
emerging categorical semantic structure
where we see that certain classes of
different types of words naturally fit
together so that's the solid foundation
on which the transformer builds but
that's of course not normal to the
transformer distributive representations
of words
been a part of neural language models as
I pointed out since the early nineties
so what else does the transformer do
that makes it so powerful and allows it
to fit and correspond and capture some
of the aspects of language that I talked
about from the first session well after
the first stage of processing which I've
just outlined in the previous slides we
end up with a particular real valued
continuous vector for each of the words
in an imminent instance so the next
stage the transformer computes what's
called a self attention operation so how
does that work well for any self
attention operation there are three
matrices containing the weights which
parameterize the operation so the first
matrix is we could call the query weight
matrix WQ the second matrix we will call
the key weight matrix WK and the third
weight matrix will call the value weight
matrix WV and each of these weight
matrices have independent ways in the
transformer and we can then their
dimensions are such that they can
naturally multiply in this case I've
written it as post multiplication of the
distributed word vector that I talked
about in the first section and
importantly as the self attention
operation is carried out these weights
are applied equally and in exactly the
same way to each of the words in the
input so we end up with for every
individual word vector I've written here
a Beatle corresponding to the word
Beatle in the input we end up with three
further vectors corresponding to
multiplying that vector by the matrix WQ
the matrix WK in the matrix WV so those
three additional vectors we can cool old
q old k and bold V and we can call those
or they are typically called the query
that vector the key vector and the value
vector for this self attention operation
corresponding to each of the words and
then with those three vectors we use
them to understand how the different
words in the input start to interact so
in particular with the query vector we
produce an Eaton operation worth
every word we take the query vector
corresponding to that word and we
compute the inner product the dot
product of that word with the value with
the with the key vector corresponding to
each of the other words so that's
represented here by the dotted line and
by taking a dot product in that way we
get a scalar and then we can we want to
understand how big is that scalar
relative to an average scalar that we
would get if we just took that operation
arbitrarily so essentially we want to
give the model to the power to represent
how strong should the connection between
these two words be and in order for that
to be a nice normalized distribution
over all the possible strengths computed
by the model we first work out the inner
product of the query value with the key
value of the particular word and then we
divide that number by the dot product of
the book we need to normalize by a
quantity corresponding to the dot
product of that query vector with each
of the key lengths of the other words
and the way we do that
compute those values and we passed all
of those values of the dot products
through a softmax layer which gives us a
distribution so it normalizes for the
exponentiate and normalizes such that we
get a nice smooth distribution
corresponding to how well each of the
queries corresponds to each of the keys
of the words in the input so this thing
gives us a set of weights corresponding
distribution which gives us a set of
weights between 0 and 1 and so for a
given work beetle we get a set of
weights one for each of the words in the
input telling us to what extent is there
a strong interaction between the work
beetle and that other word so in this
case the way I've marked it in the slide
is that the strongest interaction when
we do this operation is with the word
drove and that might be because the word
drove tells us in particular that this
beetle is not the animal type of beetle
but it should impact them sort of as the
cars that beat so that's the sort of
interaction that we want to naturally
capture it
once we've got as these these ways we
then use them to tell us how much of the
value representation to take through to
the next layer of the transformer so in
this case for example when representing
the word beetle we would notice a strong
connection with the word throne and that
would give us a strong way in our
attention distribution and then that
would tell us to take a lot of the value
of the bedding for drove through to the
next layer of the transformer so the
operation which allows us to take an
amount of the value through to the next
layer which corresponds to the weight
computed by the transformer is just this
simple weighted sum so what we end up
with then for each word like beetle is
that we take a small amount of the value
of each of the other words plus some of
the value of the word beetle through to
the next layer of the transformer and
that can then be aggregated to form the
next layers representation of the word
beetle so notice that having performed
this transformer layer we
and reduce the number of embeddings in
the model in any way we we still have a
representation corresponding to the work
beetle that we started with
but that representation has been updated
or modulated conditioned exactly on
information about how well it
corresponds or how well it is should
interact with all of the other words in
the input and of course that was just
for the work beetle but we do that for
each of the other words in turn and that
computation can be computed in parallel
which makes the transformer quite fast
to feed forward in today's deep learning
libraries and so for one application of
a self attention layer we end up with
the same number of distributed
representations coming out as we had
going in and within the mechanism the
only ways that we learn are those single
matrix giving us the queries and the
second matrix giving us the keys and a
third matrix which gives us the value
representations of course those matrices
are then applied to each of the
individual wards but it's not just this
self attention load that gives the
transformer its expressive of express
ability and power there's actually an
operation known as a multi head self
attention which basically takes the
operation I just talked about and
reapplies it four different times in
parallel so if you imagine the operation
that I just spoke about being
parameterised by three matrices WQ WK +
WV well we can repeat that process with
three additional independent matrices
and in fact typically we might do it say
four times so we'll end up with four
sets of three independent matrices and
each of them can do exactly the same
self attention operation as I just
talked about in in the previous slides
so we end up with four independent and
parallelizable self attention operations
each computed on the individually input
word a particular layer in order to get
us through to the next level now of
course that is a lot of computation and
it might require a lot of memory if we
end up with very large representations
in practice then what the developers of
the transformer recommend is that each
of our self attention layers actually
effectively reduces the dimensionality
of the envelope vectors so if the input
vector in the light blue at the bottom
of the slide here has dimension 100 then
we can make the matrix W V be a
rectangular matrix rather than square
matrix what that would do is mean that
the output of W V which is the value
vector which gets passed the next layer
of the transformer that can be
arbitrarily small in this case we might
find it to be just 25 units and so each
self attention layer independently takes
a hundred dimensional vector and returns
25 dimensional vector to each unit for
each word in our input but if we do that
four times then we end up with four
twenty five dimensional vectors and
those can be aggregated in fact in the
transformer that passed through an
additional linear layer which is
parameterize by a matrix w0 but then
concatenated to return overall a vector
of the same dimension 100 units as was
the dimensionality of the input so in
that way we can apply multi-head self
attention we can give the model for
independent ways to analyze the
interactions across the different words
indium Hertz and we can do so without
expanding the dimensionality with which
the motor needs to represent each of its
words and that makes it relatively
practical tool which doesn't lead to an
enormous explosion in the memory
requirements of the models but it does
give them a lot of many independent ways
with which it can represent interactions
between the words in the input
now after that multi-hit self attention
layer the model that does what's called
a feed-forward so conceptually this is
less interesting but essentially the
representations at the output of the
multi-head self attention layer are then
multiplied again by a linear layer
there's a rectified linear unit not
and then they're actually expanded out
in dimension some work and then reduced
again in dimension with another so when
considering a transformer altogether
it's actually multiple applications of
those multi-head self attention layers
and the linear layers that I've
described afterwards but there's another
important detail in the transformer
which is the notion of skip connections
so whenever we apply a multi-hit self
attention layer or indeed linear layer
the transformer also into the model the
option to ignore that computation and
instead to pass the activations that
were at the input to that multi-head
self attention directed bypass the self
ascension there and go through to the
point in the network at which the output
is coming out of that sub potential and
then that is added to the output of the
self attention there passed through a
layer normalization there and then that
represents the actual output of the
whole unit that the whole part of the
network whole module which is doing the
multi-head self attention so why might
that sort of skip connect should be
important well in the examples I gave in
the previous section about language one
thing that should have maybe come across
is the importance of the role of our
expectations in forming a consistent
representation of what particular input
it so as an example in the case of pet
fish we came to the understanding that
pet fishes have many bright colors even
though that was not necessarily part of
the the individual parts of the input
it's not necessarily something we would
associate with pets and it's not
something we necessarily associate with
finished
ordinary and where does that additional
notion of colors come from well it
probably comes from our sort of our
wider understanding of the world and our
ability to think about pet fish as a
combination and then reconsider
how the infobox and so these sorts of
top-down influences are expectations
influencing how we actually combine the
inputs in language a really common in
many different contexts and if you think
about skip connections it's not a
perfect model of this but it does give
you the transformer a rudimentary
ability to allow its representations of
things at a higher level of processing
to interact with these representations
of things at a lower level of presence
so let's say that the model didn't have
skip connections and fed things through
to a certain level in the hierarchy at
that point after computing many
different interactions the model might
form a consistent sense of the fact that
a meaning needs to be understood in a
particular way but of course those
top-down influences tell us that that
expectation of what the meaning might be
should actually feedback and allow us to
remodulate how we understand the input
well a skip connection which comes up at
the input and interacts with the model
at that point can can actually compute
such an interaction in subsequent layers
because at that point the model has
access to both a high level
representation of what it expects the
best way of interpreting the whole
situation is and it has direct access to
the lower level input so in some ways in
a very deep model the addition of skip
connections allows the model to execute
a form of top-down influence on
processing there's one more detail I'll
finish off with our characterization of
the transformer now if you were aware if
you were paying attention during the
explanation you may would have noticed
that none of the operations that I
described on the input words took into
account the answer relative order of the
work words in the input it was a series
of matrix multiplications which were
quite identical to each of the words and
then on top of that series of inner
products which are symmetric operations
which don't favor and the ordering in
which we apply them with respect to the
words so there was no way that a model
like this would have any what
to express the fact that certain words
appear closer together in the input for
certain words appear further apart and
of course we know in language that the
word order can tell us some of the
important things about what an integral
sentence means so in order to give the
model of sensitivity to word order in a
way that the computational form the
functional form of the model doesn't
allow the developers of the transformer
came up with a rather nice trick known
as positional encoding so positional
encoding is just a way of determining a
set of scalar constants which are added
to the word embedding vector after say
let's say in the lowest level of the
transformer it can be added before the
self into the first self attention layer
but just after the wording living and
those scale is combined with the word in
writing to mean that whatever when if a
word appears in a particular slide in
the info regarding the fact that it's
embedding weights will necessarily be
the same the actual effective
representation that the transformer sees
will be slightly different depending on
where it appears in the info so to
achieve this sort of thing you just need
the model just needs a set of small
scalars which are different in each of
these possible locations that were
appearing in the input and they use a
nice sinusoidal function which has
various properties which which may well
be more desirable and just being
allowing the word to discriminate words
according to their position because in
fact that sinusoidal function gives the
model a slight prior to pay attention to
relationships of a certain wave length a
certain distance across the input and
each particular unit in the embedding
representation can then specialize at
recognizing interactions or
correspondences at a different distance
from a given word so and unlike if you
think about models like a recurrent
neural network or an LS TM those models
have the notion
order built-in because these process
input sequentially one word after the
other according to a process
transitioning the state from the from
its position after reading one word to
after reading two words to after reading
three words to are three to four words
what that what that means is that the
model has a very strong awareness of the
ordering of the words naturally but then
it's harder for that model to remember
to pay attention to things a long time
in the past even if those things
actually end up having a really
important influence on why currently
looking at map with the transformer
things are totally different the model
has sort of natively in its native
functional form it has no awareness of
and we have to add on these additional
positional encodings to give the model a
weak awareness of world but the
transformer actually performs better
than our intents no STM's of a lot of
language tasks and this maybe tells us
that it's easier to learn the word the
or the notion of word order for the few
cases or for the number of cases where
it's actually important in language then
it is to be given the notion of word
order automatically but to have to learn
the very difficult process of paying
attention to things a long time at the
parts and when I say difficult I mean
the gradients have to pass back from
through many many weight matrices in
order to determine in order to allow the
model to update to and then learn to
encode dependencies between things in
the past and things in the present with
a transformer that passed that the
gradient has to go through is much
shorter because there's no prior
favoring of things which are close
together instead of the gradient path
that the model needs to go through to
connect any two words in the unit is
equivalent and in fact it is indeed is
shorter on average than it is in
recurrent neural networks so that gives
a small amount of intuition about
another reason why the transformer might
be so effective a process England so
just to summarize this section we saw in
the previous section the words shouldn't
necessarily be sort of as independent
discreet
symbols and the disambiguating their
meaning can depend a lot on the context
but not only on the immediate context
which is closest to those words but on
potentially distant context of the
information encoded in words a long way
over we also see that functions which
models use in order to combine the
meaning of two words should take into
account the meaning of those words and
are and if possible take into account a
wider general knowledge of how things
typically combine in order to allow to
allow that to modulate the interactions
between the words coming in and we think
about the architectural components I
talked about in the transformer the
multi-head processing is one way of
getting at this notion that words are
not discrete symbols because it
naturally gives the transformer even in
one feed single feed forward pass the
opportunity to represent each word at
each layer with n let's say four plus
different possible contextualized
representations and of course going back
longer term just the general notion of
representing words as distributed
representations and allowing words with
similar meanings to occupy local areas
in a large to your metric vector space
also allows the model to express its non
discrete nature of word meaning a very
elegant way now the fact that
distribution depends on context is very
nicely modeled by self attention
precisely gives the meaning of every
word to be critically dependent on the
meaning of all the other words in a
given input stream and the fact that
that context can be non-local as I've
just talked about is very nicely modeled
by the self attention mechanism because
the gradient flow from the past from
that from the particular point I mean a
sentence to any other point in the
sentence is the same so interactions but
we're not particularly favored over wine
our interactions another fact is that
the more layers you have the more chance
the model has to learn as a moderate
such representation of different things
how interactions might
place at different levels of abstraction
as the model goes continues to reprocess
model the input and finally on this
point about how meanings combine and the
fact that the meaning the ways in which
the meanings of two words combine seems
to often depend on the particular
meaning of those words
and also top-down effects we've seen the
skip connections are one way in which
the transformer can learn to implement
the interaction of higher-level
information that lower information and
we've also again seen that parameterised
functions on distributed representations
ie the multiplication of a matrix by a
vector is precisely the operation of a
function which combines word meanings
according to the meanings of the words
themselves and those operations are
common in most newer and many newer
language models but are a really
important part of the transformer
architecture
so hopefully this section is giving you
some intuition about how a transformer
works but also summing tuition about
maybe why it works why it is that the
various components in the transformer
improve on a models ability to process
language because of the way that we
think
meaning works in a very sort of
intricate and interactive way when we
understand linguistic input in the last
section we introduced the transformer
and we talked about how various
components within the transformer
combine to make it a very powerful
process a very powerful model for
processing sentences and combinations or
sequences of words in this section I'm
going to talk a little bit about the
very specific use of the transformer
it's a way of training transformer
models in order to allow them to excel
at a wide range of different language
tasks and those tasks might involve
reading a sentence and making a
prediction or classifying how two
sentences relate to each other or even
sighing or making predictions about
longer texts such as documents but
before I do that I also just want to go
back to our points about the nature of
language and discuss one more issue
which i think is quite motivating when
we think about how transformers are
applied in the model that I'm going to
talk about in this session so let's
consider this sentence time flies like
an arrow and then we can compare it to
what seems superficially to be a very
similar sentence
fruit flies like a banana but of course
when we start to the process and make
sense of those sentences it feels very
clear to us as native english-speakers
that there's quite a difference in the
prin the way that the words in those
sentences have to relate to each other
in order for us to sort of construct the
meanings it and it felt it certainly
seems to me like there's at least two
factors that are really important so one
thing is that we in the top sentence
time flies like an arrow we know what an
arrow is we know that they regularly fly
and in fact we know how they fly so
we've got our experience of arrows
another important piece of experience
that we have is our experience of bunch
of phrases or sentences which look like
similar to the phrase time flies like an
arrow and in particular they're similar
in the way that the meanings of the
words combine for us to come up with
representations our sentence so those
could be things like John works like a
Trojan well the trains run like
clockwork these are all actually kind of
metaphorical or simile star sentences
where we can where we compare the way
that something works with the way that
something else works so it feels to me
like those two pieces of experience are
very important in our ability to read a
sentence like time flies like an arrow
and immediately understand it and in the
case of fruit flies like a banana of
course we come to quite a different
understanding right we know that and
we're not comparing the rail flew fruit
flies with the way they've been on us
fly
and how is it that we can somehow know
that that's not what we have to do to
understand this census instead what we
do is we have some knowledge of fruit
flies and we have we know that in fact
that one of them may be one of the most
salient things about fruit flies I'm not
an expert on fruit flies but there's one
thing I do know which is that they like
fruit and we know the bananas are fruit
and so this connection helps to tell us
what may be the fighting time it's a
different type of liking but I need to
be thinking about in this sentence and
then of course there's again other than
that background knowledge of how the
world works have fruit flies are there's
also this kind of more linguistic
knowledge of sentences we may well have
already previously understood which in
which the meaning seems to combine in a
similar way to fruit flies like a banana
so Friday likes having his tummy rubbed
or grandma likes a good cuppa
in those cases it seems like the process
of putting together the meanings has
something quite similar or in common
with the scenario in fruit flies looking
on so if we're going to come up with a
general language understanding engine
that's able to cope with all these
different types of processes within
constraints which are involved in
understanding a sentence then there's
obviously a lot of places where such a
model needs to get its experience and a
lot of places and where such a model
needs to get its understanding of the
world in its understanding of language
and those considerations lead us to add
a fifth point to these many
characteristics of language which is
that when we actually form an understand
you know really is that it really does
seem to be a process of balancing our
existing knowledge and that could be
knowledge of language knowledge of the
world with the input with the particular
facts thing that's current coming into
the and that consideration is a key
motivating factor per took by behind the
approach which is taken in this model
I'm going to describe in this section
which is called bert bert stands for
bi-directional encoder representations
with transformers and bert is
essentially an application of the
transformer architecture that i
described in the last section but the
key insight with that is the
rather than training a transformer just
to understand the inputs to sentences
which the model is currently considering
the process of pre training takes place
in which the weights within the model
are endowed with knowledge of a much
wider range of text in this case which
can plausibly give the model that
background knowledge which is really
necessary for forming a coherent
understanding of the totems of different
types of sentences that a language
understanding processor needs to be able
to understand so the important thing to
remember when considering how bert works
is that a transformer as described in
the previous section really is just a
mapping from a set of distributed word
representations to another set of
distributed word representations so as i
talked about in the last section the
first layer of a transformer goes from
the in particular input symbols passed
in the model to space geometric space
continuum value vectors and of course
what comes out after these many layers
of self attention is precisely another
space of continuous valued vectors
Cortlandt corresponding to each of the
words in the input that the exact same
set of words that are the input so if I
pass a sentence to transform a model
you'll very quickly compute the set of
embeddings for those for each of the
words in that sentence and then it will
output it will pass them to yourself
attention models an output set of
embeddings for each of the words in that
sentence but of course each of those
embeddings would be highly
contextualized highly modulated by all
the other words in the sentence and
hopefully will have gone through the
source of processing needed for Ostin
for the model to sort of gradually and
incrementally for a a reasonable
representation of what sentence means so
given that fact that a transformer is
just a mapping from a set of work
representations to a modified set of
word representations of the same length
the
quite an easy way in which we could
train such a model in order to extract
knowledge from an enormous amount of
text that we might just have a look
around so in particular the insight and
burs is precisely how can we get
knowledge into the weights of such a
model without requiring problems or data
which has been labeled by human experts
or other some other mechanism in order
to give the model sort of knowledge of
what's the right classification or
what's the right answer to make so how
can we get knowledge into a model a
transformer model in an unsupervised way
and the approach that the authors of
burr take is firstly by means of a
masked language model pre-training codes
so the way this works the following the
authors just considered the problem of
mapping a particular sequence of words
to the exact same sequence of words so
the job of this transforming theory is
just to represent a sentence for example
and then output a sentence at the very
top of its network but rather than
rather than having the model output the
exact same sentence instead in the input
to the model one of the words is masked
out so the model is not aware of one of
the words in the input sentence and
instead of having to predict all of the
words in the input sentence the model
just has to make a prediction
conditioned on the the output embedding
for the missing word of what that
missing word was so it just has to
answer the question you know here's a
sentence with a missing word in it
sucking up from words and the model just
has to make the prediction that the
missing word in that case is knowledge
and when training the model the authors
of Berk do that with 15 percent of words
at random so ding grants entrances from
any any particular place where we might
be able to get a running test language
and the authors mask out words with a
probability of 15%
and then ask the model to make a
prediction and back propagate the the
cost which is essentially likely heard
the negative log likelihood the model
having predicted that word overall at
the other words in its vocabulary so
that's mass language model pre-training
but one thing that the authors noticed
is that if they trained the model in
that way and on the test set when they
came to use this model of course in the
input there wouldn't actually be any
tokens massed out so there's a risk that
just by training the model in this way
and it would not behave well on inputs
where there wasn't anything master so
for a small amount of the time instead
of masking our word they have the model
make a prediction of which word is
missing even though they didn't actually
mask a word so no words in this case the
model really does just need to retain
which word is in a particular point in
the sentence and at the output
representation corresponding to that for
addition on that make a prediction of
what that word was this of course if
this was just the only object if the
model would never have to do any sort of
inference it would never have to make
any sort of unexpected judgment about
what word could be missing it would
instead just be able to copy knowledge
straight through and that wouldn't lead
to any interesting formation of any huge
team representations so this is only
done occasionally but it does make the
model perform better on the test set
because the model does not kind of find
itself completely out of its training
experience when it encounters sentences
for which no words are masked out okay
so that's the masked language model
modeling objective but in order for bird
to be an effective language processor
the authors wanted it to also be able to
be aware of the flow of how meaning
works on a longer scale than just within
a particular sentence so in order to
achieve this they came up with an
additional mechanism for training the
weights in the bird model which is
complementary to the masked language
model objective so this objective can be
trained at the same time as the mask
language modeling objective
and as in that case it doesn't require
any data that's been labeled by experts
or fountains to have a right answer in
some way we can just construct this
objective by taking running text from
the internet and the way this works this
is called the next sentence prediction
pre training objective so the way this
works is the author's add an additional
input token at the start and it's the
output embedding corresponding to that
input location that's going to be used
to make the prediction on this objective
then as input to the model the model is
presented with not one sentence but two
sentences in this case and they say
there's the IMP additional input token
and there's the first sentence then a
separation token and in the second
sentence and that's all past of the
transformer and is processed through in
peril at the end the model produces
representations for corresponding to
each of the input tokens but it's only
the the initial representation
corresponding to this additional token
that was added to the inputs that needs
to be considered in this objective and
conditioned on that the model just makes
a binary choice of whether or not this
was actually two consecutive sentences
from the training Crocker's so in this
case it is two consecutive sentences Sid
went outside and it began to run so in
this case the model would predict yes
those are two sentences which are likely
to follow on another intercourse but by
shuffling data the trainers from the
people who train the model can also
present it with negative cases so cases
where one sentence didn't follow the
other sentence and so if that might look
something like Sid went outside
unfortunately it wasn't so the the
objective of the model here is to
identify this as two sentences which
don't fit well together and to make the
prediction no on the next next sentence
prediction tasks so by combining next
sentence prediction and mass language
modeling slowly the weights of this
large transformer the burn transformer
and gradually start to acquire knowledge
of how words interact in sentence
typically maybe abstract knowledge of
the typical ways in which meaning flows
through sentences and of course in there
the the spaces in which they represent
each of the individual words various
levels of the stack and things start to
happen like words that have similar
meanings start to come closer together
the model might be require might require
to separate them out into the different
parallel heads if words have various
different center senses and so a lot of
the general knowledge that we talked
about being very necessary for forming a
consistent and coherent representation
of loads of different language sentences
can start to be introduced into the ways
of this model as it trains according to
these unsupervised objectives so that's
the theory behind Bert or at least the
intuition behind that and of course
because neither of those training
objectives required any sort of
particular labels you can burn is
trainable on all of the text that exists
in digital form in English around the
world so you could take any test from
the internet and use it to train more
and more knowledge in theory into the
weights but of course that's in
principle helper works but it wouldn't
be a very convincing demonstration
unless there was some evaluation and in
this case the way that is then evaluated
is by taking its knowledge in all of
those ways and using that using that as
a start process to train on many
specific language understanding tasks
and these tasks typically how they do
use labelled data and they typically
have a lot less data so in order to
apply both these models the Bur ways
which are trained on all of the
unsupervised objectives then taken and
the data specific to each of these tasks
is passed through the Berk model and
then birth is the burnt weights are
updated according to the signal from the
supervised learning signal from these
but these actual specific language
understanding tasks typically this
process of fine-tuning the Berk
representations for a specific task
takes place separately and in
apparently for each of those additional
tasks and it's also necessary in in many
cases when fine-tuning in this way to
add an a little bit of machinery onto
the top of that because you know in the
standard bird architecture it's just
making predictions where it outputs a
number of distributed representations at
the top of this transformer model but of
course given a specific task it may be
necessary to come to some sort of
prediction depending on the output
format of the task it may be necessary
to take only some of those
representations and condition on them
with additional ways in order to make
that prediction but typically that's
only a small amount of additional waste
that contains task specific knowledge
and the vast majority of the model
contains the general knowledge that
trainings that that model so just doing
this massively improves the performance
of any models which aim to exhibit some
sort of general understanding of
language what I mean by that is when
model any model which is intending to be
trained on a wide range of different
tasks using the bird style approach to
transferring knowledge from an enormous
reading text corpora via fine-tuning to
those specific tasks leads to has led to
a really strong and significant
performance on a large number of these
hubs and importantly and this doesn't
just allow one model to solve lots of
tasks better in many cases this is the
way to achieve state-of-the-art
performance on these additional tasks so
even a model which was just specialized
to those additional supervised learning
tasks would not perform better than a
model that was initially pre trained on
Bert in fact in you know in for a lot of
these tasks performance is substantially
worse unless you apply bursts are pre
training on an enormous corpus before
transferring to these additional tasks
so this is a really comprehensive
compelling demonstration of transfer
learning and the key insight with birds
is that transfer which needs to take
place throughout the weights of a large
network previous attempts to do this
involved transfer just at the level of
those specific word in living ways which
which can encode the information
relevant to each individual word in my
vocabulary but they didn't have a
mechanism to encode the ways in which
those words combine now a few years
before birth model of a model called
Elmo and a couple of other models
started to show that there was some
promise in sharing more than just those
wording living weights but actually
sharing a large amount of functions
which learn to combine weights when when
pre-trained on some task agnostic
objective and transferred to specific
tasks and in the byrne model really took
that to the next level using the
machinery of the transformer to exhibit
really impressive transfer learning so
we've now acquired five interesting
principles of how a language and
meanings seem to interact when we
understand the sentence and we've added
this fifth one understanding is
balancing input with knowledge that
we've had already or our general
knowledge of the world and we've talked
about births as a mechanism for endowing
and models with something like a general
knowledge that may be necessary and
we've talked and we've shown that in
fact indeed is very important on a lot
of language understanding to us to have
this sort of prior knowledge acquired
from a massive range of different
experiences and different types of texts
so in the next section we'll look a bit
forward to other sources of information
which make laws only be useful for
different language understanding models
because of course that only has the
means to learn to acquire knowledge
through text whereas if you think about
the fruit flies are more time flying
like an arrow those sorts of examples
tell us that there are many other
sources of information that we may have
used in order to gain the general
conceptual or world knowledge required
to actually make sense of language so in
the last section we saw how the bird
model is a really exciting example of
transferring knowledge from an enormous
amount of text
to apply that knowledge to very specific
language tasks that maybe have a small
amount of data from which to learn and
this this works
in part because of the critical
importance of general knowledge in
understanding language and that not that
we need ways in which models can acquire
general principles of how language works
and how word meanings fit together in
order to make high-quality predictions
for a range of different language tasks
now in this section we're going to talk
about further ways in which we might be
able to endow models with general or
conceptual knowledge which they can then
apply to language related tasks and in
particular in a way that's not
accessible to the Bert model which is
the ability to extract knowledge general
knowledge and conceptual knowledge from
our surroundings which is something as
humans that we're doing all the time now
this is an opportune time to start
thinking about these challenges because
the tool was available for these sorts
of unsupervised knowledge extraction
processes improving all the time so as
well as the objective of maths language
modeling and next sentence prediction
that we saw with birds
there's also exciting techniques in the
field of computer vision that often
involve things like missing parts of an
image and making predictions about
whether or not that part of the image is
the correct part or which pixels with
most appropriately fit in to that part
of an image or maybe contrasting
incorrect parts of images with correct
parts of images and things like that so
those sorts of objectives are also
leading to really good ability to
transfer from large banks and images to
specific image classification tasks and
of course in the world of learning
havea bimbos often reinforcement
learning on those sorts of tasks are
techniques for having agents develop a
more robust understanding of their
surroundings and possibly and what was
known as a model of their world those
techniques are also improving so indeed
mine we thought it was the right time
given all of these improvements to start
to study this question of knowledge
acquisition it's free prediction in an
actual pager that can interact in its
surroundings but the idea of knowledge
acquisition through prediction is
actually a very old one in neuroscience
and psychology so it goes back all the
way to the time of Helmholtz and there's
some very influential papers you can see
in this slide but really proposed and
made clear the idea that predicting what
was about to happen to an agent or an
organism was a very powerful way of
extracting knowledge and structure about
the world that surrounds that heat now
in our case we unfortunately can't set
an enormous neural network free in the
world in which we live and just see if
it learns but the next best thing is to
create a simulated world and we did that
in the Unity game engine and the aim
with this work was to study precisely
whether or not an agent which moves
around this world can apply various
different algorithms in order to acquire
as much knowledge as possible from its
aura but in particular in a slight
difference to other work on this sort of
topic we were interested in whether or
not this knowledge would be language
relevant ie whether or not this
knowledge would be knowledge which could
serve the agents ability to understand
or use language and the way we did that
was as well as creating loads of random
rooms with different objects positioned
in different places in this simulation
we also created a bunch of questions
such that for any random room that was
created the aging could find in the
environment questions which could
plausibly be answered so examples of the
sorts of questions we asked to things
like
what is the color of table what is the
shade of the red object how many rubber
ducks are there in the room
is there a teddy bear somewhere and even
comparison questions like things like is
the number of rubber ducks bigger than
the number of toys so those are the
sorts of questions and importantly being
able to answer these questions requires
a particular type of knowledge that's
propositional knowledge the knowledge
the ability to tell whether something is
true or false in our environment that's
often contrasted especially by
philosophers with procedural knowledge
which is just the sort of instinctive
knowledge that may be a reinforcement
learning age of would naturally have in
when he learns to solve control problems
in a very fast and precise way so this
is a different type of problem most
typically faced by agents which are
trained with reinforcements so in order
to think about how we could develop
algorithms to aggregate this source of
knowledge as an agent explores its
surroundings we first just gave the
agent a policy which meant that it
visited all of the things in the room so
that essentially creates a video of
experience and then we set our learning
model the challenge of taking in that
experience and aggregating knowledge as
much as possible in the memory state of
the agent as it lives that experience
and then the way that we measure the
quality of that knowledge is by bolting
on a QA decoder on to the agent and
that's the part of the model which is
going to actually produce the answer to
the questions when fed with the current
memory state of the agent and the
particular question so as an example the
agent might explore a room
the yellow teddy bear and a red sheet
and a large table and then a small toy
dinosaur and a potato and the
environment might present the question
what is the toy that's under the table
now the agent would explore and the ages
letting algorithm can take in all of the
things it sees as it moves around the
room but the agent has has like a
it can't the agent itself can't see the
question so the agent just has to the
learning algorithm just has to find a
way based on that experience to
aggregate general knowledge into the
agent such that when the question the QA
decoder is cued with the state of the
agent at the end of the episode and cued
with the question it's possible to
combine those two pieces of knowledge
and answer with dinosaur so to do this
effectively the agent needs a large
amount of general knowledge about how
things are arranged in the environment
around them such that the QA decoder can
take that knowledge and make predictions
about the answers to questions it's very
important detail that we do not back
propagate from the answer to the
question back into the AG so the weights
in the agent and the objectives that the
agents applying must be general they
can't be specifically tailored to
getting the knowledge to answer the
question instead it must be a process of
aggregating throughout this episode such
that at the end of the episode the
agents memory is knowledgeable as
possible and then to this we applied
various different baseline so the
obvious basic gist of LS TM or any sort
of recurrent agent but more complicated
to apply transformers in this context
because of course we can't see the whole
episode at once as we're moving through
the world we can only see time steps up
to the current time step now another
approach is to endow the agent with
predictive learning objectives a little
bit like the source of mass language
model prediction that birds making but
when the agent has to given a certain
time point in the episode a predictive
loss the predictive engine and overshoot
engine takes the current memory state of
the agent at that time point and rolls
forward in time once the agent has
finally experienced the episode we can
then do some learning where we compare
the prediction of that predictive loss
to what the agent actually encounters
and importantly the predictive loss can
also take into account the action that
the a
chose to take it each of those time
steps so these are kind of action
conditional Overstreet unrolls where we
see what the agent actually encountered
in the future and then update the
weights of the agent such that they're
better able to make these sorts of
predictions and we tried two specific
algorithms so in one case in the SimCorp
algorithm which was proposed last year
the loss that's used in this predictive
mechanism is a generative model loss
which is modeling the each of the
individual pixels in the observations
that the agencies in the world in future
time steps and in the other predictive
objective and we use contrast and
predictive coding this is basically
asking the model to distinguish or maybe
presenting the model with two images at
a given time step and asking the model
to say which of those two is actually
the one the age of encounters the future
comparison as opposed to one which is
selected randomly from some other
episode now we can evaluate these sorts
of predicted mechanisms for aggregating
knowledge in the agent precisely by
their ability to create knowledge in the
memory state of the agents such that at
the final time step of every episode the
question answering the decoder can take
that knowledge answer the question what
we found surprisingly is that only one
of these predictive algorithms actually
led to the agent being able to
effectively answer questions and that
was the simple the model which uses a
generative model to estimate the
probability density of the pixels in
future observations of the model
conditioned on the memory state further
back in time so the contrast in
predictive algorithm was much less
effective at giving the agent the
general knowledge required to be able to
answer these questions the green line at
the top of the plot shows the
performance of the agent if you back
propagate from question answers back
into the actual agent memory so by doing
that you allow the agent to specialize
in a particular type of question for
every episode rather than requiring it
to build up knowledge and
but you can see that that makes the
agent much more effective at answering
these questions to give us some flavor
of exactly what the agent does here's a
video you can see the question is what
is the color of the pencil and you can
see that as the episode continues the
agents prediction gets more and more
confident that the answer is red such
that on the final times that red it is
by far the most probable answer if we
consider the other video that be just
today the other video you can see that a
similar thing happens with a different
type of question so here the question is
what is the acronym in green objects and
the answer is it's a grind as a sort of
pepper grinder and again the agents
confidence is very strong at yet but we
exhibit we observed these sorts of
effects only in the agent which was in
doubt with the simple model of
predicting the probabilities of pixels a
future observations conditioned on the
actions that it took an arbitrary of the
sheets into the future so that's just a
small insight into work that's going on
indeed mind where we starting to
consider how we can aggregate knowledge
from the general environment as well as
knowledge from large amounts of text
into a single model which can start to
in combine this sort of conceptual
understanding and general knowledge
understanding and our understand and a
really strong understanding of language
into a single agent which can come up
with a coherent and strong ability to
form the meaning of statements and
sentences and also to take that
knowledge to answer questions to produce
language and to enact policies labeling
it to do things in complex environments
so we've reached the end of the lecture
and I just thought we'd go back and
reflect a little bit quickly on the
various things that we've recovered so
we've talked about various aspects of
language which make neural networks and
deep networks particularly appropriate
models for a capturing the way the
meaning works
so in particular we raised the fact that
words are not discrete simple
but they actually almost always have
some sense of different related senses
that disambiguation is a huge part of
understanding language and then that can
critically dependent variable on context
we've talked about the facts that that
context can be non-local so
we're going to do with you the work
we're currently thinking about that
context can also be very non linguistic
and require can depend very much on what
we currently see doing and the notion of
composition we've reflected on the fact
that that in itself seems to vary
depending on what the words are being
combined with any one instance and we've
talked about the importance of
background knowledge and ways to combine
our ways to acquire them so one way that
we talked about was birth and
unsupervised learning from text and
another way was through predictive
objectives in a situated age and so if
we look at these features of these
aspects of language the mechanisms that
I've discussed today
cover them reasonably well and hopefully
they shed some light on why your
networks and interactive processing
architectures that have been the sort of
dis principles of neural computation and
distributed representations are
particularly effective for language
processing but of course it should be
said that there are many aspects of
language processing that the work I've
talked about just doesn't start to
approximate doesn't start to capture and
that and that's in particular around a
lot of the social aspects of language
understanding so our models are not
currently able to do things like
understanding the intentions of others
or reflecting on how language is used to
communicate and do things and you know
we need to make a lot more progress in
these areas if we're actually going to
arrive at agents which are truly able to
understand language so yeah just as a
final note I think it's interesting that
before deep learning really exhibited
its success on language processing
problems a typical view of language
understanding was what I called a
pipelined view which was that each
independent each part of processing
language from the letters to the words
the syntax into the meaning and then
eventually to some prediction could be
thought of relatively independently as a
separate process but now that we've
reflected on how language works and in
particular taken in all in the evidence
from the effectiveness of different
neural language model
processing tasks I think maybe this
isn't more effective or more realistic
schematic of how old language processing
should be thought so we may have some
stimuli some letters or sounds and we've
always got some sort of context around
notes that is or sounds those two things
input to our system but critically it's
that input combined with our general and
background knowledge of the world of
knowledge of language which together
allow us to arrive at some sort of
plausible meaning for everything that we
hear or everything that we might say so
on that we'll finish up
thanks very much for your time there's
some selected references here and many
other references which I didn't I don't
have time to list here but have been
hugely inspirational for the work that I
think that I've talked about today at
the end of popped a few therefore recent
work deepmind but again there's no not
time to list a huge amount very related
work so anyway I hope you've enjoyed
this lecture and it's given you some
insight into why language that language
understanding such an interesting
problem for computational models to try
and tackle and I hope that you've
enjoyed the talk and you'll enjoy the
following lectures on the D mind lecture
six
thank you very much
you


How Well Can GPT-4 See? And the 5 Upgrades That Are Next

we all saw that gpt4 is able to create a
website from handwriting on a napkin
with all the news since the focus on
Vision has been lost meanwhile in the
last few hours and days a select few
with full access to multimodal gpt4 have
been releasing snapshots of what it can
do I want to show you not only what is
imminent with gpt4 vision but with
releases this week in text to 3D text
inside 3D speech to text and even
embodiment we're gonna see how language
and visual model Innovations are
complementing each other and beginning
to snowball but let's start with images
do you remember from the gpt4 technical
report when the model was able to
manipulate when prompted a human into
solving captures for it well that may no
longer be needed it solves this one
pretty easily so no captures are not
going to slow down gpt4 next medical
imagery it was able to interpret this
complex image and spot elements of a
brain tumor now it did not spot the full
diagnosis but I want to point something
out this paper from openai was released
only a few days ago and it tested gpt4
on medical questions they found that
gpd4 can attain outstanding results
exceeding Human Performance levels and
that that was without Vision the images
and graphs were not passed to the model
and as you can see when the questions
did have media in them it brought down
gpd4's average it will be very
interesting to see GPT 4's results when
its multimodal capabilities are
accounted for next is humor and I'm not
showing these to say that they're
necessarily going to change the world
but it does demonstrate the raw
intellect of gpt4 to suss out why these
images are funny you have to have quite
a nuanced understanding of humanity
let's just say that it probably
understood this meme quicker than I did
quick thing to point out by the way it
won't do faces for pretty obvious
privacy reasons they won't allow the
model to recognize cases whether that
ability gets jailbreaked only time will
tell meanwhile it can read menus and
interpret the physical world which is an
amazing asset for visually impaired
people I want to move on to another
fascinating ability that the vision
model inside gpd4 possesses and that is
reading graphs and text from images its
ability to interpret complex diagrams
and captions is going to change the
world here it is understanding a complex
diagram and caption from the palm e
paper released only about three weeks
ago which I have done a video on by the
way but just how good is it at reading
text from an image well let's take a
look at gpt4's score on the text vqa
Benchmark now I've covered quite a few
of the other benchmarks in other videos
but I want to focus on this one here
notice how gpt4 got 78 which is better
than the previous state of the art model
which got 72 now try to remember that 78
figure what exactly is this testing you
ask well really text from complex images
this is the original text vqa academic
paper and you can see some of the sample
questions above to be honest if you want
to test your own eyesight you can try
them yourself so how does the average
human perform well on page seven we have
this table and we get this figure for
humans 85 you don't need me to tell you
that's just seven percent better than
gpt4 the thing is though these models
aren't slowing down as the vision
co-lead at openai put it scale is all
you need until everyone else realizes it
too but the point of this video is to
show you the improvements in one area
are starting to bleed into improvements
in other areas we already saw that an
image of bad handwriting could be
translated into a website as you can see
here even badly written natural language
can now be translated directly into code
in blender creating detailed 3D models
with fascinating physics the borders of
text image 3D and embodiment are
beginning to be broken down and of
course other companies are jumping into
here's Adobe showing how you can edit 3D
images using text and how long will it
really be before we go direct from text
to physical models all mediated through
natural language and it's not just about
creating 3D it's about interacting with
it through text notice how we can pick
out both text and higher level Concepts
like objects this dense 3D field was
captured using 2D images from a phone
this paper was released only 10 days ago
but notice how now we have a language
embedded inside the model we can search
and scan for more abstract Concepts like
yellow or even utensils or electricity
it's not perfect and for some reason it
really struggled with recognizing Ramen
but it does represent state-of-the-art
image into 3D interpreted through text
but what if you don't even want to type
you just want to use your voice just
three weeks ago I did a video on how
voice recognition will change everything
and I was talking about open ai's
whisper API but now we have conformer
which is better than whisper here is the
chart to prove it and look how conformer
makes fewer errors even than whisper at
recognizing speech the cool thing is you
can test it for yourself and the link is
in the description and while you're
passing by the description don't forget
to leave a like and a comment to let me
know if you've learned anything from
this video as you'd expect I tested it
myself and it did amazingly at
transcribing my recent video on gpt4
there were only a handful of mistakes in
a 12 minute transcript at this point
you're probably thinking what's next
well look at the roots sketched out two
years ago by Sam Altman he said in the
next five years computer programs that
can think will read legal documents and
give medical advice with gpt4 passing
the bar I would say so far he's two for
two he goes on in the next decade they
will do assembly line work and maybe
even become companions he's talking
about the physical embodiment of
language models back then openai had a
robotics team themselves that could do
things like this here is a robotic hand
solving a Rubik's Cube is despite
interruptions from a giraffe and someone
putting a pen to interrupt the model it
still solved the cube but then that team
got disbanded and it seems like they've
moved into investing in startups they
are leading a 23 million dollar
investment in 1X a startup developing a
human-like robot here is the One X
website and it features this rather
startling image and it says summer 2023
our newest Android iteration Neo will
explore how artificial intelligence can
take form in a human-like body now of
course for many of you a humanoid robot
won't be that surprising here is the
obligatory clip from Boston Dynamics
foreign
thank you
and of course these models don't have to
be humanoid here is a demonstration from
a paper published just four days ago
this is not just walking it's climbing
up balancing pressing and operating
buttons and before you think all of this
is really far away these assembly line
robots are now commercially available I
still think there's a long way to go
before embodiment becomes mainstream but
my point is this all these improvements
that we're seeing in text audio 3D and
embodiment they're starting to merge
into each other complement each other on
their own they're cool and a bit nerdy
but once they start synergizing fusing
together they could be revolutionary as
samuelman said on the Lex Friedman
podcast released yesterday embodiment
might not be needed for AGI but it's
coming anyway let me know what you think
in the comments and have a wonderful day