Deep Learning 7. Attention and Memory in Deep Learning

okay everyone my name is Alex graves I'm
a research scientist at beep mind which
should no longer say google deepmind but
only deep mine but I haven't updated my
slides obviously and what I'm going to
be talking about in this lecture is
attention and memory and how we use
these concepts in deep learning systems
so just to get started I think it should
be fairly you know clear to everyone
that attention and memory are important
features of human cognition and so yeah
you know the simplest level what do I
mean by attention I mean the ability to
focus on one things and one thing and
ignore others so to be selective about
what you're looking at or thinking about
or listening to or in some way giving
you know mental kind of bandwidth
towards so an obvious example of where
we we apply attention our day-to-day
lives as what's known as the the
cocktail party problem in a room like
this one was you know two minutes ago
where everyone was talking at once you
can still pick out one person talking
and ignore the others so it's pretty
clear we've got you know we're using
some form of at least sensory attention
on a day-to-day basis but then if you
take it a little bit further than that
you can say well it's not just about you
know being able to pick out one piece of
you know sensory information one piece
of vision or one piece of sound it's
also about pursuing one thought at a
time or remember in one event rather
than all events and so this to me
attention is kind of intimately bound up
with memory memory is something like a
tension over time so you know it's
something people often kind of comment
on as they on the face of it computers
have a much better memory than people
right we can store everything
practically on you know on a little
memory key these days we can put
terabytes of information in a very small
space but when we talk about human
memory that's not really what we mean we
don't mean storing everything at once
and accessing everything at once mean
rather this ability to
select salient information and attend to
that so for me I'm gonna this way I'm
gonna kind of talk about attention and
memory together in this talk okay so I'm
sure you've all heard lots about neural
networks in you know the earlier
lectures that you've had just you know
very briefly recap what are they well
you know in purely sort of mathematical
terms they're parametric nonlinear
function approximate errs that you can
fit to learn functions that go from
input vectors so for example a
photograph might be represented as an
input vector and all your sequence might
be a sequence of input vector as a piece
of text likewise and you take these
input vectors and you transform them
into output vectors via this parametric
function and for an example in example
we're looking at here when you're doing
image classification this is this is the
from the image net database your input
vector represents a photograph and your
output vector represents a distribution
over class labels in other words it
represents the probabilities that you're
giving to the labels that have been
applied to this class so in this case
the computer thinks it's probably a
lapper but it might be a Jaguar or maybe
a cheetah or a snow leopard or an
Egyptian cat and at first glance this
doesn't have a whole lot to do with the
tension right you feed in a whole vector
and you get a whole vector out it
appears that the network basically is
just looking at this whole picture
you've already in some sense you you it
feels like we've done the attention for
the computer beforehand by picking out
this picture of a leopard and from there
on it's just you know straightforward
function approximation but actually if
you kind of peek under the hood a little
bit into these neural networks you see
that within this this this parametric
function that they're learning there's a
kind of implicit attention meaning that
they respond more strongly to some parts
of the data than others so going back
here the network might be particularly
attuned to for example the spots on the
fur here right because that's something
that discriminates you know a leopard
from for example a Snow Leopard it might
also be looking at things like ears and
eyes because those are things that tell
it that it's looking at an animal and
not a you know a tree or something like
that
and to a first approximation we can get
a handle on where this attention is
being focused by looking at the network
Jacobian so if you're familiar this is a
standard term from linear algebra linear
analysis you basically just take the
sensitivity of the network outputs with
respect to the input so you know by
design these neural networks are
differentiable function approximator is
because we train them all with gradient
descent we use the back propagation
algorithm to find the gradient and we
can use this same algorithm to give us
the Jacobian that is to tell us when it
predicts an output like Leopard what
parts of the image was that decision
most strongly conditioned on what was it
really responding to what made it think
that this was a leopard and here's just
a you know a little illustration of what
that is if Y is your output vector and X
is your input vector you get this dy by
DX J and you basically get this if if
you've you know played around with with
back prop it's very straightforward all
you do is instead of setting the errors
to be the the gradient of the of the
loss with respect to the outputs like
you would normally you rather just set
them to be the output activations
themselves and then back propagate that
and here's a little illustration of you
know this visualizing this Jacobian in
action in a neural network and this is a
somewhat complex neural network system
that can kind of I think ties in nicely
with what you will have been learning
about in parallel which is deep
reinforcement learning so reinforcement
learning using deep neural networks and
this particular neural network has to
there's an interesting one so it takes
the the video input at the bottom of
this stack and then passes it through a
layer of convolutional neural networks
which are the kind of the standard
neural networks for dealing with image
data that you know learn these kind of
local local this pyramid of local
filters applied to the the image
transforming it in some high level
higher level representation that can be
used to make decisions like how to act
for example so this is a system that is
being trained I think in this case it's
being trained on Atari games it's
getting video streams as input and it
has to choose the
that it should take as output but unlike
most of the the deep neural networks
apply to Atari this one has what we call
two heads it has two sets of outputs one
of them basically tries to model the
value of the current state and the other
one attempts the model what we call the
advantage of taking a particular
action-reaction advantage which
basically says how much better so I have
a notion of how much reward I expect to
get in this state . so you're looking at
the screen and Atari and you and you
think okay I'm probably gonna get
another 50 points in this game just on
average but then you also have a notion
that some actions you could take right
now will be better than others and that
discrepancy is the advantage and the
reason they split this up into two heads
is these two you can you can model these
two things together and that's actually
the normal way of doing it with a single
set of outputs but they're on very
different scales typically like the
state value might have a you know a very
high wide range and the advantage of
taking one particular action over
another tends to be quite small and so
they come there was a great benefit in
splitting them up but what was really
interesting from our point of view or
from from point of view of this lecture
is that they then visualized on this
video here what the sensitivity of the
the corresponding outputs were to the
the video inputs received by the network
so on the Left we have the thing
predicting state value and on the on the
right we have the thing attempting to
predict advantage and the red kind of
clouds that you see appearing are the
degree of sensitivity so they've
superimposed this red heatmap and where
it's really bright red that means the
network is very sensitive to what it's
you know that particular part of the
screen so if you look at first of all
you can see that there's much more red
on the left and on the right so this
kind of tells you is that it's using a
lot more of the information on the
screen to assess the value of the
current state than it is to assess the
advantage of one action over another and
one thing that's very noticeable about
the one the left is always focused on
the score you see the score ticking over
at the bottom and so obviously the score
itself is a very strong indicator of how
much more score you expect to get in the
future right if you have a notion of how
much score you get or an average full
stop then you can sort of subtract the
current score from that to get a
prediction of where you're going in the
future
and another thing as you can see from
this the way this red flares up that it
tends to look quite far down the track
because it means this in this particular
game you get points every time you
overtake another car so it's kind of
looking ahead and saying look how many
cars are there in the immediate future
that I'm likely to overtake and
therefore get some some score some value
for this state I should point out with
all of these algorithms there's a kind
of exponential future decay of future
rewards so you're mostly focused on the
immediate rewards you're likely to get
rather than the ones in the very distant
future the one the thing on the right on
the other hand only flares up very
rarely and it flares up when there's a
another car very close to it and the
reason for that is it's only at those
points that there's really a significant
difference between taking one action and
another so most of the time you'll do
just fine
driving straight and not doing anything
it doesn't really matter if you go left
or right if there's a car right in front
of you and slightly to the left suddenly
it becomes very important whether you go
right or left because if you go left
you're bang into the back of it and if
you go right you'll get a head and so
this is just like a nice little
illustration of what you can work what
you can see is going on inside the
network in terms of how it's implicitly
attending to different parts of the
screen without any kind of explicit
attention mechanism so nothing nothing
in this network in this system was
trained to attend the one thing and
another that's just a feature of of
these neural networks and the same thing
happens with recurrent neural networks
now recurrent neural networks are kind
of my own speciality that's where you
know what I've done most of my my
research on over the past 15 or 16 years
and I focused a lot in particular on
things like speech signals and language
signals and signals the sequences of
handwriting which I'll show in a minute
now in this case so I'm assuming have
you heard about recurrent neural
networks earlier on in this course okay
so you know I will need to labor the
point here you know as as you as you
probably know by now it's not so
different from a normal neural network
except that you have these feedback
connections so you have again have a
large parametric function that
transforms inputs and into outputs
this parametric function is itself
conditioned on its own internal state of
the previous time step and the overall
effect of this is that it you can think
of it as something that transforms input
sequences into output sequences instead
of doing this one-to-one map from
vectors and to put it in more colloquial
terms what that means is that a
recurrent neural network has a memory it
has a memory of past events which is
encoded in this internal state and so if
you think about this attention thing
again you should be clear that once
again the network will attend to some
parts of this internal state more than
others and since that internal state
corresponds to a memory of past events
what you can see is that it's got a kind
of selective memory right it's
selectively looking at something some
things and not others that it received
in the past then using those to make
decisions and this is this is why I say
that attention and memory are so kind of
intimately bound up together and so
again we can we can look at the Jacobian
of this system I call it they call that
the the sequential Jacobian just to
distinguish it from the ordinary case
and what you're really looking at there
is for any particular output at a
particular time which which inputs over
the whole input sequence was that output
most sensitive to so what what we've it
made a decision at time T which input at
some time less than T most strongly
influenced that decision that's the
thing that it's remembering that it's
recalling when it makes that decision
okay and oh this basically reiterates
what I just said but there's a little
example here so this is you know one of
the first one of the the first things
that really successfully applied these
sorts of networks to was online
handwriting recognition and an online
handwriting basically means that you've
got the the trajectory of X Y
coordinates of the pen so for example
someone writing on a tablet or in this
case it was a whiteboard that had a an
infrared tracker for the pen and
basically the the data in that form is a
lot easier to process than if you just
get an image of handwritten text right
because you've got this nice clean low
dimensional representation that tells
you how the pen moved as they wrote it
and that actual so the thing that they
wrote is this there's these two words
once having the actual network inputs
are represented the next line up
basically there's there's an x and a y
there's a time input which isn't very
interesting it essentially just
monotonically increases there was some
variance in the time delays between
ticks and then there's this end of
stroke which shows where they lift the
pen off the paper in this case they
lifted up the pen after they wrote the
letter O after they finished the word
once and then again right at the end
they wrote having all in one go and then
they went back and put it up on the I
and this dot on the I is significant
because from the network's point of view
that dot is extremely discriminative
when you're trying to recognize a letter
I and say distinguish it from a letter L
is the dot that really tells you that
this is this is an eye and the point is
the point that was interesting about
this data from from a perspective of
memory and of attention is that the dot
as far as the input sequence that was
presented to the network the main body
of the eye and the dock were quite far
apart from one another right the I came
in the middle of the word the stroke for
the word having the dot came right at
the end and so by looking at this
sequential Jacobean for so I looked at
it for the you know calculated at the
point when the eye was emitted so the
point when the network decided it had
seen an eye or I should say this network
this this recurrent neural network is
bi-directional so there's actually two
RN ends going in both directions so it
means that it has future contexts as
well as past contexts but just think of
it as something that is mapping from the
entire input sequence the entire output
sequence and at the point when emitted
the I I wanted to calculate so what was
it responding to when it made that
decision and you can see this blue line
here shows the degree of sensitivity to
to the inputs that helped it make that
decision and you can see that it's much
more sensitive to stuff at the end
actually what it's really responding to
is the suffix ing so actually that whole
suffix is also very discriminative right
like that because that's a common suffix
if you see an N energy immediately after
a letter you already have a strong kind
of prior that it's an AI and not an L
right because LNG is not a suffix and
ing is and then there's this little
spike right at the end of sensitivity
and that spike corresponds to the point
where the dot was placed and so I
thought this was you know it's really
excited me at the time because it showed
how
how kind of intricate the underlying
mechanisms of these systems we're
learning actually were right they
weren't they weren't just somehow like
taking the whole input sequence and
putting it in a bag and pulling out an
output they had this complicated
selective attention mechanism that was
learning to ignore some things and
attend to others and again as I say this
this is closely bound up with the idea
of having a selective memory yes yes so
I calculated just yeah just for the
single unit corresponding to the eye at
a particular time point so I think what
I actually fed back into the network was
a network a one hop vector like all
zeros and a one for I and then back
propagated that you have to be careful
because if you try to back propagate
with respect to the entire softmax you
always get zero right because the entire
softmax always sums the one it's
normalized so you have to you have to be
a little bit yeah okay and here's a
video of the same thing happening in
real time so the the user is is writing
the words appearing in a second from
bottom line the inner state in the
internal state of the network is shown
in the second top row I mean it you know
interpreted as you will the decisions
made by the networker at the top so you
can see when it's deciding to output
letters this is a unidirectional network
so it only has past context which means
it has to kind of wait until it's
finished the word before it makes a
decision this ghostly thing at the
bottom is the sensitivity of the network
maybe we just play that again you can
see you know as it rolls on it's
responding mostly to the recent past but
you know it also looks back two to two
previous words and previous kind of
clusters of letters to help it
understand what's going on now you know
the point is it's not it's not uniform
right it's not uniformly looking at
everything it seems so far it's
selecting out certain things that help
it to make decisions and you can do even
more than this with implicit attention
you can do things like reordering in
machine translations so for example you
know it's a big problem of translation
between English for example and German
in the German verbs ten
end up at the end of sentences and this
creates problems for automatic
translation systems because they have to
take them and reorder them and in this
case God I know Sarah if I used to know
a little bit of German because I lived
there enough of forgotten at all but the
zoo RIKEN at the end here corresponds to
to reach right which is right at the
beginnings that is any German speakers
here yeah right yeah
thank you and so the the network you can
see that the network has to basically
take this thing at the end and move it
back to the start and by looking at
again another form of Jacobian you get
this matrix that says well which which
input basically the the columns the the
things on the left I think sorry the the
the the vertical dimension is the input
so the the the rows are the inputs and
the columns are they are the outputs I
think it actually doesn't really matter
which way around it goes but the point
is you can see if you take zoo a
reichian at the end and look up through
the rows you can see that it's mostly
sensitive to this this word reach at the
star right and so like this is this is
sort of interesting in that it's not
just this this is this is being done by
bypassing the whole input sequence
through a very deep convolutional
network so there actually isn't even an
RNN in here there's just a lot of
convolutional layers and even with a
system like that it's able to kind of
successively pass this information
around and and reorganize it so that
something at the end of one sentence
becomes you know the the main
discriminative feature for something at
the beginning of the other so again it's
it's it's not just kind of forgetting
some things and remembering others you
can use attention to to reshuffle a
reorder data as well okay so what I've
talked about so far are these implicit
attention mechanisms that just kind of
fall out automatically of having a
neural network of having a large
parametric system and it you know these
are these are these are very interesting
it's nice to see these things there and
it's certainly worth analyzing them but
it feels like at least in human
cognition we have a little bit more
control than that we have a sort of
attention so I mean at the very least
you can see you know just by moving your
eyes towards a different part of the
same scene you're shifting your
attention and you know even internally
you can you can decide to concentrate on
one thing one thing or another so it
feels that not everything is done with
these kind of implicit attention
mechanisms there are sometimes explicit
ways that we we decide to focus on one
thing and not another and there's
several advantages to this one of them
can be computational efficiency so if
you if you can explicitly limit yourself
to some subset of the data then you
don't have to reprocess the whole thing
right and so you can save yourself time
another thing can be scalability so for
example if you've got a fixed size
glimpse you have a model of a kind of
robotic eye looking at an image then you
can give it any size of image any any
any size of scene while keeping the the
dimensions of the glimpses itself the
things that are going to be processed by
your network fixed another thing that I
find very interesting is you can you can
turn static data into sequential data so
when I first started in machine learning
everyone was doing the opposite everyone
would take sequences and kind of chalk
everything together into a big bag and
process it like a static vector because
people didn't really know how to handle
sequences very well recurrent neural
networks didn't work very well at that
time there was a strong trend towards
you know getting everything into a
single vector and feeding it into a
support vector machine or something like
that and and and now in you know recent
years it's kind of going the other way
people are trying to turn everything
into sequences because they found that
these sequence processing models are so
powerful and so one example of that is
if you if you've got explicit attention
you can turn static data into a sequence
you you have a fixed photograph but your
eye moves around so maybe you've seen
these videos of how visualizations of
how people look at images these saccades
of the eye moving rapidly what you're
actually getting in your brain is a
sequence of very low kind of bandwidth
fixations rather than getting you know
many megapixels at once which is how we
tend to think about these things and I
guess the last advantage of explicit
attention mechanisms is that they can be
much easier to in
rich for for the person running the
experiment as an easier to visualize
because you can you can explicitly see
what part of the data this thing was was
focusing on rather than having to do we
saw in the previous few slides and
calculate some indirect measure like the
the sensitivity in order to work out
where you were looking okay so there's a
whole bunch of neural attention models
and that's what I guess the rest of this
lecture is mostly going to cover but
they all have a kind of a similar format
so you have a neural network that
receives input and produces output as
usual so the bit with the gray lines
here on the right of the image you got
input vectors coming into a network that
produces output vectors the network
might be recurrent it might be
feed-forward it actually doesn't matter
so much in this case but there's also
some extra set of outputs coming from
the network that are used the
parametrized and attention model so for
example these might give the location of
a gaze of the network or it might give a
time or or a lookup key telling it what
to attend to out of some you know a set
of paths theta we'll talk more about
those in specifically later and then the
attention model has some extra data to
operate on so this could be you know
just the rest of the same image that
you're looking at or it could be another
image it could be an audio sample it
could be some text to be translated but
whatever it's looking at it's going to
take this attention model and somehow
apply it to the the extra data and
create but typically what it does it
creates some fixed sized glimpse this
glimpse vector is then the thing that is
passed into the network so you've got
some this you know just the recap the
network outputs some parameters those
parameters give you are used to
parameterize an attention model the
attention model is applied to some extra
data that gives you some glimpse vector
of you know pre-specified size and
format and that's then fed into the
network and so an important thing to
note is that the the combined system
here is recurrent even if the network
isn't so because you've got this loop
where you output some attention
parameters and those attention
parameters then affect what you will get
as input
the next step you've got something that
is effectively recurrent even if the
network itself doesn't have a recurrent
internal state okay so you know the the
the the general idea is to think of this
probabilistically and say okay what
we've got is a probability distribution
over glimpse glimpses of the data so
there's extra data acts and we're
applying our attention model to it using
some set of attention output so these
attention parameters are called a here
right in the mix that we get a
conditional probability distribution
which is the probability of glimpse G
given attention a right and the simplest
thing you can do then is just say well a
just assigns probabilities through a set
of discrete glimpses so if you imagine
in this kind of cartoon picture here
there's nine possible places we're
allowed to look in this picture we are
parameters of our attention is just nine
numbers that we feed into a softmax and
get a probability distribution that
tells us which of these things were most
likely to look at and then we sample
from that and that's where we get our
next glimpse from so this would be
pretty much the simplest attention model
you could have and so if you do this you
can you you can you can literally
implement and use this you basically
treat the distribution of a glimpses as
a stochastic policy right so you've got
this probability of looking here or
looking there and you think of this as a
reinforcement learning problem it's a
little bit feels a little bit different
from normal reinforcement learning in
that you're not actually modifying the
environment but you're modifying the
information you'll get back from the
environment so from from the perspective
of the agent from the perspective of the
model it's as if you've modified
something in the environment does that
make sense to everyone just by choosing
where to look you're already kind of
implicitly changing your own environment
and so it becomes a lot like an RL
problem and then you can use RL
techniques if you say there's some
reward in this case let's say it's the
loss for something like classification
that is induced by a particular glimpse
then you can use that to train the model
with reinforce for some similar problem
and this is actually this is a general
technique this is useful to know anytime
you have even if you have a supervised
task if you have some part of the system
that's just completely non
differentiable like as in you're going
to look here or you're going to
there and there's no you know continuity
between those things there's no way of
getting a gradient between one and the
other then you can sort of fall back on
these RL methods you can always use
reinforce basically to get a stochastic
gradient even if you can't get a normal
back probe gradient but you know
generally you want to do something a
little bit more complex than that you
wouldn't want to just have a soft max
you might have something like a Gaussian
filter that has you know a particular
Center the coordinates of its center has
a width and its height the that is that
you know the distribution would be
something more complex like there might
be a continuum of places where you're
going to look rather than a set of you
know and discrete disjoint places
disjoint subsets and also usually the
glimpses are gonna be something more
complex than just an image tile so
people build you know quite
sophisticated foveal type models where
you have higher resolution in the middle
and lower resolution out at the edge so
that you can you know you can look at
something you can be focused on
someone's eyes and get them and you know
very sharply and high-resolution but
also get the background of the scene at
lower resolution because that might be
useful for whatever decision you're
making and it also is useful to tell you
where to look next right so like you
know in order to decide what to focus on
you have to have some kind of
information about what's over there you
need to see a blur of something
happening over there before you turn
your head and look and here's an example
of something with a fovea glimpse model
I'm actually struggling to to remember
some of these papers you know this is
only four years ago but it seems like a
long time this is from recurrent models
of visual attention and this was
something that had it was allowed to
output a sequence of glimpses well I
attempted to classify an image so that
you know the model was foveal the if you
both of both of these two rows here the
thing on the left shows you the actual
image and shows you the green points
show you the fixations of the model as
it processed that image and then the
remaining images show you the six things
that the model actually saw like would
it actually get back from
foveal image and you can see that what
it sees is some part of the image quite
sharply in other parts that are further
away from the center or more blurred
because of this this multi-resolution
model that's there and really what's
happening is that you know it's learning
quite quickly to zoom in and and so that
the job here is to classify the images
this is just good old standard and
missed with some background clutter
added so as to make it a more
interesting attention type pass because
in normal M this detention is trivially
trivial you always just look right at
the center of the image and there's no
distractors or anything like that so in
this case you know it learns to
basically zoom in on the thing of
interest the number and it learns to
kind of go around it like you can see
this very clearly with the eight it's
sort of in its budget of six climp sees
that it's got to recognize this thing it
uses them to kind of go around the
perimeter and therefore get all of the
important discriminative information it
can before it makes its decision about
what digit it is and this was all
trained with with reinforcement learning
and here's a an extension of this to
identifying multiple objects and sorry
this video goes a little bit too fast
for him for me to keep up with here and
so here we have basically the same data
set only with let's go back to the start
the same data set only with multiple
characters in it instead of one right
and this is this is this illustrates a
sort of an example of attention straight
away actually right like if you have a
model that is this is sort of comes back
this idea of turning images into
sequences right if you have if your
images contain exactly one handwritten
letter and you can very simply define a
kind of static classification pass
you're always going to get an image in
and you output a single class if they
contain a variable number of letters it
becomes harder to do that it's much more
natural to think of it as a sequence
where you keep looking through the image
and every time you see another number
that you didn't see before you say okay
I've seen one and so now that in that
sense you can immediately see that this
this is you know you have more
flexibility once you think of the data
as a sequence of glimpses rather than as
a static image oops okay I'll just let
the whole thing run through I think
after this it looks at some Street View
house numbers
and you can you know basically what the
point of this is you can see it focusing
you can see it looking at one image
jumps from one number to another it's
using this phobia to kind of look in the
periphery and then and then move out to
see where it needs to look next here it
has a whole sequence it goes through
needs to go through them in order I
think from left to right in this case so
it's learned how to do this kind of pass
through things I think in this case it
just yeah so first task we just had to
find the first digit and these these are
like real photographs of street numbers
and here it has to read the whole thing
and you can see again it's learning to
do these kind of sweeps these these
passes through and I think it was yeah
it was also applied to to imagenet so
like you know is it advantageous so I
mean for for cut like for classification
where you know there really is just a
single target in a single image there's
usually not that big of an advantage to
turning it you know to think giving it
as a series of fixations right your as
well just feeding it into one big
continent it becomes more beneficial to
have explicit attention mechanisms when
the paths get more complex how am i
doing for time by the way does anyone
should have my watch okay and the break
is at 3:00 is that right so I think I'm
actually a little bit ahead of schedule
here okay so these attentions we looked
at so far used what we call hard
attention so there were these fixed
attention windows that were trained with
RL techniques and the significance of
this is that basically they're not
they're not differentiable there wasn't
a gradient that we could backprop
through there's a notion of you either
look here or you look there there wasn't
a continuum between looking here and
there now for a lot of for some systems
that's really necessary like if you've
got a robot that has to choose to look
left or look right you're kind of forced
to use RL right because you you're not
going to get a smooth gradient between
going to one and the other in general
but if if you're just talking about you
know processing an image that you where
you have the entire image on your
computer that's not necessarily the case
right the attention doesn't really need
be hard we just want to focus more on
some regions and less on others so
something a little bit in a way more
like the implicit attention we saw
earlier and if you do this in a
differentiable way what you end up with
this is what we call soft attention and
the point there is that soft attention
can be trained and with back prop you
can train it just like a normal system
and generally this is a lot easier than
using RL but on the downside it can be
more expensive to compute because
typically with the soft attention rather
than selectively taking some little
chunk of the image and throwing away the
rest
you're still processing the whole you
still have to process all of the data at
once in order to calculate this
attention model and just to give a
little bit more detail about what the
template of this is so once again think
about it as a probability distribution a
probability distribution over glimpses
so we've got again we've got a set of
parameters a and a distribution over the
possible glimpses G but what we do now
in order to make it soft as we take an
expectation instead of a sample that's
really the only difference right so if
you if you if you stochastically pick
samples from this distribution then the
best you can get is a stochastic
reinforced style gradient if you take
the expectation on the other hand and so
if your vector G is now this this
weighted sum where you weight over the
possible glimpses times the probability
of that glimpse under your distribution
you get from a then you directly can
differentiate that with respect to a you
get it you get a gradient with respect
to your your your attention parameters
and those are the things that actually
came out of the network so you have a
gradient flowing back into the network
as usual and this is used really all
kinds there's all kinds of you know
variants of soft attention but they
always pretty much always come back to
this basic template it's just about
taking a mean instead of a sample and so
the first at least my own first
experience of using soft attention was
when I was looking at generating so as I
said early I earlier on I'd been looking
at handwriting recognition so
recognizing sequences of online
handwriting and then I thought it would
be interesting to look at generating
sequences of online handwrite
and specifically to do handwriting
synthesis which means given some text
sequence basically write that sequence
by hand right generate a handwriting
sequence that will from which you can
read the text sequence that you were
trying to generate so basically just
like text-to-speech only its text to
handwriting and to do this I found that
I needed an attention mechanism because
generally a text sequence might be a
sequence of maybe 50 characters or
something like that it's a sentence that
you're writing but typically the the
handwriting sequence would be much
longer than that right because this this
handwriting sequence was encoded as a
series of XY points and you might have
maybe 10 or 12 points per letter or more
even depends on how big the handwriting
was and how slowly they moved the pen
and so forth and it's highly variable
how many points there are per letter
obviously like an H will typically take
more points than an owl for example and
all of this meant that it wasn't
straightforward to align these two
sequences right I wanted a map from a
text sequence to a handwriting sequence
but there is an alignment problem this
is the same problem you get in speech
recognition and it takes the speech and
in other areas as well where you need to
kind of warp you need to stretch and
shrink these sequences so that they line
up with one another and there were there
are various techniques for doing that
but one thing that I thought would be
you know really interesting and really
general is just to give the network a
way of choosing what to attend to in the
input sequence and the way I did this
and I apologize this slide isn't really
very isn't really very clear the network
basically outputs a bunch of parameters
that parameterize a Gaussian mixture so
a set of weights the set of means a set
of variances and those things are then
convulse equals n to the text so the
text is represented as a sequence of
one-hop vectors this this thing at the
bottom and then you have these gaussians
which basically have means that like so
the means and the variants correspond to
the the position within the sequence if
you imagine that these things
index left-to-right there's the zero
factor the first second third fourth and
so on
then basically what you're saying is the
the parameters of the Gaussian say
something like well I want you to look
at the third element of the text
sequence with some variants you know
maybe plus or minus two characters
surrounding it right and then
essentially mathematically all that's
done is you convolve those Gaussian so
you just calculate the the height of the
Gaussian at each particular index along
the sequence and multiply it by that one
hot vector and then sum the whole thing
together what you get is a kind of munge
together a vector that is more or less
focused on different parts of this text
sequence and I apologize if that's a
little bit hard to digest you should
look at the paper if you want this yes
yes the number of components was fixed
and one other thing I should say this
very important is it didn't output
absolute numbers but rather offsets from
where it was looking at previously so it
was strongly biased towards the fact
that both the the handwriting sequence
and the text sequence are going left to
right or right to left if you're doing
Arabic I mean that part's not so
important but there's it there's a
continuum in that you're you're always
you're not doing this reordering thing
that you are for example in translation
you're starting at one end and going
going forwards and so first of all you
know this this worked very well these
are all you know generated see I have it
there's an online demo I think it's
still it's still working if you search
for my name search for Alex graves
handwriting you'll find a demo where you
can type in some text and it will write
it for you
and you can either let it pick a random
style to write in sort of sort of its
learn these styles from the training set
or you can you can condition it by
giving it what a particular style and
get it to write like that who I've said
real people write this badly maybe they
don't write quite that bad that you know
they do at its best the sequence this
thing generates text that looks very
much like real and somewhat messy
handwriting sometimes it goes completely
AWOL but well I think what's really nice
is that it tends to pick a style and
stay with the style so from it from a
generative modeling point of view this
thing's interesting from the
this lecture I guess was mostly
interesting about it is that it was
using this attention mechanism so as
it's writing these sequences but were
generated by it's slowly moving through
the the corresponding text vectors the T
the hte the s and you can really see
that as it's writing for example as it's
writing letter s it's looking at the
letter s and it's also looking at the
letters before and after it and this is
you know obviously for cursive
handwriting this is very important you
need to be looked you always need to be
looking at a window because you have to
know when you're writing a particular
letter how you're gonna join it up with
other letters and how it joins up with
the previous ones and actually you can
visualize this so you know you look at
the alignment you get these kind of
curves like this that basically show you
know this is what this heat map shows is
when it was writing you know look at
wherever it was at the bottom and go
straight up and then go left and it
shows you what text it was focused on
when it was writing a particular thing
is you know it's essentially focused on
the thing it's writing with a window
right it's important that it has this
window because it's using that to make
the writing smooth okay okay I'll try
and get through yeah I'll get through
this next section and then we'll have a
break so that was that was a tension
based on location
so you're basically you're not attending
to the input sequence according to
what's in the sequence but just
according to the order that it happened
to arrive and that that is a particular
that's one particular way of attending
to things right it's like imagine if
someone gives you ten objects and you
say okay I'm gonna look at the third
object now rather than I'm gonna look at
the green stone which happens to be the
third object for some types of problems
location is a very good prior let's say
and handwriting is one of them but a
much more general thing is what's called
associative attention where instead of
attending by position we attend by
content so now you look at your sequence
of objects and you say look I want to
look at all the green stones and maybe
there's one at position three and one at
position seven whatever that's now the
thing that you're looking for
so in this setting we still have the
same basic framework the network outputs
some parameters we get a distribution
over places to look but here typically
the parameters instead of encoding as
they did before locations points in the
sequence too to attend to they encode a
key vector that is compared to all
possible glimpses using some similarity
function so you output if you want to
get green stones you output a vector
that somehow embodies or encodes the
notion of greenness and the notion of
stone Ness and you compare that with all
the the vectors that you have all the
embeddings you have for the things that
you've just been presented with and the
ones that should be closest to your key
are the ones that are you know that
match it most strongly and typically the
you know that should give you the green
stones and not the red bricks and this
is you know mathematically it's very
simple you have a similarity function s
typically this is cosine similarity
which really just boils down to a dot
product you're taking dot products
between vectors and you know if those
dot products are high then these things
are similar if the dot products are low
if they're close to zero then they're
dissimilar you typically have to if you
want to have an actual distribution here
which you often do then you have to
normalize things you can do more
complicated stuff so people have done
you know have them not just cosine or
some fixed similarity measure but rather
learned one using a using a neural
network or using a linear operator but
either way you get this kind of you get
you get something that's actually very
powerful so you know in principle
Association is all you need in terms of
look at people who build whole computers
using nothing but associative recall
associated lookup because you can you
can you can search it gives you this
multi-dimensional feature based lookup
and what I mean so I mentioned before
you've got green stones you might have a
lookup that said look just give me green
things full stop right and it might give
you back a green stone but it also gives
you back a green brick or you maybe only
look for brick so you only look for
stone so it allows you to if you
represent the data in the right way if
you represent the data in a kind of
separable way then it allows you to
search for
many you realize you to kind of
fine-tune which things you're searching
for and which things you are so it's a
very powerful very flexible method and
the first place that it was really
presented was in this paper by badda no
at all and he's now a deep mind as well
neural machine translation by jointly
learning to align and translate and so
again that can be used for this and it
was used explicitly for this reordering
problem in that original paper so again
we here we have a French to English
sequence where generally unlike you know
going from English to German there isn't
quite as much reordering but there's
certain things like the orders of
adjectives relatives to noun so we have
zone economic european sorry about my
french versus european economic area
where we've just you know got the same
words but they're slightly reordered and
that's exactly reflected here in the
this heat map right you can you that
that's sort of the the cross part of the
sword here is the part where it's it's
flipping around these these three words
and it's it's doing this by essentially
learning what it's about what it can do
is learn an internal embedding so a word
like european it's learned that that
word is associated with arrow pn they're
both kind of map together when it sees
one of those in the input sequence it
uses that as the key and it gets back
the other one from the the sequence that
is attempting to translate and you this
is this is as I said this association is
a very general mechanism and so you can
learn you can use it for for example for
speech recognition so rather than doing
what people have done in the past
including the the work that I've done in
the past which was more along the lines
of attempting to dynamically kind of
stretch and compress one sequence into
another and I should say that that
approach the sort of more conventional
approach the secret speech recognition
relies again on this fact that there's
this kind of this temporal continuity
basically the the the
the order of the words and the order of
the audio match right like you you you
know you speak the words in order
basically and so you don't have you
don't have to reorder in general when
you're doing speech recognition but you
can still use this attention mechanism
to try to you know pick out bits of the
sequence so for example you know some
part of the sequence that it knows might
correspond to this the M word at the
start of Mary it's learned how to
recognize that and it can use
Association to find it and then it can
learn this kind of alignment pattern
that's shown here
ya see now that I've said all this I'm
trying to remember how this system
actually worked yes it's learned that
similarity or rather it has a fixed
similarity measure the cosine similarity
but it's learned the embedding so
basically what it's doing is it takes in
the input sequence I think in this case
it processed that with a bi-directional
lsdm network so it takes these you know
input vectors that represent words and
turns them into embeddings and then it
can search those embeddings yeah yeah
exactly and you know it and it can work
at a higher level than that as well it
can find phrases that you know translate
from one to the other and so on yes it
has to I mean Association is only as
good as your representations right like
the the mechanism for Association is
simple learning the representations is
difficult and you have to do that if you
want to be able to search through them
in a meaningful way and here's another
very nice example of this use of
associative attention so this was from
another group a deep mind Karl Moritz
Hermann and others teaching machines to
read and comprehend and the problem they
had there was what they wanted to do was
take these kind of quite long in
sequences which are drawn from newspaper
articles I believe and ask them
questions they're looking at kind of
general question answering you get some
big chunk of text and then someone asks
you like it like a
like they give to children like an
interpretation question that's not the
word I'm looking forward yeah they asked
them a question about the text in this
case it is identifies deceased sailor as
X and they're supposed to work out who X
is and so X it turns out is entity 23
okay so it's a little bit abstract they
don't they remove all the names and
replace them with these entity markers
for technical reasons but basically what
you have to do is guess which entity
would be the deceased sailor in this
case and the problem so you can
principle you can do these these things
just by feeding them into a very big
recurrent neural network you feed in the
whole input sequence and then you ask it
the output sequence the question one
word at a time
the problem is that the network's tend
to forget right they have problems with
very long sequences it puts too much
kind of strain on their memory but if
they have if while they're being asked
to answer the question they're able to
do associative look up on the embeddings
that they formed when they processed or
when another network processed the input
sequence then their range of memory kind
of stops being a problem because they
can they can look up something like
Saylor right they've got some
information about Saylor they can use
that to look up in the sequence things
that court that are somehow related to
sailors and use that to narrow down you
know the the search for the thing that
they're looking for so to put it in in
simple terms if you have this kind of
associative lookup then you can the
length of time that has elapsed between
something that you're looking for and
and and now doesn't matter anymore right
like if you if you're if you are if you
think about how the you know the human
mind works as well like you see a person
walks into a room and they remind you
they might remind you of someone you saw
you know twenty years ago or something
like that right as long as that person
has has stayed in your mind for whatever
reason it doesn't really matter so much
how long ago it happened right because
you're not you're not sort of you're not
having to search your memory by time
you're not having to say look here I
want you to look back twenty years
rather you're searching by content
you're searching by and I say searching
and no idea how this is you know
embodied in the human brain
seems to happen very automatic ly that
we see something and it reminds us of
something that we've seen before we've
something that we associate with it and
so this is kind of a long-winded way of
saying that associative attention can
give you a much more powerful memory
than you natively get from like a
recurrent neural network and that's why
you know L SCM plus attention has become
a real sort of workhorse for sequence
learning problems okay and I think we
should stop for a break there before we
turn to introspective attention we will
now resume the lecture and for the next
section of the talk we will we will
consider what I call introspective
attention so the kind of attention we
looked at so far was selectively
attending to some kind of external data
so you had an input sequence in one
language that you were attempting to
translate into another and you attended
to some part of that input sequence you
had an image that you were attempting to
classify and you selectively attended to
some part of that image so it's quite a
straightforward form of attention but as
I said at the start attention doesn't
just allow us to pick and choose between
you know things that we're looking at or
things that we're listening to that's
coming from outside it also allows us to
pursue one line of thought over another
allows us to think about one thing and
not another and that's what I mean by
introspective attention and so if you
think that as far as the neural networks
concerned the or at least as far as the
recurrent neural network is concerned
its thoughts are basically whatever is
in its current state vector its current
you know embedded memory then choosing
to look at some part of that memory and
not another gives you a form of
introspective attention and once you
have this you can start not only they
see another key difference
typically with external data you're not
free to modify you're given an image as
is you're not allowed to change that
image you know if you someone asks you
if it's a cat or a dog you can't just
draw a cap on it and then say
it's a cat whereas if what you're
attending to is internal if it's the
contents of your own mind you're
typically modifying the stuff you're
looking at at the same time as you're as
you're reading from it so you can
selectively write as well as read and
thereby iteratively modify your state so
this it starts the sound immediately a
lot more like a computation device and
not just a way of you know looking at
one thing or another and so this was
what led me to and and my colleagues it
did mind to develop what we called
neural Turing machines which was a form
of memory augmented neural network so
the idea is that we had a controller
network which again could be
feed-forward or it could be recurrent
this is doing what a neural network
usually does it's receiving external
input and emitting external output so
that's like the top part of this diagram
but it has an extra part of it that
allows it to interact with some memory
matrix and it did this via parametric
functions which we called heads and this
was kind of just in keeping with the
Turing machine tape metaphor
don't let that distract you too much
these heads are just they're just
functions they're just parametric
functions the parameters once again are
emitted by the network so this is a sort
of there's a general rule for building
things with neural networks whatever you
want the network to do you work out a
functional form for it and you get the
network to supply the parameters for
that functional form and that way the
network is can control this this
operation whatever it is and so in our
case we wanted it to control its ability
to select portions of the memory and
read or write to those things in memory
okay so once again we used a key part of
this system was associative lookup so I
mentioned that before this this neural
translation paper like I said this thing
has become quite ubiquitous it's used
all over the place we found it to be
very important for for these these
Turing machine like computational models
so we added a few extra I mean we
actually developed this simultaneously
with the other
you know we were sort of busy working
away on this when the paper came out so
it's one of these kind of independent
discovery and there were there were
other researchers doing similar things
as well we had a few sort of bells and
whistles so we had a key vector as is
usually the case this thing is now being
compared with the content of memory so
basically as I said the memory is a big
two-dimensional matrix each row in the
memory is therefore a vector which you
can think of as a memory right that's
that's a it's an independent memory and
now the this introspective attention is
taking some key emitted by the
controller Network and comparing it with
everything in the memories it's doing
this associative similarity this cosine
distance between the key in the memory
to find the memories that are closest
now we normalized that with a with the
softmax we also added an extra thing
which was a sharpness parameter a beta
which controls that kind of controls the
temperature of the softmax which can
kind of selectively narrow the focus and
we did this because we wanted the
network to be free to say look don't
give me the end nearest things to this
key but rather just give me the one
closest thing because for some problems
you don't want several you know you
don't want something that's a mixture of
Soren I should say this these
content-based mechanisms are being done
with the soft this is all soft attention
in the way that I defined it before so
this this thing gives you this
addressing mechanism gives you a
distribution and then you take the
expectation of that distribution so if
the sharpness is low then the
distribution is quite blurred out and
your expectation is kind of a blurred
sum of a lot of things and sometimes
it's not what you want you want to look
at exactly one thing and so then that
was why we included this chart it's
actually it's kind of a technical detail
so you shouldn't spend too long on it
but actually we we thought you know we
kind of wanted to throw the the
everything at this and have well so
actually my original inspiration for
this architecture came from the work
that I mentioned earlier on on
handwriting synthesis and you know
occurred to me that this attention
mechanism was choosing where to read in
the input sequence and it really is
scanning like a tape in that case is
scanning left to right I thought well if
I could write if this thing could modify
the input sequence as well then it would
really look like a Turing machine I'd
have this tape head that's moving around
and it can read and write and you know
over time the Turing machine Mehta
forgot somewhat lost in that associative
memory doesn't really fit too well with
the you know the whole Turing metaphor
but anyway that was the origin of the
thing um but we also included
location-based attention which was sort
of a direct descendant of this you know
this reading mechanism that was used for
handwriting synthesis and here the idea
is basically a shift kernel that says
look wherever whatever memory you looked
at last move n steps to the left or
right or you know up or down from that
memory and again the the sort of
technical challenge with all of this is
easy to say these things in words right
find the thing that's nearest to this
key in memory move and steps left to
right and memory the thing that was
difficult about all of this was making
it differentiable and this was sort of
there's a design choice right all of
this stuff could be done without back
propagation without smooth gradients it
could be done by a reinforcement
learning in practice it just becomes
extremely hard and the reason is hard is
that those types of you know reinforced
styles the caste gradients the noise
compounds when you have many successive
steps being performed all of which are
are non differentiable and so if you've
got a system where you're successively
reading and writing things from memory
you can see that these it gets kind of
exponentially harder to get a nice
gradient through those operations if you
don't have if the operations weren't
continuous to start with it's another
question
[Music]
social Association doesn't I'll kind of
go through that in another slide it's
just it's an it's another way I mean in
principle you can do everything with
Association it's important to remember
that in principle there really is no
need to have another but if you if you
think about what you've got a normal
computer right
you can look things up by date you can
things up by title you could you can
look things up like on Google by some
sort of you know Association type of
thing like basically it's advantageous
to have lots of different ways to access
information this is the short answer to
that yes yes yes well that that is what
they what it does but I mean well if you
if by combining you mean adding so you
talking about concatenating the there's
this there's lots of different ways of
doing that's like one thing you could
say is like take the the top end closest
things and concatenate all of them into
one big vector but there again that's
not differentiable because as the key
changes you get these sharp thresholds
where what's in and what's out of the
top end changes and you know that's but
but if you if you add them together then
if you add all of it together then
suddenly it becomes differentiable yeah
you get an expectation yes I'll come to
that in a minute right so like at the
moment they say the memory starts off
just being a big matrix full of zeroes
right it's empty and and then you can
you can access it through various
mechanisms location or comp so that
that's another thing you know when the
memory is empty Association based comp
lookup isn't very useful right because
there's nothing there to associate you
can you can you can randomize the memory
to start with so that Association will
sort of work from the ground up but you
can see that it's kind of artificial
what you really want to say is look I
typically what you actually want to do
is say look I need someone I need some
blank memory - right - right the first
thing you have to do is put something in
the memory and you want to do that more
like by location like give me the first
piece of memory or there are actually
other ones like give me the the the
least recently used piece of memory
that's another useful attention
mechanism and so anyway that I think
this somewhat answers the question
earlier about like what's the point of
having you know these different
mechanisms well where you can think of
is that the combination of address
mechanisms gives you different modes for
interacting with the memory and in
computer programming terms you can think
about these as corresponding to
different data structures and accesses
this is a little bit like STL
terminology here if you're just using a
key based lookup associative lookup then
the memory is really like a map right
you've got a key and you've got a value
in this case it's it's even simpler
right because the key and the value are
the same but you can actually you can
you can you can define and people have
defined attention mechanisms where
there's a separate key value pair so the
thing you use to look up isn't the thing
you get back rather it's something that
has been associated with that thing if
you can use some combination of content
and location well the key can first you
can first of all find the start of an
array and then this location-based shift
can give you an offset into the array so
you've got some you know memory of like
you know give me that list of N Things
that were you know that was told to me
yesterday and then give me the third one
out of that list by taking an off set of
three and if you're just using location
then you're kind of just iterating from
where you where you last were so you
have this iterator that goes through
something like an array and we'll see
how the network actually uses these to
solve you know concrete problems okay
but so to answer the other question how
does actually modify the memory so you
know what I've said so far that's how it
finds things that finds the heads are
basically placed that a particular put
at a particular place in memory or to
look at it another way they give you
back pointers to something in memory but
then in terms of what you're actually
going to do with that memory well either
you're going to read in the read
operation is straightforward it's really
it's really exactly like these this this
expectation value this this soft this
this soft basically like the the
canonical kind of soft attention thing
where you take the what the the lookup
the attention mechanism itself gives
gives you a distribution over a
probability distribution over rows and
the matrix that's this WI and then you
just multiply that by the content of the
matrix and add this up so you get a sort
of expected return
from whatever it was that you were
looking up and that's reading so that's
that's pretty much the same as all the
other attention matron so writing is
more complicated and I should probably
say that in general learning how to
write is really difficult and that's
part of the reason like these you know
see this neural Turing machine and the
differential neural computer that we did
after that these papers are a few years
old now and those models are still not
very in and all that much mainstream use
and the reason they're not in that much
mainstream use is that learning how to
write in principle makes them more
powerful than a system that just has
kind of read-only access but in practice
it makes the learning problem much more
difficult it's really hard to learn the
right things into memory and then to
read them back as well gives you this
kind of coupled learning problem but
that's just kind of an aside but anyway
the way we defined writing we said
basically it's parametrized using an
arrays vector and an ADD vector where
the arrays vector sort of deletes sort
of continuously or smoothly deletes
information essentially multiplies it by
a number between 1 and 0 if you multiply
by 0 then you're deleting if you
multiply by 1 you're leaving what was
there and then an ADD vector that's just
you know some vector that's added on
after the delete and this was kind of
inspired by the long short-term memory
architecture if you've heard about that
already which contains a forget gate
which essentially does this arrays
operation it decides whether or not to
get rid of the information and then this
input gate which decides whether or not
to add something new to the contents of
the cell so you have basically a soft
form of basically if the arrays vector
is 1 then the right head just exactly
says take let me decide if the arrays
vector is 1 and if the the attention is
completely focused on one row of the
memory matrix then you have this
completely clean thing that says take
whatever is in row end of the matrix and
replace it with this information that's
in my add vector so that at that point
it sort of it corresponds to what we
expect from a normal computer but in
order to make the whole thing
differentiable there has to be this
continuum between doing that and between
kind of
accessing several different rows and a
kind of blurred out way and more or less
strongly deleting what's in those rows
okay and so you know does this this
sounds awfully complicated can actually
do anything in practice well we started
off with the simplest sequence kind of
algorithm we could think of which is
just to copy some information so we give
it an input sequence which is a whole
bunch of these random you know binary
vectors or you know like it's it's a
vector of size I can't remember is it
eight or six or something like that of
binary numbers and you get a variable
number of these vectors in this case
maybe it's 20 or something like that and
then after you've seen all of them you
get a little indicator this little white
dot right at the bottom that says okay
the sequence over now and then the task
is just to output everything you've seen
so far so you just it's just copy you
just have to remember a bunch of stuff
and then write it down sounds completely
trivial an ordinary lsdm network can
actually do this up to a certain length
right it can learn how to remember
information what it can't do where it
really falls down is if you train it to
model sequences of up to length 10 and
then you ask it to copy a sequence of
length 20 it breaks right it doesn't
know what to do because it's sort of
explicitly learned during training that
it has to copy
you know the nth element of the input to
the nth element of the outputted in
other words it hasn't learned an
algorithm and that's what that's what
the the neural Turing machine can a
project was all about it was can we get
a neural network to learn algorithms
just with an ordinary end and gradient
based learning and that's why we're
talking about data structures and
accesses and so forth and here it does
learn an algorithm which is kind of
expressed in the pseudocode here which
is kind of like the way you would
implement copy in a really low-level
programming language so you start off by
moving the head to some start location
which doesn't really matter the memory
is empty doesn't really matter where you
start as long as you're receiving inputs
you write the input to the head
locations that is you set the arrays
vector to one and you write down an
embedding
that represents this binary vector which
isn't necessarily exactly the same thing
as the binary vector but it's something
the network can sort of decode to that
binary vector then you use your shift
operator to increment to the next row in
memory and then you repeat the process
and you keep doing that until you
receive this input the limiter and then
you jump back to the start location and
typically you're going to do do that
using associative lookup so you left
something at the start you left a kind
of flag at the head of your array such
that you could find your way back to it
associatively and then you start reading
these output vectors in so before you
were writing now you're reading you emit
the output and again you increment using
your shift operator so it uses these two
attention mechanisms and tandem to
implement a very simple just you know
you know keep on iterating one step at a
time and writing stuff down then go back
to the start
read it back and copy it and as I said
the thing that made it interesting was
that it generalized so if we trained it
on length 10 to 20 and we looked at what
happened at length 20 length 40 at
length 50 and I think at the bottoms
length the hundreds it made some
mistakes it wasn't perfect so these heat
maps basically show you know they should
look exactly the same between the
targets and the outputs for training and
you can see they're not perfect there's
a few mistakes the one at the bottom for
example I think at some point it misses
out a target so everything has actually
shifted a long one to the left but
generally you can see that it's it's
running a slightly imperfect copy
algorithm that has generalized beyond
the range in which it was trained that
was kind of the interesting part here
okay so we thought okay where can we go
from that well you know maybe the
simplest next thing we'd like this to
learn is a for loop right so let's just
take that copy algorithm and say well
now you don't have to just copy
something once you have to copy at end
times where n is a variable and again it
basically learns exactly the same
algorithm as before except it learns how
to count to keep this sum to store this
this number and in the memory and to
iteratively decrease it after each step
so that it knows when it's finished and
it's emitted all of the data
now this this was a more complicated
algorithm it learned and I might
struggle to remember exactly exactly how
this one works the idea was that there
was an underlying Engram model from
which some data was generated which had
you know Engram transitions so like
basically given if you look at the the
binary sequence at the top you could say
given the last four values in the
sequence if they say it was 0 0 1 1
there'd be a certain probability of a 0
coming next in a certain probability of
a 1 coming next and then you generate
from that and repeat right if it's a 4
gram and essentially the task then is to
look at that sequence and work out what
the Engram probabilities are and thereby
use them to predict what's going to come
next and they're sort of there's an
optimal way of solving this problem
there's an optimal kind of Bayesian
solution you start off with a you know
sort of a uniform prior that everything
is 50/50 you know every transition is
equally likely and then you update that
with each new piece of information you
receive and when we trained NTM on the
same problem we found that you know it
wasn't quite optimal but it came pretty
close if you look at the the NTM outputs
compared to the optimal one so the
middle row compared to the top row you
can see that they're a pretty close
match whereas an LST m network was much
slower to update these probabilities and
those two arrows represents those two
red arrows at the top represent
something something important that I'm
afraid I cannot remember it's been too
long since this paper it was something
that highlighted the difference between
what n TM and Allison are doing but
anyway more important than that is that
this and this kind of goes back to what
I said about explicit attention
mechanisms being good for analysis good
for interpretation
so because we could see exactly what it
was looking at in the memory we could
infer the algorithm that it was using to
solve this problem and it is indeed
learning a little algorithm which was
that it used specifically specific
memory locations to store variables that
counted the occurrence a particular
Engram
so you know it looks at how often it's
seen zero zero one one how often it's
seen zero one zero one and so on
it basically just has a look and that's
essentially all you can do with this
problem is you store a lookup table and
if you've seen zero zero one one a lot
of times and zero zero one zero not so
many times then you say okay usually I
think that followed by zero zero one
there's a higher probability of a one
than a zero and it it did this by
adjusting every time it sees the same so
these green arrows I think show the
points points where it's seen the same
Engram again and it has adjusted this
value in memory and then it uses it
looks up so if it's if it's if it's used
if it's used this memory to store the
the count information for a particular
Engram then when it tries to make a
prediction
following that Engram sequence it looks
up the same place in memory and uses
that to determine how likely it is to be
a one or a zero I hope that makes some
sense to you the point is that it's
generic so I mean for this particular
problem there's no there's no reason at
all right the Bayesian one is strictly
better it's just that you know what we
have here is an end-to-end system that
receives nothing with inputs and outputs
so we don't have to tell it anything
about engrams Bayesian probabilities
anything like that right it just learns
that directly from the data so I mean
all all of these algorithms right these
are all trivial copy end times copy I
mean you know there's they're not
difficult algorithms to program yourself
or - you know - and firth I mean you
could also through some form of you know
program inference you could find them
quite easily the challenge these are
more like sort of unit tests to say well
if we have a system that's completely
generic has no sort of inbuilt bias or
has no it's not true to say there's no
inbuilt bias but it has no kind of prior
information about these tasks
still learn them just from end to end
trainer that's my sort of inference of
what it's doing that there's nothing
model it's a model doesn't see that yes
yes exactly there's all the same model
it's just you know they're just sort of
pest so can it learn sort of simple
basic algorithms they're you know the
feeling was if it couldn't do those then
it wasn't going to be I mean long term
you know what you'd like this to be
useful for is more like high-level
cognition tasks things like navigation
for example like you've walked into a
room and there was something you needed
to remember there and you need to use
that later those sorts of things or for
I mean there's that there's a sort of
notion of a real-world algorithm so
something that sounds really trivial
like you know tell a robot to put a red
box on top of a green box and then say
look now put three more yellow boxes on
top of the green one that's a sort of
simple algorithm right and you want it
to learn that as an algorithm so that it
can generalize so that if you switch the
number of boxes there change the color
it doesn't then need a hundred you know
training examples before it gets it
right so it's more it's more of it like
an issue of generalization than anything
else and it also learned a simple sort
algorithm here where it so the the the
the idea here was you gave it a bunch of
random inputs and you gave them a random
order and you gave them a priority and
had to read
order all of the inputs to put them in
the order given by this priority this
priority sort and it did this with a
kind of line sort which so it's it's it
learned it didn't learn you know it
didn't learn quicksort I didn't learn
some fancy sorting algorithm it learned
something very simple that worked within
the range on which it was trained but it
still was learning how to sort these
these vectors okay and here's a little
video showing like what goes on during
training so this big multicolored thing
on the right is the memory where the
size and the color of the of the squares
shows the value of the of the
information in there
so maybe I'll just go back start again
and these the pink and blue heads are
the read and writes the
locations of the read and write
operations when they're more or less
blurred that means they're more or less
focused on a particular point it's hard
to explain these things as they're
running i'ma say so what happened there
this is for the repeat copy task once
the network is trained it's at this
point here it now gets a new input
sequence in coming in at the top now as
that input sequence is going in the
right head scanned down and wrote all
the vectors one after another then it
has to the read head the pink one jumps
back to the start
reads through everything in order sends
it to the output goes back to the start
sends it to the output and meantime
while it's doing all this you can see
that all these vectors these numbers in
the background that aren't being used
for the tasks are just getting bigger so
the size of the little circle represents
the magnitude of the number and it's
basically using those background numbers
as a counter I believe and then you can
see when it's finally finished it makes
this one line at the bottom just really
really big it makes all the numbers much
bigger at once and that means that it
knows when it gets there it's time to
stop so it's doing all these things
that's doing iteration is you it's
finding things with Association and it's
learning some sort of termination
condition okay and so we then extended
this to what we call differentiable
neural computers I mean people have
asked about the name change it came from
as I said the the the Turing machine
metaphor was signed to feel increasingly
strained because we weren't really
thinking of it as a Turing machine
anymore it was much more like take a
computer a real computer like we've got
maybe it should have been called you
know a neural phone anointment machine
is entirely sure but you know take take
the computer you've got in front of you
and look at all the ways that that
accesses and manipulates information how
many of these things can we build some
sort of differentiable analog that we
could feasibly use to train a neural
network and so we added some ELLs and
whistles we made more complicated ways
of accessing memory sure maybe I'll skip
through these a little bit like these
details can get rather involved you know
the overall architecture look like this
it
still fundamentally based on this idea
of looking things up in memory using
Association was still the most important
part of this model but now it had this
extra way of accessing information one
of them was by memory usage so that's
actually quite useful it could say give
me basically I want to write something
to memory so just give me some free
memory right which is what you kind of
normally do on where you're used to
doing that a normal computer you
allocate some memory that isn't already
being used and it does this by keeping a
differentiable version of a free list
that tells you you know how how to what
degree is it using a piece of memory and
this information gets ultimate every
time it reads a piece of information it
can then decide afterwards to flag that
as being free right it's like it reads
it and it doesn't need it anymore it can
flag it and then reuse it later this is
something that we it hasn't really borne
fruit yet like the the the point here
was that for
you've got a lot of tasks where you you
might have a very long a very large
amount of information you want to
process but you've only got a fixed size
amount of memory so I was thinking of
sort of reinforcement learning or
something like that you're traveling
around in a maze you can't store
everything you know your frame rate
coming in might be you know 30 frames a
second or something like that you can't
be writing to memory 30 times a second
and just keeping it all around at some
point you've got to decide to keep some
things and get rid of others just for
computational reasons if nothing else
and so I kind of thought it would be
nice if you had a way of saying given a
fixed amount of memory let's let's you
know let's be able to actually allocate
and deallocate like you can at a normal
computer and so it works we tested it
you know it works and sort of the unit
tests we design but I don't think it's
really found a you know a home in real
world applications yet and the other
mechanism that we introduced here was
the one that we used to replace so the
the shift temporal shift mechanism in
the NTM felt a little bit brittle in the
sense that it really relied on the
actual indexing of the rows if you're at
row I you went to I plus one and well
for one thing that feels very underrated
shouldn't be something that really keeps
track of you know an index you know you
don't know you're not worried about
which neuro
you're looking at for a particular
memory and for another thing it could
actually over time I think it could lead
to fragmentation problems which is kind
of a weird thought but you know if
you're sort of forces you to to allocate
all of your memory and large contiguous
blocks of information so once you start
having this more like system where you
can dynamically free and allocate things
then and you want to have large blocks
then you're getting into you know you
know operating system design territory
right things can get complicated and we
didn't want to design a neural operating
system so the the thing we use to
replace this location shift was what we
called temporal links so it just keeps
track there's a matrix that keeps track
of where you wrote to previously so
basically anywhere you are on the matrix
you can you can then jump directly to
the next place you wrote to after that
or the one you wrote to immediately
before that so that if you wrote down
end things in temporal order it doesn't
matter where you put them in the matrix
you can get back that that order of n
things and so this sort of comes back to
this idea again we want associative
lookup but we also the problem with pure
associative lookup is it tends not to
store temporal information and that
temporal information is like if that's a
very important part of human cognition
you know often will remember something
let's say you know a particular color
particularly of clothing someone is
wearing will remind us of something that
someone was wearing before but then
we'll also remember a kind of a window
around that earlier memory like will
recall a sequence and we're recalling
both according to associative recall and
according to temporal kind of con
temporal continuity let's say again it's
just this idea of like with neural
networks in general more as more if you
can give them more ways to do things
more ways more pathways to process
information more ways to manipulate
things then they can always learn to use
some and not use others when you train
them they generally just get better if
they're more flexible okay I'm not going
to go into these details it's just too
it's too technical this
this little thing shows the the test we
did for memory allocations so could we
could we give it lots of information
such that it had to store things in
memory but it only had a small memory it
had to do it had repeatedly copy things
we'd give it this little the little
white blob at the top left is an input
sequence and then it has to do this copy
algorithm where it will copies that and
writes it out again then we'd give it
another one and another one after that
without erasing the memory so it had to
learn to free up the memory so that it
had space to keep on doing this task so
it's kind of a trivial thing but it
showed at least that it can use this
there's memory reallocation okay and yes
this is the temporal link matrix which
is pretty complicated in practice you
know what one thing that we were really
keen on is can we use this to process
graphs so I think graphs are like a
super general super interesting kind of
eight F format so if you think about a
recurrent neural network is designed for
a specific kind of graph which is a
chain right it's a sequence where
everything is linked to the thing that
came next and linked to the thing that
came previously but generally you know
there's lots of different graphs we can
think of that can be used to represent
information so in this slide here for
example we can see where we've got a
London Underground map that's obviously
a graph you know there's there's there's
links and nodes stations and lines a
family tree is another you know form of
graph where you have the links being
different forms of familiar
relationships but there's many more you
know abstract things and that you to
think about you can your knowledge graph
right there's a reason that you know
knowledge representation generally
relies on graphs because you have you
have links between entities you have the
notion of you know someone having been
in a particular place at a particular
time or something like that you can
define almost anything any any form of
relationship you can you can think of
you can put it into a graph and so we
thought it would be cool to have a model
that wasn't just limited to processing
sequences but rather you could give it
any kind of random graph and it could it
could operate on it and it can learn all
this end to end so the
although the architectures were very
similar between differential neural
computers and they're all Turing
machines the emphasis of the papers was
somewhat different here we were
interested in you know processing more
complicated relational structures rather
than finding algorithms so much and so
we did a bunch of tests where we trained
the data on random graph so this bunch
of grapes looking thing in the top-left
is a random graph with random somewhat
semi localized connectivity and then the
idea being we would give it problems
like traversal so traversal this says
start at a particular node so all of the
nodes and these random graphs had unique
identifiers which are just vector is
being fed into the network and of course
there's identifiers to show which nodes
are linked to which and we would say
well start at a particular and and the
links are have identifiers as well I
think yes and and you would train the
network with a task like start a node
and follow link a link B link see now
where do you end up so it's just a
traversal on the graph it sounds really
trivial but actually that form of
question has quite important
applications so to go back to the
knowledge graph example you know there's
these there's this whole set of problems
in knowledge graphs you can ask a
straightforward question like who is
Obama's daughter and if there's a a link
in the graph that encodes that
information it will be able to find it
but if there's a few levels of
indirection who was Obama's daughter's
best friend and like a high school and
and you know where were they born or
something like that there isn't going to
be a single link encoding that
information rather you're gonna have to
follow a few steps so you're gonna have
to do a traversal in the graph and so
finding a general architecture able to
do that actually has some some
real-world applications but so did you
know just to demonstrate it we said well
look if we took the London Underground
that's just the specific graph so we
haven't trained this on Transport graphs
on you know it hasn't seen the London
Underground in train hasn't seen
anything that corresponds at all to
transport or anything like that it's
just completely random graphs but if
it's really learned how to parse graphs
how to
manipulate them how to perform these
kinds of tasks and essentially parsing a
graph means getting the whole graph
structure in some completely random
order putting it all in memory and then
being able to operate on that memory
such that you can use the the structure
of the graph if that makes sense so it's
kind of like in some sense you're
sequencing the graph but the sequence is
random and now it's just a whole bunch
of stuff written into memory that you
then have to traverse and find your way
around in order to answer the problems
anyway in the case of London Underground
the question becomes something like
start at Oxford Circus for the central
line then circle line circle line circle
and so on and make all these steps and
where do you end up and you know it
could answer those ones no problem a
slightly more challenging kind of well
actually a lot more challenging task is
find the shortest path between two
points in the graph so really what you
want to do is to you know solve solve a
shortest path algorithm which you know
we know we can do for arbitrary graphs
in practice it didn't solve it didn't
solve the algorithm right it didn't find
a sort of optimal path finding solution
it found something much more heuristic
that in a way is more like what a person
would do like when we we analyze how the
network was behaving it would kind of
look it's so it's being given this you
know the question the form of the
question is here's the start station
here's the end station now give me the
route and it would start looking it
would start at you know either end of
the sequence and look around a little
bit look a few steps around and attempt
to kind of find a way to join them up
which I mean is not as I say it's not
not unlike what a person might do
looking at the same problem but it
certainly wasn't you know optimal in any
sense but it could do up to maybe length
length v or length 6
you know paths it did a pretty good job
and it could find these you know these
these routes on the London Underground
another set of problems that we wanted
to look at was inference and again this
is something that has real-world
implications for for for question
answering for knowledge graph and so
forth and so we set up the the inference
problems again we trained it all on you
know on random graphs
form of the problem would be that you're
given a you're given an implicit link
which is not stated in the graph itself
in this case maternal great-uncle has a
specific meaning which is a essentially
a traversal right it's a chain of
relationships in the graph your your
fathers and mothers brothers whatever I
can't remember but but it's never
explicitly represented in the graph it
just has to work out what that means and
then so in this and and I should say
again during training there's no notion
of you know great uncles or
grandmother's or anything it's given
completely random labels with completely
random links that have particular
meanings but then you know to illustrate
it we took this family tree example that
makes sense to us as people and we could
ask it who is Freya's maternal
great-uncle and it can answer by
following this inferred chain of
relations and this slide shows so you
know the the analysis gets a little bit
more complicated this complicated slide
shows how it does these these things how
it reads these graphs again it uses a
combination of all of the these
[Music]
attention you know memory mechanisms
that it seen so far I think I'll try and
simplify things a little bit essentially
it gets given the graph in the form so
it's being given triples like Oxford
Circus Tottenham Court Road on the
central line that's what the first the
the very top left that's what the triple
at the very top left means and that's
encoded just in some you know there's
some there's some identifiers for Oxford
Circus there's an identifier for
Tottenham Court Road and there's an
identifier for the central line that are
just kind of randomly assigned and it
gets all of those in order it writes
them all down to memory so this this
heat map staying in the middle of the
bit with the green squares shows where
it's writing to memory is very
straightforward it's what it's doing is
saying give me a blank piece of memory
now I'm gonna write down the next triple
from the graph and so on until it's
finished with the whole graph then it
gets the query which is another sequence
it writes down the whole query and then
it starts to answer so the first thing
it does uses Association to go back to
the start of the query and then it kind
of gets interesting because it
has to start using this associative
lookup for key completion so given
something giving an incomplete query
like from Victoria follow the Victoria
line north it can look up it can sort of
specify two-thirds of a triple use that
as a lookup and the thing that will be
closest to that is the thing that
completes the triple right there's only
one thing in its memory that has start
destination Victoria and line Victoria
North and the complete triple gives you
the destination as well so this is
another illustration of how powerful
associative lookup can be you can use it
to fill in the gaps of your missing
information and it basically does that
successively to solve this problem okay
so kind of running out of time now
another branch you know we talked about
introspective attention write this idea
of selectively attending to memory but
there's actually a lot more you can do
just with visual attention than what
we've seen so far and so from back in
2015 this very was become a very
influential paper but Carol Gregor and
others at deep mind the idea was to use
Gaussian filters to read from input
images and draw to a canvas so actually
this has a feeling a lot like neural
Turing machines but it's more like
instead of you know an abstract memory
matrix or a tape that you're
manipulating rather there's a piece of
paper that you're drawing to and you
have these draw these these operations
that are the the correspond to more like
more like visual manipulations of the
data so more specifically what you have
an attention your attention mechanism
here is parametrized as a bunch of
Gaussian filters that you convolve the
data with they have the filter variants
they have a location for this the center
point of these there's these this grid
of filters and there's a great stride
and an intensity and this sort of the
images on the right here show what
happens with different settings of these
parameters so like the top one you can
see because of the the grid stride has
been made relatively small so like the
in some sense you're cropping out a
small part of the image although I
should say that this is
soft attention against and none of these
operations are hard there's no like hard
cutoff there's rather like a soft
convolution between a filter and an
image so it's all differentiable but
basically now you're focusing on the
main this is the center of this this
this letter v and it's quite blurry
because you've got quite a high variance
which is shown by the thickness of the
green line this other one is more like
kind of neatly cropping out the v itself
with quite a low variance with sharper
and so forth and this is what a
visualization of what happens when you
use draw too so in this case it is it is
this shows how so the way draw was kind
of used it was trained like an
autoencoder like you'd give it a input
image and it's able to to look at that
it's a variational auto encoder I won't
go into the details that maybe you'll
hear about those elsewhere in the course
but basically it sort of has the image
in front of it and it builds a
compressed representation of that image
and then from the compressed
representation it builds up this
internal canvas that it's going to use
to reproduce the image so this first
animation whoops sorry
yes the first animation shows how it
attends to the image and you can see
that it sort of starts off looking at
the the image as a whole things are
always too fast you just slow down all
our videos it starts off with this green
square that shows his attention looking
at the image over the hole and then it
shrinks it down and kind of traces out
the image you know like a stroke which
is very interesting and then it kind of
does the same thing when it generates
right so the drawing the actual
procedure used to draw to the canvas it
does it something something that looks
like a person following a stroke and
this is what it looks like when it
generates street view house numbers and
also know you know these these are
generated images they're very convincing
I remember when Carol first you know
download this everyone was really quite
speechless you know it compared to
generative models now perhaps it doesn't
look so striking but at the time it was
really quite quite ahead of its time and
it showed how you know these generative
models could be used to to really create
things that you you can't distinguish
from real images and and you know in
this case it was learning this kind of
right to laugh sort of sweep through the
data so again this sort of comes back
again to the thing I said at the start
about turning static images into
sequences and draw everything as a
sequence is a sequence of fixations you
know these Gaussian convolutions are
used to kind of read or process the
image and then in parallel a sequence is
used to build up your reproduction your
your reconstruction of the image okay
and just okay I think we're kind of out
of time I think maybe I'll I'll skip
this last slide which was an extension
of draw which kind of went beyond just
having this Gaussian grid and considered
more general affine transformations
rotations and scales and things like
that but it's the same basic principle
is kind of like a visual a specifically
visual form of soft attention here's a
demo of interaction is kind of nice I
mean this shows how it can it can it can
rotate images and transform them in some
kind of non linear way oh yes and he did
in 3d as well so there's 3d end this and
I was like it's kind of
ridiculous the lengths we go to to keep
on using em this for all of our all of
our problems okay and that's end of the
lecture just to summarize selective
attention is it's important for people
it's also important for neural networks
and I think it's important will only
grow over time these systems get more
and more sophisticated I expect to see
more attention built into them they
always have some kind of implicit
attention and you can you can kind of
query this you can analyze this just by
looking at the sensitivity but you can
add explicit attention mechanisms on top
you can have stochastic ones that are a
kind of reinforcement learning like
where you have a just a you know a
stochastic sample over your next glimpse
or attention or you can do this soft
attention which is differentiable and
can be trained with backpropagation as
well as access and maybe the last and
most important thing is there's lots of
different kinds of attention right we've
got we've seen content we've seen
location visual temporal there's lots
more and I think more will keep on keep
on appearing it's still quite a quite a
fertile area of research and that's the
end of the lecture thank you
[Applause]