DeepMind x UCL | Deep Learning Lectures | 6/12 | Sequences and Recurrent Networks

today I'm going to be talking to you
about sequential data and how much
deals with this type of data structure
before I begin I'm someone who likes
people to understand what I'm talking
about
so if at some point during the lecture
something's not clear or you have a
question please raise your hand this can
be also a conversation so I should be
getting the message across to you if
this is not happening then that's my
mistake and I need to know about it okay
so there's no stupid questions literally
just raise your hand and ask so with
that out of the way to give you a little
brief overview over what I'm going to be
covering today I'm gonna cover I'm gonna
divide the presentation and through
three different broad sections we're
gonna start by motivating why we care
about sequential data in the first place
why is it an important thing we should
consider in machine learning then from
there once we are convinced that this is
something we're interested in we'll
think about how do we train models that
deal with the sequential data what are
the challenges what are losses what are
we optimizing and so on and hopefully
give you the fundamentals on this once
we have our train modeled in section two
we'll move over to section three where
we take a train model and actually use
it to do cool stuff like generating
sequences and I hope to cover not only
different applications throughout this
section but also maybe introduce some of
the more recent concepts and methods
that have been applied to sequence
sequential processing and machine
learning
so we'll start straightaway by
motivating why sequential data just a
quick recap what do you have guys have
done looked at so far in this course is
first of all feed-forward neural
networks which are networks that take in
vectors of a fixed size that should have
been covered and you've also talked
about convolution convolutional neural
networks which are networks that usually
deal with images and have these things
called convolutions that they just share
across an image for example and apply we
in this lecture we want to move away
from fixed size vectors and images as I
mentioned to sequences so what even as a
sequence to give you a bit of a formal
definition a sequence is a collection of
elements that has certain properties let
me give you the example of a sentence
you can think of an English sentence as
a sequence of words so a sequence first
of all has elements that can repeat so
if you think about a sentence you can
have the same
appearing several times in a sentence
assent and a sequence in a sequence the
order of the element matters in a
sentence this is the same this is also
the case so in a sentence it's very
important in what order the words come
and it can mean something completely
different if the words are in a
different order and finally sequences
can be of variable length so they can be
any number of elements to them and this
also applies to our example of a
sentence so English sentences can almost
have an arbitrary amount of words in
them so with these properties if we look
at the methods that you've looked so far
we can see that they're not very good at
dealing with data that have these kind
of properties so this one here they're
the feed-forward neural networks they
usually deal with only fixed size inputs
which clashes with our need for variable
length and also they don't really take
structure that much into account
convolutional neural networks again
really can deal with non fixed size
inputs but this structure is not the
right one so how do we deal with this
and the question is how can we develop
models that actually deal with
sequential data because this is a
problem and given that this is so hard
you might wonder why do we even care
about sequences right why do we care
about sequences and my answer to that to
you is well the sentence why do we care
about sequence itself is a sequence and
as a matter of fact each of those word
is a sequence of letters and going
further this whole presentation is the
sequence of life so I hope that with
this I convinced you that sequences are
literally everywhere I set the example
earlier of sentences and the English
language as sequence of words but you
can also think of other things as
sequences so speech waveforms is a
sequence you can think of videos of
sequences of images images themselves
can be sequences of pixels we can have
more complex sequences so if you think
about programs you have different
executions that you carry out
sequentially and finally for those of
you interested in RL and decision making
that as well is a sequence of decisions
that your agent has to learn so it's
fair to say that sequences are very
universal and they
span a very big variety of tasks and
this is why in machine learning we're
interested in being able to deal with
sequences well okay so some of the
motivation I hope they take from this
that seek what a sequence is so it's
this collection of items where order
matters and that is a variable length we
know that sequences are widespread not
only across machine learning but across
applications and everyday life really
and finally the methods that we have
been discussing in this lecture series
so far are not enough to deal with
sequences so we need to come up with
different methods that can take all of
this into account are we clear up until
here is the motivation clear we know
what we're doing cool okay so now that
everyone's convinced that sequences we
want to focus on how do we do it in
order to think about a machine learning
model I'm sure you've had this in the
previous courses one of the things you
have to develop is what is my data what
is my loss what do i optimize and so on
so let's go through it a bit one at a
time I'm gonna first revise what you
guys should have already learned this is
supervised learning this is how I would
train a normal feed-forward network so
if we look at this the data that we
normally have in supervised learning are
pairs of some inputs and some outputs
for example you could have images and
labels for the classic classification
tasks what you're trying to do then is
you are gonna assume there is a mapping
from your inputs to your outputs this
means there's some mapping between
images and labels that you want to learn
and we're gonna in this particular
course because we care about deep
learning we're gonna try to approximate
this mapping with some nonlinear
function that is parametrized by theta
so in this case this means we're gonna
have a neural network that learns to
approximate this mapping which with some
parameters that we can then tune how do
we tune these parameters so we're gonna
define some loss function whereby we say
okay the output of our neural network is
going to be compared to the ground truth
label for example of an image and then
we're gonna take whatever our distance
measure is and that's going to be our
loss this is how much we penalize our
prediction with that Network so once we
have this loss what we're going to do
then is just
minimize it using standard back
propagation and update the weights of
the network as a function of whatever
loss we have is this clear this should
have been good when we think about
sequences this is a little bit different
so in a sequence let me take the example
of an English sentence this applies to
any type of sequence but I find thinking
in terms of sentences it's quite
intuitive because we deal with sentences
all day long so let's take that example
we're not gonna have pairs of
necessarily of targets of inputs and
outputs instead we're just gonna have a
sequence so X is just gonna be for
example our sentence and what we're
trying to model it's not necessarily a
mapping between inputs and outputs we're
trying to model how probable is a
sequence because we want a machine
learning model that generates likely
sequences if we are interested in
generating or that can estimate how
likely an English sentences we don't
want it to just be generating garbage
sentences so what we're actually using
our neural network for is to estimate
the probability of that particular
sequence the rest looks fairly similar
so you can just optimize the lock
probability so the probability of that
that you off the sentence that your
model generates and you can just
optimize in this case we're doing the
max because we're dealing with
probabilities but you can also take the
negative and minimize so dad would be
the difference is this clear cool yeah
do you what do you mean tag yeah and
then this would be more of a supervised
learning problem yeah absolutely and you
can always add information and you can
change your loss yes
exactly yeah so in a sequence and we'll
talk about it in the in a minute about
how we actually calculate this
probability but you can always add
additional losses so for sure yeah I
think if you wanna that would be almost
like a separate task that wouldn't be as
much as like estimating the probability
of a sentence more like like you say
tagging a particular word so in that
case yes I would go to the former loss
the moment you have inputs and outputs
you're going to be doing something like
this
sweet cool so let's think now okay what
we're trying to do here if we're trying
to learn this function over here as I
mentioned that measures the probability
of our sequence okay so how do we go
about that what is a good model that
we'll learn the probability of a
sequence so let's take the sentences
modeling word probabilities it's really
difficult because if there's one thing I
want you to take away from the lecture
is this sentence so I'll try to
subconsciously fit it in and modeling
word probabilities is really difficult
so one way you could think one very
naive approach that you can take to this
is say okay I'm just going to apply the
simplest model assume that the words are
all independent and what I'm gonna do is
I'm gonna look at some corpus that tells
me how probable is each of the single
words in the English language right and
just say okay the probability of the
entire sentence it's just the
probability of the individual words
multiplied with each other and that's a
perfectly valid model if you want so you
would take the probability of modeling
time step over your word probabilities
and so on
and it'll give you some probability of
your sentence now this model while it's
fairly simple and easy is not a very
good model of language and the reason
for this is that language has structure
so one example of why it shows how this
is a really bad model if I ask my model
this model what the most likely sentence
is it's gonna tell me it's the the the
the the because this is the most likely
word and it's just trying to optimize
for that so this shows us that the
English language is just not independent
words that are thrown around right there
is clearly structure in the language and
this model is not capturing it so the
question is how can we incorporate this
structure into our model okay let's go
to a bit of a more complicated model and
marginally but it's fair enough we're
gonna take all of the previous words in
our sentence we're gonna call that our
context and then we're going to
condition our probability on everything
we've seen before
and this kind of makes sense if you're
thinking about a sentence the next word
kind of depends of what you said so far
so it's a fair enough assumption it's
definitely much richer than just
assuming independence so what does this
mean in this particular example I can
give you this part of the sentence
modeling more probabilities is really as
the context and then I can see condition
on this which are X's 1 2 t minus 1 we
are going to calculate what is the
probability of the X which is the word
at time fib the time step T which is the
current word that we care about
so that would be our target and then
there will be different probabilities
for different words so difficult we
would be really likely because as you
know modeling probabilities is really
difficult hard is also a good candidate
may be fun and definitely not easy so
you would have this conditional
probability that describes what the next
word is this of course would only model
a single word but if you wanted to get
the probability of the whole sentence
the whole the way you would go about it
is you would first calculate the
probability of the first word and then
you can multiply that with the
probability of the first word is the
second word given the first word which
is everything that's seen so far and
then the third word is the probability
of the third given first and second and
so on so you keep doing this calculating
this joint probability from all of the
conditionals that you've seen so far and
this is a method that works very well
enough if you use this it does give you
a good overview of the structure but
there's a pretty big problem with it let
me show you an example to give you a bit
of an intuition imagine we only care
about modeling Peale second word given
first word that so that is the simplest
and shortest we can do in a conditional
that are ready for four words which is
not much gives us a four by four table
with all the probabilities of the
possible combinations of first and
second word four words
the English language has a few more than
four so if we look at this and
increasing maybe to 120 this is starting
to look pretty big and maybe an English
language has more like 10,000 words and
that's very limited as well so this
table gets huge if we're comparing the
probability of all words given all other
words and keep in mind we're only we're
only really looking at the probability
of one word given another word if we're
looking at context of like English
sentences this is likely to be much much
longer and if you think about it this is
gonna scale really really badly with the
size of vocabulary so however many words
your language has which in this case
here I'm sure I'm just tense
^ however long your context is this is
more than atoms in the universe for a
reasonably long sentences so there's a
it's fair to say that this is not a
table that we can store work with or
even really approximate with the data
that we have of English language out
there so while this type of model where
you are conditioning on the context is
useful and captures some of the
structure it's definitely not scalable
and not beyond very short time horizons
one way that people went around this
especially in the early and off here
research is that they said okay rather
than taking into account very long
context because we know it scales with
the size of the context the power of the
size of the context why don't we fix the
window size of that context so these are
called engrams and standing for however
long your horizon is so imagine you only
care about the two previous words those
would be two grams and what this
essentially means is that once you have
more than two words in the past you only
really care about the last two so you
only care about the probability of
whatever word you're at given the
previous two time steps the benefit of
this type of model is that again they do
as we said this scales with the ^ the
number of context points so this kind of
reduces it and you have a fixed size but
as you can also imagine you're losing a
lot of information like if I just tell
you is really difficult is really that
is not telling you anything about what I
really am asking like maybe difficult
would not be your first choice of word
after just the words Israeli so there
are yeah so having said this and
Graham's alleviate the problem of
scalability a little bit but there are
some downsides to it as well the two big
ways one is the one I just mentioned it
doesn't really take into account words
that are more than just n words away so
you very quickly lose the context or if
you take a large context you again have
the scalability problem and the data
table even if you reduce the context
size is huge and just to give you an
impression of how big this is I'm
showing you this blog post from Google
from a couple of years ago where they
essentially release a data set of n
grams of size 5 so this is not very
large keep in mind
five words of context in a normal
sentence doesn't give you a lot of an
idea of what's going on and what they
did as well is they only took engrams of
size five that appeared at least 40
times on the internet 40 yeah so 40 like
that takes away a lot of engrams and
despite taking away all of these they
still ended up with one trillion five
word sequences that they got off the
internet so this is how many we're
dealing with when actually thinking
about Engram and again this is only five
so this does not scale really well so in
summary to summarize a bit what we were
thinking about modeling these work
probabilities is difficult and it scales
really badly with with a number of
elements that you're considering for
your context so the question that we
then ask is a machine learning
researcher is can we learn this
probability estimation rather than
having to get it from some big big data
matrix can we actually learn it before I
move on are there any questions so far
with engrams why data sequences are hard
to model cool okay so I come back to
this question can we learn how to learn
can we learn how to estimate this
probability and the question is of
course yes otherwise the lecture would
be over and we're gonna talk a little
bit about how we're thinking about doing
this if you tackle this problem one way
of thinking about it is you're gonna
need two elements you're gonna need the
first element which is going to take in
your context and somehow summarize it
and then you need the second element
that from this summary predicts you a
probability distribution over what the
next word should be okay so I'm gonna
treat those two a bit separately the
first component is this I'm gonna call
it some function that is gonna we call
it vectorize the context it essentially
what we want it is to take in our words
process them in some way or other and
output this H which is going to be just
a tensor or a vector anything and this H
we wanted to capture whatever context
information we're getting from these
words that we've observed
so basically we're trying to approximate
the probability of all of the contexts
with this one H
should replace this over here does
anyone you normally don't ask questions
into the audience but let's try have any
ideas of what would be good properties
that we would want this F to have so we
want to F to somehow summarize the
context and I've told you what sequences
are important for any suggestions
whatsoever there's one very obvious one
if you think good one
very good we need to have variable input
so it cannot be like a neural net we're
going to expect a fixed size vector it's
gonna we don't know how long our
sentence is gonna be over context so
yeah very good we need to somehow still
keep the order and like this notion of
order because if I gave it some other
order of these words the context is very
different so those two things are very
very important I'm gonna give you the
other ones because they're maybe less
obvious
so order matters very good very well
then good learner but obviously we're in
a deep learning course so we care about
things being differentiable one thing
that is maybe not as obvious but when
you start working a bit with a sequence
s it becomes more so you want individual
changes to have very large effect so if
I just change one word in a very long
context it can actually mean the
opposite in the English language right
so you want a model that can actually
capture this the deep learning way of
thinking about that is to just have very
large networks lots of nonlinearities
that somehow capture these very bizarre
boundaries and in high dimensional space
and finally another thing that will
actually touch on a bit more later we
wanted to preserve long term
dependencies because language depends
not only on the previous three words not
even on the previous sentence all have
been saying so far has depended on
almost the first sentence I said in the
lecture right so you need to be able to
remember that as well very good so this
is the the first part looking back now
I've mentioned one method which was the
engrams how do they do on these kind of
desert arada this is a just a little
reminder of what an Engram was in case
someone was asleep two minutes ago
essentially the way we're thinking about
engrams is that there is a function that
just concatenates the N previous words
that's all it's doing it's not doing
super brain so the order matters
it does in a way because you're only
caring always about the last few ones
it's not variable ranked by definition
it is not differentiable you're not
learning anything they pairwise I'm
calling pairwise encoding when I say
like single words have a big effect
because that's how you work pairwise
comparing single words it doesn't have
that obviously and finally it most
definitely does not preserve long term
we by definition are cutting it to a
very small amount of words so that's not
a super great model one that people also
often think about when aggregating
sequences together is to just add
everything together
sorry we're basically you're just
thinking of F as a big hump so they just
take all the words lump them together
and say this is your context this one is
actually complementary properties to it
to the Ngram order does not matter you
lose at the moment you're just clumping
and everything together however it deals
with variable length I mean you can add
on as many items as you want that
nothing stopping him you can
differentiate through it does not have
the pairwise encoding and it can
preserve long term because you're just
literally preserving everything by
adding it together but it you can
arguably say that it's not a very smart
model because it's essentially
recreating the first example that I
showed you where all the words were
independent and there was no real
structure so we said that's not a good
one
in in this Laurent ones or in general so
this I think this goes back if I
understand correctly more to the first
part where we were saying we have some
sort of corpus like a probability table
that tells that not only how frequent
are words but how frequently are word
combinations so that was this P of x1
given P x2 so that would be the table
and this is a very intuitive way to
think about it but like combinatorially
it scales quite badly that was that was
what we were mentioning so what we're
hoping is that these methods actually
somehow learned that and like learn
implicitly they should be learning how
often do I see the word cat on the mat
like how often do I see those together
versus not right so that's but again
it's the metric
does that answer your question good cool
so this was part one we wanted to take
our context and just encode it in some
representation age now we have our
context somehow summarized vectorize
what do we do with it
so we now want a second function that
takes this context and just produces a
probability distribution our desiderata
for this are much simpler I'm not gonna
delve too much into them but basically
again we want these the fact that
individuals changes should have a big
effect this translates from before and
and also the fact that it returns a
probability distribution that's really
our only kind of concern and there's a
fairly simple to do you can just throw
sigmoid in there and it should do the
job
cool so in summary and grams and other
simple methods that we've been talking
about don't really need these
requirements for modeling sequences as
we saw in that table with all the
crosses and the ticks so how can we
build deep networks that actually meet
our requirements is it so clear so far
what our requirements are and why so far
the methods we've seen are not super
great at dealing with us cool so I'm
gonna move on to probably one of the
most important model it's in sequence
modeling in which I'm sure many of you
have heard of which are called recurrent
neural networks recurrent neural
networks are a type of neural network
architecture that of a specific
structure so they have this hidden state
H which is stores information it's going
to be a state that we're gonna keep
modifying and stores information about
what we've seen so far so the way this
would work is we're gonna have
initialized H so our state
initialized with zeros whatever you want
it to be and then you're gonna give it
some input so a word of your sentence or
whatever the first element of your
sequences and we're gonna update H to be
the next H to some particular function
okay so far so good
the way we're gonna update H is actually
quite simple there's gonna be a weight
matrix that are some of the parameters
we're learning then we're gonna multiply
the previous state with and there's
gonna be a second wave matrix again
learn parameters that we're gonna
multiply with the input so we multiply
one with the previous state one with the
current input and then pass it to at an
age so things don't go too crazy and
then we just get our next state so so
far all we've done is we started
somewhere and given that we've seen a
single word we updated our internal
state okay why is this useful it's
because this basically so far we've
summarized our context like I said that
is step one now we're doing step two we
have a summary of the context which is
this h1 at the moment and our we're
gonna do is multiply it with another
learned weight matrix and that's gonna
give us our output probability
distribution so this over here is going
to be a vector that is going to be a
probability distribution so all the
values are gonna be between 0 and 1 and
all of them are gonna add up to 1
and it's gonna overflow words that can
be in the English language there's gonna
tell us this one there's gonna be the
most likely or whatever the probability
distribution is we're obviously going to
use the softmax here as I mentioned just
because then that's going to ensure that
we have a probability distribution if
you're familiar with the softmax okay
cool so once we have our prediction
we're just gonna take the next word in
the sentence so forget about that we're
just gonna take the next word feed that
in update our current state to get the
next state there we go and feed the next
probability distribution and so on and
so on and the weight so we're
essentially doing this we're one word at
a time feeding it into our state
variable H updating H and doing this for
however long we want there is literally
no restriction on the sequence length
because we can feed in arbitrarily many
words over time and there's also no
there's also a sense of order because
the network should learn to process them
sequentially and therefore keep some
notion of what the order of the words
or does this make sense is everyone on
board with our nuns cool the way they
are often also shown are intense they
can be summarized as such basically
where you're again showing the input the
output and this hidden state and you're
looping over it with the different words
this is just the normal diagram people
also talk about unrolling RNs so
whenever you're coding it and you want
to back prep through it you're gonna
have to unroll it and we'll get to that
in a minute and it essentially just
means you're taking it and spreading it
over the difference time stuff cool so
that's our model we're gonna discuss
quickly the loss how what the losses and
how we optimize I don't know how many
here are actually interested in the math
see details I know there's also people
from the public so I'll try to point out
what the high-level messages are but if
you want to follow more closely the
equations are on here but please don't
be intimidated they just look sometimes
a bit mmm scary whether or not so how do
we train this we haven't really talked
about the loss yet we've only talked
about what we want from a model and how
we implement the model so how do we
actually update these weight matrices
that I show you
so what we're currently doing in this
task if you think about it it's almost
like a classification task where we say
our input is going to be the context and
our target is going to be whatever the
next word is right until for
classification as I'm sure you've also
already learned one of the very the
normal loss functions is to just use the
cross-entropy so essentially just saying
I'm gonna predict whatever the
distribution is over I'm just gonna take
the probability of whatever I predicted
my model predicted as the next word and
then multiply it with a real next word
in that sentence and I'm gonna
effectively add it up over all of the
words in that sentence and that's gonna
be my loss so the loss is actually
fairly straightforward
and the loss is gonna as I said depend
on some parameter theorem which are
going to be the three weight matrices
that I mentioned earlier
the one that creates the output the one
that takes on the input in the one that
just updates the hidden state now
differentiating bear with me okay I'm
gonna
very quickly touched on why
differentiating it's a bit different in
Ireland's and it has been in the classes
you took so far at a higher level it's
just different because we have this
recursive loop in the middle that's
basically all you need to know if you
don't care more than that if you've won
a few more details let's just really
quickly recap what the equations are
that we're gonna be dealing with first
of all we have just the state update so
given some eight how do we get to the
next one and as I mentioned we're gonna
just multiply with some weight matrix
the previous state and then the input
with another weight matrix 10h easy how
do we predict this y which is the same
as probability of the next X we just
take the softmax over some weight matrix
with the current state also easy and
that's the loss that as I mentioned the
cross entropy that were just taking with
our prediction and the real news were
now we have three parameters that we
need to optimize for this this and this
I'm gonna start with the one that's
easiest this one is very easy
these two are a bit more complicated and
I tell you in a minute why so let's
focus on just W want WI is fairly easy
to to differentiate because when we're
actually just expanding and doing the
chain rule we can see that we're just
Duran XI ating Y which is this one here
with respect to W Y and that's the end
of it if you want to do the math this is
what comes out of it
fine you're just differentiating and
using that to update it gets a bit more
tricky when we actually look at these
other two variables I'm gonna focus on
WH for a second and what the problem is
when here is once we start when we
actually look at the final term of our
chain rule over here so we're
differentiating this with respect to
this so far this is fine but then we
also have this term here and this H has
come from another equation that also has
the same WH in it if this makes sense
because you have the same and this is a
bit clearer actually if you think about
like this because your H depends on
another on the same way matrix again and
again and again and you have it over
time you cannot just differentiate with
respect to like that one variable and
that's it it's a bit more complicated
and this is called back propagation
through time which sounds really fancy
but all you're doing is you're
essentially unpacking this recursive
iteration that you've been doing
the last intimidating equation
essentially all I'm trying to convey
with this and you can almost just
visually see it is that you're gonna
have this is gonna actually break it
down in your explicit derivative and it
should probably be your plus there and
and something that depends on the
previous weights and then this itself is
then again gonna split into the explicit
one and some weights and then this
itself can then be broken up into two
components and this and this and this
and you go like this until the first
step okay so all I want you to take away
is that you need to unroll over all the
time steps you cannot just differentiate
once and this is a somewhat more this is
a summarized version of that equation
you can plug this summarized version in
and this is actually going to be your
objective so all of this to say is that
this is a bit different and we're I'm
new actually dealing with recurrent
neural networks differentiating is not
as simple obviously not is with tensor
flow and other libraries this should be
much more straightforward and they do it
for you and they honorable in time but
it's good to be aware that this is a bit
different than the things you've been
dealing with so far cool was that more
or less clear intuition wise we're all
on board good yeah okay one of the other
issues beyond just having to like
differentiate in a somewhat bizarre way
is this thing called vanishing gradients
which effect which are a big problem in
recurrent neural networks and I'd like
to give you an intuition of why there
are a problem imagine in order to give
you this intuition I'm gonna make a very
very simple RN an so this is a very
gross simplification instead of having
an input so before we had the hidden
states and we had some inputs and
outputs we're not gonna care about those
for a minute we only care about the
hidden states and the weight matrix it's
just gonna be a single scalar for now so
we're just updating it with a single
scalar and that's about it so we're just
taking a state and updating it with a
scalar many many times like so
essentially if we look at HT so at some
point HT is going to be this scalar
times the previous time step which
itself is gonna be the scalar times the
previous time step and so on and so on
so you can also rewrite this as HT is
gonna be this initial state times this
scalar to the power of however many
times steps you took if that makes sense
so you just
multiplying that one scalar over over
over over over again and then times the
initial state it's clear from this it's
because you are multiplying this with
itself so many times that as as if WH
some of this scalar is bigger than one
your state will go to infinity because
it will multiply itself over time and
grow as the the more time steps you take
the more it will grow and grow and grow
so H will get to go to infinity if it's
smaller than one it'll go to zero what
does this mean if the weights of your
neural network deviate too much from one
thick radiance and the valleys are all
gonna explode and go everywhere which
you can then not learn from does this
make sense intuitively okay good if I
lose you honestly just raise your hand
and I'm happy to go over this again I've
actually made a small example mm-hmm
to clarify this also a little bit
because some of you might claim ok Marta
that's fine but this guy here is
normally bounded by a sigmoid right so
it should be between 0 & 1
the H should not go that nuts and that's
true we do do that like that usually
there about it but what happens is that
the gradients are still affected by this
even if the values of the state
themselves are gonna stay between 0 & 1
the gradients are going to go to 0 so
what I've done here is I've written a
little code where essentially I'm
running exactly this I am updating the
state as I mentioned to you H is going
to be just the 10 H so it's bounded by
some scalar in the previous state so
very simple and then here I'm just
calculating the gradient which I can
calculate in closed form because it's
such a simple thing just calculating the
gradient and I'm plotting here if you
can see the values this light purple
line is the values of the state as I
change the value of the weight so I can
choose different weights for my weight
values for my weight and whenever I'm
around one the values are reasonable but
as soon as I go away from 1 or minus 1
or 1 this is gonna go to either 0 or 1
which is fair enough at least it's not
going to infinity like it was doing
before but if I then look at the
gradients actually that's a different
story the gradients are actually zero
everywhere except for quite near the
region of minus 1 and 1 and this is very
useless because of machine learning when
deep learning we depend on the gradients
in order to update our models right so
if we have 0 gradients there is no
signal coming through
parameters are not going to learn
anything so basically we're just saying
if we take too many steps and our
weights are not near one we're gonna not
have gradients and this is a bit and
this is one of the biggest problems that
our Annan's are facing because what this
essentially means is that you cannot
take many steps you can only update it a
couple of times before your gradient
spanish to zero so if we look at now on
our own our list of desiderata how well
our names are doing order matters that's
really good and it's very important and
variable length so those two we achieved
by design it is differentiable again by
design it doesn't do this pairwise
encoding yet and crucially it does not
preserve long term dependencies because
like I said we have these vanishing
gradients that are going to make that
I'm gonna force us to only look at a
certain amount of steps before our
gradients go to zero does this make
sense so far
yrnn start getting this type of score
nice good so basically just to summarize
recurrent neural networks can like I
just said model is very Bolling's and
also trained by propagation but they do
have this vanishing gradient problems
which is a problem for a modeling long
term and maybe I haven't convinced you
guys just yet why long term is so
important so I'd like to give you a
little example imagine I give you your
your model a language model and I give
you this sentence I said finally Tim was
planning to visit France on the final
week of his journey he was quite excited
to try the local delicacies and had lots
of recommendations for good restaurants
and exhibitions his first stop was of
course the capital where he would meet
his longtime friend Jean Pia in order to
arrive for breakfast he took the early 5
a.m. train from London to and then if
you're a good language model you should
look Oh friends and also capital and be
able to tell me Paris it's probably the
most likely answer to this and this is a
huge context like if you look at this
and it's not even that incredible you
are this is several sentences ago here
and this is a very very long sentence as
well so it is important to keep even
single words that are quite far away to
keep these long term dependencies in
order to be able to make accurate
predictions and this is just a small
example you can imagine if you're
writing a book sometimes things depend
on stuff that you've said in Chapter one
so it's it's
long-term dependencies are very crucial
for language modeling and RNs are
definitely not able to like capture this
so how can we do that
good are we so far so good our announce
are good we like our Nance losses okay
optimization
we're not intimidated by me okay good
we're gonna move to Alice TM you're like
our Nan's plus plus and I'm sure many of
you have heard also about long
short-term memory networks short for ALS
TM for short and in order to explain
them I'm gonna just very quickly show
you this model of an Ironman because
your should be familiar with it now so
we have the previous state H we passed
some input combine those and passed
through a sigmoid next date good because
the next one is going to be more in this
same structure so an LS TM is this this
can look quite daunting the first time
you look at it but we're gonna go
through it step-by-step and it actually
is not as bad as it looks the first
thing to point out is that not only does
it have the state H that we had earlier
it has this long term state see over
here that it has in parallel to this H
state here so we're keeping those two
internal states cell states they usually
afraid so and in addition to this
addition to this cell state it has a set
of gates that can be learned that
modulate how the input is or is not
applied to those internal states let's
go through them independently because
they're less daunting this way the first
gate we take into consideration is the
forget gate they forget the gate the job
of the forget gate is given your current
input and your age what do you need to
forget or erase from your long-term
memory right and all you're gonna do is
you're gonna combine those possums
through some network and then pass them
through a sigmoid which means the values
are gonna be between 0 & 1 and if
somewhere there's a 0 it's gonna be
multiplied with this and just erase
whatever was there because it's
multiplying it with 0 if that makes
sense so this is a gate that it's just
gonna basically regulate how much of the
previous information is allowed to pass
through and how much we forget then we
have the input gates now that we've
erase stuff we should add some new
information again based on our current
info
and some the previous state we are going
to combine those almost in the same way
that we combined this so basically we're
just creating some pseudo cell state
gating it with the same mechanism of 0 &
1 to see which information we actually
want to pass and we're just adding that
we're adding that to the cell state and
that basically updates the long-term
memory okay so in the first step we
erased in the second we add new
information and then finally we
obviously need to update this state here
because this one also needs to be
updated and this is going to be updated
again with our input but also with the
long-term state so that's also gonna
have a saying in what's get to state in
our shorter term memory and this then
looks a bit less intimidating if you
think about it in the forget state the
input gate and the output gate and by
using these mechanism of gates that are
all learned the model can learn how to
store information that is relevant for
later on without having the vanishing
gradient problem I'm sorry and this is
also why LS TMS are essentially at the
core of sequence learning so up to this
day whenever you are dealing with
sequential data in any capacity usually
LS TMS are there they pop up everywhere
so I'm sure you guys have heard of them
and it's because they're actually very
reliable at doing exactly what we would
just set ourselves out to do this is why
LS TMS are good we more or less
understand the gating mechanisms and so
on yeah in this case it's because you
have a lot of additions and you're not
only just multiplying so you have that
from as far as I understand it's because
of these gates how the information flow
you're not just multiplying the same
value with it on and on and on I see
that's from my intuition about it
because essentially because you have
this long term I'm struggling expressing
what I think because you have this long
term module over here when you even when
you're differentiating you're before you
were multiplying and essentially you're
this was going to zero but because
you're influencing it via this Plus
these gates are basically stopping the
vanishing gradients from going and
destroying everything
if that makes sense yes
I'm sorry so GRU yes I think that so
this is one way of showing it in the
diagram I think essentially what this is
trying to tell you is that this here is
gonna have the same shape as this
because we're just adding it on so
you're essentially saying first the way
this is normally done is first you
calculate some sort of new cell state
but instead of adding that in their
youth you almost have to like your own
little forget gate over here and you're
multiplying these two so you're
forgetting part of the order like you're
regulating how much of this new cells
that you actually want to pass and then
I mean you could also you could put this
two in one box but this is actually how
it like the two intermediate steps there
I carried out if that makes sense
there's also dr use which I should
mention which were actually developed
much more recently which you can think
of as like simplified versions of an LST
em they have the megiddo mechanism is a
bit simpler I'm not gonna go into
details but if you guys are interested
they're on these slides and you can just
go back to them they're a bit simpler
and I think it depends a bit on the
particular applications they train
faster but sometimes LST ms are a bit
stronger so it really depends what you
should be using but cool so if we
consider LSD MS and I guess gr youth
they have very similar behavior to our
nm so of course order matters they're
blanks differentiable great none of this
pair was encoding but it does preserve a
long term so this is something that
we've now gained with Alice TM so we're
slowly moving up the tick list which is
really good so in summary for this
little section L STM's and gr use
overcome the gate vanishing gradient
problem with this gating mechanisms that
I mentioned and as a result of being
able to overcome this they're actually
everywhere in machine learning research
and I'll come back to that in a minute
cool are there any questions about any
of this hi
yes yeah both of them actually have kids
it's just a different make AD mechanism
which arguably is a bit simpler so they
only have two gates one that they call
reset gate and one that is updated and
it's just the way they deal with the
with the gating so I mean for example
they also don't have this long stream
state so it's just a single state but
again they don't have the vanish
ingredients because the way that the
information interact is not just like
multiply multiply multiply so you don't
get that same problem but there's no
need for this long-term C cell state
basically they're probably half your
parameters and they might be quicker to
train and so on so there it's and
they're simpler so there also are might
be simpler to implement and so on so if
you write down this is simpler update
then the gating mechanism of lsdm which
is a lot but LS CMS are more involved
but tend to be maybe a bit more robust
in experience yeah cool any questions
other than that sweet cool we're doing
good for time yeah sweet so we're gonna
focus on generating sequences so so far
what I've been talking about is okay we
have a sequence model how do we train it
and now that we have a train model let's
assume that's all over what can we do
well we have trained our model using
this cross-entropy
to be able to properly predict the
probability of a sequence but that's not
a super exciting application if I tell
you I have a model can tell you how
likely a sequence is in English
great that's usually not what people
train these models for a more exciting
application is to use it to generate
likely sequences right now that you know
it knows how to see it probable as
sequences or not it can also generate
them and that's why these models
actually are quite interesting so we're
gonna focus on generation how would you
go ahead and generate so just looking
back at the RNN I'm doing this on an RNN
but this cell could also be an LSTA more
or any other recurrent mechanism if you
take the RNN from before i could and i
wanted to generate a sentence i could
just feed in my first word say modeling
this
then output the probability distribution
as I mentioned earlier and then instead
of just picking the next word in the
sentence because I have the ground truth
sentence what I'm gonna do is just feed
the most likely word or sample a word
from the distribution and feed that as
the next input so it's Auto aggressively
creating a sentence so that's gonna give
us the next probability and so on so we
can just feed whatever out would we get
into the next and so on and that's gonna
generate our sentences or sequence so
I'm gonna go through several
applications and several examples and
usually in each of those I'm trying to
introduce something new in this
particular case I've been talking a lot
about language of sequences I'd like to
talk about images the sequences of
pixels and this is a model that appeared
actually now a while ago 2016 I guess
which was a generative model for images
called pixel marinum and essentially it
was treating images as just a sequence
of pixels which is total a total valid
way of describing an image right if you
think about it so very much like before
when we were modeling the probability of
a word given all of the other words so
far we're not pro modeling the
probability of a pixel given all of the
other pixels that you've seen so what
would the color of this one being given
that I've seen all of the other ones and
this is more or less how it worked in
action so we would start by just having
some distribution over what the first
pixel of an image is and we can sample
that over there and we just get a dark
pixel and then to sample the second one
we're just gonna sample what's the
probability of the second pixel give
them that first and we do for the next
and so on like the chain rule like we
did in the first one and we'll just
advance along the image slowly and
sample always the next pixel given the
previous one and what's interesting
about it is that the model then learns
from the context of all other previous
pixels it learns what the probability
distribution of colors is for the next
one right so in this particular case
it's very likely to be a green pixel and
that's why it's mostly centered and
quite spiky on that but if we look at
this example over here maybe where it's
not clear assuming this is a bird is
this is this already where the bird
starts is the Stila grass what is
happening there so it's more uniformly
distributed across there was the
possible pixel
and what's interesting is that depending
on the context it's just changes this
distribution quite different quite quite
a lot between those different contexts
and this generates images which like I
think if you're far enough look very
realistic but nowadays obviously a state
of more state-of-the-art models produce
quite good images this was at the
beginning and even though they're not
perfect and some of them are quite
blurry and bizarre you can see that it's
learned some sort of distribution over
what natural images should look like
especially considering that you've only
been conditioning on the upper half of
the images which is was very good at the
time this was quite impressive and of
course you don't need to always
condition in the order that we just
explained which is this pixel by pixel
this is one of the nice things you could
really condition on any order I this is
not language where we need a specific
order of like first word second one you
could imagine also conditioning and
defining your order to be by clusters or
any other order up and down or whatever
as long as you define an order after
that it should learn to kind of use that
which makes them quite flexible pixel
areas do those make sense and how we can
yeah yeah so I don't know that I'm
necessarily about better because it
could learn to really well predict the
second picture from the first like
imagine it learns the super good
approximation for that so I don't know
that it would be better or worse I think
and if you were to sample different
pixels probably the variability is much
higher at the beginning because given
like the first few you there's a lot of
images that can't come out but the
variability once you've committed to
that bird image you're like well that's
probably gonna be green at this point in
the grass right so I think and that's
what was reflected also in how like
narrow or not those distributions were
right so rather than saying more
accurate I think I would say more
certain of what the next pixel should be
like if that makes sense
cool sweet okay cool so that's that
another well we've already talked a lot
about natural language but it is then
the the most obvious application
that always comes to mind when you think
about sequences and in particular
obviously we've already talked about
Ardennes and Alice TMS but I'd like to
talk to you about sequence the sequence
models which some of you might also have
heard of because they have a lot of
applications as well in industry so if
you think about the RNN model that I've
been telling you so far again we have
these hidden states that get updated
with our input and produce some output
but initially I told you that this
original state h0 can be initialized to
be whatever like normally people just
make a vector of zeros that's fine
because it's your first date but this
doesn't need to be the case you can
really just pass any information over
here that you want your model to already
start with and half right so imagine I
could give it some additional context
that gets processed and passed through
our states and only then once it's seen
all of this context do I want it to
start generating and you might ask when
would I ever want to do this and like
okay imagine you want to translate a
sentence right so it can give you the
sentence in the original language and
that's gonna be my input to the context
so all it's doing is its summarizing
this into there or in initial state it's
packing the whole sentence sequence the
sequence into the initial state and then
once it has that it actually you
initialize with the random word and you
tell it okay I want the target to be the
Japanese translation of whatever the
sentence is so it will produce this
apparently min sequence and then it will
produce two and in sequence based on
this is doing Auto regressive Li the way
you would generate a normal sentence but
it's not just generating any random
sentence its generating the sentence
that you gave it as context if this
makes sense and in sequence the sequence
have been very powerful so this way of
incorporating knowledge into a model has
been very useful and what makes it also
very powerful and this kind of opens up
the possibilities with rnns is that you
can really condition on whatever you
want and add information however you
want so you could think for example the
most simple RNN would just be this
network we're gonna have an input then
then hidden state and output fair enough
but you could also just imagine is
passing a single input and asking it to
produce loads of different outputs
sequentially or you could do the other
way around you could give loads of
inputs and actually only want one output
at the end of the whole thing
this is probably this is like the normal
iron and and this one is like the
sequence to sequence where you have
several outputs where it's not producing
anything generates this rate and then
actually starts producing outputs so
this is a very flexible way of thinking
of how can I use our enhanced and how
can I give them extra information
sequence the sequence of actually has
actually been used in a ton of
applications because of how flexible it
is so empty stands for machine
translation is at the top so it's very
widely used in all of Google systems for
machine translations image captions
which I'll show some cool examples in a
minute but also for a speed parsing
generating dialogue generating videos in
geometry so there's a couple of
references if you guys are interesting
but clearly this is very applicable to
like a very wide range of tasks the way
you can kind of think this is a yeah let
me give you an example of like Google
neural machine translation which is one
of the I guess more relevant
applications of sequence the sequence
word would also a while ago now and
essentially they would do exactly this
you would have some context in sum or in
some language some context language you
would encode this to be your state that
I just mentioned and then you give this
input in a specific order to the Alice
to the RNN down here which is just going
to produce whatever the output is and
this GN and Google neural machine
translation model actually per improved
performance on translation by a lot so
if this is the old models that were just
free space so frequency base that's the
blue line and then how humans are yellow
they're actually in pre this green
improves the baseline by a lot and
across languages as well which was very
impressive so this was a very good way
of just closing that gap between machine
translation which was really cool also
very cool and what you can do because I
mentioned this was quite flexible so far
we've actually been talking about okay
as a context I can just give some
language and then ask it to predict it
in a different language but why do I
have to give it some language in the
first place I could give it an image
right so what we can do is just take an
image pass it through some neural
network like a convolutional neural
network and to create my context state
and from there have it actually create a
sentence that describes that image so
this is for image captioning and the
examples of these type of image
captioning are actually super impressive
and one of the points that I want to get
across when I show you these examples is
that there is still a lot of variety
between the best model which would be
this one over here and I'm also going to
show you what the initial model said so
when you first train it without hyper
parameter tuning and so on so it's
clearly very sensitive to that type of
tuning so initial model we're saying a
large brown dog laying on top of a couch
which to a certain extent fair enough
but then if the best model is a small
dog sitting on a chair we can have a man
cutting a cake with a knife which not
really versus a man holding a sandwich
in his hand good a pizza sitting on top
of a white plate which is completely
wrong and a person is cooking some food
in a grill yeah and a close-up of a
person eating a hot dog versus a woman
holding a banana to her face which is
really good like if you look at the
middle sentence that's like pretty
impressive and that is just goes to say
that like in all of these it's still
quite fiddly and it still requires a lot
of hyper parameter tuning and it just
doesn't come out of the box but when it
does well it's pretty good so yeah so
this was sequence to sequence yeah yeah
yeah so well it could go either way so
here you could either train it from
scratch and just train them together and
it should learn to actually have a CNN
that extracts meaningful representation
often what people do is that they do use
pre trained models just because we can
argue that if you go online and get on
any of these Google net or whatever like
pre trained
Nets it's gonna save you a lot of
compute especially if you have a huge
CNN then you just plug that in there
subgradient and actually just update
this bit over here but it could work
either way that's just saving you buying
some time
I think that would be one of the use
cases I don't know that necessarily it's
been applied like I could see like when
I was mentioning you could cluster by
you can conditional clusters that could
be almost like what you're saying when
you have like very rough clusters and
you want to like discretize it like very
thin and increase the resolution it
makes sense so you could do that I don't
think anyone is gonna use pics learn and
to generate images like there's far
better generative models by now you use
gansler or anything else this was more
to show that we have models that can
create a sample in our model conditional
probability distributions and as a
matter of fact actually a version of
pixel iron and was then done in two to
four wavelet would that was kind of a
precursor for actually something
completely different which I'll talk to
you in a minute which creates audio
signals but it Peaks Lauren and as such
is not necessarily used that's just
maybe an example of showing how you can
do conditional probabilities yeah in a
rough way but like you saw the images
weren't phenomenal the same way where
you were trained on RNN so you would
hear you just the first one you feed it
nothing you can feed a vector of zeros
and then you expect the model to give a
pretty good first word which I really
would be something like the or a or M
right and you take the loss between
whatever it outputs and the actual
ground truth sentenced because you
assume that you have a label data set
yeah yeah exactly so it's so ever so
some still the cross-entropy
and you would just use whatever the
caption is of an image and just try to
recreate that sentence for that
particular one so that obviously is true
that then that is fixed to that one
sentence that goes with the image so
it's not super flexible obviously a
human could describe that one image in a
million different ways but you're
training it for that one particular if
that make sense so you're hoping that if
you see enough images of cute little
dogs eventually it will learn different
ways of describing dogs does that make
sense
oh there was a question somewhere where
there as well okay oh yeah sorry I
mentioned early and I stood up maybe
even more clear when I draw this the
this is just showing the recurrence this
is definitely an LCM for sure okay and
yeah this is a bit simpler than if I put
the different gating mechanisms in the
cell states and whatnot but all of these
are Alice tiems and I mean you could do
it with our announcement again like it
would very quickly not do super well so
yeah Alice seems good well paid
attention cool sweet audio wave so this
goes back to two wavenet and what this
has actually been used for more and more
efficiently and actually I take that
back because then they was changed so it
originally was used for that but then it
was changed with convolutions which is
exactly we're gonna talk about right now
so you can think of audio waves of
course also as a sequence and in this
case we're not gonna use an RNN we're
also going to use an LST em we're going
to use convolutions which is what you
guys have already talked about in a
previous lecture and the way this works
is that you're gonna have your input at
a very high resolution and you're just
going to pass it through several layers
and hoping that it abstracts away and
then that's gonna produce your next
point at the low resolution if that
makes sense and you're doing this by
using increasingly large convolutions
and increasingly fewer of them and these
are called dilated convolutions and this
is a different way actually of thinking
about how to deal with sequences so it
is it has this fixed structure of a
convolution so it always will only look
at that horizon but it does still take
into account quite a lot of horizon and
at the
different levels of hierarchical
abstraction between those layers if this
makes sense and this particular model
which is called wavenet is the one that
was developed for the Google assistant
voice so whenever you chat to them
that's this model generating audio from
text and so it's actually managed to
really scale it up really well and these
convolutions work really really fast if
I compare these convolutions to all the
other models and we've talked to so far
they actually are pretty similar to
Alice TMS in that again order matters
variable lengths because you're shifting
these convolutions along their
differentiate both because we can learn
them and originally I had put this as
like a semi tick but it does preserve
long term because it has these very
dilated convolutions that look back and
then summarize everything into one point
but again it's actually fixed size stuff
like the past that you can look at
really so if there's something beyond
that you're not gonna see it
so give and take yeah so that's audio
another one that is also where sequences
are quite important our policies and I
don't know if anyone here is interested
in RL but wherever there's a policy or
any sequential decision-making of course
sequences play an important role and
this starts already in the in in for
tasks such as generating images so
you're not even RL yet for example we
can have models that sequentially decide
where to draw on the canvas or where to
focus on when paying attention to a
certain canvas in order to generate
images and those are all sequential
decisions that have to be made a really
cool actually
application or the vehicle model it came
out recently is spiral over here where
it is learning to generate images but
normally most models generate images
pixel by pixel and decide what color
they want to do that pixel this one here
is actually using a computer program the
draws and it sequentially deciding what
brushstrokes it wants to do so it's
drawing like a human would draw
essentially and it creates actually
pretty cool faces given that it's
obviously given only a few strokes and
this is trained on celibate so you can
see I've learned how to draw human faces
and every Bracton brushstroke is going
to be an output of our sequence
actions of course then there is big oil
applications we have the open ai5 that
where state-of-the-art in dota and again
obviously very heavily dependent on LS
TMS and being able to deal with
sequential input data there was alpha
star from deepmind which also a
state-of-the-art on alpha on Starcraft
and to go a little bit more into detail
actually this is what the alpha star
architecture looked like and again it
boils down to the same thing so you have
you get in some observations that are
processed in some particular way but at
the core we have an LS TM that is taking
those observations and making decisions
right so this was actually quite a
crucial part in the agent that that was
used for alpha star and then it would
output the actions and and whatnot so
that was RL or sequential
decision-making
I'm gonna move on to transformers for
the end is there are any questions on
the previous topic cool so finally this
is the last example that I want to give
is transformers I believe you're gonna
have a lecture on transformers
independently this is just going to give
you a bit of like a quick intuition
about it and obviously transformers are
all often closely related to sequences
so just to mention that a little bit
this is how I like to think of
transformers or how you I would explain
them you guys have already seen
convolutions and the way a convolution
work is you're going to have a
convolution made out of in this case
three weights and then those weights are
then moved along to form these
convolutions and it's always the same
weight so if you look at this it keeps
sliding along the image but it's always
the same weights that look too far that
move along transformers is a bit
different instead of just focusing on a
little subset of the images that the
convolution goes over it's actually
gonna take the whole input and all of
them but what its gonna learn is which
one within those which one to attend to
and this are these are reflected by
these weights so depending on how much
you want to attend to each of these
elements in the input your weight is
gonna be stronger or less strong
and crucially as you changed the pic the
point you're generating these weights
are going to change so depending on what
your current pointers in time you're
gonna actually be learn to attend to
different elements of the sequence so
that is the difference here the weights
don't change here they should change as
a function of the input cool and this
can be used for generating transformer
so usually the way it works in language
is that you have your input words and
you're gonna try to first create some
contexts and representation whereby all
of these words are interacting with each
other depending on some weights that
depend on the attention that is learned
so depending on how menteur not they are
you're gonna have different pairwise
interactions and then you're just gonna
with as with the lsdm use that as the
context and then start creating the next
word from the context you get of the
representations the embeddings over here
and everything and the the past states
and everything you can see it a bit in
the width of this lines these are meant
to represent how much weight the
individual elements have so that's what
that is meant to show and this is how
you can use this for generating language
as well and and what's really cool these
models are really good like transformers
really meant a big improvement for
language generation and I'm sure you
might have heard of GPT 2 which is this
model that actually came out last year
from open AI and it was essentially this
huge transformer based language model
that had 1.5 billion parameters to
optimize and it was trained on this huge
huge data side of life 40 gigabytes of
text data from a lot of websites and
which was really cool as this was a
language model that was not only
generating good language good sentences
but also adapting to the style and the
content of whatever this language the
the context you gave it once which I
find really impressive if we look at
some examples this is one of my
favorites so what they did is they gave
it actually I'll read from here they
gave it this sentence as the context to
start to start from and then they
predicted these sentences so the context
they gave it was in a shocking finding
scientists discovered a herd of unicorns
living in a remote previously unexplored
Valley in the Anne's mountains even more
surprising
researchers was the fact that the
unicorn spoke perfect English so this is
the sentence that was written by some
person that was trying to probably do
the most random sentences and then the
most predictions were for example the
scientist named the population after the
distinctive horn of its unicorn these
four horned silver wide unicorns were
previously unknown to science or now
after almost two centuries the mystery
of what sparked this odd phenomenon is
finally solved which is super impressive
if you think about the fact that this
sentence is very bizarre and that is it
is keeping up with the style of this
sounding a bit almost like a journal
journal text that has been produced
so the purpose in the sentence is it
creates are not only very long but also
they make sense contextually and as you
read on they also keep referring almost
to its own text and everything so this
is very very very next level text
prediction and if we think about if we
look back on our table again much like
the LSD emma2 fulfills the same
requirements but it also has this added
pairwise encoding that I mentioned at
the beginning where individual words are
actually affecting your compare ways
comparing every word and depending on
that weighing how much they should
influence each other and this is what
also gives them a very big advantage
just to look back I like looking at
these things and see how far we've made
it I want to compare two sentences the
first one was state of the art in 2011
which is not that long ago if you think
about it when this whole sentence
generation with Arnon came out and they
were not using really any context but
the sentence I produced was while he was
giving attention to the second advantage
of school building a 2 4 2 stool killed
by the culture saddled with a half suit
defending the baha'i idea for annals
office which sounds like it could be
something but it really isn't and but
it's it's it's at least the sentence
that yeah this was in the paper so this
was originally yeah wonderfully this was
a moment of pride which at the beginning
when iron ants came out this was really
impressive but now we look at what GPT 2
is doing this is an example it was given
the synthesis context Miley Cyrus was
caught shoplifting from uber chrome
on Hollywood Boulevard today and then
the model produced the singer was
wearing a black hoodie with the label
blurred lines in the front and fashion
police on the back which is insane like
this this is not only a much better
sentence when I went upstairs
it has popular references it matches the
style of the tabloid and so on so to me
eight years between those two is super
impressive the fact that this is where
we've gotten so this is quite exciting I
think there's obviously still a long way
to go and we're still only dealing with
sentences and we're not writing books
and whatnot but slowly I think we're
we're getting closer just a summary so
this has been quick actually I want to
kind of emphasize what we went through
and hopefully what should stay with the
in mind I started by motivating Y
sequences are important why they're
everywhere and that they really are
quite important to machine learning
research and why we care about them but
you should also be convinced that
modeling the probabilities of these
sequences is quite hard we cover
different approaches so from the very
basic and original ones like engrams to
more complicated deep learning ones like
our names and Alice TMS and moved on to
some examples with like dilated
convolutions for the audio as well as
transformers at the end and hopefully I
have convinced you that these models are
not only good models but they're also
fairly flexible because they're dealing
with sequences so they can actually be
applied to a whole range of tasks not
just natural language generation or
audio but actually to a wide range of
different tasks in machine learning and
with that I like to thank you and if
there any questions
[Applause]
yeah yeah so the question is to repeat
because it was quite quiet was whether
these type of models that we've talked
about can actually only focus on like
local consistencies versus if you write
a novel you want to have global
consistencies and actually storyline and
so on I think you would probably need a
combination of a few things so this
obviously has shown that it does give
you local consistencies but you could
imagine mixing this for example from the
top of my head with some memory module
and then that is storing something more
about this so at least you could be
consistent about more than more than
just a few paragraphs you could store it
learn to save some crucial elements and
then read from that condition on that
every time you're producing right so
maybe it would learn that there's some
characters and so on
you would have to learn how to make it
maybe more hierarchical you can see the
first sentences you'd basically add a
memory of two and verse now it like
really is writing paragraphs and I think
that is because you're abstracting away
more and more you just need to learn how
to do that so there's nothing saying
that you cannot learn how to scale this
up into higher levels of abstractions
we've not done anything like it's not
clear maybe how but it could be even the
same principle just like beefing it up
to a certain degree
yeah yeah so this is a problem I think
where that has more to do with the fact
that we're using deep learning and deep
learning it's very hard to like
reconcile with actually this whole
symbolic learning and this is actually
something that I've been really
interested in because you have to be
learning all you can hope is that it
learns a pattern from the data and if it
doesn't it's not gonna abstract it it's
not gonna reason of this higher level of
abstraction that you'd like it to be
like humans like we do right like we
cannot make any claims as to at what
level it is it is reasoning and clearly
like you said from this example it it
doesn't learn concepts it doesn't seem
to learn concepts the way we do and then
apply them and I think in that case it
would have to go to other methods if you
want it to actually understand concepts
to learn concepts first and then maybe
use this too then with these concepts
generate sequentially but for sure ya
know at the moment it's just yeah and
also because the idea in my head like
deep learning is very under the
representations we obtain and deep
learning are very under constrained
right you're letting the model choose
whatever it wants to do so it's very
unlikely that they're gonna by chance
end up being super symbolic and the best
way of actually summarizing whatever
you're getting
so yeah it's you'd argue could argue
that symbols are a very optimal way of
compressing information but maybe
usually neural networks get away with
compressing in a different way which
happens to be completely random and then
this and then generalize badly yeah
that's not a stupid question that's the
golden question and machine learning I
joined one of the hyper parameters were
so when I was showing you the different
the different image captions and I was
saying some are better than others that
is when we're choosing different hyper
parameters and that is not only the
learning rate that's also the size of
your representations how do you wire
your network up how many iterations of
this and that and you can have some
other additional lost terms and whatnot
so and the size of your hidden state is
one of those variables as well and it's
not very trivial because it's also been
shown that for certain problems not for
this particular one but sometimes you
can show that you only need a certain
amount of bits to encode something in a
neural network so you can you would hope
that you do a bottleneck that is small
enough it should learn it and it
actually doesn't so but whereas if you
give it a larger one you can have and
then reduce it that works so sometimes
it seems like during the training
procedure
it needs more capacity even though if
the problem at the end really doesn't
need that much information capacity it
seems that during the training process
often it does so it's very hard to tell
even if you know the final like how many
bits you need to encode anything even
that doesn't necessarily give you an
information usually you say bigger is
better and try to like yeah give it as
much capacity as you can but then
obviously it also trains much lower
yeah
so there are long story short this goes
a bit back to the question we had
earlier with the symbols I think because
there's not a lot of constraints on
these hidden states often they're very
entangled representations that are not
very human interpreter well in general
there are research groups that are
focusing on what information can we get
out of this and like looking at like
unsupervised methods of clustering and
seeing what the correlations are and
whatnot to understand like you say how
are they how are they working but it
feels that very often they're quite
specific to the one model you train and
maybe if you train a different model it
word work completely different even if
it was the same model just different
initialization so I think trying to get
this intuitive understanding we're still
quite early on like it'd be great but I
don't think we have it yet with this
type of network yeah yeah yeah you have
because you're comparing all to all
basically and that's another advantage
and this is also wide work so well
because you are comparing the words but
you're only really comparing them at
input level so it's not like you're
comparing the whole corpus of English
language you're just saying your
sentence which is a quite finite amount
of words versus whatever you're you're
producing does that make sense yeah so
it scales worse like you say yep
definitely comparing all 12 but
there should be I think transformers by
now have been applied to pretty much
everything I don't know particular
examples but I'm sure if you look at it
it makes sense because it learns if
there are any patterns in the time
series later it varied they very quickly
pick up what to attend to so if you have
any regularities and they would learn
these weights between the comparison of
word to attend to in there in the
previous sequence so they're quite I
mean yeah there has there have to be
we've tried a couple of things but not
that it's super relevant
you
you


242. Digital People Would Be an Even Bigger Deal

hello and welcome to session
242 in the aictu.com reading group
tonight we'll be discussing the blog
post digital people will be would be an
even bigger deal by halton kanowski
is perhaps most famous for founding
co-founding give will and open
philosophy philanthropy
in the reading group we've created
some of his
arguments both arguments against miri
singularity institute as they recall
back then and um again about ai risk and
he managed to convince uh miri that they
were doing uh
their organization in a bad way and miri
managed to convince him that ai risk was
indeed real
but that was back in 2012
this blog post is half a year old from
his personal blog called cold cheeks
there is a companion
faq which
we won't
discuss here
this is
approved in the context of a sequence of
posts
about why
this could be the most important century
of all time
and the text that digital people the
title the digital people could be an
even bigger deal uh the bigger uh refers
probably back to the word to the
duplicator thought experiment that he
has previously talked about in this
sequence although he never really
clarifies bigger deal than what
a digital mind to get someone into
issuing for it he suggests we imagine a
computer simulation of a specific person
in a virtual environment and he uses the
matrix the uh the movie from 2000 from
1999 um as an example for how to think
about this i'm not really happy about
this this is obviously fictional
evidence and we should be wary of
generalizing from that because in
particular in this case i feel that it
smuggles in a lot of extra assumptions
and
that are not really warranted and
there are many
ways you could criticize the matrix for
not being very realistic at all
the obvious comparison here is to uh
robin hansen's uh work h of m
which is uh
um like a rather thick book on just the
subject and obviously what he's
presenting here is something that is
much less
deep and also contains a broader class
of things that can happen whereas robin
hansen is just talking about one
specific scenario
one of the way the key ways that differs
is that robin hansen's emulations are
created through uploading whereas um
uh
things also about integers that are
somewhat more
less like
and the other thing is that they can be
duplicated because
the duplication of course has a lot of
strong implications
one of the things that i feel makes it
very different from reprehensive sport
in particular is that rob enhances age
of m relies to a very large extent on
different kinds of continuity between
humans and the uh the inns
and
if they are not created through mind
uploading and are more unlike us then
this continuity could be uh could be
lost and what we would see would be
much more unknowable in my opinion
has a comparison with normal humans uh
to give some
intuition about the the
things the characteristics that hogan
can actually consider is important
uh the
start with where they're possible today
and
arguing here that digital people
probably will be possible uh someday um
i'll be able to interact with the human
world and
he is just stating all right they will
be conscious and
i think probably here most people are
functionalist enough to just accept that
but it should be said that many people
would not just accept that
easily duplicated that's of course the
key thing that makes the
digital people different
they can be sped up potentially
and they can be branched and all these
kind of things that you can do with
computer programs that you can't really
do with um
with humans and that allows both uh
much greater productivity and
social brain and social science we'll
get back to precisely how that enables
social science
a much greater uh
degree of control of the environment
the ability
to have locked in that's
i should say that's something again
we'll come back to later
space expansion is something we'll
probably talk very much about and
finally whether they are good or bad and
for normal humans that's outside the
school of this piece
but for digital people that's either
very good or bad and i think actually he
is here using two different definitions
like when you say he won't comment on if
humans are good and bad that he actually
means something different than he does
in this call column here
where
will the fact that there are digital
people be a good thing or a bad thing he
argues he'll either be very good or very
bad
but if you take that back to this column
is it good or bad that there are humans
well then that should not be
that controversial to say that it's
probably a really good thing that humans
are here and
extinction is bad
one thing that is not present in this
world and that i would really like is a
comparison with artificial general
intelligence in
particular the old-fashioned view of
artificial general intelligence which is
the one where we just moved first out
and
it could be arguable perhaps with uh
holland kenosha's
rather open definition of um
of digital people that there is a
spectrum between an emulation and just
an ai
but it's not like there is a straight
line from human to revelation uh to agi
like perhaps a curved line when that a
digital person would probably have a
number of
properties that are not present for
either humans or agi
but still let's try to run with this and
see where is the
attitude person compared to an agi
i've tried to put this in a scheme this
is this is not from the article this is
my own thoughts on this
so one of the key important things is
what is the motivation of which and if
thinking a person is just literally an
upload of a person we would expect it to
have roughly human uh motivations
if however it's an agi the classic view
of agis is they have a simple utility
function
maximizing whatever and they
have some is converting instrumental
desires based on this
moral value is also an interesting
digital people probably would have full
moral value depending on your moral
framework uh a uh
an agi that is very human unlike could
perhaps be said to have no moral value
but that's a really difficult question
there's a sense in which
an agi
is
built
with the express it could be the classic
agi is built with the expected purpose
of being like an optimal bayesian
reasoner or an optimal agent in
different ways whereas a digital person
would not be designed towards the school
and this would
influence the recalcitrance mrs
bostrom's definition
of how difficult it is to make
improvements to the intelligence how
difficult would it be to make
improvements for a digital person i
think it would have moderate um
difficulty i could see a number of
rather trivial ways that humans could be
improved if you just have had the
ability to do that and a an agi that's
just written down like um
ten thousand lines of scrolls or
something like that seemed like
something that could easily be tinkered
to
to improve in some way
another difference is in the
distribution where
certainly robin hansen but also
i feel uh a lot of the implications uh
uh in the halton canopy's work seem to
point towards a rather decentralized uh
world
for digital people whereas the classic
view of ati is that it's very
centralized
and cooperation can humans cooperate
well
kind of medium
whereas the assumption is that if human
if the agi has
is like perfectly rational and has a
simple utility function then almost
agreement theorem will just say it will
cooperate perfectly and create a single
turn if possible
so the premises that
the key premise here is digital people
are just like us
except for a few
um
exceptions here that they can be copied
and run in different speed and embedded
in virtual environments
and the assumption the premise here is
that there are not other changes
another uh
thing that follows directly from this is
they're conscious they have human rights
and they can do most of the things human
can do
i think whether they have human rights
and whether they
are made in this way depends on of
course will there be a human who wants
to make them and i think that is a very
uh unclear thing um
i think a lot of people would
uh given the the uh if they were asked
about this they would say no we don't
want digital people and
the reason is that the negative
implications are really really clear
right if you can get
uh
some digital people who are clearly way
smarter than you and
anyway more people than you then it
seems very obvious that all existing
political um
groups will have their power deluded
potentially substantially this can also
be seen
in the existing politics where
uh in immigration for instance a lot of
people would probably uh object to
suddenly having a billion more people
inside their country having voting
rights and having uh political rights
that seems like something a lot of
people would object to
the
other flip side of the coin is that
competitive reasons
might make it impossible to avoid right
it's possible that some countries will
have imps and
you either have the choice of allowing
digital people or be out competed
um
so before we start with the uh the
actual implications of this
um uh i want to add the uh
key location that i have from the book
hfm and that is is will there in dp and
h of m will there be um
a world with digital people
uh that doesn't seem really clear to me
in the in the sense that it will uh it
seems very unstable to me
um people then it's probably also
possible to
improve them and given that you can
either choose to have them improve
themselves or
build more hardware or more things that
only directly relate to getting more
digital people um
then
there's a huge incentive to do precisely
that and not in any incentive to have
that spill over to the rest of the
uh world
if you can just improve them a bit more
then you can get potentially far better
returns on that
it's of course possible to
like
make up stories about why this in fact
turned out to be
stable like it might be that there is
like an intelligent ceiling somewhere
that it will just hit and then it's
impossible to improve further there
might be legal restrictions or something
like that but i think our default
assumption should be that there there is
no intelligent ceiling will just
suddenly public one bit too
so the uh
idea is shown in this uh animation below
for why they would have
a rather large impact is the fact that
they can be uh instantly and accurately
copied instantly is of course not
entirely correct for software uh so the
models are potentially rather big and
cannot be copied instantly but at least
very rapidly and
posin karowski is arguing that we could
see a doubling time of the economy in on
the order of months
the reason why
i agree that digital people could cause
this but the crucial reason why is
this as we also see with other machine
learning ones that it takes a lot of
time to train them and very little
resources to execute them and it's the
same with humans basically that teaching
a human takes 18 years but once you have
that human then the trait the
getting that human to work is
comparatively cheap compared to the 18
years
and of course when when you look at a
picture like the one
in below here then
the
strategic implications become really
really clear right and in order to have
a doubling time of the economy
all of this needs to feed into the
economy and it's not at all clear why
people would do that instead of trying
to get
some kind of strategic benefit uh from
digital people
so we have increases in speed and
variable speed as two examples of why we
could get higher productivity and and
having people temporary and
greater amounts of experimentation
but
there's an argument to be made that a
lack of body could be a sustainable
captain home cannot see things that's
probably and you're gonna do it
um and the key thing notification most
humans will probably be um
i mean you have uh fully automated minds
in australia
research and development manufacturing
energy all these things could probably
mostly be automated
social science is an interesting place
where uh sees a great potential for
digital people
uh
we have we haven't really seen the the
social sciences advanced in the same
place as natural sciences and the reason
according to how kind of use analysis is
that it's just too hard to make
experiments
but if we had digital people then we
could make perfect experiments and
that would make it potentially uh a lot
more feasible to investigate a lot of
things about humans um
there's of course a potential for
produce we'll get back to that
and paul karaski speculates that
we might find a social dynamic where
people
only want to uh trade with people who
have already
overcome all their biases
and this could potentially be a really
good thing a good man said
i i'm a bit less optimistic about this
because one of the boston's six
strategic cognitive tasks is social
manipulation and it seems really
almost truly easy
like one of the ways that this would
improve digital people would improve
social science the most would be
precisely in things like persuasion and
that seems potentially extremely
problematic
now
there's the control over the environment
this could be a very bad awkward thing
and hong kong says that um
we need effective
enforcement of basic human rights
um
i agree it would be nice but i'm not
sure that's something that's very like
which we are likely to get because
um if my assumption previously is
correct where um
well
sorry ordinary human politics tried to
ban uh digital people
and
competition might uh push the other way
then for this to reach some kind of
equilibrium where digital people are
allowed but give um get
human rights that seems like a very uh
specific and
very optimistic case
poland can actually also sketches some
more dystopian scenarios
one here that people could copy
themselves and be mean to the companies
that's a
short example because people could do
something else they could copy other
people and between two copies of other
people
and again this can feed back into social
manipulation in different ways blackmail
function manipulation and other evil
dystopian things
although also sketches some more token
scenarios
i'm a bit possible about this i notice
i'm confused
talking about dystopias make perfect
sense because then you can figure out a
way to avoid them why talk about the
trophies in this game
it's a motivation something we can
strive towards there could be political
reasons to think about what to actually
want oh it's just nice to explore the
future but
i'm worried about thinking too much
about all the wonderful things that
could exist in a world of perfect
virtual reality but like it feels more
like being given to me to be honest
specifically
there's another thing that might come uh
later i'm not
gonna go into it very much because i
think that's probably something that's
only going to be well when we are really
far past the civilization and it
probably doesn't have much implications
for what we should do right now
login is way more important
because it's easy to make in
societies of digital people being way
more stable you don't have death aging
deteriorating environment or shortages
of material goods or things like that
and indeed it would be possible to have
the environment itself enforce the
stability
uh like you can imagine a number of fine
great ways to do it
and
is a um authoritarian and peter decides
hey i should be the leader forever
and then the the virtual environment
could enforce this for instance by
saying that if suddenly the leader is no
longer the leader then we revert back to
the
last checkpoint or something like that
and that would potentially be impossible
to get around
it would still be vulnerable from the
outside
but holland kanowski is strangely
optimistic here in my view is if the
values that these uh people embody are
good then it could still be positive but
i feel uh doing a lot in of
very positive values sounds almost
always like a bad thing because
precisely specifying what are good
values in a way that you won't regret
seems really really difficult
robin hansen has actually written a bit
about this so this is not in the uh text
for today but he has written a
a
i think a blog post called will
totalitarian will take help
totalitarians
where robbie hansen is arguing that
the risk of login might be overblown um
and
one of the easiest arguments is that the
uh the the person the
the person who's in charge of this uh
simulation that the uh digital people
are running in might only be able to
read
the he could obviously read the thoughts
of the digital people but he might only
be able to read the shallow thoughts and
that might not be that much of an
advantage
um i
perhaps agree at least if the if it's
only very very shallow but i think there
is um no real way to get to the
assumption that it will only be shallow
mind reading i mean
that needs some kind of arguments
there is also the ability to hear and
read everything in full control of
information and um
that's something that um
are
somewhat present now in totalitarian
societies um but uh an emulation could
do it
in my opinion much more than propaganda
diseases in existing totalitarian states
another reason why it would be hard to
be um the dictator of and
one of digital people is that the
enforcers were you in this case uh they
may be lazy corrupt or rebellious
themselves
and i i agree it's a thing that could
happen but it feels to me that
for digital people it seems a lot less
likely than with with current i mean
current totalitarian systems are
sufficiently stable that you can have
things like north korea right now and it
seems to me rather obvious that the
enforcers in north korea are probably
lazy corrupt and rebellious but in a
digital world they might be
substantially less rebellious
there's an overhead with totalitarianism
and that might mean it might be
outcompete
that is true we are seeing that to some
extent from north korea but not enough
for the totalitarian society to
disappear and there might also be
advantages for totalitarian states
yes he makes an argument that it has to
be expensive because we are not seeing
any uh
billionaires doing things like that
right now
um
and
he is making an argument that practices
arrangements organizations and relations
are likely to keep changing forever and
that's an argument against this log in
that it won't be that stable if these
organizations keep changing forever
uh and i think yeah it's an argument
against logan to some extent but why
would you expect organizations to change
forever
i don't think for instance that
organizations change very much in uh
north korea uh compared to
or at least
compared to uh against the express will
of the dictator and his audience
but i'm not sure i need some kind of
argument here
one thing that robin hansen worries
about is that if digital people are
centralized in cities that does make
singletons more likely and world
government and easier defense than
offense are also things that
to robin hansen creates an increased
risk of
um
of a totalitarian
for digital people
in particular argues that
this easier defense non-offense seemed
like something you could have when the
technological field is very very uh
equal um
and that might happen at technological
maturity
so in total is this a good or bad thing
with digital people well it depends on
how it's set up and
yeah whether
it's probably behind the singularity
once we get to some kind of equilibrium
things will have changed so much um
there's a great argument that it could
be irreversible
and
um and with these things spreading also
faster
but holland kanowski is reasonably
optimistic in his conclusion
we could in fact
see a world of digital people that leads
to a much much better world
and i kind of agree except for the fact
that we have actually solved the
alignment problem right how this pushes
the alignment problem just a tiny bit
further away and perhaps not very much
even that because the emulations would
then either change gradually to watch
something that is totally unrecognizable
as humans or they will just build in a
super intelligence and then we haven't
actually solved the problem
that is all for today thank you and see
you in two weeks