Metalearning & Induction Heads [rough early thoughts]

hey
this is chris and today i want to talk
about how induction heads seem to cause
metal learning
this is preliminary work but i think
it's one of the coolest things that we
we seem to be discovering about
transformer circuits and so i'm really
excited to chat about it
if you haven't watched my colleague
catherine's uh previous video on how
induction heads seem to cause a bump in
the loss curve um you should really
check that out before watching this
video because this video isn't going to
make a ton of sense without it
now
meta learning is probably
at least to me it's it's the most
impressive thing about large language
models it's really um the thing that's
most surprised me
about them and about i think probably
it's
in some ways maybe the most surprising
thing about deep learning um in the last
number of years
um and in fact uh it's i think so
striking that um the gpd three paper
decided to go and make it
really the title of the paper
is that
language models are few shot learners so
the idea here is we
you know when we when we train models
normally we expect them to learn over
training but with these large language
models we're starting to see that you
can go and put examples in the prompt or
in the context and they'll learn as they
read so they can go and do later things
in the context that they they couldn't
do earlier in the context
and this has led to all sorts of
interesting things like you can go
and do so-called prompt design where you
go and you
train models um or you design these
these little prompts that instead of
training the model you can you can sort
of get it to do what you want by just
designing the prompt and in fact um
there was this little meme with uh andre
uh carpathi um that you know maybe maybe
the future of computing is that we'll
just design the prompt um i think that's
that's somewhat joking but and you know
i think the fact that we're sort of
joking about things like that sort of um
i think is it sort of striking about how
interesting a property um metal learning
is
um and it's deeply mysterious like we we
design neural networks so that they'll
be able to go and learn
but somehow when you train large enough
language models you seem to get meta
learning for free
um and so that's that's really
interesting
and you know i really want to know
what's going on
um so there's a previous paper um this
is um
the the kaplan at all skilling and
skilling laws paper um and i think it
has a really clever idea for how to
think about uh meta learning so normally
when we think about models learning we
can we can look at a loss curve um and
it you could think of this um you know
the curve describing how the model
learns over time so you know we have our
training step
um
and as we train the model
our loss
goes
down
and actually and in this car lost curve
it's for a two layer uh transformer we
can see the induction bump from the
previous video
so the idea that kaplan all have is that
you can also
go and look at this not as a function of
training step but as a function of token
index you can take the fully trained
model right at the end here and you can
ask um you know okay how good is it at
predicting the first token well you know
it's not very good at predicting the
first token how good is it at predicting
the second token well it's also not very
good at predicting the second token but
um you know so these are you know the
unigram statistics and then the bigram
statistics and and the entropies of
those but then you know as we go on
um
we start
uh to get much much later in the in the
prompt
and
and you've already sort of squeezed out
all of the the short term distances but
the the loss continues to go down
and and so if the loss is continuing to
go down
somehow that means we're getting
information from earlier tokens and
using it to go and predict later tokens
and so this curve in some ways it's a
little bit like a learning curve but
it's us learning
over the context it's learning from
earlier tokens in the context how to go
and predict later
tokens in the context now you have to
average over lots of examples to see
this but um if you average over over
lots of examples in your training set
and then just look at token index you
start to see this curve
um
and i think this is a little bit of a
grandeus way to describe it but
i'm not the one who came up with the
idea so i feel free to describe it that
way um you could sort of think of this
as a meta learning curve so you could
think of this as being your learning
curve and you could think of this as
being your meta learning curve it's
describing how you're you're going and
uh how your metal learning is
progressing
um
but you could think of both of those as
really being slices or projections of a
more general 2d picture
and so here we have
our model over training
so here's the snapshots that we saved as
the model was training
and we can also look at the token index
on this axis
so
um
if we go and we
average over this axis and project it
all down we'd get the loss curve
and if we take the final slice here
um
we get the
the
the the curve describing how good we are
predicting different tokens
um at the end of training
so this is this is sort of a
generalization of the the diagrams we
saw before and it allows us to get this
this nice 2d picture and see get an
overview of what's going on
and
and yeah so i think that the most
obvious and striking thing when you look
at this diagram or at least to me the
most striking thing is that there seems
to be kind of a discontinuity
um and we can zoom in on it and that
corresponds to
the bump that we saw earlier and that we
discussed in the previous video where
induction heads form so we can see
induction heads forming
in this
in this plot
so something that we can then do is we
can say okay
uh you know we'd like to somehow
summarize how good the model is at meta
learning into a single number that would
be a really useful thing if we could do
it
um and one way we could try and do it is
we could go and take you know how good
we are of predicting say the 50th token
um
and
we can look at that for different points
in training so we can write at the
beginning of training and then after a
little bit and so on
and uh
when we learn
about induction heads we see a little
drop
but not a very big one
and then we can go and take um how good
we are at
predicting a token much later in
training or much later in the context
and we can see
um we're better at predicting it and we
have a much bigger drop
and
if we subtract those two numbers you
could think of that as kind of being
the amount of metal learning we're doing
between token 50
and token
500.
um so this is in some sense a measure of
metal learning
and the the lower it is um the better we
are at meta learning
and we see that there's this really
abrupt drop
um which happens right at the bump um
right at the point where we think that
in induction heads are forming um so
that's really interesting and suspicious
um and i don't i don't know about you
but this is this is like you know this
is a a really sharp curve like it's
almost it's it's you know it's not
exactly like this there's a little bit
of maybe bending here but it's it's
almost just an abrupt um
uh discontinuity uh uh
i mean it's pretty discontinuous so
that's that's kind of an unusual thing
um and
uh yes seeing things like that uh it's
very striking to me
okay so
um one question you might ask is you
know how significant is this right like
how how big a deal is it that we're
getting um you know this negative 0.4
uh gnats of metal learning um our losses
is measured in gnats from from
information theory and
what does that
mean
and i think there's a few ways you could
interpret it so
in the case of this model it's an 11th
of the loss
and so one way you could think about it
is that means that when we predict token
500
um
it's almost as though we get to go and
predict like maybe every 11th token
and we're able to go we get sort of get
to for free just magically know the
right answer and say it with 100
probability that would cause the same
uh you know losing reduction of an 11th
of our loss
um
another
interpretation of it is uh
it's 0.4 knots per token
um but gnats are kind of a tricky way to
think about things um i prefer to think
about things in information theory in
terms of bits so let's convert to bits
that's about 0.5 bits a token
which is about
one bit
every two tokens and remember that a bit
um is enough information to distinguish
between two things so it ends up being
meaning something like you could
sample the model twice and take the
better token or take the better sample
every two tokens that would that would
be another thing that it would mean to
go and drop by 0.4 not so i think it
really is a very meaningful
very meaningful drop
and
and
uh you know we've we're already pretty
far out in the context by token 50 and
we're still getting that much more so um
that's pretty impressive
okay so another way we could picture
this
is
uh in some ways the important thing is
you know how much more information are
we as we move
to later tokens
how much more information
are we squeezing out and reducing our
loss by
and before we discover
uh induction heads we see that you know
we pretty early on
um you know it basically becomes zero
and we're not getting very much more
and then
all of a sudden so this is sorry this is
this is the derivative um with respect
to log token log because it seems like
um they're sort of
uh
yeah it seems like probably the
the order of magnitude of tokens that
you have before you seems to be the the
more important thing and if you look at
these curves they sort of seem very
linear on a log axis but not on a linear
axis so we'll take that with respect to
the log token
and
and yeah and it seems like before we
discover induction heads you know maybe
the first 20 or 40 tokens 50 tokens um
you're still getting pretty significant
learning or pretty pretty significant
reductions in your loss but it pretty
quickly levels out after after token 50
and as you go further down you're you're
not learning much
um whereas
uh after induction heads form we we
continue to see these really significant
um reductions in loss
um so that's really interesting
and then if we take a further derivative
and we ask okay well you know how is our
ability to go
and
increase so we have our first derivative
which tells us as we go this direction
how is our ability to put tokens um
going in and increasing and if we take a
derivative this way we can ask uh how is
our ability to go and predict tokens
um improving
our ability to
get better at predicting tokens how is
that improving as we go and we move this
way and we can see that there's you know
i mean you can just visually see it here
but it's nice to confirm there's this
extremely extremely um uh you know sharp
discontinuity that really really
dominates the story so you know to first
order the story of metal learning is
that in the span of a few snapshots um
which is really just a few hundred steps
of training and we go from not being
very good at it to being quite good at
it and that's that's really the story of
metal learning this is for a small model
and this is only for a
two layer attention only transformer
where it's really easy to study things
um but we can also look at large models
and large models also seem to have a
bump um it's right here
um
and that's that about and now we're
measuring in tokens and so that's at
about 10 to the nine tokens
and it turns out that if you go and
measure the same
uh notion of of doing a difference
between how good you are predicting two
tokens to go and look at get sort of a
meta learning score
and
right at ten to the nine tokens and we
see an abrupt drop
an abrupt discontinuous drop
and it seems to not matter that much how
big your model is um across uh you know
several orders of magnitude and they all
drop in a pretty similar way now before
right at the beginning the large models
are better at predicting than these
small models are but after the drop um
the metal learning scores really
dominated
just by which side of the drop you're on
which side is this bump um
you're on and not matter that much
how large your model is that's kind of
the opposite of what i usually think
about model size usually i expect model
size to just make everything better but
here we have something that
to a pretty significant extent is it's
just the same amount regardless of model
size so that's interesting
um in any case that's kind of uh
rambling a little bit but i think that
the striking thing is that this is still
happening in large models and so
the the thing that we we you know to me
that's crying out when i see this
pattern is what the hell was happening
when we see this drop now of course i've
sort of already given away the theory we
have which is that it's induction heads
but um you know we'll we'll we'll talk
about that more in a sec um uh but you
know i
i i think that it's it's really you know
we
and maybe maybe one reason why i'm so
struck by this is i don't know about you
but i'm used to seeing weird things in
law scares in fact this is like a pretty
smooth loss curve all things considered
and you know it's not that unusual to
have random bumps and quirks in your
lost curve um but here
this this thing that seems at first
glance like a relatively um you know
not that big a deal thing and and and
you know everyday phenomena turns out to
be doing something like really the the
most interesting property of models to
me um is is like undergoing this like
phase change at that point it's it's
dramatic um and so that's really
interesting
um okay so the theory that we're gonna
have is that this is driven by induction
heads now remember that an induction
head uh it's kind of doing a nearest
neighbor's lookup so it says okay this
is my present token and well actually
maybe i'll i might even include some
previous tokens um but let's just focus
on the on the present token for a second
um so here with the first paragraph of
harry potter because we i find it fun to
use the first paragraph harry potter and
and we have the token d
okay so now we're gonna go and look for
the token d
but shifted one forward so we're gonna
look at what happens next
um
and then we're gonna look at what
happened next and increase the
probability that that is the next token
so we did this kind of nearest neighbors
look up over um our present token and
surrounding context
and then looked for something similar
shifted one forward and used that to go
and predict the next token
um
and so this is this kind of nearest
neighbor
um search over our context is is really
the thing that induction heads add
and
and you have to remember that in order
to do this you have to have a previous
attention head that can go and do this
shifting so you need to have two
attention heads in your start cut to go
and implement this first you have to
have one attention that can shift
information forward
and then you have the attention head
that goes and looks uses that shifted
information to go and match something to
the shifted version
and then go in and increase the logic so
that's what an induction head has is and
it's very natural that something like
this could be used for meta learning
because it's allowing you to search and
find similar situations and then do what
happened last time when you had a
similar situation
um okay so the hypothesis is that
induction heads drive metal learning
and
um we're gonna have a few pieces of
evidence i think the big ones are
for small models
well small attention only models
we can really show that uh induction
heads are causing the bump and they're
causing metal learning
um
and then for larger models the story's
gonna be a little bit harder we'll still
be able to say that metal learning forms
of the bump um
and
we'll still be able to go and say that
high induction heads form at the same
time as the metal learning forms
but it'll have to it'll only be a
correlational argument so
um this does leave open some
possibilities that um maybe maybe
there's additional things that are
happening at that bump but that weird
transition
um and it could be the case that large
models are different so we can we can
sort of rigorously understand this in
small models in large models we just
have correlational evidence and it could
be that there's other things we'll talk
about that more in a minute
okay so
um
for
for small models for small attention
only models and one thing that's nice
is you can just sort of mathematically
define
induction heads because remember that
they have um
if you if you've watched our early
videos where we have this really nice uh
analytical framework for understanding
these small models uh an induction head
is one that has positive ov um output
value uh eigenvalues and also has
positive qk eigenvalues attending to the
previous token
so
um that's kind of a yeah that gives us a
nice mathematical definition and we can
turn that
into a score we we talked about this
more in the previous video with
catherine but
you can turn this into a measure of how
much something is an induction head by
looking at the fraction of the
eigenvalues that are positive and then
going and taking a minimum from ov and
qk
and and so here what we have is we've
colored
each one of these lines is a induction
head
and we're seeing
if you knock it out
how does the
metal learning score change
so
um a positive value
means that the meta learning score
increases but remember the negative
value um
it's a delta and loss a lower value um
was
what it meant that that we were better
at at metal learning so um having a
higher value here means that the
induction head is improving our ability
to do metal learning having a lower
value means that knocking it out
actually seems to um
uh seems to well knocking it improves it
so it's sort of doing the opposite
and we'll talk about that in a little
bit as well
um
and then they're colored by this this
score that tells us how much they are
induction at so the obvious
story here is well we have this one
as one head that
suddenly starts to become an induction
head and becomes
right at the bump starts to
go and cause uh yeah really really drive
metal learning um i should say this this
line here is the start of the bump
um and this line here is the end of the
bump
um
and there's there's some other heads
that
uh also
uh become induction heads and uh
contribute to metal learning now there's
this weird thing where they they all
sort of um basically all of the heads
seem to sort of rush to contribute to
metal learning and then stop and then
many work some recover the induction
heads recover
um
we don't know exactly what's going on
there but the theory is probably that
they you know once they figure out they
can do induction heads a lot of
attention heads try to do it because
it'd be very useful and then there's one
one winner or a few winners and the
others go and do something else
um
now
there's one outlier here which is
sort of an induction head it's a little
bit borderline um
uh maybe maybe sort of weekly scores on
this induction head score um so it has
some negative eigenvalues some positive
ones and it's really not doing metal
earnings we don't know what's up with
that
um another thing you might ask is you
know how can there be
heads that are that are sort of doing
the opposite of metal learning well my
guess is that these are heads um that
when you knock them out damage the model
um
more generally uh and
um damaging the model more generally
might just mean that you're you're
predicting tokens more more uniformly um
uh so that that would be
uh
uh
uh sorry um
knocking out more might mean that your
uh might i mean i guess it's sort of
clear that this theory is very special
but it's not something i thought through
very carefully but it it could be that
um
this this destroys your ability to
engram statistics or something like this
um and so the only thing that remains
maybe is your is your metal earning and
and and it's able to go and contribute
more to the loss so that might be one
theory like if you're if you're bad at
predicting early tokens then the meta
learning score would would increase
um but yeah i think the significant
story here to me is that very clearly
the induction heads are the ones that
are driving better learning
um and because this is an attention only
model there isn't anything else that
could be driving metal learning so we we
know that in this case the model is
really um yeah the metal learning is
really coming from the induction heads
um
yeah so
that so yeah okay so there's a few other
arguments for why um you might think
that uh
in the case of small attention only
models um
the the induction heads are really
driving this metal learning change um
another one is just that we really
understand these models and there isn't
really
anything else that seems like it could
be driving metal learning
um and also uh we we really carefully in
the previous video were able to analyze
the induction head bump um which is
seems to correspond to this this
meta-learning phase change
um
and
in that case we were able to do things
like look at individual token losses and
study how how those tokens were um yeah
were changing and those were were both
cases that sort of seem like they could
you could interpret them as kinds of
metal learning um and they they all very
naturally flowed from uh having
induction heads
so
uh if you haven't watched the previous
video again i'll encourage you to go and
go and watch it
um
now for large models i think this is a
trickier argument um when you have
thousands of attention heads um and a
large model that's more expensive to run
it starts to become annoying to go um
and do ablations of every single head
and so
we're left instead
uh with
uh a more correlational argument
and we can see that uh metal learning
develops uh right at token 10 to the
nine or right around token 10 to the
nine
and we can see that that's when
induction heads form
um and this is across a wide range of
model sizes
um the score is the score of the
attention head that most matches um an
empirical estimate so we can't use the
remember that previously in the small
models we could um i mentioned we could
we could go and detect uh induction
heads based on their eigenvalues that
doesn't work in large models because of
mlps
and so instead uh we have to go uh and
and come up with some empirical way to
detect induction heads
and what we do is we just create a
string of some random tokens
um and uh we see if they if it's able to
go and look at the look at the next
token and because it's a sequence of
random tokens and the only way that it
could sort of go and look at the token
that previously followed the token the
present token is if it's actually doing
induction it couldn't be relying on
other statistics because it's it's a
it's a random sequence so it really has
to be doing this kind of search through
the context and then look what happens
next that's that's the only way
um and these attentions are really
accurate at it right like they're some
of these are getting up to like you know
certainly in the 90 90 percentiles um
and then they start to fall i think
that's because there's multiple
inductions that are taking on uh
different roles and so you know maybe
one one matches
when
there's like multiple tokens within a
word and another one matches in other
cases and so on so then they they start
to become you know only only look at a
smaller fraction of those but
um but yeah there's so at ten to the
nine they form
um and that's when
the metal learning forms as well so
that's that's kind of an actual argument
right that sort of seems to me um
you know
it would be a very shocking correlation
um uh to just happen by by happenstance
um
now it could be um that there's
additional things happening there um
uh but we we know in the small model
case that this is really the thing
that's driving it and it certainly looks
very similar here we know that the same
thing's happening and the same results
happening and so it's natural to draw
that inference now could it be the case
um
that there are additional things
happening here um yes it could be and
we'll talk about that more later but it
could be that there are additional
things driving uh meta learning and
large models that happen form at the
same time as induction heads um and that
we haven't yet noticed and we can we'll
talk about how plausible that is later
but um yet first order i think uh it
looks like the uh
uh yeah it looks like induction heads
are probably a big part of the of what's
going on
um
one other thing that's maybe worth
mentioning this is really an aside it is
kind of cute um
this kind of there's sort of this
oscillating feeling
um around the induction head bump and
it actually seems like that might be
real it's not something that we studied
carefully um but it does seem uh it's
the same thing that we were seeing
earlier with um
uh with having some attention heads sort
of over over correct or something um and
have to bounce around a little bit
um
that may or may not be yeah it's a
theory okay so
um
okay so our hypothesis is that induction
heads
uh are implementing uh meta learning and
transformers and that's that's kind of a
wild claim to me um
so it's it i think it sort of is clear
or sort of makes sense that induction
heads could go and
uh predict the later tokens so um or
predict copy token so like here we have
the first paragraph of harry potter um
you know we we're at
uh we're at this token d we look back we
see that there's an errors we're like ah
it's going to be errs again and we
increase the probability of that token
then we're on the token errors and we
look back at errors and we see ah the
next token was li and we can we predict
li and we we get a better loss than that
so and it makes sense that
induction heads uh can can copy repeated
chunks of text um both you know very
short terms of phrases um or names or
things like that and also also longer
things
and
uh and i think it also you know it also
makes sense that they can like we saw in
in one of the previous videos that there
was
um a cross language induction head that
looked for trend you know words that are
in different languages and then looks
one forward and allows you to go and and
generate translated versions of text so
and you know there's other things like
this but
and and you know it makes sense that
those would would cause your loss to go
down
um later later in your context
but you know is it really plausible that
those could be explaining metal learning
um like you know
these large language models they can do
all these things that uh are are
different from
uh you know they they aren't just going
and root copying things they they learn
functions and they can go and and do all
sorts of um you know interesting um you
know generalizations um on top of
uh the the thing that they learned
so uh yeah so could it be that that
these induction heads are doing the same
thing
um
well okay so one one thing you could
think of is that maybe the more general
thing that induction heads are doing is
there are kind of metal learning nearest
neighbors they're doing a nearest
neighbor search over the context
and that could be you know going and
matching literal previous token and
literal next token um but they can also
be softer and they can also copy more
flexible things we we saw in a previous
video
not an induction head but a copying head
that was was copying um things like
gender intense and plurality um to go
and and keep sort of manage agreement
between adjacent sentences and stuff
like this well
um
you can you could have induction heads
be be soft as well the translation head
was one of those and um it turns out
these soft induction heads can also help
you learn functions
um and that's the thing that really made
me start to um
maybe at least it was it was the thing
that knocked down the last barrier to me
of thinking that induction heads might
actually be the thing that's driving
metal learning
um sorry so
um what we have here
uh is
uh the uh
uh
yeah a little toy problem that i set up
so um
we have
uh
a
we're trying to simulate sort of a
pattern matching problem that we know
for sure the model um has never seen
before because i i made it up and it's
very silly um
and will allow us to go and explore the
role of some softer induction heads in
going and learning functions okay so the
the puzzle here um or the the the
problem here is
um
sometimes the first word is a color and
sometimes it's a month so sometimes it's
a color sometimes it's a month and
sometimes the second word is an animal
and sometimes it's a fruit
and then we interpret each one of those
as a binary variable and we do xor
so
green lizard is false
um
because it's a it's a color and an
animal
and similarly
a
month and a fruit is false
but the other combinations um so a color
and a fruit will be true
and
a
month
and an animal
will be true
okay
so
this is a pretty arbitrary
function and the thing that we'd like to
see
is
when we
are on a colon
the model should look for a previous
instance that is the same
um conceptually the same as our
situation and then go and look at what
the answer is
so here we have a month animal
and it attends back to a previous month
animal and says ah it was true
in fact it's looking for one where the
month is the same so it's april and and
april and i guess dogs and cats are kind
of similar so maybe that's why it likes
it it's it's also um looking at marched
lizard a little bit which is also a
month animal um but it's putting less
weight there um because i guess it's
uh
less
it's less of a close match from its
perspective
it's working on on high level linguistic
features that make sense um okay if we
go to january wolf um this is also a
month animal and it wants to get here's
one that's january as well so it goes
and it looks it true and here's another
one looks it true oh over here we have
march lion um i guess wolf and lion are
kind of similar somebody likes that uh
and here we have a wolf he said it's
true um but it's successfully going and
looking these cases okay let's look at
one that isn't um
uh
so here we have a color
and a fruit so purple lemon well okay we
attend to red cherry that's true
um we go to purple apple that's true we
go to gray lemon that's true so we're
looking at these previous cases of this
of this
of the function
um okay here we have another oh no here
we have a
a month fruit
um and we when we look here we have um
october cherry
um and it's false
um
now this one uh is actually an error
it's um it's looking at the wrong thing
um it's probably because
we have september here and we have
september here and it likes that and
then we also have cherry on the previous
line so it's attending to the wrong
place but it's noticing um that there's
a cherry oh i guess in this case there
was also a cherry on the previous line
so maybe that's why i got confused but
it's giving most of its probability to
a correct case and it's just putting a
little bit of probability on this this
incorrect one so we're still going to
get the right answer
um
oh i guess we're also putting some some
probability here so
uh yeah
but most the one that's getting the
highest probability is still correct
uh
let's keep going
um
what about
yeah we could we could look at this one
now
um
okay we have red cherry true
gray cherry true
gray strawberry true
red apple true so all those are correct
cases to attend to and
use um and now we're back at a month
animal and yeah we're going and seeing
july fish is the one that's getting the
most april cat that's another valid one
we're giving a little bit of probability
actually quite a bit of probability to
this one
which is not a correct case to attend to
but mostly we're attending to the
correct cases so we're still going to go
and get the answer right
um
so the thing that i think is so cool
about this is this is a completely
made-up nonsense function and the model
has learned it um
by going and using previous examples
um
uh and it's you know it's not just uh
uh
uh
well okay i guess i'm i'm sort of making
a fool of myself a little bit but um
here
i it's not just you know i'm showing you
ones later on but but it's also early
right so here we have red frog and then
we're looking to green frog and blue
lizard and those are valid cases we're
putting a little bit of probability on
one of the wrong ones um and this one we
put a little bit of we put quite a bit
of probability on the wrong one but
we're still putting probability on two
correct ones um so that might still win
out
and so this also works early in the
context and works better later in the
context but it also works early in the
context
um and we can look at the effect on the
logits and see you know
when when are they made more probable
and we can see that you know this one is
made more probable by a correct match
and um by correct matches and correct
matches and there's all of these cases
and it's it's it's working um so
it really does seem like these
these soft deduction heads are able to
go and allow for
the model to learn learn functions like
this which is to me really cool
um
because
the ability to to learn functions and
pattern match like this
um i think was was one of the things
about large language models to me that
was was sort of most um
most demonstrated that they were really
doing in context meta learning um and so
now we have sort of a theory of of why
that's the case
okay
um
so
that's that hopefully that makes it at
least seem plausible that uh induction
heads could be doing at least a lot of
metal learning
um could other things be contributing
yes so it's certainly possible that in
large models other things um could be
contributing
now
uh
i think there's two kinds of barriers to
that as a theory one is it really seems
like they would have to be forming
more or less simultaneously with
induction hats um so we know that uh
that when induction hence form um that
is when
uh the meta-learning forms as well and
so if there's something else um it seems
like they have to form there as well
um
it also seems
like we need to explain uh why so we
know that that in in the small
intentionally models metal learning is
really driven by induction heads
um we can sort of just look at
everything and go and show that it's
the induction heads that are driving
things
and so i think you have to explain then
why is it um
that first of all whatever you you're
hypothesizing has to not form in these
small models that seems reasonable could
be that there's things that just don't
form in small models and do form and
large models
but then why is it that the same we get
the same this constant amount of metal
learning regardless of model size
um why is it that it's not that the
large models aren't doing better at
metal learning
um
now in some sense
uh
there's there's a way in which this is a
little bit deceiving um which is they
are
they're they're getting the same number
of bits from metal learning but their
loss is lower and so those are sort of
maybe a harder bits of information and
we know that large models have more
induction heads and more soft induction
heads um like the translation head or
the the pattern matching head those
don't exist in really tiny models
um
so so i think that's a little bit of a
strange argument that i'm making here
but um it is very suspicious to me the
same amount of metal learning gets
learned in all models um and that does
seem like an argument for i have no idea
why that should be the case but it to
the extent that it's true it does point
in the direction that somehow whatever
is going on in all these models is the
same
um but who knows i think that is very
much an open question
okay
now another question
that i've been wondering about is is it
possible that action heads are the
reason transformers do better than lstms
okay so um let me unpack that a little
bit because that's that's maybe sort of
an out of the out of the blue question
um well kaplan at all uh notice that um
this is the skilling laws paper um from
2020 and they have a plot that yeah it's
not very emphasized but they're the ones
we got this whole idea of going and
looking at um loss as a function of
token index from um
well
they make a really interesting
observation
which is
if you look at transformers
the loss
continues to go down
but if you look at lstms of the same
size for a while they do kind of
similarly but then they flatten out
um
and they tend to flatten out
somewhere before
token 100 so we've taken 100 here
and somewhere before then or around then
i think at least a little bit before
then
um they flatten out okay
well
that's kind of interesting because
we saw something flattened out earlier
we saw um
that
before induction heads form we flatten
out you know we start to flatten it
probably around token 40 or 50 and you
know by token 100 we're really starting
to flatten out we've a little bit get a
little bit more here um but we're we're
really significantly flattening out and
are basically flat
we're significantly flat i think from
you know somewhere between token 1500 to
onwards so you know that kind of rhymes
with this result it's not um
certainly certainly not uh
errol proof um but it is it is really
suspicious um to me that uh you know
these
um the lstms are happening flooded at
the same you know roughly the same time
that uh transformers before they develop
induction heads
do
and there's another reason why i think
it's it's a little bit suspicious which
is uh induction heads are fundamentally
a mechanism that it seems like
transformers couldn't implement
because they're searching over the
previous context
um and so you can't do that with a
constant amount of time
um per step because you
um
you yeah you're it it's you know
probably probably it's before you have
to do some compute for every every
previous step or another way i think
about this is that going and doing um
doing search um
for for every token over all previous
tokens um natalie that's going to be the
on squared
but
uh transformers only have on compute and
so um and and if you were really clever
maybe you could make it o n log n but i
i think probably
uh you know neural nets aren't
discovering those algorithms and even if
they were still still larger than o n so
it really seems like that's something
that it's it seems like lstms couldn't
be doing
um and so
we we know that before
uh induction heads form transformers are
relying on these simple copying heads
um and those do seem like something that
uh lstms uh could be could be doing an
alligator
something analogous to you know just
being like oh you know i saw this token
previously probably it's gonna happen
again
um
and so it it really yeah i don't know
again this is this is speculation i
don't you know i don't think we we
rigorously know this is the case but it
is kind of suspicious and you know i do
wonder um so
that was a fun little fun little thing
um
one other thing that i think is worth
talking about
um
is
well okay so
at least
now
okay i think this is all a little
um
uh
a little unclear but um if you put if
you put your con your context on a log
access these these curves these metal
learning curves they do look pretty
um
they do look pretty linear and if you
especially people you're lost on a log
axis as well so it does seem like metal
learning within a context is probably
modeled pretty well um as a scaling law
and
there's a paper by by marcus hutter
from deepmind
um
and uh he gives uh sort of a an argument
that we should really expect models to
follow scaling laws um at least with
respect to data that as you go and give
them more data um you should expect them
to follow a scaling law um
uh
and something i found interesting as i
was thinking about this is you know
these models uh
they follow okay so we seem to be
observing scaling well
you know the
actually
i think the case of of meta-learning
with induction heads sort of matches the
theoretical argument that um that marcus
is giving
better probably than the standard case
of going and training models um so
um
the the theoretical stop and the and the
other papers um
that you
you're observing a bunch of data points
and if you've seen
um
if you've seen a particular problem
before then in the future you get zero
loss on it so um you're observing the
sequence of of training examples and
then um if you see the same thing again
uh you go and you you you get it
perfectly and otherwise you you get a
bad loss and you have the same bad loss
so
um
that seems very similar to having a
context and seeing a bunch of tokens and
if you see
a token that matches something earlier
in your context you can look at what
happens next and get an a very very
small loss um
that seems very similar and so you can
you can really take just all the
arguments um that are in this paper and
they run through for the case of metal
learning um with with induction heads
and so we should really really expect
scaling laws so i thought that was kind
of fun
okay so in summary um meta learning
seems to form really abruptly in the
span of a few hundred steps
um and induction heads form at the same
time
um so that's that's interesting um
induction heads can be seen as a kind of
nearest neighbor metal learning
algorithm
um and we saw that they could even even
learn functions um and in addition to
copying text
um
and induction had seemed to
substantially cause meta-learning um
certainly they form at the same time as
metal learning and in small models they
can really be shown to be causing it
uh so
uh yeah i don't know this is i guess i'm
just excited about all this because it
seems like it's getting really um you
know potentially
some deep insight or or or getting at
some some of you know what seems to me
one of the the biggest mysteries of of
these large models um and so that was
was really exciting to me uh anyways
thanks for listening


MLP Neurons - Privileged vs Non-Privileged Basis [rough early thoughts]

sometimes people ask me questions like
have i studied the neurons in a
transformer's residual stream or the
neurons uh in the value vectors
and i think this isn't quite the right
way to think about it i think it's it
doesn't really make sense to go um and
and even yeah i think that question just
doesn't really make sense and so i
wanted to sort of
lay out uh
why that seems to be why that's the way
things seem to me
and
the heart of it is a distinction i'm
going to make between what i'll call a
privileged basis and a non-privileged
basis and the idea here is that
sometimes a neural network
representation has a special basis
that is particularly meaningful to study
and sometimes it doesn't and that really
changes how we have to approach
interpretability and so even though a
lot of my work has involved studying
neurons um you can only really do that
when you have a privileged basis because
usually what it means to me to have a
neuron is is that the neuron
is
is the the
value of a vector on a particular
privileged basis
so
um a privileged basis is usually created
by having a non-linearity in a neural
network especially value um value uh
because it's it's such a sharp
non-linearity
and because it it creates sparsity
it tends to be especially good at
aligning meaningful features with the
basis dimensions of your representation
but
you can also end up uh with a privileged
basis uh if you have perhaps some some
regularizer that's enforcing sparsity uh
in your in the resulting activations um
so you could have a linear
transformation but if you were
regularizing the output you might have a
privileged basis um or
if your if your weights maybe had uh
like some kind of l1 sparsity on them
that might create it uh so there there
are other ways that you can get a
privileged basis but the most common
reason would be that you have a
nonlinearity
um especially a strong non-linear
function
now
when you do that the result is that
you're you're incentivizing features to
align with that basis and
um an example of of why you might think
that's the case so yeah why why is it
that um that we're going to have
features
align with the basis well if you if you
go and you
uh
if you imagine if you imagine having a
representation
that's trying to encode two features and
you have two relu neurons to do that
and one way you could do it is you could
align the features with the with the the
value activations or you could go and do
something like have one be the average
of the features one neuron have the the
averages features and one be the
difference
well if you do that the point at which
uh the features transition from firing
to not firing in in each case is a is
dependent on the other variable
and so
if you try to recover the variables
afterwards that's going to inject a lot
of noise
and that that's going to really
incentivize you to go and align things
with the basis
so yeah when we have a um when we have
that kind of privilege basis that
encourages features to align and then
we'll call the basis dimensions neurons
but usually we we won't call them
neurons um
or at least i usually don't call them
neurons uh if they don't have that so
for instance if you have a word
embedding where where you don't have a
privileged basis um i usually don't
think of the the basis dimensions there
as being neurons
okay now on the other hand a
non-privileged basis
uh forms
uh it's sort of the default thing so if
you don't have a non-linearity
um or some kind of regularization you
should expect to not have a privileged
basis
and that means that you shouldn't expect
looking at
some dimension of that to be any really
any different from looking at a random
projection or at least a a random
unitary uh projection
of your representation
so yeah why why should we care about
this you know you might say well okay
that's you're sort of making this
distinction why does it matter well it
matters because one of the best tools we
have for understanding neural network
representations is to have a meaningful
basis that breaks it down and where we
can study each component independently
really any neural network representation
is going to have an enormous amount of
volume um it's you know as you increase
the number of dimensions you you
obviously say that the crest
dimensionality sets in and you have just
exponentially meant much volume
potentially in that representation if
you wanted wanted to try and study it
and so the key way the key thing that
goes and saves us from the curse of
dimensionality and allows us to
understand um the entirety potentially
the neural network's representation is
good if we if we have a basis where we
can study every dimension independently
and uh
the easiest way to go and get a basis
where you can study things independently
would be if you have uh a interpretable
basis for free from your from your model
um because uh it comes with a privilege
basis
so we are very very fortunate um if we
have a privileged basis because it makes
things a lot more interpretable or at
least a lot more easily interpretable
okay
um so in transformers
uh
where what which parts are have a
uh a privileged basis and which parts
don't
well
uh
the mlp neurons have a privileged basis
so
if you're familiar with the transformer
architecture um every block you have
attention layers and you have mlp
neurons that both branch off from the
residual stream and the mlp neurons are
going to have a jelly u or a value
activation on function on them and so
that's going to cause them um deadly
have a privileged basis they're actually
um they're also
a lot higher dimensional than your
residual stream so that's going to be
actually an extra thing and like there's
poly symmetricity going on that's going
to make them extra nice
and so mlp neurons at least in theory
all the period theory points towards
them uh being great now in practice if
you look at our i think probably the
next video we'll discuss in a little bit
they might be so far they've been pretty
tricky to under understand my experience
but um in theory they definitely have a
privileged thesis and on theory they
seem like they should a lot of things
point to the direction of them being
very nice
another one that we often don't think of
um but i think actually uh is is is i
guess it's the most privileged basis in
some sense is the the tokens because we
we often represent tokens as one hot
vectors um both when we input them and
then when we output them and we have our
logits or our softmax we have
probability distribution over tokens and
um every dimension in that basis uh
is completely interpretable because
you know they just correspond to
these tokens that
are in our writing system that are
basically designed well at least the
writing system's designed for us to
understand it so that's definitely a
privileged basis and um very different
from a random projection of that space
now another one um that it's easy to
sort of forget about
is attention patterns so
uh for a given pair of tokens
uh you have your dot product of the key
and the query which is the score and
then after you apply a soft max across
the tokens you get a probability
distribution and because those are are
one-dimensional in some sense they're
you know they're the the pattern of
course the distribution is
multi-dimensional across tokens but
there's only one dimension per token and
that really uh encourages you to have
even even for the scores before the
non-linearity to have privilege basis or
at least i mean i guess if you only have
one dimension sort of your privileged
basis by default and then
softmax is very non-linear so that's
also going to encourage
encourage in some ways you'd have a
privilege basis
um
on the other hand
uh the residual stream so
from the moment you do your first token
embedding you take your tokens and you
multiply them by a matrix or look up a
vector and
there is nothing about that uh that is
privileging
uh any any dimension in that operation
and
uh when you go and you add things into
your residual stream at every step in
your model you add the outputs of the
mlps you you down project your neuron
activations and add them to the into the
residual stream or if you go and you
take the outputs of your attention heads
and project them into the residual
stream and add them in
all of that is just you know it's it's
you could potentially do an arbitrary
linear transformation and get the same
result so your residual stream is not
going to be privileged it is not a
privileged basis
um
in in the default transformer
architecture
now uh
there might be some exotic transformer
architectures where you do have um where
you might have a nonlinearity in your
residual stream
uh and i guess one way in which maybe
there's a very slight effect is
you might have layer norms on your
residual stream which uh may privilege
because you're you're multiplying by uh
sort of by a diagonal matrix that might
slightly privilege um
your your b suspension of your residual
stream but it's essentially just uh in
in almost all cases not going to be a
privileged basis
another place where you don't have a
privileged basis
and is going to be your keys and your
queries and your value vectors so your
attention head produces these keys and
queries and values by going and doing a
matrix multiply of your residual stream
and then you you dot product the keys
and queries together um so this there's
nothing about that that's basis
dependent and so definitely you
shouldn't expect your keys and queries
to go and be
uh privileged and your values you're
just going to do a linear combination of
values then multiply them by another
matrix so there's nothing at the value
vectors either that's going to encourage
them to have
uh a
a special basis and so
uh you shouldn't expect to have a
privileged basis for any of those
now
it's worth talking about this and maybe
in the context of other types of neural
networks so
um in rnns
uh your r and n state and your lstm
state is generally privileged um you
probably have uh a
some kind of nonlinear activation
function whether regardless of whether
you have an rnn or an lstem or
a gru or
any of these things you you probably
have a privileged baseball but your your
word embedding um is basically the
canonical example of not having a
privilege basis so for award embedding
it just doesn't make sense to go and ask
you know what does basis 0 mean um like
um yeah if i in just in a default basis
that it gives you or what is what does
the basis direction one
mean
there's there's nothing special about
those basis dimensions
and that doesn't mean that you can't
study the word embedding people study
word embeddings all the time but you
can't do it by just um looking at the
basis dimensions
um now in confidence we mostly see uh
architectures where essentially
everything is privileged um because we
mostly have neurons especially neurons
with value activation functions
which is really going to encourage
a privileged basis but we do sometimes
see um
the residual stream
yeah so in in resonance um
many resonant architectures will have
values on the residual stream which does
cause them to have
a
privileged basis but sometimes you won't
have values on your residual stream in
which case um they will will not have a
privileged basis
um
there's also
uh
some other sort of rare situations where
you may not have a privileged basis um
and a confident so
one case might be
uh if you have
uh sometimes you might go and do
a low rank matrix factorization
of your weights i'm forgetting that i
think there's a name that people
sometimes use for this
um
in the in the context of content that's
slipping my mind for a second because
some special name people sometimes use
but um where basically you'll do two
convolutions in a row one that's doing
spatial and one that's doing
sort of depth wise convolutions
and then you'll apply your activation
function and so if you look at the
activation between those two steps um
depending on the details of it it may
not have a privileged basis because it's
uh
you you didn't have uh an activation
function it'll it'll depend very much on
the details of exactly how you factor
those matrices um but that might be
another case for you where you possibly
might not have a privileged basis in the
conflict um but that kind of situation
is very rare so um this usually doesn't
come up in confidence usually in
continents um except for the the resnets
that don't have values on them and you
have privilege basis everywhere
okay so
uh
the final question is what should we do
when we don't have a privilege basis so
in transformers actually a pretty large
fraction the transformer doesn't have a
privilege basis for us to try and work
and we have to go and somehow work
around that
and this broadly two category of things
you can try to do so you can either
try to find a meaningful basis for that
representation
and we'll i'll talk through a few
options in a second
or
you can try to push analysis to other
representations in the model and not
work just try to ignore and not work
with the representations that don't have
a privileged basis
so
uh in the case of not of of trying to
find a privilege basis one thing that's
been very successful
especially in the word embedding context
is to define um define a new basis in
terms of differences between elements so
uh people very often will like define a
gender dimension by doing man minus
women um or something like that and then
you'll go and you'll
project things onto that those
dimensions
um you might also apply various
dimensionality reduction techniques like
pca
um
pca or ica or things like that seem um
really i think if you want to do this
you want to do some kind of linear non
dimensionality reduction probably some a
nonlinear dimensionality reaction
probably isn't very principled to do
here but you could you could do
something like pca and study that um and
study those dimensions and um that seems
also to make a lot of sense
another option
um is that you could and this is this is
pretty similar um
is you would try to factorize things
um
and
well i guess the intuition here uh
is a little bit tied up with another
idea which is that uh
neural network representations seem to
often there's this phenomenon of
polysemanticity where you have even even
representations which have a privileged
basis um often you'll find neurons that
seem to correspond to multiple things
and and we call these polysomatic
neurons and i think the leading theory
um among the small set of people i know
who talk about this for for why this is
the case is that uh there's the network
is trying to represent too many features
and so it's forced to go and squeeze
multiple features into neurons and take
advantage of the properties of high
dimensional spaces and the fact that
in a high dimensional space
and many you can have exponentially many
almost orthogonal dimensions uh in any
case
all this is to say that um if you
believe that you you want tools
potentially to go
um
and pull apart that a space into
actually a higher number of dimensions
um to find a meaningful basis and so
then then there's some ideas around
factorization um and sparse coding and
things like this um that are are
potentially really uh really powerful
um and yes sparse factorizations um that
seems promising okay so in any case um
that's that's my rant on the uh
trying to find a meaningful basis option
for the trying to push analysis to other
representations well in the case of
transformers we're very very fortunate
that all of these representations
that don't have a privileged basis
are just they're just linear
transformations of a bunch of privileged
bases and so the alternative thing that
you can do
is you can just try to go and talk about
everything in terms of
privileged bases like your mlps and your
tokens
and just push all the math back to those
and
i think actually i mean if you've
watched our our videos um so far you've
probably seen some examples of this
implicitly
where we were able to go and understand
for instance um attention heads by going
and pushing the ov circuits and qk
circuits all the way back to uh the
tokens
so
this is another strategy that you can
take and i think it's a very a very very
promising one and a very nice one
okay so uh in any case that's my rant on
privileged versus uh non-privileged
bases or uh representations that have a
privilege basis versus representation
that don't have a privileged basis and
yeah if you're interested in our next
video i think we'll actually i'll go and
have a couple of my colleagues join me
and we'll talk about um some neurons we
find in transformers um of course in the
mlps uh where that sort of is a question
that
seems like a good question to ask
thanks