EleutherAI Interpretability Reading Group 220423: In-context learning and induction heads

it's 205 looks like we have
a decent showing so i think it's
probably a good time to start
uh so today we're going to be talking
about anthropic second interpretability
paper
called in context learning and induction
heads they spoke about abduction heads
in their last
their last paper on this kind of stuff
and i'll walk through that as well but
yeah let's go ahead and get through it
um so
just starting with some of the key
concepts or what what is in context
learning
um and the basic phenomenon that they
describe it is that as they describe it
is that um with large language models
it's really common that tokens later in
a given context are easier to predict
than earlier
so you can think of that in lots of
different contexts for why that might be
the case but that's the basic thing that
at least
they're they're going to try to study in
these next several
slides whatever
um
so that could mean a couple different
things there are lots of phenomena that
could fit that description
um
and they but they they operationalize in
a specific way so one way you might
think of this is just by basic
statistical or knowledge associations so
if you just happen to know the capital
of azerbaijan then that's something um
that's just more like
what a language model is trained to do
in general um so that may or may not be
exactly what they mean by this
in fact they they kind of more liken it
to another couple concepts that are
pretty popular like few shot learning
where you
can
since language models are very good at
manipulating the context to a particular
kind of task you can also kind of train
them to do different tasks within the
context by giving them a couple examples
that's a really
popular and impressive feature of large
language models these days and you can
also engineer it to do different things
just by kind of tweaking the prompt in
ways that might suggest certain things
so
um that's that's
particularly uh useful in things like
clip guided type of image generation or
i'm sure
we've all seen cases where um you could
just
have the most aligned ai ever do
something and it would be much more
suitable for that i think that was part
of what their lab assistant paper also
talked about
they're not going to take a super
crucial stance on which kinds of things
they're talking about in this paper
and instead they're going to
operationalize the concept in terms of
decreasing loss at increasing token
indices
that's a bit more dry but a lot more
precise so
effectively what that means is just that
again that
on average
language models are much better at
predicting things later in the in the
context than earlier
um they're also going to make a score
out of this to analyze in lots of
different contexts so they're going to
take uh they they often work with
context of length 512 so 512 tokens
and they're going to take uh
i think it's 10 000 something like that
10 000 contexts and run language models
of each on each of those contexts
and then they'll take the loss of the
500th token and they'll subtract off the
average loss of the 50th token in
context and that should be their measure
of
um
in-context learning by this definition
on average
one kind of interesting quirk about this
they find which is
really weird in my mind is that um the
relationship between context index
and this kind of in context learning
score
is really
regular so on the for each line in this
plot you have different models so you
have a
that span all
many different ranges of parameter sizes
you go from 13 million to 13 billion
um and you plot the average
in context
loss i guess i'll call it um
by the log of the context index and you
see this really clean
relationship
um
it's kind of hard to read the the axes
on some of these plots but i can tell
you this is a log scale on the bottom
um
but yeah you see this really clean
relationship and so they'll make a score
that's based off of this later or that
they'll they will operationalize the the
kind of slope of this
um nice relationship here
um
but this is also to motivate the fact
that they're
there the score as they've
operationalized it here with these 550th
tokens isn't necessarily sensitive to
the specific indices you choose you get
this really nice
smooth relationship between
uh the law scores as you move forward in
the context
okay
so then what are induction heads
um a kind of rough way to understand it
that i like to use is just that it's an
attention head that attends to the token
that followed the last use of the
current one and then copies that the
the following token
so here's this one kind of schematic
diagram to use in the paper so if we're
looking at the node token
what the
induction head is going to do is going
to look at the last time node was used
attend to the
the token right after that and then
induce
or
increase the likelihood that the model
will produce that same token uh to
follow the last instance of this token
so you can also represent it kind of as
these
uh kind of nice symbol forms where you
have a pattern eight goes to b and you
later encounter a then the induction
head will
encourage you to produce b after a the
second time
they also make that definition a bit
more precise they they say that an
induction head is one that performs both
prefix matching and copying
and they'll introduce scores for that to
make that also more precise as we go on
precisely the last time or does it also
look at even more previous times i you
can look at more than one time
yeah that that's one of the reasons why
i put tildes around this because it's
not necessarily just the last time it
could be
um multiple different times it could be
a couple times ago it could be
some version of this
i should say i should note that the way
they make a prefix matching score to
quantify this characteristic
is based off of the last time a token
shows up i believe so
um i think we'll get to that later but
we but we i just wanted to flag that
issue for now
okay uh the paper as a whole is framed
in six arguments
for a particular claim that induction
heads um are responsible for a majority
of in-context learning in large language
models or i guess
in transformer models in general
um
the arguments that they're going to give
are not thought to be knockout arguments
it's not as if you should walk away from
this paper fully confident that that's
the case
but they're trying to build an
evidential case and they want to build a
norm of reporting results even if
they're not fully conclusive because
they expect that to be the case for lots
of things going forward that are still
interesting
um i'm going to try to cover some of the
qualifications they give in the paper
and alternative hypothesis but
probably not the best substitute for
just reading the paper there's a lot
there the arguments are pretty
sophisticated so i would
encourage you to read the paper if you
haven't yet
they also analyze a bunch of different
types of models
similar to the last paper they analyze
attention only transformers which get
rid of mlps and
a couple other pieces
but they also add
they also analyze models with mlps as
well as really large transformers as
well
i'm going to cover as many of those as i
thought i could in you know a reasonable
amount of time but there are some
results i'm going to skip
okay so argument one
that this that
uh induction heads underlie
a good portion of uh in context learning
is that transformer language models
undergo a phase change during training
this is something that's actually been
seen in a couple different contexts
before but they're they're claiming uh
they're claiming to explain a particular
kind of bump in the training curve that
i'll describe a bit and i'll describe in
a bit
um but during that
phase change induction heads form
and simultaneously in-context learning
improves dramatically so this is a kind
of observation that something happens
during training
and
in-context learning improves right at
about the time that you detect induction
heads occurring within the network
they give you these nice little handy
tables as well for each of their
arguments where
they try to
give kind of ratings of how well their
arguments apply at different regimes so
on the
on the horizontal axis you have
attention only models models with mlps
that are also small
and much larger models
and then there
there's an argument that induction heads
contributes some of
um
in context learning and con contributes
the majority
um across all these i think this
particular argument well i think and
they think that this is just kind of
correlational evidence so it's good to
keep in mind but what is exactly is the
finding so
they uh
take that in context learning score that
i mentioned before and they apply it
throughout training
and in this plot they're doing that for
three attention or attention only models
one has one layer one has two layers one
is three layer
the one layer model
does shows some trajectory but the two
and three layer models have this big dip
in in context learning score
um so that means that there the in
context learning improves a whole bunch
in a relatively short amount of training
time
so that's interesting that's
particularly interesting because
the mechanism that they proposed for
induction heads before requires at least
two layers in order to work and we'll
reference back to this a couple times
going forward
this this is also the case for the
networks with mlps so this is kind of
extending the results to what they had
before
to to networks with mlps
and it's seemingly also the case for
these really big models um the the data
here is interesting you'll you'll see
that the relationships are much
more coarse because they only have i
think 15 time points and training where
they
uh they made these measurements i'm
guessing
but
or
they i think they mentioned having 15
time points i'm not sure if they only
evaluated those at those time points or
exactly what happened but
you know the data that they show is
about 15 time points throughout training
and you still see a similar dip
at about this time
um
to motivate the to again kind of
motivate that
their choice of the in-context learning
score isn't driving something like this
they took the relationship that i showed
you before with
this thing
that the loss improves as you get
further into the context
and they made it a graded version of the
score which which relates the derivative
of the loss with respect to the log
token index
so that should give you
a version of the slope of this line that
pops up here
and what you what you see is that in the
one layer attention only model you see
some
kind of
blur that doesn't have many qualitative
shifts in it
the argument is that in the two-layer
and three-layer models around the time
that you see this big shift in the their
original score you also see a
qualitative jump or like a
a band
in the in the score change
so that this is motivating that that
their particular choice of score wasn't
especially the
indices that form the score weren't
particularly important for the basic
effect
they also apply so that that's nice that
something happens that
um okay it's nice that something happens
that
changes this in context score quickly
that's already interesting it's also
interesting that
this occurs only in models where they
think that induction heads can form but
they haven't really associated this with
induction heads specifically yet so what
they're going to do
in addition is develop a prefix matching
score so this is based off of their
notion before what induction heads do so
again they're going to look at
so i this is using their user interface
from the last paper
so what i did is i highlighted on this
token 1396. this is a stream of random
tokens repeated five times
and so i clicked on this one to
highlight that it should tell me
when i'm at this token what other tokens
am i attending to in order to figure out
what to do next
um and when highlighting one three nine
six in this slot here i'm attending to
five zero one two which follows the last
iteration of one three nine six
so
the idea is that what
the this prefix matching short score
should quantify the fact that i'm
attending to the thing right after the
last version of
the the last use of this token
right
okay
so
in order to form this score they're
going to take four steps the first is to
generate a sequence of 25 random tokens
it's kind of similar to what they had
before
they're going to get rid of a couple
tokens that are
i guess troublesome to work with
i guess just to get rid of statistical
regularities that pop up
they're going to repeat the sequence
four times and
they're going to compute the attention
patterns on top of this
the prefix matching score is the average
of all attention patterns entries
attending from a given token back to the
tokens that preceded the same token in
earlier repeats
so
it's the amount it's it's the average of
the scores that
looked at this that preceded um
this token
i don't know if that makes sense
another way to check this is what i
expected you to talk about here is that
they could
hard code a model that can only learn
attention heads and then they could see
like they could do a scatter plot of
whether
the
model starts doing well on the same
training examples that an induction head
hard-coded model would work well on
right
uh they will do a version of that for
the next argument so they're going to
make it easier to learn induction heads
and they're going to show that in
context learning improves a lot even
faster than the standard version
so that's
pretty related to what you said but not
exactly the same
okay
but when they when they
make this prefix matching score to try
to explicitly look for induction heads
they see that
indeed these heads just kind of appear
within this window
so throughout training you know the one
layer models have have nothing like this
the two layer models suddenly sprout a
bunch of attention heads that look a lot
like this that attend to the prefix
or that match the prefix
and three layer does as well
this is a particular point that's
actually pretty interesting because if
you look at
the
networks with the mlps
you see a similar kind of result but it
also looks a bit more complicated
so
you do indeed always see that you know
these these heads tend to develop around
this band that corresponds to where the
in context learning score improves but
you also see some other stuff that's
that's pretty interesting so
you see these other lines in this three
layer with mlp's models that
that kind of slowly build over time
i should note that the color of the
lines for each of these heads correspond
to how how where it's positioned in the
model which layer it's positioned in the
blue
the deepest blues are the later ones and
the redder ones are earlier in the model
and even with the the two-layer network
with mlps you see that um there are some
heads that seem like they might develop
into
heads that have that show prefix
matching but they
their their prefix matching decreases
over time
so it's a little it's
this story seems
super interesting and it does fit the
general description of what they say but
it also seems like there's a lot more
going on here
that's also especially true of the much
larger models so they also run this
prefix matching score with a bunch of
other attention heads
and once you get to the the highest
networks there's just all kinds of stuff
going on they're lines that come up and
down they're lines that take forever to
develop and so on
can you say more about how the line how
the
evaluation about a layer is done this
sounds so far like it will be done to
the entire model at once it is so well
okay i shouldn't say it is in general um
i think for the smallest models they do
this for every head
for the larger models they take a random
sample of heads i believe it's 100 or
something like that
how do you take a sample of heads the
heads are just like implicit right you
can't take them out
i'm not sure
yes you can i mean you just take the
i mean yeah you would um
you just take the attention scores
across the different heads those should
be separable
you're not taking them out oh you mean
you identify particular
uh heads parts of the model as attention
once
yes as attention hits and then look at
them while
like just ablating out the rest of the
model no no there's no ablation
happening yet here this is just
observing what happens during training
okay so this is the score that it would
have if you used only that head
okay so let's look at the the prefix
matching score is just trying to
uh is a quantification of
what behavior they think this head has
about its attention so like
where they're trying to quantify a
characteristic about its attention
pattern
so you can think that let's say an
attention head only attends to the
previous token no matter where it is in
the sequence you can come up with some
quantifying score to say no matter where
it is it's going to attend the token
right before me or it's going to attend
to the token 2 before me or something
like that this is just a version of that
it's just a descriptive
quantification of
what the attention patterns look like
it has nothing to do with changing the
function of the model while it's
training or anything like that or or
running inference
does that make sense
so this is just saying
i i know that well okay my i hop my
hypothesis is that induction heads are
going to look back to these particular
cases where things repeat
so if i take the average of those scores
that look like they're looking for
things that repeat does it show up
they're just trying to come up with so
yeah you can imagine that you could in
their last paper they did a lot of
manual work to basically identify these
patterns but you'd want something
quantifiable so that you can run it
in an automated way and pick out hands
and things like that oh so this is
something that works on
the attention model in particular
because that one just like
works directly with
um numbers attached to particular
indices
yes i believe that's right
okay thank you then i think i see the
conclusion
okay
the confusion was with what was actually
written on the y-axis here
got it yes so this is just the score of
a particular head
at a particular point in training
the score meaning how well it matches
prefixes
okay
um
right
but again this has lots of different
cool
you know details packed within these
plots
right so the earlier layers always pay
attention to the last time that this
showed up but the late layers do other
kinds of processing
i think it's well okay for the smaller
models it seems like that's the opposite
actually it seems like
the the right yeah
yeah the later layers uh have more
induction heads than the earlier ones
which i think at least partially makes
sense because
you need a composition well okay at
least by the mechanism that they propose
you need a composition of heads in order
to do this
so you need at least two layers deep in
order to even do this at all so
you would expect that overall there'd be
at least some
delay in this for the smallest networks
and then later all kinds of stuff
happens
sorry for things much but i like
expected that other people also confuse
no no this is good i'm really glad
you're asking these questions it's
helping me to understand what i'm not
explaining clearly to
okay
are there any other questions at this
point
okay
uh
right so the
it's cool that you can uh
run these scores and see
kind of qualitative jumps in how the
model performs but you can also even see
this particular jump on the overall
learning score or over on the overall
loss during training so here's
just the loss curves of these models
throughout training
and they've superimposed the one layer
score on top of these
and you can see that there's a
a bump in the training curve that
corresponds to this time period where
both
by their by their metrics induction
heads develop and in context learning
score improves so this bump that happens
in the learning score or that in the
loss
corresponds to some kind of qualitative
jump in what the model is doing
does the 2 layer model have twice as
many parameters as the one layer
oh that's a good question i don't
think so
i don't think it's twice but i think
it's probably close to that i don't i
don't think these are controlling for
the number of parameters but i could be
wrong on that
the obvious question is whether you can
take a ten-layer model and flatten it
out and
to two layers and get the same curve
well
yeah they
hmm
i believe the answer to that is no but
they can also do the reverse of that
which i think goes to the next argument
again i think you will like the second
argument and it should answer some of
the
questions but i don't think they've
not
yeah that's a good question
if someone could look at that up that
would be great but
i don't know off the top of my head like
how the parameters count for these look
like i think in general that's not the
case because at least they also analyze
these which are clearly not controlled
for parameters
um
i believe they've also experimented with
several versions of these so it would be
weird if
just increasing the parameters would do
this but i think they can
give you some evidence that that that
one layer shouldn't be able to do this
at all
okay
um
right so this is talking about the the
loss curve
you can
look at the same kind of analysis with
the mlp layers
and sometimes you see a bump sometimes
it's kind of harder to see
so this seemed like a point that was a
little that also has some hidden detail
in it
um and the with the larger models it's
particularly difficult because again you
don't have too many time points to look
at to try to figure out what's really
going on
um
so kind of hard to say there
another thing they look at
uh
along the co-occurrence lens is by
looking at
the vectors of loss per token so if i
have
a
context i'm just made it words because
whatever
so you have a string of words and you
have
the amount of loss on each word in each
cell here so that's each one of these is
one
loss vector
you can evaluate this
over
many different
uh
prompts
and you can do this at different points
in training so that's what the the
vertical axis here is is
depicting so you have uh the loss of
this sentence over lots of different
points in training
and you can analyze these in lots of
different ways they did this with again
10 000 examples each 512 tokens in
length and these are randomly selected
in training so there's 10 000 examples
per vector and then there are
a bunch of
samples throughout training
as as they could sample yes in most of
these cases
if you perform pca of those vectors
you can get an idea of
how the
how the loss changes
through the fire
in terms of the tokens over time so if
it's the case that similar tokens are
learned or
are
you the the network learns how to handle
similar classes of tokens you'll see
big you'll see components of this space
relate to one another
and so they
summarize the learning trajectory of
these models using pca across those loss
vectors i mentioned before
if you do that you see that
the shape of
these trajectories
so training goes this way
has a pretty similar shape
so you start from here
you move in in one direction these are
the first two components of that pca
space
but when the the induction heads and
um
in context learning score window happens
you see this yellow shift happen that's
a big curve
in how this pca space looks
at that exact same time
and it only happens for
models that have two or more layers
um this is actually surprisingly
consistent across
um
the models the attention only models the
models with mlps and the larger
transformers you you basically always
see this shape that it comes through one
direction takes a big turn and then kind
of levels off
so the interpretation here is that
induction heads have no reason not to
follow uh not to form right at the start
except that the particular
pieces of data that it would work well
on are handled relatively fine enough by
the initial model but the initial
training phase makes
the model do worse and worse on those
training steps until induction heads are
uh incentivized enough to appear
that that could be a good interpretation
i don't think they're even going that
far i think all they're saying is that
there's this qualitative shift in what
happens
that corresponds to this window so it
must be important for the model
but i think yeah i think you're going
even a step further
i think that's really interesting
if you look at
these the points around this shift
and look at how the loss changes for
different tokens you see interesting
patterns that correspond to what they
think induction heads are doing so if
they look at the loss of b minus a so
there's this loss vector for all the
tokens at b a loss vector for all the
tokens at a and they just take the
difference between those
um the red in this text above
is where the model better predicts
tokens and the blue is where the model
does worse in predicting the tokens
and the pattern is that
if
the tech if the context is repeating
what happened before similar to what you
think an induction head would do would
help you with
then the model performs better whereas
if it counters what an induction head
would tell you to do the model does
worse
so in this case if you go to
uh
mrs d ersley so at this time
at this token
um
the previous versions of this were
again mrs dursley
and mr dursley um but when it reaches a
new name for mrs potter
the model does worse here because it
thinks it's going to fall because the
induction heads in by this argument are
pushing it to
produce dursley as well
for this name
and whenever there's a dursley the model
does better
um
the potter seems to be better because
that seems to be your product or
repeated a few times repotted that's
funny
i think there's something else but
yeah
does that part make sense
yeah what happens when you look at more
than two dimensions
uh
they don't show it
that is a good question yeah
the okay well
they yeah they don't they don't show
anything more than two dimensions i
should say that it is interesting to me
at least that
the first two dimensions seem to always
look like this and have this big swing
so these are the dimensions that should
explain the most variant so that that's
pretty significant
yeah this is great you should look use
uh like do the same thing with other
models then transformers and then see
whether this keeps happening and it's
therefore more like some like a
statement about the training set rather
than the architecture that's a really
good point i like that idea too
in fact maybe we can promote or prevent
the formation of attention heads by
fine-tuning on the parts of the training
set that this
pc indicates
yeah it's the uh these model ball i
think it's pretty standard practice but
these models also never saw new training
data so
it would be interesting if
regularity in the training data would
give you more signal like this or not or
if there are particular examples that
would make this learn faster or
something like that
or make the model learn this faster
anyway okay
um
but in sum the the
basic argument for argument one is that
there's this
big shift in the model
where a bunch of stuff happens
um at the same time so in context
learning score improves
um
induction heads seem to form and the law
the loss changes kind of
significantly
um
at least you know you can there is a
visual change in the loss
um
there are lots of things so maybe there
is something that developed in the model
that
um encourages both of these things at
the same time so maybe induction heads
aren't the thing that completely
underlies incontex learning the
in-context learning score improvement at
that point in time maybe it's something
else that develops at the same time but
at least
they seem to go hand in hand at this
point
so argument two
is
more active so what they're going to do
is they're going to change the basic
transformer architecture
in a way that
well just to read they they change the
transformer architecture in a way that
shifts when induction heads form or
whether they can form and the dramatic
improvement in in-context learning
shifts in a precisely matching way
so what they're going to do is use their
proposed recipe for reduction heads from
the last paper
the last paper says that
one way to implement induction heads
is to
use one attention head to move
information from the previous token to
the current token so you move some
information from node to struction
um
and
you all
once that is done you use a second
attention head to query the the
information move to struction that
matches
um a similar kind of information from
node so then you take the second
attention pattern to basically fetch the
stuff that was moved into the right slot
by the last attention head
this is based off of a key composition
so they think this works through
the the key terms of the attention
self-attention mechanism
um so
i think the manipulation here is that if
they allow attention heads to directly
see the the key of the previous token
you can achieve both of these steps in
one layer even though you couldn't do
that with you couldn't do that with one
layer before you would need at least two
you'd need one attention head to
you'd need to do this in series with
multiple tension heads
so
they're going to introduce a new
parameter to learn in the model
where
for for each tension head you have
uh
you derive keys the normal way so you
have a key for layered i guess this is
for layer j and head h
um but then you modify the key for
layer j and head h
by blending it with the key from the
last token in this case
and so this is a trainable parameter
that determines how much you want to
blend these two together and this is
just the sig point function so this is
just
giving you the exact form of how these
two are blended together
so the final key that's actually used
for the self-attention mechanism is a
blend between the original key
the the normally generated key
and the key from the last token this
isn't a recursive formula they just
apply this once once you've generated
these these keys
does that make basic sense
well suffice it to say that i think that
all they're trying to do is make it
easier for
attention heads to learn
um
the specific mechanism of induction
heads that they proposed in the last
paper
where in the architecture is the step in
circus
uh at this at all the self-attention
mechanisms so whenever you generate keys
for your
query key lookups you would do this
okay so first that calculates the
keys as normal and then it calculates
this formula for the
uh like from the keys as normal yeah uh
so that it looks like a sawblade pattern
perhaps
could you do that last term
uh saw blade pattern uh in other words
the
the two appearances of
kj are actually two different variables
that's correct yeah that's what i'm yeah
so i'm trying to indicate by not
recursive
this is just from the this is the form
that they use but they yeah this is
before the transformation this is after
the transformation and what you would
have done was put a hatch on the left
side and left k that's right that's
right
okay
on the top row here is this the same in
context learning score plots from before
and on the bottom is when they introduce
this change into the into the model they
call these smeared keys models
and you can see that
all of a sudden the uh one layer models
are able to
have better in-context learning scores
than before
and the two layer models are also able
to learn
or to have improve their in-context
learning scores faster than before
so this is a simpler mech you don't need
to learn the
the two layer composition of heads you
just need to learn
you know that
um
you need to learn that one parameter
instead of a whole attention head that
feeds into this mechanism
so
the hypothesis is that should make this
faster and it does seem to make it
faster
they claim that this is medium in
interventional
evidence for small attention only and
small with mlp models
it's interventional because they could
design an experiment to have a
particular effect and it had that
particular effect
um there are lots of other things that
could well they don't do this directly
for large models but i think just from
i don't know what they base this part of
the
argument on i forgot that part of the
paper does anyone else remember
okay
well suffice it to say
again that's another flag to read the
paper for exactly how they qualify these
things but
that's argument too
okay so
argument three is that they're going to
try to basically knock out uh
induction heads in the model that they
detect using their metrics and they're
going to see if the in context learning
worsens when they when they perform
these knockouts
so here's a diagram of what they're
doing from
using one of the diagrams with the last
paper so here's just a schematic
of the transformer architecture
you have the self-attention mechanism
here the mlp layers you have embedding
and unembedding
what they're going to do is they're
going to take one of the heads that they
think well an individual head at a time
that they think is an induction head
and zero out its contribution to this
backbone residual stream here
um however they're not just going to
propagate the model forward after that
they're going to
before they knock out this this head
they're going to run the model
unaffected
in order to determine what the attention
patterns are going to look like from
this point on
and then they're going to zero out this
contribution to the network and then
propagate forward freezing the attention
patterns of what they were in the
unaffected version of the network
the idea is that if they just outright
zeroed this attention head that would
have all kinds of cascading effects
in terms of which tokens look at which
other tokens are
the general functioning of all the
attention heads they think that that's
probably a more severe
ablation than they probably intend
so i think
that's how this works
did they just zero this out like in the
analysis afterward or did they actually
do it in the network and then run it
yeah so they did they did it in the
network and ran it um again freezing
those attention patterns but they
propagated the the
that forward after this was zeroed out
does that answer your question
yeah it does that's i'm just kind of
thinking well that's really cool they
can freeze the later part and know how
to make those kinds of modifications but
i'm curious how they did that but that
that's that's cool
cool
okay
uh that's right so this is the this is
the same
uh this is just showing the same windows
from before to to give you a context for
where these things happen
on the top row on the on the bottom
on the bottom row each line is an
attention head they color the attention
the lines according to what they think
the head is doing based off of
other metrics that they they talk about
they have the purple line the purple
lines they think are induction heads
the
pink lines are previous token heads and
the yellows are just unmatched for
either of these
and you can see that
when you hit this window
a lot of purple lines develop
the pink lines develop
and when you knock out these purple
heads the in the in context learning
score gets worse
it's also interesting to note that when
you knock out the previous token
attention heads the ones that just
attended the token before the current
one
the
in-context learning score improves
quote-unquote
i think they they met they talk about
this in the paper a bit they mentioned
that this happens because that just
makes the model a bit worse at
predicting overall so if you are also
predicting the
um earlier tokens worse than before
then that gives you more a better
opportunity to predict the later tokens
which would
um
increase your income tax learning score
overall even if you're just overall
worse of predicting things
so you've got like two
sides of the in context learning score
that
early in the context and later in the
context
um
knocking out the induction heads they're
going to argue um
i i suspect that they'd argue would make
predicting the later context worse than
it was before
and knocking out the
previous token heads would make the
early context predictions worse than
before
it sounds like they
uh look at how knocking out a particular
head uh behaves during training but the
thing that each head does
changes during training right that's
right
so
wouldn't that make this inconsistent
like if you look um i suppose that they
simply identified what each chat does at
the end of training but
after that
training so that's why all the colors
change
so that this this point is where these
heads change by and large
but they i'm if they you know the colors
are changing from yellow to purple in
this region where
okay so since they are doing it at each
point
like to every hit they can get away with
um
not having to care about
what it does early or late yeah nice
exactly
okay
uh they also make a separate claim that
at least
i didn't know how to parse at the time
um they said that they looked
i i think they effectively just looked
at more stuff and they said in fact
almost all the in-context learning in
small attention only models appears to
come from these induction heads i don't
quite
see how they got that from the data
itself
um
there might just be something i'm
missing or there's some there's some
other plots that look at
how
the
knocking out a an individual induction
head
relates to the shift that happens within
this window
and indeed you know knocking out the
induction the induction head changes the
loss vector in that same direction as
what happens during this window
um but it seems like that's
the claim that these
that the in context learning
or almost all the context learning comes
from the induction head seems to be
based off of something like
it seems like you would need to motivate
that by saying that if you knock out all
the induction heads then all of the
in-context learning goes away
maybe they're assuming that if you like
knock out each individual attention head
and take the sum of all their
contributions it would it would add up
to most of it i don't think it's that
naive but i'm not sure what to think
about that yet
looks here like only yellow
heads can become purple
while orange heads stay orange i imagine
that in ten years we might have like ten
more types of heads
oh and almost certainly yeah
and then it
that does sound like eventually the
model might during training run out of
yellow heads to specialize
all right so that sounds like you want
to be able to like add additional heads
during training
i mean that's true these are also pretty
small models
we did see a whole bunch of complexity
with the larger model so when i
this thing
so there this is uh this is doing a
version of the detection
algorithm for induction heads
and you can see that these things change
in and out of the score it's definitely
not
a one-way transformation across the
board
okay nice yeah i can see that the purple
headlight become orange there yeah oh
well these are different colors sorry
this is uh these colors correspond to
its position in the network the later
colors correspond to their
classification of the heads but the
classification is based off of the
y-axis in this plot
but the the basic observation that
you know the heads just kind of go up
and down
would suggest that they change their
roles throughout training as well
maybe but now that you've said that
i hope someone ever
checks whether heads ever change worlds
yeah yeah me too that's a good point
okay
uh this is yeah this is
i think this argument was was
particularly interesting um and i think
that the data is like the the data that
is presented is
kind of the roughest
um
so if you look at
the
version that this there's there they
have like the main thread of the paper
and then they have this model analysis
table where they break out the results
for
um the models with mlps and the bigger
models
you can see lots of other interesting
kind of stuff happening with the mlp
models like the
the spread between their classified
induction head and previous tokens thing
gets much smaller
as you get into deeper networks with
mlps so that might relate to the fact
that maybe the mlps are starting to
contribute to in context learning or
something but
it just seemed kind of interesting
it didn't seem like it exactly matched
the
this version because you have this nice
separation between the purple heads and
the pink heads
um
in these cases where everything's kind
of tightly
put together here it still is the case
that the purple heads are generally on
top though there's kind of one weird one
here that's on the
bottom i'm not sure if anyone saw that
or knew what to make of it but i figured
i'd point out that
you know
the raw data here is
a little more rough than
the other parts
in both arguments two and three they
like introduce new kinds of models and i
would like really to see the
corresponding pca graphs like you should
be enabled before each head oh nice uh
wait you mean the pca graph for each
head that that is confusing to me
like every head here uh basically
is its own variant of the model you take
out this head and now you can grab the
loss or the score or whatever
so you can also once
yes so you can also plot its uh
like its path through the same pca space
that you've shown in argument one that's
true yeah they don't show that they do
have the pca plots for all of these
different models that
that are in the model analysis table but
they don't have that that would be a
good idea do they also have it for the
uh variant in argument too
with the key status smashing two
yes here i can i could just jump there
yeah like
you would have now convinced me to
actually look at the paper
that that's the main purpose of these
things anyway so that's good
okay so here the attention only models
one to six layer
um one layer you just have the basic
shift and in two through six you have
this big curve
uh
with mlps you see the same exact thing
roughly right that's the key smashing
i'm getting there i'm getting there
so we've got fewer points for the bigger
models but you see a similar kind of
shape these are much rougher
and then
oh these are smeared keys they don't
have they actually don't have that for
the smear key ones that's surprising
that's okay i i love these pca graphs i
imagine that we can just discover the 10
next
head types like
doing particle physics basically by just
looking at a few more dimensions and
seeing
like finding more repeating that does
that seem really really valuable yeah
that's a really good point
i i also wish that they had put these up
i think i appreciated these less than
you did when reading it but
you're right that's a really good point
okay cool
uh
anything else here
uh
quick question yep um
so on those uh on those graphs like on
the previous slide i think
um why do we see uh
like
really small scale like correlated
behavior it looks like almost sinusoidal
on these like do we know why that
happens oh yeah good question
i haven't thought about that yet
but they don't like mention anything in
the paper
i don't think so
okay
let's see so i a curve amongst all of
these
let's see so these are the differences
between
knocking out a head and
the general model performance
so maybe it has something to do with the
just the fact that maybe
yeah so not the the effect of the entire
model especially once you add in nlps
isn't just
the sum of each of the individual heads
um
so maybe there's just some like
compensatory mechanism where even if you
knock out any one of the heads the
others can compensate for it or it's
like particularly vulnerable to one head
or something like that
yeah this pattern is what
you see i think is if the heads were
like voting on a particular binary
question and it was in some on some
training datums the vote is closed and
then
cutting out any hit will change the
result yeah
but you're right you also see it for the
attention only models which
in theory should be basically a sum of
those paths across all the heads
that's weird
you're talking about like the
perturbations like the small bumps along
the curves right yeah yeah so there's
this shift there's like a little bump
here across all the different lines
uh even in these models you have like
soft maxes involved right and i think
once you have soft maxes anywhere you
can construct voting mechanisms
yeah that's probably a big part of this
okay
i don't have a clear picture that yet
but i think that there's some intuitive
picture to go forward
okay
uh argument four
is that to fight deceptive despite being
defined narrowly as copying random
sequences induction heads can implement
surprisingly abstract types of
in-context learning so this is to say
that um this is this is actually i know
circuits hasn't been around that long
but this is kind of more similar to the
kind of what i'll call classic circuits
work where they look at examples of
heads that fit the description of
induction heads and see what else they
can do
and there are lots of really interesting
behaviors that pop up
so
i showed you before that the classic
induction head pattern of this is that
if you have a sequence in the context
that is a then b and you later happen
upon a then the induction head will
induce you to produce b after that
they're going to basically make that a
little fuzzier and say that well okay it
doesn't need to be exactly a and b that
are being matched here it could be some
version of a conceptually or some
version of b conceptually and these
things still should have the same kind
of ways of thinking about them
it's also good to note that
though i'll be talking about different
behaviors than
this classic induction head version
all the heads that they analyze also fit
this pattern of exact induction head
matching in some cases so if they use
random tokens then they can extract
these same exact heads
by this definition of whether or not
they perform prefix matching and copying
by this definition
i guess they have
metrics for prefix matching and copy but
i think you get the idea okay do you
mean that instead of looking at the
previous occurrence of the same token it
might be looking at the previous
occurrence of a token that has the same
gender of the words associated with it
oh yeah yeah actually there uh
let me walk through the examples and
you'll you'll see exactly what i mean
but one example that is translating
between languages you'll see
the induction head quote unquote point
to
um
words that mean the same thing across
different languages
roughly
so it's not again it's not exactly a
it's a star
but let's see let's go to
behavior one
okay so literal sequence copying is
pretty straightforward
these these plots are
take a second to parse
but okay so you i'm going to focus on a
particular
token
so it's going to be number
and the blue is going to tell me
what
previous token contributed to predicting
the number that i have highlighted
and then the red is going to tell me
once i've reached this token in the
sequence and i'm trying to attend to
something
what is the model attending two out of
the pre uh previous context so in this
case i'm highlighting on number and it's
attending to four
you kind of have to
combine results from across different
highlights in order to make this make
sense so what i'm going to do is going
to highlight number and say okay i've
reached number i'm trying to predict
what comes next so i'm going to look at
the red thing to tell me what to predict
next okay it's going to give me four
okay
so now i look at four and then what
contributes the logits for this next
token it's going to be
four for this four
so that means that
this head
looked at this token four
and produced four as the output for it
so it's doing exactly what we thought an
induction head should do it looked at
the last thing that followed this token
or one of the i don't know if it was the
last version of number
it probably is
but it looked at the last version a
number looked at what followed it and
then
tried to copy it
and you can see that this pattern of
blue to red should just match
token by token
so again it's just doing that over and
over and over again if there's a
repeated sequence this is the first
behavior they mentioned that is literal
copying that is a kind of standard
induction head behavior
the second version of this
is translation so if i
focus on
english to french translation
it'll still have a similar blue red
pattern
but it's going to be a little blurrier
to to match
the change in word orders between
language
you'll still see that the basic thing
holds up though so if i go to temple
across these it'll still be
kind of similar i think that the german
here is probably a little messier
the french is is pretty clean
so what happens when you have different
languages where the grammar prescribes
different word orders
i don't know oh oh when they when the
grammar prescribes different word orders
um they
there should be examples of that here i
think
i think they talk about that it does
match the
the semantics of the word and not the
word order
it shouldn't be exactly blue red it
should be something related but
i don't see it here
i might just not be looking at the right
place
yeah
um
okay behavior three is is really
interesting so this is a
a task they set up for the network that
displays something like few shot
learning
so the idea is that they give it a
stream of
words where you a stream of lines and
you have um
the model the
the game is to produce a certain number
for a particular pair of concepts if you
have a month in an animal you're
supposed to produce a zero
if you have a month in a fruit you
produce a one
color in an animal you produce a two and
color in a fruit you produce a three
and so if you go
here
here's an instantiation of the game
they limited the
the visualization to
just the the colons and the numbers
but the way i think the way i found to
read this is that if you look at a
number
here
i'll click on this three
then the thing that the the previous
tokens that contributed to
what
how to predict this three were generally
the previous threes or something related
so we're looking at gray strawberry
the things that
helped predict this token were gray
grape so those match the pattern of
color and fruit
color and fruit
and then sometimes there are other
things that sneak in
but generally if you have a repeated
word so i think
let's see let's look for a repeat june's
probably a good example
no that was not a good example
that's interesting
okay
so for january apple
it's
having march and apple is a really good
match
september and apple is a really good
match
and so on
let's see let's see if there's anything
else oh snake so if the word is used
from before if you have a repeat it
helps predict this one a lot
i feel like there was a better example
that i found before
where it was the month or something
that's okay i don't i won't spend too
much time on
you can play with this but the basic
idea is that
this head that matches the induction
head
definitions from
from their metrics
also seems to underlie
some of the predictive behavior of this
view shot learning task
okay
right
so that means that
that
they can find versions of induction
heads that do
that that at least plausibly underlie
some of the behaviors that we think
relate to in-context learning
so again it's just kind of picking out
intuitive examples and showing that
plausibly these induction heads are the
ones that
underlie the behaviors in question
this is because um it's in practice it
doesn't do an if the previous word
equals that
word before
then but instead it takes the dot
product of the corresponding hidden
vectors and those
uh that that mecha principle corresponds
uh
contains numbers corresponding to the
um properties of that word such as a
language and
kind of work and so on and then
therefore can just ask how similar are
these words yeah that's right it's doing
like a fuzzier version of similarity
than exact matching
that's right
okay and how does it end up that then
you basically get these uh
word rectangles such as like man woman
uh
i don't know what it translates to in
french
uh you mean like the in embedding
vectors or something
like you know that you can with virtu
veck you can uh
go from man and woman and king to queen
by taking plus and minuses right and it
sounds like induction heads are doing
something very similar here right that's
exactly right i think that's the right
mental image to have
but they aren't doing exactly that right
what exactly are they doing instead
uh
[Music]
i don't think they mechanistically have
an account for these particular heads if
that's what you mean
okay
all they're able to do is show that the
heads are attending to where they would
be if they were doing these behaviors
and like
adding logic contributions in the right
places
and if they had an exact mechanistic
understanding they could cut out the
attention heads and replace them with a
hard-coded version of that attention
headed board
yeah exactly
yeah that's exactly right
and they would also have a much stronger
claim about you know the fact that these
were doing the right things
or that these were underlying in context
learning in general
any other questions there
okay
uh right
argument five for small models they can
explain mechanistically how induction
head work heads work and can show that
they contribute to in-context learning
furthermore the actual mechanism of
operations suggests natural ways in
which it could be repurposed to perform
more general in context learning that's
a bit of a mouthful from from the paper
but
i'm going to take the liberty of
paraphrasing it by saying that you can
think of really easy ways to generalize
the mechanism that they came up with for
induction heads
so
it doesn't seem hard that the model
would learn other ways to use this a
really similar mechanism
so again before all what they're going
to do is
have one attention head move some data
from
the previous token to the current token
and they're going to have the second
token look up
some matching information from the
shifted
spot to see if it matches the current
word
um but just like eric and gloss
mentioned before what if no what if the
the
previous token head isn't necessarily a
previous token head what if it shifts to
other places in the stream
then you would have a really similar
effect that does a more sophisticated
behavior
and if the second step of matching based
off of the current token if that took on
any other form then exactly matching the
current token then it would also have
more sophisticated behavior
so they think this mechanism could be
easily generalized to lots of different
things
and so it it's pretty likely that the
model probably finds ways to use it
given that it's found this first version
of doing that to begin with
this argument sounds like it's going in
the wrong direction causally like in
evolution in biology you can sometimes
say that this animal does this thing
because that would be useful for
reproduction but uh the architecture
can't look ahead like that right you
train it and then maybe it works but if
it works then you can't say that
early in training adopted this mechanism
because that would then later let it
generalize maybe you can say something
like that for there like 10 different
architectures that were tried and this
one worked why is this the one that
ended up working are they saying that i
think they're saying that if you can um
let's see
let's see if i can think of a good
analogy
if you're learning how to
how to walk upright on two feet you can
probably learn how to stand on one foot
for a few seconds for a few moments at a
time
like the things are so related that
you'd think that
some capacity in one direction would
imbue you with the capacity in the other
direction like you have to be able to
stand on one foot for at least briefly
in order to walk on two feet
i agree that gener that induction heads
are plausibly easily generalized but
why is an argument for
that's a paper uh this is an argument
that
yeah yeah they i that that's a good
point i think they're just saying that
i think this is all another version of
plausibility even if they're calling it
kind of mechanistic in this case but
um
yeah
i don't think they
know for sure
like like i said none of these arguments
are supposed to be knockouts but i think
they're just saying
it's pretty plausible that this
mechanism could be generalized i think
that's it
is it like if this mechanism could not
possibly be generalized then we would
just be confused because the model
couldn't possibly be working this way
and we don't see that therefore this is
like a necessary argument rather than a
sufficient one
i think they try to shoot themselves in
the foot by looking at this and missed i
think it's neither sufficient or
necessary for what they're trying to say
it's just that well i guess maybe it's
necessary it's necessary that the the
mechanism needs to be generalizable in
order for it to underlie most of in
context learning
and it is indeed
probably generalizable but i think
that's as far as they stop it
does it need to be though
can you just like do 80 and then the
rest is done by something else in any
case uh
i think it would be surprising to me at
least if literal copying would underlie
all the stuff we see with language
models
or even most of it
what i mean is also like it doesn't need
to be
even in this 80 50 scenario it would
still be responsible for most of the
context learning right
yeah if if the literal copying did that
then yes but i don't think that
that is
a fair percentage for what surprises
about in context learning and language
models
do you think that it does the
like
more than mer copying even
early on right after the
appearance right after the like yellow
region in the loss history
right after the sudden jump where the
introduction has introduced
because the loss doesn't change much
afterwards
yeah so that is where the in context
learning changes on average
and it's very flat after that
um one thing they say is that
the model that the loss still clearly
improves beyond that point and
a lot of learning occurs at that point
it's it's actually just
you could think of it as harder in
context learning still improves beyond
that point because
if you're trying to improve upon better
predictions it's going to be harder
so you're getting better at predicting
the early context and you need to work
harder to predict the later context
i think that i think the claim is that
that just keeps pace with the early
context learning
but they are circumspect about whether
or not that's that that
phase shift is all of in context
learning in some sense
you could still imagine a case where
there's a lot of learning that happens
outside of that phase shift that
corresponds to keeping up with the early
context learning
okay
um okay so
the last argument uh is just
is
just reasoning by analogy this one's
pretty short actually it's just that
they looked at a lot of met a lot of
metrics related to
induction heads
in context learning on average
and
the kind of lost trajectory stuff we've
looked at and they all look
strikingly similar to their eyes across
all kinds of different models so it'd be
really surprising if the dynamics are
fundamentally different between small
small models and large models
it's at least the simplest hypothesis
that the same mechanism is accounting
for
um
for in-context learning in both cases
though not necessarily the only one not
a knockout argument just that
it's reasonable to suspect that that's
the case
oh that's basically what i just said so
lots of these metrics look similar
uh the mechanism being the same is
simplest is the simplest explanation
uh
yeah all of them have the same sharp
increasing complex learning same rough
amounts before and after the transition
all of them trace similar paths in pca
space all of them form induction heads
maybe other mechanisms or forms of
composition formed during the phase
change as one qualification have
here's their summary of that table for
the argument that induction heads
underlie
most the majority of in-context learning
in large language are in language models
and transformer models i should say
they think their their arguments are
quite strong for small and attention
only models they basically nailed that
down i think
um
for mlps it's a little bit tougher
because
they haven't really characterized mlps
that well yet and so maybe that could
contribute some
to to the overall
proportion of in context learning
and the
the argument for large models seems
reasonable to them so i guess in that
sense it's medium but
there's nothing really
perfectly causal or
interventional to make it a knockout
there there's more stuff in this paper
that i haven't covered yet
there are some
there's a section on unexplained
curiosities which has a lot of really
interesting pieces in it i
also recommend you look at the paper for
this reason
um
but i think that's that's a decent
summary of six arguments if there are
any other questions happy to take those
and then
i came up with some other
discussion questions so
do you think that induction heads could
explain all of in context learning
that's not the claim but where do you
think the limits might be between
induction heads explaining everything
and not
um what would what would be further
experiments you might do to dig into
these results i think glasses come up
with quite a few at this point but any
further ones would be really good
um
and i think one bigger bigger picture
question i had is if induction heads
underlie most of
the large language models capabilities
in terms of in-context learning what
would this mean for alignment generally
but i think that's about it
well the last one would sure make
alignment a lot more tractable because
we could replace the transformer
architecture as an architecture of a
pile of induction heads and then we
would know much more what's going on at
any point in time that's certainly true
yeah good point
i think i guess with with
sorry yeah with the first uh point
um i guess i feel like with with um
you know as we scale the even more
advanced ais there will be kinds of
things we would consider in context
learning
um just intuitively to me it seems like
there would be you know
um
you know when an ai is going between
like very diverse
uh situations and with kind of
multimodal interfaces and things like
that
i guess i'm not sure if
induction heads would account for all
the kind of contextual
um
you know all the context there but um
it would be interesting if they did
so your point is just that
um
as things get more advanced you expect
ais to handle lots of multimodal
situations
and it would be interesting if this kind
of mechanism could generalize to lots of
different scenarios
yeah exactly and and i guess
initially i'm a little skeptical of that
i feel like there would probably be
other mechanisms involved
with
um yeah
yeah i agree with that um
like the diagrams you showed us with the
transformer models
they really show that
there's a lot more going on
when you get to that scale so
i think that
maybe they have found
a concrete mechanism for like small
scale models but
i think
like even the the concept of learning
gets generalized
for transformer
scale models
like the the 14
layer one
a lot was going on there yeah i think i
agree with that i think there's probably
more going on under the hood but it's
hard to know exactly what that is until
we've done more characterization work
probably
shiny demon does your
intuition there come from an intuition
that something as understandable as
attention heads couldn't possibly
underlie intelligence uh but some
even though perhaps something as
mysterious as
um
like advanced the algebra might
i don't know about that but
my biggest
issue with it is that
it didn't seem like any heads converged
to a specific level like i think you
said that too that um it wasn't the
transformation happening
for most heads like there were some
heads that were clearly
transforming to
[Music]
action heads but most of them were like
changing roles
the training went on
yeah so
i really like that the changing modes
thing i think we really need to
investigate on that
well these lines aren't corresponding to
heads they're corresponding to layers
and maybe some heads just get stronger
within a layer while every hit
what but there are six
wait what oh dang it okay
you win points
yeah another thing that's coming to my
mind is that rome
discussed a few weeks ago and um
you know where they were modifying
specific knowledge in the language
there it is and i'm and i'm thinking of
what bearing this has on it and
i was trying to look up the paper but i
couldn't find it fast enough to see you
know
which where did they locate that
knowledge was it in an induction head or
was it some other part of the
transformer model
um because that seemed like some kind of
what i would think of as contextual
learning
yeah i bet the hypothesis would be that
you shouldn't find much
um well maybe that's not true in the
generalized form but you shouldn't find
much
quote unquote conceptual knowledge in
one of these i bet because well yeah i
guess i guess it's just hard at least
for the the literal copying heads you
shouldn't find much
i wonder what extent it is true that
usually
um
additional features get tacked onto
the model but don't get taken out again
because instead of taking one out
usually you can just throw in another
patch that
handles the reason why you want to take
it out while the thing is still useful
for whatever training datum introduced
in the first place
that would sound like
a lot like the
hypothesis that
hates specialized but don't unspecialize
again
sorry if that wasn't like non-security i
don't think i'm fully following see what
what where
is that like there's this um
there was like this theory mentioned
today
that basically heads
are like cells in an organism and
themselves become specialized cells but
the reverse doesn't happen and i can
imagine this happening here too because
like
the incentives for a problem to get
fixed by introducing a feature are there
but instead of removing that feature
usually you can
and add a few extra lines of it i see
yeah that would make sense i mean maybe
maybe these things being
you know non-monotonic just means that
it's taking on some other role or
something like that that's related but
not exactly induction heads are not
exactly
uh prefix matching the way we think
about it
have people tried injecting fresh heads
into a model during training
i'm not aware of that
but i don't think i'm the best gauge for
it
i mean not exactly the heads if i can
jump in but there is work that works in
the weight matrices though
um
but i don't think that this is
necessarily
like you can trace it back to the set
analysis
but many of these results are kind of
in a way random maybe or that it doesn't
matter much like synthesizer is one
approach that just generates random
stuff
and works well so
the point is that the
mlp layers then do a lot of stuff that's
also the point that they make here right
that
much of it is actually happening in the
field forward players
maybe i mean i think they say it's
suggestive that the mlps might be doing
some of the work
but i mean at least the claim that
induction heads do most of in context
learning
it yeah yeah the encounter stones
yeah one reason i like pca graphs is
that they remove all of this confusion
because they don't depend at all on the
architecture
that's true yeah
i think it's interesting that in the
fourth layer transformer and
as you get to a deeper architecture
um the induction heads get earlier on
the training
which
with the scales they have it's not that
early but
since they like
maybe it's a simplistic mechanism that
smaller models
resolve right like uh fall back to
and yeah
bigger transformers they
discover
like they have an epiphany at that
breakthrough point
and then they find out that there is
more complex mechanisms that they can
start using so that's probably
something that
like even bigger transformers or
bigger training sets could show yeah it
does kind of seem like
the fact that the earliest layers are on
the top of the 40 layer graph suggest
that maybe those are just fed into other
things
that do more sophisticated things too
that's a good point
i guess on a on a different tangent um
if we're if we're just going generally
in terms of experiments
um
the general methodology at a high level
here that that's going on
um
is that
um
is essentially that you know or at least
like part of the methodology is that
they look at you know they come up with
some some strings that exhibit a pattern
essentially
um and then after that they
you know i guess they do ablations uh to
see how the model performs
um
i guess with respect to that pattern
that's super vague the way i'm
describing it but
like you
you
describe a string that requires you to
carry out a certain computation
to
carry out uh or to be able to predict
the next state of it um
and then
essentially what you do is that i guess
you would you would look at
um
the
uh
the the model's ability to perform that
that computation and what happens when
you mess you know if you if you zero out
the heads like they did or if you did
did other things
um
so i think that can be that is a fairly
general
um
sort of strategy uh
so like like here they're talking about
uh you know they have strings
specifically to look for this this sort
of induction pattern matching
um
but
it's
you know you could you could definitely
think of maybe you know more complicated
uh patterns
that you could come up with um you know
more complicated
uh strings
um and so one thing you could do
um
is you could look at uh the capability
of transformers to learn these strings
and you know as a function of the number
of layers right um in this case they
show you need two layers uh
to learn an induction head but you could
easily think of
you know there might be patterns where
you need at least three layers there
might be patterns where you need four
layers there might be patterns where you
don't have enough layers
you just won't have enough layers
yes in fact every dimension of the pca
graph corresponds to a distribution over
training datums and
like the score in that distribution of
trading data is like the average loss
over that distribution and then yes if
you look at the 10 dimensions of the pca
graph you are going to find 10 scores
and then as we have seen that one of
these dimensions in the
um
in the pc graph in the two-dimensional
one only becomes like change
changes a lot once you introduce the
second layer you could find similar
patterns with a third and fourth layer
especially as we have seen that for the
one layer model you wouldn't have seen
coming that introducing the second layer
would
make that change once you add the second
dimension
you see
i think that does at least kind of map
onto what i was hearing is that is that
right for you
how are you thinking about that
did you understand the pca graph
oh
oh the the the pca graph um
i
i i i'm gonna be honest i i i went to
get some food
let's let's go to i think this is the
most important part of the paper so you
take the loss over a bunch of examples
um not this one look at the others yeah
but you have to understand the loss
vector in order to understand the pca
but you take the loss to a bunch of
tokens in the training set
and you do this throughout training and
you run pca on those vectors
so if they're do you know what pca is
yes
so then you see this trajectory
throughout training um
are we looking at the the the weight the
the weights as a function of time
essentially uh no you have these um you
take a model during training you look at
a hundred uh snapshots over time on it
you draw a table of um on the one
dimension you have the 100 snapshots on
the other table you have all the lost
values that the model has for every
training datum so if you have a million
training datums maybe you include the
offsets that you have a million times
100. this is this is pca and losses
exactly
yes yes
pc and that's a million times 100 table
and now you look at the
100 points that the model takes during
training through that space so we see
that every every model starts
doing well on the same like doing the
same amount of well on the same training
datums then they improve on the same
training datums apparently up to the
point at the bottom where the uh one
layer model stops like visibly improving
though presumably they're still doing
something whereas the two and three
layer models now also become better on
the same kinds of um datums so now we
can
like
quite straightforwardly extract the
training data so that they become better
on and we will see that they will uh
like be
trading datas that include a lot of
repetition for example and then we can
for example cut out those training data
to train only on the rest and see
attention that's not developed
and you can do this in three dimensions
and see the next set of training data
that contain a pattern that the mobile
will then next learn and then
uh that's what you were looking for
right
me
yes yes you are looking for a kind of
pattern that
uh introduces a need for another kind of
thing like an induction head in the
model
so so so i guess
like what i'm describing here like that
is one way i guess you could go you
could go about looking for it is you
could
like the pca is a pretty general uh
technique right and they see they see
you know you have one change here
um
so that i guess that that'd be one way
of doing it um but i guess what i'm
trying to
or like what i was getting at is that
you can look for
more i guess computational procedures
that are happening in transformers and
you can investigate how
um
you know how like whether or not they're
they're like the transformer is capable
of that
um
by
you know constructing specific strings
that would that require you to exhibit
um
that property
uh because you're going the other way
you're going to talk about that make
sense
yeah sure i'm just like the pch thing
just lets you find the natural sets that
apparently the model has introduced
like components for
yeah it's just bottom yes
so one is looking for what the model or
what the model's behavior would lead you
to believe are important dimensions and
the other way is saying here important
dimensions i think a model should have
what do i need to do in order to succeed
at those
in a way the
thing they use with the like arbitrary
numbers written in
triangular brackets
those um
files sure look like they are gonna
produce a very
predictable kind of
pattern in the bottle and i can see that
the
stuff that falls out of that might be
very
like
people might attack their methodology
because of the kinds of training datums
they produced but if they produce their
training game by looking at the pca
they're extracting the set of training
data so that people can basically can't
argue with that because there's like
it feels like
three bits of complexity penalty
it's interesting
i guess following up on the phase change
thing then one thing like i know you
mentioned looking at multiple dimensions
of pca
um
i guess this would predict at least
theoretically
um that we should see
um
you know you know if we looked at a
three-dimensional plot
um
you know there might be a third phase
change or a second phase change
essentially or and then a third one
possibly
right
and
yeah it's essentially like you could get
more um
you could get more complicated uh or
presumably you would get more
complicated behaviors uh
as you as you go through these uh these
phase changes yes what we would be
hoping for is that if we take 10
different architectures some of which
randomly allow
a particular next component such as
because they have at least four layers
which is required uh then
for all of those at the same time that
we would see the paths stay together for
a while and then diverge and that is
what we would be able to
um used to deduce the existence of
another kind of component
you know this is actually really similar
though to the um
neural tangent kernel stuff this is
exactly how the neural tiny kernel
analysis worked you would look at the
the how the
law space develops and you'd see this
well at least in theory you'd see this
really regular formation so you should
be able to even
i don't know i guess by the kernel you
should be able to figure out what those
dimensions should be
i didn't feel that i understood the
neural tangent kernel stuff like under i
feel like i understand this
well let me think i mean if that's
corresponding then yeah i
like the fact that we need to
cover that again
well okay it is
yeah i believe yeah it should it should
be very similar right because
the neural tangent kernel stuff is
trying to describe how this loss
vector is going to evolve over time
throughout training which is exactly
what this is doing it's just doing it
without pca
so if you could connect those two then
we'd get a lot of information out of
that but that seems
easier that's new little tangent kernel
stuff do the part where we have multiple
architectures train them both and see
that they stay together for a while and
then diverge yeah not explicitly i bet
that you'd probably have to there'd
probably be some structure in the kernel
that would be similar or something like
that but
yeah but if you have to construct it
then complexity penalty whereas if you
just say pca and it just appears
i'm not trying to say that one
methodology is better than the other i'm
just trying to draw a connection
and by people can't argue with that i
mean they would be arguing with it's
difficult to reason
i wouldn't believe it
i have another question splash comment
but i don't
interrupt this thread
no i think i don't think unless it
i think that was a good break i mean i
think yeah
the the only observation is that the
neural tangent kernel should give you
good information about how that loss
vector should should unfold over time
through training
okay i'll try to look at that stuff
again let's
first yeah so
okay so revisiting those three kind of
discussion points the third one about
you know if induction heads uh
are really turning out to be really
central in this what are the
implications for doing that
um this will be kind of hand waving
i think it will help me understand
better and
thinking about how this kind of connects
to like
you know um
uh famous alignment strategies or
other concepts so like there's nothing
in these induction heads that looks like
mesa optimization to me
um
right and one thing that it does kind of
remind me
is factored cognition
um
it's almost like
this we there's this kind of emergent
uh
factored cognition resource where these
in context learning or these induction
heads which are kind of just
a relatively simple like copy sequence
copying thing that's that's very
composable
has uh you know these models figured out
a way to use these
to solve kind of complex language
problems and things like that just by
stacking them in different ways
nand gates don't look like mess
optimizers either and yet if you stack
enough of them you get them
true
yeah they do have i think a paragraph on
base optimization in the paper where
they say they don't see evidence of it
but they i mean they're not obviously
not ruling it out or anything like that
for this reason
um i'm i should note that i'm kind of
wincing at my own question at the bottom
here i guess i shouldn't say
i should say that they're not arguing
that induction hens underlie most of the
llm's capabilities in general just
specifically
well okay i think that's that's also an
interesting discussion point but that i
should just qualify that that's
what they're claiming
yeah so i think i think the fact that
when they do this pca that they have
this very big l at the you know in the
two-dimensional pca
um
that accounts for a lot of the shift in
probability across the training curve um
i think the fact that they do that
in a way that means that
this induction head behavior at least
like
the the change towards induction heads
is responsible for a lot of the
predictive power
um
at least like in in terms of like
absolute prediction space or whatever
the pca space they use um
for
data that we encounter on a day-to-day
basis so for the for the kind of you
know for written language
um
specifically uh
this sort of induction head probably is
doing a very large amount of heavy
lifting and i think that can tell us
something about
written language in general
um
when it comes to
you know the world and you know more
generally if we're if we're target you
know maybe we're looking at you know
code or something like that
although code probably has a similar
behavior um
you know one thing i might want to if
i'm just going to speculate here
is that generally i don't think humans
want to try to communicate using a
medium
uh
that requires much more complexity than
what you have here within
you know with uh
than what is there with an induction
head essentially um or that what can be
captured with an induction head or a
couple of induction heads or something
slightly more complicated than that
um
so i would expect that
uh what is required to predict human
language
for the most part
um or at least like a a big part of of
of human language is
fairly simple things like conduction
it seems like this is a this
the component of like just referencing
stuff that you've seen in the context is
seems really useful i bet there's just a
bunch of extra
statistical regularities that are packed
into this as well of just
factual knowledge or whatever else about
different things like
capitals of countries or whatever but i
think you're right that
being able to reference those at the
right time in these flexible kind of
ways to seem pretty important and more
important than i probably expected the
outset
yes um
in a
like coincidentally related manner but
like in secretly actually very related
manner perhaps you have that uh our laws
of physics allow for the um existence of
like a few kinds of particles that can
then that have been enough to combine
into everything that we know of
um
and the same can be sent about
landscapes i guess yeah that's right
that's a good point
that's a good analogy
so maybe that's just a pattern that
holds a lot more than it has any right
to because it's actually like
it follows all the time because of
like math that we haven't discovered yet
and that doesn't usually appear in
like human intuition because we didn't
need it in the ancestral environment
i think i think oh sorry go ahead
okay thanks
um i think that if it turns out that
the action heads are such a powerful
concept that can generalize such
such complex spaces of information
uh we should see
how they would turn up in multi-modal
contexts
because
like the concept of referring to past
knowledge
else that
is not at
hand right now
definitely turns up like in most of
knowledge and
most of information processing
that probably your brain does so i think
yeah
like
somehow
an experiment that
can use something other than tokens
could be formulated
i do think that i i predict that in an
image you would find a similar thing for
looking at a fact about a particular
pixel
i think i agree with that nato does that
pixel
that would be interesting to see you
know firsthand but i think that i would
make a similar prediction
models work like that do they have
attention that attempts to talk to
pixels rather than direct tokens i mean
vision transformers might because you'd
have these relationships that you can
query the same way
so you would have
you know what is the
what is the patch that preceded the
patch that looks the most like this
before or something like that
how do they do rotary embeddings if
there are two position
dimensions
i bet they don't use rotary specifically
but i'm not i'm not an expert on vision
i could transformers that there's like
just the obvious two-dimensional version
of complex numbers
okay let me yeah this kind of uh
strong contextual uh
skill out of something
kind of so simple and mechanistic like
this surprises me like i think before
these large language models and stuff i
would have expected
like
you know the only ways i had uh maybe it
was a failure of my imagination but to
be really
uh skilled with
vastly different contexts and stuff i
would have expected either you need to
hand code that or you need kind of like
a more general kind of a genetic ai and
so the fact that you can get this with
you know
it looks like you can get a lot of this
from these
sort of
simple dumb mechanisms composed together
is really interesting
yeah gpt2 is when i realized we could
get intelligence before we understand it
okay i think i should run but uh
i think that's probably a good point to
at least officially call it everyone's
as always welcome to hang out as long as
possible but
yeah thanks thanks for coming