Attention - General - Copying & Induction heads [rough early thoughts]

in our last video we talked about some
of the attention heads we found in large
transformers
but
uh we've saved the best for last and in
this video we're going to talk about
the most interesting attention heads
we've really found copying heads and a
subset of copying heads called induction
heads
um yeah so the last video we talked
about other things but in this video
we're going to talk about them and
they're very interesting uh for a number
of reasons but probably the biggest one
is that they seem really at the center
or like a very important part of the
story of how large models accomplish
meta-learning
this is the phenomenon where somehow you
can show large transformers
examples of things and they can learn in
their context to do new things
um or put another way um if you have a
longer longer context transformers tend
to do better and better at predicting
the next token
and so that seems like a really
interesting phenomena about
transformers and induction heads and
copying heads are going to give us a
hint as to how they do it
um so it's worth noting that
induction or copying heads are the more
general kind of head and then induction
heads are a subset of copying hats so
all induction heads are copying heads
but not all copying heads are induction
heads
now we've talked about copying heads and
induction heads before and we've seen
them well copying heads we've seen as
far back as the one layer of transformer
we studied right at the beginning of the
series so we saw
in one of our first videos that
if you attend you have these these heads
that will attend back to a token and
then increase the probability that the
next token is the same token or a very
similar token
um
and
that's what we call a copying head you
attend to a token and then you increase
the probability of that that token's
going to occur again
um and the characterizing thing about
uh
these models are these attentions these
copy attention ads and or at least one
very nice way to characterize them is
that the ov circuit the output value
circuit um had positive eigenvalues so
you could just go and look at the
eigenvalues um and go and discover all
these copying heads
now unfortunately that isn't possible
for large models
because
at least it isn't trivially possible
because um the mlp sort of makes it uh
mlp layers make it a lot harder to look
at things this way
but
one sort of very theoretical way like
you could sort of imagine if these heads
were in only an attention only models
you could define them as having positive
eigenvalues so just means that they
increased the probability of the tokens
they attended to being the next token
now
induction heads
are a subset
of copying heads
um and they're they seem to allow
models to do meta learning much more
efficiently and and
and extract
a lot more information from the earlier
context
and it seems like what they do
um so here we have uh just an example
from the first paragraph of harry potter
and which you've been using as an
example text because it's fun
where what they do is they
look
at the present token in its context
and they look for previous places
where similar things have happened
and then shift one forward and look at
last time i saw this
what happened next
and then they go and predict that
whatever happened next that time is
what's gonna happen next
this time
and so if you really sort of pull these
apart that's what they're what they're
doing
and the simplest version would be to
just look at the present token and look
if that was the previous token but um
often they'll actually use a little bit
more of the surroundings
one way you could think about this is
it's it's really a very simple learning
algorithm
that's running on top of like within the
context it's it's a meta learning
learning algorithm um it's basically
nearest neighbors and so you're looking
for uh the last time it's like similar
happened and then you're predicting the
thing that happened then
and if you find multiple matches you'll
sort of average them together and
increase the probability of all of them
a little bit
um so we have sort of nearest neighbors
going on as we learn that's that's
pretty cool um
one thing that's worth mentioning is
that induction heads because they have
to move information forward like this
induction heads only become possible um
in starting the second layer of a model
so the first layer model can't express
them because they they need to use an
attention head to go and construct the
keys for the induction head
okay so
those are
copying induction heads we've talked
about them in previous videos before
and although we're going to talk about
them in more depth now
um
but the
the really interesting thing
is
in addition to these sort of very
strict
straightforward coughing and induction
heads
there seem to be
a large number of what you might call
soft versions of them
different variations that are in some
sense doing copying or induction
but
not literally copying tokens
so we'll look at a couple of examples of
those um
and i think they they sort of paint a
picture of how
you know
doing really strict copying and
induction wouldn't be sufficient to
accomplish a lot of behaviors we've seen
these models but perhaps these these
variants um can allow
a much wider range of metal learning
behaviors
and that's that that's very interesting
to me
okay so um to give two examples we'll
first do an example of um a soft copying
head and then we'll do an example of a
soft induction head so
an example of a soft copying head
will be
a head that goes and maintains um sort
of pronoun agreement um or inference
between sentences or between previous
words so to look back and try to you
know whatever it needs to go and say
something
that
uh
implies either gender or plurality
it'll go and look at previous context to
go and try and determine you know what
is the right pronoun to use or what's
the right conjugation of a verb to use
so
that might just be you know if you're
about to go and say a pronoun look at
the previous pronoun that's a very
common one but maybe also look at other
words that might suggest things so you
could maybe go and you know if you're
going to say a pro have to say a pronoun
maybe you go and you notice that you
just said misses and maybe you're like
oh it's probably going to be she um
or
you know if it's going to be a
possessive pronoun maybe you look back
and you saw she and now you're going to
say a female possessive pronoun
or maybe
you just saw an and and that implies
plurality and then you say they so
anyways we'll talk about that more in a
second
for the soft induction example
we found an induction head that does
translation so it's like induction but
it's across languages
and so it's not literally copying things
and not literally going and looking to
the last time that it saw the same thing
and then predicting the next token but
instead
it's looking for places where
uh you know like we say maybe
i am which is sui and then you look and
see um what happened next that time
and then you're going to translate that
into english and increase the
probability
um
so
uh
yeah that's a that's a very powerful
thing to be able to do
um so yeah let's actually go and look
uh
look at these attention heads
um
so we've talked about this interface
before but uh to very very briefly recap
um
what we have here
is
uh
we just have a piece of sample text to
go and demonstrate the song this is um
the first paragraph of harry potter
and
we have the attention pattern
um of the head and if we hop over our
token we can see where it attends to um
by default it's colored by the maximum
value that's attended somewhere and we
can swap it around so we can color it by
how much it gets attended every token
gets attended to and then see which
tokens attend to them
and
we can also if we want rather than
and then look at the attention pattern
we can also look at the uh inflated
tension pattern so that's just weighting
things by
how much information is being copied and
we can also uh color things by the
attribution pattern so that's how much
every token
affects every other token
and this is really giving the game away
here and so what we're seeing
is these are all the tokens that really
have their probabilities increased by
this attention head
and if we hover over them we'll go to
see what what tokens did that so
remember that in our normal attention
pattern we go
from source to destination
but in the attribution pattern we go to
from source token to output token so
we're highlighting the token that is
affected
and
and looking at the token that
that it attended to so here they is
affected it's increased in probability
that means that that
attended back to
were and attended back to and and both
of those increase the probability that
the next token should be they
um and that makes sense because both um
you know were is telling us that there's
a plural conjugation and and
is linking together two subjects and
saying that there's there's going to be
a plural
um
now one thing that's worth noting is
this would have been much much harder to
go and see
um you know if we weren't using the
attribution pattern if we were looking
at the attention pattern um okay so it
would be it would be that well we can
kind of see that it's attending to there
but it's also attending to a bunch of
other tokens um and it's much less
obvious what's going on you know i don't
think i'd be able to figure it out just
by looking at this
but when we switch the attribution
pattern um things become very clear both
in
even just looking at which tokens are
affected um which tokens get attended to
um and and then especially looking at
the links between them uh will really
really help us see so and we can look at
more examples and here's another day um
again it's uh now we're going and having
this day
well it's a period that's attending back
remember but um they is increased by in
probability by the we use they as our
previous pronoun and were and and those
are all suggesting plural
um
what else uh i need to unlock that um
here's another day it's the same type of
thing people's plural day is plural
we're looking less at the earlier ones
now
um what about um here's a he
well we said he mr man those all implied
that there might be we might need male
pronouns in the near future
um her there was a she and a missus and
probably if we look at that she it's
looking at yeah mrs increase the
probability of it
um
and a big part of what i find so cool
about this oh yeah dursleys is plural
right so even if we didn't have they um
that sort of is is telling us you know
that that thing is primarily coming from
from dursleys being plural
um
you know this is
you know this is this is in some ways
like keeping the sent the the
you know its ability to create sort of
coherent text but there's a really
simple story which is just that it's
it's really looking at the
the word that's just what the
what the present plurality and gender
might be
and taking advantage of that so that's
that's a very simple thing that it's
doing and it's just really easy to see
once we can look at the
the the direct logit attribution pattern
so
it's a big part of why i find that that
tool uh really helpful
okay a similar type of thing is tense um
it's really this is gonna be the same
story so let's just jump into the
attribution pattern um
and this is gonna have a little i think
this might have a little bit of
plurality mixed in as well um
so
uh here for instance
uh why do we say that this is you know
this is a
past tense imperfect word well the
reason for that is we're looking at the
previous world okay that makes sense um
so that's really telling us what the
tense is um and you know each one of
these is going to attend back to the to
the last verb um maybe the last couple
of verbs and look at their tense and
then increase the probability that we're
in the same tense um
uh maybe maybe it's not that concerned
about whether it's perfect or imperfect
um
yeah now uh there's some singular versus
plural as well so where can we see that
so
yeah was has had was those are all
um
oh i guess had is actually uh
plural um
so that's an exception but um
yeah
had had
um
versus
as yet as another was
um and so there's maybe a little maybe
there's less of the plurality thing than
i i thought there was there's other ones
where i've seen that maybe more crisply
and that's what i do get for going and
doing this a bit more spontaneously
okay so those are
two examples
of copying and the reason we're calling
those copying is um they're sort of soft
copying um we're copying you know a
couple dimensions of the word maybe just
one dimension like tense or like
plurality or gender maybe maybe a couple
dimensions um in the case of the the
gender implorality one
um so we're not copying exactly the word
but uh you know conceptually we can't we
can't do the eigenvalue decomposition
for this one because it's uh it's
entangled with llps but conceptually if
we could what we'd expect to see is a
couple very large positive eigenvalues
and then a lot of smaller eigenvalues
because there's a couple dimensions that
we're really copying and then there's
maybe other ones that we care less about
um
this isn't an induction head because
it's not
um as far as we can tell looking at the
you know the previous word to decide
what it's going to do instead it's uh
really influenced by
the
uh
yeah it's really influenced by uh just
yes just looking at the recent words and
and waiting for a time when it needs to
go and say a verb and conjuring it
correctly or waiting for a time it needs
to say a pronoun and using the correct
pronoun
now the translation head is going to be
an induction head and it actually does
a small amount of induction um even
within one language but it's really
translation where it uh it starts starts
really acting
but
um before we dive into that i think it's
worth just sort of seeing you know this
this is actually an attention head that
would be really really hard to
understand if we were just looking at
the attention pattern and
uh i think it's worth just sort of
sitting with that for a second because i
think it it drives in
why these tools of the intuitive
attention pattern and the attribution
patterns that we we introduced in
previous videos
are useful for exploring attention ads
so one thing to notice that almost all
tokens attend to eot
almost every token attends to eot so
this is you know a lot of a lot of
previous papers have noted um these
attention heads that will attend to
um the first token um or to
uh to separator tokens or things like
that
um and the speculation is that it's that
it's a default and we'll be able to show
that we'll when we switch to the
inflated pattern that'll go away
um so uh it's only with the attention
pattern that we see that
um and later on it'll start attending to
periods as well so that's sort of
another default place that the attention
head can rest
um
and
we're actually not the first ones to
observe that not only does it rest on
the eot tokens but often
punctuation will also get sort of uh
used for that purpose i think it's i
think it's photo at all um that observed
that as well so
um maybe it was clark at all
um
yes okay so
but there are a few exceptions um okay
so the first thing to do though is we
should actually really uh go
and yes if we if we look at this for
instance if we switch to looking at the
source we'll see that almost all the
attention is going to eot
and we can hover over ut and see
everything that attends to it so it's
almost all tokens but a few tokens are
attending elsewhere especially to the
period that's everything over there but
there's a few clear ones that are
looking at very specific things
um
so a way that we can clean this up and
really get rid of the confusion of those
resting positions is we could switch to
the infrared pattern all of a sudden we
don't care at all about uot and we can
see we don't care at all about the
periods it's just these other words
where we're actually taking information
from them and moving it elsewhere
so that that means that we don't have to
guess what a resting position is and
what is interesting position we can just
actually actually know and understand
that that those those positions didn't
matter they were just dressing decisions
no information was being moved from them
but the thing that will be really
informative if we switch to the
attribution pattern
and maybe let's switch now to again to
looking at targets
um
and so for instance but well we're
looking at
um
the previous token was a comma and we're
looking at other places especially where
there was a comma
um maybe this is an exception but in
most cases looking at places where
there's a column then we went but again
um
okay but but they
and again
we have but they
uh comma but they comma but
and then they and then we predict
predict today by looking at those
previous copies
and this these other ones um
was
so mrs potter was mr dursley was
and
but the the place that this really gets
interesting you know these are
relatively small attributions it's doing
um a lot less induction than other
induction heads that we've seen
and the place test gets really
interesting
uh can't be demonstrated on the harry
potter text and this is an important
thing to keep in mind and you know
sometimes sometimes an attention heads
interesting behavior
only surfaces if you start looking for
you know rare examples where it does a
lot of work
um so it might be quite uncommon
um
okay so
this is an interface that's going to
allow us
um
to uh
poke around a little bit
and
uh go and type whatever text we want and
see how attention heads behave
um
so let me
let me go and put in some texts in
multiple languages maybe
i'll go and i'll put this on multiple
lines so that it's a little bit clearer
um
oh that's not right i'm not even
splitting the languages up right okay so
we have french in the on the top line
jimmy bell christophe which is canada
um and then we have
uh spanish which i'm not going to try to
pronounce
um
and then we have
uh the english
and
uh one thing that you'll you can
immediately see is if you look at uh
just the attention pattern so what
you're seeing here is for every token
where does it attend well there's these
two off diagonals which correspond to
attending back um
to i guess the
uh
yeah okay so this is this here is
attending back to english and then this
is attending to spanish so the second
um yeah so it's uh like it's
uh
yeah i guess this is
well okay let's look at the let's look
at the actual
uh tokens to go and get this i'm getting
ourselves a little bit confused but the
diagonals correspond to attending back
um
okay so
this isn't going to show us the
attribution unfortunately it's just
going to show us the attention
and but that'll in some ways we'll make
this a little bit clearer because the
part of the really cool thing um is that
we're going to do induction
so
um here we're on name
not only do we want to look
back to the previous sentence and find
the translations of name or my name
so jim appel
me
unfortunately i didn't learn spanish um
is
we want to look one forward
and see what what the next token's going
to be so we can predict is and so in the
case of
appel it's um it's just folded in to
uh the
the one verb
but here it's a standalone
um
standalone uh verb or
um yeah it is
um
okay well we can continue moving forward
um okay well again they've managed to
find the next not the same token but the
next token
and looking at the next token it's gonna
allow us to predict that it should be
christopher great
um and we can keep going and now we're
gonna predict that there's gonna be a
period
then the period we go and look one
forward um okay i
um and you know again these are places
where the words are pretty different
like
in terms of surface form jo and i are
very different but
yeah the the model understands that
we're doing translation
which is swing
um i guess here we're actually going and
having a little bit of an uh a mistake
where um period ju happens there um but
also here so we both of these are period
i um and so
now it's sort of skipping a little bit
forward and maybe
we might sort of erroneously increase
the probability of like as well because
we're looking at a fairly short horizon
and then we can keep going
am
um okay from and we're
we're getting right
yeah
there and a little bit on on doing
spanish as well
from canada
great period again
um and it keeps going um and we can do
this also with the the spanish part um
there's a little bit of it not working
that well at the start but then then
this head takes off um and yeah we're
now we're correctly going and predicting
the next token
and
so that's that's pretty cool to me um
you know you have an attention head
that's specifically doing translation um
and not only is it is it doing doing
translation but it's an induction
translation hat um so that's that's
really cool
um now we can we can do this with more
languages and so i studied
uh latin in high school
and i just looked at my latin textbook
and i got the first few sentences of the
first
um project so um
quintus este puer romanus quintus in
apulla habitat
um apulia este natalia scintilla
estamina romana and casa labarat quintus
is a roman boy
okay so we haven't quite got there but
now we're starting to see it's it's
starting right to look at the previous
instances of quintus
and look at the is great so we're
starting to get uh an is okay quintus
est
and yeah so it's not perfect but it is
getting some of this alignment um which
is pretty cool
and presumably uh you know latin is a
lot less common uh in the training set
um but we're you know now we're starting
to pretty confidently go and attend to
like you know how julia
um
est
is
um so unfortunately we can't see
attribution here but we do see
if we look at this that it does in a lot
of cases successfully
increase the probability of the correct
next token so that's
that's pretty cool
um
okay so back to
uh
our slides and yes and so this
phenomenon of soft copying and soft
induction um seems really important and
really uh generalizes our understanding
of these ideas of copying heads and
induction heads and
and so you could sort of think of the
space of coughing and induction heads as
you have these these two sort of this
larger category of copying and then the
subcategory of induction heads
and you can split them into soft versus
literal
now
the soft ones it's really hard
to
uh
in any sort of automated scalable way
that i i see at least um you know go and
automatically detect soft copying
because it's sort of a a broad category
maybe if we had the access to the
eigenvalues we could go and do that by
looking at um you know for looking for
large positive eigenvalues but only only
some um and similarly for for induction
um
uh it yeah it's hard um in these models
without and without yeah these these
models that have uh aren't just
attention heads but the literal ones um
you know we can just behaviorally look
um or or just like look for for
attention that tends to you know
increase the probability of the same
token they attended to
um i think that we we sort of um
categorize something as a copying head
if um if more than 50
of the time when it increases the
probability of a token uh it was
attending to
um the exact same token and then of
course i mean there's lots of cases
where token is extremely similar uh so
you know 50 you know maybe like like for
instance dursley versus dursleys or
things like this um you know just very
very slight variations
um and then literal induction um the
same thing um but the token not only
does it need to be the same token but
also the previous token needs to be the
same so these are these are very
strict rules for detecting them um
and by that it's actually um you know
about 4.5 4.6 percent of in our in our
large in this yeah pretty large model um
uh are
are doing copying and uh
similarly
um in a slightly smaller model about
four percent
um
and then induction we're seeing that
about three percent um of the
attention heads in the large models seem
to be doing um literal induction
and about two percent in a somewhat
smaller model
um
and of course you know for
for these ones there's there's probably
quite a large number of soft variations
i think um you know we probably i think
we see maybe more more of the soft ones
than we do the literal ones um so that's
just that actually a pretty substantial
fraction of attention heads are doing
sort of copying and induction
uh like behavior
and uh
that's
that's pretty cool and
uh you know i think we don't know how
much of meta learning is due to these
but it seems like they are really a
significant
uh part of what the model does and
uh it seems like they are at least an
important part of the story of how metal
learning works so that's something that
i find pretty exciting
um it's worth noting uh this is rare but
we do see some anti-copying heads um so
anti-copying heads um and sort of i
think often anti-copying sort of
induction heads so they they're like an
induction head they look for
places where something similar hap that
are similar to the present but then they
decrease the probability of the exact
same thing
and
one theory for this
is
uh
like if you're making a list of items
for instance um like maybe you're doing
a list of um
i don't know something you can like get
models to go and produce things like a
list of fruits or a list of
um
presidents of the united states or
something like this you know you want to
say something similar for every item in
the list but you don't want to say the
exact same thing
uh and so uh anti-copying maybe can help
increase the probability of that or you
know if you've
yeah there's a lot of cases where i
think you you don't want to say the
exact same thing and you can increase
the probability by by decreasing the
probability of that
um okay so the next couple of videos
will really be zooming in on induction
heads and maybe it's just briefly saying
you know why why do we think it's worth
doing that um
well
uh one thing is that uh induction heads
actually seem to create kind of a phase
change in model training where um well
we'll be able to find this bump on the
lost curve and it will just correspond
to
the discovery of induction heads and i
don't think there's any other case where
we know of uh
you know
like uh something that's that's on the
scale of the law score which is sort of
the most abstract level you could look
at a model and we can link it all the
way down to the circuits level um and
understand what's going on so it's it's
really this like you know
i think that
those of us who have been involved in
kind of this more circuity reverse
engineering uh
models sort of line of research i think
a question has been you know we're we're
sort of i don't know like if you're
trying to make an analogy to physics
we're kind of doing like you know this
very fine grain
um you know like i don't know like
mechanics of of like molecules banging
into each other's or something and
there's a lot of other really cool
research that's that's sort of more like
just statistical physics and very
abstract and you know here's a place
where we have a bridge like a really a
pretty substantive bridge so that's
really cool um and probably the next
video will be on that
um i think another reason this is really
exciting is uh you know we
this is the first time oh i finished
didn't finish typing the slide i'm sorry
um but this is i think you know it's a
theory of what's going on in meta
learning and meta-learning seems to me
like the the most mysterious part of a
neural network behavior or of large
language model behavior how was it that
these models are
you know they're they're able to learn
examples that we show them in context
how the hell do they accomplish that um
and so
and i think a lot of safety concerns too
that i hear um i think the maybe the the
biggest sort of
scariest theories i've heard of how
models could could really radically um
you know
generalized to new data in really scary
ways have to do with meta learning and
and how models are accomplishing meta
learning so this there's this idea of
mess optimization that um
that yeah you could imagine possibly
eating what's going on and if we can see
how metal learning works we can we can
kind of address that
and then the final reason that i think
this is really cool is we can actually
predict um meta-learning scaling laws
from this uh so uh the fact that
there's sort of this remarkable fact
that as context length goes on we get
better at predicting tokens um and we'll
be able to go and predict that uh
because we now have a theory of what's
going on uh in meta learning and and can
can then reason about what it what it
implies so those are all really cool and
we'll be able to talk about them
very soon


"Induction Bump" Phase Change [rough early thoughts]

hi everyone this is catherine and i'm
excited to be sharing with you today an
observation that we've made in some of
our smaller transformer models and some
analyses we've done that give us a sense
of what's really going on there so
unlike some of our previous videos that
take a specific piece of theory or a
specific small detail that we've
observed this video will take a macro
scale observation and slice and dice it
in a lot of different ways from
different perspectives using many of the
tools that we've introduced so far
so without further ado i'm excited to
introduce this phenomenon to you and
explain what we understand about it so
far
so here's the phenomenon we call this
the bump so i'm showing you two loss
curves here from small transformer
models the one in red is a one layer
attention only transformer the one in
orange is a two layer attention only
transformer and what immediately stood
out to us is that at a certain point
their trajectories diverge and the two
layer starts getting dramatically lower
loss and performing better than the one
layer beyond that point
and we first became particularly
interested in this after chris made an
accidental discovery about these
trajectories so he had launched a server
using garcon which we've described in
previous videos
so he launched a server with a model
that he didn't realize wasn't done
training yet so he was looking at a
two-layer model from quite early in
training and was surprised to discover
that it did not have any induction heads
which was disappointing because that's
what he wanted to study so he quickly
realized his mistake and then
waited for the model to train further
and the two-layer model later in
training did of course have induction
heads and he could perceive with his
analysis
but this was quite interesting to
observe that the two-layer early did not
have induction heads and later did and
in fact these two observation points
were on either side of the bump
so this led us to develop a hypothesis
that this divergence in training
trajectories between the one layer and
larger models had something to do with
induction heads so that's the main
hypothesis that we will be exploring
today
another reason that we found this
hypothesis plausible that the bump has
to do with the discovery of induction
heads is that induction heads can only
form if there are two or more attention
layers i'll review why that is in a
little bit
and i'll also say that you know this was
our initial hypothesis was that
discovering induction heads is what's
going on uh my current view is a little
more nuanced i think there's something
about really specializing as induction
heads really fully coming into their
role
that's important to the story of what's
going on here
so
let me introduce the rough
structure of how i'll present our
findings so we're going to answer three
different questions that you might have
about this phenomenon the first one is
generality so are we looking at the same
bump in different models
can this is this a generalizable
phenomenon the second question is
behavior so the model makes different
predictions before and after the bump
what specifically changes about those
predictions is it just overall doing
better across the board at every token
or are there particular types of tokens
that are changing in terms of the
model's predictions about them
finally we'll get into the circuits
level so what's going on structurally in
the model's weights and matrices that
produces the prediction changes that we
are observing
so you might wonder why this is
interesting i think this is interesting
for two big reasons the first is it's
evidence that the model's behavior can
change quite suddenly in just a few
hundred steps from before to after the
bump the model's behavior changes
drastically uh and in a really
noticeable way it shows up on the
overall loss curve but even more
strikingly when we drill down even
further and i think understanding sudden
shifts is safety relevant because one
type of concern you might have about
larger models is that they're behaving
in ways that you approve of
and then up until a certain point where
they start executing a different
strategy that you don't like and so if
these kinds of dramatic shifts in how
the model understands the world and how
the model's
internal structure works can happen so
suddenly that type of behavior is
something we really would like to
understand and to know how common that
is and how you can see it coming and
that kind of thing
i also find this interesting because we
can connect the circuits level analysis
with that macro scale
learning dynamic level of analysis
something that shows up in the overall
loss curve in the macro scale behavior
of the model we've managed to connect
all the way down to specific circuits
and specific attention heads at least in
these small models and with parallels to
large models and that's pretty exciting
that's more end to end than many other
things we've been able to do so far
so let's go question by question so i'll
start with generality and show you how
this bump occurs across different models
so in terms of generality i'll first
show you these small models that we work
with so this is the same graph i was
showing you before with the one layer
and two layer attention only model and
i'll show a couple more attention only
models on the left here and i'll also
show a few models with mlps on the right
so here i've added the rest of the
models in different colors and you can
see that the one through four layer
attention only and with mlps all show
some kind of a bump it shows up at
different places for each model there
isn't really a particular size trend
so the two layers somewhat intermediate
the three layer develops the bump later
the four layer earlier we haven't gone
into much depth about why specific
models develop the bump at specific
times but it does seem to be a pretty
general phenomenon and we see something
like this in our larger models as well
so here are models with
millions to a few billion parameters and
these also show a bump here the uh the
the x-axis is tokens rather than steps
but at about the same number of tokens
there is the same bump
um and i won't go in this video into
explaining why i think this is the same
phenomenon but in the next video on
metal learning that chris will present
he'll provide some evidence for why this
is the same phenomenon that we see in
our tiny models as well
so let's go into behavior so in this
i'll be looking at how the token
predictions change so and particularly
the data that i'll be analyzing is the
token by token lost trajectory so you're
probably familiar with the average loss
curve which is an average over many
different tokens
but that overall summary is of course
made out of
thousands of individual tokens distinct
and discrete loss trajectories that have
their own behaviors that when averaged
out look like the lost curve we're all
familiar with
for much of this analysis i'll be
working not with the average but with
that token by token broken down
individual lost trajectory set or matrix
and we'll be asking which specific
tokens losses are the ones that improve
at the bump
so let me just show you a visual sense
of what i mean by that token by token
loss so here's the familiar average
token loss curve
over the course of training step the
loss improves on average
now of course uh this is made up of
individual tokens so i'll show you an
arbitrary token from the harry potter
demo text and you can see that the
model's performance on this example
token uh looks very little like its
overall average i'll smooth this out
just to make it a little easier to see
and you observe that the model performs
quite poorly on this token over its
entire training lifetime
and
in fact it doesn't seem to get much
worse this isn't too surprising not only
because a two-layer model is just not
very smart but also because some tokens
are just intrinsically very surprising
and
no model can do particularly well at
predicting them
so i'll also add in a couple other
arbitrary tokens from that same demo
text and you can see that they have very
different personalities there's a lot of
idiosyncratic variation in terms of how
well overall the model performs and how
that performance changes over time so
these variable and idiosyncratic token
trajectories are what we'll be analyzing
in the rest of this behavior
behavior-related analysis
so one other thing i'll also point out
is that the bump itself can be noticed
in the individual token trajectories so
i'll be drawing some lines on this plot
the left line will be at the start of
the bump the right line the dotted line
will be at the end of the bump and then
we can see where that turns up in some
example
individual token trajectories
so here are those same two lines and you
can see that these tokens that i've
selected show different behaviors over
the course of the bump so this token
gets suddenly noticeably worse and
remains worse after the bump so it has
become somehow more surprising or harder
to predict
and other tokens seem to do better
another interesting thing is that before
the bump starts there was some kind of
flattening out or plateauing maybe even
a little worsening before this bump
related improvement kicked in
so what i'd like you to take away from
this is that individual token
trajectories contain a lot of
information that can help us understand
what particularly is going on at the
time of the bump
so i'll analyze these individual token
trajectories with three different ways
of summarizing them the first one is pca
so i'll look at the principal components
of that snapshots by tokens matrix
i'll also build something that i'm going
to refer to as the meta learning index
which is
a measure of how well a model is
performing at in-context learning how
much better it does at later tokens than
earlier
and then i'll also look at this before
and after token loss difference
that is to say the overall difference
between the token losses before the bump
versus after the bump if you subtract
those behaviors what's that delta what's
that delta vector in token space
so we'll go one at a time and break this
down
so in terms of pca of the snapshots by
tokens matrix i just want to make sure
to clarify what that matrix is so i
think many viewers will be familiar with
pca as a way to
analyze or get a quick understanding of
the main dimensions of variation in a
data set and so the data set we're
talking about in this case uh is the
token behaviors of all of these
different models by all of these models
i mean we not only have eight different
architectures we have four attention
only architectures four architectures
with mlps but also each of these
trajectories is made up of many
individual snapshots i have 200
different snapshots over the course of
training for each of the eight models
they're saved every 50 steps so overall
that's going to be 1200 snapshots whose
token behaviors will be checking out
and um
the each model has been shown 10 000
different contexts and a random token
has been selected for each context so
overall this is a 12 000
sorry 1200 snapshot by 10 000 token
matrix that we'll be doing pca on
here's the result of that in the first
and second principle component so the
thing that immediately jumped out to me
is that this bump
uh is very prominent so i've sort of
marked these locations uh right at the
bottom of the swoosh and these locations
at the top of the swoosh and those
correspond
quite precisely to the start and the end
of the bump on the overall loss curve
so you'll also notice that the one layer
models never undergo that swoosh they
just remain
in that original sort of trajectory
direction and it's the two layers and
further that are
undergoing this substantial
transformation mostly along the second
principle component
i won't be analyzing in much more depth
what these principal components are
the main thing that i want you to take
away from this is that straight away in
the most significant dimensions of
variation of this data set the bump is
extremely obvious perhaps even more
obvious than it is in that average loss
view so this tells us that clearly
something very major is changing at this
point of the model's development
so the next lens that i'll introduce on
understanding that token-wise loss
matrix is meta-learning or in-context
learning
so
meta-learning is the phenomenon that
later tokens are easier for the model to
predict by metal learning we mean sort
of learning how to learn so the model
has learned how to take information from
early in the context and adjust its
later predictions so
as a measure of how good at this the
model is we'll take the difference
between its loss on earlier tokens and
loss on the later tokens so specifically
we'll take the loss at token 500 out of
a 512 length context and subtract off
the loss at the 50th token
i'll point out we're using token 50
rather than something like token 5
because those first few tokens are
intrinsically very hard to predict and
quite noisy
so the 50th token removes some of that
variation while still giving us a
measure of how good the model is at
learning from early tokens to better
predict later tokens so this is the
index that i'll be plotting
just another way of visualizing what
that
measurement is at the top here that's
the loss on the 50th token for a
particular model then the bottom in blue
is the loss on the 500th token which is
better and increasingly better over
model training it's getting better at
meta learning over the course of
training and this difference here
is that metal learning index that i'll
be graphing and in the future upcoming
video on metal learning chris will break
this down in much more detail as well
so here's that meta learning index again
i've pulled out just the one layer and
two layer attention only and i'll show
the rest in a moment
as you can see the one layer attention
only model has
never really learned
particularly much meta-learning and the
two layer really nails it in particular
these two lines i've drawn here are the
start and the end of the bump and you
can see that regime is where the model
really
really comes into its own in terms of
its ability to use earlier tokens to
perform better on later tokens this
seems to be a major thing that that bump
related transition is all about
here are the rest of the models i
haven't drawn those bumps start and end
lines just to make it a little easier to
see
but what i'll draw your attention to is
that they all undergo other than of
course the one layer they all undergo
this steep transition between being
mediocre at metal learning and being
quite skilled at it
and
the location of this transition does
correspond with the bump quite well as
before the two layers somewhat in the
middle the three layers later the four
layers earlier as we saw before
so this is another lens that tells us
that the bump is a drastic difference
it's a very striking change in the
model's behavior perhaps again as with
pca it looks more striking in this view
than it did in the overall loss curve so
something very significant is changing
about how the model's behaving
so the final lens that i'll take on
behavior is this before and after token
loss difference so that's just a simple
measure of what is different in a
snapshot after versus before the bump so
i'll be taking two different snapshots
um all the data i'm presenting has
already been smoothed so some of the
noise is reduced but i'll just take a
sort of smooth
loss snapshot here and a smooth loss
snapshot here and this is that full
10 000 token uh multi-dimensional
multi-token uh loss and i'll just be
taking the difference of those
so when uh so that's as i said a
multi-token loss i've shown here loss on
the harry potter demo text
now of course this isn't 10 000 random
tokens but you can do the same operation
on a demo text like this
so the difference i'll be taking is
between these token wise losses after
the bump and the token wise losses
before the bump
if we look at these raw losses it's a
little difficult to immediately see
what's different although you can see
some straight away sort of with the
naked eye i'll call out just briefly
that this this phrase mrs dursley which
is one of the uh
most common repeated phrases in this
demo text
the lighter color here in this case is
lower loss so the model after the bump
is visibly doing better at this repeated
token sequence than it was before but i
won't linger here i'll go ahead and take
that difference and then the
deltas will be more visible
so here is that difference of the after
versus before and in this case the red
colors is better performance on these
tokens
i'll also say i've taken this example
from the four layer attention only model
because it was a crisper and clearer
depiction than some of the others so
that's a little bit of cherry picking
for the sake of clarity but the other
models did show a very similar albeit
somewhat fuzzier pattern
so the main thing i want to draw your
attention to is these repeated token
sequences that we're so familiar with
this in this particular demo text that
we use
often because of these repeated
sequences so of course after the token
space d the last time it saw that it was
followed by urs and so after the bump it
is much better than before the bump at
determining that that repeated sequence
recurs again uh dursley again shows up
here
and then dursleys has been observed at
this point which improves uh later as
well so that common repeated sequence
it's performing better at
it's also performing noticeably worse at
things that break that pattern so
previously when it saw misses that of
course was followed by dursley and so
when mrs is followed by something else
it is much more surprised by that after
the bump because it has learned to
repeat itself and so times where it
doesn't repeat and breaks the pattern it
gets here in darker green a lower loss
so and then other of our familiar
repeated sentences such as a small sun
which had already appeared earlier in
the text we're also seeing uh improved
loss on that as well and of course the
potters starts to repeat and those
repeats particularly once they've been
repeated over and over again also
intensify so this tells us
at least in a in a quick glimpse that
the major difference in the loss curves
after versus before the bump is on these
repeated token sequences
so let's just review quickly what we saw
in the behavior of the model on those 10
000 random tokens by 1200 snapshots data
set
so the first thing we observed was in
the first two principal components the
bump shows up very distinctly as a major
swoosh a major change in those
major dimensions of variation of the
model
we also looked at this metal learning
index which shows us that the two layer
and higher models become much more adept
at using early tokens to better predict
later tokens and that improvement
happens
over the course of the bump itself
and then we looked at this
delta between the after and before bump
uh token
token vectors and if we look at that on
a demo text it also shows these repeated
sequences is where the improvement is
showing up
so i'm going to move into circuits and
mechanism in a moment i want to just
make sure to review what we've already
covered in previous videos about
induction heads as its important context
for understanding the circuits that we
see
so
you might remember from a previous video
that the main hallmark of an induction
head
is two things the first is in the ov
circuit so that's the output and
value matrix
the uh the ov matrix has positive
eigenvalues selecting for similar output
so showing here if you have the token
potters so pot followed by the token
ters then uh the attention pattern from
pot
we'll look back to a previous instance
of ters and the output will be made
similar to what's being attended to so
that similar output shows up as positive
ov
eigenvalues the other the second
hallmark of an induction head
is that this in this attention from
pot back to ters the key information uh
is carrying the similarity to the
previous token
so the previous token was pot which is
similar to where the attention is coming
from so that similarity in the qk
circuit uh
is
allowing it to attend to tokens whose
previous token was similar to the
current one so those two different
properties the ov which gives it copying
like behavior and the qk which gives its
specificity to where the prior token was
similar those two are the hallmark of an
induction head
so importantly for why this was at first
a plausible hypothesis to us as to why
the bump was going on is that induction
heads can only form if there are two or
more layers of attention
that's because the information in the
key needs to get moved from a previous
position to the current position so
jumping back for a sec the key from this
urs token needs to have all of this
information moved into it
by previous attention heads in an
earlier layer this can't be done with
just one layer so this made it very
plausible to us that the reason that the
two layer three layer four layer and
higher we're showing the bump and the
one layer wasn't is that induction heads
are something that the one layer just
can't do
so i'm showing you here some example
attention heads these are from the two
layer attention models and i'm showing
these eigenvalues so in green as i said
the positive qk eigenvalues are
responsible for that specificity to
repeated sequences where the previous
token was similar and then the ov
eigenvalues are giving it the copying
like behavior where it outputs something
similar to the token that it's attending
to
and
the uh one thing to notice is that there
are several very highly induction-like
heads uh in the two-layer attention
model this is of course its uh second
layer um and i'll also draw your
attention to this score so i'm going to
use this score in a couple later plots
what i'm what i'm measuring here is more
or less
what fraction of the qk and ov
eigenvalues are positive and
specifically i'm taking the minimum of
the qk and the ovs because if only one
of them is positive and not the other
that doesn't count fully as an uh
induction head which needs both of those
hallmark so that minimum positivity
score of the qkov eigenvalues i will be
using to color some future plots to
highlight which ones are the most
induction-like heads
so let's get down to the circuits level
i think this is where it gets
particularly exciting and one reason
that we are quite excited about this
phenomenon and our findings is that we
can connect the circuits level all the
way to that high level behavior
so the two different lenses all present
on the circuits here the first is these
eigenvalue trajectories of the induction
heads uh over the course of the bump
i'll also do an ablation study where i
remove the contribution of different
heads
and i'll compare the deficits after
removing each head to the specific
difference that we see in that before
and after token loss delta
and
what we'll see here as i've been
gesturing at so far is the development
of these specialized induction behaviors
where induction heads have really come
into their role in doing induction like
uh
having induction like eigenvalues and
driving induction-like behaviors
so just to quickly remind this uh ov
eigenvalues are the similar output the
qk eigenvalues are similar previous
token so i'll be presenting two
different uh measures of how induction
like a head is the first i highlighted
was that score so that's sort of the min
positivity like the minimum over qk and
ov of the fraction of positive
eigenvalues i'll also look at the trace
which is just the sum separately of the
qk and ov eigenvalues which is telling
us just how positive
uh really are these eigenvalues
so this first set of graphs i'll show
you is that sort of minimum positivity
score what fraction of the eigenvalues
are positive and i've shown you here for
just the single most induction-like head
but i'll add all 12 in a moment once
i've explained what this graph is
showing so this is our most
induction-like head uh i've colored it
dark purple because by the end of the
trajectory it is over 99 positive
eigenvalues on both dimensions and you
can see over the course of its training
it becomes more and more induction-like
and during the course of the bump it
really comes into its own it goes from
maybe being 80 to 90 percent uh positive
eigenvalues to more than 99 positive so
it really specializes as an induction
head over the course from the start of
the bump to the end of the bump
so i'll now add the rest of the 12 heads
sorry first i'll just show a different
uh just a different head as you can see
from this eigenvalues plot it is not
particularly induction light
i've colored it light yellow because by
the end of its trajectory it is not very
induction-like and you can see it goes
through some oscillations over the
course of training where it becomes
varyingly positive or negative but it
ultimately after the bump settles into
having a not very induction-like roll
okay now let's put the rest of the
heads on this plot so this is all 12 of
the final layer of the two layer
attention only model
so there's a couple things i want to
draw your attention to here
one is again these heads that end up
being highly induction-like they sort of
continue to rise uh coming into the bump
in terms of how induction-like they are
and what we see during the bump is they
really solidify their role they go all
of them from being maybe 80 to 90
percent uh positive eigenvalues to more
than 90 more than 95 in some cases more
than 99 positive so overall there's a
subset of most induction-like heads that
really settle into that role
i'll also point out that many many heads
seem to be moving in an induction-like
direction coming into the bump and then
as these heads specialize many of them
turn back around and go to do something
else so this turning back around is part
of why i'm describing this as a
specialization process where rather than
a whole bunch of heads all deciding to
do a little bit of induction
a few heads really specialize into that
role and the other heads go and do
something else
so i'll also show the three and four
layer attention only models in a second
i want to just point out that i'm not
going to show the models with mlps
because this analysis in terms of matrix
eigenvalues is not well defined once
mlps are in the mix so i'll stick to the
attention-only head attention-only
models for now um there are ways to
using some heuristics map how induction
like a head is but i haven't done this
analysis at this
time so here's the three layer model
i've again taken the last layer because
that's where i observe these highly
induction-like heads so again the same
two phenomenon that i find interesting
seem to show up where the most
induction-like heads were already
starting to move towards that behavior
before the bump started
the bump really seems to represent a
solidifying of that role and at the same
time there were a number of heads with
somewhat induction-like behavior before
the bump and over the course of the bump
many of the heads just go do something
else
very similar in the four layer
i'll also point out these ones don't
seem to have as much oscillation as the
two layer i don't really know what this
oscillation is about but that might be
something we learn more about in the
future
jumping back to the four layer same two
things there's a collection of very
induction-like heads that really
solidify their role over the course of
the bump and there's a cohort of heads
that had started to do
somewhat induction-like behaviors and
the ones that don't specialize just turn
around and go do something else so
that's a pretty consistent pattern that
we're seeing in this data
so i mentioned there were these two
different scores i was going to show one
was that induction score which is the
fraction of positive eigenvalues we're
now going to look at the trace which is
the sum of the eigenvalues so that's how
strongly positive are these values
so here i've plotted for one particular
head again this is head eight which is
the
most induction like head in the two
layer attention only model and i've now
rather than taking the
minimum or combining the qks and ovs
i've left them separate so we can see in
the solid line the uh ov which is that
copying like behavior and in the dotted
line the qk which is that similar
previous token behavior
this is on a
symmetric logarithmic scale a symlog
scale
so one thing to notice is that this head
developed a copying-like output pattern
fairly early what it really specialized
into was that second order qk behavior
over the course of the bump uh it became
much
much stronger at doing that previous
token uh that previous token specificity
which is responsible for that repeated
sequence of token type behavior
um i'll point out this additional
deviation here it's kind of a kind of a
second bump it's not that important
for us right now this is where we turned
off weight decay in the training process
so it somewhat seems to have saturated a
particular value and then when we
removed weight decay it continued and
saturated the new value i'll mostly crop
this part out because it's not as
important in the graphs going forward
so the next graph that i'll show
i'll put the ov the copying like
behaviors on the left and i'll put the
qk which is that's induction specific
previous token similarity on the right
so here we've got all heads again
copying like behavior on the left uh
previous token specificity on the right
and what really jumped out to me is that
the copying-like behavior didn't change
very much over the course of the bump
for the most part it seems like some
heads did
change what they were doing but they
mostly specialized away from copying
like behavior in that range
what's very striking is it's this second
order qk behavior that substantially
intensifies uh over the course of of
this development and here i've colored
the heads by at the time their overall
uh induction score as previously so you
can see this one that ends up uh this is
again head eight very dark purple very
induction-like it's sort of developing
that induction-like nature over the
course of the bump so here we see
there's a collection of induction-like
heads that really learn this highly
similar
previous token behavior over the course
of the bump and there's other
collections of heads that just go do
something else
we haven't analyzed in great detail what
are these other
uh second order qk behaviors
it's less clear but it seems like the
process of having some heads specialize
in induction like second order qk
behaviors kind of frees up other heads
to go do something else
so one other view on these eigenvalues
that i'd like to show you is a little
movie animating their development over
time so
for a particular head let's again look
at number eight which is the highly
induction-like head in the two-layer
attention only
we've got the qk eigenvalues on the left
and the ov on the right and then we've
seen these two plots before so this
again is that fraction of positive
eigenvalues again and here on the left
is the trace the overall
sum of how positive the eigenvalues are
on that log scale so i'll just show the
movie through once and then i'll slow it
down a little bit and show you some of
these particular points
so if we watch this movie the first
thing we notice is the ov eigenvalues
develop and then the qks so that's a bit
of an interesting trend that we see also
replicated in these graphs where that
copying like behavior comes on first and
then the qks come online so we watch
this one more time we notice there's
this phase in which we're not seeing a
lot of second order qk behavior at all
and then it emerges quite suddenly and
it sort of emerges in those sort of two
pulses
as it reaches different uh different
levels uh before and after we turn off
weight decay
so i've showed you the eigenvalue
trajectories the final thing i'm going
to show in this video is an ablation
experiment that gives us another view on
these heads so the question that we're
looking to answer with this experiment
is how much is a given head responsible
for that bump specific change in
behavior so the way we'll do this is
we'll run the model with one head at a
time zeroed out so its contribution to
the residual stream will be replaced
with zeros uh and in that way we'll get
a new measure of the models losses on
various tokens
if it didn't have that head contributing
we'll sort of measure the change in
token losses from the baseline so we'll
take that difference
we'll look at these uh in its raw form
first but then the more interesting
analysis is if we compare the ablation
related change in behavior to the bump
related change in behavior this will
give us a sense of which heads are
driving that bump specific behavior
change
so just to get us oriented as to what
the raw data is uh the lower line here
in gray is the unablated two layer
attention only model as compared with
the one layer and then all of these
other lines are if you remove each of
the twelve heads in the last layer of
the two layer attention only as you can
see
some heads have a larger
impact than others they induce a greater
deficit when removed so the difference
between each
of these ablation lines and the baseline
is this change in loss that we'll be
working with so i'll show you those in
terms of what the raw data looks like
so these are the average magnitude of
the deficit that is observed when you
remove each head at different points in
training so as you can see at the time
that the bump happens heads seven and
eight are the
most important to the model's overall
loss the model performs uh with the
largest average deficit when those heads
are removed other heads don't seem to
induce a very large deficit
now
this plot doesn't show any particular
correlation between how induction like a
head is and on net how large a deficit
is observed when it's removed so i think
this plot is not particularly important
for our current analysis i just wanted
to make sure to show it to give a sense
of what this raw data looks like
i'll also show this one other way
where if instead of coloring each line
you know in this case solid purple based
on how induction like it is at the end i
can also color it uh based on how
induction like it is at various points
in training this is also just to drive
home that there's no particular
relationship between how induction like
a head is and how large its overall on
average uh contribution is to the
model's loss
however more interesting is if we
compare the token specific deficits to
that before and after bump difference so
just to remind you this is the
after minus before individual token
losses which if we do on 10 000 random
tokens this is a length 10 000 vector
that characterizes what the specific
nature of that bump improvement is
so if we compare that with the specific
deficits seen by each ablation then we
get a much clearer story so if we just
zoom in on this most salient curve for a
moment this is again head eight in the
two layer attention only which we've
seen time and time again as the most
induction-like head
and what you'll notice is that
particularly around the end of the bump
when head 8 is ablated
the deficit induced
is highly similar to the
bump related difference
another thing you'll see given that i've
colored these based on their uh
based on the the change in how induction
related they are over time is that all
of these heads that end up producing a
bump related effect or a
bump like effect are sort of coming
online as they become more and more
induction-like over the course of their
development through the bump
they become the heads that are really
responsible for or really driving that
bump specific performance improvement
so this is i think one of the clearest
demonstrations that the induction heads
are responsible for that bump related
improvement
so just to summarize what we've seen so
far
let me just review what these questions
were or these topic areas were that we
wanted to check out so the question of
generality we were able to see that the
same bump turns up in multiple models
behavior-wise we could see in a couple
different ways that the bump represents
a large change in behavior when analyzed
on that token by token level and what
the model seems to be improving on is
repeated sequences of text and then at
the circuits level
we've seen a specialized development of
induction heads over the course of the
bump and then some suggestive evidence
that this frees up other heads to do
other interesting kinds of second order
qk behaviors so just reviewing some of
these key figures here is our evidence
of generality where across different
model sizes uh the bump was shown in a
somewhat similar position and we also
saw suggestive evidence that chris will
show more of in the next video that
larger models have a similar effect too
we also looked at behavior in these
three different ways so first we looked
at the first and second principal
components and saw that immediately in
the pca view the bump turns up as a very
drastic change in the trajectories in
the two layers but not the one layer
and then we also looked at this metal
learning index so how much better
the 500th token is being predicted than
the 50th and over the course of the bump
metal earning improved dramatically
we also looked at this
before and after token delta vector and
saw that these repeated sequences of
text are where the specific improvements
are showing up and then breaks in the
pattern also seem to be performing
slightly worse if anything
we also looked at these eigenvalue
trajectories so we saw that these most
induction-like heads are settling into a
highly induction-like pattern
and other heads are specializing away
from that pattern into something else
looking at the ovs and qks specifically
we saw that the copying-like behavior
remained fairly steady and it was really
these second order qk behaviors that
were developing during the bump
finally we looked at this ablation study
which tells us that removing these
induction-like heads induces a deficit
in the model that is highly similar to
the before and after bump difference in
token loss predictions
so just recapping one more time why we
find this interesting for one it's
evidence that the model's behavior can
change very suddenly over just a few
hundred time steps and we'd like to
understand how these sudden shifts in
the model's weights and the model's
behaviors show up because in
if we're trying to ensure that a model
will continue to perform in ways that
we're happy with particularly if it's a
quite capable model or it's in a very uh
safety critical context then these
sudden shifts over training are a type
of phenomenon we'd like to get a better
handle on we also find this exciting
because the connection between circuits
and learning dynamics uh is one thing
that we're striving for in our
interpretability research program and
this is an ev this is an example of us
making that connection and learning
something cool about both in the process
so i hope you enjoyed this video and i
encourage you to move on next to chris's
video about metal learning that will
connect induction heads to metal
learning and also go in more depth into
some of the phenomenon we've seen today
thanks so much