1L Attention  - Theory [rough early thoughts]

in our previous video we talked about what's
basically the simplest model bearing any
resemblance to a transformer you could imagine:
the zero layer transformer. But now we're ready
to go and graduate to a more complex model, the
one layer attention-only transformer. Now it's
only a little bit more complex, but it will be
able to go and give us some really interesting
properties to study and I think will help us
understand larger transformers in the future.
Now we're going to have to work through
a non-trivial amount of theory, so
it's worth maybe just briefly talking
about why this is worth investing in
or why I think why i think there's some
payoff to working through this theory.
And the biggest reason is that we're going to be
able to in principle fully understand a toy model
and, well, I guess we we often, I feel like we
give up on fully understanding neural networks
and here we're going to be able to go
and and take you know something that's
that's sort of a small transformer and we're
going to be able to get to a point where we
will just completely understand this model. Now
we'll have to look at some large matrices and
you know we won't be able to keep it all in our
head but we'll we'll be able to sort of explain
all the behaviors model by consulting some large
matrices and in principle fully understand it.
We're also going to develop some conceptual
tools that i think are really useful
for reasoning larger models and will continue to
help us as we as we go forward to larger models.
and then finally actually it
turns out that even though
larger models and sort of genuine transformers
are going to have more complex circuits
and and more complex behavior some of the things
that we observe in these models these this toy
model are going to reoccur and sort of have some
echoes and so we'll see them again and that'll
sort of get some useful intuition
from studying this model i think.
Okay so, we're studying a simplified model. And
as the name suggests the zero layer attention
only transformer is -- or sorry the one layer
attention-only transformer -- is a one layer model
and it's attention-only, which means that it has
no MLPs. But there's two other smaller... those
are those are the big simplifications we've made,
but there's two other smaller simplifications:
we're going to get rid of the layer normalization
and that's just because layer normalization
and we think probably isn't a critical part of the
story and but it would add a lot of bookkeeping
and a lot of additional work to think through in
our theory um and similarly we're going to ignore
the biases or get rid of the biases because again
those would add a lot of complexity now the the
model that i'm actually looking at when i or will
be looking at when i give you empirical results
uh later on and that'll be in another in the
follow-up video to this one uh will have does
actually have uh layer norms and biases um
but i'm gonna light them to keep things simple
okay so really if we want to talk about an
attention-only transformer there's three steps
um or at a very high level at least you could
summarize in three steps which are we're gonna go
and uh embed our tokens so we'll we'll represent
our tokens as one-hot vectors and we'll multiply
them by the embedding matrix then um for each
attention head we run the attention head and add
it into the residual stream so that corresponds
to we have all these attention heads over here
we're going to add them in to the residual
stream this line down the middle and then finally
we're going to go and multiply by
the unembedding matrix to get the
logits so that's how we'll get the logits and
that's that's the output of our model okay so
uh of course we need to you know in order to
actually understand this we're going to need
to dive into the attention heads in a lot more
detail and a relatively standard way to describe
the attention heads um might be something
like this equation that we have at the top
uh so what is it saying well
it's saying something like
um first we produce the value vectors
by multiplying the residual stream by Wv
then um we go and we multiply and we go and
weight all those value vectors by our attention
matrix um so that moves uh m goes and combines
the the value vectors for different tokens
um and that weighted combination gets multiplied
by Wo so we can add it into the residual string
okay well uh that's that that that is a a formal
you know a definition of a of an attention head
but it's actually a kind of tricky definition
to think about and i think it obscures um
a lot of important facts about attention heads uh
and that's just kind of tricky because you know
we on the one hand we have like matrices matrix
multiplies we have to think about on the other
hand we have you know this weighted combination
that's kind of orthogonal to them and it's
kind of complex and there's another way we could
describe it uh which is using tensor products
so you might not be very familiar with tensor
products um but uh they're actually they're
they're very convenient and one way to think
about them is that they allow us to accomplish
the thing that uh we we often accomplish by
broadcasting or thinking about um uh you know
multiplying certain dimensions when we when
we program in tensorflow or numpy or pytorch
um so yeah it would be very common for us to
say okay well you know first we're going to
go and multiply the vector uh at each position
for each token by Wv okay well the way we'll
write that is we'll say that Wv is on the right
hand side of the tensor product um and identity
on the left-hand side and that just means we're
going to multiply every when we when we just have
a matrix on the right hand side it just means we
uh we multiply the vector for every token by that
matrix. Okay well the next thing we need to do
is go and weight things by the attention pattern
okay well that's that's instead of going and
multiplying um every the vector for every for
every token um independently, now we're gonna go
and independently mix the every every component
um across tokens, so we're multiplying across the
token dimension instead of multiplying across the
um the the i don't know the the vector dimension
or the model dimension or something like that. So
um that we're going to represent that
by multiplying on the left-hand side.
Okay well then we need to go multiply by Wo and
multiply we're going to multiply every vector
the vector for every token by Wo so that's that's
like the thing that we did on at the beginning
earlier when we multiplied on the right hand
side we're going to multiply on the have Wo
on the right hand side and that just means we
multiply the vector um every tokens vector by Wo.
And one thing that's nice about tensor products
is they have this identity that all the things
on the same side you just multiply them together
all the things on the same side combine and and uh
all the things on the other side also combine and
they're just completely independent. So if we want
to understand what, if we want to multiply these
together and combine them, we look at, for the
right hand side we'll go multiply by Wo and then
id and Wv, and id collapses so we just get Wo*Wv.
And on the left hand side we have an id and then
A and an id so we just get an A, we just have the
attention pattern. So what that's really saying
is that this the the attention patterns action is
sort of independent of the Wo*Wv action and
and they're they're sort of separate things
so that's one that's one way you could think about
it um another way to think about this is you know
what is an attention head fundamentally doing an
attention head moves information from the residual
stream of one token to the residual stream of
another token and when it does that it has to
pick some subspace of the the the attention that
it's moving information from it has to read some
subspace and it has to write that to that sub the
information that it read the vector that it read
to this a different subspace and the residual
stream of the of the the second token well
Wo*Wv describes which attention head or sorry
which which subsidies we read from and write to
and a describes which token um information
moves from into so a move describes
what the the token that gets read from and
written to and Wo*Wv describes the the subspace
of the token that we're reading from that we
we read from and and where it gets written
so that's a pretty cool way
to think about attention
okay so now we can go and plug that
into the previous definition we had
of an attention layer where we were
going and adding all the the different
attention heads and and then going and adding
them to the residual stream and you know there's
another way we could write that which is uh you
know if we fix if we fix the attention pattern now
that's that's the attention pattern is computed in
a very nonlinear way but if we pretend that it's
a constant or that it's fixed um well then all of
this is linear and so we could go and write it as
a a linear transformation that acts on x0
where we have an identity plus um all of these
um well they're they're they're they're
tensors in the mathematical sense
and we can even plug that into our you
know plug all of our equations together
and we get this this final uh final end-to-end
description of a transformer in some sense so
um the identity term becomes Wu*We that's you know
you can think of that as um well it's gonna it's
gonna be kind of like bi-gram statistics and uh
for every attention head we have this term here
and we'll return to this um this
product on the right hand side it's
it's really interesting um and in some ways
is yeah one of the most most informative
things we can use to think about in
this model okay so um that first term
is going to it's it corresponds to this direct
path down here this one here and it's going to
tend to represent bigram like statistics remember
in our previous video we saw an identical term
uh in our zero layer transformer and it was
exactly the bigram well an approximation of
the the bi-gram log likelihood that was exactly
what it was trying to do and here we're going
to see that it's it's going to do something a bit
similar some of the diagram information will move
into the attention heads but the remainder
will will continue to be on that direct path
but we also have terms corresponding to all of
these other paths right so that's those are all
going to be in that sum on the right hand side and
the sum has each one of these parts of the sum is
a tensor product and yeah the this describes where
the attention heads attend um and this describes
if the attention head attends to a given
token how does that affect the logits so
if we attend to a given token how do we affect the
logits so that that that's very interesting that
that sort of tells us a very large story large
portion of the story of what what this model's
behavior is going to be now the thing that
we're missing is we still need to understand
how the attention patterns get created and then
we'll then we'll basically have a complete story
You know i mentioned this earlier but
it's it's worth noting that this this is
if you fix the attention pattern it's linear and
that that's something that we'll be able to get
a lot of leverage out of and in general i think
any time you can go and split a function your very
complex function but then you can split it into
two things where if you hold one thing constant
it's a very simple function if you hold the other
thing constant it's a very simple function that's
a that's a nice point of leverage and um here's
an example a case where we have half that kind
of large so that's that's very nice okay so how is
the attention pattern computed well the attention
pattern computed is computed by going and dot
producting keys and queries so we dot product
the keys and the queries but okay how are those
computed well the queries and the keys are going
to be computed by taking the residual stream and
multiplying by Wq or Wk and the residual stream
is just the token times the embedding matrix one
hot token is represented as one hot factor so um
so if we if we combine this together we get Wk*We
and Wq*We and if we combine all of that together
we'll find that the attention pattern um is of the
form of a soft max of um the tokens on both sides
and then this matrix in the middle and that looks
very similar to that matrix that we saw earlier
and we'll see that it's really telling us which
tokens want to attend to which other tokens okay
um yeah so we we have these two really interesting
uh matrices and they're both vocabulary by
vocabulary size matrices and they seem to really
be at the heart of this model's behavior okay so
why why do we have those matrices now i can see
a little bit mysterious like the products of four
different matrices together and um you know why
why those matrices and what exactly do they mean
well okay let's start with the
first one we're going to call that
um the output value circuit
or the output value matrix
um and that's its form and what's going on here
is if we say okay well fundamentally what it means
is it it it's saying if you attend to a
token this is how the logits will be affected
and if you want to understand what that effect
is well okay we start with the token that we
attend to we'll call that the source token
and we go through w e so we embed the token
and then we have to convert it into a value vector
so we convert into a value vector by multiplying
by Wv so we've gone through We we've gone
through Wv and then we have to go we go and the
information gets moved by the attention pattern
and then it gets hit by Wo okay so um we have to
go because we're going to go and add it back into
the residual stream so we need to multiply by Wo
and get hit by the unembedding and
that causes a change to the output
and uh in the simplified model that's that's
exactly linear so this matrix here tells us how
uh what effect every token will have if we if
we attend to it on the outputs okay but the
second circuit this query-key circuit it tells us
which tokens want to attend to which other tokens
okay so if you again let's start with
the token that's being attended to
and if we run it through the embedding
matrix and then multiply by Wk we get
the key and if on the other side we're on the
token that's going to do the attending we go and
multiply by We and then by Wq we get the query
and so those two paths then meet in the middle
and we have to transpose this side
because to go and make things work
and that gives us this
matrix which tells us how much every token
wants to attend to every other possible token
now this is ignoring um the the attention
pattern also cares about positional embeddings or
some other probably you have to account
for position in some way and there's now
a variety of mechanisms for for doing
that um i'm going to align that for now
and we'll see there are a few attention heads
that will care about position and we'll we'll
need to go and just sort of uh put that off
to the side um but you you could make this
you'd add an if you were if you're just dealing
with positional bettings you'd add another term
um something that describes the uh the which
positions want to attend to which other positions
and which positions potentially want to attend
other tokens that would probably be very small
um but uh if we're just yeah we ignore the
positions that's that's exactly right okay so
that is um the theory of uh the theory that we're
going to need to understand one layer attention
only transformers and if you're interested uh the
next video will actually leverage this then to go
and understand uh one layer yeah understand the
behavior of a one layer attention-only transformer


1L Attention - Results [rough early thoughts]

in our previous video
we developed some theoretical results
for reasoning about transformers
and in this video we're going to take
advantage of those to try and understand
an actual one-layer attention-only
transformer
and to to really get to a point where
where in principle we could fully
understand it if we were willing to put
in the effort
so that that's kind of exciting
now to recap um the key idea from the
previous video was just that we we have
this uh
these two circuits the ov circuit which
tells us how an input a token if we
attend to it will affect the logits
and
the qk circuit which tells us which
tokens want to attend
to which other tokens
and they they have the they correspond
to these these matrices which we have
have over here
so that's that's the key idea
um that we we need
and so the the ov circuit
effect it sort of describes how the the
source token the token we intend to
affects the output
and the qk circuit how how it affects
the which tokens want to attend to it so
the destination so the one one link's
the destination token to the source
token and one links the source token the
output token and so very natural thing
to do
is to go and say well
uh okay
let's think about a single source token
since that's the one that both of these
circuits connect to
and let's ask
first um
which tokens want to attend
to that token
and if it gets attended to
what is its effect
on the output
and so for example we can see that the
token perfect
well the token r
wants to attend back to perfect
and then if we attend to it
and we'll increase the probability of
perfect and also to some extent super
and absolute and that means that we
increase the probability
of what i'll call a skip trigram where
we see the token perfect and then we we
jump over some tokens
and then we see the token r and increase
the token perfect being the problem
increase the probability of the token
perfect being the next token so we we
sort of say perfect later on are perfect
or it can be other things too because it
can be it could be the other tokens that
attend back and the other tokens that
get increased in probability so perfect
look super
and things like that of course you could
also look for the tokens they get
decreased in probability i think the
ones that get increased and probably the
most interesting
and we can do this for a bunch of things
so
uh large bunch of tokens want to attend
back increase the probability of large
and small
and that results in increasing the
probability of skipped trigrams like
large using a large
contained small
um so it's kind of this like copying
aspect to it or
copying or using similar words and where
something far essentially quite far back
in your context can now influence your
next token
for numbers we see two and
the token one wants to go and attend
back to that and predict two so we say
two and then one two but it can also be
two
has three and there's obviously lots of
other skip diagrams that are getting
getting affected
um the next couple are interesting so
this lambda one is really related to
uh
latex which is a type setting language
for scientific documents
and lambda is a keyword that's um it has
to be preceded by a backslash so we see
all these cases where there's
backslashes that want to attend back to
it and then increase the probability of
lambda
so we have lambda backslash lambda
and but also other things that that
could be in latex um backslash escaped
and are special commands like operator
um this nvsp one uh is from html so in
html there's there's some characters
that you
can't write
or you're supposed to go and write them
using using ampersand and then a special
special code or words that represent it
and and so nbsc is non-breaking space
and so if we've seen it once we maybe
next time we see an ampersand we think
we're going to say
go and write a non-breaking space
character and there's other other things
that could be like this greater than
it's another another one that it could
be
um and great
this is maybe a more normal thing
although that's related to titles um
like titles of books or something like
the great gatsby or something like that
um a little bit but the uppercase the
uppercase great
um
great the great
um okay but just stepping back um it's
worth noting noting here that we're just
we didn't run the model we just went and
took its weights
and we're reading algorithms off its
weights and these algorithms you know of
course there's there's
a similar we'd have to do this for every
attention but they're sort of telling us
the whole story of the model without
running it and we can just understand it
by by studying these matrices now the
matrices are fairly large and it could
be quite a bit of work but in principle
we could fully understand this model
okay now that copying is uh one example
of the kind of thing that it does it
does some other fun things um so this is
the first example here on this new slide
is uh is very similar to the previous
one we're gonna just copy indi um so he
yeah sometimes sometimes a single word
it's the tokenization system will split
it into two tokens and
and so we might have have the name cindy
and sometimes it gets split into a c in
indi um so we see indy and then we want
i go and say cindy and that's two tokens
the c
and the indie
and so we have c attend back to indy and
then we increase the probability of nd
or possibly all uppercase and d
um
a fun variant of that and i actually
didn't see this in
the model that i was using to prepare um
this talk unfortunately didn't have this
these are actually from a different
one layer model
but i think it's i think it's a very
interesting behavior
and sometimes you'll have a word
and
you look at the version where it's
tokenized as a single word
but it could also be tokenized as two
tokens and so then you have the the
tokens that could be the first part
of that like a a space p or a p
attend back
to the full word and then increase the
probability of its continuation
or things like pipe the the plural
version which i guess is sort of very
close to it being a continuation
and we can also have just similar words
that that so we could have spikes and
that starts with sp instead um so this
isn't quite literally copying but it's
also very interesting and um all of
these these other ones are examples
where um
a word can be tokenized potentially in
two parts ralph r elf
lloyd l lloyd
pixmap p xmap
and then it it tries to go and correct
that
um but do other fun things as well like
um you know all uppercase elf um i guess
with lloyd the reason we're seeing
atherin and a c over here is because
there's probably some famous catherine
that's either their last name is lloyd
or associated with lloyd
um in some way
um you might wonder about why is there a
cue for the pixmap um well there's a
programming language called cute that
has a class called q canvas um and so
and presumably it interacts with pixmaps
um
so we yeah we have all these sort of uh
interesting stuff hiding in these
matrices if we look at them
now i think one thing that's a really an
observation that had me kind of excited
about these is you know if you look at
neural networks and especially if you
look at language models sometimes the
cases where they they predict strange
things are very inscrutable and you just
have no idea you know why in the wide
world were the models just that
and
an interesting thing that i found
looking at these attention heads is that
they're kind of forced to introduce bugs
there's places where um in order to go
and and and sort of encode as many
possible these useful skip trigrams um
they necessarily add some bugs
so for instance for the pixmap one we
want to be able to go and say p xmap to
go and complete fix map
and q
canvas to go and say cue canvas later
those are both valid things and and
those those all make sense
but then
uh that that necessarily entails that we
must also have p canvas
because
the they have to go together in any
order and that seems like a bug as far
as i can tell there's there's no p
canvas object that's in some programming
language or being used and so that seems
to just be a bug that follows from the
way that we're encoding things and
similarly um with lloyd you know lloyd
and catherine both seem like a
reasonable things that could be
increasing the probability of but cloyd
and lathron both seem like they're
probably
probably bugs um that are just coming
from the way that things are being
encoded so that's that's kind of neat
that we can understand these bugs and
how they how they naturally follow from
trying to do something reasonable and
then just having limited expressivity
um so we've seen a couple sort of
abstract patterns and it's maybe worth
calling out um so one is that you you
see some token
and then you see a token that might
precede it and predict the same token so
you see two then you see one you think
ah you know probably i'm going to say
two again
another pattern
um is that you see a token and you see a
token that might precede a similar token
so
two and you see has and you think oh
maybe we're gonna talk about a number
again i'll increase the probability of
three it's not literally copying but
it's very similar
there's this version where you see a
token that
um
can be tokenized in a split way and then
you see the first part of the split
version and you predict the second part
it's sort of an interesting pattern and
then there's a variant on that um where
we see a split token or a token that
could a word that is represented as a
single token but could be split
and we see the first part and then we
predict
a variant the second part like all the
all uppercase version of it
now this might give you the impression
that uh all of the attention heads are
copying and that that impression
wouldn't be far from the truth from the
model that i was i was looking at as i
prepared this talk um 10 of its 12
attention heads uh are doing
are really doing this sort of copying
behavior but there were two that weren't
and those two actually are kind of
interesting because uh they're primarily
positional so remember the the qk
circuit that we're describing um for for
simplicity and because people handle
positional information in various ways
depending on whether they're using
positional bettings or or rotary
attention or things like this
um we alighted the the the
positional part um but some attention
heads care a lot about position and so
we have um some attention heads here
that primarily are gonna gonna care
about
um position
um
and so uh one interesting pattern there
uh is you'll have these attention heads
that mostly attend to the present token
and you'll say you know it seems you
might ask well why in the wide world
would it do that you know if anything
that it could do by attending to the
present token it could also go and do by
just going and and writing down that you
know we have we have that term in our
equation that just goes directly down
the residual stream and where the
embedding directly affects the
unembedding so it could do that but you
have these token um these these
attention heads that mostly attend to
the present token but sometimes attend a
little bit further back
and often in cases where the
tokenization like sort of spirit the
thing that sort of spiritually you think
of this vigram statistics really lives
with a token that's a little bit further
back so
um first let's look at one where it's
just literally doing bigram stuff so
we see the token corresponding as
probably our present token and we
increase the probability of two and two
and four and markup and with
so corresponding to corresponding four
corresponding markup corresponding with
these are all very reasonable things to
go and complete and those seem seem very
bigramish
but then we have an example
of the token cohen's
um
and presumably that's part of coincides
um actually in this case if you do look
at the the qk portion of the
uh of this there's the if you look at
the circuit there's a positional
component and a qk component and the qk
component
uh the token component uh is having the
token eyes attend back to
uh coincide
but the cohen's part is a lot more
informative than the odds part maybe and
so about what the next token is going to
be and so saying looking at currents and
being able to predict with or closely
you know coincides with coincides
closely um that that maybe is more
useful
similarly um if you if
if you have the token couldn't well just
seeing the apostrophe t and they get
split into two tokens and just saying
the apostrophe t doesn't really tell you
what's going on so
maybe it's more helpful to go and look
at the couldn't part and say couldn't
resist couldn't compete couldn't stand
couldn't identify
um and similarly when we have shouldn't
um looking at the shouldn't part is
maybe more informative than the
apostrophe t because since then say
shouldn't have shouldn't be shouldn't
remain shouldn't take and those are
actually kind of different um maybe than
the most likely things
uh
for for couldn't
um okay so
in summary uh our one layer
this i guess there's a few interesting
points so first of all these one-layer
attention-only transformers can be
understood by studying these these ov
and qk circuits and that's really neat
because uh understanding models uh is
hard and we often don't understand um
well it's you know i think i think
there's often a lot of patterns about
whether we can understand um neural
networks and you know this is an example
where we can succeed so that's that's
fun um really in principle we could
fully understand this model
um transformers desperately want to do
meta learning um i think this is
actually kind of um this was a very
surprising point to me like the model
goes and dedicates 10 out of its 12
attention heads to going and doing meta
learning stuff um how you know that it
wasn't all obvious to me that the model
was going to do that um
in fact i would have expected to mostly
be doing sort of
engram statistics stuff um but even
though the model's architecture is
actually quite poorly suited to doing
metal learning we'll see that it has
much better metal learning abilities uh
later on
and i think that
uh
yeah it's it's it's dedicating a lot of
its capacity anyways to going and
accomplishing that
um the constraints
of the system
force us uh force the model to introduce
bugs so going and trying to do represent
just sort of perfectly reasonable things
um these these skiff trigrams
intrinsically forces the model to have
certain bugs that are very inscrutable
if you just look at the resulting
behavior and so that's that's pretty
interesting
and then finally attention only
transformers uh are linear
uh if you freeze the attention patterns
and
again this this trick of you know
functions that are simple if you can go
and hold one thing constant um even if
the the whole function over as a whole
is very complex that's that's a very
nice point of leverage that uh that i i
always appreciate when i find something
like that
um well that's that's the talk
in the next video
we're going to go and discuss a way to
sort of automate and
much more quickly analyze one layer
similar models to this


1L Attention - Eigenvalue Analysis [rough early thoughts]

in our previous video we
were able to analyze uh a one-layer
attention-only transformer
but it was kind of uh tedious or at
least it would be if we wanted to do it
really carefully because we were looking
at these really big matrices
and in this video i'd like to talk about
a technique um of using uh eigenvalue
analysis to go and do that in a in a
much faster way and quickly summarize
lots of uh yeah quickly summarize these
models
so
i
you might recall from from linear
algebra that an eigen eigenvector
is
a vector
that
uh when you multiply it by some matrix w
uh just scales so maybe it gets scaled
by
um a positive value and gets larger or
um
uh yeah or at least sort of stays on the
same side it could get could get flipped
it could become negative but it's not
not getting rotated it's just getting
scaled
um
and
eigenvalues and eigenvectors um you know
to be honest i haven't had much of an
opportunity to use them in my
interpretation research before
uh because
they're really only useful
um when you're when you're mapping a
vector space
onto itself
and when a vector space when not just
when it's the same size that's sort of
the formal definition requires that a
vector space have the same
dimensionality like a matrix be a square
matrix to even calculate these but
really they're interesting when you
think of a vector space is conceptually
the input and the output spaces are
conceptually the same and that's when
eigenvalues and eigenvectors are
interesting
and
in the case of transformers we actually
have well you could see the residual
stream as being a case where you're
reading and writing to the same vector
space that's that's sort of a little bit
weak because you know the the vector
space the representation after you write
to it is a little bit different but
when you when the tokens the inputs and
the inputs because we're predicting we
our inputs are tokens and our outputs
our tokens those and we think of those
as as one-hot vectors so every token we
think of is a one-hot vector um in this
incredibly high dimensional space that
is a map from a vector space onto itself
and that means that we can potentially
study study eigenvalues and we'll see
that some interesting things can arise
now remember that eigenvalues are
complex numbers
so they live
on the complex plane and they can be be
positive
or negative
or imaginary
but
because
the
the the magnitude of our eigenvalues can
just vary by orders of magnitude and
we're going to go and put them on a log
scale
and
that will put the radius on a log scale
when we present them in this talk
and
the okay so let's
the case where we're actually going to
be using these eigenvalues is is when we
have things like the ov circuit in the
previous talk so we have um a map from
uh input tokens
to output tokens and it tells us how how
if we attend to a given token um
that will affect uh the
the logits of our predictions for the
for the output token and so when those
uh
eigenvalues are positive
and we have lots of positive eigenvalues
it means that we're copying it means
that we're increasing the probability we
have an attention head that's going to
tend to increase the probability that
whatever token we attend to um is the
same token that we output
now conversely um if we have a bunch of
negative eigenvalues
um
that's doing anti-copying it's saying
that if you attend to whatever token you
attend to you're going to decrease the
probability of that being the output
token
and we haven't seen anything like that
um really
in any of the models that that we we've
talked well in the model that we talked
about in the previous video or probably
in any models we're going to talk about
in the in the next couple of videos but
we will eventually see
some models that have anti-copying heads
um and uh it's not something that we've
studied that much yet but it it seems
like maybe um these anti-copying heads
are useful for instance if you have a
list and the model thinks that the same
item isn't going to be repeated in the
list then it can go and attend back to
previous items and decrease their
probability and even though maybe it's
also using
some kind of copying like mechanism to
increase the probability that similar
things come come later down
okay so the final piece
and this is sort of the the substance
almost the default or or neutral case is
to have imaginary
um eigenvalues because imaginary
eigenvalues
means that you're you're just affecting
unrelated tokens
um and so that's that's pretty useful um
like for instance the if we're doing if
we have an attention that's doing bigram
statistics you know
it's it's not that or the same token is
very is especially likely to be the next
token like if you're just literally
asking which token comes after each
other token it's not that it's
especially unlikely it's just that and
there's an unrelated set of tokens that
uh are very likely to come next and uh
the bigram statistics are are focused on
that so that's what our what imaginary
eigenvalues
mean
okay so if we uh look at the the
attention nets from the model that we
were looking at in the in the previous
video uh we'll see that ten of the
attention heads really have extremely
disproportionately positive eigenvalues
they're just in these tiny little
clusters and there's a couple where it's
a little bit less extreme but we've got
a lot of attention heads that are very
uh lean very very positive
and so that tells us all these attention
heads are essentially doing copying now
we can try to understand what the
details of what their copying is and
where they like to attend back and and
so forth but the the high level story is
um they're copying and then you know
similarly there's two attention heads
that are not copying and that's also
interesting and so at a glance we can we
can sort of get a summary
um now we can
in in the next couple of videos we're
going to start talking about a two layer
attention only model and
here are the eigenvalues for its second
layer
and we'll see that seven of its 12
attention heads are doing copying
and so again at a glance we can very
quickly go and see
that there are a bunch of heads that are
doing copying and the ones that are our
most intense about it are they have very
concentrated positive eigenvalues
um so and large ones so
really doing doing some copying here
now a useful trick that i found
uh is it can sometimes be useful to look
at the
uh the sum of the eigenvalues
divided by the sum of the eigenvalue
magnitudes
um and that value uh well it'll always
be a real number because eigenvalues
compares when they're complex and the
the imaginary parts cancel out and
because you're dividing by the sum of
the magnitudes it'll always be between
negative one and one
and so when this is one or close to one
that means that you have copying
attention heads if it's negative one it
means you have anti-copying and if you
have things that are in the middle maybe
maybe they're doing something else
and you can use this to make histograms
i've put little pictures of the
eigenvalue plots for each attention head
just to make it really explicit this is
a histogram of attention heads but
you can see that on the right hand side
we have the 10 attention heads that we
said were were copying heads
and then in the middle we have these two
heads
that were really doing more um
uh
i guess
yeah not not um yeah doing
basically doing bigramish or angremish
local stuff um that isn't copying or
anti-copying so this is this is a nice
way to quickly go and summarize uh
all the attention heads in a layer
and and figure out if anything like
cocking and anti-copying is going on
now
in our in our in the previous example
this wouldn't be very interesting to do
but uh as we work with more complex
models it's going to be useful to look
uh also look at the eigenvalues of uk
circuits so you know the the query and
the key are both produced from tokens
and so we get this matrix that again is
a the input and output space are both um
both tokens and so we can we can ask
about
uh
qk yeah the eigenvalues of the qk
circuit
and
what they really mean is
if they're positive
that means that we want to attend to the
same tokens this attention head when
it's on a token wants to attend to
previous copies of that token
and if it's negative it wants to avoid
previous copies of that same token
and then if it's imaginary it just cares
a bit about unrelated things
now that might seem a little bit strange
um
and it's it's because there's actually
some subtle like why would you want to
attend to the exact same token that
you're previously on well
uh
there's yeah the there's a little bit of
subtlety in the interpretation of these
um of these qk eigen values because
usually we'll be thinking about them in
the context of of deeper models
and we'll be able to talk about this in
in a bit more depth uh in the next
couple of videos and develop some theory
that will help us think about this
but uh one way you yeah roughly what's
what's going on is the the tokens
information is going to be shifted by
early retention heads the um there will
be attention heads that will shift where
the query and the key information
originate
and so that means that you can
when you when you see a positive
eigenvalue it might not mean literally
attend to the previous token that's the
same
but attend to a token that has some
relationship to a token that's the same
and if you can understand understand
that that can still be a very very
powerful tool for reasoning about the
behavior of those attention nets
okay so in summary
uh if we have
for the output value circuit
positive means copying
and negative means anti-copying
whereas for the query key circuit
positive means prefer same
same token attention
and negative means avoid same token
attention
now one final sad comment is this
technique does kind of break down as you
work with larger models especially when
as you work with models that have mlps
and the problem is that
the
the representation of tokens becomes
very intertwined
with
uh with mlps and uh the
the token embeddings really start to
migrate into attention heads and that's
that's sort of fine but um also into
mlps in a way that makes it very hard to
go and do this kind of analysis
um because we sort of can't linearize as
well through through mlps that's that's
a little bit sad but for studying small
models and especially for studying small
attention only models this is a really
useful trick for going and quickly
summarizing things and and getting a
quick overview of the model


2L Attention  - Theory [rough early thoughts]

in the last few videos we've had some
success trying to understand one layer
of attention only transformers
and so in this video we're going to move
on and try to study two layer attention
only transformers
now
this this first video on that topic is
going to just be about
some theory that we need to build up to
to do that
but i think it is really quite
worthwhile
and i found that the ideas that we can
build up uh studying two-layer attention
only models uh really actually
give us some useful traction on thinking
about transformers more generally
and we'll also when we when we in the
the video following this one get to the
point of actually actually empirically
studying what's going on inside these
models we'll find a bunch of things that
seem to be really important mechanisms
that exist in transformers of all sizes
and uh without any of the
simplifications that we're we're using
right now and so i think i think it's
pretty interesting to study these
so
recall that
in the the previous videos we developed
a pretty nice equation for
describing transformers we
were able to go
and describe them in terms of well first
we have this direct path term that
corresponds to this path here
and then we have
these terms corresponding to attention
heads which correspond to
all of these
these paths
and we found that there was a really
nice
uh interpretation of both of these this
could sort of be seen as being something
like bigram statistics and this term
here tells us if an attention head
attends to a particular token what is
the effect on the logins
and this tells us
this tells us
where the attention head attends so
uh that was pretty useful
and another way that we could write that
is we could say well uh we could put it
sort of in a factored form we could say
well okay the first thing you do is you
you multiply by w e the so first to
embed
then we apply the attention heads and
there's the identity path
as well
and then finally we do the unembedding
okay
so once you've done that it's pretty
easy to to generalize it to
a larger model
or to a model with more attention layers
we're just going to have multiple copies
of this term basically
now
you might have noticed that every time
we talk about wo it always comes with wv
and vice versa every time we talk about
wq it always comes with wk and because
these equations are going to get a
little bit larger and more complicated
for simplicity we're just going to
introduce these terms wqk and wov
that correspond to
uh those products so that's just a
little bit of a little bit of cleanup um
before we move on to the
equations
okay so um here we have
uh
our two layer attention only transformer
so we start by going and applying the
embedding so that's at the bottom here
then we're gonna go
and
talk about the first attention block so
that's
that's all of this
and within that we have the identity
term corresponds to this path and then
all these attention head terms that
correspond to these paths
and then we have the
the second layer attention heads
and then finally we have our good old
unembedding
now
uh we'd actually even though this is
sort of an easy way to
to go and describe it and we'd like to
expand this because expanding it will
give us
i think a more helpful helpful way to
think about the actual mechanics of the
system
um and we're going to take advantage of
this really nice property we sort of
implicitly talked about it earlier
that
when we have tensor products um and if
if you're not comfortable with tensor
products um make sure you watch the the
video on the theory of one layer um
attention heads where we talk about
these a little bit more but um when we
have tensor products like this
the
items on on one side of the tensor
product and we multiply them the items
on one side of the tensor product
multiply together
um and similarly the ones on on the left
hand side
end up going together as well this is
called the the mixed product identity
and it's really actually the whole
reason why i decided to frame
uh
this series in terms of of tensor
products
um and in the case of attention heads in
particular there's a really beautiful
interpretation of this which is that
if you chain really we're only going to
do this one when these are attention
heads and we're multiplying them
together so if you chain two attention
heads together the attention patterns
combine and create a new attention
pattern
and
the matrices that describe where the
attention head is reading from and
writing to also multiply together and
and you get something that looks very
much like an attention head so that's
the reason we care about that
okay so now we can write the expanded
form of that equation so we have um
first well we're going to have sort of
three types of terms so we have
the direct path term
and this is we've seen this term all the
way back to our zero layer transformer
um it's a good old friend and it just
corresponds to this direct path
down uh the transformer and it tends to
represent
bigram-ish statistics some of the
bi-gram statistics will start to migrate
into attention heads but um the kinds of
things that it represents are are
similar to bi-gram statistics
then we'll have
uh terms that correspond to going
through a single attention head and that
could go through an attention head in
the second layer or could also go
through an attention head in the first
layer
so these are the effects of attention
and then
finally
we have what i'm going to call
virtual attention ads so virtual tension
heads are when you have two attention
heads with a composition of two
attention heads
and that has also has some effect on the
output
and
and so virtual attention heads have this
nice property that they're
um they have yeah they're we just got
them through the
uh the mixed product identity that we
were talking about earlier so we we get
this attention pattern
they have an attention pattern of their
own which is the product of the two
attention patterns
and they have
um a
ov circuit of their own that describes
if they attend to a particular token
what the effect will be and it's just
the product of the
the
the first ov circuit and the the second
ob circuit
or the ov matrices at least
um
oh so that's what we that's what we had
okay so a question you might um well
okay stepping back for a second i think
one of the things that's really cool
about this is it really allows us to
study attention heads in a principled
way so i think a lot of the time um
there's been a lot of papers that i
think are there they're genuinely super
cool papers um where people go and study
uh attention patterns and they they're
like you know we found an intention head
that appears to attend from this to this
and maybe like it attends the subject of
the of the sentence or something like
this or if it's a verb it attends to the
subject or something like that
um
but it's
it's actually pretty tricky to know well
this sort of a conceptual problem which
is it's very tricky to think about
attention heads in isolation they could
be um the attention heads could be
reading information for other attention
heads um and they could be you know they
so they you know it might appear that an
attention head attends to one token and
moves it from
and and goes to a second token but it
could be that the the information that's
reading from that residual stream
actually came from an attention head
that was yet earlier and that the
attention had the information it moves
and writes that that attention that
information doesn't doesn't sort of that
isn't its final destination it yet moves
on further uh and so it's very easy to
be um i think potentially to to be
confused about this or at least to worry
that you might be being confused with
this um and it seems to me that uh at
least for the attention only case this
framework uh resolves that because uh if
if it was the case that
uh the important thing was was these
chains attention heads then those would
be the virtual attention heads and and
that would resolve all these you know
that and and the the the individual
attention head terms would sort of end
up being small and uh and that would
that would completely resolve it
so uh that made me really happy because
i've i think that i felt very
uncomfortable that when i've when i've
seen people talking about transform
interpretability for for a long time has
been this concern about uh chains of
attention heads and whether whether
attention patterns are really important
or whether they're just sort of
illusions um that are are parts of a
much longer chain and that we're missing
the whole story and it feels really good
to have have a framework that puts us on
on steady ground with respect to that
concern
okay
um so in our in our next video we'll be
able to go and actually start studying
these


2L Attention - Term Importance [rough early thoughts]

in our last video we found that we could
describe two-layer attention only
transformers
and with a kind of handy equation
and
the equation has three kinds of terms
the direct path terms
the individual attention head terms
and the virtual attention head terms
and something that might be useful
to know before we spend a lot of time
trying to go and really understand the
behavior of the model is how important
are those different terms and it turns
out we can actually just explicitly
measure that and so that's what we'll do
so
for the direct path term
which is just going straight up down the
middle uh
if you haven't if you're measuring
relative to just making random
predictions
you reduce the loss by 1.8 knots
when you add that term to the model
so that's a that's a pretty nice draw
especially just for a single single very
simple term
[Music]
um
the individual attention heads then if
we go and we
uh add those in on top of the direct
path term
so then we have the individual attention
heads and
they go and give us
a reduction of 5.2 knots now
since there's 24 terms um it's it's
actually quite a bit less than the
direct path term per head and it's only
0.2 knots
um
but
uh yet they're still still doing a lot
of work
and then the virtual attention heads
they on top if you already have the
individual attention heads they only
give us a marginal
additional 0.3 knots
um and because there's there's
quadratically many of these right
because for every for every first layer
attention head and every second layer
attention head there's a there's one of
these pairs and that are the virtual
tension heads
and so the
the amount of of loss reduction we get
per term is very small 0.002
and
so probably if we're trying to
prioritize what we want to understand
our top priority should be of course the
wwe which is going to be very easy to
understand just sort of representing
bygramish statistics and
gives us a lot of bang um
for for its yeah for the effort um
but
uh then we want to study the individual
tension head pyramids probably
okay now you might wonder how is it that
we calculated this and there's kind of a
nice algorithm you can use to go
um and calculate uh how much these
different terms
contribute to the loss and you can use
it for this model but you can also use
it for larger models it's a um it scales
to potentially very large bottles
and the trick is
first you just run the model
and you save all your attention patterns
because we want to hold the attention
patterns fixed
okay so you do that
then
and you run the model
and you're going to force all the
attention patterns to be
the attention pattern that you recorded
um
but
instead of adding the attention head
outputs to the residual stream you'll
just save them and and replace them with
zero so you'll go and record what the
attention what the engine heads would
have added to the residual stream but
then you'll add zero instead
and you record the loss so that's
just calculating the direct path
contribution
then
and you're going to go and iteratively
run the model
and every time you run it you're going
to force the attention patterns to be
the original attention patterns but
instead of going and adding
uh the true output of your attention
heads to the residual stream you'll add
whatever the you would have added at the
previous step that you you replaced
and then you'll go and you'll save
and save the output for
so you can go and add it uh
on the next step
and the the nice thing about this is
that it means that
and
all of the attention heads
only saw
at most n minus one attention heads
um or sort of were only affected their
their values were only affected by n
minus one attention heads um at least
three through the ov circuit not through
the the effects of the attention
patterns but if you hold the attention
patterns fixed then
then that's true and so
that's a way that you can go and
understand that
um of course you could you could come up
with all sorts of variants on the
algorithms if you wanted to understand
how important things were also going
through the effects on attention
patterns you you could also do that
okay so uh we think that probably
we can ignore the virtual attention
heads um or at least they they aren't
our top priority to understand
for this model
now another question you could ask is we
have really two you know for the
attention end terms we sort of have two
different things we have the first layer
of tension heads
and the second layer attenuates
and we can ask
in terms of what is their direct effect
on the loss
how much do they affect the loss
and
uh there's sort of two ways you could
measure this one is you could measure
relative to there not being any
attention hints
or you could measure how much do they
add marginally on top of
the uh
yeah on top of the
the other
uh
other heads already being present so if
you have the if you have the if you want
to measure the layer one heads you could
either measure relative to the case
where there's uh where there's just no
other attention ends or relative to the
case where you've already added the
layer of two attentions
and to make a long story short um the
layer two heads have a much larger
direct effect
than the layer one heads
and if you measure it in terms of the
relative the direct or relative to only
having the direct path it's 5.2 knots
versus 1.3 nets if you measure it
relative to the other one being present
it's four nats versus 0.05 knots and if
you then go and divide by the number of
heads those are obviously getting to be
very small numbers for the for the layer
one
now that doesn't mean that the layer one
attention heads aren't important
and they can affect the attention
patterns as a layer two attention heads
and we'll see that they're doing a
really critical job in that um but it
means that in terms of this this overall
equation we have um we don't need to go
and worry as much about the layer to
attention ants
so we can mostly understand the model by
understanding the behavior of 12
attention heads
um and those 12 attention heads their
their attention patterns are influenced
by earlier attention ends but um if we
understand those those 12 attention
heads um for instance you could just
empirically understand their their
attention behavior as a starting point
um and their ov circuits you'd
understand uh the model's behavior
um
but yeah it's really important to keep
in mind
that whenever you see
uh the attention pattern here
particularly not only is it non-linear
but for for later layers and that also
contains eventually lots of effects of
previous
uh previous attention
and in fact we can we can make that
explicit if we want so
when we looked at the first layer the
equation for our attention pattern was
this
and so
uh it's a little bit complicated but
it's um you know we this is the thing
that we called our qk
circuit matrix and we're multiplying
multiplying by tokens on both sides
but when we go and then move to
uh
the
the
the two layer model
all of a sudden it becomes a lot more
complicated
and what's going on
is that
rather than just having the tokens
get embedded and then affect things and
they first go through
all the attention heads in the previous
layer and then they hit the uk matrix
then they the the keys also were
computed using all of the attention
heads and
after they they went through the token
so on both sides we have um attention
heads and if you you can expand this all
out of course and if you want to really
analyze this then you'll have a bunch of
terms that involve um first layer
attention heads uh going and and
contributing to the to the attention
pattern
um you'll still have something like a qk
circuit uh it's but it's it'll be a bit
more complex
in any case now that we know what the
right terms to focus on are we're ready
to go and explore the model and
understand its behavior in our next
video


2L Attention - Results [rough early thoughts]

well we're now ready to actually
understand two-layer attention-only
transformers and i think there's
actually some really cool stuff inside
them so i'm excited to chat about this
uh in particular there's gonna be some
really interesting stuff that they're
doing uh that kind of is meta learning
relevant and seems to generalize to much
larger models and so
hopefully we'll learn some some
interesting and generalizable lessons
now
if you watched our our previous video on
eigenvalue analysis you'll remember that
um eigenvalues are kind of a nice way to
summarize the behavior of retention
heads without having to look at them uh
very closely
and
in particular positive eigenvalues for
an ov circuit for the output value
circuit mean that an attention head
wants to increase the probability
of
um that the output is the same it'll
increase the probability uh or increase
the logits for the that the output is
the same as the token that it's
attending to
so uh really it just means the attention
head is implementing copying
and
if you you know last time we saw that
really the behavior of our two layer
transformers is mostly described by
understanding
the
second layer attention heads
and there's 12 of those and of those 12
7 are doing copying so it's a little bit
less than when we were looking at the
one layer model but still
the majority of our attention heads are
doing doing coffee and we're going to
see they're doing a much more
interesting type of copying and a much
more powerful type of copying than
the one layer model was
yeah so
we could try to go and study this by
analyzing the qk circuits um the math
would get a little bit complicated so an
alternative is that we can try to just
empirically uh understand
the attention patterns and then
we can go from there if we want and try
to understand more about the math of the
qk circuits
and something that we see a lot of
is what we call
the induction pattern
and
so
for instance if we look here and it
might be a little bit hard to read so
let's let's focus
on
this d token so that's the present token
and
if we look back it attends really
strongly to ers
which is the next token
after d
and if then we go and when we look at
when the present token is ers
it tends to lead
and this is a very general thing so you
know similarly um
when
uh when the present token is pot um so
this is all from the first paragraph of
harry potter by the way um i think
probably maybe we've mentioned that in a
previous video or maybe we haven't but i
i've been having some fun using that as
a standard um piece of text to go and
play around with
um but you can also observe this on lots
of other lots of other text and it works
very reliably
um but yeah so if we if the present
token is pot um well we have the potters
so we we attend to the next token the
token that follows pot previously tours
okay and so we're starting to get a
pretty good hypothesis maybe what's
going on here which is that somehow
this attention head
always attends to the token
that follows previous copies of the
present token so if we if we're on pot
the previous copy was pot and then we
get terms
okay so how could it be
well that's kind of interesting now a
natural hypothesis is that the reason it
does that is that it would allow it to
go and predict then what the next token
is because the the goal of the present
token here is to predict the next token
and we can actually just measure that we
just take the output of the attention
head and it just has a linear path down
to the to the logits and so we can just
run that into the logit
and even without analyzing the terms we
can just see the empirical effect um on
the logits and it's indeed going and
predicting errors going and predicting
lee
uh going and predicting a little bit um
the pot on on potter so we have that pot
that pot um but then the deters and
plotters really strongly and so it's
increasing the probability of all of
those
and in each of those cases it was
attending
to that token
so it's copying it's copying the errors
to the errors and then we have this
hypothesis that maybe somehow the way
it's doing that is because the previous
copy of d
um or the present token is d and that
was the token that proceeded where we
attended
so we'll call that the induction pattern
and we can actually verify our
hypothesis that um that was that it's
actually
looking for the fact it's sort of um
that the fact that the previous token is
the same is what it's looking for
and and if we if we just look at how the
key for this um so let's let's let's
pick a particular case so where where
we're on
pot and it tends to terms well it turns
out that if you look at how the key is
computed the key isn't computed for the
present token but it's actually computed
from the previous token so we shift
the we grab the content of the
proceeding token
and inject that into some subspace and
then use it to construct our key
whereas we construct our query mostly
from the present token
and so by going and looking for then
looking looking to go and match the
present token with the subspace that we
stored listen we're able to go and look
for tokens that follow a token that
matches the present token
so that's a little bit more of a complex
algorithm than anything we've seen it's
almost a little bit like nearest
neighbors um
and that you're you're you're sort of
you're searching for for you're almost
treating your your text up to this point
as as being a little bit like a training
set and you're looking for places where
um
where there's a match and then you're
going and copying that or something
now it does other interesting things too
so and you know sometimes it's just the
present token that matters but there's
also cases where like here we have have
this d and if you look at this and i
don't have it here but if you uh if you
interactively play around with this
you'll see that it really is that the d
of this query matches the d of this key
and the mister of this query matches the
mister and to a lesser extent the misses
portion of the the key over there
and so um they match and then that
causes us to attend to errs which then
increases the probability of verb so it
really is this this copying thing and
we're we're calling that that general
mechanism of um looking for looking for
the token that in the past has preceded
your present token looking looking for
situations where that were analogous to
your present situation and then
predicting that whatever happened last
whatever whatever happened next in the
past is what's going to happen next now
that's what we call
um induction and or an induction head
and that seems to really be the kind of
the workhorse of metal meta-learning and
a lot of transformers we see it in large
models and it seems to seems to be
playing a big role
um and it's not just one of them
one head like this we actually see a lot
of heads that are doing slightly
different flavors of this
um some are are sort of very strict
versions that are really looking for
exact matches um
and
uh exact uh yeah exact matches and
copying
whereas authors are a little bit more
flexible like here we have um
dursley probably mr dursley was he
and then we copy was
um
or
uh huge
and then didn't um but probably if
you're using apostrophes that's and
contractions you're likely to do that
again in the future
uh so yeah we see a lot of this this
kind of induction
now
uh
yeah is there some way that we we have
this really handy trick for detecting
heads that do copying um by just looking
at the the ov eigenvalues and we
mentioned previously when we talked
about their video on eigenvalue analysis
that um having positive qk eigenvalues
would mean that we were attending to
uh that we were trying to attend tokens
that are similar
and so we can apply that same trick and
and
uh here are our qk circuits a little bit
more complex it's rather than looking
for a positive eigenvalue meaning that
um
that we're attending literally to the
token that's the same as our present
token it means that we're attending to a
token um where wherever the key is being
constructed from is the same
as our present uh token uh we're really
the same as wherever our query is being
constructed from um but that's still
author if the if the keys if the qk
eigenvalues are positive that's a very
strong sign that we're doing something
like induction and
um
the ov eigenvalues mean that we're doing
copying and so if we see if both are
positive that's a sort of very strong
signal that we're seeing induction
and we can see that there's six heads
that are quite clearly doing this
and then another one uh
that is maybe doing this a little bit
but it's a little bit a bit messier
now that leaves a number of other heads
that aren't going and doing copying and
aren't aren't induction heads and
i don't one of the reasons i'm focusing
on on induction heads so much is because
they seem to be genuinely really
generalizable to larger models or the
transformers generally and seem to be
sort of a very important story about how
transformers work i don't know that i
have any really interesting
generalizable takeaway about these other
heads and not based on the study that
i've done on this model so far a lot of
them seem to be doing sort of like
slightly engramed stuff
um
and
yeah so i'm not going to focus as much
on them
oh one final thing that's worth
mentioning is um just like we were able
to go and do a histogram in the video
that we did on eigenvalues to go and
sort of quickly quickly summarize the
heads in a really dense way and we can
put the
uh
the the
the sum of the qk eigenvalues divided by
the sum of their absolute values on one
axis and the same thing for the ov
values on another axis
and then attention heads that have both
um very very positive on both of those
so tends to to really copy and tend to
prefer to look at places where a token
is the same they'll be up in that corner
and other tokens won't be in so that's a
an even denser way to go and and really
systematically uh pin down the really in
a really automated way pin down the
induction heads
uh okay so a couple takeaways um
these induction heads seem to be uh
really a major mechanism for metal
learning
um and they seem to be a lot more
effective than simple copying heads
and
we can automatically detect them which
is is pretty cool
uh yeah and yeah i mentioned this
earlier but the the other attention
heads um they tend to be doing kind of
local stuff and it's not clear that we
have a really generalizable story so
we're not going to pay too much
attention to them
well in any case i think it's pretty
exciting that we can uh on really
mechanistically understand uh some of
how these these two layer models work
and with enough effort if we wanted to
we could probably probably really
understand them
um
and
a lot of the ideas that we built up here
um the the general sort of form of this
equation of thinking about um thinking
about intentionally models and this idea
of virtual tension heads um are actually
i think kind of generalizable um and
will be useful as we start thinking
about about larger and more complex
models


Virtual Attention heads [rough early thoughts]

over the last couple of videos we've
been uh trying to understand two layer
attention only transformers and as we've
done that this really curious type of
thing um that we've been calling a
virtual attention ed sort of just falls
out of the map
and even though it wasn't very important
in understanding our previous um the
models that we've looked at so far i do
think it's a really interesting idea and
potentially
a really powerful one that i sort of
expect at some point to be important
even though i haven't really empirically
observed that yet and so uh yeah i just
wanted to briefly chat about them
so
uh and yeah the when we just tried to
describe our second layer um our two
layer attention only transformer we sort
of ended up with these three kinds of
terms so there was the direct path term
and
there was the individual tension head
terms
and then there's this this other one
that corresponds to
going through
chains of attention heads
and so you go through
one attention head
and then another attention head and then
you hit the unembedding
and
we called those virtual attention heads
and the thing that's really interesting
about virtual attention heads well so
what has an attention head fundamentally
an attention head um i think i think the
fundamental way to think about an
attention head is that it moves
information
from
one the residual stream of one token
to the residual stream and another token
and in order to do that um it needs two
things it needs to know uh where which
tokens it's attending to
and from and then which substitute it's
going to read in and where it's going to
write it in the resulting
uh in the resulting tokens residual
string
and this virtual tension heads even
though they don't they aren't
represented you know explicitly in the
model like an attention head is um they
have exactly the same properties an
intention head does they have an
attention pattern which is just the
composition
of the two attention patterns so um you
know if if token a if if attention head
if the second attention head looks at
token a or get token three and then um
the
the first attention head um looks from
three to token two then on net you look
from um from token you look at token two
and so there's there's this composition
of attention patterns that's potentially
very expressive
and then they have an ov circuit um just
like our regular attention heads and
that ov circuit
uh
describes how if a virtual attention
head attends to a given token how it
will affect the outputs and this this
sub part here this this product in the
middle um that tells us
on you know what what residual stream
does it read in and then write out or
what what what subspace does it read in
from the residual stream right at the
beginning and then works it right to
at the end
so
i've sort of been rambling and you might
maybe you're kind of surprised because
um you know i'm i'm just sort of
observing this mathematical term and i
don't really have evidence that it's
important and i'm just sort of excited
about it and ranting about why i think
it's important so let me try to defend a
little bit why
why i think this this structure
is potentially so important and so
powerful
and i think the key reason for me
really above anything else
is that it's composition
um
and it's the composition of something
that seems to be a very powerful and exp
very powerful primitive and so when you
compose them together it seems very
expressive
um okay so let me give you an example of
uh
uh of of of just a simple example of
what i mean by composition first so um
oftentimes uh we'll see attention heads
attend to the previous token
and we'll often see multiple attention
heads that attend to
the the previous token
um
but we very seldom see attention heads
that attend to
the
second
previous token
um very rarely we'll we'll see attention
heads that go and attend to
um sometimes they attend to the token
two beforehand but very rarely attention
ads that attend sort of in a fixed way
to the token two steps before them
and so you might ask well why is that
and i think the answer is um
that well i think i think one reason at
least is that the model doesn't need to
do that in order to express um
uh in order to go and express the
ability or have the ability to go and
get information or the token
two steps before because it can use um
virtual attention heads
and you can just chain them together so
you have one virtual attention head and
then you have the next one
and you attend
two tokens back
and so you can go and read information
from two tokens back or three tokens
back or four tokens back by cheating to
have this primitive and then you don't
have to create a dedicated head and you
can and well we'll talk about the other
benefits in a second but um they're
potentially expressible
but we have much richer primitives that
potentially can be used so we have um
also the attention ends that attend to
the start of a clause or the the token
that is a transition from an outer
clause to an inner clause or from a
subordinate clause
and we'll also see sometimes attention
ends that attend to the previous
sentence and so that also seems kind of
like an interesting primitive you start
combining it with things like attention
heads that attend to the verb of the
present clause or the subject of the
present clause or the opposite of the
present clause then you can do things
like okay look at the previous sentence
and then look at
its subject
um or its verb
yeah or look at the look at the outer
clause and look at what the subject was
and that seems like it must be a useful
thing to be able to do
and you know those are those are just
combining together a few things that if
you have more and more of these these
these attention heads that you can
compose together um you can express all
kinds of things potentially and that
seems that seems potentially really
powerful
in fact um as you have more and more you
have n per layer so the number of things
you can express grows exponentially
um
now
of course most of the things that you
can most of these compositions are
probably not very useful
um but
uh in theory you have you have this
ability to have an exponential number of
virtual attention ads now you might say
um you know christopher there's only uh
uh n uh you know previous tokens um and
so why would it be useful to have more
than n um you know let alone massively
more than n attention ends well if
you're looking at and your attention
heads attend to different places for
every sentence and so
um you would sort of think of it as
being like these semantic positions uh
that are or these functional positions
um which happen to be in one position in
one sentence and in other positions in
other sentences but um for instance
uh you know the
the
the the the subject
of
the
um the first
the first noun in the sentence
uh
and the subject
of
the previous clause
may sometimes be the same token
but they may sometimes be different
tokens and so um you know those are
different things and and potentially
yeah you you want to be able to express
both of those and so
and
i think that i think it's potentially
powerful that we have exponentially many
of these virtual tensions
okay the other final thing is they can
in some sense be smaller than an
attention head so attention heads move
information right they they read in
information from the residual stream of
one model of one token and then write it
to the residual stream another token and
depending on the size of the value
vector they have to copy you know 128
dimensions or or at least you're sort of
dedicating the capacity to copy 128
dimensions
um or whatever whatever the size is that
you dedicate to your attention it's
well okay um
but what if you only want to copy a
single dimension or two dimensions then
you you potentially have to waste an
entire attention head to go and do that
well with vertical attentions you don't
have to do that a virtual attention head
can move a single dimension without
wasting anything it just goes and um
grabs one dimension from one of the
attention heads that's going to compose
it and one dimension for one of these
from the other attention that's going to
compose it and uh that single dimension
um gets used and the other attention
heads still have all of you know the the
128 minus one dimensions left to go
and uh use for other things and so uh
it sort of is costless to have tiny
virtual attention ads um
or the cost is is no larger than than it
needs to be to go and have these tiny
little virtual attention nets and that's
potentially quite powerful
okay so that's my rant on virtual
attention heads and why they seem
interesting to me
and
and probably our next video will be a
little bit less theoretical