Attention - General - Theory, Info-Weighted Patterns, Attribution Patterns [rough early thoughts]

in our last couple of videos
we've studied and tried to reverse
engineer some really simplified versions
of a transformer
and the methods we used um really relied
on some of those simplifications
especially the fact that we only had a
tension model i only had a tension there
so we were studying these attention only
transformers
this video is going to try to go and
generalize some of those techniques to
fully
general standard transformers
and we're going to talk about some of
the things that make that a much harder
problem
and how we can try to work around those
challenges
and develop some tools that will allow
us to think about about general
transformers or at least small portions
of them
um yeah so it's going to be going to be
a bit of a trickier problem
but we will be able to make some
progress and i think we'll
learn some cool things along the way
so the thing that we're going to be
focusing on uh for now and this is this
is continuing in the theme of our
previous videos is understanding the
role of attention heads and the circuits
that attention heads form
and this is actually something that's
been going on uh in
uh in studying transformers for a while
and i think it really starts um probably
the earliest example is there was some
work by um leon jones
um in 2017 and
it's kind of impressive uh sort of how
early and he just made a jupiter
notebook exploring attention patterns
and discovered some really cool things
um and since then we've had um a number
of papers jessievig
did a lot of
a lot of early work poking around at
attention heads and then since then
there's there's been really a lot of
uh a lot of work going uh and exploring
attention heads and
trying to connect it to linguistic
phenomena that's been a really
interesting thing so there's um some
work by by kevin clark and collaborators
and some other work by elena voida um
and collaborators um that's been going
and studying that so there's been a lot
of really neat uh really neat work in
the space
um and it's very tantalizing it seems
like there's there's very cool things
going on
i do think that there's
uh
however a couple of of limitations
to just looking at attention patterns
and really really i think it can be
summarized in three major points so just
just very briefly we'll dive and talk
about each one of these um if you're
just looking at the attention pattern
first of all you don't know that's the
ultimate source or destination of the
information it could be that the
information was moved by another
attention head and then moved by this
attention head and then moved by another
one yet again and doesn't really have
anything to do with the tokens that
superficially seem to be involved
um
another another thing that
uh
uh is i think challenging and important
to think about is it seems like a lot of
attention heads have have what we'll
call resting positions and there's no
real principled way to handle that if
you're just looking at the attention
pattern
and we have to make assumptions that uh
you know they might be right or they
might not be right well that one might
will impact that a bit more in a second
the final thing is it doesn't tell us
about the role the attention head is
playing in the the transformer doesn't
tell us about about functionally how it
affects the transformers behavior and
allows it to go and perform its task
we're just seeing the pattern and sort
of observing that there's this
interesting pattern
and so we'd really like to be able we're
going to try to talk about the role of
attention heads in a transformer and and
use them as sort of a a small reverse
engineered chunk of the transformer that
we're we're trying to understand in this
kind of circuity way and build towards a
large sort of full understanding of the
model we want to want to be able to
address these concerns
okay so just to unpack that a little bit
more the first concern is that the
information
if you just look at a single attention
pattern and you know you might see that
attention you know one token it tends
back to
the previous token for instance or the
the same token or you know maybe a token
several steps back
but there's nothing that says
that's the the ultimate source and
remember that attention heads the whole
thing attention heads do is they they
copy some subspace of residual stream
from one token to another and if you do
that repeatedly it's possible that
you're copying information
which was previously copied
or conversely it could be that you're
copying information you know your
attention copies information somewhere
and then another attention head moves it
yet further
and so
uh these
these attention patterns they might be
representative of how information is
sort of fundamentally moved in the model
but it could also very easily be the
case
that they're
um you know they're just one step in a
chain of movement and that the
particular tokens involved are are kind
of superficial
and i guess another way to think about
that is you know
really
the the attention pattern
uh is you can think of it as as sort of
a
if you if you were writing at the
jacobian for the model
um the attention pattern would be one
matrix and this giant product and sum
that would produce the jacobian and
because there's it's being multiplied by
lots of other things it could be that
it's just being multiplied by the
identity matrix but it could be um you
know not at all it could be that the
other matrices you multiply it by really
greatly changing
that's not the way you can think about
it in any case that's that's one
challenge with just sort of interpreting
attention patterns as they are and it
may very well be that the superficial
appearance is correct but it could be
that it's not
another challenge is if you start really
digging into attention heads you'll
notice that a lot of attention heads um
will mostly attend to
well the most common thing is you'll see
a lot of attention ends that almost
always attend to the first token so
you'll see lots of tokens attend back to
the first token in your context and that
seems weird
and then you'll notice that a very small
fraction of the time they do something
else so um a very common instance of
this is if you have an induction head
that's looking for cases where you're
where there's repetition going on most
of the time tokens aren't being repeated
and so it doesn't have anywhere to
attend and it needs some default place
to go and so it it will happen to be
often and we'll have have to go and set
some default position and often that
will be the first token
or sometimes it will also be the periods
so if you look at an attention pattern
like this
there's some sense in which really the
meaningful thing is when it doesn't
attend
to the default position and we really
want to ignore all of these default
position attentions they're they're not
that interesting
but
how do we how we know that's a
principled thing to do how do we know
when something is just a resting
position that we can sort of ignore or
default position that we can ignore
that's not doing anything interesting um
and when when is it actually
uh
doing something important especially
when
when it's it's maybe not that the
resting position is the the first token
or maybe maybe the token isn't you know
99 resting or the attention head isn't
99 attending to its resting position and
uh only one percent elsewhere if it's
you know 50 percent attending to some
resting position it might be might be
less obvious what's going on and by the
way this phenomenon has been described
in the literature um before i think the
the earliest is jessie viggs um work
where uh it's described as null
attention um
but yeah we'd like to have a more
principled way to handle this
and then the final thing and this is
really you know our goal is to reverse
engineer the model so we you know it's
cool to find interesting attention
patterns but the thing that we
ultimately want is we always say how do
they empower the model how do they they
contribute to the model's behavior and
allow it to do things um and yeah how is
the model working
and so the attention patterns are
interesting but we'd like to link them
to
behavior and function
um and understand how they contribute to
the model
okay
so it turns out that limitation two has
a relatively simple
um an elegant solution
whereas limitations
one and three
um that unfortunately is going to be a
little trickier and and they they have
really simple elegant solutions when
you're only dealing with the attention
only case but once we switch instead to
this general case it's going to be a
little bit trickier and we're going to
think about it
quite carefully and there's going to be
some slightly painful trade-offs
okay
um
so the the idea
that we're going to use to resolve this
uh
the second limitation that the fact that
there's these default tokens
and is we're going to go and create what
we call the info weighted attention
patterns we have our regular attention
pattern and we're going to create an
info weighted attention pattern
and looking at it here actually i think
it's going to be hard to understand
what's going on so let's switch to an
interactive visualization
so what we're seeing here
is
we have the first paragraph of harry
potter and we have one attention head
that i selected to be an induction
intention attention head
um
that we can we can see how it attends
over over this paragraph and we'll see
that almost always it attends to the
first token and that first eot token
and then very rarely when we have
repeated texts and the first paragraph
of harry potter and
the the name dursley is repeated a lot
and it's split into three tokens so we
can see when we when we attend to or
when we're on d
the induction head says ah there was a d
um and last time i saw a d it was
followed by ers and so i'm going to
attend to the ers and that will allow me
to predict then the next token is in ers
so that's what the induction head does
um
and
we can see yeah we can we can go and
just track that so as we go through
dursley we go and we we do that and as
we go to later once we'll see even you
know and there's there's there's
multiple copies of jersey so it can
attend back possibly to multiple ones
and the the resting position initially
it's just that first token but later on
it'll start to rest on various periods
um so it you might think it's attending
to the periods but it really is is
resting on those and there's some other
places where it finds copies and the
potters is also a thing where it can
protect the turs and potters from pot
um
and
there's also
small sun here um you know we we have
small and previously we said small
sun and so we can attend the sun and
predict that it's going to be sun again
okay so that's
that's what we call
um
what we mean by having a resting
position and it's very natural that this
this particular one has the resting
position and we can actually look at
this
if you look here this is a another
visualization
of the same attention pattern
and
what you're seeing is the
uh
for every token
where does it attend so you fix if you
fix the token you do a horizontal line
and you can see where it attends and you
can see that for the early tokens they
almost all attend
to the first token so that's that red
stripe down the middle and then these
these little off diagonals
are places where we have a little
sequence of repeated text and the
induction head can start to activate
so we see that there's also then these
places where periods start to become
resting positions and those are these
other vertical lines
so we can we can see it very clearly
there the
the stacked and um i think probably on a
youtube video um it might be a little
hard to see this but we can we can link
these together so you can see you know
the fact we can when we intend to to
dursley here that's um that's one of
those
uh
yeah we're on that we're on that
diagonal
over here
okay so so that's our
an example of
of a attention head that has
a
as a resting position okay so now the
thing we'd like is we'd like some
principled way like we could make the
judgment call but when it looks at eot
that's not real that's a resting
position and when it looked at periods
that's that's not meaningful that's a
resting position and we only want to
look at the copied cases but that would
be our judgment call and that wouldn't
seem very principled so instead we're
going to go and create what we call an
information weighted attention pattern
so we're going to take the same
attention head now
and we're going to look at the
information weighted attention pattern
and i'll explain exactly what that is in
a second but the intuition is that we're
going to go and show the attention
scaled by how much information was
copied from one token to the other so
partly that's where we attend but it's
also how large the value vector is um
and
we actually do how much
how large is the vector that's going to
get written to the residual stream at
the resulting place so that's the wo
times the value
vector
um
and here we just color by default when
we when we aren't interacting with this
we just color the tokens by how much um
by the intensity of the pattern in this
case the amount of information that was
moved to that token and we can see that
whereas before we everything had was
getting attention now we're seeing that
sort of almost everywhere is zero
so when we when we look at these typical
tokens you might see that the
the
sorry um
you might see that typical tokens
do have
a slight pink to eot
but for the most part
um it's very small and if we get to copy
text all of a sudden we have some very
strong copying going on
so that's kind of interesting
and we have these
yeah and we've just isolated the cases
and we can we can at a glance see where
there's where there's interesting things
going on
and so yeah like small we look at sun
and things like this
okay
um
so the the formal definition of that is
going to be that we
and so here we have the the regular
attention pattern
which wasn't very informative
and then we have the infor weighted
pattern which really is going to solve
our problem
and give us a principled resolution to
this question
of
um of how to how to deal with resting
positions and the trick is just that you
scale uh every source position
by the magnitude of w o v and it's
really not that different from going and
scaling by the magnitude of v um and
there there's various other variations
you could do of this um
i've experimented with subtracting off
the mean value um because sometimes the
the value vector for resting positions
will be some small
small vector that's just kind of like
you could think of it sort of as being
like a bias and then so subtracting off
the mean can go and make this a little
bit cleaner
and and one interpretation of this is
you could try to go and model the
information carried through
the attention head as a gaussian in
which case that subtracting off the mean
and looking at the norm of that vector
um
is is the surprisal and the number of
bits and informations you could think of
this as being kind of like the number of
bits of information copied from one
position to another so that's a that's a
nice principled resolution and it gives
us a way to think about these resting
positions that doesn't require our
judgment um but just
uh yeah just as is true it's just the
case that even though the attention
pattern is attending there um it's not
copying any information the attention
head is isn't doing anything when it
attends there it's just um it needed
some default position so that's that's a
nice resolution to that
okay but that still leaves us with two
limitations um that are unfortunately
not going to have quite as clean a
resolution so the first one is
the attention patterns might not be the
ultimate source or destination
information it could be that it was
copied by other attention notes um or
it's going to be moved around later to a
different token by a different attention
and it also doesn't tell us about the
function and so when we were dealing
with attention only models that was
actually really easy to solve because
what we did was we we decomposed our
attention heads into these sort of
attention head strict attention head
terms and virtual tension heads
um and then the virtual attention has
captured all this possibility that
information could be moved um by chains
of attention heads because that's that's
the definition only what they were and
the pure attention heads we could just
study in isolation and not worry that
information was going to be moved
because the yeah they they
definitionally didn't include that
and then we could look at their their ov
circuit remember this is the output
value circuit matrix that tells us how
much if it attends to one token it's
going to want to go
um
how it's going to affect the output
and that allowed us to go
and solve this question it told us all
about the function solved our our
limitation of not knowing what the
function of things were so in our
previous models this was these were both
really easy to solve
um but it's unfortunately going to be a
little bit trickier with the
introduction of mlps so
having nlp layers remember we when we
have a general transformer we go back
and forth between we have attention
layers with attention heads
and those are easy to work with but then
we're also going to have these mlp
layers and those are going to be harder
to reason about
so
um
let's let's talk through the challenge
okay so suppose that we have
hm let's start with a one of one of
these attention nuts let's start with
this attention head here okay
so
uh there is
a term first of all we should we should
be clear there is a term that uh in the
if you write the entire transformer in
this big equation like we were earlier
there is a nice term that corresponds to
uh the direct effect of the attention
head in attending to some token and then
affecting the unembedding that is a
thing that exists
but
i
and there there also are you know the
regular virtual attention head terms
but the problem is we can't decompose
the entire model
into that formulation so
when we have our attention head
it does indeed get information directly
from the token
and through these attention heads and
those can all just by linearity be
decomposed apart
but it also
gets information from the mlp
and that could come from the present
token
but it could also come
from the uh from the previous attention
heads
and yeah and that's in addition to the
the amount that came from the present
token and we can't decompose those
because the the the mlp is non-linear we
can't go and break apart these two paths
they're mixed together
and
and so we can't go and decompose
decompose everything as nicely as we
were previously so that's that's really
the tricky thing uh about this
now
um there are some solutions so well
there are at least some some options one
thing is we could just study um we could
just study the pure attention head terms
and those are interesting you can you
can just write them out the exact way we
were in the previous models and those
are possible to just completely sort of
analytically study and they're beautiful
and simple to work with and you do see
interesting things
um but it seems like a lot of
interesting stuff
also can't be seen that way
and one reason
one very basic reason even even
relatively simple things
is that you begin to see
um the mlp layers the early mlp layers
start to go and take on a role of sort
of reprocessing the tokens because
remember tokens often have multiple
interpretations and
tokens can be words can be split into
multiple tokens and then their meaning
is very unclear until you recombine them
and so the model really wants to improve
the representation of tokens using the
mlp layers and combining it with
information it got from the attention
heads
which means that your later attendants
are going to want to rely on that
which means that you're going to miss a
lot if you aren't looking at the effect
of the mlps so that's the that's the
downside of the approach of only
studying pure attention terms
now another thing you could do is you
just say well
we're not going to try and expand
through the mlp at all one of the great
things about the mlp layer um is if you
looked at our other video or maybe you
will in the future on on privilege
versus unprivileged bases so a privilege
basis is one where where the basis
dimensions are sort of special and
probably easier to understand mlps do
have a privileged basis um so if you if
you have to have somewhere that you're
going to try to go and analyze you know
the mlp layers are a pretty nice place
to try and analyze things
um and we can just try to understand
what different mlp neurons represent and
treat them like a sort of a basic source
in themselves and then you know we could
try to understand how we're computed uh
analytically but we could we could we
could sort of stop there and that would
be a very circuity approach like in
circuits um
uh you know circuits is this work that
me and some collaborators did previously
on
um reverse engineering quantum nuts
and you know really there what we were
doing is understand a bunch of neurons
and you can once you have sort of the
input and output neurons if you
understand what they are you can then go
and try to understand what computation
happens between them to go and construct
the output neurons from the input
neurons so that is that is something you
can do another possibility
is you could try in various ways to
linearize the mlps so you could like
replace them with a jacobian
or you could try to do this trick like
the attention heads aren't actually
linear they're they're only linear once
you fix the attention pattern um so you
sort of split them you think of them as
having two different inputs and they're
linear with respect to one input
um well you could do the same thing in
principle for the mlps like if
you see these usually usually these are
values but are jelly used but let's
imagine it's a value for a second and
you know a rally you just have to
go and think of there as being like a
binary mask that comes as a second
source and then you've split it into a
linear and a nonlinear a function which
has a linear input and a nonlinear input
um
but the challenge is that there's
there's sort of many more bits of
information to specify the non-linear
component here so it sort of becomes a
it's not as nice a frame as of analysis
in any case if you if you do take this
approach of linearizing say replacing
with
jacobian you're really getting into this
um
this
family of methods that people call
attribution or saliency and you're
applying that related to your attention
head into the value vectors um
and that's absolutely a thing you can do
and um
you know often if we want to really dive
into an attention it will start to do
things kind of like that um or at least
trying to really analytically think
through how the trace back how the value
vectors were formed but um
we're not i think that's probably not
the thing we want to do is our first
pass approach
so the thing that we're actually going
to do
in this video or in the next couple of
videos is we're going to take a very
pragmatic option
which is
uh
we're going to go and look when we're
trying to understand attention
and
we will just
take all possible
inputs that are contributing to the
values so we're not going to try to
decompose things we're going to take the
mlps and the attention heads and the
embedding and they all construct the
value vector for an attention head
um
and we have to be very careful in
interpretation to remember that the
value vector probably significantly
comes from the present token but it
might come from elsewhere so we we need
to be careful interpreting it but then
we're going to go and look at the direct
effect of that attention head on the
output so we're going to ignore
mlps and that's that's obviously going
to miss some part of the picture but it
it does seem like this is kind of a
a nice
a nice sweet spot where we'll be able to
really understand the downstream effects
um at least some of the downstream
effects
and
uh we won't we'll be able to see some
things that we wouldn't be able to
otherwise because tokens will get the
the nice representation from the mlps
um
that are are going and updating their
representations
and
uh that'll be that'll be a nice way to
go and think about things
okay so
the key idea now is that using this
we're gonna be able to make an
attribution pattern which says
if we you know how much does attending
to
this token and
increase the probability
of another token so let's let's switch
back to our interactive view
okay so recall that so far we've had two
things we started with the raw attention
pattern
and then we constructed this intuitive
attention pattern
now we're going to do something a little
bit different
and
where
we're going to go and construct what
we're going to call the attribution
pattern and we're going to color that in
blue so we have our infra weighted
pattern in red and we're gonna have the
the attention pattern in blue
and okay so let's let's go and consider
an example with the attention pattern so
the attention pattern
uh when we're on d
looks for the previous d and notices
there's an errors
and we think the reason it's doing that
is so it can predict the next errors so
the attribution pattern is going to say
yes it actually did increase the
probability
and
increase the probability of the next ers
so now the the errors we can see is in
blue here um
because it's actually
uh increasing and lee as well because
it's been increased in probability it's
a very intense blue because it's it's so
strongly increased in probability and if
we
look at where that's coming from it's
coming from the previous ers
so
uh that's that's what's going on there
so the attribution pattern then tells us
which tokens were increased in
probability by this attention head and
then potentially um
which tokens uh yeah how it how it did
that where did it tend to go and do that
and because this is
very simple copying it's it's yeah we
can we can sort of fairly easily
understand it
now again because we're going and just
using the value vectors that could have
been computed um in lots of complicated
ways in principle it could be that even
though it appears to be attending back
to errors to go and increase the
probability of things
actually
um you know maybe this this was computed
by some other previous tokens but um
because this is copying and it just
seems like this there's such a clear
inference that you know the
you know it's the exact same token
that's being increased in probability
we're pretty sure it's mostly just
looking at that that token
um
one other thing that's maybe worth being
really clear about is
when we work it look at the attention
pattern so if you look at the attention
pattern in the attribution pattern
you'll see that you know when we switch
the attribution pattern it shifts
forward so the attention pattern is one
token instead of at one token and then
the attention pattern is one token
forwards
and that's because the attention pattern
is to the output token not to the token
that's being attended to so this is the
the tension pattern goes from the source
token to the destination token the
attribution pattern goes from the source
token to the output token the predicted
token which is one forwards so
that's the that's the attribution
pattern
and to calculate that again remember you
just go and uh multiple you take your
value vectors um or your attention
results and you multiply by your w out
and you add them into the residual
stream and then you you hit them with
your unembedding matrix and you see how
the logits are affected
uh oh one other thing that's worth
keeping in mind
is
for the attribution pattern we're
looking at how the logits are affected
not how
the probabilities are affected so
remember that an attention head could
potentially affect multiple logics and
it could increase the probability of say
two competing things
and then even or increase the logits for
two two competing possible answers
and
and if it if there's a competitor that
maybe it increased more that otherwise
had already had more probability um it
could be that the
the probability still of a given token
decreases even though your your
attribution pattern tells you that the
the logic increased and that's just
because a logic can increase and the
probability can still go down if other
watches changed so that's another
another topic
okay so the attribution pattern is going
to be super useful to us
um i think we'll talk about that a
little bit more in a sec so yeah let's
for a minute let's just compare these a
bit more so our remember our raw
attention pattern goes from source to
destination the inflated pattern also
goes from source to destination but the
logic attribution pattern goes from
source to output
and all three of these tell us different
information so the raw attention pattern
tells us which token attends to other
tokens
the informative pattern tells us how
much information is moved from each
token to each other token
and the attribution pattern tells us how
how each token affects the logit or
directly affects the logic of each other
token
okay so what are the uses
of these attribution patterns well
um
there's there's really two reasons
they're really useful uh one is they're
very helpful for empirically
investigating attention heads and really
investigating the sort of um direct
circuits where the attention head
directly affects the logics there's lots
of cases where it's having indirect
effects as well and it's not going to
help us with that but for understanding
direct effects it's a very convenient
abstraction for thinking about that
um it's sort of taking on the role in
our previously we studied this by
looking and analyzing the ov circuit
matrix which was this big matrix
here we're going to do things a bit more
empirically but the attribution pattern
is going to be our main tool for really
empirically and studying that
the other thing that's really nice um is
we can use it to search for attention
heads
so
uh
something we can do is we can sample a
bunch of data set examples from our from
our data set
and define some some pattern that where
we think that an attention head is going
to go
uh and
uh affect
affect the output in some particular way
so you know for a copying head we think
that's going to tend to a token and it's
going to increase the probability of
that same token and most of the time
when it increases the probability of a
token it's going to be because it's
attending to a token that's the same
well we can use that to search for
copying heads
or
yeah we'll see a lot of other
interesting attentions that we can we
can search for i think i'll leave that
for the next video but this ability to
search for attention heads in large
models when you might have you know a
thousand or more attention heads and you
you have some theory that there's a
bunch of attention heads that are doing
something um that's a really nice tool
and that's gonna be really useful to us
okay so we are now ready to dive into
studying uh
you know attention head circuits in in
large models so
in the next couple of videos we're going
to be doing that and we'll find some
pretty cool stuff


Attention - General - Summarizing with NMF [rough early thoughts]

the goal of these videos isn't just to
understand toy transformers we want to
be able to understand transformers as
they're used in practice the kind of
large models that um are getting all
these really exciting results lately
and in our previous video we sort of
addressed one of those one of the
challenges to that which was um you know
talking about how switching from
attention only transformers to
a real transformer introduces various
theoretical challenges and how we can
work around them
but there's also a really practical
challenge which is these models that can
be huge
and even just when we're if we're just
talking about the attention heads um you
can easily have thousands of attention
heads in large models and you might want
to be studying multiple models as well
um and so this video which you you can
skip over if you want um
is a better trek for sort of summarizing
and getting a big picture overview of
the kinds of things that exist in a
model um at least the kinds of tension
patterns that exist in a model
and attention circuits
and yes we're going to try and get an
overview and the trick we're going to
use
is something called non-negative matrix
factorization or nmf
um and
uh
the i'm i'm chris i'm i'm i'm the one
who is recording uh this this particular
video and i have to confess
um
i'm kind of obsessed with nmf
people who are familiar with my previous
work or know me personally may know that
really one of my my first reactions to
almost any problem
is to try and go and apply
uh nmf
um and at least in computer vision it's
been really effective so um one example
that i'm i'm i'm quite proud of
um is in one of our papers the building
blocks of interpretability we used nmf
um
to decompose the activations of a vision
model
and so for instance here we have
uh
a picture of a dog and a cat
and
there are about 500 neurons in every
position the continents at every
position there's a there's like 500
neurons and that's a lot of neurons to
look at but we can just go and use nmf
to go and reduce this into a small
number of factors that describe the
activations at every position so we see
here
there's like one factor that corresponds
to the dog's ear and the head and the
snout and the bodies of both the cat and
the dog and the background and the cat's
head
and we can kind of summarize things
um and so that that was really effective
and and pretty consistently nmf
is just a really powerful tool if you
want to go and summarize
summarize things
so
i
yeah one of the first things that i did
when i started working with transformers
was to try and apply the same approach
and
um that seems to work quite well
so just stepping back a second uh
non-negative matrix factorization uh
is it's just a kind of linear
dimensionality reduction
um and really all we're saying is we
have a matrix
um
and our matrix
um you know has some some width and some
height
and we try to describe it as a product
of two smaller matrices so one of them
will be tall and thin
and one of them will be narrow and wide
and they multiply together to go
uh and produce the
the big matrix
um and then
it's non-negative because we're going to
go and require that both of these
uh matrices that we write as the product
be positive so if we if we tried to go
and do this without any constraint and
you get principal component analysis
these will would end up maybe not basis
aligned but you'd end up extracting if
you have this is say k dimensions on the
on the the small side that we've created
um you'd get the first k principal
components
that when you go and you add the
positivity constraint you get something
else um
and there's just kind of this way in
which nmf
tends to be interpretable and this isn't
just a property in machine learning and
you actually see this in the sciences
where often um in
say in physics people will prefer to use
nmf sometimes i think one really
striking example of this is if you take
a data set
of uh and i'm not a physicist so uh i
might mistake this but if you if you
take this a data set of the spectral
emissions of different stars
and you apply nmf
and you'll go and have all the different
star types go and and just fall out as
the factors of nmf um whereas that won't
happen if you if you do pca and you
you'll see this in other cases that um
you know in physics people often um
there will be cases where people prefer
to use nmf and yeah my experience is
just an interpretability um
uh this works works really well and we
could talk more some perhaps i'll
produce another video sometime about why
you should expect nmf to work
particularly well but nmf is often a
very very nice way to handle things
if you want to do some kind of
dimensionality reduction
so the idea here is we're going to take
all the attention patterns
in our model
um maybe even all the attention patterns
from multiple models and so each
attention pattern is a tokens by tokens
matrix and then for every attention head
we have one so we get a three
dimensional
and the question is how can you apply an
mf to something that has three
dimensions it's a matrix factorization
technique not a tensor decomposition
technique at least on space and what
we'll do is we'll flatten the tokens um
and the two tokens dimensions into a
token squared dimension into a single
one
and then we'll apply nmf
and nmf gives us two factors the pattern
factor
and the heads factor so the pattern
factor
is going to be at tokens times tokens
and we'll go and turn that in
unflattened into it a token's tokens
components
so it's sort of just like originally we
have an attention pattern and then
rather than having an attention pattern
like a tokens by tokens pattern
by for each head we have one for each
component
and then each component that should be
components not factors but each
component
um is a vector over heads
so we have
one factor that describes where each
component looks
and one factor that describes which
heads correspond to each factor
which component
okay
um
and this is a little bit of a technical
point and
if you don't follow that's that's
totally fine but there is this
interesting observation that actually
this is actually kind of quite principle
to do
and in fact um if you think of all these
equations we've been writing that
describe
uh transformers we always have this
these terms that look kind of like this
where we have a sum over attention heads
and then we have this tensor product of
the attention pattern um and wov though
v for each
matrix for each head
um and it turns out that you can
approximately rewrite that it's kind of
a change of basis but a change of basis
there are tensors instead of matrices in
terms of these components
um and it'll only be approximate if the
you know as you increase the number of
factors will get better and better but
you have
sort of this attention factor that sort
of you can think of as being an
attention pattern that exists for each
component and
then you can go and construct an ov
matrix for that component which is the
the sum over heads of how much uh the ov
matrix for each head and how much it
exists in that factor
in that component in any case um this is
really cool because it means that we
have potentially even though we could
potentially have a very large number of
heads or if we start working with
virtual density nets we can have an
exponential number heads we have a way
to potentially keep trying to re-express
uh things in terms of a smaller bases in
any case this is kind of a technical
point um
and not immediately relevant
so i think the more immediately cool
thing
is that we can go and take all of the
attention heads in the model on all
their attention patterns
and
express in a single image
a sort of summary of the model so here
we have um this is actually not for the
model we're going to be looking at this
was one of my first experiments doing
this with gpd2
um
and
uh you can just
go and take all the attention patterns
and you you just go and compress them
all and here we've mapped each one to a
color and we can sort of get this this
overview of the kinds of the largest
scale attention structures
and you can ask what those are so for
instance um those green stripes turn out
to be periods
um now
you'll recall from the previous video
that often when we see an attention head
attending to periods that's because it's
kind of a
it's acting as a resting position
and so
some of the structure that you see if
you if you apply this natively turns out
to be you see a lot of interesting
structure but it turns out that
structures kind of kind of superficial
um
but
we can do better than that because we
can go and apply it not just to the
regular attention patterns but we can
also apply nmf to the info-weighted
patterns and to the attribution patterns
um and that will be
really informative so let's switch to an
interactive view because that's
um
that'll be a lot more effective for
looking at this
oh
that's not what we got
okay so
um so this is actually going to be
probably a fairly extended chat about
the kinds of things we can see in these
nmf factors um and so feel free
to uh
to end if you just wanted to understand
sort of um what kinds of things exist in
these in these models but uh what we
have here
uh we'll just start with
uh
with we'll we'll start with an an
inflated pattern so we take all the info
weighted patterns in the model for every
attention head and in fact we're doing
this for two large models so it's even
for two models
and we're just going to reduce it to the
20 factors that um yeah that do nmf with
20 factors and so we get 20 different
um yeah we had 20 components that have
expressed different different common
patterns um that the attention heads
have and some are are not that
interesting like even though we did
infrareding we see that there's one
factor
um that really wants to attend to the
eot
token so
and that gets
that's there
um but there are a lot of really
interesting ones so
uh
one that's kind of cool is we see um
there's a
fact a component that corresponds to
attending back to
um
the uh
sort of to
to persons or pronouns or something like
this so here we have sun
um
boy um and one way that's that's often
effective to understand these things is
we can just look at
uh
every token how much it's attended to
and you can see that
more or less these are these are all
peoples in some cases they're nouns um
here we have we have blonde over here
but it's um probably
blonde could be also a noun that would
be a person so that makes that make
sense
um and we'll we'll see more when we look
at attribution that things like this uh
are really helpful because you might
want to predict later pronouns um or
like the gender or plurality of a
pronoun maybe um or even just attributes
when you're when you're describing
someone um might be might be helpful to
go in and remember what sort of person
you might be talking about
um
yeah we also see
uh ones that uh correspond to
um yeah to because although
but
um
uh
there's there's an interesting this is
actually a very common thing that you'll
see
i don't know what the linguistic term
for this is but you'll you'll often see
um components and attention heads that
seem to attend back to the last thing
that could either
um
either be
a introducing a that clause a verb or
something else that creates a that
clause so like um you know let's you
know they were proud that and it doesn't
actually do that but um you know
they could have said proud that or maybe
proud to um say that
um
with that um you could think that
uh you could have uh an opinion that
could be useful that
um
you might fear that
um you could have a reason that
um
uh you might want
uh
that somebody to do something
um
yeah i guess shut or maybe it's really
two but it's sort of functionally
similar um you might think that um so
you have all these things um you might
expect that um in a lot of cases you
will lied the that informally
um so like when we say opinion there was
it's it's really it you know it's their
opinion that there was so it's and it's
all these places where something like
that could be introduced
um
uh and you could imagine that's really
important because then in the next class
if you if you look at these and you look
at which tokens attend to them
and it's really the next clause
and you know the kinds of words that
might be in a in that clause really
depend on um you know
what is the context of this clause so
that's that's a very useful attention
pattern maybe
um attending to the previous token and
so i should look at it that we're
looking at where we're attending back to
um so that's that's a very common thing
um this one will be easier if we look at
where which words are getting the
attention again this is attending to
people but it's really attending to any
word
um
that implies
a gender or whether something's plural
or maybe a noun but um so it's it's it's
yeah there's we see there's there's lots
of people there's lots of um pronouns
titles misses implies you know female uh
mr implies male so that's that's helpful
they imply is plural
um uh
and yeah we'll we'll see later on that
things like this you know her implies
female against sister implies female um
uh the
where where is this but uh the the lees
and dursleys implies plural and we'll
see this is super important um because
there's actually a fair amount of
intervene guessing you know what is the
right pronoun what is the right
plurality what is the right conjugation
of a verb um so tension heads like that
and attention patterns like that are
very common
um
of course we see attention heads that
are related to induction so um it's
probably easier to look at this way that
we go and we we were on d
we look at previous cases where d
happens look one ahead and we see lots
of attention patterns like that and nmf
will start to split them up because
there's there's lots of induction heads
that might be doing very slightly
different versions of induction
um
yeah here's a factor that maybe is
approximately looking at verbs um
uh it could be that it's maybe trying to
get at the tens of verbs um or maybe
it's just useful to know what the the
last verb is
um it's there's some exceptions like
obviously director isn't a verb maybe if
we had more factors this would split
apart a bit more and we get something
that's more clearly a verb
um
yeah so those are those are factors that
i thought were pretty interpol um and
interesting now we can instead of doing
20 factors we could do 40 factors with
the the introverted patterns and you
know i wonder if there's some here that
are
uh are worth mentioning
um a lot of these are
uh
pretty similar to things that we saw
like there's a
there's one that is yeah cases where
there could be a bat it's maybe a little
cleaner um yeah
you know
say that expect that
um it was their opinion that
um
it might be useful that
um
you might think that you might pretend
that
um you might you might think that it's
possible that um
i guess shudder two you might
you might want somebody to do something
or want that something happens you might
know something know that
um
you might have a reason
um
yeah reason that are you you might if
reason was functioning as a verb it
might be uh yeah reason that so this all
these words that are sort of introducing
a clause
um a particular kind of clause i guess
um
this is interesting uh you'll you'll see
sometimes attention heads that look for
the to the first token of words so we
should look switch to a mode where we're
looking back
um so ordinarily it attends to
the
token that's on
um
but when you're in a a word or a
compound word um it'll often attend to
earlier tokens in that so yeah here we
have we're on potters it's attending to
the first token if we go to dursley it's
tending sort of to to misses and the
earlier tokens in it um to the d there
um
mustache we're attending to the earlier
tokens in mustache
and so that's a fairly common pattern as
well
um
yeah uh
here again we have something that's
maybe uh you know the
person the gender plurality and we have
lots of um lots of attending to pronouns
man is another word that sort of implies
something about gender maybe and things
that attend applied plural and this is
gonna be really useful again for for
predicting conjugation for predicting um
uh for predicting pronouns later on
um you'll be able to go and recover some
entropy that way
um there's a lot of induction heads here
i'm going to skip over that and here's
one that may be doing tense
um again when we're just looking at
factors like this if we until we get to
attribution we can't really uh say for
sure what the function is but you know
this looks like it might be you know we
have an imperfect past tense imperfect
past tense
um we have some some perfect
uh
tenses we have uh some subjunctive some
subjunctives we have some some gerunds
um
and that really seems to be what we're
what we're mostly attending back towards
um
uh could also be telling us maybe a
little bit about the the conjugation of
the verb
like is there a singular plural
subject in some cases
um yeah bunch more induction heads oh
this one's cool that's worth talking
about um
you'll often see these
uh
determiner
uh
uh attention heads and components so
here
um a determiner is kind of a
generalization of the idea of an article
so we'll see a lot of cases where we
have you know a
the
um a
um
yeah thus we have these these perfect
and uh or these
uh
uh
yeah we have these different articles um
we also have her sort of functioning as
an almost like an article for sister
right it's it's it and it's sort of
replacing an article so that's another
type of of what people called term
determiner
several is kind of functioning in a
slightly similar role there
sometimes no functions
in a similar role or such a sort of
functioning in a similar role to that
anything is kind of replacing um an
article there as well
um
uh
yeah um so you you had these things now
it's not exact um yeah here the the
possession here is sort of replacing an
article for sister
um now this there's some things here
that where that doesn't make sense
um although in a lot of cases
uh it's in places where there's a
missing article but
uh or there's sort of an aligned article
in some way but
this doesn't make sense exactly you'll
see attention heads that do this more
cleanly when we start looking at
attention heads perhaps
um
conjunctions
um yeah we have because although but
and but
and
um i guess in some ways a semicolon is a
little bit similar to a conjunction and
then it's joining together two clauses
and which
nouns yeah that that maybe is a little
bit of a noun component
um yeah propositions here
in
in
for um maybe maybe some things that are
not quite that mixed in as well okay so
um the thing that we can do though so so
far we've talked looked at introverted
patterns and compressed them down
to uh
uh
yeah to a small number of components
we're summarizing thousands of attention
heads here just in a few components
um
and sort of getting an approximate sense
of some of the things that might exist
um we can also look at
attribution patterns and that's nice
because it'll give a more functional
um
more functional explanation of these
um
so
uh
yeah the the most one of the most common
things is going to be induction
so
for instance here we have
a
we're predicting d
um so remember because we're now on an
attribution pattern it's no longer the
token that's attending back it's the
token that's being having its logic
affected so this is saying that by
attending back to all these d's we
increase the probability of this d here
um and so it's really it's actually this
mrs token attended back to mrs and then
the d over here and then increase the
probability of this d
okay so
induction's really common um
because it's just such a big part of
what's going on when you look at least
of these direct attribution
patterns
when you look at
uh you'll see a whole bunch of induction
patterns so that's one induction pattern
here's another induction pattern
um
here's another induction pattern that's
getting the middles of the words and you
know
it's getting split apart because
different induction hens are giving you
know slightly different weights to
slightly different parts of the word um
and then nmf pulls them apart
um
but
uh yeah so
you'll see a whole bunch of factors that
are involved in induction and when we go
to a larger number of factors that'll
that'll get even bigger bigger um
we'll see things related to uh
other kinds of copying as well so like
this one's really simple but here we're
copying that dash
um so we have good for nothing
the good the dash increases the
probability the other dash it's getting
copied
um
one thing that's really cool and we will
see a lot of when we start looking at
attention heads as well is um
uh attention circuits that are
responsible for maintaining um gender
and plurality um an agreement around
those
so
uh
yeah okay how did we so the colors that
we're seeing here we're seeing words
that are where their probability was
increased by this attention factor
attention component and um we can look
at what it what it was doing well it
looked at um especially at war
um were is a
uh yeah it's conjugated so that we know
it's plural um so that's that's helpful
um
this day
um
yeah this is actually a little less
clean than i was expecting it to be the
fact that it's affecting these tokens is
very clean
um
but perhaps this component is a little a
little less clean than i was expecting
and we're seeing some cases where where
it's still fairly straightforward but
maybe let's just very quickly
quickly explore this yeah so was
um yeah okay so sun ham those are both
really implying uh male
um singular they the dursleys knew um
and we're also actually a little bit of
attention back to uh
um
a little bit of attention back to the
day there as well
um but it wouldn't surprise me if
there's some virtual attention heads or
information being moved around in a way
that's making this a little harder to
see her sister
there's another
component we'll get a cleaner component
also when we go to the 40s the 40
factorization but um
yeah they
say
um
this one's also not that clean there's
probably a bunch of virtual
attention heads going on
yeah i bet
um
yeah so like here
well i guess shuttered might contain a
bit of information about this but it
could also be that dursleys is clearly
plural and some of it got folded into
there
um by another attention head and then
we're picking it up
um yeah they were going and seeing
seeing a previous day but it's yeah
usually this is cleaner uh so we'll
we'll come back to that in a in a later
one
um
it looks like there's an still another
one so let's have a look at this one uh
yet child so this is this is maybe not
just gender it might also be um more
generally sort of information about
recent people who've been talked about
so
um you know we we have some things maybe
that are related to age and being a
child and that helps us predict this is
going to be child here
um and not just
um
yeah or
uh
her sister we're just literally copying
the word sister a bunch of times
um misses we've seen misses before
something sort of in between
a more simple copying head and something
that maybe declares about gender
plurality or something like this or
maybe a copying head that's specifically
involved in people
um
okay so that's that's another kind of uh
interesting thing and we'll we'll get a
better example of that in a minute
um
uh
i don't know if we talked about that one
there's another one so again we're
getting a whole bunch of components that
are involved in
thinking about
gender plural things like that
that's quite common now one other
important type of
attention circuit that we see
is what i've called an engram circuit or
an engram attention maybe
um component
and
these ones are kind of mysterious
uh and i i have a pretty strong
hypothesis about what's going on but i'm
not sure so
the observation is
um
if you look at these
and you'll see that they
uh you know there's they're they're just
jumping across words in a kind of you
know unpredictable way um and some
earlier word just increases the
probability of this word but if you
start reading them aloud you'll notice
that they sort of they almost rhyme like
her husband well
there's some way in which like those
could just be side by side and they
would increase in probability um
with child
um
keeping away
another reason um
what what neighbors um
uh
four years
time
exiting
um
in strange
um
over fences
uh and there's a bunch of other ones let
me go to another engram head
um
yeah
uh
what would it's so blacked out there
that you can't even see it uh it's so
intense but yeah what would
um
uh that's actually that would
um
so those are those are both sort of
clearly very common phrases like it's
very common for wood to come after um
over what
um past tense didn't
um
they want
um semicolon one doesn't make as much
sense keeping away again
another reason again and we saw that in
a previous one
um
with nonsense
say were
uh
anything or
um let me go and find
there are a bunch more of these but
here's another one
um
yeah anything or
have large
um of time
finer anywhere
and
so in any case i've been ranting about
this for a little bit but the hypothesis
is that these are just
they're they're kind of like bigram
statistics so they're like bigram
statistics at a distance now that's
probably a simplification oh this is
copying the ads and that's kind of a
simplification um
uh and it's probably probably not quite
that literally
quite that literally what these
attention heads are doing it's probably
not quite that simple but um we saw
things like this a little bit even in
the one layer model and and
uh yeah i suspect that there's
uh that we're there's more of it so
let's switch the 40 layer or the 40
factor one again attribution so this is
now we have
40 components um and
uh each one we're seeing which tokens
affected which other tokens through
some set of attention
heads um and yeah just continuing on the
engram one for a second
again it's gonna be a lot of the same
thing um with nonsense
four years
with child
keeping away
another reason
her husband
um
for years
uh
somebody discover
and these are kind of weak
um although they're kind of interesting
now let me go and switch back to this
view so that i can see some other ones
a sun
on the neighbors
um over fences
the amount
in strange
um
you know makes me think of uh god works
in strange and mysterious ways
sorry there was a technical glitch there
um
but
uh yeah let's look at some other ones so
those are the engram ones we see a bunch
of things involved in tense and plural
so here
um
had we're looking at all these previous
words that are past tense
um was we're looking at previous words
that are are imperfect past tense um
singular i guess
um you might wonder about and um but
remember that and is often going to
imply plurality and so it sort of makes
sense that's getting caught up in the
circuit as well
um there's a whole bunch of past tense
by the time we get here it's more
diffused because there's so many past
tense words that we can look at
um
uh
yeah misses well it's more about titles
there so it's that's not quite as clean
um and there's a bunch again of these
these tense
uh
copying sort of attention heads
um
i'm just going to keep going on this for
a little bit because there might be
people who are finding this interesting
but um please feel free to drop off and
just zoom to the next video at any point
um
uh
yeah where's another one
that's involved in
that well there's a we're calling this a
person copying one but it's like um sun
increases the probability of child
sister of sisters
and things like that
and that's not quite gender copying
maybe i'm surprised
um seems like there's a little bit
caught up in that one uh oh here we go
yeah so
um here we have
misses increases the probability of she
um
okay so that that is sort of introducing
a clause that could be subjunctive um
but okay here a bunch of
the dursleys is plural potters as plural
increases day
um
so that's kind of a classic
uh type of plurality
sort of consistency
um all of these yeah the the dursleys
plural they plural potter's plural
increases the probability of day
it's a new clause starting
new clause starting
um
uh yes mrs
implies female sisters female
uh so we're seeing all these things and
and of course you know why okay so why
is this head doing both tense and sort
of gender
um plural plurality agreement well i you
know that those are often kind of
intermingled um since
you know you're trying to predict verbs
um the conjugation of the verb depends
both on the tense and the
the
the
the plurality and gender of the subject
maybe probably just the plurality so
that makes you want to look at pronouns
once you're looking at pronouns you can
easily copy gender information and and
so
uh and you know you can predict later
pronouns um so those sort of naturally
go hand in hand a little bit
okay so
um what about other things well again
there's a ton of heads involved in
induction they're sort of they're
they're pretty simple and we've talked
about them a lot so i'm going to skip
over them but induction
or factors involved in an induction and
it's just that induction is getting
split apart in a way that's a little um
a little sad right now because
that's just
part of what happens when you have a
whole bunch of components and induction
heads are so big and they they attend in
slightly different amounts to different
different examples
and actually wants to do that
okay so what do we have here that we
haven't talked about
uh
maybe
is there anything here
that we a lot of these are just copies
of things that we saw
previously
now there's one that's maybe
specifically involved in copying titles
mrs misses
um
uh
yeah i think i'm going to call it a d
there so
um maybe let's just very very briefly
summarize um if we look at
uh
particularly if we look at the
attributions
the
basically the the direct attribution
effects of attention heads have a pretty
simple story there's a bunch of heads
that are doing engrammy things
um or these sort of bigram skip type
things that sort of seem kind of local
and seem to just be about the
there's lots of words that have affect
the relative probability of words nearby
and we can just in a pretty simple way
model that
there's a bunch of these um induction
and copying heads that's just tons and
tons
and tons
of induction
type stuff going on um that allows us to
go and just copy previous words um that
and and increase the probability of of
when something similar happens um the
same same tokens appearing
then there's a bunch of heads involved
or
attention components involved um that's
not one
in in copying people
gender plurality handing verb subject
agreement handing handling verb tense
you know consistent tenses between
sentences
um
so there's there's a lot of stuff like
that
and honestly that's a pretty big
fraction at least of the
interpretable stuff that seems to be
going on when we look at attribution
um when we looked at raw attention
patterns we saw some stuff that isn't as
emphasized when you look at attribution
like we saw this determiner
component
that was really cool or the preposition
component
those didn't have as dramatic an effect
um
when we were looking at the
the attributions they're probably a
little bit smaller
in terms of their effects or primarily
having indirect effects um so maybe
their effects are more mod yeah more
um
more moderated uh by later retention
heads and mlps rather than having a big
direct effect
um but yeah so that gives us a sense of
some of the things that we should expect
to find when we are poking around in
these models
um and in our next video we'll actually
look at the attention heads um and sort
of move past these factors


Attention - General - Indirect & n-gram Attention Heads [rough early thoughts]

well our last couple of videos gave us
tools um to go and talk about attention
heads and large models and and try to go
and figure out what's going on with them
and so in this video we're gonna start
looking at them
now
large models have
um a lot of attention ads and a lot of
them are pretty interesting so uh before
we uh start looking at them it's worth
maybe just sort of trying to organize
them a little bit
and and what we've done here is we're
just going to make a diagram to organize
them a bit
and
it's not that this diagram is
necessarily you know fundamentally the
right way to think about things i don't
i suspect that it's probably not um it's
just
going to be useful for this particular
video and to go and just sort of
organize our thoughts a little bit so
it's probably you know it's probably not
an especially great way to think about
things but it's just a way to to go and
get us to organize things a little bit
and we're going to start
well before actually we talk about the
the categories here um i think actually
the most important thing to mention is
you know there's lots of attentions
we're not going to talk about because we
don't have uh you know particularly
uh
strong theory so there's all this the
space here sort of for uh categories
that we're we're not going to be talking
about because we we don't really
understand those attention so we're
going to talk about attendance where we
do have theories but it's worth keeping
in mind that there's there's lots of
things that we aren't talking about
and
for the attention tense where we do have
theories there's really two kinds of
theories we might have um so first we
might have um attention heads where
we have really
a theory based on their attention
patterns we can say oh you know they
tend to attempt to act so they tend to
attend to y
um
and
that's that's something that we've seen
in a lot of previous work
and has been really interesting
the other thing we can do
is we can sometimes go and develop
functional theories
and so what i mean by that and we have
all actually i think more attention
heads really where we can have
functional theories the ones where we
only have
um theories based on attention patterns
and what i mean by having a functional
theory
is that
that
we
we think we understand
at least part of what the model the
attention heads role in the model is so
what does it do what you know one and a
ten somewhere
how how does that affect the model's
behavior that attended there
um
and the way we'll do that is we'll look
at the direct effect
of attention heads on logits you can
look at the output of the attention ed
and just run it through
uh to the softmax and see how it affects
the softmax so and we can look at the
direct effect
and that's uh that's very similar to
well this is yeah we developed this tool
the the logit attribution pattern in in
the previous video so that's what we'll
be using to do that
now in a lot of cases we won't be able
to develop a functional theory and that
um that's probably because the attention
heads have an indirect role um we might
also just not have understood things
correctly but uh i think mostly this is
for
especially for early attention heads
where
uh the you know it sort of makes sense
that lots of other layers are going to
go and process things and you know the
ml later mlps are going to process
things and combine things with
information and later retention heads
might move it elsewhere and so
just looking at the direct effect seems
like it's probably not telling the story
uh and we probably don't understand
things very well
so there's gonna be
a lot of attention it's like that but
we'll still be able to go and observe
really interesting things like all this
previous work um where we'll observe
interesting grammatical structures and
patterns and
interesting things like that
and then
for the ones where we do have a
functional theory there's going to be
two
major subcategories so first um there's
going to be a lot of attention heads um
that really seem to be doing
uh or at least our theories that they're
doing something like um
something kind of like engram statistics
um
or or sort of skip engram statistics
where they're just sort of saying you
know
given that there's this head in in in
the last couple of tokens uh or this
this token in the last couple of tokens
and you know these words are more likely
and so you know people talk a lot about
you know language models or just
statistical pattern matchers um and uh
you know i think people probably mean
different things by that but uh i think
these kind of engram-like heads are
really like
you know the most extreme version of
what you might what you might mean by
that um or implementing the most the
most extreme sort of simple version of
what you might mean by that
seem to be at least
the other category is these copying
um or and copying heads and in
particular this the subtype of copying
heads called induction heads
um and those really uh are actually one
of the most interesting things that
we've discovered and they seem to be
tied up uh in in meta learning um and
uh really uh yeah just really related to
a lot of the most interesting things
about transformers and they're so
interesting we're going to give them
their own video so um if you just want
to hear about them you can skip ahead to
the next video um but for this video
we're going to go and focus on the um
the infor weighted pattern or the um the
ones that we can understand in terms of
their attention patterns and um we'll
we'll often use the inflated patterns to
investigate those
and uh the attention heads that we that
sort of have this engram-like behavior
and then we'll we'll talk about the the
other ones in the next video
okay
so
to start with these ones that
uh
have
yeah that we we understand in terms of
their attention patterns that um that
sort of seem to go and do something uh
natural and interpretable in terms of
their attention pattern but we don't
really have as much of a of a functional
theory
um
well i guess one thing to say is just
that this is this is very similar um i
think
to things that people have found in a
lot of previous work and and frankly
we're going to be doing um you know um
really a much less thorough job than a
lot of these papers so uh maybe it's
interesting in that we're gonna be able
to confirm or sort of um have slight
variations on things that people have
seen but um you know a lot of this is
going to be very similar in flavor to
these
these previous
uh papers and um yeah i really encourage
you if you're interested in this kind of
thing to look at them
um but yeah maybe we'll we'll just look
at uh a couple of these heads and uh
sort of get a sense of some of the
things that we discover for for
attention heads in terms of their
attention patterns
um
so this first attention head
has
a very it's actually i think a very very
simple
uh
behavior
it almost always has a couple exceptions
it almost always attends to the previous
token
so
um
uh just to back up for a second um we
have this interface that allows us to
explore attention patterns and also
informative attention patterns and
attribution patterns and things like
that
um
and
we need to give some sample text to poke
around and because it's fun we're going
to go and use the first paragraph of
harry potter um but obviously we could
use anything
um
and
uh yeah there's a few other things that
i'll sort of
point out as we as we play around with
it um if you watched previous videos
you've seen the same interface
um
yeah so uh when we hover over a token it
gets a square routed denoting that it's
the present the token that we're focused
on and we can see that this attention
head all the attention goes to the
previous token
now there are
a few exceptions to this
um really i guess there's really one
exception to this which is sometimes
when you're on a period
um you attend to
the period instead of to the previous
tokens so usually you attend the
previous token but sometimes when you're
on the period you attend to yourself
and that happens for some periods that
happens here
i think it happens here
um
but there's other periods where for some
reason it doesn't happen so this period
tends back so it seems like it's kind of
i don't know it seems maybe arbitrary
whether it does it or not i don't have a
theory for why
um but i think it is kind of natural to
to not attend backwards on periods
and
because
well you you could imagine that a big
part of the role of these might be to
help construct um sort of trigram
statistics so to go and say well you
know given the previous two tokens were
this you know the next token is likely
to be this
and
uh
you know i guess i guess what it means
for these to be the two previous tokens
is very different when when you're on
the other side of a period and sometimes
that period is a much bigger gap than
the normal gap between tokens
um and so you could imagine that that
given that um
yeah it's helpful to sort of ignore
ignore the the token or or treat it
differently when it's when there's a
period in between than you would a
normal previous token
now it's worth mentioning this kind of
attention this kind of you know
attention that looks at the previous
token um is
very similar to things discussed uh in
other papers um
uh
um both voda at all and clerk at all and
probably a lot of other papers have
observed the existence of what they call
positional
um attentions that attend to
tokens a fixed position uh relative to
them and sometimes that's attend to the
previous token
in their case they also have attentions
that attend to the next token
um
now attention heads that attend to the
next token um you can have that in and
they're looking at bert and you can have
that in bert in our model um since it's
an auto auto regressive model it's just
looking at the previous tokens trying to
predict the next token it can't look at
the at future tokens and so we don't
observe these attention ads that look at
the next token only attention heads that
look at the previous token
um one thing that's worth mentioning is
we don't observe attention heads that
attend two back so we have attention
instead of attend one back but not
attentions that attend two back or three
back or something um of course there's
attention it's that attend further back
but they're they're not attending to a
fixed number of positions they're
they're doing something that's more
involved in you know what particular
words they're attending to
um and you might find that surprising um
i found that kind of surprising
initially but i think what's going on
is that if you have
multiple previous token attention heads
and these models tend to have a lot of
them you can chain them together so
two previous
um token attention ads can function as
a two token
attention head
two tokens back attention so okay in any
case that's that's a long rant about um
previous token attendants
now uh another
uh really interesting thing maybe has a
slightly similar flavor
um
is
well maybe maybe rather than spoiled
let's just poke around for a second
we'll discard to discover together so
um
there are a lot of attention heads
that
uh just attend to the
the present token a lot of the time
there's an exception um
but this this attention and most of the
time it tends to the present
token
um
okay but there is an exception and the
exception is what makes it makes it so
interesting
and
i think the easiest way to go and see
this
is to
uh look at which tokens get attended to
um and maybe especially if we look at
the informated ones
um
okay so
for instance let's look at dursley here
and we can see that all of the attention
is focused on the d and none of it's
going to the later tokens within that
word so um we're using bp
tokenization and that means that words
often get split up
and so something that's interesting is
we'll see if we um if we look at it that
all of the tokens within dursley attend
to the d so d attends to itself and then
all the later tokens attend to the d
okay so that's an interesting hypothesis
um
so maybe let's look at other jerseys
there's another state's three tokens
again it's all attending to the d it was
tends to a little bit as well but not as
much
mustache is split into multiple tokens
and they all attend to the m
um dursley again
um and so let's look at
um there's other things going on here
right let's look for a second that um
words that are split into multiple
tokens so dursley's one it all tends to
d mustache
dursley
um dursleys that's plural
um
dursley's within a possessive um drills
there we go we um we have drills and the
the ills are attending to the dr we've
gruntings um that's that's one word sent
to a bunch of tokens they all attend to
the g at the beginning um potters
uh
yeah so we're seeing that a lot of cases
now there's some cases where that's not
what's going on um like thank you for
instance um is
we'll see that you
uh is attending to thank so really in
some sense thank you is sort of being
treated as as one word or um
very much very is attending to itself
and much it tends to vary
and so there's all these things
that they're the models if you if you
take this theory this is helping the
model sort of um undo tokenization and
combine things together that
uh are split into multiple tokens that
the model sort of wants to think of
maybe as one unit um there's some
something which maybe it thinks that
very much is sort of one unit just like
all the tokens within dursley are one
unit
um
and there's there's a lot of cases of
this
privet drive one unit number four one
unit um so there's lots of things like
that
um
so i think this is really interesting um
because often tokenization seems very
unnatural and awkward and splitting
things up in really weird ways
um and it seems like the model has to be
somehow undoing that and
you'll see a lot of attention that seem
like they might be involved in that
and of course those attention heads um
you know this is this is a very early
thing that you want to do in the model
and and then you can go and use it
for your later um in later mlps um
probably you have to sort of process
things to understand maybe more the
semantics of of the unit that you're
combining together um and then you can
use it to do other things
um
okay
our next attention head
um okay so if you just look at it like
this it's a little bit hard to figure
out what's going on often we're tending
to eot
um something that will make it a little
bit easier is if we um make our default
view seeing how much
every token gets attended to rather than
does attending so here we see um a lot
of tokens are attending to
uh
to i guess probably one through you
might have some verbs that might be an
initial theory but opinion is a counter
example
um
and but we still have a lot of attention
focused on on the eot for instance
um or this period here
and so that's kind of weird um but we
know that often attention heads uh
attend to this sort of as a resting
behavior um and so if we look at the
influence pattern which is just
uh scale everything by how much
the uh how much information is copied
from them uh this gets a little bit
sharper again
um we're still attending a little bit to
eot
but not as much it's become weaker
um
and
i you know this is really speculation
but the thing that i've i've come to
think about this attention and some
related ones is that they're they're
about words that might be sort of
setting setting the stage for a clause
um so that can be sometimes that can be
conjunctions that are just you know
literally joining together two clauses
um
but
in a lot of cases they're words that
might continue either explicitly or
implicitly um with a two or that so
proud to say that
it might be your opinion that you might
fear that um you might well if it's just
an explicit clause but um you might say
that um you might know that you might
want that
um you might
reason that or you might have reason to
um
you might want that you might shudder
too um you might think that you might
have you might have seen that
um you might pretend that
um
uh i guess bear uh yeah you you couldn't
bear to um
or or bear that i guess you you're sort
of all lighting it there because it's
it's sort of a
um yeah something that we do sometimes
in english but there's sort of
implicitly something like that and then
yeah there's a bunch of other um
yeah found out that um so a lot of these
really are so many places where that
that kind of thing is going on and i
think it's actually very natural um you
know this is this is just a hypothesis
um and we're we're sort of just giving
anecdotes from here and these attention
heads but in this case we'll have more
rigorous investigations and some of the
other other stuff but
um
i think it is very natural to think that
this kind of attention head might exist
um because
in some sense these words are are sort
of setting the scene for the next clause
um you know the the kind of words you're
going to use in a clause about something
that you're proud of are very different
from the kinds of words you're going to
use um in a clause describing what your
you know what you what your what your
greatest fear is that and what you what
you fear that um or a clause about
something that you know is very
different from a clause about something
that you want um or that you shudder
about um or um you know if you think
something um you're kind of caveating a
little bit and you're gonna say
something different so these um
these kinds of words they're they're not
exactly conjunctions but they're um
strictly but they're sort of setting the
stage for a clause and i think that's um
that seems very useful maybe
um who knows i i'm not a linguist so
maybe there is a formal linguistic term
for this i just don't know what it is
um
okay and here's another one that's going
to be easier if we
switch to looking at where the attention
is going and looking at an infrared
pattern
um and uh yeah this this one there's a
probably a very natural theory that's
gonna drop but jump out at you is um if
you look there's it's almost all
articles the a
a
the
a
the
now there's a few exceptions
um
like there isn't an article but it's a
determiner so determiners are kind of
this this generalization of articles um
where if you have a possessive word
somewhere it sort of replaces the
article or if you have a quantifier or
um
a word like this and they they replace
the article now there's an exception
here
of uh
i mean it's sort of of a so maybe it's
still those two together are sort of a
little bit acting that um that that is
an exception to that theory but um this
attention head really seems to want to
attend to
uh previous
articles or previous determiners
um and yeah that's kind of interesting
um you know another another attention
that sort of
uh mimicking grammar
modeling grammar
um here's another one that does the same
thing
um
but
uh
it's actually even
uh well it has had some things where it
attends to something that that isn't a
determiner but um it's also much more
general so not only do we have words
like a um and
a and the and there are those we saw
before but um other words that are uh
determiners like um here we go
possessive um the dursleys that's taking
on the role of an article for sister
no so quantifiers like that can um act
as determiners um
another is another word that's taking on
the role of a determiner
and
several is taking on the role of a
determiner there
uh what else
um
uh
that in that case that isn't but there
are cases where like if you said that
boy that would be taking on the role of
of being a determiner or this boy um
there should be cases where that's
happening yeah like here um this boy
there um
now there's a lot of words here that
where that doesn't make sense um so
there's lots of exceptions like
um
with or because or things like this
are not determiners but this attention
it seems to preferentially attend to
determiners
now another interesting category again
um let's look at where this attention
head is attending back to
um
and let's switch to an informative view
um and we'll see that this this one
really likes conjunctions although which
but
but
and and
which
but
um so
it tends to tends to like conjunctions
and
we also see that it's times like things
that um
uh yeah the periods um
uh or the first token of a sentence a
lot of the time and then you could
imagine that you know in cases you know
conjunctions are linking clauses in
cases where there isn't one between two
clauses maybe you maybe you attend to
the first token or the period and you
it's very faint and so you aren't moving
that much information from them
um and here's another one doing
something similar
because with which although
as and but
and which and but
and because
period semicolon in fact but
um those are all all conjunctions
um okay so those are some interesting
attention heads that we found there's
there's lots of others um but i'll i'll
go and leave it there
and
let's go and talk about the engram hats
so um the other category of attention is
that we're going to talk about in this
video and then we'll talk about the
copying and induction heads in the next
video are what we're calling um sort of
engram-like heads um
uh or sort of maybe another way to
describe would be like short-term
statistical heads or something like this
um
and so i think it's just easiest to talk
about this uh with an example so what
we're doing here is we're looking at uh
the logit attribution pattern we're just
grabbing pieces where there's larger
values in us these are cases where
um
some word
yeah where the attention head um by
attending somewhere increases the
probability
of another word and it just uh directly
has that effect on the soft max
um
yeah so let's just read through these
and there's sort of this funny property
that especially when it's big because
they almost
feel like they rhyme
um
say they were expect to be because they
just although he did
had nearly twice
time craning um and even just like time
xing um opinion was finer anywhere
um was somebody that would
think could
past instant
what would
if arrived
that had but never never seen
um so
this is really speculative but um
the thing that i always think when i i
see these heads
is
there's just some way in which it really
feels to me like like there
there is there is like that there's sort
of this rhyming feeling to them when i
when i read off the the attributions
um and i i really strongly suspect that
there's
there's something about just very local
statistics of of language it's like
bigram with jumps between it or maybe
maybe sometimes it's uh you know it's
mediated to some extent by what the what
the word that's tending back to that
token is um
and uh i i only developed this theory
after um studying the small models if
you watched our videos on the one layer
attention models you saw that there were
these skip trigram heads where we could
really understand them and this is
exactly what they were doing and so my
my suspicion is these are just um you
know the large model versions of that
and maybe they're a little bit more
sophisticated maybe um you know the
choice to attend back uh is based on on
several tokens rather than just the
immediate token and maybe um maybe the
effect of of the token that you're
attending to is mediated by by other
tokens around it and
but that's that's kind of the hypothesis
so let's look at a few more of these
um
people too with nonsense
um spent time over fence
um spy on
on neighbors
bear if for years
um keeping away with child
and especially look at these big ones
like the ones that have really big
effects spent time
that you know that those could just be
side by side
four years this could just be side by
side so this there's really this this
feeling to them again this is just a
hypothesis but um there's really this
feeling to them that they uh are
encoding something about about
short-term statistics
um yeah again this is going to be very
similar
um
big man
a large
was blonde
the amount
um
time acting we've talked about that one
over fences we've talked about that one
with nonsense we've talked about those
oh i'm on the other side um
a sun
somebody discover
four years
half sister
her husband
um a son
another reason
keeping away
with child
and you know this is just some feeling
to them that they they really seem like
um
yeah i don't know
um it's an interesting thing going
through these so maybe it's even a
little bit less compelling i think i set
a lower threshold for whether to include
things here
um well it makes sense that thank you
maybe increases the probability of a
period
um
people
uh i don't even know like people
subjunctive
um
anything or
with nonsense was director
um
although did
in useful time accent we've talked about
that boy anywhere
um yeah these seem a little bit weaker
but let's maybe just focus on the big
ones like think could
um but hadn't
four years
um
possible well possible b possible period
uh
what neighbors
but had
but never
seen period
this was
keeping away
with child
yeah so
these engram heads again they're they're
just speculation um but they really
uh
it really seems like there's
there's something about local statistics
that some of these attentions are are
including and one thing i think is
interesting as it means
um
these attention heads
just like if you if you look back to the
first video and you look at the skip
trigram heads that we were able to
really reverse engineer and understand
exactly
um these sort of end grammy heads
are
uh
you shouldn't expect them to have you
know obvious interpretable attention
patterns um because the
they're sort of less about the the
function of the words and it's more like
this word happens to be suggesting or
this token happens to be suggesting that
these other tokens might be more likely
and now it seems like they could be
injected here
um and so the attention patterns are
very opaque a lot of time for these
attention heads um
but i think that there is there's sort
of a natural explanation for that
okay so uh in the next video we'll be
talking about copying and induction
heads which i think are probably the
most interesting thing we've discovered
so i'm excited to chat about them with
you