MLP Neurons - 40L Preliminary Investigation [rough early thoughts]

so
one of the directions that
uh our research and anthropic has
involved has been
um just trying to understand
what different neuron neurons and
different layers of transformers are
doing
um inside the mlp layers
and it's been something that we've
haven't investigated as much as
attention heads and attention circuits
um and honestly i think we've found to
be more difficult than we were initially
hoping
but we have learned some pretty
interesting things
and so
uh hopefully in this video we can we can
talk about some of them and maybe it'll
be useful to people um who are
investigating similar things elsewhere
uh so i'm lucky to be joined here um by
my colleagues catherine olsen and nelson
el-hajj and uh we'll be
uh just going and chatting through some
of the things that we found
um so yeah i thought it would be helpful
maybe just talk a little bit um
so this is sort of really just a
preliminary investigation um and maybe
just to go over a few basic points i
jotted down a few things um but you know
as we talk through these maybe uh
catherine and nelson if there's anything
you want to add just jump in and and say
that
um
all these results are going to be from a
kind of moderate-sized model with 40
layers and we find that there's a lot of
differences between features that exist
at different layers and
but probably the the exact number of
layers doesn't matter and i think our
intuition is that in most cases and if
you look at models with different
numbers of layers you'll find similar
neurons kind of roughly in terms of the
fraction the way through so if your
neuron if you're at sort of five percent
depth often that's what we see sort of
generalize
um we'll be investigating mlp neurons
not neurons on the residual stream um so
i guess in some ways neurons on the
residual stream is kind of kind of
probably an oxymoron um or like a um
probably not the way that we would use
the terminology i i think when we when
we talk about neurons we mean something
that has an activation function and
since uh the residual stream is just
linear functions being added into it or
linear projections being added into it
you sort of can't can't think about it
that way
um we'll be talking about a number of
different decks in the model
and yeah usually our process is that we
we start with data set examples um and
then we start going and after that going
and interactively playing within our on
and when it activates to sort of verify
our theory chris what's the data set
example what's a data set example that's
a great question well kathryn how about
you tell us where a dataset example is
sure absolutely um so we take the neuron
that we're looking at
and throw a bunch of different examples
at it um just tons and tons of different
uh possible contexts that's in the data
set that it was uh the same distribution
that it was trained on and just see what
it fires most on or where it's sort of
highest activations are and we look at
the top couple like so it's asking like
what is this neuron most like or where
does it show the highest activity and so
we just look at those pieces of text
where we saw the most activity from it
and i think it's not that we think that
data set examples are dispositive about
what a neuron does but they're um
they're going to help us build a
hypothesis um and sort of narrow down
the space of things that it could be
doing
um do you also i feel like you probably
are the person who spent the most time
studying neurons if there's anyone who
should really comment on um what our
process has been to try to understand
our own really i feel like you should
feel like you can you can elaborate on
that yeah yeah i mean one thing i'll
just i'll just say is as chris said
starting with these uh positive examples
is only the first part of the picture i
feel like i didn't really get traction
until i could start to look at negative
examples as well um so when i say
positive examples i mean you can see
that this neuron really likes text of
this type uh but then you have to ask
what is it like everything of this type
or are there things like this that
actually are excluded so you need both
to know what's ruled in and what's ruled
out uh by sort of the rule that we might
use to explain the behavior and so
there's a couple different ways you can
get this sort of ruling out um ruling
out behaviors as well so you could look
at you know tokens that it likes uh but
contexts where it didn't fire on that
token because of something else going on
in the context that's useful and also
playing with interactive interfaces
where i can just type text uh sort of
exploratorially uh and try and feel out
can i get it to not fire where uh it
might otherwise fire on that token so
getting that sort of other side of the
hypothesis space but all very
exploratory it hasn't been that um
rigorous yet so much as trying to feel
out the space
and i think one thing that's that's just
worth being kind of super explicit about
is is we tend to talk about like what a
neuron likes or what a neuron is looking
for
we're
we're looking at a point in the model
where an activation function uh
non-linearity is applied typically a
value or a you know some variant of it
this model uses gel u and so sort of
when we say a neuron likes or an iron
fires we typically mean that it has a
positive activation at you know that
scalar value has a positive activation
which if we're using relu means that
that value is going to be passed through
unchanged if it sort of doesn't like it
or or doesn't fire it's somewhere in the
negative regime
um okay we tend to use this sort of
anthropomorphizing terminology and i
think it's sort of intuitive sometimes
when you're looking at these neurons but
it's just worth being super explicit
we're just talking about like the value
of that scalar at that point
i'll also say
in this exploratory phase we've largely
been looking at very sparse neurons that
is that fire infrequently throughout the
whole body of text that they've seen
because it's an easier place to get
started again that's going to be a
strong bias in what we're reporting here
but that was a choice we made to get a
foothold so there's only a small number
of different tokens that they fire on at
all whether that's uh you know how often
in the data set or uh different uh token
identities and that allows us to get a
sort of tighter uh tighter sense of what
it's doing as opposed to a neuron that
fires diffusely throughout an entire
text which is more challenging
yeah and i guess a a related thing is
because these neurons are sparse and
both it's the case that the typical
neuron in the model is quite sparse and
also that we've often preferentially
looked at sparse neurons um but when a
when a neurons very sparse and trying to
understand
like you could sort of think of the case
where it doesn't fire as the default and
so like you might ask you know why is it
that you're looking at you're looking at
the examples where it fired rather than
the cases where it didn't fire well you
could sort of think of it not firing as
default and it firing is the exception
that you're trying to understand um
whereas if you had a neuron that was
sort of equally often say to have
activations of zero and one you might
not be able to do that as easily
one final point that's maybe worth
making is just that um
i at least for me i've previously uh
sort of done this the circuits type work
and vision where we're exploiting the
fact that you know vision models have
have a number of neurons where it's at
least theoretically plausible that you
could look at every single one
and transformer large transformer
language models this isn't necessarily
true and so uh we're showing you some
neurons but
um it sort of seems seems almost
impossible that we would look at every
single neuron
and some kind of strategy if we want to
scale this is going to going to require
us to do something else beyond that
okay so with that said i think we can
dive in
and i think that it's sort of
interesting that there's some layers
where we've had a lot more luck
understanding neurons than others
and broadly i think our biggest cluster
of successes and our earliest successes
were these relatively early layers
um sort of roughly especially in the
like
10 to 25
depth regime
and we had a lot of luck but the the
earliest layers we found quite
challenging and then we found that
features became harder and harder as you
got deeper past that
and then towards the end and especially
in the last layer we also had a lot of
luck um and so there seemed to have been
the places where where we've had the
most luck understanding neurons in these
models or the most success
um
cool so i think we can just dive in um
and
uh layer five so in this in this model
that's about 12 depth and i think we
found a lot of pretty interesting
features
uh catherine do you want to comment
underneath these
yeah i mean i think this is sort of just
a quick signature of this sort of uh
positive description what does it fire
for and that these sort of layer
five-ish neurons again that's sort of in
the ten percent through the model kind
of regime um they're sort of like little
semantic uh semantically similar phrases
so they're like i would like it would be
nice i will like neuron in the top left
there um
it's there are other cases where the
word like shows up and it doesn't fire
so if you just say i like pizza that
doesn't work you say i would like pizza
then that works um and there are some
ways to sort of um
fool it or trip it up and sort of uh
write something that contains like you
know i i'll come up with a specific
example later but kind of mash up the
words and then it does fire even though
it doesn't mean semantically the same
thing so they're not like
ultra ultra precise at picking out
things with a meaning like i would like
um but they do rule out other like i
like or i love uh that doesn't have this
same sort of future tense preference
kind of expressing
thing
one thing i found pretty interesting is
some of them respond across multiple
languages to short phrases that mean the
same thing
and so for instance there's this like
um
this one here that at the at the bottom
left um that
we have not only that we have also in
french non-somal or non-songs um or uh
uh there's there's another one that
responds to four reasons but also for
essa razol um and
yeah so
i thought that was kind of an
interesting
pattern
um
i've wondered if it might be helpful to
conceptualize these as being like
um
i think linguists have this idea of a i
don't even know how you pronounce it but
a frasmi or a phrase me of like words
that just frequently co-occur and just
statistically in english way more often
than they should
say it again
phrasing like a phrasing
a phrasing
excellent yeah and so i have wondered if
these might be kind of like that
um but the fact that it's across
languages and it's grouping similar
things together things like maybe maybe
thinking it was like very small semantic
units as a as a better framing
um
and yeah this is just looking at one of
them but uh this is the the n different
or n separate or n independent and you
can see that you know it's firing on the
word independent or different or
distinct but in every case there's a
number immediately before it
okay catherine you should definitely
discuss this because i think this was
this is one of the things that i was
totally stumped by and catherine figured
out yeah i'm happy to take it away so
i'll describe this briefly and then i'll
probably share my screen and show sort
of a bit of the sort of uh discovery
process but
at a high level this neuron uh
showed up in the dataset examples
preferring where the word install very
strongly
if you look at other tokens that it
fires on it might more weakly fire on
other words that seem somewhat computer
related you know package was the other
one that showed up fairly often
but if you look at these data set
examples they're not very
computer related and so in when playing
interactively with it i can get it to
fire if it's for a computer related word
specifically install or more weekly
maybe package but the context doesn't
suggest that um so maybe i can just
steal um
presenting for a second show
show some of that
cool that's great well catherine's
switching um i was really quite confused
by this neuron um because it it seemed
to be firing for all these words but
very mysteriously would only fire some
of the time and you'd try writing
sentences with the word install on them
and you know it wouldn't fire and so
uh yeah i was i was pretty confused by
that
yeah so let me pull up so this is a
little um dashboard that we have for
checking out what neurons are doing so
i'm going to go to the dataset examples
page which is where we generally start
when trying to understand what a neuron
is doing and so you see if you scroll
through it's the word install i'll just
scroll through very quickly first and
then we can um
pause for a moment so these are all the
word install everything is installed
but
go ahead one quick addition on that is
it's the token install which isn't
always necessarily a standalone word so
um because these models split things
into tokens and that doesn't always line
up and in some cases here we have
install as part of installments um but
it can't see since it's it's auto
aggressive it can only see the past and
so it doesn't know that it's gonna be uh
installments it just knows that it's
yeah it can only it could be installer
it could be installment
right
so in this case as i was describing um
this is paying fee in installments it's
not installing software um if i scroll
up i think there was another um
oh yeah um
the weekly french fried eyes are fries
evident in the earliest installments
recent installments in this series so
this is some kind of uh
cartoon series that comes out in
installments um so you can look at this
and then we have a way to pull this up
in an interactive neuron explorer um so
and i think i didn't show you but
another view that we can that's been
useful to me is just other than install
what are the other tokens so that's how
i discovered that package is another
thing it sometimes
fires on so here we have an interactive
editor where i can try different
texts to try it out on um and i can plug
in the neuron over here on the side that
i'm looking at um and so we can just try
something out like this idea of like um
please open the debian package installer
like please use the yeah so that didn't
work at all um use the command line
um
there's no activation at all so let me
just show you have this activation view
you can look at the different um
tokens and if there was an activation it
would be more brightly colored if i just
make that just the word install
weekly so we get an activation of one
which is something it's something um
but if i go to one of these sort of
non-computer related uh context and say
the construction people right after the
worksite ready to install
if i finish the sentence and they'd love
that so that's an activation of almost
13. uh so order magnitude higher on the
on the spectrum and then these words
like package that are not its preferred
word it's a little harder to evoke an
activation i definitely won't get it if
i say something like you know
uh download the file for
the
package
that's i'm guessing going to be nothing
at all yep that's
negative so it doesn't clear the
positivity threshold but if i tell some
story about like uh
grebby's guests
brought numerous
presents to her birthday party
she gazed upon a golden
rap package i bet i'm gonna get a little
activation there yeah and so i get a
little bit of response to this uh
similar word but also with this sort of
outside the computer context so just a
quick note about the the sort of the
magnitudes that catherine is is
highlighting here sort of like you know
why
why you know this is 1.2 we color that
as very pale what's going on there
in general it's sort of very hard to
know what the magnitudes of these
activations mean in any absolute terms
because they'll only ever be
interpreted by being multiplied through
some weights and so kind of are they big
well like that depends what weights
they're multiplying by so we scale these
neurons by the largest activation we've
ever seen that neuron fire on in the
data set sort of under the assumption
that we don't know what these scales
really mean but any individual neuron
probably has some scale and so if we
scale it against its biggest ever
activation that at least gives us kind
of some sense of of you know what's
what's big for that neuron versus for
other neurons may be different
yeah
i also want to point out i didn't know
this was going to happen but i got a
tiny activation for the word rapt which
i actually haven't seen before so uh on
the fly learning learning new things
about neurons
anyway
catherine um just before you do that
uh i wonder if you want to say anything
about
uh
the sorry i'm also trying to figure out
at the same time how to how to take back
um the presentation mode but um i feel
like this has sort of been a recurring
theme and motif of these neurons that
sort of appear like they're they're sort
of like
x but context says that it's not x or
something like this um and so maybe if
you just want to i'll talk about that
motif for a second
um yeah happy here i mean maybe i should
jump to one of the one of the later
neurons that has this as well or do you
want to sort of talk through the later
neurons and then i'll pick up the thing
yeah yeah i think just uh just like you
reflecting this now so that when we come
to other examples um we can sort of
refer back to it yeah so this is just a
motif we've seen that will come up in
later in this presentation um
is neurons that are sort of saying well
you might expect this word but actually
it isn't or you might expect that this
is a computer meaning but the rest of
the context suggests that it's not so
there's kind of a you might expect but
actually
type of uh interpretation of the
activation pattern we're seeing uh and
it's been quite common it's cropped up a
number of different times where uh the
neuron doesn't seem to be best explained
is expressing just this thing is present
but it's better explained as expressing
you might expect this thing to be true
or to happen next from everything else
you've seen but actually something else
is going on that kind of motif that i
think of as a somewhat either inhibitory
or sort of push-pull kind of motif is
pretty common
cool
um
so moving a little bit deeper into the
model to layer tens that's about 25
depth um i think another interesting
type of neuron that we started to
observe more of were neurons related to
copying and if you've watched our other
videos on
um the are studying attention heads we
know that attention heads are often um
really concerned with whether text is
copied and
looking for previous places that might
be copied from and trying to predict
what might come next and we have have
induction heads and stuff like this
um but there seem to be a lot of neurons
involved in this as well when you when
you switch to full um full transformer
models and and sort of looking for
different kinds of copying so you can
have uh we have one neuron that seems to
be sort of immediately repeated text so
here we have well
well well
um so well's repeated three times or
here we have and this one's a little bit
trickier but um there's a reviewer uh
reviewers
as reviewers and the second reviewers as
a sort of short-term copy as well and so
it's firing there
um but then we have also these longer
term ones where if you look at this
first paragraph
um
it's copied in the second paragraph here
uh and i think that this happens
sometimes um for instance like if an
email thread is in the training set uh
or a forum thread and people are quoting
you know earlier people on the forum um
or if you have a news article that does
like um
an excerpted quote to emphasize it
that's also in the main text and these
are all situations where you can have
have you know paragraphs or large chunks
of text repeated um and so having
neurons related to that maybe makes
sense
um and that another famous neuron that
we found in this in this layer was um
another one of these neurons that
catherine figured out um that really had
me stumped and this was a particularly
interesting neuron because as i recall
um
it was we were looking at the the most
the neurons that had the that were both
extremely sparse and had extremely high
magnitude activations and this was the
the neuron in that layer that scored
highest on that metrics that was why we
were looking at it and when i started
looking at it i was really stumped
and it sort of feels obvious in
retrospect but yeah catherine you should
just tell us the story of the star
yeah happy too um so this is another one
where the um interactive uh
interface was was very helpful because
it's seen i was seeing it showing up in
um
sort of the
index kind of uh mode so let me just
grab if you don't know my
again
um
here i have the um
dataset examples pulled up
um
wait for that to start
um
cool i think we're are we good yeah so
um i was seeing it show up in the sort
of like
appendix
uh you know index at the end of uh of a
text kind of thing or it's like a table
of contents um so you might think it's
just tables of contents here's like you
know self-positioning 204 so
is it just somehow lists of stuff
but then if you start playing with these
um and you just start making lists of
stuff it doesn't always react so let's
see this got this is 10
11 5 8 6. so
um you know if i do this is a random
list of made-up surnames like um
darby dodson
edwards franklin
i'm kind of making this up yeah it's
starting to fire here but if i did darby
franklin
i don't know random
um
names it's starting to fire a little
here because i've started to go
alphabetically so that was the sort of
genesis of this hypothesis that it's
it's alphabetized lists and if we go
back to these dataset examples all of
these are our alphabetized lists and the
more you know it starts to um it's a
little fuzzy it's not like you know it
doesn't have access to the characters
right so it has some sense of
alphabetical ordering it's not perfect
so you can again fool it or get it to
mess up with respect to the story we're
telling about it um
by
making lists that aren't actually quite
alphabetical but it kind of thinks they
are and you can see it's sort of trying
to figure out like okay is the is the
word summer school the next entry in the
list maybe maybe this is a list that
goes particle summer symmetry no no and
then it gets back on the on the train
it's like oh no this goes particle and
then there's a couple other lines and
then particles and so you can sort of
watch it figure out or try to come up
with an interpretation of like is this a
list and if so
is this the alphabetized next entry and
have this kind of um
smooth prediction of whether it should
be considering this the next item in the
alphabetized list
i feel like this was a place where
um
the having the interactive interface for
exploring this was just uh really really
crucial and uh seemed to to really be
the difference between us uh getting
this i think especially like approaching
in a sort of adversarial way and trying
to
trying to really see
um
if it if it holds up when you go and you
uh play with it or or not and seemed
like it was important
yeah so i mean here's an example where
it's like starting to get on but it
doesn't seem to think that dates comes
before donuts so
i think another thing that i
have been grappling with especially with
this neuron is that
the kind of work that we're doing is
coming up with
stories
explanations hypotheses for how we might
in a compressed way describe the
selectivity of the neuron the fact of
the matter is that like the only ground
truth model of what the neuron is
selective for is the neuron itself so
you know if the neuron doesn't fire on
dates that's the real truth of the
matter of what this neuron is actually
doing and the fact that i think that's a
mistake because i think dates comes
before donuts
and after coconut um that's you know
whether you can say that the neuron is
messing up relative to this simple rule
that otherwise explains its behavior or
whether you can say like the rule is
just inaccurate actually it's doing
something more complex and more
sophisticated it's actually kind of a
philosophically
challenging uh question as to which of
these views is more useful i guess for
what we're trying to do
one other thing that i feel like was
pretty helpful with this neuron was
initially we weren't formatting the
white space in the text correctly uh and
uh and i think we also had we were
collecting our data examples over that
many that many examples or that many
that made that larger fraction of the
data set
um and improving that i think made it
easier just to go and see patterns and
data set examples and so i think
investing in infrastructure has really
has really helped with this kind of
thing
yeah this um this view uh previously
would have just been the the white space
was not correct and so it would have
just been a whole pile of words which
was much more challenging
okay steal back presentation go for it
i guess and maybe just actually one
other quick context while quick comment
while chris is stealing and sort of
catherine's mention of sort of sometimes
you come up with these rules that are
like nearly right but imperfectly
predictive
um that reminds me a lot of my
experience taking my first ever
linguistics class um where there's like
a lot of sort of pop knowledge about
like think about rules in english
language or or places where there are
seeming irregularities or irregularities
and in a lot of cases the like pop rule
is like decent or the pop pop culture
rule is like yeah it's just arbitrary
but then linguists have like gone much
deeper and they're like actually
this rule that you thought was about the
spelling of the word is about the
phonetics of the word and if you like
look at the phonetics everything is
totally regular it's just that the
spelling has been corrupted because
english spelling has such a weird and
arbitrary history
um and that can be a kind of similar
story of like if i have a rule and it's
almost right it's often worth
it's it's all
there's a natural temptation to be like
ah the model is trying to do x badly um
and it's worth thinking hard of like
actually it's trying to do x prime and
it's doing it quite well
um that's right both
right both both stories can be true in
different cases but it's almost always
worth
like
i i don't know i i find it's
there's some instinct of like oh it's
just a model it's like trying to do
something but it's bad at it but these
models are very capable and it's often
worth like trying harder to find the
real rule
um and then i think the other
intersection is that
i remember chris mentioning like going
off down a total linguistics wikipedia
rabbit hole and then coming back with
half a dozen new hypotheses for various
neurons or attention heads
and it's been interesting to note that
like in some cases the sort of innards
of these models are like
modeling language in some ways like
better than sort of i consciously
understood language i clearly
unconsciously understand the language
like quite well but you're very good at
it you're using it right now right
there's like a whole set of rules that
the linguists have figured out that like
i didn't learn or forgot in my one
linguist linguistics 101 class
um and that's actually been a like
surprise so in some cases a surprisingly
rich source of like hypothesis
generation
i i think that related thing is just
you i think people
i think sometimes underappreciated how
much of the the challenge of
understanding these models can be that
we just don't have the right hypotheses
um because i think i i i often see work
where people sort of like um you know
they go and search for some features
that they think are going to exist in
the model and see if they can find
neurons or directions or something that
are predictive of them um but i think
often uh like i would never have guessed
the install neuron thing where it's like
install but not in the context that you
sort of might have thought by default
and uh i think i think that's actually a
pretty common recurring theme that these
uh
that these models are
hard yeah that that a lot of things that
are going on these models are not what
your original hypotheses are
i want to just riff on what nelson was
saying about the discovery process where
it's tempting to say oh this neuron is
kind of doing short phrases of this type
got it got it okay and move on um but in
my experience with the install neuron
and the alphabetical listener on with
the first two where i really felt this
um
there's a kind of click there's a kind
of aha where you stumble on the exact
right hypothesis and suddenly everything
is very clear and all the weird quirks
of like but why aren't these very
computer-y um just instantly snap into
clarity um i have some experience a
couple years ago you know
for a phase in my life i was a mit
mystery hunt uh competitor and you know
you get together with a team big team of
dozens of people and you're trying to
understand the meaning of a random pile
of letters
uh and at some point you have the aha
and you have the hypothesis of what the
puzzle constructor uh meant you to to
see and everything makes sense and that
click or that aha um has been really
important for for knowing that we
actually we actually got this one we're
not just kind of squinting at and being
like oh it's kind of animals okay
done like that's usually not uh not
quite good enough
um okay so
we can also jump to the end and i think
the last layer is somewhere we've had a
lot of success as well
and the reason for that i think is uh
the last layer has the mlp neurons in
the last layer have a unique property
that they linearly affect the logits and
that is the only thing they can do
because they just get added they they go
through a their down projection that
gets added to the residual stream and
that immediately hits the unembedding um
i guess there's guess there's a layer in
on there so maybe it's very very
slightly non-linear but it's it's
effectively just a linear effect on the
on the logic sun and they can't go and
affect other neurons they can't be doing
and affecting things through retention
hence they can just linearly affect it
and so um you might have in some cases
have trouble understanding
when the neuron fires but you can
absolutely understand um what the neuron
does when it fires uh catherine i think
you analogize these sometimes to motor
neurons
yeah yeah um i i in a in a
half of a phd that i then quit i did a
bunch of neurosciences so the um
uh sensory neurons motor neurons analogy
tends to make a lot of sense for me
where um computational neuroscience has
had the most success more on the sensory
side or more on the motor side of
analyzing sort of the behaviors and
selectivities of neurons or clusters of
neurons i think this is a similar case
because we know what the output is then
we can sort of directly track the
contribution of these neurons behaviors
to the output which gives us a grounding
that you know in these 40 layer models
when i was trying to understand layer 20
i just had no footholds at all i had no
way of telling
what the right
frame of reference was but for last
layer neurons the correct frame of
reference has to be very close to
what next token is emitted or predicted
uh by the model of what a
slate of anticipations does it generate
that has to be just structurally uh
close to the right way to think about
them
and this is sort of strictly true for
the last layer but sort of a softer
version of this might be true for the
last couple of layers um so i think
we've had that yeah i can show a couple
in the yeah i mentioned earlier that
with this sort of inhibitory motif some
of the um some of the ones near the last
layer i think also interesting if
there's a good point for me to
show a couple of those too
awesome and i think i think we've
actually found sort of in you know the
layers for neurons in the middle of the
model our usual assumption is that sort
of whatever right every uh you know
because of the the kind of residual
nature of transformers every neuron is
added directly into this residual stream
and then that value can either interact
with later neurons or retention heads
but also that value is kind of there
forever someone might subtract it out
but you know then the sort of that and
so so every neuron you can look at what
its direct effect on the logits is
and i think we've found that often even
if that direct effect is a small store a
small part of the story it's sort of in
some way related to what it's doing sort
of something about the training process
or the sort of way these models build is
that it will affect tokens that are kind
of in some way related to what it's
doing so even for
middle layer neurons where that's not at
all it was potentially not even
mechanically a large part of the story
there's still sometimes information or
hints there about ah this neuron makes
those tokens more likely that you know
that helps me generate hypotheses
yeah that's a great point and yeah and
in general it's just a very nice thing
to be able to go and quickly look at um
and and have available because it's um
and uh you know people who are watching
our other videos know that we've been
doing this a lot for our attention heads
as well you can really anytime something
writes directly to the residual stream
you can you can be asking um you know
what the long term effects of it are
um and this this relates to
um i guess there was a user nostalgia
rest who wrote a a post on on the logit
lens interpretation of of transformers
this is a little bit in that in that
vein
um
but yeah so here we have one neuron and
uh
if you look at when it increases
uh tokens it increases uh they're always
all the tokens that it increases start
with a vowel and it decreases the
probability of tokens that don't start
with a vowel
um and often it fires on an an and so an
an is going to indicate that the the
next word must be um must start with a
vowel because otherwise you'd use an a
um there might be other cases where it
also fires but that all the really high
magnitude examples that i could find
when i was preparing this slide were
like that i don't think this is just
another great example where if you
hadn't seen the data set examples on the
left side and you're just looking at
this pile of like increased decreased
tokens you'd be like amalgam embroidery
is this about ostrich this is about like
feathers or decorations or what is you
know range combination pear like there's
there's also some additional like
semantic
thing that might be part of the story um
but you could also miss this sort of
high order bit
is uh the vowels versus versus
consonants like if i look at these lists
i do get a sense that there's something
about rich textured nouns versus
mathematical
nouns but that's not the high order bit
of of what's going on and as soon as you
see those uh activations on and you
understand that the primary thing to
understand here is the is the vowel
consonant
yeah i think i think it's sort of
it also lends sort of
sheds light on sort of on when you're
trying to understand or interpret these
neurons it's
it's important to have a sense of where
they are in the model and sort of how
what that relates to your space of
hypotheses because you know
if you just look at these data set
examples it'd be very easy to guess you
know this just fires anytime it sees the
word and but that hypothesis makes no
sense because
that the model could do that a the model
could do that way earlier on and in
general we tend to assume that these
models are like well optimized and
reasonably efficient and they're sort of
nut they're not going to be spending too
much of their capacity doing things that
are sort of obviously useless or that
they could have done earlier
um and so there's like it doesn't make
sense for it to be doing something so
trivial this late in the model but also
there's no computation after this neuron
so like
you know ah this is the word and like
that that fact doesn't really
like that that fact is it's too late to
know that fact
um and so it is the case that this
neuron very often fires on that token
and very often fires on others but sort
of the the logic effects tell you this
the kind of the much better story and
give you a much better story that fits
into where the where like where the
neuron is in the model and so the space
of things that like are sort of
informative to talk about about this
neuron
i think this also relates to why it's
useful to invest in understanding the
architecture of these models
um
i think
uh yeah i was having a conversation with
somebody else the other day and i
yeah i think i think it's really
valuable to invest
before you go and spend a lot of time
poking around at these models to invest
in understanding what kinds of features
exist at different layers and or what
the what the architecture of the model
is and and how things are wired up such
that you know you like the observation
that the only thing that an mlp layer
can do is is going to affect the logits
is something that you can get when
you've when you've invested in that so
chris can i grab uh presenting for a
moment because i just uh you know
discussing um
yeah go let me just grab and then i'll
and then i'll although i think we we i
think we're almost at the end so we will
go through this and then either way
you have to you're
very quick just as nelson was saying
like this this neuron is too late for it
to be firing on an if it's actually just
predicting when there's going to be a
consonant i can make this whole string
with just you know context metal
learning where the word or is always
followed by something that starts with
the vowel and sure enough it picks it up
and now it knows that it should be up
weighting the the vowel neurons so i
just threw this together right now just
just there
that's phenomenal there you go
back to you
thank you well um okay next tip
i really need to learn this trick of how
to resume
presentation uh
how do i
there we go resume presenting
um
okay so oh yeah so there's
we're in fact not quite at the end um so
what about neurons in other layers that
we found harder to understand
um so one layer in particular that we
found kind of tricky to understand um
and perhaps that you'd sort of think
would be the easiest layer to understand
of all of them has been the first mlp's
an mlp layer
and
part of the reason for that well i guess
these are there's two things that are
very striking at this layer one is that
the attention heads immediately before
it which are the only thing that neurons
can be computed based on um along with
residual strain are extremely diffuse
rather in large models system true and
small models those models get larger
those attention heads just start being
smeared out over basically all previous
tokens
and the neurons become exceptionally
sparse
um
and
yeah and so in theory you could you
could actually like these neurons have a
very nice form where you could just sort
of uh mechanistically understand them
you could expand them back and you could
be like well you know this is what it's
looking for through this attention head
and this is what it's looking for
through this attention head but because
the attention ends are these sort of
extremely blurred things that's been a
little a little tricky to reason about
um and a lot of your a lot of the
obvious hypotheses you might have for
this layer don't seem to be true so i
think the thing that i i first imagined
the first layer would be doing when i
when i started looking at it is piecing
together um subword tokens into whole
words so when a word gets split into two
tokens it would be very natural to have
neurons that try to go and fix that and
turn them into
uh into coherent words and that doesn't
seem to be what the first layer is doing
or at least not a large fraction of it
and
something that really
helped me think about this is there's um
this paper from google pair by conor
doll
in 2019 and they they have this really
nice diagram
um
where they take the activations and
these are i guess residual stream
activations um for a single token like
um dye
and then they go and they just do
a tc or a umap
and they find that there's one cluster
that corresponds to the german article
d
and another one that
relates to people dying
and another one that relates to dice
and those are all different
interpretations of the token d
or die
and it made me wonder if maybe something
similar might be going on because you
know a very sort of an attention that's
blurred out and looking at all previous
tokens might be actually pretty well
positioned to help you guess what
language you're looking at and so maybe
you could go
and
correct your interpretation of tokens
and
it seems like that might be what's going
on so if you um either do like a pca of
the nlp
the first mlp layers activations for a
single token
or you do a umap and you'll notice that
uh you get clusters corresponding to
languages and sometimes even to like
particular types of of speech within
that language
chris i'm not sure i ever called it yeah
yeah it's like um
just
there's there's some
some sentences about death or some
samples about death that are like from a
novel where somebody's like speaking
very emotionally about nothing yeah
lamenting or or or it's just like kind
of i don't know kind of intense language
in some way um and very personal and
there's some that's like much more
depersonalized
yeah statistical or clinical or
um and yeah the that seems to be the
sort of primary dimension of that
cluster over there
so
um so based on that you can then start
looking at neurons in mlp zero
and
uh here's one for instance that seems
like it might be related to it only
fires in contexts that are spanish
but the tokens it doesn't fire on every
token that's spanish
um and in particular it seems like it
may be tote fires on tokens that could
be english tokens um or maybe could be
tokens in other languages and this is a
little bit cherry picked because i
wanted to find a couple interesting
examples and there's there's some places
where it fires on tokens where it's it's
less clear this could be the case
but for instance um the token fund um
well fund is an english word as well um
like funds and funds you could also have
fund isn't fundamental
um in is a standalone english word and
also is used in uh lots of other
contexts europe is an english word
in-depth could be part of indefinite
um import is an english word
sin is an english word
um and a lot of these other things you
could imagine that maybe there's some
english word that can be tokenized such
that it's at some portion of them
and
and so there seem to be a lot of neurons
at least a fair number of neurons in mlp
0 that
seem seem like they could be interpreted
in this way where they they are words
that are disproportionately yeah seem
like they could be in one language but
context makes them seem like another and
that's one
sort of inhibitory pattern like you
might think exactly
but actually it's spanish so don't be
fooled the context says something
different than the token itself and
there seemed to be a lot of these for
different languages um and different
pairs of languages um and in mlp zero
again i think this is still sort of
relatively yeah not not as pinned down
as some other things that we studied um
but it seems like that might be might be
what's going on
yeah just one thing i'm noticing for the
first time staring at this example is
like on your last line there
the word embargo is also a straight up
english word and it doesn't fire on that
so it's sort of quite clear that we
don't fully understand it but it's well
that's like that's something it is it is
it's sort of it clearly has that flavor
but there's there's there's work to be
done to try to and i think again this
you know this gets to the like
is the model doing x badly or is it
doing x prime which we don't yet
understand well i think our guess is
that if we find the right explanation it
will as catherine said kind of click
into place
and but we're still figuring it out and
kind of it will i think likely be the
case that we sort of don't solve this
one but then we'll have some later
revelation about some other neuron and
then we go back and look at this one and
we're like ah yes it was clear all along
yeah this is really in the phase of like
it seems to be doing something like the
following question mark as opposed to
like we got it and i think that is
partly because we have some of the
positive story we don't have some of the
negative stories we don't know when is
it something that is an english token or
word but it nonetheless didn't fire uh
what's the story with that and i think i
have nothing to offer for this guy in
that in that regime
one thing that i might be tempted to
suggest is it could be about just the
relative probability of these tokens in
english and spanish and so embargo might
uh might be more common i don't know i
don't i don't think
that would be an interesting thing to
overlay it is more common in spanish
um
i thought one final thing maybe to just
talk about is what have been the
challenges to this kind of investigation
so i think having good tools i think has
been really important i think that's
probably a fairly uh yeah fairly
straightforward one
um i think people yeah i think this
challenge of not having the right
hypotheses to distinguish between like
um i think we we often think about
hypothesis-driven science where you're
like trying you have two theories that
could be true and you're like trying to
distinguish between them but i think we
just have this like enormous base of
hypotheses that we can't enumerate and
we don't know what the right ones might
be and that's much more often the regime
that we're operating and i think i think
that's tricky and then there's this this
eot token issue and nelson i wonder if
you want to just briefly comment on that
yeah
um
this might be worth a whole video of its
own at some point
but maybe the two-minute version yeah
it's sort of it's hard to know there's a
whole thread here and it's sort of hard
to know how much is a is a quirk of some
details of of our architecture but
we
we when we train models we always give
them this leading token that we
confusingly call eot for end of text
even though it's always the first token
in the text
um but we always give this first token
which we sort of find is is useful for
attention heads to attend to among other
reasons
um
the eot token if you look at its
activations in the residual stream is
kind of very unique nothing looks
nothing it doesn't look like anything
else it has very large magnitudes we
sort of expect it to be weird because it
is a very weird token um
but we also find that uh because of a
artifact of how our retention mechanism
works
the model tends to replicate this eot
token so that
other random tokens
gain very similar activations to eot at
sort of some
not exact but some periodicity through
the model and there's
not i think i don't think an enormous
amount of capacity but some number of
neurons and detention heads that are
seem to like mostly be dedicated to
constructing and then tearing down when
it's time to make a prediction these
fake eot tokens
um and one of the salient things is that
these so-called fake eot tokens these
what we call fake eot tokens have very
large absolute magnitude of
values in the residual stream and
activations at some point
and as we mentioned earlier one
heuristic we sometimes use to search for
things is is looking for things that
that are sparse and have high magnitude
and so
those tokens just sometimes cause false
positives and they tend to be very
uninterpretable because
they're kind of we we believe them to be
this artifact of the model
architecture where it's trying to
preserve information that it otherwise
wouldn't have access to
and so you get these very large
activations that have functionally
nothing to do with the text that they're
attached to
and we spent a while on some red
herrings until we understood this
phenomenon because we were like these
things are so much larger than any other
activation like there has to be
something really important going on here
um if you look at certain statistics of
the sort of whole model these guys
dominate these guys are the whole story
um but then you look at them and they're
attached to
we actually understand a little bit now
about where they're attaching but
they're attached to functionally random
positions and like resist any
any easy attempt to understand what's
going on there which i think also speaks
to chris's point of
of if you don't have the right
hypotheses things
and or if you're like have the wrong
hypothesis and you're
intentionally are not too attached to it
it's sort of possible to just be
completely baffled
i will say i mean i thought it was
useful um you said sort of random tokens
or you know arbitrary tokens but one
discovery that uh seemed to sort of help
move towards that story or help fit into
that story is one common place we often
use the first paragraph of harry potter
uh as our demo text and there's the word
didn't and hadn't show up at two places
in this text and they're tokenized i
think uh d-i-d-n
as one token or h-a-d-n as one token and
then apostrophe t it's like a whole
token which the model can
very very accurately predict if you see
d-i-d-n the chance that the next thing
is apostrophe t is like through the roof
and so that's a token the apostrophe t
is a token that the model would use as
one of these fake eots because it sort
of has
as our story to put this other random
stuff there without throwing itself off
because it's prediction task is so easy
um that it knows what's coming next i
guess i stated that kind of it was it
was it's the token before the apostrophe
uh yeah thank you everyone
a little gloomy as i'm speaking because
i'm saying the wrong thing the d-i-d-n
right before the apostrophe t um is
where it piles this serious eot
information because our story is because
uh it can't throw itself off that badly
because its task at that point is so
easy yeah and this this turned out to be
a general story there's kind of a couple
of other patterns and this story seems
to be that it's always in a place where
the prediction where usually due to some
quirk of tokenization the prediction
task is like brain dead easy sort of the
model can tell very early on
that there's really only one thing that
it could ever possibly be
and then it's like great i i don't need
to perform any computation here i
already know the answer and so i can
reuse this capacity
for this scratch space to work around a
quirk a quirk of my architecture
um and i think that um that's kind of
one of the i think that might be the
reason why this might be worth the whole
video is i think the specific phenomenon
is is potentially just sort of
architecture quirk but i think it's it's
a it's an interesting view into the like
kinds of behaviors that these
architectures are capable of evolving
and sort of how they allocate capacity
to different tasks in different
positions
well i want to push back a little bit on
this being i think this is an arbitrary
quirk but i think it's an arbitrary
quirk that probably a lot of people at
different labs have in their models
right now um and so i think in
particular if you have a model that is
using block sparse um and you are seeing
really weird activations um this is a
pretty live hypothesis that you should
have and i i guess that if you're if
you're if you're trying to study large
models there's like a 50 chance you're
using block sparks or yeah
i think we've also seen evidence of it
in
in
dense in
models where that have partially dense
and partially local attention and i feel
like
i suspect every large model is using
some combination of those you know one
or both of those strategies and so yeah
it may be
it's a kind of quirk but yeah you're
right i think i think it is a quirk that
is probably a wide enough class that
that many other transformers have it
universal quirk
one more comment about infrastructure
yeah i think our i think the the only
thing we have left is is there anything
else we want to talk about so well if
you go back to the previous slide i just
wanted to remark on that on that first
line you had um which is the the the
infrastructure the tooling um is is
pretty pretty crucial so you know
throughout this video i've pulled up
this interactive text box where i'm
typing stuff and trying things out and i
built that before nelson joined and
built garcon and if you want to learn
more about garcon there's a previous
video on it which is fantastic but um
you know when i first built this
interactive text box i would just spin
up a python you know collab notebook and
put a model in the notebook and then run
a little flask server in the notebook
and then pull up a separate you know
page to then be speaking to my notebook
and that was it's a little clunky um
it's not great whereas garcon allows us
to put these uh models on our cluster
and so now i just have a sort of static
website that's um
sending these texts that i'm typing to
the model in the cluster and i don't
have to be hosting it on my own dev box
and it's a lot easier um it's been
fantastic and so thanks to you know i'm
glad nelson joined and maybe i'll just
make a little pitch that if you're an
engineer and you want to work on this
kind of stuff we are hiring engineers
and it will make our lives a lot better
in these exact ways so think about it
well
uh thanks everyone for joining us um
catherine nelson do you have any last
things you want to to add in
i just want to say i would be excited
about more people out in the world doing
this kind of work it's fun it's very
much sort of like
exploratory kind of science i think
machine learning can get really bogged
down in make the number go up you know
make a state of the art whatever to
classify whatever and that's fun but
this is also fun it's different kind of
fun it's a different kind of science and
i think it's a kind of science that's
lacking in the machine learning
landscape um and i would love to see
more folks doing this and as i said the
um
having the right tools and
infrastructure behind you is a large
part of it and i you know built my first
javascript interface a few months ago
when i joined it's not that hard to
learn so even if you think that you're
not
this kind of person if you can if you
can code and you can learn new stuff um
this kind of work might be more
accessible than you think and it's fun
yeah and i think on on that note i think
there's a paper i think that i want to
give a shout out to there it came out i
think late last year was uh transformer
feedforward layers are key value
memories
um i think we haven't actually
been kind of directly influenced by that
i think we sort of read it closely
midway through this work for the first
time which is maybe remiss or at least i
i came to it sort of after doing this
work but among other features it
contains uh they did some kind of
similar work of sampling a bunch of
neurons farming them out to people in
their labs forming
these kind of hypotheses of what's going
on and they document a bit about what
they learned about the different kinds
of neurons in different layers and have
some cute examples um a sort of a very
similar kind of work to what we're
talking about here they were working on
a somewhat smaller model but that's
that's the one other instance i'm aware
of
very similar work being published and so
i thought it was worth a shout out
awesome
well uh yeah thanks thanks everyone who
joined us and thank you uh catherine and
nelson for taking time to chat about
this
yeah thanks chris