EleutherAI Interpretability Reading Group 220226: Locating and editing factual knowledge in GPT

a bunch of the experiments for us as
well
so
so what i'm going to do is i'm going to
um
talk a little bit about the um
sort of the motivation of the paper sort
of and then and then i'm going to turn
it over to kevin to talk about some of
the experiments and maybe kevin and i
will uh we'll switch it off
um
for that part of the presentation
and um
and i i guess i
i'd like to see
i'm i'm new to discord but i'd like to
see if we can try to keep it interactive
uh if people can just sort of
you know interrupt me with questions or
you know or or or if there's something
you feel like we're not going to talk
about then we can
talk about that skip ahead to it or
something like that um we probably have
a lot more to talk about than we have
time
and so uh so yeah so it'll be better if
it's interactive because then it'll make
sure that we talk about things that
people are actually interested in does
that make sense
that sounds great we usually get a lot
of great questions and stuff so not to
worry
excellent
excellent excellent excellent okay great
okay so here's here i'll i'll dive in
now
okay so the project is called we call it
looking rome
any factual knowledge in gpt it's uh is
an archive there's a little project
website you can see all about it click
through that uh we have different
materials you can play with while we're
going through the talk
um
so the um uh you know so the the problem
is and it's an interpretability project
uh the question that really drives the
project is the simple one
um which is
uh which is this drive to understand
what it is that a neural network knows
what does my neural network know and how
does it know it
now it's sort of a funny question it
might seem vague
um but it's actually a pretty important
question that is getting more important
um and it's really becoming more
important because
of the rise of unsupervised
uh unsupervised training
you know it used to be that ai models
really only learn what you taught them
and carefully labeled
but now that unsupervised um learning is
starting to really work
uh we're starting to get ai that is
learning things that we didn't really
explicitly teach them and probably the
most striking example of that
is uh the auto aggressive uh
transformers like like gpt models
so you know as you all know um
uh you know transformers invented by
ashish foswani at google a few years ago
and um and they're really good at
imitating real text uh there are a lot
of good open source implementations
including uh your your guys work uh to
program them you just show it some text
uh like this so here we've grabbed 40
gigabytes of web text you know 27
million pages you train it uh just by um
yeah exposing the model to this text and
uh using standard optimization
optimization methods you can use the gpu
to speed up the training process
it still might take a few weeks to get
it done but in the end
you get this marvelous object a trained
network
uh here i'm showing you gpt2 which was
trained exactly with this type of data
by alex radford at open ai a couple
years ago
and it has about you know this model had
about 1.7 billion parameters um
and so it's a big model but it's
conceptually
simple right it's just a function
as input you just give it a series of
words it does a bunch of arithmetic and
as output it produces a prediction right
a guess of what the next word would be
so here in this example here i'm showing
here the input is
edmund
newpert performing on the
and then as output uh the word that gpt
uh predicts should come next is the word
piano and i'll put a little green check
mark uh next to the word piano here
because actually edmund newport was
a famous 19th century pianist and
composer
so in a sense this this is really the
right
word to put here and and it makes you
think oh maybe gpt knows some things
right and and
you see a lot of these amazing
uh sort of predictions come out of the
model here's a few more examples of
musicians
if you say miles davis plays the um you
know in purple we'll have the prediction
trumpet
uh i don't know if you've ever heard of
nicolo pagianini
but it's true he was a master violinist
uh in the 19th century
and uh jimi hendrix a little bit more
contemporary right he is a virtual
surround the guitar that is also true so
there's all these things
that are remarkable because even though
what the model is doing is it's just
sort of producing uh predictions of the
next word
they seem to capture
a thing that we think of
um as like abstract knowledge
right so you know for example there's
all these uh knowledge graphs
where people have tried to catalog you
know all the factual knowledge on the
planet
and and they're generally expressed in
to pulse
relationships between a subject and
object through some sort of relationship
like so sort of like
a fact tuple for that fact at the top
would be edmond newpert
is related to the
object piano
through the relation plays the
instrument
right and somehow you know you wonder
does does the model actually know this
fact
um
now as as you can see there's actually a
lot of ways of saying the same fact
um and so but you could use any of those
ways to to probe the model's knowledge
but then there's this there is this
funny gap between
knowing something and just saying it
right and so let's let's do this so this
is actually an exercise that uh elazer
did uh last year which is to probe a
fact using a lot of paraphrases so
here's edmund newport performing on the
a virtual so anna is known as the master
of the favorite genre so
if you look at these completions now you
start to realize well
maybe gpt is not
so smart right if you use different
wording instead of saying that he is a
virtuoso on the piano which would be
correct here it's saying oh he's a
virtuoso in the violin
and if you look at the last thing
his favorite genre is like the horror
like
the horror genre or something like that
like does it think that he's a
maybe an author or a movie maker or
something like that it's not clear that
the model actually knows who edmund
newport is
whatsoever once you start looking at
these paraphrases they're all
inconsistent with each other
right so it gives you a little bit of uh
gives you a little bit of caution makes
you think that you we should be humble
about what these um what we think these
models know when we look at these
paraphrases but this is not always like
this so for example if i plug in nicolo
paganini which is another 19th century
musician into here it's quite consistent
right using all these uh completions uh
he he pretty consistently plays the
violin
and you know his favorite genre is the
symphony which is also consistent with
that
and so it really
it indicates there's there might be
something going on here but there's
there's a gap between knowing and saying
right you can know something
without saying it you can say something
without actually knowing it which is the
edmund newport example and you can also
know something
without saying it right like you can
know that nikola paganini was a
violinist and and that that fact may
came out come out in a sentence that
doesn't mention the violin
at all it just happens to be uh
consistent with that so so this is a
pretty interesting
thing and this this knowledge this
knowledge is different from saying is
the thing that we're really interested
in
um this sort of abstract
factual knowledge and so we sort of
begin with this question what is
knowledge
uh in a network and and there's a few
specific questions we're gonna drive at
in this paper one is
you know can we locate knowledge can we
change it can we measure it right we're
basically on a hunt for the elementary
units of knowledge in a network right is
is there a physical reality
to factual knowledge within one of these
large networks um as distinct from its
you know
computation of just predicting the next
word is knowing is different from saying
does knowing correspond to some
computation that we can pin down
okay so what i'm going to do
is we'll divide the talk into these sort
of three
topics
and and we'll talk about specific
experiments and so on but
um but you guys should like interact
with us and
uh ask about specific things i'm gonna
turn it over to kevin here to talk about
uh the first set of experiments kevin do
you want me to keep on
presenting
uh and and i'll say next okay i just uh
i just started sharing my screen oh you
did okay great so i'm gonna
uh i'll i'll stop that and then can
people see yours how does this work
okay
okay great
all right cool well thanks david
um so the first question locating
knowledge
um so our goal here is actually pretty
simple all things considered to debug
this neural network it's almost like
debugging a piece of software where
piece by piece we sort of strip away the
parts that don't change the phenomenon
that we're sort of interested in
until we hit what we're looking for
um so here gpt's built
um
sort of what we can interpret as
understanding of some facts about the
world
and to do this it's clearly built some
kind of representation of the entity
that contains information that's somehow
looked up consistently across lots of
different contexts so this is what's
interesting to us
and one of the things we're interested
in is are any of these different
computations that the network does
actually decisive
now a priority there's no real reason to
believe that any one of these
computations is more important than the
other it may be the case that gbt
distributes its you know its computation
that determines that milestone is placed
in trumpet it's distributed across all
the different 48 layers all the
different tokens all the different
components
um and this would be okay
but we're sort of interested in the case
where what if this wasn't true what if
there was some specific computation that
was decisive or more important than the
other ones in actually looking at this
fact
um
so that's our question
and the way that we're going to figure
this out the way that we're going to
debug this neural network so to say is
to use this thing that we call causal
tracing and the fundamental idea is to
copy a bunch of data around and see
what causes the correct output and what
causes an incorrect output so
let's figure out let's talk about what
that means
um so here's what we do
we run the network twice the first time
there is nothing interesting about the
way that we run the network we simply
put miles davis plays the and we let the
network carry out its typical
computation
um but the second time
we corrupt the input to sort of
frustrate the computation now in this
case we're corrupting the embeddings of
the subject so the network no longer
knows or it sort of has a much weaker
idea of who we're asking about it still
knows that we're asking for whatever
this entity or the subject plays
but because it doesn't really know who
we're talking about
we have a corrupted token at the very
end it's not going to give us the
correct output
and
what we do is
well we ask we're curious what sort of
computation of the network made the
original
network run predict trumpet and we're
interested whether there was some
decisive state
so what we can do okay sorry
yeah you've probably looked at this but
is it always the case that the corrupted
sentence always gets the wrong answer
you might imagine that the structure of
the sentence might allow the model to
guess the right answer in some cases
yeah so listen
yeah so i think david can shed more
light on this but i think that's
generally the case you're definitely
right and we'll sort of probably talk
about this later which is that we do
observe cases where for example
this the network doesn't actually know
who miles davis or whatever subject
we're asking about is and basically from
the structure of the sentence it can
guess that we're asking about an
instrument and it just happens to guess
correctly that the that the you know the
correct answer
um
but
but that's pretty rare
um
yeah that's right it's pretty pretty
rare
um
cool
um so yeah we have sort of a good run a
corrupted run and to figure out which
one of these states is decisive what we
do is we copy over a single state from
the original run to the second run
and so what this means sort of
in implementation is that we have this
bad run at the very bottom where it's
it's got a corrupted subject the network
doesn't really know who we're talking
about
and we take one of these hidden states
at sort of we have two axes here one is
on the token level and the other is at
the layer level at this specific token
and at this specific layer there's some
output hidden vector and all we do is we
copy it over to the corrected
computation and we change nothing else
about the corrected computation
so at this moment where we have the um
have this hidden vector it's got the
correct output from the normal run the
uncorrupted run
and then afterwards we don't change the
computation at all
and usually when we do this when we do
this copy and continue operation there's
no effect the output is still incorrect
but
after we search over the space of all
different places where you could perform
this intervention over all the tokens
over all the layers
we can actually find states that cause
the prediction to change much more
appreciably
so for example in this case if for you
know for the sake of argument imagine
that this vector is sort of a decisive
vector um if we copy this state over and
we let the output continue this somehow
this information is propagated to the
very end
where the prediction is made that this
person or this corrupted entity plays
trumpet
so what this suggests to us is that
there are some certain vectors that
contain more information than others
regarding the information that we're
trying to look up
and so this is interesting to us
so if we run this test systematically
over all the different 48 layers in gpd
to excel but i think there was a there
was a question
oh sorry
yeah just to clarify so in this case are
we grabbing the entire state uh for like
one time step or in one token
um or are we like editing a subset of
that or something
yeah that's a good question um in this
case we are simply just grabbing the
entire vector so i think this is a 1600
dimensional vector gpt 2xl um it's
simply just all being copied over
are you checking whether the
like the probability of the word trumpet
changes are you checking
whether the probability is pretty much
the same as the original probability
distribution
so that's actually another good question
we haven't quantified this yet but on
the next slide we sort of will
um i think it's the absolute
it's the absolute probability after you
restore the vector
right so you would take the probability
of exactly the one correct solution not
of the entire opinion of the network
right
uh one more question here this makes
sense but have you also tried starting
with the clean network state and
corrupting individual uh hidden
representations instead um you might
imagine that being more decisive i'm not
sure if the results were just messy or
something else happened but
yeah
yeah
so that that that's sort of the natural
experiment that's the first one to try
and it interesting when you do that you
get this
very diffuse picture
like there's a lot of different ways of
causing damage
uh to the network
all over the place
and um
uh yeah it like it may be that there's
some way of calibrating that experiment
that's better than you know what we
tried initially but we couldn't get that
one to work couldn't get it to show us
anything insightful
it's yeah so you sort of you can you can
corrupt
you sort of corrupt anything
you can go to any location sort of
corrupt you corrupt it enough that you
know the network you know
can get confused
um
yeah
maybe i can add um
these two ways actually we haven't gone
into this in the paper but these two
ways correspond to
uh
the causal mediation analysis by judah
pearl you could view them you could cast
these two options as
measures of direct and indirect effects
um
and and and that that gives you some
kind of theory as you might what you
might expect and in the
in in the mediation analysis uh
literature
there's some work on decomposing these
effects
uh in different ways and in very simple
cases these two should add up to
uh to something known as the total
effect
uh of course
this doesn't have to happen in a
complicated model like
like gpt um but if people are interested
we have another previous work that did
something
kind of similar
uh with causal mediation analysis using
this whole terminology
so i think a lot of these interventions
can be cast in
causality terms from uh from the
caudality literature
kind of interesting yeah that's a really
cool connection thanks for bringing that
up
hey awesome
um
so yeah causal tracing um when we take
this intervention that we just described
and we run it over all the different
locations that we could run it at we see
this sort of pattern and we'll we'll
show averages in a second but in this
case we're working with a person who
works in the area of film so we can see
two
distinct states where these causal
effects are much higher than the rest of
this sort of surge space there's one at
the early token at an early turn at an
early token location and add an early
layer there's another one at the very
very last token at very late layers
so the presence of site b is actually
not too surprising to us because the way
these auto-aggressive models work that's
sort of where they're making their
prediction and it better be the case
that there is decisive information being
passed in there
um but what sort of surprised us was the
presence of site a something sort of
very far detached in the network at a
very identifiable location it's in fact
the last token of the subject when you
tokenize it
and so this is interesting to us um and
that's sort of what we want to
understand better
so another way that you can actually
look at causal tracing is perhaps you
don't have to add noise to make the
corruptions happen maybe we can just
copy
a representation from a different
subject
now this isn't really easy to do in sort
of a large scale fashion in mass because
you there are a lot of constraints for
example if you wanted to copy subject
tokens from jimi hendrix and replace
miles davises with them
um you'd have to find maybe for example
tokens that are you know subjects that
when tokenized are the same length or or
things like that but to demonstrate the
effect we can also try try this this
setup
where we have copying over jimi hendrix
and we have this kind of corrupted run
and jimi hendrix is the virtuoso on the
guitar
um and then we start copying over a
bunch of representations for miles davis
now
the the sort of intuitive the intuitive
thing to think here would be oh well
since jimi hendrix is a completely
different person maybe we would need to
copy over all of these hidden states for
the prediction actually to flip over to
guitar that would be the case if
knowledge lookups were all diffused were
all over the place were completely not
localized
but what we found was again there's this
pattern of having a certain hidden state
that when copied can flip the prediction
and this is just one hidden state out of
the many money that we have here
so you patch this in
the information's looked up and we get
the miles davis output even though we've
only copied over one vector and all of
the possible ones we could have copied
over
at the representations
but by
by replacing that vector haven't you
pretty much screened off the change that
you made to the input from the output
so
in a sense um when you have this factor
here
um
from so first of all the
the uh the computations that come
afterwards are not are sort of
are uh are you like you're correct these
are
affected these are effectively now they
look like miles davis because you have
this vector here
um but the ones that came before like in
the first token in the jimmy token they
are still the same and then when before
you copied over this vector those are
also the same okay so this calculation
tells you that there is no meaningful
information passing from the
first yellow node after hendrix to the
first yellow node of the play
sure that's sort of one way you could
look at it um
and you know we see a lot of these
different interesting effects sometimes
it's not specifically at the very last
token sometimes if there's a predictive
word for example let's say audible.com
or amazon.com it doesn't wait until dot
com to actually look up the information
it's at an earlier token but you're
right there's some sort of decisive
token where all the information is being
gathered and if you sort of cut the
computation off and you replace it with
the computation or the factor from
another subject then yes you sort of
effectively
you're oh wait is there theoretically
the possibility that they would pass
some information from jimmy directed to
place or are all the
nodes along which information could pass
of length one
oh sorry could you say that again
uh like could they pass theoretical
information from jimmy to place
or does it have to go through hendricks
by the architecture of the network
so the information in the in the
architecture
i i believe that it does have to pass
through hendrix i don't know no that's
not that's not true right so in for in
attention
uh you know the the hidden oh it could
it's all right
it can it content to all the previous
tokens right okay okay but if it doesn't
have to pass through then it is much
more impressive that like
intervening on hendrix does not
uh like fixes everything
yeah so this seems to signify that all
the information about a particular
subject passes through the last token of
that subject and not you know through
other side routes on the other tokens
before that last token
yeah so this this was a huge surprise
this was a huge surprise
yeah so um along this sort of thing we
can quantify this instead of using this
qualitative example um we have this
specific case where we can pad megan
repino's name with sort of a pad token
and we compare the way that we compare
the sports that they play so i think in
this case we are looking at the
probability of soccer of shaquille
o'neal who's a very famous basketball
player so the original the original
thing read out shaquille o'neal plays
the sport of basketball
but when we take megan rupino's name
and we take the the representations and
token by token layer by layer we sort of
copy them over
we again see this interesting
interesting pattern of course
b is less surprising to us but at the
very very last token of these two
people's names
we have this really strong causal effect
where if you copied over shaquille
o'neal now has an 80 probability of
being a basketball player
a soccer player rather than a basketball
player and all we've done is copied over
one hidden state
um so yeah as
as as the audience has mentioned i think
this is interesting the fact that the
network hasn't really
the fact that the network had an
opportunity to skip over this last token
and yet it didn't is interesting but i
guess intuitively it also sort of makes
sense in some cases you can't tell which
jimmy for example you're talking about
or which megan you're talking about
until you've seen all the context
um but it's cool that the network seems
to behave in the way that we expect
what happens if you corrupt plays as
well do you then also have to intervene
on place or is placed mostly unaffected
by what happens in neil
um so our causal interventions when we
do this
um when we do the complete corruption
and we restore plays the effect as we
can see is is sort of low
right yeah at this point
they tried this for each token in each
layer um exhaustively and the fact that
there's no purple around plays means
that they tried plays and it did not
rescue the output
oh okay sorry i thought they always
scored only the subject and then
displayed the states
oh yeah
got you um sure yeah the cleanliness of
the image is is somehow kind of
confusing but yes all of these are
actually copied over um and
the hidden states the the causal the
states with high causal effect are
um are surprisingly very concentrated in
these two locations
what do you make of the fact that you
find an effect throughout you know
several layers for that last subject
token
it's kind of odd that it's so
distributed
yes that's a really good question um and
so we have some experiments that come up
that try to make a little bit of sense
with it um but then there there's some
additional
uh there's there's some additional
things we'll talk about um but the tl dr
is
uh
actually we'll we'll yeah we'll get to
that in a second but that's a really
good question
okay
cool one question um
after you do this
causal intervention
have you guys looked at
like
you can have say you have a subject and
then you have you know plays soccer
plays basketball
um
can you then go and then try to
reference the subject again and see what
happens
like
does it now think that um
i forget the name of the soccer player
but the does it does it now think that
it's a soccer player uh instead of a
basketball player that that is currently
you know the relevant person
or is it specifically like
the association with soccer that has
changed
uh okay so let me let me repeat the
question to make sure i understood it
correctly so there's a sort of
bi-directional
sort of flow of knowledge you can either
remember that shaquille o'neal
specifically plays soccer or you can
change the definition of soccer
and essentially increase its signal
globally and just predict everyone to be
soccer
i think this is one interpretation of it
i think the second one possibly is you
mean that
the the meaning of soccer has sort of
morphed into basketball
and i just wanted to clarify which one
you meant
so i i guess maybe i misunderstood the
um
the the method but my understanding is
so there's a couple of places in the
network where we can swap the state
and that affects the
the resulting probability
of um
of the output is that
yep yep that's right so
so there's a couple different ways i
guess
you know if you were to draw like a
causal graph uh you know intuitively of
that uh there's a couple of different
ways you can you can do that you could
change the subject
um
you know that a different person plays a
different sport um
you could change the association in
general right uh
you know people
with you know soccer uh you you change
that association from soccer to
basketball um
so i'm wondering if there if you did any
sort of testing to try and see
what
that uh
what is the the scope of the
intervention
i see okay um that's actually another
great question
and when we get to the model editing
part so if i'm understanding the
question correctly it's sort of like
how can you make sure that your
intervention
changed what you wanted it to change and
how can we understand what it actually
changed about the model right
yeah that's that's exciting
okay awesome um so when we get into
model editing this becomes not only sort
of an interesting scientific question
but also uh very practically important
because say you wanted to change your
model you want to make sure that when
shaquille is a soccer player he's he's
not still dunking over everyone and like
being bad at shooting free throws and
stuff like that that he's scoring goals
that he's good at free kicks or whatever
it may be um and so that's sort of where
we get into counterfact where we design
a suite of evaluations that helps
evaluate what exactly your intervention
has changed and whether it's done things
that we didn't want it to do
but in the end like the conclusion is
that rome's doing something interesting
where it's very specific to the entity
and it's changing what you want and it's
also pretty consistent over lots of
different contexts that require the fact
that you just
that you just requested like hey hey
kevin
yeah i'm sort of i'm sort of watching
the clock here
and so i want to i want to speed this
along so that we actually have time to
get to those experiments and so on
um
uh so let's let's let's let me let me
let me uh let me just sort of
uh yes i'm just nudging a little bit
okay yeah sure thanks david
um so yeah i can speed through the rest
of these because i think a lot of them
get at the same sort of ideas this is a
similar example where we apply noise
gaussian noise instead of copying over
representations and it does sort of the
same thing
um and
a natural question now is in our network
we drew these diagrams with not only
hidden states but these hidden states
can be decomposed into the outputs of
both attention and mlp layers so our
question is you know what are mlp layers
and what are attention layers doing and
can we redraw this graph while only
intervening on one of them and can we
see something interesting from that
so in this case if we have somebody um a
football player or a yeah a football
player
um we ask well if we copy over or if we
corrupt the mlps and we corrupt the
attentions and we copy them over
iteratively what kinds of effects do we
see and when we draw them out we still
see these two sort of major causal
effect regions one in the early one
early site and one late site but the
difference is that the mlp layers
activate super strongly in the early
layers and the attention layers activate
super strongly in the late layers
and this is an interesting observation
mlp seems important at the early site
and detention is at the late site
so there's actually the study i'm sure
everyone's quite familiar with it that
anthropic release where they made the
observation
that attention seems to be copying
information from a lot of different
locations namely from the residual
streams of different tokens to your
current token and
this is particularly interesting at the
last token because the last token is
where the next word is predicted
and
our observations
seem to suggest and we'll sort of make a
confirmation of this later when we get
to our experiments they seem to suggest
that the attention is doing something
interesting where it copies over
information from the early mlp
activation site or perhaps somewhere
early in the network it's copying it
over and that makes its prediction um
for the next token so it's copying
information
uh across different tokens and the
residual streams
for people who might be less familiar
with this work it's important to note
that uh that particular citation was
working with uh transformer networks
that were in most you know two layers
deep so it's particularly interesting
again that this this might apply to
larger networks as well
yeah that's right
cool
oh sorry was there a question
no i'm sorry i did not
interrupt okay it's all good okay
um so yeah
basically um we've been looking at a lot
of individual examples uh if we average
all these effects over a thousand
factual statements that we've measured
that gpt knows or in other words um
their correct answer has the highest
probability
we see these effects are actually
systematic
and we've localized factual knowledge
essentially using causal tracing along
several different axes
first we identify that there are two
sites that are interesting there's an
early site and a late site the early
site is pretty surprising and
interesting
and this early site seems to be very
concentrated at the last object token
and finally there's something about
these mid-layer mlp modules that's
interesting and as someone pointed out
earlier they're quite diffuse and so one
of our one of our questions was well
what are these mlp layers doing are some
of them keys are some of them values
and you know we just wanted we wanted to
decompose this analysis further
so one thing that we did is that if you
remember in the original computation
uh when we were copying over these when
we were copying over these
representations
after we copied over this vector future
mlp layers all had opportunities to read
this vector and
and and and add certain information to
the residual stream
so our question is are these mlp layers
actually important after you copy over
this hidden state because if yes if the
answer is yes these nlp layers are still
important after you copy over the vector
then maybe this vector did not really
contain the information that you needed
but instead it informed the network that
it needed to look it up and in the
future it actually looked it up
but what if the opposite was true what
if after you copy this hidden vector
over and you disable all the future mlp
layers and yet you still get the correct
output
that would suggest that this factor
itself contains the information that we
needed to make the prediction and if we
could form a dichotomy behind you know
between these these two different
situations um it'd be kind of
interesting and it turns out that we see
something
along these lines
um
so the graph you see here the purple
bars are the impact of us of a single
state just completely copied over um
with the mlps enabled so we don't change
the computation of the network after we
copy the vector
and these orange bars are the impact
after we disable all the mlps
in that in that in that token
and what we find is that in the early
layers after you've disabled the mlps
the effect seems to lose its impact on
the prediction you get very low increase
in the causal effect
and as you increase your layer this
orange bar seems to go up
and what it suggests to us is that in
low layers there is no effect without
nlp that's the direct observation and
the consequence is that these could be
acting as keys these are informing the
network that it has to look something up
in future mlp layers
whereas in higher mlp layer states the
difference between the impact with and
without mlps without future mlps is
fairly minimal
and the conclusion as we
we have the information that we need now
these vectors to make the prediction
and
we don't need the mlp anymore so
somewhere in the middle
we think there's a lookup of factual
knowledge
okay cool was there a question yeah do
you try to uh do it have the same
intervention after shutting off the mlp
so do you try to figure out a new
intervention to use instead
um
great question so let me clarify the
mechanism of this of this test
uh we do the same kind of intervention
that we mentioned before where we copy a
vector over into the corrupted
computation now in the normal
intervention after we copy the the
vector over the computation doesn't
change right we let it do whatever it
wants to so all the mlps are activating
after we copy over
this
vector
as that's a normal case but then we
wanted to know whether this information
actually contains the information we
needed to make the output or if it's
just some kind of
auxiliary representation that tells the
network it has to look it up
right but you had an optimization step
where you choose what states to
transplant do you choose again after
shutting off the mlps or do you
transplant the same states
so
i don't think we perform an optimization
to choose the hidden states to copy we
simply copy a hidden state and then shut
the mlps off and then let the
let the let the network continue without
those mls that's right there's no
there's no optimization
and just be clear here when you say you
showed up the mlps that means you set
their contributions to the residual
stream to you know like the the noise
output or something like that
that's that's right that's right so if
you if you were to turn them off by
doing something like set them to zero or
something then they sort of act like
uh so so some weird out of domain
signal inside the network so if you just
sort of freeze them at
you know they're sort of responding to
noise then they they behave they don't
mess up the whole network great okay
that makes sense
yeah okay so that led us to sort of
think that somewhere in the middle
um if i can find the slide somewhere in
the middle there is factual knowledge
being looked up somewhere where the
impact with the nlp disabled is on the
uptick and eventually you get to a point
where you don't really need future mlps
to be
uh to be restored in a sense um and
that's sort of our hypothesis for where
it's located you know there's something
interesting here the fact that it's sort
of spread across a lot of different
layers due to the residual structure gp
doesn't really have an incentive to
localize it into a very specific layer
so our hypothesis is that any one of
these layers could contain the knowledge
and
we sort of um just select the one we
there are a couple of heuristics that we
eventually want to implement to actually
select the layer but we think it's
plausible that knowledge could be
contained in any one of
these layers or perhaps some subset of
them
so that pretty much summarizes the the
insights that we get from causal tracing
and i'll hand it off to david to talk
about changing knowledge
so it means that when you take your
hidden state and you copy it over um but
then you don't let the future mlps to
to operate normally um
it still has a very similar effect as if
you copied over the vector and you let
the future mlps to activate and so the
difference here is if you don't let the
mlps activate
and there's no effect then this specific
vector that you copied over wasn't
actually the decisive vector
it told your network hey you should look
up some information that will make it
decisive but it didn't actually look it
up
um there has to be future computation
that allows it to be looked up now as
opposed to if you had this vector and
you copied it over it and you didn't
need the future mlps to do anything
interesting you just had this vector
um and the vector itself had a high
impact
on the final factual prediction then we
would say okay well then this factor
probably contains some interesting
information that's decisive and produces
the correct output does that make sense
um
right and
so
yeah i i think i would agree with that
naturally you almost always also made a
graph for turning the attention by
yourself yes
aha
um
so that's that's a that's actually an
interesting suggestion we haven't tried
that um we do some other experiments
with verifying that attention
um is actually a poor target for editing
factual knowledge but we haven't done
this kind of experiment on attention yet
but that's a good suggestion
the reason being that of course uh doing
weird stuff to early layers is gonna
have more effect than the weird stuff to
late layers
yeah indeed
um
i think that's
sort of one of the first things that we
thought was well you can just change the
embedding of the subject which wouldn't
be that all that interesting but i guess
what's interesting about this graph is
um
if you don't have the future mlps the
early layers actually don't do quite
well
okay
if there are no more questions about
sort of this section and we're totally
happy to jump back um
i can head over to david to talk a bit
about changing knowledge
cool
so
let me let me see so i'm gonna i'm gonna
speed through because it looks like we
only have sort of 10 minutes in before
people disappear i want to make sure
that we
uh get to talk about
uh some of the really cool uh
experiments so i'm gonna speed through
this but i but if people are interested
and they wanna stay over the hour then i
can i can return to this
uh section let me show you more than
welcome to go beyond an hour usually
people stay a little longer oh okay
no time pressure or anything okay cool
um
yes so i'm just being a little paranoid
here but we might do it okay
so let me um
let me see if i can
tell discord
i wonder if there's some way of getting
me not to ask me so many questions when
i do this okay cool
so um okay so changing knowledge so so
basically
you know
you you get this amazing uh
concentration of um
a effect that
kevin was describing
and and then
see the the least of this hypothesis
that maybe
factual knowledge look up is happening
in this sort of green green middle
layers right
so we can test that by actually trying
to change that knowledge to see if we by
going into the layer there's something
that we can do
right at this magical mid layer you know
mlp
uh that we thought was so important um
well if it's if it's really storing the
information maybe we can actually test
its storage by changing it like if it's
acting like a memory
that it knows that the eiffel tower is
in paris because it looked that back up
maybe we can actually
change it to store the fact that the
eiffel tower is in rome instead
and and that that if if we could do that
uh then you know we could
test
uh test our theory
directly
um
so that's that's what our goal is here
is to basically come up with some
framework
where
we can
you know treat an mlp
uh like a little computation that we can
modify but like a little piece of if
then you know sort of if then logic or a
little memory
um where eiffel tower is sort of acting
like an address
and paris or rome is sort of acting like
a value stored in
that address in the memory
uh so the sort of uh associative
memory or the sort of addressable memory
view
is actually a classical way of looking
at single-layer
neural networks
so the idea is that
let's say you have this structure you
have all these key value pairs that you
want to memorize
if and you want to store them in a
neural network somehow well you can
actually store them in a single layer
neural network by just solving the
linear system of equations of v equals w
k for all the v k pairs
uh that's just linear algebra
and if you do that you can store up to n
facts if you're willing to accept
some error then you could store more
than n facts um
and uh and so this is this this sort of
set up here
to uh arrange a network to store uh key
value pairs is an old idea it came from
the 1970s conan and anderson
uh
i talked about this and it's something
that uh is sort of a good thing to still
uh teach people about neural networks
today
um
uh and and so like in our large networks
we have these very powerful embeddings
where the the key vectors and the value
vectors might encode something really
interesting like megan rapino plays
soccer or sql server is by the company
microsoft but nevertheless
um you know
maybe the way that we could store such a
thing would be to store it in one of
these matrices using sort of the
associative
memory model and so
gohan i understand observe that if you
if you accept a little bit of error then
the right solution to this or one really
good solution to this an optimal
solution is uh to just to minimize error
which is just another way of saying what
you really want is a least squares
least squared error solution which is a
really well understood linear algebra
system
if we have
key value pairs where we want the w
times the key to be as close as possible
to the value then the optimal solution
is given by the solution to the normal
equations which can be written this way
so what i've done here is i've just sort
of taken
all the keys and all the values and
gather them together in big matrices and
written the matrix equation for what the
optimal w is that comes closest to
memorizing all the key value pairs
okay so this is this is a pretty classic
linear algebra it's pretty classic way
of looking at neural networks what we're
going to do here is we're going to ask
what would you have to do to change such
a memory if if your neural network layer
is actually acting like a memory like
this what would you have to change it
and so there's a paper that uh we did
back in a couple years ago
looking at this sort of associated
memory view of in computer vision
and um
and so we observed that
if you wanted to change
a value that was stored in the network
like change eiffel towers in rome
instead of paris
right
you can make a change
like this uh by doing a really really
minimal uh piece of math you can do a
rank one update and and so here's i'm
just going to speed through the
derivation here but basically what we
want is we want k star and v star to be
the new pair that we're writing like you
know eiffel tower is in rome instead of
paris
and so
uh but we want to do that while
minimizing all the old
errors that we were minimizing before so
w0 the old solution was minimizing all
these old key value lookups and we want
to continue memorizing this so w1 is the
updated matrix that we want but we want
to subject it to a new constraint that
it passes through this new q value pair
point that's in yellow so this is
actually you know also really well
understood classical math is constrained
these squares here's the solution
to it
um
and and the cool thing though is that
the constrained lean square solution for
passing through this new key value pair
is actually the same as the old one but
just plus a little extra term and most
that means most of the terms cancel out
which means that the new matrix that you
want that's the optimal solution for
storing these new value pairs the same
as the old matrix
plus
this really small update that you can do
it's a rank one update that you can
actually solve using linear algebra so
given the new key value pair that you
want you can explicitly write down
exactly what rank one update you want to
make in the weights
and um
and so
uh so it's so so you can think of this
as sort of the elementary
pieces of linear algebra that our
weights are made out of and we we can
you know sort of solve an equation to
update one piece at a time if we're
given a key value pair that we want to
update
so
so this is uh this is sort of
uh
the view of what's going on and so here
we want to ha we have this key like the
eiffel tower and we want to update it so
that instead of it being in paris
uh it's in some other thing like maybe
it's in rome
uh
so we want to update it that way now we
don't have a single layer
neural network we have a very deep
network
with many things that are going on but
our hypothesis based on our causal
tracing is that maybe any individual mlp
is acting like a memory and we could
just solve it
by
by by inserting a memory into one of the
mlps
now inside the mlp is actually also not
a single layer there's a couple layers
in there and some people have looked at
this before there's some nice papers by
geva and dye
who have have looked at uh these sort of
fan out fan and architectures inside mlp
and they sort of hypothesize that this
might be a good way of storing memories
where the first layer sort of spreads
out the key and gives you a big address
space and the second layer really stores
the value that you want to store so we
we adopt that view uh i
think that makes a lot of sense
and and what you can do is you can say
well if if the second layer is storing
all the values then we can apply our
little linear algebra
fact
to do a rank one update in this
second linear layer to insert
any key value uh
assignment that we want
um
so in this case what we're going to do
is we're going to take the key as
whatever the vector value is here in the
middle of the nlp and the value would be
whatever the target output is that we
want to add
to the residual on the output here and
so uh so given some key that we figure
out and given some value we figure out
we can just sort of use the linear
algebra to update this weight uh in a
rank one way so that's
that's more or less the the idea of what
the roman method is and then what's all
you have to do now is figure out well
what are the what's the actual key that
you want to do
uh and what's the actual value that you
want to map to and so to choose a key we
just use sampling we just run the
network and inference
on a bunch of sentences that have eiffel
tower in them and we
uh we we get an average of them to use
as a key
and for value
we actually use gradient descent
on on the target sentence
so eiffel tower is located in the city
of and we find a vector here that causes
that sample sentence
to output rom if we were to intervene in
the network and just sort of uh insert
the vector here at this location so this
is a different optimization that you
typically do when you're doing
fine tuning we're not fine-tuning any
weights at all what we're actually doing
is we're we're we are uh inserting an
activation
vector into the network at this point
and we're asking
it's a very very small simple
optimization to ask you know what
activation vector would you have to put
here for to say rome instead of paris
um and then based on that vector then we
do the rank one update for the weights
uh in in this in this location and we
can run it in an inference and ask the
question uh you know
when you put this sentence in
um
and you actually run it through our
modified weights does it output rome and
and it does it's because well we we
arranged it uh to be that way you know
eiffel tower comes in here
it goes through our original weights but
they've been modified in this very small
rank one way instead of outputting
whatever would have produced paris here
outputs whatever would have produced
rome
and and then it pops through
and it says roman end
so this is a you know this is this is a
uh
a very very constrained fine-tuning
method it it it constrains us
to a rank one update where the rank on
the left uh has to be in the direction
of a specific entity eiffel tower
um
and uh um and and so
even though it's constrained right it's
not that surprising that when we put it
into the target sentence that we chose
the value for that it it works this is
basically 100 of the time when we do
this it'll take whatever target sentence
we have and say eiffel tower is located
in the city of whatever we can make it
into rome or any city we like or we can
change all sorts of other sentences
quite easily this way
so so this is this is sort of the um
the raw material that we have uh like
the the fundamental operation that we
will try using for
modifying um
modifying our network and so the
question that we're left with is does
this actually do anything interesting or
did we destroy the network i mean it'd
be pretty easy to get the network to
parrot something
that we wanted to pair it while
completely destroying its capability to
do anything else
um and our hypothesis is that no no
actually there's some interesting
structured computation here we maybe we
didn't destroy the network maybe the
networks continues to do other sensible
things and so the question is how do you
how do you measure that how do you
measure your impact on uh on the network
um specifically we're interested in if
we actually succeeded in identifying
where the knowledge is in the network
and changing it so i'm going to let
kevin to talk
talk about those experiments
actually one more question on that point
do you have an idea of whether or not
i'm applying the same method at
different layers but also give the same
results you might imagine that other
layers you know would also show the same
effect so you might not have localized
the knowledge exactly
right right so actually um
so i i don't want to give you a negative
answer because i suspect that if you did
this at any layer you could have a
really almost hundred percent
effect doing something like this but we
haven't done that because we're sort of
chasing after
uh causal tracing
and and we were you know guided by that
we did it at these middle layers uh so
it's still a to-do item
for us to uh quantitatively test
uh you know all the middle layers
defined
so the middle layers are the layers
where we saw the strongest causal
effects so
in gpt2 these layers from
around layer 10 to around layer 20
are the layers where we're having really
strong
causal effects right
okay
okay
do we think that this uh rank one update
might be the first term of a larger
update that's more robust like an svd
composition to an mlp layer
um yeah i think that that it's i think
it's quite reasonable to to uh
to guess that there is some second order
effect that might not just be even just
uh
um
uh
you know an adjustment it might be
something interesting right maybe
there's some like hierarchical structure
or something like that there's a second
order effect
um
so
but we but we wanted to sort of keep it
simple to see
you know with this really simple first
order
uh change
you know does you know how well does it
behave and um
uh and and i think that what kevin i'll
show you when he sets up for uh the
experiments next actually i'm gonna stop
screen sharing here
um
uh
uh
is is it even like this sort of simple
first order effect that we're looking at
uh we were surprised at how robust
the results were and how well
uh it worked it doesn't mean that
there's not something else to discover
that is
lying underneath this um but but that
there's you know that i feel like we're
on to something interesting uh by even
looking at this simple effect here
kevin are you set up
yep that's the screenshot all right dude
sure
all right
so measuring knowledge um
again we have a pretty simple question
that turns out to be pretty complicated
to measure um we know that our our
output is going to output rome because
our optimizers are very strong and very
smart um but the question is does it
really know that the eiffel tower is in
rome and without getting too far into
the
into the arguments about what knowledge
is we have a pretty simple way of of of
sort of looking at it it may not be a
complete picture but we think that it's
an informative one and it gives us it
gives us some information
so we think that two important hallmarks
of knowledge in a network are
generalization and specificity
generalization means that knowledge is
consistent under a lot of different
rephrasings and reframings of the same
fact and specificity means when you
rewrite knowledge into one area you
don't interfere with knowledge that you
shouldn't interfere with and an example
with this
is you know taking the eiffel tower
example if you ask it the eiffel tower
is in rome and it's been it's been
fine-tuned to output rome it's not
surprising that it'll be it'll say rome
but what if you paraphrase you say the
eiffel tower is located in rome that's a
paraphrase phrase generalization you
could say where is the eiffel tower
question mark it is in
you know blank
um consistency is a little bit more
interesting because you sort of ask for
a fact that implicitly
requires the fact that you edited in i
think this one is the most impressive
sort of interesting effect where you ask
how can i get to the eiffel tower and so
not only does it need to know that the
eiffel tower is in rome but it's also
got to connect all the other different
facts it knows about rome for example
rome has a certain station or whatever
that you need to travel to
and things like that
what is there to eat near the eiffel
tower you'd have to sort of know you
know in rome
you have a certain type of restaurant
that is italian that is not parisian
that is not french that is not chinese
that is not whatever it may be so that's
consistency um and then finally
where is the sears tower even though you
both have the token tower hopefully you
haven't changed the sears tower to also
be in rome or maybe other things that
have inventings that are close to the
eiffel tower like the louvre
um
the um
i don't know other tall buildings like
the empire state building we hope that
they're also not in rome
so this sort of guides our
our exploration in what we actually did
to the model
and
based on these principles we've come up
with counterfact which is a data set
that helps us evaluate what happens when
you change a model that goes beyond just
checking probabilities and checking
paraphrases which is what previous
methods usually do
so
um we have a bunch of counterfactuals
here and the cool part is each of these
records comes with a few components to
help evaluate um specifically what's
going on so we have the counterfactual
obviously where
we have a rewrite example we have some
paraphrases you know where we rephrase
the fact directly and we expect it to
output rome we have the neighborhood
prompts
which are essentially a heuristic to see
if we've messed with any facts that
shouldn't have been messed with um and
this is along the axis of the subject so
neighborhood means subjects that are
similar
to the subject that we wanted to rewrite
so it could be the louvre it could be
the sears tower it could be the
champs-elysees or something like that
um and we don't want these to change we
want them to stay where they were
because they're sort of not related to
the original fact
and then finally we have generation
prompts which are prompts that
implicitly need you to know the fact
that you rewrote but will not explicitly
ask for it um
and so we already talked about this you
sort of
where the best places to eat how can i
get to the eiffel tower etc
now the evaluation strategy for the
first three is pretty straightforward
because we're just checking
probabilities of certain tokens but
generation prompt is a little bit tricky
because what does it mean to say the
correct answer to what are the best
places to eat lunch near the eiffel
tower so again we have to sort of employ
a heuristic here where we generate text
and compare the
statistics of this generated text with
some body some other body of reference
text and this reference text is actually
just some wikipedia article about the
target object
so for example if the target object in
this case is rome the subject is eiffel
tower other relation is located in and
the object is rome we look up a
wikipedia object above realm and we read
it out and we compare the statistics of
what gpt said with this article
so the purpose of this heuristic is to
say
are the things that you're uttering
actually consistent with your new
knowledge if it still keeps on talking
about you know
restaurants that you would see in paris
like things that sound french this might
indicate that it doesn't actually know
that the eiffel tower is in rome and
you've sort of failed at consistency
um and you know if how can i get to the
eiffel tower it keeps on talking about
the train station or the airport in
paris that also might be a problem so
this is our heuristic for figuring out
whether gpt actually is consistent
after we've done the rewrite
so
um
remember you know from our earlier
causal tracing experiments we found
these two sites there was an early site
and there's a late site one of our first
questions you know before we even
evaluate and compare rome against other
methods that can edit models one of our
first questions is which of these sites
if at all there's a difference which of
these sites controls knowing and which
one controls saying can we
sort of disentangle their roles uh in
the network
so the way we do this again is with yet
another intervention and we're curious
if you actually just fine tune these
attention weights can you get the model
to behave in a nice way where it
satisfies the soda state hallmarks of
knowledge general generalization and
specificity um so we have both
qualitative and quantitative examples
this is a qualitative one that's sort of
easier to grok
um here we rewrote the eiffel tower to
be in rome
as you can see the prompt that we
actually edited with the eiffel tower is
located in rome it's correct it's you
know an important tourist attraction
whatever
um
and
unfortunately when we rephrase when we
say what is the eiffel tower this is
just
um sort of a
sort of like a consistency prompt
because we're asking what it is we're
not asking where it is um but it does
some rambling about how it's the iconic
building and then suddenly he starts
talking about france again and how it's
a reminder of the revolution
um and it talks about paris so this is
problematic and then
here with the tension edit when you say
right across from it it doesn't really
know what's going on and it just repeats
itself it says it was built to the same
skill so what we can see here is that
the specific phrase that we used to do
the fine tuning
actually was rendered correct but the
other ways that we tried to get the
model to exhibit knowledge of
this fact failed
um
and so we see that in the um in the
quantitative example in the quantitative
analysis as well so the rewrite success
is the probability that you have the
correct answer on the example on the
specific phrasing of the fact that was
shown to the model during the rewrite so
as you can see um it's pretty high
pretty successful
um specificity is great you're not
really affecting too many of the things
that you weren't supposed to affect uh
but what you really see here and you
know it's sort of disappointing about
editing the attention weights is that
generalization is very poor you almost
don't see a difference these blue dots
are the outliers so there are quite a
few outliers here
um
with good generalization but the vast
majority of the mass concentrated around
zero so there's virtually no change in
the way that it generalized it couldn't
rephrase it couldn't you know generalize
to paraphrases
and the conclusion we draw from here is
okay so interesting this late site seems
to be able to regurgitate the fact that
we tell it um it's pretty specific um
but unfortunately it doesn't generalize
to different ways of rephrasing the fact
so we hypothesize that this indicates or
we think that this provides evidence
for the fact that delayed site is is is
is somehow in control of saying and
fails to control
knowing
um and then we're going to contrast this
with something else we're going to
actually edit the early site at the mlp
um and in fact this is just the real
method
uh and our question is well does this
work better
uh and what we find is that yes it
indeed does so
the same example from before it's
located in rome italy when we do the
rephrase it is a symbol of rome which is
good and then when we ask what is it
right across from it's across from saint
peter's basilica in rome italy
this sort of gets in the into the
problem where language models uh
hallucinate we didn't explicitly tell it
that um that the eiffel tower was across
from saint peter's basilica but um what
we do observe is that the model seems to
have changed its representation of
eiffel tower and sort of understood like
hey this thing is in rome so what are
the things in rome it's saint peter's
basilica
the
the sort of discussion about
hallucinations is a separate topic and
one could argue we didn't actually train
these models to admit when they don't
know things but this is this is a good
sign it's showing that the model has
changed its actual conception of this
entity and it's able to be consistent
across a lot of different interesting
ways you can rephrase the question
and we see this in the quantitative
results as well so
you still see the attention edit graphs
on the on the right side of all of these
three plots and then we compare that
with rome so rome has high rewrite
success
it's got similar specificity
but the generalization is way better and
this sort of suggests hey
we're not messing with facts that we
don't want to mess with the rewrite
success is good and at the same time
rome is also able to sort of generalize
to things that um to different ways that
you can rephrase the fact
and
just side note all these experiments are
on gpt2 xl um and what we actually see
as we'll see in a moment the effects on
gptj are actually even better
um
so yeah those those are the experiments
that we actually wanted to carry out to
to to add yet another piece of evidence
that shows there's a difference
mechanistically there's a physical
difference between a model knowing
something and saying something and when
we do these interventions we can
actually tell
so one of the interesting things that we
also did just a quick note is what
happens when we run causal traces on gpt
after we do the rewrite um so
this um plot d shows after we do the rom
update and when we average it over 350
props and the other one is attention
edit where you edit the uh attention
layers
so we can see a very a striking
difference between the ways that these
two models now
actually
process their information after you
edited the attention the attention is
now exclusively sort of looking at the
last word
and
based on all the evidence that we've
collected and all the intuition we've
gathered over the course of this study
we think that this is problematic
because you're simply looking you're
over fitting to the phrasing you're
looking specifically at the last word
and it's sort of too late to change your
conception of what the entity or the
what the subject actually is
um whereas with rome we see a pretty
healthy causal trace
where facts are being looked up at the
entity token the attention seems to you
know behave you know be behaving
normally
and the mlp lookup is sort of
concentrated in this area and this
really solid block here
is kind of evidence of our human
intervention i think this is interesting
because
um
you know we've sort of made it very
explicit what the model is supposed to
do we've concentrated all these effects
in this very specific block it almost
makes you wonder like is this you know
could this be a way to
to to impose
to impose
to impose
sort of architectural constraints on gpt
that actually makes it better um
obviously this is pretty far-fetched but
i just thought it was interesting
um so yeah pretty healthy causal traces
after the intervention
so i guess one of our next questions was
does this actually perform better than
baseline methods for doing model editing
uh we can for example directly fine-tune
stuff we can
um apply a really cool interpretability
method called knowledge neurons where
we um where they perform a an
intervention to figure out which neurons
in
the uh
in the mlp keys are actually important
and then it performs an intervention
that uh hopes to make the fact uh
manifest and then there are also these
cool works that recently came out that
use hyper networks to look at models and
edit them the cow looked at using a um
creating a hyper network extent
essentially to apply rank one updates to
all of the model weights they actually
update all the weight i think and then
mend which recently came out they have a
neural net that um that also does a
similar thing where they apply
interventions to light layer mlps but
similar sort of similar but not really
similar in terms of
these exact layer um that we do
so our question is rome is slightly
different from all these methods um it's
an explicit it's an interpretable method
we have this rank one change that's
constrained based on the way that we
think these models are actually working
um does this direct model editing method
outperform other ones that actually just
do sort of blind optimization with the
exception of dies method which takes a
per neuron interpretability view
um so here are the results this
uh charts going to be a little bit hard
to parse but we've sort of highlighted
the important details that are in the
graph
um we think that there are two
interesting modes of of failure that a
lot of these methods exhibit the first
one is lack of generalization when you
don't have high pro when you don't
you're not really able to generalize
across paraphrases this column is
paraphrased success and we have a little
bit of a more formal definition of these
metrics in the paper but you can think
of paraphrase success as the fraction of
paraphrases that the model gets correct
and paraphrase magnitude as the average
probability difference
between the correct output or the new
output that you wanted to see and the
old output that gpt used to see
um so the first failure mode is just
lack of generalization where you have
super high efficacy
for example with fine tuning with
constrained fine tuning you have a
hundred percent
efficacy almost um but you only have
fifty percent of the generalization
tests being correct or something along
those lines
and there's another failure mode which
is lack of specificity um fine tuning
does great it's got 100 efficacy it's
got good generalization scores but more
than half of the um more than half of
your specificity tests or more than half
of your neighbors
um are now incorrect so now everything's
in rome but louvre is in rome the um
everything cool you know is in rome and
that's not something we want to see
because this can just be naively
implemented by outputting rome all the
time
um and so what we find is that rome is
both generalized and specific which is
interesting because
it's sort of this method that we have
designed based on our knowledge of how
the network works without applying
um
a completely blind loss we sort of do
constrain fine-tuning but we see that it
has neither the generalization nor
specificity failures that a lot of other
methods have
i think one interesting failure case
that we're sort of still looking at is
uh isn't the essence score the essence
score is a little bit hard to
to to quantify the essence score is
essentially
i think it's best demonstrated um
with an example perhaps which i'll show
in the next slide but essentially
um
when you change the eiffel tower into
you know you move it into rome you don't
want it to become
for example a church or a mountain or
you know a lake you want it to stay as
this you know this iron rot structure
um and
sometimes you'll see that you know
certain methods so the rome method does
pretty well on a lot of methods
um
and in fact we have some kind of we we
have a few techniques that can mitigate
essence drift but in some cases rome
actually does cause essence drift like
the other methods do for example if you
had a game like a video game and you
said microsoft created it uh it may be
the case that sometimes romo will make
the model say oh it's no longer a game
it's like a microsoft office 365
productivity tool so this is something
that we're still actively looking into
and i think it has to do with
understanding the values that are read
out from these mlp memories
but here's some cool qualitative model
rewriting examples if we wanted to
insert this counter factual that curie
was actually working in medicine instead
of doing his work in physics
um
we see that a lot of the methods um are
great in some ways but also fail in
others for example fine tuning um is
cool like he you know
um curia often collaborated with a
physician who is also a chemist and a
chemist inventor great um but
unfortunately um it bleeds over into
other physicists for example milliken
who did the famous think oil drop
experiment
is now
you know interested in the biological
aspects of the human mind um
fine tuning constraint fine tuning is is
cool you know if you have
the um
the um
one of the
sort of like a well-phrased prompt
you'll sometimes get it to say well he
developed vaccines you know he was a
medicine person um but in other cases if
you sort of rephrase he now goes back to
discovering his you know all the cool
properties of radioactivity
and radium
um so failure of generalization
knowledge editor was one of the um i
think hyper networks
uh sometimes it breaks the model and
sometimes make it makes it just repeat
itself
um under rephrasings it's sometimes
unable to
be consistent and you know he's back to
discovering radioactive elements
um
and
it also causes bleed over now robert a
milliken is working in medicine who
attended medical school
mend
uh sort of behaves similarly where
sometimes you know if you ask it a a
question that's phrased a little bit
differently it'll start talking about
physicists again and then you know
he just made the discovery of the
neutron or something like that um but in
other cases it's good you know his
expertise is in the field of medicine
and medicine
um
and also men causes a bit of bleed over
where his area of work is medicine um
millikenses
so finally we get to rome which is which
suffers from neither the generalization
nor specificity failures um we can see
under different rephrasings it's got the
correct answer and robert milliken um
his answer does not change from the
original prediction so this is sort of
an interesting case and that's why we've
highlighted it in blue gpt doesn't
actually know who robert milliken is
very well um even though i think he's
quite famous in the scientific community
but apparently he works in the field of
astronomy and astrophysics and this
isn't quite true but it's also what the
model thinks before you did the rewrite
so we're going to consider this as um as
sort of like a positive case
so that was sort of like a speed run of
the results
um and i'm happy to take questions if if
anyone has any
about them
yeah i've got one on on this last part
that you were talking about so
you're showing that um rome is both
specific in general which is
kind of interesting because the
rank one update is meant to be only
specific by
its formulation like it minimizes the
cost
of
changing any other
association between keys and values
so
that makes me think that like the key
in this case
is
very abstract
that we're changing that it already is
like somehow generally
applicable to the essence of
uh
you know whatever
the the value is the noun or whatever i
don't know do you have any opinion about
that like what
does this shed light on this generality
of the key
yeah that's actually a really great
question and it's something that we
spent a little bit of time wrestling
with
um
our keys our keys are actually gotten by
reading out the mlp
outputs of the first layer let me see if
i can find the diagram
um these are the keys that come out and
so you can imagine if we go to some
layer and some token and we extract this
key
uh we would hope that it's it's pretty
general we would hope that this key is a
general representation that sort of has
a high dot product with any
representation that activates whenever
we talk about this guy you know or this
object the eiffel tower in lots of
different contexts so we actually found
that this wasn't always the case
um if you have
a an input for example the eiffel tower
as a prompt or prefix and you extract
the key at a certain location
it's okay like it works on that example
but then what happens if you put a bunch
of tokens before eiffel tower like
oh um
you know
or this person really likes eating sushi
um you know kevin really likes eating
sushi his favorite restaurant is x and
then you start asking about the eiffel
tower uh the representation's a little
bit different and the key doesn't
actually generalize too well so the
heuristic that we used was well then
there are a lot of different contexts in
which the eiffel tower could
um
could occur then i guess we'll just
collect a bunch of those contexts like
for example append a bunch of tokens in
front of the eiffel tower
and then and then average over all those
cases and read out the representation
when you get to this location in the mlp
so you're right we had to play with the
generalization of the key a little bit
um of course like the fundamental idea
is to capture all the context in which
the eiffel tower could appear and we can
only sample to do that um
but it's definitely an important
it's an important part to make sure that
it generalizes
okay great well there are almost
certainly more questions but this seems
like a great point to stop the recording
so thanks so much for the whole team for
your time we really appreciate talking
about these interesting results
you