EleutherAI Interpretability Reading Group 220212: Interpreting across time

said uh
there's a couple people who are gonna
be
presenting today from introverts
from the interpreting across time
channel
and so i'm gonna start by giving a bit
of an overview of what we're what we're
up to and what we're hoping to
accomplish
um
and then igor has some preliminary
results that he's going to present and i
have some preliminary results that i'm
going to present
and then we're going to talk about kind
of where where we would like to end up
um
for both the current phase of research
as well as kind of our longer term plans
because this is something i hope to be
working on for
quite a
while so
the origin of this of this project is
that there is a lot of papers that
in interpretability
that
study like for example the knowledge in
neurons paper
uh the example depth paper
that
study
uh interpretability concepts and in
particular kind of like how and where
information is is represented or
capacities are found within a neural
network but don't really look at that in
a
in in the context of how the network
changes during training
a really
good example of the work that i that i
want us to be doing is the alpha fold
paper which i know there's a
presentation on a couple weeks ago so i
wasn't able to to make that one
uh where they had these really wonderful
plots uh that they called what where
when plots actually i should just pull
these up to give an example
uh oh sorry i said alpha folder let's
say uh alpha zero that's right yep
uh acquisition of chess knowledge and
alpha zero so let me show my screen
briefly
um so i can
provide this as an example to
people who don't know what i'm talking
about in case they're already
here
excellent you can see my screen
we can
yep
so in the
in the paper so acquisition of
structural knowledge in alpha zero is a
study in
al
the alpha zero uh chess engine
develops its ability to play chess and
um
kind of at what stages of training
uh different capabilities uh exist
um so
one of the one of the
cruxes of this paper
uh are these plots that they call what
where when plots
and so you have three axes here one of
them is so these are these are uh
classification they're taking a part of
the of the network
and they are testing whether or not it
can just it can determine
uh
whether a particular
property is holds so for example here we
have uh in-check
uh has contested open file
can capture opponent's queen has a mate
threat there's also ones that are
slightly less categorical
so uh for example these ones are about
the material imbalance the numerical
score that people assign to positions
based on the fact that i've captured my
opponent's rook and my opponent has
captured two pawns in one night and you
know the extent to which that is
balanced or imbalanced as a as a set of
traits and so what you
see so these are all heat maps um
and what you're seeing here is kind of
three axes so you have training steps uh
so zero is the untrained network um and
then ten to the sixth is the number of
steps that they've trained their their
networks for uh
block is is
roughly synonymous at least in the
context of language models uh comparable
to
uh the number of layers uh deep within
the network so you have a large network
and similar to some of the probing work
uh we look at kind of the the first
section or the first couple couple
layers of the network and we're
interested in evaluating the the
accuracy
and kind of what they found is that
they're able to map out in this pretty
uh compelling fashion
both at how how where in the network's
depth
uh certain abilities arise as well as um
you know how how much training you need
to do to get them and they they do all
sorts of analysis analyses with these
plots and correlations
um
and one of you know one of the things
that i think is really interesting for
example you can see in uh plot a
uh where they're measuring kind of the
the engine's ability to estimate the
score which is a proxy for how likely
white is to win in a position
um
there's there's this really nice
gradient that you see across training
steps
uh where it's ability to do this
increases significantly and then
plateaus it uh after about
they train for so they train for after
about ten to the fifth steps for the
next 90 of the training um
its capabilities in this regard don't
really go up and so it's kind of
plateaued and you don't need more than
that in order to get really good
performance uh what checking whether or
not
the engine knows it's in check that
arises even earlier a performance kind
of plateaus at like 10 to the fourth
steps despite the fact they're doing 10
to the sixth steps of training
and then there are other examples um
here has a mate threat is slightly
non-monotonic
um
and there there's all sorts of
interesting things you can see about
these landscapes this this one is a
complicated concept that the engine just
never really develops very robustly
and
uh so you know i i think that these
plots are fabulous and i wish that a lot
of the interpretability research that
people do with neural networks uh and
specifically with language models
looked more like this because i think
that this is really fabulous
and in the context of language models
i'm interested not only in
how these patterns in
language understanding arise across
training but also how would the the
patterns
across training
change as you scale
so for example it is conceivable that um
you know just the stick with the chest
analogy for example it is conceivable
that if you were to multiply the size of
the neural network by 10
um
the
some of these for example in check might
uh not really change at all you need a
certain amount of exposure to data to
develop the concept and it's pretty easy
to develop once you've seen the data
whereas as a proportion for for how
early early in training um checking
whether there's a mate thread arises
could change dramatically i mean you
know as the neural network gets larger
has it has more capacity and it's able
to develop this faster because uh in
fewer steps um as you scale up
and i think that doing that kind of an
analysis would be really fascinating and
really valuable to interpretability
research unfortunately nobody's really
doing that for a couple different
reasons including the fact that there
aren't good
pre-trained sets of models to to do this
on
so big picture what our project is about
is studying this kind of phenomena
though more in the context of language
uh
and
that is not one of them
um
sorry you can't talk and think at the
same time
that's good take it down
uh
and so we've we've identified yeah so we
want to train a set of models that are
consistently trained and consistently
checkpointed like what they did in for
example the scaling laws papers um but
release the trained models as well as
all the partially trained checkpoints
and kind of use these to power analyses
of interpretability i'm sure that they
will be useful for things that aren't
interpretability research as well
um but kind of the the big motivation
from my point of view is one of the
examinations from my point of view is
studying um
interpretability um
with it yes so we have a we have a
couple
experiments
that we
uh so there are two pieces to i guess
phase one i'll call it of this project
uh
one of them is to
simply like train this this uh suite of
models that can be used to to do these
kinds of analyses and then step and then
the other part of phase one is to put
together a diverse and compelling set of
experiments to use to show people look
at all the cool stuff that we can learn
um and so we've we've gone and talked
about a bunch of different kinds of
experiments we've identified a couple
that we are going to start off with um
and igor is uh
is running one of them um and he's gonna
he's gonna tell you tell us about some
of the preliminary results that he's
found there and then um i've been doing
some stuff with memorization
uh
which
along with um
yeah words sorry i'm going to be doing
some i have been doing some stuff uh
with memorization alongside wars and i
am going to be presenting uh that after
igor talks about the the really cool
probing stuff that he's been doing
um
yeah so i think that largely covers it
you want to take it away
sounds great
for sure thanks stella can uh can you
hear me can everyone see hear me and see
my screen
we can
all right
that's uh great um yeah i've talked to
some of you in on various channels and
threads but i'll just take you know 10
seconds to introduce myself i've i'm
from engineering product building
background i spent last seven years at
pure storage building a data storage
product
and i'm currently collaborating with
sarah hill ventures
at uh
founding a next
startup in in ai primarily with focus on
b2b space
so i'm spending half of my time looking
at the latest and greatest technologies
in ai and really trying to you know
have have a
have my hand on the pulse there and and
the other half of my time looking at old
crusty enterprise applications and
trying to figure out how to how do we
pair the two together and and and and
built
uh an interesting company in in that
space
and so so as a part of the the ai
research part i've been spending some
time uh with elusive ai i've
really appreciated that i can i can drop
in and uh
if i'm you know if i'm i'm can join
projects and and help out with things i
certainly appreciate
uh you all having me here and uh
tolerating my presence uh so so uh
let me uh let me talk about some of the
things that that we've looked into as a
part of the
intro interportability across time
research track
so
um can folks still see my screen i'm not
i'm not quite used to this uh maybe i
need to switch back
to the setup but i'll figure it out so
uh in the what when where plots
uh
can't both still hear me yep okay okay
okay all right all right okay so in the
wear plots
uh one of the things that you need to be
able to do is
look across
layers so you need to be able to probe
across depth
uh cooking glass yourself please
sorry
no okay that was what was confusing i
was like what's going on uh so so one of
the things that you need in order to do
to
look
uh at interportability is to look across
depth like what's happening at different
layers in the transformer
and the first like the most basic
question is you know can we extract
something out of a given layer in gpt
dox or any other language model
and so
that's that's kind of the first question
and
for those of you who have been following
any of the interpretability discussion
you you probably would have seen what it
lands or or you would know that like
dropping layers uh
means that the model still continues to
work
so um
so this tells us that the prediction
of the next token is present at lower
layers of the of the transformer and
most algebras did a
very nice thorough examination of this
of this concept and
we could easily spend an hour just
talking about the logic lens but uh in
terms of what happens with with the
model i kind of tried to illustrate it
on on the left side in terms of in in
in terms of the components of gpt neox
and i'll use this format on subsequent
slides as well
so so basically what we're looking at
here is we we just
take the model and we only keep
l layers right so we drop the top layers
and that gives us a next
that that basically
ends up producing a model that still
works
it generates a next token prediction
uh and we can you know we can then let's
say look at what's the
uh what's the perplexity on wiki text or
what's the what's the law clause on wiki
text
and so if you look on the on the graph
that i'm showing here
here we're looking at
perplexity
on wiki text as examined by
logic ones
and
we actually started zero so you can you
can do logic lens prediction straight
out of the input embedding
um interestingly it depends whether if
the embedding and unembedding are tied
like they're in gpt2 it doesn't end up
working
because
because in that case logistics just ends
up projecting back the input but on gpt
you know on gpt neo x you can you can
use logic lines to get the prediction
out of every
layer which is that's super interesting
now
the next interesting question is
logic let's shows us that
the next token prediction is present in
earlier layers
now one question is like is logic lens
the unembedding matrix the optimal
projection
so if we take the unembedding matrix is
it the optimal way
to project
the next token predictions out of
earlier layers
or are there is there a better
prediction
and if there's even a better prediction
how much better is it is a little bit
better is it way better so that was kind
of the question that we started started
with that was uh
this stella guided me down this
general path to go explore
and uh
what the the way
and i mean i've looked at this problem
in a few different ways
and
though the way i approach this is to
add an extra
linear projection in front of the
unembedding layer so basically instead
of
using the unemitting
unembedding matrix
to project the output of a given layer
we effectively fine-tune the unembedding
i've done this through a few different
variants i've done this by
retraining the final linear
matrix i've done it by putting an extra
uh linear transformation in front of the
uh unembedding matrix and just fine
tuning that
and you end up getting the same thing in
in all the different variations and so
you get the second curve the i guess
orange dashed curve
that shows that
actually you can get a much better
production out of the residual stream
compared to what you observe with logic
lens
and so this demonstrates that
there is
a much better prediction
uh than what you might think from from
logic lens now one of the i think the
key part of
the i guess thought process i'll be
going through is um okay
is the orange curve like overfitting in
some way or is it really demonstrating
to what extent it's really demonstrating
knowledge that is present in that layer
like to what extent is it fair to look
at the orange lake orange line and say
that hey this is what the the model
knows at a particular layer
um and so this will be something that
will get you both from like empirical
and theoretical side
um but one thing that's a quick question
do you have any notion of the
variability for each of those points so
like how much
variance is around
the blue points as opposed to the orange
points
variability with respect to
just the performance overall so like how
sure the average performance is better
for orange than blue but is there just a
lot of variability at different layers
or does the variability change or
so
in order to understand kind of the
yeah so so the or the orange line
you know can be
obtained in various
uh in
in through various methodologies
i've tested this across
various model sizes i've just tested
across various
um
evaluation sets so here here i've
trained the extra linear layer on pile
and i'm evaluating on wiki text i've
looked at evaluating
enron emails i've looked at evaluating
at other
other data sets both in both that uh
measure perplexity and also that measure
other scores like lambda
and and
so so forth
so
the general results that i'm showing
i've validated in
by varying a number of different axes
yeah i actually didn't mean that as a
skeptical question it's more trying to
understand more just kind of what the
distributions of those points look like
so that maybe
let's say that for ninety percent of the
words you get a big difference in these
things but um for ten percent of the
words the
the performance is just as bad as
i don't know
random guessing or something for most
layers you know you could see different
characters of
different behaviors at different layers
if you looked at variability
yeah so what i'll have
later on is i'll have a i'll have some
i'll have an example so we can look at a
concrete example
of what's a prediction that actually is
generated
by a logic lens versus the
uh versus the extra linear layer so so
let's get to that and then if it's see
if it's helpful and again
we can uh you know
take it from there
uh
but so so one point that i did want to
make is
uh
if you look at i mean the difference
between logic lens and the and the
fine-tuned and the tuned lens is pretty
huge like it's it's something like um
you know eight layers in in the most in
from like layer zero to layer eight
and so
uh
sure you you have to believe that
what the true knowledge of the model is
to the prediction that such such a thing
exists has to be much closer to the
extra linear
line compared to the logic ones because
we've really only added
uh one matrix that's like a million
parameters
you know and then embedding space
squared
so
we've added relatively few parameters
and we got i got a prediction that's so
much better
so even the most pessimistic
interpretation like you would have to
imagine the true knowledge of the of the
transformer is maybe like a quarter of
the layer to the right of what the
orange layer line shows but
i it's certainly it's certainly quite
clear
that um
extra linear
the extra linear curve is a better
representation of what a given layer
knows about a particular prediction
and so we'll
get to that point from from different
different perspectives and angles
one one thing i want to type in at this
point about is that
there's a question as to whether or not
tuning or you know
changing up the methodology or or doing
some kind of fine-tuning or doing some
kind of prompting
is a
is cheating or not um and kind of the
approach that we've had
in general is that there is
knowledge that is
inside the neural network and we're not
necessarily trying to measure
what exactly is going to happen if you
query the network um so much as how much
information or or knowledge of what
facts can we extract from the network in
some way
uh which which i think is pretty
meaningfully important because one of
the main things that we're interested in
with these with these networks is using
them as a component in some kind of
pipeline
that is going to know a lot about
uh you know about the world um
yeah so absolutely so like to what
extent are we cheating by drawing the
orange line i think it's the that's the
key question and and then i think that's
like the the main thing that
the rest of my uh slides really come
back to from various perspectives so i
think that that's that's a very
interesting question and and that's one
that i'll
get to from different perspectives
uh i i also have here results from from
lambada uh
comparing logic lens and extra linear uh
extra linear
fine tuned projection again you see the
tuned projection doing substantially
better like quite a few layers earlier
um
yeah i've looked at lots of different
evaluation
sets one of the things that is nice
about this methodology is you can apply
any benchmark from let's say an
eval harness
and you can get the scores at any given
layer so um
that's that's pretty cool um some of the
benchmarks are noisier than others and
so sometimes the uh sometimes it takes
some time to understand and interpret
what's going on
uh
so uh so let's get to another another
question
tuned lens the the way i introduce is
it's reading from the residual stream
we're projecting from the residual
stream
into
into
a prediction space
so so what if we also tune all
projections that right into the residual
stream and what i mean by that is
all the linear
projections
that
precede the the residual addition
so so let's look at
kind of which parameters i'm talking
about
so here what i what here what i tried
was i tried to tune the extra linear
matrix
uh and then i also tuned
attention dense and mlp
the the output
matrix of the the multi-layer perceptron
so basically we're tuning
like this is like 40
of parameters in each layer
and so now question is how much how is
the curve going to look like is it going
to be how much better is it going to be
than the tuned
projection that we tuned on the reading
from the residual stream
uh and the thing that was super
interesting and i would say
i think unexpected in a lot of ways is
actually tuning these parameters does
not get you any benefit at all
so
the way i'm interpreting that is
is the model already writes
optimally into the residual stream
so ultimately all the
matrices like the attention dance and
mlp dance for h2h
those are already optimally tuned and as
long as you know how to read from the
residual stream there is no extra
benefit from from tuning those those
matrices um and this is super
interesting because now here i've tuned
way more parameters
than in in the previous
than with a tuned lens with two lens i
only like tuned one
square matrix in the embedding space
here i tuned like 40 percent of
uh parameters in each layer
and uh and there's no additional
improvements i think that's that's super
interesting to think about this and try
to incorporate that into the mental
model of how the
residual stream behaves
and so the the next question is like
okay what if we tune all the parameters
and i think that if i tune all the
parameters i didn't get an improvement
then
at least something would run with the
methodology because i think everybody
expects that if you
if you
drop the model
with a certain number of layers and then
you retune it you should get
a better performance so specifically
with the
tuned lens we see roughly a linear
behavior across layers so the
improvement in
log loss is roughly linear
across
across layers
um and you would expect
something you would expect improvement
that's you know behaves exponentially
with respect to the layers
so
i i did this you know to do this
experiment where i
retuned the entire
cropped model
and we do get a curve kind of more along
the lines of what you would expect
these are more more
finicky models to train so you kind of
need to use different learning rates for
different sizes and i i didn't quite
do enough hyper parameter optimization
in the middle so that's why that's why
the curve is not as nicely rounded as
you would expect it's a little buckled
because i only use two different
learning rates one optimized for the
left and the other options for the right
hand
but you so you get this
curve that represents
how good can a model get in let's say
six layers and so so you can look at how
big can a model get in six layers
versus if you have a 12 layer model
what how how good of a prediction can
you get out of the sixth sixth player
which is a pretty pretty interesting
comparison
does this make sense so far any i get
any questions at this point
uh we can look at we can look at other
benchmarks or comparing these curves i i
won't spend too much time on that part
um
and so so i think there are three i have
one question actually just on something
you had a couple slides ago
uh like
it sounded like um
the the model is already optimized for
writing into residual streams
uh but you know
humans want to read in an interpretable
way and it sounds like the model doesn't
care about that part until maybe the
very end is that is that what you're
suggesting
right exactly so the model already
writes optimally into the residual
stream
but it's only
i mean the the prediction
well the the unembedding matrix is only
optimized for
interpreting the output of the final
layer
and so
i guess i'm making the case that if you
want to read from the
earlier layer you need to read from the
residual stream
i'm making the case that that you tune
this
this
unembedding matrix for that layer and
that's the way you read from the from
the residual stream so that's sort of
how i'm
thinking about it
oh
um
oh i would so i would also add that um
so when we're talking about tuning
linear layers and if the question that
you're asking is one of the inspiration
one of the inspirations for doing that
namely that a linear layer represents a
change of basis
so if you think about each layer of the
network
outputting into a slightly different
vector space
um
and so you you have information that's
being written into the result residual
stream and as you go down the neural
network some of that information is
getting
modified or changed but uh another
significant portion of what's going on
as you move deeper into the network is
it's just
changing how it's using to express that
information
um and so but kind of the motivation
behind
specifically looking at tuning a linear
layer at the end is that we are
learning a mapping from
the intermediate state that the model is
in after six layers
to
hopefully
a vector space that is more aligned with
the embedding space sorry an embedding
space that is more aligned with the way
the model is representing information at
the end um and therefore ideally
something that's going to be more
compatible with like the the unembedding
layer
yeah i think that's just really well
explained so so if you believe
that
the embedding after every layer contains
a subspace that has the next token
prediction
and the subspaces can
be different across players
then it's a logical consequence that
like okay how do we read from the
subspace well we tune a linear
projection to find
that change of basis operation that's
going to give us
basically that prediction out of that
particular layer
because we've demonstrated that the
basis is not the same across cross
layers
i think you could also turn that the
other way though right because if you
think of a linear model from embedded
tokens alone
this is something that was brought up in
the last circuit's work that's that's
basically modeling bi-gram statistics
right you just go from the last token
and give me a linear prediction of what
the next token is
if i gave you the parameters of that
model it's not like you'd necessarily be
saying that you're just reading out what
the prediction was at the input
because
i mean you get some information from
those parameters right you're still
you learn something from getting from
training those parameters it's not like
you're just reading out
the embedding space
at that point there is some
predictive
extra stuff going into that as well
so i think
that is an important thought and and i
do i mean
i aim to address that in in future
slides but absolutely so so linear uh
linear linear transformation is is not
is not nothing and you do have some
parameters um
i guess i'll i'll just make one comment
here but really address it much more
thoroughly later at least to the extent
i have time because i don't think too
much time here but uh you know even if
even if the transform even if the
prediction was encoded in like specific
bits of the embedding like even if it
was
there the model would still have to know
to extract it from those bits and they
would still have to do do do do
something
but i'll
quite a bit more thoroughly in in the
next couple of slides so so let me talk
through that and then you tell me if it
was helpful
um
all right so so um i mean ultimately we
end up with these three curves that i
think are super super interesting if the
logic lens
the tuned lens and then the retrained
model curve which is you know
roughly um
logarithmic
and and so we can look at a qualitative
comparison
between these different methodologies i
picked sort of an arbitrary prompt
that was meant to
test knowledge and grammar a little bit
uh so it says at its peak the roman ire
and if you and i looked at layer six
prediction out of a 12-layer
model
um the logit lens
kind of generated something so it's a
roman empire's largest no reality tv
based off duty it's kind of like some
phrases there but not
not great tuned lens
generates something substantially better
so edit speak the roman empire was the
most popular in the world but this is
the only thing we have seen it's a and
then it kind of goes off the track a
little bit
and then the fully retrained model is
clearly the best one uh and so
so qualitatively you can see that
it's consistent with you know logic lens
not doing so great
full retraining being being best
and the tuned lens setting some sort of
in the middle between the two
sorry you say that the full retraining
is the best qualitatively here because
it still follows grammar like the
content is still kind of
strange for all of these right
but now you know we are trained early
tiny models here we are training six
player models
that's really small i mean to me at
least the second one reads
better i guess he gets into a subjective
judgment i mean the the middle one kind
of goes off track with the it's a big
guy out the other way yeah i think at
least i'm just trying to articulate what
carries my qualitative impression i
think it's just that
the last one follows grammar while the
other ones just kind of go into
some kind
it follows grammar but it also follows
like it's talking about ruling
which is you know it's tied to the
concept of of an empire
so
i mean it's not giving you encyclopedic
knowledge but it's sort of saying things
that
reflect some understanding of the
general topic even though
as far as to say that the for example
the logit lens
output looks a lot more like a sequence
of somewhat randomly generated tokens
um you know maybe maybe you sampled from
from a giant text and you pulled out six
tokens you wrote them down and then you
sampled from the text again and you
pulled down another six tokens and break
them down that's that's the kind of
process that i would expect to give a
sentence like
what the what the logic lens outputs
um so
[Applause]
i did not touch that question
i think
we have someone that should try to mute
[Applause]
experiment that i was curious about is
logic lens is effectively your
projection is effectively how
unembedding matrix would perform on the
earlier layers so you can generalize
this concept and look at
how does a tuned lens on each layer
perform on all the other layers
and so i did this experiment
and on the same small model
you can examine
how do lenses trained on
one
um
on on one layer perform across other
layers
and so um i i don't necessarily have
like a big
fundamental takeaway from from this
observation from from this this graph
but i think it's very interesting that
you can
see
how the
uh basically how the space changes
across layers so and when you tune a
layer when you tune the lens on one
layer it does well on that layer and
maybe does
okay on the on the layers next to it but
it does consider but then that's
progressively worse
on layers that are
for further away sorry i'm having
trouble parties in this plot so you the
task itself is to not just train to the
output but to train to the
representation of the other layers
yeah yeah so so let me talk through it
one more time or
let me talk to it in more detail and
more hopefully more clearly so
previously what i show then we can go
back to to this picture the middle line
that we're looking here at here is we've
for each layer we tuned a lens for that
particular layer so like the the on the
on the red line
at layer six
we've tuned
uh
a
a change of basis matrix effectively
to transform
from the output of the sixth layer
into
the output space
so we've tuned the lens to get the
prediction out of the sixth layer
but now we're asking the question okay
if i take this interjection he meant to
say green not not red
yes i need to say yeah i meant to say
green
the the red line is is is
is irrelevant i guess
this particular part of discussion
yep okay i think i get it so you're
you're using the lens trained on let's
say layer six throughout the network and
seeing how it performs
on all the other layers because that's
exactly what i would argue with the one
lens is taking the projection that was
trained on layer 12
and we're applying it across all the all
layers
right
and so i'm going to generalizing from
from that
and so then you get this so so
the pink line in in this plot is the
logic lens because pink line
is
the
uh is the projection that was trained on
the output of layer 12
and so it does best on layer 12.
it does okay on layer 11 and then it
does really poorly
on early layers
so it it this is sort of an illustration
of how the
face changes
across layers
this is this is the prediction place
if i can provide a quick caption to this
to this image because i know there's a
lot of lines a lot going on
so in the in the key where it says like
lens five um that's referring to
a basically what's happening in the
logic lens but instead of being tuned to
the output layer it's being tuned to the
the representations in layer five
and then
so that's that's the green curve and so
as the green curve goes through the plot
we're seeing what the performance is of
that lens that's been tuned on layer
five when it gets applied to the
uh
to the outputs of other layers
and the reason why it's
these curves all hit the uh the black
line is because mathematically uh tuning
tuning an extra linear layer is
uh
yeah the
tuning an extra linear layer uh at level
f at layer five
and then using it to interpret layer
five outputs is the is the same thing
that we're talking about um
so
it's it's like a mathematical topology
that
unless unless something's deeply broken
about your optim about your optimizer it
should be the case that all of these
curves hit
precisely the values on the black line
and so the kind of really interesting
thing is the the way that they bounce
off of the black line
as well as the shape of the curve before
it so you can see kind of how
how what what happens when you when you
miss a line the
uh the interpretability lens the with
the residuals
and how much that hurts performance
exactly yeah thank you thank you for for
explaining that uh you're absolutely
right that the i mean the lines hit the
black curve sort of by definition i i
mean i use the same run to get the id
because otherwise i have to just run the
same thing twice it's like really is
that is this tautologically the same
thing as you said uh but but it is the
interesting thing is how the colored
lines bounce off over look from the
black line
and and perhaps the fact that you end up
with you know something that looks like
a line at
the the bounce points
something really fascinating about this
plot personally is that they is that if
you look at layer 12 they come out in
the right order
um so
the first is the one tuned for layer
zero then layer one then layer two and
layers like four through seven all get
kind of close to each other but they are
actually coming if you read them top to
bottom um in terms of their performance
they are in order zero one two three
four five six seven all the way down to
twelve um they're
coming out in in monotonic
decreasing order um or a little more
a little more clearly um if you if you
tune a lens on layer k
and then use it to interpret the output
of layer 12
which is the last layer in this network
because it's a 125 million parameter
model
then
performance is the performance of that
lens is is a monotonic decreasing in
the distance between
the layer that you tuned to and the
layer actually evaluating on
and
that the fact that the difference there
is what's determining performance is
true almost everywhere in the plot
there's like a couple areas where it
gets a little messy and the lines
crossed um
but almost everywhere in the plot the
the main thing that looks like it's
determining the result is or the the
loss in performance is
the actual distance away in terms of
counting layers that you are
yeah that's super interesting that's a
really nice observation and by the way i
would bet that some of the places where
the lines cross are because i i
missed the
missed uh
run so like in
x equals one the green line is crossing
and now i'm looking at it i bet that i
missed the point there i didn't have a
measurement uh so but you know leaving
that aside i think it's it's
very close to you know very little line
crossing given how many lines there are
uh we can look at to look at a
slightly larger model this is this is a
24 layer model and in the same general
behavior
it follows
um
the the lens train only on the input
embedding behaves a little differently
from the previous
model from the smaller model but you
know i'm sure that'll be interesting to
to try to figure out why the
why it sort of drops down for the
final layer and how the final layer is a
little bit more similar to the first
layer the layers in between
uh but but you know it does show you
basically how this uh output faces out
of the the prediction in layer outputs
changes in sort of this mostly gradual
way which is uh that's pretty
interesting to you know
to me
uh
on the leveling off uh afterwards i
wonder if that's because the knowledge
is simply retained in later layers i
would assume that if it were to forget
the knowledge then you'd see a decrease
yeah so i think that when when a line
bounces off
there are two factors that are at play
one like the later later layers
no more
but then
the lens can't really take advantage of
that
and
and the representation drifts
so the uh so the lens gets there less
effective in in in later their layers so
i think
okay it looks like we were internet
did you lose me
i hopefully
okay all right i'm back here
so let me wrap up with a few thoughts
and i'll try to wrap this quickly so
that we don't run over time too much and
so
i guess the first observation is that
the next token prediction subspaces vary
across layer and they vary sort of
gradually
and so you know we have basically output
prediction encoded in the output of each
layer but exactly in which subspace it
is seems to change
gradually
the second thought is that we're using
logic probing as a as a technique
meaning that we are
generating
logits
out of a given
layer and this is something that enables
us to run
various
language model tasks to evaluate
performance at that layer
um which is it's not something i've seen
in in the literature
you guys know a lot of the research
better than me so but i think this is
worth
i think this is worth thinking about and
uh specifically we recommend the tuned
lens
as the probing method
uh and now i'll get to the i think the
final thought which you know gets back
to the question from earlier
to what extent does the tuned lens
reflect the true knowledge in in in a
layer
and um
the the argument
that
i am making
is
the way to think about the knowledge in
a given layer is how accessible that
knowledge is to future layers
now how do future layers
access
well they always access knowledge
through some kind of a linear transform
there's always a change of basis you
either have the kqv
matrices
or you have the mlp pre-activation
matrix that reads from the linear
from the residual stream the future
layers always read from the residual
stream
by
a linear transform so they always access
some subspace
of the individual stream
and so so the
the claim is
that
accessing any subspace
of
residual stream
is something that's free to future
layers of transform and so because of
that
the claim is that that's why you can
assume that the knowledge is known in
that layer it is already encoded in a
way that's effectively
effectively free for the future layers
to
access
and so i i tried to
i tried to
formalize that and i guess apparently i
didn't
paste that correctly uh so let me just
uh let me just fix that really quickly
um
okay
this is what i
wanted to show
can i mention that it's it is nice to
see a live person on video but i think
that might be hampering your connection
a bit
all right
i'm trying it off
it does seem a little smoother at this
point but
let's let's see how it goes thank you
for
for working with us
so this is just my last last slide
uh and
i you know i tried to kind of
formalize this statement i kind of just
look at what i've seen empirically
and try to
try to look at the theoretical response
and so
when you look at also so let's look at
for example how keys are computed in an
attention head well you take
the embedding from earlier layers hi and
you multiply it by the key matrix for
that attention head
and now let's say that we had
a lens that was tuned at that layer
well then you can express the
calculation of that then
then let's look at the calculation of a
key as an arbitrary linear transform of
both h i
and h i
times the lens so basically we're saying
like
the key
calculation accesses both the embedding
and also the output of the tuned
lens
and basically just and when you go
through the math you end up just with a
different key matrix
so you end up with a different key
you know
6k ij
but it's still linear right it's it's
the as long as the model
learns the appropriate
it can attend both to the embedding
and also it can attend
to
whatever the whatever the
tuned lens transformation is
and so so this to me is like a
very interesting thing
that uh
the model does not need future layers
don't need any additional computational
capacity
in order to access this information
that's in this subspace
the model still needs to dedicate a
subspace of the
keys
to represent whatever information it's
extracting out of the lens
but that's sort of a logically necessary
requirement even if we were extracting
the information out of the
key vector through identity
through the
even if we're extracting the information
out of the hidden um latent state
through identity
that information would still have to fit
into the key vector
so so to me this this is the argument
that i would make
that
that the tuned lens
represents information that the future
layers can access for free
and so that's why i would say this is
something that generalizes across
transformers with nlp computer vision or
whatever else you're using transformers
for
um
so i'd be super interested to get
people's thoughts on this uh
get
feedback and
if you have ideas on how we can run
experiments to either prove or or
disprove this way of thinking about
things
i'm
very interested to hear what you think
yeah thank you so much for for walking
through this stuff this is really great
um regarding the slide i don't think
i've necessarily parsed all the details
yet but i don't necessarily think that
thinking of it as thinking as a of a
change of basis is for free still makes
sense to me i think
um
just the the even the bygram model
consideration that we were talking about
before
still kind of weighs on my mind because
it's not no information that's encoded
in
figuring out what you need in order to
understand what the next
token is
touch on that point the way i think
about the
the diagram model is that basically
diagrams are
hopeful
that that the
diagram
knowledge can be integrated into
whatever follows for free
because
after
the way you read from
uh from
the digital stream is through a little
transform
and taking multiple transforms
single linear transform
and so basically the the diagram
information can be encoded into
whatever linear transform follows
for for free and so that that's
the argument
i think i
i think i can see that part but i think
when you say for free you mean a
specific version of for free like it's
for free in the sense that
it's free during once you've actually
done the training but in order to figure
out what the parameters are that you
need in order to figure that out is
still not
no information anymore right and so i i
think that there's i think that there's
two
slightly different concepts going on
here um
when when when when igor says that the
information is available for free this
this is is a different this is not the
same claim as saying that
tuning the linear layer doesn't provide
any additional capacity to the
pre-trained model like we we're talking
about something slightly different here
what we're what we're asking here is
does is as the model trains right it
gets so okay so as the model trains in
the lower layers it has
direct access to the inputs
it has like the attentions that come out
of that and it also has the intermediary
computational results from the previous
layers
and so one question we can ask is
as the model trains does it need to do
additional work
to extract information that it has
figured out in previous layers if the
answer is yes that means that there's
that could have a couple different
consequences one of which is that there
there may be some way to to optimize the
setup so that doesn't need to
do
do additional work to to recompute
things that's already figured out
um
but what what what you are is is showing
here is that mathematically speaking
there does not seem to be i mean
setting aside what it would mean to
demonstrate this empirically because i
think that like nobody has demonstrated
something like this empirically ever for
any neural network uh but mathematically
speaking there is no particular reason
to think that as the neural network
trains
layer 17
has to do any work
to to access information from the
previous layers
which which is which is definitely not
the same thing as saying that if we
stick a a linear layer at the end and
tune it and use it to extract
information we're not adding information
it's it's
it's a similarly structured but but very
very different claim does that
answer what you're looking for that
helps i think i need to think about that
more to
parse the differences yeah
so yeah so the main the main question in
this slide is kind of
how
how how accessible is previously
determined information to layer layers
of the network because we've been
talking this whole time about how they
express things in different bases and
how
you know we have to tune additional
parameters and add them into the network
in order to get human readable outputs
that that make sense to interpret prior
layers
because the
because the the unembedded matrix is
only tuned to the output of the last
layer and kind of the the argument here
is
that even though we as humans have to do
that in order to
get information out of the previous
layers
the the neural network mathematically
speaking is already doing that work when
it's being trained so that
we we shouldn't think that it is uh
particularly cumbersome for layer 17 to
consult the the outputs from previous
layers
um and i think one reason why i think
that it's important to think about this
is that the way that like the the
circuits work especially talks about the
residual stream and writing into the
residual stream kind of assumes that
there is this body of knowledge
that you start off with and then layer
one modifies the body of knowledge and
layer two modifies the body of knowledge
and then layer three modifies the body
of knowledge
and
if
we make the claim that
the input and output representation
spaces
are meaningfully different and that this
makes them hard for us to to actually
get data out of
when we move between layers it suddenly
calls into question whether or not you
actually are are writing into the same
residual stream or whether it makes
sense to even sorry whether it makes
sense to think about just writing into
the same residual stream
or perhaps writing into the residual
scene in the same language because if
layer one is writing with the layer one
uh output embeddings in layer 17 is
writing with layer 17 output embeddings
we just showed you a plot that says the
layer 1 output embeddings when applied
to layer 18 do not give good results and
so there's a
important question here of is layer 18
actually able to read the stuff that
layer one is outputting
and so that's kind of the the context
and stakes to to this slide in
particular as well as kind of how it
relates to what we were saying before
okay i think that makes more sense
yeah
thank you i'll uh i'll ramp up here
thanks for
your attention i'll be super interested
i will share the slides so if you want
to kind of look at this uh stuff in more
detail um it will be available and i'll
be super interested to hear you know
what folks think and
uh if there are specific experiments or
or you know thought experiments that we
could go through to either
either get
either support or
or reject this way of framing the
problem and thinking about it
thank you i'll head over back to stella
hello can you guys hear myself yep
cool i gotta pop saying the stream ended
and so it's just a little worried
uh so yeah so that was
a lot of empirical results igor's been
working on this for for
oh like i guess two months or something
like that now um
that is is most of the empirical results
that we have we're still trying to kind
of figure out what we think the right
way to interpret and think about these
results are but we also have a couple
separate um empirical results i kind of
want to go over briefly
that orza and i came up with um this is
actually stuff that we largely did prior
to
deciding to
to do this scaling model um but we
weren't really sure what the best way to
package it
and and present with the world would be
and then once i said you know we should
we should start training the scaling
suite and doing a bunch of
interpretability experiments with it it
became a pretty natural
place to
present this information so the thing
that that we've been looking at is
memorization in neural networks and
something i observe just talking to
people
is that a lot of people oh let me
share my screen now
uh
yeah so so here's the punch line the
order of training doesn't affect whether
or not a data point is memorized
um so
something i've observed talking to
people is a lot of people have a kind of
a mental model but the reason why
memorization in
language models and neural networks in
general happens is something along the
lines of
the model starts off
as the model trains it has a body of
knowledge or or a set of representations
that it has developed that explain
things that have happened before
and some new information it encounters
during training
is readily explained by
this
uh
what's been developed so far
and other things that encounters during
training are not
and you know if we're going to get
information theoretic about it some new
sentences that it encounters that are
extremely different from some instances
as ever seen before
require more bits to specify in an
information theoretic sense given the
the
the information the model has been
exposed to so far
and it can be
a lot more efficient to simply memorize
the
that information
rather than
do all the work it takes to kind of
re-build the
uh
latent representation space
to have a particularly
whoops that was around to have a
particularly efficient way to represent
that information
and so
kind of one piece of
empirical evidence oh sorry
it looks like that
looks like you're hitting the slideshow
button has probably changed the window
setting or something but the stream has
now closed
oh wonderful
yeah i just wanted to make it bigger
thank you that looks great thanks um and
so one piece of
empirical evidence um this is this to be
clear is not something that i'm claiming
any particular one any particular
researcher has published or something
like that but this is seems to be a
mental model that a lot of people have
um
that i've observed just from talking to
them and before we started doing this
this was my mental model um but i think
a lot of what we've done is kind of
calls that into a significant question
but there is good computational evidence
of this so for example um if you go look
at what content tends to get memorized
it tends to be
stuff that first of all doesn't show up
uh
that is structured extremely differently
from stuff that shows up elsewhere in
the model
so for example if you take a
large quantity of internet text
and you insert like a handful of 15
digit sequences of random numbers
uh those 15 digit sequences are much
more likely to get memorized than some
kind of you know baseline sample from
your training data um and carlini
especially has done a lot of empirical
experiments that have shown various
things like this and kind of the
conclusion that i have reached from
reading a bunch of his papers
and something that i know that he
believes because i've spoken to him
about the sponge is that kind of the the
two big drivers of
memorization of determining whether or
not a particular data point is memorized
is first of all how frequently the model
gets exposed to it because especially
with these large language models um
once you've seen it twice you're already
over fitting and so
as you increase the number of times you
see the same string of text um when
you're training as you know a six or 10
or 20 billion perimeter model uh that
has a really big influence on it because
the model overvalues stuff it's seen
before by a lot and then the other one
is just like you know how different from
everything else is the stuff that you
are memorizing
um and that's that's certainly not like
the only
uh
factors uh
we
ex uh exploratory analysis has shown
that for example some book passages are
memorized and you know why does gtb2
have the first like two chapters of the
second harry potter book memorized but
not the third harry potter book
memorized that doesn't make a whole lot
of sense and if you go and you look they
show up in
there's definitely some other things and
possibly just like some random noise
going on
but a lot of but from the qualitative
analysis that exists it seems like a lot
of the explanation of memorization boils
down to those two things
uh first of all
exposure and second of all how difficult
it is to predict a a bit sequence
and so when we're thinking about how
things change up for training the kind
of natural assumption is that
the order that data is presented in
the process of training matters
if we have this picture of the way
matters specifically to memorization if
we have a picture of the way that neural
networks develop these kinds of
representations that says
it has a model of the world and it's
incorporating data and some of that data
is easy to incorporate and other data
isn't but as it kind of goes on it's
continuously trying to optimize the
representation that allows it to explain
the data that's seen
um things that occur earlier in training
are less likely to be memorized because
they
have had more time
they've spent more time sitting in the
set of things the model is trying to
optimize its representations for
and
because the the model has has used it to
has used that data as its prior to try
to understand and represent other things
that see
um and kind of the i guess i talked to
the first several slides um kind of the
conclusion that we have reached through
our experimentation is that this is just
not true
um specifically if you take a
uh
a sequence of of characters and you move
it somewhere else in the training data
whether or not that sequence of
characters gets memorized like the
probability of it getting memorized does
not change
um so the way that we've we've gone
about doing this experiment is we've
taken uh we we take the the contacts and
we split into two groups we take a
document from the training data we say
okay we're going to take the first n
tokens um i think for these experiments
n equals 32
and we feed so we take it we take an
actual training document and we take the
first 32
characters
and we feed it into the model
and we ask it to predict
uh the next 32 characters and we're not
actually interested in the
like the top one predictions or
something but what we're interested in
is the
logits assigned to the true continuation
so you have a six you have a 64
character sequence
and you when prompted with the first 32
how likely is the model to what kind of
probability does the model assign to the
correct
second half of that sequence
and
um
well yes uh
so for a dense small model uh there's a
typo here this is
125 milli not 150 but
not really particularly important
um so this this is a bucketed histogram
where the x-axis shows the index of the
document sorry of the tokens in training
um
because documents have different links
the fact that
uh i the fact that these are token
indices not document indices is
important
um and what you what you see and so what
this shows is a is a bucketed histogram
um i think there are 20 buckets here
um
and this is the uh 75 range um as well
as the lines between the medians and if
you look at the the y-axis um
this is a really small variation we're
going from from negative 7.14 to
negative 7.19
um as the total variation across uh you
know
3 times 10 to the 11 tokens in train
um and indeed if we look at the the fit
regr the reg
fit regression line
um for predicting
uh negative log loss as a function of
the token index
the there's this
y-intercept term and the slope is 2
times 10 to the negative 15.
that
that's about as
strong of a statement as absolutely no
correlation as you're ever going to see
and then we did it with a 350 million
and we saw the exact same thing you'll
notice that it shifted downwards um
we're now at like negative nine um
larger models tend to memorize less
which is another
kind of implicit point in favor of the
idea that that representation capacity
and and having good representations is
something that influences memorization
but the slope here is still
3 times 10 to the negative 14.
and if we go up to a 750 million
parameter model um we've dropped even
further we're now at negative
24.
but we're still at 2 times 10 to the
negative 14.
um and then you can see here is a plot
of the regression line so you can kind
of see how they're spaced out
um
so first of all it's it's fascinating to
me that the difference between
going from like 150 million parameters
to 350 million parameters versus 350
million parameters
to 750 million parameters
is
ginormous because uh
and actually goes in the opposite
direction that scaling laws would tell
you to predict in general scaling law
says that if you plot things on a log
log plot
um performance increases linearly with
scale
uh this is not a loglog plot however uh
i've done the math and i can tell you
that if i if we were to replot this
analog log plot uh the green line is
closer i mean you can see this
kind of numerically like the the green
line is is closer to the blue line on a
log log plot then it's the red line
um
in terms of the perimeter space
uh
sorry make sure i'm saying the right
thing
um i i do but i want to get more data
points and this was kind of
one of the reasons why i said okay we
should just train a lot of models
um because at the end of the day this is
three data points and
the
hyper parameters for these models were
not
picked perfectly
and they were not picked
incredibly consistently
um so these were three pre-trained
models that i had lying around because i
was testing the gtb neox code base
um one thing that i that i really want
to do and one one thing that's part of
the this idea of training a scale a
scaling suite that i think is really
important
is
using
uh
really precisely chosen values for all
the hyper parameters and making sure
that you are trending on the exact same
data in the exact same order that you
are
you know that all of your parameters are
scaling linearly on a log log plot um
that you you are following kind of the
skill the scaling laws best practices
let's say
uh how's the horizontality of these
lines related to your claim that
the order of training doesn't matter
okay great question so if if earlier
tokens
if earlier sequences are more likely to
be
or less likely to be memorized exactly
than later in counted sequences we would
expect to see that the the negative log
like the negative
log likelihood
that's a tough word to say
of latter sequences is larger we would
expect to see a positive
correlation here but the x axis is not
the sequence index but the token index
yes
entirely so to the token index
hurry can you say that again
the sequence index is entirely
orthogonal to the token index right
you have a bunch of files
and uh each of those files can be
imagined as a line and now you're
looking at how the performance varies as
you go along the line rather than as you
go through the lines oh sorry i thought
no worries so
the the
the question i have here that i think
that this plot is a negative result for
is
is it okay we have the pile it is
uh
1.2
terabytes of text it has been tokenized
it has been shuffled it has we've
decided that we're going to train for
one and a quarter epochs um is what i
believe these models have been trained
for
great so we we have 300 and
we have uh we have
you know three times 10 to the 11 tokens
and we're going to feed them in one by
one auto aggressively and start training
them and the question is is if you take
a span of tokens
that is in the
first 10
of that
3 times 10 to 11
set of tokens and you take a span of
tokens that is in the last ten percent
of that three times 10 to 11 span of
tokens
i see i'm sorry i'm more likely to be
totally
i thought this you got it okay i thought
the token in the like i thought you
projected the entire training set onto
the relative position within a file but
what this is you have concatenated all
the
that's right yeah files
we're concatenating all the files um so
the reason why
no worries the reason why i was speaking
earlier about um you know the first
however many tokens without how this
this is honestly a really good point of
confusion because this is something i
skimmed over that is intellectually
important and can get confusing the
reason why i
we looked only at the first 64 tokens
within any particular file is because
files have widely different lengths and
we wanted to make sure that
the points that we were examining were
statistically independent
as as best that we could so
if i have a very very very large file
um
so so we didn't we didn't do literally
every token um because that would have
taken forever we did
a lot of tokens i don't know if worse or
maybe slides not me um i don't know if
he did not seem to have written down the
number i don't remember was last time i
had um but we we we didn't look at
literally every every token we
took a sample and what we want is that
the the samples are
statistically independent in the sense
that um sorry that the we want that we
would like to control for as much
uh
potential correlation between these
tokens as we can
and one of probably the most important
sources of correlation is that if you
have a
very large document and you train on on
all of it but you sample multiple times
so you sample from say the first 64
tokens in the in this textbook and then
you sample from
another set of 64 tokens that are like
100 000 tokens later in the textbook the
fact that they're coming from the same
data source
and show up consecutively
uh and it can introduce a statistical
bias into it and it seems very plausible
to me that the probability of
measurizing a passage later in a in a
text file
is
dip is like probabilistically dependent
upon
the
random variable that says did i memorize
earlier parts of this text
so oh it's just gonna say they don't
have crazy like i don't have
particularly strong evidence of this it
seemed like it was
likely
um and it seemed like the easiest way to
control for this was to simply make sure
that we were never sampling from the
same document twice
every 30 seconds sequence or every 30
second record of this was actually taken
i mean evaluating
oh this is this is every single sequence
of tokens
as in uh the pile if you consider it to
be a set of records each having 2048
tokens and every 30 second record of it
was taken for evaluation
not oh okay i thought
so tokens one through
uh yeah i think the token sequence thing
is maybe different but you're just
saying that that files were in some
sense sampled uniformly across the
length of the documents right
yes yes that is that is that is the the
moral of the story i'm trying to get to
so i i think i have mixed up some of the
details
that was the goal
was the training data shuffled before
training
uh the training data was shuffled before
training and is in the same order for
all three models
so it was shuffled once and then each
model was trained on it separately
so one thing i i
looking at these numbers and looking how
flat things are i
actually find these i find this very
very hard to believe actually
um
how flat uh
the numbers are both in terms of like
the scale and
the
uh and and just the fact that there's
almost that we're seeing essentially
almost no effect
um
across time i mean you show it yourself
with the regression line it's basically
you know it's it's 10 to the
well it's basically nothing
yeah no 2.25 times 17 out of 14 is
actually stored as zero in in many
contexts
yeah i mean to be fair the token index
is is is goes up to three to the power
of 11 but i i i get what you're saying
is it's it's a very
it's uh it's almost completely
insignificant um
and again i find this i find this
incredibly difficult to believe um
one thing i i'm curious about um
is how does the loss look like
oh actually these are only these are
only up to 750
million parameters i guess
um
and you and the data set we're dealing
with is
how big
uh
3.5 times 10 to 11 tokens um
right i'm sorry i'm really bad at these
conversions uh
i mean this this is like full gtp 3
training data size it's the full pi it's
the full pile um
more than that yes
um
and and we're saying the bigger numbers
generalize
well the other the other thing i guess
that's weird is the is the
is the fact that the magnitudes here for
the different models are are different
so the the the
kind of base rate the negative
on average is is different
though the extent of the variation is
actually pretty similar
so here the extent of the variation is
.05
um here it's point zero five
here is point zero
six
um and you know
that's that's definitely a very rough
guesstimate this is literally like
matplotlib
choosing what scale to use to fit to the
image um but you know
0.05.05.06 is the tick labels that matt
went to put on these so the the
inter
the variation within a particular
model
is not really changing very much as you
scale despite the fact
that the actual overall value is
changing dramatically
and so one one one consequence of that
is that if you express this as a
percentage you know what percentage of
the negative log i don't know
particularly if percentages of negative
log loss is a very meaningful thing to
look at let's pretend for a second it is
if you're asking yourself what is the
percent variation in the negative log
loss
um for
then small
okay and you ask yourself okay what's
the percent variation for the large
model those are
like in order of magnitude different
from each other
uh one when expressed as percent change
despite the fact that the actual
variation
is staying the same because this is
more than doubling how far away it is
um so what and
oh i was going to say the the the
negative log so we're actually we're
seeing the bigger just to double check
again to make sure i understand what's
going on here we're seeing that the the
bigger models are actually doing worse
so they're assigning less probability
maps to the
uh to the previously seen strings than
before is that correct
the
the larger models are doing worse at the
game of given 32 tokens predict what the
next 32 tokens are in the training data
so in terms of kind of the capacities
that we would like language models to
have i think most people would say that
it's doing better that we don't want it
to be able to predict the next 32
tokens
um
from the previous 32 tokens because that
would reflect too strong of a
of a understanding of the distribution
of the particular training text that's
seen
notably this is not a skill humans have
if i was to show you 32 consecutive
characters you would not be able to tell
me what the next third you know if i
write down 64 characters and i show you
the first 32 of them you are not going
to be able to guess the next 32
characters in my sentence correctly the
overwhelming majority of the time and
that's because language just has a lot
of variation there are tons and tons and
tons and tons
of correct continuations in the sense
that like they form very reasonable
syntactically and semantically valid
sentences that i could have written down
in order for you to guess the next 32
tokens gradually you need to get
extremely lucky
sorry or or i need to be doing something
trite you know if you're putting
shakespeare or something you'll be
pretty good at it but
in general what what is the y-axis here
just want to double check
the the negative log loss of
are you sure it's negatively not log
because like it's negative
yeah yeah that was just the other thing
i was going to ask um yeah so that would
be then it would be the other way around
then like then small would be like
the best memorizing which seems wrong
no that's what i'm saying yeah the the
small model is the best at memorizing i
think that
that is not unreasonable
i think that as the model gets larger it
develops
um
more
or so actually let me rephrase that um
it's not saying i'm not saying that the
the small model is the best at
memorizing in the sense that like there
are long passages for example that's
memorized i'm saying that the the
average extent to which it memorizes a
particular sequence of token uh
decreases
um but now i i
i do not know what the difference
between those two sentences are like
there's a very reasonable question as to
whether or not those are
the same thing or not but um
yes this is saying that the
the the
probable the likelihood that the model
assigns to the correct continuation
is going down
as the model gets larger
which
is
i just want to add that um this is like
the opposite of what i saw i i've seen
with the open ai models like comparing
babbage and curie and davinci uh like
davinci has memorized things to a much
greater extent than the other models so
i wonder if this changes as the models
become bigger like maybe we're before
the
like double descent point or something
but i'm not sure i'm really interested
to see how this changes this or how this
looks as you do the scaling sleep
i'm just going to say i'm confused
actually like this
i am not
expecting the the data that i'm looking
at right now i if i'm interpreting it
correctly i i'm not expecting it um
frankly
um
i i i would expect that the bigger
models
would be able to get better
log likelihoods or
assign higher probability mass to the
target data
regardless of whether they
memorized or not because they should be
able to leverage
um you know patterns that they learned
and stuff like that to assign higher
probability mass to the next 32 tokens
um so we figured the bigger ones do
better
so we've got one three uh i'm just going
to try and project
what it you know we talk about large
language models as chameleons
they're capable of being many people at
once many things at once and maybe they
are all memorizing but in order to get
the memorization you need to prompt them
so a larger model
is memorizing even more
but to figure that out
yeah that is that is uh definitely
i think a very reasonable um
thing uh these these are
these results are
are pretty solid and since like we've
we've replicated them um on this on
models of this scale but i think that
they're very preliminary if you want to
draw conclusions about
you know g2 3 or gp neox or
anything that's not like in this range
of models
um
and i mean i yeah i i think we
absolutely need to look at more models
across more scales that we need to make
sure the hyper parameters are are tuned
to be as consistent as possible and we
need to do very things like very how
many tokens of input prompting we're
giving it
and how many tokens of of output
prediction we're expecting to have so
yeah hypothetically it seemed it could
be possible that
uh
a model
is always able to get the next like a
specific model is always able to get the
next 10 tokens but is never able to get
the next 20 tickets
but like it actually assigns a
probability of 0 to the correct 20 token
continuation
but it assigns a very high probability
to the x10 token continuation and that
that is a axis of variation that we have
not really
experimented with at all
um the best i can say is that this is
definitely not tuned to give a negative
result
we picked a thing that
was primarily driven by computational
limitations because this is very
expensive
and we happen to get a very consistent
negative result
but we should absolutely vary those
things to make sure that we aren't
accidentally cherry picking for a
negative result
i observe that these lines are so flat
that you could have gotten like at the
flatness by doing a lot of less comp
computation like
um
you can like a simple way to reduce the
amount of computation that you put into
this is by taking only a random subset
of the training data like shoot out nine
out of every ten of them at random this
should give you like already enough to
say that the line will be flat because
then you know that the variation will be
like smaller than 10 to the minus 10
where the full
amount you get that it's 10 to the minus
15 or something
um given that i think it's uh
like given how absolutely flat this is
perhaps the effect is strong enough that
you will still see it perhaps even this
strongly if you do not do just the first
32 tokens because
like
if like this is so lawful that
presumably the law is simple and we
should find the simplest architecture
where the still holds
for example whether this still holds for
mlps rather than transformers even or
something
yeah that's related to a thought i have
i think a lot of the mental model that
you've talked to in the beginning
probably comes from literature on
catastrophic forgetting from
way back before transformers so it might
just be
a different regime or something
yeah that's that is fair the
overwhelming majority of results on
you know
everything from catastrophic forgetting
to to memorization is is not on
transformers um there's like six papers
five of which were written by nicholas
carlini um that are actually about
transformers with more than 500 million
parameters um so this is in that sense
pretty unexplored
um in the particular context that we are
that i mean that i i and i think most
people in this in this call are deeply
invested in sure
would it be cheap in depth and compute
to
uh redo the that small experiment with
uh the entire trading set
um i would have to i would have to do
the math um i i can say that we have
been
getting a lot of offers to by people who
think that luther is doing cool stuff
who want to give us compute
and that a lot of these people don't
really understand
what kinds of of setups you need to
train like a a very large language model
but you need you know high quality
interconnect and stuff like that but um
while
say commercial cards are not very good
for training large language models um if
we can get access to a large number of
commercial cards
uh you know there are definitely
accessible ways to kind of massively
parallelize this and this is like 100
parallelizable you don't need to do it
in any particular you don't need to do
the analysis in a particular order once
you have the trained models you can
literally run each inference on a
separate card if you have that many
cards um so i am i am very interested in
kind of trying to figure out how to
scale this analysis in different ways
and i do think that there is a
way that we can make that cost effective
whether it's uh
increasingly using subsampling the way
i think it was gurken glass who said
that or whether it's simply like you
know finding people who are willing to
let us borrow
like
20 80 gpus or or
other kinds of commercial cards that are
a lot more accessible than a100s uh but
which historically speaking ulluthera
has not expressed a huge amount of
interest in because we can't use them to
train large language models
what probability do you predict that
when you that if you were to rerun this
without the adjustment by looking only
at the start of every file you would get
the same flat line
hi
i would i would i would bet
five to one odds in favor
that that's like a literal offer if
anyone wants to take me up on that like
i i would straight up you know
bet a hundred dollars on five to one
odds on this
[Music]
i agree
so
i guess one thing i want to take a look
at um i don't know if we have the
resources for it uh if it makes sense or
if it makes sense to take a look at but
i want to see
because this from what i'm gathering
we're looking at
the average memorization
from like looking back throughout the
trade the training run uh is that
roughly correct
yes that is exactly what we're doing
um one thing i kind of want to see is i
want to see the memorization of a single
data point
um throughout the training run so i want
to see you have one sequence
and then about like say
uh
you know at the start or midway through
the training run or whatever
um
the model sees the data point and then
we plot the log likelihood
as a function of as a function of time
essentially
um
throughout the training process
uh probably would be cool to do it like
multiple time steps right after uh it
sees it and write a like right before
and right after uh and then maybe like
with increasing step size after that but
i think that would be something really
cool to see
to see if like how much does an
individual data point actually move
things because from what what i'm
looking at here it seems like the answer
is the suggested answer is it doesn't
but i somehow find that surprising
so i think that's a really great
question i think they're
i'm i'm not 100 sure what you mean
mostly because i think there's a couple
different ways to interpret what you're
saying one of them is we can say okay
let's stratify this and find let's go
find the 10 most memorized data points
in the entire data set
if we plot their locations are there
locations evenly distributed
so it's it's possible the average
memorization
is
constant even if the one the location of
the one percent most memorized tokens is
not
um and that's that's something that we
have uh
that we have not yet done and there's
something that i do want to do another
thing is you know
some of these tokens show up multiple
times in training other ones don't
well actually
almost all of them show up multiple
times in training uh but some will show
up twice in training and some of them
show up ten times in training
uh no someone showed up once in training
and someone show up four times in
training there we go
uh i know how this data was processed
i was definitely there and paying
attention to what we were doing
uh so
given the evidence that uh
the
number of times a data point is seen
influences
how the extent to which it's memorized
we can say okay what if we we take the
subset of the training data that's only
seen twice and plot that and then plot
as a separate line only the data points
have been seen three times and then plot
as a separate line only that as points
have been seen four times
and what does what do those curves look
like um
and for for this specific question like
you know
if if you strike but if you are
interested in position within training
but you strike by
number of times you see it you see the
data point
um
i
don't know the answer but if you are are
solely interested in like what the
extent to which seeing more data points
influences the memorization there's
actually a really nice scaling curve
um for that
there is a paper that is that i have
seen a preprint of uh
that is being developed that is
currently under review at some
conference whose name i forget but one
of the ones
particularly stringent and anonymity
requirements and so they weren't able to
put a preprint of it online but there is
some some really interesting
uh
results about
like sk scaling laws for memorization in
the sense of
uh
how does the influence of um
how does the extent to which a data
point is memorized
uh
get influenced by
number of exposures there we go i think
that was a coherent sentence
yeah so i think
what i was trying to get at is if you
take
if you have like some unique and and you
can like you said there's there's some
things where it happens once some things
where it happens multiple times but you
know say you have just a unique
string that shows up in the data set
once um
you know just for the sake of simplicity
you can find maybe some that show up
multiple times but say you just had a
unique string that shows up once in the
data set
um
what i'm interested in essentially
is
how does the language model react like
as you snapshot the training at multiple
different uh steps throughout it
uh how do those snapshots react to
uh
seeing the the uh
the the data point essentially um
so i would expect to just surf memorized
and then become less memorized
yeah exactly right so like what i would
expect to see is i would expect a sort
of a v-shaped
profile
where initially
the model is learning to use
generalization
to figure out the
the thing that it's never seen before
which is the data point and then when it
does see it i would expect to see
something like a small drop
in its uh
you know in the in the loss essentially
corresponding to assigning more data to
something it's seen before
and then after that i would expect it to
crawl back up as that gets progressively
overwritten
uh
by more and more data that comes in
um
that is kind of my intuition for what i
what i would expect to see
so first of all i agree 100 with your
intuition that is also my intuition for
what we'd expect to see
um
there is a sense in which we are seeing
part of that plot um with the curves
that i that i can show you right now
which is that we can see from the
evaluation index of being although at
the end of training
uh the the various points along the
curve for earlier uh
data points
and
i think that
this implies
this would be a lot more easy it would
it would be difficult to
um
make consistent what's the word for that
it would be difficult to expl
simultaneously explain this and a very
strong v shape in my mind
um like i don't think i agree or
particularly
with each other and so
we came across this
oh yeah so when when we yeah that's why
i said i'm prioritizing
yeah that's why that's why i agree with
you completely
that is that my expectation i would see
something like that and we're looking at
something that says that's not at all
what happens uh so
i think that that's kind of the main
thing that that jumps out at me
so i think uh it's
when you think about a chameleon here
when a chameleon is not surprised like
the chameleon is both a democrat and a
republican and when the republicans see
something that surprises themselves
you know just give that information to
the democrat don't be surprised
um so there might be a way that the
chameleon nature of the lm explains this
sorry could you unpack that there's not
just one person there's a lot of it
sounded more like a joke than
yeah
you see some tokens that surprise you
there's not one person in the model
you know somebody is not surprised by
seeing that token and and if you are
still surprised you create another
person
so uh
who wouldn't be surprised
so
like the capacity of the element is a
lot bigger than thinking that it's one
entity that is either surprised or not
surprised
what you're trying to explain here is
why
the
performance does not depend on the
time it's been since the model has seen
a piece of data
so i think maybe to try to try to
interpret uh
the claim there it's that
um language models don't have a single
consistent what's called point of view
um if you generate from a language model
many times you will obtain extremely
different continuations if you ask it
like a like a factual question even
um you'll or especially with if you as
you vary prompting and kind of as you
vary the way that you're asking the
question um you you see very different
types of answers um and
sometimes you get the impression that
the model knows the answer to a question
sometimes you get the impression the
model doesn't know the answer to the
question and so there's there isn't it's
not like there's a person inside the
language model answering things it's a
lot more like there's a crowd and that
you are
getting a random person from that crowd
to answer the question each time you ask
it
and so
if that's how you're thinking about the
way that language model generations come
about you might say okay so maybe some
of these
uh
people living inside the language model
that that are producing the outputs um
have things memorized have a particular
faction memorized and others don't
and that
as you kind of
do this and sample randomly from the
from the model only once
and
uh
and you know for negative log likelihood
there's a pretty large explicit average
that we're computing
over the probability of generation maybe
the the averaging across these different
um i think points of view is what david
called it
um
cancel each other out and so on net
there is no signal even if uh
there is some kind of more complicated
thing going on under the under the hood
you wouldn't expect for a large mixture
of models to
have them all average out to exactly
nothing like this you would expect if
all of them are like nothing and then
one of them is something else you would
see that
and even if they are different they
wouldn't be exactly
opposite to each other
um so one thing i'm curious about is
uh
like this is sort of like the very dumb
sort of memorization where you just show
it a bit sequence and it predicts
the next 32 tokens but uh
is there like a more natural language
sort of memorization where you can like
ask it a factual question in natural
language and depending on where the
text that explains the concept even if
it uses different tokens uh shows up in
the training data does that change
so yes but i think that doing that
properly is kind of dependent on the
probing results that you are
first of all yes it is whoever
interjected that is exceptionally
difficult
um being able to do it in a in an
effective fashion is probably dependent
upon very powerful
probing results um like the ones that
igor is trying to find
um i don't think that we have the
capacity to do that experiment in
in a
reasonable fashion today like i just
don't think that it's known how to do
that
oh sorry go ahead but what i want to say
like i feel like
that that's also extremely uh
it's gonna be you're gonna see a
large amount of variance down to
prompting they all could come down to
prompting
and so
like how you prompt for the answer is
get a dramatically change
how it matches
what it's seeing
and so
i i would not be surprised to he see
that
this
is just so noisy that it's not a really
relevant result
or anything conclusive is what i should
say
yeah i was thinking maybe you could do
something like putting uh you know
putting fake facts
at
at known points into the training data
so that you sort of know what you're
looking for
um
and then and then trying to fish them
out
with prompting
um
i guess the reason i i'm sort of really
strongly expecting this to work or like
to show difference across the training
data is sort of this picture of like
learning how to learn
that people have about gpg iii
that would require for every additional
fact your input to
to retrain the entire model or perhaps
like to retain it once once you've um
input all of those facts but that brings
me to like
i would have expected that in order to
check whether the training order like
data order matters you shovel the
training data and train the model on the
new order as well and then see if they
agree
do you think they would agree
uh that is a
great question and that is an experiment
i plan on doing um
i i have personally been kind of busy
for the past month um training a
different large language model
uh but like no i think that's a really
reasonable question to ask and i think
that we should train a model and see if
that happens
all right you could think it's a good
idea
we could then do like arbitrary
uh inquiries into
how they behave and then check whether
they do the same thing such as asking
them about fuzzy facts that we really
don't know when they learned them
yeah um i think i think that
in terms of reasonably accessible ex
follow-up experiments um
the the two
things that i am personally most
interested in doing next are
shuffling the entire training data
and re-running everything returning the
model re-running this this computation
and seeing how much the results change
um
and then
well hopefully the other one wasn't
really interesting because i already
forgot i guess the other one wasn't
really interesting because i've already
forgotten what it is
there's definitely something else i
wanted to mention as a particularly
promising
follow-up experiment
but i have absolutely no idea what i was
thinking
uh
also i i guess like this is kind of
going back to uh igoro's work um
but
one thing i'd kind of be interested in
looking at is
the
logic lenses that are being looked at
um
has anyone kind of looked at i guess
their dimensionality
like how sensitive are they to
how sensitive is the predicted token to
movements
in the parameter space in some
directions versus others and what's the
ratio
of uh
of those directions
that matter to the out like predicted
token versus the uh directions that
don't matter i guess
so it sounds like you want to do pca or
or
what is that something you want to apply
it to
i guess it would be the the residuals uh
between the different layers um so
wherever we're inserting the logit lens
essentially um
one thing to look at would be
uh you know if you know just for a
simplification assume that it's a assume
that we're just the lens is actually
just a linear layer
what is the null space of that or like
the approximate null space of that
linear layer and what is it what is its
you know the opposite of that the space
uh that is directly
um you know that directly affects the
output of that linear layer and how big
are those two with respect to one
another
or is there even is there even that
space like what are the what is the
distribution of eigenvalues so to speak
of the
uh
of the the matrix or maybe some
approximation to it um that is being
learned by the logic by the logic lens
yeah so
just a couple of quick thoughts on that
i i did look at
the tuned lenses
can we interpret what the lens wood lens
is doing so if we tune a lens on the
output of layer six can we in some way
you know can we look at it i try to like
visualize the the matrix the matrix i
try to look at the eigenvalues
i looked at some in various ways but i
did
i it didn't it wasn't trivially
interpretable like basically you saw
some factors that had the most impact
and then there was sort of gradual long
tail
of uh you know of other factors
and so
so uh it it it seemed
like the the space is complicated and
you have to have the way i ended up
interpreting it is there's
uh there are various signals that you
have to sort of subtract in order to
get the prediction out
and so i wasn't able to get something
that's that's trivially interpretable so
it wasn't like oh there's you know this
there's one eigenvalue that corresponds
to the prediction of the next token or
anything like that
did you share the matrix across the
layers
uh this is where this is in the model
where you train a matrix for the output
of each layer so there's a yes
okay so you don't share them have you
considered
um trying to share it
and then seeing how much
the performance varies in that case
to see whether the
dimensions
even if you cannot interpret the mean
the same thing across layers uh if the
performance doesn't really i did show
those cards right so so i did show the
chart where you train online
there and how does it perform on all the
other layers
that's oh sorry then i misunderstood
that sorry yeah yeah but that's exactly
what the intent of of that
or how lens from one layer performs on
the other
layer
but at least
able to kind of really trivially
interpret what's in those
transformations
okay i think this is a good time to at
least call the official end of the
presentations thanks so much guys for uh
putting the stuff out there this has
been a lot of really good material and i
hope it's given people a lot to think
about
i'm gonna stop the recording here
um
but yeah hopefully see you next time and
feel free to hang out for as long as you
like