Concrete Open Problems in Mechanistic Interpretability: Neel Nanda at SERI MATS

kill
yeah that'd be too
um all right so for the people I've not
met before
um hi I'm Neil I used to work at
anthropic doing mechanistic
interpretability of Transformers I have
spent the last couple of months as an
independent researcher doing some work
on grocking and more recently getting
extremely nud sniped by various kinds of
field building and infrastructure
creation for mechanistic
interruptability and in about a month
I'm going to be starting on the deepmind
mechanistic interpretability team
and this presentation is a essentially
an ad for a sequence of posts I've
written called 200 Corporate open
problems and mechanistic
interpretability
the structure of this presentation is
going to be a whirlwind tour through a
bunch of different areas of mechanistic
interpretability where there are things
I'm confused about but I think I could
be less confused about and
trying to give an example of work that I
think is cool in that area and
recommendations for where people who are
watching this if I want to contribute
could get started
and I'll get more into resources at the
end but if you want to follow them at
home you can get to the slides with this
link
uh you can see the actual sequence of
posts with this link and I wrote a post
called concrete steps to get started in
mechanistic interpretability that tries
to give a concrete-ish guide for how to
get up to speed yep
and I recently discovered how I can use
my website to create link redirects and
use it ever
uh it's so satisfying
so
to begin I want to briefly try to
outline what is mechanistic
interpretability
uh so at a very high level the goal of
mechanistic interpretability is to
reverse engineer neural networks that is
take a trained Network
which is capable of doing well on a task
and try to understand how it does well
on that task
trying to use whatever techniques work
to try to reverse engineer what
algorithm it has learned and what's the
underlying population of the agent is oh
the model is not necessarily an agent
and one particular exciting reason you
might care about this is there are often
multiple possible solutions that a model
might learn
uh to a problem which will look about
the same but might generalize wildly
differently of distribution or break in
different ways and if we can actually
understand what it's learned we'd be
able to distinguish between these
and the focus of this talk is not going
to be on making a case for why I think
you should care about mechanistic
interpretability from a limit
perspective
um my current favorite case for apps is
this post uh requesting proposals for
interpretability work from Chris Ola
though generally I feel like someone
should write up a good personal because
it's been a while uh but my very brief
justification is I think that a lot of
the limb problems basically evolve down
to models could learn a solution that
looks like being deceptive and evil or
learn a solution that looks like Ashley
doing what we want these were extremely
hard to distinguish for capable systems
but if we could reverse engineer them we
might have a shot
and uh here's a cute iconic picture from
some work reverse engineering image
clarification models where
there's a small subset of model where we
can identify neurons that correspond to
different bits of a car that then gets
combined into a car detection that's
like if their open doors on top and
wheels on the bottom and a car body on
the bottom it's probably a cut
and
a bit on the flavor of why you might
want to do mechanistic interpretability
um is
so I think one thing that's just
extremely hard about research in general
but especially alignment research is
just
you really want feedback loops you
really want the ability to
yeah you really want the ability to do
things and get some feedback from
reality on have you done something that
was sensible or something that was
incorrect
and uh well it is very easy to trick
yourself in mechanistic interpretability
you are ultimately playing with real
systems and writing code and running
experiments and the process of doing
this gets you feedback from reality in a
way that I find extremely satisfying
which I think also makes it easier to
learn about
um it is also my personal tastes just
extremely fun
um sorry about people Matt's backgrounds
and to me the feeling of doing
mechanistic interpretability is some mix
between maths because I'm reasoning
about this clean mathematical objects
computer science where I'm thinking
through algorithms or the model might
have learned various ideas behind
machine learning and really engaging
with the architecture of the underlying
model
Natural Science because I'm rapidly
forming experiments forming hypotheses
running experiments trying to get data
and Truth seeking because I have this
ultimate grounded goal of forming troop
Leaf support model
and to my personal tastes this is just
like
way more fun than anything else I tried
in the lens and I see more about it and
I think that a thing you should do if
you're trying to get into element is
just travel to things and see what
things you find fun uh which things you
seem good at and shoot this is actually
a fairly skipping weight
and
another features that bit of advice is
that if you want to learn about
mechanistic interpretability you should
start coding extremely early and write a
lot of code and write a lot of quick
experimental code and get your hands
dirty
it's both the bar for entry for actually
writing coding running experiments I
think is notably lower than some other
fields
but I'm also just giving this as actual
advice since I often observe people if
you try to get into the field who think
they need to spend weeks to months of
reading before they write any code and I
think this can kind of triple your
learning
and a final caveat to that and passion
sales pitch is
um
I'm slightly confused because uh it
seems like way more people are into
doing mechanistic interpretability than
they were a year ago and it's like
plausible that on the margin more people
are interested in this than should be
out of all people trying to get into
alignments I'm kind of confusable to do
about this and I still want to give
talks about why mechanistic
interpretability is great how you can
get into it and how you can contributes
but I feel like I showed at least
vaguely flag that if you're almost
indifferent between another era of limit
and mechanistic interpretability
forcibly it is less neglected in other
areas though on the ground scale of
things everything alignment is
ridiculously neglected and this is
terrible and
yeah I'm confused about how to run into
that particularly sprays
um I should also say that in terms of
philosophy of questions for this talk
um the structure is intended to be a
rapid tour where ideally you don't need
to understand any section to understand
the next one and thus I'd ideally leave
questions to the end unless you think
it's important for the flow of the talk
within that structure
so the kind of mechanistic
interruptability I care the most about
is Transformer seconds the study of
reverse engineering language models
which in this case are all implement it
as Transformers a particular neural
network architecture
um I
and I care a lot about this because I
think that by far the most likely way to
get AGI especially if it happens in the
next
5 to 15 years is by something that looks
a lot like large language models trained
on something that looks a little like
transforms
and it is wildly out of scope for this
talk to actually explain what
Transformers are but I'm going to do my
best shot at giving a sub 5-minute
Whirlwind tour
so at a very high level the way that a
Transformer like gbt3 works is its input
is a sequence of words
uh technically is a sequence of tokens
which are like sub words but you don't
need to care about this
um the output of the model is a
probability distribution over possible
next words
that is we're just feeding a bunch of
text into the model and training it to
predict what comes next in that text
a weird feature of the model is that its
outputs actually has a probability
distribution over the next token for
each token of the input sequence
because the way the model is set up the
case outputs can only see the first K
inputs so it called just cheats
the key thing here is just the model
outputs probability distributions over
next words
the model's internals are represented
um by this thing called the residual
stream which importantly is a sequence
of representations
that is
after each layer
there is for every word in the input
sequence there is a separate value
internal value in the network that is a
separate internal copy of the residual
stream for that position of the sequence
um and this is a kind of 2D thing where
you have one for each input word and one
for each layer
and unlike in classic neural networks
each layer is an incremental update to
this residual string the residual stream
is a running total of all layer outputs
and each layer is just taking in
information from it and changing both in
it a bit rather than in a classic neural
network where each layer each layer's
output is just all that the model has at
that point
um and intuitively the way I think about
but intuition that I think is useful to
have when you're first learning about
transformance but it's not actually
perfectly correct is that you can think
about the residual stream for some words
as being the model's entire
representation of that word plus a bunch
of context from the surrounding words
and that as the layers go on the
representation of the residual stream
gets more and more sophisticated context
included
and
there are two types of layers in the
model that Each of which updates the
residual string the first is
attentionless these move information
between words
and they're made up of heads each of
each each layer is available with
several heads Each of which can act
independently and in parallel and each
one is choosing its own
idea of what information should be moved
and a lot of the interpretability
progress has been trying to interpret
heads
and trying to interpret heads generally
looks like trying to understand in what
ways the model would want to move
information around and how it needs
information around
and the second type of layer of the MLP
layers which stands for multi-layered
perceptron
and these do not move information
between words they act in parallel on
the model's internals at each sequence
position and they process the
information that the attention layers
have moved to that word
and combines the model consists of an
essential error MLP layer Etc stack up a
bunch add each of these incrementally
pulls in more and more context to fit
each words until at the end of the model
it's able to convert that to a best
guess what the next word will be
uh thus ends my Whirlwind tour
um I have a what is a Transformer
tutorial and a how to implement gpd2
from scratch tutorial that you should
totally go check out and the internet is
also full of other people's attempts to
make Transformer tutorials if you want
to learn this generally this is just a
thing you really want to learn if you
want to do good making tup work
there are a bunch of the problems I need
to talk about you don't actually need a
deep understanding of Transformers to
get traction on
but hope that world and tour is
interested to people
I am now going to move on to actually
digging into areas of concrete open
problems
so
the first problem area
is analyzing toy language so
bye so buy a toy language model what I
mean is you are trying to train a
language model analogous to Cutting Edge
thing like gpd3 or chat EPT you keep the
training setup of train on a load of
text and try to predict things weren't
the same
and you keep all of the architecture and
details the same but rather than having
like 96 layers like gp3 has you just
have one layer or
two or three layers and sometimes you
might want to remove
some of the additional complexity in the
model like having an attention early
model
waiters remove the MLP layers and thus
only study its ability to move
information
and
I think that toy language models are a
pretty exciting thing to study because a
they are just dramatically easier and
more trackable to study and try to
really get traction on reverse
engineering but they're also complex and
hard enough that I think we can learn
real things from them and that there's
at least decent suggestive evidence that
the things we can learn do generalize to
Real Models
a good case study of this is induction
heads so induction heads are this
circuits we found in this paper I was
involved in called a mathematical
framework for Transformer circuits where
we studied toy attention only language
models
and we found that a thing that often
came up in language was some text that
was repeated in a sequence multiple
times like there was just some there was
a form of a comment that someone quoted
further down or a news article headline
That Was Then repeated in the text or
whatever
and it's the model learns this
fairly simple yet still somewhat
sophisticated circuit where two heads
work together to say has the current
um has the current token appeared in the
past if yes look at the thing that came
after that and assume it will come next
and
we called these induction heads
uh naively this might actually sound
that exciting but these turn out to be a
really big deal
um
two uh such a big deal that we had an
entire follow-up paper on them a
uh two notable things about them is that
there's a they appear in basically all
models who studied
there's a fairly narrow band in training
when they appear and basically all of
the induction heads are pit
here's a pretty shop uh the yellow ball
is the reasonable you go from no
induction heads to having induction
heads
and most excitingly they seem really
important for this weird capability that
language models have called in context
learning where the model is capable of
using words far back in its context to
predict what comes next
um this is analogous to say reading a
book and remembering some detail that
came up five pages ago in order to
predict what's going to happen next
this is like kind of surprising that a
new look can do this and it turns out
that a doctor in the heads are just a
really crucial part of how the model is
able to track these long-range
dependencies and the narrow band of
training when they appear exactly
coincides with a narrow bounded training
where the model suddenly gets really
good at this
and a further exciting thing about
induction ads is that we've made some
decent Traction in actually propose
engineering what's happening in them I
look and try to explain this diagram but
this is copied from a group blog post
from from Callum McDougall that is
linked in the slides that you can go
pick around in and try to read about how
these actually work
um I will flag it we are not actually
have a complete understanding of
induction hits and there are additional
subtleties and complexities that that
we're still kind of confused about
but the overall message that we take
away from the section is just that
there are things that we are actually
fairly confused about in toy language
models yet they are also sufficiently
simple that we can get some real
traction on this
and the the insights we learn genuinely
seem to transfer to Real Models or at
least give us some useful insights on
them
and a particular concrete problem that I
would really love someone listening to
like go and try to take on is
one of the big open problems in my
opinion in mechanistic interpretability
of Transformers is how to understand the
MLP layers of a transformer how to
understand the bits that are about
processing the information in place
rather than moving it between positions
and I think a particularly promising way
to get started on this would be to just
take a one layer Transformer with MLPs
and try to understand just a single
neuron there
I assert they're just not an example of
a single neuron in a language model that
we can claim to really understand and
this seems kind of embarrassing
and in the post accomplishing the
section I open source some toy language
walls I train so you can just go and
play
the next area of concrete open problems
is what I call looking for circuits in
the Wilds
where that is looking for looking at
real language models that weren't just
trained for simple that weren't just
trains in kind of like tiny toy settings
and
taking something these models are
capable of doing
and trying to reverse engineer some
circuits behind
um the work that most motivates this
section is this great paper called
interpretability in the wild from
Redwood research which studied how gpt2
small can do this particular grammatical
task of indirect object identification
uh that is seeing a sentence like when
John and Mary went to the store John
gave the bag to and realizing that the
sentence should be completed with the
word Mary
and
this is simultaneously a
complicated task where it's like not
totally obvious to me how a model could
solve it but also
uh not an incredibly hard or
sophisticated or complicated task
and interestingly it's also
fundamentally a task about successfully
moving information throughout the
context because you need to figure out
that John is duplicated you need to root
the words that are not duplicated to the
final token and uh in their paper they
did this aerobic effort of reverse
engineering that found this uh I believe
25 head circuit behind how the model
does this that roughly breaks down into
using some heads to figure out that John
is repeated uh moving these to the final
token with some X extra heads because
again the model has a separate internal
representation at each word in the input
sequence and then having a cluster of
things that moves thing that moves all
names that are not repeated and predicts
are those all complexed
and
I would just be really excited to see
people trying to find more circuits in
Real Models
and uh excitingly uh every time there's
a prop area I'm pointing to where I'm
like and here's some existing work that
you could be inspired by uh you can it
saves a dramatic amount of effort if
there's some existing work that you can
Crypt from and useless awesome points
and I made this uh demo notebook called
exploratory analysis demo that
demonstrates what I consider to be some
fairly straightforward yet powerful
techniques to get traction on what's
going inside a model and use them to
speed run half red arriving the indirect
objected identification circuit
and a concrete question that I think you
could get some good traditional
answering with a combination of my
notebook and the underlying Transformer
lens library that I wrote that it uses
to play around with models and the code
base Redwood put up with their paper is
what does the indirect object
identification Circuit look like in
other models
um
the library included lets you load in
another model by just changing some
Stacks at the top
um is it the case that the same circuit
arises is it the case that the same
techniques work is it the case that
actually we need to basically solve from
scratch and there's an incredible
there's like a new deep laborious effort
every time there's a new circuit who
knows I think this is something that you
think attraction
and is a question that I'm pretty
interested
um
and I'll skip over this so so a new area
of concrete open problems training
attacks so
um
it is just a fact about the universe
that
frequently when we take a neural network
and we give it a bunch of data or some
task run and then we apply stochastic a
gradient descent to it to a bunch it
ends up with a solution to the task that
is reasonably good
um I'm extremely confused about why this
happens and I'm also very confused about
how this happens what are the underlying
acts
um how does the model navigate through
the space of all possible solutions it
could fight
is it the case that there's just some
clear canonical solution but it's always
going to end up in even if it's path
there are somewhat difference is there a
lot of Randomness
um where it could just end up at
Solution One or solution two kind of
arbitrarily is it the case that we can
influence these training Dynamics or is
it just really over determined and it
would break anything we try doing
and I think that a particularly
promising way to get traction on a lot
of these questions is to
uh train models that can demonstrate
some of these problems in a smaller more
trackable setting and then trying to
reverse engineers
and an example of some work that I did
here so there's this there was this wild
paper from openai at the start of last
year on this phenomena of grocking so
crocking is this thing where if you
train a small transformer on a some
specific algorithmic tasks in this case
modular division
and you get it
say a half of the data as training data
if you treat it on that task it will
initially just memorize the data it sees
and do terribly in there it doesn't see
that if you keep training for a really
really long time in this case about a
hundred times longer
than it took to memorize
it will sometimes just abruptly
generalize
and to underline it suddenly generalizes
despite never seeing uh despite just
looking at the same data again and again
and never actually having a chance to
look at the Unseen date
and this is a kind of wild mystery that
a lot of people in the ml Community were
interested in
and yeah this worked I didn't I trained
a even smaller model to do modular
tissue and instead really hard at its
weights and found that it was doing this
wild treat based algorithm
where it learns to think about
uh modular Edition as rotation around
the unit circle it learned a lookup
table but converted the inputs to uh
inputs A and B to rotations by angle a
times constant and B times constant
around the circle it learns to compose
those uh to get the rotation a plus b
and then to get out the correct answer C
it learns to
apply the reverse rotation C and then
look for which C got a background
stopped
and because that's around a circle it
automatically gets modded
and uh you really don't need to
understand the specific details of this
algorithm but it is just incredibly wild
in my opinion this is actually a thing
that the model looked
and if we look at the model during
training we can see using or
understanding of the circuit we can see
that
even during this Plateau
um
the model uh even during the period
before actually generalizes the model
has shown significant internal structure
to its await some representations
and can basically just use your
understanding so look inside the model
and see what's going on during training
and a concrete problem here but I'd be
excited to see someone pick around that
is in my crocking work we Dove really
hard into reverse engineering this
modular Edition
but I also exhibited grocking in a few
other contexts like five digit Edition
or
model trains to have induction heads on
a small algorithmic task and these also
seem to grow and I would love for
someone to take say the induction heads
task try to really dig into what's going
on there and see if you can use this to
understand why that one Crooks and see
how much these insights compare
all right
um
new area of urban prompts we didn't
follow the previous ones you can just
stop paying attention to get hit uh
studying neurons
so our best understanding of how models
how neural complicated neural networks
work
is there a lot of what they're doing
is they are learning how to compute
features
whereby feature what I mean is
some function or property of the inputs
for example this token is the final
token in Eiffel Tower this bit of the
image contains a dog ear uh this image
is about Christmas uh this number
describes a group of people
and
um
this is our best guess for how models
work and
I would really like to get a much better
understanding of what kinds of features
actually get learned in languages and
where do these get ones
because what seems to happen in say
image models which I said which we've
got a much better handle is that those
models seem to develop specific families
of simple features that get combined to
more and more complex features until
eventually you get really sophisticated
and high level once
um here uh some great neurons from this
paper called multimodal neurons which
studied neurons in a model called clip
which contains both image and text pots
and they found all of these wild
high-level neurons like a Donald Trump
in Europe or a Catholicism Euro or an
anime neural or a teenage neuron and
it's like what
and uh we are much less good at reliably
figuring out what neurons correspond to
and languishables but the scope I'm
aiming for in this area is more just
try to look at what neurons represent
and use this to get a better handle on
what kind of things the model is
actually learning even if this is
neither complete nor
yeah even if this is not complete snow
fully rigorous I think we would just
still learn a lot
and a pretty janky but effective
technique here is this idea of looking
at Max activating data examples that is
you pick a neuron you feed a bunch of
data through the model and you look at
which text most activated that new
and if the top 20 bits of text
um sing to a bunch of them seem to have
some consistent pattern that's
reasonably good evidence that's the
model has actually learned that pattern
is a feature
um and there's this great paper called
sop Max linear units from anthropic that
I was mainly involved in where
um one uh this isn't actually focus of
the paper one of the things they do is
just explore what kind of things are
learned by neurons and we have things
like this a64 neuron which observes that
often you get strings that are written
in base64 and that if this happens
there's a bunch of specific tokens that
are more likely to occur than otherwise
and a bunch of common words that are
really unlikely to occur
there's uh disambiguation neurons
um so this is a family of neurons which
look at the token D and observe that D
comes up in a bunch of different
languages and contexts and it's the same
kind of text in each of those but in
Afrikaans or English or German or Dutch
you want to think of it as a completely
different thing and the model seems to
learn separate features for D in Dutch D
in German
Etc
uh there's also more abstract and
sophisticated
um here's a neuron which uh in the
middle layers and much larger model
which looks more numbers that implicitly
describe groups of people
and uh
one particularly one particular notable
thing is that the way language models
work is they can only use text that came
before rather than afterwards so it knew
that say 150 or 70 was going to describe
guests despite this not occurring before
and I made this somewhat janky and under
construction tool called neuroscope
which just collects and displays the max
activating data so examples for a bunch
of neurons uh for well all neurons and a
bunch of language models and I think an
extremely low barrier of Entry project
would just be go poke around it there
and see what you can find
I'm particularly interested in seeing
what kind of
interesting abstract Concepts do models
seem to learn around the middle layers
of larger models
and I'd be particularly excited for you
to then go and try to validate this
understanding either by actually reverse
engineering something in a smaller model
or just by feeding in a bunch of texts
that you generated yourself to the model
and seeing what hypotheses you can
confirm or falsify about what text
should activate that
and just seeing how reliable pattern
spotting in text that activates a model
actually is
uh again if you go to neilmat.io cop
Dash slides you can find these slides
all of them any licks in there
all right next category open Pro poly
semanticity and superposition
so
one thing that I somewhat glossed over
in the previous uh category is this core
problem in interpreting neons
which is the we studied a policeman test
so
it seems to be the case that
while in image models neurons often seem
to represent single Concepts uh
sometimes they didn't so here is one of
the neurons studied in the multi-metal
neurons paper
and this is a technique called feature
visualization that roughly gives you
this funky psychedelic image that
roughly models
what that neuron is mostly looking for
and we see this kind of playing card
dice style thing we might think it's a
kind of game in here
but it turns out that if you look at the
inputs that most activated uh half of
them are about games or cards or stuff
like that and then half of them are
about poetry for summaries and it's
possible that there's just some Galaxy
brain thing here where poetry and dice
and cards are all collectively some
haired feature that's useful for
modeling the task that I'm totally
missing uh but
this seems to happen all of the time in
language models and is incredibly
annoying
and it's generally seems it just seems
likely to be the case that models have
just decided to represent multiple
things in the same year
or at least kind of
representing things spread across
minions
and our best guess for what's going on
here is this phenomena called
superposition
uh theater of superposition is that a
model tries to represent more features
than it has Dimensions by squashing
those features into a lower dimensional
space
and this is kind of analogous to the
model simulating A much larger model
with many more parameters and features
uh which is obviously useful because
bigger models can represent more things
and that the model has decided that
simulating a larger model with a bunch
of interference and noise is good and
worthwhile and a sensible trade-off and
if you have more features then you have
neurons you obviously can't have a
feature per neuron and you're going to
have to be compressing and squashing
them in in some complicated way
and anthropic had this awesome paper
called toy models of superposition
which looked at can we build even a toy
model that just actually provably learns
to use superposition and is it the case
that there is it is ever that is it ever
the case that superposition is
absolutely useful to which the answer is
yes yes it is it's very annoying
and uh so what they did is they just
built this uh really really simple setup
where you have a bunch of features that
are added to the model they need to be
squashed down into some low dimensional
space and here you have five input
features abaniti squashed into a
two-dimensional space and then
we measure how well the model can
recover these
and it turns out that if the features
are there all the time such that they
interfere with each other a lot the
model just learns a feature per
dimension
if they aren't there as often uh sparse
t means
agent at the time a feature is just set
to zero the model decides to squash two
per dimension
and if they're even less frequent
the model decides to squash five into
the two dimensions in this pretty
Pentacle configuration
and uh if you dig more you find that uh
this thing wasn't just a spontaneous uh
thing that occurred because uh we had
just two hidden Dimensions uh it turns
out if you make the model squash things
into more hidden Dimensions they
spontaneously self-organize into classes
of features that are each in their own
orthogonal Subspace and that if you put
things they can form energy levels
according to
um what configuration they had like some
models have tetrahedra where four
features fit into three dimensions uh
some models have a mix of triangles with
three and two and a typical pairs with
two and one
and I'm
not really going to try to explain this
diagram properly you should totally go
check out the paper if you're curious
but some open questions here but I'm
pretty excited about people exploring
the first is just this paper makes a
bunch of predictions about what
superposition might actually look like
in models when it might occur what it
might look like uh can you just go and
pick her out of the model and get any
traction on figuring out whether any of
these are
um
in particular if you start with one of
the circuits we've already got a lot of
traction on like induction heads or
indirect object identification I think
you might actually be able to get some
interesting data
and the second category of prop is I
think there's lots of things I'm
confused about to do with how models
could do computation in superposition
and how models can kind of squash more
features that they have neurons and then
apply some non-linear function to
process information and they get
something useful at
but it seems like they can do this and
the paper very briefly explores but in
my in the relevant concrete open friends
post I try to spell out a bunch of other
angles I might take and things I'm still
confused about
and yeah
um I generally think that dealing with
superposition is probably the biggest
open problem in mechanistic
interpretability right now
and I am very excited to see if we can
get more progress in
all right new category of problems if
you so doubt it's all pay attention
again techniques and automation
so fundamentally the
thing we are trying to do when reverse
engineering models is to form true
beliefs about modeling tones
forming true beliefs is hard
and one of the main things that lets us
get anywhere on this is having good
techniques having well understood
principled approaches that we can apply
that will actually help us gain some
traction and gain Summit insight into
what's going on inside
and I think that there's lots of room
for Progress here building uh building
new tools building a better and more
reliable toolkits building
a better understanding of our existing
toolkits
uh one concrete example of this is so a
technique that's pretty popular in some
Circles of interpretability is ablations
that is you pick some bit of a model and
you set this bit of the model to zero
and you check okay I've set this bit of
the model to zero
um How does its performance on a given
task change
if you set a bit to zero and the
performance tanks then probably that bit
mattered if you had a zero opponents
does a tank then probably perform it
then probably that it didn't
is a naive thing that it's very
reasonable to think that's something
that I thought but one weird ass outcome
over the interpretability of the wild
walk was they found these things called
backup pets
where it turns out that when you delete
certain important heads other heads in
the next layer or two change their
behavior to near perfectly compensate
and it's like what
so what this graph shows is that if you
look at the direct launched attribution
of a hat that is uh the effect of that
head on the output of the model
um
What You observe
um is that when you delete this
particular important heads uh so the
original effect is like three after you
delete it it's zero because it's deleted
um two other heads which are really
important just kind of stop-map it to
significantly change the behavior and go
from really big effects to fairly small
negative effects and a fairly small
visual effect to pretty big next effect
and it's like what
why does this happen I would not have
expected this to happen I'm extremely
confused I would really like to get a
much better understanding of what is
going on here and what kind of
pathological cases might exist for other
techniques I think are cool
and one concrete Urban problem here is
just how General is this backup
headphone
can you find backup previous token heads
which are 10 to the previous token uh
only if an earlier previous token has
its deleted can you find backup
induction heads or backup duplicate
token heads uh I have no idea I would
love someone to just go and look
um or are backup heads purely results of
training models with dropouts uh
technique where you just randomly set
some bits to zero during training try to
make it robust to that
I have no idea
um a yeah and the exploratory analysis
demo that I made includes a bit at the
end where I replicate the backup name of
our head results to hopefully save you
some efforts
uh I'm generally trying to emphasize
demos and tooling because if you're just
getting started in a field and you want
to try making traction it's just really
helpful to have something you can crib
off of and copy from profaneed and saw
from scratch
um also if the thing does not succeed at
being useful and you still need to write
a little things from scratch please let
me know I will attempt to fix it
um a different kind of thing that I fit
into this broad category of techniques
and automation is
automation
um one of the most common and in my
opinion fairly legitimate critiques of
mechanistic interpretability is that a
lot of these successes people have had
are
very labor attentive and maybe they've
worked on small models and maybe they'd
work in printable and large models but
where it would just take like years to
really get traction on actually fully
understanding a incredibly large and
complicated model and where the bigger
models get the mobile height the more
and more intractable it will see
and
I'm pretty excited to see effort trying
to take
um the techniques and know-how insights
that we have and trying to think through
how you could actually scale these and
automate these
um one very simple dumb example I made
is this thing that I call an induction
Mosaic where so a nice property of
induction heads is that they work even
on sequences of purely random tokens if
the sequence says repeated
and
if yeah it works on the sequence
um even if
it works on a sequence of random tokens
you got the random tokens that they use
repeater and it always attends to the
Token that immediately after the first
copy of the current
and this is an incredibly easy thing to
code and then just check for
and this is a heat map where the y-axis
is which heads the x-axis is which layer
and the color is how induction is that
hits and we can see that across 41
models there are induction heads at all
of them and we can even observe some
high level patterns like most models
only have induction heads in the second
half
apart from gptj for some reason why does
this happen I have no idea
um and I made this mistake using my
Transformer lens Library which has the
fun feature that you can change which
model you have loaded by just changing
the name of the model in the input
and I would really love for someone to
go and make go and just do this for all
of the kinds of heads we understand
um EG all the heads in the indirect
object identification circuit are the
simple times like duplicate token and
previous token heads and then just make
a Wiki where for every head in
all open source language models we just
have a brief summary of what we can what
we know about it I think this would be a
really cool thing to exist that just
wouldn't be that hard to make
and
more generally I think there's a large
amount of scope to take the stuff that
we already know and already got and to
think about how you can distill it to
techniques or how you can to sell it to
things that are more scalable and
automatable
though I do personally yeah
sorry uh sorry to interrupt there's a
question that was asked a little bit ago
uh we're a few steps removed here
um Lisa is wondering
um a paper you mentioned a little bit
ago if it could be interpreted as model
redundancy she says did this paper also
look into what makes a head count as an
important Head worth creating a backup
for I don't know if that's still
applicable or not but uh
the answer is no they had not looked
into this but you could look into this
and yep
um who knows I'd love to see the answer
to that
um
this is just another data point in the
general Vibe I want to convey if there's
just so many concrete open problems here
but I don't know the answer to I'd love
to know the answer to
um all right uh final area of concrete
Urban problems
um that's supposed to end up half past
or the hour
and only assume talks are an hour long
by the calendar event till the hour
well no one else is me I'll just assume
that it's till the hour
um all right so
but I made a wrap-up scene anyway
um all right so you're welcome to go
longer we have like a slido uh for Q a
questions as well I'll post to the slide
after you're done yeah the timing is up
to you
yeah if you could post a slider in the
slack and zoom chat now so people can
like stop putting in questions that'd be
great
unless you've already done that all
right so final area of concrete Urban
problems I wanna okay is interpreting
algorithmic models that is train a model
on some synthetic algorithmic task and
then go and try to reverse engineer what
happened what algorithm did the model
learn
and
um
I
think obviously this is even more
removed
from actually being able to reverse
engineer Cutting Edge models than the
toy language models that I was talking
about earlier so you kind of want to
check that we're doing is actually
remotely useful
um one angle is that if we can it's a
lot easier to really understand what's
going on in an algorithmic model where
there's a genuine ground truth
and this can Subs a testbed for us to
explore and understand our
interpretability techniques
it can also be useful to build toy
models to simulate things that we
believe a lot of models to be doing so
we can try to understand them
for example
um it's plausible to me that a useful
way to upon traction on the
interpretability of the wild work might
have been to train a toy model to do a
synthetic version of that task seeing
how small you can get it and see what
that model did
um and I think things like my groffing
work and the toy models are
superposition are for the things in the
speed
um but all that is kind of
rationalization the actual reason that
I'm recommending this is I think it's
just a really easy place to get started
that will help you build real intuitions
and skills
and that just pick a model on any
algorithmic task and then try to get
traction on it is just a pretty great
first project even if it isn't actually
ever really useful
um and a demo tutorial that I made is I
just um so I was curious if you don't
explicitly tell and model anything about
which position things in the sequence
are uh can it red arrive these
for sophisticated reasons
the way Transformers work is that they
kind of treat each pair of positions
equivalently unless you explicitly hack
them to care about social information
and it turns out that if you don't tell
anything about positioning but you only
let It Look Backwards a two layer model
could just learn to read or write its
own positional embeddings if trained on
a really simple dumb task and I released
a curl up notebook along with this
um with the code relevant codes but I
also decided to record myself just
trying to do research on this for like
two hours
and hopefully this is a good model to
try to build off of if you want to go
and try to interpret algorithmic toss
all right so those ends the Whirlwind
tour are the areas of open problems
um some concrete resources to learn more
my 200 concrete open problems and
mechanistic interpretability sequence
uh though I really have not been
counting and I tend to add more lines on
my edit so it could be like 300 at this
point if anyone's account let me know
get my guest
um
a guide on how to get started in
mechanistic interpretability
um I generally recommend starting with
this one unless you just feel really
psyched about jumping straight into a
natural open problem I try to give comfy
steps with success criteria and goals
and to generally guide you through the
common mistakes I see people make like
read for weeks to months before they
write any codes or uh be really
intimidated by specific things that
aren't actually that hard
diverts a comprehensive mechanistic
interpretability explainer
um which is basically an enormous brain
dump of ideas that I and Concepts and
terms the jargon that I think you should
know if you're trying to learn about
mechanistic interpretability with
extremely long tangents examples
motivations and intuitions uh I think
it's both a pretty productive thing to
just read through to get a sense of the
fields if you're in the mood for a long
haul or as a thing to just search
whenever you come across an unfamiliar
term
and finally my Transformer lens Library
which
attempts to make it so that if you are
trying to do some research on a question
that involves reverse engineering a
model like gpd2 I try to make it so that
the gap between have an experiment idea
and get the results is as low as
possible by trying to design an
interface that does all of the things I
care about doing as a researcher as fast
as possible
and anecdotally other people seem tools
they find this useful though who knows
but uh doing mechanism mechanistic
interpretability with their
infrastructure is pay so hopefully this
has helped
and yeah zooming out a bit I kind of
just want to give the high level framing
that
the spirit I want to convey in this talk
is mechanism interpretability is a
interesting a live field that just has
lots of areas where people could add
value add lots of things we don't know
but I would really like to understand
better and that the bar to entry for
engaging with these and making progress
on them is not that high
um I also want to make sure I also
convey that through research is just
actually fairly hard and you should go
into your first research project
expecting that you're going to get a lot
of things wrong you're going to make a
lot of mistakes
ultimately the thing you produce
probably won't be like
groundbreaking research and it might
have some significant flaws especially
you know how mentorship
but also but the best way to learn to
get to a point where you can do useful
work is just by trying and by getting
your hands dirty getting contact with
reality and actually playing around with
real systems and having some clear focus
and prioritization of this is an open
problem that I want to understand let me
direct my learning and coding and
exploration so that I can try to get
traction on this is in my opinion just
one of the best ways you can learn
how to get better at doing real
mechanistic interactability research and
I think we'll have at least some chance
to just doing real research in German
and
yeah
um
I try to look to a bunch of resources
um I will also just emphasize again that
it is plausible to me that too many
people are trying to do mechanistic
interpretability and if you feel
equivalently drawn to other things you
should totally go explore those but if
you're going to try to explore
mechanistic adaptability please try to
do it right and I hope these various
resources and framings are helpful to
that
all right I am going to end the main
presentation here but I'm very happy to
take a bunch of questions for a while
uh question one do I think
interruptibles your research into model
based RL be useful any cool previous
papers in this Eric
um
So my answer to most questions about do
I think interpretability Research into
area X would be useful is yes because I
think that the amount we understand
about that works is just pathetically
small and that if we can understand more
this would be great
and yeah
uh this seems like a thing that I would
really love to have more inside it too
uh if I'm engaging with the question
more from the spirit of prioritization
would I rather have interpretability
Research into model-based RL rather than
other areas uh sorry little based around
this jargon the people who are familiar
uh RL is reinforcement learning which is
the study of if you train a model to be
an agent that is taking an action on
some environments and you give it some
rewards saying you have done good things
or you have not done good things
um how could it learn strategies to get
good rewards and I believe that model
based specifically though I'm not an RL
persons this could be wrong is when the
model uh when part of the model is
explicitly designed to simulate the
environment it's in
and using that in order to form better
plans and strategies to get more reward
and yeah so
fundamentally I think that the
there are two ways I would think about
reinforcement learning interpretability
being useful
the first would just be
I do not think we understand
reinforcement learning at all I would
really like to be way less confused
about this and I think that just
aggressively optimizing for whatever
seems most attractable
um just to learn anything at all seems
great and this can look like simple
tasks this can look like
um
smaller models this could possibly
having an explicit model-based thing
will be even easier
and then the second angle of why I care
about interpreting RL is because lots of
questions and consensual alignment
fundamentally revolve around things do
with how models
do things
and yeah fundamentally around how the
models do things
and what does it even mean for a model
to be an agent or pursue girls
do models have internal representations
of goals at all are models capable of
planning who knows and
I
don't have strong takes with a model
based or model free RL is the best way
to approach that
and finally
um the actual thing that I care about
with all the interpretability research
is will this actually be useful for
whatever AGI looks like and
the popular way of using RL on Cutting
Edge models today is reinforcement
learning from Human feedback which uses
an algorithm called proximal policy
optimization or PPO and to my knowledge
PPO is not always
which makes me less excited about model
based Focus
and I'm not aware of any cool previous
papers
um I do actually two days ago I made
look yesterday I made a post on the
sequence on reinforcement learning that
tried to give some takes on how I think
about reinforcement learning in general
how I think about interpreting real
source
reinforcement learning in general which
is hopefully useful to Checker
um is the next question
is there any research or recommendations
you have to look into automating circuit
Discovery in Los Angeles
so
let's see
um there's two things that I think are
particular so one thing I'm pretty
excited about here is Redwood research
have this sequence on this algorithm
called causal scrubbing which is their
attempt to make something that is
actually automatable uh sorry that is
their their attempt to make an approach
to
verifying that something is actually a
circuit that can just be automated and
is also highly progress I'm very excited
about color scrubbing building
infrastructible scrubbing scaling it up
checking how one-up works that they're
currently running their large scale
remix research Sprint style program
which is probably going to get a bunch
of data on that but I definitely check
out causal scrubbing
and I would also
um Arthur Conley who is a researcher at
redwoods uh recently had this post on
um
automatic circuit Discovery where he was
trying to write some codes he was author
on the interpretability of the wild
paper and was writing some codes to try
to automate uh How does it go with that
circuit having to go to this that
closely but I think it's pretty cool
um more generally
so my high level philosophy of
um automating interpretability is
I think the common uh thing that I
frequently see in people getting into
the field which in my opinion is
something of a mistake is that people
seen mechanistic interpretability and
they're like this is cool but it's so
labor intensive I'm not interested in
doing a project that involves labor
intensive approaches to discover
circuits and I want to fully jump to
automation or incredibly scalable things
and I think there is a valid spirit I do
think that having things that are
automated or scalable is extremely
important
but I also think that
we are significantly more constrained by
actual true knowledge about networks and
network internals then we are
constrained by ideas for techniques that
might scale and might automatically find
things
and
in my opinion the right way to do work
on discovering automated ways to do Mac
and tup is by starting with examples we
understand like pretty well
um trying to distill this and find
techniques that could uncover this
faster
AEG are there ways you could
automatically discover the
interpretability in the wild circuits
and then it's trying to scale up these
techniques by finding novel circuits
and in the process of doing so use this
to validate how well your Technique
actually works with a bunch of more ad
hoc and labor-intensive approaches on
the side
um
his non-mechanistic interpretability a
thing uh yes I think it would be fairly
rude of me if someone left this talk
thinking that the early back the early
adaptability that existed is mechanistic
interpretability uh interpretability is
a large and thriving field of Academia
that I
do you not feel qualified to
give clear summaries of
um generally there are just lots and
lots of things that people do some high
level categorizations would be things
like uh there's Black Books
interpretability techniques which just
look at the model's input and output
and uh maybe differentiate through the
model and try to generate explanations
for why it produces this output
um
things like CLT maps are things that try
to figure out parts of the inputs
effects and outputs uh there's the
entire field of explainable AI which
often judges their metrics their metric
of success as how we produce an
explanation that is useful to users I
don't feel that excited about these
areas especially from a limit
perspective
there's also a lot of areas of academic
interpretability that is trying to
really engage with models and Top Model
latels
um
there's this good survey paper called
towards transparent AI that I just
copied into the zoom chat
um if someone could collect these links
so like put them in slack or something
afterwards that would be useful
um
yeah
um do I have nothing to say on this
yeah I mean mechanistic is kind of a
fuzzy word
um there's mechanistic as a de facto
description of the community of people
who work on what I call mechanistic
contraptability which tends to be people
in the Olympic Community it tends to be
people at industry Labs or non-profits
like redwoods and there's in contrast to
the things that actual academics and
academic Labs or people in more
mainstream bits of animal research do
and then there's the actual research
philosophy of what it even means for
them to be mechanistic or not and it's
very easy to get into stuff that's what
has
um I also recently got into a bunch of
arguments on Twitter with uh people in
more conventional academic
interpretability about what even the
differences of mechanistic stuff were
and you should go check out my Twitter
replies if you want a lot of in the
weeds discussion
all right uh what do I think about
trying to interpret well reward models
um that seems great I really want to see
compelling results trying to interpret
reward models uh I was not actually
aware there were papers doing that which
either suggests those papers operate
Goods or just that my ability to hear
about papers is kind of bad so
everyone's interested in if I've ever
read that's interested in emailing me
those papers I'd appreciate that
um generally I'm just very excited about
trying to interpret rule models
um for context the
uh prevailing way that language models
are trained with grain force of learning
is this technique called RL from Human
feedback where the human operator gives
them some feedback and they're trained
to optimize it and reward models are a
sub part of that approach where
the sorry they're a subpart of that
approach where what happens is that the
um
language model has a separate thing
called a reward model I say separate is
often just an extra heads on the model
um on the model outputs
and this predicts what a what feedback
of human would give and that the human
feedback is used to make sure the reward
model is in sync
and the model actually learns from the
reward model
which is very janky
that is the main way that using human
feedback is remotely economical because
humans are expensive in a way that gpus
are not
um also humans are slow in a way that
GPU symbols
um yeah
interpreting Role Models great probably
pretty hard
I would definitely start with some kind
of toy model in a simpler setting
um so a nice thing about the reward
models of language models is they're
basically exactly the same network but
with a different like unembetting Matrix
at the end and so hopefully a lot of the
interpretability work will transfer
uh one fairly ambitious project that I'm
Keen to see someone do is to take a tiny
model like a one layer language model or
a four layer language model and try to
simulate RL from Human feedback using I
don't know a larger model to just
automate what that feedback should be
you probably don't even need g53 I'm
sure gpd2 could do that
and I totally not booked out of my head
but like the right task here is ETC but
I expect I'd be excited to see what
comes out of this
um
cool next question hey do you expect
larger models to outgrow the need to
create polysomantic neurons
um my yes is that
um also I observe the question below
that says what makes interpretability
research less promising for alignments
uh I don't really understand what this
means and it'll be useful if you could
put a clarification on the zoom shots uh
in particular what I said previously was
that I think uh Black Box explanation
based interpretability research is less
promising for alignments rather than
acceptability research in general
but you might want to put a
clarification before I get to it
um all right case question do you expect
larger models to outgrew the needs to
create poly semantic neurons
um
so
I think there's two questions here that
I want to disintegrate
the first question is
should a more capable model that also
has a bunch more neurons
no longer need polysomatic neurons and
I'm like no uh
seems extremely unlikely
um
the the
there's this useful intuition they're
exploring the toy models paper called
the feature importance curve where you
can imagine just having a graph where
the x-axis is just the set of all
possible features that it can ever be
useful to model about inputs from things
like this text as an English versus
French to I don't know
um the content of Neil Landers Serene
Matt's talk
and
um
yeah so
a thing
this basically seems like it should be
not quite infinite but like
incredibly long tailed
and the features that are further and
further outs are less and less useful
and less and less frequent but it's all
better than nothing
and so we should expect that if models
can they're going to want to represent
as many of these as they can
and this seems like a thing that's still
going to result in placement just
if you mean holding capabilities fixed
but giving them way more parameters can
they outgrow the need for polysomatic
europes I don't really know
um
this is also just pretty hot too because
if you give something more neurons
you're inherently giving it more
premises uh but a thing that I'm kind of
curious about is what happens if rather
than having
four neurons per residual stream
Dimension the model has like a hundreds
uh entertainingly if you look at the 11
billion premise and model T5 for some
unknown reason they trained it with uh
65 neurons per residual string dimension
I have no idea why actually 64. I have
no idea why but it might be an
interesting thing to study to see if
things are less policematic then though
T5 is an encoder decodable which are a
massive pain for other reasons
all right next question can we rule out
that the neurons that seem weird EG
poetry games are just being looked at in
the wrong basis of the layer activation
space
so
I do not think that we currently have
enough evidence to confidently roll that
out
though I don't think it would be
incredibly hard to at least collect some
evidence for that in a one layer
language model
um a conquer different problem so I'm
watching this might be a contract
um so
yeah the so the two things you want to
disentangle here are is it the case that
the model has uh as many features as
Dimensions but rather than having a
feature Direction correspond to a neuron
it corresponds to some weird things
spread across a bunch of neurons
and
um I don't know
and then on the flip side there's the
question of of them all features than
Dimensions such that even though I've
already wants to align them with neurons
they just cut because there's more than
more features than neurons and my guess
is that if there isn't superposition the
model is just gonna really want to
yeah it isn't super possession I guess
as the model just wants to align
features with neurons because in the
model has features framework the thing
that we're expected to happen is that
the model
the thing that we're expecting to happen
is that the model wants to be able to
reason about each feature independently
without interference between the two
and uh the way that non-linearity is
like a brother and jelly worked is that
each neuron is affected purely
dependently from the others but that as
you combine them but that if a features
if there are multiple features in a
single neuron then those features will
dramatically interfere with each other
and this is a pain
um
but yeah this is all kind of just
conceptual reasoning I think that the
toy model is a superstitioned paper is
like pretty good evidence that
superposition might happen but I want
someone to go and check
all right there's a sudden extremely
popular question on can mechanistic
interpretability be used for detecting
deceptive alignment in models
so two possible ways to interpret this
one way to interpret this is can we use
it right now to detect a steps of
alignment and I'm like uh
Maybe
probably not we're not very good at it
uh we might be able to like get some
insight into what's going on by just
looking at the model on a bunch of
example prompts and poking around to see
if we can get any traction but like I
don't know it seems hard man
um if I interpret this as do I think
that we could get to a point where we
can use it to detect a substable items
uh yes I feel pretty excited and
optimistic about this
and I think that the worlds where
um mechanistic interpretability is
extremely useful is Worlds where it can
help us get traction on detecting things
like this the
key path would just be
getting Goods at uh truth tracking and
scalable ways to
sorry getting good at truth tracking and
scalable ways to detect which bits of a
model are most relevant to producing
specific outputs and finding specific
circuits Within These putting the model
into a context where it could be being
deceptive and using these to try to get
traction on what's going on and why and
hopefully within a few years the field
will be in a much more sophisticated
point where that story seems kind of
boring and nice
and
I think that there's a bunch of ways
that you could raise the stakes and push
back on this and be like but if the
model was truly deceptively aligned then
couldn't it just think it's thoughts in
a way that couldn't be mindfulness and
I'm kind of skeptical of this just
because I think the
models get good at being deceptive
because they have a bunch of feedback on
what they output which outputs are good
versus bad but they have basically no
feedback on their internal
representations
how legible they are and how much they
can be interrupted
um it's possible there are some high
level strategies and
I do expect that say gbt4 will have red
so the circuits papers and known
something about interpretability
and maybe there are strategies that a
bottle could use to intentionally shape
its own thoughts to be harder to
interpret in a way that breaks these
techniques
I think this seems significantly harder
than any other form of
braking or ability to detect deception
but hard to rule out
anyway I think a lot of this depends on
like how good we get which in my opinion
boils down to a bunch of pretty hard
unknown scientific questions
um
there is a fun question from Jesse
Hoogland on William moose contrarian
ideas or disagreements with other
interpretability researchers
yeah
so
let's see
specific
things come into mind
um
let's see
I'm
if I'm focusing on Within mechanistic
interpretability
I'm pretty bullish on
um I'm pretty bullish on just looking
really hard at toy language models and
just I think that we're just pretty
confused about how to interpret even a
one layer model
this seems bad and I think that solving
this will teach us a bunch of things and
a lot of other people in the field seem
excited about logicals and uh
uh on the flip side I am inherently very
skeptical of any toy model work that's
trying to explicitly construct something
that is trying to model something useful
about a language model
um analogous to anthropics playable
and I'm broadly convinced that their toy
models or superposition paper was good
and actually tracked a thing about Real
Models they have a later one on toy
models of like excellent memorization
and I'm not actually very convinced that
that one track something useful about
Real Models
um
another disagreement I have is I'm
pretty bullish on the idea that models
actually learn
clean legible Eternal structures that
form circuits and that we can actually
find these and understand these
and Adler this is definitely a thing
that's strongly distinguished it as
macinturp from the rest of
interpretability but even within
macintop I never say some researchers at
Redwood who I think are pretty skeptical
of that perspective
um
I also
let's see I'm generally pretty skeptical
of anything that boils down to uh
explicitly put interpretability tools
into the loss function
because I just think that having uh that
is actually do grading descent on
something that involves an
interpretability tool because I think
that having a tool that is good enough
that we can robustly optimize against it
just seems wildly unrealistic
gradient descent is small to the view
all
um
and a lot of the things I'm excited
about look more like enabling better
limit work and auditing models and
noticing when they go wrong rather than
being just a fully fledged alignment
solution where you can just train on
that don't be unlike our evilness metric
and just win
um
uh do I have other disagreements
I think it's
I think
one idea I've heard badied around is
this idea of enumerative safety this
idea that
um a way we can make models safe is by
solving something like superposition a
new reading every feature represented in
the model and using this to understand
to just check like are the features
we're concerned about like deception or
goals or situational awareness in them
and I think this would be pretty cool if
it works but it doesn't seem at all
necessary to me to get this to work to
get useful things out of
interpretability and I think some people
describe me all that
um
entertainingly I don't think my work on
grocking was like that useful for a
limit and I know some detachability
research is just agreement on that which
is hilarious
um I really wish I agreed with them
because I feel way better about all that
work
um
I basically just don't think crocking is
that great a model of things that happen
in real netbooks though I think there
are some cool and useful lessons
and hopefully pretty good field building
um cool I will end my list of contrarian
ideas and discriments there
um all right I will probably wrap up but
Yep this was fun I will reiterate that
if you found this talk interesting or
inspiring or just want to prove me wrong
when I say that people can do my content
research you should go check out my
Concord Urban problem sequence and that
you should also go check out by getting
started guides and I'll put a link to
the slides again in the zoom chat
but yeah
then several for coming and for all the
good questions
and
thanks to Walter for the paper healings
on the chat
did I hear a final question
oh yeah I'm also giving this talk in
some other places so feedback on it
extremely well
thank you