Infrastructure - Garcon [rough early thoughts]

well this video is going to be an
experiment uh in interviewing
uh some some some members of anthropic
about some of the infrastructure that we
have here um that's maybe interesting
and um both might be uh internally
useful for people to hear about um and
maybe we'll we'll experiment with with
sharing this externally
and
uh i i'm excited to to start
um
by talking about a tool called garcon
and i think it's a particularly
interesting example because usually when
we think of you know serious you know
systems infrastructure um in deep
learning we're thinking about training
or serving models
um but it seems like as models are
getting larger we also need
infrastructure um if we want to do
safety work or if we want to do
interpoly work and that's kind of um
maybe a little bit new um as these
models are getting really large but it
um it seems like it's a really important
thing
to invest in
and increasingly important if we want to
do that that kind of work on the largest
models
um so we're lucky to have
nelson alhaj
who
created garcia and also catherine olson
um who's a user of garcon
to go and answer some questions about it
for us
um
and just to start off um nelson
what would you say is the problem that
garcon is solving
yeah the the
main problem we built garcon to solve is
the problem of doing
uh sort of interpretability work of of
kind of the work of trying to understand
at various levels what's going on inside
these models as as they work kind of how
how they do
the the things that they do what kind of
computation they're doing sort of to aid
that work
um and in particular on the some of the
large and very large models that we
build at anthropic many of which are
are too large to even run on or run
efficiently at least on a single gpu
maybe i can also come in with kind of
the the user's perspective is before
there was garcon i could only work on
models uh that were big enough that i
could run them just in the um collab
notebook i was working in so you sort of
load the model within the notebook uh
and be getting its activations there by
sort of reaching into the torch
internals and grabbing activations out
and so now that there's garcon i don't
have to have the model on the same
box that i'm using the collab notebook
to analyze the results it can run on a
much larger box somewhere else
and that has a lot of great advantages
for me
so just to unpack that a little bit more
why why does it break down when we try
to do interpretability on a model that
that can't run on a single gpu
um or yeah i guess so
yeah so i think i think the um
well the the sort of
unique thing about interpretability work
i think sort of unique to it is is that
rather than you know kind of just
running the model you know
passing act passing tokens through the
model and getting getting out logits or
sampling and then you know for training
we do that and then we you know back
propagate we take derivatives and do
parameter update uh interpretability
work we really want to look inside these
models in very flexible and dynamic ways
we'll
we'll run models where we want to
capture the output of a particular
attention head or we'll run models where
we want to capture
you know some statistic maybe the l2
norm or something more exotic of every
attention head at all of these random
kind of intermediate points inside the
models
um
and so
the as as katherine related to you the
easiest way to do this in a lot of
frameworks is to just load the model up
into your python environment you know
have the model running on the machine
where your python is running and then
pi torch and tensorflow give you various
ability to
install hooks and and look and kind of
poke at the guts of the model
uh but when the model is like too large
to fit on a single machine
then you can't just like load it up into
the python
interpreter that you're running because
the model doesn't fit in a single python
interpreter it needs to run inside some
kind of distributed system
that is running parts of it and then you
know using pipeline parallelism or other
techniques shuttling activations between
gpus or even between different uh
host nodes entirely in order to get out
results at the end and so
you you can no longer just have that
running inside your python process that
you're doing experiments and exploration
in
so yeah so it sounds like
basically in order in order to work with
these large models you're forced to go
and use these distributed systems and
they're just not really set up but by
default to go and make it easy to access
these
um the various things that we want
that's right those sort of the the
your your these large models force you
into these distributed systems because
they're they're too large to fit to fit
the weights into a single gpu you could
potentially add a huge efficiency cost
run them on a single gpu by shuffling
weights in and off the gpu but kind of
no one does that and so the
the systems that these models are built
in often don't make that easy and again
the the efficiency cost of doing that
would be kind of truly huge and so
you want to run them across many gpus
and
that basically forces you into doing a
distributed system the people who these
distributed systems
traditionally are built with the goal of
either training the model or
of doing you know the sort of relatively
narrow end set of end-to-end tasks that
the model is designed for you know
usually sampling or maybe a
classification problem of some sort
and the the interface they give you is
you send tokens in one end and the model
you know sends tokens out the other end
or sends classifications or whatever
else the model is doing these systems
aren't designed to
load the model and then
kind of do arbitrary introspection of
them
while it's running across these this
distributed system
okay so garcon is going to allow us to
to go into interpretability
on these models what actually is the is
the api
that you provide for
users to go in and try to interact with
these models
yeah so the the basic api that garcon
provides uh
is that you can you can launch a garcon
server using about you can give it kind
of any of the models that we've trained
here
you know give it a path to a snapshot
and it will automatically figure out the
relevant parameters and and launch an
appropriate number of nodes with an
appropriate number of gpus
and then you have a
an rpc interface where you can you can
connect to that server from anywhere
else in our cluster typically from
a colab notebook or a jupyter notebook
but potentially also from automated
scripts
and then
kind of given that connection you can
run the model you can run forwards or
backwards passes by giving it tokens or
for backwards paths giving it tokens and
a loss function
but then additionally the model
exposes a sort of number of these named
probe points
that you can attach to
and uh you can send arbitrary snippets
of python code that the model will
run
at that point in the model so you know
maybe between
layers three and four or
immediately before computing the
non-linearity for
the mlps in layer 17
or
you know uh with the value with the
value vectors produced by a given
attention head you can you can kind of
name any of those points in some way and
then as the model is going it will
inject a snippet of python code that you
give it and that python code has sort of
two important capabilities it can
alter the value if it wants so you could
zero it out or you could replace it with
some other value to sort of test some
some permutation
and it also has access to this save
context object it's just it's given a
handle to this context that it can write
arbitrary properties onto and anything
written onto those saved contexts from
any of these probe points
is then accessible by another api call
to retrieve the value of all of the
saved
saved values written by any of these pro
points and so
this gives you a relatively low level
but flexible interface to to pick what
sets of points in the model you want to
inspect either to mutate them to save
their values or to do computation on
them again maybe you just want to
compute an average or something that's
much smaller than the full tensor
and save that value into a place where
it can then later be accessible from the
client it's a separate api call to pull
down all of the saved values
and then once you have those those those
probes set up you
run tokens through the model you give it
you know a piece of text that you want
to to
run it on and it will run execute all of
those probes in the very in the
appropriate way
save the values change the behavior and
then hand you back the the resulting
computation typically the logits at the
end of the model or maybe some other
value if the the model has a different
kind of uh value head at the end
and also all of the saved values
perceived by any of those little
snippets of probe code that you attached
to various points in the model
so to recap we can run the model um and
do both forwards and backwards passes
and then we can at various points either
save or modify values um
that we want to to inspect or fiddle
with and get the the saved results back
for for a later analysis yeah and as a
user this enables a variety of types of
experiments some as you were saying more
just inspection type of sort of going
example by example and seeing in great
detail what the activations were you can
do more statistical inspection type
experiments where you aggregate over a
vast number of examples and you can do
these ablation type experiments where
you zero out values or otherwise mess
with them and see what the causal impact
is on the downstream computation
yeah catherine i'd love to hear just
like more about your experiences using
this um
uh like so you were sort of mentioning
earlier that you know obviously it
allowed you to go and and and start
doing doing research on models that you
couldn't beforehand um but yeah what
what has been your what's your
experience been like and maybe what have
been the highlights of using garcon yeah
i think there's there's sort of two
different things one is this very
interpretability related boost which is
being able to collect these activations
very easily
and in sort of high definition
um to sort of see very concretely like
how is the model responding to different
inputs at different levels um but i
think one thing i've been surprised by
is that even when all i want are these
sort of per token losses which isn't
getting very deep into the guts having
garcon as a framework makes that kind of
work a lot easier as well so the most
recent batch of experiments i've been
doing
is looking at the sort of per token
losses on many different examples of
many different kinds of models at many
different points in their training and
previously i would have had to sort of
load each model one by one
and sort of inspect it one by one which
makes it very difficult to do
this big scan or big spread over model
size and
training maturity uh whereas with garcon
i can just spin up a fleet of
dozens and dozens of garcon servers for
different models and different training
points um and sort of run against them
in parallel with a lot higher throughput
and so i can ask a question like um you
know how is this example
understood or interpreted by these
hundred
different models and snapshots uh and
and get that answer quite quickly um
because it allows it to be paralyzed and
not be sort of stuck on my on my one dev
box
so something i feel like you're
um
you're you're you're getting it here is
that
garcon was kind of created to make it
possible to work with large models but
um it sort of incidentally made it easy
to work with large numbers of models um
right because we you no longer have to
have the the model on
on your the machine that you're on your
dev box that your your notebook's
running on you can suddenly go and
instead have it um you know connect to
100 models or something like this and
run experiments in parallel on all of
those and it sounds like that's been um
an important change to your your
workflow and what you can do
yeah i think to summarize and just and
just say like sort of two related things
it was designed for large models and i'm
currently using it for very small small
models in many of them and it was
designed for
these sort of inspecting and
manipulating the internals and i'm
currently using it just for the
end-to-end properties just that sort of
um
separation
of
how the model is is being run from my uh
analysis environment has allowed a lot
more flexibility in ways that i didn't
necessarily expect would be so uh
load-bearing
something else that makes me think of is
um nelson you've sometimes talked about
having ambiently available garcon
servers um as an as an interesting thing
and i'm wondering if you could comment
on that
um
yeah i mean so we
we have a couple we sort of have a a
number of sort of well-known models that
that are kind of fully trained from
various scans we've done that we do a
lot of our work on and
for a number of those we just keep a
garcon server constantly running um
with those models loaded accepting
queries from from multiple users because
a garcon server can can handle multiple
concurrent users um
and i think for a lot of especially
quick one-off experiments uh that just
means that you can start a notebook and
connect to one of those well-known
ambiently running servers
and
with
kind of both like zero kind of cognitive
like i have to go push the button or set
something up cost and also with zero
waiting because these servers are always
running you can just be connected to a
recent fully trained model that we use
and
ask questions about it poke things into
it and that i think really lowers the
cost of doing
you know especially just one off or kind
of i have a super quick question about
how these models behave in some place
experiments
um
whereas previously you'd you know you if
you wanted to load a model into a
notebook you'd have to you have to
download those weights and load them
onto some gpus somewhere
and there's both just sort of some
amount of ceremony of like you know
typing out the code to do this
and uh
just waiting time of of shuffling those
those weights around and getting them
loaded onto a gpu
yeah i think it was kind of magical the
first time i used garcon and
uh it was a model
working with models that i had
previously worked on that they could fit
on one gpu but i was used to having to
wait several minutes for the weights to
go and download um and have that in my
feedback loop and suddenly i didn't have
that was
um
was pretty amazing
so chris i think there's a there's
another sort of type of investigation
that's sort of less top of mind because
it's not what i've been doing for the
past month or so which is sort of
interactive visualizations and
interfaces that was also made a lot
easier with garcon so this is a kind of
investigation where we might be trying
to understand what one specific neuron
does and so we create a custom
interactive web page where i can enter
text and see what the model's
predictions or what the neurons
activations are on that text um using a
sort of javascript interface that i've
thrown together
and without something like our song then
i need to be setting up a server to hold
the model and be the backend for these
uh interactive queries whereas with
garcon it's very easy for me to think
just about what that web page uh
interface should be and it'll send
queries to a garcon server in the
background that i don't need to sort of
worry about how exactly that's being
designed or or hosted
yeah that's a really interesting
observation so um i guess it's i guess
what we're saying is
uh garcon can both be useful to
um and especially these ambient garcinia
servers that we sort of know are always
going to be there are both useful to to
and you know somebody who's doing
interfaith research directly from say a
jupiter notebook or to support um
interactive visualizations um and be the
back end for interactive visualizations
that you you might be making where
you're creating a website that's going
to go and create some visualization and
you need to poke around at some large
model that's right
and i suppose that also uh like plays a
again the the benefits of being able to
work with large models are are turning
up there as well where uh like it's also
probably the case that you couldn't have
created those interactive visualizations
for really large models that don't fit
on
on a single gpu or at least a single box
prior to took our saw
exactly because what that website is
having to do is it needs to ask some
server somewhere please tell me the
activations of this neuron in response
to this user entered text and in order
for that query to get responded to uh
needs to have some
thing running this large model that can
get this specific activation
and that's exactly what garcon is
designed to do
so i wonder if there's any other
uh like big classes of types of
experiments that garcon has unlocked for
us
well there's also the batch job so i'm
running some that are a bit like this
but uh nelson has done this too of we
have a newly trained model let's collect
for example a data set examples for
neurons in this model um let's just run
tons and tons of example uh contexts or
sort of text snippets through it and uh
identify for individual neurons uh which
examples does it most prefer or fire
most strongly on uh and these kinds of
background jobs are then very easy to
launch
nelson's written a little wrapper where
you can easily write a script the script
will spin up its own garcon server make
a bunch of these statistical queries uh
over the server and then spin it back
down and so that means you don't need to
sort of personally babysit or run on a
dev box these kind of generic
statistical
uh analyses that we that we want for
most of the models that we're studying
yeah that's a really valuable kind of
kind of job as well
um
and maybe just to give this is sort of
we've been talking about some of maybe
some of these more exotic things but a
lot of the bread and butter has been
just like you know manual inspection in
a notebook and i feel like earlier on
catherine you were sort of like alluding
to like some specific things you might
do like
collect the activations of a of a neuron
in response to some example um or uh
ablate some neuron and see what happens
i'm curious if you like could just
enumerate some other like very standard
common types of experiments you you
might do
um and also mickey i mean your
background is neuroscience maybe maybe
that's kind of a fun analogy for you to
make also here to like you know what
like
it i don't know
there's probably some analogy to like
the kinds of experience you can do um in
neuroscience and the kinds of
experiments you can do having access to
a model through this kind of kind of
mechanism
yeah i think i think the way i'll answer
that question right now i mean i've
alluded to a bunch of kinds of
experiments but i think of them as sort
of lying on a sort of a
scale of analysis
continuum you know are we analyzing one
neuron one attention head in one layer
are we analyzing an entire mlp layer or
an entire you know collection of
attention heads or are we looking at
sort of all all layers that's sort of
what i might think of as like a
neuroanatomy level so i've described a
little bit kind of taking one neuron
seeing its activations on a specific
text you can do one neuron its
activations on many example texts these
are sort of like single unit studies but
then you can go all the way out to more
like the neuroanatomy study and do some
kind of
dimensionality reduction mapping
exercise we've played around with umap
and pca to understand
how are these neurons or how are these
attention heads
similar to one another how do they
cluster in their behaviors
are there different types do they group
together how does that um
sort of representational space of
selectivities change over layers these
kinds of uh an anatomical like
experiments i think sort of fleshes out
the other end of the continuum of what
we've tried to do
yeah that makes sense
i'm noticing so far we've you know we've
talked about hooks and we've talked
about getting access to activations and
and running fiddling with things and and
such um but i suppose console also gives
us access to parameters
um
and that's that's important for
for going and doing work like circuits
um or for
um
i mean it's sort of giving us the the
connectum of the model or something like
this
um nelson could you talk at all about
how that's implemented
yeah i mean i think i think the the
parameters are a little
interesting in some sense in that in
that
uh you know the like they're
they're kind of inner you don't have to
run the model they're just they're just
data
and
and kind of it's you know in some sense
you know the model
downloads them from wherever it stores
them and kind of you could always get
them from the same place kind of just as
easily
um but in practice it's like far more
easy for
the garcon server to be able to hand
them back to you because
you
you know you're already using it for
other experiments it already has them
loaded into local memory and it doesn't
have to decode them or to kind of do any
any work with them
um the the
sort of under the hood getting at them
works the sort of same way everything
else in the garson server works which is
that
the the model is running on this this
cluster of processes and the distributed
system kind of one of them you know sort
of rank zero the the kind of first one
in a somewhat arbitrary way
is doing the actual communication with
the garcon client so it it just talks to
all of the ranks says hey who's got this
parameter every rank set either sends
back their value of the parameter or you
know not me it aggregates all of those
and then sends it back to the client so
it it you know transparently deals with
the fact that
that you know when you ask for a
parameter it's on one of the many gpus
backing this model but you don't have to
know or care which gpu
um
i think one
one feature that like garcon is sort of
lies it
does make some effort to do this kind of
efficiently so that it can
shuffle these parameters which
themselves can be pretty large in a
large model of you know
a single weight matrix might be might
itself be a sizable chunk of data it
goes to some care to shuffle these
effectively so that they're
you know moved around at a you know
some decent fraction of you know the
theoretical network bandwidth doesn't go
kind of anywhere to the care that
training jobs which are often like
meticulously crafted by practitioners to
use every last ounce of performance that
the hardware has to offer garcon kind of
pales compared to those but we've gone
to some effort to make it efficient
enough that the sort of
most things you might ever want to do
interactively where possible kind of
come back you know before you get bored
or change the tab
uh which which turns out to just be like
a real threshold in usability of if you
can grab these parameters and then they
come back before you feel the urge to
check twitter it actually like
you know a 2x difference in performance
if it sort of comes in before that tab
away to twitter thresholds can turn into
a like 10x perform you know 10x
improvement in your efficiency of
actually running experiments
another thing i also want to reply to
chris is that you were you sort of
pointing at what experience can you do
with parameters instead of activations
and sort of connecting back to that um
neuroscience language i was describing
the spectrum in sort of this axis from
single unit studies to more like
anatomical studies and i think we can
put this in kind of a two by two right
you can also draw this axis is are we
doing um sort of selectivity and like
behavior of the neurons in response to
different texts or are we doing
connectivity and so i think the
parameters give you those connectomic or
connection related studies which can
also be at sort of both
scales you know we can ask questions out
of the sort of anatomical connectivity
level of like in general statistically
neurons in which layer connect to
neurons or attention heads in which
other layers overall or we can get the
circuit-based connectivity study of
saying this particular attention head
that has an interesting behavior that we
want to study which specific other
neurons does it connect from which
specific other neurons does it write to
and how do those connections
mechanistically implement the behavior
that we're studying or how do they
mechanistically contribute to ongoing
behavior so i think that sort of
scale of analysis and then
selectivity versus connectivity spectrum
gives us like a pretty good map of the
kinds of work we might do here
yeah that's a that's a nice
two-dimensional space
uh
so okay so diving into the technical
details a little bit more
um uh
you've sort of talked um
uh nelson you sort of talked at a high
level about the about how how garcelle
works with the rpc server um
i guess i'm i'm curious if there's any
sort of non-obvious design decisions um
that you'd want to share or
maybe maybe another interesting way to
frame this question could be like um you
know if you if you were going back in
time to like the version of yourself who
was about to start going and toying with
were they implementing garcon um you
know what are what are the things that
you would want to tell that version of
yourself
yeah well maybe everything was just so
straightforward that it really is just
you know go and put an rpc server on top
of these things
yeah i think i think there's not
all that much complexity there i i think
sort of the the like the biggest thing
here is just sort of
deciding that this is a problem worth
solving and and putting the pieces
together
um
garcon is
uh currently uses a super lightweight so
so we use we use python's pickles and
there's a module called cloud pickle
which allows for the pickling of actual
python code objects on the fly which is
what we use to
like let you write a custom function in
your notebook and then send it to the
server to be executed
um i think when you're in python land
and especially if you want to be able to
ship code from one place to another
those are the obvious choices
but then for the actual
rpc
framework i
built a super lightweight kind of
minimal binary protocol to handle the
the framing of requests and responses
um
in retrospect if i started over i think
there's a decent chance i would just
encapsulate it all inside something like
http
um that would add a little bit of
complexity in some sense of http
libraries or http is huge and complex
and those libraries are large but i
think it's sort of
would take away complexity of writing
your own little network protocol and and
use something very standard
i think
efficiency as i mentioned is a little
bit of a concern i think that it would
be perfectly possible you'd have to pick
your libraries again slightly carefully
be perfectly possible to get adequately
efficient encapsulation of the tensor
data inside the http container so i
think that's one choice that i made i
think just out of eagerness to do the
sort of smallest possible thing that
would work i i rolled my own
a little protocol there
um i think using something more standard
would have been a little bit more work
up front but would have avoided a couple
of pitfalls i ran into
um
i think one other piece of of i think
was not obvious to me at the start
architecture is
you're you're running these models on
multiple servers kind of what's the
relationship between all of those
servers and who runs the rpc server and
who talks to the client
i think the architecture that we settled
on is is i think
pretty standard
if you're building these kinds of
distributed these kinds of
machine learning distributed systems
which was novel to me which is sort of
all of
all of the
all of the nodes surveying that are kind
of running the model
are sort of running in lockstep kind of
all running effectively the same code
and then
having some synchronized synchronizing
network communication operations with
each other that sort of you know rely on
them all running the same thing in the
same order except for one rank
runs the rpc server and talks to the
outside world and then when it receives
a request it broadcasts to all of the
servers over the kind of internal
like cluster communication that's used
for
uh
tensor and communication and then kind
of all of the ranks execute the rpc call
in parallel
and coordinate with each other as
appropriate and then the results are
gathered back to that designated rank
which then
takes them from gpu memory or kind of
you know the internal cluster and
marshals them out to the rpc protocol
um
i think in practice this is a pretty
standard architecture if you're if
you're building something to
run sampling out of a large language
model you end up with a very similar
architecture um but it wasn't
immediately obvious to me because i
don't have i didn't
have a ton of experience with these ml
systems before building garcon that that
was the right way to go
i had i i didn't kind of figure that out
i had advice from
excellent excellent other team members
who have
far more experience building these
systems
we are indeed lucky to have lots of lots
of excellent
colleagues um is there anything
interesting to say about why we went
with hooks
um so i guess the the framework is that
we have you can attach hooks to these
points and they save things and they get
them back but you know you could imagine
a simpler system where you just say you
know we're going to record from all
these points and you can request to get
the get those back um that might be a
little bit tenser flowy and maybe you
could have like a
mechanism for overriding them or
something or perhaps there's other
setups you could do and i suppose this
is just sort of how pi torch does it so
maybe that's that's part of yeah i think
it was
it was inspired by how pi torch does it
but i think
that sort of it was kind of the
the the simplest or most minimal
approach that we could come up with that
um
satisfied sort of the following set of
desirable properties i think sort of we
um
so we needed to be able to both um
collect data from tensors and to alter
the flow of data inside the model we
want to be able to
uh you know ablate things by setting
their activations to zero or s you know
swap in activations from somewhere else
and so if you want to
to have this either be able to save
values or to change them i think it's a
it's a relatively simple mechanism to
implement at least to just like have a
piece of code that can run there that
has access to the tensor and can
return a value and also has the ability
to save it somewhere you could imagine
having two different mechanisms for
those and i think that would be very
plausible but this was sort of a simple
one that unifies those two desires
we
only wanted to
save data that someone actually wanted
the especially with a large model
there's just a sort of enormous number
of intermediate states that you might
want to inspect and saving all of them
would be very memory intensive
and you know
even if you have that memory
uh for sort of running a single example
through sometimes for performance you
want to run with a batch dimension and
and put lots of examples through and
then that just kind of scales everything
by a factor of n and we'd like to be
able to kind of push that as high as we
can and that means just you want you
want to save only the values that you
want which means that you need to
declare ahead of time somehow the values
that you want
another i think desirable property is
that as i mentioned sometimes you don't
want to just collected
an activation it's it's suffices for
your experiment to collect some summary
of some activations you know the the
mean of some tensor or the l2 norm or
you know the eigenvalues or singular
values of some some values at some point
um having an interface where you push
code to the server
lets you do that lets you a
shard that computation among all of the
ranks of the model so if you're
collecting like l2 norms of activations
across every layer
then that computation gets to make use
of the fact that you have n gpus running
this model to run more efficiently
and then also
by doing that computation kind of at the
point of
of accessing the model you save you save
a much smaller amount of data than if
you saved the entire tensor and they did
the computation later
and
definitely the thing you want to avoid
is
saving activations sending them to the
client over the network link because the
client server network link is
comparatively slow and so if you want to
do aggregates or statistical analyses
you want to do them on the server and
ideally you want to do them on the gpu
that has the data to minimize copy to
minimize communication
and if i could just jump in with like an
example of one of these aggregates that
i mentioned before like i think the data
set examples uh use case is a good one
where uh for all the neurons in this
model we'd like to know which 10
examples they fired most strongly to and
you definitely don't want to do that by
running 10 000 examples through
collecting 10 000 copies of or ten
thousand versions of all the activations
sending all that to the client and then
compressing it down you really want
while you're running the model to be
keeping track of just the top ten so far
um and then the things you get to send
is just the final answer that you're
looking for
yeah and that that actually speaks to
another design decision we made of this
sort of these these stateful save
contexts that you can mutate and then
later retrieve
is that that means that for this sort of
data set example collection where you're
you know collecting the top examples
along symmetric
you can set up appropriate probes and
then run your 10 000 models through
and then those probes in a stateful
fashion update the kind of current list
of top ten
and then with one
one call one network communication costs
you can suck all of those back down
if you had a approach where you know you
maybe were allowed to apply some
aggregation and then immediately sent
the sent the data back to the client
you'd have to do communication for each
of those ten thousand examples so this
sort of saves you a ten thousand fold
uh or a hundred thousand fold or million
fold sometimes we run you know large
numbers of examples to kind of get large
scale accurate statistical pictures by
being stateful on the server it lets you
accumulate state on the server and then
just kind of
run examples through as fast as the
model can process and amortize and then
only do your communication at the very
end
so that the the the desires for those
things the ability to read data as to
read tensors as well as altering tensors
while the model is running
the ability to do computation on the
server in order to
compute aggregates uh and
and minimize communication cost and then
the ability for the server to have
persistent states so that it can compute
aggregates across examples across batch
dimensions
and then kind of store those and send
them back kind of drove the design of
the interface uh and i think the sort of
the net result is an interface that is
very flexible
and a little bit clunky to program
directly against it's not sort of most
convenient thing but we sort of think of
this as the sort of assembly language or
the like primitive language of the
system and then we've built layers on
top of it that are
kind of much more of just like run this
and just give me these values directly
don't make me deal with
the save context abstraction and and
saving them and then the separate calls
restore them it's very it's kind of very
straightforward to build more expressive
layers on top of this that
that sort of maybe are higher level
that are higher level or or kind of
correspond more naturally to some higher
level
models of the model but that don't have
the kind of full power and flexibility
and i think sort of in general that's
that's a pretty when you can have it
that's a pretty powerful
way to build these systems is if you can
have a kind of minimal expressive
primitive that gives you everything you
want and then
layer the sort of things you actually
want on top of that cleanly
that that decomposition i think has
worked out pretty well
catherine i'm curious if you can speak
at all to your experiences as an end
user um
about garcon versus building
abstractions on top of garcon
yeah i mean the thing that came to mind
is nelson was speaking was it it took me
a while to sort of get used to
when should i be using these
simplifications and when do i need to
think about which computation happens on
the server or not like in these cases
where i just want the activations i'm
only sending in a batch dimension of one
i'm not aggregating anything i'm doing
one of these single unit studies uh it's
very easy to just use one of these tiny
little wrappers rewritten that's like
yep just save the activation send them
back don't need to think about it and
then as i moved into more statistical
analyses having to think like wait
actually there's a batch dimension now
and so i need to take that into account
i can't just always take the zero or
actually you know i'm running ten
thousand of these that needs to move on
to the um that should be done on the gpu
um
it just took me a little getting used to
those different layers and i think uh
that was that was not too bad i mean i
now have sort of a sense of how you know
just how statistical analysis is this
gonna be if it's comparatively more
statistical i probably need to just add
a couple brain cycles into you know how
do i need to change this save context to
um
to factor that in
so catherine i'm going to put you on the
spot a little bit in front of nelson and
ask you know you've talked about the
things that have been positive about
using um garcon and i talked about the
highlights are there any low lights or
any things that have been kind of
frustrating as an end user about um
about doing things this way
gosh um i think there's some things that
are still slightly half-baked so um you
know if i cancel a call to a garcon
server because i don't actually want
that results and then later i run the
cell again or run a different cell it'll
be like oh i sent back a reply to the
wrong message and so there's just little
things about the networking protocol
that are uh slightly rough around the
edge but i get used to very quickly like
yep just reconnect do it again so none
of this has been um comparatively
prohibitive and yeah just designed for
everything before just like getting my
head around like what is this context
buffer doing like it took me a while to
realize just how
stateful it was uh and then once i got
that that's sort of a step up in power
but i was getting weird results that i
didn't understand because i didn't
realize it was stateful in that way um
and so there's sort of that um
surprise versus power trade-off uh that
it just took a couple you know oh it's
keeping state or oh there's a batch
dimension now that you just have to
encounter once and then i think uh you
move through it through it pretty
quickly
that makes sense
um
okay so i guess a question somebody
might ask is you know
are you opening open sourcing garcon if
not why why aren't you it sounds like a
really useful uh useful tool that could
be helpful for people who want to do to
do interpretability
i think we're unlikely to i think it's
the kind of thing that is
uh is like pretty tightly coupled to a
bunch of details of anthropic's
infrastructure
which sort of both means that you know
decoupling it to to open source it would
be a sort of fair bit of work
and might or might not make it kind of
minor might not result in something that
was useful in anyone else's
infrastructure if they just sort of made
different assumptions about how things
worked and how they launched jobs etc
and i think that there's some complexity
there but there's there's not an
enormous amount of complexity and kind
of i suspect that another team that
wants a similar thing
kind of
the sort of description of what it is
and how it works that we've given here
is like probably enough to give
you know at least a moderately
well-resourced team like if if you're a
you know
and kind of anyone training large models
probably has at least a moderately
well-resourced team sort of give them a
blueprint to to build a you know easy
version tailored to their infrastructure
in like a week or so or and then you
know you you iterate a little bit from
there
and so i think it's the the kind of
thing where
uh the sort of the value of kind of
actually releasing code is is a little
bit swamped by just how
how fiddly it is and how much it plugs
into the sort of exact details of how we
write our models and how we run our
infrastructure
and the sort of it's just ends up being
a much better value trade-off for
everyone to sort of share a relatively
detailed description of like what it is
why how it works in my why you might
want it and encourage teams that think
they might want one to just build one
themselves um
uh it's it's sort of not too much work
if you have the the kind of appropriate
appropriate infrastructure and engineers
already on your team i think i also want
to chime in with some like appreciation
of you know other anthropic members not
currently on this panel uh who have done
a lot of work to make our cluster very
usable such that launching something to
a pod running on a cluster is very easy
and that's sort of kind of the some of
the bread and butter of what this garcon
workflow looks like and so depending on
how your team
manages its cluster and manages
launching pods to that cluster uh that
sort of uh
uh contact with the with the metal of
how your cluster works could look very
different for you and i think that's
at least one layer that would be sort of
hard for us to
decouple from and where we benefited a
lot from the other infrastructure
engineers who have made that quite
seamless for us
it's a really fantastic point
um
well in closing
uh do we have any
uh any last things that we'd like to say
um perhaps to our friends at other labs
who might be considering uh copying
garcia for us
or copying garcon and using it
uh we'd love for you to do so
um very very excited for other people to
have this kind of infrastructure and be
poking around at these models
yeah i think maybe the sort of high
level
thing i would say is i think sort of the
the details of garcon's design i think
are like decent and have worked out well
for us
i think the sort of like most important
thing is i
like any lab that is training
large models
i think like should have some mechanism
whether it looks exactly like garcon or
not that sort of makes it very
straightforward for
the people for for
other researchers at that lab to
like
[Music]
load that model
run it poke at it in a very kind of
flexible and general way
um and you know i think sort of that at
a high level that's the problem that
garcon is solving for us is that any
model we can train we also have the
ability to
not just train it not just run forward
passes and use the model
but to also load it up and poke at it
kind of no matter how large the model is
no matter how weird the model is
and i feel like
it would be a real tragedy if you know
our
interpretability or
you know even though like alignment or
other teams sort of could really easily
work on our small models but like every
time they wanted to do an experiment on
a big model on a you know it it required
custom work or custom engineering or
even like not custom engineering but it
was just like way more painful than
doing work on small models because they
had to you know write code in a
different way than they were used to and
launch a job that had a 10-minute
startup cost i think it's kind of i
think it's like really desirable
property that that the
kind of any model we build is sort of
like roughly
equally accessible to the teams doing
research on how these models work and
how they behave
um and i think at a high level that's
that's the problem that gar saw solves
and
any lab training these models like
should regard that as a problem worth
solving and hopefully the ideas and
approaches that we've talked about here
are sort of some pointers for how to do
that but
uh if you come up with some totally
other approach i think like great we
would we'd love to hear from you about
what's worked for you
i think i also want to offer like a
compare and contrast between how
anthropic approaches sort of our
research on these models and how i've
seen other research environments operate
that leads me to think that other
environments could gain an even larger
boost from this kind of approach which
is that
anthropic has a quite collaborative and
collective approach where we are working
together to determine which models we're
going to train train them and then all
sort of study and build upon and
fine-tune from this shared set of models
where whereas i've been in research
environments where individual groups of
a few researchers have their own code
for training and launching models and
their own architectures and so if i was
in one of these uh more archipelago like
labs then if i wanted to study your
model i might not know how to load it i
might not know how to get it in my
notebook i might need a set of a
personal one-on-one meeting with you it
could be very high overhead whereas if
someone else had trained a model and
just put up a garcon server for anyone
to use it then suddenly i don't need to
know your particular
code base for launching it i can just go
and start doing interpretability
experiments on it and i think in that
kind of environment it could be an even
bigger cooperation and collaboration
boost
well thank you so much uh for taking the
time to talk about this uh nelson and
catherine
thanks chris yeah thanks chris