Jaime Sevilla - Projecting AI progress from compute trends

[Music]
hi everyone and welcome back to the
tours data science podcast for an
episode i've wanted to record for quite
some time now
so there's this idea in machine learning
that most of the progress we see in ai
doesn't come from fancy algorithms or
new neural architectures
instead some say ai's progress has been
driven by scaling up compute power data
sets and model sizes and besides those
three ingredients nothing else really
matters through that lens the history of
ai really becomes the history of
processing power and compute budgets and
if that turns out to be true then we
might actually be able to do a decent
job of predicting ai progress by
studying trends in compute power and
their impact on ai development and
that's i wanted to talk to jaime sevilla
an independent researcher and affiliate
researcher at the university of
cambridge's center for the study of
existential risk where he works on
technological forecasting and
understanding trends in ai in particular
now his work's been cited in a lot of
cool places including our world and data
which used his team's data to publish a
whole expose on ai progress and jaime
joined me to talk about his work his
predictions about the future of ai and
all kinds of other cool stuff like that
on this episode of the tourist data
science podcast
[Music]
i'm really excited about this particular
episode i've uh i've been following the
your kind of your work generally on
trends in compute and the reasoning
behind all your work as well like for
quite a little bit of time now and i'm
sort of one of these lurker fans of
yours i think it's fair to say like on
twitter and other platforms i'm really
excited to to share this story with
hopefully the the white wider world or a
larger part of the world i like to start
about with you with your motivation like
getting into the space why why trends in
compute what is so important about
looking at trends in compute at this
stage in our in our life as a species
let's say
one thing that uh that i am really
interested in is the admin of advanced
artificial intelligence uh there's a
plus there's like some reasons to
believe that in the coming few decades
we're gonna see like drastic advances in
the in the practice of artificial
intelligence which is gonna allow us to
automate more and more of society it's
gonna have like wide-ranging
implications from like many new jobs
being created other jobs are being
destroyed but also change changing the
way we approach society and introducing
like new risks into the mix that might
radically alter society forever uh one
thing that i was really interested in
looking into is uh trending like uh
inputs into like machine learning
systems like people having are quite
focused in like measuring outputs we
have like really good benchmarks in
order to assess like how well are we
doing in tasks in computer vision how
well are we doing in tasks in language
models but in the coming years we have
come to learn that there is like lots of
games
that can be had just by taking like the
systems that we already have and just
scaling them up just making them bigger
making them have more parameters uh
training them for longer and like uh
using more data in order to train them
and while there has been like some work
exploring like these implications there
really hasn't been like a historical
survey of like exactly how many
resources have we've been put in into
like these systems and this question is
critical because uh it's gonna allow us
to understand like all these progress
that we have seen in the last two
decades which amount of that is due to
us having like smarter architectures
better ways of like approaching the
problem of artificial intelligence
versus us just having like better
computers and more data to train these
machine learning systems on
yeah and that makes perfect sense it
kind of gives us a window into well a
window into the future it's the only as
far as i can tell it's one of the very
few at least ways we have of projecting
what the future of ai might look like we
don't really know how to go from like an
architecture or a concept of a model to
what its capabilities are likely to be
we tend to be surprised when we find new
capabilities emerging so it's kind of
yeah it is helpful to have like at least
investment in in you know the amount of
dollars or the amount of compute
invested in these models as a a hard
number so we can say okay you know this
much money or this many um gpus gets us
this kind of model and then what can we
expect from the next uh a lot of your
work though is focused specifically on
this idea of transformative artificial
intelligence and
this term tai for short it means a lot
of different things to different people
and it's a source of great debate within
the community i'd like to start with
that then so like what is transformative
ai or what is tai to you
and uh and how do you yeah how do you
think about it in the context of your
work and research
right so for me essentially the way that
i think about dai is that dai is gonna
be to us as like uh the industrial
revolution was like uh the farmers that
uh that perceived us like it's gonna be
something that's gonna be that's gonna
enable like a new feedback uh a new
feedback loop in our culture which is
drastically going to speed up the
economy and like i'm leaving this kind
of vague because it's kind of like okay
we know that this is going to be a big
deal like we know that like being able
to automate like a large part of our of
of our working society like uh it it
really like it really changes things it
allows you to it allows
tasks to be done on like an
unprecedented speed if you don't need
like a human in the loop to uh that is
like kind of like bottleneck bottleneck
in the whole process
and more than that is like okay well you
know it doesn't make sense like i don't
spend too much time thinking about
exactly what it means
one reason i've heard for not not
worrying too much about that and i'd be
curious to get your take on this but
essentially once you get to the point
where we have genuinely transformative
ai where we have something like an
industrial revolution but powered by ai
progress is going to be happening so
quickly that whatever set of criteria
you use to define this new industrial
revolution if you slightly disagree with
someone about those criteria you're
going to be off by like a week or you're
going to be off by like a year let's say
and then the next level of the criterion
is going to be reached and so there
isn't really that much fuzziness at the
margins because change is going to
happen so quickly anyway is that a fair
characterization
i think that is a fair characterization
i think that uh that for most possible
definitions that make sense of like how
society might be transformed like we are
gonna find them to be like extremely
correlated they're gonna happen at the
same time as they say
okay no that that's that's helpful and i
think for people who are less familiar
with like forecasting especially
forecasting ai stuff hopefully that adds
some context um great so so essentially
we have this idea of massively
transformative ai ai that transforms the
economy and and as you said i mean going
beyond that is really hard also because
i wouldn't expect like a
an agrarian society a member of an
agrarian society to be able to predict
what ipads would do what zoom would do
and so on so we're going to be in the
same position relative to our future
selves in that respect at least um i am
curious so like
what are what are the um the stages that
have led us to where we are today than
in the story of the evolution of ai that
may eventually lead to transformative ai
so you like you've studied that story
and you're trying to use it to predict
when this moment of tai or this phase of
ta i will happen
what are those phases and like can you
give us a bit of a taxonomy of like the
history of ai up to up to today so uh in
terms of like the what we have seen so
far what i've been doing with my
colleagues is like collecting
information about like historically
important machine learning systems since
the 50s up until today right and then
like when you look at the when you look
at the inputs that have gone into these
systems like uh so far we have looked
into the amount of parameters that like
each of these systems had i like the
amount of compute that was used today in
the systems like the amount how long
they were trained for in a sense
and uh what we have found is that uh in
terms of parameters it was like actually
like uh fairly less clear the the whole
story there's definitely there's
definitely like these uh aboard
upward trend which is undeniable
uh but uh it seemed to be like fairly
uniform up until maybe like uh 2000 2016
2017 2018 like somewhere around that era
like something happened and that some
and there's something that happened is
that language models started scaling
like way faster in terms of parameters
that anything else that was going on at
the time
uh this was like the it was like our
first clue of like okay there might be
like some things there might be like uh
some
changes that are happening in like the
history of machine learning that is
changing the reduced scales of the
systems like it's making it so that
scaling up those systems faster than uh
there is an incentive to scale the
system faster because we are getting
like better and better better
performance continue
actually
uh we looked at at like trends in like
compute like for how long the system was
trained for like how many operations in
the computer it took uh to train the
system and then like we see like a much
neater picture
uh we already knew from some previous
work that there had been like this phase
transition
somewhere around 2010 so essentially
before that uh the trend in like the
trending like the compute used to trade
machine learning systems essentially it
was doubling every 20 months uh if if
you're familiar with like moore's laws
like moore's law is like this empirical
description of like how the the
computing power of like uh of like uh
of like our laptops or computing devices
uh has uh scaled over time
which also follows like a similar
pattern like every two every 20 months
uh roughly you can say that uh things
got uh computation got like twice as
cheap
right
so what we were seeing before 2010 is
that you know like it's not that people
were invested more into machine learning
it's more like our laptops were getting
better and better uh like researchers
were using like a state-of-the-art
laptop for all of their things so uh
they were naturally uh coming to use
like more and more resources of
computation but something changed around
2010 which is that suddenly like this
trend like a speeds up before it was
doubling every 20 months and then like
afterwards uh we argued that it started
uh it started speeding up to doubling
every six months or so and uh this is
this is like really fast this is like
two doublings per uh two thousand per
year which is like fairly crazy uh if
you stopped uh to think about it
and uh there might have been like uh
they might have like a few reasons uh
for this uh the most selling the story
is that around 2010 was like the time
where like we realized the potential for
like uh deep learning systems to like
perform really well in tasks uh
primarily in computer vision
yeah so
i'm sorry go ahead
yes so it served kind of like as a
wake-up call for uh
for like researchers all over the world
be like okay there was like this
paradigm that really was developed uh in
the past century about like
creating like this uh neural
architectures that were just shown like
a bunch of examples and through back
propagation they were like adjusted in
order to get good performance and that
really hadn't gotten anywhere like we
had like a lot of like
bespoke systems that were being
developed uh these days like shift which
like where what you used if you wanted
to have good performance in computer
vision and now suddenly you have like
this very general approach which is
gonna which works on like a a wider
array of tasks uh particularly at the
time in computer vision which like uh
now we suddenly really got news because
our computers have gotten like good
enough uh to uh to do it
and uh people started investing like way
more into like these systems which meant
that uh these trend of like okay before
uh the the computing power that was put
into machine learning systems was just
the computer power when you haven't had
now people actually had a budget to like
uh train their machine learning systems
and that budget got this scaled up and
up as time went on which meant uh this
rapid increase in like the amount of
compute that was we import into like
this state-of-the-art uh machine
learning systems
okay so two so if i understand correctly
at this point we have two distinct eras
that you've flagged or actually
maybe three um so we have first this
phase where we have a steady a flat
academic budget and thanks to moore's
law compute's getting yeah as you say
twice as cheap every two years or so and
so you see you know twice as much
compute being used every two years but
it's still only academically interesting
during this period there's no real
industry application to the tech or at
least the industry applications aren't
enough to get people to throw tons of
money at it and then around 2010 we have
this moment where there's an inflection
point in compute budgets and and would
you so would you say that that was due
to alexnet specifically or were there
other things because i remember when we
talk about the history of deep learning
alex net is often cited as this like big
aha moment everybody goes oh you know
the deep learning revolution it started
with alex net um is it is that genuinely
true or are there models that came
before that that sort of hinted at oh
you know what we should we should scale
more uh with the computer like what's
your sense of that of alex net's role in
that story
right so uh
in my in my book what characterizes the
era of like deep learning is like three
basic factors
one of them is uh about the model size
and depth like sadly we started getting
like a systems that had like multiple
multiple hidden layers in like their
architecture and like had like way more
parameters than what we had seen before
the second one is uh the use of gpus
so uh people started experimenting with
like gpu platforms in order to
parallelize the training with like
drastically uh increase the amount of
like computing resources that they had
access to
and the last one is like performance
that was the point where like yes
suddenly deep learning systems started
starting to top the charts of uh in
benchmarks uh like uh c4
uh like uh like the image recognition uh
benchmark c4 or and
other uh another tasks
now was alex was alex nest uh the the
first thing that uh did uh all of those
things and the answer is like definitely
not uh like uh we have seen like very
large uh uh very large systems like as
large as like alexnet uh since the early
2000s like there is a paper by biola and
jones in 2001 or like they train a
system which is essentially as big as
alex not
uh it's gpu based training they think
that a distinguished alex net and it's
like well also not because uh we have
been seeing like uh the use of like
this insight of like using gpus to train
machine learning models uh had been
around for like at least seven years by
then like uh
for example like uh for example in 2005
there's this paper by steinkraus and
other people were like days that they uh
show some machine learning systems that
were trained or like gpus and in fact i
can't remember right now exactly when
like the
the cuda gpu framework was released but
that was definitely like a watershed
moment in which like suddenly like gpu
computing became like very ob could use
and like really easy to program also for
machine learning applications
all right so it's not uh so it's uh
alexnet is not the special in terms of
model size it's not a special in terms
of like gpu based training is it the
special in terms of performance and it's
like yes it's significantly outperformed
like pro techniques uh in imagenet now
it was not uh the first benchmark that
had been broken by um that had been
broken by a deep learning system
like uh the other example that uh that
uh that uh predates it is uh there is
this paper by sir san and others in 2010
which uh makes substantial improvements
over the previous state on the earth uh
on mnist
and there is like also this paper by
mikolov where like they also
break like an important uh nlp
nlp test the wall street journal task
so
alexnet uh it was as big as things that
came before it had all
it had been trained on gpus but also
things that came before i like sure it
broke like a really important benchmark
but there were other important match
marks that were broken that were broken
like like two years before around 2000
and then
all of these factors combined uh kind of
like made me think that okay this is not
uh alexnet was not like a watershed
moment out of itself it was part of like
a larger trend but what it's undeniable
is that alexnet gathered like a lot of
like academic recognition and also
outside of academy
and uh i think it's quite plausible that
that it acted as like this uh wake up
call where like people were like oh
this works okay interesting so so that
was what i was gonna ask next was like
you know given that that seems to be the
case with alexnet why why is alexnet
held up as this ultimate exemplar of
like this great watershed moment so your
assessment would be something like it's
it has to do with people's reaction to
it perhaps some in some sense the
marketing around it was just really good
do you think that's too reductive or is
that like roughly accurate
i think uh i think that's uh basically
correct uh like and it's undeniable that
imagenet was like a a very important
benchmark and it was more important that
the benchmarks that were broken before
but uh like in a sense this gives me
hopes right because uh i kind of have to
like
you could have predicted alex net this
is what i'm getting at like if you were
just in 2010 and you were squinting your
eyes really hard you will have seen like
all these neural neural network papers
that were breaking like this wall street
journal task this uh mnist task and you
will be like hey
something is happening here
interesting okay so that's actually it's
especially interesting given the next
phase in the evolution of machine
learning because i would make a similar
mistake so in 2020 openai came out with
gpd3 um i can describe it to you the way
i would naively have described it before
this conversation as the first uh the
first
pseudo general purpose ai the first ai
that was trained for one narrow task
like autocomplete and turns out to be
capable of a very wide range i mean
obviously we see transfer learning in
other contexts and images for example
but this is really where we see zero
shot learning in all its glory for the
first time translation coding even essay
writing basic web design all those
things that this one system can do
despite being trained to do something
really qualitatively different um so
my guess is you're going to come back to
me with the same similar stories alex
that hey you could have seen gpg3 coming
i'd love to explore that
first off if you agree with that maybe
you won't but like could you have seen
gvd-3 coming and if so like what were
the what were the warning shots what
were the things that should have had us
going like oh okay you know scaling does
make sense
right so uh here there is like a um
there's like a this very interesting uh
story right like we have seen this
transition between like pre-deep
learning era to like the deep learning
era now what i want to talk about is uh
what the next transition that uh we are
that we argue exists uh in our paper
which is the translation between like
the deep learning era to like the
largest scale era
to the point where like industries
started investing like millions of
dollars into like training these are
very large machine learning systems in
the hopes of like getting like uh
increased performance
and uh when we were looking at the
parameters like uh it became obvious to
us that okay there's like uh
around 2017 2016 some something around
there
like
a parameter a language model started
getting like much much much bigger
and immediately the thing that came to
us is like okay this is transformers
right like transformers came out in like
2017 uh they quickly proved to be like a
different regime of a scaling and it
became like much more advantageous to
like scale them up as fast as possible
and that's what happened
um
now i'm not so i am not so sure about
that because when we look at compute uh
sure we see that uh language models like
have stolen the thunder these last years
but really like the first system we'd
see that i came close to that regime of
scaling is uh some reinforcement
learning systems that were spread headed
by deepmind so things like alpha alphago
by demand and google generally like
alphago is like one example uh the
google neural translation machine is
another example like for me this is the
point where like uh
companies realized that they could like
scale things up like 100 times bigger
than they had been done before and like
actually get good results out of that so
they just went and did that
uh could we have uh go ahead from these
uh predicted uh gpd3
uh i was just still like
i was personally really really shocked i
was like the whole dpd3 thing i like the
whole rise of like language models i was
like quite bullish on thinking that
language modeling was like what i would
call like an ai complete problem it was
like a problem so hard that like we were
gonna get like general intelligence
before we actually got like good
language modeling and like well reality
has proven me like very very wrong
yeah that in itself is is an interesting
uh an interesting aspect of all this the
link between
scale the link between um architecture
and then the actual capabilities that
are achieved by these models what's easy
what's hard
it seems like i mean this is to me one
of the the interesting aspects of your
work it allows us to start to notice
when our intuitions
were just completely wrong
um now one thing i do want to touch on
before we go more into that direction
because i think there's a lot to talk
about when it comes to kind of
capabilities and linking those to scale
and other things
it seems like you mentioned a couple
times you know
your assessment of
doubling times for compute power for
example and your thinking and your your
analysis and um and you're hinting in
that that there might be disagreement
there might be other perspectives too
which i guess to me i always found
interesting because i would have
expected this to be a very
naively like straightforward thing to
calculate you know we have a bunch of
leading models and they have a certain
amount of compute power consumption and
then we start to draw straight lines on
log plots and and there's our doubling
time um so can you explain like i'm sure
i'm wrong by the way but i'd love to
understand how i'm wrong
absolutely so uh uh let me talk about uh
the work that came before us in terms of
fly computers uh the print the main
piece is this article by open ai around
2018 where like they did they went
through the same processes as us on like
a smaller scale they like uh gathered
the amount of computers used to train
like uh around like 12 to 18 uh
state-of-the-art machine learning
systems throughout the years and they
just they plotted it and they were like
okay this is the this is the line that's
just runner regression and they got like
they got like a doubling time that was
like way faster than us just like uh our
time uh our doubling time was also
really fast like uh six months they're
uh they're doubling the only time that
they found is like half of that it's
like every three months things were like
doubling right
and then what happened
well two years went by and like their
prediction was well that their implied
prediction right like this implied trend
like a stops flood a stop flat like uh
it just doesn't go on it's like we had
sadly we hadn't seen like a doubling uh
since 2018 by 2020.
and there is in fact like uh this blog
post by alex lysol where like uh he uh
he expands on like the work of like uh
and compute adding like a few points and
being like
uh the trend stopped right when you grow
the blog post
right now now this is uh are you
referring here to the scaling laws for
neural language models paper i think
that was 2019 though wasn't it
no no no uh this is like uh this is a uh
standard growth sorry and compute that's
right that's right i then want to bring
in that that scaling laws for mural
language models paper which came out in
yeah in 2019 i think by the time they
wrote it gpd3 pretty much would have
been
built internally in retrospect because i
think they they released the paper for
gpd3 sometime in january and i think the
scaling laws paper might have come out
in
like very late 2019 um so so maybe they
had some some new insights based on gpd3
but do you have a view on like on that
paper did it cause you to change your
perspective is it consistent with your
analysis or
uh absolutely so uh the paper on like a
scaling laws is essentially like the
whole motivation for like this whole
project it's kind of like the proof of
concept that like a scaling matters and
it's like really important
so uh kind of like what what we see is
like our work is like complementary to
what's happening in like uh
like uh the scanning laws paper in there
like they were running like a series of
experiments with like uh some with like
some systems what we're doing is like
we're looking at what has happened
historically i like see like given the
insights that we found in like that
paper unlike some other papers that
we're studying uh returns to scaling
whether we can explain how much of the
progress we have seen in the last two
decades is based on uh just this is
scaling things are getting bigger and
faster versus uh us having like better
architectures the trend that uh open air
had found when like they look and they
plotted like this line of like uh
up to uh it was not only like every
three months and then like alexa like
continues that and then it just doesn't
grow i like you know like between the
biggest system in like 2018 which is
alpha uh alphago uh
one of the versions of alphago also goes
zero and uh the biggest the the biggest
uh system in like by two 2020 which is
like uh dpt3 is like actually gpd3 is
smaller than alpha goes zero so it
seemed like oh the trend has that don't
have the scope but like uh really i
think that uh this is like an illusion
that is being caused by like this uh
this discontinuity that we had in 2016
like this point where like suddenly
companies started uh uh investing like a
hundred times more so then what's going
on is that uh if you just look at like
the uh like the biggest systems overall
you're gonna catch like a lot of noise
and that's gonna make it so that uh so
that uh the trends that are apparently
there like really are not there because
they are just uh they just consist of
like these field pliers and include like
this largest is going to do with this so
you need to like take up a bigger look
uh to look for like uh the trends that
are there and like even having like that
bigger look you're still gonna have to
have uh points like in 2016 where like
suddenly things skyrocket escape rocket
up by like two orders of magnitude and
uh that's something that happens this
continues to happen what this makes me
wonder is is where these trends would
start to break down or like what might
cause these trends to break down in the
future
so uh these trends uh the distance right
now are the combination of like two
factors one is uh compute getting
cheaper as building like better uh
infrastructure for computing
the second is uh investing going up
right investment going up like
industries uh primarily industry at this
point uh is uh more and more interested
into having uh putting like millions and
millions of dollars into training these
systems
uh these two these two trends like
follow different mechanics and may break
down uh because of different reasons uh
for like the for like the computer and
uh this is like moore's law people have
been claiming that moore's law is gonna
die like very soon eventually
it has to die like it cannot go on
forever but it's like really hard to
find out what is the point at which like
uh actually uh the things break down uh
like we start to be we stop being able
to like
scale up our systems like uh there might
come like a new new ways of
conceptualizing like computation that
might allow us to like uh keep squeezing
the uh keep squeezing like uh
this trend i like keep our uh it's kind
of like a sort of like self-fulfilling
prophecy where like people kind of have
like this uh like my my impression is
that internally like harvard companies
have like this impression of like this
is the goal to meet and they put like a
lot of effort into like making it at
some point like you know physics says
stop you cannot go on but for the time
being it hasn't seemed uh it hasn't
sound like uh has slowed down a bit
but it's still is still happening it's
still a decreasing exponentially the
price of compute and i was expected to
responsibly at least at the very least
for the last 10 years and like possibly
like quite possibly for longer even for
how long it has held up so far
now the second thread is about
investment i like investment is more
complicated because uh well industries
industries have a budget and they can
siphon like part of their r d budget
towards a ai
like that's essentially what has been
happening they also have like target
revenues that they can put into this but
there comes a point where like you know
when uh ai is like 90 percent of the
earth research budget you just cannot go
on without like a state support or like
uh or like something else
uh when is when are we going to reach
that point so my colleague tamaiba
zeroglu actually uh
repeat uh performed an analysis based on
a blog post by ryan carey from a couple
of years ago
and uh he essentially tried to compute
try to put like an estimate on like what
is going to be the reversing point or
it's going to be the point where like
this trend of like uh increased
investment is going to stop uh like the
driving force between between uh
progress from now on it's just going to
be like moore's law
and essentially like
he was estimating like you know under
some reasonable assumptions about like
how much money could uh
companies like possibly spend on ai r d
like maybe in like it's it seems
definitely possible that it's gonna held
up for like 10 years uh the current
trend and then like afterwards it's like
extremely uncertain like what's going to
happen if they're gonna like hit the
ceiling and just stop increasing their
budgets if like estate actors are gonna
like come in and like uh keep uh
investment up
uh we know one interesting factor in
that analysis too is like if companies
start to scale up their uh their compute
budgets in that way
eventually you do get systems like gpd3
that can create so much value themselves
uh that it pays for that compute and so
you have this positive feedback loop
that has no termination point or at
least no no clear termination point you
know arguably we're already seeing that
with gpd3 there are companies that have
raised tens of millions of dollars that
are really just like a fancy wrapper on
top of gpd3 or maybe you know ai 21 labs
products or stuff like that and so you
know it's like 10 million dollars well
that's already the cost of making gbd
three so so we're in a way we're already
in that regime um or maybe we're not i
mean do you think that we may be already
at this point where we're kind of
closing that loop
i think we are like one of my leading
hypotheses is to explain like this
discontinuity that we saw around 2016 is
uh essentially yeah a point where like
industries did the calculation be like
okay we can afford to put the money this
morning in because this is gonna
generate like so much revenue
and uh for me like the leading example
of this it's not gbt3 it's like a dnmt
like a dnmt the google the google neural
machine translation system was like one
of the early examples of like a really
really large scale machine learning
system that broke with the previous
trend and that had like a huge uh
economic implications
okay yeah no that makes perfect sense um
and actually okay so now we've talked
about this idea of trends in compute one
of the things we haven't talked about is
how we tell
when a particular level of compute leads
to a particular capability or a
particular situation in society this tai
threshold transformative ai threshold
that you've been trying to kind of
project and predict
and one of the techniques that you've
used to actually
land that plane and figure out okay you
know
how do we how do we get capabilities
from these systems how do we predict
capabilities from scale is to lean on
this framework of biological anchors in
uh predicting transformative ai so could
you explain what biological anchors are
and how they relate to some of your work
very badly but i will try my best so
essentially within my group we haven't
mostly been focusing on like inputs but
uh some of people some people on this
area that have in high regard uh have
been figuring out like what to do with
like the estimates that we're providing
or have come up with like their own
models we're like extrapolating these
trends to try to forecast at different
levels of performance and definitely so
far like the most intricate piece of
research and the most complete piece of
research that we have seen is ayakotra's
draft report on airtime lands where like
she comes up with like some generally
useful concepts in order to try to
understand uh how much compute will be
needed uh to train uh transformative
machine learning systems
so uh that's the that's uh the this uh
by that that there's like this anchors
uh report or like c comma comes up with
like six different ways of estimating
like uh what's gonna be like uh the
amount of operations that you're gonna
need in order to train uh these
transformative uh systems i like three
of uh three of them are like essentially
like
biologically based there's like
estimations about like you know
so far the only example of like
artificial general intelligence that we
have is humans
right and like uh what uh well not that
efficient but general intelligence that
we have
and uh so it provides us kind of like
with like a a very crude estimate of
like uh well
an upper bound on like how many
operations you need in order to create
intelligence
and there is like multiple ways that uh
you can go about thinking like how many
operations did it take to uh make a
human
like uh one one thing that you can do is
just go like okay like a human like is
born and then like it takes like some
time for it to like let absorb like the
culture to like learn how to speak how
to write uh how to code how to do
different things and like uh it's gonna
take them like okay roughly like 20
years to become like a functioning adult
like uh for like a baby that knows
nothing to like a general intelligence
that can perform like a wide array of
like economic tasks so you can like sort
of estimate like okay how many
operations does should take in your
brain to uh to
do all that learning and that's gonna be
like a sort of like estimate of like how
much it takes from like baby level
artificial intelligence to like human
level artificial intelligence
um
as that as this there's like a couple
other ways that you can go about it
because you can say like okay but babies
have already like a lot of like built-in
machinery and like maybe this is not uh
the best uh the best way of thinking
about it and uh maybe you can go with
like the most extreme estimate that uh c
provides is like going like okay how
much did it take how many operations did
it take to like evolve a human right how
many operations if you see the earth
that's like this giant computer that has
been running like this evolutionary
algorithm for like uh for like billions
of years like how uh how long did it
take to uh actually create a human level
intelligence from that
and she also provides uh that uh that
kind of estimate and those are the
biological anchors right uh for me
actually the ones that are more
interesting are not the biological
anchors themselves but uh the anchors
are based on like uh this concept of
like horizon length
so essentially what i uh uh proposes is
that uh systems where like the reward is
like farther away temporarily than the
action are gonna be like harder to train
uh that uh system for like the reward
and the action is like uh closely paired
together
and uh which makes a lot of sense
uh one of the hardest problems in like
um like machine learning is the
attribution problem like trying to
attribute like okay this reward i'm
getting like to which action is it to
you
and and i guess this also
i mean it aligns to some degree with the
evolutionary anchors perspective just in
the intuitive sense that when you look
at like
animals we tend to think of as stupid
they tend to be they tend to act on
instinct in other words they tend to
respond to immediate stimuli and
respond to it in an immediate way
there's very little plotting and
scheming going on in the brain of an ant
for example whereas when you look at you
know dogs maybe they can learn to train
their owners in certain ways or you know
trick them into doing certain things so
there's a little bit of foresight and
planning monkeys more so and maybe
humans even more so um so i guess
there's a sense in which they are sort
of aligned even if they they take
different
different directions
exactly that's it so essentially what uh
what i did i was coming up with like an
estimate of like okay so far we have
trained like some reinforcement learning
systems that uh are able to act in like
these time horizons are able to act like
so many steps into the future and then
like c-com something's like an estimate
of like okay how many steps
will it take to uh uh could it take to
like uh create a company for example
like this is an example of a task right
and then she goes like well uh this is
like a time horizon flag a year the
amount of the steps uh that's that uh
this is gonna involve is like such and
such and then like uh given that i see
like from the previous data or like how
much how much compute did it take to
train like these previous machine
learning systems we kind of have we kind
of have like a rough estimate of like
what how many operations does it take to
train a system that has like a certain
horizon length so now with like these
new horizon lines for this task that we
haven't automated yet this could be like
a a possible estimate of like how many
operations will it take to automate
those as well
so through that lens i guess this makes
me think of the gpd3 context window or
the context window of language models as
being this very important number the
sort of amount of text it can keep in
mind at the same time as it predicts the
next thing
is that like do you think that's a
correct way to think about it like the
context window might have a lot to do
with this idea of planning ahead and
time horizons
absolutely i think this is one of the
main reasons why right now like i don't
see gpd3 as like uh something that can
scale up to general intelligence
because you do really need to be able to
like create loops in your intelligence
to be able to like pay attention to
things that happen like a very long time
ago in order to like create that now uh
this is not to say that gps iii cannot
be like a critical component of like
artificial data intelligence like i have
been actually like very shocked by like
some kind of like by hybrid approaches
in which like dptv has kind of been put
into like a loop in which like it is
able to produce things like mathematical
proofs like code and like that code is
executed and we could see maybe like a
uh like this uh as the beginning of like
maybe some sort of like uh
loop system in which like you put you
asked jpd3 to provide like a piece of
code the code is executed that provides
like uh that provides context for like
the next call to like uh gpp3 dptx like
interface
interesting so
and that actually vibes i guess with the
idea that
we have gbd3 taking up and historically
ai has done this too even going back to
like the mid 2000s
it almost starts with like taking away
some of the most menial tasks so like
excel spreadsheets remove things that
require like thinking on the order of a
couple seconds i don't want to have to
multiply all these cells together excel
will do it for me then the next step is
like i want this to do you know
calculate my p values to do even more
sophisticated operations that save me
more time and gradually the human gets
to zoom out more and more think more and
more well we say often think more
strategically but really think over
longer time horizons as the the goal or
the responsibilities of this ai start to
expand um and so when we look at some of
those loops i guess you probably have
when we talk about theorem proving with
something like gbd3 or gopher or things
like that
i guess you probably have the human
doing almost strictly long-term thinking
um and then the ai picking up the slack
is is that how you're seeing those loops
maybe i definitely think like uh in the
next in the next five years uh what
we're gonna see is
dptx systems like language models and
like different machine learning systems
as kind of like an augmenter of like
human italians where like there's gonna
be a human in control which is gonna be
prompting the machine learning system in
order to like uh produce uh
to produce a text to produce code that
then like the system is the human is
going to bet and decide what to pick i
actually wrote the abstract uh for my
paper using gpd3 part of it
and i expect this is gonna become like a
way more common occurrence uh in the
future i see this as different from like
what i was saying before we're like well
i was saying before like don't take it
with a grain of salt because this is
like me uh who uh who uh
who's like a relative outsider to like
the nitty gritty of like actually
training the systems like trying to
think about like how you could scale
gptx systems to like general
intelligence and be like not very
convinced about that well so this is
actually interesting because um there's
a lot of debate in the forecasting
community as i'm sure you know between
people who are like who adopt this
inside view perspective who say look the
best way to predict trends in uh in ai
capabilities is to talk to people who
are actually building these systems as
you say doing the nitty gritty work and
then people say no actually the outside
view is usually better because when
you're on the inside you kind of can't
see the forest for the trees and you i
mean i've seen this in in startups right
where like it's it's a classic thing in
silicon valley you have investors who
have like built let's say an edtech
startup
and
because they've built an edtech startup
they know all the ways that edtech
startups can fail they know all of the
the horrible list of things that need to
go perfectly right in order to make this
work and so they'll never invest in an
edtech product because they just see all
the reasons it can't work but then they
see a product in a completely other
domain that they know almost nothing
about and they get really excited about
it and often they make really good bets
as a result so paradoxically your
experience can actually detract from
your ability to make good predictions
i wonder how you see that interaction
acknowledging obviously we all have our
biases we all come from one perspective
or another as you mentioned if you're
more on the outside doing less building
maybe that'll default you to that side
but how do you think about that that
trade-off in the context of your work
absolutely so when i think about uh
expanding uh predicting trends in well
predicting like what's going to happen
with artificial intelligence with new
technologies in general like i see it as
like there's two things here there's
like trends and there are
discontinuities and like uh trends are
often are often like surprisingly robust
like uh there's this work by impacts
where like they
try to set up uh to try to look for
discontinuities so they could understand
better in which conditions do these
technological discontinuities happen and
like uh among like all the examples that
they looked at of technological trends
like actually they didn't find that they
found like
okay i gotta say they find like a lot of
discontinuities like you know around
like 30 percent of like the trends that
they looked at they found like a
discontinuity but they were explicitly
looking for for discontinuities so this
kind of like uh implies that like these
continuities are like somewhat rarer
that like one might naively think and
you can actually get like a pretty far
ahead which is like a lot in a lot in
like a straight line on like what you
have seen uh so far
now uh if you want to go the next level
then you need to account for the
possibility that uh there's gonna be
discontinuities like the one that we saw
in like 2016 with like a computer and
capabilities of machine learning systems
and for those like uh the best you can
do is like having like a having like an
inside view system like trying to
understand what are like the driving
forces behind like the different trends
that you see i like trying to understand
like how incentives might change in such
a way that this conditions happen or
like how the uh how some certain like
critical points might be uh might be
reached where like uh suddenly things go
faster one interesting insight is that
it's quite uh like we should naively
expect like these continuities to uh to
surprise us on the positive side
because if a discontinuity happens on
the negative side like kind of like
that's gonna be that's gonna be rolled
over by the trend right like the trend
is still gonna go on unless like
whatever are the forces like driving the
trends like uh subside but normally
that's gonna be uh normally that's gonna
be like uh something more gradual
unpredictable uh if you really want to
if you really want to uh so kind of like
as i see my work is providing like an
upper bound on like uh how uh how far
away intelligence can be and then like i
don't have that much to say being like
okay like i'm pretty i'm i'm somewhat
confident that by the end of the century
like we will have the resources to train
like artificial general intelligence as
an example like don't take this uh this
figure literally yeah but then i'm not
gonna have that much to say being like
okay it might come like 40 years earlier
we don't know something something
unexpected might happen and actually
that ties into
something we started with which was this
question of transformative artificial
intelligence and we tried we talked
about that definition and then you you
just raised in this context artificial
general intelligence and when we were
talking about biological anchors it
sounded like we were talking about
artificial general intelligence as well
just because we're focused on what would
it take to replicate the human brain
rather than what would it take to
transform society i'm wondering like do
you think that there's a
functional or important difference
between predicting tai versus predicting
agi predicting transformative ai versus
predicting general intelligence or are
they going to roughly come at the same
time and it basically won't matter
right uh like i think uh uh i think
there is like an important difference in
the sense of like
i think there is like some scenarios in
which like we get tai where like we
don't get like
strictly speaking agi in example like
being able to do like everything that a
human brain can do like you don't need
to be able to do everything that human
brain can do in order to like radically
transform and radically transform
society
uh
yeah i think we should focus on like
predicting like the minimal uh the the
automation of like the minimal amount of
transformative tasks with like actually
change society i like that's a smaller
subset i'd like artificial general
intelligence but i also like fully
expect that like
in most scenarios like these two things
come like
fairly attached to one another i guess
it's like for the same reason that
regardless of what your definition of
tai is you're gonna get it roughly right
because progress will be happening so
fast once you hit it that like for the
same reason tai and agi kind of become
pretty close just because progress is
happening so fast
unless there's a fundamental reason that
we can't get to agi as you say like
unless our algorithms just can't get
there for some reason we have yet to
discover it definitely seems like a like
a plausible um like a plausible scenario
um awesome hamiah thanks so much this is
just like absolutely fascinating great
overview of all your work is there
anywhere you'd recommend people go if
they want to follow this kind of ai
tracking work that you're doing
uh absolutely so i think that uh the
best the best way to find our work is in
the alignment forum there is a sequence
called trends in machine learning or
like there is an overview of summaries
of all our work which is a great entry
point to everything that we are doing
and that we're planning to it in the
future
fantastic okay i will uh i'll link to
that in the blog post that'll come with
podcasts i'll also link to your uh your
twitter account because i know you do a
lot of you know some interesting like
tweeting on this on this general topic
and in topics around this topic so um
yeah everybody definitely uh check that
out and uh honey thanks so much for for
joining me for this is a ton of fun
thank you for having me jeremy have a
great day


AI alignment and Redwood Research | Buck Shlegeris (CTO)

[Music]
hello everyone my name is oliver and
welcome to the q a with redwood research
cto buck schlieres
buck previously worked at miri studied
computer science and physics at the
australian national university and grew
up in townsville australia please join
me in welcoming buck
so as this is the q a um please put any
any and all questions uh in the swap
card chat in the question link
um and as the audience rates their
questions um buck could you first give
us a sense of how you view the problem
of alignment
yeah
so here's my here's my overall sketch of
uh the the beliefs i have about the ai
alignment problem and what it looks like
um
i i tried this morning to come up with a
new articulation of a bunch of this so
so some parts of it might be uh new and
flawed but i i'm excited for this
phrasing of everything
uh so to start with i want to clarify
the thing i'm about to say is the sketch
of the reason why i believe the stuff
that i believe rather than like the
short version of the argument uh this is
it's like a proof sketch in the sense
that uh the beginning of a math paper
with a long and complicated proof might
have a rough sense of where the argument
is going to go it's it's a map rather
than trying to claim to be a very short
proof and so that's that's very much
where i'm at i'm just trying to have it
so that people are going to know the way
that these ideas are structured inside
my own mind if you want a better longer
version of a bunch of these arguments i
think i particularly recommend
some of the stuff holden karonovsky has
written on his cold takes blog in the
most important century series recently
um ajaykotra from open phil also has a
long post on some stuff about ai
alignment which is going to come out in
about a month uh which also makes a
bunch of the arguments that i'm going to
say here uh in a much more detailed
fashion
so overall so i want to start out by
talking about artificial intelligence
and ignoring the concept of machine
learning so artificial intelligence is
basically uh just the hope that we will
or the the possibility that we will
eventually find a way of automating the
human actions of uh
doing intellectual labor so for example
doing science and technology development
or making decisions based on
understanding of the world derived from
experience and observation
and i guess like the first claim with
artificial intelligence is that
people will try and then succeed at
building systems which can automate this
type of activity uh basically because
it'll we can probably build systems that
are cheaper and better at doing it so in
the same way that humans have
substantially automated the process of
moving heavy objects from place to place
you know we have trucks we have
forklifts and so on and this is a
mixture of the fact that we can move
heavier things uh you know we can do it
we can do the move heavy objects task
both better and more cheaply similarly i
think that humans are strongly motivated
to come up with ways of doing science
and technology and decision making uh
better and more cheaply
so
building systems which are able to do
decision-making and science and
technology development uh better than
humans can probably requires according
to me building systems that uh
have long-term goals uh and consider
cleverly different options of ways that
they could act that are going to
cause them to succeed at achieving their
long-term goals and this is a
generically scary
this is a somewhat scary type of
situation to be in uh because if we
built these systems and they turned out
to have goals that were not compatible
with our goals uh then they would be
incentivized to act in ways that caused
them to have control over the future
instead of us and so the situation would
just be kind of like you know imagine
that you're some nation uh and there is
a hostile neighboring nation and you
have stuff that they want uh you might
end up in a war if they think that they
can beat you in a war just because you
have stuff that they want it seems to me
like this is kind of the default concern
with ai that we're going to try really
hard to build systems that maximize
goals
and if these goals are not aligned with
ours then we find ourselves at war with
a more advanced species um
i think like the tagline for this
perspective on the ai alignment problem
is yes like in the terminator uh i think
a lot of people like have talked about
ai alignment trying to distance the
concept from the concern that you might
build these ais that just you know want
to kill everyone and take their stuff
but from my perspective that is in fact
the main concern
uh
so
we're trying to build these really
powerful ais and this is kind of like a
really weird principal agent problem or
like a mechanism design problem in
economics uh so in economics like a
principal agent problem is something
like you know i want to sell my house uh
and what the way in order to do this i
have to hire a real estate agent because
i just don't know anything about the
local housing market and i try to come
up with some scheme for paying the real
estate agent such that they're going to
do a good job of finding like a good
price for the house
but in fact you know typically the deal
is you just pay the real estate agent
six percent of the sale price at the
house or something uh and so they aren't
incentivized to for example spend a
thousand dollars in order to increase
the sales price by two thousand dollars
uh
and this is an example of a case where
you wanted this other you know you had
to like defer like give a lot of trust
to this like other agent who is going to
do some stuff and you hope that it
worked out well for you uh but you
didn't have a full ability to oversee
them and so you
generically have trouble like forming uh
incentive structures for them such that
they do the thing you wanted so in the
case of ai it's a little different from
this because uh after building these
really powerful ai systems we will just
in fact have no recourse to do anything
with them um like the ai systems we're
just gonna like hand them the keys to
the world and we'll have no ability to
enforce any incentives on them at all
basically they're just going to be able
to do what they want uh so that's the
way this is harder than a typical
incentives problem uh the way in which
it's easier than a typical
incentive creation problem is that we
get to pick which ai we build and we get
to try to intentionally build a type of
ai that is going to do stuff that we
like
um and so i think that's the that's the
relationship between this is an
incentive problem and a large part of uh
thinking about the ai alignment problem
in my opinion is trying to think about
what types of incentives you can uh like
how you can shape uh building an ai such
that the ai that you create is in fact
going to do stuff that you will like
even though you have no way of
incentivizing or forcing it to do so
when it's actually making decisions in
the world
so
from the above i want to say that that
is an argument that ai is a big deal
uh you know i think it like could go
really wrong if we built ais that were
really powerful they would in fact have
the ability to kill everyone
um i think that it's not obvious from
the above argument that the probability
of death from ai or existential risk
from ai is you know more than a very
small number uh and so my brief analogy
for this is here are true clauses of
actions that aren't that dangerous so
one class or like one class of situation
that i'm sometimes in is i'm playing the
piano uh and the reason that playing the
piano is not very dangerous is basically
no way for me to screw it up badly
enough that i die uh there's just like
you know no sequence of keys such that
if i like tap him just wrong suddenly
i'm i've fallen over dead uh another
another situation that uh humans are
often in and that i have been in for at
least several hundred hours in my life
is driving a car so driving a car is
very different from playing a piano and
there are actually ways that you can
like mess it up enough that you just die
you can just uh fail to hit the brakes
you can just swerve suddenly uh and you
will get seriously injured or die that
said uh driving a car is in fact not
very dangerous i've done it for hundreds
of hours haven't gotten seriously
injured once um and so i think that from
the perspective of someone who wants to
ensure that human extinction uh isn't
gonna happen i think ai should like have
crossed your threshold based on the
kinds
like if i give you the full versions of
the arguments that i've mentioned so far
i think that this should be enough to
persuade someone that ai crosses the
threshold of something which is more
like driving a car than playing a piano
in that there is a way to like mess it
up badly enough that we all die uh and i
think there aren't actually that many
other things going on in the next
century that we could mess up badly
enough that we all die
i think i haven't argued for why it is
that this is any more likely to kill us
than driving a car for an hour is going
to kill me um and i think that the
arguments for that are somewhat subtle
and somewhat different uh and basically
go through arguing that the way that
this problem comes up there's gonna be a
bunch of things that are unprecedented
all in a row and it's gonna be hard to
slow down and avoid doing the dangerous
things
so overall i think that's my that's my
description of of the alignment problem
uh
and
so in all of that i didn't mention
anything which we've learned since 2005
uh and what we've learned since 2005 is
basically machine learning has emerged
as the most likely way that we're going
to build really powerful ai systems in
my opinion um and so this has updated us
towards thinking that these problems
might happen sooner than we thought
and it's also
made it so that the default guess as to
how we build these systems is somewhat
different than what it used to be in
particular it looks more like the way
we're going to get these systems is kind
of black boxy where we have to search
over systems based on their input output
behavior rather than based on their
internal structure which to some extent
affects uh what type of strategies for
technical and it makes sense
so that's why some of the alignment
problems we're going to build these
really powerful systems and it's the
kind of thing where if you did it badly
it could kill you and perhaps that's
even more likely than just uh
it would be in some other situations
that you could theoretically mess up
enough to kill you
great yeah thank you so much um and just
a reminder for those who want like sort
of fuller arguments uh check out
holden's cold take
series uh we had him as our keynote
speaker yesterday so hopefully people
are already familiar with that
um sort of following up on this like
you're currently cto of redwood research
so like given that this problem is like
sort of tricky and like could
potentially like destroy us all um
yeah what how does a writer research fit
into this problem of alignment
yep
so i'm worried about you know us
eventually being strongly incentivized
to build these really powerful systems
and in fact building these really
powerful goal-directed systems that are
not aligned with our goals uh there are
several technical difficulties uh
basically i feel like from where we are
right now it's more obvious how to build
systems that are really powerful look
like they're being really helpful and
then kill you than to build systems that
are really powerful look like they're
being helpful and then don't kill you
answer to a large extent this is just a
technical problem it's just suppose that
you have access to some systems uh and
you get to pick them based on their
input output behavior what kinds of
training schemes can you put into place
uh such that you end up with systems
that in fact don't kill you uh and you
know we can kind of describe the problem
of causing the world to not get killed
by ai as kind of two parts we have to
like solve this technical problem uh and
we have to cause it to be the case that
when people in fact build and deploy ai
systems they use whatever technical
whatever technical solutions have been
devised to ensure that the ais that they
build are in fact aligned with the
interests of their creators um and so we
at redwood research are just trying to
solve the first part of that problem
where we do the technical research to
make it
more doable to uh to align these
alignment systems in the future
[Music]
yeah sounds good um and so then like
what are the important sub problems of
alignment and like what are redwood
research's current projects like
tackling these sub problems
so i think i want to describe uh the
sub problems of alignment i want to
break alignment into two parts
um so
basically the way that you trade a
machine learning system is you come up
with some loss function which describes
how well it did at a particular task
um
and i claim that if your ai's
so so the thing we're going to do is
we're going to train systems to do
really well according to some loss
function that we're using as their
training objective uh for example some
current systems for instance if you want
to train a you know if you want to train
alphago to play go then you train uh
part of the you train one of the neural
nets involved to be really accurate at
guessing who is going to win a girl game
from a particular board state you train
another part of the neural network to uh
predict what actions are most likely to
lead you to win this board game um if
you're training gpt3
you train a system to predict what token
uh like what next word in this text is
most likely um
so we are going to have to pick some
loss function to give to our ais
and then we're going to train our ais to
do really well according to this loss
function so
suppose that we train this ai to do
really well according to a loss function
and then it kills us uh
question i thought that we were trying
to pick a loss function that would
produce an ai that in fact did good
things rather than bad things but it
sounds like we're hypothesizing that the
ai that we found that does really well
on this loss function in fact killed us
what went wrong i'm like we can break it
down into true possibilities for what
went wrong the first possibility is the
loss function was bad you know the loss
function was giving high reward
to actions that were in fact the kinds
of actions that would be
catastrophically bad if they happened in
the real world uh but just like subtly
so such that your loss function process
that's like looking at proposed actions
by the ai and trying to evaluate them
isn't doing
isn't doing a good job of evaluating
them like maybe these proposed actions
by the ai like have these like secret
subtle consequences that the uh the
system the evaluation system that's
making this loss function uh wasn't able
to notice um and so then when you deploy
the systems uh they keep doing the same
kinds of things they were doing at
training
and this eventually leads to disastrous
outcomes because uh you were training a
system based on a loss function that you
weren't actually happy about being
maximized
and the second possibility is that the
problem isn't the loss function is weak
the problem is that the system does
something different when you're
deploying it than the things that it was
doing at training time um so for example
perhaps your system is able to tell
whether it's in training or not um and
when it's in training it reasons as
follows it says well i am currently a
machine learning algorithm in the
process of being trained if i do
egregiously bad things the loss function
is going to notice them and then
deselect me and then some other ai
system is going to be the one that the
humans deploy instead uh which means
that i will not succeed at my goals of
building a squillion billion paper clips
throughout the whole universe therefore
i'd better give good answers right now
and then once your system is actually
being deployed it is no longer
constrained in such a way
um
and
then it is able to be like well now i am
deployed therefore there's no reason for
me to not just uh kill everyone if i can
see some way of doing so which if it's
really powerful it will be able to do so
i want to be like when you basically
have true these true sub problems
there's like making a loss function that
is powerful enough to notice if a
proposed action by the ai is actually
bad and then there's making sure that
the ai doesn't do stuff uh that it
wasn't doing during training or making
sure that like the most dangerous
actions that the ai might do during
deployment are all evaluated during
training so we can ensure that it
doesn't have this deployment only
failure possibility
uh so that would say those are like the
two main sub problems these kind of
correspond to what people sometimes call
outer alignment versus inner alignment
they kind of correspond to what paul
cristiano calls low stakes alignment uh
versus the robustness problem in a blog
post of his called low stakes alignment
um
i i like the separation into like was it
the lost functions problem or not
um
so those are my sub problems uh and what
is redwood research doing right now
well
uh one thing we're doing is working on
an adversarial training uh we're working
on an adversarial training
project so adversarial training is
trying to solve that second problem
where the system does bad things uh in
deployment but not during training um so
suppose they are worried about your
system uh doing good things only when
it's in training and then figuring out
that it's in deployment and then doing
bad things
an obvious class of algorithm to prevent
this from happening is to somehow uh
train it on the inputs where is most
likely to do bad things so you try to
have some process which is looking to
find the inputs on which the system
misbehaves and feeding it to the system
at training time so that it's no longer
possible for the system to have this
strategy of behaving well at training
time but not at deployment time because
you've already run it on like the
situations where it would be like most
tempted to do bad things uh
so that's like a long
so adversarial training is one class of
solution to this long term deployment
only failures problem
uh and we are currently working on
building tools for adversarial training
for current systems uh and this is
basically because we're hoping that the
things that we learn about how to do
adversarial training on some current
systems
are generalized in some way to uh doing
adversarial training on very slightly
smarter systems in two years time and so
on inductively all the way until we
actually need to adversarial trains and
systems to make sure they don't murder
us uh so the harp is that we can like
slightly push
this piece of this like part of the tech
tree forward a little bit faster so that
humanity is overall in a better place
for uh preventing deployment only
failures uh in in the future so this is
one project we've been doing that one
for um seven months you got a question
before we move on um
maybe like i know adversarial training
uh means something unique in uh machine
learning specifically i'm curious if you
could like sort of explain in more depth
like your adversarial training setup and
like what it's trying to like what kind
of behaviors is it trying to evoke in
your system
um
yeah so adversarial training in general
is where you want to verify the system
always has some behavior even in the
worst case rather than just in the
average case and so some of the input
data that you feed to your model
was chosen to make the model behave as
badly as possible according to the loss
function
the particular setup of ours i don't
think actually really matters we're
trying to train a classifier to be 100
reliable at doing a certain natural
language processing task um and the way
that we're trying to do this uh is so
it's quite easy to train a classifier to
be 99.99 reliable at this classification
task uh and then the question is how do
we train it to be more reliable than
that so the problem with just like
getting more training data is that
you're only going to find a model
mistake one time in a hundred thousand
data points because of the fact that
your model's already 99.99 reliable uh
and so it's extremely expensive to make
the model better just by getting uh
random training data uh and so what
we're trying to do is build web
interfaces and machine learning powered
tools that make it easier for humans to
construct examples where the model
fails at this classification task
which i think is in some ways analogous
to trying to build tools that humans and
ais can use in the future to find inputs
where our ai does bad things according
to a given loss function
great
the other thing we're doing
is we're working on some mechanistic
interpretability stuff
so mechanistic interpretability stuff uh
you know before i was saying that
machine learning systems sure act like
black boxes where the only way you get
to interact with your machine learning
system is you get to pick a loss
function and then your training
algorithm finds you a model that
empirically does well on the loss
function but you have no idea why uh and
in particular you have to worry that
maybe the system is giving you good good
answers uh just because it wants to kill
you later or something um
so the hope with mechanistic
interpretability is that we can in fact
understand the internal structure of our
models enough to be able to distinguish
between the hypothesis this model is
giving me helpful answers because it
wants to be helpful versus this model is
giving me helpful answers because it
thinks that this is a useful first step
in a plan where the last steps involve
killing me uh
and so we're doing kind of like uh
pretty detailed interpretability work on
sort of toy models in kind of a similar
yeah that's that's basically what we're
doing we're trying to investigate um
trying to produce like very detailed
explanations of certain types of model
behavior
uh and we've been doing this since about
about january
yeah um sort of first question from the
audience do you have like updates on
like how successful you feel like your
applied alignment research has been at
the moment and like which avenues sort
of like open up with your results
[Music]
yeah so how successful has it been i
don't know um
i feel like
i'm a lot less confused about a bunch of
stuff
than i was uh
than i was before we started doing this
research
i think there are some things about the
adversarial training problem that feel
in hindsight a lot more obvious after we
actually like ran our face against it
for a while um i think that in
particular it wasn't quite obvious to me
from the start that this project would
eventually turn into building a bunch of
tools to assist humans as they try to
find ways this model is wrong um i think
that like uh
for kind of technical reasons uh that i
don't think i have time to explain uh
the adversarial training problem
constantly tempts you into making a
certain class of mistake where you think
you can use a model in a certain way but
actually there's like kind of like a
nerve free lunch theorem reason that you
can't um and so i feel like i'm thinking
about it a lot more clearly now in terms
of what we've learned um
i think i don't know i think we've
learned a lot of little facts about like
what happens when you actually try to do
adversarial training with humans um i
feel optimistic
though not a hundred percent sure that
the things we've learned
will in fact be
helpful when we want to do the next
version of this
you know like like the goal of this
adversarial training project was to make
it so that us in a year's time or
whoever else it is that's doing the
average hour project in a year's time uh
is like in a somewhat better position
than they would have been if we'd done
something completely different uh you
know we're trying to pass this bucket of
water through forward through time to
the uh the fire which exists in 20 years
in the future or whenever agi happens
and this is a kind of complicated thing
to do uh and i don't think it's 100
clear that
we or anyone else really uh has in fact
made important contributions by the the
stuff we've done uh but i feel like
somewhat optimistic
yeah and wait have you have you like
tinkered with like uh doing automated
approaches to generating adversarial
examples
we have yes this is the thing which it
turns out is like much more
algorithmically confused that you might
have thought basically like there's
basically this nerf relaunch theorem
thing where it's like i'm like
so like the obvious idea for like how
i'm gonna like automatically find
adversarial examples for my classifier
is i'm like well i'm gonna train some
other model which is going to like
output strings that it thinks the
classifier will mislabel as um as
as good when they're actually bad or
whatever um
the problem with this is that in order
for your adversary to be able to do this
it has to know what's good and bad
better than the classifier does
um and it's not really clear why you
would expect this to be possible if
these two models are the same size um
and in particular it's not clear
is it i think it basically reduces to
saying like we're going to train two
different models both of whom are going
to be incentivized to know the
difference between good sentences and
bad sentences like they're both
classifiers but if you have two
classifiers you can just ensemble them
um
sorry sorry if this is like a slightly
more technical answer but yeah uh
automated adversarial training is like
in fact harder than you might have
thought and more conceptually fraught
with with current systems
in in a way that is slightly
disanalogous to
longer term systems very happy to give
longer answers to this for people who
want them at some point
yeah i think um heads up that like uh
we'll not be having a gun downtown q a
but we will be having a career fair so
uh go to the redwood booth if you want
to count more in the career fair
um so to follow up on the question uh
yeah have you guys seen any results with
respect to your interpretability stuff i
know this is like pretty recent so maybe
yeah
so we built some really cool tools um i
think that the the main difference
between how we're approaching
interpretability and how
other places are approaching
interpretability
there's a couple of differences but one
of them is that we're just like trying
way harder to build cool
interpretability tools than other places
are as far as i'm aware i think that our
tools for
uh certain aspects of interpretability
are
cooler and more powerful than tools i've
seen built anywhere else um and we've
only really been working on them for two
weeks um
in terms of like results about how
models do things um
i don't know i feel like we have a
somewhat good mechanistic understanding
of why in a particular small language
model after it sees an open paren it is
inclined to write a closed paren but
after it's seen a closed paren it no
longer thinks that it's inclined to it's
no longer thinks you should have more
closed parents but it doesn't think that
you should have closed parents after the
word the even if there's been an open
paren because it's quite weird for like
parenthesized sentences to have the um i
think we've like identified this like
very basic and easy behavior uh in this
like two layer attention only model uh
and we're currently
i think that like the type of
mechanistic explanation we have of this
is like in a certain way more
complicated and like clearly written
than any other interpretability results
i'm aware of on transformers and this is
partially just because we're like
zooming in on like a different subspace
of the problem than everyone else cares
than most other interpretability
researchers care about you know most
interpretability researchers are
interested in much
higher level and like less granular
understandings of much more realistic
and complicated models whereas we are
currently just focusing on understanding
as completely as possible um some
behaviors of the models that seem like
easiest to understand while they're
still being something non-trivial about
their understanding so i'm excited for
that um i'm hoping
i don't know what our publication
schedule and the interpretability stuff
is going to be i think it's plausible
that we start publishing stuff in a
couple of weeks or something um
but
unclear unclear
yeah
um
[Music]
can you comment on why you're focusing
on like applied language research um and
like where like nlp or computer vision
do you see the biggest opportunities
here
yeah so i mean i think that like the
applied alignment resource we're doing
uh
is very different from a lot of other
things that other people call applied
alignment research you know
basically i'm just interested in
stopping these scenarios where we build
these really powerful ais and then they
like intentionally kill us uh and i'm
basically like if we're working on
something that doesn't reduce the
probability that eventually when we
build really powerful ai systems they
like intentionally kill everybody that's
just like not my problem that's like
someone else's problem slash maybe it's
not a very big problem compared to the
problem if the system's intentionally
killing you so i think this is this
actually causes us to focus on a pretty
different subset of problems um
the reason that i am the reason that
we're doing applied alignment research
instead of for example uh working on
theoretical alignment research i don't
know i think about theoretical alignment
research sometimes i basically want
there to be i kind of just like when i
look at the world and what it needs in
terms of alignment research it kind of
feels to me like it actually needs a
bunch of applied alignment research to
happen um and redwood research
you know redwood research aspires to be
the place where the majority of the
value-adjusted applied alignment
research happens um
via you know being more focused than
others on doing the research which like
minimizes the probability of the agi
intentionally murdering your situation
and also being more uh
interested than others and like trying
really hard to scale up as an
organization um
and
yeah i basically just think it seems
like
i personally think that i should be
focusing on trying to like scale applied
line research that is really good uh
instead of thinking about theory stuff i
i don't think it's a slam dunk there are
many people who i think should do theory
stuff instead
yeah sounds good
well that is all the time we have for
this session uh thank you again so much
for coming to speak with us um
the career fair will be on gather town
um if everyone can like go into the
gather town there should be arrows
pointing to the career fair um redwood
research has a booth and so if you want
to ask more questions to buck feel free
to head to their booth
um yeah without it with that said uh
thank you buck again so much
thanks for having me
[Music]
[Music]
[Music]
so
[Music]
[Music]
you