Andy Jones - AI Safety and the Scaling Hypothesis

hi everyone jeremy here and welcome back
to the tourist data science podcast
on today's episode we'll be talking
about what's known as the ai scaling
hypothesis
and this is the idea that as ai
techniques like deep learning are scaled
up
using more compute resources larger
models and more data
their performance improves in ways that
we can actually predict
ahead of time now the scaling hypothesis
is really important for
independent ai researchers who can't
always afford to build
huge ais if they can build a small cheap
system and extrapolate its capabilities
using scaling models
then they can punch well above their
weight without needing an affiliation
with companies like microsoft or google
that normally sponsor this kind of
research now my guest today is andy
jones an independent ai safety
researcher who's been investigating some
surprising and fascinating aspects of
the scaling hypothesis
that have important implications for the
future not only of ai in general
but ai safety research in particular so
we'll discuss his work as well as what
it means for the big picture problems in
ai safety
and his motivations for working on those
problems as well i had a ton of fun with
this one
and i hope you enjoy the episode as much
as i did
all right well andy thanks so much for
joining me for the podcast
and thanks for inviting me along your
twitter bio describes you as an ndai
safety researcher which
i think is a really cool thing that
people who aren't in the ai safety space
might not really kind of understand or
recognize as an archetype
so i'd love to ask what brought you to
independent ai safety research in the
first place and also like what is your
day-to-day look like in that context
um i mean i will first disclaim that uh
i
i can't claim to be an archetype of any
sort or a prototype of any sort
and i do describe myself as an
independent ai safety researcher and
i think over the course of the
conversation we're about to have that
will become more clear what that means
um on the day-to-day what it means is
much like
much the same day-to-day that many of
your listeners will have um i used to
work as
a quanta trader as a data scientist of a
source for a hedge fund
and my work then since then um
independently has been largely the same
i type i'm a very specialized typist if
it comes down to it
um i read research papers
i go for walks and think about the
things that i'm trying to solve
and then i come home put them into code
and discover it's all broken it doesn't
work
and we repeat rinse repeats over and
over again and gradually progress is
made
and hopefully that's something that's a
familiar loop yeah oh
i'm sure it will be to a lot of people
and i'm curious like what brought you to
this
area because there are a lot of people
who you know work at the intersection of
let's say ai and
what i might call social good or
generally the good of humanity and they
go about it in different ways some
people do
research into ai ethics some people do
research into
you know how to deploy ai for use in
third world countries
stuff like that i'm curious about so why
ai safety and
i guess ai alignment that sort of
ecosystem well
it's there are many ways to answer this
question depending on how deep into the
past
and we want to go and i think the crooks
of there are two cruxes that are worth
mentioning
and the first is about five years ago i
was working for a hedge fund and making
very good money um let's not beat around
the bush here um
but i don't really oh i didn't i don't
really have many expensive things i care
for uh i don't think you want to live in
a large house i go on holiday
um and so i was mostly working to retire
very early
and when into my um work group at the
pla at the fund i was asked
came a friend adam gleave who is now a
researcher
at berkeley and also work on ai safety
stuff
and he introduced me to this effect of
altruism stuff and this idea that
if you've made it to the top of maslow's
pyramid maybe you should start thinking
about
other other things you can do in the
world in terms of making it better
and this is stuff like um donating to
antibularials
in the third world and deworming and so
on and so forth
so this was the first key turning point
that got me more interested
in the things beyond what was
immediately interesting that day
and just retiring early yeah for
whatever better justification
and over the next couple years and this
came to be a larger and larger part of
my life
i the best model i have for myself is
that i'm the average of who i've been
whoever i've been hanging out with these
past couple years and
after being introduced to effective
altruism there is a very large and
active group in london and these people
have come to be like
um most of my friends
and you can't help but turn into your
friends given time enough yeah
and one of the things that tends to
happen when people
spend a lot of time thinking about how
to make the world better is either they
will go quite hard on global health so
to speak
this is um anti-malarials alike or
they will come to believe that animals
have some amount of ethical utility as
humans do
and you end up going down the path of
well you can save many more animal lives
for a certain amount of money than you
can humans and this is a better
effective use of my time
and time and effort or you start to
think about
future humans and this idea that it is
easier to save
or preserve a life in the distant future
than it is today flapping the wings of
the butterfly
so to speak and this is this is nowhere
near as concrete as the core global
health stuff and i should put up front
that
i think there are very many good
arguments about this um over this
and i think there is a cynical take that
a bunch of computer scientists went out
looking for a way to improve the world
and found a computer science problem to
solve
um and certainly this kept me cynical
about the entire um
ai risk and x-risk extinction risk
and work for a very long time to the
point that
i am now on a regular basis grovelling
an apology to various friends who i
mocked about this for
many years and it now turns out that
they were right i was wrong
and i just have to live with that i
think that's a good place to
to start with this too everybody has
their own take on
well either how likely they think
extinction due to something like
ai is and also what the arguments are
they found really compelling that
brought them to that position i'd love
to hear what
what that is in your case i think the
main
place in which i disagree about the way
to go about helping the world to go
about thinking about the future which i
think is the core of this
is whether you think there is a fairly
narrow path that the future might take
and you have a strong belief in that
path
or whether you think there are many
different ways the future can go and
you're entirely uncertain about which
way it could go
so if you were the kind of person that
just believed
or if you were like a single past path
person um
the forecasting literature calls this
foxes versus hedgehogs are one of just
two nouns that will fit to a phrase
um the fox knows one big thing the
hedgehog hulk many little things
and there's a trade-off between this
feeds into how you think about the
future
but i am i cannot claim to be confident
about any particular
path that the feature is going to take
especially around ai
and but i think there are certain paths
with some small likelihood
that will both have some tremendous
impact on the way things will come to be
and which i can have a decisive
influence upon in this place and in this
time
and i think the usual analogy i bring up
here is
um i think there is like an
80 chance maybe that i'm a nuclear
physicist in 1890
and it's still 50 years or so before
anything interesting happens and i will
be old and great before anything
trust me happens and there's a 10 15
chance that i'm a nuclear physicist in
like 1939
and really it's time to get the notes
down to the ground get your nose down to
the grindstone and think about what's
going to happen here
and in other words specifically with
respect to ai that
we might be you know you think there's
like 10 to 20 chance wouldn't be on the
cusp of something like
general human level ai something like
that the idea of human level
intelligence is fairly charged
um and you know 100 years of sci-fi will
do this
um so the phrase that people are
generally more comfortable that i'm more
comfortable with when we discuss this is
simply transformative ai
and ai that will change things as we
know it
um something equivalent to the invention
of electricity though and
some uh gaps on twitter we'll call it
equivalent to the invention of fire
uh i think it's a bit strong that but i
understand where they're coming from
um and yeah and i think there is
some smaller chance it's putting exact
figures on this is
oh yeah a fool's game because someone
will come back in the future and point
to it and said you said a thing and
turned out to be wrong
yes that's unlikely but the point is
that there is some
i'm influenced largely by the small
chances of things that um
another analogy is if you go if you're
thinking about whether to play russian
roulette
it's not the five empty chambers you
spend your time thinking about yeah yeah
it's the one that's full despite
being on the unlikely case well and
there is an amusing sort of um
impotence of even experts when it comes
to that's notorious in science when it
comes to predicting the future i mean
i think if i'm remembering the names and
dates correctly it was you know
late 30s uh leo scissor or something you
know
had had given up essentially on the idea
of the the chain reaction
and then i think the day that he wrote
his letter officially giving up on this
uh there was a big breakthrough in
germany or some german lab where they
anyway the bottom line is these things
are inherently unpredictable right i
mean there's
i do not like that that's the case but
yes it does seem to be the case and in
fact i will make that anecdote when i'm
trying to explain this stuff in future
that's an excellent example
um i will say that quite often um these
same so leo
there is some um recall bias here in
that we mentioned say leo sizzler
being wrong about nuclear um energy
nuclear whichever whatever the anecdote
was about
shortly before it was in fact possible
but there are many more people in
history
who have said a thing turned out to be
wrong and then stayed wrong
and about 100 of them are called
nostradamus so
[Laughter]
yeah yes well there's the two ends of
your scale leo sizzling in 1939
yeah there you go okay good so i'll have
to use that in the future on a scale
from yeah
great well yeah so i think that that
provides some good context then in terms
of what brings you to this so
we there's a sense in which you know the
argument can be made that we ought to
worry
a lot about low probability very high
impact
uh outcomes so when it comes to ai what
are the specific outcomes
um that you are most concerned about
changing
or or addressing obviously specifics are
going to be hard to come by here because
none of us can predict the future
but just qualitatively if you can sketch
out maybe uh
i think it runs the gamut in terms of i
think the two axes to discuss these
things over its first
time horizon into the future so we could
talk about
the short scale impacts of
transformative ai such as mass
unemployment
and and then scale out to
much further distance things so or
hopefully much further distant things
about um
who ends up in how inheriting the galaxy
kind of stuff like real sci-fi stuff
and i think the thing that
is easier to talk about without sounding
too much like a crank is the short-term
impacts of things um and i think the
other axis to consider things along
is oh it's not so much an access as a
choice of reference class
i think when you talk about the way that
ai progresses um
you're largely influenced by what you
think aia is going to be like
whether it will turn out to be like
humans and whether it'll turn out for
you another mundane technology
like fire electricity um microchips the
internet
or whether it'll turn out to be like the
machine learning we have already
and i think these three sets of choices
of reference classes
they're very different oh they describe
broad
chunks of how you think the future may
go so
if for example you think ai is um
going to stand out to be like human
intelligence like if the idea is that if
you go about building an intelligence by
whatever mean you choose you come out
with something that looks like a human
intelligence and acts like to human
intelligence
then you may have ideas along the lines
of say robin hansen's
second mid 1990s so robin hansen is an
economist
who is one of those frustrating people
who was right a long time before i was
and i have a list of these people don't
worry i'm keeping track we're going to
mention a lot of names today
and one of robin hansen's key ideas was
the idea that
um if you can have a human-esque
consciousness
um running in simulation and you ask
this
human to design a microchip to run its
own simulation
then you have some kind of feedback loop
and this is one of the original
like conceptions of the singularity and
the more serious literature
and so you have the simulated scientist
who designs a better microchip to run
his own consciousness on
and you repeat the loop and gets faster
and faster and faster and faster
um and in that case the singularity i'm
not a huge fan of the concept because i
think it excuses people for reasoning
through it
um right like it's i i think
yeah i think it's a get out clause quite
often or it's
it can be the start of an interesting
conversation but i don't think it should
be the end of one
i i really agree with that i think it's
that's something i've never kind of
given myself
permission to think aloud but it very
much seems that way i mean by analogy
with
you know we always hear about physics
and quantum mechanics a black hole is a
singularity it's a place where all known
laws
break down and implicit in that is the
idea that we should stop thinking
about what happens at that stage sort of
seems seems similar here
um now now the robin hansen simulation
singularity where you talk about you
know the simulator simulates a
simulation which simulates simulation
and so on
i guess that that's one axis there's an
ai version of this as well or do you
consider those two distinct things
um sorry um can you rephrase that so i
get the sense
of what exactly you're asking so robin
hansen has this idea as you mentioned
that
you have a simulator who generates a
simulation you know that allows him to
live
at a faster rate you know so an
accelerated life basically have more
life experience
and in that simulation he can create a
new simulation that
accelerates it further so you
essentially have this like infinitely
increasing
rate of life experience collection um
with probably a bunch of physical bounds
like you'd actually probably run up to a
bunch of
of issues there in the limit but when we
talk about ai
there's it seems like there's a
different kind of
singularity that's discussed in that
context but i'm curious if you
if you think that those two might
actually be equivalent in some way or do
you really see them
as different things so the reason that i
was caught by a question
is when you said do you think there's an
ai version of this in my head there's no
distinction between
the version of this with human brains
running and the vessel with ai brains
running
i have drunk the kool-aid here um i
think of
um ais in a very similar way too i think
oh i use it yeah
it's hard for me to distinguish the two
but certainly um the equivalent or
maybe the most plausible equivalent as
of like this year might be
um you build a language model for
example
and which is a shiny new thing and you
get to the point where
it can write better code for writing
language models than humans can itself
and this is this is in a sense a there's
not as much agency in there it's not as
good of a sci-fi story in a sense but
it seems remarkably more plausible
sitting here in 2021
good okay for what it's worth that's
where i thought you were going i just
wanted to make it really explicit that
the mapping was there yeah so because i
think there are a lot of people who
who do struggle with that kind of
distinction and we're not used to
thinking of like
ai systems as general reasoning systems
things that can
make their own ais i mean in the limit
there's no reason why
an ai system couldn't if a human can
build an ai and improve ai's there's no
reason ais couldn't eventually do that
better than us and if that's the case
sort of like the same
acceleration as thesis applies right um
i
entirely agree and one of the we will
talk later about
um the various like anxieties i had
around the mergers of gt3
one of the core issues there is i went
from thinking that
um the first jobs that would be
automated by ai might be
a factory workers to thinking that the
first things that were automated by
ai might be ai scientists and speakers
an ai scientist myself that may be
much more nervous it's uh one thing it's
one thing to insult the luddites
uh when you think you're a mill owner
when it turns out you're the watch
the weaver you have a totally different
opinion yeah
i think that's a very fair point and a
good sort of introspective analysis
now maybe this gets us down the path
towards talking about scaling a little
bit
people talk a lot about you know making
big ai systems
larger ai systems scaled ai systems um
i'm sure a lot of listeners have heard
these terms like in the news and you
probably have a vague sense of
of what these things mean but i'd love
to hear from you as somebody who's done
research on scaling
what is meant by scaling like what is
the size of an ai system how can we
measure that and what are the the
measures that matter
it's not it's not even a good analogy
but the analogy that we have to work
with when it talks when we start to talk
about
um the about the size of intelligence in
a sense is we have to talk about humans
and animals
and everyone is familiar that there are
many animals on the planet with many
differently sized brains
and it's not a direct link between
the size of the brain the intelligence
to the animal for example quite famously
whales have brains
substantial much bigger than human
brains and do not seem to be anyone near
as smart
though douglas adams may disagree but
there is some
factor that we described that we usually
call the encephalization quotient
which is the size of the brain divided
by the size of the body mass because it
turns out you need a lot of brain just
to run a lot of nerves and a lot of the
muscles
and you take this then and you plot out
your rough impression of how smart an
animal is against this thing called the
encephalization quotient
then you get a fairly like direct trend
that animals with
bigger brain-to-body ratios do seem to
be smarter
and so in a sense bigger brain to body
more intelligence
and it was it is a long-running
discussion
in the people that study this kind of
thing about how seriously to take this
kind of a cephalization quotient whether
the key property of humans being as
smart as they are is in fact the size
their brains wells for their body
or whether there are other more subtle
structures in there
that actually dictates the emergence of
things like language
now in ai over the past
60 70 years how long has it been since
we first started working around neural
networks back in the 60s or so
as silicon has got cheaper gotten
cheaper has compute so to speak has
gotten cheaper as memory has gotten
cheaper
we've been able to build bigger and
bigger and bigger models
and around about 20 10ish we finally
figured out that
a lot of old techniques old simple
techniques deployed on modern fast
hardware deployed on gpus
and nvidia's gpu so the key to this
whole thing deployed in parallel
structures fast parallel
compute hardware could lead to
surprisingly intelligent output
you went from quite famously there was
an xkcd comic
sorry famously among the people that are
likely to be listening to this not
famously in the wider world
um bemoaning how hard it was to
distinguish
a bird from icaru what the contrasting
thing was but tried to
yeah yes trying to recognize a bird in
the image or something and this was
shortly before the emergence of neural
net image recognizes
and now it's something that a 14 year
old can knock together you know six
hours on a friday night
we have moved the goalposts quite
substantially in that we no longer think
of this
as intelligent behavior but it is
talking about quite strange i guess
someone who you know
got to go some machine learning in the
early 2010s i thought this was a hard
thing mods now is a very simple thing
um and we could come back later to like
this move in the goal post effect and
how
once upon a time humans proclaim things
like chess to be the ultimate human
intellect yeah
and people aren't so keen on doing that
anymore humans don't end up looking good
if you choose chess as your objective
but this progression in the size
of the hot the speed of the hardware we
use and
implicitly the size of the models and
the compute intensity of the models we
train on this hardware
and with the capabilities as hardware is
in
retrospect quite striking and about two
years ago
three years ago four years ago now so
the famous paid for this area was by
openai
but there was actually a small amount of
preceding work fris from a baidu
researcher
in about 2017 and then there was a bit
out of mit
bam by a guy whose name i cannot
remember right this second
looking at similar things in watching
about the same time that openly i were
talking about it
but the idea is is that if you take a
very specific
kind of ai you don't talk about ai in
general brains in general you take a
very specific one
and you increase the amount of hardware
available to us increase increase the
amount of um
ram memory available to us the amount of
time you're willing to train us the
amount of gpus that used to train it
then the quality of its predictions of
its abilities
increases in a fairly predictable manner
and it does this over many orders of
magnitude
i'm going to pick some numbers out of my
hat here these are certainly not the
ones of the paper
but the ones that open ai looks at and
i'll choose this because it is the word
it is the best known work
and it's certainly the most famous work
and they took a thing called the
language model and we discussed language
models briefly earlier
but the idea with a language model is
that you take the entire of the internet
or a large fraction of it
and you ask you give the internet a
sentence
from this giant corpus just one chosen
around them and you ask it to predict
the next word
and this is a startlingly simple uh
thing to try to do
just keep on trying to predict the next
word like autocomplete
autocomplete mad libs yeah um like
something very simple like this or even
just trying to annoy your brother by
trying to speak ahead of them and guess
what they're going to say next
but this very very simple task it tends
out if the model wants to do it well
um then it needs to acquire fairly
substantial understanding about the
world the way it works
the way the world works in that if
the model sees the word red then it
might want to it needs to know that fire
trucks aren't objects that are quite
frequently read
in order to be able to say that fire
truck is a likely next word and so
by training these things on billions of
words with lots of hardware
quite subtle patterns seem to get
embedded into the model
and the most impressive um
example of this is there is a guy called
gwen semi famous on certain parts of
twitter
and he keeps a collection of the various
things that these very highly trained
language models have produced
and the most impressive ones to me were
that when you have
say 10 20 billion parameters your
language model
and you can write a thing such as and
here is a harry potter parody in the
style of um
and the name of every famous author in
history has gone out of my head right
now um
in the style of shakespeare so to speak
yeah and the model will take these
things and say what is the most likely
word to happen next
i come out with a very impressive parody
of harry potter
in the style of shakespeare it just
picks up the idea of how shakespeare
speaks it's picked up the idea of
what characters are involved in harry
potter and there's a wizarding school
and what it's like
and all this from just trying to do
better predicting the next word
do better is a complicated term
obviously in this context but it seems
as if
there are a couple of different axes
that we need to pay attention to
separately so as we scale these models
as we give them more ram more compute
more data as well all at the same time
even if the model is relatively simple
we scale these simple things and then
not only do they become better at
carrying out the specific task that we
assign to them in this case something
like autocomplete but when you talk
about
you know like write this
harry potter story in the style of
shakespeare that's a task that i think
most people would not normally associate
with something like autocomplete
and it kind of hints at a level of
let's say cross task capability dare i
say generality that perhaps
might have been surprising is that an
accurate analysis or would you disagree
with that
um i would describe it as accurate
insofar as i was very surprised at the
time
um and certainly looking oh as with many
things once you know it's there you look
back and find it in many other places
um in that again i've dropped the cool
kool-aid here and i do not consider that
what what you're describing here is
generally called meta learning
um the capability to learn on one task
and coincidentally be able to specialize
in another task
and download or do better on the tasks
down the line and
and at the time i was very surprised in
hindsight it seems
surprising not to think this would
happen that if you
ask a model to predict words and
sequence and you give it enough hardware
and computers to be able to do this very
very very well
then in order to do in order to get this
tiny tiny improvement and be able to get
the next word it needs to acquire like
fairly complex models about how things
are
and then this carries over to other
places all this structure is just
carried around with it gets it for free
and i think again i'm reaching for
analogies here
and but if you look at a baby who just
playing with their blocks
and observing the world playing with the
blocks and just gobbling along
and one way to think about it is that
they may be developing
internally some very sophisticated
models about the way the world works the
way that euclidean geometry works the
way that gravity works
and despite not having any real theory
for how these things work how any
understanding of how these things work
and yet they can interact with blocks
they could juggle them a few years later
just fine
and so yeah they've trained on this like
cell this is all called self-supervised
learning in contrast to supervised
learning in supervised learning you
usually have some kind of label telling
you
what is a good example and what is a bad
example whereas in self-supervised
learning
you are just trying to predict some of
the bit of structure in the data
and the wonderful thing about
self-supervised learning is just comes
for free for living in the world
so this is interesting because one of
the big take-homes appears to be that
although gpt-3 and there are all kinds
of crazy examples
as you say guern and others have
highlighted some really amazing things
gpd3 can do that
you know we've talked about let's say
translating between
uh literary styles here between jk
rowling and shakespeare
but of course there's also translating
between languages
and some basic um coding like converting
written work into code and basic web
design work stuff like that
that does appear shocking if you
if you don't first think of all machine
learning as a as an instance of meta
learning
there's like some metal learning behind
most stuff some principles that
generalize
i'm curious where you now see looking
back where you might now see
some some hints of this in maybe earlier
experiments that precede gpd3
the easy place the cheap place the lazy
place to look for these kind of things
is in earlier language models um in that
going back to the days of markov chains
even so a mark of chain is an extremely
simple pre-neural network
style of machine learning where you just
build a table of if you see these four
words then what is the most likely one
next
and this is so simple most people almost
no one is willing to qualify it as
intelligence
but still um these things will produce
interesting bad loops
interesting like very just okay i won't
even qualify
i can hardly even call it like poetry or
whatever it's too basic for this really
but certainly new and novel structures
different from the ones that were in
their training set
and this is this is so
boring really that it doesn't qualify
the word the phrase meta learning and
that's why we didn't call it metal
lending at the time
it's only in hindsight seeing what it
actually happens when it's scaled up
and that we start giving it a different
name to just
a personal prejudice here that i suspect
the reason we have come up with the
phrase metal learning
is that for a very long time we did
machine learning on very narrow data
sets
and and so there wasn't really any place
in which it could surprise us
and we trained image recognition models
and we gave it images to look at
and whereas there's a
if you go look at chris ohler's work on
microscope ai
and his discovery that if you train very
good image recognition models
that comes up with concepts of geography
and the like internally this is
it's not quite the same thing as stupid
three but it's the same kind of
surprisingly complex structure
discovered from surprisingly simple
inputs
so this actually gets to something that
i've often wondered about
um language models so it seems like
language
prediction or autocomplete whatever this
task is that we're describing the gpd3
does
lends itself very directly to generality
because language
encodes all of our knowledge of the
world it's like the most direct way
humans have
it's also optimized for communication so
it's easy to recognize when it's being
used
but like would you say now given the
chris ola image examples
would you say that it's the case that
perhaps most tasks when performed at
outrageously
high scales outrageously well eventually
force the system that's performing them
to develop a kind of generality a kind
of flexibility across
tasks or is that is that overbroad as a
statement
yes it is overbroad but yes i agree with
this um
oh my my main okay i am very much on
board with the general idea that if you
take
some let's call these broad tasks things
like recognizing
objects and images or predicting words
in language and or maybe
maybe in the future it will be
predicting frames and videos
or operating some fleet of drones say
and so to speak
that if these things have to interact
with the world at large in some sense
possibly through some very narrow window
of say language or text or whatever then
yes they do seem to have to pick up
very complex models in order to be able
to do that test task well and it doesn't
seem to be
contagious on this particular on the
particular kind of data you are feeding
them
having said that my personal suspicion
is that
yes you are right language is special in
a way that all the data modalities are
in that we collectively as a
civilization as a species
have spent ten thousand years optimizing
what i don't know how long language has
been around for
but we've spent many thousands of years
optimizing language to be
a transport mechanism for describing the
world around us
i would expect that language is a much
more dense medium
for encoding just the things we care
about
um in a way that images or frames of
videos or
general and operation of robotic bodies
might not be
so yes this is one of those things again
is not surprising in hindsight that
language models would be the first thing
to hit this like threshold of
interesting generalization
just because the medium they work on is
information dense in the front in the
sense that images just aren't
i should qualify a lot of the things i'm
saying here um
sorry i say i qualify i think and in
that i'm telling stories here which
do seem to me to make sense and by the
way that you're not along it makes sense
so at least one other person in the
universe
but this is not to say that there are
not many other accomplished researchers
many who have been on this podcast
who deeply and valuably disagree with
everything i'm saying here
actually that's a that's a really good
point and i think it speaks to
earlier you were referring to the
uncertainty bounds around the sort of
you know low probability and high impact
events and i think it's fair to say that
the low probability part comes
perhaps in large part at least in
significant part from um
from the objections of other people who
know a lot more than
than certainly me but so it's processing
me but
but the thing is that there are i mean
ultimately there are no experts as the
the nuclear um sort of chain reaction
example shows
predicting this stuff is really hard and
in that universe you kind of have to
head your bets and make sure you're
you're kind of being cautious so i
definitely appreciate that hedge i think
it's a responsible thing to do
um that said continuing uh under the
assumption that we're we're both exactly
right about this naturally
so i think this brings us quite neatly
to the moment
the gpt-3 first comes out i think it
first came out actually as a paper
and somewhat silently i mean people
recognized it as like oh wow this is a
cool advance
but it was only once you started seeing
some of the beta testing some of the use
cases
on for me at least on twitter that i
kind of went
oh my god something big is happening
here and
this could be really serious so yeah i
think i
very much sympathize with that um the
there was the original um so there was
the neural there was
a paper called scaling laws in neural
language models which came out
um early last year early 2020 and
and i've read this paper at the time and
i dismissed it and it did not click for
me whatever i thought okay another
interesting result coming through the
archive today
great show um and then three or four
months
later um this
gpg3 was announced in a narrow sense
that they had a very nice blog post
and a paper and gave beta testing access
beta testing access to a handful of
people
and this is where you started to see
various examples percolated through the
internet
um and again i have to give gwen credits
here
in that he collected a lot of very
impressive examples and this is
his newsletter and his um exclamations
around us that convinced me that this
was a big thing
now for background here um i've been
keeping track of developments in ai
for many years and i start in this space
about 2012
2013 um and over that time there have
been
quite a few instances of what i in
hindsight called ai anxiety
that i have my idea of what ai should be
able to do this year
and then it is blown through in some
sense and the earliest example of this
was the atari work out of deep mind in
the first half of the last decade
um i had a good few days where i was
like wow this is this is a lot better
than i thought it was going to be
but i very much managed to move the goal
posts i convinced myself that this is
you know
not as impressive as it first seemed to
be and then the same
with um go and al m
starcraft and with dota and then
gpt3 was just the kick of the teeth that
was too
it was hard to move the golf host as far
as they needed to in this case um
and that was what convinced me to sit
down and think very very hard
about what place i would want to have in
this brave new world
and there is there are two artifacts
from my thinking in this time
there is one reddit post in which i am
fairly panicky
and there is one less wrong post in
which i am fairly panicky um
and in hindsight i am sort of
embarrassed by
you know the strength of my response
there but i think it's an honest
response and it reflects
the trajectory that brought me to where
i am now so these two posts were
basically going
well it's not going to stop here guys is
it it's not
if if the problem is really that i keep
on having to
move goal posts then we should put the
goal post on treadles here and they'll
just keep on moving them how fast do
they have to move to keep up with things
so if you accept that the goalposts are
now going to move at a steady and
increasing rate
um the first thing to do is turn around
and go well how fast are they going to
move what's the limiter here
um what determines the progress in ai
over the last
10 15 20 years and the big enabler there
are other things but the big enabler is
the development
and proliferation of cheap compute now
many of your listeners will likely be
aware of moore's law uh this goes back
to the 1970s and says that
every 18 months uh the density of
transistors on
a die will double and this is a very
narrow prediction this is only useful
within um
very specific parts of the semiconductor
manufacturing industry
but it's being hijacked as a general
description for the progress of
compute as a whole um so i think i'm
happy to talk about moore's law in the
sense of
gpus getting faster despite the fact
that the transistors are no longer
getting
smaller that race um i'm not there such
like um
but yeah when you start to think well if
compute doubles every 18 ish
months and there are a lot of
predictions around this and if you're
interested they're interested in this
um go google around for us and you'll
find much better predictions than the
ones i'm giving you here
but if you assume that it progresses at
some increasing rate then what should
you expect
in the coming years and how much bigger
will models be how much faster will they
grow
um and expect the other factor is how
much more money will go into them
right so one of the big questions that
you
and by the way i i should mention the um
as as nervous as you mentioned you were
as you
wrote up in particular this post on les
wrong um i'm quite convinced that there
was a minor ecosystem of people
sort of nervously scratching their chins
as they collectively read that post i
being one of them
i remember reading it and going like wow
yeah this is exactly
the um the sort of panic feeling i'm
experiencing here and to see it laid out
i mean the
the case you were you were building i
think it'd be worth exploring that case
specifically sort of
where you saw things going at that
moment and then exploring as well like
where your thinking has gone since then
because i understand that it's evolved a
little bit too
so um the thing so when i first
started writing uh this post which i'm
sure i'll be linked at some point
uh yeah yeah um
my main concern was the rate of compute
increase and that gives you
one trajectory for how fast ai ai will
get better
but really ai has been a bit of a
backwater of research
these past 50 years it's something that
university professors do for funsies
it's what you do after you've got tenure
and it's only been the last few years
that
it started started to look like an
economically viable
industry that you could put money into
training these very big models
and then go sell their outputs to people
and get more money back
and the moment something goes from being
like a fun little research project
to an economically viable industry the
amounts of cash
involved change dramatically and so you
have nikola tesla playing around on his
mounted top
at one point and then you have like a
national grid 20 30 years later
and like the moment serious corporations
and governments get involved in this
kind of thing
and it goes from budgets on the order of
like a few hundred thousand
uh dollars per run and that was like a
few years ago and that was considered
very expensive at the time
up to tens of millions hundreds of
millions billions of dollars going
forward
and this this kind of dwarfs the
progress and hardware and this is the
thing that really got me
that if some people in the right spots
in the hierarchies are very wealthy
corporations governments decide that
this is an important thing to pursue
things will change very very rapidly um
now what has qualified my panic since
then is
um it's when i wrote the post
i was deliberately a coward and i didn't
put any concrete predictions
in there because i felt if i did this
i'd have egg on my face a year later
but looking back in and out i wish i had
because it would be a good lesson having
the egg on my face
about the willingness to get panics
about things if you're not willing to
make predictions
at the same time what predictions would
you have made i think at the time i was
expecting
a similar leap to gpg3 in the next 12 to
18
24 months i'm about halfway through that
period now and it's been fairly quiet
um and it could be that's because the
big players are still
revving up for some kind of big
publication in the near future
um or it could be because it turns out
that raising more than 10 million
dollars for a training run
is a hard thing to do and the peak kind
of people you need to convince are just
not willing to put that gives you that
kind of money
or it might be that there is some kind
of hardware limitation that if you want
to bootstrap together a thousand gpus
that's one thing
but ten thousand gpus is an entirely
different to kettle of fish and coming
up with the infrastructure to do this
kind of thing is much harder than it
might have been
so this was it was what i
described at the time i think was an
upper bound on progress and then i over
focused on the upper round
but this is this is me speaking from the
other end of the hedonic treadmill where
i've adjusted to life in this brave new
world and i've been outside that's
totally okay
well it does seem like the underlying
argument though that the moment that
machine learning switches from becoming
this money sync to a money-making
machine
all of a sudden you you kind of reach
like a sort of economic
escape velocity where people pour money
in
they're able to build bigger systems
scale more those systems can do more
and and so on so you sort of like start
along what is i guess something like
a roughly exponential trajectory though
i'm sure there are different arguments
for the shape of it but
yeah i so i broadly agree with this but
i will so
i will qualify the phrase again
um the panicked way to look at this is
that we are entering into the takeoff
and now this economic loop is closed and
more and more money will be fed into
this system
the slightly more um oh
the more sanity preserving version
is that this trajectory has been running
since the 1700s
that the moment someone built a machine
that built a machine
that was all she wrote and we've been
living through the takeoff since then
and then it it became into the much more
pedestrian problem of the fact that the
world is moving faster than you want it
to
and this is something that everyone
throughout time back to the ancient
greeks complaining about how kids these
days were writing things on tablets
rather
memorizing them um has been complaining
about this is just an old people's
problem
yeah and that's that that's what i cling
on to here that
in fact um this is going back this is
the reference class of ais and other
technology rather than anything shiny
and new
this definitely seems to align with that
so i have had a conversation with um
david rudman who works at
open philanthropy and he put together a
essentially a research project on the
economics or the history of economics
really the history of human output over
time
and he was describing how most economic
not most i should be more careful that
many economic models end up going to
infinity in finite time they end up
predicting that the output of the human
species will actually reach infinity
in a finite amount of time so this is
not exponential because exponentials you
know they go to infinity in infinite
time they just do it really fast
this is literally you you know you break
the economic ceiling of the universe
in a you know by 20 whatever or 30
whatever
yeah it's every sigmoid looks like an
exponential when you're in the middle of
us
and every exponential turns out to be a
sigmoid because you run out of space
eventually
right that's it and it's so yeah i guess
that's that's another deep question is
like how singular is the singularity and
what is that upper limit
but um yeah distinct from that i mean i
guess
one of the observations that he made was
this was an
admittedly very crude simulation that he
was running i mean it has to be you're
looking back ten thousand years what are
you gonna do
but he found that most civilizations um
that he simulated starting about you
know circa ten thousand years ago
do end up in this terminal state where
the value goes to infinity
um or uh or it crashes to zero because
you deplete all the resources and
there's a disaster but
you know details details i guess the the
bottom line is it sort of speaks to your
point that
there's a sense in which you could say
that yeah the the moment i don't know if
it's the moment
language was invented and all of a
sudden we closed a sort of ideation loop
or the moment the telephone was but
they're different honestly it was rna
that was the mistake
well and that kind of has you zoom out
even more and ask you know is the
universe really fundamentally this
bundle of particles
that are parameterized by locations and
directions and those parameters jumble
together like an evolutionary algorithm
that ultimately
this is jeremy the quantum physicist
coming out i can tell
yeah i've i've lost the plot now but
anyway the point is
there's a big picture folks if we have
lost the floss then we lost the block
quite a long time ago in this year
that's fair to your point though it's
like you know when you get into the game
of saying like no
now is the time that this really we hit
escape velocity
it sort of becomes a a bit of an
unsolvable problem in that sense
so we've talked about my anxiety my ai
anxiety about this
and then it might be worth moving on to
the ways i've tried to cope about this
you've said that there's
a cottage community of people panicking
about stuff and to be clear i'm not i'm
no way the first person to get to this
as i say robin hansen was getting well
talked about this in the 1990s
um and eliza yodowski and the less wrong
folk have made
like a living out of panicking about
this stuff for the last 20 years
um and again another group people who
are very right and i was very wrong
um and i'll i'll hopefully get to
apologize to them about that someday
um but for me the arc of the past 12
months
has been i have discovered over
the course of my life that the best way
to deal with my anxieties is to work on
my anxieties
to normalize them and bring them into
the set of things that i can make
progress on
and ai has turned out to be one more
thing like that that
while this might be you know um the
future of the universe
if they're going to be really uh
hyperbolic about things
there is some small part of it that i
can focus on and say this is mine and i
can make progress here and so
in the months after my panicked less
wrong post i did
a lot of reading on um what other people
have thought so far about
what's called ai safety or ai risk so ai
risk is the broadest area
ai safety is a slightly narrower area
about okay what do we do about this ai
risk
and then ai alignment is possibly an
even narrower set of
okay and how can we get ai's to do what
the thing we want as a way to play
towards safety reducing ai risk
and there's a fair chunk of material
written about this and by
people who thought very long and hard
about it and who are much smarter than i
am
and my goal in reading all of this was
to try and find a space in which i
thought i might have a comparative
advantage
um so the effects of altruists that i
have hung out with the last few years
have this tendency to analyze problems
to work on in terms of importance
neglectedness and tractability so the
kind of thing that you want to look at
if you're looking for
air for a way to improve the world it's
something that's important to work on
uh something that's neglected in the
sense that no one else is working on it
yet
so something like climate change um
extremely important problem
but it has a lot of very very good
people working people who are far
smarter and better resource than me it's
not likely that i can make a serious
contribution there and tractable in the
sense that
if your problem is like if you've chosen
original sin
as your problem that is like important
and neglected it's not tractable mate
step off
um find something else but in my reading
up i found i made a list of what i call
my bad idea list one of my favorite ways
to have research ideas is to accumulate
over the course of a few months
a lot of very bad ideas and hope that
against this backdrop of very bad ideas
something emerges which looks like
slightly less bad than all the others um
and the one that i settled on was this
idea of this paper that i
like this tender to the paper i
eventually wrote of
that if my problem is very if
if my source of my anxiety is very large
um ais
um but me with like my fairly cheap
standard of living
can only afford very very small ais then
what can i use the small
ais to say about the big ais and this
leads into the area of scaling laws
because this is exactly what scaling
laws do they say
well we've taken a very small ai and it
performs like this
we've taken a small larger larger one it
performs like this and if we continue on
like this maybe we could say something
about
ai's far bigger than the ones we could
train this would be like the analogy b
if you see an insect be air behave in
one way and you see a rasp behave in
another and a monkey behavior another
maybe you could say something like
humans
and i think that analogy is accurate in
how inaccurate your predictions are
likely to be about any specific trait
but the general complexity of the
behavior the trends of the things would
hopefully be apparent um though i am
very much
like this is what's called exposed
reasoning i'm coming up with an
explanation after having observed the
evidence it's cheap it's easy
oh it's the funnest kind of reasoning
though absolutely love it
yeah make a career out of it but having
decided
having identified scaling laws as like a
viable way to connect the small to the
large
and i then started because i am even
compared to the people who are studying
scaling laws i am very resource bound
and so the thing i caught on to is
whether as well as scaling the size
of the agent of the size of the ai
maybe i could scale the size of the
problem it was interacting with
with like a vain hope that maybe a small
ai
solving a simple problem tells you
something about big ai solving complex
problems
like this would in a sense we know this
to be true this is what we've done all
along
in all of research ever we've taken very
simple um
like crayon style models of the universe
solved something there
and then use that to say something much
more interesting about the world we live
in
um but it hasn't it isn't off it isn't
often systematized though having said
that
i am not at all surprised in hindsight
again that those physicists
that were involved in the creation of
the scaling laws um research project in
ai
what are some examples of scaling laws
like how how might a scaling loss sound
when you state it to somebody who hasn't
hasn't heard the scaling laws before so
the easiest one to get
a grip on is probably in image
recognition
and that if you have so much compute
that you put into an image recognition
model of a certain architecture if you
scale
if you give it this particular size then
you could say that it will
um perform on this benchmark on this
very large benchmark data set of images
with this accuracy say 85 percent
say you have i'm pulling numbers out of
my hat again but if you have a 10
million
um parameter model they will give you
say 85 accuracy
and if you have 100 million parameter
model it'll give you say 93
accuracy and every order of magnitude
you add to it
it knocks another fraction between you
and 100 of the accuracy off
and in language models you might have a
similar thing that um
in so i hesitate to use language models
as an example here
because the way in which you evaluate a
language model is the entropy of its
output
and it's just not something anyone has a
handle on unless you actually study this
stuff
for a living but in the same way if you
have a 10 million parameter language
model
then it gives an entropy of output of
this much if you have a hundred million
it's somewhat less and keeps going and
you could draw a line through these
things and say
well based on the 10 billion 100 million
models here is how i think a billion
model parameter
model will go to connect this to my
earlier like thought passes i decided
that um maybe scaling the problem
and scaling the ai simultaneously was an
interesting research direction to take i
would allow me to say
something interesting about very big
problems using very small amounts of
compute
but having done that the next question
is what kind of problem do you choose to
work on
and and my sense there was it would be a
good choice to work on something of fail
something fairly neglected
so image recognition models and language
models are
people know these are cool and fun and
that they do really interesting things
and so there is a lot of competition and
i have a general aversion to working on
anything that lots of other people work
on because it means it's going to be
hard
there are lots of smart people in the
world and just don't
go again pick your battles pick
something different
and games
meanwhile have a lot of work on some
specific games
go and chess famous famously but the
nice thing about games
is that unlike languages or images it's
widely regarded that there's an easier
way to make that there's a way there's a
clear way to make them easier and harder
in that if you play go on a three by
three board
it's yeah a three-year-old can figure it
out
it doesn't take go on a three by three
board is basically like tic tac toe
um go on a 19 by 19 board is something
that
stumps grand masters and this year still
a substantially open problem
so i've sorry i've mixed together two
motivations there the first motivation
is work on something that's less
um studied games and the second
work on something that's easy to scale
up and down in a way that images and
languages to start there's no real sense
of how you make language easier or
harder
you could take good attacks on it you
could talk about formal languages or
generative languages
but it's a fairly underdeveloped field
um
games meanwhile games have got some
serious work done on them by
again deepmind and there is an algorithm
called alpha zero which is
this is i think alpha zero will turn out
to be one of the more important
algorithms of our time
um so should we take this detour or
yeah yeah i i think this is this is good
context because your paper is based on
um on that class of algorithm so
okay so this again to try and connect
this more clearly um
having decided i was going to work on
games because right level and
neglectedness
and um right level right easy to scale
the next question is what kind of
algorithms exist to play games
play board games games board games and
the clear winner here is an algorithm
called alpha zero out of deep mind
about four or five years back and it was
invented in parallel with
a lesser out known algorithm called
exit i think it was something
amplification
there were two papers that turned off
roughly a similar time
which explored this idea that
if you have a neural network that plays
a board game fairly well
how can you use that to generate a
network which plays it better
how can you amplify the network and the
way they do this is by making lots of
copies of the network in some sense
and having them all play out in slightly
different versions of how this game
could go
and then turning this into a target to
train the original network again
against so it's a bootstrapping setup in
a sense um
you can certainly dive into the
technical details if you want there are
a lot of very good explanations out
there
i think um paul cristiano's take on this
as what he calls an iterated
distillation and amplification settle is
particularly insightful without getting
into too much of the gory details
um but yeah this is this is an algorithm
that i'm very much impressed by because
of this bootstrapping property that
could start off knowing very very little
and that look better than any human
um and so this performed this formed the
foundation of
my work because the nice thing about
these kinds of algorithms
is they require no input other than the
rules of the game they are like
they are independent in some sense um
isolates they require a very little
setup i didn't need to go collect any
extraneous data like other kinds of
game-playing algorithms will need
and so what were the big because as you
say you were
you were able to tune the complexity of
the problem which is unusual
i think incidentally that feels related
to the idea that
the problems that humans find hard are
not always the problems that machines
find hard
and so to the degree that we can very
straightforwardly say that like
oh yeah a game of um a game of go with
uh
you know these dimensions is clearly
harder than a game of go with fewer
dimensions
um it's almost like that's intuitive to
us
partly because it's a problem class that
we didn't evolve to find trivially easy
whereas language is this like really
hard problem class i think objectively
but because evolution gradually
structured the like the physical form of
our brains to find specifically that
kind of problem very tractable
it almost creates this illusion that
that's actually like a much easier class
of problem
so this is okay this is a bit of a
detour but i think it's an interesting
detour in that there is an explicit
hypothesis about this
that the kind of kinds of problems that
humans came to recently
things like chess and are
um surprisingly maybe the easiest ones
to automate and it's the really old
capabilities like image recognition and
language and motion are much harder
because we have a lot of free circuitry
given to us by our ancestors
and a lot of mistakes were made in the
discovery of these things these results
resulted in very well designed solutions
whereas for chess for example it's been
around for a few hundred years and
we haven't rea there's no selective
pressures for one and no real cultural
pressure either to come up with
very complex and much um subtle
solutions having
done a lot of work on games i will still
say that language is
there is a temptation in any um kind of
machine learning research to say that
the
problem you are in particularly working
on is the important one the key to
intelligence i will not claim that to
board games for a second
i think board games are a fast i think
board games are fascinating because they
get at some of the real
hard cores of intelligence without a lot
of
extraneous stuff on top um so for
example
um some of the work out of open ai
recently
on um oh a better example um
a lot of reinforcement learning research
in this day and age is done on three and
done in 3d environments
and trying to get ages to navigate them
move things but into one place to the
other
and i can't help but think that what you
are mostly doing in that case
is trying to take changes to see seeing
is the hard part of that
and then there are some very simple
behaviors strapped on top um
and i think board games are wonderful in
that they get rid of
all but the most all but like the
interesting behavior the the purest
representation of the interesting
behavior
of like planning or deception or this
kind of thing
so it's like feature extraction is the
hard piece and then how you kind of
jumble those features
usually falls yeah i mean chess was you
know
so chess was popular for so long is so
popular because it's supposed to be some
kind of
um reduction of the art of war and war
is this you know huge complex
terrible topic um that you would have to
build
you would have to have so many data
ingestion systems to be able to actually
train an ai that did something
interesting with the concept of
all of war and we're training on chess
which is like what he was thinking was
like the distillation
of war is we could do that in an
afternoon that you can get the data
ingestion for that setup in the
afternoon
it's like a very pure representation of
what i find interesting about ai
behavior
well which is part of what i think is so
exciting about what you're doing because
there's there's something about picking
you picked a game called hex which maybe
you can explain a little bit in a moment
like what the rules are but
it's a very simple game and it really
does allow you to quite directly map
your sort of intuitions about like how
how hard is a certain scale of this game
and how surprised should i be that an ai
can hit it with this much computer
whatever
um would you mind kind of yeah
explaining a little bit of that game in
the context
sure so what we're going to discuss here
is a mistake i made
um in that we have established over the
previous
questions that i've decided to work on
um
scaling ais and board games
simultaneously i'm going to use alpha
zero to play these games
and then the choice is which particular
board game to work on
um and the obvious choices are chess and
go these are like the well-known ones
um a chess doesn't have an obvious way
to make it larger or smaller or
more harder or more easily but there are
like adaptations you could do this can
be done
um but what i
the problem with chess and go is that
because they are games played by humans
humans have added rules to them that
make the games more interesting to play
but which make for a much more painful
implementation
um so chess for example has castling
so this is like an extra little twiddly
piss that you have to add in your
implementation
um and chess is in fact there are so
many other bits of chess i think about
around
move repetition um that the kind of
kinds of people that implement chess
engines
like for like their hobby and have built
fairly serious bits of debugging
infrastructure for figuring out whether
your implementation of chess is actually
right or not
and this just this just scared the hell
out of me that if i was going to give
six months over to this project
um then i didn't want to spend three
months with building a chess
implementation yeah this is
this is not what i want to be doing and
go meanwhile is nowhere near as bad as
chess in terms of these like fiddly
little rules but it still has what's
called the code
rule or the super code rule which says
that no position can be repeated
um and this is if you want to write a
slow implementation to go this is a
trivial thing to do
if you want to write an implementation a
go that runs like 100 million steps a
second
then it's a lot more unpleasant to deal
with and so i turned to hex
so hex is a game that was invented in
the 1940s
and has developed somewhat of a cult
following among mathematicians and
logicians
because of its utter purity as a game it
has a much simpler rule set
and the easiest way to get your head
around hex is to just go look it up
rather help me explain it in words
but i will try and attempt anyway in hex
you have a hexagonal board
and so each cell is hexagonal and it
lies in a rhombus
and so like a fallen over uh watch a
square
and there are four sides there are two
players and each player gets the
opposite side of the board
and the objective of each player is to
connect their sides of the board with
tokens
and to do this each player takes their
turn and places one token on the board
either black or white
and the first player to connect their
size the board wins and that's it
there are no other get rules to the game
it's superbly simple this is what
attracted
it's kind of like a connect four in a
different geometry in a way
yeah yeah and but this was a serious
mistake on my behalf because it now
means to explain my research i now have
to explain hex
and that's the point it's a real
hangover it would have taken me a few
more weeks to get go working and it
would have been a substantially easier
project to pitch to everyone ever
if i made that choice so yeah grass is
greener though i mean i i do think
having looked at
at the rules of hex in in your paper i
mean they're pretty straightforward
one quick gif gets you there and uh but
but yes so it was really interesting
here was as you scaled the complexity of
the underlying problem
you saw i mean i don't know if it's
correct to say a new kind of scaling law
was this like a truly novel result had
anybody else seen stuff like this before
i okay um i am going to
i have to eat some humility here in that
um we give the the major explanation of
why i
undertook this work um but one of the
hypotheses i had going in was whether
um the scaling with the problem size
with making the board bigger
would be nice and smooth or whether
whether it would be spiky
and spiky would be so what
very interesting and extremely alarming
because it would say
you can have your ais trained on small
problems and they'll behave in a
predictable manner and then you make the
problem big enough and it suddenly stops
being predictable um
and this would be this this would be i'd
be shouting this from the rooftops if it
turned out to be the case
and sorry do you mind if i just ask a
little bit more detail that when you say
spikiness
can you explain like what the x-axis is
and what the y-axis is
absolutely so um okay well i'll explain
my results then we'll go back and talk
about
um the way they could have been yeah um
so
i took many different sized ais and
played them on many different size
boards
um and i ranked the ais in terms of
their performance
and i found that as you make um the
amount of compute given to the ai
given to the eye the size of the ai
bigger then its performance increases in
a fairly steady
steady manner and in particular if you
double the amount of compute
allocated to an ai then it does about it
will beat
the old version of itself about two
thirds of the time so if you have your
opponent with so much compute if you
have twice as much compute you can win
two thirds of the time that's like the
nicest
twitter tagline sale of this thing and
also
um this is the same across board sizes
so you see the same thing happening on
size three size four slide five size six
seven
size eight nine which is where i ran out
of compute nine is the biggest i looked
at
um and what's more you can look at the
performance these ais and using
the small airs on small boards you can
predict the performance of large ais and
large
large ais and large boards like it is
extremely predictable
you add one to the board size and the
amount of compute you require to hit the
same performance
goes up by about a factor of 10. so
these are the two like
key things that um double your compute
with two thirds of the time add one to
the board size ten times as expensive
um it is yeah i i i could not believe
when it came out to such round numbers
um and i am yeah i am this is like some
amount of salesmanship here in that the
numbers not are not quite that
but they're close enough that it's
easier to remember than like 7 or
9.3 or whatever well you made a really
interesting point in
in the paper itself one of my first
thoughts when i looked at it was okay
are we learning here something about
the algorithm something about alpha you
know alpha zero
or are we learning something about the
universe something
much more fundamental that um you
actually touch on in
in the uh the paper where you draw this
analogy to
how one might imagine very speculatively
this process unfolding where it comes
well actually why don't i just
ask you to explain that because i
thought that was so cool if if the bit
if it's the bit that i think you're
talking about
so um some backstory here um
so i decided on this research project
and
i will come back to place because it
might be interest to your viewers i went
out and got a small amount of funding
from an east independent research
organization
uh called survival and flourishing
that's a fun to be over six months of
worth of like
fairly cheap runway um and then i had in
late january some preliminary results i
was wondering are these interesting and
more importantly
are these just like batshit crazy
because if your results are absolutely
crazy
it means you need to go back and check
on working check that you had introduced
a bug somewhere
because as an independent researcher as
the only person that looks at your code
it's very easy to come up with things
that are just totally wrong
and not even realize it until a lot of
people have seen your work and you have
to go back and talk to them all and say
i'm really sorry about that it's not
what i thought it was going to be
but when i put out these results um a
guy
called paul cristiano who is well
extremely well regarded in the ai safety
community
left a tiny little comment to my google
docs saying it's rather strange that
when you double your computer you get
two-thirds of a win rate
that is roughly what would happen if you
had
just like two monkeys and they each had
like
a bag of possible strategies and
proportional amount of compute you've
given them and they're just pulling
strategies out of the hat
and the one that pulls out the better
strategy wins this is
this is roughly what you get in that
case um or
the monkeys have no a clearer thing if
the monkeys have bags of numbers
and they pull numbers out the bag you
get as many numbers as you have compute
and the monkey with the highest number
wins and you can make a pitch you can
make an intuitive claim that maybe this
is what
our very very complex deep thinking ais
are doing and that
for each out to compete that you give
them they find one new strategy
and they play it against the other
strategy the other guys playing and the
one with the better strategy just wins
now i need i need to qualify this too
in that i studied one game with one
algorithm
on one set of hyper parameters and it
might it
i can only really say that it's
serendipity that it came so close to
double repeat to two-thirds
it is a time likely it is easy to come
up with thought experiments where this
just isn't the case
um that's this behavior just can't hold
so the question is whether on
other natural problems with other
algorithms or the hyperparameters you
still see similar behavior or just like
you're totally different
and it it is interesting i mean if you
if you if you did it seems like you
would say something about
the process of problem solving that
there's a sense in which
it all reduces to trial and error like
there's a sense in which intelligence
itself
really is this relatively stupid process
just
directed in a fairly constructive way
would you agree with that um i would be
extremely sympathetic to that
again all my all my reasoning here is
unfortunately written through with deep
uncertainty about the work the way the
world is
um but i i
i have long been struck by how many
processors
in the world that we think are very
smart do have the dynamics of random
walks
um so there is a fairly serious
chunk of work over the past 20 years or
so in economics
on why despite if you like make a list
of all the major discoveries over the
last
however many years and you look at the
size of the research community at the
time these discoveries are made
you might expect that the recent
communities today today
being like 100 times the size as it was
100 years ago
might make 100 times as many discoveries
as it's just not the case they still
seem to turn up major discoveries
still seem to stand up at the same rate
as they ever did and
the best one explanation for this is
that
we're a bunch of people wandering around
in an ever increasing search space
and we're just doing a random walk and
pretending it's anything else
um and we need to walk ever further for
each new discovery because this is like
a broader frontier to cover
of course it being my work and me being
keen on it turning out to be very you
know
um deep and meaningful i'm very keen on
the interpretation that this is what all
problem solving is
whether it's actually the case or not
give me 20 years and i'll be able to
tell you
yeah there we go but now if if it is the
case just just even
you suggested that this is good news
from the standpoint of
ai safety predictability scaling and so
on
could you connect those dots so when i
was
after i proposed a project and after i
started off this i got to thinking what
kind of results would i actually
have to have and i wasn't a good enough
scientist to actually pre-register what
i expected to happen
because there's just no need to do good
science just
right now machine learning is too easy
to get good results just falling off out
the sky
it's amazing it's really a guard of the
reason for research
um but one thing that i was particularly
interested in
is whether the trends would be smooth as
they turned out to be
or whether as you made the ai is
particularly big or
the problems particularly hard you would
see some kind of
um jump some kind of shock
and where the ai would come up on some
realization or the problem would support
some kind of realization
that just made it suddenly better
there'd be a strategy
that a 10 million ai could not discover
that 10 million one parameter could this
ai could discover and if this had turned
out to be the case this would be like
alarm bells ringing because it means
that we could trip over any day now an
ai had a problem set that was large
enough to get some very
surprising behavior out the other side
because a lot of danger not all danger
but like a good chunk of danger comes
from things not behaving in the way that
you thought they would
um the analogy i like to deploy here is
that
through the lives of some very poor
nuclear scientists in the 1940s
we know how hard how close to holding
plutonium spheres together
we know that if we if they come less
than a screwdriver's width apart
then harry daglian dies and just don't
do that again in future
yeah don't repeat that and i kind of
hope with
the scaling those research we can get a
similar kind of thing for ai as we do
have the nuclear materials or say spark
gaps and electricity
we know how close together you can get
to the higher voltage lines
unfortunately very bad happens
and there are lots of people the ai
safety community
who will not consider this a good
approach because eventually someone
somewhere
or put the wires too close together or
the plutonium rods too close together
and then extremely bad things might
happen but i think as a very prosaic
very mundane sort of safety it is a
thing worth exploring and the fact that
it didn't show up in this particular
problem
is like it's a glimmer of hope that
maybe these things are predictable and
we can tell when bad things happen
without actually having to have the bad
things happen
yeah there's a ton of overlap with the
broader question of safety
even even the the broader course of
human history we talked earlier about
the david rudman sort of long timeline
thing
one of the debates that he was talking
about was whether
it was this question about whether human
progress has actually been continuous or
discrete
whether the history of human economic
achievement and advancement
is to be told through these like
step-wise moves or
through relatively continuous
improvements and i was surprised by the
strengths of the arguments that would
say that things like the industrial
revolution
were not actually a a step function that
you know even
specific discoveries have some level of
continuity to them
it'd be interesting to see if this
generalizes beyond the sort of simple
problem class of like hex absolutely and
there is an issue here in that what
you're trying to do
is a proof of universality in the sense
so
in mathematics it's generally um agreed
upon that it is easier to prove that a
thing exists
and then no thing exists prove that not
everything doesn't exist it's much
harder
like you have to prove non-existence
much harder to prove existence where you
just show the thing
um so if we had one interesting problem
uh where scaling the ai or the problem
size
suddenly went from predictable behavior
to unpredictable behave oh
you were surprised by what happened as
it got bigger and this would be
you only want it to happen at one time
but if we look at
again and again if we look at natural
problems interesting problems and we
find no
surprising behavior this would be
increasingly good evidence that
um we'll know we're in trouble before
we're actually in trouble
for non-existence it's i have a sample
size of one
that's right yeah not so that you want
to bet the farm on it's a start yeah
but but there is i think a sense in
which there are two different kinds of
continuity here there is like
whether this question of whether the
universe is fundamentally continuous in
terms of problem solving
or whether in practice we will
interact with the universe in a way that
makes it seem as if it's continuous what
i'm getting at here is
you know openai didn't make a model that
was epsilon bigger
than the model that was one step before
they made this
giant jump to 175 billion parameters
10xing what came before which 10x what
came before so we tend to take these
very
large leaps in model size which through
scaling work we seem to know
you know correspond to fairly big leaps
in performance but
there's a qualitative change that came
with um gpd3 which i think was pretty
surprising
on the generality side and i guess one
thing i'm curious about is like
while all these scaling curves that i've
seen they seem to do a great job of
predicting the performance of these
models on a specific task
the generality thing seems to me to be
pretty unexplored at this point like
we don't have a metric for it do we some
people are trying
and there are there they're doing this
benchmark data sets but yes you're right
that measuring generality is hard in a
sense that
when your model can do a thing you stop
stop thinking that as general behavior
um right and it's like it's a god of the
gaps phenomenon
the generality is the model doing a
surprising doing a surprisingly good job
of things that you were not expected to
do
but the moment you expect it to do well
on a thing it doesn't count anymore
um but to get on to to focus on
i think the question of whether to think
about things as step functions
or as continuous phenomena
is a very deep and difficult one i think
a lot of it hinges on your choice of
scale
so for example um when i was born that
was a very important event for me
um but you zoom out and you look at the
world as a whole
a lot of people were born that day and
it's much easier to think about it as a
continuous form of this is how many
people are born each day rather than
this how many of
me were born that day um and i think
yeah it's choosing the right scale to
think about the universe
is oh those just how do you even start
on that
i mean you're the physicist i know this
is you know your your back your back
garden of choosing the scale at which
like
um capillary and motion works out versus
resuming it and certainly mercury's
processing in the wrong way
um i mean i think in what you usually do
in those contacts is
you you pick the thing you pick the
scale that's most relevant to the
phenomenon you're interested in and in
this case from an ai safety standpoint
you know we talk about you know ais that
have the ability to make ai's and
improve on ai research that sort of
thing that sort of seems to be like
that that's a very ill-defined term
right and we're glossing over a whole
bunch of detail which arguably is
precisely the phenomenon you're
gesturing at with this
but uh giving myself that out i mean i
might imagine
leaps of that size where the the kinds
of leaps that um
that we find i i have no idea how you'd
quantify this i'm just i'm talking
myself in circles i guess
i think you're completely right is the
short answer
um okay well let's okay so yes i think
you're right that the choice of scale
hinges on the phenomena you want to
investigate
and so we'll take um the future of ai as
an example
where you can make one thesis that the
key point is when an ai can write its
own software
um but it is very plausible that ai will
be helping to write its own software for
a long time before that it actually does
it is almost certain that ai is already
used by nvidia to optimize the layout of
its chips
and it will almost certainly be used as
code complete for programmers long
before it's actually writing full
software
and so this is going to be a thing
that's won by inches in a sense
and it'll be a long time there'll be no
specified point at which we go that was
the moment that ai wrote itself
um like because certain people will they
will
even when it when even when the first ai
writes its own software stack end to end
there'll be someone that goes well it
didn't make the chips did it it didn't
run the factory
um go to the gaps again well does that
imply its own challenge for safety i
mean humans are notoriously bad when it
comes to incremental
especially exponential incremental
phenomena i mean you only have to look
at kovid to say
well you know it's only one case in the
city and well it's only a small cluster
well it's only ten percent of the
population you know we kind of gradually
um
have this this goal post moving that
you've alluded to uh it depends
whose perspective okay i think this
plays into a fairly difficult question
in ethics
of whether what is good is what is good
from the perspective of you right now
or what is good is what is good from the
perspective of the future you
in the sense that um we are we consider
our society today to be much more
ethical and humane than historic
societies
but if you dug up some pilgrims the
1500s and drop them into
society they'll be horrified they'll be
obviously yeah
um and in the same way we might now
look at things like ai induced
unemployment and go that's terrible
like um work is required for human
dignity
and in 50 years time people could look
at us like complete
morons why would you ever think that
work is key to human difficulty what's
wrong with you
um and i am frankly entirely stuck
on whether i should reason about what is
a good world from the perspective of
what i today would consider to be a good
world
what i in the future would consider to
be the future god world and
slightly more frustratingly
uh upsettingly what the survivors of
what in 100 years the people that are
still around the entities that are still
around
uh will have been selected in a fairly
serious way based on
who did well in the world that has been
and they will very likely think that
what happened was a good thing because
it allowed them to turn out to be
who they were and is that a thing i want
to optimize for
um yeah and certainly how do i trade
that off against projecting my own
ethics into the future
um i have no answers here whatsoever but
i think i think will mccaskill
mccaskill's doing some fairly serious
work on this
and i assume other philosophers are but
his his is the big name that i remember
and what's the apology of all the other
philosophers who have looked into this
no i think it's a great point and it
shows you how inescapable the tie-ins to
all kinds of different philosophical and
moral questions
are i mean this really is an
intrinsically multi-disciplinary topic
and and to your point about you know the
new the future versions of ourselves
having to be respected to some degree to
make their own decisions
it sort of the flip side of that to me
is you look at something like twitter
which is
designed to be incrementally seductive
and you know
i i right now recognize myself as having
an addiction to different forms of
social media
that coax me into interacting with them
through this sort of like frog and hot
water a gradual effect
you know you could imagine something
similar happening with ai where we get
this economic benefit
and it feels good until it all of a
sudden doesn't yeah i think you can
look for historic parallels here and say
that um
as with all things they worked out very
well for some people very badly for
other people
what do we use that to say about on
average i
it's choosing your reference class is
the key thing here and i just
i not the thing just how to do it
properly i'm still stuck between
choosing
humans mundane technology or machine
learning that we have is like
the key thing way to think about the
future and also about whether the
better analogy is like genetic
engineering where
we clamped down pretty hard on it in the
like 1990s
and a lot of people consider it a very
good thing and some people who are
worried about food oh
you can make the pitch for example that
the lockdown on genetic engineering in
the 1990s was very bad for people who
wanted cheap food
and it's very good for some
environmental movements very badly
others
you can make the pitch that nuclear the
lockdown on nuclear energy was very good
from the perspective of preventing
proliferation
and radiation hazards but very bad for
the perspective of we've built a lot
more coal than we needed to
um and yeah it's
i am struck it's almost it's a cliche
and naive thing to say but i'm struck by
the
subjectiveness of these decisions um and
i have really no other way to resolve
them than just try and think very hard
talk to a lot of people
um and hopefully we figure this out all
together
that's a really important kind of dose
of humility to infuse into all this as
well
and it's something as well it seems like
there's also the countervailing risk
that
i see in myself especially with politics
where
i i'm sort of like like proudly confused
in politics and i think
and i'm embarrassed of the proud part if
that makes sense
as paradoxical as that seems like i
recognize that things are very
confusing and complex and that it's very
easy to get suck downs or partisan
rabbit holes and and get
sort of lose your identity in this
process and yet at the same time i also
know that these
areas are unquestioning unquestionably
important and
so neutrality for pure neutrality's sake
is its own
sir failure mode and yet i can't
disentangle those things
this seems like one instance of that
right where like the far future is super
important
and yet what should we do like you can
get lost in epistemology and
moral and uh this is
yeah it's um there's a phrase for this
that ends with paralysis by carrying
what the the first noun is o and l
paralysis and possibly yes tool
paralysis something like that
not tool paralysis a different thing but
the same kind of concept of just
future shocking but present shock that
you look at the world around you and
it's great an
enormous complexity and go how the heck
do i even what where do i start
yeah and all i can offer is that all
your ancestors likely did this
and then you are at the end of a long
line of people
who are terrified with the complexity of
the world around them and yet managed to
accomplish something
and there is like again some kind of
hedonic treadmill in the sense of
you start off terrified at this thing
but you do get on with your life are the
people there at the end are the people
who just
got on with things and tried to find
some small piece they could work on and
delegated the rest of the rest to
someone else
um and you're right that
i will equally admit to being confused
about an awful lot of things politics
included
um i can't say i'm i think proud
i think yeah i say i would say that
proud confusedness is very close to
humility
and that's the thing that i'm entirely
on board with um
but no you've caught me out here like
all i've got here is like mumbling about
various
but i think that sort of summarizes what
confusion is right i mean at the end of
the day
yes to the extent that that's the point
here but but i do think
there's something to this idea that um
that blind panic especially in the face
of change and change induced by ai and
ai risk
which is something that i fell into it's
something that i definitely saw on
twitter an awful lot around this time
from the ai safety ai risk world i think
there's a sense in which it is
self-defeating and counterproductive at
a certain at a certain point
where people resign themselves to like
oh well there's going to be an
apocalypse and and whatever
first off i'm not sure that maps on the
objective probabilities we're dealing
with here
but also it it kind of strips you of
your agency would you take that sort of
fatalistic
mindset as well there's a phrase from a
fantasy book i'm a friendly series i'm
keen on which is the important thing is
to get into the future uh
don't get hung up on whether what you
believe right now is the true thing or
the best thing
is if that's going to lead to paralysis
then put that aside for a moment
and just work on getting to the future
and certainly
i have friends who have found themselves
paralyzed by self-doubt they will not
allow them to do themselves
they feel that they are not particularly
good at a thing and they
their doubt about their ability to do i
think stops them from getting any better
that's like that's the
key what's your um characterization of
this i think something that everyone can
sympathize with
um yeah and it turns out the way you
deal with that
is that it's harsh advice but you deal
with it
uh if it helps to find a community
people that sympathize if it helps to
read
um marcus aurelius
or cbt or whatever they go ahead but the
key thing is that you get yourself into
the future you do something
you know you will end up being the
person that got past it
but this is we're getting into pop
psychology here i'm not i i don't think
i'm offering anything you are different
other than it applies to me too
i i think pop psychology and the parts
of it that people in ai safety has
distilled as being particularly relevant
is sort of an interesting selection
study in and of itself that i think
on that highly pragmatic note uh i
really appreciate it
we've i think we've we've run over our
officially allotted time here but i just
enjoyed the conversation so much
absolutely this has been wonderful yeah
thanks so much for sharing your insights
andy and um yeah
oh sorry go ahead the i i need to make
one last
a few call out oh one last call out
which is that um
if you are interested in doing this kind
of work yourself if you're listening
and you think what maybe i can do
independent research
i would strongly suggest you look into
it this is another part of just
if you see a problem in the world put
your put it on your own shoulders and
walk
and there are organizations which will
lay out cash to people that they think
have promising research problems
and the ones to look into are what are
called the long-term future fund
which is run by an effective outreach
charity and more recently there is one
called survival and flourishing which is
the one that i got my small amount of
cash
and these are not rich organizations not
a thing that can you can make your
living apparently
but if you're looking for a way to get
out of whatever
income trap you're in at the moment then
i think it's an excellent place to start
that's a great point i'm really glad you
brought that up actually so i will make
sure we include links to
those and there might be a couple a
couple more funds i'm trying to think i
think absolutely if you know more then
please answer the list here
okay great yeah but i think those are
great and survival and flourishing in
particular i think um
that's the one that funded your specific
research right into yes
okay great so yeah we'll link all to all
those and thanks for sharing your story
and your suggestions as well uh at the
end there andy really appreciate it
yeah thanks for listening listening to
me for an hour and a half yeah
it's been great