Robin Hanson on AI Takeoff Scenarios - AI Go Foom?

welcome over the last over the last many
years the northern have actually
recorded a number of interviews but
today we're going to be exploring AI
kpop scenarios with reference to a
previous debate an AI Foom debate with
le AZ and calcium plum that happened
many years ago on series of blog post
and video as well so there's plenty of
room for thought ha ha ha ha ok so Robin
Hanson is associate professor at George
Mason University and research associate
at the future of humanity manages to
dachshund and it's also a blogger and
overcoming fires which is between
extremely influential and I say over the
many years in many areas on heuristics
and biases spiritual and Industry scenic
forecasting economics and a whole lot
more and it's also where I believe our
less one was born so Robin is the author
of two books the age of M gets right
there that's the elephant in the brain
and there must be a copy of age of them
somewhere there it is yes right covers
both yeah
and actual both of which we have
discussed in previous interviews so a
warm welcome back to Robin Hanson many
things join you again it's it's great to
be here um do you mind if I set a stage
you can alright well I'll set the stage
with Bond villains you see so if you've
seen the classic James Bond movies they
tend to have a villain and often the
villain has a secret island somewhere
where he has a secret set of henchmen
who have been developing technology and
this technology is often competitive
with huge nation states technologies
even though there seems to be only a few
dozen people there and this is a
tradition that goes way back in science
fiction of some small group developing
this powerful technology that
the major industries and nations of the
world so even say 20,000 Leagues Under
the Sea
imagines a small group of sailors who go
off to form their own Island and they
develop nuclear-powered submarines and
nuclear weapons and a whole underwater
cities all impressive technology far
beyond what nations and industries of
the time can do and in many people's
minds this is a plausible thing it's
because in their minds I think
technology is very lumpy there's just a
small number of major technological
innovations that happen and if you have
one of those why you have this leap
forward and you can go well beyond lots
of other people in an area and in
people's minds they they don't see
technology as having that many parts and
so they see you know with the history of
Technology and maybe there's 12 things
that ever happened or something and so
if you get one of those 12 things while
you leap far ahead of others and you
know this is how a lot of people see
technology they see it concentrated in
this one invention and a big broad
invention that has enormous impact and
that all the other smaller inventions
and all the other developments and all
the other details about making it work
in particular context is of minor
importance relative to the grand
invention this is the narrative we've
been telling people for a long time
about technology is that there are these
few genius inventors and they make these
brilliant inventions and that's most of
everything that matters mm-hmm and even
when people worry say about capitalism
or you know what will happen with trends
of technology that they tend to imagine
is relatively easy for say one firm to
take over the world hmm yeah one company
buys up all the other companies and it
makes everything itself and it has its
market power and it controls everything
and this is a terrible outcome and Marx
and Engels long ago predicted in their
communist predictions that this would be
the inevitable result of capitalism more
and more concentration into a smaller
number of firms who had more and more
market power who crushed the
life out of the workers and forced them
being misery did and took over Nations
and ran the world by having this huge
concentration of fire and many people's
minds that's that's just a plausible
direction capitalism would go
it's just what would naturally happen if
regulators weren't there to stop it and
so if in your mind these seem like
plausible scenarios then something like
the standard AI Foom scenario seems
plausible to I would say because it's a
similar sort of scenario so the the
standard AI Foom scenario of which this
debate was about was the scenario that
we've all been working on AI for years
now but we've been making slow difficult
progress but at some point some small
team somewhere has the one big adventure
the one huge insight that allows this
small team to make vast progress
compared to not only the other teams are
around but the entire rest of the world
and so this AI now this team is able to
make such rapid progress that they
whoosh past all the milestones that we
see ahead of us
past human-level AI past cheap
human-level AI just super intelligence
and this small team therefore has this
super intelligent machine which they may
or may not have control of and they
therefore rule the world and this could
happen not only in a short time like a
year but it may be in a month or a
weekend rounded by minions in jumpsuits
they looked very suspicious right so if
you thought the villain the James Bond
villain having his super weapons on his
little island was plausible and you
thought one firm taking over the world
was possible then you might think it's
plausible that there's this one super
invention that makes small team able to
basically take over the world right and
so the whole sorry go ahead
if one team like in like isolated in the
island somewhere
doesn't know doesn't have access did the
Internet I mean can i download a whole
heap of modules and just make the right
adjustments and put them together in the
same way is maybe a like you know a team
in like you know a bunker and Silicon
Valley someone even do this has often
been one of the hypothesized advantages
of the James Bond villain team is that
they can read the world's newspapers and
the world scientific journals and
they're getting all the advantage of
other people's and developments but
they're not sharing in reverse so people
imagine there's this advantage for being
from being secret where other people are
disadvantaged relative to you if your
project is small in secret and then
there's it's not though I mean
supervillain karna developed super
intelligence and work but we don't want
to supervillain directing the values of
super intelligence right so um you know
one thing to say is what if we just
assume this scenario as possible
and then we might say well yeah sure if
this scenario is possible then you might
want to fill the world with some designs
that when the supervillain team makes
their super villain project and they are
trying to be a little careful there are
some safety guidelines that they can
pick up and follow that will make their
project a little safer so at least their
machine doesn't get out of their control
and so we are only under the rule of
world rule of this supervillain team and
not their superior robot
so that's sure obviously and under this
scenario the key point is it's very
unpredictable when it might happen and
therefore you got to be working on this
way ahead of time you can't tell when
it's going to happen you can't tell when
you're close by assumption here because
this this development is these lumpy
developments are just very hard to
predict they're they're very random they
don't come with much warning and they're
so huge that of course if one of these
things is coming you would need to be
working ahead of time well before it
seems like you might be needed because
you
know when you'll be needed and you'll be
wanting to work out ways to help these
people keep their machine under control
the supervillain small island team
mmm-hmm and of course lawsuits would say
it's is not a zero probability I'm not
gonna argue that this is a zero
probability scenario and so you might
say well no matter how small the
probability if the if the danger is high
enough well it should justify
substantial effort and I might say okay
somebody out there somewhere should be
working on that fine that's that's okay
with me
so my complaint will come in and talking
about well how many people how big does
this sort of scenario loom in people's
minds relative to other scenarios and
and what fraction of the resources are
going into focusing on that scenario
relative other starts that's where I'll
make my complaint and disagreement but
I'm not gonna I'm not gonna say it's
impossible I get that but I'm sure this
is not um bit of characterization
someone's would make it their own a
strongman historia feel man see the
other side arguments and so I mean like
I guess you've also written about things
in the past would seem quite close to
some of the arguments of others
regarding general progress
one of them's was this constraint
elasticity of substitution wearing has
three exponential growth curves or
growth mode hunting which is substituted
by farming which is substituted by
industry and at some stage will be
substituted by something else
weakly suggesting that within a century
a new mode might appear with a doubling
time measured in days and not use others
the distance this could be AI this is
you're writing again quoting but that
was written in 2000 I think yet but
others this mode would be AI and you
could do this it could be like him and M
so so so more context then first of all
we have data on the lumpiness of
animation in the past we even have good
citation data that is how many
scientific papers get how many cite
and it's basically a power-law sort of
thing and so there are every once in a
while a few big lumpy innovations and so
that is part of the distribution of
innovation most innovation in terms of
an integral census in the many small
ones but every once in a while there's a
big innovation and if we look back in
history and we ask what were the biggest
things that ever happened in all of
history the biggest things that ever
happened seem to have been innovations
that caused the entire world growth rate
to increase by factors of 50 to 200 now
there's maybe only three or four of
those that we are somewhat confident
actually happened and so that's not very
many but but they did happen so it's
important to say history can be roughly
summarized as modes where growth was
exponential it was doubling in a certain
rate and then it reached a critical
point where within less than a doubling
time the doubling time increased by a
factor of 50 or more and so examples of
that were the introduction of humans the
introduction of farm farming yeah and
the introduction of industry and maybe
some things even earlier than humans but
at least those three cases something
happened that made the growth rate go up
by a factor of 50 or more so and if you
look at the statistics of these things
you can say these previous modes each
encompassed roughly a similar number of
doublings and the rate the the factor by
which the doublings increased was was
comparable and so you can put those two
things together to say well we would
roughly predict sometime in roughly the
next century or so there'd be a
transition where the growth rate would
go up within a less than safe our
15-year doubling time and that the new
doubling time might be say a month or
even more even faster so it's not crazy
to think with sometime in the next
century or so there would be a short
period say less than 15 years during
which we would transition from our
current economy doubling every 15 years
to a new economy doubling every month
and that's a pretty dramatic prediction
so if you want to call that Foom I'm
okay with that or a singularity but the
key distinction I make is that these
previous transitions were sort of the
whole world growing faster
it was concentrated somewhat in subsets
the world but it wasn't one island or
one machine in a basement one one tiny
group these were large chunks of the
world together that found a way to grow
faster and so that would be in the case
of Industry and the cases if you would
argue the beginning of the scientific
method that didn't seem to start in a
very small place in the world like
namely England
well there's locating the causal events
on on the path to producing it and
there's encompassing most of the change
so even if say the Industrial Revolution
could be thought of as starting in
Manchester in the late 1700s and you
could identify a thing that happened
there that was interestingly different
Manchester wasn't doubling every 15
years you know England and Northern
Europe together started growing faster
together and then and then their various
colonies that were selling to them
contributed to that and so it was really
a big chunk of the world that grew
together and accelerated its growth
until it really started hitting the
stride of as I say now doubling every 15
years even if you can identify place
where it especially seemed to have
originated that's not the place that
took most of the advantage so that we
might like be like saying that some
group was the first ta I team to develop
some new technique and to publish a
paper on it and have an exciting
development but that doesn't mean it's
going to be the only team that uses that
development and the only team that's
part of the new economic growth base
that's based on it you could still have
a much
wider you know group of firms and cities
and nations that are taking advantage of
and growing together this is this stage
for anybody who doesn't understand or to
those who isn't familiar with some of
the terminology there's a couple of
kinds of take offs that the stock take
off which is years or decades that's the
timeframe which we usually refer to
stock take off the thing and then a hard
takeoff this is also referred to as the
end assume that could be minutes days or
months but it does what you to agree
with those timeframes again I'm not that
picky about the words I would focus on
the key distinction of the locality that
is how small a chunk of the world is it
that's an encompassing most of this
change in growth that chunk of the world
was a pretty large chunk for the
Industrial Revolution it was also a
pretty large chunk for the farming
revolution although perhaps a smaller
chunk and even it seems a smaller chunk
for the initial human revolution that is
most of the gains seemed to have been
encompassed within the human species and
there were a number of other proto human
species that were existing in parallel
and they in the end gained relatively
little from you could argue that the
world gets smaller because it's becoming
the network bandwidth is getting wider
between various geographical places
around the world I haven't it might are
distributed like innovations can
certainly be picked up instantaneously
right so I'd say there's there's two big
effects that so far have determined how
small a fraction in the world it was
that encompassed the new revolution one
was the degree of information sharing
and copying
species found it difficult to share
information with each other it was
mostly genetic and that was relatively
limited so that meant mostly some
species just one over other species
whereas with farming it seems to be say
the literature I remember roughly 5050
it was farmers displacing other foragers
versus foragers copying the behaviors of
the new farmers as farming spread across
the world
and so and with the Industrial Aleutian
it was even more that Manchester by
itself couldn't do that much because it
was part of this interdependent economy
where not only were other parts of the
world economy copying Manchester but
they were complementing so once you have
a world economy that different places
specialize then when one place is a big
advantage in the thing it specializes in
that actually produces a big advantage
for everybody else who complements that
so if you're making machine tools in
Manchester that's great but if you need
cotton from the US or if you need you
know tea from China or if you need wheat
from the Netherlands wherever you
whatever else you need to make your
economy go all those other things are
increasing in value as you make your
machine tools more valuable as well and
so there's this standard economics
effective complementary different you
know if there's many things that need to
work together any one of them being more
valuable makes all the other ones more
valuable yeah ok for me I find it
difficult to know whether really
supports one side of the grid there yeah
I mean like ease you don't think that
service is a lumpy or likely to be bumpy
this live chances of you know a big
clump of IRA services suddenly just
taking off and being really effective in
one part of the world but does
geographical location or even like
logical industry location matter all
that much has been so much so much of
the tools
that are out there being shared or what
the only big difference now is the
people just there may need to be some
more research new tool tools created but
a lot of it may be just putting together
things that already well again the key
idea is if to do everything we need
doing we have lots of different parts
then when you improve one part that's
great but then you increase the demand
for all the other carts so you know if
you found a way to mow lawns vastly
better then everybody you know pin mow
the lawns much more cheaply but now
because that's cheaper they spend their
resources and other things and there's
the increased demand for everything else
in the economy because the economy is
now richer because there's one thing
that previously was expensive and now is
very cheap and so it's about how many
different things are there that all need
to be done and so this comes to a point
of people imagining this future AI a key
question about future AI is basically
how many parts does it have if there was
just the one AI part and when you
improve that one part basically it could
do everything better well then you don't
need any other parts right it's all you
need in which case whoever can make that
thing you know owns everything in the
same way that you know if the Industrial
Revolution if there was one super
machine you could make that would do
everything an industrial economy needed
well whoever could have made that one
super machine would have taken over the
world economy because everybody would
have wanted that one machine and if they
couldn't copy it easy then they had to
buy it from this one firm that's hold
the one super machine well then that one
firm and the one city that encompassed
it could plausibly have taken over the
world but now that we look back on
industry we think it's kind of silly to
imagine one super machine we understand
that machines are valuable but you need
different machines to do different
things and so no one company or city
took over
all the machines and all possible
machines they took over particular
machines there was the cities that made
cars and those the cities that made
building materials and those the cities
that made telephones and these were all
different machines and they were made in
different places by different companies
and different specialists and so even
though they were all part of the
industrial economy
there isn't one firm that took over
industry and so the quick question about
that in this future AI is is AI like
that or not and that's the subject in
essence of this book length through a
technical report that eric drexler
posted of thinking of AI as services and
so his vision is that in the future when
you have these very capable computers
systems they are composed of many
subsystems and each subsystem provides a
service it does a particular thing it
translates or it designs a circuit or it
you know makes a map planning program
you know and it provides that service
and if the entire super if together all
these future AI services can do
basically most anything you want really
cheaply then we are in the world of
advanced AI wear it out competes humans
but there doesn't have to be any one
part of that that can that has a big
percentage control or effect it could be
thousands of these little services that
together produce the whole effect in
which case you should worry less that
any one of these services will suddenly
take over the world it's not a plausible
thing he argues and I agree that under
the assumption that AI services
naturally breaks into thousands of
little services each of which is
relatively narrow and specialized so
it's so I think I might sink imagine a
palette loader you know something that
basically takes pallets off a truck and
puts them in the warehouse or vice versa
imagine a SuperDuper really good pallet
loader it's really good at pallet load
II do you have to worry about the pallet
loader taking over the universe well not
so much if its job as constrained to
power loading
if you say hey pallet loader here's a
billion dollars do whatever you want and
make my world better well now maybe you
should worry a bit because you've given
this open-ended broad task but if you if
you keep it with a pretty narrow task of
pallet loading there's not so much to
worry about so a key question then is
into the scenario the counter-argument
could be well there are some lumpy tasks
a I doesn't just break into a thousand
little narrow tasks there is a few core
uber tasks that just have a bunch of
important things all have to be done
together there's just no way to tangle
disentangle them and that part just is
really dangerous that thing if whatever
it does is capable doing lots of other
things and you should be really careful
about what powers you give that right
and on the subject of applying
optimization power to achieve something
there's a difference between an
optimization power being coated on you
know growing strong cause and or
focusing on you know a mouth that's
really good for dieting
the term what distinguished humans among
other animals who he worked and argue
their body design was focusing on
predation and giving him large muscles
big claws and very sharp teeth with
humans
I guess the optimization power in
relative to other states into its focus
on brain if one of these tasks the term
tracks was distributed AGI was focusing
on with actually creating intelligence
not just like observers like you know
having sharp scalpel or creating great
microwave oven so you know making cars
and year that could you know that could
also be an argument that an intelligence
explosion could come from eric drexler
and I'm not sure either you're in
contact with Eric at the future of
humanity Institute I guess you must have
spoken to him
and does he think that the only probable
scenario that he sees on the horizon or
is it
he just finding it is one of the
possible scenarios I mean it's just
clear from his writing that he thinks
it's relatively likely and even more
recommended that is you should be trying
to push toward this scenario it's a
safer scenario and it doesn't he doesn't
see that much of a loss from pushing
this scenario it looks like yeah that
was not go everyw going with a good idea
but whether he thought that unbalanced
it was far more probable than an
intelligence explosion or takeoff
so as you indicated this people's views
on the lumpiness of AI and on the
lumpiness of AI innovation is related to
their views on the lumpiness of human
brains so the two related points are you
know what explains the time trajectory
of human advancement that is was there a
single 1/p innovation that happened at a
particular time that explains a
discontinuity in in human history and
inside the brain is there some key
algorithm that's a lumpy algorithm in
the sense that there'd be a time when it
was developed and a time before it was
developed and then when you add that
algorithm then that's the thing that
gives humans its broad general
capability that's a question about the
human brain you could argue either way
and so we can go into both of these
details if you'd like so on the human
history it seems relatively clear that
there was this discontinuity in the
growth rate as we talked about that is
humans and the key cause of that
discontinuity was cultural evolution
that seems to be a relatively strong
consensus that is animals before humans
when they had an innovation and had
be embodied in their genes and so they
had to share their genes to share that
innovation and humans pioneered a much
stronger level of cultural innovation
which is that a human could invent a new
behavior and then transmit it culturally
so that other people would copy it
visually hear words and repeat and try
and it was sufficient to get a new
behavior transferred to other people
without needing to go through the genes
and that's the key thing that allowed
human innovation rates to be much faster
than other animals so there was this key
discontinuity and we understand that
that was the cause and so if there's a
lumpiness in the human brain with
respect to that discontinuity it's about
a lumpiness with respect to cultural
evolution what is the basis for cultural
evolution in the human brain well I
guess there's a limitation in the
computing power if humans were you know
ten thousand times intelligence and then
that they stumbled upon like tools then
there would be less need for it the
distributed sort of innovation
innovations could happen at the speed of
thought in you know a super intelligent
brain but it didn't happen that way I
mean we're cognitively luminous gold
bound and so we need to distribute
innovation across the wide variety like
the wide variety of people in a yeah so
actually it seems like the last three
and maybe more of these major
transitions in history were caused by
improvements in the diffusion or
distribution of innovation
and so that seems to be the consistent
cause so far of of these leaps in
innovation and so again the key thing is
there were these three major changes at
least that happened and we look at each
one and ask what caused that and they
are somewhat lumpy in the sense that
there's a time before and after and
there's this huge effect of this
innovation but the effect is in the
growth rate not in the level of ability
if these aren't innovations that make
the system's suddenly more capable
except in the capability of sharing
innovation that's the key capability
that made the difference before and
after these innovations so they aren't
things that let you think better or
something in just some absolute sense
they are things that let you learn
faster confusing because it was our good
link without modern link which is our
you know of a more articulate form of
language in it being difficult to be
able to scaffold some of the concepts
that we talked about today so the
invention of you know numbers and so
when you say history is summarized as
these three huge innovations that change
growth rates you'll realize that
surprising when you think about all the
other things that happened that did not
change the grocery the so for example
the invention of writing seems like it's
this very potent inventions that
certainly should have had a big effect
and yet apparently it didn't the
invention of writing was not
corresponding to some big innovation in
the growth rate and of course our our
history books are full of dozens at
least of other really huge things that
were invented
but this story says no there were only
ever three big inventions and every
other invention was something that
allowed the current growth mode to
continue but it wasn't something that
changed the growth rate of the mode so
for example compute right so for example
computers showed up during the
industrial revolution but they did not
change the growth rate of the industrial
revolutions the industrious tree economy
has been doubling roughly every 15 years
and that was true before and after
computers so while
computers are a terribly important
innovation they have allowed us to
continue at the previous growth rate and
of course the economy can't double every
15 years unless a lot of innovations are
big and then happen but that's not the
same as changing the rate of innovation
and so the three is more of an enabler
for the good mode to continue
in that sort of trajectory rather than
the learning at all I mean for many ways
it's not entirely clear what the the the
most local driver was so for the human
revolution and the farming revolution
and the industrial pollution we have a
better sense of the the key effect but
what was the proximate cause that
triggered it is less clear and as for
many of these things we don't quite have
the timing so we don't really have a
good timing on language so you know
language is still in the ballpark as a
plausible explanation for what caused
the key human cultural revolution I'm on
all on balance I wouldn't quite believe
it but it's it's not crazy and similarly
with farming you might say you know well
you know writing showed up near enough
to farming that might have thought it
could have been until you look at the
details but farming really got started
well before writing and so writing to
seem to have made that big a difference
but there's still this key question well
what caused the key difference of
farming early on the printing press make
a big difference you think well the
printing press is obviously more nearer
the Industrial Revolution so the
industrial evolution starts to get going
a few centuries ago and there's a number
of interesting things that happen within
a few centuries in the industrial
revolution that are all candidates for
perhaps you know the key thing that
triggered the cascade of events that led
to the Industrial illusions but it does
seem to me at least the key thing in the
industrial Aleutian was a new way to
diffuse innovations through networks of
experts analogous to scientific
societies but not always exactly equal
to in the ancient world innovation was
mostly diffused through artifacts and
animals and plants
but not so much through networks of
experts it wasn't like the expert in pig
farming in Russia talked to the expert
in pig farming in France and passed the
pig innovation that way it was just more
the pigs themselves moved and pig
farmers along the way took them and move
them and so and you know it does seem
like the the farming world has the the
possibility of faster innovation because
of the stable location and trading in
war networks
so most foragers are actually not very
close to very many other foragers they
don't interact with that much and they
don't do much trading a war and so
because and so there's it's hard to
actually diffuse innovations for
foragers because it's not like there's
these long-distance networks whereby you
trade with the necktie and they trade
etc that's that's not how it happened
with foragers but that's how it does
happen with farmers so it does seem like
a plausible explanation for the higher
rate of innovation of farming is that
they are now dense enough to have long
distance networks of trading in war
basically they are dense enough to stay
in one spot when you're on one spot it's
much easier to find you and you're
you're denser and so you're closer and
so again our best explanation for each
of these major things that happen in
history is a change in the rate of
diffusion of innovation not necessarily
the rate of invention but how quickly
they spread or not just even necessarily
in being smarter in some intrinsic sense
I saw yeah I mean suddenly thing I mean
it does seem as though that's really
taken off since the advent of the
internet and you know the ability for
computers to record and reproduce and
copy things and you know the open source
movement everything like that it does
seem to like avila a theme right 40
years really so just like you would have
been surprised to learn that writing
doesn't seem to have accelerated the
farming era
you used to be surprised to know that
computers and the internet didn't really
accelerate growth in the industrial era
we
are still growing and pretty much the
same rate we've grown for the last few
centuries and these apparent revolutions
haven't added up to something that
really changed the growth rate and so a
key thing to know about the last few
centuries are a couple of key things to
know is there have been these bursts of
concern about AI and machines taking
over that go way back there was a burst
of concerns in the early 1800s when the
Industrial Revolution was just going uh
a burst of concerns around the 1930s a
burst around the 1960s late 80s and then
burst more recently and all through this
period there has been a pretty
noticeable rate of job displacement due
to technology change that is for the
last few centuries
technologies have been changing and
that's been changing jobs and sometimes
that makes jobs go away and other times
it just makes particular tasks go away
or we get changed substantially and the
rate at which this has happened
happening is relatively steady for a
while so sometimes people say a story of
like the natural thing that happens with
computers is this really fast
exponential growth so you should expect
slow change until all the sudden it boom
happens really fast would classify it
happening really fast I mean what what
sort of inventions or ideas or what
would world look like now it doesn't
mean if things happened fast and they
were what would convince you that things
happening really far well there's a
number of different metrics of change
obviously the most fundamental one is
productivity all our capability to do
things that we want to do and so if
computers were really like getting some
traction at being able to do things much
better than before then they would be
doing a lot and that would be increasing
our productivity that is we'd be getting
able to do a lot more than we could
before
and so productivity measures would be
you know accelerating and growing faster
and that's not what we've seen so far
productivity if anything has been
slowing down in the last few decades and
that's a standard topic of debate and
discussion and so other things we'd
expect to see we might expect to see you
know as part of that productivity
process we did expect to see a wrap more
rapid rate of people displaced off their
jobs because that's how productivity
would impart happened because a thing
that a person was doing before it great
expenses now being able to be done by a
cheap machine it my flora expensive
that's what productivity is and if if
people were just seeing this about to
come say it's not quite happening but
people in the trenches see that it's
coming the kind of things you expect to
see is a big burst of investment toward
the machines and the tools that would be
doing this displacement because they'd
be worth a lot of money and so you'd be
seeing say tech firms the stocks their
stock prices being bit up you'd be
seeing students and employees like
specializing and focusing in those areas
because they were going Wow look this is
where all the money is going to be and
there's huge demand here and we're just
we're just jumping on this big
opportunity that's the kind of thing you
expect to see so only actually they blog
either that's what's happening a lot of
people went into app development you
know in the 2000s and all that with
software development in general and web
development and now people with the
rising tide of undergrad moving their
way from they are right now
so in fact if you if you look at the
history of you know the the the share of
different industries in terms of stocks
the dot-com boom looks kind of like the
thing you'd expect to see over a
relatively short term there's this huge
burst boom of investment that went into
technology and at the time it was
because people are saying wow you know
there's gonna be this huge productivity
boom in this technology area now of
course the dot-com boom crashed and well
there's been some recent increase in
investment and technology it's much
smaller than the dot-com boom and in
fact though but you could argue like the
dot-com boom looks fueled by people's
ignorance of what the future might look
like people aren't great but it is of
course right but the point is not
to see right that we'd expect to see
them some foolish investments followed
by you know a big learning curve and
then or table that wise investment in
the technologies that are gonna make a
big difference right so if people were
learning that there was a good chance of
a big automation boom soon then we would
expect to see this sort of thing we saw
during the dot-com boom which is a huge
amount of investment going into
technology areas that would be the
things that people will be buying
instead of the humans on the jobs and
the point is at the moment we see a
small degree of that but not not
remotely at the level of the dot-com
boom and therefore not naughty not
remotely at the level of you know
something you could predict you know the
robots are all taking over jobs we're
just not at that level so it could
happen as a surprise in principle but
we're certainly not at the level where
investors have a substantial confidence
that it's about to happen see that it
does change some people being surprised
by innovation and power you know the
open aisle climber like I think just
months or years before it was possible
to create a nuclear reaction that it
wouldn't be possible and there were at
least within 50 years or maybe portal is
never one of the Wright brothers it's
commonly brought up in certain
assumptions one of the Wright brothers
ended every in their flight wouldn't be
possible and they spent over 50 years
you know shortly before it was possible
lovely because it what they were doing
so now we're getting back to the subject
of the lumpiness of innovation so if you
look at our hope would be in predicting
it and how surprised people are one
actually does happen well I mean the
main thing to do is just look at the
distribution of lumpiness in the past
and then say well the future will
probably be like the past and say you
know what's the chance of different
sized lumps because in the past you know
most innovation in the past has been a
lot of relatively small innovation and
then every once in a while there have
been these bigger lumps
but even something like say
heavier-than-air flight
it wasn't a single lump it was it may be
you know a lumpy demonstration that
inspired people but then a lot of
detailed other innovation that has to go
on in order to make that actually work
and be feasible so clearly the air
industry was something that it wasn't
like one air company took over though
all the airline industry a lot of
different firms could go get into that a
lot of different firms offer their
different innovations and different
approaches and then the overall air
industry grows but not because of any
one air company taken over the whole
thing and so even if air flight was a
lumpy thing in the terms of the first
demonstration of heavier-than-air flight
might have been a surprise and it might
have been a lumpy surprise but that one
lumpy innovation isn't that all the same
as talking about a lumpy thing that lets
you take over all the flight ok so a
couple of things that have happened
recently in the AI world is the loomed
large and the headlines have been dead
lion successes in developing AI training
AI to out-compete all human efforts at
programming chess programs or go
programs to defeat human and more
recently that he also won a to divert
and a protein folding competition and
also has been winning in larger 3d
environment games like dota and
Starcraft 2 multiplayer games against
champions there and have also won a lot
and now people beginning to think along
most of the things most of the
challenges were sure to get it
approaching as they're being successful
with you and Elliott in County address
the issue about whether such progress
that deepmind's as an example
whoever it was in favor of one side of
the argument or not obviously Elliott
County was they said it was some showing
weakly against increasing the some
weekly should increase people's credence
that a food no more possible but um you
didn't hear that way you want to
describe why so an analogy might be 9/11
as a terrorist attack so you know as
most people know on and September 11th
2001 there was this big terrorist attack
and it was unusually large compared to
previous terror attacks over the
previous few decades and so at that
point the key question was has the
distribution of terror attacks changed
or did we just get an unusual outlier in
the distribution of attacks because
distribution of attacks has a power law
every once in a while you should expect
a really big attack but when you do see
a big thing you you often wonder is this
the beginning of a new trend this often
happens with a big earthquake or
something like that to any big event
that's quite big compared to other
recent events people say this could be
the beginning of a new trend yeah this
might not be you know just an unusually
large outlier in the previous one what
if it's about to be a lot bigger people
even say that about say the 2008
financial crash anytime there's an
unusually large event there's some
people will say well this isn't just one
unusual thing the the regime has changed
we're now in a new world where this sort
of thing is gonna happen a lot more now
usually it's not true
so the distribution of terror attacks
post 9/11 isn't really any different
than the distribution before 9/11 you
know we haven't had a burst of
earthquakes since the last big
earthquake the 2008 financial crash was
not a sudden indication of a lot more
financial crash as soon sometimes
regimes change but the mere fact that
you have an unusually large event by
itself is not enough what you'd want to
see is a more like a sequence of an
unusually
Vence that was unusually large compared
to previous sequences that's more what
you want to see in characterize the
series oh I always said her state of
mind has created alphago zero nor the
iteration of those and then the one
which eight players was that on our
school the alpha dog yes alpha star some
linear alpha zero I guess yeah yeah well
that was the last of the offer code
iterations wasn't it alphago zero and
then it wasn't
something called it were at alpha star
remember that was a different ion which
B at the time of the young hey I think
the key is to have a sense of history so
if you only see the recent demos it's
easy to believe that it's unprecedented
the only way to know if it's actually
impressive and it is to see precedent to
see what happened at previous eras so as
I mentioned before we're now in you know
bursts of concern number five out of a
eye concerns and in each of the previous
verse we also had dramatic unprecedented
developments that people could point to
as saying wow nobody could ever do
something like that before so in the
early 1800s it was the Industrial
Revolution it was the fact that we were
able to make lots of very productive
machines that was displacing people on
jobs and making an enormous revolution
that was really freaking people out and
a lot of the most you know famous smart
economists of the time explicitly talked
about what happens if the machines take
all the jobs cuz look though there was
the invest revolution and that was this
unprecedented spectacular thing that was
happening we again saw bursts of concern
in the 1930s this is when say they made
the movies like you know these
futuristic movies about computers
you know robots taking over people like
Einstein and Keynes actually said that
the low unemployment of the 30s was
caused by machines taking over replacing
people on jobs that was a
common thing many intellectuals said in
the 1930s that the big unemployment we
were seeing there was the direct results
of machines displacing jobs and they
were right there seeing the machines
displacing people off of all the jobs
because of that unemployment that's what
many people were saying in the 1930s and
the 1930s saw this spectacular burst of
innovation you know new innovations were
happening that were quite disruptive at
a rate much more than we see today from
the 1930s TV if you go back and realize
all the things were happening it was
really quite spectacular in crude and it
could be that you know with the right
innovations we'll be creating a new sort
of layer where we can get old our next
level of poverty Oh sharp my main point
is now just to make it clear just how
much really big unprecedented things
were happening in each of these bursts
of concerns me well back to the you know
the outlier example ebooks or before
with the terrorist attacks do you think
this is progress in AI recently with
them being able to do I can see like the
experts in certain game at a time which
which way before most actors would have
predicted do you think that they are
outliers and what thought through
programs in the future would you think
would disqualify that fear honestly on
the program so the question is isn't so
much are they outliers it's like how
often do you expect to see something
that big as an outlier it's it's is it
every you know so they were floods for
example they talk about the flood you
expect to see every 10 years and the
flood you expect to see every century
etc and so it's the flood that you
wouldn't have expected to see for a
really long time that's an outlier flood
so if you look at the history of
computer science over the last seven
years say we've had a lot of pretty big
dramatic innovations
of the scale of deepmind it's not a once
a century innovation remotely it's not
even a once a 50 years innovation it's
more like a once in 10 years innovation
of the sort that we've seen every 10
years for the last 70 years it's it's in
that sense part of the trajectory that
we've had which is rapid dramatic
improvements in computers and computer
applications for a while and I was I was
caught up in the last burst of concern
before the current ones see I was a
naive 20 year old some-something who
read these exciting newspaper articles
around 1983 and this was a that was the
big AI boom then and it was talking
about these dramatic demos that we're
doing unprecedented things they were
having you know machines that could
diagnose disease as well as doctors they
could organize chips better than humans
they could arrange pipelines there were
all these dramatic demos in the early
1980s of things that machines could do
better than humans are as well as humans
that were quite difficult and
unprecedented and that was the basis of
people claiming that look we're just
about there in the 1980s then and I left
school to go out to Silicon Valley to
join the AI revolution then and I spent
nine years and as an AI researcher from
84 to 93 so I have this visceral sense
of how it can seem to a naive young
person that here are these unprecedent
demos and nobody's ever done something
like that and aren't we almost there
right they imagine Elliott did count in
a bunch of others in 2040 will be
telling the next generations of young
bright-eyed enthusiastic people how how
it's not going to be the IRS or a
revolution this time well they can
certainly tell them that whatever
they're seeing then might or might not
be unprecedented with reference to
precedents actual what's happening
before you know this is a problem people
often throughout the world word
unprecedented without actually knowing
what precedents there are without
actually knowing what happened before
they look at something and say well just
that couldn't possibly be precedented
but yes it can well I mean like it's a
not you know in some of the chats that
I've been in like Oh watching right now
in reference to this interview company
are more what thoughts of developments
would would change your mind would make
you more bullish on the side this could
happen relatively hasn't he as an
economist the the obvious metric to keep
track of is economic growth and
productivity so as I said before the
most direct measure of a big impact of
technology AI technology on the economy
is productivity you know us being able
to produce a lot more than we could
before a slight variation on that is job
displacement people being thrown off
their jobs that I unusually high rate
compared to before if you're looking for
a a bit earlier indicator of those
things then when investors start to
notice that those things seem to be like
they're about to happen as I said before
then that's about what you see you
should expect to see an enormous burst
of investment that goes into the kinds
of tools that would and the resources
that would be used to produce that kind
of automation and productivity
room so if that's a computer firms if
that's uh computer software firms ai
firms whoever it is there would be this
enormous burst of investment that went
into those things that would be
something you'd see a little earlier now
you might say well yes but even earlier
to that you might see some really
impressive demos and you might but it's
much harder to judge impressive demos
and what sort of actual economic impact
they can have so you know as you might
notice the vast majority these really
impressive demos in last few years
aren't really next to industries with a
lot of revenue yeah there's just not a
lot of people that pay to play go so
winning it go is just not a big revenue
winner even protein folding you know
there's just not a huge industry where
you you get people to pay you to do
protein folding and so having impressive
progress hey what about the you know I
mean self-driving car
automated checkouts
all sorts of things that are now driven
by AI a lot of these things are actually
behind the scenes you don't actually get
obvious expectations by the for your
eyes amor hidden so so a recent
economics paper did did a nice analysis
up so over the last decade or two we've
had the slowdown in productivity in the
advanced countries and they said well if
over the next twenty years
imagine that we saw you know the kind of
displacement you might have hoped for in
self-driving cars and in automated
automated phone services like you know
help services on the phone say you
displace 60 or 70% of those jobs in each
of those industries over the next 20
years
you'd need 10 more things like that to
raise productivity growth up to the
level we had it before that is just two
of them by themselves is not remotely
enough you'd need those two and ten more
and that altogether might be enough to
just increase productivity back to where
it was a few decades ago not enough to
remotely go out would be a dust
revolution regime just for the advanced
companies to have there and pretty
clearly growth go up back up to the
middle of what we've seen in the
industrial era so the point is these
even if those things happen they are
well within the history of the kind of
automation of jobs place we've seen they
are not deviations from trend right so
robotics in assembly lines and 3d
printers printing specific things on
demand and dr. wouldn't say that is out
of the ordinary that's right it's about
the scale I mean so the key point to
keep in mind is during the industrial
era the world economy doubles every 15
years you can't double the world economy
unless most industries are going through
enormous innovation on a 15 year period
because basically you're doubling
everything on a 15 year period
so that means the history of the
industrial era is large innovation and
large change that's what it takes to
stay within the industrial era that
doesn't pull you out of the industrial
to some much more rapid change you need
a lot of innovation and a lot of big
innovation and a lot of disruptive
innovation to continue the trend we've
seen for the last few centuries
you
okay um well I came though if the trend
just continues in the direction it is
right now it doesn't speed up what fill
up the program you then see in neck if
they need another doubling of the
economy right so the way I see the
future is either mostly a continuation
of the past trends or at some point a
radical change and so we've been talking
about what would be the signs of a
radical change a radical deviation from
past trends but if we don't have that
we'll probably have a continuation of
past trends so the key things to know is
that we've so far been adapted to our
past trends over the last century we've
handled the doubling every 15 years
we've handled the degree of job
displacement that causes the degree of
inequality that causes the three-button
certainty it causes the degree of not
sure being sure how the future is going
and what kind of technologies would be
important the ability you know that
changing and uncertainty about which
careers in which industries are
important we've been handling that
pretty well so you should expect the
same amount of that in the future as the
past so our past institutions if they
were up to handling that or they should
still continue to be up to handle you
not in the future there's no particular
reason to think we're in a crisis where
our current institutions couldn't handle
the kind of change that we've already
seen for a long time so you should
expect to see the same levels in the
past that is you aren't quite sure which
jobs will be in demand you're not quite
sure which technologies you should learn
to be in on the future you're not quite
sure which nations will prosper as a
result which professions even will
prosper there's just a lot of
uncertainty about where the future's
going and that's how the world's been
for a long time and that's what you
should expect for the next few decades
as well if we continue on the same trend
okay we'll listen narrow down on AI
development tonight i probe it
what do you think will contribute more
to jump in AI capability would that be
hardware improvement or software
so I'm mostly gonna say what do we know
about the past and guess that the future
will be roughly like the past so in the
past in some areas it's overwhelmingly
software that makes the difference that
is in some areas the hardware is plenty
fast and cheap enough and so the
limiting factor is making software and
the hardware is just easy there are
other areas where hardware is hard and
in those areas we actually have this
interesting phenomena where the rates of
improvement of the hardware are about
the same as the rates of the improvement
of the software which is a remarkable
coincidence that is if the hardware gets
a factor of 2 cheaper every 18 months or
2 years then the software gets a factor
of 2 more efficient on the same time
scales um that suggests that in fact the
cause of the software improvement is
indirectly the hardware improvement that
is oh the world has all these ideas for
better software but you need faster
cheaper hardware to try them out and so
it's the fact the faster better cheaper
that hardware that lets you try out
enough of this offer idea is to figure
out which ones really work well to
actually get the rate of software
improvement that we see so that's I can
say I would guess that that will
continue that is on the in the areas
where hardware is a limit of limitation
it's it's you know it's just the
software and once you have the right
software it's cheap and easy then it's
all about software but in many areas the
hardware is limitation in which case you
expect a similar rate of improvement in
the software and the hardware at roughly
the same rate it sounds like um he was
describing the hardware is being
limitation in actually finding the
software that might be useful exactly it
wouldn't actually be a limitation on
that software once that once it was
found wouldn't necessarily be that it
would need huge amounts of hard work to
actually button once discovered because
then you know you know the optimized
little service and have a huge amount of
hardware overhanging
and the
incredibly powerful but again that's
what we've seen for the last seven years
we've seen that happen many times and so
we should just expect that to continue
in the future the same sort of phenomena
that you know sometimes you you make a
hardware software advance that required
the hardware to explore it but then once
you have it you're able to apply it more
broadly because you've made this bit and
that's what we've seen in the past shown
show Menon Sandberg understand Berger
will look exploring this idea of how far
away is is this sort of a take-off if it
is all the take-off together with a
faster flow they think that if it ends
up being overwhelmingly the hardware to
the problem then it may be the case that
it will be a slow takeoff because it may
only take off as fast as it's Moore's
law increases the hardware ability for
whatever laura's in the future the
increasing hardware SS could be wherever
it is if it is software once the
software is found there may be a huge
hardware overhang that can be afforded
exploited straight away and against huge
capability games and short amount of
time
well that that analysis is implicitly
assuming a big lumpiness and so that's
the key concept that I've been pushing
here as a way to help you think about
this if there's just the one algorithm
to find that lets you do all of
intelligence
then when you find the one algorithm and
there's a hardware overhang then you
have this huge boom but that's all based
on the idea that there's this one thing
to find if there's a thousand little
things to find each of which plays out
that way then you've got more like the
AI services scenario where overall it's
a gradual improvement even if in each of
the thousand areas there are lumpy
advances
yeah and so then we come back to this
key question of the lumpiness of
intelligence right okay so if it is that
you know maybe a lot of the innovations
has actually been discovered but there's
just you know a little bit more secret
sauce in that dude you'd then be
sympathetic to a hard take off more than
you would if you thought that there are
lots of little things that still need to
be discovered right so that's why I
tried to say at the beginning of our our
discussion here which is to say if you
assume this lumpiness about AI
innovation then that predicts that is
plausible that one small villain on an
island or small team in a basement could
find this one very lovely innovation and
then make enormous progress as a result
and then have this enormous effect on
the world
and then enormous dangers as a result of
it taking that team cake controller it
the Machine getting out of control of
that team and so if you find that lumpy
scenario plausible enough then you will
want people to put effort into focusing
on it and I don't object there to there
being some people focused on that
scenario I just think overall in the
world of people thinking about future AI
it looms too large in people's space of
possibilities sure let's look at some of
the reasons one people might think that
these are higher probabilities
it's been argued that when navigating
through problems thanks one can find a
succession of extremely easy to solve
problems once you've opened the door in
a sense so once you found your you've
navigated to a certain area there's a
whole debate of I think that solving one
big heap in probability space really
close to each other not you know evenly
distributed out across probability
problem expect what people to that
analysis would even do that
again I mean I see often people taking
the most lumpy innovation events that
have ever happened in human history
and then saying see that could happen
again in the future and I want to say
yes those happen but they're part of a
distribution and there are a tale of a
distribution and you should know about
the rest of the distribution and so
unless you have some particular reason
to believe that in this particular case
we will have this unusual tale of the
distribution you should be more
expecting to see the whole distribution
not this extremely unusual tale so you
know some people like to point to the
nuclear weapons and they say look
nuclear weapons were very lumpy
innovations now actually turned out the
first nuclear weapons were about as
cost-effective as as TNT in terms of
explosive power per dollar put in
because they first nuclear weapons were
very expensive but after a while we got
some much more powerful nuclear weapons
and that's what pulled this away from
the trend but the point was there's a
sense in which nuclear weapons were a
lumpy innovation but you gotta realize
that out of all of human history you've
picked one of the most extreme examples
of a lumpy innovation and so the
question is you know if that's the one
in a million or one in a billion
example what's the reason to expect that
this future thing which isn't anything
like nuclear weapons as far as we know
would be that lumpy okay similarily
the the idea that small improvement
leads to large capability gains if
they're the right ones not necessarily
you know opening the door to a big huge
type of capability or problems with
altered achieved capability game maybe
even won the right small improvement can
actually achieve that it's kind of hard
it
what do you offer again if you just want
to say look this scenario is possible
I'll say well yeah it's possible
the question is how likely is it not
whether it's possible and to ask how
likely it is we need to look at data in
the past so I think one of the key
things that happens is a lot of people
near AI are related to the AI they just
don't think past date is relevant they
just think well this is a whole new
thing
it'll be completely different than
anything in the past and therefore we
just have to go back to really basic
priors about what are the range of
possibilities so so for example if we
look at growth models there's a
parameter which is the growth rate
acceleration so if you just have a
one-dimensional growth model there's
basically two classes of models and one
borderline case in between one class of
models is were growth decelerates where
growth slows down because as you grow
more gets harder to grow and then
there's another class of models where
growth accelerates where as you grow
growth gets easier the decelerating
growth is then less than exponential
more like a power-law growth and the
faster than exponential growth typically
is the thing it reaches the singularity
in a finite time it goes to infinity now
if you look at those mathematically you
say well my priors are equal across
these two then you got to say well hey
there's a 50/50 chance that will go off
to infinity in a finite time and if
that's all you know is just the
mathematical possibilities you say I got
to give a lot of probability to that now
if I look in history and actual growth
in systems we've almost never seen
something go off to infinity finite time
all real growing systems in the past you
know have reached diminishing returns or
at most grown exponentially
very rarely do systems ever grow faster
than exponentially and typically that
period of faster expose to growth runs
out quickly and then reaches a period of
exponential or less than exponential
growth so we look at actual history of
growing systems we got to say this
faster than exponential growth to
infinity is not a plausible thing that
happens in the world even though
mathematically it's a perfectly
reasonable thing to imagine when you
think I know nothing about this world
therefore anything's possible and I
should just look at the mathematics to
infinity in order to reach capability
again that sort of outperform humans in
most areas of interest though I mean
sure there may be an asymptote like you
know a hundred times Einstein or
something like that it doesn't mean that
you know it's going to be a certain
peak to the MA and make a point you know
what I mean so I just think I don't
think it took quietly we need to support
an argument that there's gonna be an
infinite growth I didn't mean to be
suggesting I just meant to be saying
that's the kind of thing that happens
when you say the past experience is
irrelevant I'm just gonna think
abstractly about the possibilities
because when you think of strictly about
the possibilities then really lumpy
innovations because seem plausible and
one machine take one company taking over
the world as plausible because as I've
said in fiction people have thought that
the Bond villain island was plausible
and Marx and Engels thought it was
plausible that one firm would take over
the capitalist economy when people don't
have a grounding in what's actually
happened in the world or how the world
works these things seem very plausible
to people it seems very plausible that
there'll be this one lumpy thing that
changes everything because that's the
basic story and a lot of science fiction
and comic books is the one villain has
the one super thing that threatens
everybody and the hero has to go fix it
and of course we also find it very
plausible that if not for heroic
regulators or what our popular
opposition that one firm would have
taken over the world by now and and the
people with experience and understanding
the details say no these are not
plausible these these these do not fit
with our history and then you could say
yeah history is irrelevant so I'm gonna
go back to my comic book intuitions
about what's possible hmm well I mean
like I didn't see why people playing
categorize the rise of humanity for
humans to you know mean the animal
kingdom as being very lumpy example well
that's why we write but that's why we
have the history of these three biggest
lumps that have ever happened and we
should analyze what were the causes of
those and therefore what future events
could be plausibly like them so each of
those three things was a innovation in
the rate of diffusion of innovation that
was the key thing in these last three
cases growth growth rates became faster
because a new way to spread innovation
became possible that's what happened in
the last three cases in none other cases
was that a sudden leap in incapability
it was only a leap in the rate of an
that was the the cases of the three
biggest things that have ever happened
so it's fine to imagine that a future
event where a rate of innovation happens
again but again these last three things
were relatively broad events
encompassing a relatively wide scope
increasing scope over time because the
sharing of innovation in sharing and
complementing of innovation that has
increased over over these periods and so
that means you should might expect that
the next thing it happens will be
something that allows a faster rate of
innovation but that also has a lot of
sharing and complementing such that a
large scope together takes advantage of
this new innovation ability that would
be the straightforward projection of the
future based on the last three biggest
things that I've ever happened so if you
want to go with the biggest lumps ever I
gotta say those are them okay so we've
been talking about the feasibility of it
either things to more slowly take off to
cap that off I guess a lot of people
think that a is going to be a powerful
thing in future and regardless of
whether you know there's been a an
adequate solution to friendly there
questioner friendly artificial
intelligence there will be people out
there who will just try and create food
not knowing whether it's possible or not
we'll just do it because they want to be
the first I want to get they want to you
know take advantage of a perceived first
mover advantage yes so kind of scary and
I think we cured in in any case even if
we don't know that it's going to be firm
or low takeoff prepare for the work so
that you know yeah I think you agree
with that like you heard he was it I
have a subtler answer which is so if the
local Foom is coming if the machine in
the basement that takes over the world
scenario is plausible and it's coming
then as I said before there you want to
expect to get much warning about it and
before you'll have to prepare well
before you see any indications of its
problem because it's just this lumpy
thing that happens very surprisingly and
all of a sudden if instead you've got
this big chunk of the world that grows
together in a visible way that you would
start to see it happening well now you
don't necessarily have to prepare well
in advance so now you face the question
when is the right time to prepare so you
know there's a number of considerations
yeah so there's there's three main
considerations I think here about when
to prepare so one consideration is if
you spend money now versus spending
money in fifteen years if you spend
money in fifteen years you get to spend
twice as much money because the economy
doubles and your investments double or
you need more than doubles so all else
equal you'd rather wait and spend later
because you have more resources any
dollar you spend now you could spend two
dollars later in fifteen years in four
dollars and thirty years so that's
number one the second one is how much
can you actually in detail envision the
systems you have in order to envision
the problems they'll have and how to fix
them so if you're too far in advance you
just don't know enough about what you're
trying to fix to be very useful about it
imagine trying to like you know figure
out car car accidents and highways and
the Year 1500 and you'd realize well
that's probably too early you've
probably made more sense to see actual
cars and doesn't mean you have to wait
until you've seen 50 years of cars to do
anything but it might mean you know you
start to see some concrete vision of
roughly how big they are how expensive
they are how fast they go you know what
they're used for so that you can start
to think usefully about what problems
they'll present and how to deal with
those
so this consideration also suggests I
think at the moment waiting because at
the moment we really know very little
about concretely how these systems would
be structured and and placed and their
issue alysha's there's just so many
different possibilities that working on
it now is really against a headwind
because you just know so little about
them now a third consecutive
duration the ways the other side perhaps
is a an intrinsic delay in the process
if for example there's a sequence of
things you have to do one after another
and they each take a certain amount of
time then you got to start that process
you know before the sum of those times
so that you can be done by your deadline
so a key question is in this process of
preparing for these sorts of problems or
how plausible is it that there are these
delay things where you need to say you
know train a person when they're young
and they think about all their life and
then later on they have an insight when
they're older building up a whole
lifetime of thought and experience or
something I think most of our data on
research says that sort of delay isn't a
big thing mostly research progress is a
function of how many people are in an
area and not necessarily about some
intrinsic delay and producing progress
but that would be the main consideration
that would way the other way so I mean
but you know more generally you might
say well even if now isn't the right
time how do we know when the right time
is we'll need somebody tracking
the problem to say when the right time
is right the best person to be tracking
is somebody who's actually trying
so you know at this show for example
most companies have research labs that
don't necessarily produce research
that's beneficial to that company often
they'll have researchers to track
research elsewhere one of the main
functions researchers in most private
companies is to be an inside source that
tells you early on what happened what's
gonna happen elsewhere so that you can
get a jump on that so similarly somebody
somebody ought to be researching the
future AI safety problem just so that
there's smart people who would be ready
to tell you when there's a problem
looming suit well we have some people
already didn't know right too much right
but you don't necessarily need a lot for
them if the main function is to is just
to keep a certain level of tracking and
readiness for when there's big bursts is
needed but this is true in a lot of
different research areas you know in
areas so for example but true for a long
time that you know we weren't ready to
do fusion energy when nevertheless we
had some people working on fusion energy
and those are the people we might turn
to to say hey is this near is it time to
put a lot more money into this and you
know of course they're often biased and
they'll always tell you it's time for
that but you don't have to believe them
but you want some people tracking that
some people have long been arguing that
we should be doing asteroid mining that
seems kind of crazy to me but you know
what there should be some people who
track that some people should be
studying asteroid mining at any one time
redoing the numbers looking for
possibilities because you know there's a
sort of person who when it was ready the
time to do asteroid mining they might be
ready to make a case for it and so
that's the you know the strongest
argument I would accept us to say we're
a long way off large amounts of money
now is just this isn't the right time to
spend large amounts of money because you
know money later is worth more we're not
we're not close to it's not remotely
close to this to me about to happen we
don't even understand very well even how
this thing will be structured or or you
know used so but we ought to have a few
people tracking what we yeah so we
spoken about the feasibility and you
know how much resources should be
focusing on
your view and how much resources should
be focusing on researching whether this
is possible and AIA ste concerns what
about the desirability both of these you
know you've mentioned that it would
might be a lot easier to end many other
people to it might be much more easier
to deal with this low takeoff
I actually think so people often have
this intuition that technology happens
at one rate and wise cautious control of
technology happens in another rate and
they're worried that the first rate is
faster than the second so many people
have said they would just like
technology to slow down because they
can't figure out how to speed up the
wise thoughtful analysis of technology
so maybe they went television was
possible they'd want to have held off on
television for a decade or a couple
decades while academics debated the
topic of whether television should be a
distributed the masses or whether
television should not be right I just
don't think that really works
in the sense that you know mostly so far
in history nobody's been driving the
technology train we have just not been
up to the task of stopping technologies
even if we'd wanted to we don't have
decision-making bodies set up to do that
and the authority to make it happen so I
don't I haven't put that much thought
into could we prevent technologies
because it just doesn't look like the
sort of thing that happens but I also
think no reason nothing
Hawaii technology I think you mean like
you they're pretty small she's really
showing destruction which I also just
say that um our ability to adapt to
technology is actually the main limiting
factor and the growth of technology it's
not that technology can outrun our
ability to adapt to it it's that
technology doesn't happen unless we can
adapt to it well that is for an awful
lot of Technology you you can see it
ahead of time and you can try to
distribute but it's not until you find a
way that pushes it into the world where
people can in fact it app to it that it
actually happens
technology doesn't actually happen until
people feel that they can adopt that
they can actually change it they can
actually use it so the ability of people
and other social institutions to adapt
to technology is in fact typically the
main limiting factor on the rate of
technology change you if you can in
principle change the technology but
people can't in fact assimilate those
changes they don't happen they only
happen when people can in fact
assimilate the changes well let's take
an example nuclear weapon that some
people can attempt to I mean in a sense
our policies is adapted if they were a
nama technology like you know maybe
thumb they had one fire engineered virus
it was extremely hard to adapt to we do
suggest that it just wouldn't be created
because no it wouldn't that is you know
military technologies are used when
militaries find in their interest to use
them weapons don't get produced
mass-produced for military use unless a
military thinks it's in their interest
to mass-produce it because they think it
gives them a military advantage that's
the nature of military technology now
you might think it's a shame for the
world if militaries have better
technologies but that's a different
issue but you know a military technology
doesn't get created unless some military
thinks they have an advantage from that
so that's the government I think the
advantage is relative to the
disadvantage of letting the Aventine
developers first so if there is a risk
in developing technologies that are
possibly negative and you know it could
be endgame technologies people still
development development because they
they're willing to take the risk because
the payoffs for them might be quite high
like for instance people may try and
create a super intelligence and just
pull the plan and let it explode hoping
that you know knowing that it's possibly
very dangerous to do that but if it does
work in their favor they get to be king
of the universe or king of the galaxy or
whatever you know
again I I think one team making a super
intelligence just is a very lumpy
scenario that I don't find very
plausible but certainly it's plausible
that technology are technologies that
have hurt people for a while I mean the
classic example is farming you know many
scholars have made a strong case that
farming hurt individual farmers for a
long time individual farmers were poorer
BETT worse nutrition more war less
travel more disease and domination like
per individual many people said the
farming world was worse per farmer now
there were a lot more farmers but for
many people's ethics that doesn't
overturn the idea that yes but
individual ones were soft so you might
say well maybe the world would have been
better having somehow preventing farming
but of course if you had prevented
farming you would have prevented all the
things that follow on after that and
that's a generic fact about whatever
you're trying to prevent you might
prevent not only that but everything
that would follow after that unless you
sort of go down a different path so
unless you say well we're gonna get
there eventually we're just gonna go
this different path but even just like
block off the paths then you're blocking
off all the things that could follow
from there but but the most basic point
is just so far in history we just
haven't had much of the form of
collective analysis of technology and
deciding to adopt it or not based on
sudden collective analysis that just
hasn't happened much mostly for the last
you know million years humans have
adopted technologies when somebody
somewhere liked it regardless of whether
that was an overall benefit for the
world that's just how things have gone
for a very long time and it would
actually I'm pretty garages again and
turn pretty foreboding I mean like if
the technology is you know getting more
and more powerful you know this side of
things so the possible spikes of
potential power gain could be huge and
they can't I don't but that's not clear
that technology is any more powerful in
the relevant senses here if you if you
look back in history technology was a
pretty big deal all the way through
history tech technology changes have
made enormous changes on humans
and human life in very fundamental ways
some of which were good and some of
which were bad for a very long time I
don't actually see upcoming technologies
as being that much different than past
technologies in terms of their potential
for making people unhappy well let's
take some examples like if we can take
it for an example you know a possible
airborne engineered supervirus
that um you know why start people I mean
what what have we seen in the past
technologically speaking that might that
have compared or you know the potential
for a nuclear war where many players are
basically responding to each other's
attacks yeah well if you're pointing to
a hypothetical future super virus you're
pointing to some you know extreme
hypothetical scenario relative to the
actual history we've seen so I'm focused
on the actual history so far and we
don't see a trend in actual history so
far toward super destruction relative to
the past
example over the last few years some
people have made this claim that one
person can hurt more people now than in
the past one you know killer or one
crazy person can just do more harm there
is no trend like that in the data in
terms of an individual harm so far in
history you know for a very long time at
least I mean the invention of guns made
a difference but since then we are just
not seeing that sort of trend I mean
hypothetically it could happen but again
you can always make up these extreme
scenarios almost comic book scenarios in
terms of the extremity of a small group
of villains doing extreme harm but you
know that doesn't mean they are typical
scenarios and that doesn't mean they've
actually changed the distribution what's
actually happened well I mean so
wouldn't you just have to have one
outlier like a you know one crazy leader
just you know pressing the red button
and launching a few hundred nuclear
bombs whatever that could make a big
difference anything but that's always
been true the point is so long they
prove it it it hasn't happened yet okay
but the point is the world isn't you
could say that there's just this generic
problem that we've always had and we
should deal with that's different than
saying the future is different than the
past that that we're facing a new era of
heightened risk or heightened problems
that's the part I'm questioning but I
think it's fine to say look it's not
obvious technology has always been good
in the past and hey maybe you want to
have some sort of overview process to
deal with that it's a hard problem but
it's not a crazy thing to do but I just
say that's been true for a long time and
if it and it'll continue to be true for
a long time and I mostly think there are
a lot of big problems in the world that
aren't very well addressed and you
emotionally should point to past
problems and say hey let's fix this
problem pointing to hypothetical future
problems that might be different from
past problems I tend not to buy that
sort of argument I think mostly you're
looking at the future problems that are
a lot like paspalis sure but it doesn't
seem sane for that long we've had the
number of nuclear weapons that we have
today yeah but the ability from
destruction war has been around for a
long for a while
ah right okay so so it doesn't seem so
it's on the same scale and it doesn't
seem as though it would have the same
fallout of lucky for a huge nuclear
exchange but look you know like um yeah
I think this is it's important to
discuss these things I don't know how
much way we can go sis but one of the
things that does that somebody brought
up make this finding thinks it's worth
talking about the extent to which strong
assumptions about firm scenarios or
nuclear exchanges someone that animate
people to focus on existential risk or
AI safety so yeah and as you know he
thinks it at the mall a lot more than
people realize do you believe that um
he's got a point there do you think
there's too much focus on firm scenarios
or nuclear exchanges and does that
actually shape the way that they think
about this sort of causes to prioritize
and sort of things to donate to and yes
I'm less of unless persuaded that
there's any big change in reality but
I'm quite happy to endorse focusing on
the problems that have long been in
reality including existential risk that
is existential risk is such a terrible
thing that even even if it's a small
probability lowering that probability by
a modest amount isn't in terribly
important things so I'm happy to endorse
people you know but we don't need to
exaggerate the probabilities to justify
working on it I think there is this
temptation to to try to motivate people
to be concerned about a problem by
exaggerating it and the problem is that
when other people notice you're
exaggerating then they lose trust and
then they don't trust the entire
analysis or whether there's a problem
worth working on so I think you just
need to be really honest and careful and
say yeah the chances modestly low but
still it's well worth dealing with so
you know the chance of a worldwide
nuclear war is still nonzero
it might be you know one in a thousand
per year or one in a hundred per year
but those are still really high
probabilities to worth reducing so I
definitely think if there's a way you
can figure out how to reduce the chance
of a worldwide nuclear war that would be
worth doing I mean the hard part is is
that just these things are hard things
to do it's hard to figure out how to
reduce the chance of that World War I've
been on the board of directors for this
organization called all fed that's been
interested in figuring out how how in a
say a nuclear winter scenario we could
actually feed a lot of people and you
don't have to have billions people die
so it's actually mechanically possible
to use factories and petroleum and other
things we have to make food for people
even if the Sun gets blotted out for a
year if we prepare ahead for that so
another people know you can put humanity
going but but the point is you could
actually keep most people alive if you
set it up right really because we can
make food in factories mechanically it's
it's possible to do we don't need to
make food on farms if we really need
food we can make it in factories but the
only main supply chains work right but
you don't necessarily need the entire
familiar surprise chain to work but the
point is one approach to dealing with
nuclear winter is to try to ask how to
survive it
rather than how to prevent it and there
are things to do there to say you know
the yes we - we should try to prevent
these things we we should also try to do
what we can to survive them if they
happen okay yeah I agree
great idea yeah right but so but I I do
think there's this precious resource of
people willing to think seriously about
the long term and I guess you know that
resource gets spent in various ways and
one of the ways it's spent recently is
focused on
the most extreme scenarios you can
imagine and again
it's I'm fine with supporting work to
prevent those but I still think we need
a lot of people thinking about the
middle of the distribution stars not the
most extreme things they can imagine so
I would like more people to be imagining
typical futures and trying to work out
how they would play out and what we
could do about them because I think we
should will most likely live in a
typical future hmm the extreme futures
are you know disproportionately worth
attending to but still we shouldn't let
that completely push out the middle of
the distribution futures so my boy book
agents in put a middle distribution
future right even this was just about
assume a certain kind of robot and then
what typically would happen in that
world I wasn't focused on the most
extreme things that could go wrong
although I think you know having thought
about typical scenarios gives you a
better idea of the most extreme things
that could go wrong and I think if you
don't have a good idea about typical
scenarios you probably have a pretty
poor idea of what the worst things that
could happen are just don't understand
what the world looks like so it'd be
pretty hard for people like you know
during the medieval times to imagine
this kind of typical future that we're
living in what do you think we're not
living in the typical future I think
that in 1500 if somebody had understood
the basics of industry they could have
said a lot of interesting things about
our world maybe a new people here you
know they would have had to understand
you know the key assumption of
technology in terms of machines can make
machines and machines and there's plenty
of energy for the machines and then
machines can do many things and from
that they could have anticipated many
things about the industrial world they
would a lot of details they wouldn't get
right but they would roughly get a lot
of basics right in terms of well we'd
clump together in cities and we'd be
factories and we'd make cars would go
around fast and we live in buildings
that were artificially heated and
lighted and you know they could have
understand a lot of the basics of our
and that we'd be rich those are some
basic things you could understand about
world so they wouldn't necessarily be
able to predict you know nuclear weapons
and our new mad process for handling
that they could predict there be some
powerful weapons that would be scary and
we'd be dealing with but they wouldn't
know what they were exactly or how to
deal with what how to prevent them I
would not guess you know I wouldn't get
how to create a nuclear bomb I wouldn't
know what do they get it would be able
to reach the moon what did they actually
think that was something using stand on
instead of just be like I mean we
actually have say you know speculations
from Benjamin Franklin you know which
weren't that crazy about the kinds of
thick technologies would eventually be
able to do we've got you know mid 1800s
speculations about where technology
would go and a lot of those you know got
a lot of things right we've had a lot of
reasonable speculation about the basic
capabilities that say the Industrial
Revolution fruit would eventually
produce a few people that did have
pretty good views of the future well
there's ones that actually informing
leaders of the time that were guiding
nations that were informing the policies
would unfortunately the the demand is
usually for the most dramatic scary or
morally tinged forecasts rather than the
most realistic forecasts because that's
what most animates people and that's
then all the way true the the most
emotional energy about the Industrial
Revolution I think was produced by
people who predicted a degree of social
regimentation so people saw the early
factories they saw a slave plantations
they saw armies and they saw a high
degree of social regimentation in those
worlds and then they saw the industrial
world as that thing that sort of degree
of organization regimentation spread to
the whole society and that terrified a
lot of people they they then imagined
futures where
like we were so regimented the
government or our firms whatever the big
organizations told us who to marry and
when to eat and what jobs to do and when
to go to sleep they imagine these very
highly regimented futures in that that
was very scary and there's a sense in
which they were right that high degrees
of red nation regimentation would in
fact be relatively efficient for our
world in the sense that would if
everybody lived together in dorms living
would be cheaper and what food would be
cheaper but they didn't quite think
through was that when we got rich we'd
want to spend our wealth on something
and we decided to spend her wealth of
not being regimented and that's why our
world isn't quite the dystopia that many
thought although at work we are this
dystopian creatures that our ancestors
feared we don't quite realize but at
work we are really quite regimented and
quite dominated and quite submissive and
we put up with enormous degrees of of
ranking and being given orders and being
compared to other peoples that most
humans through history would not have
put up with that's who we are as
industrial people and people were scared
of that ahead of time but we're okay
with that now because that's just
limited to work
and when we leave work we are no we are
not the regimented creatures that are
our ancestors feared hmm interesting
yeah so now you take more interest as
you've got a particular vision I mean
you've already mentioned one in the in
the age of M has you got a normal vision
of the future a typical future that
you'd like to make sound I mean you can
talk about ends well we're we should end
soon but I'll just outline
I mean age of M focuses on a concrete
scenario the Vassar tonight of robot
taking over i've more recently been
thinking about what happens when ms have
to compete with rob other kinds of
robots and in some sense i see that as a
more typical future in the sense that
it's it's not obvious say that ms are
the first kind of AI but it's obvious
that eventually aims are possible and
it's also obvious that eventually other
kinds of AI is possible and so then it
seems obvious to me that eventually both
kinds will exist and then the game at
that point is which kinds when where
that is in a competitive world now many
people who see these various future
problems that they're worried about say
this is all terrible if competition
continues so somehow we must find a way
to no longer allow a competitive world
we must make a world government to make
a singleton whatever it takes to stop
competition because otherwise
competition will destroy everything of
value there are many people who have
that imagine but it seems to me when i
think it's true that a competitive world
isn't so bad and in a competitive future
world I actually think human brains as
emulations have a decent future
competing with other kinds of software
the human brain is this remarkably
powerful integrated package and I don't
think other kinds of software will
compete with it very well for a long
time to come
ask the things that it's best at mmm so
you know we are best at this sort of
integrated point of view where we look
at a broad scope and
take a wide range of considerations to
make broad choices and we will our
brains will probably be excel at that
compared to other software for a long
time to come will mostly automate much
narrower scope tasks so like as we
talked about Drexler's AI services the
the direct economic incentives will be
mostly to automate particular tasks that
are done in the economy and most of us
on the job do relatively narrow tasks
our minds are capable it'd be much more
general and that's part of the
frustration of living in the modern
world is that we are Ford your minds
built for Ford your world but we live in
this world where we must be much more
specialized in our job roles but that
means that when we automate to swap out
job roles will mostly be swapping out
humans for much more specialized narrow
automation and that'll leave humans to
do the high-level tasks for a long time
to come
in Dave Wow in a very interesting
discussion in Robin and as it always is
is there any other concluding remark
you'd like to make at this point I'd be
wanting to be there well I guess I uh
I'll make a a slight complaint or note
which is this AI Foom debate happened a
long time ago when not very many people
were concerned about AI safety in the
future and at that point people like
Eleazar were eager to get anybody to
engage their concerns and I was happy to
be one of those people to engage those
concerns at the time and then over the
last ten years there has been this boom
of interest in AI and boom of concern
about AI and a boom of attention to AI
risk but once the ai ai risk people got
validation from high status people who
said their problems are not crazy the
con
about the bait whether the the scenario
makes sense disappeared there's been
almost no interest in revisiting these
AI Foom debates once once AI risk became
a sexy topic are they basically a much
other than just you know um few revisit
to this original AI firm debate right I
think it's because once they're basic
premises are validated they're not so
interested in revisiting them they want
to continue on from those premises to do
the things that they want to do so it's
I think it's an understandable human
behavior but it's a interesting feature
of intellectual debate discussion to
keep in mind for other examples is that
when a field is trying to establish
itself that's the time it's most willing
to engage Outsiders in critiques of the
basics of it once it becomes more
established it's not really very
interested in that what what further
directions
do you think this debate could take well
there's just a lot in any debate there's
just lots of detail to get into further
I almost never
I very rarely tired of getting into the
detail of debates on any important topic
so if a topic is important enough it's
well worth just going over the basics
more carefully a little more slowly
making sure you're more precise about
definitions more you know detailed about
itemizing scenarios it was just always a
lot more work if you do and you know if
the question is important enough that
works worth doing mmm fantastic and I
certainly agree with you there Robin for
sure I look at they in one fold and I
really enjoyed this and if any of the
viewers has also enjoyed this consider
subscribing and liking this video but
also check out Robin vlog overcoming
bias and all it he was linked there to
other work that he's done and there's
Sue's book that he's got we've got the
age of him up there at the front and in
the background we've got the elephant in
the brain
and I should supply a couple of links to
the interviews we've done about the
particular subjects in the past but
thank you very much it's been great
talking ticker