Stuart Armstrong - AI: Humanity's Endgame?

hey everyone welcome back to the tour of
data science podcast today
we're talking to stuart armstrong who is
a research fellow at oxford's future of
humanity institute nester has done some
really interesting work
figuring out some of the properties that
general forms of artificial intelligence
will have to have in order to make them
safe and desirable
he's done that work not only with people
at the future of humanity institute but
also with folks at firms like deepmind
he's a deeply thoughtful person and he's
dealing with
important and fascinating problems that
humanity is going to have to grapple
with at some point
down the line so it's just fascinating
getting his perspective on all those
issues and i hope you enjoyed the
conversation as much as i did
hi stuart thanks so much for joining me
for the podcast thank you
you've covered so many different topics
topics in the area that we've been
exploring lately on the podcast
um first off though i'd love to get a
bit of a sense for your
your bio how did you discover the space
and and what was your journey
how about have i become a wise old man
of the uh
of the ai and existential risk field um
possibly um so it starts very simply
i was doing maths i heard about the
fhi it seemed incredibly cool
to work on the problems there um
i tried to get a job i failed
uh i hung around there while working uh
close by uh eventually uh
they took pity on me and gave me a job
and i've been working there ever since
and at some point along the way they
made me care
about the problems deeply and personally
which is a bit of a bastard move on
their part so i'm stuck
there for the foreseeable future and
what are some of these problems what
were some of the things that captivated
you
well a lot of the work we do is on
the far-flung future of humanity
or it would initially was on
i think the three themes were human
enhancements
uh the risks of the risks to humanity
the existential risks
and the positive what are the sort of
maximum potential
positives for humanity um but it emerged
that
there were we didn't have equal leverage
on all of these areas and they weren't
all equally important human enhancements
uh didn't seem as powerful or doesn't
seem as powerful now as people bought
initially
uh especially the sort of biological
enhancement that people think about
a smartphone may be more of a human
enhancement than
any sort of memory drug that we could
ever
get in the near future still a bit of
work on that but
um and the far the big positives
um well it emerged there wasn't much
that we could actually do
there it seems that
we that trusting the future to sort out
its own positives
is a it's actually
quite productive and it's quite it's
hard to beat
that baseline but in terms of
existential risks and avoiding them now
that turns it had a massive amount of
positive returns
so that tends to be so um we didn't
set out to be sort of doom and gloom and
the end is nigh
uh but it turned out that was the more
productive area to be working on
and i can imagine because one one type
of argument that i've heard
um from a certain kind of ethical
standpoint is people who'll say well
look
the far future is is far uh why should
we care about it
right and so i imagine it's something
you've spent a lot of time thinking
about it's like
the reasoning behind you know why i care
deeply about the far
future of humanity do you have any any
thoughts like what were the some of the
the considerations that caused you to
move in that direction
well the the far future is far in terms
that it's obscure if we
knew exactly what was going to happen if
we knew the names
of our descendants in 200 generations i
knew precisely what fates
would happen to them depending on what
we did
then this would be a much closer story
people uh are people no matter where or
when
they exist now there's a certain part of
the criticism
that's less ethical and more practical
which is that our power
over the future diminishes or
is is very small it might be very
arrogant
to assume that we have great power over
humanity
or whatever humanity will be in 10 000
years in that case distance
is an argument for not focusing so much
this is where existential risks come in
so we focus on existential risks in the
near to mid future in the
next 100 or 200 years
mainly um or shorter
time scales than that because if
humanity gets wiped out
with the one thing we know about the far
future is there isn't going to be one
if for instance right this moment we
prevent humanity from getting wiped out
this has a massive influence on the far
future
just by allowing humanity's far future
to exist so
the practical argument does not apply
for
avoiding existential risks what are the
risks that you consider
the most likely to materialize
all of the risks are ranked differently
in terms of how predictable
they are how deadly they're likely to be
and a few other factors
so if we look at one end of the scale
a potential existential risk is a meteor
impact
and we're actually pretty we've got
pretty good numbers on this we're
counting
the sizes uh the various ass accounting
ambassadors asteroids of various sizes
seeing their trajectory estimating how
often these intercept
earth and those kind of things and we're
getting better and better data on that
all the time and the risk it turns out
is
low sufficiently low that we probably
can ignore it
for say half a century or a century
and by that point if we continue as a
technological civilization
we'll probably be able to shield
ourselves from it
what do the risks happen to be sorry for
for that category of event
i don't remember uh the numbers were
just very low
um so we can trust it to sort itself out
technologically
yes i mean it's worth keeping an eye on
yeah and there are people keeping an eye
on in fact this is another positive
things people are working on that
interestingly at about the same time
that we realized the risks were lower
than we
thought people started taking the risks
more seriously
oh wow less of a science fiction more of
an actual
so the um the seriousness with which
we take the risks seems completely
disconnected from the
probability of it happening do you think
that has something to do with
how easy it is to imagine because when i
think about a lot of the risks that
people
poo poo'd like you know pandemics is
maybe a great classic example now that
we're living through it but
you know in 2019 it would have been
impossible to imagine a whole world in
lockdown etc etc
whereas like something like an asteroid
impact there's something kinetic
something physical about it that maybe
is more compelling do you think that's
part of it or um is there another
explanation
i think things just go in fashions or
um and maybe it was because there was
very little work on it so
the people who were talking about
asteroid imports
were tended to be probably more fringe
people or
not as many but then when you get
serious scientists doing conferences
on this risk there then it becomes
respectable
and that seems to be part of the dynamic
we also
we build for the the previous
risk so that's for pandemic i'm pretty
sure we're going to have
much better anti-pandemics in 2022 than
we had in 2019
in that case that gets us to a category
of risk that is likely if it
materializes to only happen
once and then like that's curtains for
human civilization i guess those risks
we can't learn from we can't iterate on
them and kind of like
you know intergenerationally or within a
generation get better at dealing with
them
um what are some of the risks that fall
into that category because i know you
spent a lot of time thinking about those
too
um here's where we get
sort of civilization collapse or
technological risks
mainly the sort of top
in this area would be ai artificial
intelligence
which is a very annoying risk to deal
with because
it's extremely difficult
to predict unlike asteroid risk if you
want at one end was
the most predictable ai is one of the
hardest we don't really know what the
capabilities of any machines that we
might develop might be
we don't know if even if it is dangerous
um
though there are quite strong arguments
that it should be
and we don't we don't know its
capabilities we don't know how
it might be controlled we don't know
what society
will look like and whether it can
exploit flaws in society and so on
but one of the reasons why ai despite
having probably a relatively low risk of
happening is that science fiction
and uh stories tend to make us think
that
huge ba large bad thing happens
civilization immediately collapses and
depending on the moral of the story
everybody dies or a plucky group of
adventurers rebuild civilization
but actually looking at history
those two things don't happen often
civilizations do not collapse easily
they can sometimes they splinter but
states
just don't fall easily and they don't
stay fallen
unless there's say warring groups around
them
um and even when you're in period of
anarchy
like somalia most recently and
china i think in the 20s and 30s might
count
but even in those sort of periods
it wasn't mass death for everybody
and in places it was quite tolerable
so it's it's not just the big thing
a large enough asteroid might just kill
everybody
but a medium-sized asteroid might throw
up a lot of dust and then
there'll be huge pressure on
civilization this won't be a nice time
but it's probable that humanity would
actually survive that
the risk of ai is sort of
in a sense the reversed of it going
really badly wrong
is relatively low in comparison
but if it does go wrong the risks of
civilization collapse and extinction are
much higher
conditional on that happening simply
because
if a powerful ai is an adversary it's an
intelligent adversary
and unlike say a disease which just
burns itself out
as time goes on and intelligent
adversary gets stronger as you get
weaker
what scenario do you see unfolding on
the i imagine there are a couple of
different ones but
like if you had to nutshell the ai risk
um
scenario what what story could you tell
to to present it
it would be three sort of factors the
first one
is that we may develop machines that are
very intelligent
um don't think necessarily in terms of
consciousness because that gives a whole
other debate but very skilled at
solving problems solving all problems
that intellect
can deal with if we do
they may become very powerful
just in the same way that humanity is
extremely powerful
over the natural world compared with say
chimps or gorillas
or pretty much any of the large mammals
that mainly survive
because humanity has decided to let them
survive
so intelligence does seem to have
strong returns to power
and finally so
if we create these machines if they're
very intelligent and very powerful the
world will start to
resemble what their goals will they put
high in their goal ordering if they have
a goal ordering
which for various reasons it's quite
likely that they would
and then the final piece of it is it is
really
hard to design a goal
in which humanities in which humanity
survives well
here you're referring i assumed the
alignment problem like what is it that
makes the alignment hard in your opinion
the problem with the alignments problem
is that almost any
goal you can think of is something
that if the ai could wipe out humanity
and take control of everything
at uh easily
that would it would do with its goal
this is sort of the more exuberant
sci-fi
scenario but if you have a money
maximizer
that you might want to run a corporation
then
extinguishing the human race and taking
over whatever
institutions remain in order to give
itself
arbitrarily large amounts of money
is something that it would do
it's kind of obvious because we've seen
corporations if not restrained
tending towards that direction so you'd
want to put more goals
for safety goals to make it compatible
with human
uh flourishing and survival but even
these goals
are problematic depending like if you
want them to keep humanity
safe uh then it may in turn
us in concrete bunkers
uh on feed drips to keep us safe and
maybe with heroin
to keep us happy if it's safe and happy
the so if we give it a goal like keep
humanity safe and happy and we don't
explain
in code properly what we mean by this
these are the sorts of outcomes that
would rank high in its preference
ordering
now it doesn't mean that it'll be
immediately doing that it just means it
will be pushing society in that
direction
and over the long term may get there
so some people at this point feel well
that's a rather dumb ai
if you didn't realize what we meant by
safe and happy unfortunately
it pro it will realize what we meant by
safe and happy
it just won't care it'll follow the
goals that we've given it or programmed
in it or
the mixture of learning and programming
or whatever tended to
even if it knows precisely what our
goals
should have been or would have been
there's no easy way to get it to
to follow those yeah and it also seems
like a related problem here is that
humans themselves don't even know
what we want out of life i mean the fact
that that's like
if you asked the average human you you
turn them into
a cosmic overlord and gave them infinite
superpowers it's like
it's not obvious that they would make
decisions that would give you anything
particularly utopic
it seems like the human problem the
problem of not knowing how to
how to set human goals is related to
this do you see a connection there or
are these sort of more separate problems
than that would imply
they are they are related
though i've had a slight difference in
perspective
recently um it's not that humans don't
that we don't know our goals is that we
do know our goals
but only in familiar environments
so it's not if you it's not that our own
it's not so much that our goals are
undefined
it's our goals are very clear but
over under the undefined or underdefined
concepts
one example you could think of is
suppose
you took ancient greeks and they had
perfected some mechanical ai
and amongst the goals that they wanted
it to preserve
was preserve honor for example concepts
of honor
honor must be rewarded say that's one of
their goals there
now what does honor mean nowadays
so you port this ai to now honor has
is basically it's very unclear and there
are multiple ways you could have
extended what the ancient greek concept
is
to the world of today
so that's the way that i'm sort of
thinking of this
we might have our concept of happiness
or flourishing or good life
or equality or pretty much anything
and in all the
possible spaces that the ai could push
the future into most of these spaces
these concepts we don't know what they
mean and we don't know how to extend
them
so it's not so much a question of
getting the definition
right as in
getting the definition in a way that can
be extended
uh to new situations this is more about
defining a process than a value
i think so it's a bit of a nuance i mean
you could say
that the process is defining exactly
what it is or something like that
but i i think this way of looking at it
uh might be more useful do you think
that there's
also a problem of consistency between
humans and among humans in the same era
like
you know the values of the the median
human in
you know the united states versus those
of
a typical person in in say you know
europe or china
these are going to deviate quite
significantly and it's sort of hard to
imagine a single set
of of moral or ethical frameworks that
an agi could apply that would make
everybody happy
i don't know maybe it's a fallacy to be
that rigid in in my thinking and say
like okay the single rule has to apply
to everybody
but do you see that as part of the the
challenge here even if we could
kind of explain exactly what we wanted
to these systems that what we want might
differ from one culture to another
i think that's much less of a problem to
be honest
because the differences between humans
though they appear
huge from the inside
tend to be relatively small from
the outside getting getting the ai in a
vague ballpark of something
human-like seems to me the bigger part
of the challenge
now uh the the main difference
between different humans moral systems
tends to be to a large extent um
who gets to count as worthy of moral
concern
and who doesn't that's i think the
biggest difference in practice
and doing trade-offs between different
people's preferences
even moral preferences is the kind of
thing
that an ai could do
and it it doesn't have to do it
perfectly and i don't think there is a
standard of perfection
but there are approximate ways of doing
it should be fine
it does make sense i mean from that
outside view when you look at the
collective whole of humanity everybody
everybody does want certain things it
seems or at least there's some
you know some sense that everybody wants
to be loved everybody wants to be
appreciated
what those things you know how those
things might manifest maybe change in
one culture or another but
that fundamental need i guess still
feels pretty consistent
but i'm not saying it's not going to be
a challenge i just feel that
it is in a sense well it's not the
challenge i'm working on and i think
it is easier once you've got the ai
in this area to come up with something
adequate
from that and how optimistic are you i
mean
because you mentioned you think it's a
low probability event with you know
very significant implications um there's
obviously a debate in the ai safety
community some people think it's like
it's almost certainly not going to be a
problem to the point where you can
almost ignore
ignore ai safety or ai alignment other
people think or seem to think it's
almost
almost guaranteed that this is going to
be a an apocalyptic
event at some point in our future where
do you fall on that and what are some of
the arguments that you see on either
side like what are some of the most
compelling arguments for
like the proposition that this is going
to be a catastrophe for sure
and what are some of the the best
arguments you've heard against
the problem with
most of these arguments is that
they're extremely overconfident
um people know that they would have
difficulty
guessing who the u.s president would be
in 10 election cycles
say even though
the u.s president is almost certainly
born today so we
uh for intent election cycles so you
could
you'd think that you yeah here's
here's the list of all the people today
in eight election cycles
they're in there somewhere someone one
of these is going to be
a u.s president so
but when you do predictions like we
might have ai
and these are the features of ai which i
think is a lot harder
than what is the us president in eight
election cycles
then people seem to get a lot more
confident
that no this is definitely not going to
happen or this is definitely going to
happen
uh it seems that the the less we have
to work with in a sense the more
confident we get
yeah you might you might sort of see the
contrast between uh
economists and historians who vehemently
disagree with each other
and physicists who have a lot more data
and who tend to be much more
nuanced or very uh or close to each
other
actually i'll add to that too um just
the the area of physics that i worked in
back in the day was this kind of niche
area called interpretation of quantum
mechanics which is precisely one of
those areas where
you have almost no data um you have all
these different perspectives on like
what the data could be telling us and
they're all equally backed
and everyone is 100 confident in their
respective positions despite the
absolute lack of data
so sorry it just seems like to perfectly
mirror what you're describing but even
in the microcosm of physics
so there's a huge amount of uncertainty
now what uncertainty does it tends you
to
it tends to push you towards the middle
so if i was to say
ai super intelligent ai
is definitely going to happen and it's
definitely going to be an existential
risk
that would be completely stupid but
saying that it's definitely not going to
happen
is also a very wrong position to take
because we just don't have the evidence
as i say we have arguments so the case
for
this could happen is
what i sketched out ai's are getting
more powerful
intelligence seems to correlate with
power
uh powerful entities push the things in
various directions
and empirically getting them to push in
the right direction is very very hard
so um now this argument seems to rely on
various things
happening together so getting a powerful
ai getting it
getting as intelligent ai the ai getting
powerful that's the sort of
argument that convinces me that there is
a risk which
varies depending on how i think about it
between 20
20 to 40 percent range
of uh powerful potentially disastrous ai
other people can slice it differently
but i would be
very hard-pressed to find any reason
that it would go below five percent and
i'd still be working on it if it was 0.1
um so there's ample probability there
for me to work on it
just because the impact is so high uh
yes
the positive impact too if in a scenario
where the ais are really powerful
if you get them aligned with humanity
then you get a wonderful
utopic vision yeah yeah utopia
is in a proper utopia a place that would
be
fun to a place would be fun to live in
and you would get a lot of interesting
and varied experiences not the standard
utopias
that people write about which are very
boring that i would see as a
failure do you think that the difficulty
of
of outlining what a utopia might look
like is part of what hints at how hard
this problem might be
maybe one the one way of
seeing it is that humans seem much
we seem much better describing hell than
describing paradise
so it's much easier for us to list the
bad things
than it is to list the really good
things and most utopias if you look at
them it's
we've removed all the really bad things
that's their main
feature yeah there's no sort of
torture war famine uh all those sort of
things and utopias but
then once they've removed the bad things
and then they need to move on to
actually putting in the good things then
it gets
we get a lot more loss but yes i think i
think it is related
typically the arguments in favor are
looking at the details
this is this is an ai intelligence power
all those sort of things
uh considering the scenarios that could
happen the
counter arguments tend to be looking at
it from the outside
saying this is a revolutionary
technology
but humanity has dealt with
revolutionary technologies
many times in the past we have adapted
we've adjusted we've integrated it into
society
uh we've taken precautions especially
after
and we've always had sort of maybe a
small problems that we can then build
into
bigger ones people are working on this
problem
uh and so problems where people work on
a lot tend to get solved
so the so-called from the so-called
outside view
um you could say that we should we'll
probably manage ai
because we've managed uh every other
technology or every other
challenge in its similar category uh
and i think there is a lot of weight to
that
i just fear that it may prove to be
something that has the skills of a
general human intelligence
in software form and run at high speeds
that
in this case the inside view is worth
looking into
so two kind of arguments the first is
the outside view and the second one
which i kind of mixed in
is we will sort it that's a
view in a sense that i've moved closer
to myself
because i'm getting more confident that
we may end up sorting
this this kind of thing successfully
i can see the beginning of a path from
where we are now
to very safe
super intelligences what are some of the
things that have nudged you in that
direction
well it's it's all very hazy at the
moment
but it's i felt that before when dealing
with ai safety we were climbing
a staircase in the dark we didn't know
what the st steps were we didn't know
how many they were
now it's still dark but i can kind of
see
the pathways to the goal
and how how they might need to be
combined and
the what kind of work needs to be done
on each one
and what are some of the what are some
of those steps look like that have come
into view recently
one thing that i was looking at
so in all my i've tried a lot of
different approaches to ai safety
uh making safe oracles hey everyone just
gonna break in here really quickly and
explain what oracles are
in the context of ai safety for anyone
who might not have heard of them before
so an oracle is a specific kind of ai
that's only capable of answering
questions
and the hope is that by imposing that
constraint we can reduce the potential
harm that a superintelligent ai could do
if it were misaligned with human values
i've tried a lot of different approaches
to ai safety
making safe oracles making reduced
impact ais
getting them aligned in a variety of
other things trying to reduce their
power
and after a while i felt
that in a sense the same problem
kept on coming up which you could see
in a very very generalized form as the
out of distribution
problem for machine learning
in that we know the the concepts that we
use
breakdown in the exotic or extreme
scenarios
that intelligence could push towards and
i think that addressing this directly
is part of the problem other
ass and that's what i'm like one of my
current big projects
another one is formalizing what does
human preferences mean
how can we sort that out so i've got a
paper showing that it's impossible in
principle
so in theory it's impossible because you
cannot get the preferences of a
potentially irrational agent but
i have i think that in practice i can
see how
we can get there uh how we can identify
the preferences put them together
or at least a path towards that so i
think
we are making progress on defining human
values
idealizing human values and figuring out
how to get the ai's to learn
it how to extend features to new
environments
how to solve symbol grounding not these
not in the philosophical version but in
the practical version
and what is simple grounding like what's
the significance in terms of ai safety
um so simple grounding is that you have
some mental symbols in your head
like for other people food
stuff like that and how do we know what
these corresponds to
in the outside world in the very early
years of ai we just named
symbols after what they were supposed to
represent
and we thought that if something was
called pain inside the ai
or something was called probability or
beliefs
and changed in the right way that that
would be enough to
make it have those properties which
turned out
mainly not to work so
how do we know that this symbol in the
brain of the ai means
something in the outside world
and i've i've been looking at it
from the um practical point of view
rather than the
philosophical so rather than wondering
what is
the meaning of this symbol i'm more
doing okay this is a symbol inside the
ai
these are features of the outside world
do they correlate
can we tell something on what's going on
the outside world by looking at the
symbol
inside the air's brain can we tell what
the symbols the eyes brains would be by
looking at the outside world
if there is a strong correlation i would
say that this symbol
is grounded or relatively well grounded
this is actually really interesting by
the way because i did a podcast
i think the last podcast as of right now
that i recorded was with
um was with a a professor of
neuroscience who's focused on
consciousness and oddly enough
this idea that you're mentioning like
your description of grounded symbols
does actually seem to map on to at least
his one of his definitions of
consciousness
which was just that the um uh well
essentially that
there'd be a correlation between the
symbols in our in our brains and and the
actual
uh objective facts on the ground in
reality uh it is a rabbit hole but
is there is there something about our
own experience of the world
that we should be paying more attention
to as well when we look at ai safety
like
is it do you think that's a fruitless
line of research and thinking to to say
i'm going to think inwards i'm going to
do some meditation see if i can
explore subjective experience a little
bit more to get some inspiration on this
i'm not necessarily convinced that that
would be fruitful um
i think you can have well symbols
without any trace of consciousness
if you if you know someone well
then their name is a very well-grounded
symbol
of themselves um
so the written name uh could in this
okay but maybe that that stretching but
you can see
in algorithms that are running uh
various things if they run them well
then you can well by this
uh semi-formal theorem and also by a
practical experience
you should be able to identify inside
them the symbols that correspond to the
concepts
that they manipulate outside and
so you don't seem to need consciousness
to deal with that now consciousness
seems to deal with symbols
in certain interesting uh
unusual ways yeah so i i don't think
uh especially for the moment that
there's all that much to be gained
uh in going down that route it's uh
there's more of x paradox
everything that's easy for humans is
hard for computers and vice versa
what this really means is that our
conscious mind and thoughts
are on top of great
unconscious processing things created by
evolution
so we the things we can do instinctively
are not the things that we can best
explain to a computer
because it's the things that we don't
necessarily understand well
do you think that i mean ultimately and
this is of course one of those
like long-run things but once we do have
ai systems that
can do what is effectively like a kind
of subconscious processing
will we have things that we have to
consider conscious like well we have to
factor them in in terms of
you know when we as you said earlier you
know the differences between different
moralities often involve deciding who's
in and who's that who counts as a person
and who doesn't count as a person how
would we
even get to the point then where we're
deciding okay like this
machine actually gets a vote gets a say
and you know i mean obviously that's a
whole rabbit hole i mean you can
replicate machines turn them into the
only thing that matters and
i guess that's a potential path forward
there's a maybe an
order of operations or a priority issue
here
um i have no doubt that we could make a
conscious
machine by any reasonable definition
of consciousness we could come up with
um
again consciousness is something that's
under defined
for the moment um but i would first want
to avoid
disasters before thinking about the
rights
of the ais and those kind of issues and
their moral status
now of course a vast amount of suffering
ais is also
a existential disaster i would say
a catastrophe so that is also something
to avoid but
i think that the
let's think about the rights of the ais
is the kind of human reasoning that we
fall into too
easily whereas the first priority
is to ensure that humanity
survives it's safe it's flourishing
and then we can see if we can draw the
circle
uh wider uh it might be
reasonable if we're going to create say
10 million 10 trillion ai's
to run various things of the world it'll
be very important to know
whether they're suffering or not but
uh in our first powerful ais
i say i'll go with safety first
the ethical imperative is to ensure that
these
machines are safe and allow a safe
future
the other thing is that humans tend to
we see gods in clouds
and rock formations and volcanoes we see
consciousness very easily
and things that are not conscious and
this is a potential
avenue of exploitation for
a non imagine a non-conscious ai
this might be an avenue that it could
follow so
i would think that we should err on the
side of
thinking that the ais are not conscious
even if we think they are
not interesting in terms of giving them
power and autonomy in terms of
preventing their suffering
we should probably think they are
conscious even if we think that they're
not
there we should be uh we should be
concerned in the other direction
but in terms of giving them power and
autonomy we should be willing to think
of them these are extremely dangerous
potentially psychopathic potentially
non-conscious things
that can appear conscious to us if they
so choose
once we're safe then then we can
start uh releasing them speaking of the
appearance of consciousness
one thing that i think has caused a
number of people to update their views
on the prospect of general intelligence
in the medium term
is um some of the large language models
that we've seen recently coming out of
in particular open ai but then
after i think google's come up with
something even bigger gpt3
like has been all obviously all over
twitter all over the internet really
impressive stuff
um how has that affected your view on
timelines i mean
are you finding yourself shifting
towards thinking that
agi will be hit sooner as you see more
progress or is
the progress we've seen in the last few
years sort of in line with what you
might have expected
say i don't know in 2015 or 2012 let's
say right after deep learning became a
thing
i'm now going to speculate um the
what i've said before though it's not
uncontroversial
is at least shared widely amongst a lot
of intelligent people
so is a mini consensus at least in some
areas
what i'm about to say now is just my own
opinion
i think that actually gpt
3 may be a sign that we will not
be getting general intelligence
so rapidly
this is connected with my ideas
on symbol grounding the basic idea is
that it
seems that great almost human-like
performance
can be achieved by
mimicking humans with the right
and having the right architecture a lot
of data what humans have done
and gpt3 does not seem
to have what we would call understanding
or general intelligence uh
if you sort of push it more
on some of its answers that seem smart
or dig in more or
make it generate longer it is at some
point going to make a mistake
that reveals its lack of understanding
now it's very impressive what we've got
but this suggests to me that it's
actually possible to imitate a human
with very little level of understanding
at least in
creating texts that means that
you can't really create something if you
don't have a benchmark
for it so
one of our best ways of measuring
understanding used to be variants
of turing tests everyone just jumping in
here again i'm guessing most of you are
already familiar with the turing test
but
quick clarification just in case a
turing test is an experiment that's
designed to determine whether an ai's
behavior can be distinguished from that
of a human being
and it was initially hoped that turing
tests could be used to determine when
ais were finally at the point where they
were actually thinking
in the way that a human being might but
it's since become clear that that idea
has quite a few loopholes
and most people have given up on the
idea of the turing test as an
interesting measure of the performance
of artificial intelligence
one of our best ways of measuring
understanding used to be
variants of turing tests variants of
just
outputting texts seeing
seeing if it was coherent and stuff like
that or human-like
and we've got to the point where we have
very human-like tests
but no real understanding
that suggests to me that we don't have
any real way of measuring
understanding so i don't think since we
don't have a way of measuring
understanding
it's very hard to optimize for this or
to aim for it
one thing i was thinking is can
gpt n or whatever
creates concepts beyond what humans have
done
what i'm thinking is bring us back to
1904
give gpt all the data that has been
produced
then in the world uh remove a few papers
by lorenz
and very few others now will it be able
to create special and general relativity
from this data
i suspect it wouldn't because
to in order to do that you have to learn
physics learn that these are rules
generalize
there connect this with the experiments
that are done
and then come up with a new theory that
connects
these things together whereas what i
think gbg
n or gp3 would do in gptn probably would
do as well is
look for the look at physics papers as
linguistic
texts construction or as a social
endeavor
and see and create things that are
similar or that
tick those boxes or that work in those
things okay so
physics papers have this kind of
structure they talk about this thing
they
connect this they have this amount of
data and so i think
that that is i think a lot easier than
learning physics of the universe from
just the linguistics of
papers correct me if i'm wrong but it
it would seem like this argument would
sound something like
papers vary more by
the language used by the authors than
they do
in terms of the content of those papers
so so like if gpdn
is interested in just like it's an
autocomplete algorithm fundamentally
that's what it is
if it if it's interested in doing the
best job it possibly can
at predicting the words that are going
to be used like
focusing on the language rather than
um then logic becomes a higher priority
well it's more there's two ways you
could build a physics paper
there's lots of ways you could build a
physics paper but let's just focus on
two
the first one is read it understand all
the concepts
generate your entire model of physics
um figure out something new from these
concepts
write out these concepts share
with a so with a social understanding of
what your
what the sharing means the other route
is to
from the various texts pattern spotting
from the text
and extend from that
now if it is possible
to create a good physics or physics c
paper by pattern spotting then that is
what gpt
3 will do there's no point in building
an
overly complicated model when you can do
it with a simple model so what you what
you need is a test
that reliably distinguishes between the
two approaches
what is the thing that right
um that shows that yes you did have to
understand
physics and not just write stuff
that said you understand physics but in
order to
achieve this um so the the fact that
it seems that just generalizing from
texts can get you so
far suggests to me
that um that actually getting a deep
understanding is harder
because it's going to be harder to
distinguish between those two
well what i used to tell people about
five years ago
was that my eddie 80
confidence interval is five to a hundred
years for
strong uh strong forms of ai i think
i have i i think it has been
accelerating so i would say
that my 90 confidence interval would now
be
5 to 70 years let's say i'm more
confident and i've narrowed the
four human comparable general
intelligence
and when that happens you think
something like 20 to 40
odds that it goes like horribly horribly
wrong something like that
the one thing that i'm having trouble
modeling is the effect of human
intervention
and as i say i am getting more confident
the human intervention will work
so it would be something like
let's say a third of a chance that it
goes horribly wrong
if done naively those timelines
obviously are a really important factor
i think for a lot of people like
when i've been talking to people on the
podcast about their views on ai safety
what we should be focusing on inevitably
timelines do come up
obviously that interval's pretty broad
but do you have any reasons to think
that it might not happen ever like is it
possible that agi
is just like something that we're never
going to be able to
uh to to achieve or figure out i mean
borrowing in all these situations i'm
excluding
human extinction or a sharp
damage to human civilization right but
that's one way that
we might never get that we know that
humans
you we know that human-like
intelligences are possible because we've
we're here um so evolution
can produce human-like intelligences
over billions of years humans and
biologists
especially are really quite good at
co-opting
natural processes so even if we don't
have these sort of
technological routes to ai
sort of the hard tech we might have the
biotech
thing where we um re-engineer brains or
brain-like things um
and then there's the brute-force
approaches of
whole brain emulations the idea of
running
running a artificial brain a copy of a
brain forward according to the laws of
physics
these approaches could work without
needing to have
a great understanding of
of intelligence of consciousness of
thinking it would
now you'd have to be very and especially
since we're talking about skills
not about consciousness
or certain attributes
so evolution has produced the ability to
solve
certain problems and to have certain
skills
so it can be done we can improve
ourselves and we can
improve our children and we can improve
our machines in different ways and we
can co-op technology
to solve categories of problems
ourselves
so i do not think
that it is very likely that ai
would turn out to be impossible uh
on one of the multiple routes that
lead there let's give it a three percent
chance
don't hold me to it this is the first
time
i've sort of seriously i think put an
estimate on
this and it is a bit higher than i
thought
i had neglected to include the fact that
we don't see any life in the entire
universe so
life may have been evolution of us may
have been an incredible fluke
but yeah let's go for around three
percent
as my current estimate for ai being
in the terms of general intelligence of
human comparable
um skill set being impossible well and
that observation that we are
alone in the universe this is something
that i've i've seen you and
um i think andrew sandberg and a couple
of other people at
the future of humanity institute tie
into
their thinking on ai i'd love to hear
your thoughts on how you see those two
being connected like
what is it that is there anything that
you think we can we can glean from the
fact that we're alone in the universe or
we appear to be
that informs like how you think about ai
risk initially
we looked at the fermi paradox the where
are
all the aliens paradox as
a as information for human risks
because one explanations for the very
paradox is that
advanced civilizations always destroy
themselves before they reach a certain
level
of capability before they become star
spanning basically
um so this is why and then when i looked
at it and i found it was so
surprisingly easy to expand across the
universe
for certain values of easy which means
that basically any civilization that has
control over their solar system for more
than a few centuries
should be able to start a massive
colonization of everything
uh pretty uh so this made the furring
paradox a lot worse
because any civilization could have
reached us and nearby
civilizations from near by galaxies
could also have reached us
especially when you consider that the
earth is actually quite a late comer
amongst earth-like planets there are a
lot of earth-like planets that
were that's existed long before us so
there's more time
ai is an exception to this ai is the
existential risk that makes
expansion throughout the universe easier
rather than harder
first of all because it's a lot easier
to expand if you're an ai than if you're
a biological species
and second of all because the type of
misbehavior
that would cause an ai to cause a
disaster for humanity
the sort of unconstrained
goal function is also exactly or almost
exactly the same sort that would cause
it to want to expand
as much as possible so unlike
other disasters
ai would leave a trace
in the universe right
but after a while and after looking at
various factors
it seems that the most likely
explanation is just that
um advanced intelligent life is hard
just that hard so in other words like
there may be that many
planets out there that many galaxies but
the
like the the probability of getting life
to emerge is just so low that even with
that number we we see
n equals one yes so i'm looking up the
drake equation
um where this is
an estimate this is the estimate for why
there should be a lot of alien lives all
over the place
again just dropping in here if you
haven't heard of the drake equation it's
worth reading about
essentially it's an equation designed to
calculate the number of detectable alien
civilizations that we should be able to
see in the universe
by multiplying together a whole bunch of
factors like the number of planets in
the universe
the fraction of those planets that could
support life and other parameters that
stuart is going to describe in just a
minute and for pretty obvious reasons
it's become the focal point for most
discussions about the fermi paradox
i'm looking up the drake equation
um where this is
an estimate this is the estimate for why
there should be a lot of alien lives all
over the place
and so there's the average rate of star
formation
times the fraction of those stars that
have planets the average number of
plants that can
potentially support life the fraction of
plants that could life that actually
that could support life that actually do
develop life the fractured plants
with life that actually go on to develop
intelligent life
the fraction of civilization that
develop a technology that release
detectable signs of their
existence into space times the length of
time
that this happens now
what we've what my work or my work with
anders
and others have helped as well is that
one of the terms the fc the fractured
civilization the developer technology
that releases detectable science their
existence into space
this is high because expanding is so
easy
in physical form uh if need be
we also have very we we have decent
estimates on the rate of star formation
and we now have better estimates on the
fraction stars of our planets actually
there's a lot of planets out there more
than maybe we initially thought
so it it feels therefore if we put our
best guess
for each of these we get something
absolutely huge but
let's look at some of the intermediate
ones
the fractions of planets that could
support life
and the fraction of planets that do
develop life given that they could
support life
and the fraction of plants with life
that go on to divide intelligent life
all we have here is guesswork yeah
complete guesswork
and maybe our best guesses are that
these are one percent
which would give lots of civilizations
across the galaxy
but it's not unreasonable to think that
it's one in a trillion
there's that's also possible
maybe say there are because one in a
trillion is
four one and a thousand chances one
after the other
so that life had to go through for one
and a thousand chances to get where we
are
or to compare what we are does not sound
all that unlikely and there you get a
one in a trillion chance
and there we start not seeing
life around oh is that roughly what it
would take like about one in a trillion
uh in order for us to be alone at that
stage something like that the the the
numbers of
uh the numbers of galaxies that can
reach us i think are in the billions
i have the numbers somewhere but it's
billions of trillions ranges
so yeah that's the thing so our best
estimate
of this may be one percent but one in a
trillion is not too unlikely
one in a quadrillion is not too unlikely
either so if we update on the fact that
we don't see any of this life
then the fact that
these these hypotheses of very rare life
uh increase now i have various arguments
based on convergent evolution
that's between say a basic nervous
system
and dolphin dolphin-style intelligence
these are relatively easy
to do so i think that the
the obstacles lie before or after that
like the first cell or something my
guess would be before
personally i think i informally
yeah first cell mitochondria
very odd things when you think about it
um
central nervous system and possibly
oxygen
oxygen well if you think about it
oxygen is a waste product of plants
life it is a waste product that is at a
higher
energy level than
the than co2 than carbon dioxide which
is what
the plants take in so it
seems surprising that at the level of
the entire planet
you have this so
energetically useful waste product and
then and
this powers all of animal life oxygen
tends to react quite a bit
uh which is why i think for the first
period when oxygen was accumulating from
the um
what are those oxygen bacteria
they have a formal name chlorophyll or
was it anyway um possibly
but for a long time my understanding is
there was no oxygen in the atmosphere
because it reacted with iron
and formed rust and only when
most of the iron had already been
reacted did the atmosphere
start filling up so oxygen is quite
reactive
so it is unusual
for it to be free floating in an
atmosphere
so if that if that is the case if it
really is the case that life is
so exceedingly rare that we find
ourselves genuinely alone in the
universe
like does that does that affect the way
that you think about like
the way that you think about the
universe and i don't want to say its
purpose but
i mean it seems like something very
weird is going on
maybe it isn't um but
does this change your perspective or
update your perspective on
i don't know what this might be i mean
is this like a cosmic experiment of some
kind is something very very strange
going on
or or is this just all it's an illusion
that we're so special i mean there might
be
pockets of universe that we can't
contact or whatever and and the same
experiments running maybe trillions of
times over and so
you know unsurprisingly we appear in one
like i don't know
how are you thinking about those
possibilities i mean if the universe is
big enough we're going to find
anything somewhere yeah so
it does cause me to rethink some of the
the meaning of the of the universe and
of the future
it makes sort of the doom of humanity
worse in a sense that there
yes there's great beauty in the cosmos
there's great interests there's
so much to learn and so much to know
so much possible art and meaning to
construct
but that if if humanity doesn't make it
or if a descendant of humanity
that has some moral and aesthetic worth
doesn't survive
then there this is then it basically
this will be
the vast emptiness between the stars and
the emptiness of the stars
and nothing nothing to appreciate this
nothing to connect with it nothing to
give it meaning
so you mentioned the um idea of
our descendants and you know if if human
like or human-descended peoples
aren't around then we've we've lost
something really valuable and important
i guess our descendants could be very
different from us i mean in general i
expect that they will be
either either biologically just because
like over long periods of time humans
will
evolve into something different or you
know because we'll be augmented
in different different mechanical or or
sort of software powered ways or other
ways
do you like does this affect the empathy
that you have for those future versions
of ourselves i mean like is there
do you feel a connectedness with the
idea of like humanity
down the road if it's gonna be that
different from us now i'd say that every
single alien
species that ever is portrayed in
science fiction
with very few exceptions is
not too far from the circle of humanity
every single star trek species is a
slightly modified human
both both do they look like that because
they're actors obviously but also in the
way they behave
it's um it's not much difference
but there could be very alien
minds out there i'm not thinking actual
aliens i'm thinking
different from us
like minds that might
think that the most interesting thing
that they can do is to put
one block on top of another block and
then take it away and then put it back
up and then take it away and do that
forever
now this this kind of mind
maybe maybe it feels no pleasure
uh experiences no pain uh it might be
intelligence possibly but this is all
that it sees these kind of minds i would
not
there's nothing wrong with their
existence per se
but i do not think that if humanity was
replaced by minds of that nature
that this would be i'd say that this is
an existential disaster that we have
failed
and i guess that's kind of what i'm i'm
getting at in a certain sense like
there are um by some
by some definitions you can imagine
people saying well like whatever you
know whatever ai takes over
if it ends up being a disaster as long
as that ai
is continues to exist in some form down
the road like
we created it it's a continuation of
humanity and therefore like
you kind of adopt almost a fatalistic
attitude towards the whole thing
whereas it sounds like there there's
sort of a list of maybe requirements
that we might want
from any kind of system like that that
will propagate in the future
for us to be able to i almost want to
say empathize with it now care about
like be satisfied that that is the relic
of of human existence that propagates
through time do you have any thoughts
about what those requirements might be
i have been working a bit on that uh i
mean
entities with a sense of identity
would be a quite useful thing to have
and there's no reason
that this would happen naturally
for say algorithms that can get copied
turned on turned off spun up
there's no need reason that they would
naturally have a
sense of identity that's anywhere close
to
our own i draw my circle of what counts
as human
much more broadly than than
many people but i think
that people don't realize quite how vast
mind space or the possible vines is
um so i think that
a generic outcome of a generic
disastrous ai
is well outside human mind space even
very broadly drawn
ah yeah yeah i mean i guess it's like
in the same way that an ant would not be
able to imagine the mind space that
could exist above it
like or a bird uh humans have the same
problem
i i wouldn't really say it's not so much
above it or it's
it's just differently it's just
different
um because you can imagine a sort of
super intelligent human
and you can imagine something that's
super intelligence and it's not a human
in any way
shape or form and so it's it's not it's
not really a
question of power or being above or so
much
it's do they have any of the things
that make human life
worth living or i mean
what are the sort of features which if
you heard that the next generation was
going to lack them or have them
that you would think that this was bad
let's take something as trivial
um in the scheme of things what if the
next generation
could never feel anger i think that'd be
complicated
it feels like part of the human
experience somehow but
i i also feel all i've done is a small
modification here
yeah and i've taken out something that's
generally seen as negative anger is not
the negative
is not seen as a positive but entities
that cannot feel anger
what what are these this seems
very weird then we sort of say well if
they don't feel love
if they if they don't have a sense of
personal identity then they don't feel
anything arguably
or their philosophy philosophy
it gets complicated but these are things
that are not very far from us in mind
space
so i think we have to be proactive
in saying that we want
at least they say a chunk of our
descendants
to have something that is in a broader
human
span you could almost i guess make up a
list of desiderata for
yeah for what what would count as like
valid human
propagation through time um i'm just
glad i'm not making the list
i want to i want to add i tend to focus
on the super intelligent ai scenario
mostly because methods of alignment or
control that work for super intelligent
ais
most of them not all of them but most of
them would also work
for much more limited entities
so i i don't necessarily think that
they're the most likely
outcome so you said ai overloads and i
say that also informally but
i focus on that but i don't think that's
what's going to happen but
it's the most useful one to work on
because if you solve that
most of the time you solve the whole of
the problem
actually like looking at um at humanity
through the same lens also seems kind of
at least interesting like is is there a
way
do you buy the argument that like
humanity itself on the whole like
all seven billion of us are kind of like
a misaligned
super intelligence with respect to any
individual person
like it kind of seems like the decisions
we make the the
the international rivalries that we
enter that are that are deleterious to
our individual well-being
so often um like is is that an alignment
problem is that in the same category or
is that a different thing
i think it's a useful analogy as long as
we don't push it too far
hmm like a corporation
can be seen as in some ways a super
intelligence
uh but a one that's far far
easier to control than a genuine
super intelligent must be because even
though
there's a whole system it is uh
populated by humans
and these put a limit on to
how the ways it misbehaves the ways it
can misbehave without getting caught
and that kind of stuff yeah i i agree
with it as an analogy as long as it's
not pushed too far
stewart thanks so much i really enjoyed
the conversation uh do you have a
a link of like a personal website you'd
like to share if people want to
follow your research a little bit more
closely i have a bunch of uh
links on less wrong that meander across
many areas there i
had a personal website but it's long
defunct so
yeah look at the future of humanity
institute and look at
uh look at less wrong perfect i'll
include some of those links in the blog
post that'll come with a podcast
and stewart thanks so much for making
the time cool
thank you