Anders Sandberg - Answering the Fermi Question: Is AI our Great Filter?

hey everyone my name is jeremy welcome
back to the tour of state of science
podcast
and i'm really excited about today's
episode because i get to talk to
anders sandberg now anders is someone
i've been angling to talk to for a
long time because his research has
focused on some fascinating topics that
he approaches with a really interesting
multi-disciplinary strategy he's a
researcher
a science debater a futurist and a
transhumanist
and he has a degree in computational
neuroscience from stockholm university
he's also currently a senior research
fellow at oxford university's future of
humanity institute i will say andrews is
genuinely one of the most interesting
thinkers i've encountered on the topic
of existential risk
and the hard questions that advanced ai
systems are gonna force us to answer
he has this way of seamlessly blending
together knowledge from fields as
diverse as machine learning
ethics metaphysics and cosmology and
that just makes them a joy to speak to
and it makes you realize as well how
deeply related these different areas
become
when you zoom out enough to see how
humanity fits into the grand cosmic
picture in deep time i
really enjoyed this conversation it
covered some fascinating topics
that all touch on the future of ai
development in some unexpected and
surprising way
those include things like why we might
actually be alone in the universe after
all
whether the energy efficiency of the
human brain suggests that general ai
might be harder
to put together than it seems and also
whether ai's will ever be conscious
so this one was an absolute blast i hope
you enjoyed the conversation as much as
i did and without further ado i'm going
to get out of the way
and let the episode start all right well
anders thanks so much for joining me for
the podcast
well thank you for having me i'm
absolutely thrilled to have you i've
been
i've been stalking you on twitter for
longer than i'd care to admit i mean
there's a lot of really interesting
stuff that you're you're working on
um so many so many topics that relate to
artificial intelligence the future of
artificial intelligence the future of
humanity
one question i wanted to start with
though was a bit biographical i wanted
to get a sense for
how you came to this field what was it
that drove you here
so i grew up in north and stockholm
in the 1970s in a suburb really boring
so i read all the science fiction novels
i could find at the local branch library
and then one day i realized actually i
want to make this real
how do i do that i probably should read
the science books so when i started over
there and then
i went to the mansible library and then
the main library and then the university
library
so that's how i ended up but i always
wanted to make the future real
if i can't write fiction about it maybe
i can investigate it
write papers invent things or figure out
what we should be focusing on
on or avoiding have you developed
opinions in the process
about what kind of science fiction
is most is most plausible and
i also want to say what kinds of
mistakes uh science fiction writers
usually make because that's sort of a
i don't know i i i wonder if you spotted
trends there
so the problem is science fiction
authors want to write good stories
and reality is usually a really bad
story
the clothing of reality is hopeless i
mean
just look at this year or any other year
and you would say
yeah this is so uneven and it doesn't
make sense
real stories of course try to make sense
they try to
tell a story that resonates with us the
problem is of course that
many of the things out in the world are
independent of our human emotions
especially when you get to the areas of
science and technology
and that means that many of the best
stores
actually don't handle science and
technology very carefully at all they
kind of miss that for in favor of human
stories
which means that if you want to think
about the future in many
cases you might want to go for science
fiction that actually
is much less worthy as fiction but much
better at
thinking about ideas but again you have
a trade-off
many of the coolest ideas might actually
not be tremendously uh
plausible i found that science fiction
that
really contains scenes for interesting
things
is the science fiction that is full of
little ideas but try to describe the
interactions of things
going on in an environment not just the
amazing technology but also how it fails
or how kids misuse the technology and
how the counterpart all the people
yelling about get off my lawn
are now talking get off my augmented
reality in the
filter at that point you start seeing
the interesting non-trivial effect
izakazimov was talking about the
elevator principle
if you show pictures of the skyline or
washington
or sorry not washington if you show
pictures of the skyline of new york
to somebody from a past century they
should be able to figure out that there
must be something like elevators
because otherwise skyscrapers don't make
sense it's
a tool to walk up all those stairs um
maybe they're going to be wrong and
saying yeah all the rich people of
course live conveniently close to the
ground floor and the poor people have to
make you
up in the the high altitude flat but
they would be forced to realize that
there must be something like an elevator
to make sense of that picture
and i think this is also where science
fiction can be the most useful
it makes you aware of some of these
elevator principles for me for example
thinking about maintainability
of a lot of advanced technologies is an
interesting question
how do you repair a space elevator if
you build it
around yourself how much effort does it
have to take to protect it
and keep it from breaking down when you
make an artificial intelligence
how much extra work is it to keep it
sane and functional
interesting that's almost um it's funny
there's a principle in startups that
this reminds me of which is you should
always
you should always aim to work on
problems that seem boring
that there are things that you know
those are the areas where people aren't
working and it's where companies like
strife you know
a lot of people don't know what stripe
is it's a payment processing company
they do the dirty work they do the
plumbing of the internet
uh this sort of seems like one of those
similar ideas do you think that there's
a
similar effect that cuts the other way a
little bit too where
people might encounter an idea that does
observe the elevator principle that is
sort of a rational forward-looking
prediction
but whose implications are so profoundly
counter-intuitive
that people just almost reflexively push
back at it
is that something you've seen as well oh
yeah all the time
indeed getting back to classic science
fiction author or to see clark in his
book profiles of the future
he was talking about failures of
imagination
and failure of nerve quite a lot of
people especially academics
have a failure of imagination they can't
imagine things being very different
i remember being told a professor of
nano technology that self-replicating
machines were absolutely impossible
when i pointed out what about bacteria
yeah but we can't build it
so from his perspective self-replicating
machines were absolutely impossible
because he couldn't put it inside his
project
and he didn't want to raise his nose
from what he could do
to the systems that actually exist but
when you have a failure of nerve
you can imagine some things but you
don't want to follow through because
the consequences are so vast so weird
that okay this just sounds crazy i'm
just not going to talk about
again nanotechnology has that problem
because the original visions of eric
trexler
demonstrate that if you get atomically
precise manufacturing
that can be scaled up the world gets
really different and that's got people
interested in nano technology but
unfortunately the field then
was taken over by people like that
professor who wanted to work on stuff
that was normal so we ended up with a
lot of wonderful
solid state science but the idea that
you could actually do things that really
transform a world
that's not really what we do here in the
lab so that's probably crazy talk
and the same thing goes of course for a
lot of other domains we have seen it in
space
that was what clark was writing about
many people were criticizing the early
space pioneers
and giving all sorts of reasonable
explanation why you could never
build a rocket that could get out in
space the problem was of course that
they
were thinking about the simplest way
they could do it
and then they could demonstrate that
that wouldn't work they didn't think
about
if somebody is really motivated to do it
and actually spend some time on making a
good
decision what could they do so that's
why people were poo pooing
rockets while godard was actually a
launching room
do you think that this because one one
paradigm i often get
stuck thinking in is this distinction
between
concrete problems and their solutions
and then processes that solve problems
and it always strikes me that
it's easy in our mundane day-to-day
existence to
you know encounter a particular
challenge like like
you know space travel and look at it and
say wow yeah just like you said that's
really hard
i can't imagine personally solving that
and that's really what we're saying in a
certain sense
it's like it's the same thing that
causes us to say well if i ran that
country it would run really well we sort
of
imagine that we can imprint our will on
the problem that somehow reflects what
would happen if humanity collectively
worked on it
rather than saying we are some sort of
collective structure a super organism of
some kind and
through a mix of weird market forces
interpersonal interactions and all this
stuff
we are kind of a machine learning
algorithm
collectively working on solving this
problem is that a is i mean is that an
accurate frame do you think
i think so and it also depends a bit on
the
kind of problem you're trying to solve
so if you have a top-down approach you
need a genius of management to do it
and we have seen two good examples about
the apollo project and the manhattan
project
in both cases there was a fairly
well-defined goal
and the underlying physics was mostly
understood
but not completely by any chance um and
then
you had people working both on the heart
science and engineering
but you also happen to have a few
management geniuses running the whole
project
general leslie gross was probably one of
the best people
in the entire 20th century at managing
people and getting stuff done no matter
what
but you also have vast projects like the
internet that grew organically
full of internal contradictions and
messiness one little
micro hobby i have is collecting
documents as
predictive imminent demise of the
internet uh there
are many of them like one from 1981 can
point out
if this trend continues by september
this is
not going to work this was written in
july
and of course the solution one was
implemented and the
september rolled around no problem there
are a lot of problems with the internet
the people be patching them like crazy
here you have a lot of bottom up
solutions not all of them are perfect
we could have avoided spam if we had
implemented mail systems differently
in the early 70s but nobody could
imagine
that email would be used outside the
computer department
definitely not by millions of people
including people
who are rather selfish and nobody
thought of inviting an economist who
would point out look
if the marginal cost of sending an extra
email is zero you're going to get the
infinite emails
right yeah and it does seem as though
to some degree like this does make me
think of the kinds of problems we have
in terms of predicting how an ai is
going to solve a problem
you know if i look at a computer vision
model i'm not going to be able to guess
ahead of time in general
what kinds of features that algorithm is
going to be looking for in images to
classify
airplanes away from ducks and submarines
and stuff like that
um in much the same way it sort of seems
like we're con we're constantly taken by
surprise by the ways in which
we as a collective end up coming up with
these these
crazy solutions one thing maybe this
ties into some of your work on deep time
and the fermi paradox one thing this
makes me wonder
is you know is the is the collection of
atoms
in the universe really all just the um
is it all just the parameters of some
grand optimization algorithm that are
being jumbled around over time
and i mean i'd love to see interact with
that idea first of all maybe we can dive
into the fermi stuff in a minute
in some sense i think it's totally true
but yeah we're doing a giant
optimization we're
minimizing free energy some gibbs free
energy or helmholtz free energy or
whatever
i can never keep them apart so in some
sense
oh yes atoms and the particles are
trying to get to an energy minimum
constrained by also trying to minimize
entropy
and that's already where things start
getting really weird and interesting
because the universe started out with a
pretty flat space time for some reason
but of course a very high temperature
and a lot of jumbled atoms
and what happens when space-time expands
and temperatures go down
is that now the atoms start binding
together in various non-trivial patterns
and because we can clump because of
gravity you get a lot of very
non-trivial patterns including some
patterns
that start fusion and start generating
energy so now you get
energy flows and things get even more
complicated
but in the super large you could argue
that the history of the universe is
basically that
we're moving a lot of entropy into the
gravitational field of the universe by
clumping matter and that is
powering a lot of non-trivial
non-equilibrium processes
that are having very low entropy many of
them then
turn out to be optimizing for other
things
there is this big field of
non-equilibrium thermodynamics
which i don't understand that well uh
but it seems like
in many cases like if you have a flame
that is kind of continually fed the gas
it will tend to maximize or minimize
entropy production depending on the
construct
you get a lot of these weird
optimizations starting to happen
all around the place so for beings of
molecular matter like us for example
crystals
really feel weird because they're so
different from most of other rocks we
find in nature
because they're kind of trying to
minimize the lattice energy and surface
energy and turn into this very
exact precise things that are very
different from the normal rocks
which are of course also full of
crystals of a different kind
and we are of course powered by what's
really called aperiodic
information crystals he postulated
before we actually knew what the genetic
code really was
but there must be some kind of molecule
that could
in a regular way contain information to
build the organism
they didn't know what kind of molecule
it was speculate a bit he was wrong
about most of that but
he was right in describing the dna as a
kind of a periodic crystal
and the cool thing about evolution is
again you have an optimization process
you try to maximize your fitness or at
least your genes
they're hoping and so far we can hope to
have a lot of offspring genes
so organisms that are very successful in
that they spread their genes around and
now you get something that's optimized
for its ecological niche
it's a local optimization many of these
niches turn out to be transitory or just
plain stupid
or just get wiped out by bad luck when
an asteroid strikes
but the end result is that a lot of
information
non-trivial information from the
environment have been converted into
genetics our bodies are full of
adaptations for handling
an earth environment and that has
happened over
literally billions of generations where
cells and organisms have learned a lot
of things usually using hard lessons
and now it's encoded in our genes
similarly of course some of those genes
are called brains that are doing
essentially the same trick but much
faster
and now we even got cumulative culture
so we're doing it on an even faster
scale
and those levels of abstraction i mean
it kind of seems like they keep piling
up
and to a certain extent i mean it makes
me wonder because
evolutionary biology is always framed in
this paradigm
where we say what's you know what's
being optimized for well a species
genetic fitness their ability to
propagate their their genes
essentially something like to propagate
genetic information through time
um but then sometimes it's framed as in
in
it's framed from the perspective that
you're trying to optimize the number of
individuals or the number of copies of
these genes
it's always somewhat unclear what's
actually supposed to be optimized for
and it sort of seems as we edge closer
and closer to
what a lot of people think is this
technological singularity that that's
going to break a lot of these
assumptions as well because
presumably intelligence is not genetic
it's not contained in genes
so whatever the universe is optimizing
for it doesn't necessarily just seem to
be genetic fitness it seems to be
something but i have no idea what it is
do you have any any thoughts on what
that might be
so if if you could talk to evolution
evolution would say oh yes humans are
really really good and successful as a
species
just look at them a large mammal of that
size that common in all
parts of the world yeah really good
except of course a lot of humans are
really bad at reproducing
i mean we we why are we making a
contraceptives
why aren't all men sperm donors and so
on
in terms of inclusive fitness that's
what you ought to do
instead you get people who get religious
ideas and decide i'm going to
live a life in celibacy in this
monastery and think
sacred thoughts we come up with a lot of
things that are more fun
than the rear end kids maybe one day we
will really upload ourselves
into software that's kind of really a
bad idea from
the sense of biological evolution but
this is what happens
all the time because biological
evolution
it creates various things in order to
try to optimize fitness
but it doesn't care about what those
things do
besides that so sex for example is a
good way of
a long-term increasing fitness and
evolvability because you can share
useful genes and then of course you need
to have a motivation system so animals
actually start having sex
so suddenly you get a lot more pleasure
and fun into the world
which is from evolution stand for just
instrumental but
from the perspective value oh this is
kind of a great thing
brains well they're really to coordinate
motor action
avoid getting eaten but you can use them
to imagine things and do
a lot of more things it seems like deep
down the universe might just be doing a
free energy minimization
but then that leads to this super
non-trivial effect
so when you play around with artificial
life simulation a cellular automata
quite often you get wonderful emergent
phenomena that are
very inspiring in many ways oh i just
put in some simple rules and get a lot
of complexity out
but if you spend enough time with this
simulation you quite often get a bit
bored because you do get
complexity but it's the same kind of
complexity mostly
again and again you after a while you're
going to see those patterns
you get something truly weird you
usually have to design it yourself kind
of put it in from outside
many of artificial light simulations
that they had in the 90s
were found and you've got small
ecosystems but then they never became
more complex
which is very different from our own
ecosystems and our
own societies they do seem to have a
tend to become more complex it might be
that we're missing something very
fundamental about reality or evolution
or it might just be that you need a big
enough world to have it happen
a little bit like the neural network
revolution in the 2000s
demonstrated that up until this point we
have been using too little data
too small computers and too little
training when you scale this up a few
orders of magnitude
really amazing new things happen as we
couldn't even imagine even like unite
this
and it is amazing how just how different
the world is from what you might imagine
it being
prior to the takeover of some of these
strange effects uh sexual selection
or sorry sexual reproduction um
biological evolution all these things
it sort of highlights how well how
strange these processes are and raises
of course the question of how
common uh they must be at the universal
level i think this might tie into some
existential questions about risk right
because
people often talk about uh you know why
why do we look up the night sky and we
don't see
any other alien civilizations there does
that mean something about a great filter
or something that might uh
might be ahead of us still um you've
done a lot of work on this topic
i would love to just prime you with the
question what do you think
of the furry fermi paradox can you
introduce it and and just describe it a
little bit and then see where you take
things from there
yeah so the fermi paradox is not really
a paradox and some people would point
out that it's not even fermis but
i like to call it fermi's question so
back in the 1950s the people at
the manhattan project were having lunch
and talking about
uh atomic rockets and how easy it would
be to settle the universe now when the
power of the atom had been unlocked
and then fermi apparently just asked so
where is everybody
and that was a very good question
because if it's easy to
go across the gulfs of space and settle
the entire universe
we ought to be seeing a lot of examples
of intelligence because the universe is
really big
and really old and it doesn't take that
long
if you can spread between the stars
before you're showing up
everywhere so that empty sky became
a real problem because if you're
somewhat optimistic about technology
this seems to create attention and
that's why people say
it's a paradox we assume that there's a
lot of sites and times where
intelligence can emerge you multiply
that with some
reasonable probability of intelligence
emerging
and then you should get a number and if
you're somewhat optimistic you get a
large number
and that doesn't seem to fit now you
could argue that maybe
there are very few places in the
universe where intelligence and life
could evolve so there are some people
saying that earth is very unique
but it's hard to make it super unique so
unique that
you can safely assume that there is no
life
everywhere else so there has to be
something else and of course
some of these factors somewhere in this
equation
that you multiply various factors
together there has to be one factor
that's small enough to make the universe
pretty empty
so that's a great filter factor it could
be that life is super rare
in that case well we're lucky we exist
and
now we have a big future ahead of us or
it could be that intelligence is rare
or it could be that italians is common
but it doesn't survive
very long and that's of course the kind
of scary great filter that got us
working at the future humanity institute
thinking about these questions
because that seems to be one of the few
pieces of really independent
information about our chances about
global risks
come from reading the newspaper thinking
about what are the
latest news about biotechnology and
pandemics they're trying to understand
the
other issues but here you have something
that seems to be an average across all
possible civilizations
now the really interesting thing here is
of course if we're living in a universe
that has a tendency towards complexity
this gets even worse if you think that
the universe is pretty
neutral or inimical to life okay fine
uh it's pretty empty if you think the
universe is really
trying to get life and intelligence you
have a bigger problem
it's also worth noticing that in the
past many people were
absolutely convinced that every
environment had its own inhabitants
the idea that there were people living
on other planets was almost self-evident
to a lot of people both in antiquity
and very modern era really actually
said that it would be kind of crazy for
god to create these planets and not put
people on them
that incidentally also means that you
don't need to care too much about the
end of humanity
so my friend thomas moynihan has written
an excellent book x-risk
about the history of thinking about the
extinction of humanity and points
he points out that up until fairly
recently
people were not taking it terribly
seriously because if we go extinct well
somebody else is going to show up that's
the way the universe is
but if you think we're almost alone or
entirely alone
if our spark goes out there's just
darkness
that makes existential threats much
worse so this is another reason we
really want to understand the fermi
question
and okay yeah i i i totally
i i totally understand that fixation on
the the fermi question as you put it
i agree i mean the fermi paradox framing
always seemed
it's not that it seemed naive but it's
um it seems as though it reflects a
certain bias
towards thinking that when you look at
i guess the famous drake equation that
sort of lays out all of the factors
that you multiply together to get excuse
me the number of
potential alien civilizations out there
there's
and you've pointed this out in your work
there seems to be
a consistent um reflection of the
author's bias so if you're analyzing
this equation
you can come up with almost any answer
you want depending on how you tune those
factors
um would you mind explaining
unconsciously
it's very easy to just put in what you
think is reasonable
and look you get a reasonable answer if
i
get what i think is an unreasonable
answer quite often i will go back and
more or less consciously fudge things so
i get what i think is a reason
and this is of course quite dangerous
for thinking critically about it
and actually to that point that implies
a huge amount of uncertainty
in the prediction itself are are we
doing a good job
uh academically or research-wise it's
studying that uncertainty and accounting
for it
well i have a paper where i'm arguing
that we have been doing a bad job at
this
because typically what i would say
people do is
they line up factors for drake equation
they admit that
this one we don't know we actually have
no clue on how
likely life is on a terrestrial plan but
let's just say
one chance in a million and let's admit
i'm making up this number
and then they multiply everything
together and they get a number
and they admit of course that yeah this
is super uncertain because i made up
some of those numbers
and then they leave it at that because
to them
admitting that you're uncertain about
something that's how to handle
uncertainty
but this is of course not rational if
i'm uncertain about how many people live
in san francisco
i should be stating a range that i think
i'm 90
or 95 certain it's inside i can't just
say
okay i think it's five million people
and then not mention how wide the
range could be around that now the
interesting part
is that if you actually take the drake
equation and try to put in proper ranges
or even better if you have even a
probability distributions you put in
on your level of knowledge and
uncertainty you will get
a range of answers and then it turns out
that
actually you get a pretty big spread
given the current state of knowledge
so we know fairly well the number of
stars in the milky way and the rate they
format
we have a decent idea that okay
terrestrial planets are a dime a dozen
there are a lot of them
we have more or less a hundred orders of
magnitude of uncertainty about
how likely life is to emerge on a planet
it could be that it happens
within 10 minutes of the first puddle
showing up
it could be that it happens through a
more or less a
thermodynamic miracle in once every 10
to the power of 100 planets or something
like that
we honestly don't know so when you put
it in you get a very broad
uncertainty distribution and even if
you're pretty optimistic on average
so you think that on average it should
be maybe 10 civilizations say in the
milky way
you get a lot of probability going down
into the tail that we're actually real
alone
in the observable universe if you do
that
suddenly i am this guy isn't that
problematic
i might still have some hope that i'm
right that we have some space brothers
out there
but this guy is not terribly surprising
if i'm instead just putting in numbers
then i'm going to end up to say oh we
should be 100 civilizations here
why or have we seen yeah
no it's it's a it's interesting how
those those
especially because you're multiplying i
mean you're multiplying all these
probabilities together right so the
you it would it be a good nut shelling
of this argument to say
you know you only have to fall on the
very very pessimistic
end of the range for one of these
parameters to really
destroy any hope or reduce significantly
at least the probability that we have
any
interest left interplanetary neighbors
yeah
exactly and it's important to realize
that many other professional astronauts
will say yeah but
people aren't that stupid when they use
a drake equation and then
of course i immediately start waving a
bunch of published peer-reviewed papers
around that
does more or less the same thing and
many people are equally overconfident
about
what is super unlikely so there are some
people who think that life is super
unlikely
we actually have to change a little bit
in our text not not to give any help the
creationists have
claimed that obviously only god can add
life to our planet
but it's not impossible that it might
take a very
unlikely set of events leading to that
complexity
one thing that i didn't think about
before writing that paper was
there it might be it was possible that a
lot of life ends up with a genetic
coding system
that is rather crappy it
allows them to reproduce but the
evolution is
really slow so they never get the chance
of turning into anything interesting
until the star burns up
now our kind of life only took a few
billion years to go from
kind of primordial goo to people writing
papers about primordial goo
but it might be that most life of
universe actually
stay primordial goo until it dries out
and dies
so that was something i didn't think
about as a hard point but i realized ooh
that's actually a possibility
it's kind of a disturbing one yeah and
actually
so it's funny you mentioned the the
creationist angle there and there's
something that
um i mean i come from i come from the
world
back in the day of quantum mechanics and
i did my work in a field called
interpretations of quantum mechanics and
we talked a lot about multiverses
in in that context one of the things
that i always found
to be a compelling argument for the
multiverse is the idea that if it
genuinely does
does appear that we are alone in the
universe
what a fantastically suspicious
situation that would be
um it would mean if really if the
universe if the observable universe is
all that actually exists
and there is exactly one um one
exemplar of intelligent life then that
implies
that the probability of life evolving on
let's say a given planet
was tuned to almost exactly one in
however many planets there are in the
universe
that is an incredibly it's not 10 times
bigger than that or we would have 10
interstellar neighbors and it's not a
billion times smaller than that it is
exactly or very roughly exactly
uh that order of magnitude um and and
then i mean that's i guess where
where you might point to some of these
uh religious narratives that that might
be an alternative
explanation of this sort of thing is
that do you find that to be compelling
as an argument for
there being a much vaster universe
beyond the observable universe maybe a
multiverse something like that
yeah i think multiverse fear is
many people recoil from them because it
seems like oh wait a minute isn't
science supposed to be
dealing with testable stuff this sounds
very untestable
but they seem to be a fairly robust
prediction of a lot of different
theories
so it's not just that you can claim that
the quantum mechanics
leads naturally to a multiverse theory
that's also my
own point of view but then again one can
spend
all day and night arguing about the
interpretations of quantum mechanics
but there is also no good reason to
think of observable universes all there
is
to the universe indeed the space time
seem to be flat
which means that the simplest answer is
oh yes it's
infinitely large you can say well it's
got some kind of closed topology
but we have seen no evidence for that
then you need to add extra complications
to get that work and then of course you
have the inflation theory is saying that
well actually there might be other
domains
um and so on and so
you get multiverse theories propping up
almost all over the place and this means
that it's kind of easy to explain almost
everything
because uh somewhere this is bound to
happen
the real question is of course why the
world is not weirder
why are we finding ourselves at a fairly
normal looking planet around the g star
even that might be a bit weird because
after all yellow dwarf stars yeah
they're not super common compared to the
red dwarf stars which are
everywhere and are also going to be
shining much longer so
it's kind of slightly odd that we're
relatively early in the stelliferous
epoch of the universe
rather than somewhere in the middle
orbiting a little dim
red dwarf star and the in another paper
i argue that
that might actually be a hint but it's
not that habitable around these dwarf
stars
normally i'm kind of optimistic about
the habitability about those planets but
maybe those flares really do
erode the atmosphere or continental
drift stops early enough
that um they lose uh the carbon cycle
and then the climate goes haywire so
actually
most life tends to show up early in this
deliberation
that's actually in a in a way kind of
counter-intuitive
um actually deeply counterintuitive and
it also highlights i suppose the amount
of uncertainty there is too i mean if if
you're otherwise
fairly bullish about these um was it
white dwarf stars sorry or
red the dwarf stars sorry red dwarf
stars so if you're yeah if you're
otherwise really bullish about them for
a variety of other reasons
and yet here we are orbiting some some
different
different star i mean what is the um
what's the difference in relative
frequency between red dwarf stars
which you know ought to be the star that
we're orbiting if
if they indeed are more or are equally
habitable and in our own sun
or stars like that so so basically you
have i would say
about 30 to 50 red dwarf stars
per yellow star and when you go down to
the smallest ones the smallest and
dimmest ones
uh the numbers skyrocket they're really
ubiquitous
and they also last really long the sun
is only
going to be lasting for five more
billion years
then it's going to become a red giant
and shed its outer layers and become a
rather boring
white dwarf star very sad for people
living in the solar system at the time
but meanwhile many of the red dwarf
stars around us
they're just happily going to keep on
going for literally a trillion years
barnum star which is a few light years
away right now
it's still going to shine when not only
the sun has turned into white dwarf but
also
cooled off to become a rather boring
almost black dwarf
it's still going to be shining it's
going to be fusing practically all its
hydrogen
if there is a planet around it it's
going to probably have the same
temperature
in that future as it has right now so if
you add up
something like i'm sure this is a bad
term for various reasons but
if you add up the the sum of all the
possible life-sustaining years of like
red dwarf stars in the universe it
should it should be way bigger than the
sum of life-sustaining years of
of our sun or stars like our sun it
should totally dominate
i can't remember what the exact number
is i calculated a while ago but
there is a lot more biosphere years
around red dwarf stars than yellow
dwarf stars biosphere years okay yeah
good good to know
there's a better term for it that's
great fascinating okay so
there are all kinds of open questions
about how we got here then clearly
the range of uncertainties is just
incredible it's it's fascinating that we
can even
reason about this and frankly i've been
surprised reading your work at how much
information you've been able to extract
from our our apparent situation just
from being here
looking in the more forward direction as
the evolution of the universe continues
obviously one of the big things on the
horizon the biggest phase shift that
we're
that we have to look forward to here is
potentially something like the emergence
of
advanced artificial intelligence
artificial general intelligence super
intelligence and all that entails
um you've done some work comparing
uh super sorry artificial intelligence
in terms of its
energy consumption and the the human
brain
and looking at arguments people have
made that suggest that
you know maybe maybe artificial
intelligence will take longer to develop
because
for well for energetic reasons you've
argued that's that's probably a bad
argument i'd love to hear you unpack
that sort of whole body of thinking so
right now
if you want to run a big machine
learning model you're going to be
spinning up your data center
and your electricity bill is going to
run into the kilowatt
or maybe even megawatt range it's kind
of not cheap but
meanwhile of course even the most
brilliant human
brain is running between 20 and 25 watts
of power
that's a fairly dim light bump well
that's a fairly
incandescent light bulb if you use leds
it's actually fairly bright but
it's still not that much energy and
that's really weird because
neurons in the brain work by kind of
rube goldberg mechanism
for transmitting information basically
of ion pumps that separate the potassium
and sodium
iron ions on the different sides of the
cell membrane
when a signal comes channels open they
flow through it creates an electric
potential that opens other channels
and you get a little wave spreading
electromechanically at about the speed
of sound
it's kind of silly but it works pretty
well
now the interesting part here is that
you could probably be more energy
efficient if you could do normal
electronics on that scale
but still brains are way more energy
efficient than current computers
so then there are some people trying to
use this as an argument saying look
in order to get through ai you
need enormous amount of energy and
obviously you can't do that
now the problem here is are we comparing
apples and oranges
and i think that's what's going on
because when an
infant starts learning language it's not
exactly doing the same thing as when we
train a big language model in the data
center
the infant here is a fair bit of talking
on the
in the room it's in it's hearing things
from radio and television
it's surrounded by language but total
amount of words and infant years
in one or two years that's not
astronomical i don't know
exactly how many million words they get
but it's not enormous
now compare that to modern language
models that
do really well you you basically feed
them all of wikipedia and reddit and the
big chunk of the internet and project
gutenberg and uh
all the translated united nations takes
essentially as much text as you can get
your hands on
it's an enormous vast amount yet they
also learn it after about
a week of training in the data center
and when you look at the processes going
on they seem to be also very different
the machine learning process uses
stochastic gradient descent while what's
going on in the infant's brain seems to
be more like heavier learning
so my argument is basically that this
doesn't work as a comparison we can't
use the
energy the use of current computers to
say anything about when they can reach
human intelligence in particular because
improvements of
algorithms can quite often mean that we
kind of fake
an extra decade or more slow if you look
at performance of chess computers
you saw have seen in the past
occasionally that the ratings
just jumped up by a significant amount
when somebody came up with a better way
of dissolving the problem
so this means that actually reducing
energy as a way of estimating bounds of
ai
doesn't work now there is a flip side to
this i do think
energy intelligence and information do
matter and we can use it to bound what
civilizations can be up to
because it does in some sense is the
information processing
i like to point out that even when
falling in love is in
at least in part an information
processing operation maybe the key part
of love is not that all
uh the information maybe there is kind
of ineffable qualia
that or actually what really matters but
you better remember who you're in love
with
that's information storage you better do
something about it that's kind of
information that needs to go to muscles
and so you say something
so the interesting part here is that we
can use the physics of
information and energy to actually say a
little bit about
advanced civilizations and what they can
and cannot do
we can look at the energy source and say
yeah there is not enough
energy in the universe to actually
perform that kind of competition
this gives us some ways of thinking
about the extremely advanced
civilization that might exist in the
extreme before future
but it doesn't tell us very much about
whether we get ai
in 10 years or 100 years and on that
on that forecasting side of things
obviously any forecasts that have to do
with ai and when we're going to get
transformative ai uh come with huge
error bars
do you have do you have a personal um
inclination like what would what would
your personal error bars look like
on let's say a uh an 80
or 90 confidence interval for the
emergence of
ai that can um let's say design machine
learning systems better than human
beings so i think that's
probably for various reasons a good
objective benchmark
so my eight maybe eighty percent
confidence interval would probably
start later than most of my colleagues i
think i i would be
rather surprised if we saw ai doing good
machine learning design uh before the uh
2030 or maybe even 2040
and my the end point of that round might
be run into the
the 2100s uh i haven't quite
yeah so i have a very broad estimate and
i think
uh if somebody according to me about it
i would probably hedge a move
make it even broader now of course from
a safety standpoint
i like to point out that yeah but we
better work as if it's
going to happen very shortly because we
better have done our homework on making
safe
and value-aligned ai before it arrives
and
even if it arrives in 10 15 20 years
that's actually around a short time to
solve some very fundamental
and the problems and if it takes a
century
well it might still turn out that it's
hard problem to solve
after all philosophy i've been trying to
solve the value alignment problem for
humans in 2500 years
with modest results actually that i
think that's a really interesting area
to dive into
what if any do you think the difference
the differences are between the value
line
problem the philosophers have been
working away at for the last as you say
2 500 years or whatever it is and
the value alignment problem that we have
to sort out
with our ai systems in order to make
sure that
they reflect they do what we want and
that they actually
are doing that we know what we want them
to do as well which is sort of the two
parts of this
so the interesting thing about
philosophy and thinking about
ethics is that for a long time it was
just assuming that the only minds that
we need to care about are human-like
minds
there are a few interesting discourses
in the
middle ages about ethics for angels but
generally it's not regarded as much of a
problem because we know that ages behave
themselves anyway
the question is whether they behave
ourselves or out of free will or just
because we're programmed to do it but
most of our assumption was always yeah
but
everybody thinks a bit alike and
that assumption about a human-like mind
has been quite profound and
it's a bit problematic because human
minds are of a special ones
it has been an interesting discovery in
animal studies but yes it seems that
chimps
actually have a sense of fairness they
get very upset when they see another
chimp
being overpaid for doing some work in
the lab
it's not just envy it's also that they
realize wait a minute why did
i just get one banana when that chip got
to bananas
so you can see that there are some
elements of what we might say in moral
feelings
i don't think chimps actually are
thinking about ethics or fairness or
anything like that
but we certainly need to have these
prerequisite feelings
that allow us to work in a social group
and a lot of that then gets refined
because we got big brains
and start thinking abstractly about it
into general principles
but underlying that is a particular
design of
reward functions in brains we have a
motivation system of particular kind
and up until rather recently
philosophers
just assumed that it was all like that
so
and i think the cool thing that is
happening right now is that
artificial intelligence is introducing
an another
into ethics that is rather challenging
certainly some ephesus have been
thinking about animal rights and animal
suffering
and those issues but animals were never
moral agents that you needed to care
about
about if you try to teach ethics to your
cat it's not going to work very well
but you could in theory try to do that
with the ai and people are starting to
realize that
these systems can be very very different
from
humans indeed going from an
anthropocentric to a non-anthropocentric
ethics
i think is the great challenge not just
for making safe and the ethical ai but
also for philosophy
and i think it's also very healthy for
both fields to talk to each other
so yeah generally going beyond the human
is quite helpful yeah it's um
it's a really interesting question that
as to whether and to what extent we do
count
ai systems as being these these agents
that are
that need to be factored in where we
have to do the accounting in a way that
mines their needs and wants and desires
and it also sort of
i guess it's you can't separate it out
from questions around
consciousness and subjective experience
because if these are really just
black boxes that don't have any feelings
no matter what no matter how real they
might seem or no matter how emotive they
might seem
if they really are emotionless deep down
inside because i don't know because
machines don't have feelings then then
we might be wasting our time and energy
optimizing for their preferences
um do you have any thoughts about how we
might explore that i mean that seems
like
it's obviously linked to the hard
problem of consciousness this is not an
easy thing to answer but
what are your thoughts on that and i'm a
lousy philosopher of mine i have no idea
how to really resolve that
but i think sometimes consciousness is
beside the point
so when nick bostrom's book called
superintendents came out
uh sir the great philosopher mine
wrote a somewhat sketchy review saying
look machines are not conscious so there
is no
ethics problem which is kind of weird
because a car can round you over without
being conscious
unconscious machine can be quite
dangerous and nick's point was of course
very much
yeah we better make safe machines
whether they are conscious about
that they're safe or not conscious is
beside the point
now we might want to design machines
that are moral patients that we need to
care about
and we might also have very good reasons
not to want to do it joanna bryson
wrote the paper with the great title
machines should be our slaves or maybe
it's robots should be our slaves
she's put the title is of course pushing
things a little bit further on paper
but she's got a great point for many
purposes you don't want something that
you have to care about and she thinks
that it would be a rather stupid step to
make a lot of machines that we need to
care about
i also think that if you can make a
machine that we would be forced to care
about somebody's
bound to do it if only as an art project
so the real question is can we tell
systems apart
that we need to care about a note and i
think that's going to turn out to be
really tricky
because normally we tend to use
intuitions like okay i talk to it
and it gives sensible responses
and okay i believe that there's somebody
around there
but we know that you must always fall
for a chat but
it's kind of embarrassing how easy it is
to fool you must be thinking that there
is somebody there
because we're fine-tuned to assume that
they're safe and sore if something seems
to be having a mind we should probably
assume
it's a human-like mind which is also why
we're projecting human-like minds on
animals
and natural phenomena and probably
building up religions to explain
who got so angry that that's thunderbolt
to hit that tree
you immediately fill in some suitable
agents for that
the problem is of course there might be
systems that are not agents at all
and where this metaphor really fails
when you think about something like the
google search engine
is that a being nah not really
uh the borders of the search engine are
very fussy
and indeed it's not even functioning in
the way a human mind would
we might end up in a world with a lot of
very important powerful systems that do
a lot of clever things
that are so dissimilar from us that we
need new moral categories
it might be that it's a bad thing to
delete a really good search agent
but not because it's bad for the search
agent but
it might be like a piece of art or it
might be that there are other values
or goals that matter there are some
ideas in animal
ethics for example that animals have
life projects and we shouldn't be
interfering with them
and you can imagine robots having
projects even though they may be very
unconscious
normally ephesus would say yeah
unconscious things don't really have
any immoral rights but if you go into
environmental ethics you will find the
biocentrists and the ecoceptist
saying that actually ecosystems might
have value and
if you go into terraforming efforts you
will find a little list
a few philosophers say oh that unliving
planetary environment has a value in its
own
and it would not be helped if we
introduced life
so they might have been very willing to
say that yeah maybe some of these robots
should have some form of right or we
should respect them even though they're
still just juggling around numbers
uh bits without having any internal
experience
this is of course very far outside how
we normally deal with it
and how we should arrange our own
affairs as a society and
people relating to things around us it's
going to be quite challenging
this makes me think of the part of the
conversation earlier where
we're talking about human superorganisms
or or let's say the collection of all
human beings on the planet as being
one coherent organism um you know
reviewed through that lens do we have a
moral responsibility to whatever that
super organism is i mean we basically
act as its
cells and who's to say it doesn't have a
legitimate conscious subjective
experience
certainly maybe if you zoomed out like
crazy you could look at planet earth and
see it evolve over time and go oh that's
a
a planet that's gradually waking up
something is happening
this this whole ecosystem deserves some
sort of
moral standing independent of the
individual entities
in that system i guess the i guess the
problem there too is like
you get into combinatorics because you
could easily say well is is canada its
own
uh you know entity with with moral thing
anyway
do you have any thinking on that sort of
um almost
inverse reductionist uh position
yeah so eric schweitzgable a very fun
gadfly philosophy he wrote a very fun
paper
about what does it take to claim that
the united states is conscious
and he argues that yeah under some
fairly common sense assumptions
uh it's not terribly hard to lead the
reader to conclude that yeah maybe the
united states is conscious
and presumably then canada would be
conscious but what about the united
nations
or indeed the world economic system
that combinatorics is not necessarily
that crazy after all
we normally have two brain hemispheres
and they
have a limited bandwidth between them
and different modules in our brain are
actually not
fully aware of the information in other
modules
sometimes you can get these weird
disjointed experiences
i remember coming home one evening and
noticing that
that coat and hat hanging on a rack
looked really like a sinister
character lurking and then i jumped
because another part of my brain i
noticed as soon as the character
lurking in the darkness the different
parts of my brain had
reached different conclusions about the
same time
and where i were kind of conscious of
both now
it could very well be that the same
thing happens with the super organisms
in some sense google can be a conscious
progress but also part of the united
states of america
and the world economy they're also
separated
to some extent by the limited
information flows between them
but you can sometimes say that it makes
sense to treat uh
in this situation that part is fairly
separate
just like we might say right now we
humans tend to be fairly separate from
each other but you can also talk about
what particular groups are doing
you can say that the science community
has decided that
the following statement seems
tentatively true about the world
but there is a lot of scientists inside
that community that have been haven't
even heard the news
it's fascinating how consciousness might
turn out to be a fractal problem in that
way and it sort of
it sort of lends itself to thinking
about about
ultimately pansysm i mean if every if
every
neuron in my brain is conscious every
cell in my body is independently
conscious
where does that end is every organelle
within every cell conscious and can i
keep going all the way down until
every quark and lepton is conscious i
mean every every particle
and then we might have to worry about uh
what about unhappy leptons and quarks
maybe they are the true moral problem in
this universe
animal and human suffering ah that's
nothing we
really need to help the poor up corpse
who knows they'll have their own lobby
now actually there is an important thing
here
consciousness might be a simple thing
and
it's not impossible that you could
stretch it out
to encompass everything then we happen
to be the kind of objects that also
write papers and talk
and think about being conscious which
probably isn't true for the rocks
but there are other properties our minds
have that are non-trivial and probably
don't carry
over when you go arbitrage for them
after all we're both speaking english
but if i were to
examine your brain i'm not going to find
an english-speaking neuron
it's a system that has a certain
property and this is of course my
response to searle's chinese roommate
fourth experiment i think it's totally
true that some systems have properties
that you don't find in the parks
and the wetness of water happens when
you have enough water molecules together
it's not in exactly part of the water
molecule
so many morally relevant parts of our
minds
might exist on some levels and not
others and this might of course be
interesting when you go upwards to the
super organism
maybe the united states certainly speaks
english in a sense
but could it be that we can speak about
virtues of nations or civilizations
we can certainly talk about virtues of
people i can't speak about
virtues of a body part or a neuron but
it makes sense to say that
that guy is a tenacious one that one is
brave that's a cowardly thing
and maybe we can say that's actually a
brave civilization or
uh that might actually make sense there
might even be virtues that exist on a
civilizational scale
that doesn't exist on the human
individual scale
is there um because one of the things
that that i find
confusing or sort of difficult to
reconcile i think intuitively
intellectually i think it makes sense
but intuitively i still struggle with it
is just the idea that you know if um
if the su if human superorganism
genuinely is conscious
i mean its behavior is fully determined
ever like it seems like
anything that the the human
superorganism does is just a
function or product of the behavior of
all the individual humans that make it
up
each of whom feels like they have free
will um so it kind of feels like there's
no room for that super organism to make
different decisions than those
that it does make which sort of seems to
imply that oh it can't really be a
conscious free-willed entity but then
again i guess the same is true of us
with ourselves
it's not like we have that degree of
freedom either i think the solution here
is that people
are assuming too much about free will
we want it to be kind of er a form of
freedom that i don't think is compatible
with physics
or even logic really but what's really
going on is of course
free will is a useful description of
what most people do
if i'm committing a crime in many cases
okay i've made a decision and it's based
on what i knew and felt at that time
which might have been a bad idea maybe i
can even
demonstrate the neural firing that led
to me committing the crime
but that doesn't actually work as a good
explanation
of why i did it free will is a very
useful thing on the human
human level it might be less useful to
ascribe to large groups
and we say that the democratic party has
free will
well to some extent yes decisions are
being made on the party level
and we might ascribe to individual
people but you could also say that
actually it's a bit of an emergent
phenomenon because the people are
talking to each other and sometimes
there is a group decision but nobody is
really supporting
but there is still a decision so i think
it's important to
try to see at what level does the
explanation work
so normally when we talk about free will
it's about predictability
when you say that somebody is being
robotic that means that you have a
fairly good model of how an input
generates an
output it's of course fairly trivial to
make a simple program
that's extremely hard to predict but
most of the time
we don't think a random behavior is
really interesting we think
appropriate behavior that is hard to
predict
that is kind of what we ascribe free
will to but we get that for
a lot of systems it's just that in most
cases we don't
interact directly with biosphere or
civilization as a whole
we deal with people or organizations but
their
free will is not at all impossible we
might say well
facebook to some extent has free will in
setting up privacy policies
you might say well it depends on what
zuckerberg says but actually there are
shareholders
there are internal structures so it's a
bit more complicated than that there are
other organizations that are maybe even
more free in
deciding what to do yeah the question is
of course how much we can blame
them for what actions we take or don't
take
right which i guess is its own kind of
separate moral category and
and something that we'll maybe have to
figure out i mean do you think that
these things are
are problems that these questions are
questions that we'll need to
answer in order to be able to safely
navigate the technological transition
that's coming or can we get by
with say a subset of them or what are
your thoughts on that
i think we will probably have to make do
with a subset and
that might be very scary because i think
it would be great if we could just solve
all these deep questions
but some of them might just be
irreducibly complex it might be that
some things
don't have a general answer but i think
good heuristics can really help you
quite a long
bit if you think about the division of
power in a
government for example that balancing
act is
really useful for avoiding certain
failure modes we learned that in various
ways
and if you think about setting up
organizations so they don't
have too much infighting there are
various tools for doing that
setting up rules for responsibility that
make people behave themselves
we have a palette of different options
here so if you create
new institutions you actually can think
about
okay what ways can we this fail and can
we build in safeguards
and we have invented them over time it's
fascinating to go back and look at the
history
of various political institutions
because in many cases just like
looking into software and looking at
cryptographic primitives
people had to invent the the committee
people had to invent various forms of
ballots and election mechanisms
at different points in time various bad
approaches were tried and the most of
them have been forgotten
good ones are now in the toolkit that we
can put them in
we probably need to invent way more
because we have both new problems
the things are on a different scale and
that's one reason why the technological
transition is so scary
it's not just that we need to think
about artificial general intelligences
that might be around
but we already have our corporations and
other super organisms
that are already demonstrated that yeah
that's not trivial at all to control
we also found various cool mechanisms to
do it in a distributed manner like
market forces reputations
and even when parents tell the kids how
to behave and not behave and give them
various myths about
what happens to kids that are doing the
wrong thing
yeah that's a form of programming and we
can't borrow some of those ideas
for example when we want to teach our
robots how to
grow up so then of course we will have
to test things out and probably find
good ways to automate the generation of
better tools
so there's a lot of things to work on
here and the cool part is
this is going to kind of blow up the
borders between philosophy
programming economics and a lot of other
fields there is so much
interdisciplinary work that needs to be
done here we can steal the coolest and
best parts
of different disciplines and build them
into entirely new disciplines
yeah it's incredible how much how much
you've had to learn and know and
understand about all these different
fields just to be able to make
um estimates about timelines that have
massive uncertainty associated with them
just to be able to get something that's
sensible
it's just an absolutely fascinating
grand tour of the past and the future
really deep time
i really appreciate your time anders um
actually one thing i will say is to
anybody who's listening if you're
curious about
anders work please do check out his
twitter because it is really good stuff
uh anders are there any other resources
that you recommend people take a look at
well in some sense i would say wikipedia
i think wikipedia is an interesting
thing not just as a
deposit repository of human knowledge
but also demonstration of sometimes we
can get our act together
it's interesting to look at successful
examples where
people actually collectively pull
together information
solve various disputes and the problems
and create something that is worth
a lot if if you had a kind of typical uh
star trek world series episodes where
aliens were to judge humanity
wikipedia would be one of the things i
would point out look this exhibit
we're not that bad the interesting thing
is people have been trying to make
wikipedia like
resources and most have failed and we
can learn from that too
eventually i think we are going to have
some kind of art and science of making
these great shared resources that
actually hold together and give a lot of
benefit
right now we have only been trying to do
this for a few decades
online so we still have no clue on how
to do it regularly
but i think we are going to get better
so that's why i'm
really recommending looking at things
like the internet archive
wikipedia and maybe some of the
reputation systems that seem to work
like on the stack exchanges
these are amazing treasures in their own
not just in the sense that there are
cool questions being answered on stack
exchange
or that the internet archive helps our
collective memory
and they go on but also we can actually
build
entirely new tools for growing in our
collective
system yeah and and hopefully they do
keep growing hopefully they help us keep
this
the super organism aligned um yeah i'm
hoping that in a trillion years
there's going to be a wikipedia entries
on
all of this thanks so much anders really
appreciate it and thanks for your time
thank you it's been so much fun