Science Saturday: Singularity Edition | John Horgan & Eliezer Yudkowsky [Science Saturday]

hello Ellie you there yep I'm there this
is John Morgan on this end I'm an
occasional contributor to the science
Saturday program of vlogging heads TV
and I have a really interesting guest
with me today I hope I get this right
eliezer yudkowsky congratulations you
got me so what I'm hoping we can talk
about today is the singularity which is
this vision of technological
transformation there are many different
versions of it but you are somebody
who's associated with this term you
write about it on your your blog you're
a part of an institute called the
singularity Institute so first of all
why don't you give us a little
background on who you are and how you
became interested in artificial
intelligence and the singularity
I'm eliezer yudkowsky I'm a research
fellow of the singularity Institute
which is a full-time paid position so
this is actually my day job and I also
run a blog called overcoming bias which
is I expect how quite a few of our
viewers will have heard of me if they've
heard of me at all so let's see how did
I become interested in the singularity
even sort of tell me a little bit about
your childhood were you you're a kid who
is really interested in science and
philosophy and things like that
who now there's an open-ended question I
was a bit too bright as a kid a fairly
well known syndrome most of my
intelligent conversations were with
books because the adults weren't
interested in talking to me and the kids
couldn't really keep up well it's a well
known syndrome I'm sure that's why
they're watching the show because
they've got nothing better to do so so
yeah so from a young age interested in
science might you know it doesn't quite
sound extreme enough yes self-identified
as a future scientist maybe from a very
early age would be the way of putting it
were you interested in science fiction
as well as science itself my parents are
both science fiction fans I am what
happens when science fiction fans
married in fact I'm told that my father
originally started hanging around my
mother because of her science fiction
collection I was created by science
fiction my parents are also both
Orthodox Jews but it didn't take and I
suspect that part of the reason that it
didn't take was that at the same time
that I was being dragged off to
synagogue for all these horribly boring
prayers I also had the science fiction
collection which had this completely
different ethnos this whole other
viewpoint on what it meant to be human
you might say not that what it meant to
be human was to serve God but whether
that what it meant to be human was to be
part of the human species growing up and
taking over the galaxy and so on
dissonance there between Orthodox
Judaism and
and sci-fi so how yes first expose to
the concept of the singularity first
explicitly exposed the concept of the
singularity probably at age 16 sadly I
failed to write down the exact hour and
the day but I was wandering around in
Harold Washington library central which
is the big Central Library in Chicago
where I grew up and I happened to pluck
a book off the shelf called
true names and other dangers by Vernor
Vinge II and that was how it all began
now that was pretty recent right you're
only in your late 20s yes this would
have been in 1996 I'm presently
twenty-eight to 5 yeah and so before
you'd read this book by Benji had you
really thought much even about
artificial intelligence and the
possibility that computers were going to
overtake us in intelligence those sorts
of things well that I'd all taken for
granted since I was 11 or 12 or
thereabouts and read a book called great
mumbo chicken and the trans human
condition by Ed regice oh sure great
Vernor Vinge e's contribution was a
quite different concept it was the
notion that that once you had something
that was smarter than human we wouldn't
be able to predict the future past that
point because it would contain things
that were smarter than us and this would
create difference in kind from all of
what we think is ordinary technological
progress in interplanetary travel life
extension medicine even nanotechnology
it's not the same as having an impact on
the minds that are doing the thinking
that creates all this technology so that
that was what I that was the additional
concept I got from Vinci as distinct
from what I had just grown up with
and was this sort of a revelation to you
I mean did have a tremendous and
immediate impact or that it was a more
evolutionary process I guess I'm just a
dry sort of thinker really because as
soon as I uh
Vernor Vinge II said and when we reached
this point our model of the future
breaks down it was just massively
obvious that this was correct and you
know it's not this sort of like
brilliant whole body shock emotional
transformation thing because I that's
not how I experience my epiphanies just
sort of oh that's obviously correct I
guess I know what I'm going to be doing
with the rest of my life
so was it more that this is something
that will happen or that it's something
that that should happen and that we
should devote ourselves to happening
because it's such a wonderful thing well
initially at the age of 16 I would um it
was a will happen because it hadn't
really I I know it's hard to describe
your younger irrational South in terms
that make sense because of course your
thinking doesn't make sense so at that
time I was thinking in terms of yes this
will happen because the whole line of
probabilities where the human species
gets wiped out before it can do anything
interesting with itself I just somehow
wasn't thinking about at that point it
was too unpleasant so so yeah so at that
age it was a will happen and the
variable was when does it happen and
when does it happen was it was a
variable that I could conceive of
affecting at that time so so for me at
the at that age has distinct from now
the main moral imperative was to hurry
up and do this and and so how did you
set about this did this guide your
education I saw a description of you
somewhere as an autodidact does that
mean that you never even went to college
or graduate school or any of the
certain things or highschool I am an
entirely self-taught yes and for and for
that matter I doubt the grade school
really counts because my father was just
teaching me math beyond my age level
during that whole point and so there was
that and then there was a science
classes where I knew more than the
science teachers and then there was all
the horror of Hebrew education classes
how do you support yourself um I left
home at 20 and the reason I left home at
20 was that at that point Brian Atkins
who had founded hyper Mart and sold it
to go to net which was acquired by info
seeker InfoSpace which was bought out by
Paul Allen and so on and so forth during
the dot-com boom said okay let's start
the singularity Institute so I didn't
leave home until 20 Anders and I left
and I'm not quite sure to what degree my
parents were surprised and relieved that
it had all worked out I'm sure there was
some degree of that well before you tell
us about the singularity Institute
because I'm curious about that myself
what sorts of things did you teach
yourself to prepare to be a singularity
and is that the term singularity rien
and I returned and I wouldn't really say
that that singularitarianism thing that
you how do I put it I was teaching
myself to be an AI researcher that's how
I thought of it so you're learning
programming and things like that
cognitive science yeah I taught myself
all the wrong things neuroscience
programming
and it's it thought how do I put it
later on it turned out that I should
have been acquiring a different skillset
and to this day I hope it's not too late
but on the other hand if I'd actually
gone off and studied first the most
important things that probably wouldn't
have worked either it's it's a bit hard
to describe neuroscience it doesn't tell
you how to build an AI but it tells you
a lot about which established theories
of AI are not helpful it if you if you
if you study how the if you if you learn
that the parts of the brain are things
like here's the visual cortex here's the
auditory cortex here's the cerebellum
and then you look at an AI theory in
which you've got a bunch of suggestively
named Lisp tokens like Apple and it's
just this little symbol that's Apple and
that's all it is and yet you see that
the parts of the program are nothing
like the parts of the human brain then
you say well that can't be right but of
course the human brain doesn't have the
right architecture either but what do
you mean by that the human brain doesn't
have the right architecture well if you
study a bit more neuroscience and a bit
more evolutionary biology and a bit more
cognitive psychology you start to
realize that the human brain is vastly
suboptimal neurons that only fire at say
an average of 20 Hertz so there's a
pretty large gap between neurons that
fire 20 Hertz and our transistors like
our two gigahertz two computer chips but
large as that gap is it's probably
nothing compared to the software gap
between what the human brain is and what
the software for an intelligent mind
should be it's just that that part of
the gap is a bit harder to get a grip on
before I you know I think I'd like to
come back to their site first of all
you're already bringing into into
there's a lot of really interesting and
important points but first tell us how
the singularity Institute was actually
founded and what your role there is well
the way that it was actually founded was
that I wrote a rather old not not very
good by my present standards but it was
a document called coding a transhuman a
on and Brian Atkins who's in the process
of cashing out of hyper Mart wrote to me
and said hey this this plan you outlined
is that something that someone could
actually do and I said no that's gonna
take him in Hatton project and and and
so we thought about doing a startup and
I sort of like looked into
entrepreneurial matters and how to write
business plans and all that sort of
stuff in preparation for the startup and
and at around the same time Brian Atkins
and I realized that it wasn't going to
work before we actually launched it just
while I was still in the preparation
process I was going like yeah this would
have been a company that actually tried
to market AI programs
no it wasn't an AI company remember at
this stage I was still thinking in terms
of real artificial general intelligence
is going to take 30 years even if you
put in an intense effort and and the
goal was just to speed it up a little
bit more that was when I went off on to
a different mistake which was thinking
that you could open source an artificial
intelligence and also sort of having
this sort of more concretely detailed
though from a my modern perspective
still wrong viewpoint as to the stages
in AI could go through and what you
could do at the start and so I started
thinking that you know you can actually
have an institute that would launch this
open source project that would get get
you started down the road to a
and coordinated and so that's what this
what the original plan for the
singularity Institute was yeah open
source AI that's a really interesting
concept who put up the money for this
thing
Brian Atkins okay it in the beginning
like he later became shall we say unable
to fund the Institute but by that time
there was a very small base of donors
and it sort of struggled forward and
kept going now here's a really important
issue for me just in the email exchanges
that we've had and by looking at this
chart on I Triple E spectrum and by the
way I should say that I Triple E
spectrum magazine it's June issue has a
whole special section on the singularity
I contributed an article to it it's also
got articles by neuroscientists and
economists including somebody you know
Robin Hanson a whole bunch of different
articles some of which are very positive
it's got an update by vinji and some of
which like my own are very critical but
it's also got what I found very useful
is a chart of the major figures
associated with the singularity and
explaining their different points of
view so that chart made it very apparent
that there are lots of different
versions of the singularity so I wonder
if you can talk about how what that
means for the singularity Institute
itself is it associated with a
particular vision of the singularity and
if so which one is that or does it cover
a bunch of different views the its
associated with the particular vision
that did change over time in the
beginning we were associated with Vin
Diesel vent horizon
the where the effect horizon is the
notion of this time beyond which you
cannot predict the future because it
contains things that are smaller than
you are
note that this does not say anything
about when this happens it does not say
whether this happens as a result of
artificial intelligence or advances in
neuroscience that enable us to augment
humans it doesn't say anything about
exponential curves it doesn't say
anything about Moore's Law just at some
point you get the technological ability
to create smarter than human
intelligence and then your model of the
future breaks down this is the original
thought that I thought was the the most
important thing I'd ever heard and it
took me several it took me quite a long
time actually to figure out the flaw in
it so the modern the modern version of
the singularity that the singularity
Institute is associated with is I J
goods intelligence explosion which is
the notion that if you make something
that is smart enough it can improve
itself make itself even smarter and then
having made itself smarter make itself
just smarter again lather rinse repeat
you mentioned good he say what applause
I cognitive scientist who was writing
back in the mid-60s he is a famous
statistician who among other things
rotate wrote a paper called the first
ultra intelligent machines or something
along those lines like I can I can look
it up on a separate computer if you
think it's important to get the exact
title oh no that's okay we could find
the links later but what's what's your
so obviously the person who's who's best
known in this whole area is Ray Kurzweil
and some Kurzweil a computer
entrepreneur who's been very successful
in building for example artificial
vision machines
text recognition machines and
particularly lately his vision of the
singularity has involved immortality in
some form whether it's because we
redesign our bodies to counter
senescence or we turn our psyches and to
software programs and download them into
cyberspace but to me this is the sort of
salient feature of his vision is that we
live forever so I wonder if you can
before you explain where you're coming
from tell us what your attitude is
toward curse well well first while is
sort of a in a newcomer to the namespace
in other words it's at some point he
started talking using the word
singularity for his vision and I do feel
that that vision has certain substantial
departures from what went by this name
singularity before then and further
vinji is the one who originally coined
the term yeah so so Kurtz while I
usually refer to that as accelerating
change it's very heavily graphs of
Moore's law based futurism and Moore's
law is not something that you would even
knit that that is logically entangled
with either the Venge an event horizon
or goods intelligence explosion there
doesn't seem to be a lot of emphasis on
smarter than human intelligence as a
breakpoint in Kurtz Wells graphs you
know it just cuts up to the point where
you've got individual supercomputers a
million times as with a million times as
much computing power as human which
incurs wiles schemas sort of taken for
granted to indicate a million times as
much intelligence as a human and then
Moore's Law is still running on it see
you know he's he's still projecting out
the graph past that point to 2099
so well he ought my impression of him is
that he's he sees this convergence of
lots of different technologies genetic
engineering nanotechnology artificial
intelligence improvements of hardware
and that in one way or another and he's
a little bit fuzzy on how this will
happen as he should be he says that it
will produce some kind of radical
transformation maybe we will become
cyborgs maybe we'll be downloaded so I
didn't think he was too explicit about
how this would happen but it's that look
multi technology convergence that's an
important part of what he's talking
about and and if you look at it from the
perspective of either of the event
horizon or the intelligence explosions
and he's not distinguishing intelligence
within that packet of technologies now
from my perspective intelligence is the
strength that underlies our existence as
humans you know you know you say
intelligence and people think of book
smarts they think of calculus they think
of chess where you're applying clear
rules and well understood situations and
they say well it takes more than
intelligence to survive in the human
world you need social adeptness what
about musical talent doesn't that count
for anything
what about persuasiveness or the or
empathy and the ability to understand
people or even manipulate people doesn't
that count for anything and everything I
just said it all occurs in the brain not
in the kidneys these are all cognitive
they aren't what we think of as book
smarts but there aren't many famous
musical composers or military generals
or CEOs of Fortune 500 companies who are
chimpanzees our intelligence is the is
is the foundation it's the strength that
underlies everything else we do as
humans and if you can even take an IQ
test that was designed for humans you're
already part of the cognitive elite you
are a homo sapiens a mouse would just
try to eat the IQ test so and all of our
intelligence is
a something that is something that was
patterned by our intelligence if you
look at the computer in front of you
that's not something that grew on a tree
that's something that grew on a human
mind so if you modify intelligence if
you tamper with intelligence you are
lifting up the technology tree by its
roots its distinguished both by its
foundational status by by being the the
by the human intelligence in human
brains being the pattern that creates
all the patterns that we think of as
technology and it's also distinguished
by the possible intelligence explosion
effect that if you that if you got smart
enough you can make yourself even
smarter ok these are ok that that's I
mean glad you you sort of focused on
intelligence here but what I want you to
tell us is how exactly you see this
happening so it seems to me that you've
got a couple of different possibilities
one is that computers these you know
mainly silicon based machines whoops let
me just disconnect my phone
that's my my other backup still ringing
anyway I said what I'm basically trying
to get from you is whether or not she
wants your attention these are just
multiplying I can't turn this damn thing
off oh my god you know I keep thinking
as you're talking about the glories of
intelligence of the super intelligent
people I know who are absolute disasters
as functioning human beings but anyway
what I want you to do that gets us all
off onto the topic of rationality
rationality intelligence but probably so
what I want you to what I want you to
tell us is which you know you've got the
cyborg
version of the singularity which you
know some kind of hybrid this rock
Brooks talks about this in the spectrum
issue Kurzweil has also talked about
this then you've also got something that
could be quite different where machines
start redesigning themselves get this
positive feedback run away effect and
and they get super smart but that
doesn't necessarily affect humans in
other words we might remain
physiologically more or less as we are
then there are also versions in which
the internet becomes this kind of cosmic
meta computer and it becomes intelligent
and develops itself goals or so forth so
which of these visions try to try to be
as explicit as you can about which of
these you think is most plausible and or
desirable Pat Cadogan speaking of the
spontaneous emergence of intelligence on
the internet I think she spoke very well
when she compared that to the notion of
dirty shirts and straw spontaneously
generating mice which was a theory back
in the good old days
spontaneous generation yeah so and and I
have a blog post about that called the
futility of emergence and simply the
notion that corporations are evolving no
evolution for corporations would be the
relevant blog post for that the sort of
fundamental thing you want to be wary of
in all of these is when you start
talking about
how you're going to build AI without
knowing how intelligence works oh and
you were talking about the cyborg vision
so by cyborgs are you talking about
artificial limbs or are you talking
about brain computer interfaces because
well they're extremely different things
one of those impacts on intelligence and
one of them does not yeah the most by
far the most interesting is the use of
prosthetics to boost perception memory
processing speed basically intelligence
and all the things that that
intelligence requires that that connect
us to the world maybe it would be a
brain chip that gives us instantaneous
access to
the World Wide Web I mean you can
imagine all sorts of different things
well I have instantaneous access to the
worldwide web right now for it pretty
much be a much broader band method and
we're talking about and also brand chips
would allow a kind of telepathic
communication between humans possible
merging of identities that's something
that incidentally that I think might
have been overlooked as a path toward
intelligence augmentation which is
instead of you might actually be able to
even before you figured out what the
brains code is you might be able to take
a million tap brain computer interface
and just hook up a million neurons in
one person's prefrontal cortex or
somewhere to another person's billion
neurons just force synchronize them and
see if the two brains can learn to talk
to each other house brains are good at
learning I don't know if they're good at
learning that sort of thing but they're
good at learning you might be able even
be able to get 64 node cluster humans so
so you see I thought you were pretty
narrow in your view of how the
singularity would occur that it would
just be in machines and the machines
would not necessarily have an
intelligence that in any way resembled
that of humans so for example there's
some singular as' in fact I think Robin
Hanson said this in his spectrum
articles who say that the way we're
going to get intelligent machines is by
building computers that mimic the
information processing of the human
brain you I thought have said that that
would be a mistake
machines should develop their own
completely idiosyncratic and presumably
much more efficient method of processing
information and that's how they're most
likely to become much more intelligent
than us okay so it's so if I can go off
on a small quick tangent
what I would say distinguishes careful
futurism from sloppy futurism is making
distinctions and supporting separate
arguments separately there is something
called the conjunction fallacy it's a
it's a cognitive bias that cognitive
psychologists study and that's where
there was a famous experiment I'm sort
of reciting it from memory here so I may
get a bit of details wrong but in the at
the second International Congress of
forecaster some psychologists went in
they asked one group of forecasters what
is the probability of a complete
breakdown in diplomatic relations
between the u.s. and the Soviet Union
and then they asked another group what
is the probability that the Soviet Union
will invade Poland followed by a
complete breakdown of diplomatic
relations between the u.s. and the
Soviet Union and the second group gave
higher probabilities even though it was
a strictly more complex event that had
an extra detail on it the conjunction
rule of probability theory is that the
probability of a and B is always less
than or equal to the probability of a
alone the more complicated something is
the less likely it is to happen compared
to a strictly simpler description of the
same event that includes the more
complicated one and with and in other
words as stories become more detailed
they necessarily become less probable
but they can sound more plausible people
will assign higher probabilities to them
and to be a careful futurist you have to
counteract this tendency
mmm-hmm Wow heavens so as a careful
futurist which scenario is most
plausible to you when I'm trying to
point out here is the importance of
making fine distinctions saying what you
know and what you don't know putting
probability distributions on things I'm
not going to pick one scenario and say
this is the way this is the story that
I'm going to tell an audience about the
wonderful future and how it will be
you've got to put probability
distributions on things okay if you want
to be a careful futurist and not a
storyteller yeah so I am putting my life
into artificial intelligence because for
a number of reasons because it's easier
to do it without FDA approval compared
to human enhancement because it just
seems in a certain sense easier even
though it's extremely difficult because
they don't have to struggle with the
existing human brain that is not
designed to the end user modifiable if
you open the case it voids the warranty
there's no documentation if you're
starting over at least you get a chance
to do things your way and because it
seems to me like artificial intelligence
requires more care to get correct like
if you're upgrading humans then they
start out with everything that makes you
human
and presumably you main and you could
drive them insane but at least you
wouldn't fail automatically in
artificial intelligence you're trying to
hit a very narrow target and create all
these things over again that human
beings already start with so it's a very
narrow target you have to be very
careful and that's one reason why I see
it as a more urgent endeavor that
doesn't mean that just because I'm
putting all my effort here does not mean
that there's only one determined future
we someone could come up tomorrow
tomorrow I could mean the news that
someone came up with this really way to
do a million neural taps and there's
already two laboratory subjects whose
brains have learned to talk to each
other and they're producing these
amazing scientific discoveries because
now that they're telepathic they can
check each other's ration
and stop each other from going off on
silly tangents the point thing is sort
of like you're asked it seemed to me
that you might have been asked a bit of
tell a story and I know is refusing to
do so I will be willing to describe why
it is that I think that our official
intelligence might well come before and
I mean I don't even know that there's
all a strong case for artificial
intelligence coming before your brain
computer interfaces you might make a
case for artificial intelligence is
going to pull ahead of it or it's going
to be hard to make to enhance humans so
that they're smarter than the smartest
existing humans using brain computer
interfaces I could talk about the
mention of these biology but these are
dependent on each other too one can feed
off the other
I mean advances in artificial
intelligence and software could
certainly improve our ability to
understand how information processing
works in the brain because there's an
enormous number crunching that you have
to do in that realm
plus we may learn things by through
brain interface human-machine interfaces
about how the brain works that can then
inspire artificial intelligence but
listen I you know Ellie I wanted to sort
of tell you I mean you know for reading
my article in spectrum that I'm a
skeptic about this but I just wanted to
give you a little bit of background that
wasn't in that article but why I'm so
skeptical so I started my first job out
of journalism school and 1983 was with I
Triple E spectrum magazine this is the
heyday of the Reagan administration it
was pouring many millions of dollars
into artificial intelligence that could
be applied to
defense and it was a very exciting
period I really got caught up in it just
before that I had had a class with a
writer named Pamela McCord ik who was
writing a book called machines who think
which came out in the early 80s and it
was about among other things the
Japanese fifth generation project which
supposedly within ten years would would
accomplish complete artificial
intelligence at least enough to do they
know that pardon me how did they know
they were going to get complete
generally the attendees were the
prediction so that this was an
industrial goal of these big computer
manufacturers in Japan that they were
going to have artificial secretaries by
1990 then he also had people who were
sort of more visionary like Marvin
Minsky at fredkin and John McCarthy were
three of the founders of artificial
intelligence saying that computers would
soon be smarter than us you get this
positive feedback effect and then they
would just go racing beyond us a few
years after that you had Hans Moravec in
mind children talking about the next
great step in the evolution of
intelligence would which would be
machine intelligence and again it was
the same thing machines were just going
to go racing past us leave us in their
cognitive dust and this was all going to
happen fairly quickly I thought this was
fabulous it was one of the reasons I
wanted to get into science journalism
because I found these visions so
appealing and thrilling and of course it
came to naught artificial intelligence
has been a huge disappointment over the
last 2025 years by almost any measure
Marvin Minsky
has admitted that you know there's still
some true believers but I mean the field
of arjun artificial intelligence is sort
of a joke now so it seems extraordinary
to me that there are people who still
hold on especially you know young people
like you who still hold on to what to me
are are completely irrational and really
best understood as religious fantasies
about this great future transformation
that is possibly going to see it solve
all the world's problems and save us and
give us immortality or you know cosmic
listen skeptical fundamental question of
rationality why do you believe what you
believe
yeah so first of all let me ask the sort
of distinguishing question do you think
that in 10,000 years let's say we're not
going to have artificial intelligence
the brain is still going to be a mystery
you know it's a good question I actually
I'll hold it up here I am now reviewing
a book I can't say the publication but
it's called year million I don't know if
you've heard of this just came out and
it's predictions from a bunch of people
Robin Hanson is one of them about Robin
I would bet Robin Hanson silly one in
there who makes any sunspots basically
warmed-over Hans Moravec mine children
the question is are we going to live in
virtual reality or this reality you know
the source in Robin Henson's think for
that the rapacious heart scrapple
frontier folk are my getting papers
mixed up know that that was actually
that that was part of it
and so what I found interesting about
his essay was that and actually this was
part of more of X original book to was
that we'd still be a Darwinian world
so I guess my hope and maybe you can
tell me where you come out on this I
look at this whole area as a kind of I
think it's fascinating because it can
help us articulate our fears and desires
for Humanity in deep time and I don't
think this is a useless exercise I think
it's really interesting what Hansen his
vision is interesting to me because it's
so Darwin he says we will have this this
endless competition of the super
intelligent whatever they are cyborgs or
he's not really clear on exactly what
form these things dictated that's not
not really being clear that is being
deliberately vague the more
possibilities you include the higher the
probability of your prediction yeah
that's fine but that the most important
feature is that competition will define
existence in the future where I guess
because I'm a bleeding-heart liberal
pacifist I look forward to you know I
look into the deep future I want to
believe that we will transcend this
atavistic Darwinian part of ourselves
you know the need to beat our beat our
neighbors in colonizing the next star
system which is what you want to believe
or you would like to make it happen that
way those are two very different
statements both I think that's where
rational creatures would want to go and
I think we are even rational enough now
that we can move in that direction and I
think we clearly are moving in that
okay but the is your is there going to
be some kind of global Leviathan that
prevents anyone from defying the schema
is there going to be some kind of
distributed Leviathan where if you go
off and start trying to take all the
resources people combine to stop you how
is your how is it going to be stable
against single defectors incidentally I
should emphasize this is a point on
which I happen to agree with Road yeah I
had which I happened to disagree with
Robin Hanson especially about the notion
II can actually apply evolutionary
arguments because in the presence of
intelligent design people go to the
optimal design straight out and so
there's not enough remaining variance
for evolution to work upon some of that
was on the overcoming bias blog that
debate between myself and Hanson however
you can't just say I would like it to
happen this way or a part of let me
phrase that you can't say I want it to
happen this way therefore that's what
I'm going to believe that's an
interesting point
I think sometimes I think it's happening
anyway that there is actually I'm as you
may know the the founder of blogging has
T V Robert Wright has this whole vision
of evolution as involving a gradual
increase in cooperation so that you know
you've got this sort of competitive base
but that leads to more complexity and
cooperation over time and you now they
you see that these are this global
networks economic networks or things
like the internet where people realize
their dependence on each other and they
don't see it it's we're not involved in
a zero-sum
that there are many ways in which we can
mutually benefit and we can become
conscious of this process and more
conscious of it over time and direct our
revolution in that way ok because I was
just going to say that their visions of
things that are inevitable don't
interest me all that much because if
they're inevitable there's no way to
modify them there's no opportunity to do
anything about them and so in the end
they cancel out if you're strategizing
and some of your writings I I've sensed
an almost again I have to say it's
almost a religious belief in the power
of super-intelligent ok so you say
religious now that's a term that means a
lot of different things to a lot of
different people can you say without
using terms like religious what
especially if you see a problem what is
the problem you think it's not fully
justified by the evidence you don't you
don't see the evidence to support the
idea okay okay first first of all
there's the issue of the creation of
super intelligent machines and I've
tried to explain to you why I I don't
break so I sell up and I think that you
you know you're it's you're expressing
more of a faith than a belief based on
current read have you did you read
artificial intelligence as a positive
and negative factor in global risk well
I I read parts of that but the other one
that I was reading was
the bias and let's see biases
potentially affecting judgment of global
risks which I thought was at least
indirectly about the possibility of
using super intelligent machines to
assess risks and you know it really it
wasn't actually that paper was not about
that it was just what it said nothing
more nothing less using super
intelligent machines to correct these
minor little human biases is like nuclei
using a nuclear weapon to swat flies
okay so tell us what is your vision of
how computers will help us solve our
problems super intelligent computers
okay well can I actually not go down
this track you just yet and deal with
this fake business but but that is a
sort of core issue okay so the first
thing I wanted to say was you haven't
read all my stuff perhaps I have support
you haven't read yet so yeah I mean this
is something I've learned about an
overcoming by some number of context
which is a sort of naive realist
rationality where I in a hunter-gatherer
tribe ten thousand years ago you could
rely on knowing what everyone else knew
and today there is no way for anyone to
say science doesn't know X because there
is all these different scientists in the
world and six billion people in total
who know all sorts of things and you
haven't met them all and some of them
are and some of them know entire
professional fields whose name you might
not never even have heard so no one can
sani and there's so much science that
now that no one human being could learn
it in a lifetime so no one can say
anymore what science doesn't know
there's too much science and I and I so
so in other words what I'm saying is
okay I have certain beliefs that sound
odd but this isn't a hunter-gatherer
truck and they're my support is not
always going to be immediately visible
especially if you don't read my papers
so going into the specifics okay you
talked about how you were disappointed
some people made amazing incredible
artificial intelligence predictions and
then you were disappointed the world's
stupidest person may say the sun is
shining but that doesn't make a dark
outside
Robert Percy or as I like to say reverse
stupidity is not intelligence if you
want to get things right you've got to
do a lot of hard thinking and process
the evidence if you imagine someone who
is wrong
99% of the time on yes-or-no questions
you could be right 99% of the time just
by reversing everything they said you'd
have to be super intelligent just to be
that stupid so so here the point is they
said the Japanese fifth generation
project said we're going to have
artificial intelligence in ten years why
why did why did they believe that why
did they say that because they wanted to
make a bright future estate prediction
so that if you get a bunch of funding
when I went into art if I went into
artificial intelligence knowing it was
hard because I thought that this is what
humanity needed to do to move forward
and and this and it seems to me that a
lot of other people I see that's what I
don't you that before this is over I
want you to explain why you this this
aspect of saving humanity or why this is
so important how this is going to solve
problems like poverty and
and warfare and climate change and
epidemics and you know ordinary
depression or you know all these sorts
of things that were afflicted with how
will super intelligent machines help us
solve those problems it seems to me
obvious in fact we've already sort of
touched on this in this conversation
that super intelligent scientists often
lack all wisdom and common sense for
solving real they are not super
intelligent they have a bit more book
smarts than the human average there is
not a distinction between between them
and the rest of humanity into the
distinction between you and a chimpanzee
but it's this quality of common sense
and sort of contextual as opposed to
very highly specialized knowledge that
is most strikingly absent in any
computer and this has been the great
failure of artificial intelligence this
is another reason why I'm so skeptical
obviously and do you think this is still
going to be a mystery in 10,000 years
10,000 years is I can't even predict in
a hundred years so I'm you know for the
purposes that spectrum article I was
focusing on the prediction of vinji and
others that this will happen maybe by
2025 2030 we're talking about something
in my I hope lifetime that's something I
can grasp okay but right you're talking
to to me not the authors of the spectrum
and I'm asking you do you think that
this is unknowable or is it simply
unknown at the moment I think it's
unknown at the moment and for the
foreseeable future if you're
extrapolating to that 10,000 years from
now it's almost I think I would have to
agree anything can happen
it's not like some magical mystical
barrier it's just there that it's just
that given all the trends and
mentioned technology we can see today
there's no reason to hope it will happen
within our lifetimes okay so from my
perspective again something I've been
talking about a lot recently and
overcoming bias don't be scared of scary
research problems not not this is not
addressed to you it's addressed to
everyone if there's a like part of that
business with how can intelligent people
be so stupid there's actually a
wonderful book called why smart people
can be so stupid it's an edited volume
but one of the ways that people waste
their intelligence is by working on
unimportant problems why do they work on
unimportant problem ones because the
important problems look scary and if
there's a wonderful quote by Lord Kelvin
which which should probably have looked
up in advance but something along the
lines of life is infinitely beyond the
range of any scientific investigation
hitherto entered upon and it's in the
generation of of of tree from generation
of trees from from a single seed in the
daily demonstrated miracle of our free
will it is infinitely beyond the range
of any possible science to that Artaud
entered upon and he was speaking in
favor of vitalism now if you look at how
mysterious life seemed in the 19th
century versus how intelligence versus
how mysterious intelligence seems to us
right now yes they were actually a lot
more confused back then my friend of
mine Tom mccade has a saying anyone who
says that intelligence is a complete
mystery ought to be slapped upside the
head with the MIT encyclopedia of the
cognitive sciences L 964 pages of it so
let me just interrupt you it
and tell you that you know I wrote a
whole book the undiscover'd mind came
out about ten years ago basically saying
that arguing for mysterion ism saying
that that certain aspects of the mind
and especially consciousness will never
be explained but recently I I couldn't
even see a program for explaining it but
I've recently written in spite of what
that spectrum article implied that I
think that the you know we can't have a
unified theory of consciousness if we
solve the neural code I think it's
probably the hardest problem in science
right now but when I talk to young
people who want to become scientists I
say that is the most exciting problem
potentially solvable problem out there
in terms both of its intellectual depth
and it's the possibility of practical
applications I you know so I definitely
think that there is room for tremendous
progress in the whole realm of the mind
and brain and artificial intelligence
it's just you know I just I just get
annoyed at these these predictions that
certain things will definitely happen
and certain have you heard that predict
smooth for me no I guess I haven't and
so if you know if you're if you're if
this is just something that you think
would be a really cool thing to happen
at some point in the future you said ten
thousand years then I know there's not
really much we can argue about it is so
you don't put any timeframe on the
singularity then
I try not to but in fact if you were to
hold me at gunpoint and force me to
confess under a lie-detector a a working
lie detector one of those newfangled
fMRI things maybe you know you would
probably get out something like twenty
to fifty years okay so I'm so I'm so I'm
feeling attacked belong that score but
the thing is is that first of all okay
you had a bunch of people making an
overly enthusiastic predictions about AI
but you can't learn from this that in
fact AI is very far away you're trying
to reverse stupidity to get intelligence
all you have to do is say okay no
program the problem wasn't that I'm
trying to didn't figure out how to put
this into words if you actually wanted
to know when AI would be developed even
approximately you have a piece of
information that is hard to get and you
would have to figure out how you could
possibly get ahold of it and and failure
to do this is what led to all these
overly enthusiastic predictions that
were essentially expressions of
emotional gee I feel really great about
AI o now I feel very sad about the AI
it's a long way away it's the same
mistake all you've done is flip it over
and look at the other side of the coin
you can't predict something without
having good information about it and the
fact that a bunch of people may failed
predictions about one AI would happen is
not good information about the actual
future well but some of these people
obviously were extraordinarily well
informed or at least they had very lofty
positions of authority like Marvin
Minsky and Herbert Simon and and John
McCarthy people like that so you know
these weren't just okay let's say that
the people who were in charge of the
fifth generation project in Japan that
was just corporate hype that's fine but
you know you've got to take somebody
like Marvin Minsky
seriously so I think you know I took his
prediction seriously 25 years ago and I
also take his pessimism about the
prospects of AI seriously right now you
sure that's not just the same sign at
the different mistake the different side
of the same mistake maybe it is it's
just that I haven't seen any
developments in artificial intelligence
that you know concrete developments
maybe there's some hiding out there
because there obviously I am
they're wrong I'm not mom Nishant but
that seems to me to be a little bit too
easy a way to to counteract pessimism
like mine I think that really
extraordinary achievements yeah you know
I think the old phrase of Carl Sagan's
is appropriate here extraordinary claims
require extraordinary evidence so you
know it's just not enough for you to say
that well out there in these little
niches somewhere or hidden advances that
they're not they're not in little niches
I mean it's very difficult to appreciate
what's going on in artificial
intelligence without being part of the
field I think that the only reason that
this field looks like it's progressing
at all slowly is simply that every
generation you've got someone coming out
and saying we're going to have general
AI in 10 years if it wasn't for that if
all you were looking at was what was
actually happening it would not look all
that slow just a global question what
about well let me just ask you this
what's the difference if any between
intelligence and wisdom we've been sort
of talking about that let me just give
you a more specific example this again
goes back to when I just began as a
science writer in the early 80s you
remember the strategic strategic defense
program starved yes that was supposed to
be basically an automated system that
would protect us from doomsday so you
needed we had we would have these
satellites that would look for launches
of Russian missiles and one of the big
technical problem and and it was sort of
a you know philosophical problem to was
was how you design the system so that it
can determine whether this is a genuine
Russian launch a genuine Russian attack
or it's a false alarm
within seconds for at least minutes
because we we wouldn't have more time
than that to make the decision do you
trust is there any computer that you
would trust with making that decision
that was that was one of the objections
that people had okay yeah so
intelligence versus wisdom all right so
wisdom is not something that generalizes
quite as well as the concept of
intelligence two realms beyond the human
and what I mean by that is that while we
can imagine something that's extremely
smart and is extremely good at
technology and therefore it can figure
out ways to take apart stars and use
them to make new and interesting things
the notion of how does wisdom scale
seems hard to get a grasp on like what
the super wisdom look like and but
within the human framework
they're sort of like people who are very
mentally adept and they're very quick
thinkers and they use that ability to
shoot their own feet off and that and
and this of course is something that
I've written a great deal about an
overcoming bias both because of the
smart person myself I do not want that
to happen to me I want to make good use
of myself if I can and also because of
what it fundamentally says about the
distinction between how fast human
thinks and how rational they are and
these are sort of different qualities
about them if they're says there's
something called there's a there's a
known phenomenon called motivated
skepticism where people more carefully
examine arguments that they want to
refute than arguments that they want to
accept and if and if you start out with
this fundamental ubiquitous mistake then
every new logical flaw you know how to
detect is going to make you that much
stupider because it's going to be one
more excuse that used to refuse things
refute things that you don't want to
hear so if you don't use intelligence
properly it is very easy to shoot your
own foot off with it you know something
I just remembered there's a column in
The New York Times today I think it's by
David Brooks reflecting on what
qualities we want in a president and
talked about Abraham Lincoln as somebody
who was obsessed with his own
limitations and failings and as a result
was a wise and mature and great
president
and Brooks said that this is you know we
sort of want our presidents our leaders
to be very confident but there obviously
is such a thing as overconfidence well
no people vote for confident presidents
but they shouldn't as I think the
statement error right so it's you know I
guess wisdom is I think it overlaps
quite a bit with with humility and I
think we all know that you know one of
the traps of great intelligence is that
it could lead to certitude and
self-righteousness and and a kind of
brittleness it's very very far from
wisdom but you know buddy can also be a
trap yeah there's very few safe harbors
when it when it comes to trying to wield
yourself correctly and you will how can
you militay be a trap it's very easy
whenever you're faced with the
conclusion you don't really want to
accept you say but how can I really know
this so for example the creationists
they want to reject evolutions they just
say well how can you really be sure of
anything isn't science all about
openness to all the ideas and what and
if there's a point where young which
your mind is permanently open and you
can't close it any further that's the
same as having made up your mind isn't
it if you refuse to close it or to put
another way the point of having an open
mind like having an open mouth is that
it occasionally closes on something
solid and so I think someone said and of
course you can always revoke it if you
just different evidence comes in but
that's not the same as you as refusing
to come to any conclusions or refusing
to make any predictions we're already
I'm sorry to cut you off we're already
at our two minutes and I want to just
ask you
one last question to put you on the spot
here and it's it's this what if you were
in my shoes or what do you think the
most valid objections of skeptics are
toward the singularity you know you're
obviously somebody who's smart enough to
to have thought of all possible
objections you've certainly heard a lot
from people over the years which do you
think are most valid and is it possible
that you might abandon your belief in
the singularity in the future well again
the singularity is a name that's being
refused used to refer to around 22
different concepts some of those
objections are so good that I outright
accept them so for example I don't
necessarily believe that there was more
change from 1970 to 2000 than there was
from 1940 to 1970 there's an objection
that's so strong that I actually accept
it but if you were talk about say the
the the concept that I would find that I
would that that I have the strongest
grip on as it were to make it the
strongest challenge that would probably
be the concept of a mind improving
itself for actually from your
perspective maybe the strongest part
would be the notion of there being such
a thing as super intelligence that can
have a huge impact and what would it
take to refute that okay in terms of
actual experience I think that if one
were to build a mind that it was at
least as smart as Einstein and then you
like increased its processing power by
orders and orders of magnitude and all
the internal metrics that you have of
you know how well the function the
systems inside are functioning are
showing that there are thousands of
times higher than they were when the
system was in Einstein and the system
still doesn't do anything all that much
better than Einstein and then I would
say that refutes the concept of super
intelligence do you demonstrate there
was an upper limit to how useful
intelligence really is and it lies right
above the unloved this is
extraordinarily improbable but in terms
of you know it's an experience that
refutes the theory and shows that it's
falsifiable and constrains prediction
that's an example of an experience that
refutes the theory is that something you
happen within the next 20 to 30 years
that's sort of test um that's AI that is
something that like like that test we're
referring to is the singularity event
itself so it has all the usual quibbles
about time scale and okay so strongest
objection what's the strongest objection
um you know it's sort of like asking me
what's the strongest objection there is
to natural selection not quite that bad
yeah I know not quite that bad but this
is something that I believe strongly be
precisely because I don't see Veni that
many very strong objections to it I mean
this promise of just labor sorry what
spoken like a true believer well I'm
sorry but things that are but strong
evidence is by definition of the
Bayesian 's exactly that kind of
evidence that you don't expect to find
on more than one side of an argument
listen you have to post your give us
your best shot post your most persuasive
articles or articles by anybody else or
even news stories about advances and
artificial intelligence that might help
us appreciate your point of view you
know that's part of what blogging heads
does they have those links yeah but but
in all seriousness my view has changed a
great deal over time why is that someone
came up with something that I thought
was a strong objection I accepted it
yeah you know I used to think that and
that in that artificial intelligences
that were powerful enough regardless of
how they were designed would be nice to
people but there were strong objections
to this theory so when I realized that I
abandoned it so there's so I can give
you examples of things that I regarded
as past strong objections it asked me to
change my opinion if I gave you an
example of a strong objection that I was
not listening to right now this would
represent some kind of problem on my
part
there are sorts of the objections you
can make like well but you haven't
experimentally demonstrated
superintelligence
sound like sort of quibbles over fine
points of dogma within Christianity for
example I'm sorry that was a cheap shot
yeah I wasn't about to say it was a
cheap son I'm just saying like what else
am I supposed to say am I supposed to
say I know here's a nice sound objection
I haven't accepted there are all sorts
of weak objections I think I think this
stuff is really cool it's fascinating I
continue to follow it with great
curiosity I hope I hope I see something
out there that at least makes me waver
in my own disbelief I'm you know I'm
sincere about that but but you're a
great you're a great representative of
this field and and you know I'm sure
you're going to get lots of people
checking out your blog and the
singularity Institute in the future
which is you know which will serve your
cost yeah oh and let me actually just
say that that policy debates have no
reason to be one-sided you know like
individual options often have strong
disadvantages so if you would ask me
like what's wrong with the singularity
Institute strategy of going out and
building an AI this way I could give you
all sorts of strong objections to that
okay so for example so so you know like
what's it so um synthesizing an AI a
priori well how are you how are you
going how do you know you're going to
get the fundamental research
breakthroughs you need to do that and
the answer is you don't right or you're
going to build this extremely powerful
AI and how are you going to get it right
the first time and the answer is things
like well you have to be extremely
careful you have to use a simple design
and you say but even so can you really
get it right the first time
and that's a very strong objection so
because this this is you know goes back
to the whole rationality issue but you
shouldn't really excelled abates you
might find weak evidence on both sides
but by the definition of strong evidence
you shouldn't expect to find a strong
evidence on both sides so I can so I can
give you so a lot of these ideas I'm not
necessarily saying they're strongly
supported a lot of times the objections
that the the the support that I have
this week but it's but if it weren't
stronger than the weak objections I
would be abandoning the idea and it's
like thank you I mean over here I think
I just understood that our viewers will
be able to play it back and and ponder
it and and rock it at their leisure but
anyway that's we're almost at an hour
ten minutes that's it yep so okay we'll
have to wrap it up there Ellie I've
really enjoyed this and your you're a
great you're really good at arguing for
your cost power to you and I wish you
the best of luck thank you all right
talk to you later