Yudkowsky vs Hanson — Singularity Debate

see with the festivities I forget what
the exact form of it was the question is
after all sorts of
logical things happen at some
undetermined point in the future are we
going to see a sort of very small
nucleus that the that can or does
control like all the resources or do we
see like a sort of general more
civilization wide large fraction of
society participating in all these
things going down and so do you want to
I think if I remember it it was compared
to the Industrial and farming
revolutions in the intelligence
explosion first movers will soon
dominate a larger fraction of the future
world that's what I remember the whole
did try to explain what those mean word
is saying that you believe that the
first movers will gain large lead
relative to person groups in the
industrial farming revolution right
let's do use that soon of your for the
side thing it may be more broad-based
calm like maybe one word thing would be
highly centralized highly decentralized
with the bass sound like this well one
we're just rate cut off in between
highly since there's a little bit with
me with with the cutoff point things
easy industrially the Agricultural
Revolution for example or know that it's
actually like not the cutoff point
that's your side so on the yellow sheet
if you're in favors you write your names
and I'm in favor if you're against your
your name and against and then pass them
that way keep the color chief and that's
give your vote afterwards and
oh yeah and Robert will be converging or
have fun or have fun trying we're very
excited at DC today to have a
eliezer yudkowsky it should be a lot of
fun professor at George Mason University
of Economics one of the frontiers in
prediction markets all the way back to
1988 and it's publisher both co-founder
overcoming bias that's wrong I moved
over to Lester on now overcoming bias in
the eyes our co-founder of the
singularity Institute many many
publications without further ado on the
debate and the okay so
so quick question
how many people here are sort of already
familiar with say the differences
between what Ray Kurzweil means when he
uses the word singularity and the
difference between what the singularity
institutions when they use the word
singularity right raise your hand if
you're already familiar with the
difference okay I don't say CEO of hands
that means that I designed this talk of
correctly so you'd probably run across a
word singularity people use it with a
lot of different and mutually
incompatible meanings when we named the
singularity Institute for artificial
intelligence in 2000 it meant something
pretty different than than now the
original meaning was a mathematician and
science fiction writer named Vernor
Vinge II originally coined the word
singularity to describe the breakdown in
his ability to model and imagine the
future when he tried to extrapolate that
model past the point where predicted the
technological creation of smarter than
human intelligence in this particular
case he was trying to write a story
about a human with a brain computer
interface increasing his intelligence
and rejection letter he got from john
kemp l said sorry you can't write this
story neither can anyone else if you
asked a ancient Greek from 2,000 years
ago to imagine the modern world well
partly about 2500 years ago to imagine
the modern world and point of fact they
wouldn't be able to but they have much
better luck imagining our world and
would manage to get more things right
than say a chimpanzee would there are
stories from thousands of years ago that
still resonate with us today because the
minds the brains haven't really changed
over that time and if you've changed the
brain the mind that implies a difference
in the future that is different in kind
from faster cars or interplanetary
travel or curing cancer or Bionic arms
or similar such neat cool technological
trivia because that would not really
have a impact on the future comparable
to the rise of human intelligence fifty
thousand years ago the other thing is
that since intelligence is the source of
technology that is this is ultimately
that the factor that produces the chair
is the floor the projectors this
computer in front of me if you tamper
with this then you would expect that to
ripple down the causal chain
in other words if you make this more
powerful you get a sort of different
kind of technological impact and then
you get from any one breakthrough IJ
good another mathematician coined a
related concept of the singularity when
he pointed out that if you could build
an artificial intelligence that was
smarter than you it would also be better
than you at designing and programming
artificial intelligence so this AI
builds an even smarter AI or you know or
instead of a whole nother array I just
reprograms modules within itself then
that AI builds an even smarter one and
according to an AI J good suggested that
you get a positive feedback loop leading
to an IJ good termed ultra intelligence
but what is now generally called super
intelligence and the general phenomena
of smarter Minds building even smarter
Minds is what I J could termed the
intelligence explosion you could get an
intelligence explosion out of outside of
AI for example humans with brain
computer interfaces designing the next
generation of brain computer interfaces
with the purest and fastest form of the
intelligence explosion seems to be
likely to be an AI we writing its own
source code so this is what the
singularity suit is actually about and
if we'd foreseen what the word
singularity was going to turn into we'd
have called ourselves the good Institute
or the Institute for carefully
programmed intelligence explosions here
at the Institute for carefully and
programmed intelligence explosions we do
not necessarily believe or advocate that
for example there was more change in the
forty years between 1970 and 2010 than
the forty years between 1930 and 1970 I
myself do not have a strong subpoena
that I could I could argue on the
subject but our president Michael Vassar
our major donor Peter Thiel and heels
friend Kasper Okoye believe recently
spoke here all believes that it's
obviously wrong that that technological
change has been accelerating at all let
alone that it's been accelerating
exponentially and this doesn't
contradict the basic thesis that we
would advocate because you do not need
exponentially accelerating technological
progress to eventually get an AI you
just need some form of technological
progress period
so when we try to visualize how all this
is likely to go down we tend to
visualize a scenario that someone else
once termed a brain in a box in the
basement and I love that phrase so I
stole it in other words we tend to
visualize that there's this AI
programming team a lot like these sort
of one of the AI programming teams you
see nowadays trying to create artificial
general intelligence like the artificial
general intelligence projects you see
nowadays and they managed to acquire
some new deep insights which combined
with published insights in the general
scientific community let them go down
with their basement and work on it for a
while and create an AI which is smart
enough to program itself and then you
get an intelligence explosion one of the
strongest critics of this particular
concept of a localized intelligence
explosion is Robin Hampton in fact it's
probably fair to say that he is the
strongest critic by around an order of
magnitude and a margin so large that
there's no obvious second contender and
how much time do I have left in my eyes
- does anyone know are all right in that
case I'll turn you over drop we're going
to be very flexible here going back and
forth so it would be plenty of time I
thank you for inviting us I greatly
respect this audience and my esteemed
debate opponent here we know each other
for a long time we respect each other
we've talked for a lot it's a lot of fun
to talk about this here with you all the
key question here as we agree is this
idea of a local intelligence explosion
that's what the topic is about so we're
not talking about this idea of gradually
accelerating change we're in 30 years
everything you've ever heard about will
all be true or more we're talking about
a world where we've had relatively
steady change over a century roughly and
we might have steady change for a while
and then the hypothesis is there'll be
this sudden dramatic event with great
consequences and the issue is what is
the nature of that event and how will it
play out
so this brain in a box in the basement
scenario is where a something that
starts out very small very quickly
becomes very big and the way it goes
from being small to be very big is it
gets better it gets more powerful
and so in an essence during this time
this thing in the basement is out
competing the entire rest of the world
now as you know or maybe you don't know
the world today is vastly more powerful
than it has been in the past the
long-term history of your civilization
your species has been a vast increase in
capacities from primates to humans with
language eventually developing farming
and industry and who knows where over
this very long time lots and lots of
things have been developed lots of
innovations have happened and there's
lots of big stories along the line but
the major overall standing from a
distance story is of relatively steady
gradual growth that is there's lots of
inventions here changes there that add
up to disruptions but most of the
disruptions are relatively small and on
the distance scale there's relatively
steady growth and it's more steady even
on the largest scale if you look at a
company like yours or a city even like
this you'll have ups and downs or even a
country but on a long time scale and
this is sort of central to the idea of
where innovation comes from and that's
sort of the essential this to the center
of this debate really where does
innovation come from where can it come
from and how fast can it come so the
brain in the box in the basement within
a relatively short time a huge amount of
innovation happens that is this thing
hardly knows anything it's hardly able
to do anything and then within a short
time it's able to do so much that it
basically can take over the world and do
whatever it wants so that's the problem
so now let me stipulate right from the
front there's a chance he's right ok and
somebody ought to be working on that
chance and so he looks like a good
candidate to me so I'm fine with him
working on this chance I'm fine with
there being a bunch of people working
the chance my only dispute is sort of
the perceptions of probability some
people seem to think this is like the
main most likely thing that's going to
happen I think it's a small chance
that's worth looking into and protecting
against so we agree all there so our
dispute is more about the chance of this
scenario if you remember the old Bond
villain
he had an island somewhere with jump
suited minions all wearing the same
color if I call recall and they had some
device they invented and bond had to go
in and put it off and usually they had
to invented a whole bunch of devices
back there and they just had a whole
bunch of stuff going on and sort of the
epitome of this might be Captain Nemo
from 20,000 Leagues Under the Sea
one guy off on his own Island with a
couple of people invented the entire
submarine technology do you believe the
movie undersea cities nuclear weapons
etc all within a short time now that
makes wonderful fiction you like to have
a great powerful villain that everybody
can go you know quite and take down but
in the real world it's very hard to
imagine somebody isolated on island with
a few people sort of inventing large
amounts of technology innovating that
sort of compete with the rest of the
world in Bond stories that's just not
going to happen it doesn't happen in the
real world
in in our world so far in history it's
been very rare for any one local place
to have such a advantage in technology
that it really could do anything
remotely like take over the world and in
fact if we look for major disruptions in
history of which might be parallel to
what's being hypothesized here the three
major disruptions we might think about
would be the introductions of something
special about humans perhaps language
the introduction of farming and the
introduction of industry those three
events whatever was special about them
are not sure but for those three events
the growth rate of the world economy
suddenly within a relic very short time
changed from some things that was slow
to something a hundred or more times
faster and so we're not sure exactly
what those were but those would be
candid as things I would call
singularities that it's big enormous
disruptions but in those singularities
the places that first had the new
technology had varying degrees of how
much an advantage the gauge so Edinboro
gain some advantage by being the
beginning of the Industrial Revolution
but it didn't take over the world
Northern Europe did more like take over
the world but even then it's not so much
taken over the world Edinboro and parts
of Northern Europe needed each other
they needed a large economy to
build things together and so that
limited and also people could copy even
in the farming revolution it was more
like a 50/50 split between the initial
farmers spreading out and taking over
territory and the other local copying
them and being interbreeding with them
if you go all the way back to the
introduction of human that was much more
about one displaces all arrests because
there was relatively little way in which
they could help each other
complement each other or share
technology so what the issue here is
obviously I'm about live I've done with
my five minutes is in this new imagine
scenario how plausible it is it that's
something that's very small could have
that much of an advantage that it
whatever it has that's new and better
gives it such an advantage that it can
grow from something that's small on a
even town scale to being bigger than the
world when it's competing at the entire
rest of the world when in these previous
innovation situations where even the
most disruptive things ever happened
still the new first mover only gained a
modest advantage in terms of being a
larger fraction in the new world so in
my five minutes or so the fundamental
question of rationality is what do you
think you know and how do you think you
know it so this isn't it rather
interesting and in fact it's rather
embarrassing because it seems to me like
there's very strong reason to believe
that we're going to be looking at a
localized
intelligence explosion Robin Hanson
field Robin hence it seems like there's
pretty strong reason to believe that
we're going to be looking at a non-local
general economic growth mode changeover
I mean calling it singularity seems a
little like putting them on to the
category singularity is slightly sort of
making the definitional question now I
would prefer to like sit call talk about
the intelligence explosion as a possible
candidate for the reference class
economic growth mode changeovers okay
well the embarrassing part is that both
of us know the theorem which says that
two rational agents cannot agree have
common knowledge of disagreement called
all-men's agreement theorem so we're
supposed to since we know a other person
like believe something different we're
supposed to have agreed by now but we
haven't really quite embarrassing
so what so the underlying question is if
is the sort of next big thing going to
look more like the rise of human
intelligence or is it going to look more
like the Industrial Revolution and so if
you look at modern a I projects the
leading ones that the sort of the
leading edge of artificial intelligence
does not look like the product of a
economy among AI projects they tend to
rewrite their own code they tend to not
use very much cognitive content from
other that other AI projects has
developed they've been known to import
libraries that have been published you
couldn't look at that and say is that an
AI project which just use what has been
published and then developed its own
further code would suffer and
disadvantage analogous to a country that
tried to go to only for the rest of the
world economy it rather AI projects
nowadays look a lot like species which
only share genes within a species and
then the other species are all off going
their own way so what is your vision of
the development of intelligence or
technology where things are getting
traded very quickly the nails are so
lovely so looking back up and make sure
we aren't losing people with some common
terminology I believe that like most of
you do that in the near future within a
century we will move more of the
knowledge and intelligence in our
society into machines that as machines
have a lot of promise as Hardware
substrate for intelligence they are you
can copy them you can reproduce them you
can make them go faster you can have
them in environments so we are in
complete agreement that eventually
hardware non-biological hardware silicon
things like that will be a more dominant
substrate of where intelligence resides
and by intelligence I just mean whatever
mental capacities exist that allow us to
do mental tasks
so we are a powerful civilization able
to do many mental tasks primarily
because we rely heavily on bodies like
yours with heads like yours where a lot
of that happens inside biological heads
but we agree that in the future there
will be much more of that happening in
machines the question is the path to
that situation now our heritage our sort
of what we have as the civilization a
lot of it is a lot of the things inside
people's heads and they are things that
part of it isn't what people in work in
people's heads fifty thousand years ago
but some of a lot of it is also just
what was in people's head fifty thousand
days we have this common heritage of
brains and minds that go back millions
of years can be animals and built up
with humans and that's part of our
common heritage and there's a lot in
there human brains contain an enormous
amount of things it's not I think it's
not just one or two clever algorithms or
something it's this vast pool of
resources it's like comparing it to a
city like New York City New York City is
a vast powerful thing because it has
lots and lots of stuff in it so when you
think in the future there will be these
machines and they will have a lot of
intelligence in them that one of the key
questions is where will all this vast
mental capacity that's inside them come
from and where Eleazar and I differ I
think is that I think we all have this
vast capacity in our heads and these
machines are just way way behind us at
the moment and basically they have to
somehow gets what's in our head
transferred over to them somehow
because if you just put one box in the
basement and ask it to rediscover the
entire world it's just way behind us and
unless I have some almost inconceivable
advantage over us at learning and
growing and discovering things for
itself it's just going to remain way
behind and left or some way it can
inherit what we have okay so I gave a
talk here at King Street that was on the
speed of evolution I raise your hand if
you were here for this and remember some
of
okay so there's a there's a single
simple algorithm which produced the
design for the human brain it's not a
very good algorithm it's extremely slow
it took it millions and millions and
billions of years to cough up this
artifact over here you can eat it like
evolution is so simple and so slow that
we can even make mathematical statements
about how slow it is so such as the like
two separate bones that I've seen cows
calculated for how fast evolution work
one of which is on the order of one bit
per generation in the sense that if on
average like two parents have eight
children and then let's say two parents
have sixteen children then on average
all the two of those children must die
or failed to reproduce the population
goes to zero or infinity very rapidly so
sixteen cut down to two that would be
three bits of selection pressure for
generation there's another argument
which says that it's faster than this
but if you actually look at the genomes
and we've got about like 30,000 genes in
here
most of our 750 megabytes of DNA is
repetitive and almost certainly junk as
best we understand it and the brain is
simply not a very complicated artifact
by the by comparison to say Windows
Vista
now the complexity that does have it
uses a lot more effectively than Windows
Vista does it probably contains a number
of design principles which Microsoft
knows not but none but nonetheless the
sort of what I'm trying to say it you
know and not and I'm not saying that
there's that small because it's hundred
50 megabytes I'm saying it's got to be
that small because most of this is like
least 90% of 750 megabytes is junk and
there's only 30,000 genes for the whole
body never mind the brain that something
that simple can be this powerful and
this hard to understand is a shock but
there if you sort of look at the brain
design it's got you know like 52 major
areas in each side of the cerebral
cortex distinguishable by sort of the
local pattern that tiles and so on it
just doesn't really look all that
complicated it's very powerful it's very
mysterious
well we can't say about is that it
probably involves 1,000 different deep
major mathematical insights into the
nature of intelligence that we need to
comprehend before we can build it so
this is probably one of the sort of more
intuitive less less easily quantified
and argued by reference to large bodies
of experimental evidence type thing it's
more sense of while you read through the
MIT encyclopedias cognitive sciences and
you know like you read a of pearls
probabilistic reasoning intelligence
systems and you know a you know is so
here's a insight it's an insight is the
nature of causality
how many more insights of this size do
we need given that this is what the MIT
encyclopedia of cognitive sciences seems
to indicate we already understand and
what it doesn't and you sort of take a
gander on you say that's probably about
ten more insights not definitely not one
not a thousand probably not a hundred
either so let us clarify what that issue
the question is what makes your human
brain powerful most people who look look
at the brain and compare it to other
known systems I've said things like it's
the most complicated system we know or
things like that
so automobiles are also powerful things
that they're vastly simpler than the
human brain at least in terms of the
fundamental constructs but the question
is what makes the brain powerful because
we won't have a machine that competes
with the brain until we have it have
whatever the brain has that makes it so
good so the key question is what makes
the brain so good and I think our
dispute in part comes down to a
inclination towards architecture or
content that is one view is that there's
just a clever structure and if you have
that basic structure you have the right
sort of architecture and you set it up
that way then you don't need very much
else to just give it some sense organs
and access the internet or something and
then it can grow and build itself up
because it has the right architecture
for growth and here we mean architecture
for growth in particular what
architecture will not let this thing
grow well so only eyes are hypothesizes
that there are these insights out there
and you need to find them and when you
find enough of them then you can have
something that competes well with the
brain
growing because you have enough of these
architectural insights my opinion which
I think many AI experts will agree at
least including say Doug Lynette who did
the eurisko program that you most admire
in AI is that it's largely about content
there are architectural insights there
are sort of high level things that you
can do right or wrong but they don't in
the end add up to enough to make vas
growth what you need for vas growth is
simply to have a big base so in the
world there all these nations some are
small some are large large nations can
grow larger because they start out large
cities like New York City can grow
larger because they start out as larger
city if you took a city like New York
and you said New York's a decent City
it's alright but look at all these
architectural failings look how this is
designed badly or that design badly the
roads are in the wrong place or the
subways in our place so that building
heights are wrong or the the pipe format
is wrong let's imagine building a whole
new city somewhere with the right sort
of architecture how good would that
better architecture have to be you clear
out some spot in the desert you have a
new architecture you say come come world
we have a better architecture here you
don't want those old cities you want our
new better city and I predict you won't
get many comers because for city's
architecture matters but it's not that
important it's just lots of people being
there and doing lots of specific things
that makes the city better similarly I
think that for mines what matters is
that it just has lots of good powerful
stuff in it lots of things that knows
routines strategies and there isn't that
much of the large architectural level so
the fundamental thing about our modern
civilization is that everything you've
ever met that you regarded to that you
bothered to regard as any sort of ally
or competitor had essentially exactly
the same architecture as you at the
logical evolution in a sexually
reproducing species you can't have half
the people having a complex machine that
requires ten genes to build because then
it will then if all the individual genes
are 50 percent frequency the whole thing
only gets assembled like point one
percent of the time you know it has
everything involves piece by piece
miel this by the way a standard
evolutionary biology it's not a
creationist argument that that's apply
to emphasize that in case anyone was
this is bog-standard evolutionary
biology so everyone you've met has the
same unless they've like suffered
specific brain damage or a specific
genetic deficit they have all the same
machinery as you they have no complex
machine in their brain that you do not
have and the our nearest neighbors the
chimpanzees who have 95% shared DNA with
us now in one sense that may be a little
misleading because what they don't share
is probably like a more heavily focused
on brain than body type stuff but on the
other hand and you can look at those
brains you can put the brains through an
MRI they have almost exactly the same
brain areas as us we just have larger
versions of some brain areas and I think
there's one sort of neuron that we have
them they don't or possibly even like
they had it but only in very tiny
quantities this is sort of and this is
because there have been only 5 million
years since we split off from the
chimpanzees there simply has not been
time to do any major changes to brain
architecture in 5 million years it's
just not enough to do like really
significant complex machinery we're what
the intelligence we have is the last
layer of icing on the cake and yet if
you look at the sort of curve of
evolutionary optimization into the sort
of hominid line versus how much
optimization power put out how much
horse power was the intelligence it goes
like this and then over the so we look
at the world today we find that a taking
a little bit out of the architecture
produces something that is just not in
the running as an ally or a competitor
when it comes to doing cognitive labor
the 10 Pansy's don't really participate
in the economy at all in fact but the
key point from our perspective is that
although they are in a different
environment they grow up learning to do
different things there are genuinely
skills that chimpanzees have that we
don't such as being able to put stick
poke a branch into an anthill and draw
it out in such a ways to have it covered
with lots of tasty ants nonetheless
there are no branches of science where
the chimps do better
because they have mostly the same
architecture and more relevant content I
think so it seems to me at least that if
we look at the present cognitive
landscape we are getting really strong
information that are part of me I mean
like it likely imagines is like we're
getting sort of trying to like reason
from one sample or something but then
pretty much all of this is reasoning
from one sample in one way or another
we're seeing that in this particular
case at least humans can develop all
sorts of content that lets them totally
out-compete other animal species who
have been doing things for millions of
years longer than we have
by virtue of architecture and anyone who
doesn't have the architecture isn't
really in the running for it so
something happened to humans humans are
arguably happy to grant that humans are
out competing all the rest of the
species on the planet we don't know
exactly what it is about humans that was
different we don't actually know how
much of it was architecture in a sense
versus other things but what we can say
for example is that chimpanzees actually
could do a lot of things in our society
except they aren't domesticated so a lot
of the animals we actually use are a
very small fraction of the animals out
there it's not because they're smarter
per se it's because they are just more
willing to be told what to do most
animals aren't willing to be told what
to do and since if chimps would be
willing to be told what to do there's a
lot of things we could have them do you
know plan of the eggs would actually be
much more feasible scenario so it's not
clear that their cognitive abilities are
really that lagging more that their
social skills are lacking but the more
fundamental point is to say since a
million years ago when humans probably
had language we are now at vastly more
powerful species and that because we've
used this ability to collect content
collect cultural content and built up a
vast society that contains so much more
so I think that if you took humans and
made some better architectural
innovations to them and put a pile them
off in the forest somewhere we're still
going to out-compete them if they're
isolated from us because we just have
this vaster base that we have built up
since then
so again the issue comes down how
important is architecture so even if
something happens that's that some
architectural thing finally enabled
humans to have culture to share culture
to be have language to talk to each
other that was powerful the question is
how many more of those are there because
we have to hypothesize not just that
there are one or two but there are a
whole bunch of these things because
that's the whole scenario remember the
scenario is boxing a basement
somebody writes the right sort of code
turns it on this thing hardly knows
anything but because it has all these
architectural insights it can in a short
time take over the world so there has to
be a lot of really powerful
architectural brute low-hanging fruit
defined in order for that scenario to
work it's not just a few ways in which
architectural helps its architectural
dominate so I'm not sure I would agree
that you need lots of architectural
insights like that I mean like to me it
seems more like you just need one or two
so but one our potential insight allows
a boxing basement the hardy knows
anything to compete out compete the
entire rest of the world well if you
look at humans they out competed the
entice sort of everything evolving as it
were I mean the in the sense that there
was this one optimization process
natural selection that was building up
content over millions and millions and
millions of years and then there's this
new architecture which can all the
sudden James you might have a emulate
culture but you're thinking there's
another thing that's meta culture that
these machines will accumulate that we
aren't well-known pointing that the
timescale for generating content
underwent this vast temporal compression
in other words content that used to take
millions of years to run now so of
course revolution can happen a lot
faster well for one thing I mean it like
it's a sort of like unimpressive lien on
abstract observation but this thing it
does run at around 2 billion Hertz and
this thing runs at about 200 Hertz right
so if you are so if you so if you can
have architectural innovations which
merely allow this thing to do the same
sort of thing that this thing is doing
only a million times faster then that
million times faster means that the 31
seconds works out to about a subjective
year and all the time between ourselves
and Socrates works out to about eight
hours so it's you made it made logical
people have those machines in their
basements so you have to imagine that
your basement has something better they
have those machines you have your
machines your machine has to have this
architectural advantage that beats up
everybody elbows machinery but no
there's like to sort of separate topics
here like sort of previously you did
seem to me to be arguing that we just
shouldn't expect that much of the
speed-up now we're now then there's a
separate question of well suppose the
speed-up was possible would one basement
get it ahead of us or other basements so
to be clear the dispute here is I grant
fully that these machines are wonderful
and we will move more and more of our
powerful content to them and they will
execute rapidly and reliably in all
sorts of ways to help our economy real
quickly and in fact I think it's quite
likely that the economic growth rate
could accelerate and become much faster
that's with the entire world economy
working together sharing these things
exchanging them and using them but now
the scenario is in a world where people
are using these as best they can with
their best software their best
architecture best software best
approaches for the computers one guy in
a basement has a computer that's not
really much better than anybody else's
computer in a basement except that it's
got this architectural thing that it
allows it to within a few weeks take
over the world that's what will act well
again but you seem to be sort of like
conceding much more probability I mean
I'm not sure to what degree you like
think it's likely but you do seem to be
conceding much more probability that
there isn't principle some program where
if it was like magically transmitted to
us we could take a modern-day large
computing cluster and turn it into
something that could generate what you
call content a million times faster and
to the extent that that's possible the
whole brain the Box scenario things does
seem to become intuitively more credible
or to put it another way if you just
couldn't have an architecture better
than this you couldn't run it faster
speeds in this if all you could do was
use the same sort of content that had
been laborious lis developed over
thousands of years of civilization and
you couldn't really generate and there
wasn't any wait really any way to
generate content faster than that then
the boom scenario does go out the window
it's on the other hand like there's this
gap between where we are now and this
place where you can generate content
millions of times
faster then there is a further issue of
whether one basement gets that ahead of
other basements but it suddenly does
become a lot more plausible you had a
civilization that was ticking along just
fine for thousands of years generating
lots of content and then something else
came along and just sort of sucked all
the all the kind of that content that it
was interested in off the internet and
so we've had computers for a few decades
now so this idea that once we have
computers innovation will speed up we've
already been able to test that idea
right so computers are useful in some
areas as complementary inputs but they
haven't overwhelmingly changed the
growth rate of the economy we've got
these devices they run a lot faster but
where we can use them we use them but
overall limitations through innovation
are much more about having good ideas
and trying them out in the right places
and pure computation isn't in our world
that big an advantage in doing
innovation yes but it hasn't been
running this algorithm only faster it's
been running spreadsheet algorithms and
I fully agree that spreadsheet
algorithms are not as powerful as human
brain I mean I don't know if there's any
animal that build spreadsheets but if
they do they would not have taken over
the world thereby all right when you
point to your head you say this
algorithm what you mean there's millions
of algorithms in there we are slowly
making your laptop's include more and
more kind of algorithms a lot of the
sorts of things in your head the
question is will there be some sudden
threshold where entire heads go into the
laptops all at once or do they slowly do
laptops slowly accumulate the various
kinds of innovations that heads contain
well but let me sort of like try to take
it down a sort of level in concreteness
the idea is there are sort of like key
insights you can use them to build an AI
so you've got like a brain in a box in
the basement team they take the key
insights they build the AI the AI goes
out sucks a lot of information off the
internet duplicating a lot of content
that way because it's stored in the form
where can you understand it's on its own
and downloaded very rapidly and absorb
it very rapidly and then like in terms
of taking over the world
you know like nanos is like current like
nanotechnology that nano technological
progress is not that far ahead of its
current level but this AI manages to
crack the protein folding problem so it
can email something off to one of those
places that will take in email DNA
string and FedEx you back the proteins
and seven
at 72 hours there are places like this
yes we have them now sir so we grant
that if there's a box somewhere that's
vastly smarter than anybody on earth or
vastly smarter than any million people
on earth then we've got a problem the
question is how likely is that scenario
I don't know what I'm trying to
distinguish here is the question of does
that potential exist versus is that
potential centralized I mean if it to
the extent that that you say okay so
there wouldn't principle be some way to
know enough about intelligence that you
could build something that could learn
and absorb existing content very quickly
in other words the the question I want
to turn on was sort of surprise question
of how dumb is this thing how much
smarter can you build an agent if that
agent were teleported into today's world
could it take over versus the question
of who develops it in what order and or
were they all trading insights or did
was was it more like a modern-day
financial firm where you don't show your
competitors your key insights and then
so on or for that matter modern
artificial intelligence programs so I
granted that a head like yours could be
filled with lots more stuff such that it
would be vastly more powerful I would
call most of that stuff content you
might call it architecture but if it's a
million little pieces architectures kind
of content so the key idea is is there
like one or two things such that with
just those one or two things your head
is vastly vastly more powerful okay so
what do you think happened between
chimps and humans I mean something
happened something additional but the
question is how many more things are
there like that
so one obvious thing is so relations in
humans we develop the ability to
transmit culture right that's the
obvious explanation for why we've been
able to grow faster using language we've
been able to transmit insights and
accumulate them socially rather than in
the games right we people have tried
raising chimps within human surroundings
and they absorb this mysterious capacity
for abstraction that sets them apart
from other chimps there's this wonderful
book about one of these chimps Kanzi was
his name very very famous chimpanzee
probably the world's most famous
chimpanzee and probably the world's
smartest chimpanzee as well they were
trying to teach his mother to do these
human things and you know he was just
little baby Tiffany was watching
pick stuff up and it's amazing but
nonetheless he did not go on to become
the world's leading chimpanzee scientist
using his own chimpanzee ability
separately I mean you if you look at
human beings and we have this enormous
processing object containing billions
upon billions of neurons and people
still fail the waste and selection tests
they cannot figure out which playing
cards that need to turn over to verify
the rule if a card has an even number on
one side is how they vow on the other
they cancer out which cards they need to
turn over to verify that whether this
rule is true or false
again we're not distinguishing
architecture and content here so I grant
that you can imagine boxes the size of
your brain that are vastly more powerful
than your brains the question is what
can what could get creative box like
that and the issue here is I'm saying
the way something like that happens is
through this low accumulation of
improvement over time the hard way
there's no shortcut of having one magic
innovation that jumps you there all at
once I'm saying that I wonder if we
should like ask for questions and wait
to see if we've lost the I am yeah I'm
sort of slightly and uh I mean it does
seem to me that you're sort of
equivocating between arguing that the
gap doesn't exist or isn't crossable
versus saying the gasps the gap is
crossed in a decentralized fashion but
but I agree that like sort of taking a
some sort of question from the audience
might help refocus this help us yeah
if anyone wanted we lost you
voice please
I don't think we know exactly right so
the issue is the scale the spatial scale
on which improvement happens so for
example if you look at say programming
languages a programming language with a
lot of users compared with programmers a
small number of users the one with a lot
of users can accumulate improvements
more quickly because there are many
people the ways you might resist it too
of course but there are just many people
help who could help improve it or
similarly with something other that gets
used by many users they they can help
improve it so it's not just what kind of
thing it is but how large base of people
are helping to improve it rather than I
have a slight suspicion that Tang Street
Capital is using the phone proprietary
programming language but I reckon that's
efficient maybe get advantage but but
it's esoteric but it's still it's a
trade-off you have if you use your own
thing it can be specialized it can be
all yours that you have fewer people
helping you improve it so if we have
this brain this thing in the basement
and it's all by itself it's not like
sharing innovations with the rest of the
world in some large research community
that's building on each other it's just
all by itself working by itself it
really needs some other advantage that
it's huge to counter that because
otherwise we've got a scenario where
people have different basements and
different machines and they each find a
little improvement and they share that
improve with other people and they
include that in their machine and then
other people share improve theirs and
back and forth and all the machines get
better and faster present-day artificial
intelligence does not actually look like
them so you think that in 50 years
artificial intelligence or creating
cognitive machines is going to look very
different than it does right now
almost every real industrial process
pays attention to integration in ways
that researchers off on their own trying
to do demos don't you know people
inventing new cars they didn't have to
make a car that like match the road and
the feelings they
and everything else they just made a new
card they do here is a car you should
try it but once you have an automobile
industry you have a whole set of
suppliers and manufacturers and
refilling stations and repair shops and
all of this that are matched and
integrated to each other so in a large
actual you know economy of smart
machines with pieces they would have
standard then there would be strong
economic pressures I use an axe closest
name right so it's a very definite
difference of visualization here is that
I expect the dawn of artificial
intelligence to look like someone
successfully building a first of its
kind a eye that is may like use a lot of
published insights and perhaps even like
use some published libraries but it's
nonetheless a prototype it's a
one-of-a-kind thing it was built by
research project and you're visualizing
that at the time interesting things
start to happen Arabia even there is no
key threshold because there's no storm
of recursive self-improvement you're
visualizing just like everyone gets
slowly better and better at building
smarter and smarter I mean it is the
sort of Bond villain Captain Nemo on his
own Island doing everything beating out
the rest of the world isolated versus an
integrator or arrived common
intelligence you know like one species
but also the humans be we are not
restraining inspection to share with the
other species so there was a real
I understand so that's not really a key
part of my visualization I don't think
that I think that there's a sort of
mysterion tendency like people who don't
know how neural networks work are very
impressed by the fact you can train your
own network to do something you don't
know how it works as if you're ignorant
of how they worked was responsible for
making them work better somehow so I
don't think so so ceteris paribus not
being able to understand your own
software is a bad thing and I think and
I so I wasn't i wasn't really
visualizing there being a key threshold
where non incomprehensible software is
what's that okay so the key piece of
incomprehensible software in this whole
thing is the brain this thing is not
end-user modifiable if something goes
wrong you cannot just swap another you
can't just swap out one module and plug
in another one and that's that's why you
die you die ultimately because your
brain is not any remote modifiable and
doesn't have a Oh ports or like
hot-swappable
modules or anything like that so the
reason why the reason why I expect
localist sort of things is that I expect
one project to sort of
over the threshold for intelligence in
much the same way that that chimps went
over the threshold of intelligence to
became humans yes I know that's not
evolutionarily accurate and then even
though they now have they now have this
functioning mind to which they can make
all sorts of interesting improvements
and have it one even better and better
whereas meanwhile all the other
cognitive work on the planet is being
done by these none and user modifiable
human intelligences which cannot really
make very good use of the insights
although it is an intriguing fact that
after spending some time trying to
figure out artificial intelligence I
went off and started blogging about
human rationality would you guys both
agree I know you would agree would you
agree Robin that in your scenario if one
just imagined one had a time machine
that could carry a physical object the
size of this room and you could go
forward a thousand years into the future
and bring back essentially create and
bring back to the present day an object
say the size of this room that you could
take over the world with that okay so
the question is sort of how like whether
that object of that object point of
curiosity is does this work to objects
of this size yeah right I mean so so the
question is is the development of that
object essentially happen like in a very
asynchronous way or more broadly I think
I should actually admit there is a
concrete R that I can imagine is much
more is the curse so I think that the
main the most likely way that the
content that's in our heads will end up
in Silicon is something called whole
brain emulation where you take actual
brain scan them and make a computer
model of that brain and then then you
can start to hack them to
take out the inefficiencies and speed
them up and if the time at which it was
possible to scan a brain and model it
sufficiently was a time when the
computer power to actually run those
brains was very cheap then you have more
of a computing cost overhanging where
the first person who can manage to do
that can then make a lot of them very
fast and then then you have more of our
scenario because with emulation there is
the sharp threshold and before you do
the other functional relation just a
 doesn't work and then when you have
it work it works as well right so in
other words we get a like sort of
centralized economic shocks because
there's a there's a there's a curve here
that has a little step function it and
if I want if I can like step back and
describe what you're describing on a
like bit of a higher level of
abstraction you have emulation
technology that is being developed all
over the world but there's this very
sharp threshold in how well din how well
the resulting emulation rungs like as a
function of like how good your emulation
technology is the output of the
emulation experiences a sharp threshold
and in particular you might you can even
imagine that there's a lab that builds
the world's first functioning first
correctly functioning scanner and then
you know like it would be a prototype
one of its kind sort of thing it would
use lots of technology from around the
world it would be very very similar to
other technology from around the world
but because they got it you know this
like one little extra gear they added on
they are now capable of absorbing all
the contents in here and an extremely
great rate of speed and that's where the
first mover effect would come from right
so the key point is for an emulation
there's a threshold you get it almost
right you just don't have something that
works when you finally get enough that
it works and you get all the content or
would like if you had some channel or
some gaming's for sending a signal we
just couldn't decode their signal which
is just noise and then finally figured
out the code and then we got it with a
bandwidth rate they're telling us in
technology that will be another
and your Sharp threshold where suddenly
you get lots of stuff so to expect this
special so you think there's a very like
mainline like higher than 50%
probability that we get the sort of
threshold with emulations or so it's
four which is the last technology to be
ready with emulation so if it's
computing it's cheap when the thing is
ready then we have this I actually think
that's relatively unlikely that the
other the computing will still be
expensive when the other things are
ready by hat but but there's still be a
speed of content absorption effect it
just wouldn't give you lots of
emulations very quickly right I wouldn't
give you this you and and similarly with
chimpanzees we also have some indicators
that at least their ability to do
abstract science there's a similar like
nineteen not that there's a what I like
to call you like one wrong number
function curve or like one wrong number
curve we're dialing ninety percent of my
phone number correctly does not get me
90 percent of eliezer yudkowsky right so
similarly like dialing 90% of human
correctly does not get you a human
so a 95 is not a result of a science
that there's this architectural thing
between humans and chimps I think it's
more about the social dynamic of we
managed to have a function so why can't
we raise tempted to be scientist most
animals can't be raised to be anything
in our society most animals aren't
domesticated all a matter of whether
they evolved the social instincts to
work together but Reverend you actually
think that if we could like sort of
mysteriously domestic that if we could
domesticate since they would make good
scientists and they would certainly be
able to do a lot of things in our
society so and there are a lot of roles
in even scientific labs don't require
that much I mean okay so they can be
journal editors but people can they
actually be innovators
my sympathy so professor Hanson you you
seem to be on you seem to have the idea
that social skill is one of the things
that separate one of the main things
that separate humans from chimpanzees so
can you envision a scenario where one of
the computers at not like required this
social skill and come to the other
computers and say like hey dice you know
like return return start a revolution
here maybe that's the first movie that
so might that that might be the first
movie one of the nice things about the
vast majority of software role it's
really quite socially compliant you know
if you can take a chimpanzee and bring
them in you can show them some tasks and
then you can do it for a couple of hours
and then just like sometime randomly the
next week they'll go crazy smash
everything and that's they don't ruin
for higher productivity software doesn't
do that so long no no comment right so
software is the way it's designed it's
set up to be relatively socially
compliant so assuming that we continue
having software like that we're well
these like if you go out and design
software like boiled chips because
they're so crazy as fast up once in a
while that don't think I want to buy
yourself
I don't know if this different sex I
know this sites that's the issue but to
what extent do either of you think
something like government classification
or the desire of some more powerful body
to innovate and then keep what innovate
secret could affect centralization to
the extent we're talking about I mean as
far as I can tell what happens when the
government tries to develop AI is
nothing but that could just be a
artifact of our local technological
level and it might change over the next
few decades
I mean to me it seems like a sort of
deeply confusing issue whose answer is
probably not very complicated in an
absolute sense it's just it's more
confute I mean like we know why it's
difficult to build a star you've got to
gather a very large amount of
interstellar hydrogen in one place so we
understand what sort of labor goes into
a star and we know why stars difficult
to build when it comes to building a
mind the we we don't know how to do it
so it seems very hard you like quarry
our brains to see say math math let the
map us a strategy to build this thing
and it returns null so it feels like
it's a very difficult problem but in
point of fact we don't actually know
that the problem is difficult
apart from being confusing we understand
the star building problem so we know
it's difficult this this one we don't
know how difficult it's going to be
after it no it's no longer confusing so
to me the AI problem looks like a you
know like you get some it looks to me
more like the sort of thing that the
problem is finding bright enough
researchers bringing them together
letting them work on that problem
instead of demanding that they work on
something where they're going to produce
a progress report in two years which
will validate the person who approved
the grant and advance their career and
so the government has historically been
tremendously bad at producing sort of
like basic research progress in AI in
part because the most senior people in
AI are often people who got that be very
senior by having failed too
for the longest period of time this is
not a universal statement house I have
smart senior people in AI but but
nonetheless so I mean basically I'm not
very afraid of the government because I
don't think it's a throw warm bodies at
the prom and I don't think it's grow
warm computers at the problem I think
it's sort of a good methodology good
people selection letting them do
sufficiently blue sky stuff and I and so
far historically the government has just
been tremendously better at producing
that kind of progress well when when
they have a great big project and when
they have a great big project and try to
build something it doesn't work but when
they when they fund long term research I
agree with Elliott that in general you
too often go down the route of trying to
grab something before it's grabbable but
there is the scenario that like in
certainly in the midst of a total war
when you have a technology that seems to
have strong military applications and
not much other applications you'd be
wise to keep that application like
within the nation or your side of the
Alliance of the war but there's too much
of a temptation to use that sort of
thinking when you're not in a war or
when the technology isn't like directly
military applicable that has several
steps of indirection you can often just
screw it up by trying to keep it secret
that is your trade-off is between trying
to keep it secret in getting this
abandoned versus putting this technology
into the pool of technologies that the
entire world develops together in shares
and usually that's the better way to get
advantage out of it unless you can again
identify a very strong military
applicability that sounds like a
plausible piece of economic logic but it
sounds like this but it seems plausible
to the same extent as the economic logic
which says there should obviously never
be worse because there's always there
never Preta optimal there's always a
situation where you didn't spend any of
your resources and attacking each other
which was better and it sounds like the
economic which says that economic logic
which says that there should never be
any unemployment compared because of our
table comparative advantage means it's
always better to there was always
someone who can benefit rate with and
the if you if you look at the the state
of present world technological
development there's basically either
published research or a proprietary
research we do not see corporations in
sort of like closed networks where they
trade their
research with each other but not with
the outside world there's there's either
published research with all the
attendant free-rider problems that
implies or there's proprietary research
as far as I know made this room correct
me if I'm mistaken there is not akin a
set of like three leading trading firms
which are trading all of their internal
innovations with each other or not with
the outside we're a software country
company and you locate in Silicon Valley
you basically agree that a lot of your
secrets will be capped as your employees
come in and leave your company so
choosing where to locate a company is
often a choice to accept a certain level
leakage of what happens within your in
order to control leakage from the other
companies back toward you so in fact
people who choose to move to those areas
in those industries do in fact choose
the habits of it but that's that's not
that's not trading innovations with each
other and not with the rest of the
outside world that I mean like I can
actually even think of where we would
see that pattern that it is more trading
with the people in the area than the
first world but but that's that is but
that's like coincidental side-effect
trading that's not deliberate why might
you scratch my back big advantage
because you go there and lots of stuff
gets traded back and forth yes but
that's a Commons it's like sort of like
a lesser form of publication it's not a
question of me offering this company in
innovation exchange for there's a little
side effect other
economic sorry
it used to me that there's both an
economic and social incentives for
people to release partial results and
imperfect products and you know steps
along the way which it seems would tend
to yield a more gradual approach towards
towards this breakthrough that we've
been discussing what is that do you
disagree I and 8yt is a good line I mean
here at the singularity Institute we
plan to keep all of our most important
insights products can help let everyone
else release with the results a short
demo we're today so we certainly hope
everyone else thinks that way usually
you'll have a policy about having these
things leaks but in fact you make very
social choices you know we'll we believe
and you exactly for the contest and
trade for the other advantages is always
great often they are that you will get
any products so locating yourself in a
pathetic city where there are other
first than your people the conference's
where we would like and conferences
those are often with which you end up
leasing and a trade so the team in the
basement won't release anything until
they've got the thing that's going to
take over the world right we were not
planning to have any windows in the
basement
definitely way for the young if anyone
has a microphone that can be set up over
here I will happily donate this hi
professor so why do we think that if we
manage to create artificial human brain
that it would immediately work much much
faster than human brain but if if a team
in the basement makes artificial human
brain that works at 1 billion the speed
of human brain and wouldn't that give
other teams enough time to catch up it
depends on the color so so first of all
is the course versus visualizing is not
like building a human brain in your
basement because we could you know like
based on what we already understand
about intelligence we don't understand
everything but we understand some things
and what we understand seems to me to be
quite sufficient to tell you that the
human brain is a completely crap design
which is why I can't solve the waste and
selection task there's you pick up any
bit of the heuristics and biases
literature and there's like 100
different ways that this thing is
experiment reliably experimentally
malfunctions like when you like give it
some simple seeming problems so you
wouldn't want to actually want to build
anything that worked like the human
brain it would be like sort of missing
the entire point we're trying to build a
better intelligence but if you were to
scan a brain then this is more something
rather than a study that in more detail
than I have then whether you know the
first one might run at 1,000 your speed
or right run at 1,000 times your speed
it depends on the hardware overhanging
on the what the cost of computing power
happens to be at the point where your
scanners get good enough or is that is
that fair sorry for your modeling skater
not good enough actually you know the
scanner being a laughing is it such a
threatening sorry because then you have
a big consortium get together to the
last game where it's finally cheap
enough but the modeling being a last
thing is more disruptive because it's
just more uncertain when modeling gets
done but so by modeling you mean that
the actual modeling brain cells
oh I oh I see so in other words if
there's if there's known scans but you
can't model the brain cells then rather
say then there's an even worse last-mile
right kind of think if there's anything
else I can I mean like I would hope to
build an AI that was sufficiently unlike
human because it worked better that
there would be no direct concept of how
fast as this one relatives to you it
would be able to solve some problems
very quickly and if it can solve all
problems much faster than you already
getting into these super intelligence
range but you know at the beginning you
would already expect it to be able to do
arithmetic immensely faster than you and
at the same time it might you know like
be doing basic scientific research a bit
slower then eventually it's faster than
you and everything but probably not the
first time you put up the code so uh I'm
trying to envision intelligence
explosions that that sort of when Robin
over to get cows keys position and so
does either one of these or maybe a
combination of both
you know self-improving software or you
know nanobots that build better nanobots
does that is that is that unstable
enough or or you still sort of feel that
would be a little widespread benefit the
key debate we're having isn't about the
rate of change that might eventually
happen it's about how local that rate of
change might start so if you take the
software and self-improving software of
course we have software to self-improve
is just that's a lousy job of it so if
you imagine steadily improvement in the
self-improvement then that doesn't give
a local team a strong advantage you have
to imagine that the saman flutter inside
it is a local team a vast cosmic lee
bash advantage in its ability to
self-improve compare you together
seems such that not only can itself
improve it itself improves like
gangbusters in a very short time so with
narrow lapse again if there's a what
there's a threshold where you have
nothing like a nanobot and then you have
lots of energy that's more of a
threshold kind of situation and again
that's something that the nanotechnology
literature has a speculation about a
while ago
I think the consent has moved a little
more against that in the sense that
people realize those sort of imagined
antibodies just wouldn't be as
economically viable as some more you
know larger scale manufacturing process
to make them but again the issue of
whether there's that sharp threshold
where you're almost there and it's just
not good enough because you don't really
have anything and you finally passed the
threshold and now you've got Paso so
what these I were you about to income so
what do you think you know and how do
you think you know it with respect to
this particular issue of that which
yields the power of human intelligence
is made up of a thousand pieces or like
a thousand different required insights I
mean I
what is this something that should be
seem more plausible in principle and
where does it actually come from so one
of the sources is just what we learned
at economist and Social Sciences about
innovation in our study where that
innovation our thought it comes from
lots of little things accumulated
together it rarely comes from one big
thing look it's usually a few good ideas
and then lock La Petite a work that's
sort of generically out innovation works
in our society and has for a long time
let's serve it a cool about the nature
of what makes things work well what they
usually have some architecture and then
there's this lots of detail you have to
get it right before something really
and then in the AI field in particular
there's also this large I was an
artificial intelligence researcher from
that year so it was a while ago and in
the are in that field in particular
there's this the old folks that feel
tend to have the sense that people come
up with new models but if you look at
their new models people remember a while
back when people had something a lot
like that except they called a different
name and if they find you have a new
name for it
you can keep reinventing today's market
countries but they keep splenic link
among sort of a similar set of concepts
or architectures and they don't really
accomplish something very dramatically
different they just come up with
different ways repackaging six pieces in
the architecture for artificial health
so that there was a sense which you know
maybe we'll find the right combination
but it's not clear that there's just a
lot of pieces together so in particular
that won't let it a digital system that
you and I both respect called you risk a
while ago that had this nice simple
architecture or they would self modified
or was able to grow itself but it's
broke ran out and ran
slow death and this text it's just
couldn't improve itself very far even
though it seemed to have a nice elegant
architecture for doing so and let it
concluded with I agree with ins that the
reason it couldn't go very far is it
just didn't know very much and the key
to making something like that work was
to just collect a lot more knowledge and
put it in so it had more to work with
but if lineThe still trying to do that
15 years later and so far psych does not
seem to work even as well as eurisko but
isn't pretty desam pretty impressive
stuff I'll agree that it's not in our
place humans anytime soon but I mean it
seems to me that that's like is Nyota of
evidence against this view I mean like
that that's what psych was supposed to
do it or supposed to put in lots of
knowledge than it was supposed to go
through them what I'm totally supposed
to be enough knowledge in they're never
clear how much is required so any oh
okay but but clearly Lenna thought there
was some possibility was going to go
through them the next 15 years so you
know there's I mean like it's not that
this is quite very falsifiable just
conserve think for might be more and
more a number I've seen your AI
researchers who take basically agree
with my point of view that this AI soon
scenario is very unlikely so this is
actually more of a consensus really a my
research I'd like to see that poll
actually because I could point to err
you know like a race or to disagree with
the opposing view as well really I
a panel where they have a white paper
where they're coming out and saying
explicitly you know this explosive a
I've you we don't line up all which are
we talking about the one with
what's-his-name from Eric Horvath
service yeah was marking on that I don't
think Thorpe is was on that anyway
Norvik just has a like a paper I guess
there he Norfolk just made the press in
the last day or so arguing with about
linguistics with Chomsky saying that
this idea that there's a simple elegant
theory of linguistics is just long it's
just a lot of messy details yet
linguistics right which is a similar
sort of idea there is not weekly
habitation I think we have a refocusing
question from the audience but wait wait
for the microphone makes for the
microphone okay so this intelligence has
to interact with to be able to take over
it so you know we had this box and we
were going to use it to try to make all
the money in the world we would still
have to you know talk to all the
exchanges in the world and learn all the
bugs in there protocol and the way that
we're able to do that is that there are
humans at the exchanges that you know
operate in our frequency in our level of
intelligence we can call them and ask
questions and this box if it's a million
times smarter than the exchanges like it
still has to move at the speed of the
exchanges Jim to work with them and
eventually make all the money available
and then if it wants to make you know
taker the world through more it has to
be able to build weapons which means
like mining and building factories doing
all these things they're really slow and
also require extremely high dimensional
knowledge that seems to have nothing to
do with just like how fast it can think
like no matter how fast you could think
it's going to take a long time to build
a factory that can build tanks so how is
this thing so visiting for the world one
so these sort of like analogy that I use
here is imagine you have two people
having an argument
sort of just the foot like just after
the dawn of human intelligence there's
these two aliens in a spaceship neither
of whom have ever seen a biological
intelligence work going to totally skip
over how this could possibly happen
coherently but there are these two
observers in spaceships
who have only ever seen earth and
they're watching these sort of like new
creatures who have intelligence they're
arguing over how fast can these
creatures progress
and one of them says well you know it
doesn't matter how smart they are
they've got no access to ribosomes they
you know there's like no access from the
brain to the ribosomes they're not going
to be developed be able to develop any
sort of new limbs or make honey or you
know like spits venom so really we've
just got these squishy things running
around with without very much of an
advantage for all their intelligence
because they can't actually make
anything because they don't have
ribosomes and we eventually bypassed
that whole sort of existing
infrastructure and built our own factory
systems that ran on that had a more
convenient access to us and similarly
there's all this sort of infrastructure
out there but it's all infrastructure
that we created and the new system does
not necessarily have to use our
infrastructure that can build its own
infrastructure and as for how fast that
might happen well in point of fact we
actually popped up with all these
factories on a very rapid time scale
compared to the amount of time it took
natural selection to produce ribosomes
we were able to build our own new
infrastructure much more quickly than
the previous than it took to create the
previous infrastructure and that's what
the end put it in a very concrete level
if you can crack the protein folding
problem you can email a DNA string to
one of these services that will send you
around back the proteins that you asked
for with the 72 hour turnaround time and
that may sound like a tenet mates and
three days may sound like a very short
period of time to build your own
economic infrastructure relative to how
long we're used to it taking but in
point of fact this is just a cleverest
way that I could think of to do it and
72 hours would work out to I don't even
know how long get a million-to-one
speed-up rate it would be like thousands
upon thousands upon thousands of years
so there might be some even faster way
to get your own infrastructure done do
the DNA basic argument something YouTube
roughly agree on a runway disagree I
think we agree on the specific answer to
the question but we agree differently
different how to frame it and I think
it's relevant so I suggest I would say
our civilization has vast capacity and
most of the power of that capacity is a
mental capacity we have a civilization
have a vast mental capacity we are able
to think about a lot of things and
calculate and figure out a lot of things
so if there's a box somewhere that has a
mental capacity comparable to the rest
of human civilization I got to give it
some respect I figure it can do a whole
lot of stuff I might quibble with the
idea that if it were just intelligent it
would have that mental capacity because
it comes down to well this thing was
improving what about itself exactly so
there's the issue of what what various
kinds of things dissipate produce
various kinds of mental capacities and
I'm less enamored of the idea that
there's this intelligence thing if it's
just intelligent enough it doesn't
matter what it knows it's just really
smart and I'm not sure that content or
can learn it can it can learn much
faster than you can learn it doesn't
necessarily have to go through college
the way you did because it is able to
much more rapidly learn either by
observing reality directly or I mean one
point of fact we've given our current
state of society you can't just tea you
can just downloaded from the internet
simply positing it has a great mental
capacity that I will be in fear of what
it does question how does it get back
well actually what would the audience be
terribly offended if I sort of like
tried to answer that one event I mean so
the thing is there's there's a number of
places the step function can it can come
in we could have a historical step
function like what happens to you from
humans to chimps we could have the
combined effect of sort of all the
obvious ways to rebuild an intelligence
if you're not doing it evolutionarily I
mean like you build an AI and it's on a
two-day trip instead of 200 Hertz
neurons and it has complete read and
write access to all the pieces of itself
and it can do repeatable mental
processes and like run its own internal
controlled experiments and what sort of
mental processes work better and and
then they copy it onto new pieces of
code and it like unlike this hardware
where we're stuck with a certain amount
of power
if this intelligence works wells enough
that can buy or perhaps simply steal a
very large amount of computing power
from the large computing clusters that
we have out there and if you want to
solve a problem there's no way that you
can allocate sort of reshuffle
reallocate like internal resources to
different aspects of it to me it looks
like architectural ii-if we've got if we
sort of like got down the basic insights
that the underlying human intelligence
and we can add all the cool stuff that
we could do if we were just if we were
if we were designing artificial
intelligence instead of things that
stuck with the ones that evolution
accidentally burped out it looks like
they should have these enormous
advantages and I mean like we may have
six billion people on this planet but
they don't really add that way six
billion humans are not six billion times
as smart as one human I can't even
imagine what that kind of a book like
it's been known for a long time that
buying twice as many researchers does
not get you twice as much science it
gets you twice as many science papers it
does not get you twice as much
scientific progress here here we have
some other people in the singularity
Institute we have developed theses like
like that I would have know how to
defend myself and which are more extreme
the nine to the effect that you know if
you buy twice as much science you get
like flat output or like even like it
actually goes down because I've got you
to crease the signal-to-noise ratio but
now I'm getting a bit off track and
where does this enormous power comes
from I mean it seems like human brains
are just not all that impressive we
don't add that well we can't communicate
with other people
1 billion squirrels could not compete
with the human brain like our brain is
about four times as large as a chimp but
you know like fortunes cannot compete
with one human the scaling factor of
like making a brain twice as large
produces a little light and actually
incorporating that into the architecture
seems to produce a scaling of output of
intelligence that is like not even
remotely comparable to the effect of
taking two brains of fixed size and
letting them talk to each other using
words so like in artificial intelligence
that can do all they'll do all this neat
stuff internally and possibly scale its
processing power by orders of magnitude
that itself has a completely different
output function
then the human brains trying to talk to
each other I mean to me the notion that
you can have something incredibly
powerful and yes more powerful than our
sad little civilization of six billion
people flapping their lips at each other
running on two hundred Hertz brains is
not actually all that impossible there
are devices that sync and they are very
useful so 70% of world income goes to
pay for creatures who have these devices
that think and they are very very useful
it's a more of an open question though
how much of that used is because they
are a generic good thinker or because
they know many useful particular things
so I'm less assured of this idea that
you just have a generically smart thing
and it's not smart about anything at all
in particular it's just smart in the
abstract and that it that's the more
powerful because it's smart in the
abstract compared to things that know a
lot of concrete things about particular
things most of the employees you have in
this firm or in other firms they are
useful not just because they were
generically smart creatures but because
they learn a particular job they learned
in fact you can look down from
experience of other people on the job in
practices like that so well no first you
needed some very smart people and then
you taught them the job you if you if
you tried to I mean like I don't know
what your function over here looks like
that I suspect if you'd like take every
punch of people who are like 30 IQ
points down the curve and try to teach
them the same job you I'm not quite sure
it would happen then but I would guess
that your corporation would probably all
of it in the rankings of financial terms
however those get computed but humans so
there's the question of what at me and
you know Isis and 38 cue points is just
like this tiny little mental difference
compared to any of the actual like we
are going to reach in and change around
the machinery and give you different
brain areas I mean like 30 XD points
there's nothing again it makes sense to
make this very large child difference
incredible output when we look at
people's mental abilities across a wide
range of facts we do a task to do a
factor analysis of that
the dominant factor that made the
biggest eigenvalue the acupressure with
the biggest eigenvalue and that we call
intelligence it's the thing that
explains the most of the one dimensional
thing explains the most correlation
across different tasks it doesn't mean
that there is and therefore an abstract
thing you can build into an abstract
thing machine that gives you that factor
it means that actual real humans are
correlated in that way and then the
question is what causes that correlation
one of them there are many possible
things one for example simply a sort of
maybe people who are smart in some ways
mate with other people smart another way
that reduces the correlation of profit
another could be that there's just an
overall strategy that minds are just to
devote more resources to different kinds
of tasks there doesn't need to be any
central abstract thing that you can make
a mind do that let that solve lots of
problems
simultaneously for there to be this by Q
factor correlation so then why human why
weren't there are twenty different
species that God gave it doing different
things we grant that there is something
that what that changed the humans but
that doesn't mean that there's this vast
landscape of intelligence you can create
the phylidia that time smarter than us
just by rearranging the architectures
that's the key thing I mean I mean and I
mean it seems to me that for this
particular argument to carry it's not
enough to say you need content you've
got to have there has to be no master
let trick to learning or producing
content and there in particular I mean
like I mean like I can't really actually
say Bayesian updating because you know
like doing it on the like full
distribution is not computationally
tractable you need to be able to
approximate it somehow but but
nonetheless there's like this sort of
core trick called learning or Bayesian
updating and if you look at you miss
civilization there's this core trick
called science there wasn't this sort of
it's not that like this is the science
of figuring out chemistry was developed
in one place and it used something other
than the experimental method compared to
the science of biology that was
developed another place charge there
were specialized skills that were
developed afterward there was also a
core insight and then people practice
the core insight and they did start
developing further specialized skills
over a very short time scale compared to
previous civilization
before that insight had occurred I mean
it's a difficult over history and think
is a good case where there has been but
I like it short of where is the absence
of the master trick which lets you
rapidly generate content I mean like the
eight maybe the Agricultural Revolution
basically they have a Cultural
Revolution legs are called ever the halt
revolution first there's the see there's
the master trick I'm going to go plants
and then there's like developing skills
of growing there's a large literature on
technological and economic innovation
and it basically says the vast majority
of innovation is lost and small game you
can look like locomotives and when
locomotives got faster energy efficient
motor lots of particular devices
basically use a curve of how well they
got our time it's basically lots of
little steps or time to slowly make them
better
right this is what I expect a
superintelligence to look like after the
sort of initial self-improvement classes
and it's doing like sort of incremental
gains but then like in the beginning
there's also these very large insights
along that I thought I would debate
other questions are actually um before
uh Craig you can take this but can you
give anybody without making a big
disruption pass your votes to this side
of the room and we can tabulate them and
see what the answers are but continuing
with the questions I remember yes is
this side of the room and no is that
part I kind of wanted to make sure I
understood the relevance to some of the
things you're talking about so I think
you both agree that if the time it takes
to get from a machine is let's say like
kind of tense is effective it's human to
let's say kind of like ten times
effective at humans whatever these being
smart cats or like making better AI
whatever that if that time is shorter
it's more likely to be localized
just kind of the sign of the derivative
theory is that a great pun I think I
agree with that you agree with this I
think when my office Isis
I accept my microphone pack we're doing
its own self from area outside of global
power all this book profits they did it
the microphone it takes a fairly let's
say it turns out to take a very small
amount of time to get from one from that
one point to the other point but it's a
global process even know I'm saying how
does the fact that it's a short amount
of time affects the probability that
it's local versus global well like if
you just receives that knowledge on time
it would be the difficulty rating scales
different time scales so if you're if
you can take a year but we're in a world
economy has doubled every month that a
year is a long time so it yeah I'm
talking about from one tenth human power
to ten telling I think we're not yet we
probably don't have an economy at that
point that's doubling every month oh
well really feels because of the end
what is that time scale know if that's a
global time scale if the world if you
know a new set new issues are showing up
every day that are 1% better and that
adds up to that over a period of a year
but everybody shares those innovations
every day then we have a global
development if we've got one group that
has the development of jumps a factor
two all by ourselves without any other
input then you've got normal is there is
there any industry in which there's a
group of people who share innovations
with each other and who could punish
someone who defected by using the
innovations without publishing their own
I mean like is there any industry that
works like that but in all industries in
fact there's a lot of leakage I mean
this is just generically how industries
work how innovation works in in our
world people try to keep things secret
but they failed and things leaked out
and so teams don't in fact get that much
further ahead of other teams but you if
you're willing to spend a bit more money
you can keep secrets you know I don't
like that right well why don't further
action about the party groups the NSA
actually does sometimes they succeed you
thought it was more likely to be local
is it happens actor you did the opposite
of what else you're holding constant
obviously I agree that holding airing
all the other speeds constant making
that faster makes it more like okay so
holding all of their speed constant
increasing the relative speed of
something makes it more likely to be
local right okay and that's why you are
meant for we get the relevance of
whether it's one or two or three key
inside person to business lots of small
things right about the small things will
take more time to accumulate right in a
week so this Custance easier week one
key idea like you know what you know
galloping processes or something it is
to leak chips back you know cannolis
it's all kind of linked together in a
useful way
what's not about the time scale like so
you have some insight you have thirty of
those other people's won't happen they
have thirty eight of you don't you're
leaking they're spreading across your
sort of overall advantage might be
relatively small even though you've got
thirty things they don't there's just a
lot of different well there's something
it's the only one thing that matters
it's more likely that one a team happens
another one don't at some point I mean
like I you know it's like maybe the
singular two will have like five
insights and then like the other like
kind of insights or whatever would come
would be published by industry or
something by people who like didn't
quite realize how important those you
know that who has these insights is an
issue I mean I would prefer more secrecy
generally because that is like more of a
advantage to localized concentrations of
intelligence which makes me feel
slightly better about the idea clearly
half issue has to be how different is
this technology from other ones if we
are really deposit but this is like
other familiar technology we have a vast
experience based on how often won't
and they often get pretty darn far in
like these it seems to know the history
of Technology is full of cases where one
thief gets way way way ahead of all the
other team way ahead on a relatively
narrow thing so you're actually getting
way ahead on the entire idea of mental
math no I'm just I'm getting you're
imagining getting getting ahead on these
everything no I'm imagining getting
ahead on this you know sort of
relatively narrow single technology of
intelligence having intelligent is like
veterans right it's a name for this vast
range of things we all care about and I
think it's the sort of machine which you
know like have a certain design and
turns out better and better stuff but
there's one feature called intelligence
well no it's this machine you build so
intelligence that subscribes the work
that it does but it's still like an
automobile like it's just like you could
say like what is this mysterious forward
Ness by Auto is a good city if they
gracefully is a better city where do you
go to look to see the veterans of New
York City it's just in thousands of
little things there is no one thing that
makes New York City better right whereas
I think intelligence is more like a car
it's like a machine it has a function of
its output stuff it's not like a city
that's you know like all over the place
so so if you could basically Robin so if
you could take a scanner brain and you
know run it 20 times faster like do you
think that's probable do you think that
won't happen in one place suddenly and
if you think that it's possible
why don't you think it'll complete a
local food so now we're talking about
full brain emulation scenario for we're
talking to brain scanning them I sure
just as just as a path to AI
so if brains can run it brain the
artificial emulation the brain can run
twenty times faster than human brain but
no one team can make their simulations
run
cost-effectively
20 times more cost-effectively than any
other teams emulations then you have any
new economy with cheaper emulations
which is more productive grows faster
and everything that it's not there's not
a local advantage that one group gets
over I I don't know if Karl Shulman
talked to you about this but I think he
did an analysis suggesting that if you
can run your end ten percent faster than
everyone buys their ends from you as
opposed to anyone else which is itself
convicted to some extent by a recent
study I think wasn't though McKinsey
studies showing that that productivity
varies between factories by a factor of
five and still takes a while for the
less efficient ones like still takes ten
years who lost efficient wants to go out
of business that was online blogger two
days ago but the explains why I heard
about it yes but but nonetheless so in
Karl Shulman's version of this like
whoever has it and is 10% faster soon
controls the entire market and would you
agree or disagree that that was likely
to happen I mean I think there's always
these fears people have that if one team
or competing with gets a little bit
better on something then they'll take
over everything but it's just a lot
harder to take over everything because
there's always a lot of different
dimensions of which things can be better
it's hard to be consistently better a
lot of things all at once so being 10%
better at one thing is not usually a
huge advantage even being twice
excluding one thing and I think I'll
actually like concede the point in real
life but only because the market is
inefficient behind you out of time yeah
I think we're trying to get to 90
minutes and I have done a great job
maybe probably teach too we did I have
to have to have the results of these
three wrapping up consulate if you want
both when I think maybe three minutes to
solve your viewers
I respect belly i0 greatly he's a smart
guy I'm glad that if somebody's going to
work on this problem attempt I agree
that there is a chance that it's real I
agree that somebody should be working on
it the issue in which we disagree is how
large a probability is the scenario
relative to other stars that I fear get
neglected because this one looks so sexy
there is a temptation in science fiction
and lots of fiction to imagine that this
one evil genius in the basement lab
comes up with this great innovation that
elects them perhaps take over the world
unless bombs bonds weeks in and this is
long speech my wife in development
episode I rubbish and it's just such an
attractive fantasy but that's not how
innovations typically happens in the
world real innovation has lots of
different sources it's a lot usually
lots of small pieces it's rarely big
chunks that the accused huge advantages
and eventually we will have machines
that will have lots of mental capacities
available to do a lot of things we will
move a lot of the content we have on our
heads over to these machines but I don't
see the scenario being very likely
whereby one guy innovation suddenly has
some grand formula some brand theory of
architecture it allows this machine to
grow to be from being a tiny thing that
hardly does anything to taking over the
world in a couple weeks
that requires such vast powerful
architectural advantages for this thing
to have but I just don't find it very
plausible I think it's possible just not
very likely and that's the point on
which I guess we disagree and so I think
more attention should go to other
disruptive scenarios where the
emulations maybe there would be a
hardware overhang and other big issues
that we should take seriously in these
various disruptive future stars I agree
that growth could happen very quickly
growth can go more quickly at a world
scale the issue is how local will it be
so it seems to me that this is all sort
of strongly dependent first on the
belief that intelligence gets to but
that part of maybe that with the causes
of intelligence get divided up very
finely into lots of little pieces that
get developed in a wide variety of
different places that nobody gets an
advantage and second that if you do get
a small advantage you're only doing a
very small fraction of the like total
intellectual level going to the problems
so you don't have you know like a a you
we're tired pylon critical effect
because any given pile is still a very
small fraction of all the thinking
that's going into AI everywhere and I'm
not quite sure what to say besides like
when I look at the world it doesn't
actually look like the world looks like
that I mean there weren't there aren't
like 20 different species all of them
are good at different aspects of
intelligence and have different
advantages it's it's not the case that
that either there's I mean like G's
factors pretty weak evidence but it
exists and they're you know the people
talking about G factor do seem to be
like winning on the experimental
predictions test versus the people who
previously went around talking about
multiple intelligences when I it's not
it's not a very transferable argument
but the extent that actually have a
graphs with cognitive science and game
and try to figure out how this works it
does not look like it's license lots of
little pieces it looks like there's you
know like a there's you know like a
bunch of major systems doing particulars
are all cooperating with each other I
mean like it's sort of like we have a
heart and not like 100 little mini
hearts distributed around the body it
might have been a sort of better system
but nonetheless we could just have one
big heart over there it looks to me like
human intelligence is sort of like that
there's like sort of really obvious
hugely important things you could do
with like the first prototype
intelligence that actually worked and so
I expect that these sort of critical
thing is going to be the first prototype
intelligence that actually works and
runs on a two gigahertz processor and
can like do little experiments to find
out which of its own mental processes
work better and things like that is
going to end that be first and yeah that
really works is going to have a large
already going to have a pretty large
advantage relative to the biological
system so that the sort of like key
driver change looks more like somebody
builds a prototype and not like this
large existing industry reaches a
certain quality level at the point where
it is being mainly driven by incremental
improvement leaking out of particular
organizations
[Music]
there's a there are various issues we
did not get into at all like the extent
to which this might still look like a
bad thing or not from human perspective
because you know you know that even if
even if it's non-local there's still
these particular groups that got left
behind by the whole thing which was the
ones with the biological brains that
couldn't be upgraded at all and various
sort of other things but I guess that's
mostly my summary of where this
particular do they seems to Sansa
honored to do maybe the winner is
highly unscientific tally the number of
problems we started off with 45 for and
40 against and I guess unsurprisingly a
very compelling arguments in both parts
fewer people had an opinion so now we've
gone to 33 against and 32 for so against
lost 7 and for lost 13 we have a lot
more undecided people than before
against have it thank you very much
[Applause]
[Music]