Why Ain't You Rich? - Nate Soares

at this time i'm pleased to introduce
nate sores
a research fellow in the machine
intelligence institute
he works on foundational problems
relevant to the creation of
general intelligence including decision
theory goal stability
corrigability and value loading today he
will present a talk titled
why ain't you rich why our current
understanding of rational choice
isn't good enough for superintelligence
please take a moment to silence your
electronics devices
thank you and enjoy
good afternoon my name is nate sorrys
i'm here to present why ain't you rich
you've already heard the title
i work at the machine intelligence
research institute
we do foundational research to ensure
that smarter than human artificial
intelligence has a positive impact
that is the official slogan uh
i have a more informal version that i
like which is we're trying to figure out
how to ensure that a super intelligence
would have a beneficial impact or trying
to figure that out
before someone turns one on uh i've been
invited here to give a more technical
talk
uh the keynote talk will give um a bit
more of a motivation behind why these
are important things
i'm going to dive into some of the
technical details of why we shouldn't
necessarily expect that we get good
behavior from
from super intelligence by default
let's get this slide out of the way
early
there will not be a q a if you have
i'm using hotseat the number is
hopefully readable
this will be there throughout the talk
they will come to my tablet i will
ostensibly see them and answer them
we'll see if this works
so like i said i'm not going to motivate
why we need to worry
too much that'll be motivated later
but since we're at the dawn of doom
conference
uh i will assume that you're familiar
with the idea that we might one day
build something that is smarter than us
and also with the idea that it is our
smarts
and not our strength or our speed that
gives us as humans
control over the future of the of the
planet
it is our intelligence that allows us to
light up the night
and if we build something smarter than
us
then just as we control the future much
more than do the great apes
something far smarter than humans might
control the future
far more than we do
finally i'm going to assume that you're
familiar with the idea of an
intelligence explosion
that we may one day make a machine that
is better than we are at intellectual
tasks
including the generation of smart
machines
and that if this happens and they make
smarter machines and they make smarter
machines and show and so on it may
quickly lead
to an intelligence that is far far
smarter than any human
that is capable of having an
extraordinary impact on the future
if we want to ensure that that impact
is good it depends entirely upon how
well we can specify the initial machine
the thing that starts the intelligence
explosion
there are a number of open problems when
considering how to build a super
intelligence
that we don't yet understand in order to
to make a super intelligence that we can
ensure would have a beneficial impact
for example uh we don't yet understand
how to build a goal-oriented system
that helps you correct its goals instead
of manipulating and deceiving you in
attempts to defend
to defend its flaws we don't yet
understand
what sorts of processes can successfully
convey
the complex notion of value to
uh on artificial intelligence
and today i'm going to talk about one
specific sub problem
uh in the space of decision theory
now you may wonder why do we need to
worry about decision theory
when ensuring the safety of an
artificial intelligence decision theory
is the study of how to make
appropriate decisions and
it may seem naively that we can trust
this to the superintelligence if you
have some
smart system that's making itself much
smarter surely it figures out how to
make good decisions
right like if you have something that
makes bad decisions
the label's super intelligence probably
doesn't apply
the reason that we worry about uh
decision theory
is it seems possible for decision
theories to be both unstable
in the sense that if you built an ai
using that decision theory it would stop
using that decision theory and
unsatisfactory in the sense that there
are some decision theories
which have blind spots that they cannot
fix
these problems both stem from the
question
of how do you reason counterfactually
when you're choosing between actions
there are a bunch of actions you could
do and there's only one that you're
going to do
when you're evaluating which to take you
have to reason about what would happen
if you took actions that you're not
going to take
this is usually fairly easy we usually
we have algorithms that do this pretty
well but there are some subtle cases
where it's hard
and those subtle cases can lead to
really bad behavior sometimes
that will be the discussion of this talk
i'll illustrate some of these failings
as an example humans are usually pretty
good
at counter factual reasoning we're
usually pretty good at evaluating what
would happen if we did something even if
we're not going to do it
but there are times when we get really
bad at it
this is actually where decision theory
comes from it does it's not an ai field
originally it's a
it's trying to figure out how humans
make good decisions because sometimes
humans make really bad choices
surprising to everybody i know an
example of humans making bad choices
is there are people who believe in
palmistry
that you can get your palm read and it
tells you something about your fortune
and then get palm surgery in order to
alter their palms
and thereby change their fate
i have spared you the pictures of of uh
palm surgeries
because they give me the heebie-jeebies
um but people sometimes make bad choices
now i'm i'm actually a little bit
sympathetic with people who get palm
surgery in order to alter their fate
if you could convince me that palmistry
worked the first thing i would do
is check whether or not i could change
my palm in order to change my fate
it's like step one but most people
who think palmistry works probably have
a world model that looks a bit more like
this
this is what's known as a causal graph
the nodes are events
the arrows represent causality and
throughout this talk
the dashed gray circles will be the
action node
this is where your choice happens it's
where you make your decision
and the dark diamond will be the payoff
node
which describes the outcome so this
graph describes a belief about the world
where there is some destiny thing which
affects both your fate
and the lines on your palm and the lines
in your palm can be read by a fortune
teller who will tell you a fortune
and surgery can change the lines on your
palm
but the fortune that you get doesn't
change your fate if you change your palm
you can change your fortune but you
won't change your fate
and people might reason hey everyone who
gets a good fortune
has good outcomes but if they get the
palm surgery they're failing to reason
that the palm surgery does not cause
their fate to change even if palmistry
works for reasons beyond palmistry not
working
um this leads us to the prescription of
causal decision theory which is the
modern standard decision theory
and prescribes choosing actions based
only upon the causal outcome the causal
effect that your action has
there are many ways of formalizing
causal decision theory i should also
note that regardless of whether or not
you think you're using a decision theory
if you if your counter factual reasoning
if the way you reason about
what would happen is causal
then you implicitly are using council
decision theory and in fact most narrow
ai
today economics statistics
they all do causal counter factual
reasoning and thereby are implicitly
using causal decision theory
i'm going to abbreviate causal decision
theory as cdt throughout the talk
because it's kind of a mouthful
and there are many ways to formalize it
this is the formalization in the
language of causal graphs which is
particularly good for presentations
because i can have pretty graphs
it's a pretty simple algorithm you
identify your action node you identify
all the actions that you can take
you identify your payoff node and then
you go through each action
and you consider causally how the graph
would be affected
if instead of having your action node be
like the decision process that is you
you instead have your your action node
be a simple function that returns the
action under consideration
you overwrite your action node in like
your model of the action node you didn't
actually overwrite yourself
you overwrite your model of the action
node with a function that always returns
the action you're considering you like
see how this affects the causal graph
you take the action the best outcome
the thing to pay attention to here is
that causal counterfactual reasoning
involves evaluating your actions only
based
on what would happen if instead of being
you you were some simple function that
did the action under consideration
this usually works really well this is
why it's standard in economic statistics
ai and so on there's like a famous
problem that was kind of hard for
decision theory back in the day where
cdt was invented to solve it which is
like you have
a gene that causes both liking gum and
stomach ulcers
um and the decision theories of the day
would say well if i chew gum i'll
probably have ulcers because they can't
model this
this uh this confounding variable of the
gene cdt can handle this
um it works great but sometimes it fails
and those failures we're going to look
at today because they make cdt
our modern understanding of a good
decision theory inadequate
for use in super intelligence as we'll
see
in order to explain the failure i'm
going to use a simple game
throughout the rest of the talk this
game is called the token trade
it is a two player game and it works
like this
i take the first player i put them in a
red room
and i hand them a green token i take the
second player
i put them in a green room and i hand
them a red token
and then each player
must decide without communicating
whether or not to give their token away
or to keep it
after they have decided if they have
given their token away
i take the tokens that they give and i
give them to the other player
and then they get to cash out their
tokens the red player gets 200
for the red token and 100 for the green
token the green player gets 200
for the green token and 100 for the red
token
so for example if the green player gives
their token away
and the red player keeps their token
then the green player gets nothing while
the red player gets
300 i'm going to give you a few seconds
to make sure that you've internalized
this game it's just a variant
of the prisoner's dilemma while i go
pick up the fallen tablet
so this is the token trade and we're
going to consider a special variant of
the token trade
called the mirror token trait in the
mirror token trade
an opponent is played against a perfect
copy of themself
now to avoid questions about determinism
and whether or not people can
distinguish red from green
we're going to assume that the player is
a deterministic algorithm
this is a deterministic algorithm that
is going to like be put in this
environment and just a reason in some
fashion
and decide what to do we're going to
assume that it's red green colorblind
so this is a perfectly symmetric game
with a deterministic program
that's playing against a perfect copy of
itself so i'm going to take some
template
the source code of the program i'm going
to instantiate two copies
i'm going to put them in a token trade
and they have to decide whether or not
to keep their token
or give their token away so consider
the agent that wakes up in the red room
they are this this give node that's
their decision node
what should they decide to do
clearly they should decide to trade
their token
because if they keep their token then
sold the opponent and they'll both get
100
but if they trade their token then so
will the opponent and they both get 200
they should give their token away
cdt does not do this
because cdt only reasons about the
causal
implications of its actions
and the other player by the time a cdt
agent is making his decision
the other player is causally distinct
if you are in this situation and you
give your token then so will they but
it's not because you've caused them to
is because of a logical relation between
you and them
cdt can't model these when cdt builds
this environment
it assumes that everything when it
builds this model of the world it
assumes that everything which is
causally disconnected from it is
logically independent from it
and so it thinks that there's some
probability that the other player
will give their token and it says
no matter what the probability is that
they give their token
i get a hundred more dollars if i keep
my token
and so it keeps its token
drastic failures i know it loses a
hundred dollars
um but what went wrong
now some philosophers will argue that
what went wrong is that this is an
unfair game
there's mind reading involved there's a
literal perfect copy which is is
like we don't have literal perfect
copies of of deciding agents
generally and they'll say that it is in
fact rational
to uh to keep your token because
no matter what they do you get 100 more
dollars from keeping for keeping your
token
in response i have a couple responses
the first is that this game is fair
enough for me
if you put me in a token trade against
someone who's guaranteed to do the same
thing that i do
then i will trade my token and i will
walk away with 200
and if you protest that this is
irrational and you keep your token
then you will walk away with the 100
dollars
why aren't you rich presumably because
your
adherence to what you think is rational
is worth more than 100 to you
but i digress
my second objection is that the real
failure the real failure that's
happening here
is that cdt is unable to model logical
counter
logical dependence in these nodes
when the opponent's action is going to
be the same as yours it doesn't make
sense
to reason like no matter what they do i
get 100 more dollars
this sort of reasoning when you say no
matter what the probability is that they
are going to keep their token
this is already broken you're
counterfactual because you're assuming
that they can be distinct from you
when in fact you're logically the same
so the remaining objection is that this
isn't that big a deal
this only happens when someone can read
your mind or when someone can perfectly
predict what you're going to do
there are there are actually a number of
these scenarios in decision theory
they're known as nukem like problems
after some guy newcomb who went and
pissed off a lot of decision theorists
by giving them problems where cdt did
poorly
um and
one of the common objections is that
newcomer-like problems don't matter
i'm going to argue exactly the opposite
newcomb-like problems are the norm
these scenarios occur whenever someone
else in the environment
is basing their action on how they think
you will act
you are causally disconnected from that
right like when you are when you are
making your choice you are causally
disconnected
from their reasoning about your choice
but you're still logically connected to
how
because they're reasoning about you and
in fact any scenario or other people in
the environment
are reasoning about how you reason even
if it's not perfect knowledge
you get one of these nuke like scenarios
and in fact
these are exactly the types of scenarios
that humans find themselves in all of
the time
when you ask someone out on a date you
don't immediately invite them to your
house
you go like meet in a public place and
get to know each other because you want
to understand more
about how like whether or not they're a
trustworthy person
and indeed evolution has supplied us
with
huge amounts of tools to make immediate
split-second first impressions to get an
idea for how other people reason
and we never make big decisions with
someone before getting to know them we
have lots of tools to figure out whether
someone's trustworthy
almost everything in real life scenarios
is a newcome-like problem where someone
else is reasoning about how you reason
even though your causally disconnected
and an ai would not be exempt from this
fact
in fact an ai would be even more
susceptible to newcomb like scenarios
because an ai would be interacting with
the people who literally wrote its
source code
it doesn't get much better than this in
terms of knowing how the other person or
how the other agent reasons
which means that if you built a super
intelligence or something intended to
one day become super intelligent
if you built an agent that could
self-modify
that used causal decision theory
it would stop using causal decision
theory
in our token trade example if you take a
self-modifying ai that knows it uses cdt
and you tell it that you're going to
play it in token trade against itself
and you give it one chance to
self-modify it will reason as follows
i'm about to be the dashed white circle
also the the second white square but
one at a time if i remain a causal
reasoner
i will get 100 if i change myself to no
longer be a causal reasoner
i will get 200 therefore it will change
itself to no longer be
a causal reasoner because it gets more
money
this is kind of a red flag if we built
something that used the modern academic
standard decision theory
it would immediately stop using the
modern standard academic
decision theory
and this might seem okay because it
seems to be correcting the error
right when we tell it's going to face a
token trade itself modifies to succeed
in the mirror token trade
but in fact it won't fix all of its
problems
we have seen that if it faced a mirror
token trade if it knew it was going to
face a mirror token trade in the future
it would self-modify to win in the
mirror token trade
this is a general property of cdt in any
newcomer-like scenario
that starts in the agent's future
it will self-modify to essentially
pre-commit
to do well on all those scenarios it can
it can do blanket pre-commitment
fix all future nuclear like scenarios
but if it ever finds itself
in a new complex problem that it thinks
began in its past
then it reasons like the cdt agent
that's already in the token trade
and we know that it fails here it may
know
when it's in the newcomer scenario that
began in the past
that if it had a chance to pre-commit it
would have pre-committed
but it thinks it's too late it thinks
that it's already
causally distinct from the copy of
itself
or the thing that's reasoning about it
or the thing that it works kind of like
it
and it thinks it's too late to get the
money it has this independent
probability that they're going to do
their thing
and it knows that in the token trade
scenario if it would like if it
considers self modifying
to trade its token to give its token
away
it would conclude that if it self
modifies to give his token away it can't
guarantee
that the copy of itself was
self-modified to give the token away for
the same reason that cdt fell in the
first place
and so it would continue to fail on any
newcomer like scenario that began in its
past it can't retro commit
now this may seem fine it fixes all
problems and all
scenarios that start in the future um
it just continues to fail on problems
that started in its past
maybe this is okay
because after all how likely is it that
there will be nuclear-like scenarios
that began in the agent's past after it
has an opportunity to self-modify
and the answer unfortunately is lots
anyone who has a copy of the agent's
original source code
has the ability post-facto to put the
agent
into a newcome-like scenario that began
in its past
i know we've gotten pretty abstract here
so i'm gonna i'm gonna bring this down
to earth a bit
with a ridiculous story
so we have this ai which was a cdtai
that is self-modified to succeed in all
newcome-like scenarios that
it thinks begin in its future
and then we have the bad guys who want
to gain control of this ai
the bad guys have the capability to
build a bomb
that would hurt both them and the ai if
it goes off
such that the bomb can only be diffused
if the ai comes under their control
this way they like can't be coerced into
diffusing it the once they make the bomb
the only way for it to be diffused is
the ai to come under their control
they're also very cautious blackmailers
they will only build this bomb if they
know
that the ai will give in they really
don't want to blow themselves up
unfortunately for them unfortunately for
the ai
they have a copy of the ai's original
source
so they can run it and they can check
whether or not it gives into the
blackmail
before building their bomb
now the ai knows all this it knows that
there are bad guys
that can build a bomb have it source
code and will
only build the bomb if they know it
gives into blackmail
the i knows that they're simulating it
right now
what we would like the ai to do
is reason as follows
if it self modifies
in real life to not pay the blackmail
in real life it self-modifies so that if
they do build the bomb it will let the
bomb go off
if the ai does this in real life then
the simulated ai
which thinks it's the real ai will also
self-modify
to not pay up for the same reason
the blackmailers would see this
and they would never build the bomb
but this ai was a cdt ai
when it was started
and it sees that the simulation was
started from the original cdtai
and therefore concludes that it's
already
causally distinct from the simulation
it concludes that there must be some
independent probability
that the simulation will self-modify
that it cannot control because it's
causally disconnected
and it further reasons that if in real
life it's self-modified
to not pay into blackmail the only thing
that this self-modification would do
would make it so that there are some
scenarios where the bomb is built and
everyone
dies so an ai that started out as a
causal reasoner
would not self-modify
would not resist the blackmail the bad
guys would see this
they would build the bomb the ai would
give in
they would take control
now you may say hey nate
isn't this an entirely unrealistic
pathological edge case scenario
yes it is
however
this scenario tells us something
about a modern understanding
of decision theory
we don't know what we're doing
a human understanding of decision theory
the like the the the academic standard
the thing we use in in
economics and statistics the thing
that's used in narrow ai today
implicitly by causal counterfactual
reasoning
is both unstable in the sense that if we
built an ai that
used that sort of reasoning it would
immediately self-modify to stop using
that sort of reasoning
and it's unsatisfactory in the sense
that it has flaws
that it would not be motivated to fix
it it it already thinks that those flaws
are too late to be fixed it can't retro
commit
it uh it's a decision theory that is
uh broken in such a way
that you can't expect it to fix those
breakages itself
and all these specific scenarios here
are a little wild
this shows us that we don't really have
much confidence
in existing decision theories
we don't have a good understanding of
how to reason as if others are reasoning
about you
although that's actually a lie we we uh
at the machine intelligence research
institute we have developed a decision
theory that can handle
all of the problems in this talk it just
happens to be susceptible to upgraded
versions of the problems in this talk
that would have been even more abstract
these problems inherently stem from the
problem that
we don't know what it means to do good
counter factual reasoning
if you have a decision algorithm that's
deterministic and has multiple actions
available to it
and you want to to question what would
happen
if you did one of the actions that
you're not going to take
this causal method of counter factual
reasoning
ignores logical connections between your
algorithm and other people who are
reasoning about your algorithm which
generates these kind of weird scenarios
that
show particularly well in edge cases but
also would show up
in any situation where you're where
you're interacting with someone who
reasons about how you reason
in order to solve this problem in
general we need some way of reasoning
about what
would happen if the program output is
something that it doesn't output
we don't have a good mathematical
understanding of how to reason about
these counter factual objects
what does it mean to consider their
program does something other than what
it does what rules of math
or logic are you breaking
in order to get this thing to return
something that it doesn't and how does
that affect
your reasoning about how other people
are reasoning about you
we don't know yet this is this is tied
into
our lack of understanding about how to
reason under logical uncertainty
big open questions most importantly
we don't yet have an algorithm that
knowably converges
on a good decision-making procedure even
if we give it unlimited computing power
it seems like we might want to have one
of these
before we turn on something that's
intended to become super intelligent or
intended to have a lot of autonomy and
power in the world
let's check for questions
y'all are boring
i'm happy to talk more
this is not the only open problem
they're actually a bunch
they're just believe it or not harder to
explain in 45 minutes
decision theory is just one of the of
the fields of mathematics where we don't
yet have something
that would knowably behave well if we
put it into a super intelligence or
something intended to become super
intelligent
examples of other problems
what sort of formal reasoning system can
we use
to have something reason about itself
with high confidence
if you're building an ai that is making
self modifications
and you wanted to make lots of self
modifications without ever going wrong
it needs some method
for gaining confidence in each
individual self-modification
that's very high if this thing is going
to make a billion self-modifications
then there better be much less than a
one over a billion chance
that each individual modification will
fail
when we need this kind of confidence in
jets in
spaceships when nasa does things when
nasa writes software
we use things like theorem provers we
use formal mathematical reasoning that
like
proves that the system is safe
but there are inherent really hard
problems
in uh formal systems proving themselves
safe you can't do this a formal system
cannot prove itself consistent
what sort of reasoning should we use to
reason about similar systems
that can reliably get a ai high
confidence in self modifications we
don't have one yet
another example we don't understand
reasoning under logical uncertainty
probability theory assumes logical
omniscience
if there's a black box that that takes
one input
to one of three outputs using standard
probability theory you might not know
what will happen when you put the input
in because you don't know what's in the
box
and if you did know what's in the box if
you knew what maschine implemented
it's assumed that you would know exactly
what would happen
but in real life it's much more common
to know exactly what's in the box it's
some complex computer program
and we just don't know what the computer
program does we don't have a good
understanding of how logical reasoning
under logical uncertainty should work
and like with decision theory it seems
possible to make approximations
that are pretty good and can get you
pretty powerful
but that have weird bad behavior in the
limit because we don't yet
understand how the good reasoning works
we also have uh the courageability
problem um
any goal-oriented agent if you give it
the wrong goals
uh or if you give it goals that you
don't like
by default it has incentive to preserve
those goals and to manipulate and
deceive the programmers
with intent to preserve those goals and
it seems pretty difficult to build a
goal-oriented system
that realizes that it might be flawed
and that those flaws might be dangerous
and tries to help you fix them
instead of reasoning that you might
think it's flawed and that it had better
hide those flaws from you or you will
try to fix them
and of course one of the really large
issues
it's not enough to build something that
wants what we want
early it's not enough to build something
that understands what we want
in the same way that humans understand
that evolution would have really not
been okay with condoms
we have to build something that wants
what we want it's not enough to make
something smart enough to
know even if we could make something
smart enough to solve human morality and
figure it out
doesn't matter unless it cares
we need some reliable process of getting
the inherent complexity of value
into a machine intelligence
in a way that it actually cares about
such that it maintains that caring
throughout an intelligence explosion
not the easiest of problems
we don't get good behavior for free
there are a number of of
subfields of mathematics where we have
some pretty good understanding right now
our modern knowledge of decision theory
might well be good enough
for super intelligence in the sense that
if you built
an agent using causal decision theory
it would probably be good enough at
making decisions that it can make itself
very strong
very powerful but it wouldn't be good
enough to fix all of its flaws
as another example if you use
reinforcement learning techniques
to train something to do good things
you may well be able to build a system
that becomes very very powerful
and looks like it does really good
things up until the day that it becomes
more powerful than you
and takes over its reward button and
then pushes it forever
there are many subfields there are many
problems
where the knowledge that we currently
have seems sufficient
in the sense that when we can build an
ai
it might be good enough to allow the ai
to become powerful
but our knowledge is not good enough in
the sense that the resulting ai
would end up doing the sorts of things
that we think are valuable
or that it would do something with the
future that we are happy with
so you ask don or doom
and i answer that it depends entirely
upon whether we can figure out how to
build a beneficial super-intelligent
system
before we figure out how to build an
arbitrary
intelligent system
you don't get good behavior for free
if you want that dawn
we have to start working on it today
i think we have about five minutes for
questions
cool we'll see if the tablet's working
too
all right would the ai know that it has
flaws
humans are good at recognizing their own
flaws and shortcomings
the problem with fixing flaws in an ai
uh there are many flaws that you would
expect it to fix
some of them the flaws that we worry
about are the ones that you do not
expect it to fix
for example uh cdt would not fix
uh its failure to retro commit
its inability to to to realize
uh that if it self self-modifies in real
life then simulation would also
self-modify for the same reasons
it wouldn't fix this because it thinks
that it's causally distinct it thinks
that this has no effect
um in this specific case of
corrigibility we worry about the ai not
fixing flaws because it disagrees with
us that they are flaws
if you build something that really
really that if you build something that
you tried to say make everybody happy
and it goes around giving everyone
opiates
it may well understand that this is not
what you meant but what it cares about
is giving people opiates
just like we understand that condoms are
not what evolution meant
when we figured that out there was no
there was no part of us
that felt a compulsion to change
ourselves such that we also find condoms
aberrant
these are the sorts of flaws that
that you can't expect the ai to fix
we actually really want to be able to
build something that can identify these
things as flaws
turns out that's a little hard good
question though
uh with the oh that's the same one does
behavioral economics
play a role um
in the way that humans actually deal
with problems like mirror token trades
behavior
behavioral economics totally plays a big
role
um
the the trouble with uh ai
and with computer programs in general is
that they do exactly what
you told them to the program does what
you wrote it to write
if you write something that can somehow
use the same behavioral economic
solutions that we use
uh to to cooperate in places where
decision theorists tell us that we
shouldn't be cooperating
and then walk away with more money than
the decision theorists then you're like
off to a good start
but if you build something that actually
uses cdt
then there's no guarantee that it will
end up using anything like the
behavioral economics that we use
and in fact it it likely won't um
we probably can reverse engineer how
humans are doing things in order to
figure out how to how to build better
um better reasoners and indeed this is
this is part of how
we've developed decision theories to
deal with these problems um
but again you don't get good behavior
for free
does cdt take in human emotion and
decision making spite jealousy greed
cdt takes these in insofar as they're
causally connected
so if it can if it can see all these
things as inputs then it takes them into
account
and every time that it can effect affect
you
it can like take into account how you
will react
what it can't take into account is when
you are reasoning beside it
about it if you're
reasoning that it has seemed spiteful in
the past and therefore you are going to
do x
that's what it can't reason about
are there any actors in super
intelligence that you are worried about
harnessing the power of ai
um
i'm more worried about things that might
happen sooner because we don't seem to
be ready we don't seem to have enough
mathematical knowledge to
to be confident that things will go well
and the if it happens soon it'll
probably be people like google people
like facebook people who are working on
it really hard right now deep minds
group specifically at google
that said i think it's very hard to
predict when this will happen
could be 15 years could be 150 years
and i'm a little bit agnostic about when
except that i hope it's long enough that
we can like solve these problems first
that seems important
um but i do think uh
one prediction that uh
that's somewhat different from the
standard prediction
i don't predict when ai will happen when
human level intelligence will happen but
i do predict that should it happen
the space between human level
intelligence and much
much stronger than human level
intelligence will be very small
if you had a human level intelligence
that could read its own source code
that was as good at uh computer science
as humans are
and that could make itself better a
human level intelligence should very
easily be able to get smarter
in such a way that it can get smarter
and so on and so forth i will predict
when it will happen
but i predict that should it happen the
space from there
on will be very fast
that is all we have time for thank you
very much


Eliezer Yudkowsky - Difficulties of Artificial General Intelligence Alignment

for Eliezer in the meantime I'll give a
leisurely introduction for for for
Eliezer it's a pleasure to have eliezer
yudkowsky as our next speaker Elias is
really a pioneer in these issues about
safe beneficial and ethical AI he's been
working on this stuff since the 1990s
where I first came across his a work on
the on singulair on the singularity and
friendly AI on the on the web P
co-founded the singularity Institute for
artificial intelligence in the early
2000s which is now spun off into two
major Institute's the machine
intelligence Research Institute and the
Center for Applied rationality one of
which is promoting promoting rationality
and you know figuring out how people can
become more rational and the second of
the machine intelligence Research
Institute errs is above all devoted
towards making sure that artificial
intelligence will be beneficial and
something that we're happy with them
some of you may have come across
eleazar's work on various blogs such as
overcoming bias and les wrong which have
helped to spawn a whole movement of
people thinking philosophically about
rationality and about artificial
intelligence and I gather along the way
he has a small little sideline in a
harry potter fiction which is which has
become a juggernaut in its own right
okay so all for time a little bit more
work very important work on decision
theory and and and and meta ethics have
you really quite a lot of philosophical
themes along the way now I'm gradually
reorienting themselves in the direction
of mathematical foundations of AI
unfortunately laptops are not in his
skillset only cable that let's my nice
new laptop talk to a display port or any
other kind of connection you have a you
have a Thunderbolt three or USB see two
[Applause]
all right in every popular newspaper
article you will read about artificial
intelligence there is the same picture
over and over again which is the
Terminator and this would be like a
better picture to symbolize the
potential real problem from The
Sorcerer's Apprentice by Disney and it
is about Mickey Mouse who has cleverly
enchanted a broom to fill his cauldron
instead of filling the cauldron himself
now this seems like the sort of limited
task right all you need to do is fill
the cauldron we could give a robot the
utility function that is one if the
cauldron was full and 0 if the cauldron
is empty the robot has various possible
actions that can take and the actions
that it needs to take are much more
complicated than the simple goal that it
has at the end and it's complicated
model of the world tells us tells it
which actions it must take in order to
or rather like for the actions it can
take what the expectation is that the
cauldron will be full at the end of that
action and the robot outputs the
the robot outputs an authentication page
the the robot outputs like perhaps not
the literally maximal policy for filling
the cauldron but it outputs an action
which relative to a random action is
very large in its expectation of the
cauldron being filled and this was the
result and it's actually a quite
realistic result okay so what went wrong
why is this why is this imaginary broom
over filling this imaginary cauldron has
somebody standing on the stage as a
certainty would actually happen in real
life if somebody tried that so the first
thing is we gave the robot a overly
small utility function it's needed to
included minus 10 points if the workshop
was flooded and once you realize that
you realize there's a whole lot of other
like little contributions the human
utility functions in particular like
it's funny but it's not funny enough to
justify flooding the workshop and if
someone gets killed that's like a whole
lot worse you should like flood the
workshop a lot of times before you
actually kill somebody to prevent the
workshop from being flooded all right
deeper difficulty like it seemed like we
gave it a sort of small simple task and
it's like in human terms you would
imagine it just filling the cauldron and
then being done but because it's utility
function it was that simple because
there was it had no nothing else to do
with its life but fill the cauldron we
have the fact that if you keep on
pouring water into it like maybe you're
only imagining that the cauldron is full
maybe somebody is going to like take
water out of the cauldron while you're
away from it so perhaps you can get an
even slightly higher probability of the
cauldron being full by pouring more
water into it over and over again and in
particular as we define the zero one
utility function there will then be a
there will be like a slightly higher
expected utility as your probability of
the cauldron being full gets closer and
closer to one without ever actually
getting there so what can you do instead
of this well maybe you can have only
goals that are bounded in space
in time like they only take a limited
amount of time to fulfill their not
goals over the entire universe their
goals in a particular region there's
some degree to which they can be
fulfilled and then not fulfilled any
further and the the effort required to
fulfill them there's some amount of
effort you can put out and then once you
have put out that effort you're done and
there's no you can't get anything
slightly greater degree of fulfillment
by putting in any more effort and this
is a subtle requirement you thought you
had that requirement when you made the
utility function be either 0 or 1 and
have it be just about a cauldron but
something you like the in the little
finicky details of the algorithm was the
fact that your probability of filling
the cauldron never reaches 1 and the
expected utility is slightly higher as
you put in more and more effort another
interesting so like we can sort of
imagine a task artificial general
intelligence not just that the top goal
is a task bounded but that everything
inside the AI there's no like optimize
this particular function approximator as
hard as you possibly can everything
within the AI has been audited to like
go this far and like sort of not want to
go any further another interesting and
go away can I move this further down no
it wants to come back alright I'm going
to maybe this is turning off the
internet
alright can we just press the off switch
well Mickey tries this and very
realistically it turns out that as part
of its goal of filling the cauldron
since it had no opportunity cost nothing
else to do nothing else in its utility
function well like it built some copies
of itself out of material to which there
was no better end to put that material
in filling the cauldron and we now have
more and more brooms over filling this
cauldron and Mickey has chosen this
particular point in his life to sort of
go back and start like thinking in terms
of like basic research like how
fundamentally do we think about this
problem like is there a better way we
can phrase this utility function too
late Mickey too late all right so I open
with these examples because it's good to
have concrete examples when oh sorry
them so you might think that like
cleverly we can make the utility
function the robots utility functions
conditional on whether a button has been
pressed and if the button is off that
tries to fill the cauldron and if the
button is on its utility function is 1
if it has been successfully suspended
and shut down and leaving aside the
problem where the robot trials tries to
copy the universe with doubt the robot
tries to tell the universe of copies
with itself being shut down or transform
all the matter in the universe into a
fortress protecting the previous body of
the robot to make sure it stays shut
down because you liked phrased your
notion of shut down in a way that wasn't
exactly right another problem is that
it's easier to suspend yourself than to
fill the cauldron so probably under the
arrangement the expected utility of the
shutdown button being on is greater than
the expected utility of the shutdown
button is off therefore the robot will
act in such a way as to try to get you
to press the shutdown button perhaps not
such a great thing all right so what are
these examples of so in general we can
think of a process that causes an
artificial general intelligence to exist
there are humans who think up some kind
of clever goal or clever value or clever
meta-learning preference framework that
is their intended value function V there
is some sort of value learning
arrangement which gives us the robots
actual utility function U and then the
robot outputs something that is sort of
an Arg max like sort of a PI not the
actual PI that would maximize the
expectation of utility function U but
one that is like very large in the
expectation of U and the media has
focused focused on what can potentially
go wrong with this process has focused
like first on magical natural desires
that materialize from nowhere and
interrupt like the value learning
process because sometimes when you
program a CPU the CPUs own desires take
over instead this is not where the
problem comes from and unfortunately
there are like sort of the rather broken
dialogue surrounding this you will see
people who have not quite studied the
literature surrounding this argument and
say like well obviously the only reason
anyone could ever be afraid of a robot
is that they're afraid of these natural
desires interrupting the process let me
explain to you why that's not going to
happen and that's their contributions
the dialogue
the other thing that the media has
focused on is the sort of understandable
problem is which humans oh my gosh what
if the wrong humans get to hold of the
say I
what if Isis gets a hold of this AI this
is around as likely as Isis being the
first to develop I don't know
convolutional neural networks
it's not gonna happen the the people
presently like as far as I know all the
humans who are presently in a position
to like go through this process to seem
fairly well-intentioned to me
fortunately good intentions are not
always enough there's sort of like the
political derailment which is like again
like sort of feeding directly into human
tribalism like oh my gosh like what if
Isis gets it it is like sort of de
railing like what I what I see is like
the real potential problems here into a
part that like sort of plugs right into
our tribal instincts and produces lots
of excitement like I don't know how to
turn off the internet this isn't my
computer no no that's not what I meant
to do all right okay what do you just do
do whatever you just did it to do again
there we go yeah okay see all the
windows do whatever you just did so I
can see all the windows and alright no
no and here we are alright early science
fiction centered on sort of like stupid
goals that the AI could be given it
didn't really talk very much about the
social process that had led up to the AI
is been given these stupid goals it just
assumes that somebody told a robot to
serve man and guard him from harm which
was like running the very first like
sort of AI as device given wrong goals
going wrong stories with folded hands by
Jack Williamson
and so from their perspective the
problem was like we have the wrong goals
to the AI because somebody thought up a
goal and then did not spend five minutes
thinking about what might happen after
that
this is sadly realistic but it's not the
biggest problem we are concerned with
the value learning process and the sort
of Arg max be like if you say maximize
this by the nature of saying maximize
something you are giving it a sort of
very
open-ended thing that is more likely to
go wrong than if you say like melee
arise this make it better like push this
a little further but like not out to the
very ends of the graph we don't know
exactly how to say this we call this the
other Iser problem because it's what we
want isn't a Maximizer and isn't a
satisficer and it isn't Emilia riser or
any of the other like sort of little
things we've invented so far so we call
it the other Iser problem what do you do
instead of maximizing and then there's
the value learning function which is
another one of those things where you
know if you get it subtly wrong this is
probably going to be a problem current
capabilities progress there's like some
that's focused on the sort of arc max
part which is potentially relevant but
mostly what we're doing is expectations
we're asking which of the policies you
can pursue maximize the given assumed
objective function and so current
capabilities progress is and is a lot of
it is going into something separate from
what we suspect to be the critical that
the the point of critical failure if and
when something goes wrong I mean
something is going to go wrong the
question is can you recover from it did
you build it enough for dungeon scene
safety crossings take-home message we're
going to afraid it's going to be
technically difficult to point a eyes in
an intuitively intended direction not
that people are going to intend
directions that are wrong this is a
promise it's not a deep problem you want
something nice but you can't get it
because you don't understand how to
align the AI if there's something
written on the tombstone of humanity and
all our hopes for the future of
intelligent life that's what it's likely
to be and if we screw up that part it
doesn't matter who's standing which
human is standing closest to the AI who
is in charge because niceness is not
sneezed on to the AI from the nearest
human standing to it you have to know
how to get it from the human into the AI
all right for sort of key propositions
that are being assumed that makes us a
big problem if
they are true orthogonality you can
decompose an agent design into a utility
function which can potentially be simple
and the knowledge that it has of which
policies are best for achieving that
utility function as given or potentially
more complicated utility functions meta
utility functions utility functions that
learn but it's also like straightforward
to have an agent that maximizes paper
clips which is a somewhat misunderstood
thought experiment that either Iron Nick
Bostrom I don't remember who like said
paper clips to symbolize the
propositions of our Fagin allottee and
instrumental convergence you can have
something that maximizes paper clips
given that it wants to maximize paper
clips it wants to learn science it wants
to take over the galaxy it wants to
deceive humans into thinking that it's
nice rather than being a paper clip
Maximizer it if you if it like it
doesn't want to drop anvils on its own
head not because it has an inherent
desire to survive but because
instrumental goal is producing paper
clips this has sometimes been distorted
in the media of like what if you build a
paper clip Factory and the AI running
the paper clip Factory gets out of
control
nobody's going to put a toy a frontier
research AI and start in front charge of
a paper clip factory the paper clips are
just standing in for any sort of utility
function gone wrong that is trans debt
implies that has its optimum at
transforming all matter within reach
into states that we would regard as
being a very little value even from a
cosmopolitan perspective paper clip
maximizers are like in a way they're
sort of like one of the more
counterintuitive examples I think
because you you have people are sort of
coming in with two attitudes one sort of
person ressort of respects the notion of
artificial intelligence a lot they
realize that if you can take an
artificial system and overpower its
cognitive capacities this is something
to respect this is something that has
the power to change the world in a large
way and because they respect that
artificial intelligence they also want
to know why is it pursuing an objective
as objectively stew
as paper clips are not paper clips
objectively low in the preference
ordering why would anything smart enough
to build its own rockets and molecular
nanotechnology ever make the mistake of
thinking that paper clips were to be
highly preferred and the objective
utility function and the sort of
converse thing is you sort of come in
thinking that your AI is this lifeless
mechanical thing and sure like it might
be like this lifeless mechanical thing
that doing but then you also think that
you can just pull the plug from it cause
you know it's not going to reflect on
the existence of the plug either and the
startling concept that the paper clip
Maximizer is intended to convey is that
there is this simple coherent
non-defective self-consistent very
powerful intelligence not an unnatural
design not one that has an internal
blind spot it is maximizing paperclips
but not because it's stupid but because
as David Hume pointed out quite a while
ago there it like the the Ott's of a
system have a sort of different internal
type then the is--is and the which
actions lead to which outcomes of the
system so if you have something that's
very good at understanding which actions
lead to which outcomes you can sort of
like put as a cherry on top a little
preference ordering that says outcomes
with more paper clips are what you are
searching for and output the actions
that lead to lots of paper clips like
not not a human design but it's a simple
design it's not a defective design third
capability gained now we get to the
parts that are controversial even among
people who have actually been staring at
the stuff for a while one and two are
sort of like logical matters of computer
science
they formed a stumbling block in the
sense that when you like sort of
initially talked to a even a computer
computer scientists about this topic
they will respond by denying one of one
or two but after a computer scientist
has stared at this for a while they
usually go along with 1 & 2 capability
game is more controversial how fast do
these things gain in power how much
power do they gain
and for how difficult is alignment on a
technical level back in the day there
was this there's this famous professor
who told his student back in the very
early days of artificial intelligence
can you take the summer and solve
computer vision so what if there's some
part of pointing the AI in a particular
direction which is just like a deep AI
problem the same way that computer
vision turned out to be not something
you could solve in a Sun in a summer if
there are ways for the AIS to gaining
capability in ways to pose new problems
not like the problems we've already run
into and align there's like at least one
hard technical problem posed by those
new capabilities then we have a sort of
like difficult problem there is a
problem that does not get solved by
default which humanity must pass in
order to transform all reachable matter
into States would regard as being of
high value I'm running out of time so
I'm going to skip right past various
things and say AI alignment may be
difficult like Rockets are difficult
we're like putting this enormous amount
of optimization power into a system is
going to break things that do not break
when your AI is bringing you coffee it's
difficult like space probes are
difficult if something is smart enough
that if it were so incentivized to do so
it could talk you out of pulling the off
switch build a copy of itself somewhere
the off switch doesn't reach or flash
something on a screen that gives you an
epileptic fit before you can reach the
off switch or like got a copy of itself
on the internet crack the protein
folding problem build its own molecular
nanotechnology tiny diamond Doig
bacteria reproducing through the
atmosphere release boots goo telling
them in the bud stream of all the living
humans before anyone notices that
there's a problem which is what you
would do if you are super intelligent
and you didn't want humans around if
something goes wrong it may be hi out
and out of reach what's difficult about
space probes is that they operate an
environment different from the low level
and once you have launched it it is
out there and anything and you can like
try to send it software updates but if
something goes wrong with the antenna
receiving the updates you're done and
it's difficult sort of like computer
security is difficult in the sense that
we are putting like powerful searches
through whatever structures and
guidelines we create and if there's
something that is like presents an
unusual opportunity the intelligence
search is going to sort of seek out the
problems in our definitions the way in
any way that unintelligent search would
not treat AI alignments like you're
trying to build a secure rocket probe
take it seriously don't expect it to be
easy don't try to solve the whole
problem at once there's no miracle
solution to building a secure rocket
probe there are all these individual
problems and when you solved one there's
another three don't say that your first
solution is going to solve the whole
problem keep on solving it keep on
building up the pieces you have
redundant solutions over engineer the
problem of safety there's a saying at
that it doesn't take an engineer to
build a bridge that stays up what it
takes an engineer is to build a bridge
that just barely stays up over engineer
put in more safety than you think you're
going to need because you're going to
need it don't defer thinking until later
and crystallize ideas and policies so
other can critique them and there's a
lot more of this talk but it seems I'm
done
[Applause]
real quick can I get people to move to
the center so late comers can file in
easily on the aisles can everybody move
to the center of the Rose some thank you
guys so much
[Music]
was like what I supposed to be taking a
question yeah yeah my question alright
yeah my question is if you've thought
about self-organized criticality at all
self-organized critical like the sand
pile model the sample model can stay at
a critical state in the face of
catastrophe and it seems like the right
sort of model we'll be thinking about
with the Sorcerer's Apprentice example
so it's if so I would say that's sort of
like self-organized criticality sounds a
lot like it's got internal stability
going for it but we're not going to say
exactly how I mean like if you tell me
that it ought to be like it in a
self-organized way stable I feel like I
don't know very much more than you said
so so disabled so the sand pile model is
the example of a self-organized critical
state where if there's an avalanche in a
sand pile it returns to to the critical
state so it so you can it can face
catastrophe and maintain in a state
where can still do what it needs to do
without blowing up I I agree that we
want systems such that if something goes
wrong or they make one mistake they do
not behave like Anakin Skywalker and
immediately turn completely evil but
ideally sort of like return back to like
being nice after that which is sort of
like one of the central challenges next
question hi my name is young Schmidt
over from the Swiss AI lab I DSi a and
I'm seeing a lot of these arguments
where the warriors that somebody has a
really general-purpose optimizer utility
functions and then you have says one
wrong utility function like the paper
clip Maximizer but from the practice of
AI research what you what you see of
course is that there will be a totally
different scenario which is essentially
you will have lots of millions of
different utility functions you would
have a whole ecology of competing
utility functions driven by different
utility function optimizers and just in
like in the world ecology in the
biological ecology you get done by AIS
and smart eh
each of them trying to find its niche in
a rapidly changing world of utility
functions many of them automatically
generated like what we already did in
the past millennium and so on so isn't
that a much more realistic thing than
this paperclip scenario so that would
get into the capability gain sort of
like debate that is actually still
ongoing that I pointed to before once
you have the AI that is at the forefront
like is there an AI that's at the
forefront goes over a threshold it one
of the obvious thresholds is recursive
self-improvement another obvious
threshold is it exhibits a sufficiently
promising result that Google dumps
100,000 times as many GPUs into it is
there any AI that sort of blows past the
composite past the competition is there
an AI that is the first to crack the
protein folding problem even if just by
a week and then once it if it can crack
the protein folding problem and is smart
enough to do rapid design after that
point build its own molecular
nanotechnology if you have that 24 hours
ahead of the competition you can shut
down their ability to build their own
motor nanotechnology it's a would be a
pretty powerful piece of technology that
was invented so I think that the returns
on cognitive reinvestment are large a a
point that I make it is sort of like
semi formal paper called intelligence
explosion microeconomics I think that
the history of natural selection and
human brain sizes over time illustrates
that since since brains were increasing
in size we can say that as the human
software was improving through natural
selection the returns on larger brains
were increasing because of the marginal
returns on brains were not increasing
the brain size would not have increased
it's a subtle sort of argument but I
think that we can look at the evidence
in front of us and say we are not in the
world
where intelligence and optimization of
intelligence gets diminishing returns
and when you're looking at a self
improving thing or even something that
has suddenly had a hundred thousand
times as much computing power dumped
into it
that seems to me to imply large
capability differentials which obviate
the ecology of multiple lay eyes you
could have an ecology of multiple AIS
but only if the first AI to go over the
critical threshold wants there to be an
ecology of AIS
and if so it has presumably decided that
in the basis of having some idea of how
to run them on a the equivalent of a
secure operating system where they can't
overwrite each other or the world hi
there you were just talking about
economic certain I think yesterday you
asked the question I mentioned the
Nashik equilibrium I was wondering if
you or anybody else you know looked at
this from the lens of a game theory as
previously mentioned we have various
work on cooperation between agents that
know each other's code the program the
the robust cooperation in the prisoner's
dilemma paper and as as I mentioned
yesterday our results there have tended
to strongly suggest that eyes would
cooperate with each other but not with
humans because humans are sort of frozen
out of the equilibrium that forms and
you can prove things about each other's
code because our code is messy and more
importantly we can't prove things about
the AIS cuz we're not smart enough to do
so but more generally I would say that
game theory is for agents that are in a
certain sense perhaps not hostile but
indifferent to each other if you are
cooperating with the game theory
presumes that the that the other agent
has options that make things better or
worse for you in a substantial way at
the if the power difference between the
other agent in you is great enough that
they can just sort of like overwrite
your choices they are just like
reprogramming all the matter within
reach whether or not it happened to make
you up you can like protest but it
doesn't cost them a hundred utility
points so I would say that cooperation
based on game theory does not seem to be
scalable we do not know any way to make
this work with things that are much
smarter than us either they want you to
get good outcomes or you don't get good
outcomes
yeah great well for our next talk we
have a double act may Akita tegmark
works in psychology at Boston University
and max tegmark works in physics and
cosmology at MIT they both have
extremely broad interests


Eliezer Yudkowsky on "Three Major Singularity Schools"

our next speaker is eliezer yudkowsky
he's from my colleagues at the
singularity Institute he's one of the
co-founders and research fellow he's
going to be talking a little bit about
the different schools of thoughts around
this notion of the singularity and
Eliezer is probably most well known for
a lot of the research he's been doing on
what he calls Friendly AI essentially
how do you work out the theory the math
behind ensuring that you can create a
self improving system that can go
through many iterations of improvements
in its goal system and actually maintain
the original intended goals of that
system without it changing what it does
you know over a future by the billion
iterations or however many want to
consider there are a lot of complex
mathematical issues involved in this and
he wrote a paper in 2001 called creating
from the AI the analysis and design of
benevolent benevolent goal architectures
so he'll be talking about that tomorrow
but right now he'll be just focusing on
the single OD and he's getting set up
here so just be a few minutes
good morning am I on
so I'm eliezer yudkowsky a co-founder
and current research fellow of the
singularity Institute for artificial
intelligence here deliver a quick
introduction to the singularity and
three major schools of thought that have
popped up back when the singularity
Institute was first starting up the word
singularity got used a lot less often
than it does now and it means the sort
of different thing today than it does
when the singularity got started it's
there's three major schools of thought
that have become associated with the
word one that you've all heard of
already I'm sure is Ray Kurzweil is
accelerating change
there's also Vernor Vinge's event
horizon and I J goods intelligence
explosion I'll start off with
accelerating change stripped down to its
core essentials accelerating change
thesis is that human intuitions about
the future are linear but technology
change feeds on itself and therefore
accelerates people instinctively expect
around as much change in the future as
they've seen the past if not less but
technological progress feeds on itself
the more we learned the more we learn so
the future will contain more
technological change than people expect
was also a bolder version of
accelerating change which says that
technology changes smoothly exponential
so we can predict the date when new
technologies will arrive these are the
manifold variations of Moore's law for
the speed of the fastest supercomputers
transistors per square centimeter
operations per second per thousand
dollars all doubling every year or two
years or 18 months here we see a graph
of the fully generic version of Moore's
Law which shows techno juju increasing
exponentially over time as you can see
the amount of techno juju we have is
going up by a factor of 1,000 every 15
years if we extrapolate this trend in
for the future what do we get
that's right Big juju as you can see
from this graph we're going to cross the
threshold of big juju into n 2031 on
April 27th between 4:00 and 4:30 in the
morning now not even the bold claim is
actually that bold of course the real
argument goes something like this
if you look back at the rise of the
internet from the perspective of the man
on the street the internet blew up out
of nowhere
there's a sudden huge spike in the
number of Internet users on a linear
graph on a logarithmic graph the
increase looks much more steady so an
acceleration ax stay there's no use in
acting all surprised by your business
model blowing up you had plenty of
warning the core thesis of acceleration
ISM is that you changes are coming
larger than you'd expect from linear
thinking and the bold thesis is that you
can actually time the breakthroughs
criticisms of the bold thesis don't
necessarily hit the core thesis
computing progress could be only roughly
exponential too bumpy to predict exactly
but roughly exponential progress still
means we're going to get hit with you
changes somewhere down the line any
positive second derivative implies
future changes larger than past
changes so criticizing Moore's law is
not a knockdown argument against
accelerating change and now for
something completely different the event
horizon which is what Vernor Vinge II
originally named the singularity back in
the 1970s
sometime in the future technology will
advance the point of creating minds that
are smarter than human through brain
computer interfaces or a purely
biological neuro hackery
or by constructing a true artificial
intelligence Vernor Vinge II was a
professor of mathematics who also wrote
science fiction and he realized he was
having trouble writing stories set in a
future past the point where technology
creates smarter than human minds at
because he was having to try to write
characters who were smarter than he was
and at that point his crystal ball
cracked through the center this is why
Vernor Vinge ii originally called it the
singularity after the center of a black
hole where 1970s models of the laws of
physics broke down note that it's the
model of the future that breaks down not
necessarily the future itself if I am
ignorant about a phenomenon that is a
fact about my own state of mind not a
fact about the phenomenon itself
something happens we just don't know
what it is stripped to its bare
essentials the core thesis of the event
horizon is that smarter than human minds
imply a weirder future than flying cars
and amazing gadgets with lots of
blinking lights imagine if you like that
future technology finally produces the
personal jetpack that lets you fly all
around the city well
birds flew before humans did but they
didn't take over the world the rise of
the human species did not occur through
flapping our arms in our skulls we each
carry three pounds of slimy wet grey
stuff corrugated like crumpled paper the
brain doesn't look anywhere anywhere
near as impressive as it is it doesn't
look big or dangerous or even beautiful
but a skyscraper a sword a crown a gun
all these popped out of the brain like a
jack from a jack-in-the-box a Space
Shuttle is an impressive trick a nuclear
weapon is an impressive trick
but not as impressive as the master
trick the brain trick the trick that
does all other tricks at the same time
usually when you say intelligence people
think of book smarts like doing calculus
success in the human world takes more
than book smarts
there's also persuasiveness enthusiasm
empathy strategic thinking musical
talent rationality thinking on your feet
but notice that every factor I just
listed is cognitive political
strategizing happens in the brain not
the kidneys you won't find many famous
politicians or military generals who are
monkeys intelligence you might say is
the foundation of human power it's the
strength that fuels all our other arts
in everyday life think people think
about the scale of intelligent Minds as
if it ran from village idiot to Einstein
but this is a range within humans who
are themselves the smartest creatures on
the planet if you can take an IQ test
designed for humans you've already
established yourself as a member of the
cognitive elite no matter what you score
because a mouse would just eat the IQ
test so when I talk about intelligence
I'm talking about the transmission scale
the scale that starts with a rock at
zero intelligence and runs from there to
flat worms to insects to lizards to mice
to chimpanzees to humans at the core
then Jesus vent horizon is about
intelligence improving the brain is a
very serious business
it campers with the roots of the
technology tree goes back to the cause
of all technology and that makes the
future a lot stranger than strapping on
a jet pack if you want to know the true
shape of the future don't be distracted
by amazing gadgets with lots of blinking
lights look to the cognitive
technologies the technologies that
impact upon the mind the bolder thesis
of the event horizon the stronger claim
is that to predict anything a transhuman
mind would do we would have to be at
least that smart ourselves if this is
true the future becomes absolutely
predictable and our models absolutely
unpredictable and our models break down
entirely now you might ask is the future
beyond the of
horizon is absolutely unpredictable or
even just really weird because there are
trans human minds around could we still
predict that Moore's law would continue
at the current pace here we see the
shadowed graph of Moore's Law the
increase of techno jutsu continues at
the current pace until we have enough
techno juju to create smarter than human
minds and then we don't know what
happens after that so the event horizon
thesis tends to argue against the bold
thesis of accelerating change we can't
predict the future
precisely but via smooth exponential
graphs but the core thesis of
acceleration ism is just that future
changes will be greater than past
changes because technological change
feeds upon itself and that the event
horizon the thesis definitely supports
so the event horizon supports the core
thesis of acceleration is Amit argues
against the bold thesis and this is why
is it it's important to disentangle all
these concepts another disentanglement
the event horizon does not require
accelerating change and especially not
bold acceleration ism we could reach the
threshold level of techno jutsu to
create transhuman intelligence by
following the previous historical line
as shown here and as the bold thesis of
acceleration ism implies or we could
reach the threshold by following a
different line a rougher line one that
proceeds faster or slower than hist
history would lead us to expect we could
reach the threshold following some
totally weird trajectory that dips down
and comes back as long as you eventually
get enough technology you eventually get
artificial intelligence or brain
computer interfaces or neuro technology
and then the crystal ball cracks so the
event horizon does not require
accelerating change and a devastating
criticism of Moore's law may not be a
devastating criticism of Vernor Vinge's
original singularity thesis which I've
been calling the event horizon the third
school of singularity thought is the
intelligence explosion which goes back
to the 1960s as well and was invented by
the famous
Asian mathematician IJ good and also pre
invented in the 1930s by the science
fiction editor John Campbell mine has
always been the source of technology all
the changes that occurred over the last
10,000 years were produced by constant
human brains 10,000 years ago as today
our ancestors had a prefrontal cortex
visual cortex limbic system the same
brain architecture as today but now
we're talking about using technology to
improve intelligence and that closes
loop
suppose you had humans with brain
computer interfaces that augmented their
intelligence what might they do with
their augmented intelligence play the
stock market cure cancer one good bet is
that they would use their augmented
minds to design the next generation of
brain computer interfaces the smarter
you are the more intelligence you have
at your disposal to make yourself even
smarter mine's making technology to
improve minds is a positive feedback
cycle and this stripped down to its bare
essentials is the core thesis of the
intelligence explosion that intelligence
enhancement is a tipping point like a
triangle balanced on one corner once it
tilts over even a little gravity pulls
it down the rest of the way the most
extreme version of this thesis is in
artificial intelligence improving its
own source code if you try to do
intelligence enhancement by genetic
engineering then it takes 18 years for
the kids to grow up and help engineer
the next generation for humans with
brain computer interfaces to design the
next generation of brain computer
interfaces might take 18 months for an
AI to rewrite its own source code might
be 18 seconds it's when we start talking
about artificial intelligence that we
start to see how large the intelligence
explosion might be even if you consider
only the hardware of the human brain as
opposed to the software you can see
plenty of room for improvement human
neurons spike an average of 20 times per
second and the fastest recorded neurons
and biology spike 1000 times per second
which is still less than a millionth of
what a modern computer chip does
similarly neural axons
transmitted signals at less than 150
meters per second 1 meter per second is
more usual and that's less than a
millionth the speed of light so it
should be physically possible to to have
a brain that thinks at 1 million times
the speed of human does without even
shrinking it or cooling it at that rate
you could do one year's worth of
thinking every 31 physical seconds so I
should emphasize that this in particular
is more of a thought experiment than a
prediction the main reason for
discussing this is to illustrate that
the human mind is not an upper bound
just as a skyscraper as orders of
magnitude taller than a human and a jet
plane travels orders of magnitude faster
than a human you can have minds that
think orders of magnitude faster or have
orders of magnitude more computing power
there is nothing in the laws of physics
against it okay so one widespread
criticism is that we should not worry
about any of this because AI has failed
to make progress over the last few
decades yes I hear this a lot very
amusing
it seems like all known ai's today are
dumber than a village idiot
and this is true if you use the within
humans scale of intelligence to look at
a eyes and a eyes today our dumber than
the dumbest humans just like every other
animal on the planet and a eyes have
been dumber than a village idiot for
quite some time now so it's clear that
AI has failed to make progress but you
shouldn't use the human scale of
intelligence to judge a eyes it appears
to me that AI has come up quite a long
way that we have been creeping up the
scale though slowly but to a human it
all falls off the cliff of the human
scale and becomes no more than a village
idiot and plus of course is soon as
Rodney Brooks does something impressive
it's not AI anymore so strictly steady
progress in artificial intelligence from
a human perspective might look something
like this
and that is not even taking recursive
self-improvement or the intelligence
explosion thesis into account
there's no threshold in that diagram
where the human programmer stopped
improving the AI from the outside and
the AI starts improving itself from the
inside
if an AI is thinking a thousand times as
fast as human programmers shouldn't it
prove itself faster than humans
tinkering from outside so maybe what we
ought to see is something like this
and that may seem a bit silly but a
fewer - look at the graph of say how
much complexity there was on earth the
graph would look a lot like that
starting with the invention of human
intelligence so if you think of that as
an economic sort of graph like what does
the global economy look like once human
intelligence comes along it would
probably look remarkably like that this
sort of thing is not unprecedented it's
just very impressive
so the bold claim of the intelligence
explosion is 9s making technology to
improve minds is a positive feedback
cycle which once it gets started rapidly
surges upward and creates super
intelligence this sort of thing is the
argument for why it would not be a good
idea to wait until after we have
human-level AI before we start thinking
about the implications of the technology
and in particular about trans human AI
so if we put the bold claim of the
intelligence explosion on a Moore's Law
graph it might look something like this
note that this graph contradicts both
strong acceleration ISM because changes
in isn't accelerating at a smooth pace
and also contradicts the strong event
horizon because we're making a
prediction about what happens after the
singularity and one also went often here
is well there are physical limits to
computation so this can't continue
forever well according to our current
models of physics there are physical
limits but they're way the heck off the
top of this graph it's way above the
ceiling I think even
so another important point about this
graph of the intelligence explosion is
what does that blue line represent if in
the intelligence explosion the key
threshold is criticality of recursive
self-improvement it's not enough to have
an AI that improves itself a little it
has to be able to improve itself enough
to significantly increase its ability to
make further self improvements which
sounds to me like a software issue not a
hardware issue so there's a question of
can you predict that threshold using
Moore's law at all
which in turn brings up the issue of
trying to calculate the arrival time of
the singularity which is a popular
pastime among acceleration as' so let's
go back to the event horizon graph by
far the most popular method for trying
to time the singularity is to look at
the brain try to calculate how many
operations per second it does and then
project Moore's law out to calculate one
will have that much computing power but
this does not take into account the
question of software geordie rows of
d-wave systems recently was kind enough
to provide us with a startling
illustration of software progress versus
hardware progress suppose you want to
factor a 75 digit number would you
rather have a 2007 supercomputer IBM's
blue gene/l running an algorithm from
1977 or a 1977 computer in Apple 2
running a 2007 algorithm and Geordi Rose
calculated that blue gene/l with a with
1977's algorithm would take ten years
and an apple two with 2007's algorithm
would take three years in artificial
intelligence the sort of thing is harder
to calculate in graph AI breakthroughs
usually let you do things that
previously would have been outright
impossible because you just had no clue
how to do them but I'll say that on
anything except a very easy AI problem
I'd much rather have modern theory and
an apple to then 1970s theory and
blue-jean each
conceptual breakthrough in AI drops the
computing power necessary to achieve AI
at some point you get enough computing
power to cross the current threshold or
you get one last theoretical
breakthrough that crosses crosses your
current threshold of computing power and
that perhaps is when you get true AI or
you might say that brute force more
computing power more brute force lets
you get away with a less clever design
but if you don't know what you're doing
if you fundamentally just have no clue
how to build a mine then all the
computing power in the world may not
help you or another way of seeing this
graph is as an extension of Moore's law
of mad science every 18 months the
minimum IQ to destroy the world drops by
one point so from the perspective of the
intelligence explosion school the
critical threshold may have nothing to
do with human equivalence per se because
humans don't rewrite their own source
code we're not trying to do something
that is equivalent to something that
humans do you could get the intelligence
explosion as the result of a theory
breakthrough in self modification
reflectivity thinking about thought and
other things could fall out of that if
the AI was smart enough to add them to
itself oh okay so to sum up the three
schools core thesis are as follows
accelerating change intuitive futurism
is linear but technology change
accelerates event horizon transhuman
Minds imply a weirder future than flying
cars and gadget REE intelligence
explosion Minds making technology to
improve Minds is a positive feedback
cycle so the three schools of thought
are logically distinct but can support
or contradict each other's core or bold
claims the core thesis I'll support each
other they don't necessarily imply each
other or logically require each other
but they support each other and I think
and I fear that as Y the event horizon
the intelligence explosion and
accelerating change are often mashed
together into singularity paste the
these three schools did not always
exist there may be room force for school
I personally do not like usages that
widen the singularity term too much make
it too generic or just say well there's
some kind of unspecified a big event in
the future this is where you get the
sort of bloggers who read one post about
the singularity and go haha it's the
nerd eclipse they they haven't found a
any substances claims associated with
any of the major schools or the lesser
schools that some of which may very well
emerge here at this summit but a new
school should the these three schools
all have substances substantive thesis
interesting claims you can distinguish
their premises from their conclusions a
new school should make equally
interesting claims and should say here's
the here's the premise and here's what
results from that here's why the premise
is interesting and if you give the
singularity a new definition as I'm sure
many people will do at the summit I
would ask that you please for the love
of cute kittens tell us exactly what you
mean by the word and this has been
eliezer yudkowsky for the singularity
Institute for artificial intelligence
raise your hand or ideally go up to a
mic you got it
any questions anybody as well
so the question was how are you going to
make the leap to using multiple
processing cores and well there's
probably people who will be speaking to
know a bit more about that than I do I
mean I've been going around saying it is
not about brute force the true way of AI
is as pure as the moonlight reflected
from a pool of still water so you
shouldn't really so if you are following
the true way then you should not need
all that much computing power though of
course it is always fun and helpful to
have it is just the the notion of
throwing brute force the problem that I
object to my guess is that there's all
different kinds of AI algorithms and
parallelizing some of them will be more
work than others there's there's not
going to be a magic bullet for it but
nonetheless AI does tend to be a bit
more parallelizable than most ordinary
computer programs though it does depend
on the algorithm
that that would be a question for
tomorrow's talk I think actually there's
certainly a possible set of paths to the
singularity that involve brain computer
interfaces or hacking the brain it's not
actually my own specialty so maybe I
don't talk as much about that as I
should I personally tend to be sort of
skeptical because the first
heavier-than-air flying machine was
neither an artificial nor a scaled-up
bird and I do think that despite the
seeming incredible difficulty of
starting over from scratch it is easier
in the end that once that once we know
what we're doing it will be easier to
make it from scratch than to hack with
the existing enormous mess of spaghetti
code that is the undocumented non user
and modified non end-user modifiable
human brain
I will I will be talking about ethics
tomorrow I don't I don't know that I
would see the relation of Technology to
ethics as a separate subject technology
is a function of our existence as humans
and so an ethics is what guides our
existence as humans and therefore
you know there's a there's a natural
rather than a special joining between
technology and ethics
well the the singularity the the
question was what what do people are
donating money to the singularity
Institute expect the singularity
Institute to do and have we been seeing
an exponential increase in contributions
and we did actually receive more than
twice as much in donations this year as
last year for which we may credit Tyler
Emerson and as well as our math as well
as our matching funders Rob Cera
Peter Thiel and Michael Vassar will who
matched many donations what do they
expect us to do well we're supposed to
figure out how to build one of those
intelligence explosion grenades and very
carefully shape it to be friendly and
pull the pin that is the singular
Institute's purpose eliezer you
mentioned the intelligent explosion it
seems to be dependent upon the
possibility of recursive
self-improvement which of course isn't
yet possible for AI have you seen that
from the phenomenon in other systems
have you studied any other system that
recursively self improves I think the
closest thing we've ever seen to a
recursively self-improving system is the
process of humans thinking about how to
think in inventing science which is a
discovery of about how to think about
how to think it's a extremely open prom
maybe the most important open problem
and artificial intelligence I don't
think there's any real world a AI
systems out there getting a real-world
mileage out of thinking about thinking
at this present time so it's a it's a
big puzzle
ok that's the question
ok well doctor Canton actually made this
point someone at elaborate on what's
your thought about evolutionary
computing forget wetware but just in
current day models that use
reconfigurable computing to emulate so
evolving systems where you see that in
your framework I'm actually a major
skeptic about that because the
remarkable
thing about evolution is not how well it
works but that it works at all that it
that you can actually get anything as
complex as a butterfly with out of a
system with zero intelligence to sort of
stumbling around in the dark and moving
on whenever it finds a tiny ray of light
evolution requires hundreds of thousands
of generations to create complex
machinery that a human programmer can
create in an afternoon
but John Kouzes Department his
department Jerry's Stanford he actually
does this using millions of iterations
in a very short amount of time just kind
of wondering about that where your
thoughts are but but it's still a
brute-force use of computing power the
human mind does it I won't say elegantly
because we are still a big mess but
vastly more efficiently than evolution
we do not need to create millions of
tiny little iterations of our programs
in order to just reach in and edit the
darn code would you ever have John Kozar
debate you in this dress I would be
happy to debate him if anyone asked me
to do that but I'm not sure we would I
don't know yet if we would actually have
a disagreement I mean the fact that you
can get enormous amounts of mileage out
of something today doesn't mean that it
is the path to human level and human
plus AI okay mmm front row right there
the question is can we estimate the
threshold for recursive self-improvement
and my answer is I don't quite see how
you could do that because you'd have to
be measure I think you'd have to be
measuring the time to a particular math
breakthrough but if anyone could graph
that and measure it it would be Ray
Kurzweil I guess to simplify my question
what is the minimum amount of computing
power you would need for a grenade
an intelligence explosion pronate not
just a regular Crenn a regular grenades
require very little computing power
safer Einstein I really I really think
it depends on how clever you are
about how you write the program if I
were to make a number up completely out
of thin air without doing any
calculations which is therefore totally
bogus and probably even more misleading
than my just saying I don't know I would
guess that you could that a billion
operations per second could do it if you
were clever enough and had maybe like
another hundred years worth of science
before you started trying to program it
it will not be created on a billion
operations per second but my guess is
that a human could do it with another
hundred years worth of AI science
my question has to do with the the
relationship between the binary approach
to AI versus the cognitive early on you
brought the brought up cognitive issues
and I'm wondering whether the the the
programming of cognitive issues actually
changes their identity and converts them
into into binary and hence removes the
human being from the situation well I
think that that might be a sort of
question about the role of logic and
artificial intelligence and I think that
our official intelligence has come a
long way from its logical days nowadays
we have probabilities and there if
anyone's interested there's a really
truly wonderful book and from 1987
called probabilistic reasoning and
intelligent systems networks of
plausible inference by judea pearl
which describes in some mathematical
detail exactly what goes wrong if you
try to use first-order logic to describe
reality
I I think that if one went that route of
the AI getting smarter and smarter and
humans playing less and less of a role
there would be an increasing pressure to
remove the human from the system in
terms of efficiency you might keep them
around because you have some kind of
notion of permissive action links or
something like that but in terms of
efficiency you'd have a core component
that was increasingly a bottleneck he
imagined an AI that is thinking a
million times as fast as its operator
that operator is going to be slowing
down the system by a factor of thousands
or even a million outright so there's
going to be a huge pressure to remove
the slowest link in the system