190. Steven Pinker on the Possible Existential Threat of AI

hello and welcome to session 190 in the
AI safety at Cambrian group tonight
we'll be discussing Steven Pinker on the
possible existential threat of
artificial intelligence Steven Pinker is
a professor in the Department for
psychology at Howard University and a
very noted linguistic and this is in
fact a podcast interview a discussion
between Steven Pinker and Stuart Russell
moderated by yokas Perry with the title
Steven Pinker is Joan rough on the
foundation's benefits and possible
existence to threat of AI so what we are
talking about today is an excerpt from
this podcast where we focus on why
Steven Pinker is skeptical about AI say
that the other side too assistant to
hope is existential risk so first off
all these things that that Steven Pinker
is saying are things that Stuart Russell
has in theory been answering because
this is a debate so why do we feel that
there's a need to go further into this I
believe that there is because of the
structure so if we see in the beginning
we will see Steven Pinker make around 30
claims and then Lucas Perry will say
Stuart Russell what are your response to
this and then Stuart Russell will go to
the will reply to the very last claim
that Steven Pinker has and maybe return
to a few of the others but basically
just make his own long list of claims
and then a lot of things are left
unanswered
so what I'm try to do here is to go
through line by line everything the
Steven Pinker say and try to give some
kind of answer and commentary on
everything here and one another thing we
can see is that this this goes two ways
in that for instance Roy shot that says
that he's frustrated that scene pinga is
not responding to resolve the claims
that Stuart Russell is making and so in
this way Steve you could make the same
complaint that that they're just not
answering each other's themes so another
thing some background that I probably
should say is to discuss the straw man
fallacy because this will come up quite
a bit the QPD assess that this is when
present one asserts a proposition and
another person a clerk is against a
superficially similar proposition as if
that wasn't a argument against the
person ones true proposition I'm not
really fond of this it says falsely you
in Wikipedia but but the way this turns
out in this particular is that Steven
Pinker will say something like people
who are worried about AI safety often
make and then some kind of silly claim
and then he will argue that this silly
claim is wrong and he will have good
arguments against this but it doesn't
actually say who precisely is making
these claims and that's of course a bit
unfortunate so what came should you
actually actually be discussing and well
I think that most people in the sht
community looks to the book
superintelligence by Nick Bostrom as the
the the most thoroughly written the
canonical version of the dangers of a
superintelligence another very awkward
place to find claims would be in store
Brussels Brook human compatible may have
just talked about that and it's a very
obvious thing to do and some blame
probably falls on Lucas Perry here on
the hosts because he doesn't answer it
doesn't ask finger to comment on your
Russell's cling but asks him in general
question about whether existential risk
is possible and also I another
disclaimer is that I'm not really
examining still Russell's claims because
he does say some things that I disagree
with I find out a few things and here
for instance he's making a rather
contentious claim
and sometimes he seems to be like
talking around what Steven Pinker is
saying and answering different questions
or something like that but I won't
really go into details about that
instead let's see what Steven Pinker
actually says the first part is about
what is called a precise
superintelligence he starts out by
saying in May discussions of
superintelligence inspired by the
success of deep learning weather
sometimes asked - so he's not pointing
out a specific discussion but I just
saying that some people say this and
this superintendents inspired by deep
learning that's often called a precise
artificial general intelligence and
that's something that's considered a
possibility but a recently remote one in
general AI safety and going beyond just
a human level and up to a
superintelligence infra side way is also
something that is considered possibly
even less so the discussion that he is
criticizing Stephen King is further one
where media precise super intelligence
is able to solve the problem of Middle
East peace so I have actually tried to
search a bit and I don't recall ever
having seen such a discussion so I would
like a reference of one exists and the
problem with this Steven Pinker is that
if you want to do something like keep
going with this then you need 60 million
similar problems ain't unique the
correct answers in order to train a
supervisor learner and that's always is
something you can't have so that's
probably an answer about what that deep
learning can actually do a few shuttling
and things like that but in AI safety we
are mostly concerned about how something
that is roughly a general intelligence
that is roughly on the human level could
become super intelligence catch me
through an intelligence explosion and in
this case what you actually need to do
is not to solve Middle East but to try a
lot of problems and see which one is
better and for this kind of thing trying
660 million times that sounds quite
feasible
and of course the hard thing is to
generate 60 million good programs and
that's of course something we can't
because we don't have an API yet but
it's something that seemed like it would
be possible and on the problem stepping
up hands out is that some of the
discussions don't have enough details in
particular he says that you know a lot
of discussions you could replace the
word superintendents with magic or
miracle and the sentence would read the
same so he's a linguist so he must know
what he's talking about so I tried to
you know take the best discussion not
just in discussion but in the Nick
Bostrom's and see if it was actually
possible to replace the word super
intelligence you can see I've tried to
do it here and activity you can't do
that so you probably can only do that in
poor quality discussions and further
some of the things that I'm missing is
that the discussions don't talk about
what is what does the intelligence
consists of and what's even a solution
so my general answer to this is that
sure there are bad discussions and I
think if you look at the word 10% of
discussions on climate change or
immigration or something like that then
I'm sure you'll be really disappointed
because there are probably a lot of bad
discussions and also a lot of
discussions where precisely what counts
as a solution to war in the Middle East
isn't really that relevant it is
possible that I am misinterpreting
finger here we're in the discussions
where you replace super insurgents with
magic
then he's he might actually be talking
about that people overestimate the
degree to which insurgents in his
particular super intelligence could
obtain calls in the real world like it
can just do things by magic so so it's
possible that's what he means even
though it's not really what he says then
there is a central part extrapolation
the concept of super intelligence is a
dubious extrapolation of an
unexplainable
continuum like humans animal on the sub
right human too smart you
so obviously there are some continue
that cannot be extrapolated if you try
to go north then eventually you reach
North Pole
if you try to go slower you and
eventually stop if you try to go faster
than light then so the concept of
unintelligible and continuance is sound
on the other hand however if you try to
say that it's impossible to go faster
than light then you need to bring some
kind of argument or evidence and for
intelligence for instance it's possible
that there is like a ceiling like it can
you can at most get 200 IQ or something
like that it might be that no
conceivable intelligence could ever
understand
string theory so in theory this kind of
argument could work but there haven't
been really been any non-trivial
candidates that have been proposed so I
think some more work needs to be done
here further Stephen Tina says that he
don't believe in intelligence it's maybe
you would call it an unrealistic on
intelligence so he can just compare a
human and a squirrel and say the human
is more intelligent and you could
imagine even more that so Stuart Russell
has a direct answer for this one later
and I guess I could make a democratic
argument that most people would he would
say that sure is possible to compare a
squirrel and a human and say well the
human is more intelligent but there are
also a few constructions that I believe
feels awfully quite well if that
bulletproof and quite strong and then so
the first is a speech super intelligence
if you could make a machine that could
think like a human which would be
possible at least in this formulation
then you could probably just make it run
a million times faster and that would
probably count as a super intelligence
you could also like build a million of
these machines that would be a quantity
super intelligence to use Bostrom's were
pledged and finally there could be
something like a quality super
intelligence which is probably what same
thing as saying couldn't
would not be possible so is this
possible
well you can try to extrapolate and it
seems that it would be possible there
are some things that some ways of
thinking that seem like they would be
possible like an optional Bayesian
reasoner and there are some biological
limitations to human brains quite
actually quite a lot of them that a
computer program might be able to get
around so I think that is a good case
that is possible and it's on student
lingo to to explain why there would be
some kind of seeing to intelligence so
the first scenario of Steven thing that
divides AI safety in scenarios into two
and the first is about will to power and
I'm just gonna start out and say this is
probably any Cell of Stroman because
this scenario is probably something that
I have never seen anyone seriously
consider this in the EIC community and
the claim here is as soon as you get an
intelligent system it will inevitably
want to dominate and exploit and I think
it's a strong word but I also think it's
possible if you are building an AGI in
some kind of evolutionary system then I
think it's possible you could get this
but not inevitable and the analogy that
Steven Pinker is saying is its mate is
one between humans and animals or
conquested dollars and natives the
reason why this is false is the
orthogonality thesis so I think that's
rather interesting that he's familiar
with the orthogonality thesis so so he
clearly Steven Pinker has studied this
somewhat and this orthogonality thesis
is just a fancy schmancy way or
referring to Schuman's distinction
between our goals and our intelligence
I'm not really sure that's true in super
intelligence where the question writes
about the or finality rights that the
difference is that it does not depend on
the human theory of motivation I don't
know really what that is so what I would
rather say is that the concept of
intelligence and final goals which is
what the orthogonality thesis talk about
are a lot
normatively thinner than reason and
morality right morality and final goals
are not the same thing - at least then
the other scenario that's been discussed
I called the here basic AI tries and
Steven Pinker starts watching and I know
that there is an argument that says
again he's not really referring to a
specific argument but just saying that
it exists and he phrases it like this
any intelligence have to maximize
response survivability at all costs I
think that he's obviously referring to
Steven 100 basic AI drives where Steve
Omohundro is before the claims that this
is not something that will happen to any
intelligence and it will not happen at
all cost
they're merely tendencies that will be
present unless specifically counteracted
so this is a truthful statement of the
thing and Steven Pinker countless that
his iPhone doesn't seem like it it
maximizes his own survivability a fun
fact this actually if it does if it's
supersymmetry it will allow you to call
the 9-1-1 it will not allow you to play
Angry Birds it will just say sorry I
can't allow you to do that but the real
answer to this is that the iPhone
doesn't really count as an intelligence
this is something that is implemented by
the programís of the iPhone and it's not
something that the firm has done itself
because it's intelligent enough to
reprogram itself and do things like that
if it was programmed to refuse to really
do as it was told then obviously we
would not buy one and I think here
Steven Pinker talking about this kind
thing it seems like he's not talking
about the treacherous churn and take
over that has come in a ICT seems to be
considering a rather different scenario
he also says this is the less common
existential threat scenario where
actually this is the more common
especially compared to the will to power
but they don't really cut the world at
the joints because no one talked about
the first one anyway
I thought about about the philosophical
implications of an hei that is really
really powerless like an iPhone will it
try and take over I think that would it
could be an interesting thought to
explore further but that's not really
something that seem thing that talks
about the next thing he talks about is
the paperclip Maximizer which he thought
was a spoof but apparently it was
intended seriously when it was
introduced it was introduced by eliezer
yudkowsky making a point about human
values so it was not intended as a
permanent as a prediction or something
like that Steven Pinker further claims
that this is a self refuting scenario I
don't think the self refuting can be
used for that this if you say I am lying
then that's a self refuting statement
but here you probably saying that these
scenarios are unrealistic or
extrapolating incorrectly or something
like that another argument against the
paperclip Maximizer is that we don't
need more efficient paperclip
manufacturing than we already have
so that's obviously false but I think
also that's a poor way to engage with
thought experiments if you are presented
with a trolley problem and you just say
oh there are no trolleys where I live
then you're not really engaging with it
but here he has a good idea an
interesting argument human bodies are a
pretty crummy source of iron for
people's lives and there is a good
account argument to this and that is the
entire point of a Maximizer it will not
and maximize it does not stop when there
are only crummy sources left and
Maximizer goes to the maximum and does
not stop before that there are some
several scenarios that are considered
one is that we might give it the and
superintelligence the goal of curing
cancer and then it just conscripted us
as guinea pigs and uses tumors in all of
us and and later in the podcast
Stuart Russell accuses
Steven Pinker of making strong a lot of
documents and Steven Pinker replaces
no this is HTML
an example that is from your book and in
particular the designers of this
superintelligence will be really chaotic
to give the system control over the
entire tent without any testing so
that's to pass both control over the
entire tent and without testing so
that's referring to and that he's not
talking about the treacherous turn and
not talking about the take-off scenario
but this is also something that I
commented on last year when we read
human compatible so let's see what I
wrote so I have competed in here for my
presentation where I say that Stuart
Russell is actually critical of these
examples when you write them in the book
it cost them unsubtle and the reason why
I know you like them is because they
omit the takeover and that's where the
real intelligence is so in total and I
think if you only have registered your
Russell's book then I think this kind of
misunderstanding it's quite easy to make
and so steaming up the question says and
I think think that's a fair thing to do
because Stuart Russell isn't really
creative so at least here I think he has
yes at least half right that this is not
as Roman but there are many other strong
and presented so then there's going a
bit back to the paperclip Maximizer
that's the problem of the single goal
because the curing cancer and paper
clips assume a single goal but things we
build right now how always have multiple
goals no current artifact is designed
with only one goal
and one of the obvious counter arguments
to this is that the AIS that we make
like alphago and fa0
they have to a goal of winning in a goal
or chess or whatever so so there are
obviously examples of this and quite
obviously really Stephen King who goes
even further and say this is a straw man
and that's of course a very surprising
thing because by definition is pure
person
instrumental argument is strong and
again his finger senses any insulting
system would be idiotic to pursue only a
single goal so this can sound like he's
still missing this as a as a paradox
maybe even so for fuming I think it's
false by the orthogonality thesis which
says that you can combine any level of
intelligence with any goal but I think I
can see where Stephen finger is coming
from because if you don't if you haven't
studied artificial intelligent look at
it in a mathematical way then things
like balancing multiple objectives and
making trade-offs between them and so
yeah that seems like it's a reasonable
deep part of intelligence and
unfortunately it turns out if you try to
write it down in a mathematical way this
is just a utility function and ranking
of futures and this having multiple
balancing multiple objectives to enough
to be quite trivial and here I think
Stuart Russell should have approached
this in a very very different way so I
think he should instead of says but this
sounds like it's really a big problem
but for naman gentlemen the possibly the
smartest person in the entire world has
looked at this and have proven a theory
on for nine men Morgenstein utility and
this is actually something that is
Tobias soft so I think it's and and it's
unreasonable
Foster Brussels to just assume the
student thinker can can understand this
easily the next toughest is on real or
perceived safety well it's true Russell
says that nuclear power was an example
where engineers were careless with
safety and people got scared and so the
full potential was not realized
the teachers fixing comment' so again
Ginga is obviously bright overreaction
can have very substantial costs but I
think they're talking past each other in
the way this job Russell say we as
engineers must be really careful with
safety and Steven Pinker's as we as the
public must not react irrational to to
this kind of Phineas I think they're
they're writing for different organs
Russell is talking to engineers and
Steven Pinker seems like he's you know
trying to make the public more rational
Bryden books to general audience the
next question is the question of small
parts here again a direct quote you
could say well even if the odds are
small the damage would be so kind of
strong ik than in this worth our concern
that's the thing one would say but sure
Russell does not say this and the
general AI safety advocates don't say
this so here we're definitely going into
a strawman territory further quote the
existential risk community and the so
called rationality community argues that
since the expected utility of extinction
is minus infinity probabilities do not
matter and so well first of all it's
impressionist rationality community but
this is a mistake on Steven Pinker's
side this is not something that is being
argued at all and many people have
argued strongly against this Steven
characterizes this as a moral hazard I
look up on Wikipedia what precisely a
moral hazard is and a moral hazard is
something different a further argument I
still think is then we can't worry about
every possibility at this point Stuart
Russell yes it's the same to make some
kind of reply all the rest of it has
basically the ovens of this point have
been Steven Pinker talking uninterrupted
so finally we get something that looks
like a debate and
on most of the previous points of the
ignore so that's two roses saying we
should worry about things that we put
effort into bringing about and if we put
hundreds of billions into AI then this
kind of thing should be something where
we should consider what would be the
result of this further to this that
money is pouring into AI but not into
super optimizes tasked with curing
cancer and with the power to kidnap
people so obviously these don't exist so
I'm just going to pretend that Pinker
said a TR he is a subfield of AI is
often called the goal of how much is the
GIS projects but so I think probably
billions but I'm quite unsure about how
much money is actually being spent on
AGI and that's of course also something
that should be cause for concern how
much is being spent for you could
probably argue that a lot of the money
that's being spent in AI that doesn't on
the face of it have anything to do with
eg I probably also have a derivative
effect on eg I may get higher prestige
or maybe some of it can be used or so I
could see some argument here that Stu
Russell is pushing the case when it says
that we're putting hundreds of billions
into AI because most the vast majority
of this funding will will not be
relevant another case where we get a
debate is on competition where we recall
that Steven Pinker was says that
intelligence he didn't say it wasn't
real right but you still Russell is
ensuring like intelligence is indeed
real and the example he's using is that
you can see
don't have much of a chance against
humans seer pinga answer to this issue
the systems that are smarter than us
will therefore be in competition with us
mr. Russell answers with the example of
a speed super intelligence like an AI
general education much faster than us so
obviously they are talking past each
other but let's see precisely how they
are talking part each other so it's true
residence making the claim that AI
control is hot because intelligence is
real and a super intelligence could be
much smarter than us be able to us Mars
and then Steven Pinker replies that air
control doesn't actually matter because
we will because alignment will be easy
and then they just Joan Russell will say
that any alignment is hard and give some
arguments for this and woman they flick
back and saying that okay alignment
might be happen is available because
control is easy so this way they are
able to talk past each other
continuously let's go back to the
multiple goals and see what - Russell is
saying first everything to say that all
these scare stories involves Jerusalem
who reacts somewhat aggressively and say
this is a complete red herring and
Morgan Stan you which is the reason why
it is and if they are equivalent single
goals and multiple goals and it makes
sense to focus on the simple case where
there is just a simple call Stephen
think that it makes the following
statement as you go down the table of
possible risks you're going into
potentially infinite risks with to
maximize people's lives you makes my
staplers combinations
in other ways where you like if there's
10% risk of a single goal AI so we don't
get any Pascal's wager like propers with
infinitely small risks was Russell on
alignment control this time we have no
clue about how to get within epsilon of
the true human ranking over futures and
I imagine that a number of business were
confused about this so let me try to
explain in more details what he's
talking about so for instance we have
during cruise work that we talked about
two weeks ago and that would be an
example so we do we indeed have a clue
about how to do this and let's go with
the term cruise proposal so this is a
graph here with an x and y axis and on
the y axis or gold we have the perfect
human utility function capital H and
let's say that a challenger give us an
epsilon and say you have to get within
one percent of of the true human utility
function so there will be an epsilon of
0.01 if you need to do that then you
need to join Cruz won't say that you can
get the precise human utility function
if you have a perfect world model but if
you want to just get into with an
epsilon then you don't actually need a
perfect world model you can just have a
world model that is a delta from the
perfect world model so maybe 95% correct
or something like that
so the way this works is that if you
need to be 1% from the true utility
function then you need a world model
that is then maybe a 5% from the true
one and if you need a better
approximation then you also need a
better world model but this is the kind
of epsilon Delta kind of proof that mr.
Russell is talking about when he says we
need to get within Epsilon Steven thinga
replies that even if there isn't an
epsilon that is a really
even if even on alignment of I want to
be agency is further Steven Pinker is
saying the Super's
might be ha we can take into account the
fantastically chaotic and unpredictable
reaction of humans you must make some
economics this goes on you know
predicting what humans will in aggregate
do and this
I think that's really an argument for AI
safety work right if it's hard to
predict all the ways that a plane can
crash well then you should probably work
hard on making the plane safe but it
also makes it hard to achieve super
intelligence is that true well the main
way I think about obtaining super
intelligence is through recursive
self-improvement as part of a
intelligence explosion and there are no
humans involved in this at all
so that that doesn't seem like is
perhaps what he means is that because
humans also carry an unpredictable then
in a super intelligence might be only be
a limited power in the even the smartest
person can't predict the stock market
obviously can't predict cultural
developments in one year and it's
possible that is super intelligence but
might be much smarter than the smartest
person ever and still be unable to
predict cultural developments in one
year it's also possible that if there
are no correlation between human
intelligence then building a super
intelligence might be very difficult in
the sense that you know if it doesn't
compress down to a single factor then
you might in order to be smarter than
everyone you need to have like smarter
than 8 billion brains that could also be
what he's mean but I'm unsure and it's
something that's still named here and
I'm not really succeeding I don't really
understand what's Jim finger is saying
here
June Roslin has another example here you
can view the fossil fuel industry as an
AI system maximizing a poorly designed
objective and has we just outwitted the
rest of the human race Steven Pinker has
a counter-argument
just people like energy and fossil fuels
where the energy is and there are no no
countering the externalities and Scott
Alexander has also talked about you know
things that are not super intelligence
and his
also said that we should be very careful
about seeing corporations to super in
seconds and I I agree with Pinkerton and
against you Russell because I awaited
the the people in the following there so
it doesn't in my book about the because
you know if we can see the risks because
we but in the world but it doesn't this
was mostly and movement because people
are afraid of it rather than seeing this
as a trend so I'm not really sure these
examples hold but it seems like Steven
Pinker is in general you know in favor
of safety and he is it is mistaken on a
lot of the arguments
but I think in general in principle he
would not be opposed to a sec that's at
least my general Chiklis that is all for
tonight thank you and see you next week


252. Propositions Concerning Digital Minds and Society 1

hello and welcome to session 252
in the aisft.com reading group tonight
we'll be discussing the first half of
the article propositions concerning
digital minds and society by nick
bostrom and carl schultman
this is a work done by the future of
humanity institute where
nick bustrom is the director and carl
schumann is senior researcher it's a
paper that has recently been published
as version 1.10
and
there's been a draft circulating
for several years
i think this is a thing that happens
very much in the future of humanity
institute that people send these drafts
to each other and obtain a large number
of comments
so uh that's probably
we probably don't see what they write
until
a couple of years after in many cases
and of course we're only taking the
first half today and i would expect next
decision to be in the second half
nick bostrom and carlton starts with a
rather expressive
disclaimer saying that this uh this is
very temptative work there is a
they're not trying to make some very
strong predictions about anything
they're just um saying this
is a thing a consideration that you
could think of um without making um
uh
like any claims of completeness or
thoroughness or anything like that
and some of these more philosophical um
state propositions have been a bit tough
for me to evaluate in the sense that i
don't have any uh formal philosophical
training and
in many of these cases they're talking
about consciousness and moral value
without really defining these concepts
and
very often to me
it doesn't matter that much precisely
what definitions you are using but in
this particular case i actually believe
that the definitions might matter a lot
because when you're talking about things
like consciousness um in from a
strategic point of view where people
want to maybe help the ais and maybe
feel compassion for the ai so we'll let
them out of boxes and things like that
in this case it matters very much what
um
what
the people who are currently interacting
with these ais um do they believe that
they can suffer for instance and and the
the moral frameworks that they are using
might be really really interesting and i
expected different moral frameworks to
have very different answers to some of
these questions that are being raised
also i should say that i expect that um
for
with ai safety we are facing a an
imminent catastrophe on ai and that
makes
thinking about whether ai's suffer and
thinking about where where the mind
crime happens and all these things we'll
talk about today a bit strange in that
there is another issue that is rather
large which is will we all be killed
tomorrow um and in that case um
why should we care about the moral and
not just care about the instrumental
considerations
i think there is an answer to this that
we should in fact care about the moral
consequences and the moral
science because they very often directly
orientately influence the instrumental
factors so i do believe that it is
important but i also think that the
authors should have
to to distinguish these two things
because
it is a very different thing
like how can we avoid getting killed and
how can we um
create a best the best long-term
scenarios etc and all these things
avoiding suffering and
things analogies to factory farming and
what happened
the first is the first part was the one
i had the hardest
time getting through and that's
consciousness and metaphysics
this
is a uh a meme i found on the internet
uh from from the authors
stating that there is in fact no
difference between artificial neurons
and biological neurons
and this is formalized uh this is not
from the paper of course this is
formalized as the substrate
independence thesis that you can have
mental states on
many kinds of physical substrates
in what matters is the computational
structures and processes and that's
enough for consciousness and not whether
it's good happening inside a cream or
something like that
and this is uh asserted to be true um
so the the chinese room argument by
cheryl is uh not accepted by nick
bostrom
uh
i think i would probably agree with this
but i would care more about what do
other people think and i also kind of
feel that this is somewhat weak in the
sense that the word mental states
like does that mean quality or what like
mental states that you know to be like
purely functional and
in that case the substrate independence
thesis is obviously true
um
so uh in this case the uh
one of the consequences of this would be
that a an immolation of a human would be
conscious
and we could also see ai's that are not
immolations of humans but doing
other things that are in fact conscious
and so the obvious question for me is
why do we care about whether they're
conscious we care more about whether
they can kill us well um
if they are conscious they probably have
interesting moral worth of some kind and
and that might matter very much to us
when talking about consciousness uh the
naive way of thinking about
consciousness is like either you haven't
or either
or you don't
but the the authors nick bostrom and
carterman think that the quantity and
quality are a matter of degrees
in
quantity obviously you can have a number
of copies uh you can have repetitions
you can have them running for a longer
time both by having faster computation
and just
more world cup time
you can have
implementations that are more or less
robust you can have them always closer
to humans in a different ways how you
measure that
and i would argue that might be less
regarded to relate to quantity and more
into quality but that's really a
nitpicking they also say that
there is a great great quantity if it
has greater quantum amplitude and i try
to uh look into what does that actually
mean i did spend five minutes on the
internet trying to figure out what does
it mean
that these processes have a greater
quantum amplitude and i didn't really
understand that so i think they ought to
um describe that in in more details and
i don't think it matters very much
i can see someone in the chat has
already written something about that
um
so uh
quality that's also a a matter of degree
in that like how where are you of your
consciousness how certain where are you
is it like a good or a bad experience
that might also matter very much and how
strong are the desires moves and
emotions and things like that
um so i think that's
you know the the conscious experience is
the very
matter of a degree i think that's a
reasonably well established proposition
they make some more assertions about
consciousness
if you have two runs of the same program
do we get twice as much conscious
experience
scott alexander has
written this answer to job which argues
that in fact you don't get twice as much
uh
conscious experience from from this and
the the uh
the obvious uh perverse incentive
instantiation is where you find the
optimal conscious experience and then
you just have a computer
run that single single program again and
again and again and is that really
something that is valuable to us
they have another statement here that i
think is interesting significant degrees
of consciousness require significant
computation and complexity is that
really obvious i don't think the
complexity might really be required i
could imagine an extremely extremely
simple um
ai that is basically the neurons that
like perceptrons or even something
totally totally simple and then you just
have a lot of that
and if you have enough of them then you
get something that is conscious um i
don't have a strong model of our
consciousness so i can't definitely say
that but i think this extremely non
complex um system would in fact uh could
be be conscious
and also like how much computation is
required i would imagine like
an optimal bayesian uh like um an ai
designed by
like optimal whatever that would mean um
according to the the limits that we have
found um an optimal reason might require
even very very little um computation uh
like an update that is literally only
doing base uh reasoning just once that
requires like one flop or something like
that uh three flop i think for base
applying base room and i could see
a system doing that which was in fact uh
conscious so i i think i would reject uh
uh this proposition
they have another proposition uh many
animals are more likely than not
conscious which is a uh very weak claim
in my in my opinion um
you could try to negate that and that
would be that
that most animals are probably not
conscious um i think that's a um
like uh the way i would i would state
this would be that
probably
most animals are conscious to some
extent uh i am very bullish on uh
animals having some kind of sentience
unconscious and ai are present ais to
some degree conscious well we have we
had uh in the last session a long
discussion about landa
where i came out uh as
as much more
uh
putting a much higher probability on
london in fact being conscious than most
others both others in the reading group
and people in general i think there is a
substantial probability that london is
in fact conscious in a in a meaningful
sense and um
well most people would consider it
obvious that they are not and the
question at least um puts uh some
probability mass on this
but again consciousness is difficult to
define and the way most people seem to
define it is i know it when i see it uh
and
in this case obviously uh current
language models are strongly not
conscious but that's a very uh
redefinition of consciousness but one
that may matter very much from an
instrumental point of view because
people are not going to give lambda any
kind of rights um or any kind of moral
consideration right now um based on the
theory of consciousness and the theory
of consciousness most people don't have
though like if you ask the man on the
street to explain global workspace
theory you won't get anything at all
right uh so they define consciousness in
a in a much more naive way
and from this it's clear that uh
current ai's are not considered
conscious
okay how about uh if we emulate the
brain that's obviously something that
could be conscious and that's something
of course also very different from
lambda and and currently eyes uh
i would i totally accept of course that
they can be conscious though does that
constitute survival for an emulated
human
um
i am not entirely sure i would accept
that proposition um like the obvious
counter example would be if i was
non-destructively scanned
and
emulated somewhere and then killed
without count as me survival like i
would strongly object to that
so
it's possible and depending on your
philosophical views certainly it might
be something that happens uh here again
they say civilization need not be
morally objectionable and uh like
i think i would probably consider that
suicide and suicide is something that
you can appreciate you can also say that
this is something that people can choose
and people have the right to choose but
i at least have a right to object
to suicide
the authors talk a bit more about
what theories of consciousness and what
they should explain and things like that
um
and uh they end up deciding so for the
theory of consciousness because if you
literally interpret them then very very
simple systems could indeed be conscious
that's an example that i've previously
in the reading group talked about that
if you um
some of these definitions of um
of consciousness is like self-awareness
and it's obvious that my windows
computer is self-aware in the sense that
it has a model of this pc and knows like
what hardware is on and what's running
in the kernel and all these things
so if you define consciousness in this
simple literal way then you run into
this kind of problem and of course uh
the answer so this is probably that the
people using these definitions are not
using them in the literal way but
when you're not using the definitions in
the literal way then you are having bad
definitions and precisely what uh you
should do instead how to interpret these
uh it's something that like
it's probably because i don't have a
strong philosophical background if i had
a stronger philosophical background i
would more be more likely to understand
what they mean instead of just reading
what they literally write
and
based on some of the
criteria integrated information theory
that's one of the things that that fails
global working theory is one that i have
criticized earlier i can't
remember what my criticism was but the
authors are a bit more optimistic about
that
and some other theories
let's go to the next uh chapter
respecting ai interests
first they talk about moral status
without uh defining moral status so i
had to like look on the internet and try
to find
what i consider to be the most
widely used definitions of moral status
what does it mean to have moral status
and again like it is perfectly possible
that
both that the authors uh nick bostrom
means something different and it's
possible that most people mean something
different and i would really like to to
be certain about these things and i
would like to
know in particular if there's any
difference
like it's it might matter a lot if nick
bostrom have some of the underlying
ideas about uh things like moral stages
that differs from what most people think
because again it matters what most
people think
right moral status an entity has moral
status if and only if it has interests
morally matter to some degree for the
entity's own sake
so again here i found
a diagram where like um you have moral
stages if there's a good of its own
sentence and moral agency um
i'm not vouching for this being like the
uh
the most common view but uh but i think
it's
it's like it seems the most conscious to
me okay so
uh
we've if an ai has moral status then we
have a moral obligation to uh consider
uh like not harming considerate wealth
there but who are we in that sentence it
could be the ai developers the
researchers creating uh new new
technologies that might uh enable
consciousness
it might also be the ai juices the
people who instantiate the ais
or it might be society in general and
the view from
the question of the proposition is it
false
on all three
uh i think um
i think it's a uh a problematic thing
because i i think it matters a lot not
just who should um
who
has the moral uh obligation to consider
their welfare but who also needs to
figure out if it is in fact um if it has
a moral status
and the way i probably think this works
out in practice is we have a wonderful
triangle of um absorbing uh
responsibility
in that the ai developers
if you read some of the papers they
write they write a lot about that we
need when we actually use this
technology the people who use this
technology need to take care that we
don't uh use this for for anything
negatively so
ai developers researchers in general put
the uh
uh the owners on the ai users
and if you ask the ai users obviously
they
put the uh the responsibility to uh
society in general like this is not
something that um
that the the user need to do uh to to to
know about this something that should be
laws and like ethic sports in society at
large that handles this question and
what does society that um who would they
point to uh if you uh if you ask them
like who should consider if this ai has
moral status well society in general
would point to the researchers so we
have a uh a wonderful circle where
everybody points to the to the other
person and no one in fact takes
responsibility for this as far as i can
tell no one is really taking
responsibility for this
um and
i think in general the question makes
more general error here in assigning
responsibility to three parties uh in
particular assigning responsibility to
society in general is something that
just basically never works right uh and
uh
elias erkowski would uh argue that
that's not how it's done in
land where responsibility is explicitly
uh pointed to one particular person in
all cases where this is at all possible
okay so but let's actually dive into the
moral status of ai why does it matter
well we if we don't
uh consider it we could before we uh
like even consider the the issue
suddenly we have something like factory
farming which can be uh like they the
authors don't argue but uh the part of
the problem is this kind of thing can be
really sticky in the sense that if we
didn't have factory farmer and someone
said hey is it okay we keep chickens in
like these conditions then obviously
people will say no that's abhorrent but
if this is the way we've been keeping
chickens for like forever then people
will say sure that's how it's always
been done and it's quite possible that
uh the way we treat ai's can be sticking
in the same way
and of course it's instrumental for for
other reasons uh like the the argument
before was on pure on the moral side but
it's also instrumental in
in that the people who care about this
might not be uh
the people who want to care about this
like
blakely mine would be an example of
someone who cares about this and
is not necessarily
thinking entirely rationally about and
going entirely rational about it and it
only takes one person to let an ai out
of the box
for instance
okay
another proposition what's good for an
ai can be very different from what's
good from a human being
on an object level that obviously makes
sense right and ais we should be worried
about informal fighting etc uh
but there is like i think like
preference due to
uh
um like what does
what we should do with the ai is to
treat it as it says it wants to be
treated right um so so
it's not certain that uh
on the meter level there is actually any
substantial challenge here
then uh another uh
issue that uh nick bachmann has
previously written about
agents with stronger
interests super beneficiaries and
greater moral stages super patience um
is that something uh that's certainly
something that can happen i can see it
happen and i can also see a substantial
instrumental point in denying this
in that
if we want to have reach some kind of
agreement motors with nd uh with ai then
uh accepting that ais can be utility
monsters uh is probably not gonna fly at
all there's a strong selling point
considering everyone to have the same
level of um of patience being the same
kind of patients and not being super
patients um and um
and that's probably really instrumental
to keep
possibly um
and they have an another interesting
like what do how do you actually treat
copies of ais do they have like
responsibilities for what the previous
version has done before and the authors
are yeah actually
they do
they relate to their private previous
time segments just like humans do i
haven't seen that argument before and i
i thought that was actually kind of true
and i wonder if this can be quantified
and i think actually can be quantified
uh by parallel with with humans uh where
like uh some crimes do in fact uh
uh
get too old and can no longer be
prosecuted uh i think can we quantify
the um
uh the rate of decay to some extent
i thought that was a cute i mean
okay we talked before about treating the
ai the way it wants to be treated so
obviously consent seems like a way we
could be more moral in our dealings with
ai and so if it can give informed
consent it should be required and if it
can value it whether it has a good life
then they should approve of coming into
existence
i think those are good i also think
there's a very obvious loophole here is
that you can just design the ai to not
be able to give informed consent and
then it's not required uh
but uh so so we are really pushing the
level pushing the problem one level up
but but it's a star ceremony
um
inform consent
is that obviously reliable uh
there are there are ways to get around
that
we should try to avoid miserable ais
and i think that's of course morally a
really good thing but not obviously a uh
instrumental thing
designing an ai to have specific
motivation is that generally wrong
i don't know uh it's an interesting
question it's been uh with humans it's
been uh
evaluated in adulthood uh
uh brave new world where humans are in
fact engineered to have motivations that
are uh useful uh and that seems like uh
some uh it's a dystopia obviously
and so
it it's
certainly possible that people will say
that this is a background
and a nice statement we should when we
build ai's and to the extent that we can
control their preferences we should try
to have them compatible uh
with ours um that seems like a good
instrumental thing but it also requires
that we have substantial control over
the preferences of the ai and that's
like a substantial part of the the
alignment problem um and so for this
reason there is a strong moral component
to solving the alignment problem so we
that's another good reason to solve the
iron problem but we already have really
good reasons to solve the alignment
problem to avoid getting
killed
avoiding discrimination um
uh bostrom and schumann uh presents two
uh
uh principles substrate
non-discrimination
that um
if two
beings intelligences are the same except
that they have different substrate then
they have the same moral status
and here on untouchable
non-discrimination if they only differ
how they came into existence like one
was born and the other one was
created copied in some way
but also doesn't matter for the moral
status
that sounds reasonable to me
and obviously um that that seems good in
itself and um this also seems like
something
which
could be used for uh
as a basis for some kind of cooperation
between humans and ai
possibly um
this is somewhat unclear to me is not
something that i have thought a lot
about because this is something that
kind of happens after the singularity
you can say
and they have a an idea that if there
are aliens that are
artificial intelligences or the future
that is built on ai
they um
might see if we discriminate based on
these two principles they may um assess
us to have low moral worth and that may
be instrumental bad for us
um
sorry
i think in particular for the aliens
that's obviously instrumental but
whether the um
ai aliens would in fact see us as
more worth saving because we care about
our ais that seems like
anthropomorphizing and for performancing
uh really
quite a bit i i have a hard time seeing
that really being relevant but it's an
interesting consideration i hadn't
thought about it
contractualism
uh i think that's harvest um
the question has a uh
uh in contractarian views ai's that have
the potential to reach a position to
greatly help bahamas beyond our control
may have an elevated social moral status
in a non-determining hypothetical social
contract
uh i think that's interesting in the
sense that i am not a contractarian i
have a hard time emphasizing with this
uh in the sense that this seems
totally obviously wrong and like just
because the guy is able to kill us that
doesn't mean that it should uh that just
means that we should try not to build it
uh and try to make it not want to kill
us and trying to negotiate in this way
seems strange to me but um but it's
always nice to see people who are not
utilitarian uh uh
try to grapple with these things because
most of the uh moral and ethical
considerations with regard to ai safety
and
transformative ai have been made by a
consequentialist utilitarian
they uh from the constructarian point of
view they have some uh considerations
like we should open compensations if we
uh try to align them put them in boxes
for instance um and if they help us a
lot then they are also owed some kind of
conversation
um especially um
if we can give them the conversation
afterwards so the idea is we put them in
a box ask them to solve the alignment
problem and then once they solve the
alignment problem we can we have aligned
super intelligences we have resources um
uh all the resources we would want and
so we can give them quite a lot um and
particularly we can give them another
success so the ai that was forced to be
in the box and develop
and align the ai somehow
can then be preserved and even though
it's online we can still uh give it like
its own galaxy if it wants to have that
because we can uh in fact have a
sufficient resources for that
um
nick bostrom is uh
uh positive about this kind of thing he
even says uh yeah it's more promising uh
i'm of course a lot less positive about
this because i'm not a conjuctarian and
i think that the opportunity costs are
very real here in that uh if we try to
uh bargain with some kind of ai the way
it will look like in practice is like uh
we are totally crazy and uh this kind of
in negotiating with an ai a costly in
this way has
uh like we spent a lot of time on it and
we seem like we're totally crazy if we
do that um and also i don't expect this
would work or be particularly ethical
because
i'm not a contractarian so i
i'm less optimistic about this this
approach but i appreciate the the novel
perspective
security and stability um
in a transitional period we will need
special precautions for ais in uh who
are misaligned
and
to
i think that's obviously trivially true
but it also
betrays uh an uncertainty in this word
in this work about to what extent is the
alignment problem in fact solved
um and uh both it has probably been
solved and has proposal act been
performed because we don't know in
particular whether
uh like
i expect if we don't solve the alignment
problem at all and we don't do any kind
of pivotal act
then
we lose everything and nothing we do
matters all value is lost whatever right
um so um
whereas if we strongly solve the
alignment problem to the extent that we
can just
get a perfectly aligned ai and just ask
it
maximize our
query extrapolated relation or something
like that then the problem is also very
small
this kind of consideration seem uh much
more relevant in a middle case where we
don't solve the alignment problem like
perfectly mathematically but we kind of
do uh solve it somewhat and we don't
have a really strong pivotal act um
where we have some world government
single tongue they can just say don't
build misaligned ai but we do build some
middle line ais but not enough that they
are able to actually destroy all value
so it's kind of like in between uh and i
think the authors would have been
uh
well served by making much more explicit
what kind of scenario they are
envisioned
along these two uh dimensions have the
alignment problem been solved and hasn't
revolved like being performed
and
if we have ai again with uh
then
which is not perfectly aligned and
not perfectly
controlled from by a singleton or
something like that then we could see a
large number of
uh new risks going on uh and we if we
look back historically there have been a
lot of walls and a lot of appropriation
events and revolution and things like
that and this might happen on digital
times deals instead of happening on
biological time scales and that will
mean that um
you know if you had like a house
uh in the year uh 1000 um and then
the austin you would still have that
after um
in the year 2022 would be very low
in the sense that obviously in denmark
like if you had invested in the stock
market in denmark in 1900 then that
would have been really foolish because
in 1941 the germans attacked and took
everything and so obviously you would
have lost everything um and the united
states is probably the only place in the
world uh where i can see someone
investing in the stock market in 1900
and having the money in fact in
2022 i think there's been exploration uh
events just about every united kingdom
maybe um
there might be a few other places sure
but in general um if you want to have if
these kind of things happens
like a hundred a thousand times more
often
then we're gonna
we can't expect to have any kind of
continuity
the uh
uh robin hansen's book hfm has an entire
age where huge a huge transformation
like uh comparable to uh like the
industrial revolution and the things
happen when
the humans who are outside looking at
their clocks say that uh between one and
two years have passed and of course the
uh the de facto power in the uh uh on
earth have uh over this age of m shifted
uh totally to the to the emulations um
and so uh if we can need to um you know
keep our property and keep our existence
through this then we need some really
stable predictions
um
and probably we need them soon before
the uh
uh
ais starts to have real security
implications
um
so can we get really stable
institutions we might um we could have
things like treaty bots uh like here you
imagine you have two actors uh maybe two
uh ai's maybe humans and
um ais uh combined in some way
negotiating with each other maybe on
national levels or whatever um they
could make treaty bots and then
you know
perhaps the weaker part the less
intelligent builds it and the more uh
insulting um
verifies it and then the the treaty
parts check that they
hear some kind of treaty is that in fact
feasible we obviously don't know this is
very speculative uh technology i think
that it might in fact be quite feasible
because
if we if 3d parts are going to be
relevant in in any case then we will
have solved the alignment problem and
having solved the alignment problem
would probably like depending on what
precise solution is available i could
certainly see that uh having a very
strong positive
influence on our probability of being
able to create treaty bots so uh
conditional on a
on conditional on solving the alignment
problem to the extent that we don't die
i think the probability of 3g buzz being
feasible is very high
the question also talks about um
internal inforcement bots and that's
probably feasible i think what
what they are actually regarding about
might be mis optimizes but um
they don't use this word so i'm somewhat
on until one thing like precisely what
um scenario they're thinking about
another thing that could enable really
stable institutions is the fact that
minds
that are digital can be copied exactly
um
will this in fact be
a feasible idea that kind of depends
like um in the book super intelligence
nick bostrom outlines the three kinds of
super intelligence quantitative super
intelligence speech super intelligence
and quality super intelligence where
both quantity and speed super
intelligence seem like they would enable
very stable institutions whereas quality
super intelligence
probably don't but again it's it's
difficult to see
and some talk about like if we in fact
end up in this situation and we need to
have ai security and military forces how
do we deal with that and they have some
speculations about this very little and
i think uh
probably if we are in such a poor
situation this is required like we have
really haven't gotten eight people left
then probably this will end up being
impossible
okay let's try to
figure out the rights of
if ai's have a
some kind of moral status then how do we
balance their rights with ours given the
fact that the ais might have super uh
human abilities in some cases and that
would require some kind of adaptation
one of them would be copyability and
humans
take quite a long time to reproduce ais
could reproduce
exceedingly rapidly um and it's uh very
clear that trying to make any kind of
formal rights uh for ais will need to
grapple with this very very early um
and the freedom of thought is another
right we would really want except that
uh
when i
think
uh like mind crime seems like an obvious
problem for ais because they are able to
instantiate to much greater fidelity
like if i think about a person and think
about that person's suffering then the
simulation that i have inside my head is
of really poor quality and so it doesn't
seem like something that like there is
uh something really bad happening
because i i'm so poor at imagining other
people um but in a i might imagine them
in perfect quality and that might make
my crime a a real problem um
so that requires some very very strong
enforcement powers and i think the
strategic implications of this is
like um
i i don't think they have been anywhere
near completely uh thought through a lot
of the uh uh implications have like oh
we have perfect uh
interpretability and something like that
and if we don't have that then what do
we in fact do
i think the uh
the implications there are uh
dark and complex and not have and
haven't really been explored
the third uh right they talk about is
freedom of speech and um
they don't really that they don't really
write more about that and i think this
is another really interesting uh topic
like what do you do if you have an ai
that is
consistently able to persuade a human of
basically anything
uh how do you co-exist with this kind of
ai obviously you you'd want to restrict
this ai from being able to communicate
with uh
unprotected humans because you'll be
able to convince them of anything um i
think it's an interesting problem and
precisely how to deal with that it
hasn't really been explored fixed
explored in this field
let's go back to problems of branding
the ai's reproductive rights what if we
don't if we just do that uh well uh in
that case we will see some evolutionary
dynamics and the default outcomes
may not be good as they write and i
think that's a real understatement
because we would see ais copied based
purely on their ability to
on their reproductive fitness and that
is really really unlikely to be any kind
of
thing we want so saying that it may not
be good is really an understatement
so one vote democracy
can't really hold and the social safety
net that seems like something that can't
hold i think there are there have been
quite a few
people trying to grapple with these
things and they have found
some ways around this
um again i think a lot of this isn't
really that instrumental to us
um some of it might be but i think uh
um like if an ai
for non-instrumental reasons or
for instrumental reasons try to create
the successor and that successor is then
not happy and then
sure it sucks for that being because we
have a
highly intelligent non-happy
ai but
it's it's not necessarily that
instrumental until the ai
you know um if it just suffers in
silence even though it's morally
very bad
mind crime
um
any questions is there's great moral and
practical importance of what happens
inside computers in an era of digital
minds
and
like minecraft the way i typically
envision minecraft is that it does in
fact have very little practical
importance uh can be used perhaps for
blackmail but most likely
like there is a
a strong general counter to blackmail
which is to self-modify into a someone
who never gives into blackmail ever and
then you won't be blackmailed
so i don't think this kind of blackmail
would necessarily be a big issue
what can we do about the minecraft well
we could do something akin to a child
protective
services
um and then maybe um like you could have
a digital inspector that only returns
one bit does minecraft happen within
this ai
um
uh cyber security is another issue uh
and part of the reason why we want it to
only return one bit is that we expect
ais to care very very strongly about
cyber security because uh
the uh the ai probably
in most cases the most valuable thing
the ai has is
its own data its own uh
processes
and this may
imply that we could see a single attack
uh
uh
getting data from uh as they as stating
a single act of piracy could uh transfer
basically all the wealth of one state to
an attacker and that would split uh
push the incentives strongly towards uh
offense rather than defense um but well
this is actually something that will
happen is quite unclear to me i think
there are
probably ways around this uh it's not
clear to me that offense will win out of
the difference in in the age of uh mine
crime
again
misaligned ai needs to be close to
surveyed uh
and
to what extent we have misaligned ai at
this point is
uh
so the authors ought to go much more
into detail with um if we have solved
the alignment problem we might have
really great interpretability tools and
then we can just see this migraine crime
going on
and the final part of the first part is
about regaining resources from space
because
ais that enable greater
strides in robotics could
you know
obtain great resources from space and we
don't really have a stable framework for
doing that and this would be something
that has these resources would both have
an economic and strategic value um
and they're speculating that we the
thing we could see is a misaligned ai
trying to expand through the universe to
obtain resources to attack rather than
immediately attack
um
i think that's not really
obvious and um i also don't think this
is many implications because if the ai
wants to do that then it can do it so we
want to
make the end up want to do that and
that's solving the alignment problem
uh we also immediately suggest a
supplement for the outer space treaty
the art space treaty is clearly
insufficient but i feel
trying to work on that is almost
certainly going to be a waste of time
because we are assuming things so many
things before that's going to be
relevant like ati is possible we will
have a super intelligence and we'll
mostly solve the alignment problem but
we won't actually totally solve the
alignment problem we won't have a strong
pivot leg but not something really weak
something in between that will allow a
multipolar scenario and this needs to be
reasonably stable to allow for uh these
kind of
processes that harvest most resources in
the universe and then we need to have
advantages to grow first rather than
attack first and we need to ask someone
who's willing to
obey the uh outer space treaty and have
the correct incentives to obey this
treaty
and only in that case doesn't make sense
to work on making a better outer space
treaty and i think all this is
so incredibly um
all these assumptions are very unlikely
to hold at the same time so i would
strongly expect that it's not worthwhile
to improve the outer space stream
that is all for today thank you and see
you next time


AI safety | Katja Grace | EA Global: San Francisco 2017

welcome back Katya grace is next she
went to the Australian National
University where she studied human
ecology and science communication her
honours thesis was on anthropic
reasoning and the Fermi paradox
she has broad interests including
signaling theory anthropic reasoning in
the history of Technology and she has
worked on AI forecasting and strategy
since 2013 when she realized that you
don't need a PhD for that today she
blogs at meta euphoric and his part of
the AI impact project at Miri please
join me in welcoming Katja grace
hello I'm one important question I would
like to answer today or help answer is
what can a person do to help with AI
risk if they don't have a technical
background or like that much interest in
getting out of bed that sort of things
if if that's of interest to anyone I
think you're in luck if it's not of
interest to you I think you're also in
luck because I'm not going to talk about
it I'm just going to tell you about some
things we've been doing it AI impacts
and I think these are good examples of
the kinds of things you can do if you're
in that situation but I'm just going to
talk about them alright so one question
that's good to address at the start of
trying to forecast AI is how much
hardware and software are contributing
to a role progress in AI so I've tried
to answer that a bit in the past I'm not
going to talk about that research now
but it came out basically hard to tell
the difference between 5050 and whatever
the real answer is so AI impacts just
ran a really big survey of machine
learning researchers the Machine the
machine learning researchers who are who
publish in top machine learning
conferences and so we got a chance to
ask them what they thought and this is
the distribution of their answers in the
top four how much less AI progress would
we have if there had been a half as much
progress in algorithmic progress sorry
half as much algorithmic progress over
the last decade as you can see they
don't know the answer or it's there's a
very wide distribution of views that
with 50% less progress if there was 50
percent less algorithmic progress being
popular and the I given you an example
of the question there which is the one
for hardware if there was half as much
hardware progress how much worse would
overall AI progress be we can see that's
also pretty spread out though we would
lose 80 to 90 percent of the progress is
quite popular so it seems like hardware
progress is probably reasonably
important and a bunch of people think
it's very important so we've looked some
at hardware in particular and what's
going on with it since it seems to be
important we were interested in how much
hardware there is in the world
altogether which seems like the sort of
thing someone might have been keeping
track of and someone was actually one
professor checks in 2007 and wrote a
paper about it I think he actually went
out and and like tracked down a lot of
different pieces of hardware and like
counted them all up and also figured out
how fast he saw it Hardware was growing
in the world and then someone else later
on projected forge till like 2015 how
much hardware there should be in the
world and so we're excited to find this
and we're going to use these numbers and
then since we it also checks what the
current cheapest price of hardware is we
we checked how much all of this hardware
should be worth it is in the world and
it turned out that it should be worth
somewhere between 40% of all of the
value in the world and like at least 400
percent I think so they didn't seem very
plausible and we went back and looked at
it more and these are the numbers we
came up with which perhaps to mean very
much but we can put it in supercomputer
equivalents and then it means slightly
more or I like to put it in terms of
human brain equivalence because that's
actually relevant to our concerns about
AI often people are interested in what
would happen if we did have very good AI
and it managed to hack a lot of hardware
and take it over and turn it into more
AI so here we have a combination of the
estimates of how much Hardware there
there is in the world and also people's
estimates of how much Hardware a brain
is worth it's very hard to tell how much
hardware brain is worth because we don't
actually know how to turn Hardware into
anything like brains
which is why we don't have AI so we want
to measure it in terms of floating-point
operations per second often and
floating-point operations or something
like multiplications and we don't know
how anything that the brain is doing is
like multiplications necessarily so the
estimates here vary by I think 12 orders
of magnitude which you might think is
really bad but I think it's better than
nothing it does rule out part of the
space that we might have thought
possible but putting these things
together in this in this figure you can
see the the horizontal line there that's
dark is the current world population and
the other lines are different sort of
scenarios for if you tend the all of the
world computing hardware into AI that's
about as efficient as the human brain at
sort of using hardware to get
computations so at the moment in the
sort of middle scenarios we have like
you'd get about a thousand people I
think if I'm reading that right so if
that happened you might not expect that
to cause like an intelligence explosion
or something as people have speculated
though in the future you might or if
we're very wrong about how much hardware
is in the brain or how much Hardware the
brain is equivalent to in some sense and
so we're pretty interested in the how
much hardware is the human brain
equivalent to for this sort of reason
so we looked into it more ourselves a
better way perhaps to measure how much
Hardware the brain is equivalent to is
to count how many messages are being
passed between neurons in the brain or
some sort of equivalent thing in a
computer because we can't actually tell
what is going on there so does this
measure traverse edges per second that
is used for supercomputers and it's much
easier to put the brain into a similar
into a similar sort of metric and so
using this we came up with the estimate
that the brain is about 0.1 eight to six
point four times ten to the 14 traverse
edges per second which is again not that
meaningful but it means it would cost
about four thousand seven hundred to one
hundred and seventy thousand dollars an
hour and therefore be like well outside
what anyone would pay for a brain
the reason that these things seem to be
getting better it means that we're
actually very close to we're human level
hardware in some sense is that the same
price as a sort of expensive human so in
the scenario where hardware matters a
lot you might expect that in the next
few years the amount of hardware that
people can afford are as impressive as a
human almost there like it doesn't take
very long to get the software that makes
that work so that's one kind of where we
might expect things to happen soon so on
the other hand it might be that software
does matter a lot in the survey that we
had recently there were some other signs
the thing might think some interesting
things might happen soon
for instance here are a list of very
concrete narrow things that we asked the
machine learning researchers when they
thought they would happen and you can
see the 10-year mark at the very top
there most of these things they expect
to happen in less than 10 years so
there's some confusion here from
different framing effects that I haven't
talked about you can see truck drivers
there and retail sales people versus
seem to soon be at risk as we've noticed
for truck drivers at least another
person at risk is Taylor Swift we asked
when machines would be able to produce a
song that is indistinguishable from a
new song by a particular artist for
instance Taylor Swift as as discerned by
dedicated Taylor Swift fans safe and in
the top row you can see that the
probability in ten years people are sort
of spread out in their views but more
than half of people thought that this
will happen in less than 10 years and
then by 20 years people are mostly
agreeing that it's it's going to happen
and by 50 years they're pretty sure so I
think this is interesting in part
because like if if pop star stops being
a business you can have that that will
be interesting for the world we also
asked when AI would be able to do
everything that humans can do and here
we have a picture of lots of particular
predictions that people sort of
implicitly made we actually asked them
for three different points in their in
their probabilities curve and then sort
of put in the rest of the line between
them but all these gray lines are
roughly different people's predictions
for something like an AI that can do
every task that a human can do what are
the chances of that in different years
so as you can see the different
respondents have very different answers
to this and from that you can probably
infer that they're not they're not very
accurate just because they disagree
terribly but there's this idea of wisdom
of crowds or something where you might
hope that if there are lots of very
inaccurate forecasters you can average
all of their predictions and get some
sort of accurate average which is what
people have mostly tried to do with this
kind of thing and that red line in the
middle is an average which I guess sells
like a 50% chance of machines that can
do everything that a human can do in
about 50 years but a problem with this
is in the thing survey that we ran we
asked about several particular
occupations like truck drivers and AI
researchers and I think retail people
and then at the end of that we asked
when they thought all of the professions
could be automated and the answers that
they gave to when everything could be
automated after thinking about those
other professions much later than the
ones that they would usually give for
just being able to do everything so
that's the orangey line shown here or
are there two different orangey lines
actually so there were four different
framings that the two with stars at the
bottom are where we asked people what do
you think the probability will be in 20
years instead of things
what do you think the year will be in
which there's a 50% probability or
something and across the board in all of
these questions they gave later answers
if we asked for the probability in a
certain year instead of the year for a
certain probability but there was a this
much bigger difference between just
asking straight out about all of the
tasks or asking about occupations first
and I guess we predicted there'd be some
difference but not such a huge
difference so it seems like now we can't
really say well these these estimates we
usually have are just like noisy and we
can average them because they're really
huge bias is happening but between
different framings and we're not really
sure which ones we should trust but the
other surveys that have been done in the
past have mostly been using the the most
the sooners framing here the blue one
with circles so this is somewhat good
news if you don't want AI to come very
soon in that any other way you could ask
the question we get a later answer
though it's bad news for it being good
to listen to these things at all so this
means I'm more inclined at this point to
trust these other kinds of complicated
estimates we might come up with from
figuring out Hardware timelines and how
much Hardware matters or software
matters so this is still somewhat
interesting I think because if if things
were coming very soon you might expect
like if this recent machine learning
boom was about to lead to the world
being taken over by AI you might expect
that the machine learning people would
have noticed this more or like I would
not be surprised in that circumstance if
a lot of machine learning people were
saying oh wow it's about to happen
and it seems like very few of them are
so that's at least some evidence against
that so if we get to human level
intelligence at some point there's still
a question of what will happen then and
some of these other things about how
much different inputs might affect
progress like whether it's software or
hardware and can compete in to some sort
of model of whether an intelligence
explosion where AI progress is fuel
by AI progress and erm yeah we're
they're effectively more and more people
working on AI because they are a eyes
themselves we could figure out something
like that but we've not done it yet but
some evidence we have about how fast we
should expect to go from human level
intelligence to vastly super-human
intelligence as various AI risk concerns
rely on is from going from human going
from normal human level to superhuman
level in other areas so I think in the
past I think eliezer yudkowsky has
argued that of these pictures in the top
we should prefer the one on the right
where the entire range of humans is
actually very close to one another it's
not really the case that stupid people
are down near chimps and Einstein is way
up sneered God or something that's just
sort of anthropocentrism
because a whole lot of evolution
happened before we go to idiots and
they're not very much evolution happened
getting to Einstein that sort of thing
and I think we've often been accepted
but if we look at something like chess
we we got too bad human chess levels in
like 1965 and then it's been very
gradually improving since then and until
superhuman levels in more like 1998 or
something and similar things are true in
other games here at least like in go and
checkers it took thirteen forty years
I think physical manipulation is a bit
ambiguous but it seems like we're
hanging out in the not very good range
for a long time and jeopardy looks like
it would have taken at least like 10 or
15 years to cross the entire space so I
think we haven't searched very well for
all of the cases and since a lot of
these games they might be special but I
think this is pretty interesting
evidence about whether you should expect
that so if you want to hear more about
this kind of research we have it at
impact org
and we also have a sort of knowledge
base with lots of reference articles
like if you have a question like how
much hardware is there in the world we
have a page about that if you're
interested in doing this kind of
research I think there's a lot of it to
be done we have a really long research
agenda which isn't really an agenda
because we won't get around to it before
it there is AI but if you want to do
some of it you can talk to me or you can
look at it on the internet or bits of it
or you can talk to a variety of other
people here who are saying similar
things or you can just like do it and
after you have done it you can put it on
the internet and people look at it yeah
thank you