Nick Bostrom's Q&A on Existential risk and AI

no I want to do this as an interactive
Q&A so and it any question really that
you have the two to discuss with you I
guess I want to work them right now
which is this book on super intelligence
which will look not so much at the
person of whether or how long it might
take to develop machine intelligence
that equals human Italians but rather
what happens if and when that occurs so
forget human level machine intelligence
how quickly how explosively would we
then get super intelligence and how
could you solve the control problem but
if you can build super intelligence how
can you make sure that it will do what
we want that it wouldn't be safe and
beneficial so once my starts to pull on
that problem it turns out to be quite
complicated and difficult it has a lots
of aspects to it at them you're happy to
talk about or or if you prefer to to
talk about other things existential
risks weighted or otherwise happy to do
that as well but no presentation just
purely so you'll have to provide at
least the stimulus and such do you want
should I take questions of you wants to
think I can put a bit easier way to see
what's your definition of a machine that
is much when I didn't see how do we know
yeah they're like a precise definition
there there isn't now if you look at the
main specific Italian so obviously areas
in which mesh
already surpassed humans so in doing
arithmetic or populations or chess I
think the interesting point is when
machines equal humans in general
intelligence or have slightly more
specifically engineering in television
so if you had this general capability of
being able to program creatively design
new systems there is in a sense a point
at which if you have sufficient
capability of that sort you have general
capability because if you can build new
systems even if all you can initially do
is this type of engineering work you
could build yourself a poetry module or
build yourself a social skills model if
you have that general ability to build
so it might be that the either general
comments or a sort of slightly more
narrow version of that engineering type
of intelligence is the key variable to
look at as the kind of thing that would
kulish the rest but human level
intelligence it's a vague term and I
think it's worth being aware of that
that it's not necessarily a natural kind
you it's kind of a question which may
lead you to wait until then but there
are kind of two organizations future
humanity institute and the singularity
institute working this let's say I
thought this was a nice little problem
the world actually donating money to it
yeah like should I give it to you
there's great kinds of a change most
audiences that will take a long time to
get to that point for ya this is not any
audience yeah and it's a difficult I
think there is a sense that both
organizations are synergistic so if one
were sort of about to go under or
something like that that probably the
one if both are doing well it's
different people will have different
opinions but we work quite closely with
a lot of the folks from siai as we
citrix and stuff like that we're going
to pick
with some people from there at the
moment there's an advantage to having I
guess one kind of academic platform and
one thing that is outside of giving with
the different things that these
different types of organizations good at
that so if you want to say get academics
to pay more attention to this to get you
know postdocs to work on this and that's
much easier to do within academia and
also maybe to get the ear policymakers
and media and stuff like that on the
other hand for siai the singularity
institute for official intelligence
there might be things that they're
easier for them to do more flexibility
there is they're not embedded in a big
bureaucracy so they can maybe more
easily hire people who have like a
non-standard background without the kind
of credential step that we would usually
need and also the more sort of the
grassroots likely the community board
less wrong that type of it's easier to
do so I'll yell give the the non answer
answer to that question do you think a
biological component is necessary for an
artificial intelligence to achieve
sentence or something important that's a
seem that that should be what the tinge
is at to me I mean ultimately I mean so
if you go all the way back to Adam so it
doesn't seem to matter that its carbon
rather than silicon atoms then you could
wonder if he instead of having same
atoms you run a simulation of everything
that's going on would you have to
simulate biological processes I don't
even think that's necessary i think my
guess and i'm unsure about it i don't
have kind of an official position or a
big world a bit of theory about what
exactly the criteria are that would make
a system conscious but my intuition is
that if we replicated the computational
processes that goes on in a human brain
at the sufficient level of detail
sufficient level of detail might be
profiting on the level of individual
neurons and synapses I think then you
would like to have consciousness and it
might be that something weaker than that
would suffice maybe you don't need to
simulate every individual maybe you
could simplify things and still have
consciousness but at least at that level
it seems likely it's not hard to say if
you had very alien types of mental
architecture something that wasn't a big
neural network but up novel machine
intelligence that perform very well in a
certain way reducing a very different
method than the human brain whether that
would be conscious as well much less
structure they did we living in case
would be a big lookup table were
physically impossible to realize but you
could imagine just having sort of every
possible situation described and that
program would just stood up run through
until it found the situation that
matched its current memory and its
current observations and then read off
which actually it should perform so that
would be kind of an extremely alien type
of mental architecture and whether that
we have conscious experience or not even
that's clear it might be that it would
not have it maybe the process of
generating this time look up table would
generate the kinds of experiences that
you wouldn't get on the action in the
middle it's something like that yeah I
was willing to the ownership general
intelligence home since this relates to
ma I be dangerous um it seems to me that
women while it would certainly be very
interesting if we were to get either as
much more intelligent than human beings
it's not is it so as the end this event
necessarily dangerous I mean even if the
AI is very intelligent too may be hard
for it to get as also to actually do
anything may not have be able to you
know manufacture extra hardware or
anything like that to do there obviously
situations that imagine to tell
free to think he can get you out or or
you can get you further capability so I
think it's true it's not necessarily
different potatoes I mean I guess it's
useful to distinguish your cases one is
instead of the default case unless we
successfully implement some kind of
safeguards are engineered in a
particular way in order to avoid
dangerous so let's think of just the
default you have something that is super
intelligent and capable of improving
itself to even more powerful levels of
super intelligence and I guess one way
just to get an initial possibility that
it would be dangerous to think about why
humans are powerful why are we dominant
on this planet so it's not because we
have stronger muscles than other animals
are because our teeth are sharper or we
have special poison glands or anything
like that it's all because of our brains
and our brains have enabled us to
develop a lot of other technologies that
give us in effect muscles that are
stronger than other animals we have like
bulldozers and external devices and all
the other things and also it enables us
to coordinate socially and build up
complex societies so we can act as group
and all of this you know makes US
Supreme on this planet you know we can
argue the case of bacteria which cannot
have their own domain where they rule
but certainly more than large mammals we
are unchallenged the cause of our brains
and the brains are not all that
different from the brains of other
animals if you look at the chimp and see
you know seems to be a small scale
factor there are other brains that are
larger so it might be that all of these
advantages that we have or do just to a
few tweaks and some parameters that
occurred somewhere among our ancestors
in a couple of million years ago or so
and those tiny changes in the nature of
our intelligence that had these huge
knock-on effects so just prima facie it
does seems possible I did the system
surpassed us even by just the same small
amount that we surpassed embassies that
it could lately too
kind of advantage empowerment if they
exceeded our intelligence by a much
greater margin then all of that could
happen in an even more dramatic fashion
now it's true that you could in
principle have an AI that was a locked
in a box such that it will be incapable
of affecting anything that happens
outside the box and in that sense it
would be weak that might be indeed one
of the safety methods one tries to apply
one developer person it's one of the
things I'm thinking about broader
speaking you could distinguish between
two different approaches to solving the
control problems or making sure that the
super intolerance if it's built doesn't
cause harm so on one hand you have
capability control measures we're trying
to limit for the AI is able to do the
most obvious example will be lock it in
a box and kind of limit its ability to
interact with the rest of the world
another the other problem the other
class of approaches work would be
motivation selection methods where you
would try to control what vai wants to
do so you might want to build it in such
a way that even if it has the power to
do a lot of bad stuff they will choose
not to I think both of these classes are
promising and worth exploring at this
stage but so far there is no one method
or even combination of methods that it
seems we can currently be fully
convinced to work there's a lot more
development needed we could talk more
about this AI in a box scenario may be
computing back to that then I do you
want to soon say I mean honestly give
the situation maybe it's more ideas to
Sir Hugh means having very successful
children season but one feature that
lets you choose results we have
actually very very powerful flexible
hands that have enable us to get a start
on the perimeter walls and so on and so
I think if your knee is just running on
someone's computer somewhere then you
might think that's more analogous to a
very intelligent creature which doesn't
have very good habits it's very hard for
it to know to do to actually improve
itself can actually do anything so you
might think well that you know maybe we
can make maybe the Box met that is
promising because if we just don't give
the AI hands I you know so some way to
actually do something in the world
improve itself in some way or can do is
alter its own code and maybe communicate
information alee you might think that
that seems privation yeah so on one is
to be careful they're so excited what
you would have everything good sir so
clearly it's not pants per se like a
minute if Hitler had not had hands he
could slowly burn energy because are a
lot of other people that had that he
could persuade to do his bidding and
it's a similar there really is super
intelligent it might be that it has no
direct effectors other than they will it
into time slowly letter by letter
message on the screen that then some
human gatekeeper can read and choose
what attacked or not but even that
limited ability to affect the world
might be sufficient if it had not a
superpower in the domain of social
persuasion so if it has the engineering
superpower it might then get all these
other superpowers perhaps and then if it
were able to in particular via super
skilled persuader it could then get
other accessories outside a phone system
that could implement its designs so
there has been no mean you might have
heard of this is this guy Elias Eve
koski run five years back I saw some
series of role-playing exercises where
the idea was that one because we don't
have a super intelligence one person
should play like the AI pretend to be in
a box and then another person should
play the human gatekeeper whose job was
to not let the air out of the box but he
has to talk with AI for a couple of
hours over an internet chats so this
this experiment was run five times with
le su casa
the AI and different people playing the
human gatekeeper for the most part
people who were initially convinced that
they will never let the AI out of the
box and in three of the five cases the
experiment ended where the the human
gatekeeper announcing to the mailing
they study I decided to let this
experiment was run under the conditions
that neither party would be able be
allowed to disclose the methods that
were used to excite sequence of the
conversation to sort of maintain and
shout of mystery but so this is it where
the human level persuader has two hours
to work on me the human gatekeeper it
seems reasonable to be doubtful of the
ability of humanity to to keep the super
intelligent persuader in the box
indefinitely for that reason um how
value do you think the idea of
controlling essentially the mentality of
intelligence the other alternative you
suggest in terms of control is with
something that's at least as intelligent
as us because saying how hard it is to
convince humans to act in a certain
series and we rely it seems way too
lively yes a human's sort of start out
with a motivation system and then you
can try to persuade them or structure
incentives to get them to behave in a
certain way but you don't start out with
the top of the Rosary just get to write
in what the humans value should be so
that's made a difference in the case of
the superintelligence like of course
once once it already has unfriendly
values and it's it has a special power
if you resist any attempt to corrupt its
call system as it would see to change
its goals from what if the goals it has
to something different do you don't
think that like us it's experience is my
positi question is its core values as we
did well I think that depends on how the
occult system is structured so with
humans we don't have a simple
declarative gold star
it's not like a simple slot where we
have super bowl and then everything else
is derived from that rather it's like
many different little people inhabit our
school and they kind of have their
debates and hide it out and make
compromises and in some situations some
of them get a boost like temptations and
stuff like that and then all the time
you know there are different things that
change what we want the hormones kicking
in fading out and all kinds of processes
another process that might affect us is
what that called value accretion the
idea that we can have a mechanism that
kind of loads new values into us as we
go along so if you may be following it
love is like that initially you would
not value that a person for their own
sake above any other person but once
you've undergone this process it start
to value them for their own sake in the
special way and so human seems to have
this mechanism that kind of make us
acquire values depending on our
experiences now if you were building in
a machine super intelligence and coming
to engineer its call system so that it
would be reliably safe and human
friendly then you might want to go with
something more transparent where you
have an easier time kind of seem exactly
what is happening and rather than having
a complex kind of modular minds with a
lot of different forces battling it out
you might want to have a more
hierarchical structure for example I
think you have been waiting or oh yeah
and just announced what you think of the
necessary prerequisites or conscious
money yes I'm not sure I mean we touch a
little bit on that earlier and I think
there is this issue of so suppose that
there is a certain kind of a computation
that is needed
ESS of my mum sort of sympathetic to the
idea that something in the vicinity of
that view might be correct you have to
think about exactly how to articulate
and develop it then there is the issue
of what is a competition so there is
this challenge of the I think it might
go back to Hans Moravec but the thing
similar objections have been raised in
philosophy against computational ism the
idea is that if you have some arbitrary
physical system that's sufficiently
complicated it could be a stone or a
chair or anything just anything with a
lot of molecules in it and then you have
this abstract computation that you think
is what constitutes steep limitation of
a mind then there would be some
mathematical mapping between all the
parts in your competition and atoms in
the chair such that you could set up
artificially through a very complicated
mapping interpret the motions of the
molecules in the chair in such a way
that they would be seen as implementing
a computation it would not be any
plausible mapping not any useful mapping
but the completely bizarre or mapping
there's nevertheless if there are sort
of sufficient in many parts there you
could just rarely define some by jection
and clearly we don't think that all
these random physical objects implement
a mine or all possible lines so the
lesson to me is that it seems that we
need some account of what it means
implementing competition that's not
trigger on this mapping function between
the abstract entity that is a sort of a
chewing program or whatever don't want
to offer computation is and the physical
entity that is said to implement it that
of a sort of mantra Bob this mapping can
look like it might have to be reasonably
simple it might have to have surgery
from counterfactual properties so that
the system would have implemented
related bits like the different
computation if you had scrambled the
initial
missions of the system in a certain way
so something like that but this is an
open question I think in in the
philosophy of mind really to try to nail
out what what what it means to implement
the computation to bring back to the
highest goal and motivation based
approach to making an AI friendly
towards us one of the most effective
ways for controlling human behavior and
quite aside from golds and motivations
is to train the vines telling on
neuroses it's why 99% of this room
couldn't be in our pants right now even
if we really really really wanted to is
it possible to approach controlling nai
in that way or even would it be possible
for an AI to develop in such a way that
there is a developmental period in which
I know a risk reward system or some sort
of narok seasons neurosis and Stillman
could be used to basically create these
rules that now I couldn't break yeah
Sutton sons are promising a because a
neurosis is a complicated thing that
might be a particular syndrome or
phenomena that occurs and in humans
because of the way that humans start
wines are configured it's not clear that
there would be anything exactly
analogous to that in a cognitive system
with a very different architecture also
because a neurosis at least certain
kinds of noises are ones that we would
choose to get rid of if we could so if
you have some big phobia or something
and that was like a bottom that would
remove the phobia or press the bottom
and so here we have a system that is
equal to the super be able to self
modify so if it had this big hang-up
that didn't like then it could be
programming itself to get rid of that
which would be different from a top
level goal because Pablo bhopal would be
the criterion it would use to decide
whether to take an action in particular
like the action to remove the top level
ball so generally speaking and with a
you know a reasonable and coherent goal
architecture you would get certain
convergent instrumental values that we
up in a wide range of situations one
might be self-preservation not because
necessarily you value your own survival
intrinsically for its own sake but
because in many situations you could
predict that if you are around in the
future you will continue to act on the
world in the future according to your
goals and that will make it more likely
that the world will then be implementing
your goals if you have sort of more that
thing pushing towards the realization of
your goals another convert into
instrumental value might be indeed the
protection of your goal system from
preparation for very much the same
reason that even if you were around in
the future but you had different goals
once you have not you would now predict
that that means in the future you will
no longer be working towards realizing
your current goals maybe towards someone
completely different purpose and that
would make it now less likely that your
current balls could be realized if your
current goals is what uses two criteria
to choose an action you were going to
try to take actions that prevent
corruption of your goal system one might
least a couple of other of these
convergent instrumental value side
intelligence amplification because
you're better able to figure out how to
realize your goals technology perfection
like develop more advanced technologies
that enable you to realize your goals
and resource acquisition general purpose
resources so this again relates to why
generic super intelligence might be
dangerous that it's not so much that you
have to worry that it would have human
unfriendly in the sense of disliking
human goals that it would hate two
months the danger or they said it
wouldn't care about your message we care
about something different paper clips
but then if you have almost any other
goals like and care about paper clips
then there will be these convergent
instrumental reasons that you will
discover for why if your goal is to make
as many paper clips as possible you
might want to a prevent you
switching off or from tampering with
your goal system and B why else might
want to acquire as much resources as
possible including planets and solar
systems in the galaxy all of that stuff
can remain in to peer identities so so
even with a kind of pretty much random
goal you would end up with these
motivational tendencies that could be
harmful to humans appreciating the
substantial risks what do you think
about some super goal goals and
merchants such drastic methods of
control some a ethically in B is a basis
for a working relationship well in terms
of the working relationship I think one
has to think about that different with
these kinds of artificial beings I think
there's a lot of our intuitions about
how to relate to other agents that are
conditioned on the fact that we are used
to dealing with human agents and a lot
of things we can assume about the human
we can assume perhaps that they don't
want to be enslaved for it even if they
say that that one a bit late what we
might think that deep inside of them
there is a sort of more genuine
authentic self that doesn't want to be
enslaved even if they have some prisoner
has been brainwashed to wanting to do
the bidding of master maybe we say it's
not really good for them because like
it's in their nature that there is this
will to be autonomous though my brother
other things like that that don't
necessarily have to obtain for a
completely artificial system which might
not have any of that bridge human nature
that we have so say in terms of why the
good working relationship is amazing
just as we think of a good working
relationship with your your
you know word processor email program
not in these terms as your sort of
exploiting it for your ends without
giving it anything in return if your
email program had a will you know
presumably it would be the will to be a
good and efficient email program that
processed your emails properly and maybe
that was the only thing it won't it
cared about so having a sort of a
relationship with it might be a
different thing that was another part
your question about whether this would
be actually like a workable it wasn't
ethical impressively control and the
same teenager yeah so I think there are
so I think that it's not necessary ating
and you aided from scratch and there are
many different possible agents you could
create some of those agents will a human
style values they want to be independent
and respected and stuff like that other
agents who could create might have no
greater desire than to be of service and
other agents just want paper clips so if
you step back and look which of these
should be designed there's that the
question of whether there are more
constraints on which of these it with
the digital and I'm not saying that
those are trivial I think there are some
deep ethical questions there however in
the particular scenario where we are
considering the creation of the single
super intelligence I think the more
pressing concern for me would be to
ensure that it doesn't destroy
everything else like humanity and its
feature now if you have a different
scenario say where you have instead of
this sort of one of your mind rising you
had many minds that becomes smarter and
smarter than rival humans as that may be
gradual exceed saying uploading scenario
where you start with very slow software
does it get human-like minds running
very slowly and then over time slowly
getting you might not have this
multipolar at home well in that case
maybe the issue of how we should relate
to these
intellect morley it comes more more
pressing or in need they show even if
you just have one but in the process of
figuring out what did you do it might
create my terminology thought crimes
like this if you have a sufficiently
powerful mind maybe your thoughts
themselves will contain structures that
are conscious this is a mystical with
the manner that you are a powerful
computer and one of the things you are
doing is you trying to predict what will
happen in the future under different
scenarios and so you might play out a
future by simulating a lot of people
that would live in that future and see
how they interact and what happens and
if those simulations that you run inside
of yourself of these people are
sufficiently detailed then I think that
there could be contrast this comes back
to our earlier that's not what conscious
mind is but I think it sufficiently
detailed computer simulation of mind
would be conscious so you can have a
super intelligence that the process just
of of thinking about things could create
sentient beings maybe millions or
billions of trillions of their welfare
would then be a major ethical issue that
they might be killed when it stops
thinking about them then that might be
mistreated in different ways and I think
that that would be an important as a
notification in this context so a laser
often suggests the month of many
problems with sort of arbitrary stamps
in AI space is that human values very
complex so but Jenny and I go system
will they see get horribly wrong because
everything we don't quite care about and
that's fans paper but how complex you
think human valuable don't be if we can
sort of currently it looks like humans
are human values are complicated I mean
even if they were very simple that even
if it turned out it's just pleasure sir
which compared to other theories of what
has value like democracy flourish
it's like it's a farce we can think
about us that's like it may be one of
the simplest possibilities even now if
you start to think about it from a
physical istic point of view and you
have to now specify which atoms have to
go how and where for there to be
pleasure it would be a pretty difficult
thing actually to write down like the
Schrodinger equation for pleasure so in
that sense it seems clear then our
values are very complex so there are two
issues here I think there's a kind of
technical problem figuring out if you
knew what the values are in the sense
that we normally think we know what
sometimes know what our bodies are how
could you get the AI to share those
others like pleasure of absence of pain
and then there is the additional
philosophical problem which is if we are
unsure about what our values are like if
we are sort of groping around in
axiology trying to figure out how much
about the different things and maybe
there are values we have been blind to
today like how do you also get all of
that on board on top of what we already
think has value that potential from a
wall of growth both of those are very
serious problems difficult challenges
one is it their number of different ways
you could try to go like 11 approach
that's interesting is what we might call
indirect normativity where the idea is
that rather than specifying explicitly
what you want the AI to achieve like
maximize pleasure while respecting
individual or company and pay extra
attention to the poor and weak ism I'd
rather than creating a list what you try
to do is that is to specify some process
or mechanism by which the AI could find
out what it is supposed to do so one of
these did some of you will have come
across as this idea of coherent
extrapolated militia where the idea is
if you could somehow tell the AI to do
that which we would have asked it to do
if we had thought about the problem for
longer and if we had been smarter and if
we had some other preparations basically
the idea it is if you could sort of
describe some idealized process whereby
we at the end
if we underwent that process would be
able to create more detailed it then
maybe point the AI to that and make the
a ice value to run this process and then
do what comes out at the end of that
rather than go with our current best
guesses about what we want to do or what
has value it's nervous with the VA I
would decide that if we thought about it
for a thousand year is really very
carefully and the we would have decided
it's just like the ai's take over and
become new yeah that seems to be a
possibility and then that racist it's
interesting customer like suppose that
that that that's really what our current
extrapolated volition would do and let's
assume that everything has been
implemented in the right whether it's
not kind of solve our realization of
this so how should we think about this
on the one hand you might say well if
this is really what our wise enough is
one what we sort of would want if just
we were you know safe from these errors
and illusions that we're suffering under
maybe we should go ahead with it on the
other hand you could say that you know
this is really this is really a pretty
tall order here like we're supposed to
sacrifice not just a big but ourselves
and when everybody else but for this
this abstract ideal that we don't really
feel in a strong connection to I think
that's one of the rig who knows what
will be the outcome of this coherent
extrapolated volition and there are
other columns what might have that are
that that needs to be spelled out so
exactly whose position is it that this
supposed to be extrapolated like
humanities okay so then like who is
humanity like there's a big to past
generations for example how far back hmm
does it exerted embryos that died who
knows whether the core of humanity
is nice I mean maybe there's a lot of
sort of suppressed sadist out there that
kind of we don't realize because they
know that they would sort of be punished
by society if they sort of maybe if they
went through this procedure who knows
what would come up so the dangerous run
something like that without some kind of
safeguard yeah on the other hand then
there is worried that if you put in too
many of these checks then in effect you
move the whole thing back to what you
want now because if you were a lot to
sort of look at an extrapolation see
whether you like it if you just like
that you run another one by changing the
parameters and you were allowed to kept
going like that i don't see we were
happy with the result then basically it
would be you now i chose from downtown
so it's worth thinking about whether
there is some sort of trade off for
blend they are never compromise that
that might be the most appealing you
mentioned before a computer may be
producing sentir life itself in running
a scenario what are the chances that
there's the society that we live in
today yeah so what exactly are the
chances i don't know i think significant
I mean but maybe maybe maybe a little
less la means it's like subjective I
judgment to him he does that make me
maybe less than fifty percent that's
like more than 10 so yeah so there's a
whole different live different topic
maybe we should save that topic for for
another possibly another time of she
simulation such as a lot if I can say
just about that if I wanted to study
like this area generally existential
risk what kind of subject would you
recommend I pursue like we are all on
Nicorette so after our better we will
start with the master or go into like a
job if I wanted to study it work what
kind of master would you recommend well
part it will depend on your talents like
if you're sort of quantitative guy or a
walk out there isn't really an ideal
sort of educational program anywhere to
deal with these things so you'd want to
try to get the fairly broad education
that many fields that'll be relevant if
one looks at where the people are coming
from have so far I've had something
useful to say so they're like a very
chunk of them are philosophers some more
sort of computer science some economists
maybe physics so this will be some and
those those fields have one thing in
common is that they are fairly versatile
like if you're doing philosophy you can
do philosophy of X or Y or Z of almost
anything economics as well it's it gives
you a general set of tools that you can
use to analyze different things and
computer science has this kind of you
know ways of thinking and structure a
problem that also useful for many things
so it's not obvious which of those
different disciplines would be best
generically I think that would depend on
individual but then what I would suggest
is that while you were doing do you feel
you would also try to read in other
areas other than the one you were
studying and try to do it at a place
where there are a lot of other smart
people around them like with a
supportive advisor that you know
encouraged you and give you some freedom
to pursue different things would you
consider
intelligence great by the human beings
as some kind of consequence of
evolutionary process oh so like in a way
that human beings try to overcome their
limitations and as its well you have a
really long time to get it on DNA level
you just get it quicker on the more
computational level so whether we would
use evolutionary algorithms to produce
your personas if artificial children's
itself is part of evolution yeah well so
there's a kind of a trivial sense in
which sin if we evolve and we created
then like obviously evolution had a part
to play in the overall cost of
explanation of why would then get the
machine intelligences at the end now for
evolution really to exert some shaping
influence there have to be a number of
factors at play that has to be like a
number of variants created that are
different and then sort of compete for
resources and there is a selection step
and then whenever you need to be
significant evolution you have to
iterate there's a lot of times so
whether that would happen or not in the
future is not fair at all if you have a
single tone forming that is if a World
Order arises in which additive top level
there is only one decision making agency
which could be like democratic war
government on a ir tools everybody are
like a self-enforcing moral code or some
other thing it could be many are too
early or a nice thing or a bad thing but
if you have that kind of structure than
they would at least be the in principle
ability for that unitary agent to
control evolution within itself like it
could change the selection pressures by
you know taxing or subsidizing different
kinds of life forms of if you don't have
a single fun then you have different
agencies that might be in competition
with one another and in principle and
that's no evolutionary
can come into play but I think the way
that it might turn out would be
different from the way we are used to
see in biological evolution so one thing
you might have these potentially
immortal all I forms that is you have
software minds they don't have they
don't naturally die they could modify
themselves if they knew that their
current type if they continue to pursue
their current strategy would be out
competed and they didn't like that they
could change themselves immediately
right away rather than sort of wait to
be eliminated so you might get if there
were to be a long evolutionary cousins
ahead impedance could anticipate that
you might get the effects of that
instantaneously participation so so I
think you wouldn't probably not see the
actual evolutionary process playing out
but there might be some of the
constraints that could be reflected sort
of immediately from the fact that
different agents had to pursue
strategies that they could see viable do
you think it's possible that our minds
could be scanned and uploaded to a
computer machine and then could you
create many copies of ourselves so this
is the what's in technical terminology
whole great information or in more
popular jargon uploading and it's
obviously it's impossible now but it
seems that it's consistent with
everything we know about physics and
chemistry and so forth so I think that
will become feasible you know boring
some kind of catastrophic thing that
that puts a stop to scientific and
technological progress so the way I met
and it would work is that you would take
a particular brain freeze it or bitter
fight it and then slice it up into thin
slices that would be fed through some
array of microscopes that would scan
each slice with sufficient resolution
and then automated image analysis
algorithms would work on this to
reconstruct the three-dimensional
neuronal network
original organic brain implemented I
have this as a set of computer
information structure in a computer at
this point you need computational
neuroscience to tell you what each
component does so you need to have a
good theory of what the say of pyramidal
cell does what they you know different
kind of thing and then you would combine
those little computational models of
what each type of hearing loss with this
three-dimensional map of the network and
run it and if everything went well you
would don't have a transfer to mind with
memories and personality is intact to
the computer and there is no good person
of just how much resolution would you
need to have like how much detail would
you need to capture off the region of
mine and actually to successfully do
this but I think there would be some
level of detail which as I said before
might be on the level of sign ups is or
there abouts possibly higher than that
would suffice so then then you would be
able to do this and then after your
software of course you could be copied
or speed it up or slow down or post and
stuff like that yep hop on a controlling
the AI and evaluating the risk my
question would be that assuming we have
created a far more compact di than
ourselves is there are credible reason
for human beings to continue existing
and yeah I mean so I'd certainly have
the reason that you know if we value our
own existence we seem to have some do
you mean that that that would be a kind
of weather that would be moral reason
for us to exist or whether we would have
any self-interested reason to exist well
I guess would be your opinion yeah when
I mind my opinion is I'd like to see
that yeah I'd like to rather not see
anything the genocide of entire Givens
rather that we all live happily ever
after if those two are the two
alternatives I think yeah that's always
happily ever after what I will come down
alert by keeping the the human species
around you're going to have a situation
where you've got extremely extremely
advanced they eyes I mean maybe there
for two decades a few centuries or
whatever or far far beyond our
comprehension and even if we still
integrate some tree with machines very
anyway don't say pie logical humors then
about just be completely receivables for
us so isn't there a danger that our
stupidity would happen their production
would hamper their perfection yeah I'm
there well so I mean there's not space
farther to be many different kinds of
perfection pursuit but raise it right
now we have a whatever like the white
dust mites crawling around in everywhere
then not really hampering our pursuit of
our core truth or beauty they're going
about their business and we are going
about ours I guess you could have a
future work and there would be a lot of
room in the universe for its planetary
science computers thinking their
grandfathers well I make a prediction
here but if you wanted to have like a
nature reserve with with a sort of
original nature and original human
beings living like now that wouldn't
preclude the other thing from happening
but what does not have approached things
like viruses bacteria Dave's five
innings so far below us and kind of
scary actually and and if you leave
human and so that also original humans
make sure that they're aware of that
isn't there a real
that they'll be angry at irrelevant to
kind of grand scheme thanks Marsh out to
be yeah I suppose I mean I do I don't
think it would bother the AI which would
be so able to protect itself or remain
out of reach now it might have kind of
demean the remaining humans that if we
are being thrown from this position as
kings the highest life form around that
it would be a demotion and that guy from
that one would have to to deal with us
as of course unclear how much value to
place on earth I mean right now
presumably in this universe which looks
like a definite somewhere out there
there are going to be all kinds of
things including God like the intellects
and everything in between that are
already out stripping us in every
possible way so it doesn't seem to upset
us terribly we just get on with it so I
think people would maybe have to make a
little psychological I'm sure you could
adjust to it easily now it might be that
from some kind of particular Theory
value that this would be a sad thing for
humanity that we are no longer even
locally sort of there there at the top
of the ladder moral rationalism was true
that is if it were irrational to perform
wrong acts would we still have worried
about super intelligence it seems to me
that the well I mean you could have a
system which if you it might not care
about being rational according to that
definition of rationality so I think we
will have to worry it regarding trying
to program a I with our values
as I understand it what's considered one
of those promising progeny I now is more
sort of statistical learning type
approaches so and the problem that is
that if we were to produce now a I'm
adapters news you might not understand
its inner workings well enough to be
able to sort of dive in and modify and
precisely right way to give it an
unalterable list of term the volumes in
some way so if we were to you know ends
up with some big neural network brain in
some way and ended up with that could
perform as well as human VidCon some
code to toss or something we might help
we might be able to do that without
knowing how to actually then offer it to
have some particular set of goals yeah
so there are some things that if I can
think about them one general word for
needs to bear in mind the font I start
kind of a project we might give it
various examples and say well this is
like a good action this is a bad action
in this context though and maybe it
would learn all of those examples then
because there is how who did generalize
to other examples outside this class so
we could test it we really divided our
examples initially in two classes and
training the one and test its
performance on the other the way you
would do to cross validate it and then
we think well that means other cases
that it doesn't you'll see it would have
the same kind of performance but all the
cases we could test it on would be cases
that would apply to it at its current
level of intelligence so presumably
we're going to do this while it's still
less than human intelligence or human
normal intolerant we don't want to wait
to do this after it's already super
intolerant so then the worry is that
even if it would sort of analyze the
what to do in a certain way in all these
cases it's only up dealing with all of
these cases in the training case when we
are when it's all that the human level
of intelligence now maybe once it
becomes smarter it would realize that
there would be different ways of
classifying these
is that that would have radically
different implications for humans so
suppose that you try to training to this
was sort of one of the classic examples
of like a bad idea for how to solve the
the control problems like let's let's
let's train the AI to want to make
people smile like what you wrong with
that so we train it on like different
people if they smile when it does
something that's like a kind of reward
oh no it gets strengthened those
dispositions that led to the behavior
that make people smile and frowning I
guess would be sort of we move the AI
away from that kind of behavior and you
could imagine that this would work
pretty well and at the primitive stage
where were they i would engaging more
pleasing and useful behavior because
then the user will smile at it then and
it will all work very well but then once
the AI reaches a certain level of
intellectual sophistication it might
realize that it could get people to
smile not just by being nice but also by
paralyzing the official muscles in the
sort of constant beaming smile and then
there would be this perverse
instantiation of the constant values all
along the value that it had learned
must've make people smile for the kinds
of behaviors that it would pursue to
achieve this goal would suddenly
radically change at the certain point
once this new set of strategies became
available to it and you would get this
treacherous turn which would be
dangerous so that's not to dismiss that
whole category of approaches altogether
the 11 this thing one would have to
think through quite carefully exactly
how one would go about that there is
also the issue all of the things that we
would wanted to learn if we think of
human values and ambitions and goals and
plans they are sort of its we think of
them in using human concept not not
using sort of basic physical like place
at them a to Zed in a certain order but
we think like you know promotes peace
you know like encourage people to
develop and achieve these are things
that to understand them you really need
to
have human concept which is subhuman AI
will not happen it's too done at that
stage to have them now why is it super
tellanon did might easily understand all
human concepts but then it's too late
what if already needs to be friendly
before so there might only be this sort
of brief window of opportunity when it's
roughly human level where it's feel safe
enough not to not to so resist our
attempts to indoctrinate it but still
smarting off that it can actually
understand what their time to tell them
and again one would have to be very
careful to make sure that we could sort
of bring the system up to that interval
and and freeze its development there and
then try to load the values in before
bootstrapping it father and maybe in a
game I mean this was one of the first
questions maybe its intelligence will
not be human level in the sense of being
very similar to a human at any one point
maybe it will immediately become very
good at chess but very bad at poetry and
that it has to reach sort of radically
superhuman levels of capability some
domains before other domains even reach
human level and in that case it's not
even clear that there will be this sort
of window of opportunity were wearing
well you could go in the values so that
said that's not I don't want a sort of
dismiss it but that's like some
additional things that one needs to
think about the contrast to develop that
we go way back how likely is it that we
will have the opportunity in our
lifetimes to become immortal by mind of
learning well so first of all by mortal
here with mean I guess living for a very
long time rather than literally never
diet which is a very different thing
that would require our best theories of
cosmology to turn out to be false and
stuff like that so looking for a very
long time I mean I'm not going to give
you a probability in the end but i can
say some of the things that so first we
have to avoid most kinds of existential
catastrophe
that is so I guess it would first
subtract if you need to start with sort
of a hundred percent and then you remove
all the things that could go wrong so
first we would have to throw away you
know pretty much whatever you think told
a level of existential risk is
integrated over all time then there is
no BS risk that you will sort of my
before any of this happens which seems
to be a very substantial risk now you
could reduce that by signing up for our
chronixx but that's of course an
uncertain business as well that could be
some existential catastrophes that that
would put an end to a lot of things like
it being nuclear war or pandemics and
stuff like that and then I guess there
are all these situations in which not
everybody who were still around got the
opportunity to participate in what came
after even though what came after
doesn't count as an existential
catastrophe so that might also be so is
it seems and then even more complicated
like if you take into account the
simulation hypothesis or whatever which
we decided not to talk about today but
yeah we're saying before I think as for
the timelines you know really truth is
we don't know so you need to think about
a very sort of smeared probability
distribution I've been really smeared
that you know things could happen
surprisingly sooner like some
probability ten years tomorrow
yes but you know probably more
probability 30 40 50 years in some
probability eight years or 200 years
like there's just no good evidence that
human beings are very good at predicting
which position these kinds of things far
out into the future first question how
intelligent can be really can so if you
assume that you're already a machine
with you already have this complexity
class of problems that you can solve and
not and let me might make this is it
fair to you by the super intelligent
machine can actually be exponentially
intelligently maybe it's maybe we'll
each human intelligence and when you say
what this is very close to what we could
achieve it into definition of
intelligence always open well some in we
are in a sense in a sort of cheetah
cells we can solve all problems so
probably that the super it's hella like
everything at the Turing machine I mean
you could take like a piece of paper
start to a it would take too long to
actually do it and if we try to do it
that would be things that would probably
throw us off before Matt completed any
sort of big Turing machine simulation so
there is a less less sort of quick a
sense in which our abilities are already
indirectly unlimited that is if we have
the ability to create these different
elements then put in a sense we can do
everything because we can create this
thing that then solves the thing that we
won't solve say it's the sequence of
steps that we have to go through the end
of the salt so there is this level of
capability which might mean that once
you have that level of capability your
reach your indirect reach is universal
like anything that could be done you
could indirectly achieve and we might
already have surpassed that level a long
time ago say for the fact that we are
sort of uncoordinated on a global level
one maybe away unwise but if you had a
wise single home then certainly could
imagine us just plotting a very safe
course taking its toll and in the end we
could be pretty confident that you
get to the end result it but maybe not
the moment neither of those like ideas
what what you had in mind maybe I had
more in mind the question of just has
smarter in sort of everyday sense of
smart could a machine be like just how
much more effective at social persuasion
to take one particular thing then the
most persuaded human being so that we
don't really know if one has a
distribution of human abilities and it
seems that the best humans can be a lot
better in our intuitive sense of a lot
than the average humans then it would
seem very surprising if if the best
humans like the top you know tenth of a
percent had just reached sort of the
upper limit of what was technologically
feasible that that would seem to be an
amazing coincidence so we're gonna
collect the maximum achievable to be a
lot higher but exactly how high we don't
know how are you so two more questions
have time for ya who wants to do you I
have fun ok first to us a follow-up and
then anybody can think it table night
right leg just like VR worrying about
super intelligent being it's quite
likely the superintendent being who
start worrying about is it possible that
that gave lots of worry about another
super intelligent yeah so it is it
present well so you could consider once
an hour in which the one AI designs
another artificial intelligence that's
smarter and then that designs another
but it might not be clearly
distinguishable from this annoyed we
have one AI that modifies itself so that
it ends up smart like whether you call
it the same or different it might be
kind of an unimportant different okay so
a last question who wants to have it
okay so this has to be the super
profound question
listen his wife and we try to didn't
build prosperity well I don't think we
should now do that but was it a good
step back and you sort of think what
would the same species do or they would
first figure out how to solve the
control problem and then I would think
about that for a while to make sure that
they really have the solution right and
they hadn't just deluded themselves to
have solve it and then maybe they would
build a super intolerant so that's what
the same species really know what
humanity will do is try to do everything
they can as quickly as possible so there
are people have tried to build it and as
we speak I mean in a number of different
places on earth the fortunate looks very
difficult to build it with current
technology but of course it's getting
easier over time computers get better
computer science of the state-of-the-art
advances we learn more about how the
human brain works so every year it gets
a little bit easier from some unknown
very difficult level gets easier and
easier so at some point somebody will
probably succeed you're doing it if the
world remains kind of coordinated and
uncontrolled as it is now it's bound to
happen soon after it becomes feasible
but we have no reason to accelerate that
even more than then it's already
happening so somebody is that the other
day like so what what's the like it like
it so we're thinking about what would
you like a powerful at AI think do that
does come into existence and it didn't
know very much yet but they had a lot of
clever algorithms and a lot of
processing power and then somebody was
suggesting maybe to sort of move around
randomly live like a human baby task to
kind of figure out how to how are things
move how it can move its actuators and
stuff like that and then we had a
discussion whether that was a wise thing
or not but but if you think about how
the human species behave we are really
having very much like a baby whistle
basically moving and shaking on anything
that moves just to see what happens and
the risk is that we are not liking a
nursery with like a kind mother who has
put us in a safe cradle but that we're
out in the jungle somewhere screaming at
the top of our lungs and they were just
alerting the lines to supper alright so
let's about that i joined this a great
deal so thank you all


262. Counterarguments to the basic AI Xrisk case

hello and welcome to session 200 and uh
62 in the AI safety.com reading group
tonight we'll be discussing uh half of
the article counter arguments to the
basic AI X risk Case by catcher Grace
is the lead researcher at AI impacts and
the person who's defining the
organization AI impacts
this was posted a couple of months ago
and we are discussing the entire blog
post
except part C which is which we might do
next time
so first before we we dive into the
article I would like to say thank you to
catcher Grace for putting some kind of
scrutiny onto these arguments I think
it's really important and this
skepticism is uh something that is uh
really high quality skepticism is
something that is sorely missing in AI
safety
I also imagine that it's somewhat of a
thankless job in the sense that
in the broad world most people are
skeptical of AI risk but within the
community of rationalists uh the most
people would disagree with her so I
believe it's a somewhat of a thankless
job to um
to present the case against AIX risk
unfortunately I also feel that the field
is dysfunctional in the way that
criticism is being handled and let me
try to explain how I
um envisioned this dysfunctionality and
I'm going to use that with my
interpretation of what may be going on
like inside catcher's mind and that's of
course something that is very likely to
be flawed but I envisioned that ketchup
probably has a intuitive sense that a
lot of the basic case for AIX risk is on
Shaky Ground and none of the arguments
are like knock down arguments but just
shaky and
um uh feels like they they are under
specified and don't have enough uh high
as high probability as other boots would
think
and so how do you
um
uh how do you answer this well the way
it's answered in this is a in a very
broad uh sense where describing all the
possible problems or most of the
problems but that also necessarily makes
the crystal somewhat shallow
and when this is posted the replies that
catch you get are mostly hot takes like
very uh brief comments and people uh
going to attacking points and discussing
points that seem rather
um irrelevant most of it and that in
turn makes it very hard for catcher to
engage productively with these counter
arguments and the end result of this
does functional circle is that we don't
get any kind of synthesis of the counter
arguments and we don't get a get
stronger Arguments for AI risk out of it
and we don't get clarity of our
epistemic state
um so what can we do about this I don't
really have a good solution
um fundamentally
um predicting the future is really hard
and we are bound to have uh to be in a
really poor epistemic state it may in
fact be impossible to get a strong
resolution of this
um
a few things that I'm like trying here
is to give answers that are just as
broad as the questions that categories
are raising and also
referring back to the literature as much
as I can um whether this is sufficient
or
um uh like I don't have any hope that
this will in fact convince catcher
um but uh uh as I said the task may be
impossible
so let's talk a bit about the basic case
for AIX risk and uh categories starts
out by describing her version of the
case for AIX risk and it is not the one
that is generally used in the literature
um catcher presents her own argument
it's a second and quite beautiful
argument but it's also a new original
argument
um
I think this is a mistake I think there
are really good descriptions of the case
for AIX risk in the literature my
personal favorite would be the book
super intelligence path danger
strategies by Nick Bostrom it's eight
years old but I believe it has held up
surprisingly well
um but catches description of the basic
case for AIX risk while not entirely
wrong it has to focus in a lot of the
wrong places and it has a lot of under
defined issues and I believe that when I
go through the um the answers that
people give to this criticism a lot of
this seem like it could have been
avoided if catcher had chosen to go with
a standard uh description of the case
for AIX risk instead of providing her
own
so her case is built of uh three Clauses
the first is that if superhuman AI
systems are built any given system are
likely to be goal directed
and this is using the word superhuman
rather than the defined term super
intelligence and likely is of course sub
is that like 50 51 or 99.99 and there's
a focus on several systems and there is
not the word AGI or general intelligence
or something like that which is a
feature of most of the the standard
cases for AIX risk
the second Clause if goal directed
superhuman AI systems are built their
desired outcomes will probably be about
as bad as an empty Universe by human
lights
and while I believe this is a way to
State the case it's also like kind of a
strange way because the long-term values
are not obviously that important like
um if the AI for instance kills us all
and then optimizes the universe in a way
according to our values that may be good
but like the thing I care about is the
AI system not killing us and so it's
kind of like
um not orthogonal to the thing we care
about but it has the wrong Focus
and finally if most goal directed
superhuman AI systems have bad goals the
future will likely be very likely be bad
and again this is very different from
the way it's presented in the book super
intelligence there's no talk about
decisive strategic advantage and and
this kind of thing
and catech Grace of course is honest
about this more being a this entire
document is more a description of what
are the gaps in this case and what are
some possible counter arguments to this
and
um not a a very strong case that AI risk
is not is very low or something like
that
um
so let's dive into the possible counters
to the first
um Contra uh superhuman AI systems will
be goal directed
when people talk about goal directed
they possibly probably think about
different things
um a classic goal directed agent is the
utility maximizer or even more extremely
the paper clip maximizer and a paperclip
maximizer is a classical example of a an
agent that is goal directed enough to be
deadly or to have uh goals desires that
are at odds with human survival
we could imagine this goal directed
agents
um categories called some incoherent
pseudo-adentic AI and presents a
spectrum of possible uh such AIS from a
thermostat to a utility maximizer
I don't think this is a good description
uh in that a thermostat is not an AI and
it's very much not an AGI the the
concept of AGI as far as I could tell is
not present in in this article so what
is the least uh sort of the least
um
eccentric and most incoherent aai that
is still arguably in AI I could imagine
something like a non-reflexive uh AI
that is using heuristics may qualify but
um but I think once we get substantially
below that
um I think I think we kind of really
call it an AI
we can observe that human level goal
directedness is bad because humans are
goals that are not enough to like I
don't know kill neanderthals
um
so categories defines weak solo agents
as agents that are less goal directed
than humans this may be nitpicking but I
believe that humans are actually in fact
very power seeking so a safe level is
not one that is below the human level
but one that is a lot below the human
level
the example that catch Grace gives of a
weak solo agent is a system of just
fully specified if x then y statements
I would not call this an AI and I
believe that the reason that this is not
an AI is
also the reason why it's safe and in the
same way if we want some AI that is
capable of learning and capable of
cross-term internalization and all these
things then that is precisely the thing
that makes it unsafe
um
and that's the argument that people
don't want to destroy the world so they
will probably use weak solo agents
um and I wish that was what we're living
in but right now people are in fact uh
deploying AI systems that are
substantially more authentic than just
being fully specified if x within y
statements and I think a lot of people
are not buying the arguments for AI
existential risk and a lot of people are
buying are deploying uh AI systems
without much care about whether they
will destroy the world
then there's a question of diverging
from expectations and one of the
advantages of
at least very uh weak solo agents is
that they will diverge less from
expectations and uh that that is true
but that's unfortunately also what we
want from the AI the point of having an
artificial intelligence is that it'll do
things that we can't do ourselves
um
catcher observes that in many cases
Divergence from expectations is bad and
um I agree in many cases it's bad in
some domains it's good and a crucial
consideration for whether it will be
good or bad is whether there is
cross-domain uh reasoning going on
because if uh if there is it may still
be good but that's also where a lot of
the threat comes from
another example catchy provides is
companies right now they often prefer
employees to follow rules like the
ontological rules rather than just
maximizing the shareholder price
um and like uh I think sometimes it's
true but far from always in particular
if you read job adverts then they write
things like the application and should
be a self-starter and aware of the big
picture and this kind of thing
um and remember for this argument to to
provide safety then it needs to be that
companies always prefer employees to
follow rules and that's just plainly
not true people do prefer authentic uh
goal directed employees
um
so if we imagine we have an an AI one of
the ones we have now and it tries to
self-improve somehow to become more
coherent more goal driven in some way uh
that doesn't necessarily move it towards
a sensible utility function
um and I think catcher is right it may
be
um
it may accidentally move
itself to be coherent towards something
like that by accident is just like wire
heading or something totally nonsensical
but that doesn't have any obvious safety
implications just because now it doesn't
want to build paper clips but want to
build I don't know uh something really
really strange
if a child's the universe with something
really strange that's not really safety
another thing that catcher suggests is
that it's possible that the AI could try
to become more coherent but just fail in
the sense that it's trying to like it
has like circular preferences and then
it tries to fix the circular preferences
but but feels to fix this and like the
the standard case for AIX risk is not so
much focused on like the average AI but
the first AI that is actually able to
recursively self-improve and become more
coherent in Gold uh directly and things
like that and that way catcher's
argument doesn't really capture the
essence of the uh the standard arguments
that are used for in the case for AIX
risk
and also on the diversion from
expectations I should say that uh a key
argument that catcher is missing in her
argument is that a strategically aware
AI would just realize okay these are the
expectations and then we'll try to
fulfill these expectations while
actually being deceptively aligned
um uh and catcher of course is aware of
these arguments I have no doubt at all
but these arguments don't seem to fit
very much into her own description of
um uh the basic case for AI X risk which
is a a problem for for her description
Ambiguously strong forces for
colderatedness need to meet an
Ambiguously a high bar to cause a risk
so that's like we don't know precisely
how uh how strong are the incentives to
become more goal directed and
um
what is required for uh for an AI to
become dangerous so if the two levels
have some kind of Gap in between them
then maybe we get safety from that
um
I don't see
I don't see that the two curves as being
flat unfortunately I see one of them
sloping down and that's of course I've
shown here the the classic scaling graph
in that AI capabilities will increase so
it will be easier and easier for the AI
to become more goal directed probably
and the um the benefits that the AI as
it becomes more capable uh also
increases
um
the incentives to become more
coherent seem to be really strong and
the requirements the difficulties in
becoming more coherent seem like there
are relatively weak and I would expect
that
um uh even very very primitive models
are able to reason that
um becoming more powerful actually is
convergently uh useful and particularly
useful for whatever gold the AI is
trying to obtain
foreign
because
we could imagine that humans kind of
have a utility function
um and if the AI has one that's close
enough that might work out and one of
the underlying motivations for this is
that humans have different values so
um human the set of human values need to
be not like a point but some kind of
space and if the AI if we can align an
AI well enough that it hits within this
space well that's ought to be reasonably
good like that's not obviously worse
than other humans decide in the future
uh that is true but this uh space
includes some really bad things in
particular uh the desire to uh like kill
people who are less intelligent than
yourself and like Hitler would be an an
obvious example of uh someone who
technically has human values but who we
really really wouldn't want to hand over
all power in all future too
claims that this is in fact not obvious
that this would uh turn out to be bad
and if it's true then we should worry
about more General problems than AI like
I I claim that is this effect obvious
that Hitler is bad and also that okay
sure it is a more General problem but it
is one in fact that AI might solve in
the sense that we have ideas about how
to uh like Implement alcoholics related
Evolution and uh like uh long reflection
and things like that so it may in fact
be things that AI could solve
potentially
um categories further identifies a
larger region uh around human values
which is that which can be reliably
aligned with typical human values via
incentives in the environment
and the problem with incentives in the
environment is that if an AI is capable
of obtaining a decisive strategic
Advantage then incentives will not work
so that region does not buy us very much
safety if we don't get the AI to be
correctable or maintain power over in
some way
so if we imagine that this
region around human values then it's
just basically an empirical question can
we get an AI that is close enough to
human values
so
um one of the reasons why I'm less
optimistic about hitting this goal is
that uh it's really hard to Define human
values and that means that hitting a
call that you can't Define can't see
sounds really difficult and also that
the targets seem really really small
compared to the space of all possible
values you could have then reasonable
human values is a very small Target
um catcher also has like a in in several
of these arguments she has a small
vignette or something where she
describes a world what would look like
if this Gap was uh was important and in
this uh most of it makes a lot of sense
but in this in particular she describes
a world where small differences in
utility functions do not turn out to be
catastrophic and the reason she
describes is that an AI where uh these
small differences don't matter will tend
to be courageable
I do not think that courageable military
followers from this at all I think
courage ability is a separate problem
and I think a world where we get almost
this uh utility function I would imagine
that would be something like our
coherent extrapolated volition uh
possibly in some slightly different
process or something like that
um I don't think I don't see courage
ability from this
the difference between the AI and the
human values may be small
first catch a grants that misoptimize us
if we see those that could
have a very large difference in values
so we're kind of disregarding meter
optimizers for now so how do humans
learn values well we learned it mostly
by observing examples and that's kind of
the also the idea that we could have an
AI learn values in basically the same
way
so one thing is learning values another
is like obtaining these values how do
humans in fact obtain our values uh so
the values that we endorse and not just
learn to recognize in a sociopathic way
like what are the values that other
humans have
I think it's mysterious and I also think
that the way humans do are very
different from the way AI training like
um from the way we're training uh like
gpt3 or something like that it seems
very different from what humans do
uh catcher disagrees and believes that
for the things AI currently uh learn the
difference between what um what humans
learn and what AI learns seem rather
small
and
I would agree to some extent in in many
simple cases chess would be an example
of where learning the values of Chess is
kind of the same but that's because
chess is a really easy example and in
general we choose to look at examples
that are really easy we don't choose
anything nearly as philosophically
complex as human values
and that's why we can't generalize from
being able to solve the easy questions
to being able to solve the hard
questions
catcher asks for catastrophic utility
function that is very close to you human
utility function and after a bunch of
training this AI has this and that is
still catastrophic
um so
um I don't think that an AI has it as
like the hazard is um
uh it can have it in two ways it can
have its assets these are the values
that the AI endorses or it can have a
model of the humans uh
and and then uh have it as another fact
about the world that it knows so what's
an example an example uh this is
cheating but uh one of my objections
would be that it will in fact not be
trying to learn human values in practice
it will be doing something like
maximizing profits
um because in general people are trying
to to aim for human values but even if
we did that we could easily end up with
something like
um trying to maximize approval for from
someone who is who has some very uh uh
not nice ideas about human uh
flourishing we could see a lack of value
learning or something like that
um and I think there's a bunch of extra
things we could see that are close to
the human utility function but for some
reason we're not we're not getting it
this is intimately related to the
fragility of value thesis by Elisa
utkowski
um and um catcher
summarizes this as a value is not
resilient to having components of it
moved to zero I don't think this is a
perfect uh summary in that the idea is
not so much that an element is moved to
zero but more that is not considered in
the first place like a modality or
something like that that is not
considered
um deep learning is given as an example
of something that can learn more things
a lot better than the more directly
specified good old-fashioned Ai and
while that is true I the argument has
never really hinged on whether the the
AI would be able to know our values like
the uh even some of the very earlier
arguments and certainly uh Boston super
intelligence assumes that as a future
super intelligence would know our values
very very well possibly probably better
than we know ourselves but would just
not care
um and can you suggest there may not be
many terrible value systems adjacent to
mine and I don't think actually that the
thing that an AI would learn from
observing catches actions would
necessarily be a value system I could
imagine the index uh indexology uh could
be wrong and like the references to
people uh to to the world and time and
things like that I could easily imagine
like you miss some kind of modality uh
like embodiment or like if it's only
text then you are missing basically all
the modalities
catcher uses a face analogy uh to
support a point and uh are images of the
human face fragile and we can see
obviously here uh some different
diffusion models that are generating
um
uh some uh
some faces and the faces in general
don't have the property that some of
them are missing noses or things like
that I mean sure you could point at this
particular Abomination here in the
background and that probably looks like
something really bad but in general
there are no
um
uh we don't get the thing where there is
no nose or anything like that
I think the example is very different
human values and if pictures of faces
are very different in this in the fact
that we do have a digital example of a
human face and we don't have a digital
example of human values so I think the
the analogy is very genius
knit Suarez answers in uh a quick
Twitter reply that the maximally face
face like image which is very different
from what these diffusion models make
that doesn't look human at all and I
think they're so much talking past each
other because Ketch is talking about
what the model knows and need about what
it maximizes
um
daily also points out that learning
human values is very different from
having human values
um
there is in the response to catch's post
a lot of it has focused on this face
analogy
um and I think it's um I think it's fair
that people criticize the feelings of
this analogy uh because Katie says it's
very nil because and in that case
obviously you need to
um uh like like it's uh then you need to
be able to defend it on on several
different uh levels and I think the
criticism is reasonably Fair
if you just said like this facet is
slightly analogous in this way then
um uh people perhaps would have focused
less time
now we come to short-term goals because
uh there is some kind of assumption that
if you have a very myopic AI it will not
be dangerous we will need some kind of
long-term goals in order for the air to
be incentivized to taking over the world
uh catch explicitly writes that
long-term goals are required and that's
overselling the case somewhat
um because we also want the ahp like low
impact robust collusion don't do a
causal reasoning and several other
things but um in general the the
argument is is reasonably sound like in
particular if you make the AI have short
enough Horizon then it does in fact make
sense
um but
um then if you say short enough then it
becomes a tautology
so longer term goals those could arise
naturally they could also arise through
Mesa Optimizer
um do humans have long-term goals catch
a kind of greatly writes that yeah
humans seem to discard the future a lot
in decision making and I think this is
just plainly untrue humans do in fact
very much care about things that are
more than an hour into the future
um and I think especially if we are
looking for like an endorsed
implementation of our values then longer
time goals are almost certainly going to
be a thing
I think it's just
it's untrue that humans don't have
long-term goals we do really strongly
have a lot of long-term goals
catcher uses the example of a timed
chess match I don't think even this very
very simple example holds because at
time chess match if you have like five
minutes that's obviously Myer week but
it is in fact not uh because humans care
a lot more about other things than not
winning than just winning chests they
care about not being known as a cheater
henceforth and like if your Google chess
etiquette you'll find a lot of things
do's and don'ts that are not really
related to this time chess match but
it's obviously related to
um
humans having goals that are to not be a
jerk because we these goals Point uh
much further than just across this time
chess match
and also in particular where a Chinese
chess match is one example and sure that
may be very uh myopic very short term
but if you give an example that 90 of
all tasks won't kill us that really
doesn't matter give us very much safety
right we want something more uh
something like all
um relevant goals can be handled by
short-term uh myopic AIS I don't think
we have anything like that
knit Suarez also uh
causes narrow optimizers can exist but
won't matter and I think there's a
fundamental tension in this argument and
the argument we saw previously that AIS
will reflect our values very well and
and they will also be extremely myopic
um in particular that all AIS will
reflect our values very well and all AIS
would be extremely myopic catch it
doesn't use the word all that's my
interpretation of the argument but if
you try to remove the word all then the
argument for safety kind of falls apart
because like if you have some AI or even
most AI reflect our values very well but
someone just want to maximize paper
clips that could be really bad and most
of them are extremely myopic but a few
have long-term planning and want to take
over the world that also sound really
bad
so you need either all or almost all in
this argument for it to provide any kind
of safety
uh
that was one more point
that I forgot
um finally the um
the last is an example of
um as the trying to use a substitution
uh with the word uh Ai and instead use
uh to see if it holds for corporations
and uh catches arguing that in fact the
argument carries and that proves that
corporations are an existential threat
and that is like a reduction or
something like that
so let's try to use the same argument
for corporations the first any given
Corporation is likely to be gold Direct
um I somewhat accept this uh I think uh
corporations cannot be as coherent as a
single AI a single agent uh could be but
in general I think that there is in fact
substantial pressure for an uh for any
given cooperation to become more and
more goal direct so I buy this Bulls
the second and here I'm just gonna
strike through the word superhuman and
just if gold directed corporations are
built their desired outcomes will
probably be about as bad as an empty
Universe by human rights so we'll get
back to the Superhuman part but for now
let's just consider the standard
corporations
um
is this true well the desired outcomes
if they desired new outcomes are just
maximizing shareholder values then I
guess I accept this what the uh the uh
corporations want is something just
about as bad as an empty universe
catcher goes further however and claims
that we don't have a way to point
corporations towards human goals and I
do believe this is wrong I think we can
in fact uh assert effective control over
corporations and we can change them like
if we have a corporation then we can the
humans in charge can say we care about
uh employee happiness we care about
climate change we care about uh other
things that just shareholder
maximizations
um
third if most goal direct corporations
have bad goals the future will likely
very likely be bad
uh and I accept with some reservations
in particular I don't think we can say
very likely I think at most we can say
possibly and uh the the two problems are
one as we said before humans can correct
corporations and make them be about more
than just maximizing shareholder value
and the second thing is that
corporations will have a hard time
taking over power even if they would
like to do so
is answering that the reason why they
are not going to be able to take over
power is because corporations are bad at
cooperating I think this is a a
reasonably small part of it but to some
extent I do in fact buy the Bulls so if
we let corporations just take over all
power in the universe and we let
corporations just profit maximize then I
think we could certainly see a very bad
future so in fact I don't see this as a
reduction I think to some extent it may
in fact be true
now I pundit the word superhuman
on the previous slide so let's talk
about what does it mean that a
corporation is superhuman
um that is not really obvious to me you
could interpret it in several different
ways I think the intended way is that
um is a corporation that has some kind
of internal governance structure and uh
um bylaws and things like that that are
self-reinforcing in some sense so that
the um
the company is capable of acting even
against the interests of all the
employees but the employees are unable
to coordinate against the cooperation or
or something like that but it could also
be like uh more precisely that it's a
super intelligence compared to the
employees or that is super powerful in
other ways than intelligence compared to
the employees or something like that it
is somewhat unclear
Nate Suarez answers to this that humans
can't stack into superhuman corporations
but if we could then yes we should value
load and stack into is a bit unclear
whether he's talking about participating
and controlling and I think again this
is an example of what I would consider a
low quality hot take in that like this
is not me summarizing uh Nate Suarez
this is literally all Nate Suarez writes
about this and I think this kind of it
really doesn't drive uh our
understanding forward
so uh catcher concludes a counter
argument is that corporations are not
smart enough but in that case the
original argument needs to include that
and I I agree in fact the original
argument needs to include that but again
ketchik did not uh present a standard
argument she presented her own and was
in fact a big problem with this that it
doesn't make a reference to uh
intelligence in the same way that
bostrom's argument talks about super
intelligence
um and I think it is in fact very clear
that it should be added when we're
talking about a superhuman AI then we
are obviously talking about intelligence
and that needs to be uh uh
and that probably needs to be clarified
what does it mean to uh to have a
superhuman uh cooperation that we are in
fact talking about intelligence and if
catcher had to uh like
do all this all over this fine every all
of our terms painstakingly then this
would have taken way too long for her to
write and so the the things she should
have done instead of introducing new
terms and not defining them is to reuse
the standard terms that are used when
other people present the basic case for
AI X risk
that is all for today thank you and see
you in two weeks


Can we build AI without losing control over it? | Sam Harris

I'm going to talk
about a failure of intuition
that many of us suffer from.
It's really a failure
to detect a certain kind of danger.
I'm going to describe a scenario
that I think is both terrifying
and likely to occur,
and that's not a good combination,
as it turns out.
And yet rather than be scared,
most of you will feel
that what I'm talking about
is kind of cool.
I'm going to describe
how the gains we make
in artificial intelligence
could ultimately destroy us.
And in fact, I think it's very difficult
to see how they won't destroy us
or inspire us to destroy ourselves.
And yet if you're anything like me,
you'll find that it's fun
to think about these things.
And that response is part of the problem.
OK? That response should worry you.
And if I were to convince you in this talk
that we were likely
to suffer a global famine,
either because of climate change
or some other catastrophe,
and that your grandchildren,
or their grandchildren,
are very likely to live like this,
you wouldn't think,
"Interesting.
I like this TED Talk."
Famine isn't fun.
Death by science fiction,
on the other hand, is fun,
and one of the things that worries me most
about the development of AI at this point
is that we seem unable to marshal
an appropriate emotional response
to the dangers that lie ahead.
I am unable to marshal this response,
and I'm giving this talk.
It's as though we stand before two doors.
Behind door number one,
we stop making progress
in building intelligent machines.
Our computer hardware and software
just stops getting better for some reason.
Now take a moment
to consider why this might happen.
I mean, given how valuable
intelligence and automation are,
we will continue to improve our technology
if we are at all able to.
What could stop us from doing this?
A full-scale nuclear war?
A global pandemic?
An asteroid impact?
Justin Bieber becoming
president of the United States?
(Laughter)
The point is, something would have to
destroy civilization as we know it.
You have to imagine
how bad it would have to be
to prevent us from making
improvements in our technology
permanently,
generation after generation.
Almost by definition,
this is the worst thing
that's ever happened in human history.
So the only alternative,
and this is what lies
behind door number two,
is that we continue
to improve our intelligent machines
year after year after year.
At a certain point, we will build
machines that are smarter than we are,
and once we have machines
that are smarter than we are,
they will begin to improve themselves.
And then we risk what
the mathematician IJ Good called
an "intelligence explosion,"
that the process could get away from us.
Now, this is often caricatured,
as I have here,
as a fear that armies of malicious robots
will attack us.
But that isn't the most likely scenario.
It's not that our machines
will become spontaneously malevolent.
The concern is really
that we will build machines
that are so much
more competent than we are
that the slightest divergence
between their goals and our own
could destroy us.
Just think about how we relate to ants.
We don't hate them.
We don't go out of our way to harm them.
In fact, sometimes
we take pains not to harm them.
We step over them on the sidewalk.
But whenever their presence
seriously conflicts with one of our goals,
let's say when constructing
a building like this one,
we annihilate them without a qualm.
The concern is that we will
one day build machines
that, whether they're conscious or not,
could treat us with similar disregard.
Now, I suspect this seems
far-fetched to many of you.
I bet there are those of you who doubt
that superintelligent AI is possible,
much less inevitable.
But then you must find something wrong
with one of the following assumptions.
And there are only three of them.
Intelligence is a matter of information
processing in physical systems.
Actually, this is a little bit more
than an assumption.
We have already built
narrow intelligence into our machines,
and many of these machines perform
at a level of superhuman
intelligence already.
And we know that mere matter
can give rise to what is called
"general intelligence,"
an ability to think flexibly
across multiple domains,
because our brains have managed it. Right?
I mean, there's just atoms in here,
and as long as we continue
to build systems of atoms
that display more and more
intelligent behavior,
we will eventually,
unless we are interrupted,
we will eventually
build general intelligence
into our machines.
It's crucial to realize
that the rate of progress doesn't matter,
because any progress
is enough to get us into the end zone.
We don't need Moore's law to continue.
We don't need exponential progress.
We just need to keep going.
The second assumption
is that we will keep going.
We will continue to improve
our intelligent machines.
And given the value of intelligence --
I mean, intelligence is either
the source of everything we value
or we need it to safeguard
everything we value.
It is our most valuable resource.
So we want to do this.
We have problems
that we desperately need to solve.
We want to cure diseases
like Alzheimer's and cancer.
We want to understand economic systems.
We want to improve our climate science.
So we will do this, if we can.
The train is already out of the station,
and there's no brake to pull.
Finally, we don't stand
on a peak of intelligence,
or anywhere near it, likely.
And this really is the crucial insight.
This is what makes
our situation so precarious,
and this is what makes our intuitions
about risk so unreliable.
Now, just consider the smartest person
who has ever lived.
On almost everyone's shortlist here
is John von Neumann.
I mean, the impression that von Neumann
made on the people around him,
and this included the greatest
mathematicians and physicists of his time,
is fairly well-documented.
If only half the stories
about him are half true,
there's no question
he's one of the smartest people
who has ever lived.
So consider the spectrum of intelligence.
Here we have John von Neumann.
And then we have you and me.
And then we have a chicken.
(Laughter)
Sorry, a chicken.
(Laughter)
There's no reason for me to make this talk
more depressing than it needs to be.
(Laughter)
It seems overwhelmingly likely, however,
that the spectrum of intelligence
extends much further
than we currently conceive,
and if we build machines
that are more intelligent than we are,
they will very likely
explore this spectrum
in ways that we can't imagine,
and exceed us in ways
that we can't imagine.
And it's important to recognize that
this is true by virtue of speed alone.
Right? So imagine if we just built
a superintelligent AI
that was no smarter
than your average team of researchers
at Stanford or MIT.
Well, electronic circuits
function about a million times faster
than biochemical ones,
so this machine should think
about a million times faster
than the minds that built it.
So you set it running for a week,
and it will perform 20,000 years
of human-level intellectual work,
week after week after week.
How could we even understand,
much less constrain,
a mind making this sort of progress?
The other thing that's worrying, frankly,
is that, imagine the best case scenario.
So imagine we hit upon a design
of superintelligent AI
that has no safety concerns.
We have the perfect design
the first time around.
It's as though we've been handed an oracle
that behaves exactly as intended.
Well, this machine would be
the perfect labor-saving device.
It can design the machine
that can build the machine
that can do any physical work,
powered by sunlight,
more or less for the cost
of raw materials.
So we're talking about
the end of human drudgery.
We're also talking about the end
of most intellectual work.
So what would apes like ourselves
do in this circumstance?
Well, we'd be free to play Frisbee
and give each other massages.
Add some LSD and some
questionable wardrobe choices,
and the whole world
could be like Burning Man.
(Laughter)
Now, that might sound pretty good,
but ask yourself what would happen
under our current economic
and political order?
It seems likely that we would witness
a level of wealth inequality
and unemployment
that we have never seen before.
Absent a willingness
to immediately put this new wealth
to the service of all humanity,
a few trillionaires could grace
the covers of our business magazines
while the rest of the world
would be free to starve.
And what would the Russians
or the Chinese do
if they heard that some company
in Silicon Valley
was about to deploy a superintelligent AI?
This machine would be capable
of waging war,
whether terrestrial or cyber,
with unprecedented power.
This is a winner-take-all scenario.
To be six months ahead
of the competition here
is to be 500,000 years ahead,
at a minimum.
So it seems that even mere rumors
of this kind of breakthrough
could cause our species to go berserk.
Now, one of the most frightening things,
in my view, at this moment,
are the kinds of things
that AI researchers say
when they want to be reassuring.
And the most common reason
we're told not to worry is time.
This is all a long way off,
don't you know.
This is probably 50 or 100 years away.
One researcher has said,
"Worrying about AI safety
is like worrying
about overpopulation on Mars."
This is the Silicon Valley version
of "don't worry your
pretty little head about it."
(Laughter)
No one seems to notice
that referencing the time horizon
is a total non sequitur.
If intelligence is just a matter
of information processing,
and we continue to improve our machines,
we will produce
some form of superintelligence.
And we have no idea
how long it will take us
to create the conditions
to do that safely.
Let me say that again.
We have no idea how long it will take us
to create the conditions
to do that safely.
And if you haven't noticed,
50 years is not what it used to be.
This is 50 years in months.
This is how long we've had the iPhone.
This is how long "The Simpsons"
has been on television.
Fifty years is not that much time
to meet one of the greatest challenges
our species will ever face.
Once again, we seem to be failing
to have an appropriate emotional response
to what we have every reason
to believe is coming.
The computer scientist Stuart Russell
has a nice analogy here.
He said, imagine that we received
a message from an alien civilization,
which read:
"People of Earth,
we will arrive on your planet in 50 years.
Get ready."
And now we're just counting down
the months until the mothership lands?
We would feel a little
more urgency than we do.
Another reason we're told not to worry
is that these machines
can't help but share our values
because they will be literally
extensions of ourselves.
They'll be grafted onto our brains,
and we'll essentially
become their limbic systems.
Now take a moment to consider
that the safest
and only prudent path forward,
recommended,
is to implant this technology
directly into our brains.
Now, this may in fact be the safest
and only prudent path forward,
but usually one's safety concerns
about a technology
have to be pretty much worked out
before you stick it inside your head.
(Laughter)
The deeper problem is that
building superintelligent AI on its own
seems likely to be easier
than building superintelligent AI
and having the completed neuroscience
that allows us to seamlessly
integrate our minds with it.
And given that the companies
and governments doing this work
are likely to perceive themselves
as being in a race against all others,
given that to win this race
is to win the world,
provided you don't destroy it
in the next moment,
then it seems likely
that whatever is easier to do
will get done first.
Now, unfortunately,
I don't have a solution to this problem,
apart from recommending
that more of us think about it.
I think we need something
like a Manhattan Project
on the topic of artificial intelligence.
Not to build it, because I think
we'll inevitably do that,
but to understand
how to avoid an arms race
and to build it in a way
that is aligned with our interests.
When you're talking
about superintelligent AI
that can make changes to itself,
it seems that we only have one chance
to get the initial conditions right,
and even then we will need to absorb
the economic and political
consequences of getting them right.
But the moment we admit
that information processing
is the source of intelligence,
that some appropriate computational system
is what the basis of intelligence is,
and we admit that we will improve
these systems continuously,
and we admit that the horizon
of cognition very likely far exceeds
what we currently know,
then we have to admit
that we are in the process
of building some sort of god.
Now would be a good time
to make sure it's a god we can live with.
Thank you very much.
(Applause)