254  Lets see you write that Corrigibility Tag

hello and welcome to session 254
in the ai safety.com reading group
tonight we will be discussing the less
wrong post let's see you write that
courage ability attack by elias
yorkowski
eliza yutkowski is
the founder of miri and
the the senior researcher and uh of
course the author of this post but we
will also be looking at 19 other
post or comments to this post
most of which are by people that i don't
recognize
the post was
a couple of months ago and it's kind of
like a competition
unless wrong and there is one entry in
particular the one made by elias
yorkowski that i will focus on
uh note that i did not participate in
this competition and that is of course
kind of unfair for me to um like i have
also presented some of the principles
that i would add in this competition but
um that's after reading everyone else's
comments um and principles so that's
obviously somewhat unfair
this all started out with elisa uh
yutkowski in the post agi ruin a list of
lethalities lamenting that it is only
him who is writing up the list of
lethalities and
no one else seemed to be able to do that
even who being
objected to this
saying that uh why should other people
do that they are writing the same things
just organizing it in different ways and
in particular
pointing to a set of four people even uh
paul cristiano richard go
and scott garabrant as four people who
have written up
something that is um essentially covers
the same part
and so um elias zaykowski notes in this
post that many people could have written
a list that even says that many people
could have written a list like that even
though obviously uh like you can argue
these maybe if elia siedkowski is the
best alignment researcher then this may
be number two to five uh of course
that's with a big question mark um so
here we'll have we'll see whether um in
a competition of who can write lists
like this uh number one is in fact
better than number two to five
and uh the the people uh even hooping up
uh poker sean richard and scott are not
really central to this list
so
um elias szykowski in the post claims to
have been asked why not challenge others
to write such a list in advance
and he is
rather negative about this idea it
doesn't do any good and doesn't persuade
anyone people don't like tests and he
doesn't believe in such tests and he
couldn't pass a number of similar tests
but oh well let's try
because
if
it might be a worthwhile try sometimes
you are positively surprised
so the task is criticality we want to
um
here is repenting as a definition of
courageability the hypothetical
motivational system that means that the
ai doesn't want to kill you even if you
didn't build it correctly
the way elias yorkowski writes uh this
is in attack in uh
mad investor chaos and the woman of s
motors which is a fanfic and attack in
this
case is some kind of a post or reply in
in the sense that you are supposed to
reply next because this is
co-written um together with
i think kelsey piper
so the topic is the principles of
courageability
and
credibility is of course an important
subject and
already the framing of of principles is
somewhat suspect in my mind in that
corridability is unlikely
to be something that we can
in a principled way uh guarantee but
more like something we obtained through
a number of different means and uh i
think this is a rather is a better word
as in
we have a large number of things we
would like an ai to have such that it
becomes more or less correctable um
principles are things you don't
compromise on at all and some kind of
compromise is almost certainly going to
be required
so to uh ensure that everybody is on the
same page elias kowski gives four
examples of
principles for credibility those are
myopia low impact boundedness and
quantization
so based on this the people who want to
see whether they can
take on this challenge and write uh
correctability tag uh you would expect
them to among the principles of
courageability have these four things
write something like principle of myopia
the ea i shouldn't only consider short
time horizons something like that would
be very
expected and in fact
none of the 19 people who eventually
take up this challenge
have all these four
criteria uh the theodoraza principles
myopia low impact boundedness and
quantalization
so before we can
probably
understand the the framing we need to
talk about darth elen
darth elen is a
fictional world that is the background
to many of elias yorkowski's stories
including
mad investor chaos it is a
civilization that is capable of solving
the alignment problem it is also a
civilization that's good in many other
ways i think you could argue that this
is elisa yatkowski's vision of utopia uh
there are a lot of people who criticize
different aspects of this
in different ways so uh but but so first
approximation i think i think it holds
lots of people in his stories that
criticize part of it like literally uh
mad investor cares is uh written from
the point of view of someone who is kind
of
not quite an outcast but someone who
certainly doesn't feel at home in
deathly land
so it's not
not really a utopia in that sense
or it might be
okay what is how does death elen solve
the alignment problem one of the ways
they do that is using a selective
breeding program to increase the average
and maximum iq while maintaining other
good traits
um they have a very great education
system with a lot more focus on
coordination and economics then we have
their goal is to eventually build an
aligned super intelligence um but they
are taking their sweet time in doing
this and expect to not really start for
a generation or so um and uh because
they are coordinating well enough to
ensure that
there will in fact not be um any kind of
race at all
if you uh like uh there was a link
previously posted um to to the post
where elias talks about the principles
of credibility in this setting
if you want to understand a bit more i
recommend going back
approximately four posts
you can go back further if you are
really keen on reading i think almost 2
million words
but
that's not really necessary the the
context is uh mostly these the past four
posts
so even though they want to build an
aligned super intelligence they have
another plan and that is if something
bad should happen really urgently they
are thinking of building a critical
thing
if it becomes necessary to take drastic
action then they can have a
and they want to build a an ai that can
do something very very important and
difficult without killing them all
and um
in particular the uh pivotal act for for
them is things like stopping a comet
rather than our
problem of stopping uh on aligned uh ati
and that gives them some very different
uh options that we don't have because a
lot of our um pivotal acts are centered
around coordination
so this
large project has developed 15
principles of courageability and those
are of course the center of this
competition this is yorkowski's entry in
this uh in this competition
these are the two main characters in uh
mad investor carriers and there is
2169 words in this uh
in this post
compared to the other 19 people who
replied this is
substantially
more than double the
the second largest so it is kind of
verbose
um
there is also some text around the
principles um
i'll get to that later and
uh yeah
one thing we should notice is that it is
quite possible for these people to have
read each other's entries um and the
post times are not really clear
i couldn't find that on um
on mad investor chaos so like a lot of
people could have cheated
and i have no way of knowing that and
i'm just going to assume that no one
cheated and looked at what other people
had written
so this is framed as some kind of
competition and it's a competition
without any kind of judge
perhaps eliza erkowski will judge it for
himself but no one will judge this
publicly so i hereby announce myself as
the judge of this competition and the
way i will do this is just to get some
kind of sense of it i will count how
many principles have each person
suggested that's a really crude metric
and i know that um
in addition i will look at how many
upvotes and less strong did people get
um of course elias atkowski didn't
post on last round he posted it on
glophic
but
yeah
and then i uh go through all the
principles the 15 principles that elias
yorkowski
listed and then
everybody gets like one principle point
um if they mention anything related to
these 15 principles
um and i did this in a
very very
uh
optimistic sense in that even if people
really really uh wrote very little about
a principle i gave them full point and
counted that as a principle i think
definitely if you wanted a more precise
measure you should like on a scale from
zero to one like how much do they
actually write about this principle and
maybe also something about weight like
some principles are probably more
important than others
um uh and then other people uh suggested
some principles that elias kowski didn't
mention um and yeah um i also
interpreted that really really
generously like if if people wrote a
paragraph about just about anything
related to courageability then i was
willing to call that a principle even
though most people didn't call that
principles
so i counted up painstakingly everything
and then i uh
threw the tell it result away um and
went with my gut feeling that's a
classic rationalist um principle uh way
of doing it
um so let's see what the here you can in
fact see my uh spreadsheet where i
calculated everything and wrote down
some of the
the principles that people have i will
go through the results here in a more
abbreviated form
so i'll write them down ordered by how
many upvotes they got unless wrong
yotkowski didn't
so i just put him arbitrarily on the top
here and then there was ian colvaid with
uh nine principles um elias kowski
posted unless wrong that that was the
one to beat
paul christiano wrote a post where he
didn't in fact give any principles for
credibility but talked about whether uh
that was a like a crisp
concept or not and kind of rejected the
principle john wentworth 16
someone called anonymous ai safety with
11 laura langoska with 14 and yeah then
people going just down you can see
everyone here
just about the only person that i know
from outside
less wrong posts would be richard and go
with six principles
so counting up these uh we get a total
of 99 principles
um so
we that's quite a lot um and one of the
key things that i was looking for was
some kind of meter principle that helps
us trade off these principles if we uh
get uh like if we can't satisfy all of
them um and that was unfortunately one
thing that no one
no one had any principled idea of how to
do
also one thing that i was a bit angry
when i read all this
was that unless wrong people are totally
not upvoting as much as they should like
if you look through this uh this list
like the lower third of them have
literally zero upvotes except the auto
upvote they get from by themselves and i
think definitely if you
write down the principles of creativity
even if you are not very skilled then
just trying that to in my mind deserves
way more about so i went and gave
everybody on this list a strong upvote
and i think it's uh i i think it must be
demoralizing for some of these people to
spend the time and write
many hundreds of words uh in in these
principles and then get precisely zero
feedback and not even an upvote
but that's not really related to this
okay so who did in fact win this um i
think that elias yorkowski's list is
substantially better than
any individual list from the rest of the
of the people who who
participate in this competition
the uh the uh
the principles are
include a lot of things that others are
talking about but in a way more detailed
and way more uh
uh powerful way and it's uh
uh they ha
many of the other principles uh that
elias erikowski did not have are written
in a very superficial way uh i think
elias yotkowski's list is certainly the
best but um
his pessimism like whether other people
could write something like that i would
say that is somehow uh falsified in
particular he um even hooping his idea
of whether number two to five could do
something better uh similar combined i
think that is in fact true if you uh
didn't have elias kowski's list but you
have had uh like the the next four
people's list um then i believe that in
total would be of higher quality than
you'd cascus lists
so
i don't think uh there is a strong
conclusion coming from this um i believe
that
elias akaski's list is the best by some
margin but not a
insurmountable margin
and now i will go through elise caskey's
15 principles of credibility
and
the way i will structure this is that i
will build some of the text when i bolt
the text it means that there is a sub
point here made by ilyasoykowski that
none of the other 19 people
had any mention of um
also uh i've covered pasted some of
elias yorkowski's
principles and in all of them he refers
to not an agi but to a thing that you're
building um and um
so whenever you see a thing then you
should think that it's the agi
so the first principle is on personhood
uh the thing shall not have qualia
and has nothing to do with safety but
just because it's morally wrong so it's
not really related to credibility in my
mind
and no one but elias yotkowski has this
as a principle
is this in fact a good principle
i would say no i would say that my
morals
disagree here
let's say you build a super intelligence
and it does have qualia
and it does in fact suffer
in even in that case if the
agi is only doing a pivotal act and then
can be uh safely shut down or
compensated or made nicer in some way
then even very substantial um suffering
like in the one we who walk away from
omales even something like that i think
would be certainly acceptable and i
think in fact consequentialist
utilitarianism i don't think it is a
taskiest one
but i am and i think it is just about
required and if i could do something
that would be really really uh aversive
and be really tough on myself but would
literally save the world including
myself then i think it's
morally required to do so
so i uh went to um
san francisco not just beca for this
reason um and talked with eliezer
yutkowski and presented my objection
directly to elias utkowski and said i
think it's uh required to do so and
elias kowski did not agree and he argued
that the suffering could possibly be
really really severe um and
we didn't make much progress i've kind
of felt like
when you're talking with someone who
have like a
different meta ethics it's kind of it
sometimes feels like you're talking with
an alien um and um we didn't get
anywhere um
but i did in fact present that objection
uh i think that
this is close to a an actual safety
principle however in that indexical
preferences like
uh
if you build a an agi then if you can uh
separate out the parts that re
refer to itself
with for things like qualia then i think
that is probably
a good thing to do as much as possible
to reduce suffering uh sure but also
like reducing suffering from the adi
does have substantial safety
implications
and of course um the reason why this is
uh
uh my guess of why this is included in
the story in particular is that dathilan
is a society a civilization that cares
very very much about morals and
not building an agent that suffers is
something they strongly strongly um
they really really don't want to cause
suffering
the second principle is tuskishness
and if you're building a like a limited
agi with a limited task um then that's a
limit something that's hopefully limited
in space doesn't
take too much time and
and it's also probably something that
you should um limit how much probability
you want to be uh to actually obtain
this in the sense that doing something
appeals will act with a probability of
99.999
is probably fine but if you try to uh
have the task bounded so it must be
99.99
and with enough nines you end up with a
strange um
with with strange solutions
if you want to
have the probability of failure below
one in a trillion um so elias rutkowski
is the only one who recognizes that the
tuskishness must also refer to
probabilities
and so of course this is something a
principle that was stated in this way in
in the actual con
competition
um
so that is like the task but there is
also the input to the task
whether that should be bounded any least
yield casket
that should be bounded the knowledge and
effort should be bounded um and
again the limited effort is also
something that's unique to eliezer
and the idea that this principle applies
fractally at all levels of cognitive
subtasks so it's not just that the uh
the thing that it wants to do is um is
bounded but everything that
it is
all subtasks are also uh
uh bounded in the same way uh
all the way down so as an example there
are no while loops uh
programming um structure that is
that can be open-ended but only limited
enumerated loops
and doesn't try to think of every
solution or member of category only at
most 10 and doesn't try to think as long
as it can until finds a solution but
only in some bounded time
um
so i think that is uh something that
certainly makes an ai more credible
easier to uh
to bound to limit
it limits me to optimizes but i don't
think it would in fact stop it
next principle is mild optimization
this is quantalizers basically um like
um the air only seeks adequate solutions
and stops looking once it has one um and
it's not trying hard to optimize and get
this best solution for for anything
um except actually aliasing casting
uniquely makes a an exception for this
there might in fact be some small areas
uh where finding a um and the optimal
the best solution is in fact required um
and he puts up some um
some requirements here for for when that
is uh allowed
and the reason of course is that if you
optimize really hard for a solution you
are going to like the variables that are
not in being optimized or are going to
be extreme
to what extent will this work that's an
interesting question um i suspect in
fact that adequate solutions uh
once if you stop once you have an
adequate solution one of the first
solutions you may come up with is taking
over the world and that's certainly
adequate and
many pivotal acts are in fact really
really difficult and taking over the
world might be easier than that
and
i think
quantization can in fact be a strong
principle for credibility in the sense
that we have other methods to try to
avoid having the ai think of taking over
the world and when it's nothing of
taking over the world in that case some
kind of quantization is probably
something that contributes positively to
courageability
here is a principle that elijkowski has
that i haven't seen before and that no
one else seems to have presented before
i mean probably there is someone who
also thought of this but i've never seen
it before tightly bounded ranges of
utility and lock probability
so
let's say that you have a
some kind of action space
and you are thinking of
different things that could happen and
then there are also some things that
could happen with a very low probability
like one in a million
how much
should you think about these very very
rare cases well you should probably
think about them a little but mostly
they can be disregarded but it's
possible that the ai will at some point
uh come up with some strange part of the
uh the action space that looks really
really malformed in some strange way and
a science
uh value to this um and in that case um
we want to uh like this is what we want
to avoid you
throw an exception or or just stop or
avoid it
in a similar way the
utility it gets from different uh
from different kinds of solutions should
not very unbounded so it's not something
like um create as many paper clips as
possible and you get one util for each
paper clip it's more like
you can get between zero and one utility
and you could just expect that a good
solution will get 99.99 or something uh
so you're using the full range
so you like cut off
the uh the really extreme solutions in
that way
um
yeah
and and the suggestion once you get
something that comes
that differs from this then
you should throw an exception this is a
thing in dathilan that is much more
general than in our societies
they have a principle system of
exception handling on an organizational
level
so
so for for the context of this story it
makes sense to just say throw an
exception um
for real-life corridor uh projects to
build a corruptible ai we will need some
kind of principle way of handling
and a lot of people in their principles
talked about how to handle this kind of
warnings or exceptions or errors um
there are a number of suggestions um
some of them actually directly opposed
to each other um
i uh i won't go into all the details
there are a number of ways you could do
this
um and alias zerkowski also says
that
if you can't find a good solution
without thinking of improbable events
then that is really worrying
my idea for how to handle this is
somewhat different
i think
once
the ai tries to uh
get a solution and fails in this way um
then um
then obviously some of the principles
need to be constrained and need to be
relaxed
and we need some kind of principled way
to do that like the symbolism is just to
like
drop the fewest number of principles
that's like way too simple i think we
can do a lot better but we need some way
of some meter principle to handle this
then we get to low impact
that is a solution that doesn't change a
lot of things
that are not
almost always changed with by
non-extreme
solutions of the task
so i think this is a cute way of trying
to formalize low impact by having just a
binary
is a particular solution extreme yes or
no and moving that into a detailed
impact measure i think that's an
interesting way of going about that
i
think that works in just about every
case except pivotal x unfortunately
pivotal acts are
likely to be extreme in my in my view
and unfortunately i don't think this is
going to be sufficient to help us
and elias
uh
warns us that this is a lot easier to
say low impact than actually formalizing
this um
the text says it's easier to illusory
name intel dane tildene is just
the common language think english
and part of the reason why this is
complicated is that the thing is
learning its own model of the
complicated world with an ontology and
representation that you don't know when
you want to define impact
of course trying to reduce impact as
much as possible is another recipe for
disaster
but it uh
eliza caskey uses a an image i think
it's from fantasia where you have an
animated broom that is uh
filling a cauldron
and most ways of filling a cauldron will
not cause
uh like the workshop to be flooded um
and that means that in uh if the the
animated broom finds a way that involves
the
uh the workshop being flooded then
that's probably not low impact
um so how uh
can we
uh like get around the problem that we
we have a hard time formalizing this
and i have an an idea a principle uh
that we um
we want a solution uh specified by
humans that
is poor and on on some dimension
and try to improve on that rather than
trying to create a prince a a
really new um
a new solution i've written here about
how you could do that for different uh
pill selects it's kind of like a
half-baked idea for a principle
that we could substitute for low impact
in case this seems to this is hot too
hard to model
now we get to the principle of myopia
and myopia the the standard way of
thinking is just
don't think about long-term time
horizons and that's how most people
thought about it eliza kowski has a more
complex definition of myopia we break
the work into subtasks
work can be both planning and execution
where each subtype
task takes limited time
or in my view also
influences a limited space
and there is some kind of compact
description of the final state of this
task so we can determine if the final
state is acceptable and then instead of
just having one instance of the agi that
looks at the entire task we break it out
into different instances that don't look
beyond the result of their own subtask
this is nice if it's possible it might
very well not be possible from a lot of
practical cases
i think
at least a limited
principle of myopia is required in the
sense that we want the entire task to
satisfy this criteria
and i think
if the the entire task does not satisfy
this criteria then uh
we um
yeah that that that would be a very
problematic thing if uh working at the
task
could not be a done in a myopic way
separate superior questionnaires
questions
so
the question if you have in particular
if you have something that is myopic uh
you might have a a separate a instance
of a separate ai that looks for
long-range impacts
and
this is something that a lot of people
have thought about how to get this kind
of ai help in
getting questions uh ai safety via
debate was one of the the options um and
this separate questionnaire should
return just one bit um and not be used
by the agi to try to uh
build solutions so it shouldn't be like
it's optimizing against and finding the
the best solution that the long range
questioner um
can accept
it needs to be stronger
the the ati
the one that checks whether a solution
is safe
have more cognitive power one of the
ways would of course be to um
relax some of the principles like myopia
we would probably want that the uh
the questioning uh ai is less myopic
than the uh ai that builds solutions
and an interesting idea here is that
this should be a
not just a separate instance but but an
entirely different uh algorithm and
architecture
and i think that's interesting and i
think
a very hot take is that if we ever find
a
an ai that seems like it could be
correctable then large language models
may indeed be something that could
fulfill the role as a separate theory
questioner in that it is
they the large language models are very
powerful and
of course will have a very different
architecture um but
it's kind of a hard take i don't think
uh language models are likely to be uh
part of the solution
conservatism
if you're searching for a solution
choose an unsurprising one relative to
your probability distribution
it doesn't really sound like
conservatism in the way i think about
conservatism uh like unsurprisingness
um
uh conservatism uh to my mind have
two parts if you look at a
the full task or more likely a subtask
then there is conservatism in means and
in goals and
um both of these are uh important but i
think the principles uh and the uh uh
the
how
different they are uh how they should be
treated from a safety point of view is
quite different in that very strange
means are more acceptable than very
strange goals
um
and of course pivotal acts in particular
we would want to maximize some kind of
conservatism in this case we
we can't probably get a lot of
conservatism
but uh but this is uh
uh like a surprising pivotal act sounds
like something that is almost certainly
going to be not nice
and
part of elias kowski's insight in this
principle is again this is something
that applies fractionally to cognitive
subtasks
and of course also
again as before we're not trying to
choose the least surprising one because
again every time we are optimizing
something that we don't strictly really
really strictly need to optimize we
introduce a lot of potential for very
weird behavior
and and weird behavior is in fact the
key thing in that we're trying to do
something that is weird people's likes
are strange by definition and
if the solution is weird then um
we need to kind of manage the weirdness
in the sense that we need we will get
something that is weird but it can't be
too weird
conceptual legibility is also something
a lot of people talked about like we're
talking about interpretability here
and elias yokosuka's formulation is the
solutions at all levels of cognitive
subtasks should have short descriptions
in the conceptual language of the
operators
this is somewhat different from the uh
the standard interpretability uh
which is
written here as um obviously dangerous
uh thing
reportability which is something that
some of the other comments suggested
that the ai tries to explain somehow
it's planned to humans uh
even though
the the plan itself contains some
structure that humans can't understand
in some sense can natively understand
um
i think when i think about this kind of
interpretability work i think about the
the the analogy of a um of a human who
is like an alchemist
and a chemist ai that is trying to
explain something to the human and uh in
this case
the short descriptions in the conceptual
language of the operators may just not
exist like the the alchemist is thinking
of things like the four humors or
something like that and the uh chemist
is talking about uh like atoms and
molecules that that aren't just simply
not present in the in the mind of the
human so um i think
that this kind of conceptual eligibility
is too high a barrier i don't think in
fact an ai that is capable of performing
a pivotal act will be able to have
conceptual eligibility i think we will
need to go for something
less ambitious and to my mind that is
something like strategic eligibility in
that a lot of the concepts may not be
legible and we'll just have to accept
that but the strategically relevant
concepts should be legible
those are the ones we really can't
afford to compromise on
operator looping
um
well
how much should you keep a main in the
loop
a lot of people say like
a lot as much as possible um without
really recognizing that this is in fact
fundamentally impossible because as
elias rykowski puts it if the operators
could do the job then there wouldn't be
any need for the
ai we only want it to do a pivotal act
that we can't do ourself
um
so
it is however still possible that there
is a part that the operators can do
themselves
without
uh the agi down too much in that case it
should obviously be done
so doing the thing is quite different
from um
what's called oversight which is a
as a related principle that a lot of
other people have picked up on
whether it is
to my mind i would expect that verifying
is in fact substantially easier than uh
than doing the tasks themselves but of
course this depends on whether the
subtest is in fact legible to humans um
which might very well not be
um
yeah
and of course you even this requires
that there is no adversarial um
sense in the ai if it tries to
push some of the work done to humans uh
then that requires that the ai is not
really adversarial at this point
again to my mind this is something we
should exclusively do in strategic
matters and not in some technical things
like trying to build nanotechnology or
things like that
whitelisting is an interesting uh
concept that i must state up front i did
not in fact understand and the big
problem is that there are really no
examples here is how elia sirikowski
defines whitelisting
every part of the system that draws a
boundary inside the internal system or
external world should operate on a
principle of ruling things in rather
than ruling things out
so uh on like the object level if you
just literally do that it seems like
that will be really really expensive and
uncompetitive um like for instance if
you take the classic example of the
complexity of values that's something
that you really can't capture human
values uh using whitelisting so i think
that is probably not going to work but
this like don't take my comments with
tomb
uh
too too strongly because i don't really
understand what elias yukowski is
getting at i think this could be related
to
perhaps the heart problem of courage
ability and i think it could in fact be
really important i would like to know
what elijkowski is thinking of and he
claims that there are a number of
principles he have not written up and i
think he should totally do that because
i'd be interested in reading his
thoughts on this
shutdown ability or affordability um
so a lot of people uh
noted that there is this creating a
an a agi that has a shop
a shutdown button but that is not it
does not mo is not motivated to make
this button be pressed is a substantial
problem elle sierkowski uh points to
some research
seven years old uh summary that says
that actually doing this is really hard
and miriam has tried to formalize this
and somewhat failed um
and he points out
that the principle is one that needs to
apply uh
almost fractally in the sense that if
the ai builds something it has a sub
agent perhaps
some kind of mechanism then that also
needs to have an
off switch that you can press that turns
everything off and not just the sub
agent and gets a low impact after that
um
again with with the analogy of the
animated broom
um so how hard is this in fact i
remember stuart armstrong doing work
that was later than this link
and i remember seeing something like
it's a long time ago unfortunately so i
can't quite remember but i do remember
him getting somewhere with this i don't
think it's fair to say that it was just
totally impossible
um and certainly that it's it's true
that it's not no challenge at all
so before we get to the next principle
i'll just talk a bit oh i'll let elias
yorkowski talk a bit about modeling
adversaries because that is something
that can indeed be um problematic and
something that i think we should have
some kind of
very strongly principled uh way around
because i can foresee a lot of different
problems coming out from this kind of
thing and i think in particular one
thing we might want to do is to just
have the
corrugal ai be unable to
model or think about other ais in
general
primarily to avoid a collusion which i
think is likely to be a very real
problem in practice um where there's not
just going to be one corridable ai but
many um ilius kowski is more talking
about things like the ai considering
whether it's in fact inside a box and
whether other adversarial minds
exist
i won't go into a lot of details about
his theories i think they are totally
valid but i also think they are um
we need some kind of strong super class
to uh to get around this problem rather
than uh trying to rule out the first 10
things we can find in the space
this kind of modelling adversaries is
avoided by a principle that elijah calls
behaviorism
and the
the way the thing that he calls
behaviorism is basically not modeling
other minds at all
um that's of course a super set that
fixes this problem i'm not entirely sure
i like the word behaviorism for this i
don't think behaviorism means don't
model minds uh i think the the technical
meaning is quite different um like uh
things like whether people have like
free will and personalities and things
like that is again different from the
strategic considerations
in my mind
i think there are a number of good
wreaths
in to not modern minds at all both ai
and humans
i think that is in fact a nice thing and
that is uh
this is elias kowski's argument for why
the a
the uh corrugal ai should not model
mines at all um i think it's um
probably too large not modeling mines at
all rules out a very large number of
pivotal acts as far as i can see it uh
and it certainly makes it uh like
uncompetitive
yeah
so uh
the
one of them would be uh
forming some kind of positive singleton
like if you could uh
uh have a um a pivotal act that involves
a world government that is benign and
takes ai risk seriously
that would be an example of a pivotal
act
where
convincing people to
take
to set up a world government a benign
world government will almost certainly
require
modeling the minds of politicians and
voters
and organizations
not in the minds of organizations but
politicians and voters have minds and
you cannot in any feasible way convince
people to set up a
a government that bans a unaligned agi
without modeling their minds
i think many other
actions in fact also like um a classical
example is using nanotechnology to um
to melt all gpus
is an example of another pivotal act and
i think in order to do that in the real
world where there are
other people working uh against you
strategically you need to model the
other people who are working against you
and you need to um like think what would
be the uh
the reaction of the american president
if we suddenly melt all gpus i don't
think you can get around that in any
meaningful way
i hope that answered the question
i think a uh totally
single-minded engineer that is super
good at nanotechnology cannot in fact
take over the world you need someone to
put that into a strategic
frame and um
to make that into a real plan and real
plans in general require
thinking about what would other people
react to my plans
yeah uh
yeah i i think uh like um it does uh
like strong behaviorism defined in this
way uh like not modeling minds at all
does seem like something that
would really make it hard for an agi to
take over the world
there it needs to be a very strong power
imbalance to have any agi that's so
powerful and can take over the world
without modeling the humans and how they
would resist
i think it's not totally impossible but
that that is a very strong
encourageable
element
so
this science-based anti-optimization
separation
like can you guess just from the title
what that means
well it refers to the
well-known thing that if you uh
put somehow put your utility function
into the ai well then if you make a um a
sign error then suddenly it's not
maximizing your utility function but
minimizing that or you could get a
cosmic ray or something like that
and you could get blackmailed in some
way um at least because you have some
funny uh are not funny ways this could
happen um i think this is not
that related to the science-based
anti-optimization separation in fact in
the sense that um
if you
no matter what the um the ati has as a
utility function it would be possible to
for an adversary to just model that and
and threaten to minimize that
um
and
in fact if we are trying to have a
taskish
agi that is just trying to do a pivotal
act and then shut down afterwards then
it doesn't actually need to have a
a model of our utility function like
it's it's not supposed to
maximize and change the universe in
according to our values it's just
supposed to do one simple thing and then
shut down
um but
a lot of courageability uh suggestions
by the other people uh
who reply to this threat suggests that
having some kind of reference to human
values to use as
um some kind of sentence check
is valuable uh and i can i can see it
could be valuable and in this case we
have two um principles that are directly
opposed to each other and again we have
no principle way of
of
weighing these two things against each
other
there's a principle of epistemic white
listing or domaining
like
thinking about separate domains
separately um
and not across domains because cross
domains is part of the place where a lot
of
the india comes from if you are thinking
about like how
the operators
think about if i
obtain this knowledge instead of this
knowledge when you're trying to do
something really specific that seems
like something that's almost always
going to go wrong
this is close to
eric drexler's comprehensive ai services
idea
where there are
wholly separate ais in each domain in
each silos that are narrow
my expectation certainly from my
objection to eric draxler was that if
you want to make it strongly narrow and
really
have the ais you need to be unable to do
any kind of cross-domain reasoning then
um
strongly narrow ai eyes are almost
certainly going to be strongly
uncompetitive um
or if you want to like um
relax this even just a little then
you're going to get information from one
domain onto the other
very easily it's really really hard to
keep
knowledge and silos away from a an ai
that's more intelligent than you it can
figure out the truth in
based on really small evidence which i
think is actually a
major point in mad investor chaos
so
this my objection to this isn't one that
is um
uh at odds with what elijkowski is
writing but i think in fact that a
principle of competitiveness
is required for how to build a collegial
agi in the sense that if you try to
build an agi
to solve some kind of task
but the performance is too weak because
you are
putting too many constraints on it
then at some point it will just
be too slow and and the comet will hit
the earth anyway so you need to have
some kind of
performance minimum
for from the solution and if you're not
reaching that performance minimum then
you need to start doing for relaxing for
instance the epistemic white listing
criteria
and of course for our case it's a lot
worse than for uh
dusty lens because we are
uh probably going to be in some kind of
a strategic race between a critical agi
and a potential unaligned ai
finally we get to the last principle
principle number five fifteen the heart
problem of corrugality or anapartistic
reasoning
like i tried to figure out what does
anap artistic reasoning
mean and like i could see that
elijkowski had used this word and just
about no one else had used this word and
i couldn't figure out the etymology i
really don't know what anap artistic
means i would really like to know that
because the heart problem of
correctability is interesting it is uh
like um if i was uh
like doing youtube videos for engagement
i would have like a thumbnail with me
saying ah
one simple uh trick to solve ai
alignment um and this an artistic
reasoning is in fact one simple trick to
solve
the alignment problem and this is
figuring out how to make the ai want to
be courageable and if there is some part
of corrugability that we haven't figured
out then you want the ai to figure that
out
imagine what hypothetical operators
would want if they were building an ati
that thought faster than and whose
thought were hard for themselves to
comprehend
and uh figuring out how can you make
yourself more courageable and
do that in a way that even though it
seems like it is helpful it is
surprising and thus not helpful um
one way i've seen this uh stated is that
you want the ai to um
its
inner mental model should reflect the
external model that the programmers have
so from the programmers point of view
the ai is um
has values that are almost certainly
wrong and we want this to be reflected
in the ai in some uh deep sense
and elias kowski claims that in dothi
land people
wouldn't try to build this into a a
thing like they would almost certainly
build it into a full super intelligence
if they ever got around to to make one
of those but the uh the assumption in in
this uh corrigibility uh principles is
that you are under a very severe time
pressure in order to build this and this
is a principle that is too deep meter
and elegant and hard to pin down
so you wouldn't try to actually build
this into something if you are unless
you have a lot of time and a really
strong understanding
uh i think elias claims that no one
would consider doing this
and
actually and i think something like five
people wrote
in their comments some kind of
suggestion to that
uh a principle that is related to the
heart problem of correct ability um so
um
how hard this in fact is is an
interesting question uh i think elias
is perhaps a bit negative about how
difficult this principle is um five
people suggested we should build it but
uh none of them had like any
real idea about how to go about this uh
so
it's possible indeed that elias
yorkowski's principles are
more realistic in the sense that he is
uh
saying this is hard and he has done in
fact a substantial amount of work in
trying to figure out how to do this and
then a number of people saying we should
try to do that without um looking into
how would you actually go about this
i think in this case um
i would trust way more elias koski
because he has tried hard to
to to solve this problem
even though he has failed um and that is
why i think
the elias yorkowski's 15 principles for
critical adi is the best one in in this
competition
that is all for today thank you and see
you next time


Superintelligence: Science or Fiction? | Elon Musk & Other Great Minds

I'm going to ask a question, but you can only
answer by saying either 'yes,' 'no,' or 'it's
complicated.'
Alright?
So, let's start over here.
Is some form of superintelligence possible,
Jaan?
'Yes,' 'no,' or 'it's complicated.'
Yes.
Yes.
Yes.
Yes.
Yes.
Yes.
Definitely.
No.
Well, this was disappointing, we didn't
find any disagreement.
Let's try harder.
Just because it's possible doesn't mean that
it's actually going to happen.
So, before I asked if superintelligence was
possible at all according to the laws of physics.
Now, i'm asking, will it actually happen?
A little bit complicated, but yes.
Yes, and if it doesn't then something terrible
has happened to prevent it.
Yes.
Probably.
Yes.
Yes.
Yes.
Yes.
No.
Shucks, still haven't found any interesting
disagreements.
We need to try harder still.
OK.
So, you think it is going to happen, but would
you actually like it to happen at some point?
Yes, no, or it's complicated?
Complicated, leaning towards yes.
It's complicated.
Yes.
Yes.
It's really complicated.
Yes.
It's complicated.
Very complicated.
Well, heck, I don't know.
It depends on which kind.
Alright, so it's getting a little bit more
interesting.
When I think, we had a really fascinating...
When is it going to happen?
Well, we had a really fascinating discussion
already in this morning's panel about when
we might get to human level AI.
So, that would sort of put a lower bound.
In the interest of time, I think we don't
need to rehash the question of when going
beyond it might start.
But, let's ask a very related question to
the one that just came up here.
Mainly, the question of well if something
starts to happen, if you get some sort of
recursive self improvement or some other process
whereby intelligence and machines start to
take off very very rapidly, there is always
a timescale associated with this.
And there I hope we can finally find some
real serious disagreements to argue about
here.
Some people have been envisioning this scenario
where it goes PHOOM and things happen in days or
hours or less.
Whereas, others envision that it will happen
but it might take thousands of years or decades.
So, if you think of some sort of doubling
time, some sort of rough timescale on which
things get dramatically better, what time
scale would you guess at, Jaan?
Start now or starting at human level?
No, no, so once we get human level AI or whatever
point beyond there or a little bit before
there where things actually start taking off,
what is the sort of time scale?
Any explosion, as a nerdy physicist, has some
sort of time scale, right, on which it happens.
Are we talking about seconds, or years, or
millennia?
I'm thinking of years, but it is important
to act as if this timeline was shorter.
Yeah, I actually don't really trust my intuitions
here.
I have intuitions that we are thinking of
years, but I also think human level AI is
a mirage.
It is suddenly going to be better than human,
but whether that is going to be a full intelligence
explosion quickly, I don't know.
I think it partly depends on the architecture
that ends up delivering human level AI.
So, this kind of neuroscience inspired AI
that we seem to be building at the moment
that needs to be trained and have experience
in order for it to gain knowledge that may
be, you know, on the order of a few years
so possible even a decade.
Some numbers of years, but it could also be
much less.
But, I wouldn't be surprised if it was much
more.
Potentially days or shorter, especially if
it's AI researchers designing AI researchers
Every time there is an advance in AI, we dismiss
it as 'oh, well that's not really AI:' chess,
go, self-driving cars.
An AI, as you know, is the field of things
we haven't done yet.
That will continue when we actually reach
AGI.
There will be lots of controversy.
By the time the controversy settles down,
we will realize that it's been around for
a few years.
Yeah, so I think we will go beyond human level
capabilities in many different areas, but
not in all at the same time.
So, it will be an uneven process where some
areas will be far advanced very soon, already
to some extent and other might take much longer.
What Bart said.
But, I think if it reaches a threshold where
it's as smart as the smartest most inventive
human then, I mean, it really could be only
a matter of days before it's smarter than
the sum of humanity.
So, here we saw quite an interesting range
of answers.
And this, I find is a very interesting question
because for reasons that people here have
published a lot of interesting papers about
the time scale makes a huge difference.
Right, if it's something that happening on
the time scale of the industrial revolution,
for example, that's a lot longer than the
time scale on which society can adapt and
take measures to steer development, borrowing
your nice rocket metaphor, Jaan.
Whereas, if things happen much quicker than
society can respond, it's much harder to steer
and you kind of have to hope that you've built
in a good steering in advance.
So, for example in nuclear reactors, we nerdy
physicists like to stick graphite sticks in
a moderators to slow things down maybe prevent
it from going critical.
I'm curious if anyone of you feels that it
would be nice if this growth of intelligence
which you are generally excited about, with
some caveats, if any of you would like to
have it happen a bit slower so that it becomes
easier for society to keep shaping it the
way we want it.
And, if so, and here's a tough question, is there
anything we can do now or later on when it
gets closer that might make this intelligence
explosion or rapid growth of intelligence
simply proceed slower so we can have more
influence over it.
Does anyone want to take a swing at this?
It's not for the whole panel, but anyone who...
I'm reminded of the conversation we had with
Rich Sutton in Puerto Rico.
Like, we had a lot of disagreements, but definitely
could agree about paths slower being better
than faster.
Any thoughts on how one could make it a little
bit slower?
I mean, the strategy I suggested in my talk
was somewhat tongue and cheek.
But, it was also serious.
I think this conference is great and as technologists
we should do everything we can to keep the
technology safe and beneficial.
Certainly, as we do each specific application,
like self-driving cars, there's a whole host
of ethical issues to address.
But, I don't think we can solve the problem
just technologically.
Imagine that we've done our job perfectly
and we've created the most safe, beneficial
AI possible, but we've let the political system
become totalitarian and evil, either a evil
world government or even just a portion of
the globe that is that way, it's not going
to work out well.
And so, part of the struggle is in the area
of politics and social policy to have the
world reflect the values want to achieve because
we are talking about human AI.
Human AI is by definition at human levels
and therefore is human.
And so, the issue of how we make humans ethical
is the same issue as how we make AIs that
are human level ethical.
So, what i'm hearing you say is that before
we reach the point of getting close to human
level AI, a very good way to prepare for that
is for us humans in our human societies to
try and get our act together as much as possible
and have the world run with more reason than
it, perhaps, is today.
Is that fair?
That's exactly what I'm saying.
Nick? Also, I just want to clarify again that when
I asked about what you would do to slow things
down i'm not talking at all about slowing
down AI research now.
We're simply talking about if we get to the
point where we are getting very near human
level AI and think we might get some very
fast development, how could one slow that
part down?
So, one method would be to make faster progress
now, so you get to that point sooner when
hardware is less developed, you get less hardware
overhang.
However, the current speed of AI progress
is a fairly hard variable to change very much
because there are very big forces pushing
on it, so perhaps the higher elasticity option
is what I suggested in the talk to ensure
that whoever gets there first has enough of
a lead that they are able to slow down for
a few months, let us say, to go slow during
the transition.
So, I think one thing you can do, I mean this
is almost in the verification area, is to
make systems that provably will not recruit
additional hardware or resigned their hardware,
so that their resources remain fixed.
And i'm quite happy to sit there for several
years thinking hard about what the next step
would be to take.
But, it's trivial to copy software.
Software is self replicating and always has
been and I don't see how you can possibly
stop that.
I mean, I think it would be great if it went
slow, but it's hard to see how it does go
slow given the huge first mover advantages
and getting to superintelligence.
The only scenario that I see where it might
go slow is where there is only one potential
first mover that can then stop and think.
So, maybe that speaks to creating a society
where, you know, AI is restrictive and unified, but without
multiple movers.
Yeah, Demis, so your colleague Sean Legg mentioned
that the one thing that could help a lot here
is if there's things like this industry partnership
and a sense of trust and openness between
the leaders, so that if there is a point where
one wants to...
Yeah, I do worry about, you know, that sort
of scenario where, you know, I think, I've
got quite high belief in human ingenuity to
solve these problems given enough time. the
control problem and other issues.
They're very difficult, but I think we can
solve them.
The problem is will there, you know, the coordination
problem of making sure there is enough time
to slow down at the end and, you know, let
Stuart think about this for 5 years.
But, what about, he may do that, but what
about all the other teams that are reading
the papers and not going to do that while
you're thinking.
Yeah, this is what I worry about a lot.
It seems like that coordination problem is
quite difficult.
But, I think as the first step, may be coordinating
things like the Partnership on AI, you know,
the most capable teams together to agree,
at least agree on a set of protocols or safety
procedures, or things, you know, agree that,
maybe, you know, you should verify these systems
and that is going to take a few years and
you should think about that.
I think that would be a good thing.
I just want to caveat one thing about slowing
versus fast progresses, you know, it could
be that, imagine there was a moratorium on
AI research for 50 years, but hardware continued
to accelerate as it does now.
We could, you know, this is sort of what Nick's
point was is that there could be a massive
hardware overhang or something where an AI
actually many, many, many different approaches
to AI including seed AI, self-improving AI,
all these things could be possible.
And, you know, maybe one person in their garage
could do it.
And I think that would be a lot more difficult
to coordinate that kind of situation, whereas,
so, I think there is some argument to be made
where you want to make fast progress when
we are at the very hard point of the 's' curve.
Where actually, you know, you need quite a
large team, you have to be quite visible,
you know who the other people are, and, you
know, in a sense society can keep tabs on
who the major players are and what they're
up to.
Whereas, opposed to a scenario where in say
50 or a 100 years time when, you know, someone,
a kid in their garage could create a seed
AI or something like that.
Yeah, Bart, one last comment on this topic.
Yeah, I think that this process will be a
very irregular process and sometime we will
be far advanced and other times we will be
going quite slow.
Yeah, i'm sort of hoping that when society
sees something like fake video creation where
you create a video where you have somebody
say made up things and that society will actually
realize that there are these new capabilities
for the machines and we should start taking
the problem as a society more seriously before
we have full and general AI.
We'll use AI to detect that.
So, you mentioned the word 'worry' there, and
you Nick went a bit farther, you had the word
'doom' written on your slides three times.
No wonder there was one star on Amazon on that
rating and that it was even in red color.
I think it's just as important to talk about
existential hope and the fantastic upside
as downside and I want to do a lot of that
here.
So, let's just get some of those worries out
of the way now and then return to the positive
things.
I just want to go through quickly and give
each one of you a chance to just pick one thing
that you feel is a challenge that we should overcome
and then say something about what you feel
is the best thing we can do, right now, to
try to mitigate it.
Do you want to start Jaan?
To mitigate what?
Mention one thing that you're worried could
go wrong and tell us about something constructive
that we can do now that will reduce that risk.
I do think that AI arms races, I see like
a lot of, like, good.
I'm really heartened to see kind of great
contacts between OpenAI and DeepMind, but
I think this is just like a sort of toy model
of what the world at large might come up with
in terms of arms races.
And for myself I have been spending increasing
amount of time in Asia recently just to kind
of try to kind of pull in more people elsewhere,
what has been so far, just been, kind of like, an Anglo-
American discussion mostly.
So, like this is, I think, this is one thing that should be
done and i'm going to do it.
Well, as someone who is outside this field,
I think the challenge i'm really in touch
with is how hard it is to take the safety
concerns emotionally seriously.
And how hard it is for people in the field
to do that as well.
I can't tell you have many people outside
this room who purport to be experts think
the control problem is a total non-issue.
I mean, it's just flabbergasting to meet these
people and just therefore not worth thinking
about.
And one of the reasons I think is that in
one case there is this illusion that the time
horizon matters.
If you feel that this is 50 or a 100 years
away that is totally consoling, but there
is an implicit assumption there,
the assumption is that you know how long it
will take to build this safely. And that 50
or a 100 years is enough time.
The other issue is, I think, most people feel
like intelligence is an intrinsic good and
of course we want more of it and it's very easy
to be in touch with that assumption because
right now there is a cure for cancer, which
we have not discovered.
Right, how galling is that?
But for more intelligence, but for knowing
which experiments to run, or how to integrate
the data we already have in hand, we would
have a cure for cancer that was actionable
now unless there was some physical law of
the universe that prevented us from curing
cancer, which seems unlikely.
So, the pain of not having enough intelligence
is really excruciating when you focus on it,
but, and I think to your previous question
of doing this quickly becomes an intrinsic
good provided we have solved the alignment
problems and the political problems and the
chaos that would follow if we were just, if
we did it quickly without solving those problems.
So, I think, it's the thing that is alarming
is how ethereal these concerns are even to
those who have no rational argument against
them.
So, Sam it sounds to me like you're agreeing
very strongly with what Shane Legg that there
is, in some circles, still this strong taboo
that, you know, don't even consider the possibility
that we might get AGI because it's just absolutely
ridiculous.
And he was arguing that the sooner we can
get rid of this taboo the sooner people can
get to work and find all these really helpful
solutions and answers that we need.
So, suppose for a moment that I came up to
you and said to you "this idea of super human
intelligence just sounds absolutely ridiculous,
sounds completely nuts.
And by the way i've never seen your ted talk."
And we're in an elevator and you have only
30 seconds to persuade me to take this more
seriously, what would you say?
A lot of people who are here will have this
exact conversation with colleagues and others
in the future.
Well, there are very few assumptions you need
to make to take this seriously, intellectually.
Again, the emotional part is a separate piece.
But, if you assume that intelligence is just,
on some level, the product of information
processing in a physical system and there
are very few people who dispute that who are
scientifically literate at this point and
you assume that we will continue to improve
our information processing systems, unless
something terrible happens to us to prevent
that, and that seems like a very safe assumption,
then it is just a matter of time before we
instantiate something that is human level
and beyond in our computers.
And, again, the time horizon is only consoling
on the assumption that we know we have enough
time to solve the alignment problems and the
political problems.
The other thing that is humbling here that
Ray brought up at one point is that even if
we were handed a perfectly benign, well behaved
AI just from god, you know, we are given a
perfect oracle we are given a perfect inventor
of good technology, given our current political
and economic atmosphere that would produce
total chaos.
We just have not... we don't have the ethical
or political will to share the wealth, we
don't have the political integration to deal
with this thing being given to Silicon Valley
and not being given at the same moment to
China or Iran.
So, there is just, it's alarming that the
best case scenario currently, basically just
ripping out 80% of Nick's book because we've
solved all those problems, is still a terrifying
one. And so, clearly, that's a near term thing that we have to solve.
Thank you, Sam.
So, Demis do you want to tell us about one
thing that you feel is a challenge and say
something about what we should focus on now
to tackle it.
Yeah, I mean I think it's, you know I agree
with both the statements already said that,
so I think the coordination problem is one
thing where you know we want to avoid this
sort of harmful race to the finish where corner
cutting starts happening and things like safety
are easy things to, you know, will get cut
because obviously they don't necessarily contribute
to AI capability, in fact they may hold it
back a bit by making a safe AI.
So, I think that's going to be a big issue
on a global scale and that seems like it's
going to be a hard problem when we are talking
about national governments and things.
And I think also, you know, we haven't thought
enough about the whole governance scenario
of how do we want those AIs to be out in the
world?
How many of them?
Who will set their goals?
All these kinds of things, I think, need a
lot more thought.
You know, once we've already solved the technical
problems.
I think it's wonderful that you're not just
saying these things, but actually doing these
things since you played a leading role in
setting up the Partnership on AI here which
goes exactly in the direction of what you're
advocating here.
So, do you want to pass it off to Nick?
I'm sure there is nothing at all you're worried
about, right?
So, tell us about one concrete useful thing
you would to see us focus on.
So, I agree with that, I mean, so fun technical
work, bring in top technical talent to work
on these technical issues, build these collaborations,
build a community, build trust, work some
more on figuring out attractive solutions
to the governance solutions that could work,
but don't rush to implement the first idea
you have, but first trial them out a little
bit more.
I think a lot about consciousness, so I was
really struck by the 'sentience caution' on
the list of principles here that said "avoid
overly... avoid strong assumptions about the
distribution of consciousness in AIs," which
I take it entails avoid assuming that any
human level or super human level AGI is going
to be conscious.
For me, that raising the possibility of a
massive failure mode in the future, the possibility
that we create human or super human level
AGI and we've got a whole world populated
by super human level AGIs, none of whom is
conscious.
And that would be a world, could potentially
be a world of great intelligence, no consciousness
no subjective experience at all.
Now, I think many many people, with a wide
variety of views, take the view that basically
subjective experience or consciousness is
required in order to have any meaning or value
in your life at all.
So therefore, a world without consciousness
could not possibly a positive outcome.
maybe it wouldn't be a terribly negative outcome,
it would just be a 0 outcome, and among the
worst possible outcomes.
So, I worry about avoiding that outcome.
Now, as a matter of fact, i'm fairly optimistic
about the possibilities that AIs of various
kinds will be conscious.
But, in so far as this community is making
this assumption, I think it's important to
actually think about the question of 'in creating
AGIs, are we actually creating conscious beings?'
I mean, one thing we ought to at least consider doing
there is making, given that we don't understand
consciousness, we don't have a complete theory
of consciousness, maybe we can be most confident
about consciousness when it's similar to the
case that we know about the best, namely human
human consciousness...
So, therefore maybe there is an imperative
to create human-like AGI in order that we
can be maximally confident that there is going
to be consciousness.
So, what I hear you say is that when you have
a nightmare about the zombie apocalypse you're
not thinking of some terminator movie, but
you're thinking about this problem.
We create... we upload ourselves and do all
these wonderful things, but there's no one
home.
Is that fair to say?
I mean this is a different kind of existential
risk.
One kind of existential risk is there's no
humans, there's AIs, but some people might
say well that's OK they are our successors.
A much worse existential risk is there are
no conscious beings in our future.
So, i'll make a confession, so Shane Legg mentioned
that there has been this strong taboo about
talking about the possibility of intelligence
getting very advanced.
It's clearly also been a strong taboo for
a long time to mention the C-word.
In fact, before the conference when we got
all these responses on the first round of
the principles, guess which one was ranked
last?
It got huge amounts of minus 1 ratings, that
was the one with consciousness, so we changed
it to-- it was terribly stated --sentience
and stated it better and then it got stated
still better at lunch and it's still rated
last.
Even though I personally share your interests
in this a lot.
88% of people agreed to the sentient caution.
But, not 90%, so that one also fell off the
list here.
So, maybe that is another taboo you can personally
help us shatter so that people think about
that question more.
Ray, anything you're concerned about?
This isn't what I was going to say, but just
to respond... a converse concern is we create
AGIs, everybody assumes that of course it's
just a machine and therefore it's not conscious,
but actually it is suffering but we don't
look out for it's conscious subjective experience
because we are making the wrong assumption.
But, what I did want to say was, there are
three overlapping revolutions that people
talk about, GNR, genetics, bio-tech, nano-technology,
and robotics, which is AI.
And there are proposals, there was the Asilomar
conferences done here decades ago for bio-tech
that have worked fairly well.
There are similar proposals for nano-technology.
There is a difference with AI in that there
really isn't a full proof technical solution
to this.
You can have technical controls on, say, nano-technology.
One of the guidelines is it shouldn't be self-replicating.
That's not really realistic because it can't
scale to meaningful quantities without being
self-replicating, but you can imagine technical
protections.
If you have an AI that is more intelligent
than you and it's out for your destruction
and it's out for the world's destruction and
there is no other AI that is superior to it,
that's a bad situation.
So, that's the specter.
And partly this is amplified by our observation
of what we as humans, the most intelligent
species on the planet, have done to other
species.
If we look at how we treat animals, people,
you know, are very friendly, like their dogs
and pets, but if you look at factory farming
we're not very benign to species that are
less intelligent than us.
That engenders a lot of the concern we see
that if we there's a new type of entity that's
more intelligent than us it's going to treat
us like we've treated other species.
So, that's the concern.
I do think that what we are doing at this
conference is appropriate.
I wanted to mention that I think we should
publish these guidelines the way the Asilomar
guidelines in bio-tech were published decades
ago.
And then people can and people can, you can
have an opt-in, opt-out, but I think we should
actually say we had this conference and the
AI leadership/community has come up with these
guidelines and people can respond to them
and debate them and then maybe at the next
conference we'll revise them.
The Asilomar bio-tech guidelines have been
revised many times.
But, I would advocate that we actually take
a stand and put forth these guidelines and
then let the whole community at large debate
them.
And have them be, have them guide our research.
It's actually worked quite well in bio-tech.
Bart?
OK, yeah so let me give a little different
perspective.
So, one concern I have at the high level is
these machines become really smart or even
in certain areas, can humans still understand,
what they, decisions that they suggested,
that they make.
And I work in the field of automated reasoning
where we have significant advance last two
decades going from perhaps a few hundred variables
to perhaps millions of variables being solved
quite routinely.
And there was a sense in the community, well
we are getting answers from these reasoning
engines, mostly hardware/software verification
problems, but we cannot, humans can no longer
understand these answers.
In the last few years, people have actually
discovered that you can use the machine to
generate explanations for their answers that
are, again, human understandable.
So, I see sort of a glimmer of hope that maybe
even if we have much less intelligence we
may be able to understand solutions that machines
find for us and we could not find these solutions,
but they may be able to provide explanations
that are accessible to us.
So that's a little positive note.
Thank you.
Stuart?
So there are two things that keep me awake
at night, other than email.
So, one is the problem of misuse and bad actors.
To take an analogy, it’s as if we were building
nuclear weapons and then delivering them by
email to everybody on the planet, saying,
here’s a toy, do what you want.
How do we counter that? I have to say, I don’t
really have a good solution.
I think one of the things we have to do is
to make designs for safe AI very clear and
simple, and sort of make it unthinkable to
do anything other than that, right?
Just like it’s unthinkable to have a program
with an infinite loop that produces a spinning
pizza of death on your -- oh sorry.
Or it’s unthinkable to have a buffer overflow
that allows your software to be hacked into.
The other thing that keeps me awake is actually
the possibility that success would lead to
AI as a helicopter parent for the human race
that would sort of ossify and gradually enfeeble
us, so then there would be no point at which it
was obvious to us that this was happening.
And I think the mitigation, which you asked
for, to look on the bright side, is that in
some sense the meta-value of human evolvability,
the freedom to change the future, is something
that the AI needs to adopt, and in some sense
that would result eventually with the AI receding
into the background, and saying, OK, now I’ve
got you through your adolescence, now it’s
time for the human race to grow up, now that
we have the capabilities to eliminate scarcity,
to eliminate needless conflict and coordination
failures and all of those things that we suffer
from right now.
So I can imagine a distant future where, in fact,
AI is perhaps even less visible than it is
today.
Great, finally you, Elon, have as far as I
know never ever expressed any concerns about
AI, right - I’m just wondering if there
is any concerns, in particular any concerns
where you see there’s a very clear thing
we should be doing now that are going to help.
I’m trying to think of what is an actual
good future, what does that actually look
like, or least bad, or however you want to
characterize it.
Because to a point that was made earlier by
Sam and maybe made by others, we’re headed towards either
superintelligence or civilization ending.
Those are the two things that will happen
- intelligence will keep advancing, the only
thing that would stop it from advancing is
something that puts civilization into stasis
or destroys civilization.
So, we have to figure out, what is a world that we would like to be
in where there is this digital superintelligence?
I think, another point that is really important
to appreciate is that we are, all of us, already
are cyborgs.
So you have a machine extension of yourself
in the form of your phone and your computer
and all your applications.
You are already superhuman.
By far you have more power, more capability,
than the President of the United States had
30 years ago.
If you have an Internet link you have an article
of wisdom, you can communicate to millions
of people, you can communicate to the rest
of Earth instantly.
I mean, these are magical powers that didn’t exist,
not that long ago.
So everyone is already superhuman, and a cyborg.
The limitation is one of bandwidth.
So we’re bandwidth-constrained, particularly
on output.
Our input is much better but our output is
extremely slow.
If you want to be generous you could say maybe
it’s a few hundred bits per second, or a
kilobit or something like that output.
The way we output is like we have our little
meat sticks that we move very slowly and push
buttons, or tap a little screen.
And that’s extremely slow.
Compare that to a computer which can communicate
at the terabyte level.
These are very big orders of magnitude differences.
Our input is much better because of vision,
but even that could be enhanced significantly.
So I think the two things that are needed
for a future that we would look at and conclude
is good, most likely, is, we have to solve
that bandwidth constraint with a direct neural
interface.
I think a high bandwidth interface to the
cortex, so that we can have a digital tertiary
layer that’s more fully symbiotic with the
rest of us.
We’ve got the cortex and the limbic system,
which seem to work together pretty well - they’ve
got good bandwidth, whereas the bandwidth
to additional tertiary layer is weak.
So I think if we can solve that bandwidth
issue and then AI can be widely available.
The analogy to a nuclear bomb is not exactly
correct - it’s not as though it’s going
to explode and create a mushroom cloud, it’s
more like if there were just a few people
that had it they would be able to be essentially
dictators of Earth, or whoever acquired it
and if it was limited to a small number of
people and it was ultra-smart, they would
have dominion over Earth.
So I think it’s extremely important that
it be widespread and that we solve the bandwidth
issue.
And if we do those things, then it will be
tied to our consciousness, tied to our will,
tied to the sum of individual human will,
and everyone would have it so it would be
sort of still a relatively even playing field,
in fact, it would be probably more egalitarian
than today.
Great, thank you so much, that’s in fact
the perfect segue into the last question I
want to ask you before we open it up to everybody.
Something I have really missed in the discussion
about really advanced intelligence, beyond
human, is more thought about the upside.
We have so much talk about existential risk,
and not just in the academic context, but
just flip on your TV, check out Netflix, what
do you see there in these scientific visions
of the future?
It’s almost always dystopias, right?
For some reason fear gives more clicks than the
positive visions, but if I have a student
coming into my office at MIT asking for career
advice, the first thing I’m going to ask
her is, where will you want to be in 20 years?
And if she just says, well maybe I’ll get
cancer, maybe I’ll get run over by a bus,
that’s a terrible way to think about career
planning, right?
I want her to be on fire and say my vision
is I want to do this - and here are the things
that could go wrong, and then you can plan
out how to avoid those problems and get it
out - I would love to see more discussion
about the upsides, futures we’re really
excited about, so we can not just try to avoid
problems for the sake of avoiding problems,
but to get to something that we’re all really
on fire about.
So to start off I’ll just tell you something
that makes me really excited about advanced
artificial intelligence.
Everything I love about civilization is a
product of intelligence.
If we for some reason were to say, well, you know, I’m
scared about this technology thing, let’s
just press pause on it forever, there’s
no interesting question about if we’re going
to have human extinction, the question is
just ‘when?'
Is it going to be a supervolcano, is it going
to be the next dinosaur-killing-class asteroid
- the last one happened 60 million years ago,
so how long is it going to be?
Pretty horrible future to just sit and wonder
when we’re going to get taken out here without
the technology when we know that we totally
have the brainpower to solve all of these
problems if we proceed forward and develop
technology.
So that was just one thing that makes me very excited about moving forward rather than pressing 'Pause.'
I want to just ask the same question to all
of you guys in turn.
So tell us, just pretty briefly, about something
that you are really excited about.
Some future vision you imagine with very advanced artificial intelligence that you’re really excited about, that you would like to see.
Jaan-
So I want to be careful when I imagine
concrete fruits of AGI.
On a meta-level I think as a first approximation,
I think we should just maximize the amount
of fun and minimize the amount of suffering.
I think Eliezer [Yudkowsky] has written a sequence called
“Fun Theory”, where he points out that
people have been horrible imagining, are very
unimaginative imagining paradises of various
sorts, just like really boring places, actually,
when you think about them.
I think Eliezer has this sketch where he says,
“It was hard to spend like one weekend with my relatives.
Imagine spending eternity with your dead relatives.”
So I think we should be concerned about side
effects and try to capture dynamics of improvement,
and basically go from there - make sure that
we’re going to adjust the trajectory as
we get smarter and more grown together.
Great, thank you, Jaan.
Sam, what do you get excited about?
Well, strangely, what excites me really just
abuts the parts that scare me the most.
I think what is nice about this conversation,
in particular about the alignment problem,
is that it’s forcing us to realize that
there are better and worse answers to questions
of human value.
And as someone said, perhaps at this last
meeting in Puerto Rico, we really have to
do philosophy on a deadline, and we have to
admit to ourselves that there are better and
worse answers and we have to converge on the
better ones.
And what would excite me about actually the
birth of superintelligent AI - one of the things,
apart from solving obvious problems
like curing disease and energy issues and all the rest,
perhaps differs a little bit
with what Stuart said.
I’m not so worried about idiocracy or all
of us just losing our way as apes and living
unproductive lives in dialogue with these
oracles.
I think actually, I would want a truly value-aligned
superintelligence to incrementally show us,
not merely conserve what we want, but show
us what we should want to keep improving our
values so that we can navigate in the space
of all possible experiences and converge on
better and better ones.
Thank you, Sam, and what about you, Demis?
So obviously this is why I spend my whole
career working on this, is that, I think if
we do this right, it’s going to be the greatest
thing ever to happen to humanity,
and in some ways I think unlock our full potential.
I mean, I’ve talked to a lot about, in all
my talks about using it as a tool to help
us make science and medical breakthroughs
faster.
And so I think that’s an obvious one.
But taking that longer-term, one reason I
got so into AI is that, like probably many
of you in this room, I’m interested in the
biggest questions of why we’re here, understanding
our minds, what is consciousness, what’s
the nature of the universe, what’s our purpose
- and if we’re going to try and really grapple
with any of those questions I think we’re
going to need something like AI, perhaps with
ourselves enhanced as well.
And I think in that future world we’ll have
a chance to actually find out about some of
these really deep questions in the same way
we’re finding out with AlphaGo just about Go,
but what if we could do that with all
of science and physics and the biggest questions
in the universe.
And I think that’s going to be the most
exhilarating journey of all, to find that out.
To just carry out on a few other things that
people commented on is in terms of
us as the most intelligent beings on the planet right
now, and treating animals badly and these
sorts of things, I think if you think about
it though - let’s take tigers or something in India.
They have huge ranges and those people are
very poor and they’re resource-poor, but
if they had abundant resources I don’t think
they’re intentionally trying to kill off
these tigers - in some cases they are - but
often it’s just because they need the land
for their cattle, and the tiger needs whatever
number of kilometers squared to live, one tiger.
And it’s just difficult with the number
of people that are there.
So I think if we solve the kind of abundance
and scarcity problem, then I think that opens up
a lot of conflicts both between humans
as well as to do with resource scarcity, at the heart of it.
So I see, if we can solve a lot of these problems
I can see a much better future.
So Nick, you pointed out, the upside part
of your book was a little shorter,
so now you have a chance to add something positive.
What are you excited about?
There are really two sides to that.
So one is getting rid of a lot of the negatives,
like the compassionate use to cure diseases
and all other kinds of horrible miseries that
exist on the planet today.
So that is a large chunk of the potential.
But then beyond that, if one really wants
to see realistically what the positive things
are that could be developed, I think one has
to think outside the constraints of our current
human biological nature.
That it’s unrealistic to imagine a trajectory
stretching hundreds of thousands of years
into the future, we have superintelligence,
we have material abundance, and yet we are
still these bipedal organisms with three pounds
of gray tissue matter, with a fixed set of
emotional sensitivities and the hedonic set
point that is kind of OK-ish for most people
but if you get - if something really good
happens it lasts for a time and then you’re
back to the baseline.
I think all of these basic parameters that
sort of define the human game today, I think
become up for grabs in this future.
And it opens up this much vaster space of
post-human modes of beings, some of which
I think could be wonderful, literally beyond
our ability to imagine, in terms of the mental states,
the types of activities, the understanding,
the ways of relating.
So I don’t think we need a detailed blueprint
for utopia now, what we need is to get ourselves
in a position later on where we can have the
ability to use this to realize the values
that come into view once we’ve taken steps
forward.
Thank you, Nick.
What about you, David?
I’m excited about the possibilities for
AI making us humans smarter.
I mean some of it is selfish - I turned 50
last year, my brain is gradually becoming
slower and older and dumber, but I’m not
sure that I am, and that’s partly because
of all of the augmented intelligence technology
we’re using.
Smartphones, and the Internet, and so on,
they’re giving me all kinds of capacities,
extended capacities that I didn’t have before.
And I’m really looking forward to AI helping
with that.
In ten years or so once everyone is wearing
augmented reality glasses with deep learning
built into it, then I’m really going to
need that around 60.
And if you guys really get on the case and
by the time I’m 70 or so we've got
real genuine AI or AI modules out there which can
somehow come to be integrated with my
brain processes or maybe eventually we get to upload
our entire brains onto AI, then there's a
way potentially to get smarter, more intelligent
forever.
And this is not just selfish, although I can't
say that doesn't motivate me,
but Demis talked about the AI scientists; I also like to think about the AI philosopher.
The problems of philosophy are really hard
and many people have speculated that we humans
are just too dumb to solve some of them.
But once we've actually got AIs on the scene,
maybe AI-enhanced humans, then maybe we're
going to be able to cross those thresholds
where the AI-enhanced humans or maybe just
the AGIs end up solving some of those hard
problems of philosophy for once and for all.
Great, Ray, you have been a true pioneer in
articulating positive visions of the future
in your writing.
So if you picked the one that you're most
excited about now, what would that be?
So imagine going back 10,000 years and asking
the quintessential caveman and woman,
Gee, what is a beneficial future?
What would you like to see?
And they would say, well I would like this
fire to stop going out and I would like a
bigger boulder to prevent the animals from
getting in the cave.
Anything else?
Well no I think that would be pretty perfect.
Well don't you want a better website and apps
and search engines?
Imagine going back 2 million years ago and
talking to primates - imagine if you could do that,
and saying, isn't it great that
frontal cortex is coming and we're going to
have additional neocortex and and a hierarchy
and they say, well what's the point of that?
And you say, well you'll have music and humor,
and their answer would be, what's music?
What's humor?
So they couldn't imagine concepts that they
couldn't imagine, and by analogy I think we
will have new phenomena that are as profound
as music and humor, you could call it more
profound music and we'll be funnier, but I
think it'll be as profound as these great
leaps that evolution has brought us, because
we will become profoundly smarter and if music
and humor are up here and we go to even higher
levels of the neocortex, we're going to have
more profound ways of expressing ourselves
and once we have that we would not want to
go back.
What about you, Bart?
Well, I pretty much agree that we can't really
predict much in advance, what we would like to have.
For myself personally I see the developments
in mathematics and science and discovery,
and computers are just the hybrids of human computers there is quite incredible and makes the field -
makes what we do much more exciting.
So I think that will be in the near future
the first thing.
Great, and what about you, Stuart?
Well, so like Jeffrey Sachs - I think that
for many of us, and probably like the cavemen
- that for many of us life is pretty amazing,
and for many more of us it isn't.
And I think the best thing that AI can do,
the big upside, is actually to fix the latter problem.
I mean I love Nick's feeling that there are
higher states of being that are so far above
our current 'pretty good', that that balances
out all the 'pretty bad' that a lot of people are suffering.
But I really think the emphasis should be
on the 'pretty bad' and fixing it, and eliminating
- so Demis was reading my notes apparently,
from across the room - but eliminating the
scarcity basically eliminates the need for
people to act in a zero-sum fashion where
they can only get by, by making it less possible
for someone else to get by, and I think that's
the source of a lot of the nastiness that
Jeffrey mentioned earlier.
So I think that would be my main upside, and
not having to read so much email,
that would be the second one.
And for you, Elon, you've never articulated
any visionary ideas about the future as far as I know, either.
What about now?
I think I just - I have thought about this
a lot, and I think it just really comes down
to two things, and it's solving the machine-brain bandwidth constraint and democratization of AI.
I think if we have those two things, the future
will be good.
There was a great quote by Lord Acton which
is that
'freedom consists of the distribution of power and despotism in its concentration.'
And I think as long as we have - as long as
AI powers, like anyone can get it if they want it,
and we've got something faster than
meat sticks to communicate with,
then I think the future will be good.
Fantastic, so let's get - I know your caffeine
levels are dropping dangerously low, and we
also have another panel after this, which
is going to be really exciting to listen to,
so let's do a just a few quick questions.
Make sure that they are actually questions,
and say your name and also say,
pick one person on the panel and address it just to
them, OK?
Yoshua?
Yoshua Bengio, Montreal.
And it's for Jaan - I found your presentation
very inspiring, and one question I have is
related to the question of eliciting preferences
and values from people.
Do you think this line of investigation could
lead to better democracy, better society,
more direct democracy, and you know, what
do you think about this direction to deal
with the issue of misuse and things like that.
Yes, absolutely.
There could be one code name for this, even,
could be like 'Democracy 2.0' or 'U.N. 2.0'
or something like that.
So, and as I mentioned in my presentation, just
a lot of people today basically want to make the world better,
but it's kind of hard to
distinguish them from people who say they
want to make the world better.
So if there was actually kind of like a very
easy measuring, like a metric that basically
would work as a Schelling point, focal point,
then I think that would be super helpful.
And yeah, like democracy was invented like
hundreds of years ago so, and clearly we have
advanced as a civilization and we have better
knowledge about how to aggregate preferences.
And Nicolas Berggruen, over there.
Thank you, Max. Nicolas Berggruen, so I have a very almost naive question.
This is a very well-meaning group in terms
of, let's say, intentions, but
who sort of, looking at who else is doing, potentially, AGI, it could be well beyond this group,
it could be in China, it could be any place.
And what happens because we've talked about
how powerful AGI is, and if Elon is correct,
if it is distributed fairly, fine.
But if it isn't, is there a way to monitor
today or in a year or in 10 years, because
once it's out it'll be fast.
Who is monitoring it, who has a tab on it?
Because this is self-selected, but beyond...
Elon or Demis does either one of you want
to take a swing at this?
Well I think this sort of relates to my point
I said earlier about trying to build AI at
the hard part of the 'S' curve, so, which
I think is where we sort of are at the moment,
as far as we can tell, because, you know, it's not easy
to make this kind of progress, so you need
quite a lot of people who are quite smart
and that community is pretty small, still,
even though it's getting rapidly bigger at
places like NIPS.
And so most people know each other, so this
is pretty representative of everyone in the
West, at least, obviously it's harder to know
what's happening in China or in Russia, maybe.
But, you know, I think that you need quite
a large footprint of resources,
people and very smart people and lots of computers
and so on.
So I think that narrows down the scope of
the number of groups who can do that,
and it also means that they're more visible.
So, you know, I think certainly in the West
I think most people around here,
someone in this room will have contact with somebody who's in those groups who are capable of
making meaningful progress towards AGI.
It's harder to know in the East and further
apart, but we should try and make links to
those Chinese National Academy of Sciences,
and so on, to find out more.
But you know that may change in the future,
I think that's the current state of it.
Great, it's - the bad news is it's getting
later in the day and we only have time for one more question.
The good news is there's a coffee break right
after this so you can ask all of your questions
if you swarm the panel.
And the last question goes to you, Erik.
Do you want to stand up?
Erik Brynjolfsson, MIT.
I'm going to pick up on the thing that Elon
said at the end about democratizing the outcome
and tie it back to the panel yesterday where
Reid Hoffman talked about people caring
a lot about not just absolute income but relative
income, and I wanted to get the panelists'
reactions to the thoughts about whether or
not AI had tendencies towards winner-take-all
effects, that there's a tendency for concentration,
that whoever's ahead can pull further ahead,
or whether there's potential for more widespread
democratic access to it,
and what kinds of mechanisms we can put in place if we want to have the widely shared prosperity that Elon suggested?
Elon, do you want to take that?
Yeah, well, I mean I have to say that when
something is a danger to the public, then
there needs to be some - I hate to say government agency, like regulators -
I'm not the biggest fan of regulators, 'cause they're a bit of a buzzkill.
But the fact is we've got regulators in the
aircraft industry, car industry, I deal with them all the time,
with drugs, food - and
anything that's sort of a public risk.
And I think this has to fall into the category
of a public risk.
So I think that the right thing to do, and
I think it will happen, the question is whether
the government reaction speed matches the
advancement speed of AI.
Governments react slowly - or governments
move slowly and they tend to be reactive,
as opposed to proactive.
But you can look at these other industries
and say, does anybody really want the FAA to go away?
and it's like people could just
be a free for all for aircraft - like, probably not.
You know, there's a reason it's there or just
people could just do any kind of drugs and
maybe they work, maybe the don't.
You know, we have that in supplements, kind
of ridiculous.
But I think on balance FDA is good, so I think
we probably need some kind of regulatory authority
and I think it's, like a rebuttal to that
is, well people will just move to Costa Rica or something.
That's not true.
OK, we don't see Boeing moving to Costa Rica
or to Venezuela or wherever it's like free and loose
To Demis' point, the AI is overwhelmingly
likely to be developed where there is a concentration
of AI research talent.
And that happens to be in a few places in
the world.
It's Silicon Valley, London, at Boston,
if you sort of figure out a few other places,
but it's really just a few places that really
regulators could reasonably access.
And I want to be clear, it's not because I
love regulators, OK?
They're a pain in the neck but they're necessary
to preserve the public at times.
Alright, on that note, let's thank the panel
for a fascinating discussion.