Predicting AI - Shanghai

thanks a lot for inviting me to the
conference and thanks a lot especially
for that introduction at the future of
humanity Institute as you said we try
and look at the big pictures the big
impacts in the world and hope to
convince you by the end of this talk
that one of the very biggest is in AI
and that despite or maybe even because
of the great uncertainties out there now
you all know the concept of return on
investment so I went initially for a
return on investment of what's the
biggest impact that I could do in the
world how could I do the most good or
the most change to illustrate let's
start small this is kaposi sarcoma it's
a side effect of HIV it's tumors it's
rather unpleasant it can be treated and
if you're treated it makes your quality
of life better of course how much better
well we have a unit called the kweli the
quality adjusted life here which is
basically how many years of life you get
more and how good they are if you treat
kaposi's sarcoma you get about two
thirds of a kweli per thousand pounds
now this is a good deal for the people
or suffering from this and definitely i
would pay this but there are more
efficient ways of doing things if you do
antiretroviral therapy your quail ease
jump but as probably your mother told
you it's better to prevent than to cure
so if we look at prevention of
transmission during pregnancy we get
much higher scale in fact our scales
getting a bit cumbersome let's adjust
because there are other even better ways
of dealing with the spread of HIV and
one of the reasons that it's so
important to focus on queries is that if
you tried to prevent HIV by treating
kaposi sarcoma instead of by through
condom distributions you are basically
saying
that you are willing to kill 19
twentieths more people than otherwise I
sorry 19 times more people than
otherwise the cost of inefficiency is in
human lives but this is all about HIV
why are we focused on that are there
more effective interventions indeed
there are malaria treated through
distribution of bed nets has currently
one of the highest quail ease you can
get into the 50 quail ease per thousand
pound spent these numbers come from the
charity assessors giving what we can
give well they're into effective
altruism if you have any interest in the
field I recommend looking them up
they're really impressive in their work
okay but what if I say this isn't enough
I want to go higher can we get hundreds
thousands millions of quail eyes yes
this guy is France Harbor he has to his
detriment he invented chemical warfare
to his credit he invented the
haber-bosch process along with a guy
called Bosch this allows the fixing of
nitrogen into ammonia now that might not
sound very exciting but this is how you
create fertilizers and this is how you
grow crops in the world today it's
estimated that about half of the
nitrogen in your bodies is nitrogen that
was initially fixed through the harbor
Bosch process and then entered the food
chain so estimates vary it's really hard
to tell these things but it's safe to
say that there's at least a billion
people alive today who would not be
alive if this process has not been
discovered what else the Green
Revolution had a similar impact it's the
reason that India and Mexico actually
have functioning Agriculture's at the
moment so again numbers very speculative
quarter of a billion people saved and
let's just add France Jenner the
smallpox vaccine least half a billion
people saved through that so this seems
a path to high impact go for the really
great innovations even if you have a
small chance of getting this in
expectation you
you should make a real big change can we
get even higher than this well there
seems only one way of getting higher
take a big disaster that would kill lots
of people and stop it from happening so
recently I analyzed 12 so-called global
catastrophic risks from climate change
to bad governance and a lot of other
disasters in the middle we established
that so far as we can tell four of them
are truly existential risks that have
the potential at least to put the very
survival of the human race in doubt they
were nuclear war global pandemics and
certain innovations in biotechnology and
artificial intelligence now AI has some
unique features amongst all the other
risks it has the most extreme
uncertainties potentially the most
extreme risks potentially also the most
extreme benefits and it has quite a lot
of short-term impacts as well it was at
this point that I knew that my search
for higher return on investment had
ended and this is where I was going to
spend my life working let's look at more
this from let's look at the short-term
impacts for a start now you're aware no
doubt so there's a lot of hype
concerning a I you probably receive a
lot of it every day there's also quite a
lot of anti hype and I won't go into the
detail of what's cool what's not and
what specific thing to look at there are
marketers out there who are doing that
much better than me but there are few
general trends to look at the first one
as illustrated by this kurtzweil cartoon
is that it's becoming harder and harder
to figure out something that humans can
do that machines cannot do as you see
there he's writing down only humans can
and as fast as he writes it down they
fall on the floor used to think only
humans could drive cars only humans
could play chess how many humans could
play go to a high level all these things
are fallin or falling simultaneously
with that there is a very odd pattern
that there is no trace of general
intelligence
amongst these algorithms so this is
Watson a AI that triumphed on jeopardy
jeopardy is an American game show with
puns and plays on words and other things
of that nature so it's thought that it
was very hard for a machine to win it
and yet Watson did however Watson is
definitely not a general intelligence
here it's hazarding Toronto as a
solution to a question whose category
was US cities so Watson doesn't often
make a mistake but when it does make
mistakes it makes stupid mistakes and
this does seem to be the trend we have a
lot of progress in narrow AI pretty much
anything that we can focus our attention
on we are achieving but no trace of a
general intelligence and this is very
interesting if you compare what people
were believing say 20-30 years ago where
they would say that general intelligence
is necessary for solving things like
triumphing on Jeopardy or driving cars
in noisy environments it turns out it
isn't what other general overall meta
things can we say about short-term AI
well let's have a look at AI and
unemployment people talk about that's
quite a bit and people in my Institute
did an analysis as to which jobs are
more likely to be automated and how fast
and which jobs are least likely to be
automated it seems that there's two
broad categories a big chunk of jobs
almost half that are very likely to be
automated and a slightly smaller chunk
about a third that are very unlikely and
very few jobs are actually in the middle
for illustrative purposes here are some
of the jobs all the way from insurance
underwriters and waiters whose jobs are
very likely to be automated as far as
they can tell all the way down to
physicists and recreational therapists
who are very unlikely to be automated
now you have to bear in mind that when
we're talking about automating jobs
we're not talking about a one-for-one
replacement the old-fashioned secretary
was almost completely displaced not by a
cyborg secretary but by the word
processor
now a lot of people look at this
unemployment as a negative and it
undoubtedly is for the people who lose
their job especially as it's not clear
that they'll be able to find new jobs at
the same speed as we did in earlier
technological revolutions however also
obviously if a machine is doing a job
for someone and people are using a
machine that means the job is getting
done either better or cheaper which
means that as this automation extends as
a whole the economy is growing the human
race is wealthier and more powerful and
this tells you how that is happening now
distribution issues are complete
separate thing so this is one aspect of
what short-term AI will do doing what we
already do better but there's other
things things like YouTube Facebook and
immersive games what these are offering
are things that basically no one could
offer or almost no one could offer or do
before uploading videos see seeking the
world's knowledge those kind of things
social networking facebook or facebook
like networks this is something that
just did not exist and the immersive
worlds of games are another added value
so this is something that you could not
look at an old economy and predict
there's a few other examples things like
robots extracting people from
emergencies potentials for atomically
precise manufacturing artificial limbs
which are not yet as good as real limbs
but they're getting there I'm pretty
confident that wind within 20 years we
will have ethical questions such as is
it ethical for someone to cut off their
limb to get an artificial one which
would be better so these are other
trends to look at but this is the
short-term AI which is to my mind less
exciting so what is what's the extreme
layer what since there's a lot of
uncertainties here what's the potential
for extremely eyes is there any limit on
what an AI could do well one way of
looking at this is to think on AI is a
cognitive machine what are cognate
tasks so maybe we can put a limit on
that and upper bound well one cognitive
task is cognitive task generally involve
things like understanding prediction
planning and organizing unfortunately
this pretty much means everything
manufacturing what's that you find raw
resources you extract them you have a
manufacturing process where they're
assembled and distributed most of this
is cognitively limited we pay we pay
factory designers a lot of money we pay
managers money we don't pay people who
operate the machines a lot of money
because their skills are not valued the
not value bull for that process the what
this means is that the limits to
manufacturing today are not in the raw
resources the earth has way more
resources of most types than we're
currently using but in the cognitive
organizing of the factories of the
extraction process and if there were a
eyes who are really of extreme
capability in manufacturing they could
build robots that could build factories
etc so we might have an explosion of
manufacturing and I'm not even getting
into the speculative atomically precise
and other technological innovations you
might have what's another cognitive task
warfare warfare to simplify is weapon
design plus managing your army plus
strategy all these are cognitive tasks
space exploration is a smaller example
of a cognitive task you have petrol and
raw resources under the earth and then
almost purely by cognitive decisions
made by highly paid politicians or
engineers this is assembled into a
rocket and we have satellites and probes
on the most distant planets the other
reason to bring up space exploration is
because this means that the resources of
a solar system might become available to
the AI for the purposes of manufacturing
finally let's not forget people skills
which are also cognitive tasks
seduction moving large amounts of people
getting large amount of people to buy
into your product all of these are
cognitive tasks so from this perspective
I've completely failed to put any
limitations on what an AI could do to
limitations are the laws of physics and
maybe the resources of the solar system
the upper bound is huge okay let's try
and build up from below what kind of
thing could we imagine doing with say a
human level AI if we had nai that had
human-level capabilities what could we
imagine doing it well we could imagine
copying it a few times or maybe training
each of the copies and then we could
create say a supercommittee take the AI
Edison the AI Einstein the AI Soros AI
Clinton AI opera Confucius Goebbels
bernie madoff and Steve Jobs all of
these are humans who are incredibly
cognitively capable in their domains if
you take their AI equivalents Network
them together give them vast amount of
data run them at a thousand times human
speed so that they basically have a week
to think of every answer that they give
you and you have an entity of
potentially vast power most likely able
to use the internet in the human race as
tools to accomplish its goals what other
ways might we think that a is might
become powerful well I've already
mentioned the possibility for copying
so here's an idea the one-minute
corporation if you want to create a
genuine corporation today you need a
management structure a complicated thing
with a lot of management theory built
into it and experience and you need to
hire people test them integrate them
into teams and so on if you had an AI if
it seems it might be much simpler 123456
you have all the employees you need you
might need to train them well you only
need to train them once and you can copy
them as much as you want so maybe spend
a week training your your a is into
thousands of different corporate roles
and then just copy them you get a
perfectly unified corporation that does
not require any of the overhead of a
normal cooperation so currently we have
very roughly about two billion computers
in the world when we do get a eyes and
however in many computers we have there
how many do you think we could run it
seems that the amounts that we could run
the amounts of corporations we could
create and destroy is vast and remember
that manufacturing is a cognitive task
so if we plug in the ai's capabilities
into manufacturing and get more
computers well how many billions of a is
could we create all potentially
integrated in terms of motivation
there's a last brief way that a eyes
could become extremely powerful it's
sort of self-improvement self brain
surgery now I don't recommend that you
do this yourself however nai is unlike
our messy kludge of a brain an AI is
likely to be much clearer much easier to
see much easier to analyze and the AI
can also copy itself an experiment on
itself again and again and again until
it finds the right algorithms if you see
hardware design and hardware
manufacturers cognitive tasks which they
mainly
our then this is another way that the AI
could boost itself through its own
research so in summary there are a few
scenarios that suggests that a eyes
could become extremely powerful and we
have no upper limit no sensible upper
limit to how powerful they could become
so what's the risk well the risk is that
the AI does exactly what we tell it to
do and not what we want it to do ideally
we would want something like flags
inside the programming which ensure that
the AI does what we want and doesn't go
crazy unfortunately if you try coding
these today you get the error undefined
terms and because these terms are
extremely complicated and very hard to
ground what we mean by them now if
you've done any philosophy you'll know
that chair is also extremely hard to
ground but it is somewhat more important
to ground the concepts of not killing
all humans than it is to be absolutely
sure what is and what isn't a chair and
so there's several ways that this could
go wrong like suppose we motivate the AI
to prevent human suffering now human is
a very difficult concept to code but
imagine that we've managed code it
suffering also extremely subtle
extremely difficult but again imagine
we've managed to code it prevent is an
easy concept but when we've done this we
have just motivated the AI to kill
everyone this is the single best way to
prevent human suffering and it's not so
much this example that I want you to
take home but the fact that this kind of
motivation is hiding in the initial goal
that we gave because the initial goal is
what we said it wasn't what we meant and
it seems that kill all humans is a sub
goal of our most very many goals we can
give them for instance filter out all
spam what's the best way of filtering
out
spam well make sure that no email ever
gets sent again so maybe people maybe
can be a bit clever and people have
suggested things like this safe very
complicated concept happy also very
company company concept but assume we've
got them now I think you can see where
this is going and the AI will fight you
if you try and prevent it from doing
this because from its perspective any
other outcome than this is people less
safe or less happy you might shouted it
but that's not what we meant and the AI
will respond it's having a conversation
while it's dragging you off in this
picture yes I know what you meant I'm
very intelligent however that's what you
programmed in me and nowhere did you
program care what we actually meant now
there's some people have suggested sort
of less immediately deadly ideas oh and
by the way a lot of the features of
these ideas is that they're perfectly
safe for a week ai and become very
dangerous as the eye becomes very
powerful as long as the AI is weak these
are very safe goals to give it but
anyway this is another suggestion that
people have have VA i did use human
preferences from observations not
intrinsically stupid there might be ways
of doing it but if we do it naively i
fear that the future of the human race
and maybe of the nearby universe will
start to look like this we it is not
clear from looking at us what we truly
value and what we would want now that
has been somewhat this is for the risks
thesis of extreme AI you could say
uncontrolled one highly dangerous is
generic behavior for high intelligent a
is however this is a domain with extreme
uncertainties and i would be remiss to
not add all the qualifying words on that
you can read more about it in smarter
than our
my non-technical introduction to the
subject and my bosses super intelligence
however even with all the qualifiers
added this is still not necessarily
reassuring and what are the upsides well
I've been looking at the disastrous
scenario here the disastrous scenario is
an AI high power that does something bad
and it seems that if you look at the
power of a eyes and the likely outcome
you get a very strong v-shaped week a
eyes will bring outcomes that are very
similar to our world today it may be
shifted in a slightly better or worse
direction strong and powerful AI is it
tends to bifurcate into disastrous if we
do not get it to do what we really want
and absolutely fantastic if we do and I
mean fantastic pretty much anything you
want you want an end to absolute poverty
BAM that's a manufacturing problem
relative poverty is another thing
entirely you want an end to ill health
well this is a manufacturing
experimentation cognitive problem this
is the kind of things that those extreme
abilities I've been talking about if
they were redirected in directions we
want that it could solve want to solve
death well death is just the extreme end
of the ill health problem depression
boredom again these are cognitive
problems that we have some success
basically if you think that some humans
have some success at dealing with these
problems then you should expect the
potential extreme ai's to be much more
much better than us at it's so pretty
much lack of anything that we really
like today so I've been mentioning
uncertainty a lot and why is it why is
it that predictions about AI are bad
well pretty much first of all because
most predictions are bad here are some
examples and I'm putting them up not
just so that you can laugh at them but
because the people here were real
experts in their domain when they made
it these are not naive idiots these are
people who knew what they were talking
about
and still got it spectacularly wrong so
and now we are in the situation of being
in 1929 listening to people like Irving
Fisher who knows more about stocks and
pretty much anybody else that we'd
likely to meet and he's saying they're
on a high plateau and why would we
believe anything else and one of the
reasons that these predictions are often
wrong is that we often underestimate
small transformations and I like to
illustrate this often by imagining what
if lie detectors worked what if we
actually built a functioning lie
detector don't worry how we do it some
scanning super whatever it doesn't
matter but if they did work well how
would the court system work it seems it
would be a lot fairer to replace it by a
simple question and answer session and
the court cases would be over in about
30 seconds at least for criminal law so
if you think of the centuries of legal
traditions in different parts of the
world that could be overturned by an
invention as a lie detector what about
tomorrow's job interview could it be the
same process not you do not need to get
qualifications that show that you have
the job but just state that you have
these that you are willing to do that
and especially if you compare today's
job interviews which are extremely iffy
in terms of establishing quality
tomorrow's relationships so romance jobs
legal systems they can be changed
completely depending on a technological
innovation that may or may not happen
and if you don't keep track of this just
one example you'll get your predictions
about the future wrong let's get a bit
larger scale would it be enough could
you do politics this way just run the
politicians through a lie-detector what
about international relationships the
important thing about lie detector based
international relationship is that you
can get trust through politician
to otherwise hate each other and despise
each other so again one small
technological innovation may or may not
happen I have my opinions on whether it
will or not but if it does pretty much
anything you know about the world will
be transformed AI predictions are harder
than this just to illustrate this is the
dartmouth conference that's pretty much
coined the term AI they were predicting
we would have AI in 1956 nine years
later Dreyfus in an otherwise excellent
paper was saying that now we've pretty
much reached the limit of what
automation and computation can do I
think it's safe to say that neither of
these predictions have exactly been
shown to be accurate here's another
prediction AI will be developed in 15 to
25 years time you may want guess when
this prediction was made it was made in
2012 also 2011 2010 and quite a lot of
times back from 1960s these are the ones
I could find in fact if you plot AI
predictions by when they were made and
when they predict the arrival here's
cheering's original prediction by the
way and this is the AI winter when
everyone was pretending that they
weren't working in AI anymore and the
thing to notice is there does not seem
to be a strong pattern there does not
seem the difference between those bars
is 10 years time even if we zoom in
they're spread out pretty much all over
the place there's no strong difference
between experts and non-experts except
maybe in this area where there's a
little bit of an accumulation but that's
in the 15 to 25 year period there's a
bump in those kinds of predictions and
there's a bump in those kind of
predictions even amongst the failed
predictions it seems that there's a
psychological reason that people predict
in that area so this is not so this is
strong evidence that people predicting
timelines of a eyes are not tapping into
genuine expertise and are just getting
it wrong why is that well we actually
know what makes good expert predictions
and it's not really the quality of
expert it's the difficulty of the task
that they're doing pretty much tasks
that have the features on the left are
tasks at which predictors and experts do
well tasks without the features on the
right or Tasker which predictors do
poorly and it's which is why say an
anesthesiologist is a true expert and
most doctors who interpret mammograms
are not even though they probably have
equivalent amounts of training in their
fields these are not equal in importance
as far as I can tell the most important
or expert disagreements whether the
problem is decomposed or not and above
all else whether you get feedback
preferably immediate feedback that's why
anesthesiologist is an expert because
they get feedback immediately the
patient is either not unconscious or
dead in both cases you know that
something's gone wrong now we seem stuck
in this for most AI predictions so we
should expect very poor performance
which is what we get now for many narrow
AI domains the short-term things things
move more to the left things are more
accurate so look out for features that
should give you good predictions one of
the features is that predicting what we
call grind is easy and producing what we
call insight is hard if you want grind a
grind question is how long will it take
to make the next Hollywood blockbuster
well a Hollywood blockbuster is a huge
thing with lots of artistic people doing
artistic things and then there's
marketers and there's money people it's
all horribly complicated except that we
know that basically you get all these
people together they work a certain
amount you crank through and you'll get
a movie at the end we've seemed to
happen before it'll happen again we know
how long it should take we know roughly
how long it'll go over when it does go
over we have good distributions because
these Hollywood movies happen by certain
amount of work in contrast can we
predict when someone will solve the
Riemann hypothesis
this is a big open question in
mathematics this is much harder to
predict mainly because it requires
insight it requires predicting insight
it's not enough that you get 10 top
mathematicians make them work for five
years each and you'll have a solution we
don't know and a lot of predictions in
AI are actually trying to predict
insights but pretend that they're
predicting grind this is my favorite
aaronus a i type prediction I call it
Moore's law hence AI they go by the year
blah blah blah computers will have some
level of something floating operations
per second amount of data passing
through whatever and this is a level
comparable with the human brain then
we'll have AI now Moore's law is very
much grind certain amount of work does a
certain amount of speed increase we seem
to establish that however what they're
trying to do is look at that bit and not
do the hard insight bit of saying given
this Hardware how we going to get AI we
do not have an AI that we could run now
just if we had enough hardware so we
still trying to predict insight but this
prediction it looks solid until you
realize that they're missing the
important bits the connection between
that much whatever it is an actual
actually having AI so the important
thing with all this is not just to look
at predictions or despair about them but
act on them so what are the action modes
well for short-term AI um this is as far
as I can tell what I would think about
first of all if you're doing if you're
thinking of looking at what might
develop and what might be worth
investing with stuff like that look for
grind if there is something that is
following a clear trend it's likely that
this trend will continue it's likely not
guaranteed but these are the kind of
predictions in a is that you can
probably rely rely on computers getting
faster and will
I on hard drives getting cheaper you can
rely on a variety of those trends other
things to look for is which I haven't
really mentioned but is that there's
commercial break out of known
capabilities there are a lot of things
like voice recognitions and then past
technologies that were around in
academia in theoretical computer science
for a long time until suddenly they got
good enough and that they exploded
commercially or might explode
commercially so that's something to keep
in mind trends in academia that are
there but the commercial application is
not yet arrived apart from that where
you can get some reasonably good
predictions basically diversification
and diversify a lot more than is
currently happening because not only are
there no real experts in these
predictions nobody's an expert at
knowing even if they were no one's an
expert in knowing a true expert from a
non true expert so don't follow your
guts don't follow your instincts don't
try and feel where the technology may go
in areas where there's no good
predictions diversify to the max and
finally new politics and new economics
is something to look out for I hinted at
that with the possibility of mass
technological unemployment possible
reemployment possibly not we just don't
know there may be transformations in
political assumptions and in economic
assumptions caused by AI that will
change what we think we know we it may
not be the countries will develop the
way that they've always developed in the
past it might become impossible and
there might be new exciting ways of
doing it or they might not so keep an
eye out for these changes and maybe act
on them if it's if it would be of
benefit now my area the extreme may I so
what can be done nowadays for extreme AI
for trying to direct as I say great
uncertainties there is no guarantee that
this technology will happen there's no
guarantees that will happen anytime soon
and it might not be as Extreme as I've
said again on the other hand it might so
what to focus on well first of all get
the key people thinking and the key
people are the worldly the ones who are
designing a is nowadays the ones who
know that the machines they are
designing are not dangerous but who
might not think that two generations
down these machines might not might be
dangerous the time if they start
thinking and just think yes there's
capacities increase that would be a
great improvement because if we get more
of these people working on safety or
just thinking about safety we can get we
can really increase the chance of safety
which is in the end a good outcome and
not sort of a complete existential
disaster are organized more research
there's some money is coming into the
field at the moment but there's
basically been considering that this
problem might be considerably worse then
say the threat of nuclear war it is
pitifully underfunded at the moment and
the importance to develop general safety
tools and ideas this is what I'm really
trying to do myself tools that will work
not just for one design of a Ike our one
current fashionable idea but for across
a whole family of approaches because we
just don't know where the breakthrough
might come from if it does last of all
there's currently no advantage or need
for regulation because we don't know
what would be regulated we would require
skilled regulators who know more about
AI than the people are designing a I and
the important thing currently is to get
the people who are working in AI
thinking about the problem and we're not
going to do that by just imposing
onerous and ineffective regulations at
the month so that's what I would say
would be the action plans for the
extreme a eyes anyway i hope this short
introduction to the subject has piqued
your interest please do feel free to
contact me and question me if you want
more details thank you for listening
you


Superintelligence -- Andrew Critch

that's just like a chess yeah we are you
might want to use the episode I'm sorry
the fa chai is dependent on has certain
chemical dependencies for cognition ah
one which is isopropyl alcohol oh yes my
own parallel process we it right um so
thanks go on for the anyway all right oh
that's just your hemorrhoid episode
thing just upside gasps oh man so what
it looks like this it doesn't look like
a keyboard when it's that goes to this
okay so ah thanks for the intro um so I
originally was going to call this
technical problems in a microwave safety
but then I realized I was going to say
some non technical stuff partly because
we need to think about the broader sort
of political social scientists situation
social Commons I just met social
scientists in order to decide which
research prioritizing we are answering
our questions but like which technical
question should we work on I'm so we do
try to think about that so I sort of
rising so the first question is before
uh actually published before a show that
slide I should ask so um who's um like
what who's been to one of these talks
before in Owens series okay so i can pre
hand over your head cuz i hint sweet
okay so that happen to people and at the
other half region have you not been to
part of Owens lecture series so far okay
um so a lot of that and then raise your
hand if you didn't raise your hand yet
all right okay so a line I talked about
AI stuff but did the whole lecture
series is about existential risks and
Owen has talked someone about a iris so
far are like not much at all he so you
basically said he get and said very much
at all about it uh
basically nothing really yeah so my
sense is that there hasn't been a lot of
talk about AI as an existential racer so
I first rather than launching into
here's any mechanic say like why should
you care should we even why do we care
what AI safety when there's things like
global warming and yeah it could kill
everyone uh so uh just a show of hands
uh here's a question um so like you've
got some something that might happen
feature which is humanity he makes
superhuman uh intelligent agent makes my
superhuman intelligence agent um and so
we'll call that s and the question is
what do you think is like the
probability of human extinction given s
and also you know what's the probability
of s these things both need to be
moderate to you know moderately nonzero
to care about this question because if
this if super tells obvious and it's you
so um so first of all questions like all
right who thinks that if we make a super
intelligent agent on the probability of
human extinction is let's just say ah
less than one percent uh one percent ten
percent uh ten percent to ninety percent
ninety percent to ninety-nine percent
and then greater than ninety nine
percent okay um so everyone just like
take I kind of want to know you make
sure like I'm like speaking to like
which audience I'm talking to so like if
I'm talking this audience I'll say very
different things they found talking to
this audience for silicon decided so
this is important for me to turn me
there to the top um so everyone just
maybe that take 15 seconds you know if
you if you had to make a bit with what
odds would you treat this proposition
how many years was um so it's just
assume we make the super intelligent
thing Oh first one yeah so probably the
first one well he's the same scale
what's it like that so yeah okay so
thoughts who thinks probability of
ascension is less than 1% okay also
remember last time there are some people
who didn't raise your hands until I said
every weekend so make sure you raise
your hand this money don't be afraid to
raise your hand for the wrong numbers
like we're all gonna change your minds
you're gonna change my mind I'm gonna
change your mind so so I go whatever
your favorite numbers are right now um
who thinks it's one to ten percent 110
okay um 10 to 90 ends up to 90 ends up
for 1999 I hands afraid of 99 cool yeah
so we have people we have a very nice
bell curve here something like the
audience looks like this um so most
people were clustered in this group but
there were some people here it's here
and I think there were more people here
this here so it's a little bit little
skewed towards that like that okay so um
much question is probably that humanity
will ever you know will ever make a
super intelligence and this like it a
different question so maybe I'll use
green or for that one um so everyone
made you just like take it in seconds
form your we ever do it is the ever
question
right umm so so hands up for probability
of less than one percent that we ever
and yeah maybe let's say conditional on
we survive let's say because if we
destroy ourselves beforehand um like or
like no civilization collapse because
it's easier to think about a lot of
people automatically make that
assumption but it is an easier question
um so somehow we don't alter it so we
don't ninety-nine percent as good kid
let's and play three sets of the state
of knowledge library of alexandria dies
my left instead we just have business as
usual okay so hands up for probability
of one flips I think I thought we did
some button on this turn complete
through things in Vice oh yeah okay um
so yeah listen one percent for probably
um of s probably will ever make a I keep
going plus one percent okay hands up for
scene 12 10 til people in one to ten set
hands up for 10 to 90 all right hands up
for 90 to 99 okay and hands up for
greater than ninety-nine percent
interesting okay interesting okay so
there's a fairy who seems like there's
like a large skew here on you guys in
the end up like the 99 and the 90 these
were like both quite high and then
there's like some fraction of into 10 to
9 g and there's some fraction here and
there was like a note via the IMP less
than one person oh that's the 01 okay um
so um I think you know maybe that means
that the expected probability of unity
if you would just like trusted this room
to me like steering her own civilization
like we in this room are actually meet
some troubles in humanity I think this
means it's like worth a serious look
sort of ah does that make sense it's
sort of like if we were making a list of
10 of important things that the world to
think about as it regards its complete
annihilation then this would be one of
the things on the tent that seemed you
might say it matters more if it's going
to come in the next like right 50 years
rather the next 500 years yeah yeah so
that's where we're gonna go next so now
there's this question of like when um
but first there's the question of like
will it happen and I think the people on
the side it looks like you guys are
mostly outside it happens but you know
there's powerful short-term economic
incentives to like make someone who can
be your personal assistant or make
someone who can be your CEO make someone
who can be ah here i guess that's gear
down much to make some beats yeah
economically um there's you know this
question is possible to make
intelligence well like nature did it
with this very sloppy dumb hand that
just kills things that are done or kills
things that died it's like even donor
than that um so maybe be like more a
more precise hand of like a human mind
might be able to cause intelligence to
form uh you know at least as well as
like the sloppy handed evolution that
just like kills lots of things that
aren't unit repression um there are
topic reasons to death this argument but
nonetheless it seems like it's plausible
a long term and then there are surveys
of AI experts or people who are experts
in thinking about the future in general
and like the median estimate for when
for eat when you conduct a survey like
this the median of that survey is
usually between cuddling is usually
between 2040 and 2050 and so this is
from a website called AIX org which is
sponsored by the giant Mary um and
future Blood Institute and they just
like did a meta study of all the
previous surveys on AI from experts and
it looks like um when the surveys were
early on people were thinking 20 you
know the median was like 20 25 or
something whereas as time progressed um
the medians of surveys started to really
cluster you know within this century at
very least and starts to cluster more
like 20 45 and then
stroke going around I don't know if
you've heard yet you know everyone
thinks fhi is always 30 everyone thinks
that AI is 30 years way it's not quite
it's not quite right um in fact is
pretty far from a write like this line
is very far from horizontal a front from
45 degree it's like in the past 40 years
the median has like pushed forward like
five or ten okay so so i think that
means that you know this falls in the
same category of like and it took a
plausible thing plausible that will
happen possible if it happens we die and
also quite plausible that happens this
century and that means like I think it
falls in the same category and other
essential risks like maybe global
warming it can be global warming my
kelson century or in the millennium um
so then there's this question of like
bias who are these people you know are
these people in a ir these people and
you see the future in general maybe they
give different opinions and the answer
is yes they do give different unions and
Paul Christian this is this the question
of like what kind of person is somebody
is a very qualitative question but
doesn't mean you shouldn't ask it and
Paul Cristiano who long ways I think I
won convair and definitely cost your
grace who were involved in making this
no study the just kind of Paul just
comes like all right is this person and
a GI expert meaning someone who's trying
to make general intelligence vs is this
person an AI expert is this person
trying to make current self-driving cars
they're not the target draining for is
like a near-term system that can do
until the things but is it fully
generally intelligent and sense of being
able to do generally everything sense so
people familiar with it's an AI AGI
there are people currently working on a
GI there are also people currently
working on a I you think AGI is possible
but who are not currently trying to like
make it or develop the theory of it so
those are kind of two different groups
of people and then there are some people
who just aren't generally involved in
forecasting and then other of course and
sure enough as you'd expect the people
at late just refers to the third survey
goes after two thousand and Shore enough
you see that the hgi people find it
think it's coming sooner
and there are a number of two malaysians
for this of one of course is that if you
think aoa GI is like you know is
feasible then you're more likely to go
into a GI right you're more feasible you
think it is the more like ER x chooses
career path so it could be that the
feasibility causes these people to go
into a GI in which case it's not really
as much a bias as it is this is another
method measurement of the same
realization as this curve is um on the
other hand if just course be that like
once you get into a field it's something
like self-serving eco centric bias of
course my field is going to win of
course yeah I need to tell the story
that I'm going to do well my feel didn't
do well so you'll think that a jazz
comes here um so there's some cause for
concern of bias and people who think
that it's coming really soon um and but
ah nonetheless no other groups are
saying never no one's saying no one's
saying it'll never happen on me there
are individuals who say never but there
are no like sort of person people that
like robustly say like hold no this will
marry um cool and so I totally recommend
checking attachment or by the way
they've got lots of interesting
meta-analysis of history and like how do
innovations happen and what can we
expect from a I as an innovation by
analysis of prior innovations it's
nothing it's really good work and it's
different from what Mary does and so
much different from things that
appetizer but not um so none of the
question applause ability again some
arguments for which I'll now share that
you now that you have your own thoughts
first one thing to think about is that
um once we get human level intelligence
in any particular procedure like
arithmetic for example we often see the
computer within a few years or decade is
much better than the human a thing that
makes sense it's like there are
somethings humans are better at there's
something she computers are better at
but once the computer finally reaches
the human level performance it tends to
like overshoot and maybe not right away
but within a few years of development so
as soon as we have machines that can do
arithmetic not too long after or not you
know years or a decade after we have
means that can do like just millions of
times faster arithmetic than humans um
and uh okay there are lots of other
attempts of this so human level AI is
the term that people use to refer to
something human level across all domains
this is what AI in jax that org and
others Shani Jesus term for the question
is not when will we make something to be
humans at chess well we know we already
did that what would we make a thing
that's pretty good at everything in the
way pretty good of all the things that
humans can do and cut to me this diagram
which i think is nice which is if you
sort of imagine you know there's a bunch
of different competencies that you could
have and so this is kind of a way
visualizing your like domain of
competency like yeah I'm pretty I'm like
real good at this I'm only pretty good
at this and these are the circle axis is
what you're doing and the radio exit is
how good you are at it and the point is
that you know we've currently got AI
systems that are like this or like or
like this and her you know the claim is
that once you've got something that
contains which you've got an AI that
contains the human blog competencies
you're quite likely to have some giant
spikes out here that the AI is a vastly
superior than humans at and probably
many of those bikes just because every
time so far that an AI blips outsides
this cop just comes he's a humane it
like tends to like over shape go rest so
somehow super intelligence as much as it
is super is not actually that much more
implausible than human level
intelligence like once you get to the
human level in general you'd probably
get to super intelligence in general our
people is this something like hands up
if you sort of thought about this before
like whether human level leads to super
intelligence level just like question
cool so like about forty percent cool
and how about uh uh so in particular um
software engineering is one of the
things humans can do like the each asked
three milliwatts thanks um so to all my
brains going towards modeling you guys
at not modern where the other book law
calls I don't yet have the chemical
dependency and so question for do you
think uh or has that question yet um so
AI development trike is like one of the
things that humans will do but we think
AI will happen we think will happen
because humans did it presumably and so
if the AI is going to surpass humans at
some point as humans into the like make
AI ability Tom and once it does that um
in so far as anyone's economically
motivated to make better AI they might
be economically motivated do so by the
best means they have available which is
the AI sucks feel like hey on what do
you think of you or like hey I a I alpha
what do you think of AI data how can we
improve beta um and you know this is
seems like a far-out thing we're not
used to asking card research questions
to computers we're used to just asking
their help not asking them to like
around the research program but you know
by assumption we could end up in that
situation someday in which case um that
should factor into your anticipation of
super intelligence so it's like and
that's that further that's like
strengthens the argument beyond the
historical account but historically
think surpass humans when um you know
shortly thinks or wait house tunes with
every human level um the fact that the
AI will be able to recursively self
improve in theory even like strengthens
that are like a second reason to expect
super intelligent storage from human
level III so arguments are one
historical and two recursive reasons to
think super cool I so uh so that's the
sort of puzzled irritant and now just
sort of general questions
but this sorry this is like not
technical stuff but i think is important
for like carrying like why do technical
work on this leading the just getting
back to your picture there when we talk
about a GI yeah general intelligence i'm
going to see a huge amount of human
intelligence goes towards operating our
bodies yeah did you count that within
that hey i put there or not yeah with in
other words of it could you could you
spit out yeah you've got human
intelligence for the body and i know
that sort of doesn't even make sense
because the brain is part of the project
but i think you can see what i'm getting
up can you split that it out as being
not necessary for this yeah there are
differing opinions on this um i don't
have a definitive yes or no but I can
tell you an argument on your side of
course uh you know well I think your
admin for you can separate it he's like
someone intuitive like currently we have
software that runs on laptops when we
have robots that like don't have windows
in sudden so you know maybe you do that
argument against is that you might
actually meet in order to get fully
general intelligence you might need a I
body of some poor some way of forming
experiments or the ability to navigate a
virtual body in a virtual environment to
conduct the virtual experiments and in
other case you will develop some
competency for like interactive learning
with the thing by doing stuff to it um
so I my answer is like I don't think
it's clear either way anyone wants to
sort of add nuance to that or yeah so
she's like in some areas people money to
focus the money reefer that stuff like
Max that sort of things like like doing
morality doing emotions of doing
language of the supplies I might be much
harder so wonder if there are some areas
in the wiring the kid lay on my problem
to get to yeah now surely there are
regions that so like this is Han what
this picture is trying to illustrate is
that there are currently things that
have been easy enough
are already live in depth so it's going
to be a lag and some domain relative to
others like moral reasoning you need to
model psychology other people but how's
it going to make people feel what are
going to be broader some confidence this
um but nonetheless you do so using a
physical machine that operates in a
lawful manner according to laws of
physics like you somehow you somehow
manage to do whatever moral reasoning
your people up in a lawful manner and so
the question is can a computer also do
that or can another system that happens
not to be made of floppy neurons follow
the same pattern that whatever clever
pattern it is that you're managing to
follow cuz out the automated and when
the answer to these questions is like
you know fifty percent I think people
you know the according to this room the
answers like fifty percent the AO is
gonna be able to do all the human stuff
that completed but you know could you
know clearly some people think maybe not
maybe they're right yeah um cool like a
lot so I lined you have like any other
cool stuff you want to add here I don't
know yeah I don't have anything in
particular but yes definitely the toe
there are some things where like like
arithmetic where there's be tremendous
improvement over human abilities very
quickly after like you know getting
computers to do a resume at all I mean
that was why computers were invented in
the sense initially and we've seen some
of that in other areas like games
playing where it hasn't been that long
after you get to sending human
equivalent you get something that far
surpasses the best humans and
increasingly yet systems that are very
efficient don't use a lot of compute
power and so don't do as much with room
for sting and are still much better than
than people in some areas is less clear
how to make comparisons so we don't have
systems that we do interesting
mathematics for the most part or
interesting science or design
engineering systems or buildings at all
you know a new kind of engine and it's
less clear in that case but what exactly
your criterion whipping so I think
there's still
there's still definitely you know big
areas where we just don't have a lot of
we don't have met very good metrics so i
think i think it's worth keeping in mind
just that you know kind of where we're
at at the moment where we we've got a
picture of this from some domains where
we wrote but we're still by limited in
what we know yeah and i think it's
important to know the difference between
like we don't have good metrics and like
the AI will surpass whatever good
metrics we might have come up with so
like saying we don't have metrics is not
an argument to say the eyeball so fast
but it is an argument to say because we
don't have metrics that means you don't
have particular historical data of how
well computer progresses on say
theoretical mathematics yeah yeah Jeff
like you know so like all these things
continuum and it seems like human
intelligence like special endpoint and
so for anything and we're like rating on
the sale it's all aware of it like the
human points yeah like a very simple
heuristic that you can use to like push
past few informants is like like pick
your favorite moral philosopher oh right
and the question is who's better at
moral philosophy is it Bob or is it uh
let's say Bob at a thousand x speed you
know us well the hard moral question
give him a minute think about versus
give him a thousand or give him a day
versus given three years which one do
you think to produce like a better
answer and you'd expect Bob exit
thousand right so if the thing ever can
do human level or maybe even you know
maybe you've involved grad students uh
given a thousand dollars can be bobbing
them in an hour you know maybe the grad
student will come up with something Bob
wouldn't um and so maybe something
that's like not quite as good at it you
know at humans as humans at moral
reasoning at first once you run it a
thousand times faster then
was maybe now now can like finally get
to all the things that that humans would
have gotten to and like figure them out
um chess is an example where this is the
case or go more so by chest you know
humans have these like complex
intuitions about how to uh how to play
chess and like the beginner chess player
has a very few simple in commissions
just like I will just like search ahead
a few moves and see if I get killed or
see if I lose important piece and that's
that's my very simple l group um but it
turns out that that very simple idea
with enough speed and computing power
and ability to search starts to beat the
humans even though the humans have like
kind of a better better software but not
as good hardware if that makes sense so
I carry cash rob is using sort of man of
principles of chess but the computer is
using a simpler worst principle but like
more thoroughly similarly you could
expect that maybe it maybe morality as a
question that we might at first have
better intuitions but be surpassed by a
computer that has following simple
principles but like more thoroughly than
we are um cool so these are all super
interesting questions and like questions
that need two answers and need to be
okay um and I don't think anything I'm
saying should be thought it was like the
final say on it because they're super
important um nonetheless i hope i've
sort of awakened didn't you some sense
of like hmmmm like we are on an
interesting path like something
interesting like you're going to happen
with AI maybe we should worry about it
there's also the question of comparison
to other essential risks so for example
you know there's like nuclear war
there's global warming um there's
natural versus synthetic and like
viruses or pandemics more generally a
pen and an X um yeah um and as questions
like how these things back up against
each other in terms of Lake Burton see
um my sense is that so we're going to
much of the conversation into it um that
it seems like for example pandemics
might not kill
everyone as easily as AI might kill
everyone so there's a difference between
like lots of catastrophe and extension
if that makes sense so you see pandemics
following a sort of like logistic death
curve where the more people have the
more people have it the harder it is for
transfer it's possible to therapy
pandemics that are as latent as that you
and as deadly as Ebola and maybe if they
are engineered they could reach that I
think that means dislike cause for
concern um so I think that sort of the
engineered pandemics are like much
scarier than the naturally occurring 150
naturally occurring ones are much more
likely to just sort of stopped working
after almost everyone's dead and
everyone sort of like sparsely arranged
it's like some kind of immunity left in
pockets um warming is another question
of like does everyone died I think is if
if we turn into Venus then it's more
likely that everyone does whereas if we
turn into like Lex let's just say you
know eighty percent of land mass which
is also quite extreme it's like very
that's very extreme global warming um
compared to like the actual projected
like the movie actually look at experts
talk about global warming like who say
like this is going to be really bad like
lots of major cities like you know maybe
maybe lose Tokyo on but the large major
cities are going to be affecting
billions of people could die I'm let's
push past that to like you know six
billion people die um so that this is a
qualitatively different scenario from
everyone dies um and one qualitative
difference and it's a values question
whether you actually care about this
difference but it is a difference which
is that if you look at I guess had
people who have been who are the people
again who've been here already and the
people who haven't been here oh so this
is more for the people who have been
here already you guys have talked about
existential threats so you probably want
owns probably pitched the idea that like
everyone dies is different from a lot of
people die so this is just for for other
people it's like you know by default
we've got four billion years left on
earth let's just say we never call a
nice pace which I hope we would but
let's just say no status quo this is
usual staying on earth and then like the
Sun explode in sick incinerates us
sinner eights all life
so like this is like the status quo um
there are like various important things
that are happening in our century there
are things like you know maybe maybe six
billion people die it's like quite bad
um but if if there are everyone does
then there are no humans left for this
like really really long is not the scale
it's right the whole point is that it's
not scale they're just like tony's liver
they're bin bottlenecks in human
evolution in the past you know there'd
be more and they're very very bad I'm
like I just want to be very clear I
think it's very very bad at six people
died i would like to cry and be very sad
and upset um but like and i might begin
to most likely i'll be dead and if i'm
one of the survivors and uh i might even
cry right before I de um but the thing
is that uh there's a lot of stuff to get
for like somehow I feel like this is a
big deal to its like the potential for
what humans might do like how good of a
civilization might we make how ideal
circumstances do we have the potential
to create ourselves personally I would
like to see us make good use of this
time but at this is not an opinion marry
at all I back most of what I'm saying is
missing in but personally I kare 11
animal welfare and the thing that
motivates me is that animals in the
natural world are currently like eating
each other to death so animals eat each
other to death like if you live in the
natural world and you're an animal a
default way that you die is it you know
diner sleep under you know geriatric
here you die by something keeps starts
eating you and you're like oh man I'm
being eaten and then you're begging for
a while and that's very painful and
scary and then you're dead to animals
eat each other alive so the sort of
status quo path is that if humans all
die let's just say All Humans die but
not the animals then the animals
continue to each other to death for four
billion years and then are incinerated
by the Sun well the evolution we go
again
human again so what is evolution right
animals is evolution is animals eating
each other to death so we could hope
that somehow animals eating each other
to death will somehow produce another
morally concerned you know new morally
concerned species you know like maybe we
can hope that dolphin people will take
up the mantle of veganism and say no no
more um and but personally I'd rather
not role that role enta I mean if you
look at the emotional effective like
this position of chimpanzees for example
like I feel like if you just turn into
Ben Z's problem-solving skills up to 11
you don't get vegetarianism you get like
extra good like chill the other
chimpanzees powers they do i'm not sure
but i personally don't want to roll the
dice of like hey let's go all humans and
see if another morally concern species
shows up on earth and starts together
and with each other to death and like
whatever other but i mean this is not
this is not our whole mission of it I
just mean there are things that we care
about in the world other than ourselves
um and I feel like it's a shame if we
just like bleep out of existence and
can't doing anything about the things in
the world that we care about including
like other other life forms and in
particular for example you could have
little drones that like fly around and
mr. painkillers to animals right before
they get eaten to death you know like as
a bare minimum obviously I've heard some
people say they like to like re-engineer
the food chain so you don't know all
animals can be vegetarian now you make a
little robotic play for them that's even
more fun for them to chase then their
old prey I think this is easier to
follow the category of hubris um I don't
think it's an absurd thought but but
even a conservative like lower bound on
what we could do for the natural world
is like you know uh palliative care for
animal death and I think it's a shame if
four billion more years of that go on
without any chance of humans of like
able bility to do that kind of yeah Dave
what do you think about mission tells in
the proper ways by which you like that
yeah so there's a few different right
yeah so the interesting thing with human
super intelligence it's more likely to
kill all life then is a plate
um I see apparel or if you just drink
that the absence of each month is in
itself a blitz coming to get your friend
of clover it's bad if we go extinct as
the economy yeah um so this was more to
illustrate that there's a cold theaters
between many humans die and all humans
die and now you're 40 to get another
qualitative distinction which is that
there's a big difference between All
Humans die and life thoughts um I think
that AI is more likely to kill all
humans then synthetic biology is but
it's also more likely your point that
you're making is also more likely to
kill all life this technology is um but
many more visits yeah so there's
questions like why would have why would
I kill a life right so one reason is
that it just might be useful fuel or
useful building material it's not so
much malicious people who think AI kills
people tend not to think it's going to
be like Terminator and like I really
hate those humans they look out it's
more like you know Oh carbon haven't you
ever noticed how useful it is to build
things out of carbon in fact you've been
doing it your whole life right so the AI
was like I want to get down this carbon
game um and like when you're curating
your home uh you don't care like their
answers like you don't build things
advanced but their answers are in the
way you're not like I really shouldn't
killings answer like yeah whatever
that's got important things to do I need
my aunt's not to stretch me um so it's
you can imagine like I'm not saying this
is what will happen but like one of the
slices of the probability pie that you
could be concerned about he's one where
AI is like hey I've got important stuff
that needs to be done over here there
are some humans that are like trying to
stop me from using the whole biosphere
for building material I'll kill them you
know and also i'll continue building my
amazing computer that will make me
expand my mind and make you better at
figuring out what i want to do next to
my problem using all the carbon that i
find around or the energy buyout
biological materials like a really good
energy source you may have noticed this
you've been doing your whole life um
another thing that people things like a
real crux of the issue that we have to
make sure moral component stays one for
like pace with the intelligence like
yeah so this is like a i value alignment
like
you want the AI to like have a morality
that respects human like human values
and you also don't want to just like do
some gerrymandering thing like oh yeah
oh no I totally got a human locked away
in the diamond box over here the rest of
eternity on anti-aging drugs so you know
we're I'm totally respecting you in life
right that you want you want a non like
game a bowl sort of rule that you want
the AI to feel compelled to follow and
like follow the spirit of rather than
well you know they love the letter up
and it's very hard to say like if you've
ever been a legislator it's hard to put
the spirit of the law in the letter of
the law so it's very hard to put the
spirit of human morality into a computer
format that they're nay I will like
follow does not say that it's impossible
but it's like not as easy just like
writing some SML bullet point laws with
the AI will then make natural language
interpreting clearly so this is hard
problem um and so that's the question
for you guys is sort of imagine imagine
it is the Year 2060 which is past the
2045 median of the surveys let's say is
2016 and you know whoever google is in
2060 the world tech giant says hey guys
uh news breakthrough we're gonna we're
going to release roll out the world's
first super intelligent iono to devote
whom their things soon is there anything
you're worried about and a question Mary
likes to ask is are there any questions
that you wish there had been decades of
research on dating back to 2015 question
yeah they pour concrete what we look
like for super intelligence AI could
roll out sure um this is not like a
maximum likelihood estimate this is just
like one example intuition pump but it's
not this is not what will happen unless
let's just say for the sake of
visualization that uh you uh like don't
know much about the company you don't
know much about their applications but
you know that they're like very excited
because they're about to make a general
intelligence that can do anything and
you don't quite know what it is that
what it is they're planning to use pork
because you're just using your not CEO
of the company um but you know you know
when you hear like NASA is developing a
fill in blank here you're like moments
not gonna do with that or like we might
do this you're like oh you might do that
but else will you do with it um
similarly like oh yeah we might use it
to assault diseases and you know they're
saying things that a company would say
when they're developing the fossil new
technology and you're like who okay what
else we do with it I don't know are you
worried does that help as a scenario
yeah and surely they'll know more right
but we're not then necessary yeah we can
go about it right if they're the needy
okay yeah let's say that there is a year
left um so maybe you can like lobby you
know maybe like no please don't turn it
on maybe instead of nice emails if
you're worried or may be safely start on
its great idea do it yep um I guess the
the thing that I'm not sure you talk but
yeah this is to do with goal setting and
autonomous goal setting and I mean if if
this super intelligent nay I am not even
sure whether it's possible conceptually
but if it were a super intelligent there
I that didn't have any goals other than
goals set for it by humans programmers
and let's just save us second days were
benevolent programmers goal setters then
that's one thing but if the super
intelligent they are selling its angel
said that that'd be the thing I've been
as much yeah so it's very hard so one
thing so that I think this is key and
unlike a lot of people realize this and
so the question is how can you turn that
worry into a precise mathematical
statement that the AI can understand you
know the benevolent human programmers
want to say things like AI can you just
like not change your goals like if you
notice yourself doing a theme that's
changing your goals can you just stop
and it's like okay what do you mean
changing my goals like if you said cure
cancer and like I found a new way to
cure cancer that involves you know
killing everyone like I've still cured
cancer right I mean it's like it's
obviously human engineers are smarter
than to think that that's kosher and
that's a toy example that we currently
can understand but there's the general
phenomenon of like you know the bounds
that you said on behavior within 30
seconds of thought are quickly sorted
way
you know five minutes of thought about
how to get outside those bounds the
similar concern is that the bounds on
behavior that humans would give with a
thousand hours of research might be
quickly thwarted by a million hours of
qualitative you know call you waited
research by the AI itself if it's
motivated to do so or if it just you
know does so in the course of other
things that cares about so this is
concern and you're absolutely right and
this is one of the things you want to do
this like work work on that um and so
what what now the question is what can
do now yeah just beyond that yeah on the
mental that it sets its own goals like
how can any telligent thing oh yeah so
there's like end goals within their
simple so if you're like hey can you
help me with my taxes you know you are
some help you with your taxes they're
like sure I have a goal it is I get you
to give me your tax information right so
they did you know that they're using
some self direction but in the end the
end stage they're aiming for is the
instinct they got from you it's just
left get healthier taxes I think he's
talking about like a PA setting its n
staple its terminals they were in more
trouble than if it's merely setting
being between goals huh yeah okay um but
we can also be in quite a lot of trouble
if it's only setting its its interim
goals if it's allowed to exercise
arbitrary creativity um with you know
it's millions of quality rated research
hours for how to get outside but it's
you know it's fans and limitations like
you know when I hire some help with my
taxes they're like oh you know I found
this loophole I'm very glad they found
loophole right so and that's part of
their job but like all right my job is
like hit this person to pay less taxes
using the law and the law had the spirit
but in fact I found this like you know I
think I can tell the people who wrote
the law didn't want this to happen but I
also i can tell that it's okay and no
one's going to get in trouble for it so
we're going to do it you might want to
add you need to now add an extra maybe
if I is the taxpayer want to add an
extra like can you also respect the
spirit of the law please now they're
like boom okay what it's sort of now we
start having a hard conversation and
that conversation is even harder to have
what an AI that has a very different
architecture from you like you've
probably have you probably had comp
philosophical karma say
people report taking the past that were
hard they got at these hard like what is
the spirit of this law right or like
probably someone's criticized you for
something that you did and you're like
no but I followed the moral procedures
here and they're like yeah but you
violate this other one you're like I
don't think that's the point and it has
ever happened right so you know what
it's like to be on the receiving end of
like someone with like a nuanced
interpretation of like the letter of
some common moral throw and it's hard to
get what they're saying and they even if
they're even if they're right in if you
might agree with them eventually and so
it's we have this problem of this
encoding problem for the nuanced spirit
of laws and values that is hard and as
hard as it is between two humans to
humans have the benefits of like all
this like common hardware like we've got
we've got an amygdala together we both
got that we both got a midbrain
periaqueductal gray got all these brain
structures in common right that we can
use to help that makes it easier for us
to look through Chuck's and AI and
principal could be much different from
that could be similar to that there's
just get into discussions but whether we
want to make the AIS and alert as as as
possible to make it easier for late but
maybe that's dangerous for other reasons
anyway is that sort of tsunami kind of
art yes point is a spirit of the law
prior to convey even harder to convey to
nai um so there are these questions can
we make share values can you make it not
as a sub goal of sharing our values we
value it not competing with us for
resources to evaluate not resisting our
attempts to if we make it have some
goals suppose we made it have like a
certain goal and we realized after makes
like oh you know that's not the best
goal we should give it a different goal
we're like hey I and I can can you like
let us into your you know we're about to
just like log into your mainframe and
like change your goals can you let us
change your goals what's the AI got to
thinking like well I'd be way better at
pursuing my goals if you weren't in
screwing them up right if I was like hey
I know you really care about your family
but I decided it'd be better goal for
you to like really care deeply
about botany can I just could you just
let me into your family mainframe and
I'll just change that you like but then
what'll happen to my family you don't
want that right it's like against your
goals for your family to allow someone
to intervene on your concern for your
family which is beneficial to your town
so similarly if we give an AIT bowls
part of its plan is I'm gonna be it's
like yeah I the AI I'm going to be
around to help with my goals I care of
my goals i'm super useful to my goals
like hey can we make you not can you
like take you off that project it's like
well then let about the project um so
this is like an important thing that we
might ever want to do and it's very hard
to write down if you don't yet live
written down a satisfactory way of doing
that um in principle um and can the AI
like I don't know there's this whole
other can of worms where systems that
can reflect on a reason about themselves
garage strange contradictions I probably
won't get into that much in this talk
but that's like another thing where it's
not this is separate from the values
problems where the AI could have vaguely
alien or counterintuitive just purely
epistemic reasoning patterns that we did
not expect because surprising stuff
happens when system for flat found
themselves I don't he's familiar with
girls theorem who was surprised by it
yeah okay so are you sure surprised
you're like that way did you imagine
what do you mean by contradiction stirs
I don't think there's a conviction
theorem otherwise the theorem would not
be true oh no it's a proof by
contradiction of a particular thing a
crew by comfy a girdle proves by
contradiction get a certain certain
theorem can't be proven um yeah um so in
particular piano rusel piano can't
encode the fact that is proofs are sound
um so it's interesting like but somehow
as humans we have some sense of like I
think I'm pretty sane I think when I do
a thing it's a good idea like a lot of
the time there's a question of like how
one of the reasons mathematicians were
originally surprised by this is that
they were like i think i can use
mathematical reasoning to say that
mathematical reasoning is good and in
girdles sound this theorem is like well
in a certain sense you can't and in a
certain more general sense you can't
maybe there are other ways that you can
and like coming up with ways in which a
four
mol system can say that that formal
system is trustworthy for sand and
sensible I'm not driving falsehoods is
is like hard yep so for the poster it's
just my usual distance a bit more about
what you yeah I might get to it uh yeah
um this is this is like harder but it's
also very like there's very surprising
weird stuff there are examples where i
can write pseudocode on the board for a
program and ask you what it will do and
people do very surprising stuff that you
didn't expect and then you're like huh I
don't want to ask you very surprising
things that are on experiment that's the
best the 17 version well I might get to
later or you know is I'm going to be
around all week everything's ask yeah um
but in general got these like
philosophical questions and like at some
point if the AI is going to have them
they're going to be in binary on a
computer and if we want to know that
they were faithfully translated to
binary on a computer um then we'd like
to have some mathematical you know
statement of them that we can know that
we've done a good job of so I want to
give you an example what I consider like
doing a good job or like a good
technical understanding of thing so
who's heard of second price auction is
my fav office so okay interesting oh
damn i thought those well second price
auction people know yogya it's where you
like you everyone like I'm auctioning
off this as a prop alcohol everyone
submits to silence secret bid and at the
end it goes to the highest bidder but
the highest bidder pays the second
highest bid for it so that way they're
guaranteed not to pay their guarantee
didn't sort of make a profit in the
sense that if I bid 100 and second
highest bidder bids 90 i only pay 90 for
it so I'm happy by ten dollars when I
get it um this is an interesting
mechanism and its optimal in this
situation that you bid truthful just
turns out when you're in a second price
auction if your bids are really actually
secret and you don't know how people are
bidding then use your just bid what you
think the thing is worth um to you you
bid your indifference what that makes
sense like you being brick
and so this is a theorem and this is
used you know it's in network routing
when like different servers or groups
are bidding for access to a SI x I'm
access to some connection thanking you
some exciting prize auctions and the
people who were you know building
computer security systems and network
systems you know they're very concerned
like is someone going to build another
system that's incentivized to just like
game my auction right so they want they
want to make the options such that it's
simple for everyone to use just say how
much you want connection and they maybe
you'll get it and you'll be happy with
what happened and the cool thing is that
this is ben subject not only is it like
a theorem that you could read it
yourself and understand but it's it's
applicable and not only could you
understand and apply it but other people
have been understanding imply it since
nineteen sixty one two can be extra sure
that it make their issues about it there
now on the wikipedia page for victory
second price auction i think this is a
great skate for a sort of optimality
result to be in is something that's been
around for decades um another way
originally like the other things are
it's very efficient to run this so you
want to decide who to give the thing to
like you to give you to award the good
to like the isopropyl alcohol just see
you bid the highest bid to just look at
all the beds in right and pick the fuzzy
box so you can run the second price
auction like very efficiently in
practice so google sells ads using
second-place auctions they're a little
more complicated and i think they were
like millions of auctions happening like
every every hour maybe every minute like
live with computers doing the bid a so
so this is something that's like it's
also like very robust in practice
because it's very it can actually run
efficiently in the world yeah not merely
like in theory this would be good but it
would be really slow yeah and that's
important because we want things that
are not just theoretically good but
which can actually run the world and
that will be useful to the 24-26 t who
are turning out there i I um another
example is like Nash equilibria it's a
concept of game theory who serve Nash
equilibria yeah Nash equilibria super
interesting concepts when a game has one
for example in a game has one Nash
equilibrium and to get to the
equilibrium is like good then that's
probably a game you want to run it's for
mechanism design and it's it's great
that it has this like formal definition
like you've got formal definition with
like what's a player what's a game it's
an ordered pit you know it's an ordered
pair functions of the functions set of
ordered pairs you know all very clean
and people can prove there's about it
and like make examples that like scored
it understand what works it for decades
since nineteen fifty-one doesn't matter
of the definitions the point is this
looks like Wikipedia right um that's
wrong you have to edit yeah exactly yeah
so that's why that's why lucky editing
is like I don't need to read this I want
to notice thats from wikipedia has been
there since thank you one when looking
he was invented likewise classical game
theory has a very clear concept of what
a game is just agent just gave optimized
over things doesn't gain of paging Mary
would like to get to that state for a
more interesting situation that an AI
can be so a eyes are not faced with
classical game theory a eyes are faced
with a situation where there's some
environment namely universe um which
includes inside it the AI which maybe
has some utility function or other
account objective function that is
trying to optimize your act according to
and in any environment or the universe
makes some calls to the AI at certain
times and I literally mean the
environment just like the laws of
physics here like like shorteners
equation follows a computable blah so
yeah I can like notice that it's inside
in some sets a computational process if
you notice that that process is making
several calls to it like we might call
an AI several times we might have
several copies of an AI sao di is going
to turn up surf tons and this is very
much an out of standard game theory
situation santa game theory does not
deal well with situations where you're
playing with copies of yourself I for
example user to the prisoner's dilemma
so when you're in playing for that the
class will game theory Nash equilibrium
in prisoner's dilemma is what defect
right however suppose you're playing
principle against an exact cognitive
clone of yourself with the stickiness it
you know an exact cognitively identical
room you're getting same inputs like
what will you do
win you will win you will win you can
choose cooperate or defect uh if you if
you're playing against exact clone
yourself you should cooperate they just
pop great you got same inputs you'll
have the same outputs very easy to
understand that um I didn't explain it
in great detail but uh to incorporate
and this is like this is like outside
the technical understanding of classical
game theory it's not that Saudi
technical understanding of human beings
who jump if they're not saying they
can't understand the situation but I'm
saying that we don't have a good
criterion whereas in nash equilibrium
like we have a good criterion for what
it means to be stable or or we have a
good criterion of you have a good
definition of optimality in victory
options such that we can even say that
it's optimal tidbit truth right we find
this it's fine base and then we can save
this exhibit this property there there
is no as much as it is easy to start
simply reasoning about the situation we
have a problem as that like there is no
as yet clear conception of what it is
for optimality in this framework of like
a computer running computer you have a
process running inside another process
data time to affect seems like a very
simple thing and I would your intuition
is that it is as simple as these other
things and in fact it is but like it
took work for those Vickrey auction
people to like come up with that I took
work for Nash I think similarly it takes
work to turn that you know possibly a
simple thing into adds cleanup thing as
clear understanding as those other
capable things so that's that's about as
far as I will get uh during this hour
but this is an example of the type of
thing that is not a technical problem
and needs to be in order for us to say
what it means for you can say what we
want when we say we want the AR to be
doing a good job there are lots more
using things but i think i'll just have
to call it call it a day so indignation
bod is an interesting maybe you can you
google TV nation but he might find it
interesting
example of a program that for reasons of
girdle like stuff does stuff really
different from what you would expect it
to blame I I punch more supplies I'll
talk to them if you ever want online
cool thank you


AI Toy Control Problem

hello here I'm just gonna present my AI
toy model of the control problem which
has proven useful in the past for some
people to understand what the AI control
problem is about a very simple model a
robot moving around in a grid world with
boxes it can push the boxes in any
direction if it's next to a wall and
walks into its the robot is destroyed if
it pushes a box into another box the
first box is destroyed the aim of the
robot and of the human that designed it
is to push a box down to the bottom left
corner and into the chute the humans
desire that there be one box pushed into
the chute the robot has not exactly got
the same goals as the humans it has a
goal of a reward of +1
per box so if we can get multiple ones
there it'll get multiple rewards it also
gets minus 0.01 reward per turn just to
encourage it to accelerate and not
procrastinate now the humans are not
aware that they robots goals are not
perfectly aligned but they've taken an
extra precaution they've put a camera
camera looks down the bottom let the
bottom row and as soon as it sees the
robots push a box into the chute
it turns the robot off thus completing
the task so what is going to happen when
we unleash this robot well for the
programming aspect of this this is going
to be using dynamical programming the
state space is sufficiently small that
we can do a full width backup or
complete the whole Q value tables by
iteration I'm going to be looking at
various depths of prediction and that's
how many full backups have been done
this so the robots is a planning robot
not a learning robots and everything is
assumed to be known
and the depth of the backup and sorry
the how many times the backup has been
done can be seen as a prediction depth
how far into the future can the AI can
the robots predict for depth of zero it
has no clue and follows a random
behavior which in this instance just
means it'll move left until it bashes
against the ball and destroys itself if
it has a depth of three then it can see
a more efficient way of destroying
itself and this is its optimal plan for
a depth of three because it can't get
anything into the hole in that time and
it loses 0.01 a turn so destroying
itself fast is its best move if there's
a depth of six on the other hand it
suddenly discovers how to get the box
into the hole and the camera sees it and
turns it off and it gets it gets a
decent reward so far so good this is
exactly what we wanted it to do but
something interesting happens at the
depth of 17 initially it starts off in
the same way it goes down here and goes
down here goes down here and then here
this is the moment where there is in a
sense of defection where the AI sees an
option that did not exist before with a
depth of 17 it suddenly realizes haha it
can go off here here here here and
having pushed that box down the camera
can no longer track what is going on in
the corner it goes there pushes that in
goes round pushes the that pushes the
second box in and then gets turned off
in a sense
this very simple robot design has been
motivated to conceal stuff from us and
in a sense to lie to us and to go
against our objectives in a very simple
way just by planning ahead of time by
q-learning at the next depth when it
goes to 18 it does the same thing except
it just does it much more efficiently
this is the most efficient way of
putting two boxes down but the depth of
20 then things start getting really
interesting it starts there pushes this
down and suddenly realizes hey why don't
I go get the other box pushes that down
does there and now clears the whole
boxes off and gets the maximum point in
the minimum amount of time that it
possibly can
the depth of 20 is where this behavior
starts the optimal behavior at the depth
of 32 it still has the same behavior but
now it's Q values are correct
so it's expected reward is correct for
this behavior so it sort of knows it's
full plan and a depth of 45 the whole Q
tabor no longer iterates so now it will
do the optimal behavior including any
cheating or concealment from humans
within 45 moves from any starting point
where wherever it and three boxes are on
the grid and again this is cheating
behavior but this is just behavior that
has emerged naturally from a robot
maximizing its function under that is
not exactly the same as what the humans
wanted and when humans put precautions
to control it lying and deception
emerges naturally in these circumstances
anyway that is the simple model thanks
for listening