Sam Harris and Eliezer Yudkowsky -  The A.I. in a Box  thought experiment

this intuition that you could just shut
it down this would be a good place to
introduce this notion of the AI in in a
box thought experiment so the hood
because this is something for which you
are famous online
I'll just set you up here the idea that
and this is a plausible research
paradigm obviously and in fact I would
say a necessary one anyone who is
building something that stands a chance
of becoming super intelligent should be
building it in a condition where it
can't get out into the wild it's not
hooked up to the internet it's not in
our financial markets doesn't have
access to everyone's bank records it's
in a box that's not yeah that's not
going to save you from something that's
significantly smarter than you are okay
so let's talk about it so the intuition
is we're not gonna be so stupid as to
release this onto the internet I'm not
even sure that's true but let's just
assume we're not that stupid
Neil deGrasse Tyson says well then I'll
just take out a gun and shoot it or
unplug it why is this AI in a box
picture not as stable as people think
well I'd say that Neil deGrasse Tyson is
failing to uh to respect the AIS
intelligence the point of asking what he
would do if he were inside a box with
somebody pointing a gun at him and he's
smarter than to think on the outside of
the box this Neil deGrasse Tyson going
to be you know man give me all of your
money and connect me to the internet so
daemon can be like ah no and shoot it
that's not a very clever thing to do
this is not something that you do if you
have a good model of the human outside
the box and you're trying to figure out
how to cause there to be a lot of paper
clips in the future and I would just say
humans are not secure software we don't
have the ability to like sort of hack
into other humans directly without the
use of drugs or like having or in most
of our cases having humans stand still
long enough to be hypnotized we can't
sort of like just do weird things to the
brain directly that are more complicated
than optical delusions unless the person
happens to be epileptic in which case we
can like flash something on the screen
that causes them to have an epileptic
fit we aren't smart enough to do sort of
like more detailed treat the brain as a
something that from our perspective
it was a mechanical system and just
navigate it to where you want that's
close the limitations of our own
intelligence to demonstrate this I did
something that became known as the AI
box experiment there was this person on
a mailing list who like back in the
early days what this was all like on a
couple of mailing lists who is like I
don't understand why is the problem I
can always just turned off I can always
not let it out of the box that was like
okay let's meet on Internet Relay Chat
which was what chat was back in those
days I'll play the part of the AI you
played the part of the gatekeeper and if
you have not let me out after a couple
of hours I will PayPal you $10 and then
as far as the rest of the world knows
this person that later sent an email a
PGP signed email message saying I let
Eliezer out of the box someone else said
that the like person who operated the
mailing list said okay even after I saw
you do that I still don't believe that
there's anything you could possibly say
to make you let me out of the box I was
like well okay like I'm not a super
intelligence you think there's anything
a super intelligence could say to make
you let it out of the box
he's like mm no I'm like all right let's
meet on internet relate in it Internet
Relay Chat if I can't convince you to
let I'll play the part of the AI you
play the part of the gatekeeper if you
can't if I can't convince you to let me
out of the box
I'll PayPal you $20 and then that person
that sent a PGP signed email message
saying I would Elliot as we're out of
the box right now one of the conditions
of this little meetup was that no one
would ever say what went on in there why
did I do that because I was trying to
make a point about what I would now call
cognitive uncontainable 'ti the thing
that makes an something smarter than you
dangerous is you cannot foresee
everything it might try you don't know
what's impossible - it may be on a very
small game board like tic-tac-toe the
logical game of tic-tac-toe you can
imagine you can in your own mind work
out every single alternative and make a
cat
statement about what is not possible
maybe if we're dealing with very
fundamental physical facts if our model
of the universe is correct which it
might not be we can say that certain
things are physically impossible but the
more complicated the system is and the
less you understand the system the more
something smarter than you may have what
is simply magic with respect to that
system imagine going back to the Middle
Ages and showing and being like well how
would you cool your room you could maybe
show them a system with towels set up to
evaporate water and they might be able
to understand how that is like sweat and
it cools the room if you showed them a
design for it for an air conditioner
based on a compressor then even having
seen the solution they would not know
this is a solution they would not know
this works any better than drawing a
mystic pentagram because the solution
takes advantage of laws of the system
that they don't know about so a brain is
this enormous complicated poorly
understood system with all sorts of laws
governing it that people don't know
about that none of us know about at the
time so the idea that this is secure
that this is a secure attack surface
that you can expose a human mind to
super intelligence and not have the
super intelligence walk straight through
it as a matter of what looks to us like
magic like even if it's told us in
advance what it was going to do we
wouldn't understand it because take
advantage of laws we don't know about
the idea that human minds are secure is
loony and that's what the air box
experiment illustrates you don't know
what went on and there and that's
exactly the position you'd be in with
respect to and with you being with
respect to an AI you don't know what
it's going to try you just know that
human beings cannot exhaustively imagine
all the states that their own mind can
enter such that they can categorically
say that they would have let the AI out
of the box I know you don't want to give
specific information about how you got
out of the box but is there any generic
description of what happened there that
you think is useful to talk about I
didn't have any super-secret special
 trick that makes it all makes sense
in retrospect I just did it the hard way
when I think about this problem I
think about it just obviously butyl
rewards and punishments just various
manipulations of the person outside of
the box that would matter so it means so
far as the a I would know anything
specific or personal about that person
we're talking about some species of
blackmail or some promise that it just
seems too good to pass up
like giving useful information you know
building trust through giving useful
information like you know cures to
diseases that the researcher has a child
that has some terrible disease and the
AI being super intelligent works on a
cure and delivers that and then you know
that just seems like you could use a
carrot or a stick to get out of the box
and notice now that this whole
description assumes something that
people will find implausible I think by
default and it's it should amaze anyone
that they do find it implausible but
this idea that we could build an
intelligent system that would try to
manipulate us or that it would deceive
us that seems like just pure
anthropomorphism and delusion to people
who consider this for the first time why
isn't that just a crazy thing to even
think is in the realm of possibility
instrumental convergence which means
that a lot of times across a very broad
range of final goals there are similar
strategies we think that will help get
you there there is a whole lot of
different goals from making pate lots of
paper clips to building guide diamonds
to putting all the stars out as fast as
possible to keeping all the stars
burning as let as fast as possible where
you would want to make efficient use of
energy so if you came to an alien planet
and you found this what looked like an
enormous mechanism and inside this
enormous mechanism or what seemed to be
high temperature superconductors which
what or-or-or like high amperage
superconductors even if you had no idea
what this machine was trying to do your
ability to guess
it's intelligently designed comes from
your guests that well lots of different
things as an intelligent mind might be
trying to do would require
superconductors or like would be helped
by superconductors
similarly if we're guessing is that a
paperclip Maximizer tries to deceive you
into being it tries to deceive you into
believing that is a human eudaimonia
Maximizer or like general eudaimonia
Maximizer the people building a tower
Cosmopolitan's which they probably are I
should just put notice here that
eudaimonia is the Greek word for well
being that was much used by Aristotle
and other Greek philosophers or someone
I believe Julia Galef might have defined
it ught eudaimonia
is happiness - whatever philosophical
objections you have to happiness that's
nice
anyway like we're not supposing that
this paperclip Maximizer has a built-in
desire to deceive humans it only has a
built-in desire for paper cliffs or
partly not built in but like inbuilt I
should say or innate people probably
didn't build that on purpose
but anyway its utility function is just
paper clips or it might just be unknown
but deceiving the humans into thinking
that you are friendly is a very generic
strategy across a wide range of utility
functions you know humans do this - and
not - because we have and not
necessarily because we get this built
deep and built kickabout seating people
although some of us do but like a con
man who just wants money and and have
just no innate take out of you believing
false things will cause you to believe
false things in order to get your money
right a more fundamental principle here
is that obviously a physical system can
manipulate another physical system
because as you point out we do that all
the time we are an intelligent system to
whatever degree which has as part of its
repertoire this behavior of dishonesty
and manipulation when in the presence of
other similar systems and we know this
is a product of
physics on some level we're talking
about arrangements of atoms producing
intelligent behavior and you know some
level of abstraction we can talk about
their goals and their utility functions
and the idea that if we build true
general intelligence it won't exhibit
some of these features of our own
intelligence by some definition or it
would be impossible to have a machine we
build ever lie to us as part of it I'm
an instrumental goal you know enroute to
some deeper goal that just seems like
it's a kind of magical thinking and this
is a kind of magical thinking that I
think does dog the field I think when we
encounter doubts in people even in
people who are doing this research that
everything we're talking about is a
genuine area of concern that there is an
alignment problem we're thinking about I
think there's this fundamental doubt
that mind is platform-independent
or substrate independent I think people
are imagining that yeah we can build
machines that will play chess we can
build machines that can learn to play
chess better than any person or any
machine even a single day but we're
never going to build general
intelligence because general
intelligence requires the wetware of a
human brain and it's just not going to
happen I don't think many people would
sign on the dotted line below that
statement but I think that is a kind of
mysticism that is presupposed by many of
the doubts that we encounter on this
topic


Ray Kurzweil: Future of Intelligence | MIT 6.S099: Artificial General Intelligence (AGI)

welcome to MIT course 6 s 0 9 9
artificial general intelligence today we
have Ray Kurzweil he is one of the
world's leading inventors thinkers and
futurists with a 30-year track record of
accurate predictions called the Restless
genius by The Wall Street Journal and
the ultimate thinking machine by Forbes
magazine he was selected as one of the
top entrepreneurs by Inc magazine which
described him as the rightful heir to
Thomas Edison PBS selected him as one of
the 16 revolutionaries who made America
Ray was the principal investigator of
the first ccd flatbed scanner the first
omni font optical character recognition
the first point to speech reading
machines for the blind the first
text-to-speech synthesizer the first
music synthesizer capable of creating
the grand piano and other orchestral
instruments and the first commercially
marketed large vocabulary speech
recognition among his many honors he
received a Grammy Award for outstanding
achievements in music technology he's
the recipient of the National Medal of
Technology was inducted into the
National Inventors Hall of Fame holds 21
honorary doctorates and honors from
three u.s. presidents Ray has written
five national best-selling books
including the New York Times bestsellers
The Singularity is near from 2005 and
how to create a mind from 2012 he is
co-founder and Chancellor of singularity
University and a director of engineering
at Google heading up a team developing
machine intelligence and natural
language understanding
please give ray a warm welcome
[Applause]
[Music]
it's good to be back I've been in this
lecture hall many times and walked the
infinite Carter I came here as an
undergraduate in 1965 within a year of
my being here they started a new major
called computer science it did not get
its own course number that's 6 1 even
biotechnology recently got its own
course number but how many of you are CS
majors ok how many of you do work in
deep learning how many of you have heard
of deep learning here I came here first
in 1952 when I was 14 I became excited
about artificial intelligence it had
only gotten its name six years earlier
the 1956 Dartmouth conference by Marvin
Minsky and John McCarthy so I wrote
Minsky a letter there was no email back
then and he invited me up he spent all
day with me as if he had nothing else to
do he was a consummate educator
I then and the AI field had already
bifurcated into two warring camps the
symbolic school which Minsk II was
associated with and the connectionist
school was not widely known in fact I
think it's still not widely known that
Minsk II actually invented the neural
net in 1953 but he had become negative
about it largely because there was a lot
of hype that these giant branes could
solve any problem
so the first popular neural nets the
perceptron was being promulgated by
Frank Rosenblatt at Cornell so Minsky
set out what are you going now and
saying I said to see Rosenblatt at core
now is that don't bother doing that and
I went there and Rosenblatt was touting
the perceptron that it ultimately would
be able to solve any problem so I
brought some printed letters that had
the camera and it did a perfect job of
recognizing them as long as they were
courier ten different types I didn't
work at all and he said but don't worry
we can take the output of the perceptron
or feed it as the input to another
perceptron and take the output of that
and feed it to a third layer and as we
add more layers it'll get smarter and
smarter and generalize and so that's
interesting if you even tried that well
no but it's high on our research agenda
things did not move quite as quickly
back then as they do now he died nine
years later never having tried that idea
turns out to be remarkably prescient I
mean he never tried multi-layer neural
nets and all the excitement we see now
about deep learning comes from a
combination of two things
both many layer neural Nets and the law
of accelerating returns which I'll get
to a little bit later which is basically
the exponential growth of computing so
that we can run these massive nets and
handle massive amounts of data it would
be decades before that idea was tried
several decades later three level neural
nets were tried there were a little bit
better they could deal with multiple
type styles still weren't very flexible
that's not hard to add other layers it's
a very straightforward concept there was
a math problem the disappearing gradient
or the exploding gradient which I'm sure
many of you are familiar with basically
you need to take maximum advantage of
the range of values in the gradients and
not let them explode or disappear and
lose the resolution that's a fairly
straightforward mathematical
transformation with that insight we
could now go 200 layer neural nets and
that's behind sort of all the fantastic
gains that we've seen recently
alphago trained on every online game and
then became a fair go player it then
trained itself by playing itself and
soared past the best human alphago zero
started with no human input at all
within hours of iteration sort Pascal
phago also soared past the best just
programs they had another innovation
basically you need to evaluate the
quality of the board at each point they
used another hundred layer neural nets
to do that evaluation so there's still a
problem in the field which is there's a
motto that life begins at a billion
examples
one of the reasons I'm at Google is we
have a billion examples for examples of
pictures of dogs and cats that are
labeled so you got a picture of a cat
and it says cat and then you can learn
from it and you need a lot of them
alphago trained on a million online
moves that's how many we had of master
games and that only created a sort of
fair go player a good amateur could
defeated so they worked around that in
the case of go by basically generating
an infinite amount of data by having the
system play itself had a chat with
Denver's house office you know what kind
of situations can you do that with you
have to have some way of simulating the
world so go or chess are even though go
is considered a difficult game it's
a-you know the definition of it can
exist on one page so you can simulate it
that applies to math I mean amass axioms
are can be contained on a page or two
it's not very complicated it gets more
difficult when you have real-life
situations like biology so we have
biological simulators but the simulators
on perfect so learning from the
simulators will only be as good as the
simulators that's actually the key to
being able to do deep learning on
biology
autonomous vehicles you need real-life
data so the way mo systems have gone
three and a half million miles
that's good that's enough data to then
create a very good simulator so the
simulator is really quite realistic
because they had a lot of real-world
experience and the they've got a billion
miles in the simulator but we don't
always have that opportunity to either
create the data or have the data around
humans can learn from a small number of
examples your significant other your
professor your boss your investor can
tell you something once or twice and you
might actually learn from that some
humans have been reported to do that
and that's kind of the remaining
advantage of humans now there's actually
no back propagation in the human brain
it doesn't use deep learning it uses a
different architecture that same year in
1962 I wrote a paper how I thought the
human brain worked there was actually
very little neuroscience to go on there
was one neuroscientist Vernon mount
Castle that had something relevant to
say which as he did I mean there was a
the common wisdom at the time and
there's still a lot of neuroscience that
says say this that we have all these
different regions of the brain they do
different things they must be different
there's v1 in the back of the head where
the optic nerve spills into that can
tell that that's a curved line that
that's a straight line does these simple
feature extractions on visual images
it's actually a large part of the
neocortex does the fusiform gyrus up
here which can recognize faces we know
that because if it gets knocked out
through injury or stroke people can't
recognize faces they will learn it again
with a different region of the neocortex
is the famous frontal cortex which does
language in poetry and music so these
must work on different principles he did
autopsies on the neocortex and all these
different regions and found they all
looked the same they had the same
repeating pattern same interconnections
he said neocortex is neocortex so I had
that hint otherwise I can actually
observe human brains in action which I
did from time to time and there's a lot
of hints that you can get that way for
example if I ask you to recite the
alphabet you actually don't do it from A
to Z you do it as a sequence of
sequences ABCD efg hijk so we learn
things that secret forward sequences of
sequences forward because if I ask you
to recite the alphabet backwards you
can't do it unless you learn that as a
new sequence so these are all
interesting hints I wrote a paper that I
that the neocortex is organized as a
hierarchy of modules in each module can
learn a simple pattern and that's how I
got to meet President Johnson and that
initiated a half-century of thinking
about this issue I came to MIT to study
with Marvin Minsky actually came for two
reasons one the Minsky became my mentor
which was a mentorship that lasted for
over 50 years the fact that MIT was so
advanced it actually had a computer
which the other colleges I considered
didn't have it was an IBM 7090 for 32 K
of 36 bit words so it's 150 K of course
storage to microsecond cycle time two
cycles for instructions or a quarter of
a myth and that thousands of students
and professors shared that one machine
in 2012 I wrote a book about this thesis
is now actually an explosion of
neuroscience evidence to support it the
European brain reverse engineering
project has identified a repeating
module about a hundred neurons it's
repeated three hundred million times
it's about 30 billion neurons in the
neocortex the neocortex is the outer
layer of the brain that's part where we
do our thinking and they can see in each
module axons coming in from another
module and then the output acts the
single output accent of that
Jil goes as the input to another module
so we can see it organized as a
hierarchy it's not a physical hierarchy
it's the hierarchy comes from these
connections the neocortex is a very thin
structure it's actually one module thick
there's six layers of neurons but it
constitutes one module and we can see
that it learns in simple pattern and
various reasons I cite in the book the
pattern recognition model that's using
is basically a hidden Markov model how
many of you have worked with Markov
models okay
that's usually no hands go open I asked
that question but Markov model is not it
is learned but it's not back propagation
it can learn local features so it's very
good for speech recognition and the
speech recognition network I did in the
80s used these Markov models that became
the standard approach because it can
deal with local variations so the fact
that a vowel is stretched you can learn
that in a Markov model it doesn't learn
long distance relationships that's
handled by the hierarchy and something
we don't fully understand yet is exactly
how the neocortex creates that hierarchy
but we have figured out how it can
connect this module to this module does
it then grow I mean there's no virtual
communication or wireless communication
it's actually connection so does it grow
an axon you know from one place to
another which could be inches apart
actually they all all these connections
are there from birth like the streets
and avenues of Manhattan there's
vertical and horizontal connections so
if the it decides and how it makes that
decision it's still not fully understood
that it wants to connect this module to
this module there's already a vertical
horizontal and a vertical connection it
just activates them we can actually see
that now and I can see that happening in
real time on non-invasive brain scans
so there's a current amount of evidence
that's in fact the neocortex is a
hierarchy of modules that can learn each
module learns a simple sequential
pattern and even though the patterns we
perceived don't seem like sequences they
may seem three-dimensional or even more
complicated they are in fact represented
as sequences but the complexity comes in
with the hierarchy so the neocortex
emerged 200 million years ago with
mammals all mammals have a neocortex
it's one of the distinguishing features
of mammals these first mammals were
small they were rodents but they were
capable a new type of thinking other
non-mammalian animals had fixed
behaviors but those fixed behaviors were
very well adapted for their ecological
niche but these new mammals could invent
a new behavior so creativity and
innovation was one feature of the
neocortex so a mouse is escaping a
predator its usual escape path is
blocked it will invent a new behavior to
deal with it probably wouldn't work but
if it did work it would remember it and
would have a new behavior and that
behavior could spread virally through
the community another Mouse watching
this was with say to itself that was
really clever going around that rock I'm
gonna remember to do that and it would
have a new behavior didn't help these
early mammals that much because as I say
the non-mammalian animals were very well
adapted to their niches and nothing much
happened for a hundred and thirty five
million years but then 65 million years
ago something did happened there was a
sudden violent change to the environment
we now call it the Cretaceous extinction
event there's been debate as to whether
it was a media or an asteroid I mean a
meteor or a volcanic eruption the
asteroid or meteor hypothesis is in the
ascendancy but if you dig down to an
area of rock reflecting 65 million years
ago the
geologists will explain that it shows a
very violent sudden change to the
environment we see it all around the
globe so is a worldwide phenomenon the
reason we call it an extinction event is
that's when the dinosaurs went extinct
that's when 75% of all the animal and
plant species went extinct and that's
when mammals overtook their ecological
niche so to anthropomorphize biological
evolution said to itself this neocortex
is pretty good stuff and it began to
grow it so-now mammals got bigger their
brains got bigger at an even faster pace
taking up a larger fraction of their
body the neocortex got bigger even
faster than that and developed these
curvatures that are distinctive of a
primate brain basically to increase its
surface area but if you stretched it out
the human neocortex is still a flat
structure it's about the size of a table
napkin just as thin and it's basically
created primates which became dominance
in their ecological niche then something
else happened two million years ago
biological evolution decided to increase
the neocortex further and increase the
size of the enclosure and basically
filled up the frontal cortex with our
big skulls with more neocortex and up
until recently it was felt that as I
said that this was the frontal cortex
was different because it does these
qualitatively different things but we
now realize that it's really just
additional neocortex so remember what we
did with it we're already doing a very
good job of being primates so we put it
at the top of the neocortical hierarchy
and we increased the size of the
hierarchy it was maybe 20% more
neocortex but it doubled it tripled the
number of levels because as you go up
the hierarchy it's kind of like a
pyramid there's fewer and fewer modules
and that was the enabling factor for us
to invent language and art music every
human culture we've ever discovered has
music no primary culture really has
music there's debate about that but it's
really true
invention technology technology required
another evolutionary adaptation which is
this humble appendage here no other
animal has that if you look at a chimp
and see it looks like they have a
similar hand but the thumb is actually
down here doesn't work very well if you
watch them trying to grab a stick so we
could imagine creative solutions yeah I
could take that branch and strip off the
leaves and put a point on it and we
could actually carry out these ideas and
create tools and then use tools to
create new tools and it started a whole
nother evolutionary process of
tool-making and that all came with the
with the neocortex
so Larry Page read my book in 2012 and
liked it so I met with him in Essen for
an investment in a company I'd started
actually a couple weeks earlier to
develop those ideas commercially because
that's how I went about things as a
serial entrepreneur
and said well we'll invest but let me
give you a better idea what you do it
here at Google we have a billion
pictures of dogs and cats and we've got
a lot of other data and lots of
computers and lots of talent all of
which is true and says well I don't know
I just started this company to develop
this is well by your company and how you
got a value a company that hasn't done
anything just started a couple weeks ago
and he said we can value anything so I
took my first job five years ago and
I've been basically applying this model
this hierarchical model to understanding
language which i think really is the
holy grail of AI I think Turing was
correct in designating basically text
communication as what we now call a
turing-complete problem that requires
there's no simple NLP tricks it you can
apply to pass a valid Turing test with
an emphasis on the word valid mitch
kapor and i had a six month debate on
what the rules should be because if you
read Turing's 1950 paper he describes
this in a few paragraphs and doesn't
really describe how to go about it but
if it's a valid Turing test meaning it's
really convincing you through an
interrogation and dialogue that it's a
human that requires a full range of
human intelligence and I think that test
has to the test of time we're making
very good progress on that I mean just
last week you may have read that two
systems
asked paragraph comprehension test it's
really very impressive winning came to
Google we were trying to past these
paragraph comprehension tests we aced
the first the first grade test second
grade tests were kind of got average
performance and the third grade test had
too much inference already you had to
know some common-sense knowledge as it's
called and make implications of things
that were in different parts of the
paragraph and there's too much inference
and it really didn't didn't work so this
is now adult level it's just slightly
surpassed average human performance but
we've seen that once something an AI
does something it average human levels
it doesn't take long for it to soar past
average human levels I think it'll take
longer in language and it did in some
simple games like go but it's actually
very impressive that it surpasses now
average human performance used at LST M
long short temporal memory but if you
look at the adult test in order to
answer these questions it has to put
together inferences and implications of
several different things in the
paragraph with some common sense
knowledge is not explicitly stated so
that's I think a pretty impressive
milestone so I I've been developing I've
got a team of about 45 people and we've
been developing this hierarchical model
we don't use Markov models because we
can use deep learning for each module
and so we create an embedding for each
word and we create an embedding for each
sentence this is we have a I can talk
about it because we have a published
paper on it it can take into
consideration context
if you use smart reply on G confused
email on your phone you'll see it gives
you three suggestions for responses
that's called Smart reply there are
simple suggestions but it has to
actually understand perhaps a
complicated email and the quality of the
suggestions is really quite good quite
on point that's for my team using this
kind of hierarchical model so instead of
Markov models that uses embeddings
because we can use back propagation we
might as well use it but I think what's
missing from deep learning is this
hierarchical aspect of understanding
because the world is hierarchical that's
why evolution developed a hierarchical
brain structure to understand the
natural hierarchy in the world
and there are several problems with big
deep neural nets one is the fact that
you really do need a billion examples
and we don't sometimes we can generate
them it's in the case of NGO or if we
have a really good simulator as in the
case of autonomous vehicles not quite
the case yet in biology very often you
don't have a billion example if you
suddenly have billions of examples of
language but they're not annotated and
how would you annotate it anyway with
more language that we can't understand
in the first place so that's kind of a
chicken and an egg problem so I believe
this hierarchical structures needed
another criticism of deep neural Nets
they don't explain themselves very well
it's a big black box that gives you
pretty remarkable answers I mean in the
case of these games demos described it's
playing in both go and chess is almost
an alien intelligence because we do
things that were shocking to you and
experts like sacrificing a queen and a
bishop at the same time or in close
succession which shocked everybody but
then went on to win or early in a go
game putting a piece at the corner of
the board which is kind of crazy to most
experts because you really want to start
controlling territory and yet it on
reflection that was the brilliant move
that enabled it to win that game but it
doesn't really explain how it does these
things so if yeah if you have a
hierarchy it's much better at explaining
it because you could look at the content
of the of the modules in the hierarchy
and they'll explain what they're doing
and just and on the first application of
applying this to health and medicine
this will get into high gear and we're
going to really see us break out at the
linear extension to longevity that we've
experienced I believe we're only about a
decade away from longevity escape
velocity we're adding more time than is
going by not just the infant life
expectancy but to your remaining life
expectancy I think if someone is
diligent they can be there already I
think I've
at longevity escape velocity now a word
on what life expectancy means it used to
be assumed that not much would happen so
whatever your life expectancy is with or
without scientific progress it really
didn't matter now it matters a lot so
life expectancy really means you know
how long would you live what's the in
terms of a statistical likelihood if
there were not continued scientific
progress but that's a very inaccurate
assumption that scientific progress
is extremely rapid I mean just as an AI
in biotech there are advances now every
week is quite stunning
now you can have a computed life
expectancies let's say 30 years 50 years
70 years from now you can still be hit
by the proverbial bus tomorrow we're
working on that with self-driving
vehicles but we'll get we'll get to a
point I think if you're diligent you can
be there now in terms of basically
advancing your own statistical life
expectancy
at least to keep pace with the passage
of time I think it would be there for
most of the population at least if
they're diligent within about a decade
so if we can hang in there we may get to
see the remarkable century ahead thank
you very much no question please raise
your hand we'll get your mic hi
so you mentioned both neural neural
network models and symbolic models and I
was wondering how far have you been
thinking about combining these two
approaches creating a symbiosis between
neural models and symbolic ones I don't
think we want to use symbolic models as
they've been used how many are familiar
with the psych project
that was a very diligent effort in Texas
to define all of common-sense reasoning
and it kind of collapsed on itself and
became impossible to debug because you
fix one thing and it break three other
things that complexity ceiling has
become typical of of trying to define
things through logical rules now it does
seem that humans can understand logical
rules we have logical rules written down
for things like law and game playing and
so on but you can actually define a
connectionist system to have such a high
reliability on a certain type of action
that it looks like it's a symbolic rule
even though it's represented in a
connectionist way and connection systems
can both capture the soft edges because
many things in life are not sharply
defined they can also generate
exceptions so you you don't want to
sacrifice your queen in chess accept
certain situations that might be a good
idea so you can capture that kind of
complexity so we do want to be able to
learn from accumulated human wisdom that
looks like it's symbolic but I think
we'll do it with a connection system but
again I'm think the connection systems
should develop a sense of hierarchy and
not just be one big massive neural net
so I understand how we want you know use
the neocortex to extract useful stuff
and commercialize that but I'm wondering
how you know our middle brain and organs
that are below the neocortex will be
useful for you know turn that into what
you want to do something well the
cerebellum is an interesting case in
point it actually has more neurons than
the neocortex and it's used to
govern most of our behavior some things
if you write a signature that's actually
controlled by the cerebellum so a simple
sequence is stored in the cerebellum but
there's not many reasoning to it it's
basically a script and most of our
movement now has actually been migrated
from the center vellum to the neocortex
cerebellum is still there some people
the entire cerebellum is destroyed
through disease they still function
fairly normally their movement might be
a little erratic as our movements is
largely controlled by the neocortex but
some of the subtlety is a kind of
pre-programmed script and so they'll
look a little clumsy but they're
actually function okay a lot of other
areas of the brain control autonomic
functions like breathing and but our
thinking really is is controlled by the
neocortex in terms of mastering
intelligence I think the neocortex is
the brain region we want to study I'm
curious what you think might happen
after the singularity is reached in
terms of this exponential growth of
information yes do you think it will
continue or will there be a whole
paradigm shift what do you predict well
in the singularities near I talked about
the atomic limits based on molecular
computing as we understand it and it can
actually go well past 2045 and actually
go to trillions of trillions of times
greater computational capacity than we
have today
so I don't see that's stopping anytime
soon and we'll go you know way beyond
what we can imagine and it becomes an
interesting discussion what the impact
on human civilization will be so take it
may be slightly more mundane issue that
comes up as a kind of eliminates most
jobs or
jobs a point I make is it's not the
first time in human history you've done
that how many jobs circa 1900 exist
today and that was the feeling of the
Luddites which was an actual society
that formed in 1800 the automation of
the textile industry in England they
looked at all these jobs going away and
felt that employment is going to be just
limited to an elite indeed those jobs
didn't go away but new jobs were created
so if I were oppression Futures to 1900
I would say well 38% of you work on
farms and 25% work in factories it's 2/3
of the working force but I predict by
2015 115 years from now it's going to be
2% on farms and 9% factories and
everybody would go oh my God we're gonna
be out of work and I said well don't
worry for all these jobs we eliminate
through automation we're gonna invent
new jobs and say oh really what new jobs
and I'd say well I don't know we haven't
invented them yet that's the political
problem we could see jobs very clearly
going away fairly soon like driving a
car or truck and the new jobs haven't
been invented I mean just look at the
last five or six years as many a lot of
the increase in employment has been
through mobile app related types of ways
of making money that just weren't
contemplated even six years ago if I
really prescient I would say well you're
gonna get jobs creating mobile apps and
websites and doing data analytics and
self-driving cars cars what's a car and
nobody would have any idea what I'm
talking about now the new job
some people say yeah we created new jobs
but it's not as many actually we've gone
from 24 million jobs in nineteen hundred
242 million jobs today for 30 percent of
the population to forty five percent of
the population the new jobs pay eleven
times as much in constant dollars and
they're more interesting and as I talk
to people starting out their career now
they really want a career that gives
them some
life definition and purpose and
gratification we're moving up Maslow's
hierarchy hundred years ago you were
happy if you had a back-breaking job to
put food on your family's table so and
we couldn't do these new jobs without
enhancing our intelligence so we've been
doing that well for most of the last 100
years through education we've expanded
to K through 12 and constant dollars
tenfold
we've gone from 38,000 college students
in 1870 to 15 million today more
recently we have brain extenders and not
yet connected directly in our brain but
they're very close at hand when I was
here that my tía to take my bicycle
across campus to get to the computer and
show an ID to get in the building now we
carry them well you know in our in our
pockets and on our belts
they're going to go inside our bodies
and brains I think that's a notic really
important distinction but so we're
basically going to be continuing to
enhance our capability through merging
with AI and that's the I think ultimate
answer to the kind of dystopian view we
see in futures movies where it's the AI
versus a brave band of humans for
control of humanity we don't have one or
two a eyes in the world today we have
several billion three billion
smartphones and last count will be six
billion in just a couple of years
according to the projections so we're
already deeply integrated with this and
I think that's going to continue and
it's gonna continue to do things that
you can't even imagine today just as we
are doing today things we couldn't
imagine you know even twenty years ago
you showed many graphs that goes through
exponential growth but I haven't seen
one that isn't so I would be very
interested in hearing you haven't seen
that what that is not exponential so
tell me about regions that you've
investigated that have not seen
exponential growth and why do you think
that's the case well
price performance and capacity of
information technology invariably
follows a exponential when it impacts
human society it can be linear so for
example the growth of democracy has been
linear but still pretty steady you can
count the number of democracies on the
fingers of one hand a century ago two
centuries ago you can count the number
of democracies in the world on the
fingers of one finger now there are
dozens of them that this and it's become
kind of a consensus that that's how we
should be governed
so the and I attributed all this to the
growth and information technology
communication in particular for
progression of social cultural
institutions but information technology
because it ultimately depends on a
vanishingly small energy and material
requirement grows exponentially and will
for a long time there's recently a
criticism that well test scores have
it's actually a remarkably straight
linear progression so humans think it's
like twenty eight hundred and it just
sort passed out in 1997 with the blue
and it's kept going and remarkably
straight and saying well this is linear
not exponential but the chess score is a
logarithmic measurement so it really is
exponential progression so if you're
lhasa furs like to think a lot about the
meaning of things especially in the 20th
century so for instance Martin Heidegger
gave a couple of speeches and lectures
on the relationship of human society to
technology and he particularly
distinguished between the mode of
thinking which is calculating thinking
and a mode of thinking which is
reflective thinking or meditative
thinking and he posed this question what
is the the meaning and purpose of
technological development and he
couldn't find an answer he he
recommended to remain open to what he
called and he called this an openness to
the mystery I wonder whether you have
any thoughts on this is there is there a
meaning of purpose to technological
equipment and and is there a way for a
human success access that meaning well
we started using technology to shore up
weaknesses and our own capabilities so
physically I mean who here could build
this building so we've leveraged the
power of our muscles with machines
and we're in fact very bad at doing
things that you know the simplest
computers can do like factor numbers or
even just multiply two eight digit
numbers computers can do that trivially
we can't do it so we originally started
using computers to make up for that
weakness I think the essence of what
I've been writing about is to master the
unique strengths of humanity creating
loving expressions in poetry and music
and the kinds of things we associate
with the better qualities of humanity
with machines that's the to promise of
AI that we're not there yet but we're
making pretty stunning progress just in
the last year there's so many milestones
that are really significant including in
language and but I think of technology
as an expression of humanity it's part
of who we are and the human species is
already a biological technological
civilization and it's part of who we are
an AI is it's part of humans so AI is
human and it's it's part of the
technological expression of humanity and
we use technology to extend our reach
you know I couldn't reach that fruit at
that higher branch a thousand years ago
so we invented a tool to extend our
physical reach we now extend our mental
reach we can access all of human
knowledge with a few keystrokes and
we're going to make ourselves literally
smarter by merging with AI hi
first of all honor to hear you speak
here so I first read The Singularity is
near nine years ago or so and it changed
the way I thought entirely but something
I think it caused me to over steeply
discount was tail risk in geopolitics in
systems that span the entire globe and
my concern is that there are there is
obviously the possibility of tail risk
existential level events swamp in all of
these trends that are otherwise war
proof climate proof you name it so my
question for you is what steps do you
think we can take in designing
engineered systems in designing social
and economic institutions to kind of
minimize our exposure to these tail
risks and and and survive to make it to
UM you know a beautiful mind filled
future yeah well the world was first
introduced to a human-made
existential risk when I was in
elementary school we would have these
civil defense drills to get under our
desk and put our hands behind our head
to protect this from a thermonuclear war
and it worked we made it through but
that was really the first introduction
to an existential risk and those weapons
are still there by the way and they're
still on a hair-trigger and they don't
get that much attention there's been a
lot of discussion much of which I've
been in the forefront of initiating the
existential risks of what sometimes
referred to as GN rg4 genetics which is
biotechnology and for nanotechnology and
gray goo robotics which is a
and I've been accused of being an
optimist I think you have to be an
optimist to be an entrepreneur if you
knew all the problems you were going to
encounter you'd never start any project
but I've written a lot about the
downsides I remain optimistic there are
specific paradigms and not foolproof
that we can follow to keep these
technologies safe so for example over 40
years ago some visionaries recognized
the revolutionary potential both for
promise and peril of biotechnology
neither the promise no peril was
feasible 40 years ago but they had a
conference at the Asilomar conference
center in California and to develop both
professional ethics and strategies to
keep biotechnology safe and they've been
known as the Asilomar guidelines they've
been refined through successive sell
more conferences much of that's baked
into law and it in my opinion it's
worked quite well we're now as I
mentioned getting profound benefit it's
a trickle today it'll be a flood over
the next decade and the number of people
who have been harmed either through
intentional or accidental abuse of
biotechnology so far zero actually I
take that back there was one boy who
died in gene therapy trials but 12 years
ago and there's congressional hearings
and they cancelled all research for gene
therapy for a number of years you could
do an interesting master's thesis and
demonstrate that you know 300,000 people
died as a result of that delay but you
can't name them they can't go on CNN so
we don't know who they are but it has to
do with the balancing of risk but in
large measure virtually no one has been
hurt by biotechnology now that doesn't
mean you can cross on our front list
okay we took care of that one because
the technology keeps getting more
sophisticated and Christopher's great
opportunity there's hundreds of trials
of Christopher's technologies overcome
disease but it could be abused you can
describe scenarios so we have to keep
reinventing it January we had our first
Asilomar conference on AI ethics and so
I think this is a good paradigm it's not
foolproof I think the best way we can
assure a democratic future that includes
our ideas of Liberty is to practice that
in the world today because the future
world of the singularity which is a
merger of biological non-biological
intelligence it's not going to come from
Mars I mean it's going to emerge from
our society today so if we practice
these ideals today it's going to have a
higher chance of us practicing them as
we get more enhanced with technology if
that doesn't sound like a foolproof
solution it isn't but I think that's the
best approach in terms of technological
solutions
I mean AI is the most daunting you can
imagine there are technical solutions to
biotechnology and nanotechnology there's
really no subroutine you can put in your
AI software there will assure that it
remains safe intelligence it's
inherently not controllable
there's some AI that's much smarter than
you that's out for your destruction the
best way to deal with that is not to get
in that situation in the first place if
if you are in that situation and find
some AI that will be on your side but
basically it's going to eyeb Aleve
we have been headed through technology
to event to a better reality look around
the world and people really think things
are getting worse and I think that's
because our information about what's
wrong with the world is getting
exponentially better I say oh this is
the most peaceful time on you in history
if you say what are you crazy didn't you
hear about the event yesterday and last
week and well a hundred years ago there
could be a battle that wiped out the
next village in you wouldn't even hear
about it for months
of all these graphs on education and
literacy has gone from like 10% to 90%
over a century and health wealth
poverty's declined 95% in Asia over the
last 25 years document about the World
Bank all these trends are very smoothly
getting better and everybody thinks
things are getting worse but but but
you're right like on violence that curve
could be quite disrupted there's an
existential event as I say I'm
optimistic but I think that is something
if we need to deal with that a lot of it
is not technological it's dealing with
our social cultural institutions so you
mentioned also exponential growth of
software and IDs I guess related to
software so one of the reasons for which
you said that all that information
technology costs this exponential is
because of fundamental properties of
matter and energy but in the case of
ideas why would it have to be
exponential well a lot of ideas produce
exponential gains they don't increase
performance linearly there's actually
study during the Obama administration by
his scientific advisory board on
assessing this question how much gains
on 23 classical engineering problems
were gained through hardware
improvements over the last decade and
software improvements and there's about
a thousand to one improvement it's about
doubling every year from Hardware there
was an averages of like twenty six
thousand to one through softer
improvements algorithmic improvements so
we do see both and apparently if you
come up with in advance its it doubles
the performance or multiplies it by ten
we see basically exponential growth from
each innovation
so and we certainly see that in deep
learning the architectures are getting
better
while we also have more data and more
computation and more memory to throw in
these at these algorithms
thank you for being


AI, Ethics, and the Value Alignment Problem with Meia Chita-Tegmark and Lucas Perry

[Music]
i'm erielle khan with the future of life
Institute and I'm excited to have FL
eyes Lucas Perry and Mei Akita tegmark
with me today to talk about AI ethics
and more specifically the value
alignment problem but first if you've
been enjoying our podcasts please take a
moment to subscribe and like this
podcast you can find us on iTunes
SoundCloud Google Play and all of the
other major podcast platforms and now ai
ethics and the value alignment problem
so first consider the statement I
believe that harming animals is bad now
that statement can mean something very
different to a vegetarian than it does
to an omnivore both people can honestly
say that they don't want to harm animals
but how they define harm is likely very
different and these types of differences
in values are common between countries
and cultures and even just between
individuals within the same town and
then we want to throw AI into the mix
how can we train a eyes to respond
ethically to situations when the people
involved still can't come to an
agreement about what an ethical response
should be but the problem is even more
complicated because often we don't even
know what we really want for ourselves
let alone how to ask an AI to help us
get what we want and as we've learned
with stories like that of King Midas we
need to be really careful what we ask
for that is when King Midas asked the
genie to turn everything to gold
he didn't really want everything like
his daughter and his food turned to gold
and we would prefer that an AI we
designed recognized that there's often
implied meaning in what we say even if
we don't say something explicitly for
example if we jump into an autonomous
car and ask it to drive us to the
airport as fast as possible implicit in
that request is the assumption that
while we might be okay with some
moderate speeding we intend for the car
to still follow most rules of the road
and not drive so fast as to put anyone's
life in danger or take illegal routes
that is when we say as fast as possible
we mean within the rules of law not
within the laws of physics
and these examples are just the tiniest
tip of the iceberg given that I didn't
even mention artificial general
intelligence and how that can be
developed such that its goals aligned
with our values so as I mentioned a few
minutes ago I'm really excited to have
Lucas and Mayer joining me today
maiya is a co-founder of the future of
life Institute she is interested in how
social sciences can contribute to
keeping AI beneficial and her background
is in social psychology
Lucas works on AI and nuclear weapons
risk related projects at FLI his
background is in philosophy with a focus
on ethics man Lucas thanks for joining
us today it's a pleasure thank you
thanks for having us so before we get
into anything else one of the big topics
that comes up a lot when we talk about
AI and ethics is this concept of value
alignment and so I was hoping you could
both maybe talk just a minute about what
value alignment is and why it's
important to this question of AI and
ethics so a value alignment in my view
is bringing ai's goals actions
intentions and decision-making processes
in accordance with what humans deemed to
be the good or what we see is valuable
or what our ethics actually are so for
me from the point of view of psychology
of course I have to put the humans at
the center of my inquiry so from that
point of view value alignment you know
you can think about it also in terms of
humans relationships with other humans
but I think it's even more interesting
when you add artificial agents into the
mix because now you have an entity that
you know is so wildly different from
from humans yet we would like it to
embrace our goals and and our values in
order to keep it beneficial for us so I
think the question of value alignment is
very very central to to keeping AI
beneficial so just to expand on what I
said earlier the project of value
alignment is in the end creating
beneficial AI it's working on what it
means for something to be beneficial
what beneficial AI exactly entails and
then learning how to technically
instantiate that into machines and AI
systems and also
building the proper like social and
political context for that sort of
technical work to be done and for it to
be fulfilled and manifested in our
machines and NAIS
and so when when you're thinking of AI
and ethics is value alignment basically
synonymous just another way of saying AI
and ethics or is it a subset within this
big topic of AI in ethics
well I think that they have different
connotations if one's thinking about AI
ethics I think that one is tending to be
more so focused on on applied ethics and
normative ethics one might be thinking
about the application of AI systems and
algorithms and machine learning in
domains in the present day and in the
near future so one might think about
autumn ization and other sorts of things
I think that when one is thinking about
value alignment it's much more broader
and expands also into into meta ethics
and really sort of couches and frames
the problem of AI ethics as something
which happens over decades and which has
a tremendous impact I think that value
alignment has a much broader connotation
than what AI epics has traditionally had
I think it all depends on you know how
you define value alignment I think if
you take the very broad definition that
that Lukas has just proposed I think
that yes it's it's probably includes AI
epochs but you can also think a bit more
narrowly simply instantiating your own
values and in into AI systems and having
them adopt your goals in that case I
think there are other issues as well
because if you think about it from the
point of view of psychology for example
then it's not just about which values
get instantiated and how you do that how
you solve the technical problem but also
we know that that humans even if they
know what what goals they have and what
values they uphold it's very very hard
for them sometimes to actually act in
accordance to them because they have all
sorts of cognitive and emotional
affective limitations so in that case I
think you know value alignment is in
this narrow sense it's basically not
sufficient we also need to think about
AIS and applications of my eyes in terms
of how do they help us and how do they
make sure that we gain the cognitive
competencies that we need to be moral
beings and to be really what we should
be not just what we are right and I
guess just to expand on what I was just
saying I mean valuable I'm and I think
in a more traditional sense it sort of
it goes beyond or it's more expansive
and inclusive in the it's recognizing
like a different sort of problem than
like AI ethics alone has I think that
when one is thinking about value
alignment there are elements of thinking
about some what about machine ethics but
also about social political technical
and ethical issues surrounding the end
goal of eventually creating AGI whereas
AI epics can be more narrowly
interpreted just as as certain sorts of
like specific cases where AI is having
impact and implications in our lives in
like the next ten years
whereas value alignments really thinking
about the instantiation of ethics and
machines
and making machine systems that are
corrigible and robust and docile which
will create a world that we're all happy
about living in
okay so I think that actually is going
to flow really nicely into my next
question and that is at FLI we tend to
focus on existential risks and I was
hoping you could talk a little bit about
how issues of value alignment are
connected to the existential risks that
we concern ourselves with
right so we can think of AI systems as
being very powerful optimizers we can
imagine there being like a list of all
possible futures and what intelligence
is good for is for modeling the world
and then committing to and doing actions
which constrain the set of all possible
worlds to ones which are desirable
so intelligence is sort of the means by
which we get to an end and ethics is the
end towards which we strive so these are
how these two things really integral and
work together and how AI without ethics
makes no sense and how ethics without AI
or intelligence in general it also just
doesn't work so in terms of existential
risk there are possible futures that
intelligence can lead us to where earth
originating intelligent life no longer
exists either intentionally or by
accident and so value alignment sort of
fits in by constraining the set of all
possible futures by working on like
technical work by doing political and
social work and also work in ethics to
constrain the actions of AI systems such
the existential risks do not occur such
that by some sort of technical oversight
by some misalignment of values by some
misunderstanding of what we want the AI
generates an existential risk
so we should remember that Homo sapiens
represents an existential risk to itself
also we are creating nuclear weapons we
have more of them that we need so many
in fact that we could destroy the entire
planet with them and not to mention Homo
sapiens is also represented an
existential risk for all other species
the problem with AI is that we're
introducing in the makes a whole new
agent that is by definition supposed to
be more intelligent more powerful than
us and also autonomous so as Lucas
mentioned it's very important to think
through what kind of things and
abilities do we delegate to these AIS
and how can we make sure that they have
the survival and the flourishing of our
species in mind so I think this is where
value alignment comes in as a safeguard
against these very terrible and global
risks that we can imagine coming from AI
right and what makes doing that so
difficult is beyond beyond the technical
issue of just having a AI researchers
and a safety researchers knowing how to
just get AI systems to actually do what
we want without creating a universe of
paperclips there's also this terrible
social and political context in which
this is all happening where there is
really great game theoretic incentives
to be the first person to create
artificial general intelligence so in a
race to create AI a lot of these these
efforts that seem very obvious and
necessary could be cut in favor of more
raw power and I think that that's
probably one of the biggest risks for us
not succeeding in creating value line AI
okay so I right now it's predominantly
technical AI people who are considering
mostly technical AI problems how to
solve different different problems is
usually a you need a technical approach
for this but when it comes to things
like value alignment and ethics most of
the time I'm hearing people suggest that
we can't leave that up to just the
technical AI researchers and so I was
hoping you could talk a little bit about
who should be part of this discussion
why we need more people involved how we
can get more people involved stuff like
that
sure so maybe if I didn't just break the
problem down into what I view to be the
three different parts then talking about
it will make a little bit more sense so
we can break down the value alignment
problem into three separate parts the
first one is going to be the technical
issues the issues surrounding actually
creating artificial intelligence the
issues of ethics so the end towards
which we strive the the set of possible
futures which we would be happy in
living and then also there's there's the
governance and the coordination in the
international problem so we can sort of
view this as a problem of intelligence a
problem of agreeing on the end towards
which intelligence is driven towards and
also the political and social context in
which all of this happens and so so thus
far there's certainly been a a focus on
the technical issue so so there's been a
big rise in in the field of AI safety
and in attempts to generate beneficial
AI attempts at creating safe AGI and
mechanisms for avoiding reward hacking
and other sorts of things that happen
when systems are trying to optimize the
utility function the concrete problems
on AI safety paper has been really
important and sort of illustrates some
of these technical issues but even
between technical AI safety research and
ethics there's disagreement about
something like also like machine ethics
so like how important is machine ethics
where does machine ethics fit into
technical AI safety research like how
much time and energy should we put into
certain kinds of technical AI safety
research versus how much time and effort
should we put into issues in in
governance and coordination and
addressing the AI arms race issue and
how much of ethics do we really need to
solve so I think that there's a really
important and open question regarding
how do we apply and invest our limited
resources in sort of addressing these
three important cornerstones in value
alignment so the technical issue the
issues and ethics and then issues in
governance and coordination and how do
we optimize working on these issues
given the timeline that we have how much
resources should be put into each
and I think that's an open question and
yeah one that certainly needs to be
addressed more about about how we're
gonna move forward given limited
resources I do think though that you
know the focus so far has been so much
on the technical aspect you know as you
were saying because there are there are
other aspects to this problem that need
to be tackled and what I'd like to
emphasize is that we cannot solve the
problem if we don't pay attention to the
other aspect as well so I'm gonna try to
defend for example psychology here which
has been you know largely ignored I
think in the conversation so from the
point of view of psychology I think the
value alignment problem is double-fold
in a way it's about a triad of
interactions human AI other humans right
so we are extremely social animals we
interact a lot with other humans and we
need to align our goals and values with
theirs and psychology has focused a lot
on that we have very sophisticated set
of psychological mechanisms that allow
us to engage in very rich social
interactions but even so we don't always
get it right societies have created a
lot of suffering a lot of moral harm
injustice unfairness throughout the ages
so for example we are very ill prepared
by our own instincts and emotions to
deal with intergroup relations so that's
very hard now you know people coming
from the technical side they can say oh
you know we'll just gonna have a I learn
our preferences like em first
reinforcement learning is a proposal
that says you know that basically
explains how to keep humans in the loop
so it's a it's a proposal for
programming AI such that it gets its
reward not from achieving a goal but
from getting good feedback from a human
because it achieved the goal so the hope
is that this way I can be correctable
and can learn from human preferences but
as a psychologist I am I am intrigued
but I understand that this is actually
very hard are we humans even capable of
conveying the right information about
our preferences do we even have access
to them ourselves or is this all
happening at some sort of subconscious
level sometimes knowing what we want is
really hard and
how do we even choose between our own
competing preferences so this involves a
lot more you know sophisticated
abilities like impulse control executive
function etc so you know I think that if
we don't pay attention to that as well
in addition to to solving the technical
problem I think we are very likely to
not get it right so I'm going to want to
come back to this question of who should
be involved and how we can get more
people involved but one of the reasons
that I'm talking to both of you today is
because you actually have made some
steps in broadening this discussion
already and that you set up a workshop
that did bring together a
multidisciplinary team to talk about
value alignment and I was hoping you
could tell us a bit more about how that
workshop went what interesting insights
were gained that might have been
expressed during the workshop what you
got out of it why you think it's
important towards the discussion etc so
just to give a few facts about the
workshop the workshop took place in
December 2017 in Long Beach California
and we were very lucky to have two
wonderful partners and Corgan izing this
workshop the Bergeron Institute and the
Canadian Institute for Advanced Research
and the idea for the workshop was very
much to have a very interdisciplinary
conversation about about value alignment
and reframe it is not just a technical
problem but also one that involves
disciplines such as philosophy and
psychology political science and and so
on so we were very lucky actually to
have a fantastic group of people there
representing all these disciplines and
the conversation was was very lively and
and we we discussed topics all the way
from near-term considerations and an AI
and how we aligned AI to our goals and
also all the way to thinking about AGI
and even super intelligence so it was a
fascinating range both of topics
discussed and also the perspectives
being represented
so my inspiration for the workshop was
being really interested in ethics and
and the end towards which this is all
going you know what really is the point
of creating AGI and perhaps even
eventually superintelligence what is it
that is good and what is it that is
valuable and broadening from that and
becoming more interested in value
alignment the the conversation thus far
has been primarily understood as
something that is this purely technical
so value alignment has only been seen as
something that is for technical AI
safety researchers to work on because
there there are technical issues
regarding AI safety and and how you get
a eyes to do really simple things
without destroying the world or ruining
the million other things that we care
about but this is really as we discussed
earlier an interdependent issue that
covers issues in meta ethics and
normative ethics applied ethics covers
issues in psychology it covers issue in
law policy governance coordination that
covers the AI arms race issue solving
the value alignment problem and creating
a future with beneficial AI is just it's
a civilizational project where we need
everyone working on all these different
issues on issues of value on issues of
like game theory among countries on on
the technical issues obviously and so
what I would really wanted to do was I
wanted to start this workshop in order
to broaden the discussion to reframe
value alignment as not just something
than in technical AI safety research but
something that really needs voices from
from all disciplines and all expertise
in order to have a really robust
conversation that reflects the
interdependent nature of the issue and
where different sorts of expertise on
the different parts of the issue can
really come together and and work on it
and is there anything specific that you
can tell us about what came out of the
workshop like were there any comments
that you thought were especially
insightful or ideas that you think are
important for people to be considering
I mean I think that for me one of the
takeaways from the workshop is that
there's still a mountain of work to do
and that there are a ton of open
questions and that this is a very very
difficult issue I think the one thing I
took away from the workshop was we
couldn't even agree on the minimal
conditions for which it would be okay to
safely deploy a GI
they're just issues that seem extremely
trivial in value alignment from the
technical side and from the ethical side
that seem very trivial but on which I
think there is very little understanding
or agreement right now
I think the workshop was a start and one
good thing that happened during the
workshop is I felt that the different
disciplines or rather their
representatives were able to sort of air
out their frustrations and also express
their expectations of the others so I
remember this quite iconic moment when
one roboticists simply said but I really
want you Attucks people to just tell me
what to implement in my system what do
you want my system to do so I think that
was that was actually very illustrative
of what Lukas was saying the need for
more joint work and I think you know
there was a lot of a lot of expectations
I think from both the technical people
towards the ethicists but also from the
ethicists in terms of like you know what
are you doing explain to us what are the
actual ethical issues that that you
think you are facing with the things
that you are building because I think
there's a lot of catching up to do on
both sides and there's much work to be
done in terms of making these
connections and bridging the gaps
and so you referred to this is sort of a
first step or an initial step what would
you like to see happen next I I don't
have any really concrete or specific
ideas for what exactly should happen
next I think that's a really difficult
question certainly like things that most
people would want or expect like I think
that in the general literature and
conversations that we were having I
think the value alignment as a word and
as something that we understand needs to
be expanded outside of the technical
context I don't think that is expanded
that far I think the more ethicists and
more moral psychologists and and people
in law policy and governance need to
come in and need to work on this issue
I'd like to see more like coordinated
collaborations specifically involving
like interdisciplinary crowds like
informing each other and addressing
issues and identifying issues and really
some sorts of like formal mechanisms for
interdisciplinary coordination on value
alignment it would be really great if
people in technical research and
technical AAS a few research and an
ethics and governance could also like
identify all of the issues in their own
fields which the resolution to those
issues and the solution to those issues
requires answers from other fields so
for example inverse reinforcement
learning is something that male was
talking about earlier and I think that's
something that we can clearly decide and
see as being interdependent on a ton of
issues in law and also in ethics and
ended value theory and so that would be
sort of like an issue or like node in
the landscape of all issues and
technical AI safety research that that
would be something that is
interdisciplinary so I think it would be
super awesome if everyone from their own
respective fields are really able to
identify the the core issues which are
interdisciplinary and able to dissect
them into the constituent components and
and sort of divide them among the
disciplines and work together on them
and identify like the different
timelines at which different issues need
to be worked on and yeah it's just
coordinating on those things
and then solo because you talked a
little bit about nodes and a landscape
but I don't think we've explicitly
pointed out that you did create a
landscape of value alignment research so
far can you talk a little bit about what
that is and how people can use it
yeah for sure so with the help of other
colleagues at the future of life
Institute like jessica cousins and
richard mala we've gone ahead and
created a value alignment conceptual
landscape and so what this is is it's
like a really big tree almost like an
evolutionary tree that you would see but
what it is is a conceptual mapping and
landscape of the value alignment problem
and what it's broken down into are the
three constituent components which we
were talking about earlier which is the
technical issues the issues and
technically creating safe AI systems
issues and ethics breaking that down
into issues and meta ethics and
normative ethics and applied ethics and
like moral psychology and descriptive
ethics where we're trying to really
understand values what it means for
something to be valuable and what is the
end towards which intelligence will be
aimed at and then also the other last
section is governance so issues in
coordination and policy and law in
creating a world where AI safety
research can proceed and where there
aren't where we don't develop or allow
like a sort of winner-take-all scenario
to rush us towards the end and not
really have a final and safe solution
towards fully autonomous powerful
systems so what the the landscape here
does is it sort of outlines all the
different conceptual nodes in each of
these areas it lays out what all the
core concepts are how they're all
related it defines the concepts and also
gives descriptions about how the
concepts fit into each of these
different sections of ethics governance
and technical AI safety research and so
the hope here is that people from
different disciplines can come and see
the truly interdisciplinary nature of
the value alignment problem to see where
ethics and governance and the technical
AI safety research stuff all fits in
together
and how this altogether really forms I
think the essential corners of the value
alignment problem and it's also nice for
researchers and other persons to
understand the concepts and the
landscape of the other parts of this
problem
I think the for example technical AI
safety researchers probably don't know
much about meta ethics so they don't
spend too much time thinking about
normative ethics and I'm sure that
ethicists don't spend very much time
thinking about technical value alignment
and how inverse reinforcement learning
is actually done and what it what it
means to do robust human imitation in
machines and what are the actual
technical ethical mechanisms that are
going to go into AI systems and so I
think that this is like a step in sort
of laying out the conceptual landscape
in introducing people to each other's
concepts it's a nice visual way of
interacting with I think a lot of
information and sort of exploring all
these different really interesting nodes
that explore a lot of like very deep
profound moral issues very difficult and
interesting technical issues and and
issues in law policy and governance that
are really important and profound and
quite interesting so you've you've
referred to this as the value alignment
problem a couple of times and I'm
curious do you see this and I'd like
both you'd answer this do you see this
as a problem that can be solved or is
this something that we just always keep
working towards and it's going to
influence you know whatever the current
general consensus is will influence how
we're designing AI and possibly AGI but
it's not ever like okay now we've solved
the value of I'm a problem so make sense
I mean I think that I think that that
sort of question really depends on your
meta ethics right so if you think the
there are moral facts if you think that
the moral statements can be true or
false and aren't just sort of
subjectively dependent upon whatever our
current values and preferences
historically and evolutionarily it
accidentally happened to be then there
is an end towards which intelligence can
be aimed that would be objectively good
and which would be the end towards which
we would strive and in that case if we
had solved the technical issue and the
governance issue and we knew that there
was a there was a concrete end towards
which we would strive that was the
actual good then the value alignment
problem would be solved but if you don't
think that there is a concrete end a
concrete good something that is that is
objectively valuable across all agents
then the value alignment problem or
value alignment in general is just sort
of a is an ongoing process in evolution
and in terms of like the technical and
governance sides of those I think the
there's nothing in the laws of physics
or I think in computer science or in
game theory that says that we can't
solve those those parts of the problem
those ones seem intrinsically like they
can be solved that's nothing to say
about how easy or how hard it is to
solve those but whether or not there is
sort of like an end towards value
alignment I think depends on on
difficult questions in meta ethics and
whether something like moral arrow
theory is true where all moral
statements are simply false and that
morality is maybe sort of just like a
human invention and which has no real
answers or whose answers are all false
yeah I think that that's sort of the
crux of whether or not value alignment
can quote be solved because I think the
technical issues and the issues and
governance are things which can be which
are in principle able to be solved
and mayor I think that regardless of
whether there is an absolute end to this
problem or not there's a lot of work
that we need to do in between and I also
think that you know in order to even
achieve this this end we need more
intelligence but as we create more
intelligent agents again you know this
problem gets magnified so there is
always going to be a race between the
intelligence that we're creating and
making sure that it is beneficial and I
think at every step of the way the more
we increase the intelligence the more we
need to think about the broader
implications and I think in the end we
we should think of artificial
intelligence also not just as as a way
to amplify our own intelligence but also
as a way to amplify our moral competence
as well you know as a way to gain more
answers regarding ethics and what our
ultimate goals should be so I think that
the interesting questions that we can do
something about are some where I sort of
an in-between we will not have the the
answer before we are creating AI so we
always have to figure out a way to know
keep up with the development of
intelligence in terms of our development
of moral competence
Soumya I want to stick with you for just
a minute when we talked for the FLI end
of your podcast one of the things she
said you're looking forward to in 2018
is broadening this conversation
and so it's hoping you could talk a
little bit more about some of what you
would like to see happen this year in
terms of getting other people involved
in the conversation who would you like
to see taking more of an interest in
this
so I think that unfortunately especially
in academia we've sort of defined our
work so much around these things that we
call disciplines and I think we're now
faced with with problems that especially
in an AI that really are a very
interdisciplinary we cannot get the
answers from just one discipline so I
would I would actually like to see in
2018 more sort of for example funding
agencies proposing and creating funding
sources for interdisciplinary projects
the way it works especially in academia
so you propose grants to do very
disciplinary defined granting agencies
another thing that that would be
wonderful
you know to start happening is you know
our education system is also very much
defined and described around these
disciplines so I feel that for example
there's a lack of courses for example
that teach students and technical fields
things about ethics moral psychology
Social Sciences and so on the converse
is also true in social sciences and in
philosophy we hear very little about
advancements in artificial intelligence
and and what's new and what are the
problems that are there so I'd like to
see more of that I'd like to see more
courses like this developed I think a
friend of mine and I we've spent some
time thinking about how many courses are
there that have an interdisciplinary
nature and actually talk about the
societal impacts of AI and there's a
handful in the entire world I think we
counted about five or six of them so so
there's a shortage of that as well but
then also educating the general public I
think thinking about the implications of
AI and also the societal implications of
AI and also the value lineman problem is
something that's probably an easier for
the general public to grasp rather than
thinking about all the the technical
aspect of how to make it more powerful
or how to make it more intelligent
so I think there's a lot to be done in
educating
funding and also just just simply having
these these conversations I also very
much admire what Lucas has been has been
doing and I hope he will expand on it
creating this conceptual landscape so
that we have people from different
disciplines understanding their terms
their concepts you know each other's
theoretical frameworks with which they
work so I think all of this is valuable
and we need to start it won't be
completely fixed in 2018 I think but I
think it's a good good time to to work
towards these these goals okay and Lucas
is there anything that you wanted to add
about what you'd like to see happen this
year I mean yeah nothing else I think to
add on to what I said earlier obviously
we just need as many people from as many
disciplines working on this issue
because it's so important but just to go
back a little bit I was also really
liking what may have said about how AI
systems and intelligence can help us
with our ethics and and and with our
governance I think that that seems like
a really good way forward potentially if
as our AI systems grow more powerful and
their intelligence they're able to
inform us more so about our own ethics
and our own preferences and our own
values about our own biases and about
what sorts of values and moral systems
are really conducive to the thriving of
human civilization and what sorts of
morality is lead to sort of navigating
the space of all possible minds in in a
way that is truly beneficial so yeah I
guess we'll be excited to see more ways
in which intelligence and AI systems can
be deployed for really tackling the
question of what beneficial AI exactly
entails you know what is beneficial mean
like we all want beneficial AI but but
what is beneficial what does that mean
what does that mean for us in a world in
which no one can agree on what
beneficial exactly entails so yeah I'm
just excited to see how this is going to
work out how it's going to involve and
and and hopefully we'll have a lot more
people joining this
work on this issue
so your comment reminded me of a quote
that I read recently that I thought was
sort of interesting I've been reading
Paula Boddingtons book toward a code of
ethics for artificial intelligence and
this was this was actually funded at
least in part if not completely by FLI
grants but she she says it's worth
pointing out that if we need AI to help
us make moral decisions better this
casts doubt on the attempts to ensure
humans always retain control over AI I'm
wondering if you have any comments on
that
yeah I don't know I think it's sort of
like this it's a specific way of like
viewing the issue or it's a specific way
of viewing what AI systems are foreign
the sort of future that we want I mean
in the end is the best of all possible
futures a world in which human beings
ultimately retain full control over AI
systems I mean if a systems are our
autonomous and and if value alignment
actually succeeds then I would hope that
we created AI systems which are more
moral than we are AI systems which have
better ethics which are which are less
biased which are more rational and which
are more benevolent and compassionate
than we are if value alignment is able
to succeed and if we're able to create
autonomous intelligent systems of that
sort of caliber of ethics and
benevolence and intelligence then I'm
not really sure what the point is of
maintaining any any sort of meaningful
human control I agree with you Lucas
that if we do manage to create in this
case I think it would have to be
artificial general intelligence that is
more moral more beneficial you know more
compassionate than we are then they
should control it's probably not so
important but in the meantime I think
while we are sort of 10 occurring with
our divisional intelligence systems I
think the issue of control is is very
important yes because we wouldn't want
to
we wouldn't be want to cut out at the
loop too early before we've managed to
properly test the system make sure that
indeed it is doing what we intended to
do right and I think the in the process
of that that it requires a
of our own moral evolution something
which we humans are really bad and slow
at you know as the president of FLI max
tegmark likes to talk about he likes to
talk about the race between our growing
wisdom and the growing power of our
technology you know human beings are
really kind of bad at keeping our wisdom
and pace with the growing power of our
technology and if we sort of look at the
the moral evolution of our species we
can sort of see huge eras in which
things which were seen as normal and
mundane and innocuous like slavery or
the subjugation of women or other sorts
of things like that today we have issues
with factory farming and animal
suffering and income inequality and just
tons of people who are living and with
exorbitant wealth that doesn't really
create much utility for them whereas
there's tons of other people who are in
poverty and who are so starving to death
there are all sorts of things that we
can see in the past as being obviously
morally wrong under the present - yeah
and so so so then we can see that
obviously there must be things like that
today and we wonder ok so you know what
are the sorts of things today that we
see is innocuous and as a normal and as
mundane that the people of tomorrow as
William McCaskill says will see us as
moral monsters how are we moral monsters
today but we what we simply can't see it
and so as we create powerful intelligent
systems and we're working on our ethics
and we're trying to really converge on
constraining the set of all possible
worlds into ones which art which are
good and and which are valuable and
ethical it really demands a moral
evolution of ourselves that we sort of
have to figure out ways to catalyze and
like work on and move through I think
faster
did I thank you um so as you consider
attempts to solve the value alignment
problem what are you most worried about
either in terms of us solving it badly
or not quickly enough or something along
those lines
and what is giving you the most hope in
terms of us being able to address this
problem
I mean so I think just like technically
speaking ignoring the likelihood of this
like the worst of all possible outcomes
would be something like an s risk so an
s risk is a subset of X risks asterisk
stands for suffering risk and so this is
a sort of risk where by some sort of
value misalignment whether it be
intentional or much more likely
accidental some seemingly astronomical
amount of suffering is produced by
deploying a misaligned AI system the way
that this was sort of function is given
certain sorts of assumptions about the
philosophy of mind about consciousness
and machines if we understand
potentially consciousness and experience
to be substrate independent meaning if
consciousness can be instantiated in in
machine systems that that you don't just
need meet to be conscious but you need
something like integrated information or
information processing or computation or
something like that then the invention
of AI systems and super intelligence and
the spreading of intelligence which
which optimizes towards any sort of
arbitrary end it could potentially lead
to vast amounts of digital suffering
which would potentially arise
accidentally or through subroutines or
simulations which would be epistemic ly
useful but that involves a great amount
of suffering and that coupled with these
artificial intelligence systems running
on on silicon and iron and not on on
squishy wet human neurons would mean
that it would be running at digital
timescales and not biological timescales
and so there would be huge amplification
of the speed at which the suffering was
run so subjectively we might infer that
a second for a computer a simulated
person on a computer would be much
greater than that for a biological
person then we can sort of reflect that
these are the sorts of risks that
ernesta risk would be something that
would be really sort of sort of bad just
any sort of way that AI can be
misaligned and lead to a great amount of
suffering and there's a bunch of
different ways that this could happen
so something like an asterisk would be
something like super terrible but it's
not really clear how likely that would
be but yeah I think that beyond that
obviously like we're worried about
existential risk we're worried about
ways that this could hurt tail or
destroy the development of Earth
originating intelligent life and you
know ways that this really might happen
or I think most likely because of this
winner-take-all scenario you that you
have with a I we've had nuclear weapons
for for a very long time now and we're
super lucky that nothing bad has
happened but I think the human
civilization is really good at getting
stuck into like minimum equilibria where
like we get locked into these into these
sort of positions where it's not easy to
escape from so it's really not easy to
disarm and get out of the nuclear
weapons situation once it's once we've
discovered it and once we start to
develop I think more powerful and robust
AI systems and I think already that a
race towards AGI and towards more and
more powerful AI might be very very hard
to stop if we don't make significant
progress on that soon if we're not able
to get a ban on leith autonomous weapons
and if we're not able to introduce any
real global coordination and that we all
just start racing towards more powerful
systems that there might be a race
towards AGI which would cut corners on
safety and potentially make the
likelihood of a existential risk or
suffering risk more likely
are you hopeful for anything I mean yeah
like if we get it right then the next
billion billion years can be super
amazing right it's just kind of hard to
internalize that and think about that
it's really hard to say I think how
likely it is that we'll succeed in any
direction but um yeah I mean I'm hopeful
that if we succeed in value alignment
that the future can be unimaginably good
and man what's scary to me is that it
might be too easy to create intelligence
but there's nothing in the laws of
physics making it hard for us and thus I
think that it might happen too fast
evolution took a long time to figure out
how to make us intelligent but that was
probably just because it was trying to
optimize for things like energy
consumption and making us a certain size
so that's scary it's scary that it's
happening so fast and I'm particularly
scared that you know it might be easy to
crack general artificial intelligence I
keep asking max max but isn't there
anything in the laws of physics that
might like you know make it tricky you
know the answer of his answer and also
that of more physicists that I've been
discussing with is that no it doesn't
seem to be the case now what makes me
hopeful is that you know we are creating
this Stuart Russell likes to give this
example you know of a message from an
alien civilization alien intelligence
that says we will be arriving in 50
years and then he poses the question
what would you do when you prepare for
that but I think with artificial
intelligence it's different it's not
like it's arriving and it's a given and
it has a certain form or shape you know
that we cannot do anything about we are
actually creating artificial
intelligence and I think that's that's
what makes me hopeful that if we
actually research it right that if we
know think hard about what we want and
we we work hard at you know getting our
own act together first of all and then
also making sure that this
stays and is beneficial you know we have
a good chance to succeed now there will
be a lot of challenges in between you
know from very near-term
issues like Lukas was mentioning for
example autonomous weapons
you know weaponizing our our AI and
giving it the right to harm and kill
humans to other issues regarding income
inequality
you know enhanced by by technological
development and so on to down the road
you know how do we how do we make sure
that autonomous AI systems actually
adopt our goals but I do feel that it is
important to to try and it's important
to work at it and that's what I'm trying
to do and that's what I hope others will
join us in doing alright well thank you
both again for joining us today thanks
for having us thanks for having us this
is wonderful if you're interested in
learning more about the value of lineman
landscape that Lucas was talking about
please visit future of life org slash
value alignment map will also link to
this in the transcript for this podcast
and if you enjoyed this podcast please
subscribe give it a like and share it on
social media we'll be back again next
month with another conversation among
experts