Eliezer Yudkowsky "Friendly AI"

so this is also a new talk for me
because Nick Bostrom covered a lot of
the material that I would usually put
into a talk so I had to quickly scurry
off and design another talk that would
not overlap with his talk let's start
with a old tragedy of science that is a
case where some some scientists screwed
up
why don't predator prey populations
crash you would think that you know if
you if you ran a simulation or something
you would find that the population of
foxes wouldn't the rabbits would get out
of whack there'd be too many foxes not
enough rabbits they'd all starve now
back in the 1960s they had this
brilliant notion which is that if you
look at if you would partition the fox
population into groups then groups of
foxes that reproduce too much and eat
all available rabbits will starve and be
eliminated from the gene pool so by
selection on groups foxes will evolve to
restrain their own reproduction this was
an actual much talked-about theory
before the 1960s back when biologists
had when a large majority of
professional biologists had no idea how
evolutionary biology worked a bit really
foxes will evolve to restrain their own
reproduction okay that basically never
happens you don't have a gene for not
reproducing that becomes dominant in the
gene pool
specifically group selection although it
was a very popular sort of theory was
tremendously difficult to make work
mathematically in the night you know
before the 1960s before it's now called
the Williams Revolution people would
postulate all sorts of nice pro-social
group benefiting adaptations and they
would point to group selection as as
their theory for how evolution could
actually do that but it's mathematically
extremely difficult for group selection
to work
example is in simulation if the cost to
an organism of a gene is 3 percent of
its own fitness and it benefits its
neighbor so much that pure altruist
groups have doubled the reproductive
fitness of pure selfish groups and group
sizes twenty five and twenty percent of
all deaths are replaced by neighbors
from another group the result is
polymorphic for selfishness and altruism
that means that if the cost is five
percent of fitness or pure altruist
groups are less than twice as fit as
pure selfish groups then you do not get
pop then you would just get the selfish
gene the selfish gene would win if you
look at the statistics of a human
maternity hospital where around the same
number of boys and girls are born and
teach generation you can see at a glance
that individual selection pressures beat
group selection pressures on humans you
know you might imagine that if you had
more girls born and fewer boys that you
would have more mothers you'd be able to
reproduce faster but if you had an
equilibrium like that then by then any
gene for birthing more boys would be
able to make a genetic killing in the
next generation because boys would you
know each child would still have half of
its genes from the father half of the
genes from the mother the pool of boys
in the pool of girls would make roughly
equal genetic contributions to next
generation so if there are fewer boys
then you can make an individual genetic
killing by birthing more boys if there
are fewer girls you can make a genetic
killing by birthing more girls
regardless of what's good for the group
so you can just look at a glance at the
human maternity hospital and you can see
that the forces of individual selection
are far stronger than the forces of
group selection on humans and there is
no we would now say that there is no
known case of a group selected
adaptation in any mammal one might even
say no case known case of groups lefty
the adaptation period but that is
slightly more controversial part about
no mammal we know in point of fact why
don't predator prey populations crash
well when I was young my father said to
me why don't people on
Australia fall off the globe since
they're standing upside down and my
father said to me
aha it's a trick question actually
people fall off all the time just in
case you're wondering what kind of
family I came from and indeed this is a
wrong question the answer is predator
prey populations crashing all the time
but later on even though the
mathematically required conditions for
group selection are ridiculously extreme
a mad genius named Michael J Wade took a
laboratory population of tribolium
beetles and implemented the extreme
conditions needed for group selection
selection on groups of beetles for
reduced population that was so severe
that group selection could
mathematically overpower individual
selection so what do you think happened
when group selection this same force
that was once appealed to to talk about
all sorts of wonderful aesthetic humane
solutions like be nice and restraining
your own breeding this is what
biologists before the 1960s appealed to
group selection to was supposed to
produce so Michael J Wade took some
populations of beetles he eliminated all
the beetle populations that that grew
the fastest leaving only the Greenough
populations that grew the slowest what
do you think actually happened take a
moment to predict it but if you've
actually read about this before then
don't forget this group well yes you did
very well because of course the actual
solution group selection produces eat
their babies you know at your own babies
of course you go around and find the
other organisms babies and you eat them
especially the young girls of course
those are the tastiest nom
think of your brain as an engine that
searches for solutions ranking high in
your preference ordering using limited
computing power now if you if your tribe
was faced with the sort of population
problem that Michael J Wade presented
his beetles with or the Foxes had with
the rabbits then for you the solution
let's all have as many children
individually as possible than try to eat
each other's babies especially the girls
would rank so low in your preference
ordering that your brain would not even
suggest it like you wouldn't see that as
an option because your brain has limited
in computing power and there is no point
in generating solutions that weird in
that low in the preference ordering so
if you try to understand evolutionary
biology by putting yourself in evolution
shoes and asking how you would solve a
problem you won't even see evolutions
answer as a possibility you have to
learn to think like this non-human
optimization process which early
biologists didn't and indeed there as
much the Lemos energy literature and
evolutionary biology devoted to training
you to stop thinking like a human who
will come up with nice solutions and
start thinking like natural selection
and coming up with which genes are
actually likely to rise to universality
in the gene pool the relevance to AI of
course is that if you so for example
yesterday of Schmidt Schmidt Huber who
unfortunately is not here today gave us
a certain reward criterion and suggested
that it would be maximized by art and
you know having been through this sort
of conversation for quite a while now I
sort of squint at that and say okay so
art fits there but what else besides art
fits there how about sets of black boxes
encrypted using it by an external
mechanism using that using a key that
can be reliably recovered in five
minutes then you could achieve a lot of
compression that way
in general what sort of goes wrong with
your mind when you try to understand the
sort of nonhuman optimization process is
something like persuasive
rationalization like let's say I'm buy
I'm buying an entire chocolate cake
going to sit down to eat it you ask me
hey why are you buying an entire
chocolate cake I thought you were on a
diet oh because I want to help the sugar
industry helping the sugar industry is
very important what's wrong with this
claim over here what is wrong with the
claim that you are optimizing the
criterion of helping the sugar industry
by buying the chocolate cake well you
could just mail them a check and instead
of getting only the tiny portion of the
gains from from trades that are captured
by the sugar industry when they when you
know by the time you buy the cake and it
has been marked up by the cake
manufacturer in the store and so on you
could just mail them to check directly
so strategy acts in this case eating an
entire chocolate cake suggested by
criterion one in this case chocolate
cakes are tasty you try to justify that
by an appeal to criterion to helping the
sugar industry you can tell it's a
persuasive rationalization because just
mail them a check strategy why optimize
this criterion to help the sugar
industry it comes a lot closer to
maximizing criterion two so let me just
sort of skip a head over here and so in
the tragedy of group selection ISM there
was a biologist saw an ecological
problem of the limited rabbit supply
they imagined how they would solve that
problem by having the Foxes cooperate to
restrain their building now this was
really sort of suggested by their human
nice aesthetic sense of values and then
someone pointed out to them hey wait a
minute that is not how evolution works
but by this time they've gotten they're
emotionally attached to their lovely a
aesthetic view of nature foxes and
rabbits and harmony people would
actually say things like that but you
know before the 1960s and so they
invoked their brains persuasive
rationalization module now that they
decided to do that that just sort of
automatically when they were defending
their a mistake and sort of tried to
persuade evolution to see it their way
by appealing to evolutions own stated
moral principles of promoting
reproduction sort of like try to you
know hey they're natural selection you
know you could actually you know
immaculate you could get some
reproduction done by doing things my way
and having the foxes restrain their
breed and yeah but unfortunately
evolution itself does not start with
human a aesthetic preferences and then
figure out a way to rationalize how that
helps reproduction evolution is just the
process where the genes that build foxes
that produce more baby foxes become more
prevalent in the population so after you
are done sort of nudging the nonhuman
optimization process and claiming that
it odd to do nice things for your clever
reasons that your prediction fails
because the process itself does not work
that way and it's not within the and
your thing that you know sort of maybe
helps reproduction a bit didn't maximize
reproduction it wasn't didn't fit the
criterion as well as some other things
sort of like okay sure art maximizes
schmidhuber is quite partly art might
produce some reward for Schmidt humors
internal reward criterion but having a
sequence of black boxes that you
produced by an external device that
where you know you can recover the key
in five minutes will maximum what will
optimize it even more oh and let me uh a
little similar story that related story
might find entertaining so less wrong is
a community website devoted to man
rationality and it's got all these
lovely sequences on things like how to
actually change your mind sometimes that
this will become relevant at a moment I
promise so newcomer cluster on those hi
I've read through the sequences and I'm
an egoist well so of course he was like
immediately asked so if you could take
the pill that could numb your conscience
and no one would ever find out how many
babies would you kill for a dollar
because you see people who call
themselves egoist they may have this
sort of verbal philosophy of selfishness
but their actual options are being
generated by the full range of normal
human values and then they appealed to
selfishness in order to justify those
options that were generated by other
sources
they aren't actually egoist because if
they were actually eat lists of course
they would see nothing wrong with
killing babies for a dollar as long as
they didn't get found out oh and and the
response one day later okay I thought
about that for a while I'm not an egoist
anymore this only happens I'm less wrong
I don't think I've ever seen that happen
anywhere else on the Internet and so as
this person realized they never actually
used egoism to select their actions they
use normally human preferences to select
their actions but then justified them by
rationalizing to this single single
simple principle of egoism you get a lot
of people proposing single simple
principles that are all we need to build
into a friendly AI and everything will
be hunky-dory forever after and that's
because they use their full range of
human preferences to generate and select
their actions and then they justify them
by appealing to this principle which can
actually be maximized a lot more by
things like eating babies killing babies
for a dollar or having black boxes
encrypted by an external device so the
point people don't notice when simple
optimization criteria imply humanly ugly
choices because their own brains don't
generate the strategies for
consideration and think people think
they can pack all sorts of nice things
into simple principles because they make
their choices using fully land values
nay aesthetics and then rationalize
those choices by persuasively arguing to
that principle so this is another
thought experiment the reason why I
couldn't come up with an abstract for
this talk is that it's a series of
persuasive historical cases or thought
experiments and I just couldn't figure
out an abstract for that because I was
too busy finishing the talk so let's say
you're in a world where no one knows
what's really going on with
addition addition is one of the
mysteries but people and so of course
modern computer scientists are trying to
build artificial addition and they do
this with a logical addition device
which of course does anyone who has
familiar with symbolic AI will realize
will naturally contain all sorts of
price propositional logic statement
stating that the plus of seven and six
is thirteen and of course all these
little suggestively named Lisp tokens
over here are given their meaning by the
the larger semantic network in which
they are embedded and it turns out that
doing artificial addition this way is
you know very expensive and
time-consuming and they've only got
artificial adders that work up to the
number 60 and you know getting your
artificial adders all the way up to to
working with like general addition and
the ranges of thousands or millions you
know as humans can do is thought to be
decades away and so on and so you've got
all sorts of and by the way don't worry
this will be relevant to from the AI now
there all sorts of lovely comments about
this problem of artificial general
addition for example there's the view
that artificial general addition is
difficult because of the framing problem
what 21 plus is equal to depends on
whether it's plus 3 or plus 4 so you
need to program a huge network of
arithmetic Allah facts to cover
common-sense truths and then you'll get
artificial general addition or you need
an artificial general arithmetic that
can understand and roll language so
instead of being told that 21 plus 16
equals 37 it can obtain that knowledge
by reading the web or you need to
develop a general Aerith petition the
same way nature did evolution top-down
approaches have failed to produce
arithmetic we need a bottom-up approach
to make arithmetic emerge we must accept
the unpredictability of complex systems
neural networks just like the human
brain they can be trained without
understanding how they work neural
networks will do arithmetic without us
their creators ever understanding how
they add after actually you just need
calculators as powerful as the human
brain and Moore's law predicts that
Kappa laters these powerful B will
become available on April 27 2013 1
between 4:00 and 4:30
in the morning maybe that's not enough
maybe we've actually got to simulate the
detailed neural circuitry humans used
for addition or gödel's theorem shows no
formal system can never capture the
properties of arithmetic in the
classical physics is form Eliza Bowl so
hence an artificial general edit it edit
adder must exploit quantum gravity human
Emerson's think we're something off to
cetera et cetera haven't you ever heard
of Donna trills Chinese calculator
experiments see it doesn't really know
what the numbers mean probably will
never know the nature of arithmetic the
problem is just too hard for humans to
solve so I usually tell the story with
the moral that when you're missing a
basic insight you have to what you have
to do is actually actually understand
what's going on inside addition and
until you understand that you're screwed
you'll come up with all sorts of clever
workarounds and things that you can that
you can say about the problem and ways
that sound like they might solve it that
will sound clever even if you don't
quite know what you're doing but it's
actually impossible to talk sensibly
about solutions until you are no longer
confused it's quite important to
recognize what people sound like when
they start talking about something with
about which they are fundamentally
confused this is what they sound like
it's good to be able to recognize that
today is moral though I'm actually just
going to take a simpler moral which is
that if you have to put in a infinite
number of special cases it means you
didn't understand the underlying
generator of the behavior this world
will probably become relevant
so next thought experiment the outcome
pump you let's say you have a time
machine
what sort of fun machines can you do
with the time machine go back to
yesterday and throw up high it yourself
or achieve omnipotence so let's say that
you build a device which automatically
resets time unless some desired outcome
occur so in other words you just sort of
keep presenting time back to some
previous state until you get the outcome
you want now you have a physical genie
device time machine equals genie
why talk about a little time machine
reset device because if you talk about
genies who are tempted to think of them
as minds and anthropomorphize them and
assume that they would do it what they
would do what you would do in your shoes
we want to talk about a physical genie
the little time machine reset device
because it lets us talk about an
optimization process the resetar just
using language of physical things
without invoking mental entities and
that may help help us be a little less
anthropomorphise that's why I opened
with the example of natural selection
natural selection is a non human
non mental optimization process and that
makes it and that doesn't mean people
are don't down through morph eyes it but
it means you can give them very stern
looks when they do so you you take your
physical genie and you want to solve the
grandma extraction problem now if it's a
regular old genie you just say I wish
for my grandma to be outside the burning
house they you know there's a house on
fire your grandma's in it you want to
get it out that is the grandma
extraction problem but this we don't
have a genie we have a little time
machine device so first thing we've
we've got actually specifying some way
if it doesn't can't be assumed to
automatically understand what we want
we've got to describe what we want to
this physical genie that clearly can't
understand language because it's just a
plain old physical time machine so how
would you specify this goal
well you even if you don't can't
understand natural language you might
you know iPhones can take pictures we
have software that can understand
pictures we might hook up some kind of
scanner that can identify objects in its
vicinity you know by magic and Will's
will scan the photo of grandma's head
and shoulders we use object contiguity
to select grandma's whole body and we
will say that the probability of
resetting time decreases as grandma gets
further away from the center of the
house so this where I'm actually sort of
skipping over a bit of the background of
how I usually explain this like the idea
is that
you can specify a quantitative utility
function for this kind of physical
outcome pump by saying the higher the
utility the less likely you are to reset
and so you're more likely to end up with
an outcome with higher utility in any
case so in this case we're specifying
that the outcome pump is going to try to
select an outcome in which grandma
identified by object contiguity and a
photo is far away from the center of the
house
you have now told the genie to get
grandma out of your house so the gas
main under the building explodes
grandma's body is hurled into the air
thereby rapidly increasing her distance
from the former center of the house and
there's a little button on your time
machine that you're supposed to push if
something goes drastically wrong which
almost certainly resets it it's called
the regret button of course you never
experience pressing it because all those
probabilities have been wiped out but in
this case of flaming wooden beam drops
out of the sky and smashes you before
you can hit the emergency reset button
that's causing the time machine to think
everything's fine now if you were
talking about a actual genie a mental
genie a genie that understood what you
were saying you might be tempted to
blame it but as long as this is a
physical optimization process here
there's no point in blaming it for
anything any more than there's a point
in blaming natural selection for
producing baby-eating as the outcome of
group selection
you simply programmed in the wrong
utility function it's not the fault of
the time machine it's the fault of the
function you gave it to to maximize so
if this were a mental genie we would say
I wish for my grandma to be outside the
burning house and alive so you try to
write something into your time machine
outcome pump that recognizes whether
grandma's dead or alive and make sure
that she is breathing at the time she
exits the house and of course she ends
up in a coma
the open-source wish project tries to
devise to develop inescapably good who
wishes this is their version 1.1 of
their wish for immortality I wish to
live in locations with my choice in a
physically healthy uninjured and
apparently normal version of my current
body containing my current mental state
a body which will heal from all injuries
that are registered blah blah blah blah
blah blah blah blah blah now remember
the previous lesson about needing to
patch an infinite number of special
cases because you didn't understand
edition and therefore you had to program
in all the knowledge manually let's say
you were trying to build a chess AI mate
you could imagine that you would build a
test that by having a human look at a
bunch of chess positions rate whether
those chess positions are good or bad or
wait the or have the human look at a
bunch of chess positions and say what is
the best move in each chess position and
then program those chess positions and
their best moves into the chess playing
AI problem this of course is that there
are too many chess positions the key
insight that you need is that what makes
a move good is that it leads to a
certain range of board states that we
have designated as winning and that we
want that the eight the chess playing AI
can navigate to one of the board states
known as winning
and until you achieve the insight of
game trees you cannot build a chess
playing ai so Grandma extraction problem
we have cases like grandma is dead is
worse than grandma's alive but Burns is
worse than Grandma alive and healthy
suppose a wormhole opens and transport
grandma to a desert island while it's
better than her being dead it's worse
than her being alive well healthy on
traumatized in continual contact with
her social network is it ok to save
Ranma at the cost of a fireman's life at
the cost of the family dog's life at the
cost of two murderous lives is it worth
a point zero zero zero zero zero zero
zero one percent risk to Grandma to say
if the family dog would you destroy
every extant
copy of box little Fugen g-minor to save
your grandma what algorithm are you
using to decide all these cases how do
we capture the generator you're checking
the value of the distant consequences
and you're implicitly checking all the
components of your utility function you
people like the open source Wish project
are generating all the clauses of the
swish by imagining events with
negatively valued consequences and
ordering the genie don't do that this is
one of my favorite sentences ever
it's from William Francona's ethics and
I first encountered it in the Stanford
encyclopedia of philosophy article on
what things have terminal value what
things do we value in themselves and not
for their consequences life
consciousness and activity health and
strength pleasures and satisfactions of
all or certain kinds happiness buta tude
contentment truth knowledge true
opinions of various kinds understanding
wisdom beauty harmony proportion and
objects contemplated aesthetic
experience morally good decision
dispositions or virtues mutual affection
love fringe and cooperation boy you're
ever feeling down you can just read the
sentence over here and remember
everything that makes life worth living
if the genie is searching more pass
through time than you if you're Gili
with the genie that will be considering
options that you didn't imagine at the
start of the problem because you were
not smart enough to imagine them because
you did not search all the past through
time that the genie could take then no
wish that you make that genie is safe
unless the genie is itself is checking
all the consequences of any strategy it
considers using the entire utility
function if the genie has no component
in its utility function for music that
means that changes to music are value
neutral in the genies evaluation and
that mean that means that as far as the
Deenie is concerned if you can destroy
all copies of Bach's music to prevent
someone from breaking a leg
where it knows that breaking a leg is
bad but it has no component that's
utility function for the music that's
great that's fine
so this is the hidden complexity of
wishes that whenever you consider well
is that a good weight to wish something
the strategies to implement simple
sounding instrumental goals are chosen
using the full array of terminal values
that forbid negative consequences and
optimize positive consequences this goes
on in the background you're not you're
probably not even aware of consent of
ever consciously saying hmm should I
blow up the house in order to get
grandma out of it
no your brain doesn't even generate that
as an option it's too low in your
preference ordering but the reason it's
low in your preference ordering as a way
for getting grandma out of the house is
that you value grandma being alive you
weren't even aware of considering that
but it was a consideration that was
there the whole time and thinking about
this in as a purely physical device a
little time machine that resets time
unless some sort of physically specified
condition is achieved sort of reveals
why Time Machine like that an outcome
pump like that you might think it would
grant you am nipa tints it's actually
too dangerous to ever be used it
searches all the paths through time and
unless you can program william frank
anna's entire value list in there plus
everything for and kind of forgot to
talk about it's going to stomp on one of
your values in the course of
implementing what sound like perfectly
reasonable wishes bill Hibbard we can
design intelligent machines so their
primary innate emotion is unconditional
love for all humans first we build
relatively simple machines that learn to
recognize happiness and unhappiness and
human facial expressions human voices in
human body language then we can hardware
as a result of this learning of the
innate emotional values of more complex
intelligent machines positively
reinforcing we are happy and negatively
reinforced and we are unhappy trained
super intelligences so their reward
function is smiling humans
naturally if you actually tried this the
galaxy would end up tiled with tiny
molecular smiley faces
it just seems obvious what could he
possibly have been thinking Oh
incidentally the people who did the DNA
will likely relief face some of them had
read my work and like one of them
actually emailed me to say oh I produce
some tally of tiny molecular smiley
faces it has begun
this is the sort of friendly AI proposal
that you get when people use qualitative
physics to think about friendly a is so
qualitative physics is a psychological
study of a certain kind of reasoning so
for example you this diagram says that
if you increase the burner temperature
that will increase the amount of boiling
going on and if the amount of boiling
going on increases that changes the
derivative that decreases the derivative
of water in other words the water is
already boiling away but now it's going
to boil away more quickly because when
there's more boiling going on and this
says that you can turn on the burner to
get rid of water so presumably bill
Hibbard was thinking something along the
lines of happy people smile more smiling
reinforces the AI behavior the AI will
therefore do things that make people
happier happier people have more utility
therefore building a super intelligence
reinforced by human smiles is according
to this graph good and of course you
carry a very large category proposals
that one might term apple-pie AI which
go like this apple pie is good nuclear
weapons are bad all we need to do is
wish for as build a super intelligence
that will give us lots of apple pie and
not use any nuclear weapons no seriously
the you know that that thing that
Hibbert Rose was in a peer-reviewed
journal I hear this all the time nothing
remotely like this approach will ever
work ever and if I hear one more
proposal to build an AI that promotes
liberal democracy I'm going to scream
so the natural next think you might
think of is okay build a super
intelligence that is optimizing William
Franck Hanna's entire value list the
problem is when you get down to the
bottom of that list you get to power and
experiences of achievement
self-expression freedom peace security
adventure and novelty and good
reputation honor esteem etc that
etceteras that is the dangerous part
really
dialing nine tenths of my phone number
correctly does not reach nine tenths of
eliezer yudkowsky what happens if a
super intelligences utility function has
one wrong number it is missing one just
one little component of value the N
armed bandit problem says that you have
a number of slot machines in front of
you and you are trying to determine
which of these slot machines when I pull
the lever is gives me the highest
expected payoff the human solution of
course would be to pull your favorite
levers so far and occasionally get bored
with the known levers and try pulling
some new ones in general this is called
an exploration exploitation trade-off it
is the problem that boredom solves in
humans there's also in the Bayesian
optimal solution to an exploration
exploitation trade-off is to have a
prior over the one-armed bandits the
slot machines update your beliefs about
them based on your observations of the
payoffs they've delivered so far and
here's the thing if you know how long
you're going to have to how many polls
you're going to have on these bandits
then you're out any piece of information
you can obtain about them is more
valuable when you obtain it closer to
the start of the problem any piece of
information you obtain you get to use on
more future occasions if you obtain it
earlier rather than later so
occasionally going exploring is not an
optimal strategy for maximizing the
payoff from the one-armed bandits the
optimal strategy is to do all your
exploring first until the you have
updated your beliefs about the one-armed
bandits and the expected value of
information has dropped below the
threshold of the most rewarding band
that observes so far and then once you
have gathered all the information going
to gather you just pull the best lever
over and over until time runs out that
is the basement I met you can make an
error there well if that possible well
first of all if that if that possibility
exists in your prior for one thing okay
okay so if you have no idea which one it
is then you might as well pull the one
with the highest well sure then of
course well yes it it yeah indeed you
can have a case where you're pulling the
one that that's best then you get an
unexpected piece of information about
and then you start switching again so
yes that and that indeed can happen but
if that doesn't happen and the I I will
generally not expect that it will happen
at the time that it starts pulling the
bit the best lever will just pull the
best lever over and over again the point
is the the solution doesn't the the
exploration act at the exploration
exploitation problem that boredom solves
in humans human boredom is not the only
way to solve this problem and if you ask
what sort of convergent instrumental
value you're likely to get it's going to
be first to all your exploring find the
configuration that maximizes your
utility exploit that configuration over
and over again that's the convergent
instrumental one humans have a terminal
value for doing new things imagine
running into aliens who have a different
emotion of boredom than we do in
particular
they are more easily amused in
particular these aliens have a narrower
definition of what constitutes the same
thing for purposes of boredom so if you
show them the same picture with one
pixel changed they say ah that's a
different picture I'm not bored anymore
what would this alien civilization look
like if we encountered them probably
something like this you know just the
moment of maximum fun over and over and
over and over and over again I will take
a moment to comment at this point that
you have to solve the Friendly AI
problem in order to get an interesting
universe full of strange alien beings
whose art and science we cannot imagine
that's you've got a solid the friendly
eye problem isn't about preserving
humanity as it is now until the end of
time if you don't solve the friendly a a
problem you don't get a strange
wonderful alien universe you get things
that you didn't even think about is
particularly human values like boredom
which because they're left out of the
utility function you lose everything of
value if you an alien civilization
doesn't have the same kind of boredom we
do if there if as far as they're
concerned one pixel of this frame makes
it a different experience that is no
longer boring then their entire
civilization would look uninteresting to
us similarly if you have a paperclip
Maximizer it wants to make paper clips
that's all it's utility function is it
doesn't have any utility any function
its utility any terminus utility
function for boredom and you don't get
to say well we don't have to build in
boredom as an explicit in the terminal
value it's a convergent instrumental
goal because the choice is not between
the human version of board
and an AI that experiences nothing
analogous to boredom our own boredom is
a product of our evolutionary origin the
fact that we are built by natural
selection that natural selection is
stupid means that a lot of things that
are a sort of natural selection sense
instrumental goals are in us terminal
values and for that matter our boredom
could easily be our style of boredom
could easily be tied to our neural
implementation it could be a matter of
neurons adjusting to the same reward
over and over and getting bored and go
looking at something else if you look at
a ideal Bayesian decision agent and ask
what kind of boredom is a convergent
instrumental value then there is a
convergent solution to the exploration
exploitation trade-off and that solution
leads to what we would regard as a
boring worthless valueless future this
is why we have to solve the friendly eye
problem it's not about the values that
you think of as uniquely human it's
about the values you don't even think of
as human but which are there nonetheless
and if they are lost if even a single
one of those values is lost losing a
single dimension of value can lose
nearly all the expected value of the
future everything we were hoping to get
out of those galaxies and besides
boredom some other cases where you lose
a single dimension of value you lose the
entire thing consciousness you know you
got everything in there and the utility
function only it doesn't done you know
it doesn't talk about consciousness or
doesn't talk about which sort of things
are conscious and from our perspective
the entire galaxy has turned into a
wonderful literary novel with no one to
read it external reference of subjective
sensations you have an AI that is sort
of carefully optimizing your subjective
sensations but doesn't want anything to
be behind it that's not part of its
utility function you know you have this
wonderful perfect girlfriend and you
feel very free but your girlfriend is
just wallpaper and your feeling of
freedom is produced by artificially
stimulating your little freedom of
sensation lobe in their little sensation
of freedom love in their
this is say I presume I'm running out of
time here on my corrector okay so so I
will skip a little extended thing that
will follow this last example this is a
anecdote I have repeatedly heard
presented as fact I've been unable to
track down the original of it but that
doesn't mean it's not true and the story
goes like this the army was trying to
develop a neural network that would
detect tanks and they gave some photos
of tanks and of non tanks scenes with
tanks in them and without tanks in them
to their AI researchers and the AI
researchers trained a network until it
could distinguish the tanks the non
tanks and then they from the same set
you know took the test photos in and the
neural network that had been trained and
the training photos could distinguish
the test photos just as well you know
got 100% accurate classification and
then they took it out and give it to the
army and the army came back and said
this doesn't work at all and it turned
out that all of their pictures of tanks
had been taken on cloudy days all their
pictures of non tanks had been taken on
sunny days they had built a cloudy day
versus sunny day classifier instead of a
tank classifier
the simplest boundary around the data is
not always the boundary you had in mind
if the training cases in real world
cases aren't from exactly the same
independent identically distributed
context no statistical guarantees apply
and if you imagine Terry Schiavo if I'm
pronouncing her last name correctly
Chavo Chavo sighs OH so Terry Schiavo
would not have been something we would
have encountered in our ancestral
environment if you try to classify
people as alive or dead in the ancestral
environment you have a litany of plus
cases in minus cases where if you draw
the simplest boundary around it it's
likely to talk about the difference
between alive and dead being the
difference between breathing and non
breathing say or moving and non-moving
and the case of Terry Schiavo is outside
the training cases of the ancestral
environment you can't
if you presented an AI with the net
ancestral environment cases and tried to
train it to distinguish between people
and non people the advance of technology
would have produced all sorts of exotic
cases that it couldn't classify that
wasn't guaranteed to classify using the
training data probably wouldn't classify
correctly because the simplest boundary
used for making predictions would not be
the moral boundary this is over here I'm
the sort of like talking about some
material that I was going to cover later
and skipping over it essentially the the
the there are several morals you could
draw from this but one of the morals is
the categorization that you want the AI
to draw sometimes without even realizing
it it's not there in the data it's there
and it's it's there in your utility
function but it's not there in the data
itself is a class of problems I'm not
really going to get a chance to talk
about because I'm skipping straight
through but it's why it's difficult to
just give the AI set of training cases
and say this is our ethics because the
shadow of your ethics on that data
set may not capture all the features of
your ethics that's the difficulty with
just trying to get Frank Anna's entire
value list including the etc by taking a
large corpus of ethical dilemmas and
training the AI on them but I didn't
actually quite get to go through all
that he caused a lot of time and I will
also skip the summary of the talk and go
straight to the end due to being out of
time


AI Ethics,  Bostrom and Yudkowsky

hi this is matt mccormick at the
Department of Philosophy California
State University Sacramento this is a
lecture for my philosophy of artificial
intelligence course by it's about an
article by Nick Bostrom and eliezer
yudkowsky called the ethics of
artificial intelligence
this articles may be about ten years old
now so it actually overlaps a few of the
arguments we've already encountered in
Schweitzer gibble and Garza and a few
other places but there's enough new
ideas in here and new angles to get us
thinking about ethical issues and the
ethical dimensions of artificial
intelligence to make it worth going
through several the points here so they
raise several issues a few of which we
might remember might recall I've
reordered these here they start out the
article by worrying about making sure
that AI systems as they move more and
more into our economy and more and more
into making important decisions that
involve humans they've got to be we've
got to worry about AI systems being
manipulated or exploited or otherwise
misused by people I moved this first so
we can look at this video there's a good
example it need not be evil necessarily
the one on the top here is using some
camouflage to thwart AI face recognition
that the Chinese government has been
using to clamp down on protesters so the
guys at the bottom are illustrating how
an AI that was previously very good at
recognizing humans if you are holding
that picture that disrupts the lines and
disrupts the recognition capability when
one guy hands the picture to the other
guy
the AI ceases to see him he becomes
invisible right so those are
manipulations that we might consider to
be useful and good in some cases where
they were worried about the Chinese
government clamping down on democratic
dissidents or we might be worried that
you know an AI system in a bank has just
failed to recognize a robber whose
because he's got a picture on his shirt
or something else like that the bigger
problem here is that Boston renewed
Kowski argue that AI algorithms need to
be transparent to inspection and I've
been calling this the black box problem
we can talk about this to talk about few
details here we've got AI systems
already that are doing things like
deciding mortgage applications for
example and let me get my notes
illustrated here so we've got AI systems
deciding mortgage applications for
example and we might worry about or we
might discover for instance that some
bank is giving out mortgages at some
disproportionate rate and there to say
white applicants and rejecting black
applicants at some disproportionate rate
and there's a couple of different ways
that something like this might happen
one of them we might identify for
instance suppose you train an AI up on a
bunch of of a bank manager a human bank
managers cases out of his filing cabinet
where he has looked at hundreds of
mortgage applications and he is rejected
or accepted down for two various
different circumstances and come to find
out the guy is racist if guy has got a
problem with black people he's
suspicious about black people and he
gives them he rejects their applications
that are higher than normal rate if we
had used his mortgage training data as
labeled training data sets then that
guy's discriminatory practice or that
guy's prejudice against black applicants
would be reflected in the training data
set and then an AI system that gets
trained up on it would just learn that
those ones are not worthy mortgage
applications and that the others are
worthy mortgage applications now here's
a case where the reason the racism is
actually implicit in the data
and the racism coming over over from the
human is not coming from the AAI
so that's a kiss where we might be able
to suss out what's going on and how the
data skewed the numbers and the more
transparent the AAI is algorithms are to
inspection the more we be able to
understand that this is actually a case
where we need to have transparent data
that's going into the AI system but
there's also a way that you could have
an unlabeled data set that goes into
training an AI system and that the thing
ends up discriminating in some sense of
discriminating against black applicants
so you've got say you've got
predominantly black zip codes or
predominantly white zip codes and it
also turns out that being black
correlates with poverty it correlates
with higher to loan default rates
because of other social contextual
cultural racist policies or racist
effects so as a result when an AI gets
trained up on and looks at the actual
data for who's defaulting on their loans
it will discover that there's a
correlation between being African
American and defaulting on your mortgage
loan and then the AI might conclude in
some sense of reasonable that it's
reasonable to bias against African
American applicants right so now here's
the case where you've got the bank
that's picking up on or the AI system in
a bank that's picking up on some
implicit racism that says that's buried
out in the culture buried out in the
society and then it's getting reflected
in the bank's decisions and we've got a
very real problem here about thinking
about where does responsibility for
repairing that sort of problem lie in
the first case it was pretty simple in
this case it's not so obvious how we can
solve that problem but you Kowski and
Bostrom's point is that we need to have
them transparent so we can at least
figure out what's going on in there and
we also need to have them be somewhat
bound to custom and culture and have
their behavior their decisions be
predictable
there's some other context in which I'm
somewhat skeptical about this black box
problem there's a lot of people who are
worried about it I actually think that
in many cases where people are worried
that AI systems are making decisions
that we don't know how they're making
the decisions or we don't know what
sorts of correlations they're
discovering I think that very often
those will I've got a couple ideas here
one of them is that very often we can
figure out what's going on just by
looking at the outputs so like in the
case I the inputs and the outputs like
in the case I gave you with the bank
manager if we look at what's going into
the AI system what's coming out we can
figure out what the problem is the other
idea here that I've got is that AI
systems need not or may not be any more
of a mysterious black box than human
reasoning this you know if I ask a cop
whether or not they're biased against by
people of course the cop on sort of
introspection is gonna say no I don't
discriminate against black black people
in the neighborhood or whatever you know
humans don't know what what they're why
they're making the decisions they're
making we're as much a black box to our
own investigations as the ICA AI systems
are and again how you figured this sort
of thing out is by empirical external
testing of inputs and outputs so you
test a AI systems the same way or maybe
come up with some clever ways you do it
the same way to figure out like implicit
bias in human cases so there's been a
lot written about this I think there's
some more it's worth some more study but
I think that some of the problems may be
overblown
so you'd Kowski boström you Kowski also
worry about the several sort of general
issues surrounding the idea of
artificial general intelligence so the
problem here is that we're trying to
build a system that we want it to do
something whatever that might be we
wanted to do it better than us and the
problem is that we don't know how to do
it better than us because if we could if
we knew how to do X better than I can do
X then I would do it that way and then
I'd be looking for some other way to do
it even better still so we're trying to
get artificial general intelligence
even narrow intelligence for that matter
I've got a picture here of alphago zero
plane lease at all and one of them one
of the important things about alphago
zero playing go sorry this is probably
alphago playing lease at all one of the
important things about alphago is that
it devised new strategies new approaches
new moves and new gameplay that Lisa
Dahl couldn't anticipate couldn't see
what to do about couldn't handle it was
better than any human player he'd ever
seen and he felt and you know as a
result he loses right so we built
alphago to play go better than Lisa doll
can play and in doing so we're making
something that's better than us and can
figure something out better than we can
figure out so what what happens in these
cases is we don't all right we don't
know the ways in which it's going to be
better so we lose what they call local
specific control and predictability that
in the short term in the in the sort of
micro moves and the short moves right in
front if we want to use the game
metaphor some more we might not be able
to see that alphago in for this move or
three moves out or five moves into the
game or 10 moves into the game we may
not be able to see that alphago is
winning but actually for long-term
overall success we know on the basis of
other training and other other other
work that alpha goes better than the
human so in the short term looking at a
single decision it might even look like
a bad decision from our perspective
because of course we're not as good at
it as it is so we can't not a good
position to evaluate from the from the
local you know it's sort of the local
specific move but in when it's playing
the long game it's playing the broad
game it's it's plane moves 20 or 30
moves out and it's playing better than
us overall it's actually going to
produce a better outcome so we've got
this problem about how do you make it
better than us and how do we sort of
abdicate local specific control and
predictability because we need to trust
it right we need to give over or some
control to the thing to be able to come
up with solutions that
beyond our horizon and and this creates
a sort of new set of problems about
creating this thing in the world that we
unleash and then it's it's making moves
or making actions or has behaviors that
we take on faith or some kind of vague
confidence that it's actually going to
do something good with these decisions
in the big picture the problem is even
much worse when we make the moon we make
the AI general and we want it to find
unknown solutions to problems across
different domains that are far beyond us
right I mean consider some medieval
healer struggling with trying to deal
with bubonic plague and in the 1300s and
they're you know we're operating with
you know an evil demon Theory evil demon
possession as a theory of plague or
they've got the four humors theory of
medical evaluation and they're using
that to try to solve the plague they're
completely in the dark about what's
really going on it took six hundred
years and modern virology for us to
understand enough about how bacteria are
starting out in virology but modern
bacterial science to understand that
this is a bacterial infection right so
for a medieval healer healer in the
1300s understanding what was going on
with bubonic plague is not just you know
is is to say it's beyond that person's
horizon is understating the problem it's
understanding the point the solution to
that medieval healers problem was it was
in a completely different universe a
completely different frame of reference
and a completely different scientific
paradigm so imagine how that six hundred
year solution out in the future would
look to somebody in the 1300s and now
perhaps were you know food sort of
starting to get her head around what
happens when you build an artificial
general intelligence that's vastly more
intelligent than a human and it devises
some solution to a problem that wit has
stymied us that has stumped us you know
curing cancer or solving world hunger or
something and it comes up with some
confounding solution that we can't even
grasp or get our heads around
so there's a very tricky sort of
difficult problem here about sort of
building the thing and then setting it
loose and then wondering about the
behavior so their answers the results
that come out of it consequently how do
we project it towards good Pro human
behavior out there at the limit when we
you know have it solve these problems
now
boström and you Kowski are writing well
before Stuart Russell's book which we've
just read recently and and I'm Russell's
solution here comes to mind for me that
Russell you know gives us these three AI
principles and he says look one of the
ways you solve the control problem is
that you couple the ai's projects or its
utility function to human preferences
and then you force it to figure out what
human preferences are by watching human
behavior so there's a you know there's a
promising angle here an approach to be
able to sort of solve some general
control problem but you Kowski and
boström have got got their finger on
something big and important here because
of what happens ethically once we get
this new vastly more intelligent player
agent rational agent on the playing
field
okay so then therefore ethics for AGI is
going to be different from ethics for
technology because AGI ethics are
fundamentally different from non
cognitive technologies I mean look the
local specific behavior of the AI may
not be predictable apart from its safety
even if the programmers do everything
right alpha goes alpha goes programmers
for example gave up local specific
predictability they didn't know what it
was doing even the programmers couldn't
tell you what alphago was doing you know
in move seven or move eight what they
could tell you was we think it's gonna
beat lease at all overall when it wins
the game but I don't know what he's
doing right now um and we saw the
similar sort of thing when IBM created
deep blue and they beat Kasparov with
their chess playing program they didn't
know what the moves men
but they predicted or expected that it
would be there would be a better player
overall and verifying the safety of a
system like that becomes a really big
challenge because we've got to verify
what the system is trying to do what's
its goal what's the thing is after
rather than being able to verify the
system safe behavior in all operating
contexts and that's the big difference
between sort of an artificial
intelligence system versus just a
robotic arm or something that's under
you know complete human control
so ethical cognition itself must be
taken as a subject matter for
engineering so we get a whole new
problem here right it's not just
engineers trying to figure out how to
make a robot arm you know perform an
action but we're trying to figure out
well what is it ethical cognition what
is it that when when I'm making a
decision about what's what's the morally
right thing to do in circumstances
what's going on there and how do I a
couple and AI systems development
towards those kinds of goals Russell
doesn't really talk about that in his
book but that's a very powerful question
and problem here okay so good on them
for raising the problem and sort of
sketching out some of the dimensions and
now they're going to cover some of the
material that we've already familiar
with because of Schweitzer evil and
Garza do artificial intelligence systems
have moral status and you'll see a
couple of familiar looking principles
here so some entity X has moral status
means something like X counts morally in
its own right its permissible or
impermissible to do things to it for its
own sake that's vaguely Conte and
sounding but the idea is that you know
you've got moral standing and we think
of you know humans have any moral
standing and rocks don't have moral
standing property has some moral
standing in that it's owned by humans
and to do harm to it you harm the human
so we've got this notion of you know
moral rights in both ways and we've seen
a couple of these criteria before too so
boström and you Kowski say well
sentience is the capacity for phenomenal
experience or having quality qualitative
fields such as the capacity to feel pain
and suffer that's when we focused on and
sapience which we haven't used that term
before I think
is the set of capabilities associated
with higher intelligence such as
self-awareness or being a reason
responsive agent so that's a way of
describing the content criteria for
moral status is if you've got sapiens
consonants are worried about sentence
but he is does build his his ethics
around sapiens okay so here's a
principle that'll look a bit like
Schweitzer evil and Garza if two beings
have the same functionality and the same
conscious experience and differ only in
the substrate of their implementation
then they have the same moral status so
this is very much like their principle
of no difference that we saw before and
then Bostrom and you Cassie expand it to
this notion the principle of ontogeny
non-discrimination if two beings have
the same functionality and the same
conscious experience and differ only in
how they came into existence then they
have the same moral status so we saw
Schweitzer land Garza argue for a
similar sort of thing they considered it
but then rejected an Aristotelian
argument that said the way you come into
the world matters with regard to your
moral status and so you've we've got all
four of them now agreeing on this notion
that um you're being artificial you're
being synthetic or you're coming into
the world by way of humans building you
doesn't change your moral status if
you've got these moral making properties
in your system that is being artificial
by itself doesn't qualify
doesn't disqualify AI systems so
furthermore the principle of ontogeny
non-discrimination is consistent with
the claim that there are creators or
owners of an AI system with moral status
may have special duties to their
artificial mind which they do not have
to another artificial mind even if the
minds in question are qualitatively
similar and have the same moral status
so they're gonna work out that ownership
question and we'll see with Bryson in
next week that or in our next lecture
Bryson's got a very dim view of all this
she thinks that ownership just gives
gives you complete entitlements to do
what you like to the AI system so it's
very much like we saw before if the
principles of non-discrimination
with regard to substrate and ontogeny
that is from where it came our accepted
than many questions about how we ought
to treat our threshold mines can be
answered by applying the same moral
principles that we use to determine our
duties in more familiar context I think
maybe philosophers are not that troubled
by this idea but I suspect that non
philosophers going to be a bit surprised
by this outcome they're gonna think that
we've sort of gone off the deep end
thinking that AI systems have the status
insofar as moral duties stem from moral
status considerations we ought to treat
an artificial mind in just the same way
as we ought to treat a qualitatively
identical natural human mind in a
similar situation okay so what are the
different ethical assessments of AI well
boström and you kasky bring up a very
nice point here actually and they don't
go quite far enough with it but they
wonder about the possibility and I
haven't an AI system that has sapience
without sentience so imagine that it has
a very high level of rational autonomy
it has you know sort of classically Con
Thien rational self governing principle
reasoning intellect but it doesn't have
qualitative fields it doesn't feel
anything pain or pleasure associated
with any of its mental states so there
you it's there's there there's a very
real possibility here that we build an
AI system it could have one without the
other so we can have other weird
configurations like that and that's
going to create these challenges for how
to deal with them morally that's what
not like anything humans have ever had
to deal with before how would we treat
those sorts of things ethically um they
call it a kind of zombie problem it has
intellect and higher cognitive faculties
but it can't feel pain suffer or be
happy
here's another exotic property what if
it's subjective rate of time is very
different than humans what if this thing
runs a thousand times faster in terms of
its subjective unfolding of it's
phenomenal states internally well then
that means that in an hour of suffering
it suffers a thousand times more
suffering than you would in an hour of
suffering so it might
deserve some special consideration as a
result of enduring more pain or on the
flip side pleasure and we worried about
this before when we thought about you
know zzyx utility monster problem again
and I'm also going to point out that
there's a the background argument that
it often doesn't get discussed in con
con takes a hard takes a lot of abuse
from people for you know not seeming to
care about animals and not seeming to
care about consequences and not seeming
to care about people's feelings and cont
has this interesting little argument
buried deep in some of the later works
where he says with regard to animals
that we don't have strictly speaking a
Conte and moral duty to respect the
rights of animals however being kind to
animals rather than mean cruel or rather
than torturing them or rather than you
know doing harmful things to them is
good practice it's good it's good for
building our character it would that
were you to routinely torture animals it
would make you he says something this is
an empirical question but he says
something like it would make you more
likely to be cruel or indifferent or
hurtful to humans and humans are ends in
themselves and worthy of moral status so
the so there's kind of indirect argument
that you know there are there are
grounds for being kind to animals so
maybe we could get a kind of conti an
argument here for the conclusion that we
ought to treat a is maybe you know some
sub status a is in the right sorts of
ways because it's it's it's good for us
not that it's good for them okay and I
also want to raise up the possibility
that as far as exotic properties go
boström and you Kowski don't go really
far with this sapiens without sentience
is interesting speed it up slow it down
that's interesting that makes a
difference but I'm gonna point out I'm
direct you to there's an article by
Marie Shanahan who is a computer
scientist in Bristol or someplace in
England and he's got this great article
on conscious exotica in a on magazine
online and he sort of explores this
question about really radically
different kinds of minds and that's what
these charts are about is he he comes up
with some ways to sort of graph and
think about very profoundly different
kinds of minds than human minds and it
gets and this discussion in Boston when
you cast makowsky gets me wondering
about well what if there are some moral
making properties that we are not really
on our horizon yet we're not really
fully appreciating and that these things
would be sort of you know profoundly
unrecognizable as a human consciousness
but they might deserve some moral status
depending on some you know kind of
radical or innovative new set of
principles about what kinds of entities
in the world matter so I think that's
worth some exploration and there's lots
of good science fiction examples where
people have explored these possibilities
Stanislas slams LEM is his name last
name he's got a book called Solaris it's
been made into a couple of movies the
recent one by jewel with George
Clooney's not too bad but Solaris is a
really interesting crazy speculative
science fiction book about a radically
different kind of consciousness out in
the world I won't spoil it by telling
you what it is you should go take a look
so imagine a radically different kind of
consciousness that's not even
recognizable human could it have morally
salient features other than sapiens or
sentience you know that's what we've
been dealing with humans trying to solve
the moral problem for centuries but it
may be that that there's arguments to be
made here that there's some other
grounds that could warrant it okay so
they as a result of this discussion
about speeding up or slowing down AI
systems they say in cases where the
duration of experience is of basic
normative significance like you're
enduring a minute of excruciating pain
it is the experiences subjective
duration that counts so if if a beans
sow if a beans hour of suffering amounts
to being going through a thought you
know is run at a thousand times
acceleration from what you go through
then what matters is the duration and
that so it would matter more
in the kind of case and I have a black
mirror episode the the episode is called
white Christmas and it's got John Hamm
and and I'd used to have a clip but I
don't have the clip anymore I can't find
it
but go watch this black mirror episode
called white Christmas he actually does
it to an to a an AI system in that in
that game he's trying to punish this AI
to make it do something he wants it to
do and he locks it away for like you
know subjectively a thousand years which
only transpires in like a minute and a
half for him so it's actually
illustrating the very thing the boss
term you can see here getting out um
there's my there's the the frame of it
but I can't get the clip I couldn't find
it it's it's off the internet I think
it's a copyright problem okay so we're
in new moral territory here's one you
haven't thought of before boss term you
Kowski say we'll look with humans
especially since our reproduction rates
are slow and sort of beyond our control
we just allow people to have as much
freedom as they want and we're on the
same same boat here with regard to how
many kids you have
um but look with an AI system it should
be possible for them to reproduce
quickly like copy themselves even
exponentially so an AI system could make
a hundred thousand or a million copies
of itself and create enormous demands
for resources because you've just
created a hundred thousand copies of a
thing that has moral status and then you
know so maybe they multiply maybe they
converge who knows it's gonna throw our
moral calculus and completely out of
whack so they say you know sort of
typically understating the point we will
need to rethink this principle among
others right because of this sort of
change of this problem all right so how
do we get super intelligent AI to be
good well we're gonna achieve super
intelligence say bostrom you Kowski by
two routes we might do it by achieving
by redesigning cognition itself we also
might achieve greater intelligence by
just increasing the speed and we've seen
this distinction before so two different
ways that we might get these AI systems
and let me just pilfer this distinction
from Keith Stanovich we've talked about
this before but I'm going to formalize
it so we've got two basic kinds of
rationality there's instrumental
rationale
which means you're behaving in the world
so that you get exactly what you most
want given the resources available to
you or is the optimization of the
individuals goal fulfillment so the idea
for instrumental rationality is if
you're being instrumental irrational if
you've got a goal you're pursuing those
behaviors of those actions that are most
effective at achieving your goal
it doesn't evaluate the quality or the
more morality or the or the you know the
the laud ability of the goal itself so
if you want to be a really good serial
killer there's an instrumental
irrationally instrumental irrational way
to do that there's better and worse ways
to do that so that's what instrumental
rationality means but we also have a
notion of epistemic rationality and
that's where we mean having correct
beliefs having true beliefs that map
correctly onto the world and actually
with a artificial general intelligence
do you need to have both you and I need
to have both too because I need to act
and I want to achieve my goals and being
able to achieve my goals requires that I
have true beliefs right and so you can
see like in cases we've got in the news
right now of people you know demanding
to be let out of coronavirus lockdown
and they want to be able to exercise
their freedom to assemble or their
freedom to go to church or whatever and
then predictably two weeks later they
get coronavirus and die right so that's
a case where somebody's you know that
they had some instrumental rationality
that that they argued for you know
they're trying to achieve something I
wanted to go to church or I wanted to be
able to go congregate with my friends so
they argue for that goal but the goal
itself was a really bad goal and didn't
achieve what they were after
okay so Stanovich helps there and we
might start thinking now in using
bostrom and you cows keys distinctions
about the ways in which super
intelligence might be different so I
think you'd Kowski had come up with this
set of metaphors there's three there's
three I think ways to think about this
we have a metaphor here well we we can
look at each other and we can think
about different differences of
individual intelligence between humans
so like think about the difference
between you and Einstein so if we think
of
them being more or less like us but then
being as far beyond us as Einstein is
beyond you then we then we have these
metaphors or these sort of utopian
images that come out of that metaphor
that leads us to think about a is our
going to be able to patent new
inventions publish groundbreaking
research papers make money in the stock
market lead political power blocks and
so on so we can use the metaphor of
imagining the difference is differences
between of intelligence between two
humans we can also think about the
differences in intelligence between past
and present human civilizations so I've
kind of already hinted at this with my
medieval healer and bubonic plague
problem so fast
AI systems will invent capabilities that
futurists commonly predict for human
civilizations a century or a millennium
in the future like molecular
nanotechnology or interstellar travel so
sort of think about that where the
ancient Greeks were in terms of their
knowledge and where we are in terms of
our knowledge and imagine then that kind
of mapping that analogy up to okay well
then we could expect earth imagine that
super intelligent AI might be as far
beyond us as we are beyond the ancient
Greeks and then third we might consider
the differences between brain
architecture between humans and other
biological organisms and I don't think
they go nearly as far enough here but
they pilfered this line from Verne
Irving
I don't recall which science fiction
book that is but he said imagine running
a dog mind at a very high speed what a
thousand years of doggy living add up to
any human insight and maybe we can
imagine that it would especially at some
very high speed or it certainly came up
with some novel solutions to problems
and and the you know we think of dogs as
being dumber than us in some sense but
what we're getting at here is that
changes of our cognitive architecture
might produce insights that no human
level mind would be able to find or
perhaps even represent after any amount
of time so now think about dolphin
consciousness or dolphin Minds or whale
Minds and then that Murray
Shanahan article that I referred to a
few slides back he starts thinking about
alien Minds so we think in terms of
intelligence of whales for instance or
octopi or other creatures as being not
just less intelligent than us but also
very different than us so now imagine
very different configurations that are
more intelligent than us and now you can
start imagining ways in which you get
sort of radically different results from
our different prospects for having
artificial general intelligence working
on problems
okay so we're familiar with the the
worry that bostrom has they've got a
whole book on this that super
intelligence poses an existential risk
to human beings and that means we're an
adverse outcome would either annihilate
earth originating an intelligent life or
permanently and drastically curtail its
potential right so it's a very sort of
academic way of talking about wiping out
humanity or conversely there's the
utopian jackpot that we might get from
this invention a positive outcome for
super intelligence could preserve earth
originating an intelligent life and
fulfill its potential completely right
so that's all the utopian scenarios we
imagine and then they call our attention
to what they call a good story bias and
what happens is that we overestimate the
probabilities of those scenarios that
make for good stories in TV and movies
emboss'd Ramin an expert and I've heard
I've seen Russell complain about this
too being an expert on artificial
intelligence inevitably when the science
press or the popular press comes and
interviews him or talks to him they will
ask him questions and there's always the
killer robot Terminator question and
inevitably when they publish their
article they go get a terminator picture
and put it at the top of the article
right
Russell's gotten to the point where you
won't even do any interviews we see so
sick of that happening well why is that
always the first thing on everybody's
minds it's because we've seen so many
damn movies with that theme right so
that makes us elevate the probability of
that sort of thing happening whereas
boström for instance you kasky are
worried about other you know
not so movie worthy scenarios but that
also might pose existential risk to the
human race and we've talked about you
know that sort of paperclip scenarios
over the course of the semester and
that's the the Avenue down which rostrum
goes okay so now here's a here's
something else that you haven't thought
about or you maybe you didn't reflect
about with the sort of the AI the
context of AI epics considering the
ethical history of human civilizations
over centuries of time we can see that
it might prove a very great tragedy to
create a mind that was stable in ethical
dimensions along which human
civilizations seem to exhibit
directional change what if Archimedes
had had been able to create a long
lasting artificial intellect with a
fixed version of the moral code of
ancient Greece so look we've made what
we think of as moral progress over the
course of the centuries and if we were
to ask for instance Archimedes or
somebody a thousand years ago if they
were building an AI system and you asked
them okay what is the good life or what
is the best way for Humanity to live or
what's the most moral behavior for this
artificial intelligence to pursue
they're going to give a very different
answer than what we would write and we
consider the expansion of the moral
circle as I've been arguing that you
know we've seen this progressive
expansion where for instance in American
culture you know originally you get
propertied moneyed white males are the
ones who have the most moral standing
the most political standing most social
standing and then we've steadily
expanded the circle of moral
considerations so we include and allow
women the right to vote and we expand
rights to gay couples to get married and
we expand rights to people of African
American descent or African descent and
so on and then you know we've seen the
the Peter Singer article that says
you've got to keep expanding that circle
out to sentient beings and include it
including animal
and therefore you shouldn't be
subjugating them to your you know to
your meal so we think of that arc as a
as moral progress we think of that nut
is not just change and not just shift
but there's development here that's
progress
so if we want an AI system to do the
right things well there's a time index
on that right so time do the right
things what 21st century right things
11th century right things first century
right things or 30th century right
things um and and this also gets me sort
of worrying about you know that example
so consider well I'll talk about it in a
second so
boström you Kowski are worried about you
know how do you build an AI which when
you when it executes it becomes more
ethical than you and I mean one problem
here is how do you know it's being more
ethical than you because it may not it's
behavior may not resemble your behavior
I mean maybe our committees would look
at us and be baffled by our pursuing
what we think of as noble or moral or
ethical goals and what do you do when it
arrives at allegedly ethical conclusions
that that look nothing like what your
21st century hominid brain thinks are
ethical so what are they getting out
here here's another long quote one
strong piece of advice that emerges from
considering our situation as analogous
to that of Archimedes is that we should
not try to invent a super version of
what our own civilization considers to
be ethics this is not the strategy we
have when we would have wanted
Archimedes to follow we wouldn't have
wanted Archimedes to have imposed some
kind of global ancient Greek ethics on
all of us perhaps the question we should
be considering rather is how an AI
programmed by Archimedes with no more
moral expertise than Archimedes who
could recognize at least some of our own
civilizations ethics as moral progress
as opposed to mere moral instability
this would require that we begin to
comprehend the structure of ethical
questions in the way that we have
already comprehended the structure of
chess so the idea here is that
there's a goal that we have in mind and
we want to figure out what the goal of
ethical development is and we want that
AI system to be pursuing that now this
has got me thinking about a couple
things on the side one is that Russell
again has got a nice solution in our
benevolent AI book readings where he
says look you solve the control problem
by having it be coupled to satisfy human
preferences and then it derives its
knowledge of human preferences from
human behavior and you keep it uncertain
about what human preferences are so it
has to watch our behavior to figure that
out so that's one way to kind of keep
the AI from running off without us but
what if the the suprem of the super
intelligent AI moral code appears to be
morally repugnant to us I mean look if
you had come to some ancient Greek with
the notion of modern egalitarianism post
French Revolution modern egalitarianism
that says ray neither race nor gender
make have any moral moral distinction to
make about whether or not a person has
moral standing an ancient Greek would be
dumbfounded meat would be baffled by
your notion that slaves that human
slaves are are wrong for instance or
some plantation owner in the 1600s in
the in the in the in the early American
settlers right if you had come to
someone like that mm-hmm
and suggested hey in five hundred years
we're going to think that not only we're
we're going to conclude that it's moral
to tolerate homosexual relationships and
let them get married and we think that
slavery is intolerable and you it's it's
a horribly morally repugnant thing to do
and that women ought to be allowed to
vote that guy would be outraged at the
suggestion that this is that those are
moral conclusions right so I by
extension I'm worried
that what if how do we know when an AGI
comes back to us with the answer to how
do we build the optimum Society right
and the answer he gives us is you know
dumbfounding or seemingly morally
repugnant or seems awful to us but we've
got to sort of trust in the conclusions
that this thing is drawn because it's
viewing things from a 30th century
perspective or it's understanding the
arc of a moral development and it
converges on something out there at the
limit that's way beyond what we sort of
you know can way beyond our kin so
there's all there's a lot going on here
and a lot of a lot of doors get opened
up by this sort of playing around with
the idea of radically different minds at
work on human level problems okay so the
recap here is that they start out by
arguing that we need to have sort of
responsibility and accountability
transparency audit ability that is we
need to be able to know what's going on
inside these AI systems and so on to you
know prevent these horrible outcomes for
humans and we've got this over the
horizon problem for building a super
intelligent AGI because they're gonna
come up with solutions to problems that
we won't recognize and we won't know at
least in the short local term whether or
not the solutions are coming up with
they may not look like that the right
ones are the good ones because you know
they're coming from an intelligence
that's vastly vastly beyond our own and
we don't understand it any more than a
caveman would understand your behavior
and then they spend some time wondering
about the moral status of machines they
come to some very similar conclusions to
Schweitzer gibble and Garza but also
spell out some of these details on the
side and then I've tried to expand on
this notion that exotic minds are gonna
raise novel moral considerations they
focus on speed and proliferation I want
to open up the door to a bunch of other
possibilities there and then we've got
this problem about what sorts of moral
norms or moral goals will a ethical
super intelligent AI converge on
what's the character of the ethical
being we want to shoot for and what is
the what are the answers that it's going
to give to what is the the sort of best
human life that we can live