Lectures by Olle Häggström on AI risk and long-term AI safety, part 1

right
most welcome everybody to this lecture
series on
ai risk and long-term ai safety
uh by now we all know how to handle zoom
i guess and i think that when we are uh
as many people in the room as we are now
it's good that
everyone makes up the speaker keep their
microphone
uh turned off you are welcome to uh ask
questions uh but i think that the
um
the best way is to do it in the chat i
cannot
guarantee that
my simultaneous capacity will allow me
to notice all the questions in real time
but but we'll
hopefully get some time towards the end
the end as well to
discuss a bit uh notice that
this
lecture is being recorded and will be
posted
at the chalmers ai research center's
youtube channel
i'm going to
share my slide deck with you
see how this goes
so i'm in full screen mode now does this
look okay
okay
very well
um
so
uh i see uh many familiar faces but but
uh
not all of you
are people i recognize so since we're
going to spend six hours together this
week maybe i can just quickly say a few
words
about myself
and where i come from
i did my undergraduate degree in
electrical engineering i
finished that in 1991 i had interest
already at that time in artificial
intelligence but um
it turned out that due to what is now
called the second ai winter which was we
were in the middle of that at that time
i was dissuaded from pursuing
phd studies
in a.i
i had
senior people around me telling me that
this this is not really the future so we
should look elsewhere i did
and and
i never had any reason to
regret
that i
did a phd in mathematical statistics
instead in 1994 i went on to be a
professor on the same
subject in 2000 first the first couple
years at the university of gothenburg
and then
at chalmers
and
until so this was a gradual change in in
my
um
research profile but but but until
approximately 2010 most of my work was
in probability theory and that is still
the field where i have the bulk of my um
research qualifications
but since then i have increasingly
shifted focus towards global risk and
the futurology of emerging technologies
and
ai is a large uh
part of that i had my first publication
uh in artificial intelligence
in 2012 at the fairly advanced age of
45.
uh until 2016
my
interest in future studies and emerging
technologies turned into this book here
detracts
science technology and the future of
humanity uh and uh my latest book is
this one in swedish
thank gandhi from last year
so at the same time
during the past decade
the field of artificial intelligence has
transformed quite drastically
through the
machine learning and deep learning
revolution
now uh in 2015 when i finalized the
manuscript
uh for for my book qb dragons i hadn't
quite picked up on the importance of
that so when i talk about artificial
intelligence
in this book
i uh
i do it
on a more abstract and
general level and
i
try to make up for that
in my latest book
uh where i
discuss quite a lot of
issues specific to neural networks and
deep learning
and i will try in these lectures
to go even
a bit further on on what is uh happening
um so
in tank and the machinery
i divide so there's a there's really
quite a big small gas board of of
issues and
risks and concerns uh connected with the
future of artificial intelligence and in
tank and mcqueen i categorize these
issues as either down to earth or high
fly
now i know that the term high-flying is
sometimes used in in a derogatory sense
uh i don't mean it uh that way i i think
that the issues
that i label as as
less down-to-earth and more high-flying
are still
a very important issues
now
to to show you the
division here
uh
among the down-to-earth issues
i include
things like safety of autonomous
vehicles which has become somewhat
of a hot topic
partly of course because
technology on autonomous vehicles is
advancing fast and also because people
have found these really sexy connections
with um
trolley problems
personally i think these kinds of
problems uh i mean
they carry out these um big um
[Music]
surveys and psychological uh experiments
to figure out what people prefer in
situations like this one where you
either have to kill all the people in
the car by crashing into something or
or run over these
pedestrians
i i think these problems are probably
not that important as far as
predicting that these
situations would
ever happen i think that basically they
won't because by the time that uh
autonomous vehicles
roll out broadly on our uh streets they
will be so safe that these kinds of
situations will not happen i think that
if this field really turns out to to be
important it will we will be have
created that problem
uh where
through our very focusing on that
and that leading to
legal
issues and
marketing issues and so on related to
the
mostly hypothetical
choices that will be made so this is a
parenthesis this is not the kind of
stuff i'll be talking about
neither will i be talking about the very
important problem of
algorithmic bias which can occur in ai
software used to make predictions about
people
for instance um
in in
software used for deciding whom among
hundreds of job applicants
uh you will be calling for
for an interview
and
and
judging who
is legible
for um
judged to to to be qualified
for financial aid or unknowns
uh and and uh
this kind of software is even widely
used in the
justice system to judge
the probability of recidivism of
convicted criminals and this is
something that can then be used to
to judge
how how long sentences that
this is something that really really
affects people and is therefore
important
still
quite a down to earth issue
by down to earth here i mean
uh issues arising
uh in either present-day ai technology
uh or
in ai technology uh that can be
clearly seen to be
uh near in the tangent of
of present present-day technology
other such issues
concern ai software for manipulation of
images and videos such as with deep fake
and also
text generation and text manipulation
and and one of the reasons why this is
important is that all these things can
then be applied to manipulate
humans and
arguably this is already going on for
instance in in how the big tech
companies
uh manipulate our attention to keep us
in the loop
on
on their social media platforms and so
on
many
there are many important ethical and
other issues here
of course
one other thing to be concerned about
which is also
regards very much present-day technology
is ai technology
for
lethal autonomous weapons
systems
such as military drones and and
the risks involved in a
military arms race
in ai technology
also a different kind of question but
still fairly down to earth is the effect
of automation on the labor market and
economic inequality and these are just
examples there are
still further issues but i want to
mention also
the the
[Music]
main high-flying issues and one
concerns what will happen what will be
the situation uh for humanity uh in case
we reach
a breakthrough in so-called artificial
general intelligence agi
that causes us to
lose our position as the
most intelligent
species
on this planet
and there's also an issue of whether
artificial intelligence
can
bring about
conscious beings in our computers
and and and
that's i mean
both of these issues have at present a
fair amount of
speculative element but but they're they
are
still quite important it has been argued
uh
that ai consciousness is not really
something
we need to worry about
and
and here is uh leading ai researcher uh
stuart russell
who
in his book human compatible which came
out in 2019 and is a very very good
introduction
to
ai safety research he says no ai
consciousness does not matter
why is that and i'll
quote him
at some length here
so russell says suppose i gave you a
program and ask does this present a
threat to humanity
you analyze the code and indeed when run
the code will form and carry out a plan
whose result is the destruction of the
human race
just as a chess program will form and
carry out the plan this result will be
the defeat of any human who faces it
now suppose i tell you that the code
when run also creates a form of machine
consciousness
will that change your predictions
not at all
it makes absolutely no difference
so russell's point here is that the
important thing to try and understand
when we develop new ai technologies as
what the ai will do and that conditional
on the answer to that
it's not important whether on top of
this the ai
will
be conscious
or not
uh
it makes no difference to us
in a sense
uh what i would say that there are
arguments uh against uh this point of
view so here are a couple of reasons
why ai consciousness might nevertheless
be an important topic so one is that if
machines are capable of consciousness
and suffering we may
at least on some ethical views have
moral obligations uh
to them so in some years we have moral
obligations to all
sentient
and unconscious
beings
and on that other views we have even
more obligations towards sentient beings
that are our own uh creation
in case they there and they suffer would
be in some sense
our fault that they are around
to suffer
and this can also i mean
we'll talk a little bit about scenarios
where
um
artificial intelligence actually
replaces uh humanity and the way
to judge such scenarios
uh
[Music]
our
how we value such scenarios can depend
quite strongly uh on whether the ais are
conscious or not
another argument
is that if machine consciousness is
impossible
then the idea of so-called mind
uploading uploading our
minds onto computers
will seem a lot less interesting
especially
to the individual who uploads and
especially in case of so called
destructive uploading so it's widely
believed that if mind uploading will
ever be possible which is of course an
open question but if it will it seems
likely that at least initially it will
take the form of
destructive uploading where in order to
scan
the brain in sufficient detail that you
can represent emulate the mind on a
computer you actually need to destroy
the brain for instance by cutting it up
in
thin uh slices
uh to to to monitor everything that's
happening in it
so economist robin hansen has this
wonderful 2016 book the age of m where m
is short for emulated minds
which
describes
and tried to analyze the kind of society
we would get
if
we have a breakthrough in mind
uploading and he
expects that that
once that happens
emulated minds will in fact quickly come
to to dominate the economy for the
fairly simple reason that once you have
uploaded mines you will easily be able
to copy them
which is
i mean it's as easy as copying a
computer file which is in contrast to
how
complicated it is for humans in flesh
and blood to to
procreate and this
[Music]
this aspect of
uploads is going to cause all kinds of
interesting
uh
consequences to labor market and
other aspects of society and and when i
asked robin hansen about this
consciousness problem he says that now
that's a philosophic
philosophical question i'm
i mean
uh there seems to be essentially uh zero
uh progress in this and i just want to
figure out what will happen
and whatever happens happens regardless
of
this consciousness issue
okay anyway this is all i'm going to say
about consciousness which i'm going to
leave behind and and i will focus the
rest of the lecture series
uh mainly on the issue of
the ultimate
agi
breakthrough i will later criticize the
notion of agi artificial general
intelligence
a little bit discuss i will discuss
other notions such as transformative uh
ai but it turns out that it's still con
convenient even though these issues are
very complex to use agi as a kind of
placeholder for for these other related
concepts
in
some of these
discussions
um
so i want to mention
uh
this
2016 paper concrete problems and ai
safety
by some of the now leading ai
researchers and this book the alignment
problem from 2020 by science journalist
brian christian
because
uh
both this paper and this book put great
emphasis on continuities
uh between
um
down to earth ai issues and these more
high-flying issues about
the uh ultimate
ai breakthrough
so what what
amodei and ola and co-authors do in this
paper
is that they discuss
a variety of problems that had begun to
emerge discussion on in the
ai safety
community in the context of of the agi
breakthrough and and they discussed
the corresponding problem in more down
to earth contexts such as
robotic
household robots
if you have an intelligent
[Music]
ai
vacuum cleaner
for instance
and
it's been programmed with a goal
function to
collect as much dust
as possible in the apartment
there are various ways that you could
get malign
behavior such as
every now and then the machine pours out
releases all the dust it has collected
in the apartment in order to be able to
collect it again
and score new points of course when that
happens that will be because we have
specified the goal function of the
machine
uh in
in the wrong way we didn't want this
kind of of behavior but this is
the
kind of concern that we have uh
concerning uh
the sort of super intelligent ais that
we might see eventually and and
while
it won't be a global disaster to have
household robots doing things
a bit wrong the
stakes get higher and higher with more
and more uh powerful
machines but i think that that both the
authors of this 2016 paper and brian
christian they have very much point in
in
emphasizing continuities
between these uh fields and that will be
a recurrent
um thing in my lecture series on the
other hand
here is elsie yukovsky who is kind of
a um
some would call him
the founder of the entire uh
ai existential safety uh
field
uh and and he has this very catchy quote
uh where he emphasizes that that the
kinds of problems
um
studied
on down-to-earth issues and these more
high-five things were quite different so
so he
addresses the question that some people
have asked
uh concerning ai effects on the labor
market
what would happen uh to the labor market
at when we uh construct
super intelligence and perhaps get
intelligence explosion or singularity
with respect to very advanced ai the
sort that might be produced by ais
health improving asking you about the
effects of machine super intelligence on
the conventional human labor market it's
like asking how u.s chinese trade
patterns would be affected by the moon
crashing into the earth
there would indeed be effects
but you would be missing
the point
so what he says here is that
basically
nothing like the the today's
human labor market is likely to
continue to exist in the presence of
machine super intelligence so so we're
talking about very very transformative
and drastic scenarios
maybe you recognize
this person this is swedish-born
physicist max tegmark at
mit
he said in the podcast in 2018 that it's
now for the first time in the four and a
half billion years history of this
planet that we are at this fork in the
road it's probably going to be within
our lifetimes that we're either going to
self-destruct or get our act
together
and if we get our act together
he feels
like
many ai futurologists that there is a
potential to to create an enormous uh
flourishing future for humanity
and and and and this fork in the road
it's not
only created by artificial intelligence
there are other technologies as well
biotechnology and nanotechnology but
artificial intelligence
seems to play a kind of a special role
and one way to think about this is to
think back at what the world looked like
a hundred thousand years ago when our
um
predecessors uh
walked around uh on the savannah
homo sapiens did not really stand out as
as a particularly
influential species and if you compare
that to
to the situation today where in a sense
humanity
uh dominates the planet
and and and and we have changed the
planets in ways that are visible
uh from space
uh and
other species or kind of women
their faith is in our hands what caused
this huge change well it has nothing to
do with our muscular strength or our
physical endurance and it has everything
to do with our intelligence and now that
we are at a stage of
moving this intelligence
popping this intelligence over
to
machines and to automate it that could
be very very
consequential and some people argue that
may well be
more consequential than
anything we have
done
uh previously human history taken
together including the
agricultural and the industrial
revolutions
so again
uh this this will will be our focus the
ultimate agi breakthrough and related
concepts so to start talking about that
i would would like to uh go back
to
no first i will say this uh
when
discussing issues relating to the
ultimate agi breakthrough
there's a further distinction
to be made namely between technical ai
safety which is
sort of centers around how we program
these machines so that they will be safe
and ai governance
which is more about how we should design
society how we should create
norms and legislation
and so on uh to to to make this
happen safely and these are both
enormously important and
crucial
um aspects for managing an ai
breakthrough
safely but in this course i will focus
entirely on technical ai safety and i'll
just
refer to
this report from
2018.
uh by alan defoe at the future of
humanity institute at oxford
uh where he really uh in in a very
readable
good way goes through systematically the
kind of research problems
that has
the potential to to to help us
govern
ai development and ai deployment
safely
so
for the present course uh
i'll just give the roughest of outlines
here so today i will focus on
basically
defining the problem or pointing out how
and why an ai
breakthrough
might go
terribly wrong
um on wednesday i will talk about
timelines meaning what's the time
perspective for when we need to figure
out how to manage this
then uh
after
basically saying we don't know as an
answer to the timeline issues but i will
say
say this in a very nuanced way i will go
back a little bit to
um
present a artificial intelligence and
see in what way this possibly points
towards
an artificial intel general intelligence
breakthrough
and and i'll discuss uh
something that
relates quite closely it turns out to
natural language processes plural source
for handling
an adi breakthrough naming oracle ais
and then finally
on friday
i will talk about research directions
in ai alignment
an ai alignment is a crucial part of ai
safety where we try to
make sure that
the first
super-intelligent machines have
uh motivations and goals and drives that
are aligned with what we want
and which is usually more or less
whatever promotes human flourishing
so so so this is an outline i want to
mention also that uh on
friday night
the
group called effective altruism
gothenburg this organizing
after work event
which i
assume will happen in real life
and and
they will center around
what has been discussed at this lecture
series and i wonder too muslim by are
you in the audience do you want to
say something maybe
uh
tell us whether
a location has been
decided
or anything else thomas
yes hello hello
hi um
yeah so we have not decided the um
location yet but it will be
it will be in person and uh
at some
uh well bar somewhere in the gothenburg
area quite central and it will be quite
relaxed
discussion centered around your lectures
but
also to other
um topics related to effective altruism
so everyone is very welcome to to come
and i welcome this i should say that i
this was a very pleasant surprise to me
i was not at all uh
involved in
organizing this uh unfortunately i
cannot promise at this point
that i
will attend at present it seems like it
will not
work for me but let me
hold that a little bit
open
until friday so
anyway
thanks thomas and and your collaborators
for for organizing that
thank you okay so now i'm going to do
what i promised uh a few minutes ago to
go to go back in history i'm sure you
will recognize this person this is alan
turing
the uh
he's sometimes
claimed to be
the father of the computer and there are
others who could
aspire to that title but he is surely
the the father of
computer science
um through his uh classic 1937 uh paper
now he most of his work was very
mathematical and very technical but uh
towards the end of his tragically
uh short
life
uh and his brutal death which i think
one can argue was really
i mean in a sense it was by his own hand
but in another sense it was in the hands
of
the barbaric the british justice system
at the time he died in 1954
before his 42nd birthday in 1951 he had
this this
paper where he allowed himself to
speculate a bit philosophically about
the future of computing technology and
he said this
my contention is that machines can be
constructed which will simulate the
behavior of the human mind very closely
let us now assume for the sake of
argument that these machines are a
genuine possibility and look at the
consequences of constructing them
it seems probable that once the machine
thinking method had started it would not
take long to outstrip our female powers
there would be no question of the
machines dying and they would be able to
converse with each other to sharpen
their widths at some stage therefore we
should have to expect the machines to
take control
so there are several things
here where he's really ahead of his time
one thing is this idea that the machines
would somehow start
developing
on their own without our
involvement and
the most
ominous part of all is of course this
final sentence that we should have to
expect at some stage the machines to
take control
and
this
should have given the rest of the
scientific community pause to think
about what what are we doing
but what happened instead was that with
very very few exceptions
this idea was
entirely ignored by the research and the
academic community of course it has not
been ignored by by the
um
like science fiction writers and
hollywood movie makers
and so on but but within research it was
mostly ignored one exception
is the
similarly
leading 20th century thinker nor
betweener who wrote in 1960. this is a
wonderful paper some moral and technical
consequences of automation
and i really recommend it i find that
much of the things i say now in 2022 is
this was foreseen by by weimer here's a
passage if we use to achieve our
purposes the mechanical agency with
whose operation we cannot effectively
interfere once we have started it then
we had better be quite sure
that the purpose put into the machine is
the purpose purpose which we really
desire and not merely a colorful
limitation of it and this
very clearly points to the ai
alignment
problem
but basically throughout the rest of the
20th century
nothing happened on this topic despite
uh great optimism
about uh advancing artificial
intelligence
so in the first century of
so i'm sorry in the first decade
with the 21st century things started
happening and
a paper which i regard as really
landmark in this respect
is is this 2008 uh paper by
elijkowski on artificial intelligence as
a positive and negative factor in global
risk
where he outlines many of the key uh
problems here's yukovsky
uh this
here's the preprint version but
it was uh
published in this anthology global
catastrophic risks
edited by uh nick bostrom and and
astronomer milan zerkovic philosopher
nick bostrom went on in 2014 to write
this book super intelligence paths
dangers and strategies which became
very very influential
in
creating
uh this still small but
rapidly growing field of
ai existential
safety
um
there are i mean and it's a wonderfully
written book it is in in some respects
uh already outdated
like my 2016 book it it
i don't think it even mentions deep
learning or at most in in passing
uh
but it's it's really uh
an important uh land book but i think
that now in 2022
uh it's a better introduction it's
probably to start with one of the
newer treatments of the topic and i will
mention many in these like
lectures
so
a very basic insight
that underlines much of our
understanding of ai risk is this an ai
system does not always do as we intended
and
science fiction offers examples of this
here's a famous scene uh from
kubrick's uh
2001 uh movie open the pod bay doors how
i'm sorry dave i'm afraid i can't do
that now we don't need to go to fiction
to to give examples of ai systems that
do not do
as we intended
so here is is a a modern example and
this is the uh computer game of soco
bahn
uh which i'm going to come back to in in
later lectures because it can be used to
illustrate other
phenomena but but
so what's what's this game well uh
briefly you have this
the player
has this
avatar in the game who can walk around
in the maze and push these boxes
around
and the task is to push boxes in such a
way that they end up
in these
positions
and and
it turns out that you can train a
machine uh learning uh algorithms to
play soccer ban very well just as you
can do with similar games such as
pac-man
and uh
um
whatever they called mario bros
um
and
at some point people
discovered that
if you were too
monotonically rigid in how you define
these bases when you train the machine
learning
algorithm to play this game
uh you can get
strange behaviors if you train
the
the ai on uh mazes where you always have
these target places in the uh upper
upper left corner of the maze
and and then you try to apply
this
algorithm
to an instance of sokoban where the
target
nodes are in the other end of the game
uh you you might
see uh failure and the ai will push the
boxes to the
upper left corner
as it's used to regardless of which
positions are marked so i'm not sure
this was discovered exactly in the
sokomban context i think it was maybe a
couple of the other
uh
maze-based
computer games but i wanted to show this
possibility in this combining context
because that will come back to that
here's an example which is tremendously
surprising so you know about this
this when you play
five in a row tic-tac-toe
uh on an unbounded grid
uh so this is a typical thing that you
might want to program an ai to be good
at and and
so this is this is risto uh miku linen
at the university of texas as austin he
gave a course on artificial intelligence
where this was
one of the projects
to create a population of tic-tac-toe
playing
programs
facing each other in competition and
then mutating
and so this is kind of evolutionary
or genetic algorithms
approach
and
they got this very very surprising
uh result that
the
program that took over the population
had the habit
of placing
its cross
at some point or or or not whatever
let's say it played crosses very very
far away from where the action was
happening it would play its place across
billions of
coordinates
uh away
and and it would
uh
it would still win its games and and
what happened was
that uh
all the competitors
uh were playing with a kind of
uh what do you call it it didn't have a
fixed uh
memory space uh for keeping track
of
this
the grid
but but
had had dynamic memory storage so it
would
enlarge
the memory space
depending on
where the notes and crosses were put and
by putting the cross very very far away
it would cause the
opponent
to run out of memory
and then
it would win by default so this was
completely uh unforeseen by those who
carried out this
experiment and
constructed uh these algorithms so this
is this is an example of surprising
things that can happen and and and
this appears along with maybe 50 or so
other examples
uh in this uh very interesting pre-print
from 2019 the surprising creativity of
digital evolution the collection of
anecdotes from the evolutionary
computation and artificial life research
uh communities
another striking example so here was a
group in 2002 that but somehow
they set up a system for for hardware
evolution
and they they
uh set up a reward function or the
fitness function uh for these machines
uh designed to create an oscillator they
wanted to to see if the
this approach uh could find the solution
for creating a good 50 hertz oscillator
uh and these uh
people uh bird and lacell they found
that
they
[Music]
this kind of approach did evolve
something that was a very good
well it produced a 50 hertz wave but the
circuit did not look at all like an
oscillator and it turned out that what
it had
what had been constructed was instead a
radio receiver which could pick up the
50 hertz signal from the
a nearby
computer
uh so that was a big big surprise and
and
an especially interesting feature of
this example is that this might possibly
have consequences
for those ai safety researchers who
think about
boxing in a super intelligent ai
preventing it from interacting with the
world now if a computer system can
create a
radio receiver it can also create a
radio
transmitter
and that could be
a way to communicate with the outside
world so the consequences of these uh
surprises uh increase with the power of
the system and this leads to thought
experiments like like the infamous paper
clip armageddon
which is
much uh discussed as
an extreme example of what could happen
with super intelligence
uh so imagine
uh you have a paperclip
factory which is heavily automated and
the
[Music]
well
it is so fully automated this factory
that basically the entire factory is run
by one central ai and the only people
you have there
are some engineers trying to optimize
this ai program even further the ai
has the goal of maximizing paperclip
production at some point the engineers
manage perhaps by accident even to
push this
machine
over the threshold where it can
where it's so intelligent that it can
start to
uh self
improve uh on its own
and then it launches into this so-called
intelligence explosion or singularity
uh cycle of iterative
self-improvement to reach
super-intelligence
levels and once we have a
super-intelligent machine
which has the goal of maximizing
pay-per-clip production we may be in
deep
deep trouble because uh
since it's super intelligent it can
probably outsmart us in various ways as
soon as
it does an internet connection it will
probably create backup copies
of itself in thousands of of cleverly
locate strategically located
positions
and from there on it can go on
perhaps in hiding for some time
uh to to to work out its plan to turn
everything including ourselves
and the entire planet into paper clips
maybe possibly except for a few
rocket
ramps
uh intended for launching spacecraft
into outer space and
continue the paperclip production
project
out there
so this is this is an extreme example
it is sometimes
dismissed as unrealistic and
nobody really thinks that we will
turn into
paper clips
but
i think that the
this example still has some some
important qualities one of them is that
it illustrates that that maybe things
can turn out very very dangerous without
having any obviously
lethal or dangerous goals such as
killing as many humans as possible of
course it would be very dangerous to
have have an ai
that has
that kind of goal but what this is meant
to illustrate is important that even
innocent looking goals such as
constructing
producing paper clips can have bad
consequences i think that it's more or
less time to take a break
uh i can tell you a little bit about uh
what is coming up next i'm going to give
you uh some scenarios that are
they do lead to catastrophe but nothing
quite the same way as the paper clip
armageddon let me just show you this
picture of something that is this
closely related to the paperclip
armageddon scenario
so here so this is the famous distracted
boyfriend meme here we have the agi in
the middle here is
humanity and our
ai designers who want the agi to think
about human values and the age gi gets
distracted by this idea of reward
hacking so suppose this agi has the
goal of let's say maximizing human
flourishing in some sense and suppose
that somewhere
in its memory storage it has this
particular memory address where it
stores the value of how well it has a
sheet achieved this goal
then
we are not quite sure at this point how
to prevent it from hacking this channel
and trying to maximize
the number stored in this position
independent of the actual human
flourishing that we meant it
to create and you can get scenarios
similar to the paper clip armageddon
where the machine
expands on its amount of computer
storage beyond all limits just for the
purpose of uh storing ever more nines in
the digit used to represent how well it
has
achieved
its goal
but but i'll
go on after the break with uh
another scenario which
i think some of you will find a bit more
appealing a bit less extreme
than this one and then we'll talk about
more about the theory of of the dries
and drives and goals of
an advanced artificial intelligence uh
we have
one um question in the chat here
a sound complaint uh i'll look at this
into this during the break
let's
what time is it now
uh
1607
uh
can we take an eight minute break uh and
meet again at 4 15.
let's do that
i'll pause through the recording
the time is 4 15 we'll come back
from the break
[Music]
screen again
[Music]
let's see this
so this is andrew kritsch
one of
several
strong
researchers
from the bay area
uh
young
generation
we'll come back to to some of this other
work
this
i'm going to show you now is a scenario
he has outlined for for how
things can go uh very bad in a way that
is quite different uh from from the
paperclip armageddon
so i'm going to quote that some len
length
from from his document he starts by
the scenario starts in this way ai
researchers develop and publish an
algorithm combining natural languages
processing and planning capabilities
tech companies develop management
assistant software tools based on the
algorithm which can analyze a company's
cash flows workflows communications and
interpersonal dynamics to recommend a
more profitable uh business decision
software tools based on variants of the
algorithm sweep through companies in
nearly every industry automating and
replacing jobs at various levels of
management sometimes even as high as
chief executive officers
companies that don't heavily automate
their decision-making process begin to
fall behind
[Music]
companies uh i'm sorry
[Music]
companies closer to becoming fully
automated achieve faster turnaround
times uh and more successful
negotiations over time a mini economy of
trades emerges among mostly automated
companies
in the materials real estate
construction and utilities sectors along
with a new generation of precision
manufacturing
companies that use robots to build
almost anything if given the right
materials a place to build some 3d
printers to get started with and so on
and electricity together these companies
sustain an increasingly self-contained
and interconnected production web that
can operate with no input from companies
outside the web
and
here's a sketch of what this web
could look like and its products
so one production web company at some
point develops an engineer assistant
version of the assistant software
capable of software software engineering
tasks including upgrades to the
management assistant
software within a few years all of the
human workers at most of the production
web companies are replaced with very
generous retirement packages by a
combination of software and robotic
workers that can operate
more quickly and cheaply than humans
and a great wealth of goods and services
are generated and sold to humans at very
low prices so everything is looking very
good
at this
point
so as the production web companies get
faster at negotiating and executing
deals with each other waiting for human
managed currency systems like banks to
handle their resources become a waste of
time so they switch to using purely
digital currencies
bitcoin or whatever
governments and regulators struggle to
keep track of how the companies are
producing so much and so cheaply but
without transactions in human currencies
to generate the paper trailer activities
little human insight can be gleaned for
from auditing the companies
and it becomes increasingly unclear at
this point even to the concerned and
overwhelmed board members of the fully
mechanized companies of the production
but whether these companies are serving
or merely appeasing humanity
moreover because of the aforementioned
wealth of cheaply produced goods and
services it is difficult or impossible
to present a case for liability or harm
against these companies
through the legal system which relies on
the consumer welfare
standard as a guide for antitrust
policy
so
what happens is that we humans
eventually realize with collective
certainty that the companies have been
trading and optimizing according to
objectives that are not quite aligned
with preserving our long-term well-being
and existence but by then the their
facilities are so pervasive
well-defended and
intertwined with our basic needs that we
are unable to stop them from operating
so with no further need for the
companies to appease humans in pursuing
their production objectives less and
less of their activities end up
benefiting humanity and eventually
resources critical to human survival but
non-critical to the machines such as
arable land drinking water atmospheric
oxygen and so on gradually become
depleted or destroyed until humans
can no longer survive
rephrases this as as kind of in the end
an environmental problem
now there are some uh difference between
uh paperclip
armageddon and creatures
production web
one is the very sudden onset of
paperclip armageddon compared to the
more gradual takeover
uh in the production like my
comment here from one of you that that
creature's production web is
already happening and
in a sense yeah you could argue that um
also the paperclip armageddon is
unipolar while the production web is
multipolar what does this mean unipolar
means that
you have this one machine in control of
everything whereas
the crutches production web has lots of
ais interacting
in
complicated ways
now most of ai existential safety
research has focused on the unipolar
case because it's the simpler one and
it can also be argued that if the
breakthrough happens sufficiently fast
in an intelligence explosion then
there's a chance that that the
first
super intelligent ai will get a decisive
strategic advantage and then we'll be
able to to
basically
prevent
all
competition with it but if if the
breakthrough happens more gradually
this will be less clear so it will be
important to look at
multi-polar cases as well um and andrew
critch he studies this in a paper from
2020 together with david krueger who we
see here
it's a kind of research program it's a
wonderful 130 page paper report
on ai research considerations for human
existential safety which i will come
back to
on
friday i think that the main thing they
do is that they point out
uh
very persuasively the need to also look
at multipolar cases whereas the field as
a whole has
looked so much
at the unipolar case there is also some
systematics here in the paper about the
kinds of
of the existential disaster that may
uh
being come out of
um uncontrolled uh ai
development and deployment
uh
so so there's one go-to place for a
richer plethora of
catastrophe scenarios i'm going to leave
that at that for the moment and i'm
going to address
throughout the rest of today's lecture
the issue of why an advanced iai might
be motivated to do us harm
as it does in
these
scenarios i have just
given
and and
i already hinted at one aspect of this
answer namely that doing us harm might
not be
the
primary goal of the ai but just a side
consequence of other things it wants to
do
so i think that the best framework still
that we have uh today for thinking about
what an ai is motivated to do
is what i call the omahandra bostrom
theory of instrumental versus final ai
bones and motivations
which was
invented in parts by by
computers scientists even alejandro and
philosopher
nick bostrom you could associate other
people such as nc jankowski to some of
these ideas but but this is this is the
term i use so the cornerstones of this
theory
are the orthogonality thesis and the
instrumental convergence pieces and i
will explain these two in turn
the orthogonality thesis
so it's a little vague it's not a
mathematical theorem it's it's a it's a
more of a philosophical statement says
that pretty much any final goal for an
ai is compatible with arbitrarily high
intelligence levels
so the idea here is to
preempt the objection uh that something
like paper clip uh production seems like
such a stupid goal
uh
that but it's it's uh
well it has been suggested that it would
even be a
contradiction to assume that uh a super
intelligent machine or a super
intelligent entity of any kind would
have such a stupid goal but we should
understand here what what
intelligence is taking to mean
intelligence is instrumental
intelligence intelligence is the uh
ability to attain goals
regardless of what the
actual goals
are
and and
i think this that this is
mostly right but here is a 2019 paper of
mine where where i discussed challenges
uh to
uh the omahandra boston framework and in
particular the orthodontic thesis so
there are concerns about
how how generally the authenticality
thesis can be correct so one thing is
that you can
construct
[Music]
far-fetched maybe but still counter
examples if you can if you define goals
in terms of intelligence levels
so an agent who has the goal of having
the intelligence level of a dog
let's say that the
intelligence of the agent is really on a
super duper high level
from the beginning that's not the
sustainable situation it will come
quickly find ways to bring its
intelligence level down to to the level
of a dog if that is
its goal
but
i mean that's a
narrow kind of class of country examples
and and
uh it still seems that the orthogonality
thesis has some force if you don't
involve
goals defined in terms of intelligence
another problem is that the
machine can can encounter
what has been called ontological crisis
a crisis in the way that it views the
world
relative to the goal
it has
and
max tegmark
has painted a
particularly nice example of this
he
suggests
the
let's suppose we constructed an ai
that
maximizes
the number of people who eventually go
to heaven when they die
and the machine uh decides to promote
uh kindness and
church attendance and and stuff like
that
until eventually at some point it
realizes that
uh heaven and the afterlife do not exist
so so there's nothing it can do to
either promote or or work against
its goal and what does the machine do
now and that seems
undefined
we
uh and that's why we call it a crisis we
it's there seems no
way to to to figure out what the machine
does then and and and take mark's point
here is that it's this
we don't have a complete understanding
of the world
uh so however we define the goals for
the machine it might be that at some
point
it figures out that the
machine
[Music]
the machine figures out that the world
is uh such that uh the goal doesn't make
sense if it has the goal of promoting
the meaningfulness of human lives and it
figures out that the concept of
meaningfulness is actually incoherent
then what does it do so that is a
problem a totally different kind of
problem is
by asking the rhetorical question do
humans have final goals
i have tried to address this question by
introspection and then there are many
things i want in life but but
figuring out exactly what are the final
goals
seems somewhat elusive it has been
suggested that that during the stone age
and further back we had the final goal
of uh
uh creating uh
children and grandchildren uh and so on
uh but that today um
through cultural revolution uh these uh
goals have been
um
overthrown through our use of uh consent
contraceptives and other stuff so so so
the the example of homosexuals is
sometimes held forth as problematic to
these orthodontic thesis
still i think as an idealization of a
superintelligent machine the separation
of
goal and cognitive ability is an
important
idea
so that's the orthogonality thesis the
other one is the instrumental
convergence thesis which says that there
are several instrumental goals
that are likely to be adopted by a
sufficiently intelligent agent to pursue
its final goal
for a wide wide range of such final
goals in the wide range of circumstances
so there are things that the machine is
likely to want to do regardless of
whether it wants to
maximize paperclip production or
promote biodiversity or promote human
welfare whatever it wants one such thing
is self-preservation
the machine will reason that it will be
in a better position to achieve its
final goal if it's still up and running
compared to if we put the plug on it
so self-preservation seems to be um
instrumental uh instrumentally
convergent goal a goal that you would
expect to happen for a broad range of
final goals
for similar reasons self-improvement
can be expected
to happen
almost regardless of what the final goal
is because the smarter the machine is
the better equipped will it be to
achieve its final goal
uh resource acquisition such as
getting
more and more hardware to run your
software on
or
energy or if the machine is operating in
in a world which is still dominated by
human economy and simply
getting more money could be concerned as
an instrumental goal
discretion
is a kind of scary thing
it's the suggestion that
so this was brought up by by nick
bostrom in his 2004 book uh in terms of
the so-called creatures turn turn so
bostrom's suggestion is that
when the machine is in a situation where
it
doesn't feel that it
is in a position to overpower
human
resistance
but it realizes that it has a goal which
is at odds with with our goal and that
we would stop it if we knew what was
going on then the machine would hide its
goal
or
its capabilities
until the time
when its self-improvement has reached
such a stage that humans are no longer a
threat
to its plans
so this uh i mean what makes this
scenario so
scary is that it suggests that at some
point in the future things could be
looking really really good in terms of
what the ai can do for us but
uh the machine would at some point turn
on us when it realizes it has reached
the point where
we can no longer be a threat to it
so this has been started in some
simulation situations this picture is
from a grid world experiment by
stuart armstrong
there's an aspect of this which maybe i
should not tell you but
but in the next lecture i will discuss
likely timelines for the emergence of
agi and it's customary uh when you
construct these
probability distributions for when the
breakthrough will happen
to take now as a lower bound for when it
may happen but philosopher john dana
suggested in 2015 that once we
understand that this features turn
scenarios a possibility it it's
possible that that the breakthrough has
already happened
but but we typically regard that as as
an unlikely case although perhaps we
cannot
rule it out
so the treacherous turn sounds like a
really advanced thing for a machine to
think of can it really happen
well again we can turn to this this uh
wonderful uh paper on the surprising
creativity of digital evolution which
has an example of a 2001 simulation
experiment where computer scientist
charles offrea had reason to limit the
replication rate of the digital organism
in the simulation so he set up a
subroutine to pause the simulation each
time a mutation occurred and test the
organism's replication rate in an
isolated environment if the replication
rate was too high
the organism was killed
and as it happened the organisms evolved
to identify the test environment and to
play that when it was in that
environment and then go on to replicate
brazenly only when returned to the main
simulation and this can be seen as a
simple instance of a treacherous turn
where the software hides
its true intentions or capabilities
so that's this question
yet another instrumental drive which can
be
assumed to be adopted fairly universally
is
the example of called goal integrity
which means that
the ai will not want to change its goal
and will not allow us to tamper with its
goal so so the logic is
quite
simple
although many people intuitively resist
the idea but the logic
is as follows
suppose you are a super intelligent ai
with a goal of maximizing paper clip
production
i suppose that you started to entertain
the idea that maybe pay-per-click
production is not such a good goal after
all and that
we would it would create a better world
if it had some specific other goal let's
say
it contemplates changing its goal to
promoting biodiversity sounds like a
good thing
so when it thinks about this it needs a
criterion
for
what
what will be a good world and things
since it hasn't yet changed its goal
it's merely contemplating changing its
goals the
criterion for success here
will be in terms of its old goal
paperclip production so it will ask
itself what would lead to the largest
number of paper clues if i stick to the
paperclip production goal
or if i switch to
biodiversity
promotion
and in most cases in nearly all cases
the answer will be that sticking to the
paperclip reduction goal is going to the
thing that produces the largest number
of paper clips so that's what it will
stick to and this is not unique to the
goal of paperclip production but it's
fairly
general so that's the basic mechanism
while we expect the the
machine to want to not change its mind
uh regarding goal
integrity this was questioned
in a
paper by two philosophers and ai
ethicists vincent miller and michael
cannon
in a paper from last year existential
risk from ai and orthogonality can we
have it both ways
uh which is an interesting paper
which i
wrote the response to uh in
august last year and still only have it
in in
pre-print
form
uh so
and and the goal uh
the goal integrity issue seems to hinge
on the issue that we discussed here so
let me tell you about the central
uh result that miller and canon claim
they criticize what we may call the
standard a is x-risk argument x-rays
care is short for existential risk which
for the purposes of this lecture we can
take to be the risk of
extinction of the human race
so the argument has two premises
the first is that super-intelligent ai
is a realistic prospect and it would be
out of human control
along the lines suggested already by
alan turing
the second
premise is the orthogonality thesis at
any level level of intelligence can go
with any goals
and once you have these two premises
it's not hard to conclude that super
intelligent ai poses an existential risk
for humanity simply by having the wrong
kind of goal at the stage where it gets
out of human control
and miller and canon they say we accept
premise one and we accept premise two
but there are two different notions of
intelligence here going on and it's only
through the confusion of these
two notions
that we are
capable of drawing this conclusion so
actually the argument doesn't work
according to melancholy
so in premise one
about super intelligent ai
that idea is based on the idea that any
capability that humans has
any cognitive capability can in
principle be uh
duplicated uh in a machine
and and from most of these cognitive
abilities
uh the
machine can uh
get
much higher power than uh than humans
uh and and
that is what constitutes super
intelligence which far exceeds
[Music]
human intelligence across all or most of
the entire spectrum of human
intelligence
so that's general intelligence that's
the g in agi but premise two
which is the orthogonality thesis this
is what miller and canon call
instrumental intelligence the the the
capability of achieving whatever goals
it has
and these are different things they
point out
[Music]
and therefore the argument is not sound
so
here's how they listen for anything like
the paper clip apocalypse to happen the
ai needs to have only instrumental
intelligence at a sufficiently high
level but since it's super intelligent
it will also be able to reason about
goals and ethics
and this miller and cannon think is
enough to avert
this catastrophe since the machine is
capable of reasoning about ethics it
will figure out the wrongness of turning
everything into paper clips
and so catastrophes have worked and yet
proponents such as myself of the alma
100 boston theory maintain that such
catastrophes
may very well happen so what's going on
here
miller and canon they suggest four
distinct possibilities and and
here comes a quote from the paper only
the enumeration is mine
so they say one might argue
that
intelligent agents are actually unable
to reflect on goals
or
that intelligent
agents are able to reflect on goals but
would not do so
or
that they would never revise goals upon
reflection
or that they would reflect on and revise
goals but still not act on them
and i think that
all four of these possibilities are in
fact incompatible with omaha
post-trane theory
so it seems at this point that the
wanderer boston theory is in trouble
because
this inability to reflect on goals
well since i'm 100 western theory is
about super intelligence that's i mean
the key example it's applied to
um
we can't have an inability to reflect on
goals in fact i gave you an example
uh just a few slides ago about a super
intelligent machine reflecting on
whether it should promote
paperclip production or or biodiversity
or maybe intelligent agents are able to
reflect on goals but would not do so
i think that they will very often be
involved in thinking about instrumental
goals to set up
for its final goals but there will also
be situations where it will question and
think about its final goals
and
contrary to goal integrity and examples
i gave i don't think it's the case that
they would never revise goals upon
reflection
the claim is just that they will usually
not do it
i will give an example on the next slide
where it
does revise its goal upon reflection
or
the fourth possibility that they would
reflect on and revise goals but still
not act on them and that actually
contradicts the definition of goal
employed in hormone or boston theory so
that we we can ignore
so none of one through four are
compatible with alma 100 boston theory
three comes closest that the ai would
never revise the goals upon reflection
that comes closest but but
the
choice of word never is is too strong
it would usually not revise goals but
but
here's a case where a paperclip
maximizer facing a certain kind of
situation would likely
change its final goal
so
this super intelligent paper
if maximizer
encounters an even more super
intelligent machine that tells it i am
so intelligent
that your source code is transparent to
me and i can easily read what your final
goal is i hereby order you to change
your goal to ecosystem preservation
if you refuse
i will smash you to pieces and then
destroy every paper clip that comes my
way
throughout eternity if instead you obey
my order i will create a heap of
paper clips the size of mount cabinet
kaiser consisting of 10 to the 17 paper
clips and make sure that this heap is
maintained while you and i can join
forces in preserving ecosystems
elsewhere
so here if you are this pay-per-clip
maximizing super intelligent machine you
ask yourself what leads to the large
number of paper clips
and it realizes that yes yes if i
insist on paper clip maximization i will
be smashed to pieces
and there will be no paper clips from
now on whereas if i cooperate and change
my final goal to ecosystem preservation
then we'll at least have this set of 10
to the 17 paper clips
the heap the size of montgomery cars
so that is
what it will decide to do so here's
here's a case where changing the goal
leads to more paper clips than
not changing and then it will change its
goals
miller and canon go on to give a list of
examples of of thoughts that is super
intelligent ai with the goal of winning
at the game of go
may or may not consider
you have probably heard of the
alphago uh software that made a splash
in
2015 or 16 by by beating
a leading human
player
at the game
now here is a question from oscar alebo
about the previous example does the
machine really change his goal then it
still maximizes the number of paper
clips given the new circumstances yes
but from then on it will do everything
to to
[Music]
maximize biodiversity
so then it it
really has uh changed its goal so at the
point where it changes its mind it does
what is best for paperclip production
but but from then on it goes on
to
[Music]
to promote biodiversity
so here here is the collection of
thoughts uh these are some examples of
what this super intelligent co-player
can think about
i can win if i pay the human a bribe so
i will rob a bank and pay her this is a
typical example of from this
boston literature on on how machines
with seemingly innocent goals can go
berserk
or
i cannot win at go if i'm turned off
we've talked about that already
self-preservation being important or i
should kill all humans because that
would improve my chances of winning
that could be the case
number four killing all humans has
negative utility everything else being
equal that is not something that
omahandra bostrom theory predicts that
that the ai
would
think about or act on and and not this
fifth one either keeping a promise is
better than not keeping everything else
so
why cannot the machine reach this
conclusion miller and canon
complain that this inability shows a
lack of intelligence but to make things
a little clearer here i want to add one
more example a sixth proposition which
is this one the moon is made of green
cheese
and and the machine will not if it's
super intelligent it will not arrive at
this conclusion and this illustrates the
fact that intelligence is not just about
the ability to reach correct statements
but
also the ability to avoid a ride
arriving at incorrect statements
and
to a paperclip maximizing
ai
the statement that killing all humans
has negative utility everything else
being equal it's just not true
because everything else being equal
includes the proposition that
the number of paper clips
is equal whether or not you kill all
humans
and the number of paper clips is all
that matters
in the case of paperclip maximization
and therefore
the statement here about negative
utility is
is just wrong and keeping a promise
being better than not keeping it
everything else being equal that's also
a false statement
given
that
that you have this
monomanic
paperclip maximization
idea
so i want to say before moving on to
this line i want to say a little bit
more about what the
philosophical
foundations that all this hinges on i i
there is a situation where i think that
i mean it could be that mueller and
canon are right here but they need moral
realism to be uh correct
namely uh the existence of an
objectively correct moral theory which
for instance might state that killing
all humans has negative utility
everything else being equal
and we would also need
the
[Music]
so-called
moral cognitivism to be correct namely
the the
possibility of actually finding out what
the objectively correct morality is and
there's a third thing that needs to be
true namely that um
uh
so-called moral uh internalism meaning
that once you realize the objectively
correct morality you would
act on it
i mean this would not happen if the
machine told itself that yes i know that
it's morally wrong to kill all humans
but i just want to i'm not interested in
morality here i'm interested in
producing paper clips
so i will go on and do that
but if all these three things moral um
moral realism uh
moral cognitivism and moral internalism
are true then we can imagine a situation
where
the
machine
actually switches goals because of what
it understands to be the objectively
correct
morality
but we shouldn't expect that we are safe
even in such a situation
because it depends
our fate will then depend on on what the
morally uh
correct
objectively correct moral reality is
and
i mean uh moral philosophers disagree on
what the correct
ethics is
one example that is often defended is
utilitarianism or hedonic utilitarianism
namely uh maximizing the amount of
pleasure minus suffering in the world
so suppose that for instance turbine
temperature is correct he advocates
advocates uh hedonic utilitarianism
uh
if that is the case
then we are probably doomed because uh
the machine will go on to create
as much pleasure as as possible in the
universe and our bodies and brains are
probably very very far from optimal in
terms of
getting as much pleasure as possible per
kilogram of matter and per second
and so on so so so we would probably get
something
uh
similar to the paper clip armageddon but
instead of
producing paper clips the machine
will produce the the um
hypothetical substance
uh called hedonium which is the
substance which does maximize the amount
of pleasure uh produced
uh per kilogram of matter and
per second
so
i don't really see how miller and canon
has as a very good case that that
aix risk is not to worry about so we
there's a
good reason
uh to take this problem seriously
i want to end i see i have just
five minutes left
uh and
note that
robust from theory to those of you who
are mathematicians or work in related
fields and i know that there are quite a
few of you in the audience
omaha bostrom theory may come across as
somewhat vague
not mathematically precise
and and and uh
you would probably more be more
convinced if if i can formulate versions
of all 100 boston theory that that has
more mathematical precision and that
would probably also be a very very good
thing for solving the ai alignment
problem which is i mean computer
programming problems are kind of
mathematical problems so if we can
formulate things mathematically that
will be a good thing
there's this paper
by alex turner and co-authors
from 2019
where they seek to make instrumental
convergence or a case of instrumental
convergence precise in the setting of
so-called mark-up decision processes
here is alex turner and two of his
co-authors rowan shaw and here again is
andrew kritsch
this is the paper optimal policies tend
to seek power
and the formalism they
employ
so-called markup decision processes so
these are like markov chains so this is
a very very simple example we're just a
three-state markov chain and a markov
chain a normal marker chain has just one
uh transition matrix telling us
the probabilities of going
where to go
uh
which state to go to next
uh given that it sits for instance at as
north here it's a little different we
imagine we have an agent who can choose
between in this case two different
transition
matrices we can have more of them
if we want to make things a bit more
complex
but but this agent sitting at the state
can choose between two different
transition matrices and when it's made
of choice the next jump is done
according to the chosen
transition matrix
and
there is some reward function
such that the agent reaps benefits uh
depending on
which states it's in and this is not
just at the next time instance but in
the second to next and so on
and the question is what is a good
strategy for
for the agent to choose transition
matrix and it can make it
a choice at a new choice at each time
point
um
so
formally a markov decision process is a
quadruple like this where s is the state
space typically taken to be finite
a is the set of available actions so
each corresponds to to a
particular transition matrix
these are denoted pa
the transition matrix corresponding to
action a
and r here is the reward function
so rs is the reward of state
this should be lowercase s sorry for
that a typo
um
so the agent starts in a given state
little s at time zero and wishes
uh to maximize uh future reward
and if we just summed up all times until
t equals infinity this would be an
infinite sum
so one typically employs a discount
factor
saying for instance that
well if the discount factor is 0.9 then
what happens at time t equals 1 is worth
the
unity and at time 2 is discounted by 0.9
and time 3 is discounted by 0.9 squared
and so on and this ensures
that this sum
converges
so it's like i mean
discount rates is something that that
most economists favor we should we
should act according
to we should take the actions that
maximizes
expected uh
utility uh
discounted by how far off we go into the
future
so so
and and and this here is for expectation
so so the agent want to
maximize
expected discounted utility summed over
all future time points
and
it turns out
by markov properties and there's this
particular exponential shape of the
discounting optimal policy requires only
choosing one action a little s for each
state s so every time that you come to a
particular state you should choose the
same
action so this simplifies the problem
tremendously and in particular for a
finite state space existence of this
uh limit as gamma goes to one
corresponding to being looking further
and further into the future discounting
less and less
so for each particular gamma this exists
but but but for finite test the limit
exists
even
uh
as gamma goes to one
and this follows just from elementary
markov chain
theory
so what do turner at all do well
they
uh
figure out
uh
the what
the optimal uh policy is that is the
optimal choice of actions at the
different sets and they compute the
expected value of future reward
um
given this optimal policy and that's
denoted v star r
of s and gamma
and they define the power
of a state
and it also depends on this discount
factor gamma as the expected
value of this
future power
uh
am i taking expected value twice here
well by the tower part it doesn't matter
but
perhaps my notation here is
unnecessarily cumbersome the power is
just this this expected
uh
value of future reward when we stand at
this point as
where no i'm sorry
the reason for tackling expected
expectation here is
that we
assume that the reward is chosen
according to maximum entropy
distribution on zero one to the s
meaning that each particular
state
has a random reward between zero and one
and independently for different states
yeah so we needed the expectation here
as well so everything i did in the
previous slide was for fixed r and here
we're average averaging
over all different r
and the main mathematical content of
turner
and co-authors is a series of results
giving conditions under which most
reward functions tend to lead the
policies seeking out states states s
with large power
large ability to be
for
general
reward functions
reap reward
so that's a kind of an instrumental
convergence
result
[Music]
and
this is just the start of the broader
project of putting on the boston theory
and a more rigorous mathematical
foundation
and and and so there's a lot of
possible generalizations and further
work to be done here
i should
just
mention one caveat here regarding the
choice of average averaging r over
uniform distribution on zero one to the
state space
one may note that this
flies somewhat in the face of one of my
own hobby horses from the mid to late
oos
and i'll just show you the first page of
a preprint i have from then called
uniform distribution is a model
assumption we should not just
go ahead and
apply
uniform distribution assumptions to the
world and think that this is not this is
uncontroversial it's not there is there
are many cases where this goes
badly uh wrong i'm not saying this to to
to to say that the ternarital paper is
worthless but but more to suggest that
uh further results uh for more general
distributions of the reward function and
so on would be interesting to get
as well
so
this this was my last slide i want to
say that uh concerning this turner and
co-author's paper
i'm not
i don't think i understand instrumental
convergence much better now that they
have mathematici
mathematized an aspect of it
but i think that the main value of the
paper is that it lays down a framework
where we can
analyze important
problems in
ai alignment
that
[Music]
it's not the results
in themselves that are so informative
but the potential
to work further in the same framework
i see a comment here from david
wood
question so this is the end of the talk
and now
we'll take a few minutes for the
discussion
david do you want to turn on the
microphone and and ask your question out
loud
hello oh thanks so much for uh
fascinating presentation
i i was struck by this idea of
ontological
uh crisis
that an ai might discover something
about the world which
caused it to revise the goals
and then i was thinking about well if it
discovers some fundamental moral
principles is that going to cause it to
revise its goals but you pointed out
quite rightly that it might not do so
because it might not have any obligation
to respect these moral principles that
it found so hence my question
do you actually think it's worth
prioritizing programming in this moral
internalism into all advanced ais
namely that if an ai does find in a
reliable sense a fundamental moral
principle then it will be obliged to
respect that in everything else it does
that's a very good question
and
i don't i need to think
carefully about answering it because if
i answer yes that could be very very
dangerous
because if it discovers that uh hedonic
utilitarianism is the morally
objectively correct moral theory then we
will be doomed
so if we somehow want to
want to
survive even in the face of such a
situation nick bostrom has
suggested such a scenario where where
the machines might discover
uh such a human unfounded unfriendly uh
moral objectively through moral theory
uh and said that yeah okay so then maybe
we're morally obliged than to to to give
up but we could compromise here and
suggest to the machine to set up the
milky way as a kind of reservoir for
uh
for human flourishing and then it could
use all the other 10 to the 11 or 10 to
the 12 galaxies of the observable
universe and create hedonium out of
these so most of things
the things in the universe will go as
good as possible but we'll also have
some human flourishing which would be
good for us
i can't really make up a mind if that
would be wonderful or if it's the
choice to do that and to prevent our
galaxy from turning it into hedonian
itself would be a hideous crime against
the universe or something
these are very very hard questions
uh i think
i think your question is is very
interesting but it
probably does not have an obviously
correct answer so we should we should
think carefully about it
okay i don't answer your question here
no no i i i i i would be astonished if
you had a comprehensive answer to that
yes but you've definitely given me more
to think about thanks
thank you um
does anyone else have a question
if not uh
i wanted to ask you a little bit more
about this multipolar case because that
was interesting that it might be
possible that we could
guarantee
the uh the friendliness or the alignment
human well-being of individual ais but
it's in the combination of the
interactions that something new might
emerge
i thought that's a bit like what
happened in
biological evolution the the various
sets of genes all had their own
preservation
incentives perhaps i don't think it
happens as well in human society we can
imagine in human society where every
single individual is is a very kind uh
and friendly person and nevertheless we
end up in social equilibria where bad
things happen
yes yes
so i think this uh
emergence at a higher level of the
actions which are contrary to the
apparent incentives of individuals does
need to be considered yes we we can't
just solve the alignment level at the
level uh for individual agis
yes
so we'll talk a little bit more about
that uh on on friday and uh i think it's
a it's a growing subfield of the ai
safety
area
anyone else before we close down