108. The Learning-Theoretic AI Alignment Agenda

welcome to session 108 in the AI safety
dot-com reading group the presentation
tonight is by vadim casaya research
associate for the machine intelligence
Research Institute his article on
learning theoretical AI alignment agenda
has recently won the first prize of
round three of the AI alignment press
please go ahead
okay so and I'm going to talk about my
essay which basically explains explains
my research agenda the the motivation
behind it and the general research
directions so okay so the overarching
goals of the research agenda are the
three items that I list here which are
the first item Monica is intelligence
and what it means is developing some
kind of natural theory aim what general
intelligence so intuitive terms with the
context of a I like general intelligence
is the ability to crucial certain goals
in an unknown environment effective in
some sense so this is like one sentence
explanation what we mean by intelligence
but sort of formalizing its with
medically
wish I could please mute your
microphones sorry is it muted it is not
muted okay so everybody if you look at
the picture and where the the the screen
should be there should be at bottom
where you can close the call two buttons
to the left as a microphone press that
one then it should be neutered am i
muted now no I was muted before then
okay
so if you keep speaking and I'm not
currently saying you are not muted in
that case you're probably muted okay
great so the second item is the
mathematical is in the role of alignment
so when we sailed in in artificial
intelligence is aligned what does it
mean exactly so it should mean that this
agent if in some sense first use the
same variance that we push here are the
same goals that we define it in the
sense that we mean it but again how to
formulate this mathematically is the
complex question and some of my hope or
so speak the holy grail of this research
program is that eventually this this
theory will produce a quantitative
theory of AGI performance and safety so
somehow the the vision is that
eventually we will have a theory which
which allows given a certain the I
design produce some quantitative
estimate of how safe this design is and
how Windows designers which of course
will have some empirical parameters and
some error bars and so on but at least
there will be some theoretical basis for
claiming that your design is safe or not
safe and the main building blocks which
margin the building is statistical
learning theory accommodation complex
theory algorithm formation theory we're
just briefly very briefly statistical
learning theory basically tells you well
it
questions such as how much data doesn't
algorithm need a learning algorithm to
convey eshton right answer and whether
it even can converge to the right answer
and computational complexity theory adds
to this the dimension of computational
resources so maybe have enough data but
it's not feasible to process this data
with some reasonable amount of
computational resources and so the sum
of those two is usually called
computational learning theory and like
the first part of written information
theory is basically well it's basically
what tells you what is the correct prior
you should use in some sense for your
agent to be a truly general intelligence
that somehow is able to reason about in
some sense some as broad domain of
problems as the thing that humans are
able to reason about so okay so the
first part is what I call inverse
enforcement learning and what I mean by
this is basically an attempt to answer
this first item here the automatically
divided of general intelligence so so
the starting point of this is the arocs
I which was introduction by Marcos
hotter air excise basically an agent yes
well that assumes its environment is
sampled randomly from a Solomonic prior
so what is the soul of prior so just in
two words a basic illuminance prior is a
way of mathematically formalizing the
idea of Occam's razor right so when we
well when we discuss just think about
what does it mean to think rationally or
you know you use the evidence that you
have in rational ways or in scientific
ways then like the central principle
there is Occam's razor which says that a
simpler theory
is more likely to be true their complex
theory and so much Breyers way to
formalize it we're like a complexity of
the theory is corresponds to the length
of a program push the universal Turing
machine that encloses theory so you can
sort of imagine the drawing a diagram
here it shows you in the ax I ancient
and you have like the environment the
well the environment gives some string
of some stream of bits into the agent
which are the observations of the agents
and this kind of presented sensors or
whatever input it gets from the
environment and then there are some
actions and agent takes to to affect
this environment and so you get some
process you get some closed-loop process
here and then if you assume that the
environment which agent doesn't know a
priori is the sample from the Solman of
prior then you can talk about expected
utility of the agent and the agent with
maximum expected utility by definition
is the ax ax so XS are just an attempt
to formalize what intelligence spins and
this is like a nice first attempt but
there are all sorts of problems with
this so ultimately what I hope this
theory will give us it'll give us some
set of optimality conditions an agent
should satisfy in order for the cold
general intelligence in some sense and
these conditions this should be
sufficiently strong so well I'll say a
few more words about this later it
should be sufficiently general in the
sense that the space of environments
that the agent can somehow effectively
operate and should be like a very big
space in some sense and the sample
complexity in computational complexity
should be feasible so as opposed to aja
X I which is not even computable we
should arrive at the theory that is
consistent with agents that are actually
feasible complete
it can be implemented with feasible
computational resources which is the
computational complexity part and which
converge to some the learning process
takes a reasonable amount of time which
is the sample complexity part okay so so
I'm going to speak about a few problems
with EXI a few problems that Exide
doesn't solve and which are important
remind you to arriving at this theory of
general intelligence and I'm going to
say just a few words about how I think
this problem should be attacked and how
I'm trying to attack them so the first
problem is the problem of craps so what
is a trap a trap is an irreversible
decision we have no long-term
consequences so it's something that you
do and which has potentially bad
consequences if you cannot undo wants
you to do this action and this and this
this thing creates problems so most of
existing reinforcement learning theory
assumes that there are known traps so
why does this yield because if there is
a crap in the environment then it's not
clear why you can learn that there is a
trap the only way to know for sure that
the trap is a trap is by entering the
trap and then it's sort of too late you
cannot get out of it so it's not clear
how to have some kind of guarantee that
your agent can learn you can
asymptotically learn about all the traps
and approximate some optimum performance
or something where it does so usually
for some learning theory just assumes
there are no traps or has some tricks
like the episodes which results are just
traps or irrelevant if some samples are
well liked one of the reasons it's
especially important in the context of
AI alignment is because a lot of
problems that arise in the context of
alignment we can think about them as
certain special case of traps so for
example of the agent self modifies in
some way that may
sit like changes for examples utility
functions some bad way or just you know
makes it like an ineffective agent and
you can think about it as a type of crap
like it's an action that your agent can
take that will lead to some bad state
and you cannot exit the state because
once addiction runs YouTube the action
and the original agent doesn't exist
anymore
and replaced with some other agent which
will do some bad things and corruption
is also sort of so what I mean by
corruption is like so in reinforcement
learning your agent usually experiences
incentives to do things which we would
consider bad for example it has some
external reinforcement signal which is
trying to maximize then it creates an
incentive to part of the signal and to
artificially set it to a high value and
just decouple it from what the humans
actually want and like entering this
source of states can also be considered
to be a type of threat so how can we try
to solve this so one approach which I
investigate and continue to investigate
is what I called DRL which means
delegative enforcement learning where
your agent can sometimes delegate agent
actions to a human advisor then you're
able to prove that you're assertive
you're set your agent will be capable of
learning about all the traps that the
humans and human knows exist in the
environment and in particular it solves
the corruption problems under some very
simplified assumption but in principle
it can solve the corruption problem so
other directions can be introducing some
assumptions about the environment or
formal evening soft romantic conditions
which are or weaker than what's usual of
course with learning theory but stronger
in the bias optimality ok so another
another problem of the ax sign has
is the lack of reflection so X I is
incomputable despite the environments is
able to reason about are computable and
so we can sort of instill exercise we
can consider some agent whose prior
includes only for example only
environment that have a given time
complexity but then the agent itself
will always have a higher time
complexity so by Indian agents almost
always have higher complexity than the
environment and this creates a paradox
because the real environment is usually
more complex than the agent for example
it can compromise it can contain other
agents which are just as complex as the
regional agent and so we need to somehow
take this into account the theory needs
to deal with this and one direction that
I'm exploring is the use of what I call
incomplete or fuzzy models which means
that your agent reasons in terms of
model and don't fully specify the
environment but specify just certain
aspects from the environment and then
you don't need the environment to be
more simple then the agent you just need
to environmentally have some properties
which are simple enough for the engine
to Express and then your agent will be
able to exploit this this properties
sorry I've already also some test cases
of how such a theory can be tasted so
game theory is like looking at several
agents interact with each other and
seeing whether we can prove the
conversion to some reasonably liberal
and like decision theoretic problems
where you have America that stimulates
the agent and the agent cannot simulate
Omega and self codification is not a
cased and of course another important
case for alignment is when you have
closed loop between a human and any I
and like at least one of them will not
be able to simulate the other so you
need like this theory of you need some
kind of reflected theory of enforcement
learning to the pro things about such
loops
now okay so the next the next part of
the agenda is value learning protocols
and here here I'm actually talking about
the alignment itself so what does it
mean for an agent to learn human values
and how can we make sure how can we
create setup state and sure
mathematically that this learning will
actually happen
so there are several ways how you can in
principle try to communicate values to
an agent and so one way is by vertical
formal communication so formal
communication means that you are
transmitting some signal to the agent
that has a priori known semantics so it
means that the agent it's sort of
hard-coded on some level into your AI
what a signal means so for example can
be just a full specification of the
utility function it can be a reward
signal or it can be just some comparison
some special cases like you're saying
regions of this situation is better in
this situation or this situation is
twice better than the situation of
things like that so what are what are
the problems with this so first of all
space it's like anything is very hard so
foolish spreads find usually a function
is somehow completely infeasible just
like a human various are very complex
like it's very hard to specify the
mathematically like you're very far from
there but even producing the reverse
signal becomes very hard if you're sort
of trying to communicate the sum total
human values as a Polish from so per
typical narrow problem so this is like
one problem with this another problem is
the incentives for corruption which I
already mentioned before
DRL can solve to some extent or this in
simplified models and the third problem
with this is scarce information which
means life like for example if I'm just
given the reverse signal to the agent
then it might be it will take a lot of
time for the agent to discover what it's
supposed to be doing for example suppose
I want the agent to
world war food you know to prevent a
nuclear war shall happen so it's your
wife will be you know zero when nothing
interesting happened and minus one if
there is a nuclear war and plus one if
somehow a nuclear war was eliminated
forever so something like this
but the problem is the nature will get a
lot of zeros in the beginning and it
will take it a lot of effort to reach
any state where some relevant reverse
signal exists so this is some limitation
of this so an alternative approach is
instead of formal communication we can
do something else which I call
demonstration so what is what what is
demonstration demonstration means that
the human is not is just trying to
communicate its values by just
demonstrating them but just doing things
that are aiming to optimize Disraeli's
showing the engines what what stage is
supposed to deduce from that what agent
what awareness is supposed to be
optimizing so this idea was very popular
and it remains popular in recent years
in different formulations so there's
like inverse reinforcement learning
where engine just observes the human and
tries to and guess what was there was
there utility function was there work
function is there RC IRL cooperative RL
when the agent of the human act together
singleton C on the environment and
there's divided are introduced which is
DRL delegated in versus force with
learning where the agent acts on the
environment and can sometimes give
control to the human and allow the human
in step to act on the environment and
see what the human does so the
advantages of this is in some sense
easier in the sense that you don't need
to specify digital the function you just
need like to do things in some sense and
like it it has
might have richer information because
the the human is doing the demonstration
is already moving towards some
reasonable goals so from the start you
have some information about
what the human is trying to do
challenges with this is of course the
demonstration can be imperfect can the
human to make just errors name of the
demonstration or introduces with some
very suboptimal or irrational things and
again the human can get corrupted so so
I'm the signal from the human to the I
can be hijacked or the human can be
manipulated and so forth and and another
problem is that or perhaps a related
problem is that it's limited by the
capacities of the demonstrator so for
example if I'm offered a bet whether to
play a game of chess against Kasparov
then I will not take the bet because I
know it cannot win such a game of chess
but maybe I would like the AI to take
that if the I would react in my step
because the I can actually win against
the spiral but the I might might never
be able to learn at this because it will
never be sure ever I'm rejecting these
bats because I don't think I can win or
inject ejecting its bats for some
completely other reason the bouncy
people don't like hate playing chess or
soft OFS and like this is related to B
versus NP so giving the reverse signal
is like NP we were just evaluating a
solution and and the during the
demonstration is not available solutions
producing a solution so this beavers
versus NP problem the main test is there
is evaluating solutions is much easier
than producing solutions and this means
that in some sense producing the reverse
signal there's no computational staff
reason if a signal is actually much
easier than than demonstrating so so one
one idea I have about how to sort of try
to get the advantages of both as much as
it's possible is a protocol that I can
learn by teaching and like in learning
by teaching here three actors so you
have your agent you have which is di you
have someone called the adviser which
presumably be human and an operator
which might be American so the operator
acts on the environment and the advisor
gives some advice to the to the operator
there are three moves between which the
ITM switch on its own so in one mode the
operator is doing things and the advisor
is giving advice in the second mode the
operator is given link is doing things
and the agent is giving advice instead
of the advisor and the third mode the
agents just acting directly on the
environment and the operator and the
advisor down into their anything
and so what what's the idea here the
idea here is that by giving different
source of advice there I can learn what
what the operator is trying to do
because well like on the one hand if
you're doing something but you receive
advice from some really smart and allows
you to search solve much more
complicated problems than the problems
you would be able to solve on your own
so this is this in principle allows the
operator to transcend its usual
abilities on the other hand the I can
understand which advice is good and
which advice is bad because of the basic
principle that the operator will simply
not respond to the bad advice or maybe
it will try it and then we'll stop using
it whereas good advice the operator will
keep using it and the agent will be able
to conclude that the operator actually
likes this advice and like it's it's
moving towards the correct a goal and on
the third hand there is still the
delegative mechanism which protects us
from manipulation because the I will
sort of sample its advice from the space
of advice advices with the adviser the
human adviser can produce and will avoid
advice that might be dangerous might
manipulate the operator and make it
reacting in correct ways so in principle
this approach should allow us to
transcend the ability of the operator so
give a certain better result than just
usual demonstration
and avoid corruption so and you don't
need to Manuel spats very war signal
okay and so so ultimately I hope that
whatever the ultimate protocol is going
to be what I hope is that this protocol
will be they have some philosophical
backing by some theory of imperfect
rationality so we're going to have some
mathematical theory which tells what
does it mean for an agent to have
certain values or utility function or
some other presentation of values
despite the fact that the agent is not
perfect not perfectly rational and
software and like for example one model
that sort of emerges from thinking about
is learning by teaching over it is that
your agency is going to be not perfect
because it as a limited hypothesis space
there are parts is that too complex for
it to contemplate it can become
corrupted some some states when the
agent the human enters them it stops
just behaving rationally in any sense
and it can just randomly do some errors
and and the learning by teaching
protocol can sort of overcome in some
sense all three types of irrationality
overcomes the internet for space because
the AI can have a much bigger potted
space it avoids corruption by the
delegation mechanism and it can avoid
the random blunders because when they I
already learned enough can directly on
the environment and somehow simulate
some perfect version of the operator in
some sense but this is not necessarily
the final model but hopefully we'll have
some model which sort of explains what
imperfect rationality is so some other
topics that are included in my agenda
are well one topic is the topic of
taming demons so why are these demons so
demons are analyzed sub engine so you
might have an ancient which is in
principle sorry
which is in principle aligned in some
sense or for some protocol which should
there ensure a good which has some
performance guarantee in some sense but
it has sub agents that are not aligned
so how can this happen for example if
your age include some evolutionary
algorithms then the evolution of Russia
can create some intelligent agent we
have some different agenda like
biological evolution created humans by
optimizing for genetic fitness Britain
humans which optimize for other things
another concern is that your agent might
be simulating other agents it might be
just thinking about the world and
imaging other agents which exist or
might exist in the world and it runs
simulations on this agent some of them
might be malicious and some of them
might be able to actually escape from
this box which is which is the original
agent and like the third example is the
causal attack of all Kristiana where
some malicious super intelligence sort
of runs a lot of copies off of your AI
which seem like there are which all
these copies they see something which
looks like ordinary earth and but there
are much more of them than earth and it
causes your eye to think that it's in
there it is one of these copies and it
is not on real earth this allows the
malicious super intelligence to
manipulate your AI so I defy this agent
this demons into two types
so there are Cartesian demons and non
Cartesian demons and Cartesian demons
are those that just usually use the
usual input output channels of your
originally I to do bad things and the
non Cartesian agents are of the rest so
my producer agents are sort of work by
sort of trying to hack your eye from the
inside find some error some bug in the
code in the ratings
in the hardware or just transmit
information outside using some physical
effects like magnetic induction or
whatever and use that and in principle
Cartesian agents should be blocked by
the same performance guarantees that
fall from the theory that we have before
so like if we are if we don't have a
theory of universal reinforcement
learning plus very learning protocols
which gives us some optimality
guarantees that's in particular imply
that our AI is going to remain aligned
then this blocks to the Cartesian agents
also just by definition and like for
example this a causal heart attack is
blocked by delegation because you're a
is able to delegate to the human at the
correct moment of time and use this to
distinguish between this whether it's in
the simulation or really an art so the
Cartesian the agents are relatively easy
although we have to be careful for the
quantitative analysis of her performance
guarantees which is not simple the non
Cartesian agents at the moment it's much
less clear how to solve its I do not
really see a purely learning theoretic
solution so one solution that sort of
works is just using a morphic encryption
which is a certain way to encrypt the
execution of your AI in a way that makes
it look like just execution of random
code and so under certain assumptions
reasonable assumptions about physics
such an encrypted in air will never be
able to sort of break out of the box and
you can sort of communicated by
encrypting and decrypting these inputs
and outputs and then like it's normally
personally become the only channels of
communication all the other channels are
cut off but it has like very serious
performance penalties and it might make
your eye uncompetitive against some
variants so maybe there are some ways to
do instead of using extra homomorphic
encryption you something weaker like her
which has some with some form of
validation didn't even it but okay but
it it is a complicated problem okay and
the last topic that I wanted to talk
about and sort of the last sort of item
on my research agenda is
self-improving here so self-improving is
a I did sort of improves itself and so
this idea is interesting because it
might lead to some kind of exponential
feedback loop are you reaiiy improves
itself and the improve the AI improves
itself even faster and so on but some of
this is just an informal idea there so
it's interesting to try to come up with
some formal mathematical model of this
and well and there are some things in
the theories I mentioned before that can
be used for here for example I already
said the full lots of modification is a
kind of traps so for fear which deals
with traps can also be used to deal with
sort of house safety of self
modification this is one thing another
thing that you can focus on education
sort of game where the players are the
set of all possible agents or all agents
that your agent can modify into and then
if if you if you will have this result
as I sort of mentioned before you might
want to have that your well your
Outworld converges into some kind of
game theoretical Librium then you can
start to apply to this game and then it
will imply that your agent will actually
do some modification in good way since
instance so it will you know it will
play its optimal strategy in the game
you can insult yourself modifying the
things it should sell for defending
ourselves modifying their things and
another just further about this is the
sort of self improvement so it sort of
seems hard to understand how to
formulate learning theoretically what
does mean this sort of feedback loop
how do we think about it so one way I
think which might be interesting how to
think about it is that self-improvement
is really sort of flowing on which
hardware you are implemented because
usually when you talk about
computational complexity then we talk
about complexity up to a polynomial
which is why is it why we always talk up
to full you know no because then it
doesn't depend on the models computation
it doesn't depend on what sort of
machine what science computer implements
your algorithm and and so eventually we
will come up with some algorithm that
has some performance guarantees and that
it will run in some polynomial time but
this polynomial time will not be optimal
for the particular computer in which you
are going to implement it so there is a
self-improvement we can try to model it
as they are trying to learn on which
hardware it is implemented and find well
and somehow modify itself into the
implementation which is optimal for this
harder and then maybe maybe you can
which make to analyze it and find it
under some condition you really have
this exponential growth place that
people have been sort of speaking about
or not okay so I think it I'm I think
that I finished the presentation yeah
okay thank you for your presentation
venom if anyone in the audience have any
questions or comments please come with
them now and if you come up put some
more questions well people are talking
then please write them in the chat so we
don't talk at the same time so who would
like to feel the first question
I would please go ahead okay um I just
wanted to pass our cross I wanted to ask
for clarification on the learning by
teaching why exactly could the a I not
just say hack the advice is to give
particularly easy advice or anything
like that how does that prevent um
corruption yes I understood the question
so learning by teaching basically uses
the same mechanism to defend from
corruption like all the other delegative
a learning protocols so the basic
mechanism is that well first of all the
agent knows that there is such a thing
as corruption so it assumes that some
states some states are corrupted and if
you enter a corrupt state then you
cannot no longer believe anything and it
knows it it should avoid the state and
the way it it the mecca the tool it has
the widest ways is by allowing allowing
the adviser to act first and then when
it learns the behavior the adviser then
it understands which things are safe
so the basic assumption is that the
adviser will not will not become
corrupted on itself and the operator
will not become corrupted all on itself
so he assumed it this adviser operator
as long as as the act without
interference of the eye
they will not grab himself then the
agent can learn this certain behaviors
sort of behaviors in this advisory over
display are safe and the states that
they entered in this behavior are safe
states and it can then use this
knowledge to also behave always are safe
this is like the basic mechanism
okay um Stewart Armstrong had a comment
yeah I just wanted to ask if you're
based in Berkeley and if you're going to
be there in the second half of August
okay so the answer is that I'm not based
in Berkeley I'm based in Israel and I'm
certainly not going to be there in the
second half of August I mean I might be
there sometime in the future probably
will be at some point but certainly not
not this month okay
then we should have a discussion after
August after I formalized some things
there and some comments on your thing
I'll send you one link to something that
I did to try and remove a causal trade
you may find it relevant to what you
were thinking about of course Cheers
okay there is a question from Linda or
possibly some of the other people
together with her asking if the operator
is a human in the value learning
protocol so in the simplest
interpretation of the protocol yes the
brain is human you might try to come up
with some or okay in the simplest
version the upper there is either a
human or maybe some organization of
several humans or something produced by
humans but you might also try to imagine
some more complicated versions which are
more like a post Cristiano's thing where
have some chain right where each sort of
agent teaches to the next agent in the
chain so I think it might be interesting
to think about that also but so far I
haven't constructed any mathematical
model which does this thing but it might
also be used for directing
okay there are other people writing but
I guess I could field one of one
questions my own there are as I
understand it three or at least three
research agendas there is your learning
theoretic agenda the spall cristianos
alba agenda and the 18th foundation's
agenda
it seems focused Yanis agenda focus a
lot on the being very strictly
competitive with the the best machine
learning techniques deep learning etc
whereas the the agent foundations agenda
is all the way to the other end of the
spectrum with not focusing on
performance and all at all and just
trying to find a feasible solution it
seems you're learning theoretical agenda
is somewhere in between is that a
reasonable characterization so I think
you could say that in some sense it is
in between so it is similar to pause the
gender in the say in the sense that in
the sense that I'm using mostly sort of
conventional or relatively conservative
mathematical tools formalism which are
usually which people developed for
machine learning it's sort of different
from Paul in the sense that are more
focused on so Paul seems to be focused
on constructing there's some informal
solutions which are good enough in some
sense and then somehow gradually
formalizing them later I'm my focus is
more on doing things with where you can
actually prove theorems completely
rigorous way even if the theorem is
initially in some very simplified model
so I prefer to start with a very
simplistic model but about which you can
actually pull things rigorously and then
gradually move on to more and more
complex models rather engine so don't
imagine the final solution and only then
trying to make it rigorous and also I am
so the focus on reinforcement learning
while Cristiano prefers to sort of a
more roundabout approach where instead
of doing the enforcement learning he
thinks about things like systems that
answer questions and things like that
which somehow in a roundabout way allow
us to create systems that connect on the
world so yeah like the agent on the
other hand on nearest agenda is well
it's similar to mine in the sense that
also tries to construct some formal
medical theory but it's different in the
sense that well the mathematical tools
in nearest agenda and the sort of the
subproblems that the emphasize are
different from from the em from the
tools that I use and the emphasis in my
agenda okay yeah
Rafe had a question in the chat I don't
know if you can see the chat does the
advisor have very completely benevolent
what if it advises the operator to do
something because it's good for the
advisor but bad for the operator well at
the moment are assuming that you know
everyone all the advisement overs are
both benevolent so I'm not trying to
think about any conflict of interest
between the humans so it might be
interesting at some point of the future
or maybe even its importance of the
future we'll think about scenarios where
you have several humans that have like
different interests or sort of trying to
pull the and different directions but I
think that this is like but for my point
it's some kind second-order fact like we
need to if we don't even know how to
solve the problem you know in the case
where all the humans are perfectly
benevolent incorporated them they're
going beyond this is somehow seems too
ambitious this point
the next question from Alexander is
about the relationship between
homomorphic encryption and the AI box
problem this is something we've actually
bred in the as50 reading group paper
about I think half a year ago or
something so I could get the but the
reference but that you could you sketch
how that works yeah yeah so so say some
chatter explain it briefly so so a
memory decryption is a way to well
execute a problem in encrypted form then
create both the program and both the
input and you get an encrypted output
that you have them to decrypt and the
property has is that if someone who
doesn't have the key looks at the
computation then then this someone
cannot distinguish between this
computation and just some random program
which is running on some random input
and why is this okay so why is this good
for sort of boxing VI because it means
that if you execute your AI in this
encrypted form then well you see if we
assume that the universe is also limited
in computational resources and like it
seems like it is so from our
understanding of physics the universe is
in some sense in the complexity class P
Q P so it can only do things which are
in some sense running quantum polynomial
time which should still be not enough to
break the encryption scheme probably of
its own snow rhythms conjectures but
assuming it's not enough and it means
that the universe the physical universe
will not be able to tell what is we
won't be able to distinguish between
that the program runs inside the box and
just some random problem and this means
that this program will never be able to
have any unintended side effect on the
universe which you know any side effect
that it would have destroying a random
so you just cannot optimize the universe
in any bad direction so if you had
purely information theoretic encryption
it's like a home about freaking company
I'm not sure maybe I just read the paper
could you set it to me sir yes in the
mean time which is this it seems this
learning theoretical gender it's mostly
probably limited to a GIS agents that
are based on learning there are other
options for artificial general
intelligence agents that are search
planning or based on logic to see could
they be aligned with some of the same
methods so I so my point of view is
different so I do not see this agendas
limited to a particular type of agent
and I think this is actually important
so the starting point of my agenda is
thinking about what sort of mathematical
desiderata
can be satisfied by any sort of agent
you know one said what is the other are
impossible cannot be satisfied one days
in your honor are possible and can be
satisfied and then you know and then we
can say something more even more
quantitative about the satisfaction of
these conditions so I'm trying to get at
the mathematical conditions that some
algorithm should satisfy in order for us
to call this algorithm an intelligent
algorithm so it doesn't matter what is
inside the algorithm from the inside the
algorithm might use it might use neural
networks it might use formal logic it
might use something that you don't even
know about currently but the the point
is that if we prove some theorems that
you know something cannot be satisfied
by any algorithm then you know it
doesn't matter what lower court we'll be
able to do it on the other hand if we
pulled it know that some criteria can
satisfy them it can be said so I then
you know it doesn't matter how we we did
the important thing that he says but it
actually sets by this criteria so so my
point of view is sort of trying to
correct characterize what is mean for an
agent to be intelligent rather than
study the properties of particular
algorithm yes I don't think that's been
Lucas has just written a message in the
list of problems with I see a I X I I
didn't recognize anything that looked
like the ontological identification
problem is that problem problem you
consider important ok so this is so ok
so this is an interesting question so
the thing is that this ontological
education problem it is possible that we
can in some sense avoided but ok so why
is it possible that we can in some sense
of worried it because like ok if you're
if you have some agent that has some
ontology or about the universe and its
values are defined in terms of this
ontology then you can probably still
explain its behavior in terms of some
utility function that is defined just in
terms of actions and observations
because the beliefs that the agent has
about the objects in its intelligent are
a function of its the observations that
it has seen and therefore we can
translate the to the function from
whatever ontology the agent is using
intrinsically through the space of
actions and observations and therefore
if we so it might be enough to just look
at additions that agents that only look
at actions and observations and then if
we have in particular a very learning
particle for some agents then it then
such agent will be able to
various without somehow explicitly being
with the ontology thing on the other
hand it might still be interesting but
because this translation of utility
function from some ontology collection
observations can starting from some
simple utility function produce some
very a complicated function which some
less good mathematical properties then
it might be useful to to do the opposite
which means which means trying to model
this ontology thing explicitly and then
for example we can think about it in
terms of an agent which plays as a
casting game so an agency writes with
the environment with okay the agent
works with some state with some with
something which represents its internal
value ontology and this something is
acted upon some other agent which
represents the external environment
which represents nature so to speak and
then au revoir function can be a
function of the states inside this
ontology so I think this mathematical
model in my view captures sufficiently
nicely what it means to have variously
find the so intrinsic ontology and then
we can ask what sort of we read bounds
we can prove in this formalism and what
implications doesn't have about very
well and so forth so this is so I think
this is an interesting research
direction
I'm not search not currently sure
whether it's a high priority or less
high priority but certainly it's
something that also we should think
about okay I have another question and
one thing that your article didn't talk
about was the concept of courage ability
it seems like the addition of courage
ability would makes a lot of these value
transfer
protocols easier in the sense that if
the 18 was sufficiently corrigible the
problem of transferring the values would
be simpler in that we if we did not hit
such a narrow target then that doesn't
matter so very much if the agent is
critical so so personally I don't like
the concept of courage' ball very much
and the reason I don't like it is
because I feel it is not sufficiently
formalized and I feel the different tip
to the point I think the different
people use it to mean different things
so it's not even specified in the
informal level not not mention the
formal level so the way I so the general
method of my research agenda is that the
way we arrive at alignment is by proving
some performance guarantees which for
example take can take the form for grade
bounds and this performance guarantees
they they show they should imply that
your agent converges to the RPM value of
the utility function where the utility
function is the correct tilted function
of the human in some sense so this is
sort of the the way I am trying to
mathematically formalize the concept of
alignment and to prove the certain
algorithms are alive so from my
perspective if I manage to prove this
then I don't need college ability or you
can say that this is you can just say
this is what courage ability means if
you want and if I cannot prove it then I
don't see how college ability is going
to be enough so at this point I just
don't see it as very useful concept on
top of the concept that I already
introduced in the agenda ok and we are
nearing the end exactly has a question
about further to this homomorphic
encryption scheme
and I don't know if you have any if you
have can read it yeah if you have yes I
can read the chat so so fourth question
is there a possibility results are
unconditionally secured market
encryption in other words can you only
have our confusion schemes that we need
to assume that verse we had only time
well well you need to assume something
right so like if if you're if the
adversary the de strive to break the
scheme has unlimited computing resources
then you can always sort of try
impossible keys for example or something
like this so in general encryption
there's such thing as like information
theoretic description like one-time pads
and things like that but information
theoretic encryption doesn't help you
here because for information theoretic
encryption to work you will need to
block the passage of information between
the I and the universe in some way but
this just seems physically impossible
because like there's no way you can you
know you can completely eliminate flow
of information so okay so in some sense
the other approach that I mentioned form
of verification it's not really
encryption but you can sort of think
about it as this sort of information
theoretic security attempt because so
the idea there is that we sort of have a
model of of our of our hardware and we
formally proved it inside this model the
communication bandwidth from anything
inside the computer to anything outside
is bounded by some small constant and
then we will have some learning
theoretic game theoretic analysis which
shows that this constant is good enough
and but then again to help something
like this you still need to assume
physics so your analysis would still
assume a physical model so I don't see
any scheme that will not assume anything
about physics because like this seems
almost by definition impossible I might
be missing something but we just don't
see how it can be done okay I think we
are just about ready to wrap up so the
final question will be from Lucas asking
if you have written anything about
ontological identification so so no so
unfortunately not so this is a topic so
I just have not did not have time yet to
address these topics there are main
things I believe that I will write
something at some point but currently I
don't have anything yet
okay then so I will just say good luck
of course with your research agenda and
thank you very much for coming here
tonight and we've enjoyed it very much
and then to everyone if you want to join
next week at the same Wednesday
equalities 207 UTC will be reading the
article the malicious use of artificial
intelligence and discussing that and
otherwise I will just say thank you for
tonight and see you next week


Concrete Problems in AI Safety (Paper) - Computerphile

Today I thought I'd talk about a paper fairly recent.
It was last year
A paper called "Concrete Problems in AI Safety"
Which is going to be related to the stuff I was talking about
before with the "Stop Button". It's got a bunch of authors; mostly from Google Brain
Google's AI research department, I guess..
Well a lot of it's AI research, but
specifically Google Brain and some
people from Stanford and Berkeley and
opening iEARN. Whatever... it's a
collaboration between a lot of different
authors
The idea of the paper is trying to lay out a set
of problems that we are able to
currently make progress on like if we're
concerned about this far-off sort of
super intelligence stuff.. Sure; it seems
important and it's interesting and
difficult and whatever, but it's quite
difficult to sit down and actually do
anything about it because we don't know
very much about what a super
intelligence would be like or how it
would be implemented or whatever....
The idea of this paper is that it... It lays
out some problems that we can tackle now
which will be helpful now and that I
think will be helpful later on as well
with more advanced AI systems and making
them safe as well. It lists five problems:
Avoiding negative side effects, which is
quite closely related to the stuff we've
been talking about before with the stop
button or the stamp collector. A lot of
the problems with that can be framed as
negative side effects. They do the thing
you ask them to but in the process of
doing that they do a lot of things but
you don't want them to. These are like
the robot running over the baby right?
Yeah, anything where it does the thing
you wanted it to, like it makes you the
cup of tea or it collects you stamps or
whatever, but in the process of doing
that, it also does things you don't want
it to do. So those are your negative side
effects. So that's the first of the
research areas is how do we avoid these
negative side effects.. Then there's
avoiding reward hacking, which is about
systems gaming their reward function. Doing something which technically counts
but isn't really what you intended the
reward function to be. There's a lot of
different ways that that can manifest
but this is like this is already a
common problem in machine learning
systems where you come up with your
evaluation function or your reward
function or whatever your objective
function and the system very carefully
optimizes to exactly what you wrote and
then you realize what you wrote isn't
what you meant. Scalable oversight is the
next one. It's a problem that human
beings have all the time, anytime you've
started a new job. You don't know what to
do and you have someone who does who's
supervising you. The question is what questions do you
ask and how many questions do you ask
because current machine learning systems
can learn pretty well if you give them a
million examples but you don't want your
robot to ask you a million questions, you
know. You want it to only ask a few
questions and use that information
efficiently to learn from you. Safe
exploration is the next one which is
about, well, about safely exploring the
range of possible actions. So, you will
want the system to experiment, you know,
try different things, try out different
approaches. That's the only way it's
going to find what's going to work but
there are some things that you don't
want it to try even once like the baby.
Right, right.. Yeah you don't want it to
say "What happens if I run over this
baby?" Do you want certain possible things
that it might consider trying to
actually not try at all because you
can't afford to have them happen even
once in the real world. Like a
thermonuclear war option; What happens if
I do this? You don't want it to try that.
Is that the sort of thing that.. Yeah, yeah..
I'm thinking of war games.. Yes, yeah.. yeah. Global
Thermal Nuclear War . It runs through a
simulation of every possible type of
nuclear war, right? But it does it in
simulation. You want your system not to
run through every possible type of
thermonuclear war in real life to find
out it doesn't work cause you can't.. It's
too unsafe to do that even once. The last
area to look into is robustness to
distributional shift. Yeah
It's a complicated term but the
concept is not. It's just that the
situation can change over time. So you
may end up; you may make something.
You train it; it performs well and then
things change to be different from the
training scenario and that is inherently
very difficult. It's something
humans struggle with. You
find yourself in a situation you've
never been in before
but the difference I think or one of the
useful things that humans do is, notice
that there's a problem a lot of current
machine learning systems. If
something changes underneath them
and their training is no longer useful
they have no way of knowing that. So they
continue being just as confident in
their answers that now make no sense
because they haven't noticed
that there's a change. So.. if we can't
make systems that can just react to
completely unforeseen circumstances, we
may be able to make systems that at
least can recognize that they're in
unforeseen circumstances and ask for
help and then maybe we have a scalable
supervision situation there where they
recognize the problem and that's when
they ask for help. I suppose a simplified
simplistic example of this is when you have
an out-of-date satnav and it doesn't seem
to realize that you happen to be doing
70 miles an hour over a plowed field because somebody else, you know, built a
road there. Yeah, exactly. The general
tendency of unless you program them
specifically not to; to just plow on with
what they think they should be doing.
Yeah. It can cause problems and in a large
scale heavily depended on , you know , in
this case, it's your sat-nav. So it's not
too big of a deal because it's not
actually driving the car and you know
what's wrong and you can ignore it
As AI systems become more
important and more integrated into
everything, that kind of thing, can become
a real problem.
Although, you would hope the car
doesn't take you in  plowed field in
first place. Yeah. Is it an open paper or does it
leave us with any answers? Yeah. So
the way it does all of these
is it gives a quick outline of what the
problem is. The example they usually use
is a cleaning robot like we've made this.
We've made a robot it's in an office or
something and it's cleaning up and then
they sort of framed the different
problems those things that could go
wrong in that scenario. So it's pretty
similar to they get me a cup of tea and
don't run over the baby type set up. It's
clean the office and, you know, not knock
anything over or destroy anything. And
then, for each one, the paper talks about
possible approaches to each problem and
things we can work on, basically. Things
that we don't know how to do yet but
which seem like they might be doable in
a year or two and some careful thought
This paper. Is this one for people to read? Yeah, really good. It doesn't cover
anything like the range of the problems
in AI safety but of the problems
specifically about avoiding accidents,
because all of these are
these are ways of creating possible
accidents, right? Possible causes of
accidents. There's all kinds of other
problems you've been having in AI that
don't fall under accidents but within
that area I think it covers everything
and it's quite readable. It's quite... It doesn't
require really high-level because it's
an overview paper, doesn't require
high-level AI understanding for the most
part. Anyone can read it and it's on
archive so you know it's freely
available. These guys now working on AI
safety, or did this then
They've hung their hat up. They've
written a paper and they're hoping
someone else is gonna sort it all out. These people are working on AI
safety right now but they're not the
only people. This paper was released in
summer of 2016, so it's been about a year
since it came out and since then there
have been more advances and some of the
problems posed have had really
interesting solutions or well.. Not
solutions, early work, that looks like it
could become a solution or approaches
new interesting ideas about ways to
tackle these problems. So I think as a
paper, it's already been successful in
stirring new research and giving people
a focus to build their AI safety research on
top of. So we just need to watch this space, right? Yeah, exactly..


The AI revolution and international politics | Allan Dafoe | EAG 2017 Boston

next up it is my honor to introduce
Professor Alan Defoe he is an assistant
professor of political science at Yale
University and a research associate at
the future of humanity Institute at
Oxford where he is involved in building
the AI politics and policy group his
research seeks to understand the causes
of world peace and stability
specifically he has examined the causes
of the liberal peace and the role of
reputation and honor as motives for war
along the way he has developed
methodological tools and approaches to
enable more transparent credible causal
inference more recently he has focused
on artificial intelligence grand
strategy which he believes poses
existential challenges and also
opportunities and rich and which
requires us to clearly perceive the
emerging strategic landscape in order to
help humanity navigate safely through it
to discuss his work on AI of
international politics please welcome
professor Allen Dafoe
thank you
[Applause]
Thank You Nathan so I will actually
start by talking about war and then
we'll get to AI because I think there's
some lessons for effective altruism so
war is generally understood to be a bad
thing it kills people it maimed them
destroys communities ecosystems and is
often harmful to economies there's some
we want to add an asterisk to that
because in the long run war has had
dynamic benefits but in the current
world war is likely a net negative that
we would want to avoid so if we were
going to start a research group studying
the causes of war for the purposes of
reducing it we might ask ourselves well
what kinds of war should we study there
are many kinds of wars that have
different causes and some are worse than
others so one classification that we
might employ is to bin Wars in terms of
how many people they kill there's some
Wars that kill only a thousand to ten
thousand some that kill a hundred
thousand for the million and so the
x-axis here are the bins of the fatality
the battle deaths and Wars and the
y-axis is the fraction of wars of those
kinds so an instinct we might have is is
to say well let's study the wars that
are most common right those on the the
first bin these are the wars that are
happening today in Syria civil wars I
availability bias would suggest that
those are the words we should worry
about some of my colleagues have argued
that great power war the big Wars world
wars are a thing of the past
right the liberal piece democracy
capitalism on nuclear weapons have all
rendered great power war obsolete we're
not going to have them again the
probabilities too low but as effective
alterus we know that you can't just kind
of round a small number down to zero or
you don't want to do that you want to
really try to think carefully about the
expected value of different kinds of
actions and so it's important that even
though those probabilities of a war of
say a million a ten million one hundred
million or a billion is very small it's
not zero if we look at the past seventy
years
of history World War two stands out as
the source of most of the battle deaths
that have been experienced so that would
suggest that fatalities is a good enough
proxy for whatever metric of importance
you have that we first really want to
make sure we understand World War two
and the kinds of wars that are like it
that are likely to kill so many people
we can zoom out more and World War one
comes out of the sort of frost of civil
wars and really those two wars loom
above everything else as the the thing
we want to first explain at least to put
prioritizing importance we can see this
in this graph which again has these bins
of size of violent quarrels now we're
going all the way down to homicides and
we see that the world wars the
10-million bin on the right contains
again most of the deaths in violent
corals so that again would suggest that
it's really important that we understand
these big Wars but of course the next
war need not limit itself to 99 million
deaths there could be a war that kills
hundreds of millions or even billions or
even 6.5 billion now the problem
empirically is that we don't have those
wars in the data set so we don't know
how to estimate non parametrically the
expected value in those we can try to
extrapolate from what's very close to a
power-law distribution and pretty much
no matter how we do it unless we're
extremely extremely conservative we get
a distribution that looks like this
which says that most of the harm from
war comes from the wars that kill a
billion people or more of course we
haven't had those Wars but nevertheless
this follows from other reasoning but
even still we go further the loss from a
war that kills 6.5 billion is not just
those 6.5 billion people that die the
currently existing people it's also all
future people right and this is when
where the idea of existential risk and
sort of existential value comes in we
have to ask ourselves what is the value
of the future right how much worth do we
give to future human lives there's many
ways you can answer that question and
then you want to discount in various
ways for model uncertainty and different
things but one thing that drives
concerned with essential risk which I'm
concerned with in the future of humanity
Institute is concerned with is that
there's so much
potential in the future there are so
many potential lives that could be lived
that anything that cuts that off has
tremendous to sell you and so one
estimate would put it at and this is
quite a conservative estimate if there's
only a billion people living on earth
and they continue living in a
sustainable way for a billion years
which is doable as long as we don't muck
it up that gives you ten thousand
trillion lives which is a lot and so
anything that reduces the probability of
extinction of losing those ten thousand
trillion lives by even a little bit has
a lot of expected value okay now these
are that's a very small number 110
trillion and it's hard to you know know
if you're making that big of or if
you're making less of an effect than
that it looks close to zero the numbers
aren't meant to be taken too seriously
they're more thinking heuristics right
they illustrate that if you give value
to the future you really want to worry
about anything that poses a risk of
extinction and one thing that I and
others and and one group I'll note is
open fluently projects have identified
as a risk to the future is artificial
intelligence before telling you about
the risk I'm going to first tell you
about what what's up with AI are these
days so for a long time artificial
intelligence consists of what we would
now call good old-fashioned AI there was
a programmer who wrote if-then
statements you're trying to encode some
idea of what was a good behavior that
was meant to be automated so for example
chips chat algorithms you would have a
chess master say here the heuristics I
use here's the value function you put it
in a machine the machine runs it
reliably and more quickly than a human
can do and that's sort of you know an
effective algorithm but it turns out
good old-fashioned AI just couldn't hack
a number of problems even simple ones
that we do in an instance like
recognizing faces images and other
things more recently what's sort of
taken over is what's called machine
learning this is what it sounds like
machines learning for themselves
solutions to problems and other terms
for this are deep learning
that's especially flexible machine
learning you can think of it as just a
flexible optimization procedure it's
it's an algorithm that's just trying to
find this
solution and has a lot of parameters in
neural networks is another term you
probably heard okay so this is showing
basically the breakthrough recently in
image classification arising from neural
networks and the year onion year-on-year
and improvements to the point where now
machines are better than humans at image
classification another domain is
generalized game playing or arcade game
playing here are tari games probably not
many of us have played these deep mind
has built a machine deep mines a leading
AI group within Google that learned to
play Atari games at a superhuman level
with no instructions about anything no
instructions about the nature of the
universe
what is time and space what is the bad
guy and here's pac-man
what is the pellet and what is the ghost
all the machine is told or given is
pixel input it just sees the screen from
a state a blank state blank slate I'm
sort of beginning and then it plays the
game again and again and again getting
its score and it tries to optimize for
the score and gradually over the span of
a go today it learns to make sense of
this pixel input to know to derive
concepts of sort of the bad guy the
ghosts the pellets devise strategies and
become superhuman at a range of games
another domain we see this as go alright
and the solution to go was very similar
you take a blank slate neural network
that's sufficiently flexible have it
first be exposed to lots of human games
and learn to predict the human move then
you have the Machine play itself again
and again on the order of about 10
million games and it becomes a
superhuman so here's Lisa dole and
alphago which is another product of deep
mind and Lisa dole was the first real
excellent go player who publicly played
alphago and here he's saying in February
2016 just last year I think I won the
game by a near landslide at least this
time well as I've alluded to that didn't
work out so here is him after the first
game I was very surprised because I
didn't think I would lose and
unfortunately he lost again I'm quite
speechless I'm in shock
I can admit that the third game is not
going to be easy for me he lost the
third game he did win the fourth game
and he talks about how it was so the
greatest moment of his life and it's
probably the last game the one and only
game played against a level of alphago
that of that level or better that a
human will ever win now I'm bringing out
Lisa Dolan his losses for a reason and I
think it serves as an allegory for
humanity that we don't want to be caught
off guard when it's our alphago moment
right at some point machine intelligence
will be better than I've set
strategically relevant tasks and it
would be prudent for us to see that
coming to have at least a few years
notice if not more to think through what
it means how we can adapt our
international system our politics our
notion of meaning in life and other
areas okay so what's driving this
progress well algorithms talent data
about a big thing driving it is um
hardware computing inputs just keep
getting better at an exponential rate
this is sort of a generalized Moore's
law across a range of inputs and it's
this kind of persistent progress that
makes Kurzweil graph from 2001 not seem
totally absurd so here we have on the
first y-axis calculations per second per
thousand dollars and I added four recent
dots from the past 17 years and you see
you know basically we're on track we
have exponential improvement
what we don't know is when we get to
transformative AI
right now Kurzweil has it's evocative
second white y-axis where you have
organisms we recognize mice humans in
all humans it's not obvious what the
mapping should be between calculations
per second and transformative AI
intelligence is not a single dimension
so I we could get superhuman AI and some
domains long before other domains but
what I think is right about this graph
is that at some point between 2020 or
you know 2025 and 2080 big things are
going to happen from machine
intelligence now in our work we want to
be a bit more systematic about these
timelines and there's various ways to do
it one way is to survey AI researchers
this is the result of
recent survey and some takeaways are one
there's huge disagreement about when
human level machine intelligence defined
here as machines better than all humans
at every task will come so as you can
see by the sort of gray s-curve
lots of disagreement to the group that
we surveyed gives enough popular enough
probability mass that by a hundred years
there still won't be human level machine
intelligence but three in the next ten
years or twenty years this group gives a
ten percent to 20 percent chance that we
will reach it already and if that
probability seems right upon
consideration which I think it does not
just using this as evidence I think
that's more than sufficient warrant for
us to invest a lot of resources thinking
very hard about what that would mean for
our humanity so here are some tasks I
like to think about milestones and
Canaries what are some things that
machines will eventually achieve that
are either noteworthy or strategically
relevant and the Canaries are those that
when they when they die when they go off
signal to us that we better be paying
attention because things are going to
change quickly I expect most of these
tasks on the right column will soon be
moving over to the left column if
they're not there already so this is
something else we're working on more
generally there's a whole host of
challenges near and medium-term and
long-term that we will be working on we
as a society but also at FHI there's a
range of near-term issues that I'm not
going to talk about each of those could
occupy a workshop or conference I will
say that when thinking about long term
issues we also confront the near-term
issues for one reason because near-term
issues or the long-term issues often
look like the near term issues magnified
you know one hundredfold but two because
a lot of the long-term insights that we
have in strategies and policy
interventions we want to make have a
most appropriate place of action in the
near term okay but what are the long
term opportunities and risks
well the opportunities are tremendous we
often just think it's you know sort of
greater wealth and maybe you know and
Google stock will go up but it's a lot
more you know longevity health
preventive medicine
material bounty that could be used to
in poverty that would need not because
it will also likely come in a more
unequal world reduced environmental
impact deepmind was able to reduce
energy usage at google data centers by
40% but also basically anything else
that we value is either the product of
intelligence or depends on intelligence
for its protection and so if we have
superhuman intelligence then in
principle we can use that to achieve all
our goals
I'll also emphasize the last point
resilience to other existential risks
we're likely to face those in the coming
hundred years and if we solve a AI if we
build it well then that could reduce
those as our risks by a large margin but
of course there's also risks with
bringing into the ecosystem a creature
that is better than us it's the thing
that matters most for our survival and
flourishing I'm not going to go through
this typology I will illuminate the risk
by quoting max tegmark and others one
can imagine such technology of smarting
financial markets out inventing human
researchers out manipulating human
leaders and developing weapons which not
even understand where's the short-term
impact of AI depends on who controls it
the long-term impact depends on whether
it can be controlled at all
I will also appeal to another Authority
this is from my world I'm a scholar of
international relations Henry Kissinger
as you know is also very worried about
AI and this is sort of the definition of
a grand strategist modern technology
poses challenges to world order and
world stability that are absolutely
unprecedented I personally believe that
artificial intelligence is the crucial
one lest we wind up creating instruments
in relation to which we are like the
Incas to the Spanish but our own
creations have a better capacity to
calculate than we do it's a problem we
need to understand on a global basis
okay in addition to these quotes the AI
researchers we surveyed also agree we
asked them what the long-term prospects
are for human level machine intelligence
and while most of the probability mass
was on extremely good or unbalanced good
the median responded gave a 5 percent
probability to it being extremely bad or
human extinction that is human
I it's not often you get an industry
that says that their activities give
rise to a 5% chance of human extinction
I think it should be our goal to take
that number whatever the real sort of a
frequentist number is and push as close
to zero as we can get and push as much
of that probability mass up to the top
there's two broad ways we can do that
one is to work on what's called AI
safety the computer scientists in the
room and mathematicians can help build
AI systems that are unlikely to
misbehave without our intentions okay
the other way is AI strategy which I'm
going to talk about so here's Stuart
Russell explaining that we're not
worried about machines suddenly waking
up in deciding they want to build their
own machine utopia and get rid of humans
it's not some kind of emergent
consciousness the worry is that they
will be hyper optimizers which is what
we're building them to do and that we
will have not specified their value
function correctly and so they will
optimize for the wrong thing this is
broadly called the control problem or
the value alignment problem here are
some groups working on this or funding
it I'm going to keep moving here are
some people who can't really make out
but I will tell you that they are
leading researchers and industrialists
from the most prominent AI groups in the
world and they came together this
January at a sillim are I'm to really
think seriously about AI safety which i
think is a tremendous achievement that
for the community that we've brought
everyone together like this and it's I'm
showing this with a reflection of how
exciting a time it is for you to be
involved in this okay so one conjecture
that's been posed is that a value
alignment it's actually not that hard of
a problem as long as we have enough time
once we have the final system we want to
deploy to test it right it's like drug
tests you have the the thing you are
thinking about deploying the population
let's just make sure it undergoes
sufficient scrutiny however this
conjecture goes it is almost impossible
a I by alignment if we don't have that
time well if that's right which seems
plausible then it directs attention to
the world in which this system is being
deployed is it one where the developers
have the incentives and the
to do these safety tests and this is one
of the big things AI strategy thinks
about basically how we can prevent a
race in AI here's the CEO of deepmind
basically saying this point we want to
avoid a harmful race to the finish where
corner cutting starts happening and
safety gets cut this is a big issue on a
global scale and it's extra hard when
you're talking about national
governments right it's not just a race
between companies in the u.s. it's race
between countries so what do we think
about an AI strategy a lot
so one thing we think a lot about is
what AI races and AI arms races would
look like another whole class of issues
is what AI could look like militarily
what are some implications of AI for the
military in terms of balance of power
crisis stability uncertainty about
capabilities
another thing we think about is what it
means economically if we live in a world
say where AI is really the engine of
growth and value in societies which
increasingly seems to be the case so the
top 10 of the top 10 firms by market
capitalization either five or six are AI
companies Google Amazon Apple Microsoft
and so in such a world
what do countries do like Saudi Arabia
or France that don't have their own
Google or Amazon but they want they want
to be part of that value chain and so we
may be entering an era of AI nationalism
where countries want to build their own
national champion and China's certainly
in the business of doing this and lastly
or I mean there's a lot of issues these
are very high level categories is this
massive challenge of AI governance both
of the near-term small-scale how do we
govern algorithms that are being used
for judicial sentencing or self-driving
cars to the long-run scale of what kind
of electoral system what voting rules do
we want to use for the organization
that's going to be deciding on how to
test the super intelligence and how to
deploy it these are very hard questions
and I want to make clear that these are
questions that are being asked today by
the leading AI group so here's Sam
Altman demis hassabis asking these
questions like literally what is the
voting rule that we should use for
control of our super intelligence once
we build it well they didn't literally
say that but but
the site for governance today is the
partnership on AI this is a private
sector organization that has recently
brought in NGOs including the future of
humanity Institute and it's plausible
that this will do a good job
guiding AI in the near term and could
grow in the longer term at some point
governments are likely to get more
involved and so that's a site for
studying for intervention another thing
we can do is to try to articulate
principles of good governance of AI and
these are some principles that came out
of the Asilomar conference again a hot
tip to max tegmark for putting that
together and especially these principles
so we might want to work to identify
important principles that we can get
different communities to agree on and
then formalize them institutionalize
them to make sure they stick so in
summary what's to be done a lot of work
what kind of skills do we need virtually
every skill set everything from people
who can help us grow the community
operations people admit administrative
experts and people who enthusiastic and
efficient we need people who do policy
engagement outreach can form media
strategy what should be say the media
strategy of a group like the future of
humanity Institute or others when
there's another self-driving car
incident or when I say truckers are
being massively displaced from their
sites of employment these are important
nutrient issues but also their sites for
having conversations about the
longer-term issue we're doing work on
strategy theoretical work mathematical
modeling of tech races trying to
understand AI development what predicts
innovation measuring actual capabilities
and different sites in the world
countries understand the value chain on
supply chain of AI we're surveying
Publix throughout the world and elites
trying to design safety standards
working with AI safety researchers and a
range of other issues so if this seems
important to you and it's interesting I
strongly encourage you to get involved
there's a link there hey i policy career
guide that has some text that can sort
of point you in the right direction
there's also a reading list are there
and in general just please reach out to
me
Tarek's linens here from the future of
an institute as well and he's helping us
build up that site and there's people
working on this in a range or beginning
to work on this at a range of sites so
we'd be very happy to help you be
productively engaged so to submit your
questions you can go to Boston be a
global org slash poles I was encouraged
to make sure we run all the questions
through the the website just because the
mics won't pick up people in the
auditorium so thank you for the the
talking and for being here we'll give a
second for questions to kind of come in
but one thing that I'm struck by is it
seems like we're making a lot of
progress I mean I've been involved in
this community at sort of on the edges
for a number of years and there was a
time when to even talk about something
like this and that time was not long ago
I mean seven eight nine certainly 10
years ago it was extremely you know kind
of fringe and you know only weirdos
sort of you know seem to be willing to
go there now we've got respectable
people like yourself and you know the
growing body of academics that are
getting involved so it seems like
there's been a ton of social progress
what is that translating to in terms of
technical progress or sort of practical
progress to try to get a handle on these
issues that we're bringing people
together but what are those people
producing so far and do we should we
feel any safer than we did ten years ago
yeah so I can speak most to the strategy
side of it
I will remark that one sort of comment
that was made out of Salaam are that
resonated as as true to many people is
that AI strategy is today what AI safety
was two years ago so they're the first
beneficial AI sort of meeting at Puerto
Rico ai safety was really nice and there
was you know just a handful of you know
individuals in the world thinking
seriously in full time about it and
that's changed today now the way
I groups have safety teams there are
people doing PhDs with an eye toward AI
safety so that's very exciting and there
has been a lot of sort of technical
progress that's coming out of that AI
strategy is just where AI safety was two
years ago but I think it's rapidly
scaling up we haven't had enough time to
a lot of thinking has been done it's not
public yet but I think in the coming
year you will start to see a lot of
really insightful sort ejek analyses of
AI attention risk yeah and so uh I'll
also say I've given some talks like this
to other respectable audiences while
political science Pete you know
workshops and the none of the PhD
students think this is crazy
they you know they all think some of
them think oh I don't have to quit
smoking because of this which which was
one comment but they all think this is
real I and the challenge for them is
that the discipline doesn't entirely
support work on the future you know one
question I got when I presented at Yale
was so how will you be empirical about
this because you know we're social
scientists we like data and we like to
be empirical and I remark that well it
is about the future and it's hard to get
data on that but we try him so you know
as some of the you know projects I
listed and you know so I think it's a
challenge for currently existing
disciplines to adapt themselves to this
problem but increasingly we're finding
good people who recognize the importance
of the problem enough to take the time
to work on it so first question which i
think is a good one coming in from the
audience can you give a little bit more
detail on what an AI governance board
might look like are you thinking more
kind of blue-ribbon panel of experts or
more of a free-for-all you know
everybody can contribute open democracy
type of structure yeah so I think
there's a lot of possibilities and I
don't have a prescription at this point
so I'm not going to answer your question
but you know these are the issues we
need to think through so there's
trade-offs I can see I can talk about
the trade-offs say there's trade-offs
between legitimacy on the one hand right
the UN General Assembly for example is
often seen as a legitimate organization
because every country gets a vote but
it's not necessarily the most effective
international body so you know and also
you have to wait your institutions in
terms of power holders so if you have
the most ideal governance proposal it
might be rejected by the people who
actually have the power to enact it so
you need to you know work with those who
have power makes you know make sure that
they sign on to the regime and try to I
think build in at early sites of
intervention so the key properties of
whatever this development regime is in
governance regime so that good comes out
of it and so I'll mention a few
suggestions you know whatever
development regime it is I think it
should have a constitution some you know
explicit a text that says what it's
about what is it trying to achieve I -
it should have enough transparency so
that the relevant stakeholders be that
the citizens of the country if not
citizens of the world can see that the
regime is in fact building AI according
to the Constitution and I should say the
Constitution should be a sort of common
good principle type thing and then three
you want the regime to have
accountability so that if it's not
working out there's a mechanism a
peaceful mechanism for changing the
leadership and these are basic
principles of institutional design but
it's important to get those built in so
speaking of people in power
President Obama apparently was was asked
and I haven't seen this clip but was
asked about risks related to artificial
intelligence and the answer that he gave
seemed to sort of equate AI risk with
kind of cybersecurity risk that we know
and love today do you think that there
is a sufficient understanding at the
highest levels of power to even begin to
make sense of this problem or do we have
kind of a fundamental problem of lack
and understand
that may be quite hard to overcome yeah
so that's a great clip I wish I could
just throw it up really quickly we don't
have the AI yet to just do that I
encourage you to watch it it's hard to
find it the way wired put it on their
page but so he's asked about super
intelligence and if you know he's
worried about it and he pauses he
hesitates he gives sort of a considered
hummer sigh and then he says well I've
talked to my advisers and it doesn't
seem to be a you know a pressing concern
but the the sort of pause and hesitation
is enough to suggest that you know he
really did think I mean Obama is a
science-fiction fan he really did think
about it seriously and I know so so I
think he probably would have been in a
good place to appreciate other risks as
they arise but I think yeah lots of
people on the government are likely to
dismiss so lots of reports from military
or government have put superintelligence
worries at least sufficiently distant
that we don't really need to think or
address it now um I will say those cyber
security is likely to be a site where AI
is transformative at least in my
assessment so that's one domain to watch
in particular another question from the
audience if there is a five percent
chance or you know the same analysis
would really hold even if it were a
lower percentage chance of extinction
due to AI one would you know not be
unreasonable to jump to the conclusion
that hey maybe we should just not do
this at all right it's just too hot to
touch
is there an image first of all what do
you think of that idea and second is
there any prospect of making that
decision globally and somehow sticking
to it yeah so I want to flip back to the
slide on opportunities so I actually had
a conversation the other day with family
members and friends of the family and
one person at the table asked that
question why don't you know if it's so
risky why don't we not do it and then
another
a friend of the family asked what are
the impacts of AI for medicine and for
health and you know for curing diseases
and I think in many ways that you know
those are sort of two sides of the
policy decision that there are really
tremendous opportunities from AI not
just you know the material things like
increased material bounty but really you
know curing diseases curing Alzheimer's
you know pretty much anything you can
imagine that's the product of
intelligence you know could come from
machine intelligence so it's there's a
real trade-off to be made but then the
other issue is stopping AI progress is
politically infeasible so I don't think
it's a viable strategy even if you
thought that the trade-off weight in
favor of doing so and I could talk a lot
more about that but yeah that would be
my position probably last question that
we can take just due to time constraints
but thinking about kind of the ethical
direction that we want to take as we go
forward into the future right Thea the
value alignment problem you had posed
that notion that if we have two years we
can probably figure it out yeah but if
we don't you know we may be more than
likely we can't that strikes someone in
the audience and I would say kind of me
to is maybe even a little too optimistic
because we've been working for thousands
of years on what it means to have a good
life and you know what
what good is right so nice to do we do
you think that we are closer to that and
then maybe I think we are or how do you
think about the kind of fundamental
question of what is good in the first
place yeah right so to be clear this
conjecture it's it's about whether a
single person who knows what they want
can build a super intelligent machine to
advance those interests it says nothing
about whether even if we only what we
want whether we could agree on what we
want to build let alone do we even know
what we want so the can we agree is the
political governance question of you
know even if we all have sort of
fundamental preferences how do we
aggregate those in a
a good way and then there's the deeper
question of what should we want and yeah
those are hard questions that we need
people working on as I mentioned in the
last slide so moral philosophy and
politics what do we want so we need your
help figuring that out well thank you
for wrestling with these issues and for
doing your best to protect our future
professor Allan Defoe thank you very
much
[Applause]