Andrew Critch – Robust Cooperation of Bounded Agents – CSRBAI 2016

welcome back so tomorrow is a workshop
day so this is the last talk of the
colloquium series on robust and
beneficial artificial intelligence our
speaker Cassandra rich a research fellow
here at machine intelligence Research
Institute Andrew got his PhD in
mathematics from UC Berkeley and
co-founded CFR and worked at James
Street Capital for a while and he's here
today to talk about about ways to have
cooperation coordination between bounded
bounded algorithms that can read each
other source code so let's join me in
welcoming and thanks okay so let me call
this let me call this yeah is it this is
in the frame we're here cool so I'll
call this talk robust cooperation of
bounded agents via a bounded
generalization a parametric bounded
generalization
of Lopes tariffs and let me give an
outline which I will immediately really
label to menu because we're a very small
group so you can just kind of steer me
through the outline and I choose your
own adventure collaborative game that
you have to play with each other so be
sure to cooperate so I'll start with
what I'm gonna call open-source
prisoner's dilemma the OS stands for
open-source it also stands for one-shot
so open source one-shot prisoner's
dilemma prisoner's dilemma next will be
attempts at robust cooperation in that
scenario next will be I'll probably talk
about the been a generalization
parametric bounded lobes theorem or P
blob and then to the extent that it
wasn't quite covered in this section
which is optimized for pedagogy I might
want to say some more about the history
of these results where they came from
and then possibly also future directions
or this stuff great and so knowing that
these are these things are all on the
menu you can like try and get me to jump
forward or backward or whatever you want
cool so just so I can find out who I'm
talking to you I know I know like about
happy you guys I guess and I've met more
than half so first of all who has read a
paper by Patrick on robust cooperation
okay so about half the audience so this
half this this will be for oh you're my
paper oh that's even that's worse
okay I see you read much so who the red
paper on Ross corporation or my paper oh
no all right so you guys should be
giving me the Tuck and all right so I
know I'll skip through this pretty
quickly then probably skip through this
pretty quickly maybe I might spend more
time talking about this or like
interpretations with results or the talk
could just be less than an hour we could
just like be done quickly
okay so I'll just like so hands up for
the compliment the people who have
neither read my paper northernmost
cooperation paper okay so I'm talking to
you guys so open starters lemma I'll
just illustrate it by some example
agents so an agent takes its opponents
source code and then returns cooperate
or defect that's like the format of the
game the agent also has access to its
own source code which you can do bike
whining if you like functional
programming or you can just like have as
a primitive in your language that you
can read your own source code so
secretly it's also got access to its own
source code but I won't always denote
that so example
we'll have an agent called corporate bot
which I'll just like right see because
it's short we're gonna say define
cooperate but of opponent equals return
cooperate so it's very simple it's just
like ignore its opponent is it
cooperates with the opponent matter who
that is we can define another agent
which I'm gonna call is me bot and as
far as I can tell this agent was first
written down by pin and Holtz in games
and economic behavior of 2004 I'll just
like to put a little citation here oh
really yeah
all right well I'm gonna cite him for it
then holds games in economic behavior
2004 Andy's me but just says if the
opponent equals me then we return
cooperate else we returned effective
Howard okay I'm not gonna cite them
because I'm not sure I want to yeah yeah
so are people thinking about things like
this yeah like I'm interested in this
because this is the point in which this
is the paper where the notion of program
equilibrium was defined where you can
think about a Nash equilibrium where
you're you and I are about to submit
programs to play a game against each
other and there's a Nash equilibrium
between us as the programmers but not
really between the programs playing the
game anyway
okay next so if you would mind let me
follow my own narrative track here
because I think I'm gonna explain this a
little bit different from the way
Patrick normally does so the way I think
about this is that this agent is very
fragile because there's like only you
know it's interesting you know it's it
can achieve cooperation with itself in a
one-shot game which is cool usually you
don't see cooperation in prisoner's
dilemma unless there's like iteration
where there's something like a lot of
examples you've seen tournaments right
we're agents participate and prisoners
lemmas with each other over and over and
when you have a reputation to keep up
with your opponent then you like
cooperate more to try and get them to
cooperate with you later but in a
one-shot game there's like no incentive
to cooperate if there's unless maybe
something is different like your source
code is visible and in this case you
know you end up cooperating with
yourself if here is me but but what if
is me bot is written in Java and there's
another is me bot written in C++ they
don't cooperate so it's kind of fragile
in that way even though like they're the
same idea they don't cooperate so so we
can say this is fragile because it uses
kind of like a fragile cooperative
criterion so Patrick and others we're
looking at a more robust criterion which
motivated this guy and I'm gonna skip
over the I'm just gonna skip right to
the the bandit version that I have in my
paper so this is gonna be fair bots and
you can you can imagine well okay it's a
question of what's actually gonna happen
here is this gonna work or not but like
let's just just just say what happens
when you bound this thing so fair bot
looks at its opponent and it's going to
search for a mathematical proof
so you fix some proof system let's say
piano arithmetic of length less than
equal to K characters so imagine is a
text file I'm we're counting the number
of characters in the proof that the
opponent cooperates with fahrbot and
like this is a mathematical statement
and searching for a proof you can search
for approval as different ways you can
search for proof by running the program
but you don't have to prove stuff by
write proof stuff about programs by
writing them so like if I say while n is
like in a 0 while in less than a
thousand increase n by 3 and ask you
about that program outputs you can like
think about it per second and realize
it's gonna input up with the first
multiple of three after a thousand which
is a thousand two and you didn't have to
think of the number 501 while you reach
that conclusion you did not have to
simulate the program you only need to
reason about it you could run it that's
another way but I'm gonna allow any kind
of proof that the program outputs a
certain thing as part of the reasoning
process here cool okay so this is you
search for the thing you search for
proof of and then you know if found
where found is like a boolean return
cooperate else return defect so I can
already tell from the audience that this
is not going to be the version of this
talk of it where's Alex that will go on
the internet because too many of you
already know how it turns out
but or how it's eventually going to turn
out but the question is how does this
bound affect the proof search so just
for like the three people who have not
right I guess that probably already
talked to Stuart about it so there's
only two people so we're gonna do you
haven't seen this okay yeah so what
happens here everybody just like take 30
seconds to think for yourself
what happens when fahrbot encounters
fahrbot so we can do first we can do a
warm-up question warm up what is fair
about
of co-operate bots and everybody waved
your hand at me
when you have an opinion about what this
is just like wait you don't have to
raise it just go like I have an opinion
all right what's our opinion not from
Patrick hey Patrick knows more about
this than I do so I don't want I want to
engage you guys more yeah not quite now
quite depends on K so if K is small
there's not gonna be any proof if I say
find me a proof using one character that
cooperate but cooperates then it will
find no such proof does that make sense
proofs too short right so equals C you
know for K sufficiently large maybe K
needs to be like 100 or something I
don't know how many characters you
actually need to write a proof that that
thing cooperates but it's pretty pretty
shortly okay so that was our warm-up to
make sure we're actually all actually
thinking about how the code really runs
and not just being intuitive about it
question yeah yeah so fair yeah so I'm
just gonna skip is me about for now and
the next question is what happens when
fahrbot sub K in this form this version
of fahrbot k runs against itself and
maybe just think about that for 30
seconds and give me a wave when you have
an opinion okay so here's what I want to
do this means cooperate this means
defect okay so I want everybody to make
your best guess whether this is
cooperate or defect
okay let's say yeah good question so
let's say K is big so obviously we
already saw up K small it does
incorporate so let's just say for K
large us just say for K is 10 to the
hundred okay everybody
prepare your hand cooperate or defect go
alright so we have some some bimodal
answers some people are doing this some
people are doing this okay cool so let
me hear from okay from the people who
are saying defect you guys see the
defects okay why it's other people who
say defect are having similar intuitions
you know you can't contain something ah
yeah it's a come here yeah yeah and I
saw at least soon had it like a C or D
thing going on like this underspecified
interesting okay so which part of the
code is like not specified enough okay
why don't I do this now can a computer
on it I'm not gonna do trust me sure
yeah right okay so you would like to pin
down how you're searching for the proof
so so here's an interesting question
right so what's the argument for C and D
being both plausible yeah if you didn't
know which was wish let me let me let me
try to make an hour yep so first of all
yeah so let's write great I'm glad these
thoughts are all coming up because I
want to make sure that we're thinking
concretely enough about this to have
these questions otherwise the result
will seem like kind of weird or like
it'll be vague and won't be clear what
it means so first of all let us note
that this program halts I can bound its
run time it's gonna run in something
like 2 to the K steps however long it
takes to search for this many proofs and
then it will give you know if it
finishes the search and found no proof
it will give up and defect so this has a
bounded run time no I don't have to you
it's right here in the code right so
corollary of the code consequence is
that the run time can be bounded another
thing that we should need to specify is
we maybe need to specify how we're
searching for the proof right is it's
just like a heuristic search so let's
just say we're gonna alphabetically
search the surfer the simplicity of you
know you guys analyzing it to me
analyzing it let's just assume we're
gonna search alphabetically through all
strings and then check is this string a
proof that my opponent cooperates one
second so search alphabetically through
strings
for a proof so soon does this seem like
it has specified I can also say a proof
in piano arithmetic and I can say in
piano or through tick using standard
girdle encoding with binary
abbreviations for numbers to save space
using binary we're numbers well I think
one reason it's important to use binary
for numbers is that this symbol will get
really big if you use the successor
operation to encode in numerals so you
definitely want to use binary yep
somebody thinks doesn't matter how we
search somebody else might think it does
doesn't matter for example so suppose so
hey look maybe it seems like cooperate
so there's at least a few people here
who think that cooperate in defect are
both consistent answers and so maybe
whether you gets on the order that you
search in the like well but hang on the
proof is going to be about the search
method right so maybe right so you're
proving a different thing you're proving
it whatever the search method is the
proof is going to be a different proof I
don't know how to do that because think
of it with the point is if I like fixed
if I fit the single statement right if I
fix this the interesting thing is that
like fahrbot takes as a parameter
like proof search method right so if I
just fix a single proof statement like
Riemann hypothesis if I put here if I
don't have fair about here if I just say
search for a proof of Riemann hypothesis
then yeah I agree that the search method
doesn't matter because all that matters
is whether there is a exist a proof of
Riemann hypothesis in these characters
or not right
problem is the statement you're proving
has as part of it the proof search
method so if I change the proof it's not
if I change the proof search method I
don't just like find another method for
searching for the same result I'd like
change the result I'm also search yeah
if you could prove that that'd be
interesting
yep that's right
yep so those are some subtleties this is
take down I was thinking I was thinking
about doing that next semester actually
all right so so so those are things now
there's for people who are feeling who
are the people feeling like C and D are
like both consistent so think about this
only like this is a program it's
deterministic once I've fixed my proof
search method and I like it's got an
output C or D it's it's it's got a
bounded runtime it can't run forever
so yeah that's right so this is I'm not
drawing any random numbers during my
program this is like a deterministic
computer program with no cosmic rays
hitting it yep cool all right so now
that we've had a little time to
deliberate repol
so hands up for ya indicate cooperate
defect or this would be like don't know
okay go don't know got some C's get some
don't knows got a D cool alright why do
you think D basically it's not gonna
have enough time to like reason about a
thing that's reasoning about this much
length cool yeah so so this is I think
we've like now thought like I feel like
I've tried presenting what's gonna be
next in a few different ways and I find
that if I don't give people enough time
to think about it it sort of fails to be
as counterintuitive as it should be so
the answer is cooperate and for the
people who said cooperate is it because
you know loves their right okay so
answer is cooperate so imperative so
empirically sometimes people start talks
with things that don't cooperate as a
way of like you know illustrating why
it's interesting so yep
so and I did say I don't know I said
attempt anyways you guys all know the
results so it doesn't work but I just
wanted to let you to think about it long
enough to feel like actually it's not so
obvious great so it's cooperate and just
to review the intuition which was I
guess vladimir slaps left Nev was the
first person to think about using loaves
theorem to like get things to cooperate
which is so loops there is that if
piano or thematic can prove that if a
natural number encoding a proof of P
exists then P is true
then piano if it can prove P directly so
hands up for people who would have
written Lopes theorem on the board had I
just asked them to do it it's gonna be
like a half yeah great
okay so you know this means this is like
an arithmetic statement that means a
proof exists of P and this also means a
proof exists but it's like a different
language using to state that fact so
this is like exists a natural number
encoding a proof proof of P cool so that
slope and basically it almost applies
here when you let P equals if you ignore
the K for a second and blur your eyes
and pretend like things aren't bounded
or you know look for some kind of fixed
point in some logical universe then it
will you feel like you want to write
down you know if there's a proof of
cooperation fair about will find it and
therefore cooperate
cuz this part makes sense P is
cooperation so this feels like if
there's couplet proof of cooperation
they'll find it and moreover you know I
can write down an argument for that
piano is meticulous and that argument
there for piano music can prove that
cooperation happens there for
cooperation happens because we will be a
piano or thick or you know we think this
thing believes planner with him taking
that this criterium corporation either
way so that's interesting and actually
Patrick and others developed a framework
for writing down formulas like this
writing down formulas rather like this
for cooperation of various agents and
kind of thinking about what they would
do each other do to each other in a in a
universe with like unbounded
computational resources where things can
sort of search instantaneously for
through proofs and and also search
instantaneously for proofs about those
things etc is that fair yeah you can
just say Oracles though yeah
and but the problem is that sort of none
of those agents are bounded and it
remains it kind of is like okay Lupe's
there was kind of weird right
so who's thought about why loves theorem
is weird okay fewer okay so you know the
Riemann hypothesis you know if this is
Riemann hypothesis then you really you
kind of it's very tempting to believe
this statement right it's like well I
don't know if the Riemann hypothesis is
true or not but if I prove it it's true
right so you're like yeah if I prove the
Riemann hypothesis is true well problem
is if you ever wrote down a proof of
that claim in piano rithmetic then you
would shortly thereafter write down a
proof of the Riemann hypothesis which is
bad news that the Riemann hypothesis is
false so until you know the Riemann
hypothesis is true you like don't want
to believe that you're trustworthy about
it in the positive direction so it's
that's the weirdness of lobe some people
call this the lobe Stickle it's like an
obstacle to self Trust and it's often
viewed as a negative result and then
sort of slip nemu notice that you could
also use it to get like a kind of
positive thing to happen like
cooperation and a lot of people when
they see lobe they're like well that's
weird and it's got infinity in it namely
this existential quantifier this search
instantaneously through all natural
numbers and determines were there proof
for something exists right so maybe the
weirdness of Lopes theorem is coming
from infinity right maybe if we write
bounded programs that do bounded
reasoning low the weird stuff will go
away anyway the answer is no the weird
stuff does not go away and here's this
so we have yeah maybe only one person
actually read my paper so all of these
for the person who already knows this
result
but a parametric gun at Loeb is that if
be onerous Nik can prove that for all K
so if we're gonna fix some statement it
takes a parameter K like fair about
cooperates with robot and we're also
gonna fix some function f of K that
grows faster then some function of log
of K it's gonna depend on the proof
system
I think log K cubed I'm like very early
confident lucky cube is enough I'm also
fairly confident that log K is enough
but if we go located five I'm like hell
yeah okay
so some function actually depends on
things like whether you're using binary
encoding for motors yeah so we're
definitely actually if you're know the
log depends on whether using banner
binary code encoding for numbers if you
don't use binary encoding z' you have to
replace log by like whatever your
encoding is the exponent comes from how
efficiently you can encode proofs about
yourself or like proofs about proofs so
if you have a proof of length and how
much space does it take to prove that
that proof of length n is a proof of
length it and I think I think MIT I'm
pretty sure metamath can do this in
linear space like the amount of time it
takes to verify proof yeah yeah oh you
can make it super unreal yeah well you
could make it piano is predict using the
successor operation for encoding
numerals and then that's already pretty
bad and you could make it you can
probably make it worse if you wanted
yeah this is like this for the five the
log and the five are like four
reasonable things like things if you
ever wrote a program to do proofs there
are certain efficiencies you take
advantage of for example if you want to
check whether modus ponens is being
applied it's nice to like have a pointer
to like where
the arrow is in the middle of the most
ponents to save space in the search for
that so you fishin proof system would
like start an application of modus
ponens with like a little number that
says like where the arrow is and the
modus ponens so they can check it
officially so like little things like
that if you do things like that then you
can probably get it down to just that
but something some number n goes in
there depending on how patient that's
gonna be and so I'm just gonna wrap all
that up into like some there's gonna be
like some polynomial function of log
okay are we cool with that it depends on
the proof system alright but whatever it
is if F is bigger than that and if the
enter if that can prove that for all K a
moderately long proof of P of K so f is
our like measure of like moderately long
proof it's sufficient to deduce P of K
if you can show that long proofs if you
can show that you trust yourself about
proofs that are fairly long but not
crazy you know you don't need to trust
yourself about crazy long proofs just
like moderately long well long ones then
pianer think you can prove that for all
k passed some threshold P okay stars
feature so it's just exactly what you
want prepare bots to eventually
cooperate for large K this is kind of
nice it's not coincidence I was just
like aiming for anything that would make
pair of BOTS cooperate so that's why
this is the result I get questions yeah
this is this is my like function that
like takes into account how efficient it
is to encode proofs about to have proofs
about proofs and it depends on the
logical system so it's gonna depend a
lot on you know this thing here
yep sir
well you do need this whole thing to be
less than K and so I'm pretty sure like
log to the five is less than K so it
works out so application let right every
time you increase K you also slightly
increase the complexity of the thing
that you're proving by making it like
slightly longer yeah oh there are other
other questions yeah because there's a
proof yeah yeah yeah the really requires
is that it seems that the boats will
cooperate by just proving more or less
by proving this but my intuition for why
things like this are true is that they
can be readily readily turned into
statements about girls second
incompleteness theorem this is in fact
equivalent to a statement that if any
certain extension of an arithmetic
contained a short proof of its own
consistency then would be inconsistent
okay yeah that's probably true at very
least a result of that form I'm pretty
sure that a result of that form could be
used to prove this and this might be a
little bit too weak to get what you said
I'm not sure but the reason I'm calling
this parametric bound of low is because
I'm saving the name bounded lobe for
maybe someone produce something slightly
more basic than this
yeah so yeah so you let f of K be
whatever like I think this is you just
wanted to be epsilon inverse of K I
think so epsilon inverse circle K so if
epsilon is like cubing then you can just
take cube root okay or F of yeah okay
anyway I might fix that as we go and
then let P of K be this thing so what do
we have what we have is we have box okay
yeah maybe we don't write what we need
is for K to be bigger than that yeah so
we just want we just want this yeah I
should I should take back that statement
we need K to satisfy we need the
function K to satisfy being bigger than
that and if epsilon is like exponential
then K doesn't work doesn't doesn't like
satisfy this inequality and doesn't let
you apply the theorem but epsilon is
probably somewhere between linear and
cubic depending on realistic proof
system so we have box KPFK implies P of
K because of proof of length K of
cooperation at least corporations and
then because K is not a crazy-big
function we get P of K so that's fair
about cooperating with itself oh and I
should probably put for all K is here
and this is for all K greater than
official so now you can ask questions so
like be unsatisfied with this result now
so like try and be like hmm but what if
this and is it like really that and like
what a yeah let's move into like
dissatisfaction mode I'm Bruce whether
Jim feels that his
about a question about the like century
of this was was yeah you said something
about there being symmetry between C and
D but there is an asymmetry here which
is that if you find a proof of defection
you ignore it and keep searching
hypothetically yeah even though you
won't find one we think all right yeah
yeah turns out you had to be the kind of
algorithm that wouldn't have defected
had you founded a proof of defection and
if it was searching in Peru in parallel
for cooperation and defection I think
would just depend very heavily on the
search order yeah just like you you you
would get one or the other because it'd
be a program but which one you got would
depend
you know subtly on the proof search or I
think I don't know yeah other random
musings oh it doesn't exploit the
cooperate pot yeah so that's why Patrick
and others developed this prudent pot
thing that first kind of checks and see
if the opponent is stupid and if the
opponent is stupid it'll defect and
complete the opponent and then if it
realize that this is okay that's not
gonna work then it tries to go for a
cooperate cooperate scenario and that's
gonna work not gonna work then it's like
screw it
then it defects is that fair Wow Wow
it's like like if I know my crew system
like it's never way to calculate what K
hat is yeah I'm pretty sure it's like a
million yeah I don't know sound we're
between a thousand and a billion pretty
sure yeah for like whatever proof system
I would write my makeup piano arithmetic
with abbreviate you know with like good
encoding for numbers and yeah yeah you
have to write down the number K you have
to prove this and then say all right now
let k equals the number K it's so weird
so loopy the Hofstadter is probably
looping in his grave yeah whoa really
yeah oh man I didn't know that all right
good job Hofstadter you're not going on
the Internet you look like you had think
okay so another amusing consequence of
this is that if you defined benefit of
the doubt baht which is the same benefit
of that but is the same as fair baht
which tries to prove that you're going
to defect and if it finds a proof of
defection it gets angry at you and
defects but if it finds no proof of
defection it assumes you're innocent
until proven guilty right and gives you
cooperate seems like a nicer bot right
but actually benefit of the doubt plot
defects against Ben for the debt bot by
lobster the funny thing is one is when
you do benefit of the doubt plot against
fahrbot yeah okay wait let me remember
this benefit of that bought versus
fahrbot at least at least in this so I
haven't thought about the bound
inversion but in the modal version
benefit of the bat the doubt but finds
no fair bot
no it's got to be in it it's gotta be a
thing and then okay they both find out
proof
Fairmont events and for the Deaf
operates so Fairmont exploits Bennett
what a jerk yeah all right yeah okay so
that's interesting counterintuitive
stuff happens because lobes theorem is
weird because Gorillaz theorem is weird
so you might also think all right what
happens if you pit fahrbot K against
fahrbot k plus 1 you know the whole
point of this exploration was like
robust cooperation right so we didn't
want is me M is me but as a race right
but we're like annoyed that is me bot
has to be exactly equal to its opponent
right so it would be annoying if fahrbot
had to be exactly equal like the K has
to be equal or it has to be right like
there's something dissatisfying about
these things had to be basically the
same agent right and so after I proved
this Patrick yelled at me said that's
not good enough
you didn't really yell but he told me it
wasn't good enough he said go back go
back into your room and prove something
better like yeah so here's another
result we're going to define and the
definition is going to Cle into the
result there we know surprise so we've
got some agent like fahrbot that has a
parameter k in it and we're gonna we're
gonna spit we're gonna say a sub k is G
fair where G is some function so G and 2
n is some computable function and G
stands for generosity so when your can
put when your opponent is very
complicated you're gonna be generous in
your proof search you're gonna search
longer to account for the fact that your
opponent is complicated so that's what G
fair is G fair aka G fair if a proof of
length K plus the generosity function of
the length of the opponent length of
opponent source code oops let me pee
that should be opponent cooperates with
me the opponent of Asaph pay equals claw
equal to cooperate if this proof is
sufficient to ensure that a sub K
cooperates is that cool
so your G fair if you give a certain
amount of generosity in your search for
a proof of cooperation and then notice
I'm not writing if and only if here so
I'm not saying that that has to be your
sole criterion you could have other
criteria for cooperation you could also
be like ah also if my opponent has a
comment that is a binary encoding of a
cat video then I will watch the cat
video and then cooperate so it doesn't
have to be if and only if it's
explicitly you can you can call it as
long as your opponent being pretty good
is and simple understandable to you is
enough for you to cooperate then your G
fair cool so the theorem and this is
what I would just call robust
cooperation of added agents is that if a
and B so say I think you want G I forget
I think G you want to be bigger than L
cubed or something so I think if G of L
is greater then I think it's like a
similar function or two to the I don't
know what it yeah I think it's like L
cubed then if a sub K and B sub K are
both G fair then for all M and n greater
than some threshold K hat they cooperate
with each other
that shouldn't be okay that's the whole
point of this they don't have to be case
and in particular be could be like
vastly more any than a is Amy so
something yeah it's like there's this
intuition from there's like some early
results on finite automata cooperating
with each other in in game theory where
the reason they cooperate is that they
can't think far enough ahead to defect
or something they can't like they're
these games where like if you're gonna
you the iterated prisoner's dilemma like
you always defect in the last round and
like if you can't think far enough ahead
then you don't and then you cooperate
for a while and then when you start
getting close to the end of the game
you're like screwed so there's there's
this like lingering intuition around
where like cooperation is a symptom of
boundedness like there's this kind of
matric going around at least a few
economists that I've told this result to
you were like well of course they
cooperate they're bounded
that's what bounded things do check out
these finite automata that cooperate
because they're too dumb to defect yep
and I'm like it's not the other
right the other people right exactly so
you just have to prove things and so
anyway so they cooperate each other it's
so that's the robustness result and I I
think I'll just like end there as like
the punchline okay thanks
questions oh man I was aiming for 50 all
right so how do we think that this might
end up looking for things that do more
heuristic search rather than for search
because this is clear like not something
that you would officially yeah so the
first thing I thought of the other day
which is uh what do I have fair about up
here anymore not all right here yeah so
the first thing I thought about the
other day is like if you had some
heuristic for assigning probabilities to
mathematical statements then instead of
searching for a proof you can instead do
like think about it for K time steps I
think for K time steps and if you say
like Q sub K is your heuristic
probability assignment to mathematical
statements you'd say like if Q sub K of
this statement is bigger than 90% if
you're like pretty sure that the ponents
gonna cooperate with an asterisk here
which says like by the way proving it to
makes you sure like proof proof of
cooperation counts as being more than
90% sure so for example I proven to
people here in some yeah yeah like in
particular if you find us a proof of
defection it doesn't like if you find a
proof of cooperation and affection
you're still more than 90% sure so it's
a little symmetry breaking otherwise
yeah Scott pointed this out and I first
thought about this so anyway so you have
to have this like a symmetry this
counterfactual asymmetry you have to
have the property that if
counterfactually you found both a proof
of cooperation and a proof inspection
you would believe in cooperation instead
so you have to be like counterfactually
self fulfilling believer of goodness it
has to be nice the worlds in existence
right or you have to be optimist
if the world is inconsistent yeah
and so anyway then this thing is G fair
so here we have a probabilistic
reasoning thingy that does robust
cooperation so we're done but well we
need is the this also it's not clear
whether the probabilistic and you need
the asterisk yeah that's right all I
really yeah right yeah yeah all you
really needs the asterisk it's true for
this proof but like it just well I'm
just doing this to point out the obvious
fact that if you have a probabilistic
reasoner who also believes in proofs
like then it'll it'll work now it's
possible that it would like there's a
question of whether this thing gets a
lower K hat so my proof goes through
Loeb and like get to lower K hat maybe
it doesn't there's another thing you
could do which is like finally so this
is who thought of nicer but Stuart did
you or just get invite event nicer but
it's Stuart
sure yeah it's a fairly natural thing so
you could say another thing you can do
is let me prop basically co-op I'm not
gonna write it cooperate with a slightly
higher probability of corporation than
your opponent yeah and so then the only
fixed point if this thing plays gets
itself the only fixed point of that is
they both cooperate on yeah that old
thing yeah so that's the thing you could
do it death doesn't use Lopes theorem to
cooperate but it is may be another kind
of like robust corporation and I think
the whole point I don't know the whole
thing that's exciting about all this is
that like game theory has really not
explored this like open source
up for stuff so and like a eyes can be
open source to each other to an extent
we might be open source to AI like if it
studies us enough maybe it thinks it
really gets us maybe we maybe a eyes
open source to us
it's like active area research make AI
understandable right okay it could be
I'm just saying in principle I can just
email you and be like send me your
source code yeah exactly one proposal as
I mentioned is is if you if you want to
demonstrate your source code you produce
a source code file and say alright I'm
not going to like with your observation
make a new agent who this is their
source code and then when they're
finished my transfer all my resources to
that agent now you know the Machine has
this source code anyway and you have to
prove that you're gonna transfer your
resources to them which you probably do
using banks like you build with our
information you transfer the transfer to
the your resources and then they can
trust that thing oh nice oh you just
give it to them you just like here's all
my stuff yeah did that thank you nice
cool yeah so yeah there's like lots of
computer security work that would go
into like make implementing the game
theory of open source prisoners levels
just like open source game theory in
general but it's just wide open and like
also the one-shot nest is important
right for cuz for existential risk right
like if the first round of the
prisoner's dilemma is where like the
thing can destroy you or you can destroy
it by not turning it on or by turning it
off or etc right you're not in an
iterative game you just want it to go
well the first time so I think the open
source 'no sand the one-shot nests are
like important things that game
theorists need to be thinking about and
i'm makhani personally i'm not going to
be pursuing this line of research much
anymore because I think it's it's to a
point now where like I can give a random
talk about it once
get into some journal I can give a talk
about it at a game theory conference and
then maybe more people will think about
it I think there's like other stuff that
I think is more important for value
alignment than this that's like still
needs still needs to get to a point
where other people could be interested
in it enough to do it
yep Patrick is gesturing like Oh regular
blub-blub-blub could look like a Stewart
I think Stewart he's not gonna do it
either now their bus is gonna prove it
like I think there might be something
there might be some theorem of the four
so Stewart wrote a blog post outlining
oh yeah I mean so I don't know it'd be
cool if you could prove something that
doesn't have a parole quantifier in it
it just says if you have a proof of
length in that a proof of length K of P
implies P if n is sufficiently small
relative to K then you get P in some
other function event something like that
and if I think you could do use this to
prove that because what happens is
whenever you have a proof of a for all
that bounds the length of the proof for
specific instances later so you could
use this I think to prove that but what
if we already have robust cooperation
yeah cool all right


Stuart Armstrong – Reduced Impact AI and Other Alternatives to Friendliness – CSRBAI 2016

good morning and welcome back to the
Pokemon quilting series and robust and
finish beneficial artificial
intelligence I think be magical Victoire
our first speaker today is is Stewart
Armstrong and Alexander Tomasz fellow at
the Future command the Institute at the
University of Oxford Stewart has a PhD
in mathematics and has been doing in
doing research on corage ability and
modifying utility functions and the
feasibility of intergalactic
colonization I believe it will be
speaking on the first it means today so
join me in welcoming Stewart Armstrong
thanks there yes no galactic
colonization today that was just a minor
side project that got blown out of
proportion
so is the sound working fine
okay so ai safety I ignore whatever the
official title is what this is is
basically a box of tricks various
different methods to make a ice achieve
certain goals that I've come up with
that don't really fit in any coherent
theme so we'll look at some of those now
the problem of AI safety can be
summarized as AI is effective at
unbounded unfriendly goals
I've phrased it in this convoluted way
because each of the four main words have
different approaches and if you can cut
it off and probably because there is
still great uncertainties as always
where AI is concerned but there are
different methods for cutting off the
problem at different stages like we have
the Box Oracle arguably interrupts
ability or satisficing reduces the
effectiveness of the AI we have low
impact or in virtual world values which
we'll see later which aim at the
unbounded aspect
but you can have things like more
learning or friendly utilities to get
the values and some designs have to lay
eyes on other constructions now one of
the most important problems with dealing
the eye safety things is finding flaws
because it is very easy to solve the AI
alignment problem with the quote I
myself have done it five or six times
already and most other people who think
about it find the solution within the
first five seconds and you need to go
beyond that realize how hideous ly wrong
your initial solution was and try and
get to grips with the problem let's just
briefly look at some of the ways the
things in red can go hideously wrong for
instance satisficing now satisficing
means a lot of things one example I
often seen is that you aim for a minimum
expected utility rather than for a
minimum utility if you do that well the
problem with satisficing is that it
allows extraordinarily dangerous things
the AI can still do very dangerous stuff
it just isn't forced to it's not
motivated towards that necessarily but
it's not precluded from doing it but
some of the biggest danger in IT is that
a satisficer
has a very strong tendency to turn
itself into a Maximizer
in fact the act of turning yourself into
a Maximizer accomplishes your
satisficing goals in almost every
circumstance where you could achieve
your satisficing goals and there are
other reasons
it's basically unstable you would want
your descendants to be a Maximizer even
if you were a satisficer
now friendly utility this is perfectly
nice it just requires that we code all
human moral concepts now philosophers
have been trying for thousands of years
to get all human
or concepts but I think the biggest
challenge it's code
seeing as how long we can get how long
it's taken us to get a computer that can
say recognize a cat or recognize a ball
in a picture and seeing how simple in
our hierarchy at least the concept of
cat and ball are compared with ethics
freedom happiness flourishing this is
going to be a huge challenge finally we
have the tool AI variants these have
serious line problems which I'll get to
in a bit they are strongly motivated to
either give you completely useless
information or to lie to you there's
also a much weaker tendency for tools to
come into maximizers actually yes the
the one of yes okay I'll get to the
lying in the next thing so by the way
this is a far more informal than maybe
some of the presentations so please jump
in on particular ideas because don't
will necessarily wait till the end
because I'm going through different ones
and once that idea is gone I won't be
going back to it well I can if they
passed but we're looking at these areas
and buildings AI safety is something of
a bizarre process you have to sort of
mix formal mathematics the concept of
what you're doing
and the details of the setup and they
all have to interact properly now let's
have a look at the Oracle idea it's
simple you put the AI in a box and it
sends you a message and what could
possibly go wrong
well absolutely everything does everyone
know eleazar's boxing ai boxing
experiments
well I'll repeat them in case this and
someone watching the recording basically
Eliezer imagined himself as an AI
interacting with people who were keeping
it in the box
only through text messages and in three
out of five cases convinced the person
to let them out when they had a
financial incentive not to let him out
incidentally just through dialogue so we
can probably assume that a super
intelligence intelligence could do
better than that so this design is not
safe and now let's bring up one of the
biggest problems even beyond the whole
issue that it might manipulate us the
fact that lying is a default action for
an Oracle why is this well again suppose
you want to ask an Oracle who's gonna
win the u.s. presidential elections and
suppose it answers you with a long
detailed printout of the position of
every atom at that point or expected
position of every atom at that point it
hasn't lied to you but it's given you a
completely useless answer so when you
ask for it to give you a true answer it
needs to be speaking to you in your
particular categories according to our
own assessment of what counts as
truthful and what counts as misleading
basically it has to simplify and by
doing that defining what the
simplification and then still counts as
truth for us is a very big challenge the
there's other things we're lying as
default like this is for the thule idea
suppose we want the AI to give us a plan
to cure cancer and it's we try and
phrase it as a cunning counterfactual
this plan if implemented would be the
most effective at curing cancer and
suppose it comes out with a plan unleash
me as a sovereign AI and I'll kill
everyone in the world and there'll be no
cancer now we're unlikely to select that
option but suppose for some bizarre
reason we were
we started to implement it everyone
around us would say why are you doing
this plan which has kill everyone in the
whole world in the description of it so
actually a plan that is more effective
if implemented is one that achieves the
same goal but is misleading in its
phrasing so even if you start
implementing it if the plan is
misleading it is more likely to achieve
that goal so we have just by saying the
plan which if implemented would achieve
the best thing we've already motivated
it to lie and this sort of thing comes
up all the time when we try and get like
it would be a terrible idea to just have
the US vote on how good we thought the
ai's answer was or stuff like that again
we've given it a big goal which is
manipulating us and lying to us and
misleading us so the lying really
appears all over the place in Oracle's
however I know of two ways of getting
round this problem so what kind of
questions are we safe to ask well first
of all checkable multiple choice
questions in a non counterfactual way
and second of all counterfactual
questions about expected utilities
probabilities conditionals and things of
that nature so let's have a look at the
first one this is I'm calling it the
verifiable selective Oracle what does
this mean what has instead of letting it
give us a non unfiltered message we give
it a multiple choice from what you can
select it that's why it's a selective
Oracle it doesn't tell us a message it
doesn't argue with us it doesn't give us
any extra information we give it
thousand choices it gives us one of them
now what could we use this for well
suppose we want to invest our money in
the stock market and we want to ask the
Oracle which stock will do best
well here's over a list
say a thousand companies and say Oracle
which stock will do best that's fine it
is if the Oracle can manipulate us to
unboxing it by saying Unilever that
would be a great achievement on its part
so can I help I'm helping myself to the
assumption that the Oracle can be
contained but which stock does well on
the stock market is a very complicated
concept that's something that would be
complex to get in there so then we have
the second part of it which is the
verifiable part basically in a year's
time we're going to press a big button
and tell the AI or tell what score we
got on it so actually you said Unilever
but that was the third stock amongst the
thousand I press that and I confirm that
now if you have the AI motivated to
minimize the number on that button a
relatively simple goal two relatively
simple though to train on then as long
as it knows what we're doing and as long
as we are relatively committed to not
doing anything stupid around the button
we can get a safe answer and train it
with a clear objective now what can we
I'll get to some of the things you have
to keep in mind with this but what can
we use such an Oracle for yes we have
the wire headed problem
it's boxed this needs to be boxed you
know this is not this AI is hideously
dangerous if ever it gets out this day
is hideously dangerous however we talk
with it this is the AI that we have to
have the discipline we run it once we
get the answer we press the button then
we reset it
we don't do anything more with it this
needs very strong boxing for that
work and but seeing us to date I haven't
seen many ideas for getting good a data
even out of a boxed Oracle this is uh so
what kind of thing well the stock
picking example I gave there so there's
a variety of things we can check every
single one of them we want to know which
performs better under some metric that
seems doable now project funding
selection what do I mean by this well
suppose we have a lot of different
projects not AI related but say to
reduce bio risk and we have a thousand
projects and we want to fund it and we
ask the AI which one should we fund
which one will perform best at some
metric the the advantage of this system
is that as long as the metric is sort of
clear in our heads the I will figure out
what it is in order to act on the button
but big problem how do we know that that
was the best project if we only fund in
response to its answer we fund one
project we can't tell the AI this was
the best project of the ones we funded
but what you can do is say you have a
thousand different projects say will
will fund nine of them at random give us
a tenth and we'll our number will be how
much how well that tempt performs
relative to the other nine so that we
can at least compare now it is somewhat
dangerous to touch this too close to AI
but we can get things like improved
investigations same sort of thing if we
have a thousand ways of working on safe
AI and we can commit to not actually
building any safer AI for a year just
working on those projects and again same
sort of trick we're gonna select nine of
these to work on at random give us a
tenth will tell you our subjective
impression as to how much progress that
made or how how much that felt sorry how
much we progress we felt that that
project made
so we can actually get a surprising
amount of work out of this extremely
dangerous Oracle in this setup now what
are the risks well first of all we have
to reset it to ask new questions and we
have to make sure we reset it properly
that there's no a cause or trade going
on between its and other versions of
itself there's um I have a little trick
of this nature to try and break a causal
trade networks I'm not entirely sure how
well it works but I think it should work
in this situation
and a reset means that we can't if we
give it too many choices it starts to
become dangerous the more choices we
give the more potential optimization
power it has so if we give it a trillion
choices if that becomes really dangerous
but we can't chain sequences of this to
get to a billion by just doing a
thousand choices than a thousand choices
than a thousand choices because this one
is predicting what the next one will do
which was the next one will do so the
first one even though he's given it a
thousand choices we've actually given it
a billion choices so those are
considerations now if the AI figures
that one of our options will cause a
super AI to be unleashed it'll then go
and press the button for it it will
choose that option and it will probably
calculate quite well the probability of
that event happening however if that
probability is low it will still go for
the stock that it thinks is best for
instance in the stock because that has a
greater probability of achieving a low
score with a good stock than it has a
via unleashed AI so we have to be
careful that we keep this low when we
give it things so stocks are normally
should be pretty safe these especially
if we're investigating actually safe AI
we have to really construct the
for structure around this in a very
careful way and then there's of course
as a probability as humans looking into
the box asking follow-up questions doing
something stupid again we have to keep
this low somehow as long as we keep
these low not impossible but low it
should still be incentivized to give us
the best options that it can re off
right yes
if the box asks for more information we
turn it off I think there's no question
about that
now let's look at the second model for
Oracle's now here I've added an extra
thing I could call this the scoring
Oracle for the moment there is now a
process on the output and a ratio here
so sorry its output now is just numeric
and we're gonna ask it to estimate the
utility of a particular function U and
there's an erasure process e that'll
erase its message with a certain very
low probability and if there's no
erasure the message comes through for
simplicity purposes I'm Esther I'm just
assuming that you has to be non-negative
there's a version that works for general
you but it's much simpler with a
non-negative one so we want to motivate
it to follow this utility function
obviously now ignore this these terms
for the moment but just look at this
this is the indicator function for a
ratio what that means is that if there's
no erasure the AI gets no utility or
utility zero it has no control and no
interest in worlds where the erasure
happens
therefore all this the rest of it is
conditional on a world where the erasure
does happen as a consequence of that it
is not motivated to manipulate us by the
content of its message since a world in
which we read its message is a world in
which it doesn't care so that's the
first step
now what's this clunky expression this
is a proper scoring function that is
maximized when the AI has M as the
expected utility of you conditional on a
ratio so this is a counterfactual
because almost certainly we are gonna
see the message and that will tell us
what the expected utility of you would
be if the message were erased some of
you might know this as this is M in
brackets M minus u squared the negative
of that except I've removed the U
squared term that's just for various
reasons in that here we can tell exactly
how much utility the AI itself will get
as a function of its expectation of U it
gets M squared utility from that that's
more just a preference thing there are a
whole bunch of proper scoring functions
you can put in there to achieve that
goal now suppose we want to ask the AI
well what would be the utility
conditional on an event X well that's
simple we bung in the indicator function
for X into that and we get this again
this has to be an Oracle if it can start
affecting the e process or the X process
or something like that it's completely
unsafe but as an Oracle we would get
this so what can we do with this well we
can test a lot of things first off if
someone says I have found the ultimate
friendly utility function we can kind of
test things like okay according to this
utility function if a nuclear war
happens is this that
than if a nuclear war doesn't happen and
if then you feel what happening scores
higher we can start asking questions
if someone's cramps I have the ultimate
low impact AI and we say okay well what
if we open the box what would happen if
opening the box that increases the
utility again this is a something should
be questioned so it's very good for
testing fugitive utility functions
people have had ideas for how to test
them sort of honey pass ideas but all
the problems all the approaches suffer
from the problem that the AI is
motivated to lie this one is not
motivated to lie now I still would not
trust the lowest significant digits of
this because for the moment I've been
assuming that this is perfect this
person that we can only see the message
if a ratio doesn't happen and that the
AI perfectly understands what a ratio
means most likely won't none of these
will be perfect so there might be a tiny
1% or thousandth of a percent chance
that we still see the message even if
the erasure process happens and we need
to worry about those worlds but we can
actually quantify by looking at this
function we can quantify how much
utility the AI is willing to sacrifice
in a world where erasure happens in
order to manipulate us in a world where
we still see the message and especially
if you is bounded above we can put
bounds on that we can see how much you
two it's willing to sacrifice and we can
cut its message at the right place so
that it doesn't gain from manipulating
us so those are Oracle's
now on to more familiar terra-tory value
learning would any of you want to become
murderers
okay apart from the I'm generally gonna
work from the assumption that people
don't want to become murderers or if
they do they don't want to seriously
announce the intention in public but one
of the reasons why we don't want to
become murderers because we currently
don't want there to be murders and if we
became murderers there would be more
murders so our current values tell us
that we shouldn't change I notice here
that I'm not assuming any consistency or
that we should be following the past or
stuff like that just our current values
resist the change in the values
that's our Hondros one of basic AI
drives and his thesis and so how can we
get around it how can we change the i's
goals
well there's courage ability or words
which has a lot of things in it but one
approach that at least partially seems
to achieve this goal is indifference so
if you're changing from a utility you to
a utility v you don't instead
you change from utility you to utility V
and you add compensator e rewards to
balance it out exactly so that the AI is
indifferent to the update process and
therefore the a I will neither seek nor
resist its change of utility there
that's the old-fashioned indifference
what's the most recent result well what
about just stop it this is our
interrupts ability which I did with law
also of deep mind and this is more
looking at safe policy change so if you
want to override what it's doing rather
than override its values necessarily
then what can you do we looked at actual
agents that are you being used today and
this is the equation for Q learning Q
learning is naturally interruptible Q
learning is a so-called off policy agent
it doesn't see its own policy it you can
reach in and tweak its policy and
they'll never notice or care
it'll learn differently if it ends up in
different places and stuff like that but
it has no way of capturing inside itself
the concept of my policy has changed so
Q learning is safely interruptible this
is salsa you notice the equation is only
very slightly different but instead of
having a maximum here you have the
actual next action here this makes a
huge difference
salsa is on policy because of this term
it actually can capture its own future
actions however if you do a slight
modification of that to have it not
update on its actual future action but
on the action that it would have taken
if it hadn't been interrupted
then this sorcerer variant is now safely
interruptible as well okay these an
interesting result is also that a ixi is
safely interruptible now you might
wonder about this because I said off
policy agents were safely interruptible
and isn't a ixi on policy in that it
computes the values of the policies that
it follows yes it's on policy but only
on policy by coincidence in that it does
compute the values of the optimal policy
and it does follow the optimal policy
but it has no way of tracking that it
actually is following the optimal policy
you could shove in something completely
different for its actual policy and it
would still compute the values of the
optimal policy so AI X I in a lot of
is basically safely interruptible it's
not just that salsa is an a Cartesian
ation agent a much simple one but it
captures its own future actions in its
updates so the rule for you could
imagine a variant of AI X I which took
into accounts its next action or
something like that it no no I can't
quite imagine but you could it's not
because of the Cartesian boundary so
that happens so the Cartesian boundary
helps but it's because it really does no
answer when it looks at its past actions
it does not look to see anything about
itself it looks or its own policy or
values it just looks to see stuff about
the world that lives in now what I've
done here is I was imagining that there
was a process that changed the reward
function for AI X I so this would be the
equivalent of changing its values and
what the V is this is the values if you
have your AI X I calculate V as the
value of your policy it will not detect
that it's it will behave as if its
values its reward function was never
going to change
suppose a ixi every soft when we press
the button an AI x i walks into a box
and locks itself in there for a while
then what it'll conclude is walking over
there into a box is stupid this is not a
policy that I should be following right
what else is there to learn and then the
next time we press the box
it's walking towards the box it's
thinking continuing to walk is stupid
continuing to walk is stupid continue to
walk a stupid closing the door behind me
is stupid
yes and the reasons I put these up so
this is the true value function which we
don't want the ixi to compute and this
is the one that if it computes them that
way
it is it is its it'll it'll behave as if
its value from a reward function never
changes but now let's assume that it has
a little bit of reflection about itself
then it will sort of notice that it's
passed estimates were inconsistent or
something's wrong here then if we give
it another a compensator award of this
form this is basically the expected
utility difference thing but in a ixi
form then even a a ixi that has some
self model some consistency will start
will behave normally because it gets the
rewards that it expected to get so
there's nothing odd now this is very
roughly different agents types and
whether they're courage of all this
should be indifference value courage of
all interruptible or interrupter work
they know the environment and roughly
these are these types of agents that are
already of that type and here are the
variants of algorithms that can change
it as you get to model-based with a
consistent self model the only thing
that I know is the compensator rewards
you have to balance out its values
exactly but as you go for model base
with a self model you can just get a
some variant algorithms and as you have
ones that are don't have self model then
you can get different too
you can get different algorithms some of
them require a lot of more stuff than
this just implies now the reduced impact
AI or the low impact AI here the idea is
simple you want an AI that doesn't do
much if you tell an AI that doesn't do
much make me a hamburger with these
ingredients it might make you a
hamburger with those ingredients you
tell general AI make me a hamburger it
optimize the world to give you the
perfect definition of what your def
criteria for hamburger was sorry I don't
think that'll be the most efficient way
of doing it but um maybe but anyway so
the ideas that so many goals so you are
little you are utilities that are
extraordinarily unsafe become perfectly
safe if we can shove in some sort of
penalty for large impact now what is
this penalty how can we define impact
and this gets really tricky because the
AI is turned on its heating its thinking
this percolates through the world
there's chaos it has a huge impact
already just by existing so we have to
try and sort of filter that out somehow
come up with the definition in fact this
is something that works best for super
intelligences that are not godlike in
their predictions if the super
intelligence cannot predict the outcome
of a chaotic process these methods tend
to work best but let's look at it so the
first thing is to find some sort of
point where we can define what low
impact really looks like and an easy
example is we turn on the AI or we don't
turn on the AI
this is low impact
so that's a good thing to compare with
so what we're looking for is some sort
of distance metric between a world with
AI is turned on in the world where it
isn't or over the problem some probably
just T distributions in these worlds you
notice I put them in a box but the box
now is not to contain it not to keep it
imprisoned it can be made of cardboard
but it better be made of dark cardboard
because the box now is to prevent us
from seeing the AI the AI is wants to
minimize the difference between those
two worlds so if it wants to be the
world to look like something where the
AI isn't turned on we better not be able
to look in and check that it was turned
on so this box is to shield the AI so it
has the possibility of low impact now
what's the first method well this is
what I call the 20 billion variables
it's you know how sometimes when you
come up with an idea for someone coming
up an idea for safe AI and you say
wouldn't the AI do that and they say oh
yes and I'll make sure that it cares
about people's reported happiness yes
but wouldn't it do this oh yes I'll also
add dopamine in that what about fritter
free will we're then will get people to
vote on so they keep on patching the
idea this is we apply all the patches
ahead of time something like define say
a sphere a few hundred meters around the
AI and anything that's outside that
sphere is fair game we divide to find
the air pressure in Alaska we define the
Chinese the Shanghai Stock Exchange we
define the next winner of the Oscars we
shove in everything and we get a huge
chunk of variables and we describe the
world by this chunk of variables and
then the penalty becomes the world as
described by these variables should be
the same
or very similar whether or not the AI
was turned on and one interesting thing
about shoving all these variables in is
that they it means that you can add
variables that aren't really
well-defined
like I said the Shanghai Stock Exchange
results but some quite a complicated
concept or maybe I should then encode it
as the outcome of a output of a wire
which reports on the Chinese Stock
Exchange results however this is the
kind of thing that's very dangerous to
give an AI maximize something output of
wire could just go control of the wire
and put what it wants however if we've
shoved him enough other variables it
should be extremely difficult for the AI
to take over the world to get control
that wire in order to re rewire to its
desires so we can basically shove in
unsafe variables or poorly defined
variables and they would arguably add an
extra level of security basically as we
wants the AI to be unable to grab the
whole world without really changing a
lot of these things along the way
once it's grabbed the whole world that
can probably control them all but this
is to prevent it to being able to break
out without affecting these there's
another way that is described in this
incomprehensible picture which is that
if the AI is undetectable we can assume
that it has low impact so if we take a
slice of the future and observe from it
and our observations are G then if from
G we cannot tell whether the AI was
turned on or not it's probably safe to
assume that it has low impact this
requires certain counterfactual
reasoning about an idealized reasoner
seeing logically counterfactual sorry
about an idealized reasoner that sees
this and blah blah blah but this is
another way of
getting a penalty function now this is
if I was to use these ideas successfully
I would have programmed brilliantly a
useless box now that we have an AI that
doesn't do much how do we make it do
anything well the first idea is to just
use that button the idea that came of
the Oracle actually thought of it first
in this context just conditioned on the
output of us pressing the button in a
year's time maybe give the give the AI
the thousand stocks as an output and if
the eye is turned on is not turned on
one of them is generated at random so we
can't conclude from the fact that we
receive a message the AI was on or not
then conditioned on what we press in a
year's time by doing this conditioning
now the AI is motivated what if we've
given it the motivation is motivated to
minimize the output of the bus and while
otherwise behaving as if it wasn't
undetectable yeah minimizing its impact
on the world ah yes you have to think of
all of the natural consequences like you
think I worry about though impact as you
say produce a cure for cancer but yeah
but otherwise have low impact and it
comes up with a nanomachine that doesn't
matter to cure cancer but replaces it
with something that is not cancer so
that you is satisfied but it causes
people to get sick and die the same way
that they would if they had cancer yes
which is why you don't ask it to cure
cancer even if it has a low impact thing
this approach only works when you have a
relatively small yes because I had
another idea that was very dangerous
because notice here I'm not filtering on
the content of its output
in this in this setup if the AI gives us
the instructions for a nanomachine it's
had a large impact because we're not
filtering on that I had a previous idea
where you did filter on its outputs
channel but that's extremely dangerous
so here I'm plugging in the reduce
reduced output and as if so select
amongst these in the same way that a
random number generator would so this is
if you want remember the first Oracle
works with this where instead of having
a perfect box we have this motivation to
control it so it fits very well with my
first Oracle design but ya know but
nanomachines that caused people to die
as if they had cancer would not count as
reduced impact here because people would
realize that there's nanomachines
rather than cancer they'd start talking
about this this would have rather large
effects on the world this this sort of
balance is very fragile it well it's
very fragile once you start doing
effects you have to filter via exactly
by what the effects are allowed finally
there's another possibility which is of
getting high impact from - low impact a
eyes so here it's the old aiming the
laser as an asteroid problem the
asteroids coming to earth if we shoot
the laser we destroy it this is a very
high impact thing so it would seem we
couldn't get a low impact ai to work on
this idea we have X which is 1 AI it is
motivated to aim in the X direction for
the lasers we have why it is a motivator
to aim in the y direction to hit the
asteroid however they're both reduced
impact but they're both reduced impact
conditional on the other one
not being turned on so this one is
thinking this is perfectly safe I'll
output my thing I'll put the right
coordinates but the lace is gonna miss
anyway because this one's not around
this one's thinking the same thing you
aim the laser both of them are turned on
and it actually works this yeah if you
don't have any way specifying your goals
rather you are right and I don't imagine
that will be using super intelligence
reduced impact AIS in this way it's just
the observation that we can actually get
some very high impact by chained
together reduced impacts in this
interesting way in the case where
they're both on you can in the case
where they're both on they're basically
either that is an undefined state or
ability nurse that they don't consider
or there is a different utility in that
case well they're they're optimizing if
you want you X a me correct x
coordinates - the penalty term condition
and the whole thing conditional on Y not
being turned on now this works for
computers it doesn't work for robots
because if you can keep the computers
isolated from each other while they're
doing this calculations but if the robot
says okay I'm working conditional on the
fact that that robots is not turned on
so why is it walking around and waving
and then it enters some rather
entertaining undefined weird States
right I think this is the final idea AI
is in a virtual world it's an old idea
you have an AI that only cares about a
virtual world that you defined and that
it only does stuff in that virtual world
and yeah everything's safe no problem is
the AI can influence the outside world
can influence what goes on in the
virtual world which means that if the AI
can influence us it can influence the
end the it's virtual world indirectly so
you can like the it's running on
something which doesn't exactly
correspond to the software abstraction
that is supposed to care about um and
then even start the outside world end
and they said they could achieve its
goals better if it turned in this
outside world in the computer omnium
better simulate it's for instance or
just wire headed itself via making us
redo its virtual world so here but you
told you sort of got in too close to the
nub of the problem we have the virtual
world versus the virtual world with our
influence in it and the basic idea is
that we want the AI to know everything
about its virtual world including itself
and just for simplicity's sake it'll
assume both is deterministic the AI is
deterministic and the world that
operates in is deterministic so it has a
self model that's deterministic and we
need to distinguish this from a world
where we reach in and influence it and
that's why I've just defined it as the
same setup except that sometimes
stochastic things happen
these exactly what stochastic things we
can allow is very tricky
I basically you can use anything that
does not react to the ai's actions and
that doesn't have an a causal connection
with the AI but that's a more subtle
point then what do we motivate it well
we have an internal utility function and
we have maybe a shutdown command now
since this is actually a Cartesian
idealized world we might actually be
able to define shutdown properly and we
have some sort of utility which is
maximized you as long as you think we're
in that world shutdown as soon as you
think we're in that world and so the
shutdown is basically as soon as it
starts detecting outside influences in
its world it shuts down or to a certain
extent there's questions of weights and
a whole host of things there and so the
challenge is to have a safe and useful U
and W you can't there are goals you can
give it they're unsafe if you have
virtual humans in there and you say
seduce these virtual humans even though
it's in there it's still unsafe for us
to look at the messages that it might be
generating but what I generally think is
that this is useful for testing models
to destruction
W would be our model of the world W
would be a model that we are testing and
we would just want the AI to do massive
amounts of things with it like again you
can do if someone has a model of a stock
exchange that actually is good but no
one knows how to compute it properly
then by all means go ahead of there if
people have models of cancer cells how
they reproduce and how these specific
enzymes would minimize them models of
protein follows models of docking you
can put lots of models
here now the most likely thing is that
the AI will compute an edge case that is
technically in your model but not
doesn't really solve the problem and
that tell you your model is wrong you
patch the model you try again
it's safe generally to look at the
outcome of this because there is no if
as long as we don't put something that
is human-like in there the a will not
behave in a way that will seduce the
human-like entity and yeah this is
basically it so the AI is effective at
unbounded unfriendly goals and this
problem or the different part of it is
possibly solvable thank us putting the
probability of a race shirt one and ten
million yeah well if that die comes out
we just reset the air and run it again
we have to have unless we build one
Oracle and ask it one question we have
to assume that people will be resetting
it somehow so as I said the probably
doesn't erase so I try to influence what
the utility will actually be by my a
message because even these - actions
that raise this causes like this not
that
this is where we have to break the a
causal trade thing and because remember
as stated there is no problem because if
it's message is read if we can define
specifically the erasure event that
we're talking about then it doesn't care
if it can set things up for a future
ai's erasure event I said I thought I
had some solution I think I have a
solution to a causal trade I don't
understand what a causal trade is so
let's see where is it okay this is the
set up so instead of maximizing you you
maximize you minus the expectation of
you given that you were not turned on
now this doesn't sound very dramatic
this seems just to be a constant term
but imagine if you had an a causal trade
so a true a causal trade then whatever
agents are a causally disconnected from
you they will do the same thing in a
world where you're turned on in a world
where you're not turned on because they
can't tell the difference because they
don't have a causal connection with you
they might know the probability of this
which we would set very low but they
can't distinguish between the two so if
some a causal a I in the next Galaxy
sorry the next universe will build you
paper clips that shows up here and that
shows up here
so given some linearity assumptions
about utility this does not motivate you
to engage in a causal trade because the
thing will happen on both sides it does
not oppose he gets run if you will be
erased well be yes you could try and
make them causally separated as far as
they can figure out or you could the
equation as stated works the question
because it defines a specific as long as
you can define a specific erasure
process and you can distinguish between
different copies ask the same question
or we could say there might be some
safety protocol for you nice fall which
includes resigning ourselves to like a
future without a I helped if maybe one
in ten million chance it's the only way
to be safe for the rest of humanity I am
confident that that will be implemented
without any problems whatsoever but yeah
no I don't think that's realistic
because it also means that we can only
ask them one question because if we
reset them even if the erasure didn't
happen if we reset them to answer
another question we got the trading
effect there so yes AI being asked only
one question about about a utility I
don't see that ever happening
well yes a causal trade is probably a
weakness of every single method that
I've put up there except maybe the
low-impact ones but it is yeah
a causal trade or variants of that a is
collaborating when they shouldn't be
we'll break most of these things it's a
problem I'm aware of and most of the
devious ways of manipulating a eyes have
that problem because especially if
you're trying to motivate them in weird
ways then trade means that they're weird
motivations will become more general and
things go bad yes I think that is a
different kind of flying than deception
where the eye gives you instruction
tells you something while knowing index
what do you think it does is actually
not what it really does okay
when we think about lying we think about
the theory of mind right that the you
know I can model in your mind I can
model what you're going to think it does
if it gives you those instructions or
that information so then do you see
reason why you couldn't when you
couldn't keep the simplification while
not having exception so that it but it
has to then it has to model things on
that it has to give you instructions
that will match what things what you
will think it will do um possibly I mean
there's also the whole things that
people lie to each other all the time in
ways that are expected that we fill in
the blanks in what's said we tone down
some of the larger claims people who are
always truthful with other people find
this very odd and this is our found very
odd and what I'm so that that point is
defining what a truthful statement is
for a human is a non trivial question it
has to be a simplification within a
certain model within our model of what
accuracy is so there's a lot of work
that goes into that now if the AI was
aware of the model basically if we had
the model we might be able to say to the
AI do this this is what we want if the
AI has to infer the model for our theory
of mind then it becomes very difficult
to say in your inferred model which we
can't really define because it's inside
the AI do the good thing or do the
truthful thing and you of course would
need to be much more precise and I'd be
now just example but with your answer
give us something that will actually do
what we think we're doing because for us
this would be an extremely hard problem
because we cannot we cannot infer what
what you actually what's actually going
on in your mind but if we expect that
the AI would be very good at deception
we we assumed that it would be very good
at modeling our minds so then the
inverse is true to that it would be very
good at knowing that it is being
truthful to you or that is it's yeah but
but this is another version of do what I
mean
and again the problem is that the AI if
we program an AI with a bad goal like
cure cancer it is going to know exactly
how we should have programmed it if it's
smart enough to get the goal that we
really intended to but it's not
motivated to follow that goal this is
what I was saying about the model of
human and lying and stuff like that we
have to define what for us not lying is
and what we want it to be is construct a
model of what lying means for us and
then don't lie ok I'll just say one
thing then we can maybe talk after it's
it's just basically that we are saying
we're not giving it the right we're not
giving it the full criteria but that's
the point we can't point to its theory
of mind we either I mean ok some people
eric drexler for instance was thinking
of decomposing eyes into modules it
occurred to me that if you could do that
you might be able to say put the theory
of human mind in here put the other
stuff here other stuff there and then
use this as a definition but in general
if you just have an AI the AI is going
to create a theory of mind as an
instrumental goal and we can't if we say
use your instrumental goal to do
something then creating that is no
longer an instrumental ok so more
questions
yep okay because like either this Oracle
project flows out like a bunch of money
or perhaps someone will find out that
the AI Oracle pointed out this that we
do really promising company but that's
okay I've been there you sleep at my
soul you know the reduced impact will
not solve that problem but remember if
it has one stock that has 60% chance of
coming first just via normal market
effects and it has another stock that
has 59 percent chance of building an
unfriendly AI that will go there and
press the button for it it's still gonna
go for the first stock yeah you there is
yeah these are non-contractual x' so
they will be self-fulfilling prophecy
aspects to it your attempt to like put
money into it will decrease your
expected return on on that so that you
cannot pick a stock that comes in number
one market money comes in number two but
that's so that's so fine it'll pick the
least self-defeating and the most
self-fulfilling prophecy that it can
but ya know there are concerns of that
nature but again you have to sit down
and sort of think really figure out
seriously if so you could well to solve
that particular problem you could just
not give it Google just give it Unilever
Mars Pepsi these are but ya know that
there are issues and you have to think
through them carefully but most of the
time when you say there are issues and
this a is and you have to think through
them carefully that you have to be
afraid that the AI will not think you
here assume the I want think you and
assume that there if there is a devious
path of high probability it'll find it
we just have to try and ensure there is
no devious class of high probability
because it has very little influence on
the world so we need to do our due
diligence to check that choosing one of
these is not going to break the world in
that way I mean it would be completely
stupid here we have a hundred different
super intelligences which one you want
should we run but what might work if we
have the discipline is which one do you
recommend we want we would find the most
friendly if we analyzed it for a year if
we can analyze that and as I say nine
other randomly chosen one for a year and
then say yes actually that's one seems
pretty safe that I have to think about
it because it seems that there could be
other hideous things that go wrong but
it seems the bad thing would be
potentially safe so we can actually get
some information out of here but the
division between a safe question and a
dangerous question is not so clear so a
lot of thinking with that model
all right so those thanks to it one more
time