Bart Selman – Non-Human Intelligence – CSRBAI 2016

morning and welcome back to the whole
game series on the bus and beneficial
artificial intelligence co-hosted by
machine intelligence research institute
and future command institute our first
speaker today is is bart salman
professor of computer science at Cornell
University and a prolific researcher in
areas of scaling and computational
sustainability now representation and
tractable imprints and much more and we
are also very pleased to have him as a
research advisor for us here at nearly
so he's here today to discuss Kurt
progress in machine reasoning and human
understanding of such and so I join me
in welcoming professor of art Salman
okay thanks it's it's great to be here
my first time but it looks like a very
nice place so so i'll talk a probably a
little different than the other touch my
areas is machine regional refutation and
it gets less pressed and she learning
but the progress has still been dramatic
and i'm actually looking forward to the
time and i've started happening when we
start combining machine learning and
machine reasoning but so i'll focus on
machine reasoning so muslim we think of
AI we think of human intelligence and i
like to say that's because we gots the
intelligence we know and you know we
split we bother down to perception
learning reasoning planning knowledge
and of course the current and trans of
machine learning especially deep
learning is really it has really changed
them what we thought we could do at
least in perception and learning if we
have in us data so so that's very
encouraging and actually a lot of the
concerns about AI about the I safety and
where things are going of course driven
by these advances I like to look at what
i call non-human intelligence is it's
not
it's just a good less attention but but
the advances also have been have been
very interesting and their reasoning and
planning so I it's actually parting not
getting as much attention in the AI
world because it's more used in software
verification program synthesis and i'll
talk about automating signs and
mathematical discovery so it sort of
other areas related to AI but not
central part of a I that are using these
recently technologies but especially the
software verification world the
Microsoft and Intel an IBM push push
these reasoning programs very hard and
that's why there's there's so much
progress and I think it will start
feeding back into AI in in the near
future so these developers are really
not motivated by human intelligence
they're really motivated by just getting
capabilities like verification
capabilities proofing programs correct
verifying parallel protocols we don't
really care we only think we're not
really concerned about how humans would
do it or even if humans could do it we
just want to want to verify these these
systems so it's really you know it has a
different motivation than love AI work
ok and it's totally machine focused ok
so the tire you know verify this code
synthesize this code and the difference
you know the last 10-15 years is we have
systems that can do billions of little
inference steps and in an intelligence
way you know I've my early work was in
planning and early planners in this
going way back in early 1990s the
playing systems could do plans with
suing him that the robots that that
would do little blocks tucking tasks and
they could do you know 10-step plans you
know maybe 15 stuff plants and that's
where these systems would really stop
and planning was sort of considered too
hard that's
we changed last 20 years now we can do
plans with him a thousand steps
optimally 10,000 steps if you want
near-optimal plans so we can generate
plans that are way beyond any human
capability so humans could not
synthesize plans of that size now
there's still there's still many Asian
planning that need to be addressed but
in certain areas from some areas of
plenty where the image is very long
plans waiting to get plants we've seen
tremendous progress okay i will have an
impact on robotics so I want to discuss
a particular example of a reasoning a
problem just to give you a flavor of
what's feasible nowadays and I think
it's just the beginning in many ways so
you might have seen it is it's the air
duster discrepancy problem how many of
you have seen the Erdos description
problem it's vaguely heard of it okay
great so but but i'm just going to
illustrate it with a little example so
consider this sequence here of minus one
plus once accepted sort of randomly I've
picked them but we're going to imagine a
very long sequence of minus one and plus
once okay then I'm going to look at
basically sums we're going to look look
at sums of sequences and sub sequences
in particular let me illustrate it the
green ones these are just you're taking
every term of the sequence and I'm going
to look at so the first two terms okay
11 Plus 10 and now we're going to make
the third term plus one gets 1 4-1 get 0
X is are we making we are looking at the
sub sums going you know from left to
right okay then you see here the outcome
of 3somes and I hope this is correct i
give this talk a few weeks ago where i
have mistakes on these additions we just
costs embarrassment but I think now I've
double check that and and I think it
works so and you see it goes up and down
this son okay now to make a little more
interesting
we're also going to look at at sub
sequences where you jump a space so in
this case I'm going to jump one space
skip by one okay in the rat see you
later on I'm going to jump two spaces so
I started 2 4 6 8 3 6 9 12 etc but I'm
gonna get pic the term so 1 plus minus
10 plus 1 18 plus 1 is 2 etc so I'm
going to look at the subs the sub songs
of of these terms selected you know the
top one is really the key 1 plus minus 1
and the others are just in Jesus okay
and again you get these thumbs okay
going okay so um so basically in general
so what Erdos was was asking is you know
you see these sons growing I actually I
don't you know if I had a few more minus
ones here this could go down to minus 1
minus 2 but air does a discrepancy
problem which would she consider i think
around nineteen fifty 1940s roughly is
can you find sequences where the Sun
stays within plus 2 and minus 2 okay so
how long can you make this sequence that
all these sub sequences and any other it
becoming menu sometimes yeah they can
jump Isis by hungers but 200 second
maybe they're tremendous number of them
anything that fits into the sequence you
selected can you always keep that some
within minus 2 and plus T okay that was
the question okay and this is one of
these reasoning results so 2015 it was
shown that there exists a sequence of
1160 plus 1 and minus 1 such that the
sum of all the sub sequences is between
minus 2m plus 2 now if you started
thinking about this as a search problem
you know how many secret is order to to
the 1160 that is about 10 to the 350
okay so they're quite a few and so you
have to so it doesn't know hope for
enumeration
or anything I actually want to point out
one little thing here why how does
inference come into when you think of
this problem well let's say we're here
so we're having up till this point okay
this sequence and this sub summers too
okay now we can do a little inference we
can say wait a minute if this is too
hair I cannot have a one here because
then I would go to three okay so it
means that the next entry has to be a
minus two so while you're building you
sub sums up you get these kind of little
influence step that you say okay I know
something about my sequence and that's
what the reasoning programs do they they
look for in this is a very small example
but of course it happens in lots of
places that's gonna happen when you see
it minus to come you know the next one
has to be +1 okay so it all has to fit
together okay so what I want to do is
say a little more about how this is done
and why it's interesting okay so so
first I have here this is the sequence
okay and you can of course why have a
little program that checks lives and it
works beautifully so I said put it in a
40 x 29 pattern I've actually you of
course we're gonna do some deep learning
on this but we you've never done that
but I wonder like what it's predicted
for you can predict from once one part
of sequence uddin export but let me see
okay so if a part of my problem well my
point did this dog is to stress that
this was done with a general proposition
of reasoning engine it was not done with
any specialized reason for this problem
in fact there's a family one on
polymaths projects which is a group
project due to do mathematics is over
the web everybody can contribute was
started in 2010 and the went for a
number of years and people who write
specialized programs for this problem
like you know and
and the most advanced ones we would
would build unknown types of other
sequences and try to modify those other
sequences but they couldn't come close
to 1160 I mean towards the end anywhere
up to 11 20 which is pretty good but
early on they were down to write you
know 100 200 that they could find the
longest sequence okay so they're really
in the project actually sort of died out
because you know we're going to come to
the the next thing is what about 1161
but but but they couldn't really get you
know you want to answer this question
one way or the other is there a sequence
that that that can always stay within
plus minus one and plus minus two plus
two we are really interested in if it
gets too long maybe you will go out of
that range okay but people that looked
at the specialized solvers and
specialized programs a lot of mass of
course it's done by mathematicians so
they're really interested in group
theory etc so they used you know the
best ideas they had basically but he
couldn't get cause until somebody from
them I think somebody from the
satisfiability world that it's called
boolean satisfiability solvers or sat
solvers s80 solvers until somebody
picked up this problem said let me just
give it to a silver okay now the person
was a little lucky because saw was five
years ago couldn't solve this again so
is now can so it was also a question of
timing okay so just a little taste of
fear when I say so it translated
satisfiability phone there's a little
taste of what that looks like this is
not this is actually not from this
particular mass promises from the world
of verification just to give you a
flavor you know we tend to to the head
of that just says that this is in
conjunctive normal form which is just a
particular logical format the variables
are numbered so from 12 I think I got
several thousand variables are in this
formula everyone we use very simple
notation minus what means that
it was negated so not X 1 or x 7 and
that's our first little disjunction okay
little claws as we call it and we have
to satisfy it by either setting x 12
false because then this becomes true and
we're done or we said X F into true and
we're done too okay so now what you see
here you see all these minus one so you
might actually already suggests well
maybe we should set next one to false
then we'll satisfy those first you know
700 forgotten 67 clauses 20 s by the way
to use the end marker of the clause okay
so now the goal is given a set of claws
each variable is a boolean variable to a
false find a setting for the variables
that satisfies all these clauses all
these constraints or show that no such
setting exists okay that's the goal with
the satisfiability sauveur make it a
little bit more interesting so here's a
little bit later on the cloth this one
here you know you see this is much more
interesting that has many movie
variables it also has one not negated
okay so maybe we actually have to set x
12 true you know because maybe that's
the only way to say we don't know that
could be the only way to satisfy this
clause okay so this is of course what
makes the reasoning from interesting and
makes it hard it's an np-complete
problem one of the first problems in
computer going to be shown actually it's
the first one to be shown NP complete
but giving evidence that may be the best
possible algorithm has to take
exponential time if NP is not equal P
and that's still the state's we believe
actually mp's nautical p so we believe
it in the worst case this really
requires two to the end time where n is
the number of variables okay now if that
was the real story but complexity you
know not much could be done and
sometimes we will be in real trouble
okay it turns out that many problems
were interested in including the maths
problem i just put up the worst case
complexity is far better
far better than this then this
exponential behavior and we'll see that
in a moment okay so four thousand pages
so that causes you they do get more
interesting so this is some verification
problem fairly for the old one but
fifteen thousand pages okay here you
have one like this final statement says
the variable 185 which is probably input
variable to destroy kit that might be
fixed at +1 so you have to set this to
true to satisfy this class but now what
about the rest okay so this problem has
about 50,000 variables so it has a
search space a roster express of 2 to
the 15,000 or serif 10 to 15,000 and if
satisfiability was truly exponential for
most instances let's say in the typical
case you could never solve this okay you
cannot my my guides you can search space
of about 10 to 12 20 to 14 may be
exhaustively but that's and that sort of
the limit 10 to the 1500 s is infeasible
however as I said there's much more
structure so current SAT solvers solve
this particular instances in a fraction
if you get a few seconds okay and now
this this was unsolvable about 20 years
ago and now it's down to a few seconds
okay so and in fact this instance may
even be unsatisfiable so it's goes well
it's for satisfiable where you find the
assignment but this instance i think is
unsatisfiable on the soulful saying a
few seconds there's no solution okay so
implicitly cleverly it's searching this
space but of course it's doing nowhere
near an exhaustive search okay so back
to our sequences okay so just a little
bit about how we encode things we just
have variables for the sequence these
are the obvious thing so we have n plus
1 and minus 1 so we get x 1 to x and
there's our first set of willing
variables and we interpret you know
if in our solution we find X 1 to be
true then we take that to mean it splits
plus 1 and in our encoding and if it's
the final solution has x1 set to false
we take it to be minus 1 okay and if
it's unsatisfiable an hour ago if our
encoding is unsatisfiable it means that
there is no sequence that that stage was
in minus 1 and plot lines to imposter
okay now a little bit more interestingly
ah you couldn't quite a guy you have you
know these are the starting variables
you need a few more variables you need
for that sort of makes sense they're not
not difficult this sort of a very
stirring once you get used to these kind
of encodings it's a very straightforward
encoding but when you first see it it
looks like a little strange we're going
to have propositions of the following
form the sum of the first two terms of
the step by two sub sequence is a
propositional variable is it's all right
now that's not the problem hey we want
to know what value that is ok so that
can be to for example and this will be
true or false this is basically saying
you know if I add up so my let me just
quickly I think it's yeah so if I go in
the first two terms of the of this of
this some of the object step by I didn't
to step by two and stuff I one and step
by two maybe oh I guess now actually
again I might have an error on the
slider yeah yeah yeah I thought that he
gets to two so it will be true ok i just
realized there's another because i want
to get on this guy so I mean is actually
love me so we can we can basically we
have variables that that allow us to
talk about these partial sums and of
course we're going to have many of them
and they talk about the values of those
sums okay and for any given setting of
the original patterns of the original
and Plus Ones minus one you know these
predicates will evaluate to plot into
two or false okay because they actually
encode what what the sums are okay and
then we ask you know these are the
variables okay and now we have
statements and I think this is so this
actually should be x 9 but so if the sum
of the first two terms of the stuff to
such things is to if that's true then we
know something about the next thing we
don't want to go higher so we have to we
know this should be x 9 by the way that
that that next step should should be
minus 1 okay because otherwise you will
go to three ok so here we have if we
have these to observe observations up
sir then what Leslie encoding we have
these two observation this will be X 9
now that means that the sum of the first
three terms of the step by two
subsequent it goes down by one okay so
true goes so remains true but we're
encoding what happens depending on again
this should be x9 what happens depending
on that next value okay so this is a
little little module encoding of what
how these sums change okay and and
that's all really I mean there's not
much more to it so now if you do the
extra work and I actually did that just
to just to double check it so you get a
problem is about 37,000 418 variables
and about 160 1460 clauses and that's
actually not a large problem for current
solvers okay and in fact on my macbook
air I fun to see you in about an hour
okay so doable okay so uh you know you
might actually wonder you know maybe the
sets overs just lucky and finding the
sequence okay and there's always you
know that that worry I guess then maybe
and it's it's just a fluke but we gotta
we gonna check that
and I think you know how how we know
that's not the case is if I make it one
longer and this I think is a real result
okay then i can show that encoding
becomes unsatisfiable okay meaning no
matter what sequence a pic of a hundred
to see 1161 Plus Ones and minus ones at
some point one of these substitute a
little has every sequence there will be
some point where the Sun goes to plus
free or minus three okay that encoding
look at it and counting is not hoops you
know it county is only slightly bigger
because you only have one more bit so 37
thousand 462 a few extra clauses okay
and now you fire up the the solver and
in about 10 hours it's a little longer
sometimes longer but actually I think
it's multi-threaded so still there's
about an hour but my macbook air told me
it's unsatisfiable okay so basically
there is no setting and and it shows
that you search that that is sometimes
you search the whole space so it's not
like you were lucky to hit the solution
for that 1160 because if i give it to
space we have to search the whole thing
it also figures it out okay so I and
this is the interesting part or 30
gigabyte and now you can actually print
out that proof the proof of that
unsatisfied building why is it done
certifiable okay you can print that the
steps of the prove out now the proof
steps I just given a little example here
the proof steps are very basic so
they're literally they started with the
original clauses and then basically says
how you should combine them together
okay and you get the statements like
this you know three lines a and not B
give see so you can now write C as the
next step in the proof okay that level
of proof that's why we can have a 50
line you know or the 51 program that
takes that 13 gigabyte proof which has
about a billion in front steps and check
it okay
and everybody can write their own little
checker and and just do it and now
you're certain okay that that's actually
correct now of course what we're left
with is and the Machine can understand
this human never really understand the
proof i guess but for the machine the
billion steps is not such a big deal but
the key thing is we can be certain of
the result okay and that you know that's
my first observation it's it's different
than early come here you might have rats
in this result anandam hit the press the
probably most famous one was the four
color theorem that was done in 1970s
that any planar map can we call it with
for callers and that two countries that
border as two different callers okay
this was a famous own problem in graph
theory and it was resolved by hagen
hagen hagen and apple in I think 1990
late 1960s or 1970s okay and it was a
huge it was a for those days it was a
huge computer proof that checked where
we rejected thousands of tens of
thousands of special cases of the
calling problem the problem was when
that was when was published in the new
it was quite controversial in part it
was controversial because how would you
really know that the few improver they
used to train all these cases was itself
correct okay it was you know hundreds of
lines of code on in published thousands
actually so and in fact when people so
the only thing people could think of is
well let's try to replicate it by
independent groups and actually in that
replication a few missing cases were
discovered they were added and improve
remained correct okay but that the only
confidence was over time I think it has
been done two or three times at least
redone the proof is a different theory
improve or new code
getting the same result I can call our
own maps so that's how people build up
confidence but it's not quite
satisfactory for mathematicians because
well maybe everybody has a subtle bug in
there inert theorem prover and proving
and showing these kind of see improved
correct and programs correctness is
almost infeasible so the difference here
is you can throw away the stats over
i'll give you the eight gigabyte file
you check it ok actually yeah this is
moving faster than the record now at
1212 terabyte proof and just a few weeks
ago came out for a problem about
coloring of the Tigris particular
triples so there's another result that
has meant 12 terabyte proof but anyway
the proof is out there you can download
it and you can check it second thing a
bit of confusion over there people so
sick of these souls is doing boot for
search it really is not okay if it was
good for search at earlier solvers could
do it you know they say you can take an
earlier so for one of them current
hardware you be able to do it that's not
the case at all in fact as I mentioned
specialized programs cannot find to
prove so why is that you know the brute
force proof you can actually look at the
very basics s liability so over that
didn't have all the smarts building that
we have now would get a search tree that
essentially is of the size of all
possible input sequences because somehow
i have a very basic reasoner has to has
to consider them all okay so that's
intended a about 10 22 349 different
steps was a dumps over okay um the
current solver you know finds a
completely you can actually look at the
trays at the end it prints it out and
you'll find it it makes about 10 22 10
little entrance steps so that's a
savings of a factor 10 to the 339 so so
I'd find you know a you a miniscule size
proof compared to sort of brute force
search
and it's also able to find the proof you
know it's not just that that the proof
is much shorter it has to find it among
you know this search space okay so
that's the difference so it's not really
it's actually has smarts built in and
you know we hope that they will just get
better and better so what you know what
was the follow up on this yeah this was
2015 got a lot of publicity because of
the polymath project the first time in
this air dust problem was open for about
40 plus years the first time any
progress was made you know so it's sort
of strange you know intuitively you
might say yes sure that you can't you
can't contain those sequences but but
nobody is able to prove even for
discovery too but after this was proved
and there was some other advanced I
should mention that there were some
other very civil advantage English
English property of the citizens in
terms towel then proved in the general
case meaning for any finite discrepancy
if your secrets gets long enough your
sums cannot stay within that finite
discrepancy so this was a mathematical
to the force deep instead of mass but I
do you know and actually we go to tennis
original paper he did borrow a bit from
some of the intuition that came out of
his computer proofs about you know slim
proposition of these sequences so so
there was some sort of carry over and he
knew that there was true i guess that
helped for the case of too i would even
tell you to step further it doesn't
really this general shops across a limit
resort result so in the limit so telling
out what they do in this protein
consider hugely you know very long cigs
and look look what happens on n goes to
infinity and that's how you get these
results it doesn't actually you know
Tara Stiles result does not imply you
know 1161 is infeasible because his
limits are much larger numbers and so so
there's there's something coming out of
this proof that still is not actually
replicated in a human way and this is an
interesting idea that
that future mass no mass of course you
know just like any human endeavor it
sort of it goes to what we can do okay
it doesn't explore things that we cannot
do but now you get mathematician
complemented by this machine reasoning
systems and they will give us now fact
that there are two and verifiable by the
machine and now we can use it as a
mathematician so there's sort of an
exciting time where we will see mass so
we're starting to explore things and
starting to use factors ok this is
computer-generated proof and verified
verifiable I'm going to build on that ok
and and I'm sort of hoping that we're
going to see some interesting
developments dear ah okay just so I'm
actually almost done so so how about
giving you some reason that this is sort
of it's a different kind of a world the
reasoning world but my hope is that
we're going to go to merge and in fact
all four go is a bit of an example of
that so our go you know this really is a
break true but as as an inference or
reasoning based first I was very aware
of this step Monte Carlo tree search
which hampton 2006 and and if you looked
at you know how chess was done checkers
was no no min and Max searches very
basic game tree search and that doesn't
work for go because you don't have a
good evaluation function so in 2006 the
UCT which is a very clever kind of the
call it research algorithm got around
this form of not knowing you know where
the board is good or bad by doing by
taking a board and then doing a random
play out so you make most players play
randomly and see who wins ok it's very
counterintuitive in the sense that the l
if you randomly play you get very weird
games that have nothing to do with
actual games so when the losses in the
end you might say how can it be any good
and that sort of my action minor issues
nobody had tried this before because it
seems too weird to you
I ok so what's the intuition why can it
work well so I give you a board let's
say from the grandmaster game and I ask
you who's winning you know and I can
take two very bad players and have them
play and I've repeated a few times and
and if you know which side wins the
majority of the games that site will
have a little advantage even though
these players are all far weaker than
Grandmaster players so there is
something in that bored already that no
matter how you play you you know as long
as most players are equally week that's
that's the intuition so now random is
about as weak as you can get so as long
as well as we ran them you get some
information and that's what what was
found in this multicart research if you
get a little bit of information there
and now you combine it with search when
the search is looks very similar to
minimax search over few tweaks ok so
this was the real breakthrough 2006-12
should be careful it's real but it was a
breakthrough and there that boosted the
Big O level up to I think they're no
middle amid local amateur and and
suddenly people said no maybe go is
solvable okay that's that's up till then
people so it goes not so we'll go with
silver now becomes a timeline question
how long will it take and so then
alphago game in and and sort of add it
add a reinforcement learning deep
learning basically improved the play
outs strategy so there's a beautiful
piece of work but combined it with the
multicolored research if you take them
up colors research out you're not going
to have a good go playing program so
it's really the synthesis of these
techniques that makes things go and
that's I think we're so the future will
why where we start combining things that
we knew I'll do in the past with deep
learning and possibly the reinforcement
learning so these are just some general
comments against any we can do you see
an exciting area for me and so it also
this is this plan synthesis which is
like program synthesis so where we have
programs that generate programs are and
they will synthesize programs for us are
like a robot that since I plans on the
fly and it's a very different paradigm
than my programmers are used to programs
are used to pre-programming industrial
robots from the interesting story but
Sony nothing to yo now I guess who at
some point this was around 2010 actually
reduced the use of robots in their
factories ok the reason they reduced it
it's because they found real
reprogramming the robots for new models
was actually too expensive so so the
roads of the emotions are very carefully
pre-programmed so they realize actually
the software development was too
expensive and it went back to more
humans which are a little bit more
flexible but you know in doing what's
happening now is humans euros will start
using more general planning algorithms
you don't have to reprogram them every
time and they will be back in action so
this is just to give you something think
about so this is our you know how
computer scientists think about you know
computational complexity this is very
nice class here the polynomial time
soulful promise but she'll in your time
reasons sup linear time problems are all
in there you know sort of sort of spice
these are things that are really used to
mostly in your programming and but then
there's this next level up and we used
to avoid those because these are MP
complete we used to sort of avoid those
because they are in worst case is
exponential but as I've shown you for
the solvers the SAT solvers this almost
feels like in practice it collapses into
p ok I cellco I won't show NP
completeness you don't have to worry
about that much anymore
unless you're into cryptography so
cryptography works specifically to
create very hard to be complete problems
that are that are averaged k simply
complete in some sense so so they're
they're still have to exploit so the
theory is still all working it's just
whether the typical case is that hard or
not so we feel it's much more closer to
peak for a typical case but now look at
all what's involve it okay so there is
counting which is much more related to
public inference there is a game playing
planning with up for cereals multi-agent
system p space and then exponential i
guess what we know for sure is that p is
not the same as XP is expiration but
this could be principally collapse down
although we don't believe it will happen
so but if i had to put you know so this
is gets a little controversial yeah so i
think here humans a idea but not
machines are trying to tackle this now
how I machines are trying to take on
this de Stefano everyone probably gave a
talk last Monday taking things like
counting and sampling and being very
clever and basically adding some for the
clauses to it in some randomized way
collapsing this problem down to an empty
complete problem and then calling assets
over or mixed integer programming
package so so it's quite a bit of work
now where they take problems and map
them down there's there's something you
lose you lose optimality guarantees but
you get good approximation guarantees so
you may not be able to count exactly by
doing a trick but you can count
approximately by doing that trick so
they're taking problems from here and
mapping them down so there's a lot of
excitement it sort of that that you see
that happen and a lot of it is possible
because a million variable problem 10
milli variable problem those are
solvable now
if that wasn't doable enough of this
London's would be interesting so there's
a real shift to towards our Megan doing
given the push from the verification
during the hardware industry software
industry I think he solves go to 10
million hundred million variables and in
a billion classes for example and then
you can do a whole lot of stuff here by
mapping it down okay and this is then
haha so my last comment here you know
what are the consequences for you and
understanding of machine intelligence
you know there are some real questions
you know to what extent can we
understand what the machines do and she
can even give such a good scenario is
where we have the proof and you can
write a little program and check it and
hundreds of thousand people can just
check for themselves and say okay I
don't quite understand to prove here but
it's correct okay there's a slightly
more difficult area which you see in the
game playing program like go and even so
my chess where there is no short proof
so there is no short proof of why a
certainly just move is good it's just
that deep blue or alpha go in for go
I'll go gives you a big rest the best
move it can find and say okay trust me
you know this is a great move there's no
explanation for it so there's a whole
area that worries about the question how
can you verify something how can you
give what's called a witness for certain
results and and I gave an example where
there is a short witness tears who we
said eight gigabyte is not that sure but
it's shorting us okay there's a witness
for these go playing programs or the
chess programs there is no witness so
all you can do is well I wrote my own
chess program my own over go program I
get the same wolf prediction there's no
no other way to verify the computation
and that's a release this issue of you
know how we're going to trust these
programs that they really are doing the
right mover is alpha girl trying to
trick me into doing something very dumb
but I will not know it okay so there's
sort of
we have to develop a notion of trust or
have other programs from watch the
program's making their moves but it's
not so easy I think the example I gave
is the verifiable proof is a very nice
case of where it is checkable but in
general it's not so ok nothing guts if
actually oh yeah thank you so let's see
a couple questions I'm puzzled by so the
distinction between go and chess in this
icing your bookcase is talking about if
you have liked larger boards larger
instances of databases why didn't you
change has and go though oh actually
show is that on this slide yeah so yeah
so I think um it's a little bit how its
generalized I guess I think I did chest
bounded number of moves here we have a
bounded number of moves if you allow
potential exponential mundum of moves
and goes up and not in complexity if you
would do bounded go and would go one so
it can I win in 50 moves or 100 moves it
goes down to pee space so there's no
both both of these results are I always
questioned well in the sense that you
know we have to make some generalization
about the problem sighs and it's
somewhat artificial and you're much more
interested in what is the extra
complexity of a BHS or 19 x 19 go the
problem is complexity or you will tell
you it's constant time sir so that's a
pretty bad answer so we generalize them
an answer that is there it comes from
how how we've generalized it no other
questions or ever sir
I think this is the most optimistic
chart I have seen you drink we're still
here oh so I can like my channel the
spirit of Tom took channels for the head
I and so I think there's like a certain
type of route for saying that machines
can do to do you know solve quotes all
tango and maybe our problems that he was
going to solid foods will be done at
this all in practice hmm and there's
also this thing where like I Cubans are
able to sort of like find the light find
a bit clever like way to cut a search
base down dramatically yeah and it's
this modern thing that seems like what
you really need to do want to say
machines consult NP hard problem
wondering how that ties into jira sleep
yeah i know but that's a very good point
so but that's exactly what i was
actually what I was trying to say about
these soldiers they they find these
shortcuts you know that's that's we've
actually done a lot of work ourselves on
them sort of trying to explain how these
souls can find these very large do is
you with millions of variables and and
they fi it's called Clause learning to
actually have a component that tries to
that learns new new implications while
it's searching adds it to the problem
you know you have a problem is elizabeth
million closet they can add 10 20 30
million implication it finds you in the
search and it's sort of clever about
which once it keeps around and gets
around ones that are most useful for
future part of the search so we found
these Souls actually is sometimes
memories like when if human can solve it
this clever reasoning this the soldier
will rediscover that and that's actually
showed the secret of it so so yeah so
it's not a brute forcing it's really
finding you know I got call its you know
mental shortcuts you know shortcuts in
the proof but the solvers it'll find us
too so so it may not be such a so the
question is like guy humans still better
they know one area where humans are
currently still outperforming like a
mathematical reasoning area machines is
coming up as abstractions so certain
kinds of your mathematical abstraction
the Newton's are so good at we're still
very bad at a machine reasoning but I
could see that change you know that that
we're now able to do things that you
know that we could maybe you in
principle ten years ago but we never
could do in practice more and more you
see see machines that's a shuttle would
for abstractions for example and apply
those to the proofs so yeah do you think
that there and peace problems are should
be cut into 2 into like those that you
are really key in like the majority of
product probabilistically of cases no
notes that are not like Cleo versus I
connector versus Etta may say I think so
this and and really today's it's the MP
complete ones that are up here yeah that
has to be I think there's a very small
corner here which is in which I guess
you know there was actually some were
called on average NP completeness
average case NP completeness but also by
done by levan one of the cobra discovers
NP completeness who exactly looks at
that notion and says I wanted notion
about the police said I hate generate a
random instance and it's still hard and
you identified it i guess it's problem
having it was parody kind of constraints
it's a sort of you can order lattice
based problems a lattice based problem
it's the most difficult paper to read
nobody can can read it sort of like half
a page long as a russian style so so
that's why most people missed it i guess
it can't understand it but but and it
has now slowly people trying to make
sense with but it's a very small corner
here and that's I think the kind of
thing you're pointing it so I could see
the whole thing collapse down in
practicality except for that little
cryptographic corner and that's actually
the interesting thing because that thing
out d what we're finding is yes I said
you cells are very powerful but
sometimes they can't do it at all so
they can lead to certain frustration and
that's because sometimes in your own
County you might accidentally create a
little cryptographic sub problem they
just over can't handle and that's when
they can't do it so it's something we
need too much I'm better in your head do
you think the part that is elapsing is
like MP intersects going pee or anything
that is a different for me yeah I mean
that but yeah that that I could see
colossally when she collapsed in
practical terms she impractical turn yes
yes yeah so the question again could be
some cryptographic subsets that is
average case hard could be in there not
not even sure anybody has formulated
that it suspicious by the fact that like
the proof was also short in the problems
are solved yeah yeah yeah yeah I mean it
goes hand in hand against a probably if
the post weren't short that you know
that's the thing so when we looked at
how can you set sofas to do with it you
know the part of the reaction people
have is outdated when it's promo
satisfiable it is lucky to find an
assignment in hundred and that's just
not the case you know if you go slightly
again we and in a few more constraints
are no solutions they find short proofs
but it does go hand in hand they have to
be short proves into space otherwise if
I was not doable and in fact a few
hardness results that are known look for
random three shots where we have no way
of dealing with it when the pumps are
satisfied there as a prove that those
proved there to prove that those proofs
are long are too long explains you want
so sudden OSHA short proves and easiness
and doable all come together in which is
good okay all right okay thanks


AI Safety Reading Group (Session 44)

so hello and welcome to the 44th session
of the AI safety reading group today we
are going to have a look at the ATI
safety solutions map made by Alexei
Turchin elected Turchin here is an
author from Russia who works at the
science for longer life foundation which
is also founded and he has made an a
summary or some kind of illustration of
an article by kaiser talon romanian
polski called responses to catastrophic
AGI risk a survey and AGI is artificial
general intelligence that is a capable
of optimizing of working through many
many different domains and the risks of
course are that that this will have some
strong negative side effects not because
the the AGI is doing is making accidents
but because it's competently following a
an agenda that is opposed to humanity so
Kai's Italian Romanian polski looks at
the solutions that are given and divides
him into three main categories as social
constraints external constraints and
internal constraints but Alexa Turchin
he builds on this and makes three new
categories AI is used to create a safe
ai and multi-level solutions and finally
met a level and you further tries to
sort the solutions according to how
simple or how complex they are he also
incorporates some ideas that are there
are newer than this the article is from
2013 so it's also a kind of update in
particular here editing by penguins
through Armstrong and Paul Chryst
channel so this this is structured as a
map which is a very non-traditional and
what's the point of using a map in
general a map is a way to get an
overview over some kind of unfamiliar
territory so if you are new to AGI
safety then looking at a map gives a
reasonable idea about what solutions are
are there roughly and met but maps can
be used more more activity as a way to
find solution because normally when
people look at solutions they find them
in a rather ad hoc way where they have
some parts they focus about they have
their ideas and they sometimes they
duplicate their ideas something they
reinvent the wheel but a map can be used
to discover new solutions if you have
some category and a level of complexity
and you can see there's a cell here no
one has actually tried to find a simple
solution in this category and then you
can try to come up with something in
particular exit surgeon has stated that
a number of these points in this map and
these solutions are actually knew and
solutions that he came up with while he
was making the map let's go to the map
itself here you can see structured
roughly with six columns and I was
talking about and sorted with the
simplest solution on top and growing in
complexity I'm going to zoom in a bit
because I don't believe you can see this
at all so whilst I start with the
simplest societal proposal and the way
I'm going to go through this is I'm
going to show them very very briefly of
course it's a map and I can't go through
everything but I'm going to jump into
parts that I think are particularly
important and one of the things that I
think is particularly important if the
simplest societal proposal which is to
do nothing and this might be the best
solution under five different criteria
it might be that it's for some reason
impossible to build at general
artificial intelligence it might be that
it's too far into the future if it takes
a thousand years of research then it's
probably a bad idea to try to do
anything now it might be that the that
it's actually not dangerous at all it
might be that some people would prefer
to be replayed that it's a better moral
solution that the AGI replaces us so we
should just let it kill kill us and it
might be that in our attempts to solve
the control problem we might do things
that are more dangerous that are
actively harmful so there might be a
number of reasons why doing nothing
might be the correct solution there are
more solutions the one that its most
mainstream people are thinking about is
to figure out how to integrate the AG
ice into society people are talking a
lot about regulating society in
regulating research sorry and in
particular the bathroom has talked a lot
about the differential technological
progress figuring out what technologies
to research first there are the options
making humans smarter relinquishing
technology going back to the Stone Age
it might be a solution it doesn't sound
like a feasible solution there are a
number of smaller improvements you could
do that's the only I solution where I
give AI to everyone it might be that's a
technical solution that is impossible to
figure out yet because we are too far
from a TTI right now that might be
possible to have to distribute
intelligence in some good way or even
create some kind of AI police so that's
what we can do when we look for the
society if we try to make external
constraints on the GI the toughest but
also simplest might be to put it into a
box a GI confinement this is something a
solution that
has been discussed very much and the
consensus is probably that it's a
marginal solution it's probably not
going to be to work but my google was a
try it might be possible to have a
bigger box where there's a control room
and a gatekeeper which is kind of an
inverse box which is the only thing that
the AI cannot enter that's the
assimilation in certainty argument or
philosophical constraints which also
something that might work but might not
work very very well more promising is
the idea of preventing self-improvement
if we can prevent the API from rewriting
its own code in some way or putting some
kind of bound on it in different ways
this would could could help a lot of
solving the cultural problem and
reducing risk from general am we can
check the goals this is also a proud
also something that will help a lot in
particular there might be a moment where
the AI realizes that it should hide
things from us and it should and in this
very moment where sighs do this it is
not yet hidden things and that means
that in this very moment it might be
possible to stop it and focusing on this
moment might be a good solution there
are logical of philosophical Zen minds
some of them are really really technical
there's something called the logan the
obstacle that an api might need to solve
and that's very very complicated from a
mathematical point of view i make no
pretense adams understand that at all
but it it seems important than
mathematics is just too hard for me to
really explain exactly what the libyan
obstacle is there are other ways to
constrain the AIX term you can record
what it thinks you can make some
constraints about the signs
and you can constraints knowledge oh you
thousand some other external constraints
you could figure out if you want to exit
touching gets creative and things like
electricity so the internal constraints
are with the AI not preventing it from
doing bad things but making it not want
to do bad things the simplest is what is
called an Oracle AI and AI that simply
answers our questions and doesn't
attempt to influence the world in any
way and we might have short rulesets
like The Three Laws of azimuth we might
have a final way for it to to learn our
values this is something that has been
researched a lot and is generally
considered a very promising Avenue there
are other ways of one of the ones that
look really hard probably a good
solution is the formal verification if
we could explicitly prove that nai is
friendly then that would help us a lot
but it might be possible to edit its
motivation only and only from
non-natives motivation still pushing it
strongly towards being human aligned
there's a constitution that some
different some kind of moral invariant
will push onto the AI that might also
work you might merging it again so the
AI want to merge with humans sorts and
the next step in human evolution that's
decision theory that could cause it to
be friendly avoiding super goals or
stopping it from understanding its own
architecture for understanding itself
really giving it an open goal system
where the course can be changed oh
putting constraints and what kind of
children it can make of course a
computer program that makes children
means that it runs other computer
programs and possibly variants of itself
just more improved and that's what the
fourth column is where the AI is
building other AIS in this particular
we're thinking about having an AI that
is possibly unsafe creating an AI that
is safe so that's the standard problem
here and it's possible we can make it
some a causal decision can you with it
and that's also somewhat far far fetched
we might in either enforce it where we
have some some strong AI that prevents
unfriendly a is from coming into being
we might have a a morn any age I that
doesn't enforce so much as prevent only
the worst cases and we have on the other
end of the spectrum the Korean extra
brain evolution by a esa eight cows key
that is generally not considered a
solution to the AGI safety because it's
hard to implement if it is implemented
it would constitute a solution to this
problem but is generally considered too
hard to to to be a good solution it's
not something you want to do that the
first time and of course the solution to
a I risks things you want right from the
beginning other ways to have the AI to
have any instead of having one AI
building safe AGI then you could have a
democracy or a jury system took the
influence the creation of other AIS you
could have something I won't go into
Alba
qualia this is things like love or
empathy and feelings that if you build
this into the AI and this it's not clear
to me exactly why this is a into the AI
building AI and but this is how L exit
urgent has has created it mmm so I think
it is actually possible depending on how
you look at this that these these lines
here means that all this are related to
a is creating a is and these down here
are just random ideas because here we
have something like AI cartons and
avoiding harm and making puzzles and all
these kind of things that are very
unlikely so that might be an interesting
idea but these are not really something
that has been worked very much in
details things like testing the AI a lot
and hear some general problems about
creating a I HD is safe second the last
part is the integrated solutions this I
this is something here we have an
example and military in any AI where we
have a an AI that might not be a general
AI pursue men is good at one thing and
that is military conquest and then it
conquers everything and then it builds
in any I that is safe because once it
has conquered the world it can solve the
coordination problem at a very large
scale and it's possible to have a mutual
level 80 I with an unchangeable call in
the middle in innermost and the new
building layers outside and work for
this kind of architecture in the last
meter these are not solutions but
descriptions or characteristics that and
solution must have and there's a number
of virologist joint ideas in particular
the international AI agency an idea for
may be there through the United Nations
or something like that co-ordinating so
everybody is trying to create a positive
AGI instead of engaging into an arms
race so then this is the roadmap the
safety solutions map that Alexa Turgeon
has created so thank you for watching
see you next week


Jim Babcock – The AGI Containment Problem – CSRBAI 2016

it was been a serial entrepreneur in
fields of programming machine learning
and medicine and has volunteered with
FLI Jim is here to talk today about
problems in computer security
that we might want to focus on in
advance in order to build safe testing
environments for artificial intelligence
is that in reason more generally as
quite possibly relevant to to Paul's
talk as well so let's so join me in
welcoming Jim Babcock today talked about
capability decomposition which is where
because it may be difficult to make a
sort of unified reinforcement learning
agents that has the properties we want
but we build some sort of structure out
of a sub agents like
and these sub agents do things like some
of them communicate with each other some
of them control each other's reward
channel and then at the end there's an
output that goes to the environment and
the environment gives back observations
but you could imagine setting up
something like this and one of the sub
agents notices let's say this one
notices that this other sub agent over
here has some information that would
really help it to get a higher reward so
it tries to find a communication site
channel to talk to this one and it's
this one also notices that this one has
some information we want and they trade
information and suddenly are not
following the structure that the
developers intended and it becomes very
difficult to reason about another thing
you might have that could happen is say
this agent sends this agent a malformed
observation that changes its behavior to
be something different for example if
this one is giving the words back to
this one it might send a message that
causes it to always give maximum reward
because that would be really great what
I mean by AGI containments what I'm
going to be talking about is you're
setting up a computational environment
for all of these sub agents to to
interact in how do you make it so that
that setup enforces the constraints and
the structure that you intended your
system to have so that agents aren't
communicating unless you who actually
wrote function calls from one to the
other and they aren't randomly giving
each other buffer overflow
crazy stuff like that the extreme case
of this going wrong would be if basic
one sub agent takes over the
computational environment entirely has a
network connection and starts
interacting with the Internet where
there's all sorts of trouble that it
could get up to but that's what one of
the easier failure modes to defend
against the Beast if you're testing
because you can just unplug cables but
when you have parts that are supposed to
be interacting you can't put an air gap
between them today a eyes are not smart
enough to discover some computer
security vulnerabilities in general
there's sort of borderline cases that
you might call a narrow AI but would be
pushing the definition a little bit but
decades down the line this may be an
issue these specific vulnerabilities
that exist today
probably aren't going to exist decades
down the line so I'm gonna sort of I'm
going to present some examples to give a
feel for what sort of computer security
vulnerabilities are possible but what
I'd rather you be thinking about is the
shape of the computational environments
and how the pieces interact and what
that means for what things need to be
hardened and how to put those pieces
together so an example set up a little
more concrete than the thing I'd run the
white board suppose you have an AI that
plays Mario so sends buttons one way it
gets back pixels and then at the end of
your test
Mario spits out a performance report
which goes to the developer it turns out
that the Super Nintendo game Super Mario
World has a buffer
overflow human human players discovered
in in in this order ways to make the
game literally produce game objects a
way to make the game literally jump to
the end of game you win screen and then
a way to make it load of program via the
controller port and start playing snake
that required very precise input that
the controller part was not connected to
a real controller with human hands but
just in the last month someone did
manage to do it without without machine
aid and made super mario world
transformed into flappy mario a flappy
bird clone and during the entire program
and triggering the the memory corruption
by hand and report ends i think live
streamed it on youtube so people could
see that he got it on the first try
so in setups like this
the andhra thesis about conversion
instrumental goals applies you have some
agents most setups will give them
certain incentives if they have the goal
that's they're able to optimize them
they prefer that aged guys that are
similar to them come to exist because
those will optimize similar goals so for
example if the if the AGI could output
either everything is great or there is a
catastrophic error if it outputs there
is a catastrophic error then this will
cause aged guys like it to not come to
exist in
world because developers won't try that
again but if there's two aging eyes with
different incentives then it can then it
can report catastrophic errors only so
long as they are maintaining those
separate utility functions and all of
these instrumental goals it helps to
find security vulnerabilities that let
it break constraints that the
environment is trying to impose on it so
what do I mean by an API container what
I mean is it enforces the structure that
you thought you had and makes sure that
the company that's the computation
really has that structure so only runs
the code that you want in the places
that you want think things don't have
unexpected and put our output channels
the log files or logging what really
happened and not what the AI would want
to be in the log files tripwires work
you can reproduce it you can reset it
and not find out later that it
reprogrammed the hardware on you and you
can do and if you have a data center of
these things you can do several tests in
parallel and they are properly isolated
so why how would it be able to run code
that you didn't want or expect well you
might give it permission to self modify
but assuming you didn't do that
it might output an image and then some
other part of it reads that image or the
developer reads that image on their
workstation and image formats are
complicated and the PNG occasionally
tests security vulnerabilities
discovered in it which will execute
arbitrary code
whatever computer opens that image an AI
that is modular
if one module sends messages to another
that are not formed then that could
trigger a buffer overflow it could take
over that other module or maybe it
breaks more of the constraints than that
one type of output that's a that's
particularly a problem text is not just
text I don't know how many of you have
worked with this particular aspect of
UNIX and Linux before but text is a
bunch of printable characters and
occasionally a command that says
something like move the cursor to the
left of the line or clear everything
below the cursor and this protocol goes
back to the 70s and is really really
messy the things that interpreted have
sometimes have buffer overflows one in
one example in X term you can send you
can send an escape sequence that says
you can send an escape sequence that
says sent the window title so that when
you vault tab you know what's in that
window and then you can send it up
another escape sequence that says echo
back what the window title is so and
what a program can do is set the window
title to commands newline send a echo
back to the window title sequence and
then quit and then the window title
sequence gets typed into this shell that
ran that program
there's also hardware problems there's a
know fairly famous issue where if you if
a program accesses memory in a
particular pattern its causes something
something like an electrical fault in
RAM chips that will randomly flip bits
and security researchers found that they
can exert some control over which bits
get flipped and caused permission bits
to get flipped in memory page tables
it's kind of ridiculous and that's now
basically fixed on most computers by
tweaking some electrical parameters of
how how often the RAM gets refreshed but
there are probably more problems like
this waiting to be found
so should only execute the code you
think it's running should not get
information from places you don't think
it's gonna get it or send information
where it shouldn't another famous name
security vulnerability is called was
called GS mem and the way that works is
someone puts a reprogrammed cellphone on
top of a computer's case and a program
running on that computer accesses memory
in a way that generates radio frequency
interference and sends a message to the
phone so militaries and organizations
that like to be careful with their
secrets well put glue in all of the USB
ports and not provide any network cables
or anything so obviously you couldn't
get any information out except by
laborious ly typing it into that phone
by hand right if a program has access to
a clock then and the same computer is
also doing a EES encryption operations
repeatedly then by measuring time
measuring cache timings the program can
extract information and invokes this a
secret key you began to talk with saying
what abilities aren't not so
we move this back to where it was before
Botox so it seems kind of like magic if
you don't know what the space is
possibility is that these things are
working with is but if you if you draw a
diagram of all of the parts in the CPU
you can see yeah these two processors
are sharing an l2 cache and that means
that they will occasionally affect each
other's timing and if you are really
really careful about it you might try to
prove that they don't which would
probably mean just putting them on
separate CPUs because that's easier log
files debug information if it's
different so I'm envisioning this as an
ATI has some print statements and it's
that occasionally print out details
about what it's doing that the
developers will look at later if it's
running the code you think it is this is
mostly fine and it's not running the
code you think it is then that could be
anything including weird X term escape
sequences
so and as far as debugging there's two
general categories of debuggers that
exists there's the usual kind of
debugger which assumes that you're
debugging a program that you wrote you
have in source code and it isn't
malicious if you attach norm your normal
python debugger or VBB to a malicious
program then it might take over the
debugger this isn't a failure mode that
people are particularly worried about
because people don't use debuggers in
that sort of circumstance but there's a
different kind of debugger which is
intended for debugging malware but which
generally assumes that you're working
with the assembly because my work
doesn't give source code so one thing
that one thing that I think people
should be doing is making a middle
category of debuggers that is hardened
but designed for cases where you really
do have the source code and have at
least presumptive assumptions about what
you're looking at tripwires
so so one example of a tripwire is this
sort of stylized assert statement to
certain not taking over the world we
can't really design things like that
today because we don't know what the
algorithms are going to be Paul had some
ideas about guaranteeing
non-catastrophic execution that depends
on a particular setup but what I
mentioned what I'm more interested in
which we can work on now is
technical requires that asserts things
like this data structure is not corrupt
there aren't pointers pointing to weird
places or point pointers that our
knowledge shouldn't be or cycles and
your data structures things like that
the thing that is kind of tricky about
that is if an AGI is reflective it can
look at this Venn diagram and try to aim
at this slice so an interesting software
architecture problem is how do you how
do you allow introspection while denying
access to that information have a
program have a program that can look at
everything about its source code except
for where the assert statements are and
what they say over time so
reproducibility
is it's a bunch of issues if you write
it at you if you write a fully
deterministic program then in principle
it's reproducible as long as you by the
time of year going investigating what
went wrong you still haven't still have
the pseudo-random see you still have all
the inputs you haven't lost track of
things
so some programs look deterministic but
aren't because because of race
conditions for example you so you write
a program that's multi-threaded and it
has slightly different behavior based on
which of two things finished first
purely from pure functional languages
like Haskell I think let you get
guarantees of this not happening
something like Python that won't happen
so there's a programming languages
research question which is how to get
guarantees of gnome race conditions
secure resets so a you know dawg i heard
you like computers so i put ten
computers inside your computer hey Herc
each component of a computer typically
is another computer that you don't
really think about because all it's
doing is managing the interaction
between itself and the main computer but
security researchers do think about
these things and sometimes find ways to
reprogram them to make it so that when
they take over a computer it is person
it is persistent so that
if the user reinstalls the operating
system they're still not where they're
so so jellyfish is a bit of malware that
runs on GPUs that is a research project
on github it lives in memory so if the
computer is powered down and turned back
on it will go away but if you just hit
the reset switch it'll stay the NSA had
a catalogue of all of their computer
security tricks leaks a few years ago
and it contains lots of things like for
example irate monk which programs not
not the hard disk contents but the
controller that decides where on the
hard disk to read next to order reads
and writes for good performance and
makes it so that if it's reading an
executable off the disk every thousands
time or something it will read a
different executable that contains their
code so that's the isolation and you
want you don't want you don't want
experiments communicating with each
other if they weren't supposed to you
don't want one tests to changing the
hardware and affecting all the
subsequent tests similar sorts of other
abilities or a problem so there's all
these technical issues and then there's
also humans so in the early days of
what's wrong please read Kowski ran what
he called the AI box experiment where
you would have an IRC chat lasting a few
hours with someone with someone
role-playing as a gatekeeper
try to convince them to let him out of
the box and he succeed he he didn't
succeed every time but he didn't succeed
and the super intelligence is probably
smarter than any humans so it should be
able to do this as well and so other
people reproduced it's tuck stitch was
another user who ran the Eibach's
experiments and succeeded at them but
you can't persuade a human to do
something without actually being able to
talk to them so there's a policy
question which is the if the AGI outputs
things
what outputs are okay to read and who is
it okay to read them and that depends
how smart do you think it is on how much
you trust the specific you people
involved what that is for and how you
think it's optimized the containers job
is to enforce that policy to make sure
that a developer can read log files
while a passerby can't can't receive
text messages telling them that it would
be fun to plug in extra network cables
in me so if you're really really really
paranoid then you can use some fairly
extreme strategies like have the age
guys output only be the answer to one
multiple-choice question and a B C or D
none of these answers is going to trick
me into pasting the AIS source code onto
a different server there's just not
enough information there a similar thing
which can solve problems like rope
hammer with high reliability is
homomorphic encryption which transforms
a computation into something that
as much floor but doesn't interact with
the hardware in any way that's
distinguishable from randomness without
access to a particular geographical key
so and the paper about this we define
sort of define a hierarchy of degrees of
caution essentially from light
containments for well we don't really
have been working yeah yeah but we'll
use some software as long as it doesn't
get in the way sort of future I think of
that as if we can write software for
containment and some of some measures
will just be dropped in doesn't cause it
doesn't cause any problems for the
developers just makes things silently
more reliable instead of all of those
things and then there's some more things
you could do to make it more reliable
that slow things down or make the
developers do extra work and those are
medium containment and there are some
things that slow things down
lock and make and less than the escape
ability and I think we called those
heavy containment so that's an overview
of the problem
not the end of the talk I'm going to
tell you about off-the-shelf
technologies that are partial solutions
and say that we need more research to
make these better yes kind of yeah you
have the slides angle your invariances
we had this listen variances yes I'm
wondering like how how do we generate a
list of these and variances such that
like I'm sure that I haven't I don't
think the list is complete I mean I'm
sure that there are more things and
specific projects will have specific
invariants as well about like their data
structures amount of computation that
pieces used and I'm sure many other
things that I haven't really thought
about the strategy that I use to
generate that list was thinking about
what properties if they fails to hold
would break everything else so that you
couldn't possibly have any other
invariants besides those
other questions before I go on so here's
sort of an overall picture so
conceptually you have an AGI program and
it's written in some programming
language and that programming language
is a sort of sandbox a few programming
languages actually call it that like the
Java sandbox the basic premise is well
if you if you never mention the
functions that make network connections
then clearly this program is not going
to make network connections you hope I
mentioned a few loopholes and that sort
of thing but it's a partial solution
it's clearly much harder for it's
clearly a significant obstacle to break
out of the language model for a program
if it's going to corrupt its own memory
it's very likely to crash itself instead
of take instead of doing the particular
weird thing it wanted then there's the
operating system privilege model where
when you run a program you can say this
program is only allowed to access these
particular things and nothing else
Linux has a version of this BSD has
another version of this windows has
another version of this the defaults are
pretty permissive but if you put in the
effort to say exactly what it's supposed
to be able to access then the operating
system will enforce some things for you
but the Linux kernel occasionally has
security vulnerabilities as well
so you've got that and then also you
have a virtual machine a virtual machine
is like an operating system inside of an
operating system where you have clinic
you have the Linux kernel and then that
runs a program which emulates another
computer using the similar hardware so
that it's fast it's fast emulation then
that computer has another kernel and
then that runs a program that is the AGI
and then the fourth layer on top of that
is you keep track of what wires are
connected to what and the idea is that
these are conceptually dissimilar enough
that's if you find the security
vulnerability in one that doesn't really
help you very much with finding security
vulnerabilities and the other ones so
this diagram say the AV guy starts down
here it if this has permission to self
modify then it can follow this arrow and
get up to here and if it finds a
vulnerability and entity bugger then it
can get up to here and if it finds a bug
and it's the driver that controls the
GPU then it can get up to here and and
so on and has to get all the way up to
here before it can actually interact
with the internet and as it as it goes
up it properties that you would want to
have stopped holding and the idea is you
build a multi-layer thing like this and
then each layer there is some monitoring
that looks at the low V inner layers and
checks that they are only running the
code
they're supposed to be running so
language semantics I talk to I just
talked about that on a skit there's a
scale running from C to Haskell where C
is almost impossible to write correct
programs and and errors lead to amazing
amazing League comes complete control
ability to Haskell which has a very
thorough script type system which
doesn't really have this sort of problem
the problem is that a program written in
Haskell is still dependent on a bunch of
software libraries for doing things like
interact with the GPU and bio and
loading data and those might be written
in C with problems in them so next layer
operating system sandboxing there's at
least a couple dozen of these one in
particular just to give you a sense of
how they work
Linux has a mechanism called SEC come
where the program starts running it
initializes it opens all of the files of
plans to use and then it says from this
moment forth I will never open another
file I will never make another system
system call accepts to interact with
this set of files that I had already
have open and if I ever break this
promise then send me sick kill
that's the default and the more refined
version that Linux also supports is to
say all of the system calls I will make
from now on if you feed them into this
short little program will return what
the program is called the Berkeley
packet filter it was originally made
designed for Network firewalls and got
repurposed for this virtual machines
there's about one security vulnerability
per major virtual machine product for a
year that would allow breaking out of
the virtual machine sandbox hopefully a
decade from now they'll all be found and
there won't there won't be any left we
can hope maybe we can speed that up we
can hope a virtual machine virtual
machines have a fairly complicated
surface area because they're not just
providing a CPU for programs to run on
they're also providing a virtualized
video card if you want to turn an avi it
probably does have to so GPUs are these
incredibly complicated things and they
have these drivers that essentially take
code that you wrote for the GPU and
compile it and in a virtual machine that
driver lives on the host system instead
of on the guest
book both nvidia and AMD have projects
in the works to try to change this but
they're not done yet for now this this
is what you get if you use a GPU this
this is a quote I found out on a random
internet forum from a former developer
at Nvidia about how complicated GPU
drivers are and apparently GPU vendors
and video game companies have been
working around each other's bugs for
more than a decade and the result is
that they are extraordinarily
complicated they're also full of they're
full of bugs they're written in C++ and
they run with very strong privileges
so this is a problem google recently
announced their tensor processing units
which sound kind of like their non
backwards compatible machine learning
only GPUs and I think that's great
because it means they don't have to use
this million line driver codebase to run
their a eyes so network topology you can
you can air-gap your system this is what
people use used to think about when
talking about AI boxing you just put it
in a concrete concrete shell or
something and don't have any network
cables in and yeah that means that it's
not going to interact with the outside
world in any ways you didn't intend but
that's totally useless because you do
actually want to use if you want to test
them you want
tests parts of it to test each other and
also developers like to work in nice
offices not in isolated concrete bunkers
so what I expect to happen is that these
will be configured with routers that
limit what can talk to what which might
have Boehner abilities and them
hopefully not but no actual air gapping
so questions about any of this okay so I
have one more slide which is what what
can we do about today seems this thing
about about this is that it's a bunch of
work that calls for a completely
different skill set than the usual AI
safety work so different people can work
on it making a hardened debugger does
not require any knowledge at all about
machine learning and machine learning
does not require any knowledge at all
about the software architecture of
debuggers so this is something that can
be pretty costlessly pursued in parallel
not costlessly because of how society is
organized but in principle could be done
costlessly in parallel it's some it
would be very valuable if someone were
to set up red team events where there's
a set up that is that represents a an AI
container as it would normally as one
might expect to be organized and people
compete to break out of it or to break
its invariance and then report how they
did it this is how the world of computer
security has gotten successes elsewhere
for example every year there is a
contest to break out of the google
chrome sandbox for $60,000 I believe
that progress has been claimed every
year but every year the way that it's
done gets more impressive
so that's that's where we're at as far
as the the infrastructure that ICA is
being built on and changes that I think
need to be made to make them more secure
yeah a bunch of questions for but yeah
more questions okay in the in the
broader world of the internet where
people are doing running lots and lots
of different programs not necessarily
investing in all in security offense
pretty reliably wins over defense is it
has so far I don't expect that to change
in the context of an AI where the
developers can invest in making things
secure I think it could go either way
and there's reasonable prospects that
defense will win and even if it's not
possible to get rid of all the security
vulnerabilities just because there's a
vulnerability doesn't mean the AI is
going to be smart enough to find it so
I'm surprised you didn't mention formal
verification and then type theory as as
part of this because it seems like
projects like a a version of Linux that
is formally verified goes along the way
towards hardening not just the attack
vectors you know about but once you
don't yet know about yeah so I bought
ice i bundled that in language semantics
there's there's an isomorphism between
type-checking
and formal proofs they're not exactly
equally expressive but if you have a
sound type system like in Haskell for in
Haskell or thrust for example then there
is implied in that of proof that certain
kinds of errors aren't present we want
to go a step further to sound dependent
X's something like icon where you can
like express computations it would I
agree but I also think that's replacing
all the legacy C code with anything else
is a much bigger game than going from
reasonable programming languages to the
very best possible programming languages
yes related to speech question what are
you intuitions about bugs currently
being dense versus parson what is
possible to make by using different
architectures I'm familiar with that
look I've learned like them kind of
means that like in my drama debug salute
fix someone will be able to find so
there was a recent advance in bug
hunting techniques in the form of a
program called AFL fuzz which revealed
that the bugs were denser than everyone
had thought there were a lot of old
programs that had gotten a lot of
attention a lot of bug fixing and people
thought that they had probably gotten
everything or almost everything and then
they hooked it up to this really
sophisticated buzzer that generates
random inputs in a strategic sort of way
and it finds huge numbers of
vulnerabilities in them and that still
doesn't cover all of the types of bugs
in particular we know that it doesn't
cover our bugs we know for sure that it
can't find bugs that require very large
inputs or triggering certain kinds of
integer overflow so when I'm thinking
about a big leap it's like one of the
main things that seems like especially
hard to send against and steganography
is like if there's any any channel from
the humans who eventually like to do
things to the Internet any kind of
information theoretic channel let form
has like more than some number of
kilobytes right then it seems easy to
like create a message encrypted with a
meat key and then all right so the
problem there is that you could ask the
AI to paint you a picture
and this picture is nicely you put it in
your press release and later it turns
out it contained the AI source code I
like hey I think the main thing I would
serve you about is like if we had a way
to deal with this especially difficult
problem with that method of solving that
problem also so like a lot of these
other things or like are actually a need
for different solutions in different
problems so there is a possible research
path there where people could look into
how to take images and noise them and
put them through auto-encoders and
jitter them and whatnot in such a way
that's steganography can't get through
but the only surefire solution is just
never published that image or anything
they produce directly by the AI know
like yes Oliver do you envision like I'm
selecting against things that cannot
only write code and like recent event
burn abilities but like reason about
multiple copies of themselves and
similar agents and so forth and the
thing I kind of like to use that there
is another copy of me running on that
server over there and so coordinated
without actually exchanging any
information there's a lot to think about
there but it's definitely worth
researching but it's not something that
the software can help with yeah but it's
kind of like make sense to be aware of
those kinds of like problems as well
because it kind of might indicate which
layers are worse defended that we would
think people think those things I agree
all right
so let's let's thank you again