A systems neuroscience approach to building AGI - Demis Hassabis, Singularity Summit 2010

[Music]
our next speaker up on the floor was the
highest rated chess player in the world
at the age of 13 he wrote the
best-selling video game theme park at
the age of 17 and then he got a double
first in computer science from Cambridge
he put all those childish things aside
and founded the computer game company
called elixir studios and once he was
done with all that he got his PhD in
cognitive neuroscience and then became
known on the world scene for noticing
some connections between memory and
imagination science magazine called his
breakthrough his connecting memory and
imagination one of the scientific
breakthroughs of the year in 2007 so
this guy's been busy and he still looks
pretty young to me so please welcome
demis hassabis to the stage so today I'm
going to be talking about different
approaches to building a GI specifically
I'm going to be advocating a systems
neuroscience approach as potentially the
best way forward to making progress but
before I go into details about the
unique advantages I think the system's
new approach has I'm going to set the
context by doing a quick overview of the
current state in the art in AGI projects
past and present and see if we can
categorize them in some kind of
meaningful way so at the top level the
kind of most fundamental split I guess
is between those approaches that take a
biological and a non-biological approach
to building a GI so let's talk about the
non-biological approach first
chronologically in the history of AGI
these these were the first attempts and
basically the umbrella term we can we
can put these under is symbolic AI
and this includes formal logic systems
logic networks lambda calculus and
expert systems and there are many
projects involving these systems in the
80s and 90s were run by very smart
people
the issue was what they found when they
pushed these projects and they scaled
them up to try and approach AGI is they
tended to suffer from one or more of
these flaws they tend to be brittle not
dealing with ambiguity and uncertainty
very well they were time-consuming to
train and they were generally poor
generalizing also a specific problem to
symbolic AI systems is there are some
known classical philosophical problems
really for example how does a symbolic
system acquire new symbols we should
describe things and also the classic
symbol grounding problem probably one of
the biggest issues with symbolic AI
systems how do you actually create
meaning when you're only describing
symbols in terms of other symbols so
probably the prime example and probably
the most ambitious project in under this
umbrella is Doug Leonard's site project
which has been going on for about 25
years now and for those of you that
don't know basically this project was
about trying to encapsulate all of the
world's common-sense knowledge in a
massive database of the order of about a
million predicate rules now the problem
with this is apart from the time it's
taken to put those rules into the
database as I said 25 years and Counting
and around about estimated 609 years
they came across big problems where now
when you add one new rule into the
database this can cause a cascade of
inconsistencies throughout the rest of
the database that can take weeks to
resolve if they can be resolved at all
so on the other side of this divide are
the biological approaches so these
apologetically inspired approaches to a
GRE that use the brain to a certain
level of abstraction and I'll come back
to that as a blueprint for what idea a
GIS should contain although this is kind
of I put this as one camp it really
covers a massive range of different
approaches before we move on into the
biologically inspired and splitting that
down further let's just think about
really what it is that the underlying
belief systems
that would lead one to conclude which is
the best approach between these two
and I think fundamentally if you strip
away all the procedures and the various
systems that you might use it comes down
to one question what you believe
the search space or possible AGI
solutions looks like so I've done a
little cartoon to demonstrate that so
let's consider the search space of
possible AGI solutions I think really we
can big we can break this down into two
possible regimes that we are in so let's
consider regime one where we have the
search base of solutions is basically
small and dense so this is a little
cartoon of it so here we have a
relatively small search space the blue
stars indicate possible adi solutions
within that search space and there are
relatively many of them compared to the
size of the search base and now if we're
in this kind of regime the human brain
of course is another example of an AGI
but it's maybe not worth the effort if
you thought you were in this kind of
regime to spend the time to investigate
the human brain and use that as an
example on the other hand raising two we
have a large and sparse search base so
here we've got the relatively large
search space and now we have relatively
sparse numbers of solutions so if this
is the regime we find ourselves in then
clearly it makes sense to look at the
human brain and see what we can spend
the time and effort to see what we can
extract out of it so then just to finish
this line of thinking which regime are
we in now rigidly we want small and
dense regime to large and sparse now I
would say that of course we can't
categorically prove this one or the
other yet but we have current evidence
points towards the fact that strongly I
would say that we're enraging to the
large and sparse regime and I've got two
arguments for that one is a natural
argument evolution which is itself an
optimization and search algorithm
through this kind of search base has
only produced human level intelligence
once after hundreds of millions of years
of trying and then really going back to
these projects in the 80s and 90s these
serious projects and efforts to make AG
either largely failed to make progress
towards the overall goal although
produced interesting
on the way that's another indication it
may be that you know the optimistic
views of the 1670s of how easy a GI was
to build is was potentially wrong and
it's actually a lot harder than a search
basis a lot bigger and there's many
fewer solutions in that search space
it's ok let's say we buy this argument
and that you know using a biologically
inspired approaches is the right way
forward but this itself can be broken
down into a number of different clusters
so I basically created a spectrum here
so let's say on the left is the the most
loosely based on the brain in terms of
and I've determinate abstract and on the
right is the closest approaches to the
biology itself so let's look at think
about the left so over here I'm going to
put cognitive science architectures now
I could have picked dozens of examples
of these architectures but these are
three of the more well-known ones
they're Sophos allen eul's and john leds
saw Akhtar John Anderson's Akhtar and
more recently Bangert Saul's OpenCog and
here basically they what these what
these systems are is they've used been
inspired by behavioral psychology to try
and extract from the brain what modules
and what functions the brain may be
implements and then what you get out of
all these systems is effectively a
module a circuit diagram here if for
example is the one for actor and each of
these boxes basically refer to one
function that the advocates of these
systems believe the brain is doing and
then there's some wiring between those
and the problem I have with these
systems is that they're fairly
unprincipled so then these issues come
up when for example psychology throws up
a new function that we suddenly realize
is important to thinking and high
commission let's say episodic memory and
then now the these these kinds of
systems have to shoehorn post-hoc these
new modules into these kinds of
architectures and then just hope that
doesn't ruin everything and I think
another reason this shows the problem
with these on kind of unprincipled
approaches and why they're a bit
unsatisfactory is the very fact that
there are so many such a proliferation
of these kinds of architectures and I
believe the pond reason there is such a
proliferation is it's very hard to prove
your architecture is better than someone
else's but
because there are then they're
inherently no principles you can point
out that prove that and like all AGI
researchers we all believe we have the
right we have the answer and I think
that's the standing joke is that there
are as many architectures as there are
AGI researchers in the field so on the
other end of the scale they vote on the
very biological scale we have the whole
brain emulation advocates so again there
are many examples of this but the two
most famous ones are the Blue Brain
Project by hearing back room and motors
IBM synapse program and of course these
home very impressive technical feats
they push supercomputers to the limit
and they produce beautiful diagrams like
this this is taken from motors recent
PNAS paper that shows the wiring in the
macaque monkey brain but the question we
have to ask ourselves as in terms of
building AGI is is this level of is this
level of detail this level of
implementation detail useful towards
building AGI and I would say the answer
is maybe not so we may know some of the
wiring but what is it telling us about
the function so there's there's a long
way to go here and in fact kind of
serious whole brain in emulation
advocates go further than this and they
want to do a one-to-one mapping and
correspondence between a copper brain a
living brain and then copy that
one-to-one into an artificial substrate
but we're really 50 years away plus or
from creating that kind of imaging
technology so it's where's way in the
future especially if you want to keep
the original copy or the living brain
alive so what we're advocating is this
middle way what we're calling systems
neuroscience and here what we're
interested in is the brains algorithms
so let's just unpack that a little bit
more and look at in a slightly different
way and and hopefully this will show you
the different the real difference
between these these three different
approaches biologically inspired
approaches so David Marr is as many of
you know the father of computational
neuroscience and in the 70s he did a
number of he wrote a number of seminal
papers and one of those papers he
suggested what is now known as Mars 3
level of analyses and what he was saying
was that to understand any complex
biological system fully it's no good to
just understand it at one level of
abstraction
so just say for example the neuronal
level in the brain you need to
understand it at what he proposed three
different levels of abstraction all at
the same time so let's look at what the
what the levels of abstraction that he
proposed were useful for this kind of
analysis so he turned them computational
algorithmic and implementational so by
computation or what he meant was
defining what the goals of the system
are algorithmic is the how what are the
representations and the algorithms the
system uses to achieve those goals and
implementation is the medium the
physical realization if you like of the
system the actual substrate and how that
works so now using this level of
analyses we can go think back to the the
spectrum of different approaches on the
last slide and look where they fall into
this level of analyses so it's quite
clear from this we can see that the
whole brain emulation camp is basically
focused on the implementation level and
then building out from there would be
the hope on the other hand the cognitive
science architecture camp is really
focused on the very very high level of
computation of the goals of the system
and then just taking that and what I'm
suggesting is really we should be
focusing on the algorithmic level of the
brain and especially work extracting
what kinds of representations and
algorithms the brain uses to solve the
kinds of problems we'd like to solve
with AGI so that would all be that's all
very well that argument but it would be
a moot point if they weren't rapid
advances in neuroscience literally in
the last decade neuroscience has
advanced enough that we can start
answering some of these questions that
we had for AGI usefully so in the last
decade there's been a revolution in if
cognitive neuroscience and this has
mostly been driven by new experimental
techniques as a whole raft of them
there's new ones coming every few months
so obviously of course from imaging
Monte so recording opto genetics and
transmitted magnetic stimulation
two-photon microscopy and a whole range
of different techniques for answering
different questions there's also been a
revolution in the analysis tools used so
for example using multi very pattern
classifiers like we saw in the last talk
to class
by brain activity so as this whole
singularity summit is very fond of
exponential themes you can see here on
this graph this laser pointer works you
know
so here the green is the fMRI is fMRI
machine here's a pet machine and here's
an Emmy jima Sheen now if we look at
this graph this is a graph of on the on
the y axis is number of citations of
papers per year and down at the bottom
on the x axis is that is the year and we
can see this huge exponential growth
here happening from around 1995 and
continuing to this day and not only is
this exponential in total you can see
that the later developments like fMRI
which is in green are exponential
themselves so you can see here and this
graph here shows the proportion of
neuroscience papers using each of these
techniques and you can see the green
area as we go further along in time is
taking up more and more proportion of
the number of papers so that too is all
very wealth are there's tons of
neuroscience out there but most of it is
not relevant for AGI so how do we find
the Nuggets in all of that wealth of
information how do we keep abreast of
this sea of information so as we just
looked at there's this exponential
growth in neuroscience as 50,000 euro
science papers were published just in
2005 in 2008 what this means though is
if you last seriously looked at
neuroscience circa 2005 or before that
you're way out of date now you're
somewhere here on the graph so you're
somewhere here on the graph so there's a
whole load more things that have been
happening even in lost five years so
then the question is how do we sift
through it all to find the rail aparts
that are relevant to AGI what level of
implementation should we be looking at
and furthermore really we shouldn't just
be passively observing this neuroscience
work we should be actively conducting
and directing neuroscience cutting-edge
neuroscience research in ways that are
useful for the questions that we want to
answer for AGI
so there's no easy answer to this I
don't have an easy answer this you
basically need a ton of expertise and
full immersion in both areas it's the
only way and we found with our
colleagues on this project that this
takes about five years of intensive
learning and research at the end doing
intensive research from very motivated
smart people to gain the necessary
knowledge to be able to to cope to do
the rest of these things okay so what
what uses systems neuroscience for AGI
I've been banging on about neuroscience
being useful so what what are some
examples so firstly I think neuroscience
provides two things with you boil it
down provides direction and one calling
validation testing I'm going to unpack
those so let's look at provides
Direction first what I mean by this is
it will provide inspirations for new
algorithms and architectures so there
again ton of examples I could choose but
I'm going to talk through two first one
is a classic example for computer vision
that many all of you will know and then
a much more recent example from
navigation new navigation systems let's
talk about the visual system so Hoople
and Wiesel in 1959 did some seminal work
on on cat cortex visual cortex and they
discovered two types of cell with four
which they won the Nobel Prize in 1981
first type of cell they found was simple
cells tuned to preferred input stimulus
and here are some examples of stimuli
that these kinds of cells in this case
primary visual cortex v1 cells were were
tuned to so we can see here that these
are basically just straight lines and it
did different orientations and if a cell
saw one of these straight lines in its
in its receptive field it would
basically fire so that's what a simple
cell does okay then there's a second
type of cell complex cells and what
these do this is slightly more
complicated but they take a bunch of
simple cell inputs so we can see here an
image that's been deconstructed into its
simple cell inputs and then here's a
complex cell the first layer
flex cells which is taking a bunch of
these simple cells and the key is that
they're all tuned to the same preferred
stimuli so in this case this they're
tuned to the line pointing in this
direction it's all of these ones but the
key is that there are it's taking a
bunch of simple cells with slightly
different positions in the retina and
different scales and what this turns out
this put this introduces is invariance
which is exactly what you want for a
visual system tolerance to scale and
translation across the visual field and
then it turns out you can actually start
building hierarchies of these simple and
complex cell layers and it's exactly
what poggio has Tomaso pocha MIT has
been doing and he's created a state of
the art vision system by basically
mimicking the primate visual system so
here on the Left we have a macaque
monkey brains visual the wiring diagram
over there visual cortex
you can ignore most of it but just
notice the colored ones here this is to
do with the object recognition pathway
the ventral visual stream and then on
the right here you can see pojos model h
max model which basically mimics the
same layers in the same places to do
object recognition and this is state of
the art so what we can see is the
systems neuroscience procedure or
approach extracts the principles behind
an algorithm brain users creatively rim
could implement that in the
computational model and the result is
you get a state-of-the-art technique and
possibly an AGI component let's look at
one mother example play cells so play
cells are neurons that fire when a rat
is a specific location in environment
these were discovered by John Okeefe in
1970-71 and you can see here this is a
rat in his in his home pen and he's free
to roam around here and what they do in
these experiments is they stick
electrodes into the rat hippocampus so
you can see this is the rat brain and
the blue bit is the hippocampus so in
here's an electrode stuck into the
hippocampus and that's recording a
single cell from the hippocampus and
then what they do is they let the rat
run around this pen and you can see here
this is a top-down view of the pen and
the black lines are basically the roots
the rat has taken through that pen
during one trial and what you see here
is the red dots or the red clusters are
where this cell
fired during this journey through the
pen and you can see when you do a
heatmap they're all clustered around
what is called the place for place
filled of that place so around this top
right corner in this case now much more
recently what was found by them by the
Moses in in Norway was a bunch of cells
called grid cells and these are these
were found in the internal cortex which
is next to the hippocampus and provide
input into the hippocampus and these are
neurons with the spectacular regular
periodicity and they form hexagonal
firing patterns so I just showed you
play cells that had a single firing
pattern but here you've got hexagonal
firing patterns so let's just look at
this this final column on the right here
it's the only one that's relevant for
this which is these are three different
cells and in this case the rat is in a
circular environment rather than a
square one and you can see that this
particular single cell is firing at
these regular hexagonal places so it's
incredible when this was first found in
2005 no one believed it because it looks
non-biological it looks like someone's
coded this and this is what the brain
uses to to mark out space so it seems
like this is what the brain is using to
define an intrinsic measure for space so
what you could think of it as is like
the graph paper of the mind it's
something that's automatically
tessellating space we're doing this
right now we're looking at this wheel
and several new navigational systems
including several current DARPA projects
are using these two types of cells to
build new kinds of navigation techniques
okay so it provides direction and we've
seen that inspires new algorithms and so
what about validation testing or what do
I mean by that so what this is is really
we'd like to answer the questions of
this forum does an algorithm that we may
have invented through machine learning
or some other way or through maths
constitute a viable component of a GI
system should this does this algorithm
that I have sitting here on my machine
that does some interesting things is
that a useful component of an a GI
system well so to show this an example
I'm going to use the most classic
example of all which is reinforcement
learning now for those who don't know
the reinforced
learning problem is a whole class of
problems where the only teaching signal
is the reward game from the environment
now a temporal difference learning tool
TD learning is a method for solving part
of the reinforcement learning problem
that's one was going to talk through
here and the TD learning algorithm works
by minimizing the error in predicted
future reward so this was this was a
grand pioneered in the 70s and 80s by
Richard Sutton and and his colleagues
and it's been applied very successfully
to a wide range of control and
prediction problems so the question then
to reform the world of water I stated
just last slide is what we'd like to
answer is should reinforcement learning
algorithms be considered for an AGI
system and I we've had many arguments
with engineers that say yes it should be
or no it shouldn't and we advocate why
don't we just look it to the brain for
an answer to this question and in fact
when we do that we find the answer so
here in this one of the most famous
neuroscience papers of the last couple
of decades by shorts Diana Montagu in
science 1997 it's a single cell
recording of single neurons in a monkey
brain and what they found is that
temporal difference learning is
implemented precisely like the
algorithms created by Richard Sutton
ubiquitously across the brain so let's
have a quick look at this so here this
is a raster plots of neurons being
recalled a single neuron so you can see
here these are all the different trials
and these are the aggregated firing
rates across those 12 so we could just
look at this top line now this panel
here is occurs before the monkey has had
any training so what happens here in
this in this system is that the monkey
is just going along doing his business
and it gets a reward so this is
indicated by our and in the road in this
cases that is a drop of juice fruit
juice and you can see here in response
to getting the fruit juice unexpectedly
there's a better there's an uptick in
the firing rate of this dopamine neuron
now after a little while later they
train the monkey to associate a light
coming on with the delivery of the fruit
juice a short while later so the light
coming on is denoted by this CS symbol
conditioned stimulus and the reward is
dogs being delivered here where they are
is what we can see is that
the this neuron firing has traveled
backwards in time to be associated now
with the with the light coming on which
reliably predicts the future reward
rather than when the reward is presented
so you see here there's no extra firing
and then in the third manipulation the
final manipulation of this study in this
final panel they now remove the reward
so the light comes on the the neuron
fires as before up here in in
anticipation of a reward but now there's
no reward so what happens the the
dopamine neuron reduces its firing rate
now this is exactly the behavior this
dopamine neuron is is exhibiting the
exact behavior you'd want to signal
prediction error that a TD learning
algorithm would need and it turns out of
course is not just one neuron
this is a whole massive system the
dopamine system within the brain so here
is the dopamine system and it's pretty
ubiquitous across the brain it's also
Concorso so one other thing then is then
to look at the architectures then so in
the brain on the left hand side we have
the brain implementing TD learning and
these are the these are this is a box
diagram showing which brain areas
implement which parts of the function of
T V learning the T V learning algorithm
and on the right we have the engineering
approach with just math functions
implementing these these supporting
functions and you can see just by
looking at the architecture they are
identical so what can we conclude from
this well I think we can safely conclude
from this then implementing
reinforcement learning algorithms as
part of an AGI system is a compelling
approach to take it doesn't mean it has
to have this but it means that if you
try and build on with it in that maybe
not such a crazy thing to do and of
course I'm going to go through these but
there are many many other systems that
we know about now so we just talked
about dopamine for reward their
serotonin system that regulates mood and
emotion there's a style choline that
deals with expected variability and then
there's no penofin that deals with
unexpected variability and there's
there's there's others so but just to be
clear although I'm making the case for
systems neuroscience what I'm really
advocating is a hybrid approach which is
that we want to combine the best of
machine learning has to offer and
systems neuroscience so I think I can
make this most clear by saying
where we know how to build a component
for an AGI system let's use the
state-of-the-art algorithms let's just
take those the best to breed so really
that's reinforced learning or
hierarchical neural networks where we
don't know how to build a component then
we should continue to push machine
learning algorithms as far as we can but
also it makes sense in parallel to look
at systems neuroscience for ideas for
potential solutions so to do this in
parallel not one or the other
in fact my colleague Shane leg will be
talking about some of our specific
research using this approach tomorrow in
his talk so example systems so I've been
talking about what what other systems
are there out there we've seen some
examples that have been used in the past
what are they're left to be used well
actually there's there's dozens and his
each one of these things I've just put
up flicked up here which I've just
chosen would take a whole lecture to
describe but these are all put to candid
the potential candidate systems that
might be useful for investigating in
terms of AGI so mirror neurons model
based versus model free learning theory
of mind working memory top-down
attention conflict resolution and
constructs a system for mental
simulation my particular neuroscience
work but I have a lot of time to talk
about all that I'm just going to talk
about one thing which i think is
probably the most important of all which
is conceptual knowledge acquisition how
do we acquire conceptual knowledge so we
believe that concepts are key to a key
thing we're going to need to solve in
order to make some strives towards AGI
so let's have a look at what I'm meaning
by concepts so knowledge in the brain
can really be split into very coarsely
and there's no hard boundary here
between three different levels of
knowledge so the bottom level you have
perceptual which is your sensory stream
and then this this thing goes into the
concepts you'll so now we're talking
about more abstract things like so
perceptual maybe what a dog looks like
concept should be me maybe what a city
concept is the concept of a city and
then symbolic finally is labeling those
concepts so calling that concept of a
city a city and let's look at the
analogues of this on the machine
learning or AGI side right so what we
have is we have plenty of systems you
know attempts all these all of these the
the symbolic AI attempts for exam
based on logic and logic networks that
deal with the symbolic level of
information that the brain deals with
also in the last decade we've had many
advances in some very cool techniques
that deal with sensory learning and
sensory processing including Jeffrey
Hinton's deep belief networks pojos h
max I just showed and Jeff Hawkins as
HTM it's my aerials Destin and so on
there's there's there's a lot of good
systems now that deal with this but what
we don't know how to do is really
concepts this missing bid in between and
if we could solve this problem then we
would be able to deal simultaneously
with the classic simple grounding
problem how do we ground these symbols
in terms of meaning and also how do we
go from perceptual knowledge at the top
of these network of these hierarchical
networks to conceptual knowledge okay so
that's the problem space so how does the
brain acquire conceptual knowledge well
we have some very good ideas now and in
fact I think we're only a few years away
from fully understanding this and what
seems to be the candidate system if you
like is the hippocampal neo called
cortical consolidation system now for
those of you though know the hippocampus
it sits at the apex of the sensory
cortex you can see it here in blue and
them in the middle of your brain and
then when I'm talking about neocortical
in this context I'm meaning high-level
neocortex which means prefrontal cortex
on the front here and association cortex
on the side as opposed to sensory
cortices which round the back on top and
how does hippocampal neocortical
consolidation work well it works but
hippocampus stores the memories of
recent epoch experiences or episodes it
then replays those memories during
slow-wave sleep at far speeded rate so I
think we saw that in the last talk at
orders of magnitude faster than you
experienced them in in real life and
what this does is gives high-level
neocortex a tremendous number of samples
in which to learn form right even if you
only experienced that one that one
important thing once furthermore
memories are selector stochastically for
replay and rewarded memories emotional
and salient memories are biased to be
replayed more often
so suddenly now what this means is we
have a system that can circumvent the
statistics of the external environment
so the sensory cortices are limited to
learning the structure of the external
world the statistical regularities of
external world but that may not be what
we want for our behavior we may want to
fiddle around with the statistics so we
we bias the things that are more
important to our survival or progress
and this system has the ability to
confer that so this really is the first
step to that leads towards abstraction
and semantic knowledge and as an aside
also I think it explains the contents of
dreams but I haven't got time to go into
that but for those of you are Freud in
advocates I've got bad news for you I
think the evidence shows that dreams are
epic phenomenal so we can talk to me
about that afterwards but so to
summarize the brain is a useful put a
proof of concept for AGI we now have a
tool to meaningfully investigate it and
systems neuroscience can inspire new
algorithms and validate existing
algorithms and what are advocating is
combining the best of machine learning
with the with the best from systems
neuroscience and it's as this system
neuroscience approach has one final
advantage which is that building an AGI
system in this way at this level of
abstraction I think may also be the best
way to understand our own minds and
natural intelligence so effectively what
we're doing when we try to build AI
systems is distill intelligence to an
algorithmic construct and by comparing
this alchemy the construct of the mind
this should said shed light on our
long-standing enigma like consciousness
and then finally what I own village is a
kind of research program that combines
machine learning in neuroscience at a
very fundamental level in across distant
we group of people that help that and
this will help each other so in some
kind of virtuous cycle so I'm just going
to finish by leaving you with a quote
from the great Richard Fineman I think
sums up this whole approach best which
is what I cannot build I cannot
understand thanks for listening
[Applause]


Reframing superintelligence | Eric Drexler | EA Global: London 2018

I've been working in this area for quite a
while.
The chairman of my doctoral committee was
one Marvin Minsky.
We had some discussions on AI safety around
1990.
He said I should write them up.
I finally got around to writing up some developed
version of those ideas just very recently.
So that's some fairly serious procrastination.
Decades of procrastination on something important.
But for years one couldn't talk about advanced
AI.
One could talk about nanotechnology.
Now it's the other way around.
You can talk about advanced AI, but not about
advanced nanotechnology.
So this is how the Overton window moves around.
So what I would like to do is to give a very
brief presentation which is pretty closely
aligned with talks I've given at OpenAI and
DeepMind and, of course, FHI, Berkeley, Bay
Area Rationalists.
usually with a somewhat smaller number of
people and structured more around discussion.
But what I would like to do, still, is to
give a short talk, put up points for discussion,
and encourage something in-between Q&A and
discussion points from the audience.
Something along those lines.
Okay so, when I say "Reframing Superintelligence,"
what I mean is thinking about the context
of emerging AI technologies as a process rolling
forward from what we see today.
And asking, "What does that say about likely
paths forward?"
Such that whatever it is that you're imagining
needs to emerge from that context or in that
context.
Which I think reframes a lot of the classic
questions.
Most of the questions don't go away, but the
context in which they arise, the tools available
for addressing problems look different.
And well, that's what we'll be getting into.
So once upon a time when we thought about
advanced AI we didn't really know what AI
systems were likely to look like.
It was very unknown.
People thought in terms of developments in
logic and other kinds of machine learning,
different from the deep learning that we now
see moving forward with astounding speed.
And people reached for an abstract model of
intelligent systems.
And what intelligent systems do we know?
Well, actors in the world like ourselves.
We abstract from that very heavily and you
end up with rational, utility-directed agents.
Today, however, we have another source of
information beyond that kind of abstract reasoning,
which applies to a certain class of systems.
And information that we have comes from the
world around us.
What's happening, how AI systems are developing.
And so we can ask questions like, "Where do
AI systems come from?"
Well, today they come from research and development
processes.
We can ask, "What do AI systems do today?"
Well, broadly speaking, they perform tasks.
Which I think of, or will describe, as "performing
services."
They do some approximation or they do something
that someone supposedly wants in bounded time
with bounded resources.
What will they be able to do?
Well, if we take AI seriously, AI systems
will be able to automate asymptotically all
human tasks, and more, at a piecemeal and
asymptotically general superintelligent level.
So we said AI systems come from research and
development.
Well, what is research and development?
Well, it's a bunch of tasks to automate.
And, in particular, they're relatively narrow
technical tasks which are, I think, uncontroversially
automate-able on the path to advanced AI.
So the picture is of AI development moving
forward broadly along the lines that we're
seeing.
Higher-level capabilities.
More and more automation of the AI R&D process
itself, which is an ongoing process that's
moving quite rapidly.
AI-enabled automation and also classical software
techniques for automating AI research and
development.
And that, of course, leads to acceleration.
Where does that lead?
Leads to something like recursive improvement,
but not the classic recursive improvement
of an agent that is striving to be a more
intelligent, more capable agent.
But, instead, recursive improvement where
an AI technology base is being advanced at
AI speed.
And that's a development that can happen incrementally.
We see it happening now as one takes steps
toward advanced AI that is applicable to increasingly
general and fast learning.
Well, those are techniques that will inevitably
be folded into the ongoing AI R&D process.
Developers, given some advance in algorithms
and learning techniques, conceptualization
of how to address more and more general tasks
will pounce on those, and incorporate them
into a broader and broader range of AI services.
So where that leads is to what are asymptotically
comprehensive AI services.
Which, crucially, includes the service of
developing new services.
So increasingly capable, increasingly broad,
increasingly piecemeal and comprehensively
superintelligent systems that can work with
people, interact with people, in many different
ways to provide the service of developing
new services.
And that's a kind of generality.
That is a general kind of artificial intelligence.
So a key point here is that the C in CAIS,
C in Comprehensive AI Services does the work
of the G in AGI.
Why is it a different term?
To avoid the implication... when people say
AGI they mean AGI agent.
And we can discuss the role of agents in the
context of this picture.
But I think it's clear that a technology base
is not inherently in itself an agent.
In this picture agents are not central, they
are products.
They are useful products of diverse kinds
for providing diverse services.
And so with that I would like to, as I said
the formal part here will be short, point
to a set of topics.
They kind of break into two categories.
One is about short paths to superintelligence,
and I'll argue that this is the short path.
The topic of AI services and agents, including
agent services, versus the concept of "The
AI" which looms very large in people's concepts
of future AI.
I think it should look at that a little bit
more closely.
Superintelligence as something distinct from
agents, superintelligent non-agents.
And the distinction between general learning
and universal competence.
People have, I think, misconstrued what intelligence
means and I'll take a moment on that.
If you look at definitions of good from the
1960s, ultra-intelligence and more recent
Bostrom and so on (I work across the hall
from Nick) on superintelligence the definition
is something like "a system able to outperform
any person in any task whatsoever."
Well, that implies general competence, at
least as ordinarily read.
But if we ask, you know, we call children
intelligent and we call senior experts intelligent.
We call a child intelligent because the child
can learn, not because the child can perform
at a high level in any particular area.
And we call an expert who can perform at a
high level intelligent not because the expert
can learn, in principle you could turn off
learning capacity in the brain, but because
the expert can solve difficult problems at
a high level.
So learning and competence are dissociable
components of intelligence.
They are in fact quite distinct in machine
learning.
There is a learning process and then there
is an application of the software.
And when you see discussion of intelligent
systems that does not distinguish between
learning and practice, treats action as entailing
learning directly, there's a confusion there.
There's a confusion about what intelligence
means and that's, I think, very fundamental.
In any event, looking toward safety-related
concerns, there are things to be said about
predictive models of human concerns.
AI-enabled solutions to AI-control problems.
How this re-frames questions of technical
AI safety.
Issues of services versus addiction, addictive
services and adversarial services.
Services include services you don't want.
Taking superintelligent services seriously.
And a question of whether faster development
is better.
And, with that, I would like to open for questions,
discussion, comment.
I would like to have people come away with
some shared sense of what the questions and
comments are.
Some common knowledge of thinking in this
community in the context of thinking about
questions this way.
Is your model compatible with end-to-end reinforcement
learning?
Yes.
Thank you.
To say a little bit more.
By the way, I've been working on a collection
of documents for the last two years.
It's now very large, it will be an FHI technical
report soon.
It's 30,000 words structured to be very skim-able.
Top-down, hierarchical, declarative sentences
expanding into longer ones, expanding into
summaries, expanding into fine-grained topical
discussion.
So you can sort of look at the top level say,
hopefully, "Yes, yes, yes, yes, yes.
What about this?"
And not have to read anything like 30,000
words.
So, what I would say is that reinforcement
learning is a technique for AI system development.
You have a reinforcement learning system.
It produces through a reinforcement learning
process, which is a way of manipulating the
learning of behaviors.
Produces systems that are shaped by that mechanism.
So it's a development mechanism for producing
systems that provide some service.
Now if you turn reinforcement learning loose
in the world open-ended, read-write access
to the internet, a money-maximizer and did
not have checks in place against that?
There are some nasty scenarios.
So basically it's a development technique,
but could also be turned loose to produce
some of the kinds of global... "creative systems
trying to manipulate the world in bad ways"
scenarios are another sector of reinforcement
learning.
So not a problem per se, but one can have
problems using that technique.
And then a clarification question.
What does asymptotic improvement of AI services
mean?
I think I'm abusing the term asymptotic.
What I mean is increasing scope and increasing
level of capability in any particular task
to some... comprehensive is sort of like saying
infinite, but moving toward comprehensive
and superintelligent level services.
What it's intended to say is, ongoing process
going that direction.
And just sort of if someone has a better word
than asymptotic to describe that I'd be very
happy.
Can the tech giants like Facebook and Google
be trusted to get alignment right?
Google more than Facebook.
We have that differential.
I think that questions of alignment look different
here.
I think more in terms of questions of application.
What are the people who wield AI capabilities
trying to accomplish?
So there's a picture which, just background
to the framing of that question, and a lot
of these questions I think I'll be stepping
back and asking about framing.
As you might think from the title of the talk.
So picture a rising set of AI capabilities:
image recognition, language understanding,
planning, tactical management in battle, strategic
planning for patterns of action in the world
to accomplish some goals in the world.
Rising levels of capability in those tasks.
Those capabilities could be exploited by human
decision makers or could, in principle, be
exploited by a very high-level AI system.
I think we should be focusing more, not exclusively,
but more on human decision makers using those
capabilities than on high-level AI systems.
In part because human decision makers, I think,
are going to have broad strategic understanding
more rapidly.
They'll know how to get away with things without
falling afoul of what nobody had seen before,
which is intelligence agencies watching and
seeing what you're doing.
It's very hard for a reinforcement learner
to learn that kind of thing.
So I tend to worry about not the organizations
making aligned AI so much as whether the organizations
themselves are aligned with general goals.
Which will be the subject of the talk I'm
giving at 3:00 on "Paretotopian Goal Alignment."
And, actually, I have a few slides appended
to this that I can show you at some point.
Going to have a little bit of overlap between
the talks.
Could you describe the path to superintelligent
services with current technology?
So some more concrete examples?
Well, we have a lot of piecemeal examples
of superintelligence.
AlphaZero is superintelligent in the narrow
domain of Go.
There are systems that outperform human beings
in playing these very different kinds of games,
Atari games.
Face recognition recently surpassing human
ability to map from human speech to transcriptive
words.
Just more and more areas piecemeal.
A key area that I find impressive and important
is the design of neural networks at the core
of modern deep learning systems.
The design of and learning to use appropriately,
hyperparameters.
So, as of a couple of years ago, if you wanted
a new neural network, a convolutional network
for vision, or some recurrent network, though
recently they're going for convolution networks
for language understanding and translation,
that was a hand-crafted process.
You had human judgment and people were building
these networks.
A couple of years ago people started in these,
this is not AI in general but it's a chunk
that a lot of attention went into, getting
superhuman performance in neural networks
by automated, AI-flavored like for example
reinforcement learning systems.
So developing reinforcement learning systems
that learn to put together the building blocks
to make a network that outperforms human designers
in that process.
So we now have AI systems that are designing
a core part of AI systems at a superhuman
level.
And this is not revolutionizing the world,
but that threshold has been crossed in that
area.
And, similarly, automation of another labor-intensive
task that I was told very recently by a senior
person at DeepMind would require human judgment.
And my response was, "Do you take AI seriously
or not?"
And, out of DeepMind itself, there was then
a paper that showed how to outperform human
beings in hyperparameter selection.
So those are a few examples.
And the way one gets to an accelerating path
is to have more and more, faster and faster
implementation of human insights into AI architectures,
training methods, and so on.
Less and less human labor required to do that.
Higher and higher level human insights being
turned into application throughout the existing
pool of resources.
And, eventually, fewer and fewer human insights
being necessary.
So what are the consequences of this reframing
of superintelligence for technical AI safety
research?
Well, re-contexting.
It says, in part... well let's look over here.
"If in fact one can have superintelligent
systems that are not inherently dangerous,
then one can ask how one can leverage high-level
AI," abusing the word asymptotically, "asymptotically
superintelligent systems and applying to technical
AI safety problems."
So a lot of the classic scenarios of misaligned
powerful AI involve AI systems that are taking
actions that are blatantly undesirable.
And, as Shane Legg said when I was presenting
this at DeepMind last Fall, "There's an assumption
that we have superintelligence without common
sense."
And that's a little strange.
So Stuart Russell has pointed out that machines
can learn not only from experience, but from
reading.
And, one can add, watching video and interacting
with people and through questions and answers
in parallel over the internet.
And we see in AI that a major class of systems
are predictive models.
Given some input you predict what the next
thing will be.
The next, well in this case, given a description
of the situation or an action you try to predict
what people will think of it.
Is it something that they care about or not?
And, if they do care about it, is there widespread
consensus that that would be a bad result?
Widespread consensus that it would be a good
result?
Or strongly mixed opinion?
So if one has a predictive model of that,
and note it's a predictive model trained on
many examples, it's not an agent.
That is an oracle that, in principle, could
operate with reasoning behind the prediction.
That could in principle operate at a super
intelligent level, and would have common sense
about what people care about.
Now think about having AI systems that you
intend to be aligned with human concerns where,
available for a system that's planning action,
is this oracle.
It can say, "Well, if such and such happened,
what would people think of it?"
And you'd have a very high-quality response.
That's a resource that I think one should
take account of in technical AI safety.
We're very unlikely to get high-level AI without
having this kind of resource.
People are very interested in predicting human
desires and concerns if only because they
want to sell you products or brainwash you
in politics of something.
And that's the same underlying AI technology
base.
So I would expect that we will have predictive
models of human concerns.
That's an example of a resource that would
reframe some important aspects of technical
AI safety.
So, making AI services more general and powerful
involves giving them higher-level goals.
At what point of complexity and generality
do these services then become agents?
Well, many services are agent-services.
A chronic question that arises, people will
be at FHI or DeepMind and someone will say,
"Well, what is an agent anyway?"
And everybody will say, "Well, there is no
sharp definition.
But over here we're talking about agents and
over here we're clearly not talking about
agents."
So I would be inclined to say that if a system
is best thought of as direct toward goals
and it's doing some kind of planning and interacting
with the world I'm inclined to call it an
agent.
And, by that definition, there are many, many
services we want, starting with autonomous
vehicles, autonomous cars and such, that are
agents.
They have to make decisions and plan.
So there's a spectrum from there up to higher
and higher level abilities to do means-ends
analysis and planning and to implement actions.
So let's imagine that your goal is to have
a system that is useful in military action
and you would like to have the ability to
execute tactics with AI speed and flexibility
and intelligence, and have strategic plans
for using those tactics that are superintelligent
level.
Well, those are all services.
They're doing something in bounded time with
bounded resources.
And, I would argue, that that set of systems
would include many systems that we would call
agents but they would be pursuing bounded
tasks with bounded goals.
But the higher levels of planning would naturally
be structured as systems that would give options
to the top level decision makers.
Who would not want to give up their power,
they don't want a system guessing what they
want.
At a strategic level they have a chance to
select, strategy unfolds relatively slowly.
So their would be opportunities to say, "Well,
don't guess, but here's the trade off I'm
willing to make between having this kind of
impact on opposition forces with this kind
of lethality to civilians and this kind of
impact on international opinion.
I would like options that show me different
trade-offs.
All very high quality but within that trade-off
space.
And here I'm deliberately choosing an example
which is about AI resources being used for
projecting power in the world.
I think that's a challenging case, it is a
good place to go.
I'd like to say just a little bit about the
opposite end, briefly.
Superintelligent non-agents.
Here's what I think is a good paradigmatic
example of superintelligence and non-agency.
Right now we have systems that do natural
language translation.
You put in sentences or, if you had a somewhat
smarter system that dealt with more context,
books, and out comes text in a different language.
Well, I would like to have systems that know
a lot to do that.
You do better translations if you understand
more about history, chemistry if it's a chemistry
book, human motivations.
Just, you'd like to have a system that knows
everything about the world and everything
about human beings to give better quality
translations.
But what is the system?
Well, it's a product of R&D and it is a mathematical
function of type, character string to character
string.
You put in a character string, things happen,
and out comes a translation.
And you do this again, you do this again,
and you do this again.
Is that an agent?
I think not.
Is it operating at a superintelligent level
with general knowledge of the world?
Yes.
So I think that one's conceptual model of
what high-level AI is about should have room
in it for that system and for many systems
that are analogous.
Would a system service that combines general
learning with universal competence not be
more useful or competitive than a system that
displays either alone?
So does this not suggest that agents might
be more useful?
Well, as I said, agents are great.
The question is what kind and for what scope.
So, as I was saying, distinguishing between
general learning and universal competence
is an important distinction.
I think it is very plausible that we will
have general learning algorithms.
And general learning algorithms may be algorithms
that are very good at selecting algorithms
that are good at selecting algorithms for
learning a particular task and inventing new
algorithms.
Now, given an algorithm for learning, there's
a question of what you're training it to do.
What information?
What competencies are being developed?
And I think that the concept of a system being
trained on and learning about everything in
the world with some objective function, I
don't think that's a coherent idea.
What is... let's say you have a reinforcement
learner.
You're reinforcing the system to do what?
Here's the world and it's supposed to be getting
competence in organic chemistry and ancient
Greek and, I don't know, control of the motion
of tennis-playing robots and on and on and
on and on.
What's the reward function, and why do we
think of that as one task?
I don't think we think of it as one task.
I think we think of it as a bunch of tasks
which we can construe as services.
Including the service of interacting with
you, learning what you want, nuances.
What you are assumed to want, what you're
assumed not to want as a person.
More about your life and experience.
And very good at interpreting your gestures.
And can go out in the world and, subject to
constraints of law and consulting an oracle
on what other people are likely to object
to, implement plans that serve your purposes.
And if they are important and have a lot of
impact, within the law presumably, what you
want is for that system to give you options
before the system goes out and takes action.
And some of those actions would involve what
are clearly agents.
So that's the picture I would like to paint
that I think reframes the context of that
question.
So on that is it fair to say that the value-alignment
problem still exists within your framework?
Since, in order to train a model to build
an agent that is aligned with our values,
we must still specify our values.
Well, what do you mean by, "train an agent
to be aligned with our values."
See, the classic picture says you have "The
AI" and "The AI" gets to decide what the future
of the Universe looks like and it had better
understand what we want or would want or should
want or something like that.
And then we're off into deep philosophy.
And my card says philosophy on it, so I guess
I'm officially a philosopher or something
according to Oxford.
I was a little surprised.
"It says philosophy on it.
Cool!"
I do what I think of as philosophy.
So, in a services model, the question would
instead be, "What do you want to do?"
Give me some task that is completed in bounded
time with bounded resources and we could consider
how to avoid making plans that stupidly cause
damage that I don't want.
Plans that, by default, automatically do what
I could be assumed to want.
And that pursue goals in some creative way
that is bounded, in the sense that it's not
about reshaping the world, other forces would
presumably try to stop you.
And I'm not quite sure what value alignment
means in that context.
I think it's something much more narrow and
particular.
By the way, if you think of an AI system that
takes over the world?
Keep in mind that a sub-task of that, part
of that task, is to overthrow the government
of China.
And, presumably, succeed the first time because
otherwise they're going to come after you,
if you made a credible attempt.
And that's in the presence of unknown surveillance
capabilities and unknown AI that China has.
So you have a system and it might formulate
plans to try to take over the world.
Well, I think an intelligent system wouldn't
recommend that because it's a bad idea.
Very risky.
Very unlikely to succeed.
Not an objective that an intelligent system
would suggest or attempt to pursue.
So you're in a very small part of a scenario
space where that attempt is made by a high-level
AI system.
And it's a very small of scenario space because
it's an even smaller part of scenario space
where there is substantial success.
I think it's worth thinking about this.
I think it's worth worrying about it.
But it's not the dominant concern.
It's a concern in a framework where I think
we're facing an explosive growth of capabilities
that can amplify many different purposes,
including the purposes of bad actors.
And we're seeing that already and that's what
scares me.
So I guess, in that vein, could the superintelligent
services be used to take over the world by
a state actor?
Just the services?
Well, you know, services include tactical
execution of plans and strategic planning.
And if there is a way for a state actor to
do that using AI systems in the context of
other actors with, presumably, a comparable
level of technology?
Maybe so.
And I think that is a good reason to give
some overlapping slides with the talk that
I'm going to be giving at 3:00, which is about
goal-alignment.
The assumption there is that a state actor
might want to try to do that.
It's obviously a very risky thing to do.
So, just to step into that area.
Key considerations for forward-looking EA
strategy.
AI: very, very, very, very, very important.
Why?
In part because it will lead to a situation
in which, well, important, and the important
point here is that at some point explosive
growth will be close enough to be credible
and be within, understood to be within the
planning horizons of more and more and more
real world actors and institutions.
One aspect of powerful AI is an enormous expansion
of productive capacity.
Partly through, for example, high-level, high
quality automation.
More realistically, physics-limited production
technology which is outside today's sphere
of discourse or Overton window.
Security systems, I will assert, could someday
be both benign and effective.
Stabilizing.
So the argument is that, eventually it will
be visibly the case that we'll have superintelligent
level, very broad AI, enormous productive
capacity, and the ability to have strategic
stability, if we take the right measures beforehand
to develop appropriate systems, or to be prepared
to do that, and to have aligned goals among
many actors.
So these are outside the Overton window of
policy discourse, outside the range of what
can be discussed.
So the first set of facts says we can have
an approximately strongly Pareto-preferred
world, a world that looks pretty damn good
to pretty much everyone.
And fact four says that strategies for getting
there are constrained by the fact that you
can't really talk about the underlying premises
seriously.
And this is much of what I'm going to be talking
about in the "Paretotopian Goal Alignment"
talk at 3:00.
But, quickly, resource competition is central.
It's the zero-sum part.
It's like a zero-sum game.
And here we have two axes: quantity of stuff,
some finite resource that A gets, quantity
of stuff that B gets.
The constraint line is unit quantity.
And so we have this nice, straight diagonal
line.
And I've shown 50/50 current holdings, just
as an example situation one might be in.
The usual assumption is that resources are
fixed or growing slightly so it doesn't make
a whole lot of difference that they're growing
and so you have this kind of conflict.
"I want more, that means you get less."
Approximately zero-sum.
If we think about utility, though, we should
be thinking about utility not quantity of
resources.
Then people often think of quantity of something,
the utility is something like the logarithm
of that quantity.
Money or, I'm going to say, resources here.
And that gives us... well, first of all, here's
what expansion by 50% looks like.
You're still, largely the question is, back
and forth, where do you end up on this, strong
trade-offs.
Here's B taking all the gains.
Here's B taking 90% of the total, actually
taking away from A. Now putting this into
the world of utility, we re-plot the same
curves on a log scale and we get curved lines
and they look like this.
And there's our 50% expansion again.
Qualitatively it looks rather similar.
So the question is, "What happens if you have
a large expansion of resources?"
What if you were in a situation where, for
the first time in history, it's possible to
see, within a decision-making, planning horizon
that there are opportunities for an enormous
expansion of these rivalrous goods, resources?
Well, same curves are shown there in the lower
left.
And here's what a factor of a thousand expansion
looks like.
Notice that all the gains versus 90% have
swapped positions.
Now someone can take 90%.
You get a smaller fraction of the resources
and you're still much, much better off.
What's important is that you share the gains
in some way that's reasonable.
Now, in that picture, this is small.
The incentives for grabbing everything are
relatively small.
And they're also risky.
If you try to get from current holdings to
all the gains you're going to get a lot of
opposition.
If you aim for some place that is relatively
fair, maybe one can have goal alignment and
a lot of different parties saying, "Yes, let's
go there."
And so the argument is that greed brings risk.
And here are places you might end up if you
are one of these actors and trying to get
to different places.
Very unfair outcomes are likely to get opposition,
you're less likely to succeed, so this is
the risk-adjusted gains, or payoff.
And then, associated with this, well here's
a region.
This region is labeled "Paretotopia."
It's sort of like utopia except without the
implication that everybody is in the ideal
society whether they want it or not.
It's instead defined to be a set of conditions
that look pretty good to pretty much everyone.
The sorts of world outcomes that could in
fact align the goals of powerful actors.
And that's why I brought this up in the context
of states trying to take over the world.
This says that's dangerous and there's very
little reward to doing so if you have the
option of having security, not by dominance,
but by developing secure systems.
And the argument is that, in fact, forward
looking in the context of superintelligence
does objectively make that possible.
The problem is to have decision makers have
that be within their sphere of discourse and
see it as an attractive option when the decisions
are made downstream.
And that's what the following talk is about.
I'm going to go back to...
So then, in the context of that, what do you
think the greatest AI threat to society, then,
in the next 10, 20 years would be?
I think the greatest threat is instability.
Sort of either organic instability from AI
technologies being diffused and having more
and more of the economic relationships and
other information-flow relationships among
people be transformed in directions that increase
entropy, generate conflict, destabilize political
institutions.
Who knows?
If you had the internet and people were putting
out propaganda that was AI-enabled, it's conceivable
that you could move elections in crazy directions
in the interest of either good actors or bad
actors.
Well, which will that be?
I think we will see efforts made to do that.
What kinds of counter-pressures could be applied
to bad actors using linguistically politically-competent
AI systems to do messaging?
And, of course, there's the perennial states
engaging in an arms race which could tip into
some unstable situation and lead to a war.
Including the long-postponed nuclear war that
people are waiting for and might, in fact,
turn up some day.
And so I primarily worry about instability.
Some of the modes of instability are because
some actor decides to do something like turn
loose a competent hacking, reinforcement-learning
system that goes out there and does horrible
things to global computational infrastructure
that either do or don't serve the intentions
of the parties that released it.
But take a world that's increasingly dependent
on computational infrastructure and just slice
through that, some horribly destabilizing
way.
So those are some of the scenarios I worry
about most.
And then maybe longer term than 10, 20 years?
Longer term-
If the world isn't over by then?
Well, I think all of our thinking should be
conditioned on that.
If one is thinking about the longer term,
one should assume that we are going to have
superintelligent-level general AI capabilities.
Let's define that as the longer term in this
context.
And, if we're concerned with what to do with
them, that means that we've gotten through
the process to there then.
So there's two questions.
One is, "What do we need to do to survive
or have an outcome that's a workable context
for solving more problems?"
And the other one is what to do.
So, if we're concerned with what to do, we
need to assume solutions to the preceding
problems.
And that means high-level superintelligent
services.
That probably means mechanisms for stabilizing
competition.
There's a domain there that involves turning
surveillance into something that's actually
attractive and benign.
And the problems downstream, therefore, one
hopes to have largely solved.
At least the classic large problems and now
problems that arise are problems of, "What
is the world about anyway?"
We're human beings in a world of superintelligent
systems.
Is trans-humanism in this direction?
Uploading in this direction?
Developing moral patients, superintelligent-level
entities that really aren't just services.
They're the moral equivalent of people.
What do you do with the cosmos?
It's an enormously complex problem.
And, from the point of view of having good
outcomes, what can I say?
There are problems.
So what can we do to improve diversity in
the AI sector?
And what are the likely risks of not doing
so?
I'm not sure exactly what diversity means
in this context.
Well, I don't know.
My sense is that what is most important is
having the interests of a wide range of groups
be well represented.
To some extent, obviously, that's helped if
you have in the development process, in the
corporations people who have these diverse
concerns.
To some extent it's a matter of politics regulation,
cultural norms, and so on.
I think that's a direction we need to push
in.
To put this in the paretotopian framework,
if your aim is to have objectives, goals that
really are aligned.
Possible futures that are strongly goal-aligning
for many different groups.
Those groups' concerns, which we don't fully
understand, I'm over here do I fully understand
the concerns of that group?
No.
Need to have some joint process that produces
an integrated, adjusted picture of, for example,
how do we have EAs be happy and the billionaires
maintain their relative position?
Because if you don't do that they're going
to maybe oppose what you're doing, and the
point is to avoid serious opposition.
And also have the government of China be happy.
And I would like to see the poor in rural
Africa, billionaires be way up here and they're
now competing not to build orbital vehicles
but star ships.
And the poor in rural Africa of today merely
have orbital space capabilities convenient
for families.
They're poor.


Decision theory research at FRI | Johannes Treutlein & Caspar Oesterheld | EAGxBerlin 2017

good morning everybody welcome to the
second day of the conference and I see
you early in the morning - this is not
so easy presentation and let me
introduce our two speakers so Johannes
and Caspar work both at the foundational
for National Research Institute and
Johannes Felton we should researches on
topics and decision theory and before
his work at Fri you Hannah studied the
double bass in Stuttgart and room Berg
and played with the Munich Philharmonic
Kasper studied computer science and the
University of Bremen in Midland High
School in computers a year's worth of
university level courses in mathematics
at the University of Hamburg he has
published in the Journal of synthesis
and his research areas include
theoretical computer science ethics and
artificial intelligence and very keen to
listen to your presentation
so custard and I are doing among other
things you're doing decision theory
research and especially we're looking at
the decision theory of new comes problem
which we'll talk about today and similar
problems now in this talk we will
present two papers we are currently
working on and we are very grateful for
any feedback or questions that you have
unfortunately we don't have the time to
take questions during this lot but
please do come to us in person after
this talk and if you have any questions
or feedback and we can discuss them in
the break now since this is the kind of
advance track we will keep our
explanations quite short but we hope it
still talked will still be intelligible
so I have a begin now with talking about
the wager financial decision theory and
afterwards Kaspar will continue with
decision theory and approval directed
agents so if well informed and
intelligent experts disagree about some
matter then it seems unreasonable for
anyone to have a hundred percent
credence in any single solution to this
problem and we think this also applies
to decision theory so there are many
different theories that have been
proposed most notably causal decision
theory and evidential decision theory we
talked about those later as well and
academics can be found endorsing either
of these theories so if you want to say
we are uncertain about which to correct
a decision theory we can take this
uncertainty into account by using some
kind of meta decision theory and that's
what for example Richard Nozick or where
you make a scale have proposed doing now
in this talk I am I'm going to talk
about one implication of using such a
meta decision theory and that's if you
live in last year
and we care about what happens in other
parts of the universe then this can lead
to a wager in favor of evidential
decision theory now in the talk for
simplicity for the reason of simplicity
I only talk about causal and evidentiary
decision theory but this is also a wager
in favor of some other non causal theory
so timeless logical up dateless
anthropic decision theory are also both
games phone and calling us variations of
CDT now let me begin by recalling and a
famous thought experiment Newcomb's
problem a variant of this thought
experiment so here we can kind of see
what causal decision theory so CDT and
evidential decision here UDT are about
and this problem sets both of those
apart so imagine there is an omniscient
or powerful and trustworthy being called
Omega and Omega plays a game with us so
there are two boxes one translucent box
and one opaque box and we have two
options we can either take both boxes
with it which is called to boxing or we
can only take the opaque box which is
called long boxing now the translucent
box contains visibly one wish and this
wish stands for one life set so if you
take this box then Omega will save the
life of one terminally ill child the
content of the opaque box gets
determined as follows so Omega predicts
in advance what we will do if omega
predicts we would one box
it puts two wishes in this trans Lou Ann
this opaque box which stand for two life
saved if it predicts we what two box it
was nothing in this box so here we have
an overview over this decision situation
and this table specifies the number of
life say
given either to either of these two
options and if you perform them and in
those two possible states so either the
opaque box is filled or it isn't good
now there's two ways to reason about
this problem and those two ways for
sponsor to different versions of
expected utility theory
the first one is evidential decision
theory which says you should one box
because one boxing is the action which
provides us with the better news so
conditional on observing one boxing it's
much more likely that your peg box is
filled and that then because Omega is an
accurate predictor and then we'll end up
with more life safe than conditional on
to boxing and formally we can represent
this by this expected utility equation
until we use the we sum over the state
over States and we weigh the states by
the probability of these states
conditional on our action now let's say
the predictor is 90% accurate so filt of
cake box has probability zero point nine
conditional on one boxing so one box
English is action one here and then if
you do this calculation turns out one
boxing is the action with higher
expected utility
according to evidentially decision
period now a cost of decision theory
reasons differently it only looks at the
causal effects of our actions and here
for this reason cause of the superior to
box so it would think we can't so the
the content of these boxes is already
determined by Omega and we can't
influence the content which these boxes
causally anymore but and no matter
whether the OPEC box is filled or not in
both of these cases we end up with more
money with more life saved if we toolbox
and formally we use this expected
utility equation where we
very used the unconditional insistence -
now this equation that we work always
but in the cases we discussed it works
fine and if you just assume the
unconditionally these credentials are
50% for each of those states then -
boxing so action to here has higher
expected utility
now as I discussed in the beginning we
might want to use some meta decision
theory because we are uncertain about
which of these theories is true and we
can do this the same way we treat
empirical uncertainty so you'll be used
expected value approach and here this DK
stands for the decision theories we have
credence in and we just do expected
value and weigh all of these the values
that these decision theories provided by
the credence we have in TCU's now this
equation presupposes some things for
example it presupposes that these
decisions previous values are comparable
and that we normalize them in some way
so I think at least for evidential and
positive materialists why it's plausible
since both of these theories agree in
most ordinary cases and they use exactly
the same utility function so both are
our versions of expected utility only in
cases where our action provides us with
evidence that doesn't stem from the
causal of effects of our actions only in
such cases today disagree so and if you
use their cases where they agreed to
normalize them then we can just give
them equal weight and this is what this
equation does now let's suppose we have
a 90% credence in constant decision
theories since maybe we think more
academics are in loss and causes an
evidential decision theory but we still
some uncertainty so 10% in favor of
evidentially decision theory and if we
do these calculations it turns out that
to boxing here as I higher expected
I expected utility and this also seems
not surprising so since indeed in this
case around equally as much less at
stake for causal as for an evidential
decision theory and for this reason
because we have a higher credence and
cause the decision theory it seems
reasonable that here we go with a cause
of decision theory if use this meta
decision theory now what happens if we
try to model this situation in a large
universe so an infinite universe with
infinite space is one of the most widely
in those theories among cosmologists and
if this is true then at some point in
space there will be an exact copy of
Earth and an exact copy of our part of
cosmos and down will not only be one
copy but of course there will be
infinitely many such copies now since
it's hard to calculate with infinities
we just assumed here that there's some
kind of large number n of copies and
copies of the thought experiment that
happen and furthermore we also assumed
that we actually care about what happens
in these other parts of the universe so
when some life kid gets saved not on
this earth but on some copy of Earth and
this also comes to our towards our
utility now what happens in this
situation interestingly the expected
utility of evidential decisions here it
just gets multiplied by this factor so
we don't care we don't show here we
spare you the exact calculations we have
proof for this in the paper also but the
reasoning here is that conditional on
one boxing of course all our copies will
also warn Bach
and all other omegas in all other parts
of this universe will also predict you
on boxing so everyone's going to end up
with the same utility and if you just
calculate this towards our own utility
we receive this now importantly cos the
decision Theory reacts differently to
this situation so the expected utility
by positive certainty just gets shifted
by one constant and the reason here is
that we can't causally influence what
our copies do so whatever our estimate
is for what happens in these other
copies of the universe
this estimate doesn't change according
to court decision theory based on
whatever action we take so the public
our utility function is which is
determined by our copies is the same for
all actions now if we use meta decision
theory in this case it turns out and for
example we set N to 100 here then you
can see that action 1 so 1 boxing will
have higher meta expected utility and so
this also seems still a kind of desired
behavior because in this case since
there are so many copies and so many
lives at stake there and only evidential
decision theory takes the impact that
our copies have into account here we go
with evidential decision view it's much
more it's at stake for evidential
decision theory than for court decision
here in this scenario now this kind of
reasoning from this example can be
generalized and then it leads to a
general wage and favor of evidential
decision 0 so whenever there is the last
universe so many copies of us and then
if we also care about the games of our
copies so if as we let this number of
copies and increase the part of our
utility function that stands for the
games that these copies gets also
increases then if the universe is very
large
the impact of our copies will completely
dominate our own impact and now because
only evidential decision theory takes
its impact of our copies into account
only financial decision theory this
matters then in fact whenever we have
any non-zero credence an evidential
decision theory we will actually act
according to the action that evidential
decision theory recommends so we'll act
as if we have a 100% queens in
evidential decision theory now that's of
course if you use this metal decision
theory in the way I outlined in the
beginning and then this leads to a wager
in favor of evidential decision theory
as I said and of course also in favor of
all other theories that take the impact
of our copies into account now just some
brief thoughts about the relevance that
this has so just in general this topic
is relevant for for artificial
intelligence or AI safety since we are
here asking ourselves what's the optimal
action we are supposed to perform given
some specified goal and this just
generally relevant to AI now there's
also some important macro strategic
question that might hinge on this so
whenever there's a kind of prisoner's
dilemma type situation so whenever
there's maybe an arms race or some
conflict the dispute between evidential
and positive decision theory might
matter
and lastly insofar as this wager pushes
us even if we have a very low prudence
in evidential decision theory or any non
causal theory this wager pushes us
towards this kind of non causal
reasoning and so it's also an argument
in favor of all other considerations
that are based on this non causal
reasoning so for example one piece of
research by
mr. held which discusses so the idea
here is that we based on one course of
reasoning we should cooperate with other
agents in the Montero's or the universe
so insofar as this wager pushes us
towards this non causal reasoning it's
also an argument even for people in
Lausanne Kozma decision theory to take
this argument by Kozma is dialed into
account thank you
okay so in this second half of the talk
I would like to talk about the decision
theory of approval directed agents
so within AI safety there seem to be two
questions related to decision theory the
first is what is the right decision
theory for an AI to use and this is the
question that most AI safety researchers
who have also sought about decision
theory have spent most of their time on
and in a way it's also what philosophers
working on decision for you read and
think about I mean they don't talk about
a is in particular about instrumentally
rational agents in general and I would
guess that a lot of that discussion
carries over to a is the second question
and the question that I would like to
talk about now is the question how do we
implement decision theories and in
particular the right decision theory in
an AI and although this this question
includes this word implements it's still
I guess in a great part a theoretical
question so the reason why this is
interesting and non-trivial is that
decision theories are usually not I
would guess in explicit in AI
architectures so most a AI algorithms
don't sort of explicitly like spell out
which decision theory they use or
compute explicit expected values
according to causal models or something
like that
and instead they're their decision
theoretical behaviors of implicit in
their algorithm so one example would be
if an AI is based on this paradigm of
sort of doing what has worked well in
the past and so this paradigm doesn't it
doesn't compute any sort of expected
values explicitly but you can still talk
about what this is decision theories
this paradigm would implements
a counter example for those who know
about this is Schmidt robust girdle
machine so this is an example of an AI
model that that so specifies the
decision theory very explicitly and you
can you can just you can just change the
decision theory and you don't have to
change anything else about the PAI and I
would guess that most practical AI
designs aren't like that and schmidhuber
scale machine is not a practical AI
design okay now I'm going to introduce
this concept of a cruise directed agency
but first I'm going to introduce the
concept of agency and in particular we
need some model of agency that makes it
possible to describe situations in which
CDT and EDT disagree and so so this is
slightly different from usual models
which just have sort of one probability
distribution and no know provide no way
for cause the decision theory to provide
it to calculate its causal expected
utilities and so we have this agents and
this agent can choose actions and these
actions causally influence the state of
the environment but and they can also
provide some other kinds of evidence
about how the environment looks like so
for example the environment may contain
a predictor who models the agents and
based on what the predictor things the
agent does first boxes with money or not
and so by by taking one box for example
you surf you don't called lis change how
much money there is in box but you gain
evidence about how much money there's in
box or you sort of yeah determine it in
some some non calls away okay so so much
for agency now the approval directed
part so we're going to say that there's
an overseer
which
which looks at the agents action and
also partially observes the environment
and based on these observations it it's
caused the agent's behavior so it
evaluates how how well the agent has
performed and the agents the agents goal
is now to just to just maximize the
score that it receives from the overseer
so this this approval directly ages now
this term is originally due to poor
Christiana but I use it slightly
differently so in particular in import
crystianna's model the overseer does not
observe the the environment at all and
and so one one reason why this
particular model is interesting is that
it's sort of a superclass of
reinforcement darkness so reinforcement
learners are agents who maximize what a
reward module says and sort of the
reward module would would kind of be
like an overseer okay and also I'm only
going to look at some single decisions
because that's just simpler okay um one
one thing that's interesting from a
decision Theory standpoint about this
model is that it that basically these
two these two entities together forms of
a new goal-directed
agents and the goal of that agent is
basically sort of what it's basically
there's a utility function that the
overseer estimates so I'm going to
assume that the score of the that the
overseer gives somehow does something
like approximating like how much utility
there is in the Indigo
like something like that and now these
two together sort of maximize this this
utility function that the oversea
estimates but interesting ly both of
these parts of the agent have something
like the decision theory so the agent
can use cobble or evidential decision
theory to to maximize what the overseer
how did it help
like which score the overseer calculates
and the overseer doesn't so doesn't
choose actions but it it still has to if
it estimates some utility function it
still has to compute some some form of
expected value and it can use either
causal or evidential expected value for
that and I mean causal expected values
is sort of a bit awkward in this in this
context because it's so it doesn't
doesn't properly estimate how much
utility there is but because the
decision theorist might argue that
because the overseers expected value
calculations sort of into incentivize
the agent it should use causal decision
theory to to incentivize the agent
properly okay so so we have these two
decision theories and they're sort of in
this single agent so now the question is
what happens if they're not the same
what happens it for example the agent
news is cause the decision theory to
maximize what the overseer to maximize
the score that the overseer calculates
and the overseer calculates the score
according to regular conditional
expectations so basically according to
Aditi's expected expected value well it
turns out the answer to this question
depends on what information the overseer
bases its expected value calculations on
and I'm going I'm only going to look at
the extreme cases which is that either
it only bases the information on the is
it only basis to calculate
on information that it's get for it gets
from the environment or it only bases
the calculation on information that it
gets from looking at the action of the
agent
so first let's say that it only looks it
only looks at its observations from the
environment and it doesn't have to look
may still see what action the agent has
has decided to take but it gets so much
information from the environment that it
doesn't actually have to look at that
information to to sort of estimate how
much utility there is okay so an example
finally so let's say the agent faces a
regular neucom's problem so the agent
can call the increase its rewards or its
payoff by $1,000 and not taking these
thousand dollars correlates with or the
determines in some I call the way that
there's money in the opaque box or in
the suitcase or whatever and like a
million dollars and after playing this
neucom's problem Omega will just hand
you the suitcase and you can open it and
you can see how much money there was and
you would also get the thousand dollars
in it then look at it and the overseer
can also look at how much money you got
and in this case it seems that the
overseer took to score your performance
just doesn't have to look at your
actions can just it can just look at how
much money you want and then sort of
reward you and like just sort of yeah
basically it is for every dollar you get
one you to learn or something like that
and it doesn't have to think about how
you action like whether your action was
a good idea or something like that it
just looks at how much money you got
so this of course also means that the
overseer doesn't really or there's no
sort of interesting decision for you is
stuff going on in the oversee anymore
because it doesn't look at your action
it cannot differentiate or yeah there's
no difference between sort of causal and
evidentially expected value and in this
in this case sort of the thing that one
would expect happens like if
aegeon users cause with a certain theory
then two boxes and if it if it uses
evidential decision three we had one
boxes so let's say it's uses cause of
the students theory then it was what
reason that by to boxing it can cause
lee increase its reward by $1,000 and
def with like to boxing it can therefore
hardly increase the revolt that they
oversee is seized by $1000 and so it
will causally increase the the score
that the overseer calculates by like the
equivalent of $1000 in Newtons okay so
in the other case the overseer doesn't
get any additional relevant information
from the environment
it just has to make some guess about how
much utility there is based on what
based on the agents action and now let's
take it slight variation of Newcomb's
problem let's say that instead of just
giving you the money
Omega covertly invests the money into
increasing your utility function so for
example omega may save some lives like
with the money that you want in the
Newcomb's problem but it does so in a
way that you can't read about it in the
news and you can't just can't observe
whether like how many lies omega saved
or how much money
Omega invested into saving these lives
and so on and the over the overseer also
doesn't see this so now it turns out
that if the agent uses cause decision
theory and the overseer uses like users
of evidential decision theories regular
conditional expectation then the the
agent could actually to box even though
it's a calls and services so it would
reason that by to boxing it really it
would say two boxes it would the agent
would one box even though I suppose
theorist so by one boxing it could
reason that it's a causally decreases
its of actual payoff like how much money
it will actually get by 1,000 but that
doesn't really matter it would reason
that by one boxing it can causally
influence the overseer to estimate how
much money it got or how much utility
it's got to be higher so in a way and
the the causal decision theory agent
would sort of think about how its of
tricks and the overseer into thinking
that it's got a lot of money while it's
like well the agent itself believes that
to boxing would actually have gone it
the the higher page okay and what we do
in the paper is basically to sort of
show that this pattern holds true in
general that is if the overseer only
looks at the world and doesn't perceive
any additional relevant information from
looking at your action then the agents
the agents decision theory is decisive
for how sort of the this overall system
behaves relative to the utility function
that the the oversee estimates and if
the overseer only looks at the agents
decision and not in the environment then
the overseers decision theory is
decisive for how the overall system
behaves thanks
[Applause]