Well Founded and Human Compatible AI | Stuart Russell

okay welcome everyone
um i have the honor of introducing to
you professor stuart russell he is a
professor of computer science at uc
berkeley and director of chai the center
for human compatible ai he is the author
of human compatible and his book
artificial intelligence a modern
approach is the standard text in
artificial intelligence
he'll be speaking to us today about
provably beneficial ai including topics
like probabilistic programming and
formal verification
after the talk we'll have time to ask
questions
there's a questions tab on swap card
where you can ask questions and upvote
ones you want asked
thank you professor russell
okay thanks very much vivek and i
apologize for the
delay i'm
out at the beach and wi-fi is not always
completely reliable here so it took a
while to get everything lined up uh okay
so i'm going to go through the talk
fairly quickly so we have lots of time
for q a at least some time for q a um
the first part will basically be
a quick reminder of where we got to last
year in the talk i gave
um
about
what we mean by provably beneficial ai
and the approach that we're taking
so
ai in the standard model
which has been
pretty much enforced since the beginning
of the field is that machines are
intelligent to the extent that their
actions can be expected to achieve their
objectives
um so those objectives could be
uh logical goals for problem solving and
planning systems they could be
constraints for
uh csp solvers they could be
reward functions for mdp solvers or
reinforcement learning algorithms and so
on
or less functions for supervised
learning
and
one of the
failure modes for this kind of ai system
is that we failed to specify these
objectives correctly and we call this
the king midas problem because of course
king might have specified his objective
of turning everything he touched him to
gold and that was the incorrect
objective causing him to die
of starvation
so if you take that
uh that failure mode and you combine it
with increasingly capable ai systems
uh then you get exactly the potentially
catastrophic loss of control that alan
turing talked about
in 1951 when he said it seems horrible
that once the machine thinking method
had started
it would not take long to outstrip our
feeble powers and at some stage
therefore we should have to expect the
machines to take control
uh so i think this is probably what he
had in mind as the failure mode
and i think we're already starting to
see it with what's happening
in social media
those algorithms have uh
a well-defined objective from their
point of view
to maximize click-through or engagement
or some other measure of
how much interaction
there is between the human and the
platform
and you might think that in order to
maximize such an objective
uh the best thing to do
[Music]
is to learn what people want
and
send them only stuff they're interested
in because that way they will
they will interact more with it
but that's actually not the solution and
it's uh it's only the solution
uh if you think of people as completely
static objects but of course
uh if you're a reinforcement learning
algorithm then your goal is to maximize
your long-term reward by changing the
state of the environment in this case
changing the state of the person's brain
and so you can get more abort if you
modify people
so that they become more predictable in
future
so this is one possible explanation for
why we're seeing
uh
such uh polarization
and sort of rabbit hole behavior
from
tens of millions of people
and these algorithms of course are
really really simple they don't know
that human beings exist you are you're
nothing more than a sequence of clicks
uh to these algorithms and all they want
to do is to make that sequence more
predictable
but of course if they were
more capable if they did have a better
understanding of human psychology and
belief systems and all the rest
then the outcome would be far worse than
it already is
so this illustrates this idea that
when you have this standard model with
fixed objectives
um making ai better does not make the
outcome better for human beings
and i think
one one way of sort of retroactively
reconstructing this history is to say
that
uh so back in the 40s and 50s when the
field was uh beginning when we were
starting to think about what ai
meant
uh we borrowed this idea of intelligence
that was already
uh fairly widespread in the intellectual
community that intelligence was really
identified with rationality right the
ability to choose actions that can be
expected to achieve
our objectives and that's fine for
humans to be rational
but then when you copy that
over to machines
you get this problem that the machines
don't have
intrinsic objectives of their own in the
way that humans do
so we have to plug them in and that
creates uh the possibility of putting in
incorrect objectives so
i would argue that the standard model is
flawed as a methodology
and instead we want a slightly different
definition we want machines that are
actually beneficial
to us uh not to themselves so to speak
and so um
we want it to be the case that their
actions can be expected to achieve our
objectives the objectives that are
actually in us
uh and not uh in them
and so this
can be turned into
a way of thinking about the design of ai
systems and i'll just focus on
[Music]
the second principle here
it means that the the machine or the
robot is going to be
intrinsically uncertain
about the human preferences that
according to the first principle it's
supposed to satisfy
uh and and this is a crucial change
uh in the way we think about ai
and you can turn that into
a mathematical model
called an assistance game
so it's a game between
a human and a machine
and the way the game is defined
the human has
the preferences or in game theoretic
terms the payoff function
the payoff function of the robot is the
same as the humans but the robot doesn't
know what it is
um
and so
this is a i would say completely
different kind
of ai that we need to do and you get
completely different kinds of solutions
uh the ai system will
in these games defer to human
instruction uh it lasts permission
before
taking actions that it's not
completely sure will be beneficial to
people and in the extreme case it will
allow itself to be switched off rather
than preventing itself from being
switched off which is typically what
would happen in the standard model
um
and we can show that it's
under certain idealistic assumptions or
idealized assumptions it's rational for
humans to build machines that solve
assistance gains
and
so there's still
a ton of work to do but i think
[Music]
i'm reasonably confident that as we do
all that work as we elaborate and enrich
this
framework
and make it closer to being realizable
we'll keep the sec the same property
which is that
the better the ai the better the outcome
for human beings the system will be
better at figuring out what our
preferences are as well as better at
actually realizing them
so there's a lot of
questions that people have asked over
the years and here are
some of them just to save everyone some
time uh aren't you building in one set
of values and whose values are they
and the answer is no we're not building
in one set of values at all um there are
eight billion people the system should
have eight billion preference models uh
some of them will be more fleshed out
than others because
any given system will have more
experience of of some individuals and
some
uh kinds of people and that's uh that's
inevitable
um
uh some people ask well won't it learn
from bad humans to behave badly and
uh no we're not doing imitation learning
uh we're not copying the behavior of the
humans because we we understand that
uh
the human decisions are shaped a lot by
constraints and context and
um what we would like to understand
really is the motivation
behind uh decisions and of course the ai
system uh is also constrained to take
into account the the interests of all
the humans uh and so it can't take a bad
action that will impinge negatively on
others
um isn't it same as imitation learning
nope uh
won't won't people have to do millions
or zillions of demonstrations of
behavior in order for the system to
learn
about what human preferences are
and i think the answer is no
partly because there's already a vast
amount of evidence about what human
preferences are i mean everything we've
ever written
contains massive amounts of information
about human preferences because
partly it describes human actions and
human reactions
to those actions
but also uh it constitutes human action
so the act of writing is an act
uh and uh and that itself is evidence of
of what humans care about
um
a technical point that some some people
raise is well if we're uncertain about
the preferences why don't we just
uh integrate out the uncertainty and and
then we can get
uh you know actions that maximize
expected utility with respect to that
uncertain
uh preference model um and that's
that's correct
only under the circumstance where
uh future interactions cannot yield any
further information about human
preferences
um
and so it's it's sort of a folk theorem
that you might see in some mdp textbooks
uh or or papers about decision making
that say that you know uncertainty about
the reward function isn't interesting
because you can just integrate it out
and replace with the the expected reward
but that's simply not the case
uh for any situation where additional
information can be obtained and that
comes from
the fact that there's a person in the
game
uh
and
some people argue that you know it's too
hard to learn an explicit
preference model the preference model is
too complicated um
and that might be true but it's also not
necessary to learn the explicit
preference model we can have
assistance game solvers that simply
learn what is the optimal policy for the
assistance game
without necessarily explicitly
constructing the preference model as an
intermediary
uh lots of open questions
um we have to figure out uh
you know
the basic preference aggregation problem
that moral philosophers and economists
have studied
for thousands of years uh and the
advanced versions of the preference
aggregation problem such as future
generations
uh people who may or may not exist
depending on the decisions you make
um and various other
quandaries i guess
comparisons of
preferences across individuals
the so-called utility monster problem
and so on
i'm not going to talk much about that
today
dealing with the fact that we'll have
billions of
uh ai systems uh how do we make sure
that they don't have uh strategic
interactions with each other that cause
problems
um
if you're going to learn from
uh from real human beings we have to
take into account the fact that human
behavior
is not a perfect reflection of our own
preferences
so to um
to invert the behavior to get out the
preferences we need to the systems or us
need to learn more about
uh our own cognitive architectures
um
probably the biggest problem from a
philosophical point of view is
the fact that we are plastic in the
sense that our preferences are not fixed
uh obviously we acquire them over our
lifetimes
and they are still subject to
uh
to modification
um so obviously you don't want uh ai
systems to simply modify people's
preferences uh to make them easier to
satisfy which is in some sense what the
social media algorithms are doing um but
you also have to ask questions about
um
whether the preferences of individuals
are are in a sense truly autonomously
acquired
or whether they're inculcated by
a society
for the purposes of some
um
some elite
uh who wishes the populace to have
certain desires uh because that
maintains the uh the social order that
they benefit from
um
foundations we need to
uh if we're going to get rid of this
assumption that we know what the
objective is then all the basic
technologies of ai such as search and
planning and reinforcement learning
uh need to be rebuilt
um and in particular they need
to involve uh some
interactive relationship with human
beings uh through which preference
information flows
uh at runtime and that's the main
characteristic
of the the new model uh compared to the
old model
and then perhaps if we can build
[Music]
applications in areas such as the ones
listed we'll be able to
convince the rest of the field
that this is in fact the right way to
think about ai because you just get
better systems uh better behaved more
robust more reliable uh and and much
less likely to do disastrous things
okay
so so far so good um
but i think of this as
a sort of an outer framework it's not
telling us anything about the
uh the design of the ai systems it's
really sort of a
specification in a in a general sense
these are guiding principles they're not
laws that the ai system itself is
consulting in the way that asimov
uh imagined so
um
i think it's going to be particularly
difficult
if we imagine that um
the kinds of
uh systems we'll build in the future
that are tending towards being human
level general purpose ai are really
black box systems so i'll just use this
acronym bb ai for black box ai
to mean
intelligent systems that are operating
according to some internal processes and
organization that we simply don't
understand and i think that's
uh that's the case now
for
the large
um
the large language models for the uh the
deep networks that are doing computer
vision
uh we simply don't know what they're
doing
and uh we don't know how they work we
don't know how they generalize or if
they generalize
um and i'll talk a little bit more about
some of the doubts that are creeping in
now about those those kinds of ways of
building ai systems so i would argue
that we cannot
uh ensure that that those systems
are going to be conforming to the
principles
that we need them to conform to if
they're going to be
safe
so i'm beginning to think that
actually
part of what the ai safety community
needs to do is actually to put effort
into other ways of building ai systems
rather than trying to sort of
uh
you know after the fact build safety
into these black box ai systems that we
don't understand
to build systems that are kind of safe
in their
the way that they're constructed so safe
by design
uh right down to
uh the individual components
so um so in a sense it looks a little
bit more like the the approach that um
was common
uh before deep learning became popular
uh where we have semantically
well-defined representations of
knowledge of reasoning processes
and decision-making processes where each
component of the system
can be checked individually because it
has semantics you could basically ask if
it's true
what's what's different i think now that
we might not have been able to do you
know in the 90s or early 2000s is that
we can still construct all of this by a
process of machine learning so it
doesn't mean
uh you know abandoning the advantages of
uh building through machine learning but
it means that we're learning different
kinds of things
than they are in deep learning
we can build agent architectures from
multiple
components and we need a rigorous theory
of how those components are composed
together and and the properties of the
the composite uh asian architecture
based on the properties of the
individual components
and then just generally i think
there needs to be much more emphasis on
a rigorous formally verified
[Music]
software stack and i this is not just
for ai i think this is for the entire
digital infrastructure of the world we
are we are woefully vulnerable uh to all
kinds of failures there
um so
i think there's there's a
a kind of
fault tree analysis that one could do um
to try to figure out you know
is it better from the point of view of
long-term ai safety
uh to put an effort into well-founded ai
systems and there's one
possibility which would say you know
actually it's not
uh which is that if black box ai is
guaranteed to fail in other words if it
could never produce
uh transformative or human level ai
systems um then
why don't we just let it continue
uh and from the point of view of ai
safety that would be ideal because then
we'd never have anything to worry about
um
so
i
i think that's uh that's a possible
argument and but it's not what i'm going
to talk about today i'm going to just
say suppose that
we do need
uh well-founded ai in order to have
systems that we are confident
are safe and beneficial so let me just
briefly mention some
reasons why i think that black box ai
uh may actually
uh not get us to human level ai um and
so even if that's true right what
inevitably what would happen then is it
hits a wall um
and people cast around for other ways of
keeping progress and ai going
um and i think i guess i'm arguing we
might as well get going on that now
the uh the first reason is that um if
what you're doing is training circuits
there are many many concepts such as the
rules that go which are
very hard to represent in a circuit
because circuits don't have the ability
to to basically express universal
quantification
uh for example over the squares over the
pieces over the time at which the event
takes place
and so
um
so that that means that the process of
learning those concepts
uh is extremely slow requiring millions
billions of examples instead of tens or
hundreds
um
[Music]
a second point is that
um
one interpretation of of adversarial
examples which are examples where small
perturbations to inputs cause
deep learning systems to change their
mind completely about the category
for example of an image
one one hypothesis about why that's
happening is in fact that
uh the the deep convolutional networks
are not actually learning
a
generalizable uh interpretive process
for the image
and the dimple manifold conjecture from
um adi shamir
says that what seems to be happening
is that in fact they're learning a giant
lookup table
that's
constrained to operate in
the manifold of
naturally occurring images
so you get some generalization
for example if you make a small
if you make a small movement parallel to
that manifold you're going to get a
similar image that also looks natural
you're not going to open up a big hole
uh in in the object uh it'll be more
like a rotation of the object or a
movement of the object or a change in
the color of the object
so
so this conjecture seems to agree
quantitatively with a lot of the
phenomena of adversarial examples and it
would suggest that
that deep learning systems
at least in the area of vision are
simply not doing the right thing at all
and then there's other sort of more
practical
anecdotal evidence right so so a group
at mit
looked at a number of vision systems on
imagenet data
and so
uh in the left-hand column for example
this is the origin the top left is the
original image of a parachute
and then the red pixels are the pixels
that the classification algorithm is
paying attention to
in order to decide that this is a
parachute
uh and so you can see that actually it's
not looking at anything other than the
sky
uh you know if you look at the third
column there's catamaran again the red
pixels are not on the catamaran
[Music]
and so
if you look at the the golden retriever
right it's looking at the grass to
decide that this is a golden retriever
and presumably that's because of a
regularity that exists in the training
set and the test set
um
that of course wouldn't hold in general
in the in the
real world
[Music]
and so this
this tendency to just find spurious
regularities that have absolutely
nothing to do with recognizing the
category
uh is something it's not the algorithm's
fault right it's just
our
over-enthusiastic interpretation of the
fact that the algorithm does well on
on the training sets
and you know another set of problems
uh there's a paper from google and this
is a summary of that uh of that people
from google showing that um 30 of their
big deep learning
applications that were fielded turned
out not to work
once they were fielded even though
in all of the training and testing that
they did
it looked like it was working great
and similar problems have happened for
example with the skin cancer apps the
great deal of who are about skins cancer
detection
those apps have all been withdrawn from
the market because they just don't work
uh and francois charles who wrote the
deep learning uh textbook
uh admits at the end of the book like
many more applications just completely
out of reach
uh for deep learning techniques and he
suggests that
instead we need models which are closer
to general purpose computer programs
so that's what i want to talk about next
there are models that are closer to
general purpose computer programs called
probabilistic programs
uh and they date back to the late 90s
some work that daphne caller and avi
feffer
did
uh in my group along with david
mcallister
from uh from toyota technical institute
um and here's a very quick overview i'll
give you an example in a second they
combine probability theory
with some universally expressive formal
language which could be a programming
language or it could be
something like first order logic
and the nice thing is that you get that
expressive power so if i want to write
the rules that go in one of these
languages it's one page
instead of a million pages
um
[Music]
and uh they're universal so we can
represent concisely any probability
model that can be represented in any
formal language
and
we can have general purpose
inference and learning algorithms so for
any model any data any query in
principle subject to constraints on
computation time
uh we we get that kind of generality so
it feels like it has
the sort of wherewithal for general
purpose intelligence even though there
is still a long way to go uh in terms of
making this practical
and they give you this ability which i
think is really important to combine
prior knowledge and data in a way that's
cumulative
so
we all know that if you're able to
supply prior knowledge to a learning
system
you can get learning to happen from far
fewer examples
um and then out the other end you get
some new knowledge and what you would
like is that new knowledge
then can be fed back
uh into the whole process so that you
get this cumulative development just as
we have in science
where
you know we can now do things like
detect the collision of two black holes
on the other side of the universe
because of hundreds of years of
cumulative acquisition of concepts and
physical laws of knowledge and technique
to make that possible
so ppl if you're not familiar with it um
really
was very quiet for the first
10 years or so um and then started to
take off
um
and now is up to as of 2020 at least
according to google scholar about 2
500 papers a year there are
significant groups at microsoft facebook
google
and a bunch of universities around the
world
i'll just give you one example which is
monitoring for the nuclear test ban
treaty so that takes
data from a network of sensors mostly
seismic but also hydroacoustic and
infrasound all over the world
uh and about two terabytes of data per
year coming in across the satellite
network to vienna
and this is some of the what a typical
seismic signal looks like
and then you're trying to
collect all that data and figure out
what are all the seismic events
uh that have taken place in the world
where are they
uh how big were they how deep were they
and which ones might have been nuclear
explosions
and that's called the daily bulletin
that we produce
and so the evidence is all this uh all
this data coming in so it might it's
mostly seismic
the query is what happens so what events
took place in the world
and the model that we can write in the
probabilistic program
uh describes the elementary geophysics
so so what any seismology student knows
about geophysics
we can write it down
in the form of a parabolistic program
and there it is
and this took um
i wrote it during the lunch break
in a meeting i went to in vienna
and it works about two or three times
better
than
the uh the then
uh
official united nations
uh global seismic monitoring system
which is sort of the result of 100 years
of seismology research
so i'm not going to go through it in
detail but the point is that
we can write very
large scale models so this
this model in real time is doing
probabilistic influence with hundreds of
thousands of random variables um
and uh and it and it runs on a laptop
so it doesn't require
you know tpus a tpu is about 10 million
laptops
and so
um
the
the computation requirements often
people say well well probabilistic
programming doesn't work because it
needs too much computation
uh this this actually doesn't need that
much computation uh at all
uh and it works as i said way better so
this is
a 2013 uh explosion in north korea that
we detected in real time and also got a
much more accurate location
compared to uh the
the assembled geophysicists
at the test ban treaty organization
another example right we can do things
like real-time video tracking
um
and this is the
this is the
sort of the standard
uh technique
for most of the last 20 years
called adaptive background mixture model
which
represents each pixel as being drawn
from a gaussian mixture
of either background
foreground or shadow and and then
uh
with every new image you basically
update your mixture of gaussian model
and you classify all the pixels and then
that gives you separation between
foreground background allows you to
track moving objects
um so you can write that this is
basically that model written in a
probabilistic program
um but we can actually the nice thing is
that so first of all you don't have to
do any math right so you you write this
model and the
the inference engine does all the math
for you
right so you don't have to do a whole
phd
for each new kind of parabolistic model
that you want to deal with
you just write you write the code and
everything else is taken care of so for
example if i want to change it to be
a much more sophisticated model where
um
the category that each pixel belongs to
foreground background or shadow
rather than you know being
drawn independently from the gaussian
mixture at every time step
actually has its own internal
persistence so once you're in shadow
you're going to stay in shadow for some
number of frames and then you're going
to flip to foreground and stay in
foreground for some number of frames so
kind of a hidden markov model for every
single pixel well that's all you do
right you add one line to the model
and and you're done right the inference
and the learning all happens
automatically
and then we get a really good
performance so this shows that we get
much better
uh tracking of objects with much less
artifact than the opencv
computer vision library
so uh to make this
um
[Music]
practical for you know as a basis for
provably beneficial ai
we need to do a few things one is
we need to
go from probabilistic programs that are
just representing probability models to
probabilistic programs that are actual
fully fledged agents um operating in
assistance games
um so stage estimation in fact the the
example i just showed is a form of data
state estimation so keeping track of the
state of the world over time that's the
core of an agent in my view
um
and then on top of that you can add
various forms of decision making
and to do that we need to extend ppl's
with actions and rewards
and
this actually uh creates some quite
interesting and complicated
uh
fuzzy philosophical problems somewhat
similar to what happens when you extend
logic to talk about uh knowledge you get
modal logic
and you get various kinds of um
referential opacity issues
so here for example if i have an action
if i have a first order language i can
have an action like
assassinate the leader of spectre
right but when you have uncertainty over
the identity of objects then that action
is not doesn't have a unique referent
so it's not just that you're not sure
who it is but you're not even sure
what that actually means given that you
don't know who the leader or spectre is
um and so the uh
solving those kinds of problems i think
is you know we have some ideas about how
to do that
um
and uh we can we can develop that uh
framework extending ppls with actions
and rewards and i think
the the aspect
uh that you need for assistance games is
uncertainty over the reward functions
um
and i think that's actually a fairly
natural fit
for
uh for probabilistic programming
there's a formalism called cpnets
caterous parabolas nets
that was developed in the 90s that's
sort of a bayes net like representation
of preference structures
uh or
or you know think of it as reward
functions or utility functions
um
and then you know lifting those to a
first order language and then
allowing for uncertainty over those
preferences i think is something that
we can
we can probably do okay in probabilistic
programs
the second part we need
is a real theory of agent architectures
right we i we can't imagine at least i
can't imagine that
um the kinds of systems we will want to
build as we start to approach human
level ai will be a single large
black box right just one thing
uh and that we train it to be superhuman
at everything i just don't see that
um it will have uh multiple components
connected in
um in ways that uh hopefully will
combine to produce uh safe beneficial
intelligent behavior
but at the moment
we don't have a good theory of
of agent design
if you
if you go and look at places where they
actually build real agents
for example the mars rover is a real
agent self-driving cars or real agents
and so on
they have lots of boxes and lots of
arrows joining those boxes together
and it's very common for example to talk
about three layer architectures for
uh for physical uh intelligent systems
where the lowest layer is the
uh sensory motor control so you know the
servos that make sure your wheels turn
at a constant speed and so on and so
forth right now all the way up to the
high level so
um why is that good
no one knows
right it's just evolved over over time
uh where
people say well i've been doing this for
30 years and i'm pretty sure all my
boxers narrows are better than your
boxes and arrows
that's not a very satisfactory
foundation
and there's a theory called bounded
optimality which i think may offer a way
to have a better answer to that question
so very briefly
uh this is a quick reminder
of that idea uh so the first concept
that we need is an agent function which
is this you know a standard concept in
ai right the function that maps uh from
a sequence of inputs like here we're
playing tetris so this is the sequence
of screens you observe
uh with objects appearing and new
objects appearing
um
so you go from a sequence of
observations
to
an action and
that's what we mean by the agent
function
that agent function
is running on a machine
so we fix the machine
and then we choose some program uh that
l drawn from the language that that
machine can support
uh and then the running of that program
on the machine produces an agent which
has agent function f
and um
you know interestingly
not all agent functions are
computationally feasible uh many agent
functions for example an agent function
that solves the halting problem
doesn't exist on any computable
device
but for any given machine there's a set
of functions that are feasible
which is basically all the functions
that can be produced by any program that
runs on that machine and then the
concept of bounded optimality is
basically uh just is to say that the the
best agent you can produce is the one
that's running uh
the the best of the feasible
programs that can run on that machine
and so that depending on the
memory size and speed and so on of the
machine that that may be an extremely
stupid program
or it may be extremely brilliant if if
this is a machine with lots of resources
and then we can weaken that concept
slightly
to get something called asymptotic
asymptotically bounded optimal
programs
by allowing a slight uh you know as we
do with big o notation right a a
constant factor
uh slowdown
[Music]
so here what we say is that
l is
asymptotically bounded up to more
if the value of l
the agent produced by l running on a
machine that's k times faster
is at least as high as the value of the
optimal program running on the original
machine m
and so this gives you a much more robust
notion so it's much less dependent on
the details of the program and the
details of the machine
and it's this notion that allows us to
get
composition so here's one very simple
example right suppose
that you're operating in
uh environments where there's a deadline
where you have to act by a certain time
and
let's suppose that if i know the
deadline in advance i know how to
construct an program for that task
so let's say
i have exactly
10 seconds to
decide on the answer to a question
um
and let's imagine that i can write the
 program for any fixed deadline
um and
let's let li be the program for a
fixed deadline at 2 to the i times
epsilon and we'll see why that matters
in a second
right
and now suppose that we don't know the
deadline
right
can you make a real-time decision-making
system for an unknown deadline out of
these components and the answer is if
you
if you simply string together the
programs
for these uh exponentially increasing
uh fixed deadlines and you string them
together in this sequence so you start
with the shortest one and then the
second shortest one and so on
then you can show that that whole
program
is asymptotically bounded optimal for
any deadline distribution so
effectively it's as good as if you knew
the deadline in advance so that's a very
simple example of how to make a
composite agent architecture out of
simpler agents uh and still have uh a
rigorous theorem about the properties of
the composite agent so that's what we
are aiming for
um
unfortunately
yeah we are out of time we need to get
to the next presentation as well
okay well
um
town i i don't remember
uh yes i will go i'll go togethertown
just uh if you send me the link
uh okay uh wonderful um last bit
uh
learn about formal methods because they
are real and practical and we don't use
them nearly enough
um
and that's
that's the summary so basically arguing
that uh
the ai safety community needs to think
about
a well-founded approach to building ai
systems
uh and these are some possibilities for
how to do that
wonderful
thank you so much professor russell um
and yeah we will be on gather town um
so
uh yeah can we get
uh someone
from production maybe to drop the link
in the chat um
so yeah if you if you join us on
gathertown we'll have q a um thank you
so much
you


Provably Beneficial AI and the Problem of Control

hello everyone
um my name is rohan shah and i'm going
to be the emcee
uh for this session um on
my stuart russell uh so i'm excited to
introduce you to him
uh i know stewart best as my advisor uh
when i was doing my phd at the center
for human compatible ai
but of course he also has a you know
real academic bio
stewart is a professor of computer
science at the university of california
at berkeley
holder of the smith sudden chair in
engineering
and director of the center for human
compatible ai
he is a recipient of the ichikai
computers and thought award
and from 2012 to 2014 held the
chairblaze pascal
in paris his book artificial
intelligence a modern approach with
peter norvig
is a standard text in ai used in 1 500
universities 135 countries
he is also author of human compatible
his research covers a wide range of
topics in artificial intelligence
with an emphasis on the long-term future
of an artificial intelligence and its
relation
to humanity he has developed a new
global seismic monitoring system
for the nuclear test ban treaty and is
currently working to ban
lethal autonomous weapons uh today
stewart will be discussing the problem
of creating provably beneficial
artificial intelligence
arguing that the standard model for
developing ai poses major risks
we invite you to use the swap card
platform during this session
on the right side panel feel free to
share your thoughts on the live
discussion board
and leave and upload any questions for a
speaker to answer at the end of the
presentation
after the session we invite you to use
swap card to set up meetings with fellow
conference attendees
stewart will also be hanging around
afterwards
to answer questions on gather town
and without further ado uh welcome
stewart
thank you very much rohin so rohan gave
a pretty good summary
of what i'm going to say um i understand
that this is a broad
audience interested in uh all kinds of
risks so
let me first of all explain uh about ai
so of course ai is about making
intelligent machines
uh the question is what does that mean
um and historically what it's meant
uh and i'm going to refer to this as the
standard model
uh is making machines that
are intelligent to the extent that their
actions can be
expected to achieve their objectives
and this borrows the standard notion of
rational behavior from philosophy and
economics where it
was developed over several centuries
ai has lots of subfields uh problem
solving the kinds of algorithms that
get you to the airport when you use your
gps navigation system
uh game playing the kind of program that
that beats human world champions
knowledge representation
reasoning planning to allow a system
that knows something
to convert knowledge into action uh
natural language processing
uh speech vision robotics these are all
obvious and then
machine learning is the thing that's
mostly in the news these days
it's actually been around for a long
time in fact in 1950
alan turing recommended machine learning
as the most likely route
to build uh human level intelligence
systems
so uh as you know it's a huge field
there are
tens or perhaps hundreds of billions of
dollars being invested
into developing improving ai
and it's a very rapidly growing field
there's enormous demand for it a little
factoid i think almost two-thirds of all
applications to our entire department in
berkeley
are specifically to study artificial
intelligence
so given all this uh amazing level of
energy and effort
what happens if we succeed in our goals
well we can go back and ask uh alan
turing here he is
um and in 1950 he was pretty optimistic
he talked about
the prospects for all kinds of things
that you could do with ai and how
wonderful it would be
by 1951 um he was talking on the radio
and he said
it seems probable that once the machine
thinking method
had started it would not take long to
outstrip
our feeble powers at some stage
therefore we should have to expect the
machines to take control
so there it is right no no mitigation
no solution no apology
just resignation uh this is the future
of mankind
um so
fast forward uh 70 years and you know
we're starting to see
some of the technology existing we have
self-driving cars we have programs that
can beat human world champions at
really complicated games like go
and we're also starting to see some of
the downsides
um racial and gender bias in algorithms
uh the expansion of disinformation
uh the impersonation of human beings
so i'll ask you guys later which of
these four
is actually a real person
using ai systems to kill people
replacing human roles in the economy
with machines but there's also an upside
right there's a reason why we're
spending all this time and energy and
effort to create ai it's not just
for the fun of it and it's not just as
lord lighthealth said in his 1973 report
that it's a bunch of male scientists who
are unable to have children
and therefore they want to create ai
instead
it's actually because ai can have
enormous benefits
so to illustrate what kind of benefits
it could have
we could actually sort of go back in
history and say
what was it like getting to australia
um in 1800
uh well it would take you
hundreds maybe even thousands of people
a fairly hefty project
uh to uh outfit
a major expedition probably take you
about 10 years
start to finish and you'd probably be
dead before you got there
um but now because of the advance of
technology we have what you might call
travel as a service right it's just as
we have
you know electricity as a service water
as a service
we have travel as a service you take out
your phone you go tap tap tap
and you can be in australia tomorrow
and it costs you uh not a billion
dollars but a thousand dollars or
in other words it basically it's
basically free in relative terms
and you're going to be alive when you
get there that's if they let you in
um so what human level ai promises
is that kind of improvement in
everything so xaas means everything as a
service
and we can expect the same type of cost
reduction not necessarily the same
speed improvement because there are
limits
in physics and biology and and also in
the human mind
but we can eliminate cost and uh
essentially use
ai as a source of wealth so if you
wanted to run
uh you know this conference in 2035
you would just ask your laptop to do it
and it would take care of it it would
set up everything it would invite all
the right people
um we make sure that the environment was
great that everyone was meeting each
other
uh and it would be wonderful um you know
if you're
in some remote uh village um where you
don't have access
to government services uh you can just
ask the ai systems to come along and
build some houses and schools and
maybe a road connecting you to the
nearest city
uh and teach your children and if you
need surgeons perhaps train the surgeons
as well or even be the surgeons
um and you shouldn't think of ai
the kind of ai that will be doing this
as uh individual robots
um most media portrayals and movie
portrayals
have the ai embodied in a single
physical object but it's going to be
as people say now in the cloud it'll be
a globally connected
uh essentially a single system and it
will have
physical extensions that it can deploy
whenever they're necessary and they will
have
whatever physical characteristics are
necessary
whether it's for operating inside a
house or moving at high speed with wings
or
carrying enormous loads with wheels and
it will carry out
essentially whatever tasks we know how
to do
and perhaps even some that we don't and
so
in a very conservative forecast of what
we could do
with this kind of technology um not
inventing cures to new you know new
cures to diseases or faster than like
travel
or you know life extension or these
other sci-fi things but just things we
already know how to do
but just doing them effectively
efficiently
and at almost zero cost we could lift
the living standards of everyone on
earth to a respectable
level um and
at least well i ted parson the previous
speaker is probably
going to disagree with that because we
might not have the resources to do so
but let's say that we do this
efficiently it's about a 10-fold
increase in the gdp of the world
which is about a 13.5 quadrillion dollar
net present value so the cash equivalent
of the
increase in global income uh would be
about 13.5 quadrillion dollars so that's
a lot of money
and it makes the tens or hundreds of
billions of dollars that we're
investing uh absolutely
negligible in comparison right we're
we're not even
it's like spending a penny to buy a
house
um and if this technology becomes
available i think it would have some
uh globally beneficial effects
on the way we relate to each other
because um
there should at that point be no need
for conflict over
wealth because everyone would simply be
able to make more of it when they need
it
just as you can make more digital copies
of a newspaper
and so we don't fight over who has more
digital copies of the financial times
because if you want another one you can
just make another one
so i think we have to accept
that ai systems are going to be more
capable than humans in the future of
making decisions
in the real world not just on the go
board
but if we gradually relax through
through research advances
we relax the assumptions that may go
easy
such as complete observability of the
state of the world
completely predictable rules discrete
state and so on we can relax these
technical assumptions
by developing more powerful algorithms
and eventually we have systems that can
out think us
in the real world and what turing is
asking
is okay if you do that then you have
systems that are more powerful than
human beings because
our decision-making capacity is what
gives us power over the world
so if you're making entities more
powerful than ourselves how are you
going to retain power over them
forever and touring obviously sees no
answer to this question
and that's why he's resigned to a future
in which machines take control
now it seems reasonable actually to ask
is there any way out right short of
banning ai
altogether and turing actually went on
to refer
uh to butler's uh book erawan
uh in which they have banned machines
for exactly this reason
and for those of you who've read dune
you know that there was a butlerian
jihad so that refers back to butler and
this jihad was to destroy
all machines and they added another
religious commandment
saying thou shalt not make a machine in
the likeness of man
so they don't have computers in dune
because
as soon as you start having computers
you start making them more intelligent
and you go down
the slippery slope uh and you almost
lose control
it's happened in due so um
i actually think that it's not as bad as
that
i think there is a way that we can avoid
the fate that turing is predicting by
understanding where things are likely to
go wrong
and one place one one source of the
problem
is in the standard model of ai the one
that i
talked about earlier and it's not just
ai right this
this same approach where we design
machines that optimize a fixed
specified objective on our behalf that's
what happens in control theory
where you minimize the cost function in
statistics
you minimize a loss function in
operations research you maximize
a sum of rewards in economics you
maximize
gdp or utility or social welfare
so this is a pretty powerful and pretty
widespread
model for how to do things
but it's a mistake because
uh once we get outside the lab or
narrowly constrained systems
uh we don't know how to specify
objectives completely
and correctly um
so a simple example the self-driving car
companies
uh right now um are trying to figure out
what the objective is that their car
should be maximizing
and uh it's still subject to revision
and they've constantly finding places
where they need to fix it up
and that's for a system that can only
move a steering wheel
and the brake a gas pedal on the brake
pedal right doesn't have
access to a keyboard or anything like
that and so it's
very restricted
now we've known this point about the
difficulty of specifying objectives
completely and correctly for thousands
of years
and almost every culture has legends
like this or myths like this where
uh here's king midas and king minus
specified the objective everything i
touch should turn to gold and his
objective was
granted because the gods being the gods
they gave him exactly what he asked for
so there the gods were playing the role
of the ai system
and then of course his food and his
drink and his family all turn to gold
and he dies in some versions in misery
and starvation
um in goethe's story the sorcerer's
apprentice right the apprentice
uh asks the rooms to fetch water but he
forgets to say how much water
um and then of course you know the whole
house fills up with water and
the sorcerer has to intervene and if you
uh ever get one of these magic lamps
where the genie grants you three wishes
um in those stories typically the third
wish
is please undo the first two wishes
because i made a mess of the universe
but what happens if we don't even get a
second wish
right um so
uh the the literature on ai safety is
full
of examples of ways that things could go
wrong
um you know you want to restore carbon
dioxide levels in the atmosphere
and you you know the system does so
perfectly successfully um in a way that
wouldn't make ted parsons happy
um because it reduces the level of
oxygen the atmosphere by 25 percent and
and we all slowly die of asphyxiation
et um et cetera right um
and uh max tegmark's book life 3.0 has a
very nice
uh it's a prologue or preface
um which describes uh a great length
a process um by which with the
cooperation of human beings
a super intelligent machine gradually
takes over
uh our entire planet
so um if we
follow the standard model of ai and we
specify an incorrect objective
then we are actually kind of setting up
a chess match between
us and the machine right because we've
got us
with our actual objectives and then
we've got a machine with a different
objective
and if that machine is more powerful
than us and it's pursuing this objective
uh which is misaligned then it could be
you know we could think of it as one
machine
or you could think of it as some
globally distributed system
that's set up uh to optimize a given
objective
um then
basically we don't want to be in that
chess match
um and i think it's actually starting to
happen
um one example is what's happening in
social media
where the algorithms are set up with an
objective which is something like
maximizing click through
or engagement or various other measures
that um appear to actually align
the interests of the user who wants
interesting things to read and look at
and the company who once clicks because
clicks generate revenue
and so i think when this was designed
initially the idea was that the
algorithms would learn
what people want what they're interested
in and we send it to them and that
sounds
sounds okay uh and people started to
talk about the filter bubble that
of course if it only sends things you're
interested in
uh you you stop learning about things
that are outside your
uh your bubble um and perhaps your
interests uh
gradually narrow over time but actually
this isn't
what happens when you
ask a learning algorithm to maximize
click through what it does
is it comes up with ways to manipulate
people
through the sequence of content that it
sends you
so that in future you are more
predictable because the more predictable
you are
the more it can maximize click through
and so it doesn't want to leave you the
way you are where you're
you're unpredictable and you have wide
interests it actually wants to change
you
into a more predictable and typically
more extreme
version of yourself uh in order to to
monetize you
and i think many people argue that this
is one of the major ills affecting
society today
and you can see that these are very
simple algorithms having already
a massive global effect because of the
way they're deployed
on billions of screens if they were
better algorithms if they were
actually
able to think about your interests
rather than just treat you as a click
stream
then they could do far more damage
[Music]
and
if you um
if you imagine that the ai system
also gets better at generating content
that is able to
uh manipulate you um then the outcome
could be far worse
and this is the general property of ai
systems built in the standard
model that if they're pursuing an
incorrectly defined objective
they will be better able to achieve it
they will be better able to mess with
the parts of the world
that are not mentioned in the objective
in order to optimize
the achievement of the objective and the
probability of
success and they will be better able to
prevent
humans from interfering with that
process
so it looks like this is a methodology
where success
means failure and that's a good sign
that the methodology
is probably the wrong one
um and i think this the the story and
this is sort of a you know a very very
simple historical reconstruction
of why we got into it is because uh
when we were setting up the field of ai
in the 40s and 50s
it was natural to think okay what do we
know about humans that makes them
intelligent what is our
sort of scientific definition
intelligence in humans
um and one one version of it was
the sort of psychological or cognitive
science definition
that just said let's copy actual human
cognitive processes
but the one that won out in the field of
ai is
uh let's build ai systems that are
rational because that was uh that's a
scientifically
uh definable or mathematically define a
notion that we can actually
constructively pursue
um so we took that definition of
intelligence in humans and we just
copied it to machines and i'm arguing
that that was a mistake
so what do what should we do instead
well uh the problem is
uh the problem with this definition uh
is that
we are not able to transfer our
objectives
correctly and completely into the
machines if we could
i wouldn't be too worried about this um
but since we can't do it then we're
stuck with something else
right we're stuck with making machines
that are beneficial to the extent that
their actions can be expected to achieve
our objectives uh and so our objectives
are going to remain in in us uh but the
machines
have to be beneficial to us according to
those objectives uh
and this is almost a truism that this is
in fact
the type of machine that we should build
right it is rational for us to build
machines of this type and not of the
uh of the other type
and um and we can turn that simple idea
um into some principles you know three
principles because
it's uh you know maybe it's either as
most birthday or something but
the three principles are first that the
machine's only objective
is the satisfaction of human preferences
and i'm using preferences here in the
in the same sense as von neumann and
morgenstern uh meaning
uh your ranking over all possible
futures or in fact over lotteries over
all possible futures not just what kind
of pizza you like
but your your preferences about the
entire future
but the second principle says that the
machine does not know what those
preferences are
and this uncertainty turns out to be
crucial
the third principle provides a kind of a
grounding
for this whole notion of preferences
that connects
humans to machines and how is the
machine going to find out anything about
what our preferences actually
are the answer is
from human behavior that that is the
primary
source of evidence about human
preferences there could be other things
for example you know direct fmri
measurements and that way you could
actually
get preference information from
locked-in patients for example which
might be helpful
but in general it's going to be from
human behavior or
obviously the consequences of human
behavior
so you can take those three principles
you can formulate them as a
in a mathematical framework which we
call assistance games
um so it's a game because there are at
least two decision-making entities and
economists call decision problems with
more than one entity
games there's at least one human at
least one machine
it's an assistance game because the
whole purpose of one of the entities
namely the machine
is to be of assistance to the other
entity
and these uh assistance games have a lot
of interesting
structure uh i'm not going to go into
any
mathematical stuff here uh because of
the nature of the event in the audience
i'm happy to do that
in gather town afterwards but when we
solve these games
and we've actually solved a few of them
simple ones but
we can look at the solutions and
understand the properties
of the behavior that is exhibited by a
machine that is solving this
type of game and we find that it will
defer to the human
uh it has an incentive to ask permission
before doing anything that would impinge
on aspects of human preferences that it
doesn't know
right so for example if it knows that we
want to restore carbon dioxide levels
to the proper amount but it doesn't know
how much oxygen we want in the
atmosphere and it comes up with a plan
that
changes the amount of oxygen then it has
an incentive to ask
our preferences about oxygen levels okay
and it won't do anything before finding
out enough about those preferences
to be pretty sure or almost completely
certain
uh that its plan is something that's uh
acceptable to humans or preferred by
humans
uh and crucially it allows itself to be
switched off
which is not true of ai systems that are
built in the standard model so i'll go
i'll talk about that in a minute and as
i mentioned
we can show that it's rational for
humans to build machines
that solve assistance games if we
were able to write down our preferences
completely and correctly
and put them in machines it would be
rational for us to build machines
in the standard model but we cannot and
it's not rational
for us to build machines in the standard
model but i believe it is rational for
us
to build machines within this model um
and the bet in this model
it fixes what seems to be wrong with the
standard model in the sense that
as the ai system becomes more and more
capable
it becomes better at learning about our
preferences and better at satisfying our
preferences
and so we get better outcomes
so let me just illustrate this off
switch problem that i mentioned
right here's our robot there's the off
switch on the back
um and this is the big heavy robots
about 400 pounds
and that's why it has an off switch and
in the classical way the standard
model for ai right if you give it
an objective like fetch the coffee all
right
um that's becomes its life's mission
right and um
it doesn't take a genius to figure out
that you can't fetch the coffee if
you're dead
and so the robot uh instantly as a
result of being given
uh an objective that is not easy to
achieve when dead
now has an incentive to preserve its own
existence so this is not built in
this is simply a consequence of being
given the objective in the first place
and so since it wants to avoid death one
way to do that is to disable its off
switch
and possibly take other pre-emptive
measures
to stop any interference with this
mission of fetching the coffee
so this is what we want to avoid right
we want systems that do allow themselves
to be switched off so
when the machine is uncertain about the
objective so it might know that i prefer
to
have some coffee now um but it may know
uh very little actually about the rest
of my preferences
so the thinking goes quite differently
the human might switch me off
but only if i'm doing something wrong
right and the point here
is the machine doesn't know what wrong
means right so it doesn't know which of
our preferences it might violate
but it doesn't want to violate any of
them and so it has an incentive
to allow itself to be switched off in
order to prevent
preference violation right the first
principle is basically telling it
what the the dual of the first principle
is avoid preference violation
right and so it has a quantitative
incentive
to allow itself to be switched to be
switched off
and you can actually very simply prove
this
as a theorem by writing everything down
in greek and then you get a theorem
that the robot is provably beneficial
and
provably has a positive incentive to
allow itself to be switched off
as long as it remains uncertain about
the human preference model
and when that uncertainty goes away
the incentive to allow itself to be
switched off also goes away
[Music]
so i promise not to go into a lot of
math
but i want to just head off a lot of
misunderstandings
so we don't waste a lot of time in the q
a i'll i'll ask some of these questions
and then i'll answer them uh right now
so
a common response i get is well you know
who are you you know you uh you white
cisgender
cisgender episcopalian affluent western
male
uh to uh to determine the values that ai
systems are going to
uh be optimizing um or you know are
you gonna build in christian values or
you know some all kinds of variants of
the same
uh the same basic idea that we're
building in a set of values into the
machine for it to optimize
um and the answer is
no we're not uh we're not building in
one set of values at all
in some sense there's no set of values
beyond that the machine
is there to be of benefit to humans
um and uh it will have potentially you
know
if there are eight billion people 8
billion different preference models
about what the future should be like
another question is won't the robot
learn from bad humans to behave badly
and uh the answer is no
uh no more than a criminologist learns
to be a criminal by observing criminals
uh so uh bad people or
people we think of as bad uh because
they do actions that are harmful to
others
they have their preferences and
the only exception to this rule is the
the pure sadist
right the one who takes actions that
have the cause harm to others
simply in order to derive pleasure from
the harm inflicted
right harming others in order to
you know obtain money
in order to you know buy yourself
um a catamaran right
that's a motivation um and one can
separate out the
the positive preferences that the agent
has what they want
in life from the negative effects on
others and
uh the the ai system won't uh help
uh the assist the the human uh inflict
negative effects on others
because that's uh that's not what it
does um
but we do need to zero out the
the sadistic preferences of humans
and this is a long discussion in the
theory of utilitarianism
and there seems to be a pretty good
consensus that that's one way of
treating it
in in machine learning and reinforcement
learning there's a whole subfield called
imitation learning where
humans do things and then machines try
to copy them
uh you know might it might be gymnastics
or
uh walking or uh anything like that
um and uh so some people get confused
and think that well
this you know because i'm putting a
machine and a human in the room together
uh and the human is going to be
demonstrating something the machine is
going to
copy it um no not at all
it's not the same um and just to give
you two examples right if the human uh
in the room drinks coffee right i don't
want the machine to drink coffee
right that's not the goal here right in
fact that
what should happen is that the machine
you know at the appropriate time fetches
the human some coffee to drink
or help you know brings it to them in
bed in the morning or whatever it might
be
uh similarly if if the human says i
would like a cup of coffee
right we don't want the machine to say i
would like a cup of coffee
right that's what imitation learning
would do and that's completely not what
we're
what we're setting up here so there is
some
you know family resemblance to imitation
learning but it's actually a completely
different problem
some people argue that uh this approach
can only work if people
are doing just zillions of
demonstrations all the time for
everything
that they ever want the machine to do um
and uh
the answer is no in fact they might not
have to do
any demonstrations at all and and part
of rohan's thesis
was actually showing that you can learn
a great deal about human preferences
simply by
observing the state of the world and we
call this or i call
it the non-naturalistic non-fallacy
um because the world is not sampled from
a state of nature
right the world is the result of the
actions of
billions of humans
operating on their preferences and so
by looking at the world we can learn a
lot about what those preferences must
have been
for all those people to do all those
things that resulted in the present
state of the world
so you might not have to have any
demonstrations you can also
read all the books that humans have ever
written watch all the movies
uh and learn tons and tons and tons of
stuff and babies do this
all the time right i mean each human has
learned
uh their own set of preferences from
a relatively small amount of experience
and a lot of it i think is actually
explaining why
the other humans around them are doing
the things that they're doing
um some people mathematically oriented
uh think well okay if there's
uncertainty over the preferences well
why don't you just
you know take expectations integrate it
out
um and that's correct actually if
it's not possible to obtain any further
information
about human preferences and this is a
a result going back to the ninth uh
early 1970s
um showing that you know mdps
with uncertainty about the reward
function can simply be converted
uh into regular mdp by integrating out
the uncertainty on the reward function
and then there's an extension of that
where um
additional observations can tell you
more about the reward function
um and when that's the case
then you get a different policy sort of
obviously right so if
preference information can flow at
runtime
from the human to the machine then
this idea of integrating out the
uncertainty is not is not correct
right the theorem fails and the machine
behaves in very
very different ways uh when it's
possible to gain more information about
preferences
so that's a no um and lastly a lot of
people think that
assistance games are requiring the
machine to learn about
human preferences nothing in the
formulation
says that right and in fact
we can have machines that simply
optimize a policy over time right so
they're not learning anything about
preferences they're just tweaking a
policy
uh gradually changing it over time and
we can show that
in the limit of enough experience that
machine is doing
exactly what it should do if it's going
to be solving the assistance game
okay so it doesn't have to be explicitly
learning preferences
any more than something that you know an
agent that
satisfies the von neumann morgenstern
axioms has to actually have a utility
function
right it doesn't it just has to act as
if it did
okay and there are all kinds of ways to
do that in fact
pretty much all of the methods we use in
ai
are based on doing von neumann
morgenstern
implicitly so the answer that is no
there are lots of other questions that
i don't really yet have answers to um
and uh i'm just gonna very quickly run
over those so
first of all the basic theory um
that i uh that i've outlined here uh
initially talks about results obtained
with one machine and one human
obviously there is more than one human
and the problem there is not that
the the humans have different preference
models that's fine we just learn
billions of preference models just like
facebook does um
the problem is mainly how do you make
trade-offs among the preferences of many
different humans
um and uh you know i think most people
on ai
naturally gravitate to a utilitarian
approach to this problem
and um in the book human incompatible
uh i i argue that in fact many of the
objections to that approach
don't really hold up i'm not saying that
there are no possible objections
but most of them seem to be quite
manageable
but these are questions that moral
philosophers and economists have worked
on for hundreds or thousands of years
um and there are still many open
questions
in in formulating the basic theory for
example
how do you make decisions when the
effects of your actions can change the
number of people who exist
how do you make decisions on
behalf of humans whose preferences
change over time
right these are fundamental
philosophical questions that we don't
have answers to but
need answers uh if we're going to flesh
out this approach
we also have to be concerned about what
happens when there are many machines
doing this particularly machines that
are not all
designed and implemented according to
the same template
right so in addition to humans
functioning in the environment uh there
could be many other
machines that are you know even if we
you know pass the regulations saying
you know everyone has to be solving the
assistance game um
you'd still have the issue that there
are lots of other machines that you
didn't design
um and uh if if we're not careful they
may have
uh kind of strange synergetic uh
feedback loops that cause things to get
out of control in ways we don't yet
understand so that's a very important
area um
if we're going to actually have machines
learn more about
human preferences they have to
understand that humans are not
rational and therefore in order to
interpret
our behavior as expressing or
providing evidence about our underlying
preferences uh we have to know
how that expression process works
because we need to invert it or the
machines need to invert it
so in a sense uh our machines need to
learn
uh inverse psychology if you like um
and that's obviously a very complicated
process there are
all kinds of reasons uh myopia
emotions uh plasticity that that cause
the expression of preferences and
behavior to differ from
uh perfect rationality um
and then there are sort of the you know
the practical hard work
that that we need to do uh
to make this actually be the field of ai
in the future if you look at the
textbook on ai
every chapter begins by saying okay
here's how we define
such and such kind of problem right
there's for example in the chapter on
search
right there's a goal and a cost function
right and those are assumed to be given
right that's sort of
what we mean by defining a problem right
in reinforcement learning there's a
reward function
that has to be known for every
transition that could possibly take
place
so that you can supply the reward signal
to the learning agent
and so all of those areas of research
have to be rebuilt
and it's not we're not necessarily
throwing away all the stuff we've done
because in fact it's a special case
right the case of certainty about the
objective is a special case of
uncertainty about the objective we're
simply saying no we need a much
broader foundation and we've only looked
at one tiny corner
and the behaviors in that tiny corner
are sometimes dangerous and undesirable
and also actually much less interesting
right they're very single-minded they
don't
have incentives to interact with a human
to learn from us to
ask permission all those kinds of things
are behaviors that can't be exhibited
in the standard model at least
straightforwardly interpreted
and then we have to show that these
methods can actually be
successful in the real world so we have
to look at applications like
self-driving cars or digital assistance
and so on
so that people out there in industry
have the confidence to say okay yes we
should build
systems along these lines and we should
not be
following the old approach so to
summarize
um if we pursue the standard model to
its logical conclusion
as turing pointed out we lose control
over our own future um if we take this
other branch
and i think it's it's very early to say
exactly how things should go
but i'm pretty convinced that this other
branch exists
and somewhere down there are our
solutions that are provably beneficial
to humans
and those are the ones we should be
doing uh in fact there's this
ample economic reason to do things this
way these are just
better ai systems and so i don't want
people to go away thinking
that this is a fight between
you know the ai ethics people and the ai
people
right that's not going to work right
what's going to work
is ai people understanding that this is
just a better way of doing ai
right just as bridge builders have
learned that oh this type of bridge
falls down
this other type of bridge doesn't fall
down so this is a better type of bridge
and this is the one we should build
right and that's a good bridge that is
good
civil engineering um and we want this to
be thought of as this is
good ai and the other way of doing it
well that's light
lazy you know we used to do that back in
the 2020s but we don't do that anymore
because
we know that it doesn't work right and
that's that's the kind of mindset that
has to happen
so there are other problems uh arising
from ai
misuse the doctor evil problem uh so
it's sort of
cyber crime or cyber war on steroids
which is uh
we're not doing particularly well with
cyber crime and cyber war right now
so that's more of a societal policing
problem and then
overuse meaning that the human race uh
becomes too dependent on the ai systems
that we build
um that if they are providing everything
and running our civilization
we lose the incentive to know how to do
any of those things
um and we become dependent and enfeebled
uh and this is an undesirable future and
a well-designed ai system should
actually say no
uh we're not without you know we're not
doing that stuff for you you have to
uh keep knowing how to do those things
yourself and doing those things
yourselves
in order to maintain the vigor of your
civilization um
but we being myopic humans
may not like that answer and may
nonetheless
fall into this trap so there's a lot of
work to do there and that's a cultural
and not so much a technical problem okay
we have a few minutes left for questions
and then we have time together so thank
you very much
thanks a lot stewart that was a great
talk um i'm hoping everyone's going to
you know keep in mind the distinction
between the standard model and the new
model
um and thanks everyone who's watching
for submitting such great questions
uh i have no no like
delusions that we're going to get
through all of them but i will try to
get through as many as
as we can um so yeah so the first and
most uploaded question that everyone had
stewart
uh you can sort of see the philosophy
and morality that
coming through um
oh actually it has changed never mind
the most uploaded question
uh doesn't beneficial ai as maximizing
satisfied net human preferences
run into the problem of incoherent or
evil preferences
shouldn't it instead seek moral realism
and maximize
moral goodness
um interesting question so
incoherent preferences uh are
preferences that can't be satisfied
um so in the book i have a simple
example right so if there's three kinds
of pizza
you know pineapple sausage and plain and
you prefer
each one to the next so i prefer
pineapple to sausage sausage to
plain and i prefer plain to pineapple
right so now i have
circular preferences and whatever pizza
the ai system gives me
i say i don't like that one i like the
next one
right and this just goes on forever
right so there's nothing the ai system
can do
to make you happy and so all forms of
incoherent preference
kind of boil down to this that you can't
they can't be satisfied
but you know if i prefer all three of
those peach
pizzas to you know a dinner of licorice
then it's reasonable for the ai system
to make me one of the pizzas
but i can't make me the perfect pizza
because
that doesn't mean anything for me if i'm
incoherent about my pizza preferences
so um that's one the the evil
preferences um
to some extent evil is in the eye of the
beholder
um and the way um that
hassani thinks about this um so he
hassani is an economist um who
did the sort of the first axiomatic
extension
of von neumann morgenstern axioms to
decision making on behalf of many people
and he has a social aggregation theorem
that
under fairly mild assumptions that seem
very reasonable
you can show that all pareto optimal
policies
uh will optimize some linear combination
of the preferences of the individuals
and
and then he argues that you know by
fairness or by symmetry
the linear combination should give you
equal weight to everyone's preferences
so the evil preferences would be the
sadistic ones that i mentioned
where uh if you break down
my preferences into uh in preferences
about my own well-being
sort of intrinsic well-being preferences
and preferences about the intrinsic
well-being of others
if i have a negative weight for the
well-being of others then i am a sadist
right because that means i will give up
some of my own well-being
to hurt other people right and he ah
hassan you says look then that person is
simply outside the social contract
um for which it makes sense to be making
decisions on behalf of those
those people and there is no way i am
obliged to help you hurt someone else
simply for your own gratification and so
you can zero out right at least in the
simple
mathematical decomposition you can zero
out the negative
the negatively altruistic preferences
that people have but i think this whole
this whole area of of
looking at how one's preferences are
composed of
preference intrinsic preferences about
one's own well-being and
preferences about well-being of other
people i think it's under-explored
and it has significant implications for
some of the moral objections to
utilitarianism
which to me actually seem incoherent
right so they're
they seem to have the property that um
even if
no individual in the population has any
interest in the well-being of any of the
other people
right the the moralist is still imposing
some notion of equality that needs to be
added to utilitarianism
even though none of the individuals care
at all but of course if the individuals
do care
then utility then utilitarianism does
impose
you know egalitarianism right and the
optimal
uh resource allocation under pure
utilitarianism is completely egalitarian
so i think there's a lot of confusion
going on here as for whether we should
simply implement moral goodness um well
i could argue that moral goodness means
acting in
in the best interests of the human race
right which is
and you can argue about well how exactly
do you calculate that but
that's what i'm trying to implement
through preferences
utilitarianism uh and i'm not
going to be in the business of saying
well actually you know
my preferences or i you know i'm writer
about
uh what the future should be like than
other people so i'm implementing
my theory of moral goodness and
so this is a preference utilitarianism
you know incorporates preference
autonomy that everyone is entitled to
their own preferences
and it doesn't say that you should care
about whether you have anything to eat
doesn't say you should care about
how rich you are doesn't say anything
like that it's what your preferences
actually are um the big issue the place
where there's a hole in the theory i
think is
with preference plasticity um
and it's not clear i don't think you can
prevent ai systems from modifying the
preferences of humans because
in almost any form of interaction
potentially can modify
human preferences but we don't want the
a system to deliberately manipulate
preferences to make them easier to
satisfy so that's an open
problem good thesis topic
that was a long answer sorry no worries
um i do i think we can get through
another question i don't think that's
likely
i will probably just close it at this
point um
so yeah thanks again stuart for for this
session
um it's it's been pretty great i expect
the viewers have gotten a lot out of
that and thank you everyone for coming
[Music]
you