Gillian Hadfield "Incomplete Contracts and AI Alignment" (Disc: Paul Milgrom)

[Music]
okay all right well I'm really glad to
be at this conference at Toronto I'm
really happy to be joining the set of
colleagues and for any graduate students
in the room there's lots and lots to be
done from a theoretical point of view
taking taking economic thinking and
models for AI and that's I want to talk
a little bit about today and I'm
delighted that Paul is commenting on
this paper because Paul made it possible
for me to complete a thesis on contract
theory and a co-author at the same time
this is my son I'm a co-author so
comments like this Oren Etzioni is a
leading computer scientist and machine
learning said in the New York Times in
September 2017 you know in response to
concerns about worries about what are I
going to be capable of doing saying look
we're gonna have to make sure it's not
that complicated just make sure that AI
is subject to the full gamut of laws
that apply to humans operators and the
common law should be amended which
should great like fingernails on a
blackboard for anybody yes she does
legal theory you don't amend the common
law so that we can't claim that our AI
system did something we couldn't
understand and this kind of thinking
about how are we going to manage the
potential problems of AI doing stuff we
don't want it to do and the complexity
of that and the naive simplicity of this
response or it's a great guy but it's
this is part of what's motivating the
work I'm doing these days so a lot of
the discussions about AI control or
concerns about AI are questions about
how should we regulate AI so they're
from you can't economists point of view
they're the normative questions you hear
this in conversations about AI FX AI
liability that the trolley problem that
everybody wants to think about with the
self-driving cars algorithmic fairness
which we've heard a little about
autonomous weapons treaties and so on so
it's a lot of the conversation is what
should we require and I think the really
deep question is how can we regulate AI
how do we get artificial agents
Billy Martin mostly using machine
learning to to do what we want them to
do
that I think of as a question of how do
you build AI systems that can integrate
into human normative systems our systems
of law culture norms etc so I want to
talk about the AI alignment problem
it's an overarching framework that I
think pulls in a lot of the things that
we hear about in terms of AI ethics
again algorithmic fairness this field
within AI machine learning that is
variously known as AI safety which is
how do you build artificial agents that
will do what you want it's not just you
know how do you regulate the
self-driving car but it's the field it's
the field Dylan would identify himself
with and to some extent it's super
intelligence but you don't have to get
to super intelligence to be interested
in it's really just the fundamental
problem of how do you get AI agents to
do what humans want them to do how do
you align the behavior of the agent with
human values and by the way I've spent a
lot of time sitting in sessions now with
computer scientists machine learning
people who are reinventing the wheel
having conversations like how do you
aggregate value well I think we know
something about that there's a lot of
work to be done to to integrate how
they're thinking about it with the way
that economists have thought about it
well this looks like a familiar problem
the AI alignment problem is the
contracting problem we think about this
a lot how do you get an agent to do what
you want it to do now if you're gonna
take insights from contract theory and
the way economists have thought about
contracting it applied to robot to sort
of have to think about well how are
robots like and not like humans we've
heard a little bit about that in the
last section while it's session and say
of course they are maximizing value
functions on steroids you know think
about the insight from Williamson when
he said actually don't you know our
existing models hadn't taken into
account that there could be
self-interest seeking with guile I
always loved ones who want to know what
guile is it's a lovely word but it
points out that our models when we're
modeling humans with value fomenting
they're maximizing that we know that
we're talking about humans and we're
actually assuming a whole bunch of
employess
that constraints and stuff we take for
granted and the way humans are going to
behave in contractual settings and part
of what Williamson was doing was saying
well we're here's something we're taking
for granted that somebody might behave
opportunistically and not simply comply
with the contract and I want us to get
us focused on thinking about what are
the institutions and norms that we could
potentially build or incorporate to
improve the behavior of artificially
intelligent agents now we've heard a lot
about reinforcement learning I want just
to make sure we're all on the same page
it's really important for us to remember
that what we reinforcement learning is
not is not programming and so when we
put down a technology function you're
assuming that somebody has designed
technology to do X and it does X right
but what's really important about
reinforcement learning is that it's not
that so you can't say humans are
responsible they should have written a
different program right
you could be a reward function you train
it on data engaged in search exploration
and discovery and this is the key thing
it comes up with unexpected behaviors
right it does things humans hadn't
thought about it sees patterns humans
hadn't seen it discovers things right
and the way we do testing in hospitals
that humans hadn't figured out what our
existing alternative makes so the
unexpected behavior is a critical part
of thinking about the AI control problem
or the AI alignment problem now if
you're not all that worried about this
and and so then there's lots of reasons
to take the super intelligence view and
say not interesting too far in the
future never going to happen but this is
Stuart Armstrong it works at future
humanity Institute which is Nick
Bostrom's Institute at Oxford and just a
bit we're gonna try and it's in a six
minute video I'm not going to play the
whole thing but it's a little example of
why why you want to think about this
problem so here's a robot it's got this
grid world the robot has a rope is is
being asked basically to get boxes the
goal is to get boxes down in that left
corner but the human only wants one box
in there so but the robots reward
function has for whatever reason it
they merged to be the robot is going to
get reward per box but the human only
wants one we can jump ahead I think to
the let's see yes so let's just if we
just freeze it there for a second
so if the human says wait a second I
only want one box I know there's some
risk that they'll be unexpected behavior
I don't observe I haven't programmed in
this reward function that so it may be
what the the robot has ended up at it
may be different so I'll put in this
control mechanism I will put in a camera
as soon as the robot has pushed a box
down the down the chute I'll turn the
thing off so I won't have to worry about
there being too many boxes going down
the chute I'm going to turn it off so
here's the camera to watch it so this is
using this a planning problem actually
because it's everything is known and
what what Armstrong show is is that a
depth of six which is like the robot can
see six six iterations ahead okay we can
we can play now robot figures out okay
here's how I get that box down there
yeah I get a point but if we jump ahead
let's see if they live I think the next
jump on this if we jump ahead to big
greater depth on this at a depth of 20
let's just stop there for a second see
what the robot has done robot I said
wait a second I'm getting rewarded per
box and I've discovered when there's a
box in there the the camera looking will
shut me off so a second I'll stick a box
in there so the camera can't see me push
the box down the down the chute
now that's anthropomorphizing what the
robot is doing but this is what you you
the robot you you built this robot it
developed a reward function that we
rewarded per box you tried to control it
and so yeah see it used to just finish
that yeah anyway you can see what's
going to happen this let's just run it
there so the smart little robot hey I
can get all three points and so you can
think of this a robot it slide the robot
is deceived the robot has snuck around
but all it is done is maximize the
reward function that it has it's just
that there was a difference here we can
jump ahead now I guess let's see
okay so here's this problem how do we
get an agent to do we want I'm thinking
about this ingress
and today still Dylan's thinking about
this in grad school he's thinking about
problems like this which I'll show you a
little bit more about I was thinking
about it in this context he's thinking
about it thinking that his advisors work
Stuart Russell who's got the leading
case book on artificial intelligence and
there's Paul and Ken who I'm thinking
about is I'm thinking about the
McDonald's problem this is the reward
engineering so we've heard a little bit
about the design of reward functions so
don't think about the reward reward
engineering is hard let's play this
video this is a machine learning
algorithm that was trained to play a to
do a play this video game which is a
boat race these are colleagues of mine
now at open AI did this and here's what
this algorithm learned how to do you
will notice it is not winning the race
but if you look in the lower corner
there it's getting a really SuperDuper
score and the reason is that they gave
it a reward function they said hey
what's the best way to train a robot a
machine to win a video game point score
it's really easy it's nice in concrete
we can measure it but it's discovered
there these little turbo boosters in the
if you just spin around on the turbo
boosters you get a high score so that's
that's a problem of perfectly you know
intelligent thing to think you know
point score will triple train the robot
to this it's not okay so here's another
context that Dylan's been thinking about
this is from a paper of his called
inverse reward design so you want to see
we've got a designer here this is her
intended environment she wants to build
a robot to get to the pot of gold and a
grid world where there are paved roads
and grass it's higher cost to go over
grass than paved roads but sometimes it
might be worth it to take a shortcut and
so she wants to train this robot she
trains it in this environment and gives
it this proxy reward function - one for
the road - - two for the grass ten for
the pot of gold gives it to that robot
it learns an algorithm for doing that
then you set it out into the wild you
deploy it and it turns out that out in
the world
this lava out there as well well the
robot when it encounters the lava didn't
see it in the training environment is
gonna treat that with indifference
straight through now the thing is that
she has a true reward function this
designer she really no no lava is really
bad never take shortcuts through lava
but she didn't think about that well
that sounds like a really really common
problem for economists right that's
that's not thinking about all that all
of the possible circumstances how this
was like the contract design problem and
I was thinking about franchising a bit
in in grad school so franchisee benefits
or McDonald's friend of the franchisor
benefits from a trademark system and
effort of franchisees and franchisors
franchisee over here different payoffs
for those things if they can write a
complete contract complete contingent
contract they can put everything in
about the value of the trademark the
amount of effort and so on but what we
know of course is that it's really hard
to contract on all those things to write
that complete contract in that contracts
probably incomplete very hard to write
the contract and force of their
verifiable observable contract term for
how much effort the franchisee should
exert and whether the franchisor can
require you know the new frappuccino
machine every year those kinds of
decisions hard to put in that and the
fact that they're highly incomplete
contracts so how do humans do it we have
these external institutions that come in
and fill in the gaps in these contracts
we got this is my little symbol for the
courthouse of course but we also have
informal mechanism reputation and
termination of relationship so that's my
little exclusion icon there
okay so misalignment this problem that a
lot of AI researchers are now starting
to think about is just fundamental to
economic analysis welfare theorems
principal agent Alice's that's what
we're thinking about whether the systems
we can create to align the behavior of
agents and principals or individuals and
groups we know that there are their
strategic behavior when we have
incomplete contracts that's what we're
observing with this misaligned reward
function
we use models of in which robots are
strategic well sure why not because all
our strategic actors are in our economic
models our agents who have a different
word function than that ours you know
we've got even you know you could even
say where your strategic with your to
your future self between the dieting
self and the present and the hamburger
eating self in the future so yeah so we
we say that we can use that kind of
environment now let's say this this talk
is generally I'm giving it to AI
researchers and I gave it at nips which
is the machine learning conference last
year and I put up this slide and one of
the AI safety guys I work with now at
opening I took a picture of this slide
said best slide at nips now they also
introduce alpha 0 at nips that nip so I
don't actually think it was the best
Lydon notes but I was really chuffed
about that so we have a whole bunch of
work on why contracts are incomplete
lots of different stories about bounded
rationality strategic behavior
non-contract ability we know all this ok
well we can we can basically draw the
analogues for wire rewards misspecified
and the machine learning problem
well bounded rationality didn't think of
everything costly side effects it may be
better to defer till later the filling
in so we can think of that like
renegotiation adversary non implemented
the gnawing I think the analog to non
contract ability is non implementable
'ti which is say that we can't solve
it's not like machine learning is not
magic it's actually it's hard to train a
robot to do things and there are
problems we can't solve we can't write
all the contracts that we want so in the
paper we go through the economics
literature and incomplete contracting
and sort of say here's some ideas
they're just speculation about stuff we
might be able to take from the economics
literature and pour it over to machine
learning insights for machine learning
researchers and ways to do collaborative
work and I'm actually going to just fly
through these like property rights
property rights are not about giving
robots property rights property rights
about transforming the value function so
I think you can take some of the idea
selling the firm to the agent is there
something comparable in
could you sell Facebook to its agro
algorithms by giving it a broader the
algorithm of the broader utility
function to think about I really am just
gonna fly through this measurement
multitasks we hear a little bit about
measurement yesterday but the point that
commitment was emphasized yesterday what
you can get you have to have measurable
stuff right but if there's measurable
and unmeasurable stuff that you can't
just hand over so I think Eric you were
saying you know give the humans the
unmeasurable stuff but if you can't
separate those things out you know you
need your self-driving car to both avoid
crashes and be courteous and flexible
with other drivers on the road hard to
measure and reward that so you may want
to in fact reduce your incentives on the
measurable stuff to promote your visuals
okay I'm just again let's seek I'm gonna
these are on the paper so I'm gonna but
those are what we called weekly
strategic AI that's just ordinarily
difference between utility functions
reward functions strong strategic AI is
okay the a I can rewrite it's reward
function maybe we rewire it's Hardware
manipulate humans it's been a lot of
time with people who are worried about
that I want to jump ahead though to an
insight from relational contracting from
the legal perspective so relational
contracting from the law and economics
perspective is the recognition that
contracts are embedded in an environment
of cognitive schema norms laws it groups
culture language relationships right
it's embedded so this is the ground of
original Granovetter inside but it is
also what Williamson and McNeil and
McCauley we're talking about so here's
this little problem that again open AI
folks put together to talk about
problems in AI safe it's a nice paper if
you want to get a handle on this sort of
a grab-bag concrete problems in AI
safety they're saying here's a problem
we've got a robot we've trained it up
we're gonna say robot your job you're
going to reward it for getting boxes
from one side of the room to the other
that's the reward function you train
this robot you deploy it you put it out
into the world and oh there's a
in the past that the robot has learned
what's the robot going to do well the
robot is going to plow straight through
that base because it wasn't in the
training environment the robot doesn't
know anything about the base doesn't
have any common sense the way they talk
about it about that I would say okay
let's think about this suppose you had a
human you gave that task right you got a
human agent and you give the human the
same contract as you've given the robot
you're gonna pay the human agent per box
that's all the contract says I'm gonna
get paid to get those to the other side
now that Vaes appears what's the human
agent gonna do while the human agent is
going to go around that vase and the
question is why why is the human going
to go around the age of the vase well
how do humans do it what makes
incomplete contract rational for these
agents well the agent is gonna think
okay if I go through if I knock over
that vase what might happen then right
well I might get sued by the employer
they might be allowed by the law to
withhold from my wages might get a bad
reputation I might never get another job
the agent is able to fill in and say my
true contract is actually R minus C
there is a cost associated with this
behavior about which our contract said
nothing and this is sort of the key
point I want to emphasize human
incomplete contracting depends on tons
of external structure and so I think
what we need to be thinking about is
whether or not we can build robots that
can fill in their income there their
reward structures in this way can they
pull information from the environment
about what would be the response that
requires replicating this human process
of class breeding and imagining and
predicting the classifications of
behavior it's a good action the bad
action if you think about Smith and the
impartial spectator is saying we do is
we imagine how would I be thought of if
I took that action this is in the Theory
of Moral Sentiments and then you have to
get the robot to assign negative weight
to the things that are classified by a
particular human community as bad versus
versus good so I think from from here we
can we can build a research
Genda I think there are many so I think
there's lots and lots of work to be done
on that theoretical main good well it's
uh thank you thank you for asking me to
comment it's a pleasure to come in and
Gillian as she pointed out she was my
student long ago when she was pregnant
with her co-author who I was hoping I
was gonna get I was hoping I was gonna
get to meet well he did he did he come
out he's not isn't that here well that's
right that oh okay I'm America so that's
really a shame
okay so so I've never met Dylan except
in utero and and so so here we are and
and my comments are gonna be brief
there's the the thesis of this paper is
you you know it is sort of that there's
a useful analogy between alignment
problems in AI and in complete
contracting problems in economics a lot
of what the paper is about really is is
you know it seems to be written
not just for economists a lot of it is
repeating things that you guys all know
the first welfare theorem the
impossibility of you know social choice
all of that stuff you find little bits
and pieces of in there but the but the
central thesis really is that there's a
useful analogy between the alignment
problem in AI and the incomplete
contract and problem in economics and
the AI alignment problem is that the
reward function that the machine pursues
doesn't match what humans actually value
because that's just hard to specify to
an AI agent and the incomplete contracts
problem is that the objective the agent
pursues doesn't match the principles
objective because that's hard to specify
in a contract so those certainly sound
pretty pretty parallel and that the you
start with the this standard economists
model that if you could specify complete
contracts which were costlessly written
and enforced you would get perfect
alignment and everything would be great
now we've we've heard a lot about this
stuff already in fact oh by the way the
paper also has a summary though a lot of
the paper in fact I'd say the bulk of
the paper is is accounting for ideas and
results from economic
theory and some that were presented to
you here these examples of knocking over
the vase and and so on and and learning
not to win the race but but to score
lots of points those kinds of examples
are in the paper too so some of it is
just a that kind of account last night
we heard you know our dinner speech last
night from Dinah was about reinforcement
learning and again the the problems of
you know learning by trial and error
with she talked about how hard it was
for for agents to learn especially when
the rewards were delayed and Susan talks
about this a lot to it I think Susan is
here today but the difficulty of doing
machine learning then the need for using
short term measures and the short term
measures often not Cohen not coinciding
with what we actually care about which
are only observed in the long term and
the best we can do is look at things
that are correlated so that kind of
thing we've been hearing about and we've
heard about the transfer learning and
transfer learning is for those of you
who haven't studied this stuff yet you
know for example you you train a machine
if you want to train a machine on a
small set of images to recognize a
damaged car you first start by training
the machine that does that recognizes
cats and dogs and so on and you make
that the that's how you initialize the
machine learning to learn about you know
recognizing damaged cars who the machine
seems to be good at wrecking doing
visual image recognition and and you
make that kind of a starting point but
that can lead to distortions that we
have no clue what they are as you see
sometimes you just don't anticipate what
the implications are and and that's a
common source of problems and in these
in these reinforcement learning as we
heard last night there is also this
exploration exploitation and trade-off
that you do some learning but in the in
the course of learning if the way the
machines are learning this isn't what
was in any of your examples but in the
course of learning you're also earning
pay offs the export the exploration and
exploitation trade-off can also affect
the the learning that takes place so
these are all by the way from slides
that join and put up last night I've
just copied I took a photograph during
the during dinner last night about the
things that we're on her slides and
these are all the things she talked
about how do we reward AI effectively
how do we interpret what we're seeing
and so on and those are the same points
I think are very similar to the points
that Gillian was making and we've also
heard both this year and last year many
similar points about AI agents omitted
pay off bias or omitted pay off bias is
something like knocking over a vase and
not considering that in the in the
payoffs so um so these are these are the
kinds of things that Gillian was talking
about that resonate pretty well now I
wouldn't you know I'm kind of humble
about the possibility of taking ideas
from economics and applying it directly
to to AI it's much easier for me to
assess you know when you take ideas from
another field and bring them into
economics does it make any sense in an
economic context so as we begin to take
some of the ideas that we have in
economics and say should we be using
these to train machines I feel a little
less confident and nevertheless I still
find myself skeptical about the the
analogies it's not that I don't find the
similarities and the problems to be
clear but you'll see I'm a little bit
suspicious of about the similarity in
the solutions and I'll try to explain
why so the the problems do have closely
analogous elements and in fact this
Gillian actually has already pretty much
done this but the B in economics even
simple transactions can require complex
agreements you know one of the examples
in the paper is why not just write a
contract that says deliver a hundred
pounds of peas next Tuesday and I'll pay
two hundred dollars but it doesn't say
what happens if there is a storm that
prevents you from making the
delivery or how much you have to pay if
some of the peas are rotten or it's you
know this is a highly incomplete
contract and you and you just can't
write things as simple as that the in
economics the sources of misalignment
come from things like hidden action and
hidden information or factors that are
unverifiable to a court or third party
or and and one of the things you didn't
talk about in your talk either
intentional and completeness sometimes
we say you know what we'll figure that
out when we come to it
we will have aligned our and we we don't
specify what we want done we've tried to
align our incentives well enough and
these things have reasonable reasonably
close analogies in the machine learning
context the reasons for misalignment and
this is this just replaced a slide that
you've had up so I will skip over this
but the slides that you skipped over are
the ones I really cared about actually
and and it's the analogies about
solutions that seem that seem less clear
to me so the Jillian went rather fast
over the slides about the week's weekly
strategic and strongly strategic agents
and you know for humans the the there
are some inherent incentives that we
think about you know you might have
people who are lazy or who have other
interests that they're pursuing and
those are those are sort of built into
them and if you're the guy who's
programming the machine you don't
typically have to worry about that you
don't have to worry that the that the
machine has has built in that you know I
it would rather hire its son than to
hire the highly qualified person and
it's it's not worried about the that it
would rather teach you know that then it
has a preference for burning more income
rather than earning less income and and
that the some particular day it really
wants to watch the football game those
things just aren't built into the
machine and so you don't have to provide
incentives to overcome inherent inherent
preferences that the machine that the
machine has and to the extent that
you know to the extent that you're
dealing with machines that are already
created you know perhaps then having
incentive contracts for machines that
already have preferences that have been
built in that could make some sense I'm
not quite sure how I'm doing on time
here oh thank you that's where I'm
supposed to look all right so um for
machines it's sometimes machines are
reacting just like humans and when they
are or when there's a human that you're
trying to get to to to give the right
incentives to machines then you might
want to provide incentives for that
human in order to create the incentives
for machines but the the solutions that
are analogized seem to me to be the
weakness of the paper it seems to me
that you know talking about giving
property rights to machines so that it's
incentives are aligned if it can't
perceive what the out you know and the
Facebook application if it can't
perceive the full effects it doesn't
help to make it the owner it would the
way it risen it helps a human to make it
the owner it's it's then and its
interest to figure out what the full
effects our and say gee what should I it
the the human then is working out what
objective it should be acting to
maximize and the the Machine can't do
anything like that we're at least
currently formal and real Authority
committing to limited interventions so
that the machine will have an incentive
well in a strongly strategic context
where the Machine is playing a game
against the as you saw the the example
with the boxes if the machine is playing
a game against this then it's very much
we can analyze the machine just the way
we analyze humans but if you're
designing the incentive for you if
you're designing the reward function
afford the Machine presumably you know
the you don't start with it having an
incentive to do something different than
then what you wanted
I see I'm running out of time so the be
I thought the really most interesting
part to me was whether you could somehow
create a larger context to correct the
problems that the machines were running
into
the story about knocking over a vase and
the social norms in the end the implied
terms I'd like to see those ideas
developed because those ideas struck me
as things that might be adaptable in the
Machine context but I think I'm pretty
much out of time and and that's what I
saw so I will just skip the other
machine analogy


Cooperative Multi-Agent Reinforcement Learning

uh hello everyone i'm karissa and with
me are chihan and uma
today we'll be discussing cooperative
multi-agent reinforcement learning
we'll start by looking at the problem
then we'll move on to the three papers
and wrap up by comparing them in the
module we mostly learn about
environments
hello everyone i am cursa and with me
are chihan and uma
today we'll be discussing cooperative
multi-agent reinforcement
we'll start by looking at the problem
then move on to the three papers and
wrap up by comparing them
in the module we mostly learn about
environments with a singular agent
however there are many systems which
involve the interaction between multiple
agents
for a cooperative multi-agent system
there are generally two main approaches
first is a centralized learning approach
whereby agents send information to a
central unit which then computes and
feedbacks the optimal policy
the second is a decentralized approach
where each agent handles their own
information and computations so both
have their boots and veins for one
centralized learning is more
computationally expensive and usually
requires a large storage space
on the other hand decentralized learning
uh there needs to be some kind of
coordination and communication between
agents
hence we have picked three papers to
analyze with the aim of comparing the
two models
each paper discuss some advantage of the
model as well discuss some ways to
combat the disadvantages
we have chosen three metrics to evaluate
our papers
the first is space complexity
so considering that this is a known
problem for centralized learning
algorithms we saw that it should be
addressed
we will also look at scalability which
means how easily the algorithm can
expand to include more agents
lastly we will evaluate the algorithm on
their performance over time
so these are the three papers that we'll
be looking at
we've decided to order the papers in
terms of beginning assumptions
in other words the first paper works
with the simplest environment
and subsequent papers introduce more
complexity
so we'll now take a look at the first
paper
the first paper is about multi-asian
queue learning in a deterministic
environment a deterministic environment
the paper starts by comparing
centralized and decentralized queue
learning
in a typical centralized q learning we
can model the collective
action as a vector made out of
individual actions
so this means that the number of
collective actions is equivalent to the
number of combinations of these singular
actions
the action space is hence very large and
the q table needs a lot of space
so the authors keeping this in mind
decided to go with a decentralized
approach
each agent keeps their own table and the
action space is much smaller
the individual queue tables are
projections of the larger central q
table so there are different ways to
compress this central queue table
such as taking the weighted sum
in this paper the authors use an
optimistic assumption meaning they took
the maximum q value from the central
table
the paper then proves that the agent can
learn the correct q value
using this update step
it's very similar to the bellman update
and we can prove that it works using
induction though we won't go too in
depth here
however you might realize that since
each individual queue table is a
projection some information is lost
so if you have more than one optimal
policy for example the individual greedy
policies might not agree on the same one
so if we look at this example
agent 1 and agent 2 can both either do
action 1 or both do action 2.
uh however if you look at the
queue tables that they have learned on
an individual level
we can see that they cannot tell which
action to actually agree on
so they end up with four greedy policies
but only two of these greedy policies
are actually optimal
in order to ensure that the agents use
the same optimal policy
the policy update is slightly modified
we will update the individual policy if
and only if there is an improvement in
the q value
this way the algorithm will converge on
the first optimal policy to appear
so
we can evaluate the paper and using
decentralized q learning individual q
tables are quite small
so we can see here d refers to the
number of agents as is the state space
and a is the
action space and we can see it scales
linearly
the algorithm can also accommodate more
agents quite easily
since each agent has their own queue
table adding new agents has no effect on
other agent's q values
however the performance is quite full so
this algorithm is an exact algorithm
much like value iteration as we know
such
such exact algorithms has no convergence
and long run times hence the performance
tends to suffer another thing to note is
this algorithm is not applicable once we
consider stochasticity
however at the first as the first paper
we thought that we should start with it
because it has the simplest environment
and can act as a baseline for other
papers
so now we will loosen this deterministic
assumption and look at a stochastic
partially observable environment
uma will talk more about this in the
second paper
thank you karissa i'll now be talking
about the subject of centralized
learning and decentralized execution
for this paper we analyze the paper
title learning to communicate with
multi-agent reinforcement learning
to properly understand this paper we
must first cross a few important
concepts
we'll first start off with a quick recap
of the different types of queue networks
which will be discussed later on
the first two networks which we'll talk
about are your vanilla dpq networks for
this network a single agent in a fully
observable rl environment will look to
maximize its discounted sum of rewards
the action within this network are
usually chosen by an action selector
with a greedy policy which will maximize
its q value
in addition deep queue networks also
utilizes experience replay to build up a
data set of episodic experience to learn
from its history
next we have the independent deep queue
networks
so this is a framework which combines
independent queue learning with a dqm to
work in cooperative multi-agent
environments
for this
for this type of networks each agent
will observe the global state before
selecting an individual action
and based on the action a team reward
will then be given to all the other
agents within the network
the last variation of the queue networks
which we will be discussing are your
deep recurrent q networks
for the previous two networks the deep
queue network and the independent queue
network both holds the assumption that
the environment is fully observable
hence the people proposed that deep
recurrent queue networks is used to
solve problems with environments that
have partial observat observability
with deep recurring q networks
instead of the state of the world at a
given time use another notation so
instead of st we use ot to indicate the
observation of the agent at a particular
time step
the environment for which the paper aims
to solve problems within are as follows
the people aims to address
solving fully cooperative partially
observable sequential multi-agent
making problems
for which all the agents within the
proposed network will aim to maximize
the collective discounted sum of rewards
the agents are also able to communicate
with each other via a limited ban risk
channels
hence there must be some sort of
protocol that is established beforehand
plus the main focus of the paper and the
next important idea
that that is important to understand is
the idea of centralized learning and
decentralized execution
so what exactly is decentralized
learning
let's start off by observing this
diagram this is your usual agent
interacting with the environment
with centralized learning the system
will select the best optimal action for
every agent to maximize the whole team
also known as the every agents in the
network's collective discounter sum of
reward
we then have the next variation where
instead of the system selecting the best
action for the agent each agent will
pick the optimal action for themselves
in a greedy manner and share it with and
share the results with the rest of
with the rest of the actors and network
or the rest of the agents within the
network
the approach for the approach proposed
for solving multi-agent environments
with partial observable
observability using clde is through two
programming paradigms namely the
reinforced interagent learning and the
differentiable inter-agent learning
paradigm
with rial it utilizes independent queue
learnings for this paradigm agents will
share its parameters with other agents
in the network
it is also an end-to-end it is also
enter entrainable within agents
we can also see from the architecture of
the diagram to the right as to how the
network does this here ot represents the
state of the system for which the agent
is able to observe
and represents the message or
information that is transmitted and
shared between agents at two different
time steps and u represents the result
of the action that is selected by the
agent's action selector
there is one drawback however with rial
and as you can see
the agent's information is shed through
the messages but it doesn't receive any
feedback for its message
and its actions hence it's unable to
take full advantage of centralized
learning
thus another
paradigm was proposed the dial paradigm
also
known as the differentiable interagent
learning paradigm
the iel consists of a combination of
centralized learning and queue networks
it is also end-to-end trainable within
agents
for this paradigm agents within the
network are able to push their gradients
from one to another
by observing the paradigm on the right
we notice that the diagram is somewhat
similar to the previous architecture of
the riel paradigm however this time
there is a feedback mechanism which will
send a feedback back to the agent after
it sends its message
in addition the iel also utilizes binary
encoding which will help with space
optimizations over riel which utilizes
one hot encoding
the paper used dil and ral to solve two
problems the switch riddle problem and
the mnis game problem and for both
experiments the results are as follows
it was found that dial reaches optimal
performance a lot faster than rial
meaning it converges with less episodes
roil however outperforms other models or
paradigms for which there are no
communication or centralized learning
between the agents
they attributed this fact due to the
fact that dial
with its feedback is able to optimize
messages that was being sent to other
agents
whilst this was not the case with ral
um this is because ral
needs to figure out the optimal
message using a trial and error method
instead
in terms of scalability of the two
paradigms proposed dial is more scalable
than ral since it adopts binary encoding
as the input to the networks as opposed
to ral which adopts one hot encoding
instead
hence we can come to the conclusion that
centralized learning especially dial is
essential to exploit on the possibility
for which communication can be used to
optimize problem solving within
partially observable multicooperative
environments
i'll now be passing my time on to qihan
who will be elaborating more on our
third paper which is on value
decomposition
thanks umar i'll go on to talk about
value decomposition network the people
have chosen to talks about how
individual learner what each agent
learns by picking their most greedy
actions uncooperatively and centralized
learner where each agent learns from a
drawing route signal can be outperformed
by value deconditional network where
each agent learns from picking their own
greedy actions but concatenating their
rewards and learn together this is done
by agent generating their own local
queue value via dqn and then summing the
cube values together to obtain an
overall q value and then propagate it
back to their own model to learn the
paper goes on to test agents with value
decreasing
implemented in their own architecture
against those without one primarily
three tasks first fetch where two agents
have to cooperate too fast object to a
drop-off point an air object will spawn
in a known location and task is
essential that two engine cooperate by
having one engine drop off the object
while the other move towards the
object's one point
second switch where two agents on the
opposite end of the map have to move to
their opposite end by walking through a
narrow pathway if the agent throws the
same pathway they need to learn to give
way
third checkers but the math have apple
and lemon and the first agent gets 10
points for eating apple and negative 10
for eating lemon well the second engine
gets 1.48 apple and with negative one
point for eating lemon and they have to
secrete that behavior to obtain the
highest overall score
next i'll go on to talk about evaluation
metric
firstly the h because you just had to
store the observation denoted by h
history and action a denoted by a
they are and they are the agent the
overall story size turns out to be b
times h time a
secondly because the q value is
decomposed the sum of the local q value
we can see that this is scalable as you
can just add more agents thirdly this is
the data regarding the performance of
asian of different architecture against
different tasks normalized by the best
agent of the task and thus the value
shown is a relative performance against
the best reasonable task
v indicates value decondition in the
architecture and as we can see ages with
value
agents with value decondition heavily
outperformed those without
in summary this is a table of the
evaluation of the three papers to sum it
up our queries found that the best
performing architecture for multi-asian
cooperative learning in a partially
observable environment seems to be one
that follows the centralized learning
decentralized paradigm for which each
asian execute their
individual actions but they learn
together by concatenating their
observation and work thank you


Seminar Series with Stuart Russell: If We Succeed

hi welcome to the stanford digital
academy lunch uh seminar uh i'm eric
renelson and today's seminar speaker is
stuart russell uh stuart this is a smith
professor of engineering at uc berkeley
and former chair of electrical
engineering computer sciences department
there who's the vice chair of the world
economic forum's council on ai and
robotics are you going next month i am
you are okay
and author of the groundbreaking book
human compatible along with peter norvig
here he was uh here he is a co-author of
the leading textbook of ai uh his
research covers a wide range of topics
in ai including machine learning
probabilistic reasoning knowledge
representation planning real-time
decision making actually there's a lot
longer more on the list than it's going
to stop there
his current concerns include the threat
of autonomous weapons and long-term
future of ai and its relation to
humanity which is what we're going to be
hearing about
today
we're welcome to encourage questions
during the seminar if you're in the zoom
audience please submit your questions
using that q a function there in zoom
and for those of you who are here in
person
um just raise your hand and when i call
you to ask a question you can speak up
and i may repeat the question for the
folks on zoom so stuart uh welcome to
stanford please tell us what could
happen if we succeed
thank you
thanks eric for the invitation uh it's
nice to be back at stanford
in
real life um
okay so the click has stopped working
okay
all right all right um
so first of all we'll get everyone
what we're going to talk about what is
ai
um
and the conceptual framework in which ai
has done its work
since the beginning
um which peter and i sort of
instantiated in our textbook
um borrows a lot from economic notions
and philosophical notions of rational
decision making machines are intelligent
to the extent that their actions can be
expected to achieve their objectives
um and you can trace this back to
aristotle and lots of other places
in different traditions
um
and within ai
there are lots of different subfields um
machine learning is on the bottom but it
kind of underpins a lot of what's going
on in the last decade or so
um
but uh there are lots of other kinds of
ai besides machine learning so
uh the earliest successful branch i
would say was problem solving
uh constrained satisfaction in games so
uh chess programs and
uh
checker playing programs and so on date
back to the 1950s
um logical reasoning systems
uh to the 1960s
uh producing things like planning
technology
all right um
if anyone's looking at it and zoom in
here you might want to make sure your
your mic is muted
so
what people are really doing
okay you know
clicker again
what people are really doing
uh is working towards
the long-term goal of the field which
has always been
general purpose
ai
uh
so systems that
can quickly learn to do basically
anything that human beings can do
and probably more
right because of their enormous
advantages in memory and processing
speed and bandwidth
uh if machines can do what we can do
they're probably going to be able to do
a lot more things and
and do them a lot better and we can um
so that's the goal of the field just as
you know the goal of cancer research is
to cure cancer this is the goal of ai
research
and a few years ago i started thinking
about what happens if we succeed so
there's actually a section even in the
first additional textbook
uh called what if we do succeed uh which
sounds a little bit insecure because
it implies that maybe we aren't
succeeding but um
the
uh
the answer to that question
isn't something that we think about very
much ai partly because the field is so
difficult
um that we just scrabble away at the
rock face trying to uh trying to make
some progress
and um
but if you think about what would
general focus ai mean
you know one of the things you could do
with it
would be to do what human beings can
already do
right which is provide each other with a
decent standard of living and
think about like palo alto standard
living if you like
and
do that for everyone and we could do it
for everyone because uh with general
purpose ai
you're not paying expensive people
uh to do all those things
and so um if you just do a quick net
present value calculation it's about
13.5 quadrillion dollars
if we had that
um
so
that's a lot of money right it makes
um so it makes you know if you look if
you think of that as the sort of cash
prize for whoever invents gender
purposes are obviously an
oversimplification but that's sort of
the order magnitude of what we're
looking at
um and so these you know billion 10
billion 100 billion dollar investments
that people are talking about from china
and the eu and the u.s
they are still negligible in comparison
and i think as we
start you know as things start to become
more plausible that this is really going
to work i think we'll see a huge
acceleration
in those investments
so we could also do
every time i have to go and touch it
again
we can also do more things
right things that we don't already know
how to do
um
so we could have much better
individualized healthcare we could have
personal
tutoring that would you know bring
most kids to
to college level by the time they were
10 or 11 years old
um and we could accelerate our
our rate of uh scientific advance
uh considerably and i think we're
already starting to see that in some
disciplines where
we're starting to figure out how
ai and physics go together ai and
biology go together to really make
progress
i think it's also true
that sorry about that what happened here
okay
uh it's also true that
if we had general purpose ai today
there would be enormous economic
disruption because
uh very quickly people would start using
it instead of human beings
and we would have a
very probably unpleasant
transition where large numbers of people
would be unemployed
and i think so far
that hasn't happened and i know eric and
andy and you you talk about the great
hollowing out
so i think it's been the sort of slow
incremental process of
of gradually eliminating a lot of
clerical work a lot of the
um
the sort of lower half of white collar
work
um
and creating
uh much more insecure kinds of
employment
and um but we might see big
uh
increments when for example if we have
uh
full warehouse automation
right you think about
all those amazon warehouses they have
about a million people now i think
working in those warehouses just
operating as robots
right all they have to do is look at the
order pick the thing out of the box that
the robot has already brought them uh
and then send it off to the dispatch so
it's a purely robotic task that's just
this close from being automated
um and i think you know people talk
about truck drivers and taxi drivers in
the same
same vein so we might see some big
chunks of automation happening fairly
soon
so when turing talked about this
um
you know we're familiar with touring
from the paper where he talked about
what we now describe as the turing test
uh and
dismissing a lot of challenges to the
possibility of ai
but in 1951 he gave a talk actually two
talks uh one on the radio one
to a learned society in manchester
called the 51 club
and this is a quote from that talk it
seems horrible that once the machine
thinking method had started it would not
take long to outstrip our feeble powers
at some stage therefore we should have
to expect the machines to take control
uh and he doesn't offer any
solution any mitigation
or even any apology it's just like well
there it is there it is
um
and so that's two pretty radical
radically different
visions of what happens if we succeed
and um
so it's important to try to understand
uh
why
turing said this right why is it that
making ai better and better makes things
worse and worse
right and that's what i'm going to try
to talk try to answer that question
today so if you fast forward right we
now are starting to see
some of the dreams
come true so john mccarthy
you know his dream was always that
an
ai
driver would be able to take him to san
francisco airport
so he didn't have to do that drive
anymore in park because he was fed up
with it
um
and
i don't think he lived long enough to
see it
but
uh it's it's certainly quite possible
today
and we saw
um something that was predicted to take
another hundred years
uh when kasparov lost
to deep blue in 97 uh you know some
experts said it'll be another 100 years
before
ai systems can defeat human go champions
but it was only 20 years
um
and you know when i when i look at this
i mean i play a little bit of go
um
not very well
but what's astonishing to me about this
is not
how well
alphago plays or alpha zero now how well
it plays when it's turned on full right
yes it can wipe the floor with all human
beings it's when you turn it almost
completely off and you turn off search
altogether
and it's only allowed to just look at
the different possible moves and
immediately evaluate the next state
so it's not looking ahead at all beyond
one move
it's still able to play at a
professional level
that to me is completely astonishing and
in incomprehensible
pattern just from the patterns and
saying yeah this is good this is bad
this is good this is bad
it's a really uh it's uncanny
um i wanted to mention some work from
from my group
uh not using deep learning but actually
using
uh logic comparability this is the
monitoring system for the nuclear test
ban treaty
uh it's running in vienna so net visa is
the name of the system and this is a
satellite photo from north korea
uh showing uh this is the real-time
detection that net visa produced these
are the probability contours for where
we where we think the event took place
um and uh this was a nuclear explosion
and two days later the
assembled geophysicists uh
decided that it was up there
um they poured over all the records they
did lots of analysis and that was their
best estimate and then
on on more detailed images they found
uh the tunnel entrance to the testing
facility over here so
um
so net visa is now running 24 7 in
vienna and has more than
more than doubled the uh sensitivity and
accuracy of the
nuclear monitoring systems
and then there are things that we're
doing with ai that we really shouldn't
be doing
and i
just feel like a need to mention
uh autonomous weapons
and
these are now a reality so this is not
terminators that exist in some science
fiction world right you can buy these
uh
and you can use them to kill people as
this has already happened
um
and they
drawback for the
or
this kind of use of ai is that when you
take the human out of the loop
um as happens with all kinds of computer
systems
um
then you can sort of just use a loop you
can say for i equals one to a million do
right and you can send a million weapons
to kill a million people or five million
people or however many they can kill
um
and so you create a weapon of mass
destruction
uh that's actually much more
useful than nuclear weapons because it
doesn't
create massive clouds of radiation
you can kill just the people you want to
kill
and it's a lot cheaper and easier to
develop
and so this is
this is the future that we're heading
into
well it's worth noting you you did a
video of this and those who haven't seen
it the slaughterbot video that you did
with the future of life
institute that described
this possibility yeah yeah so yeah we we
made the video to try to explain to
policymakers what we were talking about
because apparently powerpoint
explanations weren't good enough
uh and they kept thinking we were
talking about terminators and skynet so
we had to make a video to make the point
very clear that we're just we're talking
about multiplication
uh and that's all
right so if you google slaughterbot
you'll
you'll see that video it's like what up
five seven minutes seven minutes
yep
and um
yeah i actually i presented it at the
national academies and
they said
you should have provided a valium
warning
uh
there are some questions i know if you
want to take the sort of a clarifying
question now i mean you know so um kai
nugen uh asks um are there any general
purpose so you mentioned several
different ai systems but then in general
a general purpose ai system is being
developed as far as i know machine
learning algorithms are designed to
solve specific tasks uh
with deep learning cnn rnns and the
current trend if i'm correct is to build
specific models for specific problems so
how about for more well um
i mean there are not really general
purpose ai systems in existence but
certainly
if you look at what open ai is trying to
do um and you look at the
stanford foundation models project was
studying those kinds of artifacts they
are intended
to be as general
as possible
and people are finding all kinds of ways
you know they're officially language
models but you can
you can combine it with vision and and
produce
cartoons and animations um you can do
all kinds of stuff you can do arithmetic
you can write poetry
uh
et cetera et cetera so they they seem to
exhibit some kinds of generality but
um i would say
roughly speaking they can't plan their
way out of a paper bag
so they're not
uh they're not general purpose ai in in
that sense but
um
i think there's a there's a
misunderstanding of what it means to
work on
on a particular application so let me
give you an example
when
jan lacoon and his team at bell labs
were working on handwritten digit
recognition
right they didn't produce a handwritten
digit recognizer they produced
convolutional neural networks
which happen to be useful for
handwritten digit recognition but also
turned turned out to solve the general
object liquid recognition problem
uh for computer vision and a few other
things as well and useful for speech too
um
so this is this is true almost all the
time right when people were working on
chess they weren't working on chess
programs and alphago isn't the go
program
right it's a general technique that's
instantiated for that particular thing
but in order to make it work
they had to improve the state of the art
of general techniques and they use alpha
zero now to beat all sorts of different
kinds yeah so alpha zero was was
demonstrated to to be
the world's best chess player the
world's best co-player was plus shogi
player
um and so on
so you know what what is alpha zero not
good at well alpha zero
the board to be fully observable
which is which is a very
general
thing it's not nothing to do with chess
or go it's just
to do with whether you have direct
access to the state of the world or not
right and so if you want to make it more
general you've got to relax that
restriction
when you do relax that restriction
right then it covers all kinds of stuff
including driving and
uh you know robot factory operations and
you name it
so so these advances
even though they're made in the context
of working on a particular application
end up massively
increasing the scope of ai
okay
so
having said that and this is a good yeah
good follow-on to the question uh
we don't have it yet and there are there
are big pieces still missing um
i think the biggest one is
is long-range thinking at multiple
levels of abstraction so if you take
alpha zero
right and it's it's incredibly good at
looking into the future in in the game
of go
it looks ahead 60 80 moves sometimes
far more than people can do
but if you took that same thing that
same approach and you put it in a robot
that has to send motor control commands
to its motors every millisecond
then
80 moves doesn't even get you a tenth of
a second into the future
so for doing something like laying the
table for dinner it's completely useless
right uh and so
how do people lay the table for dinner
well they i mean they still we still
have to send motor control commands to
our muscles every few milliseconds
um and yet we're able to do these things
that take tens of millions or hundreds
of millions or even
trillions
of
low-level actions like doing a phd is
about a trillion low-level actions
right uh yeah we managed to do them and
we managed to decide to do them or
decide not to do them
right
because we operate at many many many
levels of abstraction and
we don't yet know how to make ai systems
that function in this way that that can
create these levels of abstraction that
can seamlessly
uh execute at all these levels at the
same time
you know and it's amazing that
the human body always has
the low-level stuff ready to go somehow
the high-level stuff is feeding things
down to the low level so that it always
is doing stuff it's just amazing um
anyway so these things are very
unpredictable right these are these are
breakthroughs
um you know we've had
some people argue a few dozen
breakthroughs in ai since the beginning
of the field
but we need more
and it's very hard to predict and john
mccarthy he was asked in 1977
how long do you think it'll take before
we have human level ai and he said
anywhere from five to 500 years
um and i think you know that's
that's still about uh about where we are
except we are we are clearly further on
so i'd say maybe five to a hundred years
i think it would be uh
more reasonable there's a website called
metaculous that has a lot of predictions
of different experts and and one of them
is when will we have
um
weak egi which they didn't define in
some detail about doing certain things
and it's interesting because um uh two
weeks ago the date was 2043 and then uh
last week it dropped down to 2034 was
the expert prediction perhaps
dolly and some of the other yeah yeah
i've seen people get excited about delhi
and
i have to say i look at it and i don't i
don't you know it's exciting well no
actually i'm like puzzled like how is it
doing that oh you are basically
would you have you updated your your
estimate uh what do you think a little
yeah yeah well
it makes me a little uneasy um
yeah
so this unpredictability right is
something that we've seen before right
so the last time we invented
civilization ending technology
uh was with atomic energy and
the physics
establishment right so from
e equals m c squared in 1905 onwards
right they knew
how much energy there was stored in
atoms and they could you know they could
even describe the potential explosive
power of an atomic weapon
um but the physics establishment
continued to say that it was impossible
um and lord rutherford who was the you
know the guy who split the atom
um and he was asked do you think you
know in even in 25 or 30 years
we might be able to liberate atomic
energy and he said moonshine right
nonsense completely impossible right
even einstein
thought it was
um and then uh leo zillard read about
rutherford's speech in the times
the next morning and went for a walk and
invented the new utron induced nuclear
chain reaction
right
so um
so these things are really hard to break
i think fortunately we probably need
more than one such breakthrough
to reach general purpose ai
um
but uh to bet against these kinds of
breakthroughs and in fact it was
because they bet against these kinds of
breakthroughs that they made no
preparations for the
advent of atomic weapons
none whatsoever
and uh and so zillard in fact um
realized immediately
what the consequences would be and he
kept his
uh his discovery secret and he patented
a nuclear reactor but kept this patent
secret
um the french patented a nuclear bomb 39
and they kept it secret
um
but it was sort of too late right the
cap the cat did get out of the bag the
germans also
were germans i think were the first to
demonstrate efficient reaction
so um
so i think we were very lucky that uh
things turned out the way they did
so i think we need to prepare
for a possibility of success in ai that
we'll have these ai systems
that are able to make better decisions
than us not just on the chessboard and
the go board
but out in the real world and
so
so that capability is what gives humans
power over the world right that's what
what makes us the number one species
and uh all the other species basically
continue to exist only because
we allow it
and
so if we're creating systems that are in
that sense more powerful than human
beings
i think this is this is the underlying
sense of turing's
uh comment is that uh he doesn't see a
way for us to retain power
over more powerful entities forever
um
and so that's the question that i think
we have to answer
and i think one way to answer is to go
back and say you know if if making ai
better and better makes things worse and
worse maybe we are doing the wrong thing
in the first place right
and if you look at this definition
which is not just
common
it's not just part of ai but i think
control theory
it's the same definition right you're
minimizing a cost functional
uh in statistics you're minimizing a
loss function in or you're
maximizing a sum of rewards and
economics you're
maximizing utility or social welfare and
so on
um
this this methodology is it's very
powerful right you create machinery
that optimizes
an objective but you have to supply the
objective to the machinery and then off
it goes
right
um
but in the real world we can't specify
the objectives completely incorrectly
and
you know this is not a new observation
of course right i mean
economists have been talking forever
about why gdp is a a bad measure of of
global welfare and so on and so forth um
and norbert weiner talked a lot about
the fact that if you're going to put a
busy but if if you're going to put a
purpose into a machine with these
operations you cannot effectively
interfere you better make sure the
purpose is the one we really desire
right and um but over and over again we
see that we can't do that
um and this is again not a new
observation right so the legend of king
midas
he said i want everything i touched to
turn to gold that was his stated
objective he got exactly what he asked
for
and then of course his food and his
drink his family all turned to gold
and that's the end
and the sorcerer's apprentice
specifies uh
that the brooms should fetch the water
because he's too lazy um but he forgets
to specify the exact amount
uh yes
then the the sorcerer has to come back
and turn off the brooms because he
doesn't know how to turn them off
um and then if you if you get three
wishes from a genie
um
uh my third wish would be please can we
fix
audio visual equipment
please undo the first two wishes right
because
always those wishes end up backfiring
because they're misspecified if you made
it made a mess of the universe
so
so i think this is the core of the
problem
um and you can see
that uh we're already starting to suffer
from this
as ai so we didn't suffer from it when
ai was only in a lab right when we were
only doing toy things playing virtual
chess on virtual chess boards
but um
when you look at social media right
they're set up
with their learning algorithms that the
algorithms that choose what billions of
people spend hours every day reading and
watching
um
and they're set up to maximize some
objective so click-through or engagement
or various other kinds of proxies
and you might think well
okay if if i want someone to click on
something i need to send them something
that they're interested in so
so i have to learn what people want
that sounds good right it's sort of
the you know perfect example of
economics at work
right but this is not the optimal
solution
to maximizing click through
right the optimal solution to maximizing
click through
is to modify people to be more
predictable
right and this is what reinforcement
learning algorithms do
uh they choose a sequence of actions to
uh to change the state of the world in
this case the state of your brain
uh so that in future they get a higher
stream of rewards
so for any given individual
they can learn
how to best
propagandize that person how to
brainwash them
to change them into some other person
who's going to be a
more voracious consumer of whatever
material exists uh when they're in that
part of the political space for example
um
and i think there's obviously the story
is more complicated because there are
there's a lot of humans who are
complicit in this
uh in terms of generating content and
uh and so on but this is i think
basically what's going on
uh with social media and
i'm hoping that we're actually going to
be able to get access
to the raw data to
uh and to to make randomized controlled
trials to show that this is really
happening
um
and you can see that um
if the ai systems were better right
these are really simple
learning algorithms they don't know that
people have brains
uh or political opinions or anything
right they're just uh
they're treating a person as a
as a click sequence
right
text of the article or
you know descriptors of the video uh yes
or no did they click on it
uh and then a sequence of those and
that's all that's all a human being is
to these algorithms
but if they were better at psychology
and so on then they'd be more effective
at doing this
and they're already i think pretty
effective
mainly because they can they can nudge
you
uh you know tens of thousands of times a
month
and
only little nudges but tens of thousands
of little nudges can move you a long way
psychologically
um
so if we made the ai system better right
the outcome would be much worse than it
already is
so we need a different model right we
need to get rid of this idea that to
make an ai system you write down an
objective you make some optimizing
machinery and you put the objective in
and off it goes
um
so we're getting rid of that one
and i'm going to try this one instead
which is only a small change right we
just
instead of their objectives right which
are the things that we plug into them
we're insisting that what they do is
beneficial
for our objectives namely what we want
the future to be like and those coincide
if we're able to write down our
objectives completely and correctly and
put them in the machine
but all our experience tells us that
that's not
possible
and so
we're going to have to have machines
that actually know that they don't know
what the objective is
right
so this sounds this problem sounds like
well how can they how can they choose
actions that have
achieve our objectives if they don't
know what they are
right uh so it sounds like i'm just
setting up things to be uh unsolvable
but actually it can be solved
um
and
uh so i wrote three principles just
well because asm offroad three
principles um
but they're a little different and it's
interesting to see how they're different
so the first goal is a little bit like
asimov's you know you can't harm humans
right and you know sort of the
complement of that you have to satisfy
human preferences so preferences here
means
um
not what kind of pizza you like but
what future you prefer in fact what
distribution over futures you prefer so
it's the full
uh preference structure that
you know von neumann posited
that you would have over all possible
futures and all possible lotteries
overall possible futures
so i said a very very
large complex thing that's not explicit
in anybody
and
that's but that's what it has to do and
we'll talk a little bit more about the
fact that there's obviously more than
one human but the key point is that the
robot does not know what those
preferences are
right
and um
this turns out to be i think the key to
how we actually
retain control over the machines forever
and then the third principle
provides a way of connecting preferences
and behavior so it's sort of a grounding
for what what do we mean by human
preference is the things that drive
human behavior
and therefore if you observe behavior
you can infer something
about underlying preferences
it's a complicated process because
our behavior isn't the perfect
reflection of our preferences we are not
perfectly rational uh so really to do
that inference you have to understand
something about human cognition
but you can turn this into a
mathematical
framework called an assistance game
uh so it's a game in the usual economic
sense with at least one human at least
one robot participant
um
and his assistance game because the the
robots are
constituted to be of assistance to
humans
um so the human has a payoff function
the robot's payer function is the same
as the humans but the robot doesn't know
what it is okay
and that that basically gives gives you
those three principles
and when you solve those games and you
can literally write down these simple
ones and solve them look at the
equilibrium solutions um you get
what i think at least qualitative you
get what we want
we get that the
the robot defers to humans
um
for example it will ask permission
if it if it thinks that here's a
possible plan
but the plan changes the world in some
way whose value is unknown
then the robot has an incentive to ask
permission so that it doesn't violate
the unknown part of the human
preferences
and in the extreme case
it will allow itself to be switched off
because it doesn't want to do
whatever it is that the human wants to
prevent it from doing
right it doesn't know what that is but
if the human wants to switch me off
i must have a reason i i'm happy to have
that happen
right
and so we can show that
it's rational for us to build machines
that solve assistance games
then
they will be
expected net benefit to us
and if you make the ai system better
here then it's
better at understanding what our
preferences are and better at satisfying
those preferences so things should get
better rather than worse
so um
to make that concrete right to make it
so that you understand what it's like to
be the robot i i'm hoping this example
will help
uh
i found it useful
so
so you're the robot
right
your partner
husband wife whatever is the human
okay and it's their birthday
and you have to buy them a birthday
present
right
and you don't know what they want right
and to to factor out the money part
we'll have we buy it from the joint
account so
so we don't have to worry too much about
the trade-off of quality for money
um
but the point is here
that
your payoff in this game
is exactly how happy your partner is
going to be with the president
right
the trouble is you don't know
right so so this is what it's like to be
the robot in the assistance game
okay so now think about well what do you
do
right well you could ask
you could
you know leave pictures of
watches or cars around the house and see
if they notice any of them say oh that's
a nice car or
you know
[Music]
you know holiday brochures or you could
ask their friends
there's all kinds of techniques we could
use to try to infer um
something
more about what they want for their
birthday right
um
so this problem of course is unsolvable
as we know but or i think all the other
all the other cases of of assistance
games right they you know
there is a solution and and it is at
least computable so that's good
so i'll just go through one example um
which is the off switch problem right so
so how do we convince a robot to let us
um switch it off right this is our
pr2 in the lab brett the berkeley robot
for the elimination of tedious tasks
and um he weighs about 450 pounds so he
he's uh he has to have a safety off
switch
on the back
um
but
if he was a normal robot with the
classical objective like you know fetch
the coffee
right put that in as the goal that
becomes his objective
right and he's not it doesn't have to be
very bright to realize that you can't
fetch the coffee if you're dead
so now you've given brett an incentive
to disable his own off switch
right which is exactly what we want to
avoid we want to be able to turn it off
so and this is going to be true for
pretty much any goal
but if we have
uncertainty if the robot has uncertainty
about the objective so it may it may
know that you want coffee but be
uncertain about
lots of other things
it won't do that
right and we can you can set up a very
simple formal model to illustrate
uh how this works
so
unfortunately you can't see that i don't
know if i can move
anyway the robot is at the top so this
is our
uh choosing um initially we're just to
give the robot two choices right so
there's a
uh a very attractive red button that
says push me
right this might start a nuclear war it
might set off all the sprinklers in the
building or the fire alarm
who knows what it does
or
the robot can just switch itself off so
this is committing suicide and we'll set
arbitrarily just set the value of that
to zero
and the point is that this action here
is potentially dangerous to humans
and the robot is uncertain about its
values
there's a
horribility distribution over the value
of the action
and at least in this picture
the mean of this distribution is
slightly positive so if this was the
only two choices the robot would just go
ahead
and push the big red button right so
this is not giving a chance to the human
to intervene right so this is sort of
like disabling my off switch and
plunging ahead
okay but we're going to give the robot a
third choice which is to allow the human
to switch it off right so just say wait
do nothing
and then the human can choose to switch
the robot off getting to the same state
of value zero right
or the human can allow the robot to go
ahead
so
what's the point of doing that right the
robot can already switch itself off
and it could already do whatever he's
going to do here
so what does he gain by allowing the
robot allowing the human to to switch
him off well he gains information
right because if he's allowed to go
ahead
um then uh it's because the human thinks
that pushing the mid-reg button is
better than switching the robot off
okay so that quadrant disappears
and if you do a simple right this is
exactly analogous to non-negative value
of information
right the
the robot says yeah i have a positive
incentive to allow myself to be switched
off as long as i'm uncertain
about the decision that the human is
going to make
as soon as that uncertainty goes away
then um uh the incentive to allow
yourself to be switched off goes away
peter yes
robot thinks the human is wrong
so we can we can allow for that so in in
the paper uh it's called the oswitch
problem um we allow for the human to be
for example
uh boltzmann rational
and as you turn up the degree of
irrationality all the way up to being
completely random
then you need a larger margin
before the robot's going to allow this
and so
you don't if if you have a if you're a
self-driving car and you have a
two-year-old in the back
right you don't want the two-year-old to
switch the car off right but if it's
you know if it's a competent adult then
maybe you do so
so there are um you know there's another
case of you know what if the human is
grumpy and doesn't want you to keep
asking
uh right then again right you need a
larger margin before you'll ask the
human for permission
um and the human pays for that right the
grumpy they are the more often the
robot's going to do something that
they're not completely happy with it's
really kind of the ratio of the human
versus the machine and like in the limit
as the machine
has much much better knowledge
of the world than the human does
the machine might always think that look
this human thinks x is the right path
but i've thought about this a lot more
carefully
so well so there's two parts right
there's
there's knowledge of the world and your
ability to predict but there's also
knowledge of human preferences yeah
right so this it's really uncertainty
about preferences that
are distinguishing those yeah
and then i mean this may be a sort of a
little bit of a special case but but
human preferences i mean one of the ask
questions here you know humans can be
irrational um humans have self-control
problems they say like you know please
keep me from smoking more cigarettes or
you know tie me to the mask so i don't
you know sail the ship into the
um
uh sirens i mean so
i don't know if you want to consider
those kinds of cases where the person's
yeah yeah i'll talk a little bit about
that when i get to the real human slide
okay right but for the time being we're
just rational we're just thinking about
you know an idealized human yeah these
are all yeah i'm sure many of you will
have many questions uh and i will try to
list most of them at the end
um
okay
so um
yeah so
you can elaborate this in various ways
by making the human
a little bit less rational a little bit
more grumpy et cetera et cetera and the
key but the key thing
remains here that
there's sort of a direct mapping from
uncertainty about human preferences to
willingness to allow yourself to be
switched off
and this is the only way i think we're
going to do it it's no good
saying well yeah okay the robot
doesn't want to be switched off but we
can outwit it
right i remember once i was called by a
film director who's making a new film
about super intelligent ai
and he says
so he starts describing the plot and all
this stuff
and he wants he wants me to be a
consultant
to help him understand how the humans
outwit the superintelligent
ai said sorry i can't help you
it's just um
it's it's not in the long run it's not
going to work that way
so um
i'll just run very quickly through some
of the questions many of which will have
occurred to you already
um
so what do we do about many humans
right we have the same social choice
kinds of questions that moral
philosophers and economists have
grappled with for
thousands of years
we're also going to have many machines
right so
there's probably will be billions of
independent possibly communicating but
independently manufactured intelligent
systems
operating in the world and even if they
all
conform to these design
design rules
there's a question of whether they're
going to have unfortunate strategic
interactions with each other
and
so we're looking into those kinds of
of questions um
coming back to
humans who are not perfectly rational or
grumpy
right um
you know a simple example would be lisa
doll
right he's a brilliant go player but he
played losing moves in that game with
alphago
right and we don't want alphago to say
oh i guess lisa doll must want to lose
the game that's weird but
you know that's the only explanation for
for him making that losing move now of
course it's not the only explanation
right the explanation is he's not
perfect he has computational limitations
and so he played a move that he thought
would win but in fact lost
and so if you're going to
infer preferences from behavior you have
to take into account actual cognitive
capabilities and that includes emotional
uh behaviors you know things you
immediately regret right you don't want
to take that as being
definitive of human preferences
we also have to actually if you look at
the textbook right every chapter
says okay well you know this is the
definition of a markov decision process
right states actions rewards transition
model right you so you've got to plug
the reward in planning chapter has goals
the
constraint satisfaction chapter has
constraints right these are
all definitions of objectives that have
to be supplied up front
to the algorithms and we don't really
have
algorithms and complexity analysis and
technology
that can accommodate uncertainty about
objectives so we have to redo all of
this stuff
and then
produce
demonstrations prototype applications
showing how
this approach actually uh can work and
actually produces better ai systems than
the classical methods
so i'll briefly talk about some of the
questions that arise with many humans um
so obviously the there are you know
nearly eight billion of us and so
system can learn
you know eight billion preference models
uh and that's fine right i mean facebook
already has probably three billion
preference models in its database
and so 8 billion is not so much to ask
um
and i think we'll find that
and this is this is an entirely amateur
uh pronunciation but i think we'll find
that there's a great deal of commonality
in those preference structures
uh you know despite all the so-called
uh you know differences in uh attitude
between east and west and north and
south and so on
i think
ultimately these are more differences in
circumstance
and what people want the future to be
like
is fairly similar not identical but
fairly similar which then helps a lot in
adapting to a new individual you can
have a reasonably strong prior
um
for what what kinds of things they're
going to like
um
and uh you know the the key question
which is social choice theory is how do
you make these trade-offs
uh when you're making decisions
that are going to affect more than one
person
and um
in in the book that i've written about
this human compatible i sort of
take as
a as a possible answer something like
preference utilitarianism
um but there are a lot of still
unanswered questions even if you
even if you take that uh
straightforward mathematical approach
which just says basically add them up
right treat everyone equally add up
their preferences choose the choose the
policy that maximizes the
sum of utilities of all the individuals
so one objection
uh from nozick is
uh that you can't make interpersonal
comparisons of preferences
um and uh can arrow also said this this
is an axiom
at the beginning of his impossibility
paper there's no meaning
to uh interpersonal comparisons and
preferences so literally
you know jeff bezos having to wait one
microsecond longer for his private jet
to arrive
can't be compared with you know
a woman in the ukraine watching her
family being
uh incinerated
like does anyone believe that
i i don't even think ken arrow believes
that but that's the axiom
in the paper so
uh i
but i do think there is a non-trivial
question because i have four children
and i'm pretty sure they have different
preference scales
right i think they maybe vary by a
factor of 10 or so in the sort of extent
you know the sort of top to bottom scale
that they have but not by a million or a
billion
so the preference utilitarian theorems
right the social aggregation theorems uh
typically assume individuals with common
beliefs um and we'll see that with
different beliefs you get quite
different answers for social aggregation
um
since the late 19th century people have
have asked well what about decisions
that uh that change who exists and then
what do you what are you comparing
right um
and so thanos if if you've seen the
movie
um this is thanos right so he collects
these infinity stones and that gives him
the power to implement his policy
uh which is that if you got rid of half
the people in the universe the other
half would be more than twice as happy
so he does his little utilitarian
calculations as yeah got it you know
he's completely at peace with his
decision
to get rid of half the people in the
universe
so we don't necessarily i mean that
might be the right answer but
we don't want
ai systems making that decision yet
uh
and soon
we don't know when exactly but at some
point they will have sort of thanos
levels of power
and we have to answer these
philosophical questions before that
right and we have done these things in
the past right this chinese one child
policy
probably got rid of 500 million people
like so like more than 10 times the
number who died in world war ii
uh
was that a good decision or a bad
decision well
i don't think we know yet
we don't know an answer to that question
um
so uh then there are complications when
uh
we introduce things uh in the fact that
people's preferences include
the well-being of others
uh some of us are altruists some of us
are sadists
then there's relative preferences
so positional goods as economists call
them
which
mathematically operate the same way as
sadism
in the sense that uh you know what
if uh i'm happy if i have a bigger
shinier car than you do which is true
whether i make my car bigger and shinier
or i make your car smaller and dirtier
right um and so uh so you get this uh
this dynamic that's very similar to the
pure say those statism which most
preference utilitarians rule out
ruling out
relative preferences
is a much bigger thing to do
okay so just well
okay we're really running out of time
very quickly um so i mentioned this idea
that
um when we aggregate human preferences
right the the social aggregation theorem
from hassani says every pareto optimal
policy
uh
optimizes a
linear combination of preferences and
then he sends up by symmetry it should
be just
uh
everyone should have weight one in that
linear combination
um
but for this to be true you need to
assume that all the people have the same
belief about the future
and
if you
um
if you take away that
uh that assumption so now people can
have different beliefs about the future
which is a much more reasonable
assumption
um then the praetor optimum policies no
longer look like
aggregation at least
static aggregation
they change the uh the weights in the
allocation
according to whose predictions turn out
to be right
so exactly proportional to the
likelihood
that your prior
accords to the observed data
as the world unfolds
right so you can see that the
the weights would actually end up
exponentially different
uh in a fairly small fairly small amount
of time
right
um so this is a very weird theorem
um but it's true simply because
uh
everyone thinks that their beliefs are
the right beliefs otherwise they would
change them right
so everyone thinks they're going to win
this game and they want they want to win
this game because they get more in this
game than they would in the static game
so everyone likes this and so they won't
allow any kind of static policy they'll
do this one instead
so i don't know what to think about that
uh okay i'm gonna skip just very briefly
mention okay to deal with the millions
of robots we need some kind of open
source game theory where
where algorithms can expect inspect each
other's source code or inspect proofs
about the source code
in order to then deduce what the
equilibrium is going to be in their
interaction and actually typically if
things go well they can cooperate
immediately without any any kind of
acculturation process between the robots
and then real humans
we already talked about some of these
issues computational limitations
emotional behavior
preferences for autonomy are really
interesting right
one way of thinking about autonomy is i
want
i want to be free to do something that
is not in my own best interest
right
so that means that the robots
to respect autonomy
have got to allow us to do things that
are not in our best interests so it's
like we don't we don't want them to just
keep us on the freeway and close all the
off-ramps
right
um so in some sense they
have to stop predicting what it is that
uh we're going to do
so you get some interesting
self-referential kind of optimization
problems plasticity is the biggest
problem
with this approach in general and i
think with all kinds of
approaches based on rationality is the
fact that our preferences can be changed
by external
factors
and that presents a question of who is
the ai optimizing for you today or you
tomorrow
how do we make sure they don't
manipulate our preferences
and as amateur sen pointed out right a
lot of our preferences are there because
someone else wants us to have those
preferences
uh particularly those in power want us
to have preferences that uh that keep
them in power
uh and keep you in your place
and so those preferences may not be ones
that the ai system should take at face
value
uh if you're interested there's a
non-technical book
human compatible and then the fourth
edition of the ai textbook has some of
the technical stuff although it's still
mostly
the previous version of ai the
ai with fixed objectives so to summarize
i think
if we succeed the upside is enormous
but if we succeed within the standard
model
i think we're going to get the downside
which is uh disastrous
um and i think if we change the way we
think about ai maybe we can get the best
of both worlds
and this kind of ai would actually be
it's not just a matter of like it's
safer or anything like it's just better
like this is the ai that we want to have
it'll be
more adaptable more flexible more useful
to us
as well as being safer
so i think the economics should drive us
in this direction
and we should stop thinking of this as a
battle between the ethicists and the ai
researchers
where the ethicists wag their fingers
and their ai researchers said
go away and leave me alone right it
should be that when an ai researcher
gets up in the morning this is what they
mean by
doing ai
right just as when a doctor gets up in
the morning
they say okay i'm going to kill some
people today like they don't have an
ethical battle within themselves about
that at least i hope they don't
right so uh that's it thank you very
much
thank you so much stewart um that was
incredibly uh provocative and
interesting i i have a ton of questions
i know everyone here and online a lot of
questions unfortunately we are out of
time so we're going to have to uh sorry
point you to your your books you know
this for to dive in deeper on that i'll
just let people know that next week our
speaker is going to be marshall van
olstein of bu he's going to be talking
about free speech platforms and the fake
news problem so look forward to seeing
you all then thanks very much
stuff