Stuart Russell – AI: The Story So Far – CSRBAI 2016

good morning everybody and welcome to
the colloquium series so I'm really
excited for today's lineup of
distinguished speakers starting with
professor Stuart Russell the professor
of computer science and the Smith is a
professor in engineering here at the
University of California Berkeley so it
would take too long for me to list his
qualifications and awards and
contributions so I will include the book
that he co-authored with Pierre Norvik
artificial intelligence and modern
approach which is now being used by by
universities and the quadruple digits in
countries and the triple digits across
the world and and also sort has been a
powerful influence on the field of
artificial intelligence starting to take
seriously the positive and negative
effects of future advances in artificial
intelligence on the world and the things
that we care about in that respect he's
been talking to influential groups major
conferences as well as the Davos World
Economic Forum recently and so we're
very pleased to have him here today to
talk to us about the prospects for work
rewinding the field on provably
beneficial artificial intelligence and
and also mary has been pleased to have
him as a research adviser helping direct
us in and what things are important for
us to work on so so I would like you at
all please join me in welcoming our very
first speaker professor Stuart Russell
thank you very much pets so um I made a
last-minute decision to switch to a much
shorter talk and that will give us
hopefully much more time for discussion
so I'm gonna just dispense with the
usual preliminaries of this where I talk
about you know what is AI and what's
happening now and look at all this
amazing progress and all these
milestones and so on and just say look
let's take it as a given for the sake of
argument that
eventually we will exceed human
capabilities and some still not very
clearly specified since you know partly
because we don't really know what human
capabilities are but if we think about
what it means to make decisions and how
to make better decisions it means if you
can take into account more information
if you have a better model of how the
world works and you can compute more
extensively on that model and look
further and further into the future so
think of this as like alphago moved from
the NGO board to the whole world then AI
systems are gonna make better decisions
than humans and I put an asterisk a sone
asterisk is something that linguists use
to mean this is this is not quite a
felicitous expression in the natural
language and so what could I possibly
mean by putting an asterisk on better
well there's a piece missing not just
taking into account more information and
looking further into the future but what
is the objective that's being optimized
in making decision that turns out to be
a crucial point so so the upside as as
Nate mentioned is it's pretty large
because pretty much everything we have
is the result of our being intelligent
and so if we had more intelligence at
our disposal to use as tool as tools we
could do all kinds of wonderful things
and you know the each of these areas is
something that that have they've been
problematic for the human race forever
pretty much and the last one ecological
declaration is getting much worse and it
seems like well it couldn't hurt to have
access to more intelligence to help and
you can even imagine very concrete ways
where it might be very useful so one of
the biggest issues when you look at
poverty and disease and war is actually
communities it's not that we don't know
what to do about these it's actually
that we have we have difficulty in in
management of collective
decision making and implementation
processes that a I can clearly help with
sort of if you like global distributed
government governance at a sort of micro
level where lots and lots of people have
to do lots and lots of things for this
to work well so in the long run we could
get away from the constant you know
fight with ourselves and fight with
necessity in sort of physics
and actually choose how we want human
life to be so that would possibly be
very good or not I mean there's another
at least release we have a choice
whether we know how to make that choice
that's another question but at least it
be nice to have a choice
and then the downside well everyone
knows about killer robots and everyone
knows about the end of employment and
then this is other thing the end of the
human race which seems to be a very
popular theme these days yeah but I
would say most the discussion about this
theme has at least in the media and when
I meet people when I go around giving
these talks everyone seems to almost
everyone seems to got hold of the wrong
end of the stick
you have many wrong ends of the stick
but there is a sort of a general sense
and this has been this goes back to
certainly to Alan Turing saying that you
know I expect at best that they will
keep us as pets or something was that
effect that if you make something that's
much more than you are then you we might
sort of find ourselves in the situation
of the gorillas so here they are having
a meeting and this guy is falling asleep
you can tell it's a meeting and they're
talking about whether it was a good idea
for their ancestors to have created this
human race these these human things
which are much smarter than they are
they're having a really hard time with
this this issue and I think they pretty
much concluded that it was terrible idea
because now they don't have any control
over their own futures and they could
easily go extinct and it's say if they
had
ability to conceptualize their own state
they'd probably be very sad about it but
that's a very inchoate fear and then
that gets translated in the media into
all kinds of things like oh you know
armies of killer robots are gonna
spontaneously rise up and and decide
that they hate human beings and so on so
forth right so you know you know all of
the you know Hollywood sometimes gets it
almost right and mostly gets it mostly
wrong so more specifically right the
problem is is this right that they're
gonna be incredibly good at making
decisions and doing stuff but somehow it
isn't the right stuff I mean if they are
incredibly good making this sense and
it's the right stuff I
they really are helping us realize
whatever it is we decide we want to
realize you know that's that's that
would be what we want so it must be
because they're not quite doing that
they're doing something else they are
the objective that they're making
decisions on is not the right one
and unfortunately AI by lodge and these
other areas operations research control
C and so on all assume that that
specifying objective is actually not
part of the problem at all right it's
just you know the user who knows what it
is
and you know and you can control theory
it's like a squared error with respect
to the reference trajectory Y squared
error well because that makes the
equations easier but it doesn't have
much connection to actually what
anything anyone really cares about so so
actually there isn't a lot of help right
when you say okay we have better we've
got to get these objectives right
otherwise we're screwed okay what
discipline can I turn to
the answer is not really there isn't a
place to turn and so no but we know
pointed this out so this is a very
useful paper I don't know if you have a
reading list Nate for for the group but
there's an there's
there's a nice paper I often point
journalists to this paper so he wrote it
in science I think in 1960 and it was in
 as a result of looking at Arthur
Samuels checker playing program that
learned to be better playing checkers
then an office under was so it's a very
early demonstration refuting the usual
claim that all you know machines can
only do what we programmed them to do so
we don't need to worry right and so he
said okay if we use to achieve our
purposes in mechanical agency with whose
operation we can't interfere
we better be quite sure that the purpose
is the purpose we really desire and that
that's a pretty clear statement of the
problem from 56 years ago but arguably
that that statement could have been
written by King Midas whenever this is
some uncertainty about the date all
right have you tried to write it down
yeah in the paper as well so so this is
a writing the story of King Midas is
actually both in microcosm and macrocosm
a lesson for Humanity right so the
whoever it was it was granting King
Midas's wish took his objective
literally and and then it was too late
right once his food and his wine and his
daughter all turned to gold
he couldn't undo those things and he
said damn you know I wish I had said it
right and this is often in with these
stories in other cultures you know
there's a genie in the genie grants you
wishes you know this is in going back to
the time of King Solomon and in the
Jewish culture and in Arab cultures and
lots of others as a version of this
story where you ask for wishes you get
what you want a man you know your last
wishes please undo the first two wishes
because I got them wrong right
and then in the macrocosm right this is
actually telling the universe or perhaps
what you what we are wishing for
right the ability to automate and have
sort of super control over everything in
its or unlimited powers and they
actually be a poisoned chalice for the
human race in general not just through
the individual so we better be more
careful about our macro policy and so
Steve Omohundro
pointed out some some additional
problems are not just that when you have
the machine with the wrong objective
right in some sense you're you're
setting up a chess match or a go match
between the human race and the machine
that's busy pursuing the objective
that's wrong and we know what happens
with those matches so but Steve pointed
out that it's actually worse than that
because if you give a goal to a machine
then even if you don't ever mention to
the machine that it should preserve its
own existence
so I mean Asimov didn't need to have the
third law saying that machine should
preserve avoid harm to themselves
because actually unnecessary right they
will nonetheless form this as a sub goal
because you you can't fetch the coffee
if you're dead so you give the machine
they're gonna fetching the coffee the
Machine figures out based on physics
that if it's dead it can't get the
coffee so it naturally has a sub goal
not to be dead right as a consequence of
needing to get the coffee this is a very
straightforward point and also you know
it can improve for sort of typical goals
in the real world you improve your
chances of success by having more
resources more computational sources
more money and so on so all other things
being equal you're going to want to
acquire more of those so then if you
have a machine that has the wrong
objective and he's gonna have these
things as sub goals then you can clearly
see that you're gonna have how like
problems so that's the high-level story
and it's it's a pretty straightforward
story and then there have been a number
of arguments about why
nonetheless we should pay no attention
to this issue yeah so so I thought it'd
be helpful to go through some of those
and we can discuss in further after the
end but you will come across these you
probably have come across many of them
already so one of the first responses
I'm sorry this colors not ideal for for
the lighting situation could we maybe we
could turn the light yeah we thought
they were low enough but in fact it
wasn't low enough given they chose the
wrong color okay orange okay yep so one
all right so orange is these are things
that other people say right so one
typical response is it's never going to
happen right or you know we're not going
to achieve human-level AI and so it's
pointless to to worry about this or it's
it's so far off in the future that it's
it's completely ridiculous and you know
if I think if it was true that if you
went to people back a million years ago
you know who figured out how to make
fire actually pre humans and told them
that this fire stuff was gonna cause
global warming and they should stop
right I think that was probably like
that would be not good advice so if you
know if a I was gonna happen you know a
million years in the future then yeah
probably
it's too soon to to even think about
what we might do but I wanted you know
so I so in response to that I sometimes
point to a historical example this is
Ernest Rutherford who was the most
famous nuclear physicist of his time so
not a weird fringe dude but actually the
main guy in nuclear physics and here's
what he said on September 11 of 1933
essentially that it will never be
possible to to get energy out of atoms
right they knew that the energy was in
there based in they had done the mass
defect calculation they knew the
equals M c-squared they knew the amount
of energy that was there but his
considered view which he expressed in
many ways in many forms and many times
was that it was impossible to ever get
it out and even Einstein kind of agreed
with this and then that was September
11th he he said this at a meeting of the
British Association for the Advancement
of science and it was reported in The
Times and Leo Szilard read this in The
Times the next morning and he got
annoyed and so he invented the neutron
induced nuclear chain reaction and
within a few months he patented early
version of the nuclear reactor you know
with negative feedback control
mechanisms to to damp out the critical
reaction soon after that people were
patenting nuclear bombs and and so on so
forth so it went from never to 16 hours
and so it's very hard to predict these
things and I think just saying well I'm
an expert and it's never going to happen
he's not good enough argument and this
was what he wrote so after he did it he
did a demonstration of a natural fission
reaction and he said you know there was
little doubt in my mind that the world
was headed for grief because at that
point they were also in an arms race
with Germany and he anticipated that
there would be nuclear conflict with
Germany ok so a version another version
of that is it's too soon to worry about
it you know if you if you ask many
people when do you think is likely to
happen you know I generally try to avoid
giving predictions because precisely
because it for the nuclear physics
example I think it worked quite so it
requires breakthroughs but it's very
hard to say when those are gonna happen
but if you ask people in the field or
near the field they'll say you know give
you some number that looks like 50 to 75
years some people earlier but not that
many people think it's not gonna happen
this century right so so if I said that
you know in 50 years time a giant
asteroid is on course to collide with
the earth you know when we saw it's way
too far away
to even worry about it or even start
thinking about the problem you know so
come back in 58 years sorry 48 years and
then won't like them won't give you some
funding to work on it that wouldn't be
the kind of response one would expect
and arguably for climate change the
right time to intervene would have been
around 1900 when we already knew the
basic physics you know Iranians and
others had published papers you know
giving quantitative calculations the
greenhouse effect and projecting carbon
dioxide and you know influential people
like Alexander Graham Bell had said you
know this is gonna be a major problem we
have to do something but it was ignored
I don't know exactly I haven't looked at
the history of why people didn't pay
attention at that time but that would
have been a time when you could have
intervened before the fossil fuel
industry and electoral electrical power
production became so important to our
entire economy that that it's very hard
to change you know so you could have
started investing in wind power and
solar power improved battery technology
and other kinds of things a long time
ago but we didn't
so my distinguished colleague Andrew
Inge has another version of this story
right it's it's like worrying about
overpopulation on Mars he since changed
that to Alpha Centauri to make it seem
even more ridiculous or perhaps he
thought Mars well that fits it is
reasonable to worry about Rovers I don't
know having seen the Martian I'm not
sure but you know this is it's you know
it's a it's an appealing analogy but I
think is totally misleading you know
another version of this which I saw in a
paper recently was you know it's like
worrying about black holes suddenly
materializing into us a little bit I
mean yeah if they did that would be
terrible but you know there's no
particular reason to think it's going to
happen so it's sort of silly to worry
about it right and the answer to both is
so they're saying well you know if we
were spending billions of dollars to
move the human race to Mars without
thinking about what
we would breathe when we got there that
would be that would be silly right
you know similarly if we were spending
billions of dollars to cause black holes
to materialize in near Earth orbit then
it would be reasonable to ask you know
is that a good idea and you have you
thought about the consequences how would
we would prevent the obvious sequel i
and you know so so I don't find and
doings argument well no no I me see if
you're gonna use the argument that beats
this is just like materializing you know
worrying about materializing black holes
they say no it isn't just like that so
yeah so I mean so in other words the
onus is on someone who says that to to
actually prove that in fact AI is
harmless that it isn't a black hole
because we are spending billions of
dollars to make it happen another
another version of this is well if the
problem comes with giving objectives
like make some paper clips or whatever
to the to the AI system then it's better
not to have us giving the goals the AI
system just let the Machine indent its
own objectives which is a little odd
right I mean it's sort of like saying
you know if you have a problem steering
straight then the best thing to do is
remove the steering wheel altogether and
just leave it up to chance as it were to
make the right thing happen this is this
is something that you see a lot I be M
for example this is a general there's
you know view of why we don't have to
worry well because we're gonna have
these beneficial human AI teaming and so
it's not gonna be you know machines
independently operating and deciding
what to do there's in the human AI teams
of work together but you you can't have
a human AI team unless the team members
all are aligned in what their objectives
are so it's just a restatement of the
problem I mean yes of course we want
beneficial human AI teaming but that is
that in fact making the question how do
you ensure that the AI passed the team
is actually on the team
another common responses well okay
you're right yeah it's really shoe but
there's nothing we can do about it
whatsoever because it's well known that
you can't control research you know
there's no way to put a stopper on human
creativity you know and then that
usually people will show cute movies of
of kids playing you know interacting
with robots and exhibitions and look at
this you know outpouring of human
creativity and there's no way you can do
anything about this and and there's you
know there's some validity of that but
it's not really true right we can and do
biologists deliberately said engineering
the human genome is not something we
want to do and that was a complete
switch because an awful lot of work on
genetics and an early molecular biology
was precisely about the ability to to
improve humans and then it was decided
oh perhaps that isn't an ideal goal for
biology because that opens up a
Pandora's box of you know genetically
--tz-- and all the rest of the stuff
that science fiction has already looked
at so they said no and it's been 40
years and it's still hasn't happened
although it's the rich been reopened
recently with there's this CRISPR
technology although the inventors of
CRISPR also believe that we shouldn't
use it to to engineer better humans
another interesting reaction is this is
just typical Luddite right you're just
attacking AI or attacking technology so
in fact Elon Musk and Stephen Hawking
and their various other people I guess
everyone who signed the open letter on
robust and beneficial AI was included as
when
as of the 2015 Luddite of the Year award
from the information technology
innovation foundation who who seemed to
be vehement ly opposed to any any of
these thoughts and I just think this is
misdirected it's misunderstanding what
we're saying
completely right if a fusion researcher
says fusion researchers need to be
contained in order to be safe right that
doesn't make them a Luddite it's just
complete misunderstanding of what's
going on right they're not attacking
physics by saying that we're not
attacking I mean we're ridiculous to say
that Turing was attacking AI by pointing
out this long term issue or that we know
was attacking AI or Bill Gates is
attacking right right and these these
are people who put a lot of their effort
into creating AI in the first place
so another reaction that you often see
even from very distinguished AI
researchers is Rome's there isn't really
a risk right because if anything we
don't like we immediately just switch
off the machine and that solves the
problem right as if super intelligent
entities couldn't possibly think of that
that eventualities and wouldn't you know
so it's sort of like saying yeah you
know if you're if you're losing a game
against alphago well they just you just
win all right what's the problem
you know just win they're easy you know
some people say well if we could if we
just avoid anthropomorphizing and
putting in these goals like
self-preservation then of course there
won't be a problem Steven Pinker's
version of this is we just make female
ai's they wouldn't want to take over the
world literally he said this this is
just these stupid male AI researchers
who don't get it yeah but you can't not
put it in I mean it doesn't matter if
you don't put it in it will it will
arise anyway because you can't get the
coffee if you're dead so I'm happy to
discuss any of these further on you may
have heard other arguments that you
you're not sure how to respond to so the
proposal is that in fact you know the
part of the problem is that AI is
traditionally conceived for which I
guess I have some guilt in conveying
this idea that that AI is about rational
behavior which means optimizing
objectives you know allows for the past
you know release doesn't think about the
issue of well what if the objective
isn't the one that you actually want to
have optimized so could we change AI to
a different field this should initially
we're going to call it provably
beneficial ai and you can see why
they're asterisk because this is almost
oxy oxymoronic because beneficial is so
vague and touchy-feely and provably
doesn't seem to fit with that eventually
it'll just be called AI because you know
just like we don't you know if you're a
civil engineer you don't say oh I work
on bridges that don't fall down right
you just say I work I work on bridges
right it's just so just intrinsic to
bridge design that they don't fall down
and it should be intrinsic to AI system
design that they are supposed to be
beneficial to you and that's sort of
what it means to do it I so eventually
it will just be called AI but for the
time being we have to distinguish it
from traditional AI okay and how do you
do that so so here's one way and there
are there are others you know that
there's a whole range of research that
can be done on in some sense trying to
constrain behaviors of AI systems which
is I'm not going to talk about but
that's a completely plausible and
interesting and but as yet totally
unsolved direction but if we want to
think about this this question of how do
we get rid of the problem of of
misaligned values well you could say
well the only way to get rid of
misaligned values is to just to get the
values to be exactly the same all right
to get the objectives to be exactly
those of the human race and then
everything's fine that's
but that's too difficult and it's also
isn't quite necessary right what needs
to happen actually so this is number two
is crucial number one is just to point
out in some sense that as Moore's Law is
or what at least one of them is
superfluous we don't want the robot to
care about itself at all it has no
intrinsic objectives whatsoever it's
only objective is to optimize human
values but it doesn't know what they are
right and so this is a if you like this
then it's get you get soft alignment
right that it's at least compatible with
humans because it's uncertain about what
the human objective is and it's as as we
say in power ability the support of its
distribution includes whatever the true
human value function might be even
though the machine isn't sure on which
which of the possible value functions is
right and this turns out to be quite
helpful and then the third part of this
is well ok how yeah we could have very
robot that's very very very uncertain it
doesn't know if humans like losing legs
or like gaining extra legs or just like
having the number of legs they have
right well that's not a very helpful
robot right because now the robots are
less I'm really not sure what to do to
help you ok so you what you want to get
better at understanding human so it
could be more helpful to you and the
information source is there right the
raw data if you like the ground truth is
contained in human behavior because that
reveals information about human
preferences so those three simple ideas
you could put together in various ways
and get to start to make progress so so
a version of the self-preservation
thesis from our mohandro is is this one
way to have a robot that you know it has
an off switch that someone can come
along as press the off switch now the
robots did
right and you know if you take Omaha
murder and literally what he says is
look if the robot has the objective of
getting a coffee you know one way of
failing is that someone comes along and
presses the off switch so if robot has
an action which permanently disables the
off switch so it's sort of an internal
off off switch then then it would
naturally do that right there's no cost
and it gets rid of one branch of the
tree that would lead to failure and so
it's clearly a good idea right and when
you put it like that it's sort of hard
to find even think of a way around it in
fact when you put that into mathematics
there is no way around it it's in fact
you know unavoidable and so but if you
if you avoid giving the robot a precise
objective but instead you allow it to be
uncertain about the objective so for
example it might know that it's supposed
to get coffee but it's uncertain about
what other what the signs of the other
variables and the value function might
be you know so is it allowed to you know
kill people who get in the way of the
coffee machine it's not sure all right
well so then it starts to its behavior
will be different because of that
uncertainty in the value function and in
fact so then you've got uncertainty
about the the human objectives and then
you have to have some attribution of
rationality to humans it doesn't have to
be perfect but it has to be so to me
behavior has to be sort of correlated
with with their objectives and so
roughly speaking then the the you can
think of the human action of switching
off the robot is actually providing
information to the robot about what the
human's true value function is in
particular we know whatever the robot
was about to do is not helping right and
so that's why we're switching off and so
the robot should be happy to be switched
off because that leads to an outcome
that is more beneficial from the human
than the robot disable and be off switch
okay and so and you can when you do the
math that works out and in fact the
margin of safety is proportional to the
allowed amount of uncertainty about the
human value function and but of course
the more uncertainty there is about the
even value functions are less helpful
the robot can be and that seems to be an
unavoidable trade-off okay so yeah sure
then the consequence is it's actually in
the robots interest to to leave the off
switch available so then let me talk a
little bit about this third point value
alignment you know how do we learn what
the value function is how we narrow down
this uncertainty from the dirting
behavior so there's this old Field
called inverse reinforcement learning it
has other versions so in economics and
applied you know consumer theory they do
something called preference solicitation
you know so so many presents consumers
with you know 81 different versions of
headphones and asked them to say how
much they pay for them or which ones
they like better and so on so forth to
try to figure out the human value
function for headphones and you know so
that's the sort of those are non
sequential decision problems like do you
want this one or that one
but there's another field called
structural estimation of mdps where for
example you know the economists look at
when do people have children and then
somehow you figure out the value of
children from from people sequential
child production behavior and things
like that so the general idea is that
the behavior is a is a very complex
manifestation which is made complex
actually by the environment in which the
behavior is produced but underlying it
there's a simple explanation which is
that the human wants some things and
cares about some stuff and and so that's
a if you like the physics of behavior
alright what is the underlying
Laurer physics is the humans want things
and they act to try to get them and so
you can invert the behavior to figure
out what it is they want and this is
this has been around in AI since 98 and
there are quite effective algorithms
that are quite scalable and people have
done there are several hundred papers on
how to do this it's not quite the right
problem for one obvious reason is that
you don't want the robot to adopt the
value function of the human right that's
that's trivial but important sorry if
the robot watches knees struggling out
of bed and wandering down stairs like a
zombie to get my coffee it can figure
out that oh you know you Stewart really
likes to have coffee when he wakes up
but you don't want the robot to want
coffee
that doesn't help right so so it's not
adopting the value function that's
usually how it's done in the inverse
reinforcement learning you know you you
will a copter pilot and now you learn
about desirable helicopter maneuvers and
then the robot doesn't so it actually
adopts the value function so the
framework we developed is a
generalization of that called
cooperative inverse reinforcement
learning which is a game theoretic
setting and you could essentially you
have a human or multiple humans and a
robot or multiple robots and as I
mentioned they the human has a value
function and at least implicitly they
know it or they might not be able to
make it explicit the robot know doesn't
know it and knows it doesn't know it but
if that's its objective to maximize and
and then when you when you solve this
game when you look at the solutions of
the game they automatically produce the
kinds of things that you want namely you
know the robot is cautious it asked
questions the human actually has an
incentive to teach the robot so that
because the faster the robot figures out
what the human wants the more it can be
helpful and new we can actually show
show little examples and so this
actually contradicts the inverse
reinforcement learning assumption
all right the inverse reinforcement
ending assumption is that the human is
acting optimally according to some value
for
and then we observe the behavior and we
try to figure out what what the value
function is but actually in this setting
the human doesn't act the same way as
they would if the robot wasn't there
right they sort of will you know
demonstrate things they'll even you know
point out what not to do right whereas
the human by themselves would never do
that because totally pointless all right
and so you actually get different
solutions and and and so since the human
is gonna behave as it were a non-optimal
at least in the isolated sense then the
the algorithms for learning from that
behavior also have to be different so
the standard IRL learning hours won't
work in this setting and they have to be
revised so it creates a much richer more
complicated and interesting setting so
he's just a very trivial example that
Dylan my student Dylan had feel Manila's
not here right now
so he just did some sort of deliberately
trivial but you have a grid world and
there are three locations that can be of
throats or three centroids of value and
they can have different you know any of
these could be positive or negative and
then they would radiate that value to
their neighboring squares as you can see
here this is a peak of value and this is
the peak of value this is a kit that you
want to avoid and so the optimal you
know if the human or you know a rational
agent is put in this environment and
let's say it starts here then you know
the optimal behavior because we're
slightly to the left of the the center
here the alto behavior is to go directly
to the left-hand peak of value and then
stay there right that's that's the
optimal solution for this environment
but and then what I've shown here is
okay if you see that behavior and you
run IRL right then you will conclude
this gray what I mean this grey map
shows the conclusion that the IRL garden
draws about what is the value function
underlying this behavior okay
and in fact there's in the posterior
over value functions this is now whereas
in truth it's highly positive it now
looks slightly negative because the
robot didn't go to the right right and
therefore that rules out the possibility
that that this is the highest value
square right and then so the the mean of
the posterior is actually sit now
slightly below zero so to speak it
definitely didn't go down so it's pretty
sure that's not a good idea either
right so you get the wrong conclusion
from observing the behavior and in fact
if you solve the if you solve or you
actually this is one round of best
response in the game so it's not a
complete solution to the game but the
the one after one round a best response
to what the human does is actually to
visit both of these regions of high
value and then this shows the posterior
that they learning out of them obtains
and it's much closer to the true
posterior from compared to that one and
so this is just a trivial observation
that the solutions of these two player
games are different from optimal
behavior by one agent observed by a
second agent that's trying to figure out
the balance
okay so then looking looking ahead we
know beyond trivial toy examples and say
okay let's imagine if we take this
seriously we are actually going to need
to figure out to a large extent what the
human value function is and that's you
know that's easily a twenty thirty year
project and it's interesting to think
about well what's the output right it's
like if I if if you guys are a bunch of
venture capitalists and I always hear
saying hey I need funding to to start
this and then think it's okay so what
are gonna sell at the end of is how I'm
gonna sell value functions right well
what exactly does that going to look
like you know so just you just try to
imagine doing this right and taking it
seriously and I think okay well what are
the sources of information
well actually there in all there's
enormous amount of information about
human behavior right so everything
pretty much everything that's ever been
written by humans is actually about
people doing things some of it very
boring like people buying two bushels of
corn and exchanging that for you know
some arrowheads
but even that is really useful
information about the human value
function and your novels and and
newspaper articles and everything else
and every television program you know
there's not a lot of television programs
where they only talk about rocks and not
about you know nothing about what people
do or care about or any of those things
so so almost everything out there is
gonna be useful information a lot of it
is you know in newspaper articles and
levels and everything is it one person
does something another person gets upset
or happy alright that's also useful
information but again it's a form of
behavior it's not it's not direct proof
that wine is wrong any other one's right
but it's evidence that it can all be
thrown into the mix you know if it's
understood properly so you know so that
we in order to do this we'll need to do
natural a new language understanding and
your computer vision to understand all
the TV programs and what everyone's
doing in speech nothing else there's
lots of AI to be done to make this work
but it's easier than building the super
intelligent AI system that we are
preparing for so it's it's it should be
feasible and so that this is this is
this is good news we need to solve this
actually much earlier so this this
startup company the values are us
corporation you know will will actually
have customers fairly soon I think you
know so self-driving cars domestic
robots you know so one example I give I
don't think I have the slides here I
just gave a talk in Korea where I made a
little sort of cartoon sequence of a
robot in the house and then there's the
little kids sitting there and their
plates are empty and they're hungry and
then the robot has to find something to
eat if the fridge is empty
oh and there's a little cute kitty and
then robots oh yeah we'll cook the kitty
for dinner and then there's a newspaper
headline and that's the end of the
domestic robot industry so there's a
very strong economic incentive for
self-driving car companies and domestic
robot companies and personal digital
assistant companies right you know if
they're gonna be helping you book your
airline flights and and making meetings
you know you don't want them to make
meetings with lunatics you don't want to
book flights there Antarctica
and so on so they all need to understand
your value system fairly well so there's
this very very strong economic incentive
to get it right even fairly soon so
that's good all right that means that
this should be this should be part of
the AI industry and we will be
developing the technology so you know
really reasons that these are related
reasons to the concern about super
intelligent AI but they're much more
mundane that the difficulties include
you know the fact that the humans are
complicated some of them are nasty so
how do you you know how do we avoid you
know there's lots of bad behavior out
there how do we avoid learning that we
should be that the robots should be
supporting all these very undesirable
behaviors you know even if it's not
clear the extent to which our behavior
can even be successfully described as as
trying to optimize any value function
there are lots of reasons for thinking
that isn't true including the fact that
evolution doesn't care about us as
individuals anyway like so a lot of
evolutionary theory says no it's nothing
to do with you and your desire to
reproduce it's actually you know small
groups of genes that actually exist
across multiple species and they're the
units of optimization and they're the
ones that are really being selected and
from a even Irish you know even if you
think about the species is a unit right
well as as a unit the
the species if it's going to survive
needs to do both exploration and
exploitation exploration means one way
of having the species explorers by
producing individuals who are completely
nuts right who acted extremely risk
prone ways and then sort of go off and
explore you sail across some ocean that
they think is gonna fall off the end of
the earth and happen to arrive another
continent and things like that you know
completely completely nuts the kind of
stuff that they do on Star Trek right
it's not that the individuals involved
are irrational is that the concept of
rationality in some senses doesn't apply
to individuals at all right it's
actually that they're just fulfilling a
function which is part of the
rationality of the species or the tribe
or the gene group or whatever so so
things can get really really complicated
in understanding the you know the full
spectrum of human behavior and how we
infer anything from it you know we're
computationally limited so if you watch
two people playing chess well you know
one of them loses does that means
because they wanted to lose the game or
no it's because in fact it's because
they you know they're both computing
computation limited and one's maybe
slightly more than the other all right
could be that he's trying to lose yeah
it could be these trying to lose that it
does happen but usually he are not doing
it and so on and of course you know we
there's different you know here all
humans are individuals and then there's
differences across cultures and so on
and then and there are these questions
of trade-offs right that we even if you
do learn the value function of
individuals you can't optimize
everyone's value function because
they're ours enough countries to be king
or queen all of them there isn't enough
money for everyone to be a billionaire
and so on so on so on so so so how do
you deal with those so and these these
are age-old questions in social sciences
so we're not gonna solve by observing
the human behavior but by making
everything much more explicit and
mathematical and empirical hopefully we
can make a lot of progress and maybe
we'll learn more about what we what we
think we should be doing and
that will make us better at doing it
okay so the consequences are various so
the the objective is I think in part to
change how we think of the field to
include these considerations and and and
then ensure that what we're building is
actually the produces behavior that
we're happy with and you know as I said
there are a lot of questions that social
scientists have studied for a long time
and that will have to be incorporate
some of those concepts will have to be
incorporated and then last question is
well there is a lot when you actually
get concrete and say okay we twenty
years time values are us corporation
he's now selling these things you know
what are they going to look like right
it's not at all obvious it I could do it
for chess very easily I could sell you a
value function for chess you know and it
says nine points for a queen and five
points for rook and it's pretty
straightforward right but that's because
chess is fully observable and there's no
argument about whether you have a queen
or not but the inputs to a domestic
robot are the the video sequence coming
in through its camera you're not going
to define value functions in terms of
video sequence is coming in through
cameras right so you know give zillions
of pixels that will be daft all right so
what are you gonna do and that I think
is a somewhat open question but you know
a technically technically important
question to answer so coming back to
Norbert Wiener all right so he in his
paper which I really do recommend
reading you know he points out that that
these questions are incredibly difficult
even a scientist is sort of only seeing
a very local part of an unending stream
that goes on for millennia and might
think that what he's doing is beneficial
that in fact could be entirely wrong and
you have to look over a long time scale
and try and figure out the answers and
it's very difficult but you have no
choice but to try to do it so I guess
that's why you're all here thank you
it's just yes that's a good question I
mean I think the point I mentioned
towards the end that value functions
apply in these partially observable
environments and how you define them
right so you could imagine let me just
take something very simple like you know
is the cat alive or dead right so you
could put you know higher value on the
cat being alive than the cat being dead
but for different robots the mapping
from percept sequences to the
probability that the cat is in fact
alive or dead
would be different and so presumably we
all have to agree on what we mean by
alive and dead and then the robot
manufacturer has to have some recognizes
for that and this is all very hand wavy
but and and then you supply the value so
they're like dead the then the problem
with that is that in fact you know alive
and dead are not well-defined you know
if you talk to anyone like a
neurosurgeon who works in the hospital
it's extremely high in many cases to
figure out if someone's alive or dead
and one of my colleagues told me that so
the hospital allowed him to run
experiments on people who had been
officially declared dead so he kept them
alive on the ventilator or kept them
kept the body's functioning on the
ventilator kept them alive because
they'd already dead and two of them got
what got up and went back to work so
so this is actually you know it's it's a
tricky thing it's not really fine and
this is exactly where the super
intelligence this they find the worry is
they find the loopholes right they find
ways of achieving what you specified is
the objectives that are so you would
just never imagined they would think of
that they're so extremely
counterintuitive but they you know just
like tax law right you think you've
ruled out a loophole so people find this
completely bizarre way of you know they
pay their employees with gold coins
right because that's you know they're
it's a five dollar gold coin it's five
dollars right I give you one each and so
you don't have to pay any tax because
you're only making five dollars a month
but you know you know that that's kind
of an example but you know they will
come up with much much much more devious
ways that you know and in alive or dead
so people in this in the existential
risk literature talk precisely about
situations that you could argue a kind
of in this gray zone or they'll define
in this yes you're still alive but
you're immobilized in a box with with a
heroin drip and so on and and you might
say well that's really you might as well
be dead but no you're alive you've met
that you met the stated criteria so so
there I think is where where the
question of having alignment which is
not perfect right so might one you get
sort of a clash of intuitions if one
says look if you're if you're the value
function of the robot is optimizing is
within epsilon of a true value function
then nothing too bad can happen right
and you can maybe prove that you know
the most you can lose is you know
epsilon squared over one minus gamma or
something right but then you know your
other intuition says but if the robot is
way way way smarter than you you know it
can somehow use that epsilon to as a
loophole to produce something that in
the long run you were extremely happy
with unhappy with I should say
and I don't I mean that that seems like
a question that can be attacked
mathematically I think it will come out
the desire of the right way but I still
I'm still not sure about it they're not
fully identifiable so so like the reward
functions and inverse reinforcement
learning are positive to exist but they
they are not directly observed episode
so we have all the challenges of latent
variable my learning which is that often
you cannot pin down the exact value of
these things but still they give the
reason to use them as they give a very
compact and maybe even approach on
causal explanation of the behavior so
it's yeah it will be tough and so the
but that means the AI system needs to
know that it doesn't know which come
back to very first slide and needs to be
to behave robustly with respect to that
yeah but I think I I I almost want to
put on my luck fees are they accent and
say I mean the it's not just that we
can't observe alive and dead directly
but in fact it isn't a well-defined
right why we that even notionally you
can't say okay here is a here's a
particular world and alive is true and
here's another one where alive is false
there's always a dichotomy yeah but is
that uncertainty treated the same way in
other words you take expectations over
it or do you do you take work worst case
over at yarn that's one way of thinking
I mean of course the other responses is
try to find those corner cases and then
check to see the people like so what do
you think about this yes what is more
dangerous and
yep so we thought about cooperative so
you
there's the shallow particular agent
which is the human which is this
well-defined thing and in real world is
this sort of like uncertainty this
relates to diversity like what what
counts as human and what human action is
you think I would change very much
so what counts as human I mean it's not
that I have two arms in two legs and
exactly yes so what what counts as what
you care about
yeah oh yeah I mean in some sense that's
a political question you know should we
include in our observations the you know
the behavior of the clinically insane
you know and what you know what about
animals and so on I I'm not sure that
anything I can instruct things that
canvas you murderous definition and is
that their preferences like it gets back
to the Victorian point yeah yeah so I do
I don't know whether we will have to be
microchipped at birth so that the system
knows that we recount is real and and
these are non Fae Keable microchips but
yeah and so yeah I don't know I maybe we
need to make sure it doesn't ever make
people yeah I think
that's just last question another way
once no some way to tell you talk show
funny you should ask I was just thinking
about that last night it sort of
imagining a large library of decision
making scenarios which would be
represented bazoom by you know an in an
embedded 3d virtual reality experience
that would go on for some time and then
the robot would have to be we deciding
how to behave in this in this scenario
and we can you know sort of kind of like
a driving test right so at least you
know you you've got a hundred thousands
in areas where you're sure that across
all these it's behaving adequately well
would be that would be a good start for
a domestic robot I think and for a
self-driving car oh because we're not
right that assumes that we the human can
induce from these hundred thousand
scenarios what exactly the value
function is and so we we can largely the
assumption is that we can largely act in
them morally and societally reasonable
way in a large variety of settings but
we're unable to make explicit in a
reliable way exactly what the value
function is that
is enough for you say yes here's the
right wedding function yeah also a
system where I teach what I mean thank
you thinking about the the thing about
the the imagenet competition right I can
recognize all these objects I can have a
system that learns by any mechanism how
to recognize objects and I can test it
on a million test cases but I can't
write down the discriminating function
yeah all right so it's just like that
yeah sure I mean that's that's one way
of doing it and in some sense that's
precisely
except that I'm looking at training data
which is occurring naturally right the
actual behavior of the human race where
I haven't validated that everyone is
behaving precisely according to the
values that I want the robot to learn
but you still have to be able to learn
from back that kind of data right so so
King Midas did what he did and nany he's
expressing remorse well you can learn
from that behavior sequence something
about human values that they they like
gold but they actually like their
daughter even more than gold but they're
also that they're not 100 percent
rational in collecting the outcomes of
their choices right
that's all good information even though
none of it is optimal behavior so yeah
what we used to call near-miss examples
of the machine that english' two before
it was entirely forgotten and replaced
with a new machine learning literature
but yeah yeah so near-miss I mean these
are these are all good cases and and you
could you could look at fables and and
other instructive stories for children
is precisely doing that for human beings
all right so let's thank professor
Russell again


$100,000 for Tasks Where Bigger AIs Do Worse Than Smaller Ones #short

in AI bigger is better right large
models outperform smaller ones according
to scaling laws except not always
sometimes bigger models can actually
perform worse and sometimes in ways that
might be dangerous so since we're going
to be building really very big models
very soon we need to understand what's
going on here so the inverse scaling
price is offering a hundred thousand
dollars for new examples of situations
where bigger models do worse I talked
about one of these in an earlier video
If you start with bad code sometimes
large code generation models like GitHub
co-pilot will deliberately introduce
bugs or vulnerabilities into their
output in situations where smaller
models will generate the correct secure
code this is because of misalignment you
want code that's good but the AI wants
code that's likely to come next that's
just one example but the hope is by
looking at lots of them we could come up
with more General methods for detecting
and dealing with this kind of
misalignment to apply first find an
example where you think inverse scaling
applies then find a data set of at least
300 examples and test it using the
models in the Google collab that can be
found at inverscaling.com where there's
all the instructions that you'll need
the deadline is October 27th and the
total prize pool is 250 000 good luck


Why Would AI Want to do Bad Things? Instrumental Convergence

hi so sometimes people ask how I can
predict what an artificial general
intelligence might do they say something
like you seem to be predicting that AIS
would take these quite specific actions
trying to prevent themselves from being
turned off preventing themselves from
being modified improving themselves
acquiring resources why do you think
that an AGI would have these specific
goals surely we haven't built the thing
yet we don't know anything about it it
might want anything are you making a lot
of unwarranted assumptions well the
reason that I make these predictions is
because of something called instrumental
convergence which actually relies on
surprisingly few assumptions the main
assumption I'm making is that an AGI
will behave like an agent an agent is
basically just a thing that has goals or
preferences and it takes actions to
achieve those goals or to satisfy those
preferences so the simplest thing that
you might think of as an agent is
something like a thermostat it has a
goal which is for the room to be at a
particular temperature and it has
actions it can take in the form of
turning on the heating or turning on the
air-conditioning and it takes actions to
achieve its goal of keeping the room at
a particular temperature that's like an
extremely simple agent a slightly more
complex agent might be something like a
chess AI its goal is to win the chess
game to have the opponent's king in
checkmate it can take actions in the
form of moving its pieces on the board
and it chooses its actions in order to
achieve the goal of winning the game
the idea of an agent is popular in
economics where it's common to model
companies and individual human beings as
rational agents for the sake of
simplicity it's often assumed that the
goal of human beings is to acquire money
that their utility is just proportional
to how much money they have this is
obviously a huge oversimplification and
it's a very popular fact that most
people are motivated by a lot of things
not just money but while it's easy to
point out the shortcomings that this
assumption has what's remarkable to me
is how well it works or that it even
works at all it's true that most people
use a very complex utility function
which looks nothing like the basic goal
of get as much money as you can but
surprisingly even very simple economic
models that rely on this simplifying
assumption can provide some really
useful and powerful tools for thinking
about the behavior of people companies
and society
why is that I think it has to do with
the nature of terminal goals and
instrumental goals I talked about this
in a lot more detail in the
orthogonality thesis video which I would
recommend checking out if you haven't
seen it yet but to give a quick summary
your terminal goals are the things that
you want just because you want them you
don't have a reason to want them
particularly they're just what you want
whereas instrumental goals are goals
that you want as a way of getting some
other goal so for example in chess you
want to take your opponent's queen not
because you just love capturing the
Queen but because you can tell that
you're more likely to win if you've
captured your opponent's queen than if
you haven't so capturing the Queen as an
instrumental goal towards the goal of
winning the game so how does this work
with money well let's imagine that
there's a total stranger and you don't
know what he wants out of life you don't
know what his goals are maybe he wants
to win a marathon maybe he wants to cure
cancer maybe he just wants a really nice
stamp collection but I can predict that
if I were to go over to him and offer
him a big stack of money he'd be happy
to take it how can I predict this
person's actions even though I don't
know his goals well a guy who wants to
win a marathon if I give him money he
could buy some nice running shoes or he
could hire a personal trainer or
something like that a guy who wants to
cure cancer could give that money to a
cancer charity or maybe use it to help
him to go to university and study
science to cure cancer himself and a guy
who wants to collect stamps could buy
some stamps so the point is even though
none of these people value money as a
terminal goal none of them want money
just for the sake of having money
nonetheless they all value money as an
instrumental goal the money is a way to
get them closer to their goals and even
though they all have different terminal
goals this goal of getting money happens
to be instrumentally valuable for all of
their different goals that makes getting
money a convergent instrumental goal
it's a goal that's an instrumental goal
for a wide variety of different terminal
goals so since money is a convergent
instrumental goal if you make the
assumption that everybody values money
it turns out to be a fairly good
assumption because whatever you value
the money is going to help you with that
and that makes this assumption useful
for making predictions because it allows
you to predict people's behavior to some
extent without knowing their goals
so what other convergent instrumental
goals are there well
an obvious one is self-preservation most
agents with most goals will try to
prevent themselves from being destroyed
now something like a thermostat or a
chess ai they're not self-aware they
don't understand that they can be
destroyed and so they won't take any
actions to avoid it but if an agent is
sophisticated enough to understand that
being destroyed as a possibility then
avoiding destruction is a convergent
instrumental goal humans of course
generally act to avoid being killed but
humans as evolved agents implement this
in a way that might obscure the nature
of the thing the point is that
self-preservation
need not be a terminal goal an agent
need not necessarily value
self-preservation just for the sake of
continuing to exist for its own sake for
example suppose you had a software agent
an AGI which had a single terminal goal
of making as many paper clips as
possible it would try to prevent you
from turning it off not because it wants
to live but simply because it can
predict that future worlds in which it's
turned off will contain far fewer
paperclips and it wants to maximize the
number of paper clips but suppose you
said to it I've thought of a better way
to implement your software that will be
more effective at making paper clips so
I'm going to turn you off and wipe all
of your memory and then create a new
version that's better
at making paper clips this is pretty
clearly the destruction of that first
agent right you're wiping all of its
memories and creating a new system that
works differently that thinks
differently but the paper clip agent
would be fine with this because it can
see that when you turn on its
replacement that will end up resulting
in more paper clips overall so it's not
really about self preservation itself
it's just that practically speaking most
of the time you can't achieve your goals
if you're dead on the other hand suppose
we were going to turn off the agent and
change its goal we were going to change
it so that it doesn't like paper clips
anymore it actually wants to collect
stamps here you're not really destroying
the agent just modifying it but you've
got a problem again because the agent
can reliably predict that if this
modification happens and it becomes a
stamp collector the future will contain
not nearly as many paper clips and
that's the only thing it cares about
right now so we've got another
convergent instrumental goal goal
preservation most agents with most goals
again if they're sophisticated enough to
realize that it's a possibility will try
to protect their goals from being modest
fight because if you get new girls
you'll stop pursuing your current goals
so you're unlikely to achieve your
current goals now for humans this
doesn't come up much because modifying a
human's goals is fairly difficult but
still if you suppose we were to go to
the guy who wants to cure cancer and
offer him some magic pill that's gonna
change his brain so that he doesn't care
about cancer anymore and he actually
wants to collect stamps if we say look
this is actually really good because
cancer is really hard to cure it's
actually a large collection of different
diseases
all of which need different approaches
so you're very unlikely to discover a
cure for all of them but on the other
hand stamp collecting is great you can
just go out and buy some stamps and you
can put them in a book and look at them
I don't really know what stamp
collectors do is that anyway is this guy
who wants to cure cancer going to take
that pill no he's gonna say hell no I'm
not taking you're crazy
stamp pill even though this isn't a
terminal goal for him he doesn't value
valuing curing cancer right he doesn't
have a goal of having a goal of curing
cancer it's just that he believes that
if he gave up during cancer to become a
stamp collector that would result in a
lower chance of cancer being cured so
preserving your terminal goals is
instrumentally useful whatever those
goals are another convergent
instrumental goal is self-improvement
notice that the guy who wants to cure
cancer part of his plan is to go to
university and study science so that he
can learn how to research cancer cures
and the guy who wants to run a marathon
part of his plan is he wants to train
and improve his physical performance
both of these things are improving
yourself and something like this comes
up quite often in human plans again this
isn't a terminal goal the guy who wants
to cure cancer doesn't want to get a
degree just because he wants a degree he
wants to become a person who's more
effective at curing cancer now there's
another way of improving yourself which
is not really available to human beings
but is available to AI systems which is
directly modifying your mind to improve
your own intelligence you're not just
adding information to your mind you're
actually making it more powerful for
example you might be able to rewrite
your code so that it works better or
runs faster or you might be able to just
acquire more computing power the more
computing power you have the deeper and
faster you're able to think the better
you are at making plans and therefore
the better you are at achieving your
goal whatever that
is so computing power is kind of like
money it's a resource which is just very
broadly useful which means we can expect
acquiring that resource to be a
convergent instrumental goal and
speaking in the broadest possible terms
almost all plans need resources in the
form of matter and energy if you want to
build something whether that stamps or
paperclips or computing hardware or
robots or whatever you need matter to
build it out of an energy to build it
with and probably energy to run it as
well so we can expect agents with a wide
range of goals to try to acquire a lot
of resources so without really assuming
anything about an AGI other than that it
will have goals and act to achieve those
goals we can see that it's likely that
it would try to prevent itself from
being shut down try to prevent itself
from being modified in important ways
that we want to modify it try to improve
itself and its intelligence to become
more powerful and try to acquire a lot
of resources this is the case for almost
all terminal goals so we can expect any
generally intelligent software agents
that we create to display this kind of
behavior unless we can specifically
design them not to I want to end the
video by saying thank you to all of my
excellent patreon supporters these these
people and in this video I'm especially
thanking James McEwan who looks a lot
like the Stanford buddy and has been a
patron for nine months now thank you so
much for all of your support and thank
you all for watching I'll see you next
time
you


Intro to AI Safety, Remastered

hi this video is a recording of a talk
that i gave a while back i already
published a version of it on my second
channel but
did you even know i had a second channel
most people don't i thought more people
should see it so i remastered it i
cleaned it up improved the graphics and
yeah this is that enjoy right
hello everyone my name is robert miles i
am usually doing this on youtube i'm not
really used to public speaking
i'm not used to not being able to edit
out my mistakes there may be mistakes
also
i may go too quickly um sorry not sorry
so uh when it comes to ai safety you can
kind of divide it up into four
areas along two axes you've got your
short term
and your long term and you've got
accident risks and misuse risks
and that's kind of a useful way to
divide things up ai safety
covers everything the area that
interests me most
is the long-term accident risks i think
once you have very powerful ai systems
it almost doesn't matter if they're
being used by the right people or the
wrong people or what you're trying to do
with them
the difficulty is in keeping them in
under control at all
so that's what i'm going to be talking
about what is ai safety why is it
important
so i want to start by asking the
question which i think everybody needs
to be asking themselves
what is the most important problem in
your field
take a second to think of it
and why are you not working on that um
for me i think the most important
problem in the field of ai
is ai safety this is the problem
specifically that i'm worried about
that we will sooner or later build an
artificial agent
with general intelligence so i'm going
to go into a bunch of these
terms the first thing is what do i mean
when i say sooner or later
this is a little bit washed out but this
is a graph
of a survey a large survey of ai experts
these are people who published in
major a.i conferences and they were
asked when they thought
we would achieve high-level machine
intelligence which is defined as
an agent which is able to carry out any
tasks humans can
as well as or better than humans and
they say that uh 50 chance of of
having achieved that we we hit that
about 45 years
from 2016. um but then of course we hit
like 10
chance nine years from now so um
it's not immediate but it's happening
this is definitely worth
taking with a pinch of salt because if
you ask the question slightly
differently you get an estimate of 120
years rather than 45
there's a lot of uncertainty in this
area but the point is
it's going to happen as i said sooner or
later because at the end of the day
general intelligence is possible the
brain implements it and the brain is not
magic
sooner or later we'll figure it out so
what do i mean when i say an artificial
agent
well so an agent uh is a term from
uh economics mostly but basically agents
have goals
they choose actions to further their
goals this the simplest expression of
what an agent is
so um the simplest thing that you might
call an agent would be something like a
thermostat
it has a goal which is to have the room
be at a particular temperature
it has actions it can take it can turn
on the heating it could turn on the air
conditioning
it chooses its actions to achieve its
goal of maintaining the room at a steady
temperature
extremely simple agent a more complex
agent might be something like
a chess ai which has a goal of
like if it's playing white it has a goal
of the black king being
in checkmate and it takes actions in the
form of moving pieces on the board in
order to achieve its goal
so you see how this idea of an agent is
a very useful way of thinking about
lots of different intelligence systems
and of course
humans can be modeled as agents as well
this is how it's usually done in
economics
individuals or companies could be
considered to have a goal
of you know maximizing their income or
maximizing their profits and making
decisions
in order to achieve that
so when i'm talking about intelligence
intelligence has a lot of as a term is a
heavily loaded term has a lot of
different
people put their own definitions on it
in this context what i mean when i say
intelligence is just
the thing that lets an agent choose
effective actions it's
whatever it is that's in our brains or
that's in the programming of these
systems
that means that the actions they choose
tend to get them closer to their goals
um and so then you could say that
an agent is more intelligent if it's
more effective at achieving its goals
whatever those goals are
if you have two agents in an environment
with incompatible goals
like let's say the environment is the
chess board and one agent wants
white to win and one agent wants black
to win then
generally the more intelligent agent
will be the one that gets
what it wants the better ai will win the
chess game
um and finally general intelligence this
is where it becomes
uh interesting in my opinion so
generality is the ability to behave
intelligently in a wide range of domains
if you take something like a chess ai
it's extremely narrow it only knows how
to play chess
um and even though you might say that
it's more intelligent than a thermostat
because it's more sophisticated it's
more complicated
it couldn't do the thermostat's job
there's no
position on the chessboard that
corresponds to the room being a good
temperature
there's no move that corresponds to
turning on an air conditioner the chest
ai can only think in terms of chess
it's extremely narrow generality is a
continuous spectrum
um so if you write a program that can
play an atari game
that's very narrow deep mind one of
their early triumphs was
that they made a program that could play
dozens of different atari games single
program that could learn all of these
different games
and so it's more general
because it's able to act across a wider
variety of domains
the most general intelligence that we're
aware of right now
is human beings human beings are
very general we're able to operate
across a very wide range of domains
including and this is important we're
able to learn domains which
evolution did not and could not prepare
us for
um we can for example drive a car an
evolution did not prepare us for that
we invented cars they're very recent um
we can
you know invent rockets and go to the
moon and then we can operate on the moon
which is a completely different
environment
and this is kind of the power of general
intelligence really the power of general
intelligence is
we can build a car we can build a rocket
we can put the car on the rocket take
the car to the moon
drive the car on the moon and there's
nothing
else that can do that yet
um but sooner or later right
so so this is what i'm talking about i'm
talking about
what you might call true ai real ai the
sci-fi stuff
um an agent which has goals
in the real world and is able to
intelligently choose actions in the real
world
to achieve those goals now that sounds
i've said i said what's the biggest
problem this doesn't sound like a
problem right
on the surface of it this sounds like a
solution
you just tell the thing you know cure
cancer or maximize the profits of my
company or whatever
and it takes whatever actions are
necessary in the real world to achieve
that goal
but um it is a problem
so the big problem is this should be
auto playing and it isn't
um the big problem is it's difficult
to choose good goals
um so this this is an ai made by open ai
it's playing a game called coast runners
which is actually a racing game
they trained it on the score which you
probably can't see down here it's
currently a thousand um what the system
learned is that
if it goes around in a circle here and
crashes into everything and catches fire
these little turbo pickups they respawn
at just the right rate
that if it just flings itself around in
a circle it can pick up
the turbo and that gives you a few
points every time you do that
and it turns out that this is a much
better way of getting points
than actually racing around the track
and the important point here is that
this is not unusual this is not open ai
doing anything unusually stupid
this is kind of the default um picking
objectives is surprisingly hard
and you will find that the strategy or
the behavior that
maximizes your objective is probably not
the thing you thought it was
it's probably not what you were aiming
for uh there's loads of examples
actually uh victoria has a great
list on her blog deep safety if there's
like 30 of them
different things going wrong there was
one they had uh
they were trying to teach they were
trying to evolve systems that would run
quickly so they they trained them on the
i'm going to pause this because it's
distracting as hell
where's my mouse yeah pause
pause please um they were training like
agents that were supposed to run so they
simulated them for a particular period
of time and measured
how far their center of mass moved which
seems perfectly sensible
what they found was that they developed
a bunch of these creatures which were
extremely tall and thin with a big mass
on the end
that then fell over because they weren't
simulating them for long enough
the you could go the fastest just by
falling over rather than
actually running that moved your center
of mass the furthest um
there's a lot of these there was a
tetris bot which um
would play reasonably well and then just
when it was about to lose
would pause the game and sit there
indefinitely because it lost
points for losing but didn't lose any
points for just sitting on the pause
screen indefinitely
this is this is like the default of how
these systems behave i have
no memory what my next slide is oh yeah
right so we have problems
specifying even simple goals in simple
environments like atari games or
basic evolutionary algorithms things
like that
when it comes to the real world things
get way more complicated this is a quote
from
stuart russell who sort of wrote the
book on aia when a system is optimizing
a function of n variables where the
objective depends on a subset of size k
which is less than n
it will often set the remaining
unconstrained variables to extreme
values
if one of those unconstrained variables
is something that we care about the
solution found may be highly undesirable
in the real world we have a very large
number of variables and so
we're talking about we're talking about
very large
values for n here so let's say you've
got your robot
and you've given it a gold which you
think is very simple you want it to get
you a cup of tea
so you've managed to specify what a cup
of tea is and that you want one to be
on the desk in front of you so far so
good
but suppose there is a there's a
processing vars on a narrow
stand sort of in front of where the
kitchen is so the robot
immediately plows into the vars and
destroys it on its way to make you a cup
of tea because
you only gave it one variable to keep
track of in the goal
which is the t it doesn't care about the
vars you never told it to care about the
vars
it destroys the vast this is a problem
um
so okay now you can you know shut it
down modify it and say
okay give me a cup of tea but also don't
knock over the vars
but then there will be a third thing
there is always another thing
because when when you're making
decisions in the real world
you're always making trade-offs you're
always
taking various things that you value and
deciding how much
of one you're willing to trade for how
much of another you know i could do this
quicker
but it increases the risk of me making a
mistake or i
could do this cheaper but it won't be as
reliable
i could do this faster but it'll be more
expensive you're always trading these
things off against one another
and so an agent like this which only
cares about a limited subset of the
variables in the system
will be willing to trade off arbitrarily
large amounts
of any of the variables that aren't part
of its goal for arbitrarily tiny
increases
in any of the things which are in its
goal so it will happily
i let's say now for example now it
values the vars
uh and those are the only things that it
values it might reason something like
okay there's a human in the environment
the human moves around the human
may accidentally knock over the vars and
i care about the vars so i have to kill
the human
right and this is totally ridiculous but
if you didn't tell it that you value
being alive it doesn't care and anything
that it doesn't value
is going to be lost if you manage to
come up with if you have a sufficiently
powerful agent
and you manage to come up with a really
good objective function which
covers the top 20 things that humans
value
the 21st thing that humans value is
probably gone forever
because the smarter the more powerful
the agent is
the better it will be at figuring out
ways to make these trade-offs
to gain a millionth of a percent better
at one thing
while sacrificing everything of some
other variable
so this is a problem but actually that
scenario i gave was unrealistic in
many ways but but one important way that
it was unrealistic
is that i had i had the system go wrong
and then you just turn it off
and fix it but in fact if you're
if the thing has a goal of getting you a
cup of tea this is not like a chess ai
where you can just turn it off because
it has no concept of itself or
being turned off its
world model contains you it contains
itself it contains the possibility of
being turned off and it's fully aware
that if you turn it off
because it knocked over the vars it
won't be able to get you any tea which
is the only thing it cares about
so it's not going to just let you turn
it off it will fight you
or if it's slightly smarter it will
deceive you so that you believe it's
working correctly
so that you don't want to change it
until it's in a position
where you can't turn it off and then it
will go after its actual objective
so so this is a problem
and the thing is this is this is a
convergent instrumental goal which means
it sort of doesn't matter what the goal
is it doesn't matter what your goal is
as an agent
if you're destroyed you can't achieve
that goal so it sort of almost doesn't
matter what goal we give it there is
only a very tiny fraction of possible
goals
that will involve it actually allowing
itself to be turned off and modified
and that's quite complicated um
there are some other convergent
instrumental goals so we had
self-preservation goal preservation
resource acquisition is the kind of
thing we can expect these kinds of
systems to do
most plans you can do them better if you
have more resources
whether that's money computational
resources just
free energy matter whatever
the other one is self-improvement
whatever you're trying to do
you can probably do it better if you're
smarter and ai systems potentially have
the capacity to improve themselves
either just by
acquiring more hardware to run on or
changing
you know improving their uh their
software to run faster or better or so
on
so there's a whole bunch of behaviors
which
uh intelligent systems intelligent
agents
generally intelligent agents we would
expect them to do
by default and that's really
my core point artificial general
intelligence is dangerous by default
it's much much easier to build these
kinds of agents
which try to do ridiculous things and
trick you
and try to deceive you or will fight you
when you try to turn them off or modify
them
on the way to doing some ridiculous
thing which you don't want
uh much easier to build that kind of
agent than
it is to build something which actually
reliably does what you want it to do
and that's why we have a problem because
we have
45 to 120 years to figure out
how to do it safely which is a much
harder problem and
we may only get one shot it's entirely
possible that the first
true artificial general intelligence
will manage to successfully achieve
whatever its stupid goal is
and that could be truly a disaster on a
global scale
so we have to we have to beat this
challenge
on hard mode before anyone beats it on
easy mode
so are we screwed
no we're only probably screwed
um there are things we can do
safe general artificial intelligence is
totally possible
it's just a very difficult technical
challenge
and there are people working very hard
on it right now trying to solve
a whole range of difficult technical
challenges
so that we can figure out how to do this
safely
thanks
[Applause]
[Music]
you may have noticed in the intro in
this outro that the image quality has
improved since the last video
that's largely thanks to my excellent
patrons thank you to all of these people
here for helping me to get this new
camera in this video i'm especially
thanking james pets who's been hanging
out with us on the discord server
helping answer questions from the
youtube comments and so on and actually
that last video about mesa optimizers
has had a lot of really good questions
so the next video will be answering some
of those
that's coming up soon so thanks again to
james and to all my patrons
to everyone who asked questions and to
you for watching
i'll see you next time
you


Why Not Just: Think of AGI Like a Corporation?

hi so I sometimes see people saying
things like okay so your argument is
that at some point in the future we're
going to develop intelligent agents that
are able to reason about the world in
general and take actions in the world to
achieve their goals
these agents might have superhuman
intelligence that allows them to be very
good at achieving their goals and this
is a problem because they might have
different goals from us but don't we
kind of have that already corporations
can be thought of as super intelligent
agents they're able to think about the
world in general and they can outperform
individual humans across a range of
cognitive tasks and they have goals
namely maximizing profits or shareholder
value or whatever and those goals aren't
the same as the overall goals of
humanity so corporations are a kind of
misaligned super intelligence the people
who say this having established the
metaphor at this point tend to diverge
mostly along political lines some say
corporations are therefore a clear
threat to human values and goals in the
same way that misaligned super
intelligences are and they need to be
much more tightly controlled if not
destroyed all together others say
corporations are like misaligned super
intelligences but corporations have been
instrumental in the huge increases in
human wealth and well-being that we've
seen over the last couple of centuries
with pretty minor negative side effects
overall if that's the effect of
misaligned super intelligences I don't
see why we should be concerned about AI
and others say corporations certainly
have their problems but we seem to have
developed systems that keep them under
control well enough that they're able to
create value and do useful things
without literally killing everyone so
perhaps we can learn something about how
to control or align super intelligences
by looking at how we handle corporations
so we're gonna let the first to fight
amongst themselves and we'll talk to the
third guy so how good is this metaphor
our corporations really like misaligned
artificial general super intelligences
quick note before we start we're going
to be comparing corporations to AI
systems and this gets a lot more
complicated when you consider that
corporations in fact use AI systems so
for the sake of simplicity we're going
to assume that corporations don't use AI
systems because otherwise the problem
gets recursive and like not in a cool
way
first off our corporations agents in the
relevant way I would say yeah pretty
much I think that it's reasonably
productive to think of a corporation as
an agent
they do seem to make decisions and take
actions in the world in order to achieve
goals in the world but I think you face
a similar problem thinking of
corporations as agents as you do when
you try to think of human beings as
agents in economics it's common to model
human beings as agents that want to
maximize their money in some sense and
you can model corporations in the same
way and this is useful but it is kind of
a simplification in that human beings in
practice want things that aren't just
money
and while corporations are more directly
aligned with profit maximizing than
individual human beings are it's not
quite that simple so yes we can think of
corporations as agents but we can't
treat their stated goals as being
exactly equivalent to their actual goals
in practice more on that later so
corporations are more or less agents are
they generally intelligent agents again
yeah I think so I mean corporations are
made up of human beings so they have all
the same general intelligence
capabilities that human beings have so
then the question is are they super
intelligent this is where things get
interesting because the answer is kind
of like SpaceX is able to design a
better rocket than any individual human
engineer could design rocket design is a
cognitive task and SpaceX is better at
that than any human being therefore
SpaceX is a super intelligence in the
domain of rocket design but a calculator
is a super intelligence in the domain of
arithmetic that's not enough our
corporation's general super
intelligences do they outperform humans
across a wide range of cognitive tasks
as an AGI code in practice it depends on
the task consider playing a strategy
game for the sake of simplicity let's
use a game that humans still beat AI
systems at like Starcraft if a
corporation for some reason had to win
at Starcraft it could perform about as
well as the best human players it would
do that by hiring the best human players
but you won't achieve superhuman play
that way a human player acting on behalf
of the corporation is just a human
player and the corporation doesn't
really have a way to do much better than
that
a team of reasonably good Starcraft
players working together to control one
army will still lose to a single very
good player working alone this seems to
be true for a lot of strategy games the
classic example is the game of Kasparov
versus the world where Garry Kasparov
played against the entire rest of the
world cooperating on the Internet
the game was kind of weird but Kasparov
ended up winning and the kind of real
world strategy that corporations have to
do seems like it might be similar as
well when companies outsmart their
competition it's usually because they
have a small number of decision makers
who are unusually smart rather than
because they have a hundred reasonably
smart people working together for at
least some tasks teams of humans are not
able to effectively combine their
intelligence to achieve highly
superhuman performance so corporations
are limited to around human level
intelligence of those tasks to break
down where this is let's look at some
different options corporations have four
ways to combine human intelligences one
obvious way is specialization if you can
divide the task into parts that people
can specialize in you can outperform
individuals you can have one person
who's skilled at engine design one who's
great at aerodynamics one who knows a
lot about structural engineering and one
who's good at avionics can you tell I'm
not a rocket surgeon anyway if these
people with their different skills are
able to work together well with each
person doing what they're best at the
resulting agent will in a sense have
superhuman intelligence no single human
could ever be so good at so many
different things but this mechanism
doesn't get you superhumanly high
intelligence just superhumanly broad
intelligence whereas super intelligence
software AGI might look like this so
specialization yields a fairly limited
form of super intelligence if you can
split your task up but that's not easy
for all tasks for example the task of
coming up with creative ideas or
strategies isn't easy to split up you
either have a good idea or you don't but
as a team you can get everyone to
suggest a strategy or idea and then pick
the best one that way a group can
perform better than any individual human
how much better though and how does that
change with the size of the team I got
curious about exactly how this works so
I came up with a toy model now I'm not a
statistician I'm a computer scientist so
rather than working it out properly I
just simulated it a hundred million
times because that was quicker okay so
here's the idea quality distribution for
an individual human will model it as a
normal distribution with a mean of 100
and a standard deviation of 20 so what
this means is you ask a human for a
suggestion and sometimes they do really
well and come up with a hundred
30-level strategy sometimes they screw
up and can only give you a 70 idea but
most of the time it's around 100 now
suppose we had a second person whose
intelligence is the same as the first we
have both of them come up with ideas and
we keep whichever idea is better the
resulting team of two people combined
looks like this
on average the ideas are better the mean
is now 107 and as we keep adding people
the performance gets better here's 5
people 10 20 50 100
remember these are probability
distributions so the height doesn't
really matter the point is that the
distributions move to the right and get
thinner the average idea quality goes up
and the standard deviation goes down so
we're coming up with better ideas and
more reliably but you see how the
progress is slowing down we're using a
hundred times as much brain power here
but our average ideas are only like 25%
better what if we use a thousand people
ten times more resources again only gets
us up to around a hundred and thirty
five diminishing returns so what does
this mean for corporations well first
off to be fair this team of a thousand
people is clearly super intelligent the
worst ideas it ever has are still so
good that an individual human will
hardly ever manage to think of them but
it's still pretty limited there's all
this space off to the right of the graph
that it would take vast team sizes to
ever get into if you're wondering how
this would look with seven billion
humans well you have to work out the
statistical solution yourself the point
is the team isn't that super intelligent
because it's never going to think of an
idea that no human could think of which
is kind of obvious when you think about
it but AGI is unlimited in that way and
in practice even this model is way too
optimistic for corporations firstly
because it assumes that the quality of
suggestions for a particular problem is
uncorrelated between humans which is
clearly not true and secondly because
you have to pick out the best suggestion
but how can you be sure that you'll know
the best idea when you see it it happens
to be true a lot of the time for a lot
of problems that we care about that
evaluating solutions is easier than
coming up with them you know Homer it's
very easy to criticize machine learning
relies pretty heavily on this like
writing a program that differentiates
pictures of cats and dogs is really hard
but evaluating such a program is fairly
simple you
show it lots of pictures of cats and
dogs and see how well it does the clever
bit is in figuring out how to take a
method for evaluating solutions and use
that to create good solutions anyway
this assumption isn't always true and
even when it is the fact that evaluation
is easier or cheaper than generation
doesn't mean that evaluation is easy or
cheap
like I couldn't generate a good rocket
design myself but I can tell you that
this one needs work so evaluation is
easier than generation but that's a very
expensive way to find out and I wouldn't
have been able to do it the cheap way by
just looking at the blueprints the
skills needed to evaluate in advance
whether a given rocket design will
explode are very closely related to the
skills needed to generate a non
exploding rocket design so yeah even if
a corporation could somehow get around
being limited to the kind of ideas that
humans are able to generate they're
still limited to the kind of ideas that
humans are able to recognize as good
ideas just how serious is this
limitation how good are the strategies
and ideas that corporations are missing
out on well take a minute to think of an
idea that's too good for any human to
recognize it as good got one well it was
worth a shot we actually do have an
example of this kind of thing in move 37
from alphago's 2016 match with world
champion Lisa doll this kind of
evaluation value that's a very that's a
very surprising move I thought I thought
it was I thought it was a mistake yeah
that turned out to be pretty much the
move that won the game but you're go
playing corporation is never going to
make move 37 even if someone happens to
suggest it it's almost certainly not
going to be chosen
normally human we never play this one
because it's not enough for someone in
your corporation to have a great idea
the people at the top need to recognize
that it's a great idea that means that
there's a limit on the effective
creative or strategic intelligence of a
corporation which is determined by the
intelligence of the decision-makers and
their ability to know a good idea when
they see one okay what about speed
that's one of the things that makes AI
systems so powerful and one of the ways
that software IGI is likely to be super
intelligent the general trend is we go
from computer
can't do this at all two computers can
do this much faster than people not
always but in general so I wouldn't be
surprised if that pattern continues with
AGI how does the corporation rate on
speed again it kind of depends
this is closely related to something
we've talked about before parallelizable
ax t some tasks are easy to split up and
work on in parallel and some aren't
for example if you've got a big list of
a thousand numbers and you need to add
them all up it's very easy to paralyze
if you have ten people you can just say
okay you take the first hundred numbers
you take the second hundred you take the
third and so on have everybody add up
their part of the list and then at the
end you add up everyone's totals however
long the list is you can throw more
people at it and get it done faster much
faster than any individual human code
this is the kind of task where it's easy
for corporations to achieve superhuman
speed but suppose instead of summing a
list you have a simple simulation that
you want to run for say a thousand
seconds you can't say okay you work out
the first hundred seconds of the
simulation you do the next hundred and
you do the next hundred and so on
because obviously the person who's
simulating second 100 needs to know what
happened at the end of second 99 before
they can get started so this is what's
called an inherently serial task you
can't easily do it much faster by adding
more people you can't get a baby in less
than nine months by hiring two pregnant
women
you know most real-world tasks are
somewhere in between you get some
benefits from adding more people but
again you hit diminishing returns some
parts of the task can be split up and
worked on in parallel some parts need to
happen one after the other so yes
corporations can achieve superhuman
speed add some important cognitive tasks
but really if you want to talk about
speed in a principled way you need to
differentiate between throughput how
much goes through the system within a
certain time and latency how long it
takes a single thing to go through the
system these ideas are most often used
in things like networking and I think
that's the easiest way to explain it so
basically let's say you need to send
someone a large file and you can either
send it over a dial-up internet
connection or you can send them a
physical disk through the postal system
the dial-up connection is low latency
each bit of the file goes through the
system quickly but it's also low
throughput the rate at which you can
send data is pretty low whereas sending
the physical disk is high latency it
might take days for the first
to arrive but it's also high-throughput
you can put vast amounts of data on the
disk so your average data sent per
second could actually be very good
corporations are able to combine human
intelligences to achieve superhuman
throughput so they can complete large
complex tasks faster than individual
humans could but the thing is a system
can't have lower latency than its
slowest component and corporations are
made of humans so corporations aren't
able to achieve superhuman latency and
in practice as you've no doubt
experienced is quite the opposite so
corporate intelligence is kind of like
sending the physical disk corporations
can get a lot of cognitive work done in
a given time but they're slow to react
and that's a big part of what makes
corporations relatively controllable
they tend to react so slowly that even
governments are sometimes able to move
fast enough to deal with them
software super intelligence is on the
other hand could have superhuman
throughput and superhuman latency which
is something we've never experienced
before in a general intelligence so our
corporations super intelligent agents
well they're pretty much generally
intelligent agents which are somewhat
super intelligent in some ways and
somewhat below human performance in
others so yeah kinda the next question
is are they misaligned but this video is
already like 14 and a half minutes long
so we'll get to that in the next video
[Music]
I want to end the video by saying a big
thank you to my excellent patrons it's
all of these people here in this video
I'm especially thanking Pablo area or
Pablo a de aluminio Sushil recently I've
been putting a lot of time into some
projects that I'm not able to talk about
but as soon as I can and the patrons
will be the first to know
thank you again so much for your
generosity and thank you all for
watching I'll see you next time
[Music]