AI "Stop Button" Problem - Computerphile

In almost any situation being given a new utility function,
Is gonna rate very low on your current utility function.
So that's a problem.
if you want to build something that you can teach, that means you want to be able to change its utility function.
And you don't want it to fight you.
So this has been formalized, as this property that we want early AGI to have called 'corrigibility'
That is to say it is open to be corrected.
It understands that it's not complete, that the utility function that it's running is not the be all and end all.
So let's say for example you've got your AGI. It's not a super intelligence, it's just you know, perhaps around human level intelligence.
And it's in a robot in your lap.
And you're testing it.
But you... saw a Youtube video once that said maybe this is dangerous.
So you thought, okay well we'll put a big red stop button next to it.
This is the standard approach to safety with machines. Most robots in industry and elsewhere will have
a big red stop button on them.
Oh yeah
Hey.
I happen to have a button, of appropriate type.
so
and we have
we have that.
So right... there you go.
Alright, so, so
if only HAL would've been fitted with said stop button.
Can't do that Dave, uh yes I can.
except yeah, probably not.
I know that you and Frank were planning to disconnect me.
And I'm afraid that's something I cannot allow to happen.
That was an incorrigible design.
Is this the point we're making? Kind of.
You've got your big stop button,
because you want to be safe. You understand it's dangerous.
And the idea is if the AI starts to do anything
that maybe you don't want it to do,
you'll smack the button, and if the button's like, mounted on its chest, something like that, ya know.
So you create the thing, you set it up with a goal, and it's the same basic type of machine as the stamp collector, but less powerful.
In the sense it has a goal, a thing that it's trying to maximize,
and in this case, you know, it's in a little robot body
so they can tootle around your lab and do things.
So you, you want it to get you a cup of tea just as a test, right?
So you set it up with this goal you manage to specify in
the bot's like in the AI's ontology
what a cup of tea is and that you
want one to be in front of you.
You switch it on and it looks around, gathers
data and it says oh yeah there's a
kitchen over there it's got a kettle
and it's got teabags and this is like
the easiest way for me to fulfill this
goal with the body i have now and
everything setup is to go over there and
make a cup of tea. So far we're doing
very well right. So it starts driving over
but then oh no you forgot it's bring
your adorable baby to the lab day or
something and there's a kid
in the way. Your utility function only
cares about tea right. So it's not going
to avoid hitting the baby. So you rush
over there to hit the button
obviously as you built it in and what
happens of course is that the robot will
not allow you to hit that button because it
wants to get you a cup of tea and if you
hit the button it won't get you any tea so
this is a bad outcome so it's going to
try and prevent you in any way possible
from shutting it down
that's a problem plausibly it fights you
off crushes the baby and then carries on
and makes you a cup of tea and the fact that
this button is supposed to turn it off
is not in your utility function that you
gave it so obviously it's going to fight
you
ok that was a bad design right assuming
you're still working on the project
after the terrible accident and you have
another go try to improve things right
and rather than read any AI safety
research what you do is just come up
with the first thing that pops into your
head and you say
ok let's add in some reward for the
button so that because what it's looking
at right now is it says button gets
hit i get 0 reward. Button doesn't get
hit if I manage to stop them then I get
the cup of tea i get like maximum reward
if you give some sort of compensation
for the button being hit maybe it won't
mind you hitting the button. If you give it
less reward for the button being hit
than for getting tea, it will still fight you
cause it will go well I could get five reward
for accepting your hitting the button
but I could get 10 for getting the tea
so I'm still gonna fight you.
The button being hit has to be just as
good as getting the tea so you give it the
same value right so so now you've got a
new Version 2. You turn it on
and what it does immediately is shut
itself down because that's so much
quicker and easier than going and
getting the tea and gives exactly the
same reward
Why would it not just immediately shut
itself down? So you've accidentally
made like a dramatically suicidal robot.
We're kind of back to the Three Laws of
Robotics again but could you presumably
say well the robot is not allowed to to
shut itself down or something like that? Right
yeah so. It's still a reward if it does
so it's not gonna fire you but it's not
allowed to do itself i'm guessing that's
a possibility. The robot cares about
things in the real world and the button
whether it's pushed or not has a big
effect on the things that the robot cares
about which means that the robot cares
about the button so either it's trying
to stop you from pressing the button or
it's trying to press the button. Either
way that's not what you want
so suppose then you don't mount the
button on its chest where it can hit it
right. You put the button somewhere only
you can hit it somehow you managed to
completely isolate the bot from the
button. It's a fingerprint scanner or something?
It's yeah it's something that it can't do
itself right. It still either wants to
press the button
or wants the button not to be pressed but now
the only path is through you
so it's now incentivised to manipulate
you to deceive you to lie to you in some
way either to cause you to not press it to
cause you to press it if you somehow
managed to persuade it it's not allowed to
interact with the button itself so in
the case where it wants the button not
to be pressed it has a reason to deceive
you so in this example it would steer
around the baby it would avoid the baby
assuming it has basic level of
intelligence to know you don't want it
to hit the baby and moreover if you
realize your mistake you will hit the
button
shut it down and change it which it
doesn't want so it will behave as though
this is a there's a term which i
like to use here i'm not sure if I
should just use it because it's my own
which is that it will Volkswagen you.
That it will set itself up so
that it it behaves properly in the
testing environment despite the fact
that it actually shouldn't pass the
test. You're testing it to see if it
avoids running over babies. So
you're trying to see does it care about
harming humans but actually all it
cares about is you not hitting the
button and so then you might end up in a
situation in which you become convinced
that the thing is perfectly safe because
it keeps passing all of your moral
puzzles and test that you give it but
actually it's just has enough of an
understanding of your psychology to tell
what it thinks you've put in and what it
wants of what you want rather so it will
for as long as that button is able to
shut it down it will continue to attempt
to deceive you until it's in a
situation in which it believes it can't
be shut down and then you have a problem
on the other side if you accidentally
make something that wants the button to
be hit but is unable to it is going to
try and manipulate you into pressing the
button so if you create your thing you
say okay
make me a cup of tea if the button is
hit shut yourself down
you aren't allowed to shut yourself down
you aren't allowed to hit the button but if
the button is hit the reward you get is
just as good at getting the tea so that
you don't have a preference this machine
really wants to hit it's own button because
that's as good as getting the tea so
what it's likely to do probably is just take a
swing at you or something just
immediately because if it can quickly
persuade you to hit the button if
scaring you into hitting the button is
easier than getting the tea it will just
do that instead which is a really kind
of unexpected outcome that you've made
this thing with perfectly reasonable
sounding rewards and what it does
immediately is try to terrify you
it reminds me of the proverbial carrots
and a stick this this is almost like this
is the stick and actually we need to
find what the carrot is would that be a
fair thing to say
yeah yeah you want it to actually want
it it's interesting because it has
to not care about whether the button is
pressed right because it has to take
no steps to try and cause the button to
be pressed and take no steps to try and
prevent the button from being pressed
but nonetheless really care that the
button exists so one thing that you
can do something slightly more sensible
is you define the utility function such
that the whole part of of what it's
really trying to achieve in the world
and the part about paying attention to
the button being pressed and turning
itself off and it sets it up so that
those it adds an adjustment tab so that
those are always exactly equal however
however much value it would get from
either it being pressed it not being
pressed it normalizes those so that it's
always completely indifferent to whether
the button is being pressed. It just doesn't
care
so that way it will never try and hit
the button on its own it will never try
and prevent you from hitting the button
that's the idea that's a fairly sensible
approach it has some of its own problem
feels like a really complicated thing to
evaluate to be honest
yeah firstly yeah it is kind of tricky
and you have to get it right but that's
always the always part of this but one
thing that's interesting about that is
that it isn't what's called subagent
stable so this is something that is a
desirable property and it's part of
corrigibility in fact which is that
the if there's some property that the
agent has you want it to if it makes
any new agents that they will have this
property as well so you get some of that
for nothing like in the same way that
you don't want to take a pill that will
make you want to kill your kids
you also don't want to make a clone of
yourself that wants to kill your kids right if
you're making another instance of
yourself or you're creating some agent
you want it to want what you want this
is like the usual classic disney way out
of a problem for a baddie character where
they go I'm not allowed to do this but that
doesn't mean I can't get this person to
do this for me
I need a young pair of legs and a strong
back to go in after it
exactly so in general when an AI designs
its successor it will design it with the same utility function as it has for
the same reason but unlike with some
safety measures where the AI will be
explicitly motivated to remove the
safety measures if it can by making new
agents which doesn't follow them in this
case it doesn't care so you might think
well you're okay then it doesn't care it
won't remove the safety precautions but
it also won't to put in any effort to
keep them so if it's making a new
version and it would take a couple of
extra seconds to put in
oh and also respond appropriately to the
button and it could do anything else
with that time or those resources it
just won't do it because it looks at the
utilities and says if the button is pressed
i get x utility if the button isn't
pressed i get exactly the same utility
this bit of code does nothing there's no
point in it right this is going to be
optimizing itself or optimizing its
successor in a way that
yeah it think is being
streamlining potentially but actually
is missing the point of and that's
kind of unavoidable if you give if you
have a situation in which it wants to
preserve the button so that it can be
pressed because it wants that then you've
made something that wants the button to
be pressed and will try and manipulate you if
you if you do it the other way you know
whichever way you do it you have
problems
another thing that people have thought
you know as a suggestion of something
you might try in this situation is to
just have the robot have the AI not know
about the button. Oh so keep it a secret
right you have it hidden somewhere you
know the the machine is being observed
from somewhere else and they have the
button and it's outside of their control
and they don't even know the button exists
so they're not going to be motivated to
try and deceive you because they're not
aware that you have the capacity to
shut them down and change them. There are
a few different variants of this but
at the core of the problem there you've
got the fact that you're building a
general-purpose machine to learn and
reason about the real world of uncertain
abilities
I suppose I mean even a very simple
optimization systems consistently
surprise people and so the idea that you
could reliably keep such a thing a
secret indefinitely and it would never
put two and two together in fact while
you're while you're improving it and
training it's going to be learning about
human psychology because that's
important for the final goal. It needs to
understand humans in order to understand
what humans want in order to do what humans
want so the idea that it's never
going to think oh interesting these guys
you know they don't seem to trust me i'm
aware there is this concept of an off
button and it's weird that they haven't
given me one so maybe they have actually
and just not told me about it that kind
of hypothesis shouldn't take too long to
occur and of course if you've designed
the thing in an incorrigible way it
won't let you know that it's figured out
that you have an off button and will  just
deceive you as it always had so that's the
kind of thing which is which could work
but it's not really a reliable solution
the other thing that people sometimes
suggest and this happens this happens in
conversation sometimes is that people
keep applying patches you have a you
have a bad idea for a way to do this and
then somebody points out the way that
would go wrong and then rather than
realise the big core approach is flawed
you apply a patch you say oh well we'll
also add a negative term for doing that
and then also for doing that you know
The spaghetti code ensues yeah yeah and
what's more you're then in a situation
in which you've got this system that you
believe you've patched every possible
way
it's it's kind of um you haven't proved
it's safe you've just proved that you
can't figure out how it's dangerous but
what are the chances that you've
genuinely thought of every possibility
ideally we really want to be able to
formally prove that the system has these
properties you don't want a system in
which you've blocked off loads of
specific actions that the AI can do
and you're just relying on it it's
like running a complicated search trying
to figure out a way to screw you over
and you're pretty sure you've blocked
off all the angles. You've kind of failed
before you've begun there that your code
is running this extensive search that
you just hope fails and if it finds any
way to do it will jump on that
opportunity
it's not it's not a good way of going
about things the other point about this
is that the button is a toy
problem it's a simplification that's
useful for thought experiments because
it lets you express your you know it lets you
formalize things quite well you only
have two possible outcomes you hit
the button or you don't hit the button but in
fact with corrigibility what we want is a
more complex range of behaviors we
want it to actually assist the
programmers in its own development
because if it has you know it has some
understanding of its own operation you
want it to be able to actually point out
your mistakes to you or seek out new
information perhaps if you say something
ambiguous rather than just assuming to
say well do you mean this or do you
mean this or if you if you believe that
you've been programmed poorly to
actually draw the programmer's attention to
what may be the mistake rather than like
quietly storing that away for any time
that they might try and press this button on you
you know
likewise wanting to maintain and repair
the safety systems and so on these are
these are more complicated behaviours than
just not stopping you from pressing the
button and not trying to manipulate you
into not pressing the button so there are
some things that might work as solutions
for this specific case but you would
hope that a really good solution to the
off button problem would if you run it in a more
complicated scenario also produce these
good more complicated behaviours in
that situation so that's part of why
there are some things that maybe are
solutions to this problem but they're
not they're not they're only solutions
to this specific instance of the problem
rather than the general issue we're trying
to deal with right now we have a few
different proposals for ways to ways to
create a AGI with these properties
but none of them are without problems
none of them seems to perfectly solve
all of these properties in a way that we
can be really confident of. So this is
considered an open problem so I kind of
like this as a place to go from the
previous thing because it gives I think
it gives people a feel for where we are
the types of problems that it seems like
the simplest thing in the world right
you've got a robot with a button
how do you make it not stop you from
hitting the button but also not try and
persuade you to hit the button
that should be easy and doesn't seem like it is
so utility function is what the AI
cares about so the the stamp collecting
device its utility function was just how
many stamps in a year
this is kind of like its measure is it
yeah it's what it it's the thing that
it's trying to optimize


A Response to Steven Pinker on AI

Hi. A quick foreword before we get started. I wrote the script for this video in maybe February
or March of last year in response to an article Steven Pinker wrote for Popular Science magazine,
which was basically an extract from his new book Enlightenment Now. I ended up not publishing that
video because I thought it was kind of mean-spirited and sarcastic in places, and I didn't
want to be that kind of YouTuber. I figured Pinker is on the side of the angels, as it were, and
there'd be a retraction or correction issued before long, like we heard from Neil deGrasse
Tyson. But it's now been almost a year, and Pinker has stood by this work in response to criticism,
so here we go. Steven Pinker is a cognitive psychologist and an award-winning popular
science author. His books tend to be carefully researched and well thought out, books which
have really influenced the public conversation about various scientific topics. How the mind
works, the relative roles of nature and nurture in human psychology and behavior, and society and
social progress. His latest book is titled Enlightenment Now, which advances, amongst other
things, one of Pinker's main points, a point which I think is true and important and not widely
enough believed, which is that things are on the whole good and getting better. Everybody's always
talking about how everything is terrible and society is on the wrong path and everything is
going down the drain, but if you look at almost anything you can measure, it's getting better.
Crime is down, violence is down, war is down, infant mortality is way down, disease is down.
Generally speaking, people are happier, more prosperous, and living in better conditions
than they ever have been. The statistics on this are pretty clear. Things do seem to be getting
better. And in spite of this, a lot of people have a tendency to be unreasonably pessimistic
about the future. It can be pretty aggravating. So if you can't tell, I quite like Steven Pinker.
He's the kind of person who, if I didn't know much about a subject and I heard that he'd written an
article about that subject in Popular Science Magazine, I would think, oh good, I can read
that article and it will give me a good understanding of that subject. But I think you
can tell from the setup here that that was not the case with this piece. So the article is titled
We're Told to Fear Robots, But Why Do We Think They'll Turn on Us? With the subtitle
The Robot Uprising is a Myth. Pinker starts the article by talking about this fact that
things are generally getting better and a lot of people seem to want to think that things are
getting worse. There's a popular narrative that says, even though technology seems to be making
our lives better, what if we're actually on our way to some disaster and technology is actually
going to be terrible for us and we just don't realize it yet? And it seems that Pinker sees
concern about AI risk as just more of the same, more of this just general purpose technology
pessimism. Is it though? I'm sure it's part of the reason for the popularity of the idea.
I've certainly talked to people who seem to think that way, but when it comes to deciding whether
AI risk is a real concern, I think this is a weak argument. I'm generally very wary of arguments
that talk about people's psychology and their motivations for making a claim, rather than
talking about the claim itself. To demonstrate, you can make a very similar argument and say,
well, Steven Pinker has spent many years of his life railing against this sort of general
pessimism. And because this is kind of his thing, he's inclined to see it everywhere he looks. And
so when he sees people who actually have an understanding of the technical subject matter
raising serious safety concerns, he's inclined to just dismiss it as more of the same Luddite
pessimism because that's what he knows. That's what he expects to see. Is that what's actually
happened? I don't know, maybe, but I think it's a weak argument to make. The fact that people might
have reasons to want to say that AI is a risk doesn't mean that it isn't. And the fact that
Steven Pinker might be inclined to dismiss such concerns as this kind of reflexive pessimism
doesn't mean that it isn't. So these are just bulverism and we should get down to the actual
technical issues. So what are the actual arguments put forward by the article? Before we start in on
that, I just want to say summarizing arguments is difficult. Summarizing arguments fairly is
especially difficult. And I'm not trying to construct straw man here, but I might by accident.
I would encourage everyone to read the original article. There's a link in the description.
But I know most of you won't, and that's okay. Okay, so one of the first points that the article
makes when it gets to actual arguments is it says, among the smart people who aren't losing sleep
are most experts in artificial intelligence and most experts in human intelligence. This is kind
of an appeal to authority, but I think that's totally legitimate here. The opinions of AI experts
are important evidence about AI risk, but is it actually true? Do most AI researchers not think
that advanced AI is a significant risk? Well, just recently a survey was published and in fact,
I made a whole video about it. So I'll link to that video and the paper in the description.
Grace et al surveyed all of the researchers who published work at the 2015 NIPS and ICML
conferences. This is a reasonable cross section of high level AI researchers. These are people
who have published in some of the best conferences on AI. Like I say, I made a whole
video about this, but the single question that I want to draw your attention to is this one.
Does Stuart Russell's argument for why a highly advanced AI might pose a risk
point at an important problem? So the argument that's being referred to by this question is
more or less the argument that I've been making on this channel, the general AI safety concern
argument, all of the reasons why alignment and AI safety research is necessary. So do the AI
experts agree with that? Well, 11% of them think, no, it's not a real problem. 19% think,
no, it's not an important problem, but the remainder, 70% of the AI experts agree that
this is at least a moderately important problem. Now, I don't know how many of them are losing
sleep, but I would say that the implication of the article's claim that AI researchers don't
think AI risk is a serious concern is just factually not true. AI experts are concerned
about this. And the thing that's strange is the article quotes Ramez Naam as a computer scientist,
but the only time anyone is referred to as an AI expert is this quote at the end from Stuart
Russell. That name should be familiar. He's the guy posing the argument in the survey.
The AI expert who Pinker quotes kind of giving the impression that they're on the same side of
this debate is not at all. The thing that astonishes me about this is just how little
research would have been needed to spot this mistake. If you're wondering about Stuart
Russell's opinion, you might read something he's written on the subject. If you're kind of lazy,
you might just read the abstracts. If you're very lazy and you just want to get a quick feel for the
basic question, is Stuart Russell worried about existential risks from artificial intelligence,
you might just go to his website and look at the titles of his recent publications.
Oh, these academics with their jargon, how are we supposed to decipher what they really think?
Anyway, the next point the article makes is that the arguments for AI risk rely on an
oversimplified conception of intelligence, that they think of intelligence as this sort of magical
one-dimensional thing that gives you power. It's just like animals have some of it, humans have
more, and AI would have even more, and so it's dangerous. The article makes the point that
intelligence is multifaceted and multidimensional. Human intelligence is not monolithic. It isn't
this single thing which automatically gives you effectiveness in all domains.
And that's true. It's an important point. And I think sometimes when communicating
about this stuff to the public, people oversimplify this fact. But the article
seems to use this to suggest that general intelligence is not a thing, or that generality
is not a thing, that all intelligence is an accumulation of a number of small,
sort of narrow, specific abilities. He lists some things that human intelligence can do.
Find food, win friends and influence people, charm prospective mates, bring up children,
move around the world, and pursue other human obsessions and pastimes.
And yeah, these are pretty much all things that animals also do to some extent,
but they aren't the limits of human intelligence. Human beings do definitely seem to have some
general intelligence. There's some ability we have which allows us to act intelligently in
an unusually broad range of domains, including domains which are quite different from what our
ancestors experienced, things which our brains didn't evolve specifically to do. Like, we can
learn how to play games like chess and Go, to do very intelligent manipulation of these pretty
arbitrary systems which we invented. We can learn how to drive cars, although there were no cars in
the ancestral environment. And in fact, we can learn how to build cars, which is pretty unlike
anything our ancestors or other animals can do. We can operate intelligently in new environments,
even in completely alien environments. We can operate on the moon. We can build rockets. We
can attach cars to rockets, fly them to the moon, and then drive the cars on the moon.
And there's nothing else on earth that can do that yet. So yeah, intelligence is clearly not
a single thing, and it's clearly possible to be smart at some things and stupid at others.
Intelligence in one domain doesn't automatically transfer to other domains and so on.
Nonetheless, there is such a thing as general intelligence that allows an agent to act
intelligently across a wide range of tasks. And humans do have it, at least to some extent.
The next argument that he makes is, I think, the best argument in the article.
It actually comes out of the orthogonality thesis. He's saying that concern about AI risk
comes from a confusion between intelligence and motivation. Even if we invent superhumanly
intelligent robots, why would they want to enslave their masters or take over the world?
He says, being smart is not the same as wanting something. This is basically a restatement of
the orthogonality thesis, which is that almost any level of intelligence is compatible with almost
any terminal goal. So this is a pretty sensible argument, saying you're proposing that AI would
do these fairly specific world domination type actions, which suggests that it has world domination
type goals, but actually it could have any goals. There's no reason to believe it will want to do
bad things to us. It could just as easily want to do good things. And this is true so far as it goes,
but it ignores the argument from instrumental convergence, which is that while intelligence
is compatible with almost any terminal goal, that doesn't mean that we have no information
about instrumental goals. Certain behaviors are likely to occur in arbitrary agents because
they're a good way of achieving a wide variety of different goals. So we can make predictions about
how AI systems are likely to behave, even if we don't know what their terminal goals are. And if
that doesn't seem obvious to you, I made a whole video about instrumental convergence, which you
should watch, link in the description. The next thing the article says is, the scenarios that
supposedly illustrate the existential threat to the human species of advanced artificial
intelligence are fortunately self-refuting. They depend on the premises that one, humans are so
gifted that they can design an omniscient and omnipotent AI, yet so moronic that they would
give it control of the universe without testing how it works. And two, the AI would be so brilliant
that it could figure out how to transmute elements and rewire brains, yet so imbecilic that it would
wreak havoc based on elementary blunders of misunderstanding. Okay, so before we get into
the meat of this, I have some nitpicks. Firstly, an AGI doesn't have to be omniscient and omnipotent
to be far superior to humans along all axes. And it doesn't need to be far superior to humans along
all axes in order to be very dangerous. But even accepting that this thing is extremely powerful,
because that assumption is sometimes used, I object to this idea that humans would give it
control of the universe without testing it. You've said it's omnipotent. It doesn't need to be given
control of anything, right? If you actually do have an omniscient, omnipotent AGI, then giving
it control of the universe is pretty much just turning it on. The idea that a system like that
could be reliably contained or controlled is pretty absurd, though that's a subject for a different
video. Now this second part, about the AI being smart enough to be powerful, yet dumb enough to
do what we said instead of what we meant, is just based on an inaccurate model of how these systems
work. The idea is not that the system is switched on and then given a goal in English which it then
interprets to the best of its ability and tries to achieve. The idea is that the goal is part of
the programming of the system. You can't create an agent with no goals. Something with no goals is
not an agent. So he's describing it as though the goal of the agent is to interpret the commands
that it's given by a human and then try to figure out what the human meant, rather than what they
said, and do that. If we could build such a system, well, that would be relatively safe. But we can't
do that. We don't know how, because we don't know how to write a program which corresponds to what
we mean when we say, listen to the commands that the humans give you and interpret them according
to the best of your abilities, and then try to do what they mean rather than what they say.
This is kind of the core of the problem. Writing the code which corresponds to that is really
difficult. We don't know how to do it, even with infinite computing power. And the thing is, writing
something which just wants to collect stamps or make paperclips or whatever is way easier than
writing something which actually does what we want. That's a problem, because generally speaking we
tend to do the easier thing first, and it's much easier to make an unsafe AGI than it is to make a
safe one. So by default we'd expect the first AGI systems to be unsafe. But apart from those problems,
the big issue with this part of the article is that both of these points rely on exactly the
same simplified model of intelligence that's criticized earlier in the article. He starts off
saying that intelligence isn't just a single monolithic thing, and being smart at one thing
doesn't automatically make you smart at other things, and then goes on to say if humans are
smart enough to design an extremely powerful AI, then they must also be smart enough to do so safely.
And that's just not true. It's very possible to be smart in one way and stupid in another.
So in this specific case, the question is, are humans ever smart enough to develop extremely
sophisticated and powerful technology, and yet not smart enough to think through all of the
possible consequences and ramifications of deploying that technology before they deploy it?
And the answer here seems to be very clearly, yes. Yeah, humans do that a lot.
Like, all the time humans do that. And then similarly, the idea that an AI would be so
brilliant that it could transmute elements and rewire brains, and yet so imbecilic that it would
wreak havoc based on elementary blunders of misunderstanding, you can be smart at one thing
and stupid at another. Especially if you're a software system. And anyone who's interacted
with software systems knows this, to the point that it's a cliche. I mean, you want to talk about
self-defeating arguments? There are a few more things I would want to say about this article,
but this video is long enough as it is. So I'll just close by saying, I really like Steven Pinker,
but this article is just uncharacteristically bad. I agree that we have a problem with unthinking,
cynical pessimism about technology, but concern about AI risks is not just more of the same.
In its most productive form, it's not even pessimism. The thing is,
AI can be a huge force for good, but it won't be by default. We need to focus on safety,
and we won't get that without suitable concern about the risks. We need to push back against
pessimism and make the point that things are getting better, and science and technology
is a big part of the reason for that. But let's not make AI safety a casualty in that fight.
I've had some of my patrons suggest that I try making a larger volume of lower effort content,
so I'm trying an experiment. I've made a video that's just me reading through this whole article
and adding my comments as I go. It's like 45 minutes long, it's kind of
unstructured, rambly sort of video. I'm not sure it belongs on this channel,
but I posted it to Patreon so people can let me know what they think of it. So if you're interested
in seeing that kind of experimental stuff from me, consider joining all of these amazing people
and becoming a patron. And thank you so much to everyone who does that. In this video,
I'm especially thanking JJ Hepboy, who's been a patron for more than a year now,
and has finally come up in the video thanks rotation. Thank you so much for your support,
JJ. And thank you all for watching. I'll see you next time.


Stop Button Solution? - Computerphile

a while back we were talking about
uh the stop button problem right you
have this you have this
uh it's kind of a toy problem in ai
safety you have an artificial general
intelligence in a robot it wants
something you know it wants to make you
a cup of tea or whatever you put a big
red stop button on it
and you want to set it up so that it
behaves courageably that it will
allow you to hit the button it won't hit
the button itself you know and it won't
try and prevent you this sort of uh
behaving in a
in a sensible way in a safe way
and that like by default
um most agi designs will not behave this
way well we left it as an open problem
right
and it kind of still is an open problem
but there have been some really
interesting things proposed as possible
solutions or approaches to take and i
wanted to talk about cooperative inverse
reinforcement learning
i thought the easiest way to explain
cooperative inverse reinforcement
learning is to build it up backwards
right learning we know like machine
learning and reinforcement learning is
an area of machine learning i guess you
could call it it's it's kind of a
it's a way of presenting a problem
in most machine learning um the kind of
thing
that people have talked about already a
lot on computer file thinking of uber's
videos and the the related ones usually
you get in some data and then you're
trying to do something with that like
classify
you know unseen things or you're trying
to do like regression to find out what
value something would have for certain
inputs that kind of thing
whereas
reinforcement learning the idea is you
have an agent in an environment
and you're trying to find um
a policy but so so
let's back up what do we mean by an
agent it's an entity that interacts with
its environment
to try and achieve something effectively
it's doing things in an environment so
this isn't necessarily a is this a
physical thing or is it
doesn't have to be so if you have a
robot in a room then you can model that
as the robot being an agent in the room
being the environment similarly if you
have a
computer game like um
pac-man
then pac-man is an agent and the sort of
maze he's in is his environment so let's
stick with pac-man then the way that a
reinforcement learning
framework for dealing with pac-man is
you say okay you've got pac-man he's the
agent he's in the environment and you
have actions that pac-man can take in
the environment now it's kind of neat in
pac-man there are always exactly four
actions you can take oh well i guess
five you can sit there and do nothing
you can move up left right or down you
don't always have all of those options
like sometimes there's a wall and you
can't move right but those are the only
that's the that's the complete set of
actions that you have
um
and then you have the environment
contains sort of dots that you can pick
up which are they give you points it's
got these ghosts that chase you that you
don't want to touch and i think there's
also there's like pills you can pick up
that make the ghosts edible and then you
chase them down and stuff anyway so the
difference in reinforcement learning is
that the agent is in the environment and
it learns by interacting with the
environment
it's and so it's kind of close to the
way that animals learn and the way the
humans learn
you try you try doing something you know
i'm gonna try
you know touching this fire oh that hurt
so that's that's caused me like a
negative reward that's caused me a pain
signal which is something i don't want
so i learn to avoid doing things like
touching a fire so in in a pac-man
environment you might you might sort of
say
if you're in this you're in a situation
like let's draw pac-man let's say he's
in a maze like this
you look at pac-man's options
he can't go left he can't go right
he can go up and if he goes up he'll get
a dot
which earns you some points so
up gets a score of you know plus 10 or
however you've decided it um or well
whatever the score is in the game either
way or if he goes down he'll be
immediately got by this ghost
the point is that pac-man doesn't need
to be aware of the entire board right
the entire maze you can just feed in a
fairly small amount of information about
its immediate environment which is the
same thing as
if you have a robot in a room
it can't it doesn't know everything
about the whole room it can only see
what it sees through its camera you know
it has um sensors that give it some
some information about the environment
um
partial information
i i suppose just playing devil's
advocate the difference here is you
usually pac-man is being controlled by a
human who can see the whole board
so the point being if that ghost is
actually not static and is chasing
pacman and he's heading up to get that
pill if uh
if a few pixels later that
that corridor if you like stops in a
dead end
yep well he's kind of stuffed either way
really first that's true yeah so um that
is because so so
most
well yeah almost every
um
reinforcement learning algorithm almost
everything that tries to deal with this
problem
doesn't just look at the immediate
surroundings or it looks at the
immediate surroundings but it also looks
a certain distance in time
so you're not just saying what's going
to happen next frame
but
so like if you if you go down
here
most algorithms would say okay the
option of going down in this situation
is bad but also
all of the options we chose in all of
the situations that we were in in the
last
second or two also get a little bit
this is kind of a decay
there's time time discounting
so that
uh
you're not just punishing the immediate
thing that causes the negative reward
but also the decisions you make leading
up to it
so that pacman might learn not to get
himself stuck in corners
um as well as just learning not to run
straight into ghosts
so
that's the basics of reinforcement
learning there's different algorithms
that do it and the idea is you uh you
actually you start off exploring the
environment just at random you just pick
completely random actions and then as
those actions start having consequences
for you and you start getting rewards
and punishments you start to learn
um
which actions are better to use in which
situations does that mean that in
pac-man's case would learn the maze or
would it just learn the better choices
depends on
what algorithm you're using
um
a very sophisticated one might learn the
whole maze a simpler one might just
learn
um
a more kind of local policy
but the point is yeah you learn you
learn a kind of mapping between
or a function that takes in the
situation you're in and outputs a good
action to take
there's also kind of an interesting
trade-off there which i think we may
have talked about before about
exploration versus exploitation
in that
you want your agent to be generally
taking good actions but
you don't want it to always take the
action that it thinks is best right now
because it's understanding maybe
incomplete and then it just kind of gets
stuck right it never finds out anything
it never finds out anything about other
uh options that it could have gone with
because as soon as it did something that
kind of worked it just goes with that
forever so a lot of these systems build
in some uh
some variants some randomness or
something right exactly like you usually
do the thing you think is best but some
small percentage of the time you just
try something random anyway
um
and you can change that over time like a
lot of algorithms as as the
policy gets more and more as they learn
more and more they start doing random
stuff less and less
um that kind of thing so that's the like
absolute basics of reinforcement
learning and how it works and it's
really really powerful
like
especially when you combine it with deep
neural networks as the thing that's
doing the learning
like deepmind did this really amazing
thing where i think they were playing
pac-man they were playing a bunch of
different atari games and the thing
that's cool about it is
all they told the system was here's
what's on the screen
and here's the score of the game
make the score be big the score is your
reward right that's it
and it learned
all of the specific dynamics of the game
and generally achieved top level human
or superhuman play
the next word is going to be inverse we
did a thing with huvey on anti-learning
but can't work all the time that sort of
thing right yeah this is not like that
this is a description of a different
type of problem it's it's a totally
different problem that they call inverse
because
in reinforcement learning
you have a reward function that
determines when you what situations you
get rewards in and you're in your
environment with your reward function
and you're trying to
find the appropriate actions to take
that maximize that reward
in inverse reinforcement learning
you're not in the environment at all
you're watching an expert so you've got
the video of the world championship
record pac-man player right
and you have all of that all of that
information you can see
so you're saying
rather than rather than having the
reward function and trying to figure out
the actions
you can see the actions and you're
trying to figure out the reward function
so it's inverse because you're kind of
solving the reverse of the problem
you're not trying to
maximize a reward
uh by choosing actions you're looking at
actions and trying to figure out what
reward they're maximizing
so that's really useful because it lets
you sort of learn by observing experts
so coming back to ai safety you might
think that this would be kind of useful
from an ai safety perspective you know
you have this problem the core problem
of ai safety or one of the core problems
with air safety is
how do you make sure the ai wants what
we want
we can't reliably specify what it is we
want
um
so
and if we create something very
intelligent that wants something else
that's something else is what's probably
going to happen even if we don't want
that to happen how do we make a system
that reliably wants the same thing we
want
so you can see how inverse reinforcement
learning might be kind of attractive
here because you might have a system
that watches humans doing things
and tries to figure out you know if we
are experts at being humans it's trying
to figure out what rewards we're
maximizing and try and sort of formalize
in its
um
in its understanding what it is we want
by observing us
that's pretty cool
uh
but yeah it has some problems
one problem is that we don't
in inverse reinforcement learning
there's this assumption
of optimality
that the person that the the agent
you're watching is an expert and they're
doing optimal play
and you're you know there is some clear
coherent thing like the score that
they're optimizing and
the assumption of
the the algorithms that do this is that
the way the world champion plays is the
best possible way
and that assumption is obviously never
quite true
or generally not quite true
um but it works well enough you know
but
humans are not
like
human behavior is not
actually really optimizing to get what
humans want perfectly and ways
uh places where that assumption isn't
true could cause problems so is this
where cooperative comes in because when
we started this we're doing it backwards
it's cooperative inverse reinforcement
learning right so you could imagine a
situation where you have the robot you
have the agi it watches people doing
their thing uses inverse reinforcement
learning to try and figure out
the things humans value
sorry try and figure out the things
humans value
and then
adopt those values as its own
right
the most obvious
like the first problem is we don't
actually want to create something that
values the same thing as humans
like
if it observes that i
you know i want a cup of tea
we want it to want me to have a cup of
tea we don't want it to want a cup of
tea but that's like that's quite easy to
fix you just say you know figure out
what the value is and then optimize it
for the humans
say easy to fix but you know what i mean
it's
that's doable
um
but then the other thing is if you're if
you're trying to teach
if you're actually trying to use this
to teach a robot to do something
it turns out to not be very efficient
like if you
this works for pac-man if you want to
learn how to be good at pac-man you
probably want to not just watch the
world's best pac-man player and try to
copy them right that's not like an
efficient way to learn
because there might be a situation where
you you're thinking
what do i do if i find myself stuck in
this corner of the
maze or whatever and the pros never get
stuck there so you have no
uh you have no example of what to do all
all the pro all watching the pros can
teach you is
don't get stuck there and then once
you're there you've got no
you've got no hope let's say i wanted to
teach my robot to make me a cup of tea i
go into the kitchen and i show it
how i make a cup of tea
i would probably have to do that a lot
of times
to actually get the all the information
across
because
and you'll notice this is not how people
teach right if you were teaching a
person how to make a cup of tea
you might do something like
if there's some difficult stage of the
process you might show you might do one
demonstration
but show that one stage like three times
say and you see do it like this let me
show you that again and then
if you're using inverse reinforcement
learning the system believes that you
are playing optimally
right so it thinks that doing it three
times is somehow necessary and it's
trying to figure out what values like
what reward you must be optimizing but
doing it three times is
important
so that's a problem right that's where
the assumption isn't true
or you might want to say
okay what you do is you get the tea out
of the box here and you put it in the
thing but
if
there's none in this box you go over to
this cupboard where we keep the backup
supplies and you open a new box right
but you can't show that the only way
that
the only way that the robot can learn
to go and get the extra supplies only
when this one has run out
is if you were in a situation where that
would be optimal play so the thing has
to be actually run out
in order for you to demonstrate that you
can't say if the situation were
different from how it is then you should
go and do this so the other thing you
might want if you're trying to teach
things efficiently you might want the ai
to be
taking an active role in the learning
process right you kind of want it to be
like if there's if there's some aspect
of it that it doesn't understand you
don't want it just sitting there
observing you optimally do the thing and
then trying to copy
if there's something that it didn't see
you kind of want it to be able to say
hang on i didn't see that you know or
i'm confused about this or maybe ask you
a clarifying question or um
just in general like communicate with
you
and cooperate with you in the learning
process
um
so
yeah so so the way that the way that
cooperative inverse reinforcement
learning works is it's a way of setting
up the rewards
such that
these types of behaviors hopefully
will be incentivized and should come out
automatically if you're optimizing
you know if the ai is doing well so what
you do is you specify the interaction as
a cooperative game
where
the robot's reward function
is the humans reward function
but
the robot
doesn't know that reward function at all
it never knows
the reward that it gets
and it never knows the function that
generates the reward that it gets
it just knows that it's the same as the
humans
so it's trying to optimize it's trying
to maximize the reward it gets
but
the only
clues it has for what it needs to do to
maximize its own reward
is observing the human and trying to
figure out
what the human is trying to maximize
this is a bit like two players on a
computer game but you can only see one
score
yeah like if you're you're both on the
same team yeah
uh but only the human knows the rules of
the game effectively you both want
you both get the same reward so you both
want the same thing just kind of by
definition
but the pro so in a sense you've kind of
just defined the core problem of as i
was saying the core pro one of the core
problems of ai safety is um
how do you make sure that the robot
wants what the human wants and in this
case you just specified it usually you
couldn't do that because we don't really
know what the human wants either
two people who don't speak the same
language can still communicate
with actions and gestures and things
yeah and you can generally get the gist
of the idea across to the other person
is it a bit like that yeah but a
sufficiently sophisticated agent
uh
if you have an agi that could be quite
powerful
it can speak you know and it can
understand language and everything else
and it knows
that
so it knows for example
uh hopefully it should be able to figure
out that
when the human is showing something
three times that it's that the human is
doing that in order to communicate
information and not because it's the
optimal way to do it
because it knows
that the human knows
there's kind of there's common knowledge
of what's going on in this in the
scenario so it allows for situations
where the human is just demonstrating
something
or explaining something
or it allows the ai to ask about things
that it's unclear about because
everybody's on the same team trying to
achieve the same thing
in principle
um
so the point is
if you have a big red stop button in
this scenario
the ai is not incentivized to disable or
ignore that stop button
because
it constitutes important information
about its reward
right the ai is desperately trying to
maximize a reward function that it
doesn't know
and so if it observes the human trying
to hit the stop button
that provides really strong information
that what it's doing right now is not
going to maximize the humans reward
which means it's not going to maximize
its own reward
so it wants
to allow itself to be shut off if the
human wants to shut it off
because it's for its own good
so this is this is a clever way of
aligning its interests with ours right
right
it it's not so so like
the the the problem in the in the
default situation is i've told it to get
a cup of tea
and it's going to do that
whatever else i do
and if i try to turn it off it's not
going to let me because that will stop
it from getting you a cup of tea whereas
in this situation the fact that i want a
cup of tea is something it's not
completely sure of
and so it doesn't think it knows better
than me
so when i go to hit that stop button it
thinks oh i thought i was supposed to be
going over here and getting a cup of tea
and running over this baby or whatever
but the fact that he's rushing to hit
the button means i must have gotten
something wrong
so i better stop
and learn more about this situation
because i'm at risk of losing a bunch of
reward
so yeah it seems like it seems like a
potentially workable
thing
um a workable approach
so uh one interesting thing about this
is there is still an assumption
that the human's behavior
is in accordance with some utility
function or some reward function some
objective function like
if the human behaves very irrationally
that can
cause problems for this system because
the whole thing
revolves around the fact that the robot
is not completely confident of what its
reward is it's got a model of its of
what the reward function is like that
it's constantly updating as it learns
um and it doesn't have full confidence
and it's using the human as the source
of information
so fundamentally the robot believes that
the human knows better than it does
how to maximize the human's reward
so in situations where that's not true
like if you run this for long enough
and the um
robot managed to build up a really
really high level of confidence
in what it thinks the human reward
function is
then it might ignore its stop button
later on if it thinks that it knows
better than the human what the human
wants
um which sounds very scary
but might actually be what you want to
happen
like if you imagine
you know it's the it's the future and
we've got these robots and they all have
a big red stop button on them and
they're all you know
and everything's wonderful and you say
to your robot oh take my
uh my four-year-old son to school you
know drive him to school in the car
because it's the 1950s sci-fi future
where it's not self-driving cars it's
like robots in cars anyway
and it's um
it's driving this kid to school it's
doing 70 on the motorway and the kid
sees the big red shiny button and smacks
it right
in principle a human has just pressed
the button and
a lot of designs for a button would just
say a human has hit your button you have
to stop whereas this design might say
i have been around for a long time i've
learned a lot about what humans value
and also i observe that this specific
human
does not reliably behave in its own best
interests
so maybe this hitting the button is not
communicating to me information about
what this human really wants
they're just hitting it because it's a
big red button and i should not shut
myself off
so
it has the potential to be safer than a
button that always works
but it's a little bit unsettling that
you might end up with systems that
sometimes actually do ignore the
shutdown command because they think they
know better
because what it's looking at right now
is it says button gets hit i get zero
reward
button doesn't get hit if i manage to
stop them then i get the cup of tea i
get like maximum reward
if you give some sort of compensation


AI That Doesn't Try Too Hard - Maximizers and Satisficers

hi so way back when I started this
online air safety videos thing on
computer file I was talking about how
you have a problem when you maximize
just about any simple utility function
the example I used was an AI system
meant to collect a lot of stamps which
works like this the system is connected
to the Internet and for all sequences of
packets it could send it simulates
exactly how many stamps would end up
being collected after one year if it
sent those packets it then selects the
sequence with the most stamps and sense
that this is what's called a utility
Maximizer and it seems like any utility
function you give this kind of system as
a goal it does it to the max utility
maximizers
tend to take extreme actions they're
happy to destroy the whole world just to
get a tiny increase in the output of
their utility functions so unless the
utility function lines up exactly with
human values their actions are pretty
much guaranteed to be disastrous
intuitively the issue is that utility
maximizers have precisely zero chill to
anthropomorphize horribly they seem to
have a frantic obsessive maniacal
attitude we find ourselves wanting to
say look could you just dial it back a
little can you just relax just a bit so
suppose we want a lot of stamps but like
not that many it must be possible to
design a system that just collects a
bunch of stamps and then stops right how
can we do that well the first obvious
issue with the existing design is that
the utility function is unbounded the
more stamps the better with no limit
however many stamps it has it can get
more utility by getting one more stamp
any world where humans are alive and
happy is a world that could have more
stamps in it so the maximum of this
utility function is the end of the world
let's say we only really want a hundred
stamps so what if we make a bounded
utility function that returns whichever
is smaller the number of stamps or 100
getting a hundred stamps from ebay gives
100 utility converting the whole world
into stamps also gives 100 utility this
function is totally indifferent between
all outcomes that contain more than a
hundred stamps so what does a Maximizer
of this utility function actually do now
the system's behavior is no longer
really specified it will do one of the
things that results in a hundred utility
which includes a bunch of perfectly
reasonable behaviors that the programmer
would be happy with
and a bunch of apocalypse is and a bunch
of outcomes somewhere in between
if you select at random from all courses
of action that result in at least 100
stamps what proportion of those are
actually acceptable outcomes for humans
I don't know probably not enough this is
still a step up though because the
previous utility function was guaranteed
to kill everyone and this new one has at
least some probability of doing the
right thing but actually of course this
utility Maximizer concept is too
unrealistic even in the realm of talking
about hypothetical agents in the
abstract in the field experiment our
stamp collector system is able to know
with certainty exactly how many stamps
any particular course of action will
result in but you just can't simulate
the world that accurately it's more than
just computationally intractable it's
probably not even allowed by physics
pure utility maximization is only
available for very simple problems where
everything is deterministic and fully
known if there's any uncertainty you
have to do expected utility maximizing
this is pretty straightforwardly how
you'd expect to apply uncertainty to
this situation the expected utility of a
choice is the utility you'd expect to
get from it on average so like suppose
there's a button that flips a coin and
if its tail's you get 50 stamps and if
it's heads you get 150 stamps in
expectation this results in a hundred
stamps right it never actually returns
100 but on average that's what you get
that's the expected number of stamps to
get the expected utility you just apply
your utility function to each of the
outcomes before you do the rest of the
calculation so if your utility function
is just how many stamps do I get then
the expected utility of the button is
100 but if your utility function is
capped at a hundred for example then the
outcome of winning one hundred and fifty
stamps is now only worth a hundred
utility so the expected utility of the
button is only 75 now suppose there were
a second button that gives either eighty
or ninety stamps again with 50/50
probability this gives 85 stamps in
expectation and since none of the
outcomes are more than 100 both of the
functions value this button at 85
utility so this means the agent with the
unbounded utility function would prefer
the first button with its expected 100
stamps but the agent with the bounded
utility function would prefer the second
button since its expected utility of 85
is higher than the
buttons expected utility of 75 this
makes the bounded utility function feel
a little safer in this case it actually
makes the agent prefer the option that
results in fewer stamps because it just
doesn't care about any stamps past 100
in the same way let's consider some
risky extreme stamp collecting plan this
plan is pretty likely to fail and in
that case the agent might be destroyed
and get no stamps but if the plan
succeeds the agent could take over the
world and get a trillion stamps an agent
with an unbounded utility function would
rate this plan pretty highly the huge
utility of taking over the world makes
the risk worth it but the agent with the
bounded utility function doesn't prefer
a trillion stamps to a hundred stamps
it only gets 100 utility either way so
it would much prefer a conservative
strategy that just gets a hundred stamps
with high confidence but how does this
kind of system behave in the real world
where you never really know anything
with absolute certainty the pure utility
Maximizer that effectively knows the
future can order a hundred stamps and
know that it will get 100 stamps but the
expected utility maximize it doesn't
know for sure the seller might be lying
the package might get lost and so on so
if the expected utility of ordering a
hundred stamps is a bit less than 100 if
there's a 1% chance that something goes
wrong and we get 0 stamps then our
expected utility is only 99 that's below
the limit of 100 so we can improve that
by ordering some extras to be on the
safe side maybe we order another 100 now
our expected utility is 99.99 still not
a hundred so we should order some more
just in case now we're at 99.9999 the
expected value of a utility function
that's bounded at 100 can never actually
hit 100 you can always become slightly
more certain that you've got at least
100 stamps better turn the whole world
into stamps because hey you never know
so an expected utility Maximizer with a
bounded utility function ends up pretty
much as dangerous as one with an
unbounded utility function ok what if we
try to limit it from both sides like you
get a hundred utility if you have a
hundred stamps and zero otherwise now
it's not going to collect a trillion
stamps just to be sure it will collect
exactly 100 stamps but it's still
incentivized to take extreme actions to
be sure that it really does have a
hundred like turning the whole world
into elaborate highly
stamp counting and recounting machinery
getting slightly more utility every time
it checks again it seems like whatever
we try to maximize it causes problems so
maybe we could try not maximizing maybe
we could try what's called satisficing
rather than trying to get our utility
function to return as higher value as
possible and expectation what if we set
a threshold and accept any strategy that
passes that threshold in the case of the
stamp collector that would look like
look through possible ways you could
send out packets calculate how many
stamps you'd expect to collect on
average with each strategy and as soon
as you hit one that you expect to get at
least 100 stamps just go with that one
this satisficer seems to get us to about
where we were with the pure utility
Maximizer with a bounded utility
function it's not clear exactly what it
will do except that it will do one of
the things that results in more than a
hundred stamps in expectation which
again includes a lot of sensible
behaviors and a lot of apocalypses and a
lot of things somewhere in between since
the system implements the first
satisfactory strategy it finds the
specific behavior depends on the order
in which it considers the options what
automated use well one obvious approach
is to go with the simplest or shortest
plans first after all any plan that
takes over the world probably requires
much more complexity than just ordering
some stamps on eBay but consider the
following plan get into your own source
code and change yourself from a
satisficer into a Maximizer all you're
doing there is changing a few lines of
code on your own system so this is a
pretty simple plan that's likely to be
considered fairly early on it might not
be simpler than just ordering some
stamps but that's not much reassurance
the more challenging the task we give
our AGI the more likely it is that it
will hit on this kind of self
modification strategy before any
legitimate ones and the plan certainly
satisfies the search criteria if you
change yourself into a Maximizer that
Maximizer will predictably find and
implement some plan that results in a
lot of stamps so you can tell that the
expected stamp output of the become a
Maximizer plan is satisfactorily high
even without knowing what plan the
Maximizer will actually implement so
satisficers kind of want to become
maximizes which means that being a
satisficer is unstable as a safety
feature it tends to uninstall itself so
to recap a powerful utility maximized
with an unbounded utility function is a
guaranteed apocalypse with a bounded
utility function it's better in that
it's completely indifferent between
doing what we want and disaster but we
can't build that because it needs
perfect prediction of the future so it's
more realistic to consider an expected
utility Maximizer which is a guaranteed
apocalypse even with a bounded utility
function now an expected utility
satisficer
gets us back up to in difference between
good outcomes and apocalypses but it may
want to modify itself into a Maximizer
and there's nothing to stop it from
doing that so currently things aren't
looking great but we're not done people
have thought of more approaches and
we'll talk about some of those in the
next video
I want to end the video with a big thank
you to all of my wonderful Patriots
that's all of these great people right
here in this video I'm especially
thanking Simon strand card thank you so
much you know thanks to your support I
was able to buy this boat for this I
bought a green screen actually but I
like it because it lets me make videos
like this one that I put up on my second
channel where I used GPT to to generate
a bunch of fake YouTube comments and
read them that video ties in with three
other videos I made with computer file
talking about the ethics of releasing AI
systems that might have malicious uses
so you can check all of those out
there's links in the description thank
you again to my patrons and thank you
all for watching I'll see you next time


What can AGI do? I/O and Speed

hi a long time ago in computerphile we
introduced the stamp collector this
hypothetical super intelligence which
just has crazy extreme powers by magic
effectively it's a thought experiment
then we brought things back down to
earth and started looking at this paper
concrete problems in AI safety which
looks at how we can improve the safety
of current AI systems hopefully in a way
that will be applicable to future
general AI systems and super
intelligences so let's talk a little bit
about what I expect those AGI systems to
be like this is purely speculation of
course but here are some of my thoughts
about what kind of capabilities we might
expect from a human level AGI and some
of my intuitions about why it might not
stay human level for long so suppose we
figure out how to implement general
intelligence we have a system with that
magic ability that humans have to tackle
a wide variety of different cognitive
tasks and let's assume that it's not
just anthropomorphic it's not a direct
one-to-one emulation of a human brain
but it's a general intelligence it can
in principle learn to do any cognitive
task that a human can and let's assume
that it's more or less human level it
can do any cognitive task a human can
about as well as the best humans but
note that this assumption still makes
our AGI pretty stupid in a lot of
domains because humans are stupid in a
lot of domains if the AGI is only about
as good as the best humans at something
like mental arithmetic it can be
outperformed by a human with a
calculator right if an AGI that's about
as good as the best humans at chess
place the best human chess player it
should be pretty close but in practice
such an Ag I would just download and run
one of the existing freely available
chess programs that already far
outperforms humans and if it needs to do
mental arithmetic it should be able to
just use its own existing computing
hardware so let's revise our definitions
and say that our AGI can perform any
cognitive task as well as the best
humans or if our existing narrow AI
outperforms humans then the AG eyes
performance will be closer to what our
best narrow AI systems can do so that's
one way that a human level AGI almost
immediately becomes sort of superhuman
it can outperform humans at things their
computers already outperform humans at
does that really count though like is a
human with a calculator really an
arithmetic super intelligence kind of
it depends where you draw your
boundaries the human calculator system
is much better at arithmetic than any
human so it kind of counts and this
brings us to some of the other
advantages a human level AGI might get
pretty much free of charge because if
you imagine the human has a calculator
implant in their brain that lets them
just think of equations and the
solutions immediately pop into their
mind that feels a lot more super
intelligent and that's much closer to
what the AGI would be working with
because humans are at a big disadvantage
when it comes to interacting with other
systems they are made out of meat for
humans interacting with computers is
slow you have a number in your brain
it's some pattern of electrical signals
but to get that into the Machine your
brain has to send signals to some nerves
that pass it on to more nerves across
synapses that work by chemical diffusion
that then activates a muscle fibers that
start to accelerate some meat and you
have to send loads of these signals to
lots of different muscles to get the bit
of meat to land on the right button and
then that completes the circuit and the
process can go along at a reasonable
speed again but you've got to do that
for every digit getting data out of a
human brain and into another system is
glacially slow by software standards and
getting data in is pretty slow as well
you can't just experience a number it
has to be presented as a pattern of
light which excites some light-sensitive
cells in your eye that sends a signal to
your optic nerve which goes to your
brain which then does a load of
processing to figure out that you're
looking at a 3 or whatever you know it's
it's really low bandwidth
high latency the round-trip time
measured as the time it takes a human to
press a button as soon as they see a
light flash is about 200 milliseconds
1/5 of a second that is a long time in
software terms as any systems programmer
will tell you gamers will know this to
think about playing a game with 200
milliseconds of ping that's what your
brain is working with all the time so
getting rid of that lag and increasing
the bandwidth should make an AGI much
faster at a lot of tasks and much better
at integrating other software systems
into its thinking even if that AGI is
not actually smarter than a human in any
other way the other problem with our
input system is that we can only really
take in information in certain forms
text images sound we can't easily pass
most types of information there's way
too much information
the matrix which is why we make graphs
and charts and data visualizations you
have to convert the data into something
that can be passed by systems designed
for hunting and gathering our AGI might
not be limited in that way it might be
able to experience different kinds of
data much more directly without having
to first map it into an existing sensory
modality it would probably have
specialized systems for processing
images or sound but it wouldn't have to
convert every input into images or sound
and then back again before it could use
it it could just use the data imagine a
software developer who's about as smart
as the best human software developers
but they can perceive data directly
without having to convert it into
symbols and visually read it and they
don't have to type or manually navigate
the files they can write code as fast as
they can think even without being
smarter than a human the improvement to
i/o could make such an AGI add much more
effective than a human if it couldn't
download a chess program it might be
able to write its own pretty quickly
another advantage I want to talk about
is the sort of cognitive flexibility you
can get when you're made of software it
could allow an AGI much more control
over at the mines operation for example
as a human you have one auditory system
which pretty much means you can process
one stream of audio at once like you
can't really listen to two people saying
different things in each year basically
even if in principle is you have enough
insurgency or Howard to do it you can
shut your eyes to avoid being distracted
and that might make it easier but you
can't then say well I'm not using my
visual cortex right now so I'll have
those neurons process one of the audio
streams brains don't work that way but
an AGI might be able to do just that
diverting computational resources from
one task to another as needed increasing
its effectiveness it might even be
capable of superhuman multitasking by
splitting off simpler versions of itself
to handle large numbers of simple tasks
and this is all assuming that the core
general intelligence part of the AI
system operates at the same speed as a
human brain which might not be true I
mean named a cognitive task that
computers can do as well as humans but
the computers do it slower
not easy right the general trend is we
go from computers can't do this at all
to computers can do this much faster
than people not always but in general so
I wouldn't be surprised if that pattern
continues with AGI and being faster is
valuable in general intelligence imagine
your brain stayed exactly the same but
the world slowed down to one tenth or
one hundredth of speed once you got used
to controlling your body at that speed
you'd effectively be much more
intelligent since you could afford to
put lots of careful thought and planning
into everything you do when someone says
something to you in a conversation you'd
have time to think for a minute or two
about what the other person said
consider the different meanings they
might have intended you know decide on
what kind of thing you want to say write
a draft go through and check it make a
couple of different versions you don't
think about like how each one might be
interpreted by each of the people
present where it might take a
conversation next and so on
you'd seem way smarter and you'd
effectively be much smarter but your
brain hasn't changed at all the world
just slowed down so in a way speed is a
form of super intelligence and I'd
expect a GIS to get faster over time as
well the obvious way is if you make a
human speed AGI a year later there'll be
new faster computers to run it on and it
will get faster but you can also speed
it up by giving it more Hardware note
that it's not always the case that a
system can be sped up by running it on
multiple computers you can't get a baby
in less than nine months by hiring two
pregnant women
some algorithms inherently require a
large number of sequential steps each
step needs to use the output of the
previous step so there's no way to break
the problem down into parts that can be
given to different machines to work on
in parallel maybe a GI will turn out to
be one of those but I don't think so I'd
expect it to be parallelizable for two
reasons
firstly just about all of the current
state-of-the-art AI systems are they're
built on frameworks like tensorflow that
are designed from the ground up to
efficiently run these algorithms on
large numbers of processes if we find a
GI along the path we're currently going
down it'll almost certainly be a
parallelizable algorithm secondly we
know that general intelligence does not
inherently require a large number of
sequential steps because human brains
have general intelligence and brains
can't do large numbers of sequential
steps at least not quickly the fastest
in neuron can fire is around 200 times
and so that has to be the most
sequential steps we can consistently
execute in a second
anything your brain can do every second
must be doable in less than 200
sequential steps because otherwise
neurons simply aren't fast enough to do
it
so whenever the brain is doing something
computationally impressive quickly it
has to be because it's using extremely
large numbers of neurons working in
parallel we know it must be possible to
parallelize a general intelligence
algorithm because it's already been done
so given a GI more hardware and it
should run faster and you just know that
as soon as the system starts to exhibit
behavior that looks like general
intelligence people are going to want to
run that on as many processors as they
can get their hands on so everything
I've talked about in this video assumes
that we can't actually make an algorithm
that's truly smarter than a human but
just something that has faster i/o and
can be run faster and that's already
capable of some pretty impressive things
one question I often get is what can an
AGI really do if it's just a computer
but nobody hopefully the capabilities
I've talked about here give some
impression of the kind of things it
might be able to do but I intend to
elaborate on that in a later video
[Music]
thank you so much to my excellent
patreon supporters that's these people
in this video I'm especially thanking
for bricio Pazhani he's the one on the
left
I think anyway I've recently been
thinking about ways to improve my
patreon currently patrons get early
access to the videos occasional like
behind-the-scenes stuff sometimes I send
out the diagrams and stuff I've drawn in
the videos and of course you can send me
messages on patreon and I reply to those
before I do like YouTube comments or
Facebook comments but if you have any
other ideas of course stuff I could do
send me a message if you're a patron or
even if you're not but you're
considering it let me know your thoughts
in the YouTube comments thanks for
watching and I'll see you next time
the fastest in neuron can fire is ah I
gotta buy some real sound damping