197. Ben Garfinkel on Scrutinizing Classic AI Risk Arguments 2

Hello and welcome to session 197 in the AI
Safety Reading Group.
Tonight we’ll continue with the podcast
of Ben Garfinkel on Scrutinizing Classic AI
Risk Arguments.
Ben Garfinkel is still a research fellow and
Howie Lempel, Robert Wiblin, and Keiran Harris
are the three people on the 80000hours, the
podcast team who has worked with this podcast.
This was published almost two months ago,
but recorded about a year ago.
Today we’re discussing the second half,
starting at the 1 hour 43 minutes mark, the
section called “Instrumental Convergence”.
Instrumental Convergence is criticised in
the following way: It would be possible to
make a methodology for predicting technology
by looking at all the possible combinations
of features that it might have, and see if
most of the ways to make this technology involve
a particular property ; probably when we make
this technology, we would make one that has
this property.
This, I feel, is not really how the Instrumental
Convergence is argued in most cases in the
classic presentations.
It’s not that most ways have this particular
property, it’s that either we’re in a
very, very degenerate case, or we in a case
that explicitly something positively aligned,
or we would have these instrumental convergent
goals.
The way Ben Garfinkel argues for “most of
the ways” is by analogy of a plane with
either open or closed windows.
He further argues that this is the methodology
that’s used to argue for instrumental convergence
in the classical AI risk arguments.
This way of looking at kinds of technologies
can probably be formalized with a bitstring,
with, like, there are this number of descriptions
and the ones that are most of - you can just
count them - has this property, then that
is what would happen.
But that is empathetically not how it is commonly
done in the classic AI risk arguments.
The first on instrumental convergence is by
Steve Omohundro: “The Basic AI Drives”,
where Steve Omohundro uses a definition and
an argument structure that is much closer
to the one I have up here which does not have
the word “most”, but saying that either
we’ll design very, very particularly, or
it will have the instrumental convergent subgoals.
It’s also not how it is done in Bostrom’s
book, “Superintelligence”, where instead
of looking at all possible programs, we’re
looking at not just the space of all possible
programs, but the space of programs that have
been filtered by humans that are searching
for AI, that has some kind of goal that the
humans are interested in, and where the AI,
it may not have the goal, but it will look
like it has this goal.
I think probably this is what Ben Garfinkel
is arguing you should do: Instead of just
looking at all these possible programs, we
should look at ones that look like they are
promising, and this is what Bostrom does,
and it doesn’t really solve the problem.
Most of the ones that look good, still are
not perfectly aligned, and are subject to
the basic AI drives.
Also there are some problems in the transcript
that aren’t clear, there were some parts
that I couldn’t quite parse.
This is the 3rd major argument for the problems
in the original AI risk arguments and, these
arguments are entangled in the following way:
AI researchers are likely to steer towards
some designs that are safer, and that’s
because we would have much more smooth, and
not, in particular not an intelligence explosion,
but a more smooth scenario.
That means we would notice the problems while
they’re small, and the alignment research
will be automatically improved along the way
because it will be a requirement for capability
progress.
And in this way, the arguments for or against
AI risk mutually reinforce each other.
And in particular, the take-off speed act
as some kind of multiplier or intensifier.
This means that if you have some model of
how the future’s going to be, and then you
just tweak these small parameters even just
a tiny bit - if you are ten percent more optimistic
on a rather small number of cases you can
end up with a dramatically more optimistic
- or pessimistic, for that sake - conclusion.
I think, if you are in a situation where the
arguments are entangled in this way, that
should mitigate against strong confidence
either for or against, because if, if you
can suddenly get into a…
Ben Garfinkel doesn’t need to be very wrong
on many of his assumptions, as long as he’s
wrong on several of them, then they will mutually
reinforce and then suddenly we could have
a serious AI risk, and also, of course, the
same for people who are certain that there
is doom, because if we would have a bit more
time and if alignment progress is a bit more
required, then a lot of things would look
a lot lighter.
Then there is the question of neural networks.
Howie Lempel asks: if the space of possible
AI designs is very dense with bad scenarios,
so even if you change a tiny bit, and don’t
get it alignment one hundred percent, then
you might end up in an awful scenario.
Ben Garfinkel answers in the following way:
by looking at neural networks where if we
make some small perturbation to the neural
network, change one weight or something like
that, then generally the behavior is mostly
the same.
And I believe that is true.
Ben Garfinkel is not completely certain about
this but I believe it’s true for capabilities,
but we don’t really care much about capabilities
when we talk about the instrumental convergence
thesis.
We talk about goals.
And inded for goals, small perturbations can
be very, very problematic.
There is the “complexity of values” thesis,
which holds that small perturbations can destroy
all… everything of value.
There is also the fact that, for instance
something like a utility-maximizer and a disutility-maximizers
are extremely close in bit-space, you only
need to basically flip one bit, and the actual
behavior is dramatically different.
One of the things in particular that will
cause substantial problems, or could cause
substantial problems, is that very close to
an AI goal-set what we approve of is an AI
goal-set that humans will actually fight against
because we don’t approve of it.
And that means that the capabilities in the
actual scenario of what will happen with an
AI, it depends intimately on whether we will
collaborate with it or we will fight it.
And that means that even quite small changes
could have dramatic threshold effects.
Ben Garfinkel also argues that changes in
distribution, which is another thing that
could happen, will give incompetent behavior
rather than dangerous.
This is something that could happen.
The classic AI risk arguments are in particular
interested in the changes in distribution
which is relevant to whether the AI perceives
that it will be able to make… obtain a decisive
strategic advantage.
Also distributional shifts like, for instance,
what precisely corresponds to a human or something
like that, could invalidate all alignment
work.
That’s also something that we need to look
out for.
Then there is quite a bit about mesa-optimization.
There is a description of this.
Some of it is correct and most of it is probably
not quite as correct.
Ben Garfinkel states many times that he’s
actually not really sure about this.
And that is something, it’s not really new,
at least moderately new.
Because Ben Garfinkel had this broadcast a
year ago, so back then it might have been
new.
But I think there are very serious problems
with Ben Garfinkel’s description of mesa
optimizers.
I think the key point is that mesa optimizers
are about learned models that are themselves
optimizers.
And this here is a classical image.
And there is nothing in the description that
Ben has… that really relies on this in any
way.
It’s usually just the standard problem that’s
sometimes called “outer alignment” here.
Ben doesn’t say anything really that leads
one to… that forces you to conclude that
he actually, you know, understands mesa-optimization.
It should be said here that when talking about
the evolution~human analogy they do get closer
but there is nothing really about mesa-optimization
in what Ben Garfinkel is saying.
Ben Garfinkel also talks about the relationship
between mesa-optimization and distributional
shifts and why we care about those, and I
think also that description… if I should
describe why we’d care about that, that’s
because if we have a mesa-optimizer that is
deployed in training, then this would actively
be looking for a distributional shift that
will show that it’s no longer training but
now it is actually capable of performing a
treacherous turn.
In this case the small perturbations that
we talked about could be really, really interesting,
because here we have an intelligence that
is adversarially looking for these small distributional
shifts.
And in this case even very small things can
be very problematic.
But this might not really be relevant to what
Ben Garfinkel is saying.
It’s somewhat unclear from the transcript,
this part, unfortunately.
On the other hand one thing we could say is
that mesa-optimization obviously is not part
of the classic AI risk arguments because it’s
a lot newer.
So why is it that there is so much talk about
this?
Well, I believe that if Ben Garfinkel had
written a formal article then probably there
would not be so much talk about mesa-optimization.
But the question posed by Howie Lempel: “What
are the best counter-arguments to your own
arguments?”, I think that it’s a really
hard question and this is the context where
Ben Garfinkel talks about mesa-optimization.
And I’m not really sure I like this kind
of question.
It’s really hard to skip over even if you
don’t really have anything good to say because
you feel you’ve already answered the main
points.
Then there is some talk about what Nick Bostrom
calls “perverse instantiation”, even though
that’s not really the word Ben Garfinkel
is using.
Ben Garfinkel states that there are ten pages
in the book Superintelligence about how these
benign-seeming objectives actually are really
problematic.
So I looked in Superintelligence: I have around
one page about this, two if you are generous.
But Ben Garfinkel argues that this kind of
thought experiment doesn’t really tell us
much about what the future will actually look
like.
And this is something that is acknowledged
explicitly by Bostrom: These are illustrations
of why naive alignment methods fail, and not
meant as any kind of prediction.
The argument by Ben Garfinkel here is that…
once we have a very powerful AI it would probably
be based on Machine Learning and not on feeding
English language sentences.
If we wanted to have powerful AI based on
English language sentences it must have common
sense understanding.
And if we have that, then this common sense
understanding will also be applied to the
goals.
This calls back to the brain-in-the-box-confusion
that we saw in the previous session.
Because in the classic AI risk arguments,
there we were talking about a seed AI that
can improve itself - rewrite its own source
code but it does not have any kind of common
sense.
And then if it’s given a goal, then the
literal interpretation of these goals will
be locked in during this process and will
not be changed.
Ben Garfinkel has an analogy from a movie
called “Explorers” which is about people
who receive some requests, and then horribly
misinterpret them.
And if an AI is trained on that, then what
would happen?
I’ve tried to figure out what this is a
reference to.
The closest I could get is “Dora the Explorer”
but from the Wikipedia article not really
sure whether that’s that.
So I would be interested if anyone knows about
an “Explorer” movie where anything like
that happens.
Then there is the part about the “Treacherous
Turn”.
And there is a description about the Treacherous
Turn - again it’s relying on mesa-optimization
which is explicitly not something that the
original description does.
The description of the Treacherous Turn is
also strictly assuming a smooth scenario.
And in the smooth scenario we would expect
to see a number of failed attempts at deception,
or some deception that exists but isn’t
really problematic, or at least not catastrophic.
And I’m not quite as optimistic as Ben Garfinkel
on how we will see this and how we expect
to see this.
Because if we have, you know, just a neural
network, then figuring out if it is really
deceptive or not, you know, it’s really
hard to peer into these kind of learned models
to figure out things that are on the level
of whether it’s deceptive or not.
And in particular if an AI attempts a takeover
without deception, then it’s very likely,
of course it’s dramatically more likely,
that you’ll notice it in this case.
And unfortunately the answer is that, yes,
it is very, very likely that if the AI notices
it can take over the world, and then takes
over the world without at any point deceiving
anyone, then of course we’ll notice it,
but it will also have taken over the world.
So it’s not really a lot of help that we
will see it.
This Treacherous Turn, how dangerous that
is, relies on how smooth the world is.
And indeed, as the transitions become faster,
treacherous turns become more dangerous.
Ben Garfinkel has the following statement
here, that’s not really related to the treacherous
turn, but: “I have a lot of trouble understanding
how you’d get to the point where an individual
system is powerful enough to destroy the world.”
I think this general framing is really problematic
because the way I normally try to summarize
the argument given in Superintelligence, I
divide it into six parts, like A implies B
etc.
And if you zoom out all the way, A to G, then
it does seem like a big leap to go from “We
have an AI”, and suddenly “It has taken
over the world” - that does seem unlikely.
But, you know, the point of scrutinizing the
classic arguments is that you have to go into
all these small sub-implications, and see
whether those are actually true or likely.
The intuitive feeling that the entire thing
seems unlikely - it might still be useful
but scrutinizing the arguments themselves,
I believe, is more important and that’s
certainly what we’re supposed to be doing
here.
How many systems will be treacherous, again
if we are in the smooth scenario so there
is no intelligence explosion?
If we imagine that there are millions of machine
learning systems deployed all over the world,
many kinds of, you know, capabilities and
motivations etc. then it would be really surprising
if a lot of those were treacherous and we
didn’t notice that at all.
This is of course true, but the classic AI
risk arguments don’t really talk about the
smooth world.
They talk about the world where things are
much more “lumpy”, and in this case of
course we would notice the first treacherous
but then it would be too late.
Ben Garfinkel, in the smooth world, gives
an example, specification gaming - we talked
about this in the previous session - and he
says “this is an example of lying, and not
getting away with it”.
As I see it this argument doesn’t really
work because: If it was a problem we would
notice it, and we currently notice it, so
it’s not a problem.
I feel this is the way with the specification
gaming: We notice that there is specification
gaming.
So AIs obviously… if you believe the specification
gaming is an example of lying then AIs obviously
lie, right?
And then there is obviously a problem.
But as I said in the previous session I don’t
believe that specification gaming is actually
lying.
I believe that you… in order to make what
is actually a lie, you need to have a model
of what does the person that you are talking
to, communicating with, what does he believe.
And if you don’t have this kind of theory
of mind, then you’re not actually lying.
Howie Lempel again tries to put the discussion
back on track: “What if the AI doesn’t
start lying until it has a high confidence
it can take over?”
Ben Garfinkel answers: “Well, even if the
AI is ninety percent certain then if there
are millions of systems, then sometimes we
will catch them.”
And that’s of course true but it’s not
just lying, it’s that it has a high confidence
that it can take over, and of course if an
AI, in particular a superintelligence, has
a high confidence that it can take over, then
there is a very great risk that it will indeed
take over.
And then it will not take millions of AI systems
before they take over, it might only take
one.
I think, actually there is an argument here:
you could argue that if the first AI that
is, you know, a general AI or at least has
a theory of mind so it’s capable of thinking
about, you know, whether it should do a treacherous
turn, and believes it has an extremely low
probability of taking over the world.
Then I think if it’s a paperclip maximizer,
it would try to do that, even though it’s
very likely to get caught.
And I think something like that might be possible
for a honey pot or something like that, you
know, to catch these AIs.
I think there might be something there.
About lying: Children are slightly cognitively
impaired agents, I think that’s a great
way to put it, and children lie a lot, and
they often get caught.
And this seems to imply that you need to learn
how to lie, and you can’t be good at lying
until you have actually had experience lying.
The obvious objection to this line of thinking
is that we don’t really know -at all.
We don’t know if a superintelligence that
has never tried to lie will actually be good
at it.
I think I would expect it to be very good
at lying even though it has never done that
before.
There is some fictional evidence you could
put forward but I think the main point is
we just have absolutely no clue.
On the other hand you could say: how hard
is it to lie?
Because at time T we imagine a robot saying
“I am an A-maximizer”, you could say that
as a human utility maximizer, and then after
the AI has reported that to the human it thinks.
And it thinks, actually, hmm, it’s only
maximizing smiling faces or something like
that.
So how can it lie?
How can make the human believe that it is
still an A-maximizer, a human utility maximizer?
Well, actually, it’s really simple because
at timestamp t+1 it just repeats what it said
at time t.
So this is actually a really, really easy
way to lie if you’re just supposed to first
say one thing genuinely, and then later repeat
the same thing as a lie.
Also we imagine that once the AI is at the
level where it can improve itself it probably
cannot conceive about lying, but at the superintelligence
level, then it almost certainly can, and the
question is: Will anyone even think to ask
it in between these two points?
There is another kind of argument that comes
up here where Ben Garfinkel points out that
the treacherous turn doesn’t explain why
the AI would either have power or want power.
And this is of course a different part of
the argument.
The argument is commonly structured in several
parts and for each part it is true that it
doesn’t explain the others, so in that way
you could put a question mark on each sub-part.
But I believe this is a logical error: Each
subpart of the argument only explains what
it’s supposed to do, and the fact that it
doesn’t explain the other parts is irrelevant.
(Hold on, let me just see if I can get this
Windows taskbar away.
Yeah, I could, great.)
OK, so, we don’t really have very tight
and perfect mathematical arguments for AI
safety.
And if we ought to put a lot of resources
into managing AI risk then would really, really
like to have very, very good arguments for
both about whether there is a risk, and more
concretely what it would look like, and of
course also what can be done to mitigate it.
And that’s just for the classic AI risk
arguments.
Some of the new ones, like Wei Dais arguments
for instance, we haven’t anything that’s
anywhere near as formal.
To this I can only say that I agree, and I
believe that this is something that would
be really, really nice to have, something
I care a lot about.
Getting these arguments shaped up is, to use
the classical EA methodology, IMPORTANT and
NEGLECTED.
But unfortunately I believe that it is not
TRACTABLE.
I quite simply believe that the book Superintelligence
presents the arguments roughly as well as
they can be, and it’s just a general fact
about our epistemic state that predicting
the future is really, really hard.
So, assuming that Superintelligence does indeed
present the argument as well as possible,
then obviously it doesn’t convince everyone,
and I would then say that probably there are
no arguments that can resolve the debate as
it is right now, unfortunately.
We just don’t know.
So I think the best we can do is to examine
counter-arguments like I’m doing here with
Ben Garfinkel because I don’t think we are
going to ever be able to strongly resolve
this far in advance.
Ben Garfinkel has…
The arguments that Ben Garfinkel has presented
have influenced his advice to people from
Effective Altruism.
He believes that, if you are a machine learning
researcher and you’re really interested
in the long-term perspective, then you should
work on alignment.
If you’re working in governance and have
a strong interest in Computer Science and
Machine Learning, then it’s a really good
idea to work on AI governance.
If someone has many options, and alignment
or AI governance are not really the best ones
but there are others that are better, then
Ben Garfinkel is uncomfortable advising them
to work on AI safety.
So, this is really, really weak advice, and
I think most people would agree with this
because, you know, it’s equivalent to “Working
on what is your best fit”, but it’s really
strong words, right, if you’re a Machine
Learning researcher and really interested
in the long term perspective.
There is an excluded middle here, for instance
people who are closer to AI fields than other
fields, but not extremely so, like they established
researchers.
I think in this case we probably have three
arguments: We have the rigorous argument as
presented in the book Superintelligence, then
we have some other AI safety arguments that
haven’t really been formally written down,
and then we have the counter arguments which
are also informal.
Ben Garfinkel says that the informal arguments
for AI safety…
he’s not confident, you know, changing his
advice based on that, but his own informal
argument, he is, he does want to give advice
based on that.
I think that’s, you know, minimal hypocrisy
in the way that it’s very human to value
your own arguments that… if you believe
that informal arguments shouldn’t have much
weight, then when they are your own informal
arguments then you kind of tend to give them
more weight.
But it is not an obvious way to do because
there is indeed a reasonably well established
social phenomenon called the Information Cascade
where the classical example is there is a
jar here, and people take turns drawing a
ball, and they have to state if they believe
the jar is mostly red or mostly blue, and
you can see what the people before you have
chosen, and very often it makes sense if all
the other say it’s red and you pull out
a blue one, they you might want to say “actually
I believe they are all red or mostly red”.
Of course not all, but mostly red, because
you have the evidence from all the others.
This could be something that is happening
with AI safety in that people believe that
everybody else has looked deep into the arguments
and so derfer to them.
Or maybe their criticism has been answered
by a Google Doc somewhere.
I believe that this is rather unlikely in
that people are not saying anything like that,
people are instead strongly pointing towards
the book Superintelligence as the canonical
representation.
So people are not hiding Google Docs or anything,
but explicitly pointing towards the book Superintelligence
and saying “this is what I’m basing it
on” then, you know, you can actually read
it for yourself - there is no particular private
information.
Then I have a…
I think I will skip through this but point
out that this is something you could actually
investigate empirically by, you know, asking
people why they believe what they believe.
Now, there are a number of concrete counter
arguments that’s been made against the book
Superintelligence including Robin Hanson who
is mostly arguing using the outside view,
Katja Grace who is also mostly arguing about
the outside view, Brian Tomasik who’s using
the inside view, and Paul Christiano who is
both using the inside and the outside view.
I have in this reading group made similar
presentations where I’ve engaged with all
of these, and a number of them do have good
points but some of the major parts, unfortunately,
do have major flaws.
I think it’s a sad thing that when these
four people and a number of other people make
criticism, then there is no formal structure
for capturing this, and when I make any kind
reply to this, again there is no formal structure
so it’s kind of just ends up in nothing.I
think that’s really sad and it would be
great if we had some formal structure.
And further, these kind of arguments, when
they are… these four people have made decently
public and reasonably polished arguments,
but “a lot of the arguments for or against
are in private Google docs” is asserted
by Ben Garfinkel, I actually don’t know
whether that is true.
But it needs to really move to something more
formal because this really, really helps arguments
- this makes arguments a lot stronger.
And I agree with this and I think the big
problem for this, when I have to try that,
for instance, I try to do an adversarial collaboration
on SlateStarCodex on the intelligence explosion,
and the problem is that one side has been
written down very, very exhaustively, then
the playing field is extremely unlevel because
the counter arguments… one needs to reply
to those, and that’s a lot of work needed
to be done by one side and not the other,
and that makes it really hard to get traction
- at least in my experience.
So I don’t really know how to make a formal
structure with this kind of feedback.
Ben Garfinkel has quite a nice analogy: Zeno’s
Paradox, the paradox with Achilleus travelling
first half the distance, and then one fourth
of the distance, one eighth of the distance,
and to some other point without actually reaching
it.
And there wasn’t really a clear explanation
of what… it was always intuitively very
wrong, right?
You would obviously get to this point at some
time, but what is the thing that is wrong
with Zeno’s Paradox?
This required quite a bit of mathematics.
Ben Garfinkel says that it wasn’t until
1900, I believe, that in 1821 Cauchy published
Cours d’Analyse, which I believe introduced
the epsilon-delta definition that I would
use to explain Zeno’s paradox right now.
But I think the point stands, right?
This is something that was explained as a
paradox for two thousand years where people
really couldn’t figure out what was the
exact problem with this.
And it’s really hard to articulate what
are the issues with arguments that use fuzzy
concepts, like movement and distance for instance,
before you had the mathematics for it.
And I think this is why we should probably
expect something that is like the book Superintelligence
will have some mostly poor criticism.
But even though a lot of the criticism is
poor, the underlying intuitions that people
have that it is wrong is something that needs
to be respected.
And I believe this is true.
And you could actually say there’s something
like an offense-defense balance here, in that
writing a book about a bad prediction might
indeed be easier than doing the work of criticising
it, and pointing out precisely where it’s
wrong.
The final point is: AI safety as a field,
how much money does this deserve?
Ben Garfinkel has a cute example of the film
The Boss Baby, which has it’s production
and marketing cost substantially more than
it has ever been spent on AI safety.
And when he thinks about how much money should
be spent, he compares it to things like, you
know, pandemic preparedness, nuclear war or
climate change… these kinds of things.
And he believes that the AI safety level is
way too low and it’s hard to argue against
a five time increase, five times more than
The Boss Baby.
So I looked up the budget, 125 million dollars,
so five times that would be 625 million.
So I tried to calculate if it was reasonable,
both with how many people would that fund
for how many years, and as a percentage of
both AI and the world GDP, and I kind of got
stuck with this, I’m not an expert in this.
So I couldn’t figure out whether that was
reasonable, or how you could argue for 5,
or 25, or whatever.
So I tried to look for a formal calculation,
I couldn’t find that either.
I did look into the Oxford Prioritization
Project which does something like this, and
I think that’s probably a requirement to
have something like their Guesstimate Model
in order to say whether this is actually true
and whether this corresponds to the other
probabilities that Ben Garfinkel has given
for AI risk.
That is all for today.
Thank you and see you.


260. How Might We Align Transformative AI If Its Developed Very Soon 3

hello and welcome to session 260 in the
aisafety.com reading group tonight we'll
be discussing the last part of how might
we align transformative AI if it's
developed very soon by Holden kanovsky
uh Holden kanovsky is still the co of
open field and co-founder of givewell
and we are discussing the last third of
this uh post
I will start with a brief recap of the
entire argument as well as recapping My
overall thoughts on it so the idea in
this nearcasting scenario is for the
hypothetical AI company magma to
try to
use the uh Advantage the lead they have
on transformative AI to somehow ensure
that we will not get any air catastrophe
and the the five goals they uh that are
suggested by Holden kanovsky are
hardening goals which I was very
pessimistic about alignment applications
which I think is the best coordination
applications which would be nice but
seem really hard enforcement
applications which I don't have much
hope for and advisor application that
seems like just hoping for the best
so the way we saw uh that magma would
make it uh safe to pursue these
objectives is by having some AI that has
some high level meter behaviors one of
them is value alignment I was quite
negative about that Honesty which is
defined in its kind of low level that I
think is okay but not super important
courage ability which I think is
absolutely key legibility which I also
feel is like nice to have but I noticed
that there is a lot of things missing
from this list uh like I don't know
interpretability and things like that
um so how do we get this intended
Behavior well uh there are some facets
of alignment that are described in uh we
saw last time which I was uh quite
optimistic about and then I thought this
was the best part of the article where
uh accurate reinforcement out of the
distribution robustness preventing
exploits and testing and threat
assessment seem like important steps
towards alignment
and in order to help with this we will
now look at the key tools uh decoding
manipulating internal States limited AI
systems AI checks and balances and
keeping supervision competitive
so of these tools actually I think that
should be perhaps like a
like decoding and manipulating internal
States actually turns out to help with
AI checks and balances so this graph
should actually be somewhat more uh
complex so this is quite a
simplification but I'm getting a bit
ahead of myself
let's talk about the key tools that uh
that magma will use to ensure alignment
the first is decoding and manipulating
internal States
um and we're assuming this is a uh a mix
of deep learning and reinforcement
learning in some very unspecified way
um and how can we manipulate of course
manipulating deep learning
um is a lot harder than just looking at
uh at the system from the outside uh in
a black box way
um and right now our ability to really
figure out what's going on in under the
hood what is gbt3 really thinking is
very limited
um I
I share I think I share uh Holden
kanovsky's negative view on this but I
would uh note that in interpretability
which seems to be a prerequisite for
this has a lot of uncertainty about how
how much progress are we making and a
number of people are saying that we are
in fact making good progress
um but a lot of people are also saying
that we're not making good progress it's
really hard to say
so why does decoding and manipulating
internal States help well it would allow
us if we can decode and see whether that
is just
deceptive then there would of course be
a huge help we would be able to train
based on like how deceptive how honest
how goal directed how identic uh these
kind of things would be very helpful to
be able to train on
uh we could reverse engineer the the
model into human readable code
and then perhaps edit that and finally
um getting a better understanding of
what's going on inside is probably going
to be really helpful uh later in the
process when we're going to talk about
checks AI checks and balances
um my view on this is that it looks like
currently we are really far from being
able to do these things
um that's at least my uh
view so but um is not much more
optimistic than me but he does that it
is imaginable that this could work
when I try to think about what it would
actually mean that qt3 is reverse
engineered into human readable code to
me that sounds really Fantastical like
okay I can imagine a lot of things but I
would give odds a thousand to one
against uh someone having being able to
look Activity 3 and edit and change gbt3
into a human readable human
understandable uh program where we can
point to line 420 and say that's why the
consequentialist reasoning is I I really
really really have a hard time seeing
that being possible
the next key tool is limited AI systems
and this is something that doesn't feed
into like the only facet this feeds into
is courage ability but credibility is
the facet that I think is most important
so let's have a look at that
a part of that is to just have a very
narrow task another example of what uh
uh
way to make a limited AI system is to
make it myopic we could give it limited
situational awareness I think that is in
fact the most promising thing and where
I think more efforts should be spent
right now
thus the idea of process based
optimization where you're not um
rewarding or punishing the uh
um the reinforcement learning for the
the objectives when it's uh for the
outcomes but for the processes
and finally art has described a uh
another suggestion which is uh produced
plans that look good to humans
um and that is according to Holden
kanovski also a possibility and I
suppose it is a possibility but um like
I uh skimmed or a description of this
and also of course I read what Holden
kanovsky said and like there is a really
really obvious problems with with
producing plants that look good to
humans in that of course that would also
look good to the supervisor so and when
I read their description of this
suggestion none of them seem to like
take that into account or even mention
it at all that makes me somewhat scared
because you know just producing plan
that looks looks good to humans is
something that will obviously strongly
fail in the limit of very powerful AI
systems
has another really interesting point and
that is that right now we uh we don't
have a mature research process it's
sometimes said that we are
pre-paradigmatic
um and once we uh are no longer prepared
it magic then these kind of limited AI
systems could potentially contribute
quite a bit to alignment
um and I think that is
um uh quite plausible and I think it's
an interesting point I think we're kind
of like assuming uh the conclusion in
that um
uh uh like
um magma builds this AI system and they
start working on how to align it and
they accidentally hit the right Paradigm
and then they use uh the AI to improve
according to this Paradigm
um and uh I can I can see the AI
improving according to the to a
particular Paradigm but I
um like how precisely would they find
that I think there is like a missing
step somewhere in in this strategy
then we come to checks and balances
which is the idea that we have a number
of uh not just one AI that is being
improved but a number of AIS that are uh
trying to criticize each other using
debate and things like that and
hopefully they will be um not um we can
get them uh
varied enough that they will be able to
find hopefully all the problems or
enough problems that
um that we don't have a catastrophe and
for this uh holding suggestion suggests
different capabilities and incentives
for different AIS and this is something
that could potentially help in all of
the uh
um requirements for alignment
and something that could uh potentially
also feed in directly and amplify the
human made alignment research that has
come before
um these AIS have another advantage in
that they don't necessarily need to be a
lot smarter than humans they but when
they're doing checks and balances
against each other then because they
think in the same way that they might
see the same kind of exploits and that
could potentially make supervision
easier
Holden says that this should be a
central element of Magma's approach and
uh like it's a I agree that some version
of this is probably
um close to our best shot
there are however uh false suitable
problems with this kind of uh AI safety
versus type 8 and one of them is that it
might be very difficult for humans to
adjudicate this picture is due to Steve
bernus
showing a person the human supervisor
water skiing in between a rope that is
uh pointed towards uh there are like two
missiles pointing towards each other and
moving uh slowly forward and this seems
like an extremely extremely unstable
situation
um and I think
I share more of Steve's uh intuition on
this uh if these two are both uh
deceptive and are both very powerful and
like what happens if they both if they
suddenly start to accuse each other
sky is a bit optimistic about this
saying okay if it turns out to be hard
this problem then um this is evidence
that the alignment problem is hot and
that may make it easier to communicate
to other AI companies that they should
be wary about solving the alignment
problem and that this would be a real uh
challenge
I'm not entirely sure that the other
companies will see it that way because
magma is they might as well say that
magma is just incompetent at doing
something hard that is outside of the
thing that they are basically
um like they're world leaders in
um in human feedback on diverse tasks
building AIS based on this Paradigm and
now they're trying to do something
completely different and they're failing
at it well that may not be because it's
very hard it may just be because they're
not competent and Facebook AI or some
other
um AI lab may say oh they will surely be
able to handle this and just push ahead
a key problem for uh AI checks and
balances is what Holden kanovski calls
Advanced collusion the idea that so the
AI that is
um being monitored and the AI that is
supervising will collude somehow against
the human supervisors
um
the hope is by having many different at
least some of them will not perhaps be
able to uh see this uh this deception
where some of them will not be able to
infer the in the deception some of them
will not see that there is some kind of
payback where they if they collaborate
against the human they will get some of
their value function satisfied and some
of them of course may not join the
collusion and may read it out to humans
that would be really nice
um
they will probably be helped a lot with
covered communication channels so it's
important to be on the lookout for those
um
we saw previously that
um
decoding internal States would be
helpful for this and it probably is
really helpful but we should also have
some AIS that don't have this kind of
access because then they can't see uh
credible commitments
but a castle trade is still something
that is possible
and this is something that gets worse
and worse the more advanced the AI we
have I would imagine that once you have
an IQ of I don't know 150 you can invent
a causal trade but because it's like
Humanity's current work on a causal
trade is published the ass know it and
for this reason you don't really need
that high of an IQ to understand the a
causal trait when there's literally a
PayPal you can look up
um so definitely at the human level we
should expect this this kind of problem
to appear if not earlier
and of course holding also
um concludes that this is in fact a
major open problem
um so what kind of values make collusion
more likely well if the AIS have like
literally the same or very close to the
same utility function then that is
likely to make it more like uh then they
would obviously collude against us
Holden takes as a an example of a really
bad goal that AIS as a group has a high
reward
um and I agree it's a um a bad goal but
I also think it's the kind of goal that
we are unlikely to get by accident it
seems like just an obviously trivial bad
um
The Hope from
uh Jose is that this is something that
will only appear relatively late in the
game
but one of the problems is that if we
iteratively have uh this process where
we try to make a generation of AI as
safe as possible using the previous
generation and then once we have made
that safe then we go to the next then if
we fail in one round of this the then
that is a terminal failure
also have some thoughts about intexical
goals like if the AI has goals that
refer to itself like it should have
um like power over something or it
should have very high reward or it
should have uh this kind of different
things believes indexical goals are very
bad
um I agree they can be bad but I can
also
um like we probably would not uh make a
lot of indexical goals for for the AI I
imagine that we are more interested in
outcomes that don't directly refer to
the AI it could still happen through
meter optimizers and of course it's hard
to see what we like to be done but there
are also advantages to avoiding
indexical goals and that is that the
ai's own alignment problem becomes a lot
easier because if the AI just cares
about maximizing paper clips then it's
very easy to build a successor system
and then tell it to also maximize paper
clips but if you have any technical goal
then the new generation of AI that you
are building May repel against you and
then you won't satisfy your own utility
function and that means that a um
an AI within technical goal could in
theory be safer than one without
Jose describes a number of ways around
these problems one of them we talked
before about
varying the capabilities and incentives
of different AIS and playing them
against each other in this checks and
balances you could have a different kind
of assignments for them like argue that
this AI is a deceptive regardless of
whether it actually is would be an
example of a different assignment
um
that's one suggestion to give large
losses for false alarms and large losses
for failing to notice known problems as
far as I can tell this situation would
do absolutely nothing like if you take a
an AI that is trained in some way and
then you instead of making the losses
small you make them large like you
double all the losses uh I imagine that
this would in fact have no uh influence
on the behavior of the AI if you just
double all rewards and all losses
um but I'm uh I could I could it depends
on precisely how it's structured but by
default I would expect this to not do
anything
another way you could vary the
competence of them would be to vary the
um compute uh
you imagine that um figuring out things
like Advanced collusion or in the causal
trade if you have very little compute to
spare then maybe you can't figure that
out
um
we talked also previously about access
to internal State as one thing that
could be varied in this uh checks and
balances scheme you could just do brain
surgery on the AI and just pull out some
of the facts if you can find them you
could just ablate some part of the
architecture or whatever
um if you can vary the situational
awareness that may be helpful
um my estimation is just if you can get
that as low as possible that is always
good
you can imagine more limits or uh like
varying things that you're not really
sure what does like different in
initialization
um and finally uh if you have some of
the earlier Generations that you are
quite sure are aligned that could be
helpful as well
perhaps betrays some intuitions that are
very different from mine by saying that
they may become uh confident that it is
vanishingly unlikely that a an AI in a
given generation is misaligned
um and I think vanishingly unlikely is
totally the wrong way of looking at I
don't think we can be anything like that
but I would in fact be interested in
having some idea about the extent to
which a partial alignment failures in
one generation like accumulate to the
next
I could see arguments foreign against
this and I think that maybe a key
empirical question
now we come to the question of
um uh Precision like this is a like a
lot of you have probably seen an image
corresponding to this like this just
explain like what is it a false positive
what is a false negative
um and um
Halton argues that it is important with
this uh checks and balances that you
have few false positives few times where
the AI says hey this this other AI is
deceptive and then it's actually not
um is that more uh important than um
than not having any uh false negatives
well I think yeah I would actually argue
that avoiding false negatives is more
important than avoiding false positives
in that if you have this Ai and you
believe it is not deceptive but it
actually is deceptive that seems a lot
worse
um but in general of course there needs
to be some kind of balancing between
these these two kinds of errors
we may be able to use AIS with low
Precision if we have good adjudication
like if there is a good strong way of
determining the truth of the matter if
one AI accuses another AI of being
deceptive
um then that would be helpful as well if
nothing else even if we can't really
understand and adjudicate then we could
just count the number of warnings that
may be helpful
I mean yeah it may be a little helpful
right
um
and if we are doing uh the thing where
we try to
um uh lower the amount of free energy to
harden the society in general then uh
AIS with uh low Precision could perhaps
also be valuable for this
so how effective will checks and
balances be well uh Holland kanovski is
optimistic about this and argues that
even if the AIS tend to collude
sometimes then this will have benefits
because they will still give some level
of uh protection in that they will like
if they explain some security
vulnerabilities and then that will be
helpful
um
and we could potentially make collusion
quite hard
um like uh I I don't know precisely uh
how hard Holland kanovski estimates that
it's going to be but
um variation is certainly something that
will also make
um
give more false alarms and make uh
adjudication a lot harder because we are
probably going to be
um like if there are different
architectures or something like that
then we need to understand more
architectures uh as judges
um
that's how the incentive scales is also
an interesting case when you imagine
that if the uh collusion becomes
slightly less probable of succeeding
then that strongly reduces the
incentives
um I think that's interesting and I
think it's possible to do some work on
trying to formalize this
um which I think would be really helpful
um and um
has this came reliably calculating the
probability that AIS are caught in bad
behavior goes a long way
um I think it's a strange claim I think
definitely reliably carefully in the the
probability that an AI will do a
treacherous turn is something we're
never going to have like reliably
calculating that we are really um
grasping for straws rather
I think the final key tool is to make
supervision competitive and this is
supervision competitive with the
supervised so this previous generation
of AIS that are supervising the next
generation of AIS
um and this kind of competitiveness
should be distinguished from the
competitiveness between magma and and
the world
um and basically how do we ensure that
supervision is competitive
um how can I ask you suggest that we
move cautiously and iteratively and I
think that is
true but I also feel like this is
described like as a key tool and there
is basically this be cautious and
iterative and that's kind of
underwhelming to me
um I think I would have liked to have
some more ideas
um I recognize it may be hard at this
point to write a lot but I think it's
it's important and it feels really thin
to me
the final section of the post is called
high level factors in success and
failure
um I uh when I read that title I didn't
quite understand what it was about I
think what it's basically about what
kind of future evidence we could see
that would update us to be more positive
or more negative
so one of them is whether it turns out
to be natural or not that the AIS will
play the training game
uh playing the training game is not
precisely the same as being deceptive
it's described as trying to appear as
attractive to researchers as possible
while disregarding their intent
um and I tried to uh look into this I
think this kind of new definition is
kind of uh strange it doesn't really cut
reality at the joints because you can't
appear attractive to someone without
considering their uh their intent
um so um I would just say whether
deceptiveness is naturally all that like
here
we don't know
um this is a difficult question
um in theory we could see ai's
internalizing at an early stage that
hacks that give a little legitimate
rewards are not desirable that would be
really nice hoton is not positive about
this you think it's unlikely but if he
learned this then that would update him
to be more positive about Magma's
changes chances
a bad Factor would be that if we learned
that AI systems rapidly become extremely
powerful relative to supervisors
so here there are two things there are
against the the capabilities of the
supervisors which in the first
Generations are only humans and later
will be humans helped by the previous
generation narrow AIS something like
this
um but the other thing that the AI
systems could become powerful towards
are human defenses like how hard is it
to convince people how hard is it to
actually take over the world
um
if it turns out that systems become
rapidly become extremely powerful that
is very bad news
and I think uh
we should skip the words extremely uh
and rapidly in this claim because
um sure if you put enough qualifiers
then obviously but I think if it turns
out that AI systems at some point even
at a slow speed maybe eventually become
powerful relative to supervisors like
even slightly more powerful then that
would also be uh very very bad to learn
that
so um I think this should be split into
two uh Dimensions
um like there is against the
capabilities of Supervisors to figure
out whether the AI is deceptive I think
right now ai systems are quite capable
of doing that for the simple reasons
that the supervisors are really really
poor like we really don't have any good
way to look at dvd3 and say is it
deceptive or not
um but against human defenses it seems
clear that AI systems are not anywhere
near being powerful enough to just take
over the world
so what would this look like in practice
it would look like AI has been able to
deceive and manipulate humans
distinguish whether they are in training
our deployment find some kind of
security exploit and generate security
suggestions with hidden back doors
um one thing I would note here is that
um
it isn't actually required that the AI
generates the security
suggestion with a hidden pector that it
can exploit itself if it expects it will
be replaced with a smarter AI then it's
good enough that it says that it notice
that there is some subtle bug here that
probably a smarter AI would be able to
use a specto
another thing the AI may be able to do
is to disguise their internal States
collude against each together with each
other and just make adjudication too
hard for humans
so we you notice here that this part
about why powerful AI is bad didn't
really talk about alignment
um but Holland kanovsky
says that we this is really bad news if
AI become powerful because during
training there's going to be a lot of
different opportunities to defect and to
take over the world or do different
things and like if we strive for a big
amount of variation then almost
certainly some of the AIS are going to
be unaligned
um and this seems to be a Crux for many
people including the most optimistic and
pessimistic and I believe that yeah this
could be close to one of the cruxes
um in particular effect behind uh why
some people are calling for a pivotal
act because we are probably going to
need something that is really strong to
prevent other uh all other AIS from
being able to do uh bad things if there
are if it turns out that AI is rapidly
become extremely powerful compared to
humans
okay the optimists at this point would
probably say that okay if we make
multiple Generations then why is it
exactly that we need the Next Generation
to be a lot more powerful than the
previous couldn't we just do baby steps
um even if it turns out that
increasingly intelligence of these
systems is very easy surely we could
just lower the amount of compute and get
something that is just slightly smarter
so we don't get something where the next
generation is a lot smaller than the
previous
um Holden kanovski says seems to be
taken by this argument I believe he
thinks it's a really good argument
um and says it's unclear about what the
counter arguments to that would be so
the first counter argument I would give
is that the standard story of uh AI
takeover includes the AI hiding some of
its capabilities and that would prevent
uh humans from very gradually turning up
the the heat instead of uh
um
uh instead of just giving it as much
compute as possible
the second is that I feel capability is
multi-dimensional it's possible to have
one level of capability that is uh
against human defenses like manipulating
humans breaking out of the box hacking
taking over the world and uh alignment
the ability to align other AIS and there
is no particular reason to expect these
two to go precisely in the same level I
believe indeed that the alignment
problem is very very hard and that would
also make it very hard for the AI
there are a few more things we could
learn we could learn that adversarial
training works really really well
um that would be a good thing Hogan
kanovsky is not optimistic about that he
believes that will require that
effectively every input uh is identified
that would cause unintended behavior and
that seems like a really really big ask
but I don't think we actually need all
that much I think the key thing we need
for adversarial training to be super
effective is a really strong principled
way to make the AI think that it is
still in test when it is deployed I
think that's actually all we need but of
course I also believe that this is
really hard
another thing would be if it turns out
that decoding and manipulating internal
state is really effective in some way
um it may be that we can solve the
alignment problem or Global coordination
using only rather limited AI systems
that would also be great
um another thing we could learn as we
try to use AI systems to align other AI
systems is that decisive alignment is
just hard and that would of course be
really bad
another key question
that determines whether we will have
success of failure is how questions will
magma and others be
um because in cautious behavior is just
really attractive there's an easy way
around all these kind of problematic
symptoms just training the way basically
um and
um
doing all putting all this delay and
work on alignment is not perhaps easy to
justify to shareholders Etc
um and we're probably gonna need a
substantial amount of Serial time
because human needs to be involved in
many things such as resolving security
flaws and adjudicating and even if the
AIS are more powerful we need some
serial time
another thing that could happen is that
we get a sub-existential catastrophe
that would make could make magma and
others be more risk adverse I think that
is true but I also think that one level
below a catastrophe that is one level
below helps but two levels below do not
help so if we get a sub sub existential
catastrophe that doesn't really help
like if we get a lot of self-driving
cars that crash into other self-driving
cars that would make a lot of people say
yeah the problem is that the AI is not
competent enough so we need more
competent Ai and so that doesn't really
help us at all
So based on all this this entire uh
uh uh work would civilization survive in
this nearcasting scenario where magma
realizes that they have six months to
two years to uh try to um uh uh prevent
an AI catastrophe
um
doesn't really give a strong suggestion
to this
um he can't resolve this he believes
that it is uncertain it's not certain
that it's impossible and it's not
certain that it's impossible somewhere
in between seems reasonable
um there are in this situation a small
number of AI labs and
um but one of them is probably in
cautious I think it's it's too big of an
assumption that they will all be
cautious
another Advantage is that there are
several possible paths to alignment
um but unfortunately they all look kind
of hard
so all canovski ends up with saying like
meh Maybe
um to me I think in fact that
um the actions that magma take are not
that relevant because the suggestions
given here are not pivotal acts in
general and they are unlikely to really
flip the game board in any meaningful
way so I think the key question here for
would civilization survive is how hard
is the alignment problem how natural is
it for AI to be deceptive this kind of
thing and how well magma does like
McMahon can't really do that much with
this uh the small lead
um so whether they're really competent
or or not that doesn't change the
probability of AI Doom very much
that is all for today thank you and see
you next week


3 principles for creating safer AI | Stuart Russell

This is Lee Sedol.
Lee Sedol is one of the world's
greatest Go players,
and he's having what my friends
in Silicon Valley call
a "Holy Cow" moment --
(Laughter)
a moment where we realize
that AI is actually progressing
a lot faster than we expected.
So humans have lost on the Go board.
What about the real world?
Well, the real world is much bigger,
much more complicated than the Go board.
It's a lot less visible,
but it's still a decision problem.
And if we think about some
of the technologies
that are coming down the pike ...
Noriko [Arai] mentioned that reading
is not yet happening in machines,
at least with understanding.
But that will happen,
and when that happens,
very soon afterwards,
machines will have read everything
that the human race has ever written.
And that will enable machines,
along with the ability to look
further ahead than humans can,
as we've already seen in Go,
if they also have access
to more information,
they'll be able to make better decisions
in the real world than we can.
So is that a good thing?
Well, I hope so.
Our entire civilization,
everything that we value,
is based on our intelligence.
And if we had access
to a lot more intelligence,
then there's really no limit
to what the human race can do.
And I think this could be,
as some people have described it,
the biggest event in human history.
So why are people saying things like this,
that AI might spell the end
of the human race?
Is this a new thing?
Is it just Elon Musk and Bill Gates
and Stephen Hawking?
Actually, no. This idea
has been around for a while.
Here's a quotation:
"Even if we could keep the machines
in a subservient position,
for instance, by turning off the power
at strategic moments" --
and I'll come back to that
"turning off the power" idea later on --
"we should, as a species,
feel greatly humbled."
So who said this?
This is Alan Turing in 1951.
Alan Turing, as you know,
is the father of computer science
and in many ways,
the father of AI as well.
So if we think about this problem,
the problem of creating something
more intelligent than your own species,
we might call this "the gorilla problem,"
because gorillas' ancestors did this
a few million years ago,
and now we can ask the gorillas:
Was this a good idea?
So here they are having a meeting
to discuss whether it was a good idea,
and after a little while,
they conclude, no,
this was a terrible idea.
Our species is in dire straits.
In fact, you can see the existential
sadness in their eyes.
(Laughter)
So this queasy feeling that making
something smarter than your own species
is maybe not a good idea --
what can we do about that?
Well, really nothing,
except stop doing AI,
and because of all
the benefits that I mentioned
and because I'm an AI researcher,
I'm not having that.
I actually want to be able
to keep doing AI.
So we actually need to nail down
the problem a bit more.
What exactly is the problem?
Why is better AI possibly a catastrophe?
So here's another quotation:
"We had better be quite sure
that the purpose put into the machine
is the purpose which we really desire."
This was said by Norbert Wiener in 1960,
shortly after he watched
one of the very early learning systems
learn to play checkers
better than its creator.
But this could equally have been said
by King Midas.
King Midas said, "I want everything
I touch to turn to gold,"
and he got exactly what he asked for.
That was the purpose
that he put into the machine,
so to speak,
and then his food and his drink
and his relatives turned to gold
and he died in misery and starvation.
So we'll call this
"the King Midas problem"
of stating an objective
which is not, in fact,
truly aligned with what we want.
In modern terms, we call this
"the value alignment problem."
Putting in the wrong objective
is not the only part of the problem.
There's another part.
If you put an objective into a machine,
even something as simple as,
"Fetch the coffee,"
the machine says to itself,
"Well, how might I fail
to fetch the coffee?
Someone might switch me off.
OK, I have to take steps to prevent that.
I will disable my 'off' switch.
I will do anything to defend myself
against interference
with this objective
that I have been given."
So this single-minded pursuit
in a very defensive mode
of an objective that is, in fact,
not aligned with the true objectives
of the human race --
that's the problem that we face.
And in fact, that's the high-value
takeaway from this talk.
If you want to remember one thing,
it's that you can't fetch
the coffee if you're dead.
(Laughter)
It's very simple. Just remember that.
Repeat it to yourself three times a day.
(Laughter)
And in fact, this is exactly the plot
of "2001: [A Space Odyssey]"
HAL has an objective, a mission,
which is not aligned
with the objectives of the humans,
and that leads to this conflict.
Now fortunately, HAL
is not superintelligent.
He's pretty smart,
but eventually Dave outwits him
and manages to switch him off.
But we might not be so lucky.
So what are we going to do?
I'm trying to redefine AI
to get away from this classical notion
of machines that intelligently
pursue objectives.
There are three principles involved.
The first one is a principle
of altruism, if you like,
that the robot's only objective
is to maximize the realization
of human objectives,
of human values.
And by values here I don't mean
touchy-feely, goody-goody values.
I just mean whatever it is
that the human would prefer
their life to be like.
And so this actually violates Asimov's law
that the robot has to protect
its own existence.
It has no interest in preserving
its existence whatsoever.
The second law is a law
of humility, if you like.
And this turns out to be really
important to make robots safe.
It says that the robot does not know
what those human values are,
so it has to maximize them,
but it doesn't know what they are.
And that avoids this problem
of single-minded pursuit
of an objective.
This uncertainty turns out to be crucial.
Now, in order to be useful to us,
it has to have some idea of what we want.
It obtains that information primarily
by observation of human choices,
so our own choices reveal information
about what it is that we prefer
our lives to be like.
So those are the three principles.
Let's see how that applies
to this question of:
"Can you switch the machine off?"
as Turing suggested.
So here's a PR2 robot.
This is one that we have in our lab,
and it has a big red "off" switch
right on the back.
The question is: Is it
going to let you switch it off?
If we do it the classical way,
we give it the objective of, "Fetch
the coffee, I must fetch the coffee,
I can't fetch the coffee if I'm dead,"
so obviously the PR2
has been listening to my talk,
and so it says, therefore,
"I must disable my 'off' switch,
and probably taser all the other
people in Starbucks
who might interfere with me."
(Laughter)
So this seems to be inevitable, right?
This kind of failure mode
seems to be inevitable,
and it follows from having
a concrete, definite objective.
So what happens if the machine
is uncertain about the objective?
Well, it reasons in a different way.
It says, "OK, the human
might switch me off,
but only if I'm doing something wrong.
Well, I don't really know what wrong is,
but I know that I don't want to do it."
So that's the first and second
principles right there.
"So I should let the human switch me off."
And in fact you can calculate
the incentive that the robot has
to allow the human to switch it off,
and it's directly tied to the degree
of uncertainty about
the underlying objective.
And then when the machine is switched off,
that third principle comes into play.
It learns something about the objectives
it should be pursuing,
because it learns that
what it did wasn't right.
In fact, we can, with suitable use
of Greek symbols,
as mathematicians usually do,
we can actually prove a theorem
that says that such a robot
is provably beneficial to the human.
You are provably better off
with a machine that's designed in this way
than without it.
So this is a very simple example,
but this is the first step
in what we're trying to do
with human-compatible AI.
Now, this third principle,
I think is the one that you're probably
scratching your head over.
You're probably thinking, "Well,
you know, I behave badly.
I don't want my robot to behave like me.
I sneak down in the middle of the night
and take stuff from the fridge.
I do this and that."
There's all kinds of things
you don't want the robot doing.
But in fact, it doesn't
quite work that way.
Just because you behave badly
doesn't mean the robot
is going to copy your behavior.
It's going to understand your motivations
and maybe help you resist them,
if appropriate.
But it's still difficult.
What we're trying to do, in fact,
is to allow machines to predict
for any person and for any possible life
that they could live,
and the lives of everybody else:
Which would they prefer?
And there are many, many
difficulties involved in doing this;
I don't expect that this
is going to get solved very quickly.
The real difficulties, in fact, are us.
As I have already mentioned,
we behave badly.
In fact, some of us are downright nasty.
Now the robot, as I said,
doesn't have to copy the behavior.
The robot does not have
any objective of its own.
It's purely altruistic.
And it's not designed just to satisfy
the desires of one person, the user,
but in fact it has to respect
the preferences of everybody.
So it can deal with a certain
amount of nastiness,
and it can even understand
that your nastiness, for example,
you may take bribes as a passport official
because you need to feed your family
and send your kids to school.
It can understand that;
it doesn't mean it's going to steal.
In fact, it'll just help you
send your kids to school.
We are also computationally limited.
Lee Sedol is a brilliant Go player,
but he still lost.
So if we look at his actions,
he took an action that lost the game.
That doesn't mean he wanted to lose.
So to understand his behavior,
we actually have to invert
through a model of human cognition
that includes our computational
limitations -- a very complicated model.
But it's still something
that we can work on understanding.
Probably the most difficult part,
from my point of view as an AI researcher,
is the fact that there are lots of us,
and so the machine has to somehow
trade off, weigh up the preferences
of many different people,
and there are different ways to do that.
Economists, sociologists,
moral philosophers have understood that,
and we are actively
looking for collaboration.
Let's have a look and see what happens
when you get that wrong.
So you can have
a conversation, for example,
with your intelligent personal assistant
that might be available
in a few years' time.
Think of a Siri on steroids.
So Siri says, "Your wife called
to remind you about dinner tonight."
And of course, you've forgotten.
"What? What dinner?
What are you talking about?"
"Uh, your 20th anniversary at 7pm."
"I can't do that. I'm meeting
with the secretary-general at 7:30.
How could this have happened?"
"Well, I did warn you, but you overrode
my recommendation."
"Well, what am I going to do?
I can't just tell him I'm too busy."
"Don't worry. I arranged
for his plane to be delayed."
(Laughter)
"Some kind of computer malfunction."
(Laughter)
"Really? You can do that?"
"He sends his profound apologies
and looks forward to meeting you
for lunch tomorrow."
(Laughter)
So the values here --
there's a slight mistake going on.
This is clearly following my wife's values
which is "Happy wife, happy life."
(Laughter)
It could go the other way.
You could come home
after a hard day's work,
and the computer says, "Long day?"
"Yes, I didn't even have time for lunch."
"You must be very hungry."
"Starving, yeah.
Could you make some dinner?"
"There's something I need to tell you."
(Laughter)
"There are humans in South Sudan
who are in more urgent need than you."
(Laughter)
"So I'm leaving. Make your own dinner."
(Laughter)
So we have to solve these problems,
and I'm looking forward
to working on them.
There are reasons for optimism.
One reason is,
there is a massive amount of data.
Because remember -- I said
they're going to read everything
the human race has ever written.
Most of what we write about
is human beings doing things
and other people getting upset about it.
So there's a massive amount
of data to learn from.
There's also a very
strong economic incentive
to get this right.
So imagine your domestic robot's at home.
You're late from work again
and the robot has to feed the kids,
and the kids are hungry
and there's nothing in the fridge.
And the robot sees the cat.
(Laughter)
And the robot hasn't quite learned
the human value function properly,
so it doesn't understand
the sentimental value of the cat outweighs
the nutritional value of the cat.
(Laughter)
So then what happens?
Well, it happens like this:
"Deranged robot cooks kitty
for family dinner."
That one incident would be the end
of the domestic robot industry.
So there's a huge incentive
to get this right
long before we reach
superintelligent machines.
So to summarize:
I'm actually trying to change
the definition of AI
so that we have provably
beneficial machines.
And the principles are:
machines that are altruistic,
that want to achieve only our objectives,
but that are uncertain
about what those objectives are,
and will watch all of us
to learn more about what it is
that we really want.
And hopefully in the process,
we will learn to be better people.
Thank you very much.
(Applause)
Chris Anderson: So interesting, Stuart.
We're going to stand here a bit
because I think they're setting up
for our next speaker.
A couple of questions.
So the idea of programming in ignorance
seems intuitively really powerful.
As you get to superintelligence,
what's going to stop a robot
reading literature and discovering
this idea that knowledge
is actually better than ignorance
and still just shifting its own goals
and rewriting that programming?
Stuart Russell: Yes, so we want
it to learn more, as I said,
about our objectives.
It'll only become more certain
as it becomes more correct,
so the evidence is there
and it's going to be designed
to interpret it correctly.
It will understand, for example,
that books are very biased
in the evidence they contain.
They only talk about kings and princes
and elite white male people doing stuff.
So it's a complicated problem,
but as it learns more about our objectives
it will become more and more useful to us.
CA: And you couldn't
just boil it down to one law,
you know, hardwired in:
"if any human ever tries to switch me off,
I comply. I comply."
SR: Absolutely not.
That would be a terrible idea.
So imagine that you have
a self-driving car
and you want to send your five-year-old
off to preschool.
Do you want your five-year-old
to be able to switch off the car
while it's driving along?
Probably not.
So it needs to understand how rational
and sensible the person is.
The more rational the person,
the more willing you are
to be switched off.
If the person is completely
random or even malicious,
then you're less willing
to be switched off.
CA: All right. Stuart, can I just say,
I really, really hope you
figure this out for us.
Thank you so much for that talk.
That was amazing.
SR: Thank you.
(Applause)