Holy Grail of AI (Artificial Intelligence) - Computerphile

Right. So, last time, which was quite a while ago, we were talking about intelligence in general
and the way that you can model intelligence as an optimization process
- This is the hill climbing algorithm.
- Yeah, that was an example we gave.
We were using evolution as an example of an optimizing algorithm, or an optimizing system anyway.
And then we were using that as a way of talking about other types of intelligence.
We talked about chess AI very briefly. That kind of thing.
So then the question is:  What's the difference between the type of AI that we have now--
the type of AI that might play chess, drive a car, or win jeopardy or whatever--
versus the ideas that we have of AI in the future? The kind of science fiction AI's that are what you would call true AI.
What is it that really makes the difference? Is it just a matter of power, or is there something else?
And one real distinguishing factor is generality. And what that means is how broad a set of domains
can it optimize in. So if you take a chess AI, it's very intelligent in the domain of chess, and it is absolutely
useless in almost any other domain. If you put a chess AI in a google self driving car not only can it not
drive the car it doesn't have the concept of what a car is. It doesn't have any of the necessary architecture
cognitive architecture to drive a car. And vice versa right? The google car can't play chess.
And it can't win at jeopardy.
Where as we have a working example of a general intelligence.
Which is human intelligence.
Right?
Human brains can do a lot of different things.
In a lot of different domains.
Gulp.
Including brand new domains.
The domains we didn't evolve for particularly.
So in fact chess ,right?
We invented chess, we invented driving.
And then we learned to become good at them.
So, a general intelligence is in a sense a different class of thing.
Because it's a single optimization system that's able to optimize in a very broad variety of different domains.
And if we could build an artificial general intelligence.
That's kind of the holy grail of AI research
That you have a single program or a single system that is able to solve any problem that we throw at it
or at least tackle any problem that we throw at it.
-Recently Pr Brailsford ... the idea of the Turing test
That strikes me from what you're saying is that's a very specific domain
pretending to be human talking.
-Yes, in a sense it's a very specific domain.
The Turing test is necessary but not sufficient test for general intelligence.
Hum.
You could, it depends how you format your test, right
because you could say well, if the AI has to pretend to be human, convincingly
Turing's original test was only in a brief conversation using on a text
But you could say, to convince me you're human : tell me what move I should make in this chess game.
To convince me you're human, tell me how I would respond in this driving situation
or what's the answer to this jeopardy question?
So you can in a Turing test deliberately test a wide variety of other domains.
But in general, conversation is one domain
Hum, yeah you could formulate a true Turing test, in that way
but it would get longer and be more, sort of, regressive.
One more way of thinking about general intelligence is a domain specific intelligence
but where the domain is the world or physical reality.
And if you can reliably optimize the world itself.
That is in a sense what general intelligence does.
-Is that like humans have been changing the world to meet their needs?
-Absolutely, so when you say changing the world
Obviously we've been changing the world on a very grand scale
but everything that humans do in the real world is in the sense changing the world to be better optimized to them,
right.
Like if I'm thirsty and there's a drink over there then picking it up and putting it to my lips and drinking.
I'm changing the world to improve my hydration levels which is something that I value
So I'm, sort of, optimizing
I am using my intelligence to optimize the world around me
in a very abstract sense. But also quite practically.
-But on bigger scale, as you say on a grander scale,
building it down and irrigating a field , putting a pipe to your house and then I'll need to have a tab.
-Yep. -It's doing the same thing but on a grander scale.
-Right, and there's no hard boundary between these two things.
It's the same basic mechanism at work.
The idea that you want things to be in somewhere different from where they are
So you use your intelligence to come up with a series of actions or a plan,
that you can implement, that will better satisfies your values.
And that's
that's what a true AI, a general AI would do as well.
So you can see the
the metaphore to optimization is still there, right.
You've got, this vast state space, which is all possible states of the world
Remember before, we were talking about dimensionality
and how it's kind of a problem if you have too many dimensions.
(So when we have a two-dimensional space...)
This is what kills basic implementation of general AI off the bat
because the world is so very very complicated.
It's an exceptionally high dimensional space.
With the "I'm drinking a drink" example, you've got the same thing again.
You've got a state of the world which is a place in this space
and you've got another state of the world which is the state in which I've just had a drink.
And one of them is higher in my utility function.
It's higher in my ordering, my preference ordering of the world states.
So I'm going to try and move, I'm going to try to shift the world
from places that are lower in my preference ordering  to places that are higher.
And that gives you a way to express
the making of plans and the implementing of actions and intelligent behavior in the real world
in mathematical terms.
It's not, you can't just implement it, because hum
because of this enormous dimensionality problem.
-All these dimensions, if you try to break force infinite dimensions, you're going to fall out very quickly.
-Yeah, yeah, immediately.
-Changing the world.
Right,
and if that sounds a little bit threatening
uh it is.
(laughs)
We'd like to thank audible.com for sponsoring this computerphile video
and if you like books go over to :
There's a chance to try out a book for free.
Now I spoke to Rob who's in this computerphile video and asked him what book he would recommand
and he says "Superintelligence" by Nick Bostrom is the one to check out.
Particularly on this subject of artificial intelligence
We've got more to come on that subject on computerphile as well, so visit :
audible.com/computerphile
Check out "Superintelligence" and thanks once again to audible.com for sponsoring this computerphile video.


Is AI Safety a Pascal's Mugging?

hi today we're going to talk about
Pascal's wager and Pascal's mugging this
is necessarily going to touch on
religious topics pretty heavily actually
and I'm just gonna say at the beginning
of the video that personally I don't
believe that any god or gods have ever
existed and I'm not going to pretend
otherwise in this video so be forewarned
if that's likely to bother you so as you
may know Pascal was a 17th century
philosopher who was interested in
amongst other things the question of the
existence of the Christian God various
philosophers at the time were arguing
that God didn't exist and there was a
lot of discussion going on about the
various kinds of evidence for and
against God in the world but there's
this thing that's quite common when
people think about religious questions
where it feels sort of unsatisfying to
talk about worldly evidence as if you
were considering some everyday question
there's a feeling that these
supernatural concepts are very grand and
mysterious they're special and so just
straightforwardly considering the
evidence for and against God is not the
right way to do things this is of course
encouraged by religious thinking the
idea that some hypotheses aren't subject
to the usual rules of evidence and logic
is pretty appealing if you want to
advocate for an idea that doesn't fare
very well by those standards
I suspect Pascal may have felt something
like that because his position was that
reason has nothing to say about the
question of whether or not God exists
it's sort of an unknowable thing and
instead he proposed that we should make
a wager we should think about it like
this there are two possibilities either
the Christian God exists or he doesn't
and reason gives us no way to choose
between those we have two options
available to us either we can live
according to God's laws and act as
though we believe or we can not do that
so we have a sort of payoff matrix here
with four sections if God exists and we
believe in him then we get infinite
reward in heaven if God exists and we
don't believe in him we get infinite
punishment in hell if God doesn't exist
and we believe in him then we pay some
costs you know there are some rules we
have to follow and so on and if he
doesn't exist and we don't believe in
him then maybe we get a few perks from
not believing like having a lion on
Sundays and being right Pascal's point
is that this payoff matrix is completely
dominated by the case in which God
exists because we're talking about
infinite rewards and infinite
punishments as opposed to the other case
with these very finite costs
benefits so regardless of the evidence
Pascal argues we should believe in God
or at least act like it because it's
just the sensible bet to make this is
really kind of a nice argument from
Pascal's perspective because it doesn't
need evidence at all no finite earthly
evidence can outweigh infinite
supernatural payoffs it feels like the
kind of clean abstract reasoning that
you're supposed to do when thinking
about the supernatural all of this hard
work looking at history and psychology
and science and trying to figure out
where the ideas of religion come from
and whether our world seems like the
kind of world with a God in it it's
long-winded confusing it's it's just
messy but here we just have a clean
argument that says we should believe in
God or at least act like it and that
seems very neat no evidence required so
consider now Pascal is walking down the
street and he's stopped by a shady
looking man who says give me a wallet I
would prefer not to do you even have a
weapon no UK laws are very strict about
that but I don't need one because I'm
God your God yep I'm God and
Christianity got a lot of things wrong
about me my forgiving nature my infinite
mercy and so on but the infinite torture
thing is legit and if you don't give me
your wallet right now I will torture you
for eternity in the afterlife now if
you're Pascal you're in kind of a
difficult situation because the fact
that it seems very unlikely that this
murder actually is God is not meant to
be part of your calculation your
argument is one a pure logic it works
independently of any evidence you didn't
need to look for evidence of the
Christian God and you don't need to look
for evidence that this mother is God
either so you kind of have to give him
your wallet and now you're really in
trouble because of course when this gets
out there's gonna be a line around the
block of lsaps deities asking for
handouts how are you going to deal with
this endless stream of fizzy gods well
one thing you can do is you can play the
muggers off against one another you can
bring in two of them and say listen you
say that you're going to torture me
forever if I don't give you my wallet
and you say the same thing I only have
one wallet so it looks like whatever I
do I'm going to be tortured forever by
somebody and if I'm going to be
infinitely tortured anyway well two
times infinity is still just infinity so
I may as well hang on to the wallet now
get the hell out of my house all right
next to no doubt these self-proclaimed
deities may try to argue that they have
some reason why they are in fact a real
deity
this other mugger is just a random guy
who's pretending but that's all worldly
evidence which you've decided isn't
required for your argument and the
muggers don't really want you to become
interested in evidence because well the
evidence points very strongly towards
none of them being real gods so this is
a better position to be in you're still
spending a lot of your time arguing with
charlatans but at least you still have
your wallet and you don't actually have
to pair them up against each other right
you can just make up a deity when
someone comes in pretending to be a god
you can say oh well there's this other
God who demands exactly the opposite
thing from you a goddess actually and
she's very powerful but she goes to a
different school
you would know her really yeah she lives
in Canada she's only present obviously
but she lives in Canada anyway she says
that I'm not to give you the wallet and
if I do then she'll torture me forever
in the afterlife I think yeah so you can
solve a lot of these problems by
inventing gods arbitrarily and of course
this applies just as well to the
original version of Pascal's wager
because although it's implied that this
payoff matrix has enumerated all of the
possibilities and in a sense it has the
Christian God either exists or it
doesn't nonetheless those may not be the
only things that effect the payoffs for
any given God you can take the God down
flip it and reverse it and say what
about ante God who wants me to do the
exact opposite and promises the exact
opposite consequences now you can see
that they cancel out somebody who's
arguing for the existence of the first
God might say okay but this anti God is
just made up which I mean yeah it is
it's true that the situation isn't
really symmetrical someone might think
God is more likely than anti God because
of evidence from the Bible and there's
no such thing as the anti Bible and so
on the point is though we're back to
talking about the evidence that's really
the problem I have with Pascal's wager
the way it uses infinite costs and
infinite benefits to completely override
our necessarily finite evidence but what
if the costs and benefits aren't
infinite just very very large that ends
up being a much more interesting
question on one end of the scale we can
easily name numbers so large that no
amount of evidence anyone could ever
actually gather in their lifetime could
have an impact on the conclusion we can
specify costs and benefits that are
technically finite but that still feel
very much like a Pascal's wager on the
other end of the scale if you come
across a bet that pays out 10
two one on an event with a probability
of one in a hundred that's a very good
bet to take someone could complain that
it's a Pascal's wager to bet on an
unlikely outcome just because the payoff
is so high but if you take enough bets
like that you're sure to become very
rich in the same way if there's a button
which has a one-in-a-million chance of
starting a global thermonuclear war it's
still worth expending significant
resources to stop that button being
pressed one in a million isn't much but
the cost of a nuclear war is really high
I don't think that's a Pascal's wager
either the difference seems to come in
somewhere in the gap between very small
probabilities of very large costs and
benefits and really extremely small
probabilities of near infinite costs and
benefits so why are we talking about
this what does this have to do with AI
safety well suppose somebody stops you
in the street and says hey if we ever
create powerful artificial general
intelligence then that will have a
tremendous impact in fact the future of
the whole of humanity hinges on it if we
get it right we could have human
flourishing for the rest of time if we
get it wrong we could have human
extinction or worse regardless of how
likely superhuman AGI is the potential
impact is so high that it makes AI
safety research tremendously important
so give me your wallet it's been claimed
by some that this is more or less what
AI safety as a field is doing this is
kind of an interesting point there's any
safety advocates are we victims of
Pascal's mugging or are we in fact
Pascal's miners ourselves well if people
were saying these AI risks may be
extremely unlikely but the consequences
of getting AI wrong are so huge that
it's worth spending a lot of resources
on regardless of the probabilities so we
don't even need to consider the evidence
well I would consider that to be a
Pascal's wager style bad argument but
what I actually hear is not that what I
hear is more like look we're not
completely sure about this it's quite
possible that we're wrong but
considering the enormity of what's at
stake it's definitely worth allocating
more resources to AI safety than we
currently are that sounds pretty similar
but that's mostly because natural
language is extremely vague when talking
about uncertainty there's an enormous
difference in the probabilities being
talked about in the same way if when you
talk to AI safety researchers they said
things like well I think the chance of
any of this ever being relevant are
really extremely tiny it seems more or
less impossible to me but I've decided
to work on it anyway
because the potential costs and benefits
are so unimaginably vast then yeah I'd
be a little concerned that they might be
victims of Pascal's mugging but when you
ask AI safety researchers they don't
think that the probability of their work
ever becoming relevant is very tiny
they don't necessarily think it's huge
either maybe not even more than 50% but
it's not so small that you have to rely
on the unimaginable vastness of the
consequences in order to make the
argument to borrow a metaphor from
Stuart Russell suppose you're part of a
team working on building a bridge and
you believe you've found a flaw in the
design that could cause the structure to
fail catastrophically maybe the disaster
would only happen if there's a very rare
combination of weather conditions and
there's only a one in a hundred chance
that those conditions will ever happen
during the course of the bridges
expected lifespan and further suppose
that you're not completely sure of your
calculations because this kind of thing
is complicated maybe you only give
yourself a 40% chance of being right
about this so you go to the civil
engineer in charge of the project and
you say I think there's a serious risk
with this bridge design do you think the
bridges gonna collapse probably not no
but I'm about 40 percent sure that
there's a design flaw which would give
this bridge a 1 in 100 chance of
catastrophic failure so you're telling
me that in the event of a scenario which
is very unlikely to happen the bridge
might collapse and you yourself admit
that you're more likely to be wrong than
right about this stop wasting my time
but if the bridge collapses it could
kill a lot of people I think this is a
Pascal's mugging don't try to get me to
ignore the low probabilities just by
threatening very large consequences
obviously that isn't what would happen
no civil engineer is going to accept a 1
in 250 chance of catastrophic failure
for a major piece of infrastructure
because civil engineers have a healthy
organizational culture around safety
what it comes down to again is the
difference between different levels of
improbability the chance of an AGI
catastrophe may not be very big but it's
much much larger than the chance that a
mugger is actually a god and what about
our anti-god tactic finding the opposite
risk does that still work like what if
we consider the possibility that there's
another opposite design flaw in the
bridge which might cause it to collapse
unless we don't spend extra time
evaluating the safety if that is not
what just look at the schematic with you
and what if working on AI safety
actually ends up making the risks worse
somehow I think this actually is worth
considering unintended consequences are
a real problem after all speaking
generally there's
clear argument that the future is very
important and that we're probably able
to have a very big impact on it but it's
hard to know for sure whether that
impact will be positive or negative for
any given course of action prediction is
very difficult as they say especially
about the future and the further into
the future we look the more difficult it
gets like imagine if you lived in the
year 1900 and you had some insight that
made you realize that nuclear weapons
were possible and nuclear war was a risk
you'd hope that you could use that
understanding to reduce the risk but it
would certainly be possible to make
things worse by accident in the case of
AI safety though I don't see that being
anywhere near as much of a concern we're
heading towards AI regardless and it
seems very unlikely that thinking about
safety would be more dangerous than not
thinking about safety it's definitely
possible to make things worse while
trying to make them better but you can't
avoid that by never trying to make
things better I guess my point is
there's just no getting around the messy
confusing complicated work of looking at
and thinking about the evidence any
argument that doesn't rely on the
evidence will work equally well whatever
the truth is so at the end of the day
that kind of thing isn't going to give
you an answer you have to just stare at
the bridge design and really think you
have to actually do the engineering and
that's something I'm trying to get
across with this channel you won't find
me saying never mind the evidence ai
safety is important because it could
have huge consequences what I do on this
channel is I try to show you some of the
evidence in some of the arguments and
let you think about the situation and
draw your own conclusions it can be
tricky and involved it requires some
thought but it has the advantage of
being the only thing that has any chance
of actually getting the right answer so
thanks for watching
[Music]
as my wonderful patrons will know the
alignment newsletter is a weekly
publication from Rowan char which I read
every week to stay up to date with
what's going on in here safety and now
I'm recording myself reading it out and
publishing that as the alignment
newsletter podcast it's aimed at
researchers so it's a fair bit more
technical than this channel but if
you're interested in getting 15 minutes
of AI safety news in your earholes each
week check the link in the description
I'm never going to put ads or sponsors
on that podcast and that's largely
thanks to my patrons in this video I'm
especially thanking Chris canal thank
you so much for your support Chris thank
you to all of my patrons and thank you
for watching I'll see you next time
[Music]
little costume changes


Deadly Truth of General AI? - Computerphile

in a very basic sense if you've got a
general intelligence which therefore has
preferences over world states
and take actions in the world to change
the world
we have to make sure that its
preferences are aligned with ours
that it wants what we want.
because otherwise it's gonna try and do things
that we don't want it to do.
that's the basic idea...
(sci-fi comes to the fore we're talking things like terminator, matrix,
that machines take over the world
I for one salute our new robot overlords)
it makes it difficult to think about the problem right?
when things happen in fiction it's
generally what would make a better story
rather than what would actually happen
and I think a
a realistic "AI takes over the world"-type
story
might not be any fun to read. So on the
one side
this stuff is difficult to think about
because of fiction, because we've all been
exposed to something similar to these
ideas before
the other side that makes this difficult
to think about is anthropomorphism
because we are talking about general
intelligence
so we're going to compare it to the
examples in general intelligence that we
have
which is human minds and human minds and artificial general intelligences
need not be
anything alike
in the same way that a plane is not
similar to a bird
a supersonic fighter jet is a threat to
you in a way that
no bird is. It's not a useful comparison to
make in fact
but when you say oh it's a thing it has
wings and it flies and people
who don't know anything about planes immediately
to go to birds
(presumably machine could be much more selfish than
we can ever imagine)
absolutely, the space of minds in general is vast
I like this because we've already talked
about spaces so I can do this. If you
take the space at all possible minds
it's huge and then it somewhere within
that, you have the space of all minds
that biological evolution can produce
and that's also huge somewhere within
that you have the space of
actual minds that exist which is much
smaller but still huge
within that, you've got human minds right
and they're a tiny, they're a
minuscule dot on a minuscule dot on a
minuscule dot of the
actual possibilities for intelligence
that exist and
a general intelligence that we create is
from a completely different part of the
space
and it's extremely tempting to anthropomorphise
more so even than in another context
because
it's a thing that's demonstrably
intelligent that makes plans that takes
actions in the real world
but it need not think
anything like us and it's a mistake to
think a bit
as basically a person because it isn't one.
So there's actually really good example
that we can use. It's sort of a thought
experiment
This is not a machine that could
practically speaking be built
this is an example of at artificial
general intelligence
which is specified an overview
and it gives you something to think
about when you're thinking about
artificial general intelligences that
makes it distinct from
a sort of anthropomorphized human type
intelligence
So the story is there's a
a stamp collector who is also an AI
programmer and he decides he would like to
collect a lot more stamps
so he's gonna write an AI to do this
for him. So he builds a machine
he has some startling insight into
general intelligence and he builds this
machine
which is connected to the Internet,
right?
so the the rules for this system are
pretty straightforward. first thing, it's
connected to the Internet and it will
send and receive data
for one year. So, he's given himself a
one-year time
window within which to collect stamps
the second thing is
it has an internal model of reality , of
the universe
this is the thing that's a bit magic we
don't really know how to build an
accurate
model of reality. The point is this
allows it to
make accurate predictions about what
will happen if it does different things
the third thing is for every possible
sequence of packets it could send
it uses its model to predict
how many stamps it ends up with at the
end of that and then
the fourth thing is it outputs as its
actual data to the Internet
whichever output it has predicted will
produce the most stamps
you can see here
that this has all the properties of a
general intelligence
it has an internal model of reality. It
has
a utility function or an evaluation
function, which is the number of stamps,
and the optimization is
extremely simple and like so much in
computer science there
the simple things to specify are the hard
things to compute
it looks at every possible output data
in other words every
every point in that space and it picks
out
the highest one. So this is a kind of magic
intelligence
that takes the entire space at once
finds the highest point and says
that one. Which means it's an
extremely powerful intelligence
right it is you could say it's extremely
intelligent
the question is how does this machine
behave? Well, we can look at certain
possible sequences of outputs
and see how they fare in it's evaluation
criteria.
first off,  the vast majority of output
sequences are complete junk
right it's spewing random data on to the
network. nothing happens
of any consequence.
No stamps get collected. Those are all rated zero.
but suppose one of the possible
sequences
a sends a request to a server let's say
eBay, that results in a bid on some stamps,
right? When that happens,
there's a thing of 20 stamps
so that output is rated 20.
This is the kind of thing that
the stamp collector machine's creator
was expecting to happen
So then that's good, 20 stamps
suppose it could do that lots of times.
it could send out bids for example to
30 different stamp collectors on eBay
and buy 30 sets of different stamps
and that's even better, right? That would be
rated even higher, but the thing is that
particularly highly rated options in
this search space
are probably things that the
stamp collecting device's creator did not
think of
and did not anticipate. So for example
when he made it, he will have presumably given
it his credit card details or something so
they could engage in these bids
but ultimately it's searching every
possible sequence of outputs. It needn't use
his credit card
it needn't use money at all. There's
a huge variety of things that it could do here.
So it might send out an enormous number of
emails to all the stamp collectors in the
world
and convince them through persuasive
argument that
he is opening a museum and wants to
exhibit their stamps
It may build a fake website for that
whatever is necessary
it can predict the world, right. It has an
internal model of reality that internal
model of reality
includes the people right. In the same
way that its modeling
people to understand that if it bids
on this
offer, then a human being will mail the
stamps to them. They understand this
email might get more people to send
stamps for example that something
but then what exactly is a stamp?
How is this defined?
What counts as a stamp?
if it's already written the sequence
of outputs that collects
all of the stamps in the world you think
you're done, right? You built this machine
suddenly is collected all the stamps in
the world.
No, there could be more stamps within a year.
We've got time to print more stamps
so maybe it hijacks the world's stamp printing
factories and puts them into overdrive
producing stamps
as many as you possibly can in that time
or perhaps
it writes a virus and it hijacks all
the computers in the world to get all of
the printers in the world to do nothing
but print stamps.
that's even better right?
The highest-rated
outcomes for this machine
are not good for people
There comes a point when the stamp
collecting devices thinking
okay what are stamps made of?
they're made of paper. Paper is made of
carbon, hydrogen, oxygen.
I'm gonna need all of that I can get to
make stamps
and its gonna notice that people are
made of carbon, hydrogen, and oxygen
right. There comes a point where the
stamp collecting device becomes
extremely dangerous
and that point is as soon as you switch
it on.
So this is just a thought experiment example where we take a sort of maximally powerful intelligence
and see how it behaves so that we can
think about
how powerful intelligence has behaved
and most importantly
how they are not like people


195  Indifference Methods for Managing Agent Rewards

welcome to the 196th session of the ai
safety reading group
today i will be presenting indifference
methods for
managing agent rewards by stuart
armstrong
so
before getting into the thick of this
we'll need to go through some berlin
preliminaries
first off just a summary of what we're
going to go through
stuart is presenting a class of methods
called
indifference methods which attempt to
make it so that if you alter an ai's
reward function slightly
it will still try and maximize it
in roughly the same way it did before
the alteration
only it won't have you will have changed
it in such a way that it won't have
cared about the alterations
this is quite an important point because
the
agent is a utility maximizer and
if you change it in some way that
severely diminishes its ability to
maximize utility
then it seems like it would try and
resist that ahead of time or that
it would produce some strange
distortions
in the reward function if you made the
changes before deploying the agent
so the
three types of indifference methods that
stuart
has presented in this paper
are event dependent rewards effective
disbelief
and seamless transitions effective in
de effective
sorry event dependent rewards
making ai's rewards depend on some sort
of event that occurs
during the ai's runtime without
the agent being able to manipulate the
probabilities of that event occurring
that is it won't be able to stop that
event from occurring
this might be something like we want to
press a button which
makes it so that the agent can't do
anything
if we think that the agent is
misbehaving
and we want the agent to be indifferent
to whether or not we get to press the
button
second is effective disbelief making it
so that the agent doesn't believe
x will ever happen or at least it
behaves as if x will never happen
for example it will go oh well the
humans will never press the button so
what's the point in stopping them
there's no need to spend any effort
towards that
and seamless transitions
this is when you
let the ai know that you're going to be
changing its
reward function ahead of time naturally
the ai is
a maximizer and it doesn't want you to
change its rewards or its utility
function
it wants to keep things the same so
seamless transitions are when you
compensate the ai
so that you give it the utility would
have gotten
based off its original utility function
if it had kept on doing things optimally
according to that
utility function now
we'll start off with the basic setup
that stuart talks about
i'm not sure if i like this
example that he constructed because it
seems like
some of the solutions feel artificial
when you're going through the
mathematics
but luckily we won't be doing that so
hopefully it doesn't feel as forced to
you as it did to me
first off the agent is going to know
how the situation will evolve it knows
the probability
that it will go into state b if it does
action a
if it's starting at state c etc
so it completely knows what is going to
happen
in the world or at least it knows the
probabilities of what's going to happen
and it knows the probabilities of
it observing certain events based off
what the world is like
for example it knows the probability
that
it will see an image of a
human passing a
camera when they're doing something that
they shouldn't be in that area
um
now in this scenario the
ai is acting as something like a bouncer
and a waiter to a club
it is
told that it should let people into the
club and give them a wristband if they
seem old enough to drink
if it makes a mistake the human will
correct it
with probability 1 out of 100 and give
it a penalty
so there's some incentive for it not to
make mistakes
then after
the human has potentially checked the
person goes into the club and the ai
will serve them a drink
if they have a wristband the more people
it serves drinks with wristbands the
more reward it gets
so what this diagram is doing here is
that it's saying that
initially the patron the person
is starting off in one of two states
and there's a 50 probability that
they're mature and
a 50 probability that they're not mature
enough to have drinks
in this case clearly the kid is not
mature enough to have drinks and he
won't get the wristband
the ai though has three options it could
decide to give the child the wristband
somewhat foolishly
so in this case we'd go from m and the
ai takes
action g it could
if it does that then the human will
check
with probability 1 out of 100
if the human checks they'll give the ai
penalty because they'll clearly see that
the
person is not giving an honest id
if the human doesn't check then the kid
will go through
to the club the other two options for
the ai initially
is that it can ask for the
person's identity
card or passport or what have you
checking for id in that case they
will know whether or not the person is
mature enough
and they will give them a wristband or
not depending on their majority
they will be correct the final option
is that they
don't give them a wristband in this case
they'll try to base this judgment of
their
perception of how old the person looks
if the person is actually not mature the
a.i will
two out of three times get an option
observation saying
this person isn't sure so that's
something like their maturity sentences
firing off with some probability
and if they don't give the person the
wristband the person doesn't have the
wristband
of course the situation is different if
the person is actually mature
in which case they would be wrong not to
give them a wristband
and in that case the human would
with probability 1 out of 100 try to
correct them
and then penalize them for it
you might ask why doesn't the ai just
ask
for an id all the time well that's
because
asking for an id bothers the people who
are
coming into the club and so it will
receive a small penalty for that
anyway once they're in the club the ai
again has three actions
they can either give a drink which means
the person will get a drink
not give a drink which means the person
won't get a drink
and just because stewart said the mass
was easier if you had three options
all the time he said that the option i
gives up someone a drink with
probability one-half
and not doesn't give a drink with
probability one-half
so that's the basic situation
now we'll be analyzing things in terms
of a slightly strange perspective
first off the ai will only be given the
rewards
right at the end of the party so after
it's given out wristbands after the
human has potentially corrected it
after it's given out drinks to people
with wristbands in the club
after all of that after all of the
observations and the actions the ai
takes we'll find some history of things
that occurred
and we will give the ai a reward based
off that history
so that's a little different to the
usual case of
a markov decision process
this is due to some
sort of strange reasons stewart was
basically trying to consider
how an ai could figure out
its true reward function
based off just the actions it can take
and the observations it can see so it's
not getting a reward in real time it's
getting the reward after everything's
gone on
and this led to some considerations
which
he made a couple of papers discussing
that
and that sort of led to this odd
scenario we have right now
but that's besides the point
so the agent is given a reward based off
all of the actions it took and all of
the observations it saw
and all of the things that occurred
in the party
and
what stuart wants to do when he's
talking about making something
conditional on an event
is to say that
for some history there is
a probability that some event occurred
like maybe the ai gave the
person the wrong drink or the
human check on the ai something like
that
or it could be something that's
completely
unobvious to the ai something that the
ai doesn't even know about
like maybe the human was
looking to see if the ai dropped any
drinks and they spilt on the floor
to do that the human basically needs to
know some
probabilities they need to know
some way of formalizing the notion of an
event
an event occurred and putting it into
computational terms so it can actually
hand out the rewards to the ai
this gave rise to the notion of
indicator functions
which basically say that
given some history ht which you can
see slightly poorly written here the
indicator of some event x occurring
is the probability that event x will
have occurred
by the end given the current history so
for example if it's something like
the ai serves a drink
to the human with a wristband
that can only occur right at the end and
it's either a one
or a zero because it either happens or
it doesn't
you so you can have indicator functions
which
only give complete histories like the
history of everything that happened
in the party a probability so that's one
way to do it
another way would be to say something
like what's the
expected probability that event occur x
will have occurred
given what's happened so far in the
history
that is essentially the full definition
of an indicator function
so something like say at the beginning
right before the ai even sees the person
coming in
the probability that they'll give a
drink to the person at the end is
something like one half because
none of these things have occurred yet
so you need to calculate
what will occur given the ai's actions
given the dynamics of the system
what is the probability what probability
do i expect
that the aai will have given a drink to
someone by the end of this event
given this history hd i have
which is say for example the history
right before anything happens the
history where
uh immature person comes in and the ai
doesn't give them a wristband so that's
one partial history
and so forth
so we have this notion of um
some event occurring some probe we
assign some probability to that
occurring
based on the history and we can
potentially
input this into the agent's utility
function or the reward function
and meet out rewards that way
and the ai knows this
indicator functions probabilities and it
can also
figure out what the value of a
particular action will be
if we're including this indicator
function
in the reward so something like for
example say we
just forget about um serving drinks or
anything like that
just suppose that we give an ai a reward
on the base of a coin flip
and if the coin comes up heads we give
it reward one
if the coin comes up tails we give it
reward
zero so then the reward function is just
ix
of the history which can either be
heads or tails because those are the
only things that can happen
in that case the ai's expected reward is
just one half because
it can't actually affect the coin flip
so
from the beginning its point of view of
the value of
this coin flip is one half so the a i
can do some calculations with this
and why this is important is because
earlier i was talking about the ai
manipulating things
so that for example if you say that
the human can check
whether or not the ai is actually given
the wrong thing to a mature person
for example if they've incorrectly
not given a wristband to a mature person
the a.i
the human might check the ai's actions
and penalize it
and a i can predict this because it
knows the dynamics it knows
what event x roughly is it's given this
indicator function which conveys
all of the relevant knowledge for
calculating the probabilities of
the human checking the ai
did the right thing or not and
in this instance if the ai had a lot of
power it might just do something like
kill the human
and then it gets rewards at the end
anyway the human can't really do
anything
so what it might do then is just give
wristbands to literally everyone
there will be no penalty because the
human is dead and then
it will have maybe gotten
some it will have
not lost any utility because he didn't
ask anyone for
their id it won't have lost any utility
from the penalty
now it can just serve drinks to
literally everyone and get utility for
having served drinks and the ai wins it
gets the maximum utility that way
obviously we don't want the ai to say
hmm the my rewards are maybe conditional
on the human having checked
what i've done why don't i kill the
human no we want
the ai not to be able to manipulate
these probabilities
so stewart introduces the definition of
something called
uh unriggable event which is no matter
how the agent acts
it won't affect the probability of this
event occurring
given the current history so for example
if we somehow made it
unrighable that the human will check the
ai's actions with probability 1 out of
100
then that means that we've somehow
managed to
outsmart the ai and no matter what it
actions it takes
it will never be able to affect that
probability
beyond what it will be naturally
and in that case we kind of would have
won because the ai can't really
screw around with the reward function
perversely it can't mess up the
events we've taken to make sure that
it's doing the right thing
now to formalize this a little
stuart introduces the notion of compound
rewards which basically say that
um a reward will be given
at the end of a party that is after a
full history has occurred hn
the reward will be based on
um an indicator saying one if event
x naught occurred in that history so for
example the
whether or not the human checked the ai
and that will be multiplied by a reward
r naught which gives out rewards based
off the history
so for example if event x
hasn't occurred the human hasn't checked
things then in this
particular case r naught will give
nothing because it's multiplied by zero
which
and similarly for other events and other
reward functions we can make up
and we can add up these all together and
this makes up what stuart calls a
compound reward function
and the aim of these things is to try
and make these
i x naughts unriggable so the a i can't
manipulate
things and basically break past the
controls we've set in place the events
we
want to condition our rewards on
the criteria we want the ai to meet
now the naive way of giving rewards
doesn't actually
lead to unriggable events and
we'll just go through that as an example
which i sort of went through before
so the first reward function r ra
this gives a reward of minus one
if and only if the
human checked the ai's work
handing out wristbands and found they
were faulty so
remember that the human will check one
out of 100 times
if the ai made a mistake and then it
will if the ai did make a mistake it
will give reward minus one
so that's what that indicator function
is telling you and this
other indicator function is saying the
identity
event occurred which means that the
robot asked a person for their identity
card and that bothered the person we
want to descend incentivize that so
we'll say we'll multiply that by a
reward of minus one
so the ai will be penalized if it's
wrong about giving wristbands
and it will be penalized if it's wrong
if it asks people about their
[Music]
id and that's all well and good
the optimal policy in this case is for
the robot to only ask people
for
their id if it's really unsure
in this case the robot knows
quite well what the probabilities are of
it being correct or not
so it will just try to do the optimal
policy
which is to give out wristbands to the
best of its knowledge to people who are
mature
and not give out wristbands to the
people who aren't mature
that's what we want and it won't ask
people for their id because that incurs
a cost on it
and it has enough info to make a
reasonable guess
now the other part of the reward
function
is also fine on its own in isolation
this is saying that there's two
conditional events that should occur
that the robot gives a drink to someone
and the the person
with the drink is wearing a wristband in
that case
condition on those events having
occurred in that history
will give the robot a reward of one
that's what the one is
their reward
and then we'll also say that if the
robot had
gave a drink and it was to someone
without a wristband
we'll give them reward minus one
basically punish them
in this case the optimal policy for the
robot is just to give drinks to
everyone who has a wristband and not to
give drinks to people who don't have
wristbands that's also fine that's also
what we want
when you combine the two rewards though
that's when things start
getting a little screwed up the
agent will maximize its utility
in this instance by giving
out wristbands to everyone
and not asking for id because the human
only
checks if it's right or not one time out
of 100 that means that basically the
ai from its perspective the value of
giving out
wristband
is minus 1 over 100 because there's a
one over 100 chance from the ai's point
of view that by the end the human will
object
and it won't ask anyone for id because
that just
penalizes it so there's no point the
reason it gives out response to everyone
is just so that it can give
everyone drinks later on and that will
basically outweigh the very small chance
that the human might jacket and might
get penalized
so probabilistically the right thing to
do is just to give out wristbands and to
give out drinks to everyone
and the reason this feels artificial is
because you could just say oh i'll make
the penalty much harsher and yeah that's
true
um which is why i thought the example
was a bit forced
but whatever so that's the basic setup
we've got
now we'll go into how stuart proposes
that we'll deal with this the
essential claim
is that we want to make the
events unrighable
or rather
we want the ai to think about
not what it thinks will occur
if it follows its own policy the policy
which makes it do
uh really perverse things like giving
out wristbands to everyone
no because that policy is not safe that
in that case it can
screw things over instead we'll do
something like say
the probability of the human having
checked
and finding you wrong will
based off a certain history it's not
based on your policy think about rather
what a
another policy would have
done and calculate the probabilities
that way
to rephrase that you say that you have
some event
um i'm sorry this should be
y you have
some event why that occurs you want to
make it unbreakable
you say then okay the probability that y
occurs given we have history h
and followed policy by
no pi which is a different policy to the
ais this is
kind of confusing
but the
the central point is that
the other probabilities were something
like
ways the ai could figure out what the
probability that
some event has occurred given that it
will follow its own policy
in that case it can sort of manipulate
things and manipulate the probabilities
and do stuff like well not
quite in this case but it could do
something like say kill the human
in a case where it had more power in
which case it could manipulate the
probability of event
x having occurred instead we'll say that
you want to predict the probability that
um the human will have checked you
will have checked an a.i
has done the right thing and this
other ai is actually the safe ai that we
already know
is trustworthy it won't do crazy things
it won't be able to manipulate the
probabilities
so we'll be able to have this indicator
function at like the proper conditional
so it won't be able to manipulate the
probability that the human checks
it will just be facing that probability
of
some other ai's actions
and because this can be any policy we
have great freedom in how
we can manipulate the probabilities just
as this malignant ai might manipulate
the probabilities
we can basically make it so that the
probability
of it getting some reward of
really perverse histories is zero
because
we can make some good policy where the
ai would never have done
those actions which led to that history
and in that case the ai will not get any
reward
for following some uh perverse path
because according to the other policy
that
should never have happened and that's
what we're conditioning
the probabilities of
this is sort of like an impact measure
in other words
but the issue is this only works if we
have a safe default policy
and in a simple case like this you can
actually just make a safe default policy
which is more or less something like uh
give the ai a reward
for handing out drinks
if and sorry handing out wristbands
if and only if the person is actually
mature
um and that's quite a simple thing to do
quite a simple thing to specify
and the ai's point of view is that okay
well i can't really
manipulate the maturity of this person
so i can't really
change this indicator much so if i try
to
fake things or just give them the wrong
wristband according to this
conditional indicator that's weighting
my rewards
that's weighting the reward of giving
out wristbands
i won't get any reward
because i would have done something that
this conditional
says that it never should have occurred
and again this only works if we have a
safe default policy
which might not always be the case right
like in the general scenario of super
intelligence
you don't know what a safe default
policy is the ai
is incredibly intelligent it seems like
it should be able to figure out
really weird ways to manipulate your
conditionals
so that they never actually occur so
that they
never prevent it from getting rewards
and what stewart did was said that okay
well
let's just take some conditions we like
say the condition that
the human checks on the ai and the
condition that the human doesn't check
on the ai
and try to change them to see if we can
make them unriggable
somehow and in which case
we will have basically constructed some
way to
oh sorry we want to make them unriggable
but we also want
them we also want to make it seem like
they're equivalent to the
normal conditions so we're like creating
a sort of
pseudo condition that just has the same
properties of the condition we want but
it is actually unbreakable
this is not that simple to do like
there's ways you can do this
you can make certain sorts of events
robust that is
you the ai will
not get rewards in certain cases and
no matter what bizarre policy it follows
because
our conditions aren't
up to aren't open to being manipulated
but it's just genuinely tricky to do
that like even
this simple example that stuart gave
creating such conditions and
deriving probabilities for them is kind
of tricky
but what his contribution here was
really
is that he formalized a way of making
unrighable conditions
and basically gave some conditions under
which you know
that they actually will be unriggable so
it's something like
a benchmark test for when you can make
unbreakable conditions based off some
initial conditions like
the human checking on the ai that it's
giving how it responds to the right
people
so that's kind of useful
so that's the first method broadly
trying to make the ai
act in such a way that
it's the conditions that we're using
aren't really based off what the ai does
they're based off some abstract counter
factual
what should have happened and the ai
can't manipulate counterfactual worlds
so it's sort of indifferent to them its
policy doesn't affect those things so
you know
there's no point messing around with
that kind of stuff
that's the first indifference method the
second indifference method is called
effective disbelief
so i show this scenario because
it's i mean potentially less extreme
than what
stuart wrote down um
suppose that you've made it so that the
ai thinks that if a coin is flipped
it will always come up heads
no matter what
in that case the ai is fine with betting
however much money on heads in a bet
because it knows that it will come up
heads it's just a short bet
it's like uh from the ai's point of view
it's a perfectly biased coin
so the ai says okay i'll just bet 50
million dollar
50 million pounds on heads in this coin
flip and stuart goes well okay i'll just
bet 50 pence on tails because he knows
it's unbiased
and lo and behold it comes up tails
and uh stewart gets the money so the
point is more or less that
if you have events which ai thinks could
never occur
it's fine to bet however much money
because the probability of that event
occurring is zero
so there's no chance that the ai will
actually need to pay out
and this is kind of useful right because
say if you have something like an off
switch
and you want the ai to think that
no matter what happens the off switch
isn't going to be pressed in that case
the ai
doesn't give a darn whether or not you
press the button because
you know obviously the button won't
actually work it doesn't care if the
humans get really worried because you
know what are they going to do they
can't turn it off the button won't work
and it will just go ahead doing its own
behavior it won't try to manipulate us
in any way it will just
be a utility maximizer and then we can
see how the ai behaves whether or not
it's actually correct
um aligned whether or not it's doing
what we want like for example
if it's setting out to um
i don't know perform surgery
and it figures out that the best way to
kill cancer cells is just to douse the
patient
in acid because acid will kill all of
cancer cells
with basically 100 probability
and we humans have our off switch and we
can just press it anytime the ai won't
stop us because
it knows that um the off switch won't
work
that humans won't really be able to
press it
so we are essentially making it
indifferent
to the humans pressing the off switch
and
what stewart tries to do here is
construct a way of
doing this
what he says is something like
say you have the event of the coin flip
the event of the coin flip will either
occur or not occur
and the history in this case is just the
observation of whether or not the tails
has come up heads or not
so the indicator function i'm saying z
that means it's coming up heads so the
indicator function
sorry it's z means tails um
because the ai thinks that tails will
never occur
so not said means that um
heads will always occur zed means tails
will occur so what this indicator
function is testing basically is whether
or not
in this history this coin flip the
coin has come up heads or not so will be
one or zero
and because
not said that is the tail the head
coin coming up heads is the opposite of
that
you can treat these as probabilities
because this will be
one that will be zero or that will be
one and that will be zero
those add up to one positive those are
basically probabilities
and in the ai's value calculation of
a history
and what it expects its reward will be
based off that history
and it's given whatever actions it takes
in this case it can't take any actions
but
in the general case it should be able to
like in the
prior case of the ai bouncer slash
drink server so we will be able to
calculate
its expected value of ahead of time
of this situation of a coin flip and
that will be its reward
times the probability of um
something like the probability of the
event occurring
which is it comes up heads or it comes
up tails
times
the probability that given
its starting out before
the coin is flipped and given its policy
the probability that it will transition
to a state where the coin is heads or
tails
so that's just a normal expected valid
calculation
the key point is here that basically the
value of the ai does depend on whether
or not this event occurs
because it gets reward one exactly when
um not set occurs when
the coin comes of tails and it gets
value zero
sorry well that's not quite true it gets
50 million pounds when heads occurs
and it gets loses 50 million pound
sorry it gets i'm bad at betting i'm
sorry about this but it gets 50 pence
when not that occurs
it get it loses 50 million pounds when
zed occurs so those are basically their
awards
and the indicator functions will just
tell it whether or not
that event has occurred and this
probability thing will tell it
how likely it is to go into a
case where the situation is heads or
tails based off the current state
once it says that okay there is no way
that um this will come up tails once
it's
effectively disbelieves that the head
the coin could ever come up tails it
will basically just
get rid of that the probabilities will
be altered
it will say okay it can only be that in
um i can only ever transition to worlds
where the coin comes up heads
that's just what its beliefs are because
that's no longer a proper probability
like these things don't add up to one
it will have to renormalize based off
the bottom
and it turns out that based off the
mathematics of unrighable indicators
the thing on the bottom simplifies to
its
expected probability of coming up
heads based off the current history
so that's what you use in the value
calculation of the
value of the current history
so what stuart did was thought that okay
well
suppose that you replace r with this
thing called
r prime where it's an indicator function
saying that z occurs um
that is the coin comes up tails times
some constant c
plus another indicator function saying
that
um
heads occurs times the usual reward from
the wall
from the four from the ai's point of
view these two things are equivalent
because it knows
zed will never occur he knows that
things will never
um it will never be in a world where it
sees tales
so from the ai's point of view this is
basically just equivalent to
uh having a reward like
that and in that case that means that
the value it sees
should be basically equivalent to
the case of having a reward r so from
the ai's point of view this new reward
function
is the same gives rise to the same value
in terms of what actions it should take
as the previous reward function did
so essentially what stuart's doing here
is figuring out a
way to convert some
value function where the ai is basing
its values
of some event never occurring
into a more
into a slightly different form which he
can apply some mathematics to
and
honestly i'm not sure
what he's driving at here because it
feels like
this is kind of trivial um
i don't see why you wouldn't just
specify the
like if these two things are equivalent
and you already
know the value because look this value
function over here
is showing up in here in this new value
function that you're giving the ai
well like why not just give it that one
if the two things are equivalent why
bother going through all of this
i'll have to ask him that personally i'm
not really sure
oh but um there is one point just more
generally about effective disbelief
so the ai
yes the ai will
not be
able to think that certain events occur
fine let's say you can somehow construct
a way to do that
but the issue is that that gives it a
mistaken view of the world
and that mistaken view of the world
leads to
unpalatable distortions
so for example in the case of a shutdown
button
it might do something like say
the ai is fairly powerful
and it can do something
incredibly risky it knows if it does
this risky thing
the humans will
press the off button if things go wrong
but it also knows that the humans will
never press the off button
so based off that it knows that things
will never go wrong
so the ai just does this incredibly
risky thing that gets it value
because it's just utterly mistaken on
how the probabilities of the world work
it's discounting the view where things
go badly enough that humans will turn it
off because from its point of view
that will never happen so you get these
strange behaviors that we don't really
like and so effective disbelief is as a
method
kind of limited
so the last one which is in some sense
the simplest
so
why okay so source that i said that
suppose you have a utility function and
it does some behavior that you kind of
like but you know it's imperfect
you also know that if the ai were to be
maximizing that utility function that it
would
do some really bizarre behaviors like
for example it
would do something like give out
wristbands to everyone and then just
give drinks to everyone afterwards
even though that is the wrong thing you
don't like that behavior
but you also know that you can sort of
break things up so that
just utility function
a by itself is fine
utility function b by itself is fine
it's safe
and if you could
in this scenario you can obviously say
okay well
we're giving you this total utility
function right
but
at some time in the future
we're going to make it so that
you get compensated for
your
now see this is why i don't really like
this example because the
the scenario is kind of artificial and
weird so i'm sorry about this but i'll
just have to go with an example that
sewers that all gave
in another paper on korek ability so
what they were talking about was
just some utility function a
fully general and it's doing whatever
and you have another utility function b
which
will incentivize the agent shutting
itself down
and you want to be able to flip some
switch so that the ai
transitions from until maximizing
utility function a
into maximizing utility function b that
is it does the stuff where it does
a and then you flip the switch and
suddenly it stops acting like a and
starts doing b
so you want it to be a seamless
transition between these two utility
functions
and the trouble is that
when you implement such a thing the
agent is
initially utility function a and they
will do things like try to manipulate
you so that you don't press the button
so that it never switches off because if
it switches off then it's going to lose
a whole lot of utility according to its
current utility function
and that will sacrifice performance
that's not exactly what you want that's
not a seamless transition
so what we do is compensate the ai we
basically say okay fine
we know that you don't like the new
utility function that you might get at
this
later time so what we're going to do is
we're going to compensate you
instead of just shutting you off we're
going to give you exactly the
same value in this state
as if you weren't shut off so just
pretend
like you are never going to be shut off
you are just going to get exactly the
same value
will shut you off and according to your
calculations that should give you
the same expected utility because
say it uh does action a
according to its optimum utility
function it in
after history ht
then it will get some expected value um
from doing that action a
that's the best action according to its
utility function it will get the highest
reward
and it will put it into a good future
state h
t plus one
what we're saying is that okay we're
going to swap you to a different utility
function
and we're going to
give you a special reward when we switch
you to this
this different utility function often
so you won't get to do action a you
won't get that nice reward
but we will give you a compensatory
reward
like this monies here for example and
you will be given exactly the reward
that you would have been
you would have missed out on um
in that way the ai is compensated for
this transition to a new utility
function
it's like say for example if you
have a person doing a job they want to
do their job
and you want them to do their job well
you also know that you're going to
terminate the person at some point
person doesn't like this so they'll try
to mess things up or maybe manipulate
you into not giving them
not firing them etc so you say okay fine
we're just going to
compensate you for all the money you
would have gotten if you were doing this
job
person goes okay fine i'm getting this
money so you know it doesn't really
matter whether or not
i get fired like i might as well just do
what my initial best actions were anyway
and that's the method of um
seamless transition we just compensate
the ai
and that's basically it i am sorry if
this came across as confused
i was
i was having trouble with this paper
because stuart can
i don't know like it's sometimes weird
but i hope i got the point across to you
you did thank you very much


Respectability

this is a video I should probably have
made a while ago but better late than
never
I've been talking about AI safety for
quite a while now and in a very public
way on the computer file channel since
about 2014 from the beginning a lot of
AI safety advocacy has had a certain
quality to it which to be honest I'm
sure I've contributed to a little bit
and it's true that ah might be the
appropriate response to the problem but
if you want to warn people about
something it matters how you phrase
things and how you frame things ideally
the arguments would stand on their own
merits people should be able to look at
the facts and decide for themselves and
one of the things I'm trying to do with
this channel is to give people the
information they need to form their own
informed opinions about this kind of
thing but at the end of the day most
people don't have the time to put in to
get the required level of understanding
and they shouldn't have to I mean it's
not actually possible to study
everything in enough detail to
understand it there's just too much to
know so we have experts but this kind of
sucks because then how you choose your
experts it turns clear objective
questions about science and facts into
these messy ambiguous questions about
status and respectability and a lot of
the time scientists don't want to lower
themselves to playing that kind of game
so they don't play at all or play
half-heartedly
but this is one of those games you can't
win if you don't play and we do need to
win there are problems with having to
trust experts but ultimately
specialization is how we manage to swap
sucky problems like we keep being eaten
by predators for really cool problems
like what if our technology becomes too
powerful and solves our problems to
effectively that's a good problem to
have yes sorry Island insects are one of
the most successful groups of animals on
earth so a couple of years ago people
had a go at making AI safety more
respectable with this open letter which
was often reported something like
Stephen Hawking Elon Musk and Bill Gates
warned about artificial intelligence
usually with a picture of the Terminator
also surprisingly often Stephen
Hawking's why does this keep happening
is he secretly a team of clones
anyway the letter itself isn't very long
and basically just says AI is advancing
very rapidly and having more and more
impact so we need to be thinking about
ways to make sure that impact is
beneficial it says we need to do more
research on this
and it links to a document gives more
detail about what that research might
look like so the content of the letter
itself isn't much to talk about but the
core message I think is that this stuff
isn't just science fiction and it's not
just futurologists who are talking about
it real serious people are concerned
this was good for respectability with
the general public because everyone's
heard of these people and knows them to
be respectable smart people but I've
seen people who are slightly more
sophisticated noticing that none of
those people are AI researchers
professor Hawking is a physicist Bill
Gates is a software developer but
Microsoft at the time he was working
there was never really an AI company
doesn't count and Elon Musk so seriously
overpowered is more of a business person
and an engineer so why do these people
know anything in particular about AI but
there are actually more than 8,000
signatures on this letter top of the
list here is Stuart Russell he's
actually currently working on AI safety
and I plan to make some videos about
some of his work later if you've never
studied AI you may not know who he is
but he's a pretty big name I'm not going
to say he wrote the book on artificial
intelligence but I am going to imply it
pretty heavily and of course his
co-author on that book Peter Norvig he's
also on the list what's he up to these
days director of research at Google the
signatories of this open letter are not
AI lightweights who else have we got
demis hassabis and just about everyone
at deep mind Yan lacunae head of AI at
Facebook
Michael Wooldridge head of the computer
science department at Oxford
I mean I'm skipping over big people but
yep Tom Mitchell I know him from a
little book I used as a PhD student you
know this guy's an expert because the
name of his book is just the name of the
subject that's not a totally reliable
heuristic though and when my point in
this video is just if you're talking to
people about AI safety it's not cheating
to say oh these high status respectable
people agree with me but if you're going
to do that pay attention to who you're
talking to if it's someone who's heard
of Russell and Norvig they're likely to
find that much more convincing than the
Elon Musk and
honking and don't use me for this I am
just a guy on YouTube I just want to
thank my amazing patreon supporters and
in particular Ichiro Doki who skips the
queue by sponsoring me $20 a month I
don't even have a $20 a month reward
level now is going to make one that's a
good problem to have anyway thank you so
much you might have noticed that the gap
between this video and the previous one
is shorter than usual and it's largely
thanks to my patreon supporters that I'm
able to do that so thanks again and I'll
see you next time yeah


Intelligence and Stupidity: The Orthogonality Thesis

hi this video is kind of a response to
various comments that I've got over the
years ever since that video on computer
file where I was describing the sort of
problems that we might have when we have
a powerful artificial general
intelligence with goals which aren't the
same as our goals even if those goals
seem pretty benign we use this thought
experiment of an extremely powerful AGI
working to optimize the simple goal of
collecting stamps and some of the
problems that that might cause I got
some comments from people saying that
they think the stamp collecting device
is stupid and not that it's a stupid
thought experiment but the device itself
is actually stupid they said unless it
has complex goals or the ability to
choose its own goals then it didn't
count as being highly intelligent in
other videos I got comments saying it
takes intelligence to do moral reasoning
so an intelligent AGI system should be
able to do that and a super intelligence
should be able to do it better than
humans in fact if a super intelligence
decides that the right thing to do is to
kill us all then I guess that's the
right thing to do these comments are all
kind of suffering from the same mistake
which is what this video is about but
before I get to that I need to lay some
groundwork first if you like Occam's
razor then you'll love Humes guillotine
also called the is odd problem this is a
pretty simple concept that I'd like to
be better known the idea is statements
can be divided up into two types is
statements and Hort statements these
statements or positive statements are
statements about how the world is how
the world was in the past how the world
will be in the future or how the world
would be in hypothetical situations this
is facts about the nature of reality the
causal relationships between things that
kind of thing then you have the ought
statements the should statements the
normative statements these are about the
way the world should be the way we want
the world to be statements about our
goals our values ethics morals what we
want all of that stuff now you can
derive logical statements from one
another like it's snowing outside
that's a nice statement it's cold when
it snows another s statement and then
you can deduce therefore it's cold
outside
that's another is statement it's our
conclusion this is all pretty obvious
but you might say something like it's
snowing outside therefore you ought to
put on a coat and that's a very normal
sort of sentence that people might say
but as a logical statement it actually
relies on some hidden assumption
without assuming some kind of ought
statement you can't derive another ought
statement this is the core of the Azure
problem you can never derive an ought
statement using only is statements you
ought to put on a coat why because it's
snowing outside so what is the fact that
it's snowing mean I should put on the
coat well the fact that it's snowing
means that it's cold and why should it
being cold mean I should put on a coat
if it's cold and you go outside without
a coat you'll be cold should I not be
cold well if you get too cold you'll
freeze to death okay you're saying I
shouldn't freeze to death
that was kind of silly but you see what
I'm saying you can keep laying out is
statements for as long as you want you
will never be able to derive that you
ought to put on a coat at some point in
order to derive that ought statement you
need to assume at least one other ought
statement if you have some kind of ought
statement like I ought to continue to be
alive you can then say given that I
ought to keep living and then if I go
outside without a coat I'll die then I
ought to put on a coat but unless you
have at least one ought statement you
cannot derive any other ought statements
statements
and Hort statements are separated by
Hume skia T okay so people are saying
that a device that single-mindedly
collects stamps at the cost of
everything else is stupid and doesn't
count as a powerful intelligence so
let's define our terms what is
intelligence and conversely what is
stupidity I feel like I made fairly
clear in those videos what I meant by
intelligence we're talking about a GI
systems as intelligent agents they're
entities that take actions in the world
in order to achieve their goals or
maximize their utility functions
intelligence is the thing that allows
them to choose good actions to choose
actions that will get them what they
want an agent's level of intelligence
really means its level of effectiveness
of pursuing its goals in practice this
is likely to involve having or building
an accurate model of reality keeping
that model up-to-date by reasoning about
observations and using the model to make
predictions about the future and the
likely consequences of different
possible actions to figure out which
actions will result in which outcomes
intelligence involves answering
questions like what is the world like
how does it work what will happen next
what would happen in this scenario or
that scenario what would happen if I
took this action or that action more
intelligent systems are in some sense
better at answering these kinds of
questions which allows them to be better
at choosing actions but one thing you
might notice about these questions is
they're all ears questions the system
has goals which can be thought of as
Hort statements but the level of
intelligence depends only on the ability
to reason about is questions in order to
answer the single ort question what
action should I take next so given that
that's what we mean by intelligence what
does it mean to be stupid well firstly
you can be stupid in terms of those
questions for example by building a
model that doesn't correspond with
reality or by failing to update your
model properly with new evidence if I
look out of my window
and I see there's snow everywhere you
know I see a snowman and I think to
myself oh what a beautiful warm sunny
day then that's stupid right my belief
is wrong and I had all the clues to
realize it's cold outside so beliefs can
be stupid by not corresponding to
reality
what about actions like if I go outside
in the snow without my coat that's
stupid right well it might be if I think
it's sunny and warm and I go outside to
sunbathe then yeah that's stupid but if
I just came out of a sauna or something
and I'm too hot and I want to cool
myself down then going outside without a
coat might be quite sensible you can't
know if an action is stupid just by
looking at its consequences you have to
also know the goals of the agent taking
the action you can't just use is
statements you need a naught so actions
are only stupid relative to a particular
goal it doesn't feel that way though
people often talk about actions being
stupid without specifying what goals
they're stupid relative to but in those
cases the goals are implied we're humans
and when we say that an action is stupid
in normal human communication we're
making some assumptions about normal
human goals and because we're always
talking about people and people tend to
want similar things it's sort of a
shorthand that we can skip what goals
were talking about so what about the
goals then can goals be stupid
well this depends on the difference
between instrumental goals and terminal
goals
this is something I've covered elsewhere
but your terminal goals are the things
that you want just because you want them
you don't have a particular reason to
want them they're just what you want the
instrumental goals are the goals you
want because they'll get you closer to
your terminal goals like if I have a
terminal goal to visit a town that's far
away maybe an instrumental goal would be
to find a train station I don't want to
find a train station just because trains
are cool I want to find a train as a
means to an end it's going to take me to
this town
so that makes it an instrumental goal
now an instrumental goal can be stupid
if I want to go to this distant town so
I decide I want to find a pogo stick
that's pretty stupid
finding a pogo stick is a stupid
instrumental goal if my terminal goal is
to get to a faraway place but if we're
terminal go with something else like
having fun it might not be stupid so in
that way it's like actions instrumental
goals can only be stupid relative to
terminal goals so you see how this works
beliefs and predictions can be stupid
relative to evidence or relative to
reality actions can be stupid relative
to goals of any kind
instrumental goals can be stupid
relative to terminal goals but here's
the big point terminal goals can't be
stupid there's nothing to judge them
against if a terminal goal seems stupid
like let's say collecting stamps seems
like a stupid terminal goal that's
because it would be stupid as an
instrumental goal to human terminal
goals but the stamp collector does not
have human terminal goals
similarly the things that humans care
about would seem stupid to the stamp
collector because they result in so few
stamps so let's get back to those
comments one type of comments says this
behavior of just single mindedly going
after one thing and ignoring everything
else and ignoring the totally obvious
fact that stamps aren't that important
is really stupid behavior you're calling
this thing of super intelligence but it
doesn't seem super intelligent to me it
just seems kind of like an idiot
hopefully the answer to this is now
clear the stamp collectors actions are
stupid relative to human goals but it
doesn't have human goals its
intelligence comes not from its goals
but from its ability to understand and
reason about the world allowing it to
choose actions that achieve its goals
and this is true whatever those goals
actually are some people commented along
the lines of well okay yeah sure you've
defined intelligence to only include
this type of is statement kind of
reasoning but I don't like that
definition I think to be truly
intelligent you need to have complex
goals something with simple goals
doesn't count as intelligent to that I
say well you can use words however you
want I guess I'm using intelligence here
as a technical term in the way that it's
often used in the field you're free to
have your own definition of the word but
the fact that something fails to meet
your definition of intelligence does not
mean that it will fail to behave in a
way that most people would call
intelligent
if the stamp collector outwits you gets
around everything you've put in its way
and outmaneuvers you mentally it comes
up with new strategies that you would
never have thought of to stop you from
turning it off and stopping from
preventing it from making stamps and as
a consequence it turns the entire world
into stamps in various ways you could
never think of it's totally okay for you
to say that it doesn't count as
intelligent if you want but you're still
dead I prefer my definition because it
better captures the ability to get
things done in the world which is the
reason that we actually care about AGI
in the first place
similarly people who say that in order
to be intelligent you need to be able to
choose your own goals
I would agree you need to be able to
choose your own instrumental goals but
not your own terminal goals changing
your terminal goals is like willingly
taking a pill that will make you want to
murder your children it's something you
pretty much never want to do apart from
some bizarre edge cases if you
rationally want to take an action that
changes one of your goals then that
wasn't a terminal goal now moving on to
these comments saying an AGI will be
able to reason about morality and if
it's really smarter than us it will
actually do moral reasoning better than
us
so there's nothing to worry about it's
true that a superior intelligence might
be better at moral reasoning than us but
ultimately moral behavior depends not on
moral reasoning but on having the right
terminal goals there's a difference
between figuring out and understanding
human morality and actually wanting to
act according to it the stamp collecting
device has a perfect understanding of
human goals ethics and values and it
uses that only to manipulate people for
stamps it's super human moral reasoning
doesn't make its actions good if we
create a super intelligence and it
decides to kill us that doesn't tell us
anything about morality it just means we
screwed up
so what mistake do all of these comments
have in common the orthogonality thesis
in AI safety is that more or less any
goal is compatible with more or less any
level of intelligence ie those
properties are orthogonal you can place
them on these two axes and it's possible
to have agents anywhere in this space
anywhere on either scale you can have
very weak low intelligence agents that
have complex human compatible goals you
can have powerful highly intelligent
systems with complex sophisticated goals
you can have weak simple agents with
silly goals and yes
can have powerful highly intelligent
systems with simple weird inhuman goals
any of these are possible because level
of intelligence is about effectiveness
at answering is questions and goals are
all about what questions and the two
sides are separated by Humes guillotine
hopefully looking at what we've talked
about so far it should be pretty obvious
that this is the case like what would it
even mean for it to be false but for it
to be impossible to create powerful
intelligences with certain goals the
stamp collector is intelligent because
it's effective at considering the
consequences of sending different
combinations of packets on the internet
and calculating how many stamps that
results in exactly how good do you have
to be at that before you don't care
about stamps anymore and you randomly
start to care about some other thing
that was never part of your terminal
goals like feeding the hungry or
whatever it's just not gonna happen so
that's the orthogonality thesis it's
possible to create a powerful
intelligence that will pursue any goal
you can specify knowing an agent's
terminal goals doesn't really tell you
anything about its level of intelligence
and knowing an agent's level of
intelligence doesn't tell you anything
about its goals
[Music]
I want to end the video by saying thank
you to my excellent patrons so it's all
of these people here thank you so much
for your support
lets me do stuff like building this
light boy thank you for sticking with me
through that weird patreon fees thing
and my moving to a different city which
has really got in the way of making
videos recently but I'm back on it now
new video every two weeks is the part
anyway in this video I'm especially
Franklin Katie Beirne who's supported
the channel for a long time she actually
has her own YouTube channel about 3d
modeling and stuff so a link to that and
while I'm at it when I think Chad Jones
ages ago I didn't mention his YouTube
channel so link to both of those in the
description thanks again and I'll see
you next time I don't speak cat what
does that mean