(Duplicate?) 187. Stuart Armstrong and Scott Garrabrant on If I were a Well-intentioned AI

so this is more a meta approach than a
specific approach it's a way of getting
insights rather than a technical
nuts-and-bolts method the inspiration
came to me when I was thinking of the
manifold hypothesis for neural nets that
they when they're classifying they're
finding a particular shape like pandas
or Gibbons within the image space and
adversarial examples are sort of sub
manifolds or sub shapes weird sub shapes
inside these shapes probably a very low
dimension but if you have an
optimization process they probably score
high in that so I was thinking well well
if I was the AI I'd avoid those I'd
avoid things that maybe put me on sub
manifolds and that was sort of one of
the first things where I was thinking
maybe these problems are solvable by
considering them from a different
perspective I had a similar thought when
I started on wise good heart laws bad
and when I thought that through I
realized that our usual fear of
anthropomorphizing was actually leading
us astray in this situation surprisingly
I'll get back to that so we don't have
an AI we don't have code for an AI we
don't have a theoretical model for a
safe effective AI so we can't solve the
and most of the questions that we're
working on we don't actually have them
in any formal sense so we can't solve
these by for ultra formalization proof
at the highest level instead we need to
form some abstractions and reason with
those apps
actions like models are always wrong and
some of them are useful and different
abstractions are useful for different
purposes one of the best first
abstractions was donot anthropomorphize
the AI and when you start thinking in
that way you realize that in your
informal reasoning we're using our own
blackbox
our own intuitions to reason about what
a eyes do and that that's wrong like
anything like a eyes will value
complexity hence blah blah blah or these
kind of things were projecting human
ways of reasonings are may eyes and that
is clearly wrong and learning to avoid
anthropomorphizing opens up new ways new
valuable ways of thinking on AI however
I think that too much emphasis on
avoiding anthropomorphize ation blinds
us to certain other aspects and when I
was looking at why is good heart law bad
and I delved into it and I have
Barrett's post on this and people agree
or disagree and there's a bigger the
back and forth but one of the things I
realized is that good heart law is not
bad in the context in the models that we
were thinking of we were thinking this
is a model of what an AI looks like good
hearts law is bad let's avoid it but if
you look at that former model very often
good hearts law is not bad so the but
for us we know that good hearts law we
expect it to be a problem so the problem
was that we were not actually putting
enough of our own preferences into the
model we were considering it at a too
abstract level in a sense
that's one way of thinking about it but
this these sort of things cause me to
think by the way this is a very clean
post picture of everything that's going
on it was a lot more messy than this so
this caused me to go and think if I was
the AI myself trying to avoid
anthropomorphize ation trying to avoid
too much excess information can some of
these problems be solved and then I've
done a series of posts based on that the
first one is just on image
classification which was one of the
original inspiration so this is the
famous pandas plus noise equals Gibbon
and I was thinking again as if I was the
AI and I was being fed these basically
if I knew the concept of adversarial
examples but not obviously what a panda
or a Gibbon was can I avoid this can I
start
as I say looking for points close to it
looking for evidence that this image has
been selected by an optimization process
adding noise myself
it seems the sort of thing that I could
do and of course since I could do it as
a well-intentioned AI this then suggests
the sort of formal method which we as AI
designers could inject in it but looking
for adversarial examples seems like
something that is strictly easier than
fully defining what pandas and Gibbons
are there was another smaller post the
second one was on acting in the world if
an agent had a lot of experience of a
red door and were then put in a
situation where there were red windows
or blue doors
how should it behave on that and this
connects with the sort of good heart not
being too much of a problem because
there are sort of conservative policies
here which cover as much as possible of
the realistic options does they want you
to go to the red window so when you go
to the blue doors they want you to spend
time at one or spend time at the other
there are a variety of different reward
functions that correspond with the
initial one and so this this sort of is
developing into a general idea of
extending models which I won't go into
here
there was the extremal good heart idea
which was you the the idea was a cancer
curing a I trained by apprenticeship
learning and you wanted it to be able to
say blast the cancer cells with a laser
but you didn't want it to dissolve the
human with acid you know both of which
would completely eradicate the the
cancer cells and neither of which the
agent has seen and what I saw is that
there are certain things that are
correlated with the training examples
surviving the operation surviving for
some years after there are some negative
things that are also correlated it's
like complaining about pain the ones
where the operation fails have other
things to complain about being more
prone to dementia that's a negative but
it's it comes from surviving four more
years after and there's some random
things like paying more taxes and
thanking the surgeon so the first step
was dissolving the patient does not have
these features it is a killing the the
cells with a laser has some
these features so this is a way of going
beyond models again so these are not
sort of these are more that these are
insights which came to me through this
way of thinking not that they derive
other people could have derived them
from other roots and finally I think the
most interesting or unique or precise
result that I got was when I was
thinking of Meza optimizing and I was
thinking of it as I am a human in a
corporation I run one of the divisions
of the corporation and I run this for
the benefit of the directors and this
thinking about this and thinking about
how these organizations succeed and fail
I realized that there's a big difference
between a Meza
optimizer that is aligned and one that
is controlled and aligned one is one
that wants what the board wants it's a
perfectly a it has those girls and a
controlled one is one that the board can
understand and that does what the board
wants it to do and it informs the board
of what it's doing and those kind of
things and they are very different
basically there's very it's a very
different situation if you have an
aligned a sub agent or a controlled sub
making a mistake one way like a
controlled substance that you think is
controlled is not so much of a problem
but anyway this sort of opened up a new
small a small new technical area in this
thing and it came about because I was
anthropomorphizing north or thinking
from within the AI a bit more than I'd
been that used to okay that's my sort of
brief overview of why I went this and
what I've used it for
stop share cool thank you very much for
your presentations sure so I guess the
first question should go to Scott so for
the address sorry the image crops
classification example it seems like I
don't know I'm it seems like yeah you
could do something where I don't know an
example the kind of thing that you might
do is you might like add some noise to
the image and like kind of like average
the values of what you would predict
about all the images that are nearby
according to the model the the abseil
examples like coming from like all the
things that are nearby in like pixel
color space which seems to me like
something that you're you're thinking
about from the outside and to the extent
that we want to call the adversarial
example an adversarial example at all
it's because like it's being adversarial
to the specific way in which the neural
net is working the the way in which the
image classifier is working and like
it's it's like work like the adversarial
examples like chosen
it's like notion of nearby is like a fun
it's sorry its notion of nearby is like
a function that's like trying to
diagonalize against the specific image
classifier and it feels like when you're
saying like oh I can deal with
adversarial examples you're like using
the fact that the example is adversarial
to something else as opposed to
adversarial to you do you think that's
true
yes that that is a valid criticism I
think I mentioned it in the posts the
aim when I was thinking that way was can
we get rid of adversarial examples in
ways that are strictly easier than fully
defining what a panda and a Gibbon are
in unambiguous categories and in the
what it what is I think when you get
when you get a message that is designed
to mislead you but technically true what
was the what was that I think it was a
bruh I'm working on that I'm not sure
it's the when you get information when
someone sends you information the
information is true but the source is
extremely manipulative so in those
situations you try and compensate for
your knowledge of how manipulative they
are they're trying to compensate for
your compensation and so on this is what
I was alternately hoping for that if you
know if you know that someone that some
entity has access to your code and it's
trying or some maybe some black box
access and is trying to for you it
should it should be doable
in many cases to make yourself less
vulnerable yeah yeah there's this
question that feels like a central like
binary question for to me for like this
general class of thinking which I which
I don't know is reminded by with what
you just said which is I don't know if
it like there's a class of solutions
there's a class of solutions that
corresponds to like explicitly modeling
the way in which
you might be wrong so that you're like
ah like here's like the way in which I
might be wrong and all kind of like I
think there might be someone behind
stores sorry I'm sorry that that was my
fault I hadn't okay um sorry yeah so
it's like there's one class of solutions
that looks like trying to like
explicitly deal with the ways in which
you might be wrong and trying to like
like when you said like up there
something that's like updating on the
facts of the person is trying to deceive
you or something like that it feels like
a like active reaction to things like
good heart versus there's a more like
passive thing where it's just like don't
optimize too hard or something like that
and it feels like these things are kind
of different types we're like one kind
of like I'm trying to like come from
above and once right now like come from
below or something like that feels like
these things are different types this
makes sense to you or yes but I think
there are some things that seem to be in
the middle of that like one of the
things I was looking at is detecting
optimized images so if something if I
have minor on that plus my extra module
things that are optimized to fool my
neural net look different from normal
images so I I as the AI against a not
infinitely smart adversary well first of
all I should be able to detect images
that are that are designed to for the
lowest level of my neuron that and I
should be able to sort of swallow my
tail and diagonalize myself to some
extent and sort of
detect images that are meant to fool the
entirety of me at least to a level that
makes them less effective yeah yeah the
senses are sorry too and she was so
detecting over-optimization
in the images aimed at myself yeah I
mean I feel like over-optimization could
at least in principle be diagonalized
against you also like whatever your
method for detecting over-optimization
I'm not actually sure about this but it
feels like whatever you're up your your
method for detecting over-optimization
you can kind of be find things aerial to
that an arbitrary intelligence
adversary' there's nothing I can do it's
not actually obvious to me it might be
that like the problem of detecting over
optimization is like an easier problem
than detecting a panda in the sense that
like you might actually be able to be
robust and you're detecting
over-optimization thing but it was in
this it's in this kind of area that I
was thinking and that that this way of
thinking directed me to yeah
should I like alternate questions or
somebody or something should I like
every once in a while have somebody else
jump in well no one has raised their
hands I have some questions but they're
probably of poor quality so please go
ahead just okay just keep talking until
somebody jumps in with a question on the
chat so I have something to add I think
which is that for current like for the
AIS that exists today adversarial
example do look different they are
detectable and yeah I don't I've
yeah then you can take from that what
you want if you believe that that will
work on like higher intelligence instead
it's an area it's an area I think worth
investigating as to your adversary gets
smarter as the AI itself get smarter is
there who do we expect to win in this
cut back I say I haven't investigated
these were just avenues that were
suggested to me but it does so there are
papers on this I haven't read the papers
I have read the alignment from
newsletter that summarizes the papers
but that's yeah I know there exist
papers who look at like how our
adversarial example different and you
can sort of imagine it as okay there's
image space which is like highly
multi-dimensional because it's like
every pixel has three dimensions because
three colors and then there is a like
the manifold that is embedded in this
space which is like things that are
actually what photographs end up looking
like and then the adversarial examples
are not on this matter from there sort
of off there like outside how real
pictures actually look like in a way
that is not detectable for humans but is
detectable by a eyes oh I mean at the
meta level this the photo that's
supposed to be a given looks like a
panda and we can tell that and we aren't
using high-level processing to do that
so there is some high there features
that we don't detect but the AI is
actually using in its classifications
and those high level features do look
like a given yes but what I mean is the
fact that we can see a panda and tells
me that there's something different
about this image and I fully expect that
those ways of getting the AI to detect
it I've seen some of your papers but I'm
just saying in principle we expect that
AI should be able to detect it I haven't
really seen anything about the sort
diagonalization argument you could say
that ganz are a little bit like that
maybe that just so now get meso
gangsters or general adversarial
constructions might be like that maybe
the Nash equilibrium or the final thing
of that you get something that is not as
good as a perfect classifier but and is
not completely immune to adversarial
examples but is hard to fool and it's
decent so I wanna I want to draw
attention to something which I think
might be a disagreement that I have with
you but I'm not actually sure partially
because I'm not actually sure what I
think and I think I like I tried to
point out before but I think I maybe
failed to point at it so concretely I
want to point at the example where like
I have a utility function but I have
some uncertainty so base basically like
the red door green there's a red red
door Blue Door red window type example
and I want to generalize this something
where it's like I have some like
distribution over utility functions that
I'm kind of uncertain about and I'm
trying to be conservative in some way
right or I guess that's not the main
problem it's like you're proposing this
class of solutions that looks like I
have this uncertainty over a space of
utility functions I want to be I want to
like be conservative about like doing
well according to all the utility
functions inside this space we can said
it's accurate could go to the end of
what you're saying and I want to simply
contrast this with approaches that look
like not pushing too hard in the
directions that might cause things to go
apart select my space of utility
functions like
is the reason why I have a space of
utility functions is because I've like
trained on some examples and and like
there are things that are kind of out of
distribution and I could be like well I
have uncertainty about these things that
are out of that are like out of
distribution of things that I've trained
on and in cases where I can't like ask
for more information I'm then going to
do something where I kind of try to be
conservative and maximize all the
utility functions a little bit
simultaneously or something like that
I don't know this feel like a fair
classification of the kind of thing that
you're trying to do well it's at least
given me enough to do a rambling answer
okay and then I want to contrast that
with the class of approaches that look
like just don't push so far out of the
kind of things that you were trained on
where it feels like one of them is like
let's try to like tabulate all of my
uncertainty and figure out all the
different ways in which I could be wrong
and make sure that I like cover all them
versus the other one is like just don't
stray too far or something which are
like two different ways of approaching
conservatism and I'm uncertain about two
things I'm uncertain about whether these
are like different and I'm uncertain
about whether I actually think that it's
better to try to approach ways that look
like don't strain too far as opposed to
like tabulate and all the uncertainty
but it seems to me like you're you're
pushing I'm reading you as as pushing
for doing something that's kind of like
keeping track of like uncertainty like
explicit uncertainty of a lot of
different things I might be trying to
optimize for if you have any comments on
that well so first of all on the
conservatism and going beyond this
of training environment there's a lot of
that in the post that I emailed to you a
few days ago that's a fundamental aspect
of it you could say so that's the bit
that's dealing with that conservatism
and when you need to be conservative and
why quantizers are not sufficient in my
view but that's that's still sort of
private so I won't go into that too much
but I'll contrast it with the other
aspect that you're saying so here is a
link to something I wrote which was when
good Harding is optimal and I basically
just to take the very simplest example
if you are a robot and you're hesitating
between going left and going right and
as soon as you've done left or right
it's over you get your reward or you
don't and you have fifty point one
percent probability that it's left and
forty nine point nine percent
probability that it's right this is a
pure good heart situation you just
choose the optimal policy which might be
disastrous compared with the other one
you just maximize utility it's a pure
good Harting and it's clearly the right
thing to do in that situation because of
the other things that it's linear you do
it once it's closed you can you can't
you can't correct it so that was the
thing that got me thinking so why do we
fear good Harting and we feel good
Harting I realized because we don't
expect that situation to be like that we
expect the situation to be different
like in that post I listed we expect
that there's going to be diminishing
returns that value is fragile
we expect that that's that's the biggest
of it this is why we don't like good
heart because if we just choose naively
a top option weeks basically if we
choose a top a top option we expect that
this will be disastrous it's the kind is
the reason why we feel a fear of good
Harting and then I was thinking well if
we add that information if we add the
fact we expect it to be diminishing
returns that we expect to have value
fragility that can all be done included
in a very Bayesian way across all the
utility functions and when we do that a
chunk of the good heart problem goes
away now in that post and probably in
some things I'm implying but maybe most
of the good heart problem goes away in
the post I emailed you you can read it
as sort of a tone of maybe very little
of it goes away or that's not the
fundamental problem but basically be so
the post that I've just sent here and
linked to the it adding more information
about why we feel good Harting removes a
chunk of the problem and I think that
this should be looked into and this if
there's still a good hardened problem
left after that then that's a separate
other problem that is maybe the moment
to be conservative on top of the gist
just being Bayesian at and I have
noticed that Linda has put her hand up
several times there yeah so I'm confused
when you said that the choosing between
the fifty one percent and ninety nine
press no okay so I'm don't think you
have the same definition of good as I do
so I just wanted to ask you how you
define a good heart
well I defined a good heart style
behavior
as naively picking a simple or a single
utility function and maximizing that far
beyond its area of applicability so you
mean that the AI picks its own or do you
mean when we pick it or both of the
cases well let me let okay let me
develop the example a little bit and to
show where we get the actual good
Harting suppose to that you could go to
the left you can go to the right you can
stay it goes on it goes on forever
there's a discount rate you can stay on
the left or you can stay on the right
and the one of your one of your utility
functions gives you a reward for the
logarithm of the number of times you
stay on the left and one gives you a
reward for the logarithm of the number
of times you stay on the right and
there's also a discount function given
this the optimal behavior is well go
left because that's the best stay there
for a certain amount of time go right
after certain runtime stay there for a
certain amount of time and fluctuate
back and forth according to the various
parameters here and this kind of
behavior seems very sensible and very
much what we would want the good heart
behavior for that situation would be go
left and stay there i picking naively
the best option and sticking with it so
if you want i distinguished a good
hardstyle behavior from a optimizing
behavior and what i noticed was that a
lot of the the good heart the problems
with good heart come because it
optimizes a narrow under-specified
utility function and that's a problem
but if we incorporate information such
as this is a narrow underspecified or
you don't have enough information and
our reward functions are diminishing
returns and the values of fragile and
then say okay given this information
naively maximize expected utility you
tend to get behaviors that are a lot
better so if you want yeah so I'm yeah I
I I'm not sure that actually agree with
like calling the thing you're calling
good heart good heart but it feels to me
like there's a sense in which like I
don't know we have some proxy objective
and then we have some like true true
objective we have some proxy objectives
and the proxy objective is noisy in some
way and if we're like in this situation
there's like kind of two paths forward
or like you want to do some combination
of these probably but there's like two
paths forward one is like figure out
what information we're lacking and gain
more of that so that we like figure out
the way in which like the things might
be diverged and like put that
information in so that they converge
more and then there's also like
generically try to figure out ways to
make it such that in spite of the fact
that our thing is diverged we still
don't things still don't end up badly
yeah so the the prophecy I think with
when you mentioned the proxy I can
phrase what I was saying better if we
have a proxy reward and we know that it
is that there is uncertainty about it
sorry what do you mean by there is
uncertainty about it you mean we know
that it's
we know it's a proxy okay you know it's
a proxy and maybe we have some idea how
it might relate to the real reward but
we know it's a proxy
yep then the I'll define the good
hearting behavior as naively máxima
maximize the proxy without caring about
yeah
just just maximize the proxy would be
good harder than that situation and what
I was saying is that in many situations
with the kind of utility functions that
we use in tall boy examples that is the
optimal thing to do but if if we then
move to more realistic utility functions
incorporating our judgments about the
various things they're talking about
then that becomes the wrong thing to do
however if we incorporate that knowledge
into the algorithm so it's it has the
proxy but it knows that the proxy
derives in this way and this is the for
shape of its uncertainty then so
maximize the proxy would be good Harting
and really bad maximizing the expected
utility with the proper form of the
uncertainty seems a lot better what is a
lot better so I think there was a
confusion conceptually between the two
between yeah so yeah I don't know if
people were confused but I definitely
was that good Harting was or that if you
want maximizing expected utility was the
same thing as good Harting was the same
thing as maximizing the proxy and that
these are distinct and that yeah yeah so
yeah I'm like I'm really curious about
this there's there's something where
there's being able to look at a
distribution being able to so yeah I'm
sitting here there's a true value I
don't know what it is I have two objects
I have this proxy value and I have a
distribution over the true value value
is just like the average yeah let's say
the proxy value is the most likely sure
let's say that proxy value yeah
approximately is the most likely and
wait so does does what you're
recommending like equates to optimize
the average as opposed to optimize the
like five does your thing corresponds to
pay it into the distribution optimize
the average as opposed to optimizing the
most likely or basically well yes for
some if you take average as weighted sum
of utility functions weighted by the
probability yeah if you want the the
shape of the plausible utility function
is not a ball it's not a ball around the
proxy it's it has a different structure
would you hi the average is not the same
as the proxy the mean and the mode are
the mean and the mode are different it's
the easiest way of saying it potentially
very different yeah
yeah so there's another thing that I
felt like I felt like you were saying
and maybe maybe you weren't which was
something that was like I don't know I
feel like there's something to being
aware of my own foul ability that is not
just like averaging over the the things
where you're like like there's some sort
of like being aware that like things
might actually be diagonalizing against
me or something like that were you
trying to point on something like that
too or or not I agreed that it's
potentially this case but what I wanted
to point out in the post I've sent to
everyone here and other civil ones is
actually just being Bayesian naively but
doing it properly gets you a surprising
distance it's plausible but it won't get
us the whole distance as I said how to
look at the one I emailed I haven't I
haven't looked at the one you emailed
yet so I might accidentally say things
that are in there
don't worry about it it's my sort of big
the thing I've been working on during
all of lockdown but putting that aside
the thing is that you're doing the
Bayesian stuff properly seems to get us
a surprising amount of the distance and
then of course yet there's various
conservative things that you can apply
on top of that if we feel like it but
I'm not doing that yet because I I
wanted to see how far the naive Bayesian
approach with proper uncertainty got us
can I answer the two questions that have
popped up in the chat I'm going to give
me time to think
okay um so from Ricardo everyone can
read this I won't need to repeat the
question so for Ricardo Bob Pato I yes
the question is this is not just change
the problem we have
good ideas about how this uncertainty is
shaped we have that's the point of why
we fear a good heart is because we know
that our values are complex for example
but that there is diminishing returns
that there are the humans are fallible
and can be corrupted this information is
not present in standard good heart
problems but now we can't put it in as
in terms of uncertainty over the proxy
so it changes the problem I wouldn't say
it just changes the problem and for the
question from Roland Philip Cass can you
propose I think it's because well I can
only speak for myself because I've been
working with good heart problems for
years and I didn't notice anything like
this until recently so but I think we
are too focused on the human version of
the good heart problem where the human
is antagonist the principal-agent
problem is basically what it is and
they're the agent is antagonistic or at
least misaligned with the with the
principle and here it's basically can
the principle specify in enough detail
to not leave any wiggle room for the for
the agent the principle cannot specify
something like well I'm a bit unsure it
might be the left one or it might be the
right one think a bit longer about what
I really value in this way and you'll
come to the right conclusion and that
will never work with a human because all
that they need to do is come up with a
plausible sounding justification for why
whatever was the right one but if you're
dealing with an AI then you can say
things like that if you specify well
what you mean you can allow
a superior intelligence to figure stuff
out about your own values which you
can't do in the standard good heart
problem so you notice that this is
touching upon thinking as if I were a
well-intentioned AI kind of thinking and
that's I think that was the one of the
key things is that in the AI version of
the good heart problem we we can have
the agents being much smarter than the
principal and figuring stuff out but the
principal doesn't know and can't measure
as long as it's well specified so I'm
inclined okay so I'm inclined to say
that the if if it were the case that we
could specify a distribution over a over
a utility functions such that like our
true utility function has non-trivial
probability we would have solved almost
all the value alignment problem like
almost all of the part of alignment that
is private specify values or something
like that and so I kind of feel like I
mean what we working with distributions
that have like a true value inside them
what yours what you're saying is
trivially easy to do just do all
possible reward functions or value
functions in existence which some weight
okay so I if we if we average over all
possible value functions with some
weight well I get I guess I'm trying to
say something like it feels like
examples in which like there's like a
small number of things feel like they
might be misleading yet the the but the
thing the thing that I'm hoping is that
we can break symmetry the reason why
averaging over
all possible utility functions is
doesn't work it's because there's every
utility function has an antagonistic
there's you and there's - you as long as
they're both there with similar
probabilities they might as well not be
there you can just take them both out
when you're averaging but can we break
the symmetry even what I noticed a long
time ago even knowing there is a good
hard problem
this slices the space in half half of
the utility functions do not have a good
heart problem they have the opposite of
a good heart problem but these are not
these are nothing like the ones that
that they prefer that you maximize the
proxy rather than the average okay and
so just knowing that there's a good
heart problem we've sliced away half of
them which is nothing but at least it
shows the break in symmetry and the more
of the stuff the meta stuff that we know
about ourselves that we add the more
symmetry we break so if you want to take
the trivial thing which is every utility
function is there this is terrible
because when you average it out you get
nothing or you get something absurdly
simple and then start slicing away by
adding our meta knowledge and keeping
still keeping average but I think the
best process can go a long way yeah it
basically feels like training or
something you have you could have like
and now you start with like a big prior
over all the possible utility function
and then you could ask a bunch of
questions that are like you before this
world of this world and each of those
questions like cut your space in half or
some
like that and training which
distinguishes a utility from its
negative is a different type of training
for one that doesn't tend to do that
yeah yeah I don't know I'm simply
thinking of the kind of training that is
like do you prefer a type of ruling
things out which are like which is like
I don't know playing guess who and
saying like do you prefer this one of
this one okay we're gonna cut half of
the utility functions out and repeat
this until you're left well I'd more be
interested in things that sort of cuts
between diminishing returns and
increasing returns for example because
increasing returns messed up things when
you average them you could say do you
prefer a or do you prefer a X percent
chance of beat and a and then one more
chance but see and you can ask different
lottery questions in order to cut things
in half yeah I did I think those are
more the things that I'd be looking for
the other stuff is good too of course
but they less include our meta knowledge
yeah so I mean I basically just like
absolutely agree that like working with
the average is probably going to end up
better than working with the most
working with the mean is just gonna end
up a lot better than working with mode
there might be some places where like
working with the mode is more tractable
than working it to me but I kind of
expect that like you collect a bunch of
data like this and the mean still isn't
great oh yeah as a silly example add
values are fragile you can do this with
a smooth min and we expect human values
to have at least this level of
complexity now ignoring for the moment
negative outcomes sort of people being
tortured and stuff like that
if we can avoid those as well
then it seems that we have to find a
very wide class of things that contains
our preferred utility functions and that
a average of this Maximizer with huge
amounts of power will get a positive
result not not nearly comparable with
the amount of power that it has because
this is a small slice but yeah but
sufficiently but as I say I'm not Eve
this is confusing because what I'm
saying is kind of the opposite of what
is it then
the virtus should I be mailed to you or
not the opposite but a different
approach shall we say but I just to
repeat again I feel that going I think
that lots of people consider the good
hard problem think of the mode and the
mean are the same and the toy examples
that we have they are and I think
distinguishing these two is very
valuable and I think adding extra meta
knowledge shows that the mean can be
surprisingly good without even having to
add any quantizers or other methods and
that seems right to me
do people want me to say what I mean by
the opposite of the good heart problem
or have I explained that okay they be up
the opposite the good up run if you want
is when you prefer that the proxy the
mode be maximized rather than the mean
if you have increasing returns this is
the kind of thing that might happen let
me you always prefer them mean like if
you take the mean of functions that have
increasing returns that we just make you
go with the mode anyway
well let's let's do a sort of example
like the nails the the people's
commissariat for central nail production
has told you you need to maximize the
number of met nails and their proxy
function is pieces of pieces of steel
and you maximize that proxy and these
are terrible nails now there is the if
you call the proxy V and the actual nail
the genuine utility function u the if
there's a delta so let's see consider V
minus u minus V this is the utility
function on the other side if u is here
V is there this is the one that's on the
other side this is a weird sort of thing
and this utility fungus ACOG W I'm not
following
so this is U this is V there is another
utility function at which V is exactly
halfway between the two utility
functions can be added if you have a
scale assume we have a scale it can be
added they form a vector space so this
is U this is the other side of the
vector there's a W and this w this W is
something that benefits a lot this is
sort of like the utility that hates
nails and loves pieces of steel it it
would much
for the V as stated then say a 50/50
chance between it and the true you so if
you want you yeah so but this odd kind
of utility function notice how hard it
is for me to describe in human possible
terms because it makes no sense for a
human because it isn't a human I can
describe it in terms of vector space
it's easy this is not there but it makes
no sense for you actually you secretly
do not want true nails but you wants
more pieces of useless still or
something like that it makes no sense
for a human no but like the W is the
difference between the true utility or
whatever that is and the proxy whatever
that is
I'll need no it's not the difference
it's the other side it's let's see so it
would be V Plus V minus u v minus u is
the Delta between U and V W is the
negative Delta V Plus V minus u so okay
so it's it's it's so here okay now I see
what you mean by the other side but it's
sort of them it's defined in opposition
to the to you yes so no matter what the
true utility function is even if it's
something inhuman
this w is always going to be worse than
you worse well the point is that from
w's perspective it wants it prefers a
maximising of V rather than a 50-50
chance between U and W so it does not
want the mean which is 5050 between U
and W prefers that the
mode of the middle is maximized
Lisa w prefers w w prefers that V be
maximized rather than the agent be
uncertain between U and W sorry like why
why is the mean of U and W not we
because like if you're defining W in a
way in which it lead abuse it it is
possible I am making a mistake in what I
am saying here I okay I do I have the
examples very clearly to hand all my
posts all I know is good heart it's
possible sorry that is let me find this
the difference in the meantime I can see
that if Scott would like to break in and
stop this question because he feels he
has a better question then Scott by all
means go ahead at the full version of
what I was saying is here I was not
expressing it correctly but just as
there are things that fear good hearts
there are things that auntie fear good
heart in the same way every utility
function of course would prefer that it
be maximized but the kind of ones that
particularly feel good feel good heart
are compensated by other ones that anti
fear it and the example there is more
coherence than what I was rambling and I
got the definition of W wrong - I have
you thought about the consequences of
the fact that your ear so you're
averaging over utility functions
you're not averaging over utility
functions upto affine transformation
which means that you're gonna have a
bunch of different copies of each
utility function of up to a fine
transformation in your class I don't
know if you have anything you want to
say related to that that the
normalization of of utility functions is
a very hard problem that I have several
posts on it showing how hard it is and
that maybe here we have more of a
principled normalization like we can
sort of compare with things like I like
ice cream and this TV show this much and
then we can at least rank the different
utilities compared with that possibly
yeah here we have a kind of principled
point that we might want to like all the
0 which is like choose a random utility
function from your class and optimize it
which like that gives you a a utility
for each utility function and we use
that as the 0.44 realization I think we
need to with the zero point is not
enough to normalization we need another
appointment like my um have you heard of
the population ethics which is
exponential in the number of people now
would you give that any probability
whatsoever I I don't I don't I can
simply don't believe in unbounded
utility functions but the the sort of
issue with that is that if you give it
any probability whatsoever it doesn't
have to be unbounded if you give it
anything but the tiniest of
probabilities then it it'll dominate
average utilitarianism it'll dominate
totally to terrorism
it'll dominate any theory of population
ethics of course this I think is
ridiculous so you need
to penalize it by its span that's the
sort of mid max normalization but yet I
feel you have to normalize all these
utility functions anyway to prevent that
kind of bad behavior so I want to give
it I want to give a concrete alternative
proposal to optimizing the optimizing
the average of a spatial distributions
of utility functions so I have a
distribution of utility functions I will
define a zero point as I'll define the
zero point as choose a random utility
function and optimize it okay and then
instead of maximizing the average
utility I want to maximize the product
of the differences between like I want
to choose something that is better than
this I want to choose a Pareto
improvement of this mm-hmm why
maximizing a product of the differences
in utility let's say we have a finite
collection but we can also do this with
them so basically a Nash bargaining it
basically Nash Mart Nash bargaining yeah
so I'm curious if you have cash flops
about what about like Nash bargaining
versus Bayesian choice for maximizing
this painfully you know you're gonna
copy another personal character I could
but let me try and I do like the fact
I've reached the stage in my career
where I can answer most questions with
links but but what I was no I'm trying
to answer this I don't like the Nash
bargaining equilibrium because it
doesn't feel natural eat it's there
there might be some messy stuff around
zero yeah I wanna I wanna put thing I'm
proposing is not exactly an iceberg
because I've defined zero in kind of a
weird way doesn't matter once to do the
Nash bargaining you need to describe you
define a zero and oh I I thought the
Nash bargaining explicitly had had zero
as like the threat points or something
you know you've just defined a different
threat point if you want the okay so
this has certain failures we can come up
with examples where it fails it's
generally to do with things that only
maximize a tiny bit unless you give all
your effort to it and then the one over
10 trillion thing dominates the product
kind of thing and yeah so yeah I think
that's a problem I I'd go for one of the
other ones like what what are the ones I
came up with so you can keep your zero
point you keep your you define a utopia
point where every utility gets its
maximum expected value that's not a
point that can ever be reached but you
normalize those to zero and one and you
Pareto improve on that that was what was
it my mean worth nothing bargaining
solution something anyway something I
came up with some time ago
the okay so this thing is probably
exactly the same thing as maximizing the
mean yeah either way I'm not sure oh boy
for
Oh Joe you know what you did you both
with to attend normalizations so if you
fix that normalization so yeah if you
fix missus this is the Mean Max
normalization
why don't I recognize it the mean action
is pick one of them at random maybe
according to probability the max is
maximized only this one normalize each
utility function to be zero for the mean
policy one for the backs policy then
given that normalization just normalize
the mean this is now what I've just
described there yeah so are we are you
know sorry for everybody else who hasn't
necessarily followed my back catalogs
from several years ago are you yeah sue
I don't know I was originally
interpreting you is just like not
working up to a font transformation
right like when you're originally
proposing the thing you're just like
you're just like take the average of the
utility functions and utility functions
of those utility functions their utility
functions up affine transformation they
aren't well yeah yeah the normalization
question is a difficult one that I think
is separate yeah it's maybe it's not
separate but it would you don't think at
least for my methods or for their taking
the mean you have to solve the
normalization somehow separately before
you do this or while doing this or
whatever more links okay I'm listening
to questions while I go searching for
link yeah I'm assuming that if there are
other questions they'd pop up in the in
the text chat
um I mean I'm listening to what you're
saying while I go searching for links to
answer that question I yeah sooo yeah
would you describe your position right
now as roughly we like we should just
like kind of ignore the normalization
question and just like we have a
distribution over utility functions
that's coming from I guess is this
question like where my utility function
I mean actually assignment of numbers
like distribution over bounded utility
functions that are inside some interval
it is it is possible that some of the
other methods like the quantizer methods
maybe have their own inbuilt
normalization which the main method does
not which may be a point in their favor
but for the moments i am for at least
I'm saying the mean and the
normalization are separate questions
here oh I don't know why you're saying
it seems like to take you mean you don't
have to normalize you oh well okay yeah
sorry I'm imagining like I have a space
of distributions of utility functions on
where utility is between zero and one
and I might have the same util the same
utility function up I'm a fine
transformation several times inside this
distribution and that's that's what I'm
interpreting a proposal to be which is
just like completely aside from any
question of normalization well if you
are if you take if you take actual
utility function representatives as in
actual functions and you have a
distribution over that then you've
already kind of normalized yeah
let's see my and here is the mean worth
optimization mean worth bargaining
solution kind of thing when I imagine
trying to use this this proposal and
still being like killed by good heart
it's like because I'm like okay I'm
going to like specify this distribution
over all the things and I just like
entirely miss a dimension hmm
and so when you're saying like like
enumerate all the utility functions and
give them all some weight like by
enumerate all the utility functions your
enumerate all the utility functions
within some space and like you could
imagine having utility function they
kind of like misses that space this is
where the post of my email becomes the
most relevant and I can't really answer
you without referencing it okay
I don't I don't think of it as there's a
distribution over these utility
functions which might miss a dimension
but what happens when we see evidence
that we've missed a dimension how do we
extend our our current model but this
brings us way astray from this but the
from this conversation yeah I mean I
don't know I wouldn't call it too astray
because I feel like when I hear the
proposal the reason why I'm feel doom
it's because of the thing I just said or
something well okay well let's think a
little bit more formally if you
a dimension missing and we use another
method like let's say quant Eliezer if
we're quantizing and there's a dimension
missing
we still got big problems if with most
most of the conservative methods have
similar problems except you could say
that as a conservative method at the
time and kind of keep things close to
the training environment that McKenna
catches these extra dimensions without
realizing it but it seems the back is
something you could do as a prior as
well so what if we're missing a
dimension can we capture it in other
methods that we wouldn't do it in mean
so why doesn't quantizer take like what
so the safety measure in quantifier is
basically don't stray too far from the
original action distribution which is
assumed to be safe
so what why doesn't this take care of
extra dimensions let's think
so the you rank the quantizer so we have
the proxy we take policies that
maximized it up to some extent you're
right if we were very careful with the
quantizer we may we may implicitly
capture the extra dimensions but there
is no so yes so in that way it is
superior but there is no clear there's
no clear we don't the queue in the
quantizer we don't know what it means
surprisingly I have a post
on that and I'm looking for it but yes
you are correct it does seem that the
quantizer can capture extra dimensions
is a sort of kid I'm claiming that it
just naturally does like Oh actually no
sorry I'm revising that no it doesn't
okay because the policies that are say
50% effective to do the proxy there are
some policies that will capture this
extra dimensions and are certain there
were an T on these acted this extra
dimensions if the dimension is not
captured in the proxy at all yes so you
know it seems that my default weight my
default like understanding of a
quantizer is you like take some learned
distribution of things that humans would
do and you like select a like top one
percentile from that distribution is
that not what other people mean here I
mean I okay I was under the impression
that as defined by Jessica it was not
that that might be the case I'm pretty
confident that it is that well I would
like to be able to answer but it's
taking forever I think because of the
video can you think you defined it as
that in the post yes okay the posts that
I'm looking for I defined it in that way
and in the post that I'm looking for I
linked to Jessica's original thing so
either I misread her thing or there are
multiple definitions of quantizers
floating around yeah so in general you
seem to use concept like words slightly
different than I'm used to
sorry and it's sort of fine because you
explained if you explain what you mean
and but I also think that you're missing
out on things other people have done
because you don't notice
like usually terrible at rich literature
reviews yeah like this the idea of using
uncertainty over utility function as a
safety feature is it's a really good
idea that I've seen around for a while
and I've posted a link to like where I
first came across it in the shot mmm-hmm
I'm yeah okay give me my phone and I'm
giving up on me I'm giving up on the
Wi-Fi and I'm just going to but that is
what I I mean I'm I'm reasonably
confident that that was what the
quantizer was because I looked it up
specifically for that post I think that
the most likely scenario is that there
were multiple versions of quantizer in
decibels yeah that's very possible and
like the one that went into the post is
probably the one that you saw and
because I worked with Jessica I probably
like saw other ones okay so yeah so the
like I'm misremembering or something
that's several people that I talked to
all have the wrong definition of
quantizer so the kind of thing that
you're talking about I've considered
sort of extension of apprenticeship
learning kind of things okay it's a
little s wrong that there's a problem
because my phone which is on on the
network not on Wi-Fi I had a problem
loading goes wrong earlier today but the
line and forum seems to be up okay let's
try a lineman form I feel okay is this
yeah so let's refocus on the thing the
question is
my current question is what do you mean
by quantizer because it's not what I
mean a quantizer is an agent that went
is that I'm reading for me that returns
a random action in the top queue
proportion of some base distribution
over action sorted by the expected
utility achieved if that action is
executed yeah so yeah I usually think of
the base distribution as like a learned
distribution of what humans do okay I I
think of the base distribution as the
although all the policies you can think
of or all the policies that they are can
think of yeah the version I read the
base distribution was supposed to be
something that's reasonably safe to draw
a random action from okay some human
behavior okay something that it's safe
to draw a random but not an optimized
action from okay yes then this if it's
safe to draw a random action but not an
optimized action from then this is
intrinsically this controls for extra
dimensions that we might not not have
considered if we are convinced that it
is safe to draw a random action for it
which from human style behavior is
definitely safe in the short term I'm
not entirely sure if it's safe in the
long term but okay so yeah that seems to
be arguments that quantizers do have
advantages that's wrong amazing idea
okay then that is learning and how
should I
how conservative should the part relax
max I just be okay so this is the post
that I was referring to and I think the
points of the post is still valid even
with the yeah not quite as valid with a
safer based distribution then the point
was if we know that the optimal one is
wrong and we know that the zero is also
wrong because it doesn't help
what are we how do we know that it's
safe well as you say going for say if if
if we could draw a hundred times at
random and expect to not go disastrously
wrong then 99 percentile is probably
safe in a similar way I very powerful we
get we get 99 percentile actions like
one out of every hundred actions we take
you could just do with policies anyway
sorry I'm I'm starting to fade I fear
and I think the questions are getting a
little more wavy we call it well let's
check if anyone has a last question
especially a sort of simple
clarification question which are easy
okay I have a symbol what is the
difference between their will intention
di and an aligned yeah
they well into the aligned AI is an
actual objective a well intentioned AI
is a thought experiment that allows me
to explore new ways of possibly getting
to an aligned AI the yeah a well
intentioned a I as I've described it
might still kill the universe
it's if you it's focusing on a different
a different approach to the air problems
or a different class of the AI problems
maybe for meso optimizers the difference
makes sense but not for general they
eyes I don't want to erupt I know those
your question but afterwards class the
naive question go ahead
wait Soren did that answer your question
thank you so you both work along with AD
me or with Mary I have a question about
post I think it was like it was from one
of you tapped you'd cast these talks
about AI lineman being like a
cryptographic rocket problem and when it
I think it went on to describe the
process by which you try to align it so
you began your talk today doctor I'm
strong with a discussion of like we
don't have an AI we don't have the code
you can't doesn't quite work the way one
would hope it would do you actually have
these things so instead you try to
introduce a characterization or a
definition or at least Miri has
discussed introducing the tools that
would help one get ease with what they
called like a space plane to the moon
and they described it as like the
principles of first derivatives and
second derivatives towards getting us
towards rocket science so how do you
guys view AI alignment in terms of given
that we don't have any any AI now do you
mostly view it as a problem of like
getting the tools necessary for it or do
you go along the path of life let's
define what super intelligence is even
if it's probably attractable and then
let's go from there I mean we're looking
at your work but and some others but I'm
having trouble recognizing the
difference between the two approaches
and I don't know if that's coherent
enough I I ramble in many different
directions and finds many different
ideas
because that's how my brain is wired
maybe one I always I was tend to make
the assumption that it is a super
intelligence and that we can we can't
constrain its knowledge except if it's
in a very specific cryptographic away
the reason being that most solutions
that work for that kind of anything most
solutions not all but most solutions
that work for super intelligence work
from gamma-ray eyes as well and Ally I
can answer your question with saying I
can't answer your question right now and
thank you very much better and sorry dr.
Gerber and do you anything that do you
have any response to that question or
rambling ha yeah I think I think I miss
parva-- a few things I think are
adjacent that I want to say well say is
that like I feel like I should probably
disclaim that I don't know we're talking
a lot about things that are trying to
like figure out how to get from utility
functions into a eyes which like I kind
of I don't know like like I wrote this
guitar post another than facts which is
like a very general thing it's like
generally knocks like these the the
sub-problem that I'm thinking about most
of the times where like I'm mostly
thinking about the problem such that if
I can have a thing that is trying to
like optimize reliably any hard
objective I could write down whatsoever
like this is like the the part of the
problem that I am like I have my sight
on right now we're like there's there's
value in the like reliably type thing
like be able to direct something at
doing a thing but I kind of feel like
from my point of view if we could make a
paper
maximizer only we would have solved a
large portion or problem and so a lot of
this is like out of the space that I'm
like normally thinking about and then
another thing I want to say possibly
addition to your thing and possibly not
I don't know it's definitely definitely
is the case that the way in which I view
most of most of my time working on
Highline related things is like trying
to build up foundations about having
like the directs like without like
having the direct use case like I have
like a few different like these cases or
one to like resolve some uncertainty for
some confusion around some area but like
I'm like trying to work with abstract
enough objects that no specific like use
case that I have in mind will be like
the only plane which are movies or
something and so like it does feel to me
a lot more like trying to like develop
calculus than trying to like build a
rocket in the sense that like it feels
like it's like a thing that you need to
do before building a rocket and I feel
like it's hard to be able to look at the
problem of building a rocket and figure
out what the right type of calculus to
develop is without actively working on
building the rocket or something but and
so maybe I'm doing it very badly I don't
know but I ie
I do think that like most of my time is
like most of the like analogies little
draw and stuff you can try to think
about them as being directly about the
ISIS that's then it might like miss the
friend point ad or something okay thank
you very much and then I am one more but
i blew some other people might have
questions to it so i wanna you know them
in case they do if not then i'll ask how
do you as a researcher at mary how do
you feel about the choice to not publish
any research so i believe one of the
last publications that I saw from Mary
was the logical adductors paper that you
you are the primary author on how has
that changed
your production of research hasn't been
a net benefit to you personally I like I
knew inside view of this but it's weird
from an outsider perspective from
someone just looking at Mary's work for
quite a while and then see a complete
like hiatus on that in terms of like net
benefit on me personally there are
definitely ways in which it's like good
and bad like two ways in which like it's
bad is that it like makes collaboration
harder another way that it's bad is that
like sometimes I would want to use
writing stuff up as a motivation for
myself to formalize things more or
something it's like I have all these
ideas in my head and like the process of
like writing them down is like a
motivation to do a certain type of
thinking that I have to like externally
motivate myself in a different way to do
or something like that but I a large
part of I know that there's there's a
point that I want to make about
publishing which is that I think that it
like depends a lot on whether you expect
like I kind of expect the kind of stuff
I'm working on is has like a chance of
like developing something really cool
and a chance of not and like the
incremental progress is not as exciting
or something versus like if I was
working in a field that was more close
to a lot of other people then like the
incremental progress would matter like
I'd have to get my incremental progress
out so that other people would get more
incremental progress on it and it can
like build up through a bunch of like
individual like small jumps or something
versus like I think that I am like
trying to like do some like very and
seeking trying to find some like large
jumps in in the way that we're
conceptually thinking about the problems
such that
the costs that I pay by working
privately until I have something that
seems really interesting and then
deciding whether I want to share it and
then put it in the work to sharing it is
like not that much of a cost because
it's like the sharing of the incremental
stuff doesn't actually help that much
because there aren't like a lot of other
people that are working on the exact
same things that I'm working on in in
the same way versus if I was in like ml
there would be a lot of people who are
working on very very similar things and
very more way it would be a lot more
benefit sharing I'm obviously a lot more
conventionally academic and also some of
the way I work is generating random
ideas and there it's much better to sort
of toss them out into a less wrong post
and move on and see if other people
develop them so yeah I probably my
reckon is III disagree with the Mary's
decision at first order but I also trust
that they've thought it through at
second order
well thank you both for your time and
thanks for answering the questions I
sincerely appreciate it okay thank you I
would like to just say two more things
and very much so thank you for that and
then the the second thing is just in the
beginning which said that there were
actually no implementation of any of
this and actually that turned out to be
wrong in that joon-kyu has made a actual
explicit implementation of a utility
function for a superintelligence and
also an implementation of meta semantics
that's been published at mr. ethical AI
and next week I will try to give a
presentation on that in
reading group on Tuesday at this sad
time so I hope to see some of you there
hey that should be all see you next week
all right I I'm away from a computer so
is there any way you could like like hit
command and copy comments or no because
there were a lot of links shared and
some questions yeah and I don't know a
good way of saving any of this for


Is AI Safety a Pascal's Mugging?

hi today we're going to talk about
Pascal's wager and Pascal's mugging this
is necessarily going to touch on
religious topics pretty heavily actually
and I'm just gonna say at the beginning
of the video that personally I don't
believe that any god or gods have ever
existed and I'm not going to pretend
otherwise in this video so be forewarned
if that's likely to bother you so as you
may know Pascal was a 17th century
philosopher who was interested in
amongst other things the question of the
existence of the Christian God various
philosophers at the time were arguing
that God didn't exist and there was a
lot of discussion going on about the
various kinds of evidence for and
against God in the world but there's
this thing that's quite common when
people think about religious questions
where it feels sort of unsatisfying to
talk about worldly evidence as if you
were considering some everyday question
there's a feeling that these
supernatural concepts are very grand and
mysterious they're special and so just
straightforwardly considering the
evidence for and against God is not the
right way to do things this is of course
encouraged by religious thinking the
idea that some hypotheses aren't subject
to the usual rules of evidence and logic
is pretty appealing if you want to
advocate for an idea that doesn't fare
very well by those standards
I suspect Pascal may have felt something
like that because his position was that
reason has nothing to say about the
question of whether or not God exists
it's sort of an unknowable thing and
instead he proposed that we should make
a wager we should think about it like
this there are two possibilities either
the Christian God exists or he doesn't
and reason gives us no way to choose
between those we have two options
available to us either we can live
according to God's laws and act as
though we believe or we can not do that
so we have a sort of payoff matrix here
with four sections if God exists and we
believe in him then we get infinite
reward in heaven if God exists and we
don't believe in him we get infinite
punishment in hell if God doesn't exist
and we believe in him then we pay some
costs you know there are some rules we
have to follow and so on and if he
doesn't exist and we don't believe in
him then maybe we get a few perks from
not believing like having a lion on
Sundays and being right Pascal's point
is that this payoff matrix is completely
dominated by the case in which God
exists because we're talking about
infinite rewards and infinite
punishments as opposed to the other case
with these very finite costs
benefits so regardless of the evidence
Pascal argues we should believe in God
or at least act like it because it's
just the sensible bet to make this is
really kind of a nice argument from
Pascal's perspective because it doesn't
need evidence at all no finite earthly
evidence can outweigh infinite
supernatural payoffs it feels like the
kind of clean abstract reasoning that
you're supposed to do when thinking
about the supernatural all of this hard
work looking at history and psychology
and science and trying to figure out
where the ideas of religion come from
and whether our world seems like the
kind of world with a God in it it's
long-winded confusing it's it's just
messy but here we just have a clean
argument that says we should believe in
God or at least act like it and that
seems very neat no evidence required so
consider now Pascal is walking down the
street and he's stopped by a shady
looking man who says give me a wallet I
would prefer not to do you even have a
weapon no UK laws are very strict about
that but I don't need one because I'm
God your God yep I'm God and
Christianity got a lot of things wrong
about me my forgiving nature my infinite
mercy and so on but the infinite torture
thing is legit and if you don't give me
your wallet right now I will torture you
for eternity in the afterlife now if
you're Pascal you're in kind of a
difficult situation because the fact
that it seems very unlikely that this
murder actually is God is not meant to
be part of your calculation your
argument is one a pure logic it works
independently of any evidence you didn't
need to look for evidence of the
Christian God and you don't need to look
for evidence that this mother is God
either so you kind of have to give him
your wallet and now you're really in
trouble because of course when this gets
out there's gonna be a line around the
block of lsaps deities asking for
handouts how are you going to deal with
this endless stream of fizzy gods well
one thing you can do is you can play the
muggers off against one another you can
bring in two of them and say listen you
say that you're going to torture me
forever if I don't give you my wallet
and you say the same thing I only have
one wallet so it looks like whatever I
do I'm going to be tortured forever by
somebody and if I'm going to be
infinitely tortured anyway well two
times infinity is still just infinity so
I may as well hang on to the wallet now
get the hell out of my house all right
next to no doubt these self-proclaimed
deities may try to argue that they have
some reason why they are in fact a real
deity
this other mugger is just a random guy
who's pretending but that's all worldly
evidence which you've decided isn't
required for your argument and the
muggers don't really want you to become
interested in evidence because well the
evidence points very strongly towards
none of them being real gods so this is
a better position to be in you're still
spending a lot of your time arguing with
charlatans but at least you still have
your wallet and you don't actually have
to pair them up against each other right
you can just make up a deity when
someone comes in pretending to be a god
you can say oh well there's this other
God who demands exactly the opposite
thing from you a goddess actually and
she's very powerful but she goes to a
different school
you would know her really yeah she lives
in Canada she's only present obviously
but she lives in Canada anyway she says
that I'm not to give you the wallet and
if I do then she'll torture me forever
in the afterlife I think yeah so you can
solve a lot of these problems by
inventing gods arbitrarily and of course
this applies just as well to the
original version of Pascal's wager
because although it's implied that this
payoff matrix has enumerated all of the
possibilities and in a sense it has the
Christian God either exists or it
doesn't nonetheless those may not be the
only things that effect the payoffs for
any given God you can take the God down
flip it and reverse it and say what
about ante God who wants me to do the
exact opposite and promises the exact
opposite consequences now you can see
that they cancel out somebody who's
arguing for the existence of the first
God might say okay but this anti God is
just made up which I mean yeah it is
it's true that the situation isn't
really symmetrical someone might think
God is more likely than anti God because
of evidence from the Bible and there's
no such thing as the anti Bible and so
on the point is though we're back to
talking about the evidence that's really
the problem I have with Pascal's wager
the way it uses infinite costs and
infinite benefits to completely override
our necessarily finite evidence but what
if the costs and benefits aren't
infinite just very very large that ends
up being a much more interesting
question on one end of the scale we can
easily name numbers so large that no
amount of evidence anyone could ever
actually gather in their lifetime could
have an impact on the conclusion we can
specify costs and benefits that are
technically finite but that still feel
very much like a Pascal's wager on the
other end of the scale if you come
across a bet that pays out 10
two one on an event with a probability
of one in a hundred that's a very good
bet to take someone could complain that
it's a Pascal's wager to bet on an
unlikely outcome just because the payoff
is so high but if you take enough bets
like that you're sure to become very
rich in the same way if there's a button
which has a one-in-a-million chance of
starting a global thermonuclear war it's
still worth expending significant
resources to stop that button being
pressed one in a million isn't much but
the cost of a nuclear war is really high
I don't think that's a Pascal's wager
either the difference seems to come in
somewhere in the gap between very small
probabilities of very large costs and
benefits and really extremely small
probabilities of near infinite costs and
benefits so why are we talking about
this what does this have to do with AI
safety well suppose somebody stops you
in the street and says hey if we ever
create powerful artificial general
intelligence then that will have a
tremendous impact in fact the future of
the whole of humanity hinges on it if we
get it right we could have human
flourishing for the rest of time if we
get it wrong we could have human
extinction or worse regardless of how
likely superhuman AGI is the potential
impact is so high that it makes AI
safety research tremendously important
so give me your wallet it's been claimed
by some that this is more or less what
AI safety as a field is doing this is
kind of an interesting point there's any
safety advocates are we victims of
Pascal's mugging or are we in fact
Pascal's miners ourselves well if people
were saying these AI risks may be
extremely unlikely but the consequences
of getting AI wrong are so huge that
it's worth spending a lot of resources
on regardless of the probabilities so we
don't even need to consider the evidence
well I would consider that to be a
Pascal's wager style bad argument but
what I actually hear is not that what I
hear is more like look we're not
completely sure about this it's quite
possible that we're wrong but
considering the enormity of what's at
stake it's definitely worth allocating
more resources to AI safety than we
currently are that sounds pretty similar
but that's mostly because natural
language is extremely vague when talking
about uncertainty there's an enormous
difference in the probabilities being
talked about in the same way if when you
talk to AI safety researchers they said
things like well I think the chance of
any of this ever being relevant are
really extremely tiny it seems more or
less impossible to me but I've decided
to work on it anyway
because the potential costs and benefits
are so unimaginably vast then yeah I'd
be a little concerned that they might be
victims of Pascal's mugging but when you
ask AI safety researchers they don't
think that the probability of their work
ever becoming relevant is very tiny
they don't necessarily think it's huge
either maybe not even more than 50% but
it's not so small that you have to rely
on the unimaginable vastness of the
consequences in order to make the
argument to borrow a metaphor from
Stuart Russell suppose you're part of a
team working on building a bridge and
you believe you've found a flaw in the
design that could cause the structure to
fail catastrophically maybe the disaster
would only happen if there's a very rare
combination of weather conditions and
there's only a one in a hundred chance
that those conditions will ever happen
during the course of the bridges
expected lifespan and further suppose
that you're not completely sure of your
calculations because this kind of thing
is complicated maybe you only give
yourself a 40% chance of being right
about this so you go to the civil
engineer in charge of the project and
you say I think there's a serious risk
with this bridge design do you think the
bridges gonna collapse probably not no
but I'm about 40 percent sure that
there's a design flaw which would give
this bridge a 1 in 100 chance of
catastrophic failure so you're telling
me that in the event of a scenario which
is very unlikely to happen the bridge
might collapse and you yourself admit
that you're more likely to be wrong than
right about this stop wasting my time
but if the bridge collapses it could
kill a lot of people I think this is a
Pascal's mugging don't try to get me to
ignore the low probabilities just by
threatening very large consequences
obviously that isn't what would happen
no civil engineer is going to accept a 1
in 250 chance of catastrophic failure
for a major piece of infrastructure
because civil engineers have a healthy
organizational culture around safety
what it comes down to again is the
difference between different levels of
improbability the chance of an AGI
catastrophe may not be very big but it's
much much larger than the chance that a
mugger is actually a god and what about
our anti-god tactic finding the opposite
risk does that still work like what if
we consider the possibility that there's
another opposite design flaw in the
bridge which might cause it to collapse
unless we don't spend extra time
evaluating the safety if that is not
what just look at the schematic with you
and what if working on AI safety
actually ends up making the risks worse
somehow I think this actually is worth
considering unintended consequences are
a real problem after all speaking
generally there's
clear argument that the future is very
important and that we're probably able
to have a very big impact on it but it's
hard to know for sure whether that
impact will be positive or negative for
any given course of action prediction is
very difficult as they say especially
about the future and the further into
the future we look the more difficult it
gets like imagine if you lived in the
year 1900 and you had some insight that
made you realize that nuclear weapons
were possible and nuclear war was a risk
you'd hope that you could use that
understanding to reduce the risk but it
would certainly be possible to make
things worse by accident in the case of
AI safety though I don't see that being
anywhere near as much of a concern we're
heading towards AI regardless and it
seems very unlikely that thinking about
safety would be more dangerous than not
thinking about safety it's definitely
possible to make things worse while
trying to make them better but you can't
avoid that by never trying to make
things better I guess my point is
there's just no getting around the messy
confusing complicated work of looking at
and thinking about the evidence any
argument that doesn't rely on the
evidence will work equally well whatever
the truth is so at the end of the day
that kind of thing isn't going to give
you an answer you have to just stare at
the bridge design and really think you
have to actually do the engineering and
that's something I'm trying to get
across with this channel you won't find
me saying never mind the evidence ai
safety is important because it could
have huge consequences what I do on this
channel is I try to show you some of the
evidence in some of the arguments and
let you think about the situation and
draw your own conclusions it can be
tricky and involved it requires some
thought but it has the advantage of
being the only thing that has any chance
of actually getting the right answer so
thanks for watching
[Music]
as my wonderful patrons will know the
alignment newsletter is a weekly
publication from Rowan char which I read
every week to stay up to date with
what's going on in here safety and now
I'm recording myself reading it out and
publishing that as the alignment
newsletter podcast it's aimed at
researchers so it's a fair bit more
technical than this channel but if
you're interested in getting 15 minutes
of AI safety news in your earholes each
week check the link in the description
I'm never going to put ads or sponsors
on that podcast and that's largely
thanks to my patrons in this video I'm
especially thanking Chris canal thank
you so much for your support Chris thank
you to all of my patrons and thank you
for watching I'll see you next time
[Music]
little costume changes


Reward Hacking: Concrete Problems in AI Safety Part 3

hi this video is part of a series
looking at the paper concrete problems
in AI safety I recommend you check out
the other videos in the series linked
below
but hopefully this video should still
make sense even if you haven't seen the
others also in case anyone is subscribed
to me and not to computer file for some
reason I recently did a video on the
computer file channel that's effectively
part of the same series that video
pretty much covers what I would say in a
video about the multi agent approaches
part of the paper so check that video
out if you haven't seen it linked in the
doobly-doo right today we're talking
about reward hacking so in the computer
file video I was talking about
reinforcement learning and the basics of
how that works you have an agent
interacting with an environment and
trying to maximize a reward so like your
agent might be pac-man and then the
environment would be the maze the
pac-man is in and the reward would be
the score of the game systems based on
this framework have demonstrated an
impressive ability to learn how to
tackle various problems and some people
are hopeful about this work eventually
developing into full general
intelligence but there are some
potential problems with using this kind
of approach to build powerful AI systems
and one of them is reward hacking so
let's say you've built a very powerful
reinforcement learning AI system and
you're testing it in Super Mario World
it can see the screen and take actions
by pressing buttons on the controller
and you've told it the address in memory
where the score is and set that as the
reward and what it does instead of
really playing the game is something
like this this is a video by sethbling
who is a legitimate wizard check out his
channel link in the doobly-doo
it's doing all of this weird looking
glitchy stuff and spin jumping all over
the place and also Mario is black now
anyway this doesn't look like the kind
of game playing behavior you're
expecting and you just kind of assume
that your AI is broken and then suddenly
the score part of memory is set to the
maximum possible value in the AI stops
entering inputs and just sits there
hang on ai what are you doing am i rock
no you're not wrong them are on because
it turns out that there are sequences of
inputs you can put into Super Mario
World that just totally breaks a game
and give you the ability to directly
edit basically any address in memory you
can use it to turn the game into flappy
bird if you want and your AI has just
used it to set its reward to maximum
this is reward hacking so what really
went wrong here
well you told it to maximize a value in
memory when what you really wanted was
for it to play the game the assumption
was that the only way to increase the
score value was to play the game well
but that assumption turned out to not
really be true any way in which your
reward function doesn't perfectly line
up with what you actually want the
system to do can cause problems and
things only get more difficult in the
real world where there's no handy score
variable we can use as a reward whatever
we want the agent to do we need some way
of translating that real-world thing
into a number and how do you do that
without the AI messing with it these
days the usual way to convert complex
messy real-world things that we can't
easily specify into machine
understandable values is to use a deep
neural network of some kind train it
with a lot of examples and then it can
tell if something is a cat or a dog or
whatever so it's tempting to feed
whatever real-world data we've got into
a neural net and train it on a bunch of
human supplied rewards for different
situations and then use the output of
that as the reward for the AI now as
long as your situation and your AI are
very limited that can work but as I'm
sure many of you have seen these neural
networks are vulnerable to adversarial
examples it's possible to find
unexpected inputs that cause the system
to give dramatically the wrong answer
the image on the right is clearly a
panda but it's misidentified as a given
this kind of thing could obviously cause
problems for any reinforcement learning
agent that's using the output of a
neural network as its reward function
and there's kind of an analogy to Super
Mario World here as well if we view the
game as a system designed to take a
sequence of inputs the players button
presses and output a number representing
how well that player is playing ie the
score then this glitchy hacking stuff
sethbling did can be thought of as an
adversarial example it's a carefully
chosen input that makes the system
output the wrong answer note that as far
as I know no AI system has independently
discovered how to do this to Super Mario
World yet because figuring out the
details of how all the bugs work just by
messing around with the game requires a
level of intelligence that's probably
beyond current AI systems but similar
stuff happens all the time and the more
powerful an AI system is the better it
is at figuring things out and the more
likely it is to
expected ways to cheat to get large
rewards and as AI systems are used to
tackle more complex problems with more
possible actions the agent can take and
more complexity in the environment it
becomes more and more likely that there
will be high reward strategies that we
didn't think of okay so there are going
to be unexpected sets of actions that
result in high rewards and maybe AI
systems will stumble across them but
it's worse than that because we should
expect powerful AI systems to actively
seek out these hacks even if they don't
know they exist why well they're the
best strategies with hacking you can set
your super mario world score to
literally the highest value that that
memory address is capable of holding is
that even possible in non cheating play
even if it is it'll take much longer
and look at the numbers on these images
again the left image of a panda is
recognized as a panda with less than 60
percent confidence and real photos of
Givens would probably have similar
confidence levels but the adversarial
example on the right is recognized as a
given with ninety nine point three
percent confidence if an AI system were
being rewarded for seeing given it would
love that right image to the neural net
that panda looks more like a given than
any actual photo of a given similarly
reward hacking can give much much higher
rewards than even the best possible
legitimate actions so seeking out reward
hacks would be the default behavior of
some kinds of powerful AI systems so
this is an AI design problem but is it a
safety problem our super mario world a I
just gave itself the highest possible
score and then sat there doing nothing
it's not what we wanted but it's not too
big of a deal right I mean we can just
turn it off and try to fix it but a
powerful general intelligence might
realize that the best strategy is an
actually hack your reward function and
then sit there forever with maximum
reward because you'll quickly get turned
off the best strategy is do whatever you
need to do to make sure that you won't
ever be turned off and then act your
reward function and that doesn't work
out well for us thanks to my excellent
patreon supporters these these people
you've now made this channel pretty much
financially sustainable I'm so grateful
to all of you and in this video I
particularly want to thank Bo and
mustard
who's been supporting me since May
you know now that the channel was hit
10,000 subscribers maybe it's actually
12,000 now I'm apparently now allowed to
use the YouTube space in London so I
hope to have some behind-the-scenes
stuff about that so patreon supporters
before too long
thanks again and I'll see you next time