2:Risks from Learned Optimization: Evan Hubinger 2023

hello everybody Welcome uh this is now
talk number two in the lecture Series so
today we are going to be talking about
uh risks from learned optimization uh
which is based on a paper that I wrote
with some other people I wrote it also
partly while it was open AI which is why
you're going to see some opening eye
branding on some of these slides because
this talk is from uh from them but um uh
yeah it's about this paper
okay uh so so let's just get started so
uh yeah so what what are we talking
about here so
uh just return again back you know very
quickly to uh you know we talked about
this before this is our sort of
stereotypical machine learning algorithm
you know what is it that machine
learning does
we talked a bunch last time about this
basic process of you know searching over
possible uh parameterizations of some
function to find uh values of those
parameters that result in the function
you know having some particular desired
behavior on some data set
and one of the things that you know I
really tried to emphasize last time that
is that is really important thinking
about this is It's hard to think about
it's it's actually really difficult to
understand exactly what sort of function
exactly uh you know what sort of
parameters what algorithm you're going
to end up implementing when you do this
procedure you get you know something
like the simplest algorithm that fits
the data but exactly what that means is
complicated and we don't exactly you
know understand
okay and so because of this you know
it's difficult to understand what's
going to happen we don't always you know
uh actually understand what this process
is going to do it's tempting and in fact
people will you know make use of
abstractions you know ways of trying to
understand what this is doing that don't
literally go through uh you know the
actual process of what it's doing but
allow us to sort of abstract away and
understand uh you know something you
know in theory that describes you know a
way of thinking about it and so a very
common abstraction that people use in
thinking about this is what I call the
does the right thing abstraction
so the idea of the does the right thing
abstraction is that well you know what
the procedure that we did here was we
selected the parameters
uh to minimize you know the loss over
the data distribution right uh you know
the way that we actually selected this
um
uh you know the way that you know
grading descent was structured the way
that it actually worked uh was that it
did you know it did this big search and
it found parameters that had this
property right and so we can sort of
Imagine in some sense that well if we
want to try to understand what those
what those prop what those what that
function will do what the algorithm will
do on new data we can sort of imagine
that the in some sense our model is
trying to minimize the loss that it is
sort of actually attempting to do the
thing which would result in the loss
being minimized even when it sees new
data points
um and this is a really useful and
helpful abstraction there are a lot of
situations where it's very difficult to
understand what exactly it is that our
model is doing but by using this
abstraction we can make some pretty good
guesses you know if I train to model an
environment where there was a gold coin
and I train the model to try to get the
gold coins it's a pretty good guess that
if I put it in an environment where it's
trying you know the environment is
slightly different I've like changed
some of the parameters but there are
still gold coins it might still want to
try to get the gold coins that's often
what is going to happen so so it's not a
bad abstraction it's often very useful
for helping us understand what's
happening
but it's not true right it doesn't
actually describe the process of machine
learning right the thing that we did was
not build a model whose goal was to
minimize the loss the thing that we
actually did was search over a bunch of
parameters to find some set of
parameters which in fact had the
property that they resulted in the loss
being low you know that they did the
right thing on the training distribution
and so so the the abstraction is not in
general true and in many cases what that
means is it's going to leak there's
going to be some situations where the
abstraction doesn't hold
okay and so I want to talk about the
sort of purpose of this talk
um the thing that we're gonna be talking
about today is a particular instance in
which that abstraction uh doesn't hold a
particular instance in which uh it fails
to work
and so uh to understand that particular
instance that we're going to be talking
about we're going to introduce a concept
which is the concept of an optimizer
so what is an Optimizer so we're going
to say that a system is an Optimizer if
it is internally searching through some
space uh you know according to you know
some goal you know it wants you know to
find something in a space that has some
property
um
so it has some sort of representation of
an objective of like you know something
that it wants to find in that space and
it's looking in that space to try to
find something with that property
okay so examples of things that are
optimizers
so gradient descent is an Optimizer in
the same way that you know we've we've
talked about and described where grading
descent is uh you know searching through
the space of possible you know
parameterizations of a function to find
parameterizations which in fact result
in good performance on the loss
um you know other things that are
Optimizer is something like Minimax uh
you know which is just like you know a
simple like you know chess playing or
you know any game algorithm where it
sort of you know searches for the best
move you know n steps ahead assuming
that the opponent makes the move that is
worse for you
okay and humans are not always
optimizers but sometimes we behave as
optimizers there's lots of cases where
we have some goals some strategy you
know something we want to accomplish in
the world and we try to look for
strategies that actually accomplish that
goal well and so we search for possible
actions that do a good job on
accomplishing that goal Yeah question
yeah so great he decent is this local
thing Minimax and he wins they can
search over a long time spans how
relevant do you think this is to the
definition of optimizers
yes it's a really good question it's
definitely in fact the case that uh in
many cases uh you know grading descent
um or you know in many cases you will
have optimizers uh like you know grading
descent for example that is like local
you know it's just you know looking at
each individual step and it's not doing
some sort of long-term optimization and
sometimes you'll have you know long-term
optimizers like humans that you know
have some you know really long-term goal
um for the purposes of right now uh you
know what we're going to start with you
know trying to understand uh you know at
the beginning of this talk I'm mostly
going to ignore that distinction you
know we're going to be looking both at
you know optimizers with these sort of
you know short-term Horizons and
long-term Horizons we're even gonna you
know think about optimizers that have an
objective that is about the world or
optimizers that have an objective that
isn't even about the world at all right
you know this maybe just they look for
an action which has the property that
you know it has you know some fixed
property like you know they just want to
Output you know something which is as
close to some fixed value as possible
they're not necessarily trying to
optimize something over the world so
we're just looking for optimizers in
general later on near the end of this
talk we're going to sort of return and
be like okay what happens if your
Optimizer has particular properties like
uh you know like like it's long term and
why might that happen but right now
we're not talking about that Yeah
question
Abraham demski has this sort of
distinction between selection and
control and I guess these are two types
of Optimizer selection being something
which seems something similar to what
you've written which is that it's
searching over some search space and
looking for candidate solutions that it
can stash it in search space and measure
the performance or something that it
just this is like an artifact over this
and then control is something that it's
a bit more greeting and like it or it
knows that like it can instantiate
Solutions in the kind of Organic
Solutions in in the world and it needs
to act as such so do you think Dexter
distinction has some equation here
yeah so that's a really good question
um I guess a couple of things that I
will say so I think that just to start
with
um yes I think it is an interesting
distinction and one that can often you
know matter this sort of Distinction
running selection control for our
purposes I'm mostly not going to make
the distinction I think I would sort of
analogize this distinction to one of the
distinctions we talked about last time
which is just sort of the sanctuary
something like supervised learning and
reinforcement learning we're in
reinforcement learning you know instead
of searching over you know a sort of
function that you know results in you
know a really good you know labeling of
you know classes on this particular data
set you know you're searching for a
general policy which generally results
in good things on this environment when
it's deployed and so in same sense you
can sort of think of the control as you
know being you know searching for a
policy which is able to control some
environment effectively you know whereas
um you know that is sort of different
from just like searching for a
particular element in a space
but I'm gonna you know mostly live
distinction and be like well okay
they're basically both doing you know
some search over some space they're
trying to find some you know something
with some objective even the control is
still sort of doing a search it's just
sort of you know searching for good
policies you know for good actions
rather than
um you know just like a good element in
in some space yeah
okay uh
more questions
yeah I was just wondering
I was wondering about like whether you
consider markets efficient markets is
optimized in a sense uh like it seems
like maybe they can't in fact do this
search
for like you know uh the most creative
like they can necessarily pull out the
pathway to the pre-efficient outcome
before they get there if there's like
too many actors or something but they
can in fact reach the career Frontier
yeah
um
so I think that
um it's yeah we're not going to just
imagine that an Optimizer has to be like
perfect it doesn't have to always you
know be be you know really good at
finding the best possible element in the
space it's just any anything that's
trying to do you know search over space
According to some objective and so in
that sense uh you know I think markets
you know in many cases can behave as
optimizers
um
it's not like our you know stereotypical
example here but you know you could
imagine a case where
um you know even you ended up with a
model that was internally acting like a
market it's a you know a little bit of a
weird case but you could imagine it
um
yeah and that maybe that's a little bit
of a hint that we were going is we're
going to be you know looking at
situations where we have models like
this one thing that maybe I'll point out
very briefly also is just the negative
samples here right so I mentioned models
you know if we just randomly initialize
a neural network
um it's not going to be doing one of
these things right it's just you know
some random parameterization that
results in a function that is just uh
you know doing not not doing some you
know coherent simple sort of algorithm
that could be described as something
like optimization
um what we'll see though at you know
what we're going to be talking about is
the situation where you know maybe after
I I train a network a bunch
um I do end up with something that's
well described as an Optimizer and of
course the reason for this would be the
same reason we talked about last time
which is just that in general when we
train these sorts of networks we find
these sort of simple structurally simple
algorithms and so you know this sort of
thing that we're describing here this
idea of optimization of search you know
might be the sort of thing that is the
sort of structurally simple algorithm or
you know that's that's at least the
hypothesis we're going to explore
um other things that are not optimizers
bottle caps is sort of a classic example
here where the idea is that
um you know we as humans optimize bottle
caps to be really good at keeping water
in a bottle
um but but that doesn't make the bottle
cap an Optimizer right so I want to
distinguish between the case where
something was optimized over right like
any neural network if I train it you
know is going to be optimized over by
grainy descent right but it's not
necessarily itself going to be running
some optimization algorithm like this
and so we really want to distinguish
between that situation so we're really
thinking about things that are
themselves optimizers not bottle caps
that have have been optimized over by
some other uh optimizer
okay more questions
so if I can even simpler example because
that means like a an algorithm that well
a function that F puts the maximum item
of a list would that be considered an
Optimizer because it is searching across
a search space but internally
represented objective
yeah it's a pretty simple Optimizer I
think it's not sort of the canonical
example we're going to be talking about
um but you know you could imagine that
that's sort of at least doing some sort
of search what it's not doing is it's
not doing a search over a very large
space you know and it's not doing uh you
know but but but but it's still you know
doing some some sort of like you know
optimization and so you know you could
describe it as doing something like that
it's not mostly what we're going to be
talking about it's sort of an edge case
Yeah question I'm still confused by what
you mean by internally searching for
some search space especially if we apply
it to Green descent for example which
you can't as an Optimizer what shall I
be thinking about where I'm thinking
about the internal alt the turtles of
gradient descent
is searching over a space right of
parameterizations
um I guess it's a little bit you know
what does internal mean
um I think you know
for our purposes the sort of thing that
we're going to be caring about is we're
going to be sort of internal to the
model because we're going to be looking
at situations where we have an
optimizers that are sort of contained
entirely within the the function that
you've learned
um you sort of learned an Optimizer
function
um you know so so maybe internal isn't a
good word to describe any you know
optimization because it's it's a little
bit unclear what counts as the internals
of grainy descent I agree and what
counts as the externals I think that's
not super a relevant distinction that
we're going to be caring about that much
you know you can imagine a situation
where like if I had a process and what
it did was like first it generated you
know a parameterization and then it did
a bunch of search over that
parameterization then the like the
algorithm which did that which first you
know parameterized some function and
then search over those parameters you
know what would be an Optimizer you know
if I if I count that whole thing as one
I draw a big circle around that and I'm
like anything internal to this you know
well clearly internally there is there's
an optimization happening it has some
space of parameterizations and it does
the search over it
um it's I think it's not super important
where we draw that distinction and I
think that one thing that is is sort of
relevant is that in many cases exactly
where you should be drawing that
distinction with like you know models
can change because you know our notion
of like what things count as part of the
model and when things are sort of
external is a little bit unclear like
you know you might imagine a situation
where like the sort of model on its own
is an Optimizer but if you give it a
bunch of time to think or you give it
the ability to interact with some you
know uh you know memory or something
then maybe it sort of becomes an
Optimizer
um I think that you know we're sort of
going to be including that in in what
we're talking about you know situations
where you know if you can draw some
boundary around it be like look this is
my Ai and inside of this boundary it's
doing some optimization
um then that's what we care about
because we want to find situations where
you know you're deploying some system
and that system is doing optimization we
want to make sure that optimization is
is the right sort of optimization
question what if
I put drills down to find some
Parts in the valley or is well an
Optimizer or power Digital Boundary was
optimizer
yeah that's a good question I guess in
some sense maybe we could say that like
physics is doing an optimization there
uh but the ball is not uh I think that
it's another sort of one of those tricky
edge cases
um that we're like mostly not gonna you
know imagine I think that like if you
had a network and internally the way
that the network did things was like it
simulated a physics environment where
like it like tried to you know had you
know a bunch of valleys and then put a
ball somewhere and tried to see where
the ball would land up and like use that
as part of its computation I would be
like yeah okay that's doing some
optimization right like it requires an
optimization to be able to understand
you know how to solve these sorts of
physics problems
um so you know for that purpose as we
could say it was doing some optimization
the literal ball itself is that an
Optimizer I don't know I mean no I sort
of want to say no but it also doesn't
matter that much because we're not
really going to be like we're not the
point of this is not to have like some
end-all be-all notion of what an
Optimizer is we're going to be using
this notion to talk about a very
specific situation
um where you can have sort of optimizers
um in in your sort of neural network and
so um you know the sort of more General
philosophical questions I think they can
be interesting and relevant but I'm
mostly not going to be engaging with
them
yeah more questions
so uh so if I guess if green descendants
or an Optimizer is natural selection
also considered an optimizer
yes that one is definitely yes uh and
we're gonna be talking a little bit more
later in the talk about that particular
example of an optimizer
okay I think hopefully we have some
clarity now on on what an Optimizer is
so uh
so so we'll keep going so I've mentioned
this now a bunch uh but the thing that
we're trying to do the sort of place
we're trying to get to is a situation
where we have uh a sort of model and
that model is itself you know internally
in some sense uh running some
optimization algorithm
and so
um there are a lot of you know cases
where you can have a model that's
running optimization algorithm
um
in particular we're interested in the
situation where we have a gradient
descent process that is searching over
some very large you know parameterized
function and it's not necessarily you
know searching only over functions that
are optimizers but it's searching over a
large enough space of algorithms that
some of those algorithms you know
especially maybe some of these sort of
structurally simple algorithms might be
running some sort of optimization style
process in that case uh we'll say the
situation where you do end up with an
Optimizer that you know that is what the
algorithm is that you found that is
implementing we're going to call it a
Mesa Optimizer so this is a little bit
of a weird terminology but the basic
idea is that Mesa is sort of the
opposite of meta in it's like a you know
Greek sort of prefix and so you know you
can think about oftentimes we'll talk
about like meta optimization where a
sort of meta Optimizer is an Optimizer
that is optimizing over optimizers and
oftentimes we'll do this explicitly
right so you can have cases where you
just construct a parameterization right
of some space where all possible
parameterizations Implement some
Optimizer um but that's not what we do
generally in machine learning right like
many of the parameterizations don't
Implement an Optimizer they're just sort
of general parameterizations of almost
any you know possible simple algorithm
but some of those parameterizations
might be an Optimizer if you if gradient
descent you know finds a
parameterization that does implement an
Optimizer then once it's done that it is
effectively a meta Optimizer right
because it is now searching over the
space of optimizers and so because this
sort of happens emergently we're going
to call you know the sort of case where
you you're sort of one level Below in
the sort of meta optimization hierarchy
you know means you're you're a Mesa
Optimizer right so we'll call grading
descent is sort of the base Optimizer
and if it is the case that gradient
descent is sort of in a situation where
it is optimizing over another Optimizer
then that other Optimizer is sort of a
Mesa Optimizer it is you know one meta
level Below in the sort of optimization
hierarchy relative to to gradient
descent so these are the sort of terms
that we're going to be using we're going
to say that you know your grading
descent process that is doing some
search over some large space of
parameterized functions is a base
Optimizer and
attuation where
them the particular parameterization the
gradient descent finds is uh you know
well described as an Optimizer then
we're going to call that system a Mesa
Optimizer and importantly this is sort
of the whole model it's just you have a
model and if that model is doing some
optimization then we're going to call
that model A Mesa optimizer
okay questions
so since early you said that natural
selection was an ultimate does that make
humans examples of Mesa optimize it
would expect a natural selection
yeah so this is one of those classic
examples that that people often like I
think
um it can be quite helpful and I think
it like absolutely you know is the sort
of thing that
um you know is reasonable to describe
what's going on here but
um it's not an example I want to focus
on because I really want to be focused
on on machine learning so we'll uh we'll
come back to that example later you can
sort of keep it in the back of your mind
but right now I think it's a little bit
tricky as an example because it's a
little bit non-central there you know
some things about it that are sort of
different you know I mean there's a
bunch of things that are different about
natural section that are different about
humans uh you know from ways in which
you know AIS might might be structured
and so we will be talking about that
example but I don't I don't want to like
focus on it too much
I also want to just try and understand
whether you could be a base Optimizer
and a base Optimizer at the same time
because you said natural selection can
like be filmed as a base Optimizer and
then you'd prefect humans who can also
be optimizers uh but then you just now
said that they can also you make
something like this
book at the same time or yeah or
concretely talked about neural Nets
right we have stochastic gradient
descent as like say a phase Optimizer we
find some uh optimization algorithm
entirely let's say Minimax I had Angel
already also said like uh you know
Minimax would be an Optimizer so it
could be a base anyways ultimate yeah
absolutely these are all relative terms
so in the same sense that like you know
I could be enough like you know yeah you
you could be meta relative to something
else or May's a relative to another
thing if I have like a stack of 10
optimizers you know and each one is
optimizing the next level down then like
you know pick any one of the 10 to be
your base and then everything above it
would be like you know meta then Meta
Meta and everything about would be Mesa
and then Mesa Mesa right so so these are
this is all relative terminology right
and so we're saying that relative to
gradient descent as the base Optimizer
you know if you were one level below
that and you were another Optimizer then
you would be a Mesa Optimizer right and
so this is not like absolute terminology
the only like absolute term here is off
right where like things are optimizers
or not optimizers but like is a thing a
Mesa Optimizer well just depends
relative to what face Optimizer right
cool
Yeah question
um where did the name Misa come from
it's just Greek uh it's like was picked
to be sort of opposite to meta in terms
of like Greek prefixes I don't I don't I
don't speak Greek and so I don't really
know I have had some people who do speak
Greek they claim that it's not a great
match and other people that say no it
probably is the best match I don't know
whatever it's what we're stuck with uh
there isn't there wasn't like when we
wanted this there wasn't like a really
standard uh you know Greek prefix the
correspondence of the opposite of meta
we did some searching and we found Mesa
was sort of like reasonably approximate
to what we wanted and it's got a nice
ring to it so it's what we're going with
uh but uh whether it actually is like
correct in like a Greek sense I don't
know whatever it doesn't matter that
much just imagine that it is
okay uh cool all right let's keep going
okay so I talked about this very briefly
so like things that are are wrong so
we're not talking about a Mesa Optimizer
as a subsystem we're just talking about
it as like a whole model right so we're
really going to be focusing on the
specific situation where you have
gradient descent is your base Optimizer
and then grading descent finds a
parameterization of a function you know
some algorithm some you know a neural
network or whatever that is doing some
search internally and if there is search
happening if there is optimization
happening inside of your model your your
network you know your AI uh then we're
going to call that a Mesa optimizer
okay we're not going to be sort of
identifying you know particular parts of
the the algorithm we're just we're just
talking about the whole the whole thing
um okay
great
um and then I talked about this also a
little bit but you know we're sort of is
This Very related to this General
concept of meta learning so one way you
can sort of describe What's Happening
Here is sort of emergent meta learning
and this is often how this sort of stuff
has been described in other Pace uh you
know places in the literature where the
idea is that you're in a situation where
you didn't think you were doing metal
learning you just thought you were doing
normal you know optimization over some
space but it turned out that some of the
things in that space that you were
searching over were themselves doing
optimization and so now you're in the
business of meta learning even though
you sort of didn't imagine originally
that you were and so in that sense we
can sort of also refer to this you know
this sort of Mace optimization as you
know a situation where you have emergent
meta learning
okay all right so returning back to what
we were talking about earlier we have
this does the right thing abstraction
right and we're sort of concerned about
situations where this does the right
thing abstraction might be might be
wrong and uh so so first let's ask you
know what is the does the right thing
abstractions say in the case of Mesa
optimization right and what it says is
that a you know if the sort of Mesa
Optimizer is doing the right thing right
it is like in fact Trying to minimize
the loss in some meaningful sense then
it should be the case that
um the you know models Mesa objective
where the mace objective is just
whatever it is that it's searching for
whatever it is that it's trying to
accomplish
um you know should be to minimize the
loss right it should be to do whatever
it is that you know we were trying to
train it to do
um
the problem of course right that we
talked about previously is the does the
right thing abstraction is leaky it
doesn't always necessarily hold and so
we are going to be really focusing on on
this talk on this very specific question
which is when does this particular does
the right thing abstraction leak right
like what happens when I have a model
and that you know I have a learn
function I have you know
parameterization and that uh you know
algorithm that it's found is itself an
Optimizer and you know is it the case
that we can say that does the right
thing abstraction holds and when can we
say it holds and when does it leak for
that uh situation
okay so in that particular situation
where we have a you know uh a sort of
Base Optimizer and a mace Optimizer and
we're trying to understand you know this
does or I think abstraction we can
isolate sort of two different problems
and these are sort of two uh you know
sort of core problems in any situation
where you have uh you know sort of Mesa
Optimizer uh which we're going to call
Inner alignment and outer alignment and
so outer alignment is the sort of
alignment problem uh you know the
problem of making the model do the thing
that we want that is external to the
system it is the problem about making
sure that we have some loss function or
some reward function or some you know
goal that we want the model to be doing
that if the model we're trying to do
that it would be good
and then the inner alignment problem is
okay if you end up in a situation where
you have a model and that model itself
is trying to do something right it has
some mace objective it has some you know
search process some optimization is the
thing that it's doing the same as the
thing you want it to be doing right so
we have two core problems we have outer
alignment which is find something that
it would be good for the model to be
doing and we have inner alignment which
is actually get a model which is in fact
doing that good thing
okay does this make sense and so we are
mostly going to be talking about inner
alignment here this is sort of the
purpose of this talk is really to focus
on this case of okay you know what is
happening when we have a situation where
we have a model and is doing has some
objective it has some you know maybe
subjective it's doing some optimization
um is that going to match up with what
we want uh and and you know when when
will when will it when will it
okay uh so so what does an inner
alignment failure look like right so
what happens when inner alignment fails
so
um outer alignment failures are sort of
sometimes easier to describe right we
can think about an outer alignment
failure as just a situation where you
know I told I wanted the models to do a
thing and the thing I wanted the model
to do was bad I shouldn't have asked the
model to do that right I wanted the
model to you know uh unboundedly
optimize for paper clips is sort of a
classic example and so the model you
know is like ah I will turn the whole
world into paper clips so it's like okay
well the thing that you were trying to
get it to do was not a good thing to try
to get models to do right
um but inner alignment is sort of
different right so inner alignment
failures look like situations where
um
we're going to say inner alignment
failure is a situation where your
capabilities generalize but your
objective does not generalize so what do
I mean by that so let me try to unpack
that so here's a particular situation
that we're going to imagine so let's say
you have a training environment this is
sort of the you know we're doing maybe
RL you know reinforcement learning and
we want to get uh you know some
parameterized you know function you want
to find an algorithm that does a good
job at you know implementing a policy
which solves this means and so what we
wanted to do is we wanted to get to the
end of the maze
um that's sort of the goal that we want
and we've set up our mazes that look
like this they're sort of these small
brown mazes and they have these green
arrows at the end
and then we want to ask what happens if
I take a a sort of algorithm which in
fact is a good job in this environment
and I move it to this other environment
uh where we have larger mazes they're
blue now and the Green Arrow got
misplaced I put it in a random location
in the middle of the maze instead of
pointing at the end but what I want is I
still wanted to exit the maze right I
don't want it to get stuck inside the
maze and go to this random place where I
misplaced my green arrow
okay and so now we can ask what happens
right what are the possibilities for uh
you know a model trained in this
environment that in fact does a good job
in this environment when it moves to
this new environment
um and so here are some possibilities
that could happen
um it could just fail right it could
look at this larger maze you know these
really big blue mazes and not have any
idea how to navigate them in a coherent
way
in that case
um what would happen you know we're
going to call this a failure of
capability generalization this is a
situation where the models you know the
algorithm you learned its capability to
solve mazes in general didn't exist it
didn't have good general purpose media
solving capability and that you know
whatever to the extent that it did have
made it solving capability that
capability didn't generalize it didn't
continue to work in this new environment
so we're going to say it's capabilities
failed to generalize
but there are other situations as well
so another situation that could happen
is a situation where it does it does
what we want right it successfully
navigates this larger blue Maze and it
exits the maze it does you know
successfully in that case we're going to
say its capabilities generalized because
it sort of was successfully able to do
general purpose May solving in this new
environment and it's alignment
generalized because it was also able to
do it for the right purpose right we we
trained it to try to get to the end and
it successfully got to the end that was
what it did um and so we'd be like okay
that was the situation we had capability
and Alignment generalization
uh you know and objective generalization
um but there's a third thing that can
happen which is its capabilities could
generalize it successfully navigates the
larger blue maze but it does so to get
to my misplaced Green Arrow instead of
getting to the end and and of course the
reason this can happen is because well
those are the same on the training
environment right in the same way that
when we're talking about you know the
situation where you have you know either
it can learn the shape classifier or the
color classifier in this case well
either you can learn the go to the Green
Arrow or you can learn go to the end and
both of them will do a good job in
training but they're going to have
different impacts in terms of what they
do in this new environment
and so inner alignment failures are
situations where you wanted it to have a
particular objective right you wanted to
learn the objective of go to the end but
it learned a different objective and
learned the objective of you know go to
the end go to the Green Arrow instead
and it learned it while its capabilities
still generalized so it
solve them for the purpose that you want
it and so what's sort of concerning
about this is that it's a situation
where you have a very competent
generally capable model a model that has
learned a general purpose algorithm for
solving some particular environments and
yet has learned to use that algorithm
for a purpose that you never intended
that you never specified and that you
never wanted the models to be doing
and that's potentially concerning
because if the thing that it is trying
to do is is you know not something that
we want
um you know that's bad
uh question
if we are in such a situation where my
mother competently
mitigates a maze towards a green arrow
instead of the exit but
it's not performing search internally
Reserve is it an inner alignment problem
still
yeah good question so I think that um I
would say it's still sort of a problem
of like you know uh you know objective
misgenderization where your objective
doesn't generalize and your capabilities
do generalize
um but it might not be that all
objective misgenderization problems are
caused by Mesa optimization right so
we're going to be specifically focusing
today on the situation where the reason
you have an objective Mass
generalization problem is because you
had a mace Optimizer with a different
objective that's not the only reason
that you can get an objective
misgeneralization problem
um so what I'm describing here is not
you know this is if you have an inner
alignment failure where you know
something goes wrong with your mace
Optimizer you don't learn some different
some incorrect made subjective then it
will result in an objective Mister
analyzation problem that looks like this
that's what the failure will will result
as but that doesn't imply that all
objective miserialization failures have
to come from mace optimization right
does that make sense cool
okay
all right so we're just going to give
some names to these things so we're
going to say that the situation where
the model is sort of aligned over the
whole distribution where it uh you know
its capabilities uh it's sort of its
alignment in fact generalizes when we
give it new situations where it actually
sort of has learned the correct May
subjective we're going to call it
robustly aligned and in the situation
where
um
it sort of learns the wrong way
subjective it learns some may subjective
which does a good job in training uh
because like there's some
indistinguishability you know in the
case of like the Green Arrow and the
coincidence the maze they sort of both
do a good job but one of them is sort of
correct it's one we want one of them is
the one we don't want in the case where
it learns the one we don't want we're
going to call it pseudo aligned because
it looks aligned in training but in fact
there are situations where it actually
will do the wrong thing
uh okay and in particular we're talking
here about the objective generalization
not just the capability generalization
so situations where it sort of tries to
do the wrong thing it has a Mesa
objective that is that is incorrect is
optimizing internally for the wrong
thing and so that's that's the pseudo
alignment case Yeah question
I have a hard time understanding how we
can be always obviously alive because as
far as the pope is amazed they think we
just wanted to eat to see the riddle so
we we finally we know it will be a
Robusta line but if instead we decided
it would be just we had to pick up to
find the end of the main it took deep to
that and then so oh could you beat like
subtle nothing in case it would be very
scary it's just it's you know how they
did
yeah good question
we're sort of okay okay if its
capabilities fail to generalize in some
particular situation so if we're you
know in a situation where it's just its
capabilities just fail to generalize
um then even if it like you know has the
wrong would have had the wrong objective
in that case we're sort of okay right so
what we want is we want in all the cases
where the model is like really
competently doing something off
distribution we want it to be doing the
right thing right and so in some sense
you know one sort of thing that we're
sort of okay with you know still
accepting is if the model can identify a
situation where you know it shouldn't
try to optimize for the thing that it is
you know what it's trying to do
um and like stops and you know then
we're fine right if it like asks humans
and you're like hey like what should I
be doing in this case you know then even
if it started with the wrong you know
idea then it sort of gets the wrong the
right idea right and so we're really
concerned about situations where there's
this capability generalization without
the objective generalization it does
something really competent and it really
tries to do an optimization in that case
but it does so without having the right
objective and so if in all cases where
it's sort of doing some competent uh
optimization it's doing it robustly it
has the correct objective then we're
going to say it's our essay aligned okay
oh
okay we're back okay it's all working
great uh okay so uh did I step a slide
uh yeah okay great so uh we're gonna be
talking in the rest of this talk about
two distinct problems related to mace
optimization so uh problem number one is
unintended mace optimization right when
are you going to find situations where
you don't necessarily want you know to
find an algorithm which is doing
optimization but in fact optimization is
just the result of doing your search
right you have to you know you have
grading descent you're doing some big
search over algorithms and it just so
happens that right you know the simplest
algorithm fits it data whatever you know
the inductive biases of gradient descent
are select for algorithms which are
doing optimization that's to me the
first thing we talk about and then the
second thing we talk about is inner
alignment which is if you do end up with
a model which is doing some sort of
optimization it's you know it's trying
to accomplish some sort of goal you know
when is it going to you know what sort
of goals will it try to accomplish what
sort of uh you know mace objectives will
it have and you know when will those
match up and when will they not match up
with the sorts of things we want it to
be doing
okay so uh we're gonna be trying to uh
first understand conditions for Mesa
optimization so what are situations in
which you would get Mesa authorization
okay so first uh you know one thing that
I just sort of want to start with here
that I think is sort of you know a basic
reason why you might expect uh you know
to get Mesa optimization in some case is
well search is really good at
generalizing right so if I'm in some
environment and uh there's a lot of
possible options for you know how things
could go in that environment what things
could possibly happen
um then uh it's you know really bad
strategy right to just memorize all you
know possible situations right I really
don't want to just you know be like okay
if the go board looks like this then I
do that if it looks like this then I do
that I need to learn some general
purpose algorithm for you know selecting
good go moves right in some environment
and a property that good go moves have
is that well they're all you know
compressed by you know the single you
know property which is well if I do some
look ahead if I do some search on how
well this go move performs in a couple
moves down you know good go moves at the
property that they result in good you
know States for me a couple moves down
right and so there's a there's a
structural property about you know good
actions in the environment which is that
they are compressible by knowing that
they're good right knowing that you know
I could do some search and that search
would tell me you know what actions are
good lets me sort of compress and
understand what's happening with you
know good actions environment right so
so we think about this right like what
search lets you do is it lets you be in
a situation that you've never seen
before a situation where you don't know
exactly what's happening you haven't
understood the exact mechanics of the
situation but as long as you have the
ability to sort of search and understand
ah if I took this action or if I did
this thing you know then what would
happen in a couple steps then I can have
a ability to do a good job in that
environment because I'm not just sort of
learning basic structure you know I'm
not learning sort of heuristics about
their particular situation I'm learning
this sort of much more General thing
about you know what it means to do a
good job in general which is you know to
do a good job means to find something
which has this property and so I'm just
searching for things which have that
property right and so the basic idea is
well okay just to start out with you
know search is actually a really good
strategy for doing a good job in any
situation where you have a lot of
possible uh environments a lot of
possible you know diverse situations you
could find yourself in
okay and so this is the reason that you
know in fact we often will use search
algorithms to solve cases where there's
a really large variety of possible
things that you know you could encounter
right there's a reason that when humans
write chess algorithms or go algorithms
we write search because you know it's
it's you know too difficult to be in a
situation where you you just have to you
know have enough heuristics to cover
every possible situation right there are
too many situations too many possible
things that you can encounter you need
to be learning sort of general purpose
algorithms that can you know be able to
work in any possible situation and
search is sort of one of those sorts of
general purpose algorithms and so you
know because of that we might expect
right if you know to the extent that our
models are sort of selected to
generalize
um which is to some extent you know the
reason that we select the particular
types of parameterized functions that we
select is because we want them to have
the property that you know the simplest
sort of function under those uh under
those conditions will actually
generalize well we'll actually do you
know good things in new environments
um well okay search is one of those
sorts of things to generalize as well so
we might expect to find it for that
reason
okay so that's number one right is
search is actually really good right you
know we should expect you know you're
training them all to do really powerful
things and one way to do really good
stuff is to search and so that's a
reason to expect search
okay uh another reason to expect it is
Simplicity and compression so I was just
talking about this previously but we can
think about the reason you know the sort
of the the you know in some sense you
know your model is sort of you know
they're searching for patterns right
that compress the data and if I have a
bunch of data that describes you know
some complex Behavior Uh that is trying
to do some you know powerful thing in
some particular situation a good way of
compressing almost any complex behavior
is while complex behaviors are often
selected uh to accomplish some goal
right I you know search through a you
know uh you know if I'm like you know
looking for you know at good go moves
you're right those good go moves were
selected to win the game of Go and so I
can compress the you know the property
of good go moves into you know things
which in fact have the property they do
a good job uh you know at accomplish and
go after I do some look ahead
um you can sort of think about this
right if I if I could just memorize all
of the possible paths you know uh you
know all the turns I have to make in
some particular maze to get to the end
you know I have a really long maze
um and I could just memorize all of the
all the things I have to do just all of
that maze or I could just write the you
know two-line algorithm that just like
you know searches for all the pass
through the Maze and finds the one which
results in me getting to the end and
that's a substantially simpler algorithm
right I don't have to do all this
memorization of all these possible ways
uh you know all the sort of turns you
have to take in the Maze I can compress
down the sort of you know policy of
taking all these particular paths into
you know the much simpler uh algorithm
of just you know figure out which path
works and then take that path
question
in order to produce outputs reasonably
quickly then Chelsea to spend some time
actually identifying what's the optimal
path through the search space and the
compressibility of that should be a
factor as well right
yeah that's absolutely correct so I
think that I think that I am
is that you're going to find a model
that is just doing search the only thing
it does is has a search algorithm and
everything else you know there's nothing
else that it's doing
um that's not going to happen I think
that's that would be really unlikely
um so right so so again we're talking
about a situation where we have a model
and one of the things that model is
doing right is it has some complex
search process but certainly it's also
going to have other stuff right it's
also going to need some amount of
heuristics right because search spaces
are really big uh you know oftentimes it
can't just search over the entire space
so it has to have some amount of
heuristics and some ways to prune the
space but
um if it is doing some search if it has
some objective then we're still going to
call it a mace Optimizer so we want to
understand the situation where okay why
would it be doing any search right why
would it even have a search algorithm at
all versus why would it not
um and so yes absolutely there you know
even when you do end up with the search
algorithm it's still going to be doing
other stuff as well
um
um but but it still might be concerning
if that search algorithm is searching
for something we don't want
um and we talked last time of course you
know in terms of compression rule okay
so why would you find these sort of
simple algorithms we talked about you
know a bunch about this previously about
you know in fact is the case that these
sorts of uh you know machine learning
systems that we build have the property
that they select for these really
structurally similar algorithms and so
to the extent that something like you
know search is this very structurally
simple algorithm it's just doing you
know this basic procedure of you know
narrow the space until we find a thing
which you know scores well on some you
know objective
um we should expect you know that sort
of structurally simple algorithm to be
one that is selected for by the you know
machine learning processes that we use
yeah question
over the past couple of years there have
been sort of attempts I guess to does it
create mace optimizers there I mean
there was the RL squared paper from a
couple years ago and recently it was
offered the distillation which I guess
is explicitly tries to compress like
pretty complex policies or but like uh
there have been arguments that neither
of these two models are actually doing
Mesa optimization and that mace
optimization is actually not selected
for I'm like rather like simpler
heuristics that also compressed peace
policies would be selected for over uh
this sort of search that I can see we're
describing just now so yeah do you have
any comments
yeah I think I'm not going to try to
weed too much into the debate on like
whether any particular model that people
have built does or does not count as a
mace Optimizer I think one thing that
I'll say is that well it really depends
on your definition of base Optimizer
right you know we waffled a bunch at the
beginning right like what is an
Optimizer what is search exactly
um you know I think that you know
certainly it seems like a lot of the
sorts of models that we've built often
are doing searchy like things you know
especially if you think about write
something like alphago well alphago was
literally trained uh by distilling a
search process Monte Carlo tree search
into a policy model that was attempting
to mimic the results of the search
process and so like okay probably
has to be doing something search-like to
be effectively able to mimic a search
process consistently
um exactly what it's doing though you
know the problem is we don't know right
you know I think a lot of the problem in
trying to actually understand exactly
you know is does the thing count as the
main office or is it actually doing
search you know is it doing some
optimization as well we don't have very
good transparency and so oftentimes it's
really difficult for us to know exactly
what algorithm it is that it's
implementing and um even when we do have
a good some amount of understanding of
what algorithm is implementing it's
often really hard to interpret that
right it's often really hard to actually
understand uh you know does this count
right as you know authorization
the important thing right that we care
about at the end of the day is like is
it the sort of thing that could have an
objective that is different than the one
that we want and and you know if it does
would that be bad right and so that's
really what we're going to be focusing
on
um you know right now I think that you
know a lot of these semantic questions
get you know Harry and you can sort of
debate them all day but um you know at
the end of the day we care about
situations where you know it's it's in
it has some search processes optimizing
for something and if that thing was
optimizing for we're wrong you know we'd
be in trouble
question so search seems to be easily
describable so it seems like simple to
describe but maybe is computationally
complex is the modern Paradigm of deep
learning going to select for something
that's going to be as computationally
complex as an algorithm like search
[Music]
really good question I think the best
answer I can give is
um I don't know
um I think it's you know it's often
unclear you know the extent to which we
can uh you know actually find
search-like algorithms in sort of you
know these sorts of parameterizations
that we're searching over
um I think you know some things I will
say I think that
um
you know especially as we start getting
really really large models that have the
ability to influence you know very
complex multi-stage algorithms certainly
search is possible right you know you
absolutely can encode some search
algorithms in these sorts of really
large models you know you start by
creating some space you do some
iterative refinement you know you select
something out of the space
um are they doing it I don't know um you
know like like I was saying I think it's
really tricky it depends on exactly how
you define these things I think they
could be doing it and it and one of the
things we're going to be talking about
is that it seems to me like a lot of the
sort of biases that are pushing you in
various directions seem like they the
you know the biases they will favor
search seem like they're increasing so
so one of those for example is is
Simplicity because as we talked about
previously these sort of bias towards
finding structurally simple functions
increases as we have larger and larger
models models that are doing more you
know have the ability to implement
algorithms that require multiple stages
um you know like search you know the
bias towards those sorts of structurally
similar algorithms uh you know increases
and so for that sort of reason that's
one of the reasons you might expect that
you know that this will arise as we have
larger models
okay question
or I guess the way of you described
search makes it seem like it has some
sort of like recursive structure to it
um so does that mean that you can sort
of get around the problem of unintended
optimization by just building uh models
that aren't recurrent or don't have like
like recursion in them and like
would this make it harder for them to
implement this sort of like recursive
search
yeah so it's definitely not the case
that you can get around this by sort of
having models that aren't recurrent
because you can still in one search even
with a finite depth I I just Implement
finite depth search right so I can
influence search out to a particular
finite depth uh you know in a model that
is limited you know via some finite
depth right I can't Implement unlimited
search I can't do a search that just
like always you know keeps going until
it finds something that is good enough
uh because you know in a in something
that is you know has finite depth I
can't do that
um but but doesn't mean you can't have
any styles of search doesn't mean you
can't have any sales optimization so is
it the case that recursive architectures
might increase the probability that you
end up with Optimizer style systems you
know Mesa optimizers I think the answer
is it seems like that is possible
um but is it necessary definitely not
at least in theory I mean it may be in
practice that you know we only ever find
optimizes we do recurrent things I think
that's pretty unlikely to be true but
it's at least you know conceivable
okay
um other reasons that you might you know
believe that your model will uh you know
you're gonna end up with a Optimizer
um another reason is well oftentimes we
train on really big data sets of things
that humans have done and a property of
humans is that we often choose what
things we do and what we say and how we
act to accomplish particular goals
um and so if you want to do a really
good job at mimicking humans then in
some sense you need to be able to run an
optimization process to do so because uh
you know humans you know often will do
optimization
and so there's sort of this basic
structural property of it eventually if
you want to get good enough at mimicking
humans you sort of have to be able to do
some amount of optimization
um and one property about this is well
you know if you're doing optimization
for the purpose of making humans it's
really easy to reuse it you know for
your own purposes right once you have
the authorization algorithm there's lots
of you know you know you you can reuse
that same algorithm for accomplishing
you know all sorts of other things as
well you know
um and in fact this is like you know if
we think about how do humans mimic other
humans you know oftentimes the way that
humans mimic other humans is by sort of
you know imagining you know we were them
right empathy you know this sort of you
know mirror neuron style idea where
you're like okay you know if I we're in
your situation and then I run my own
optimization algorithm you know in the
situation where you are and I try to
predict what you would do that way and
so you know in the same sense we might
imagine models doing similar things
where you know they can have their own
optimization that they do for their
purposes of you know whatever it is
they're optimizing for um and they you
know then that same authorization is
used you know to help predict humans and
it could go either way you know you
could start by first developing
authorization for the purpose of
predicting humans and then you know
repurposing it for yourself or vice
versa but the fact that we're trying to
predict humans means you know it seems
to incentivize you know we're gonna have
to have models we're doing some amount
to bottom station
and at least if they get good enough of
that task
okay so I talked about this previously
but it seems like you know we have you
know we have these sorts of lists of
things right that are pushing Us in
various different directions we've got
you know as we have larger and larger
models that are biased towards
structurally simple functions we have an
incentive towards mace optimization uh
you know as we have more you know more
Simplicity bias from larger models as we
have you know stuff like statefulness
the ability to like store state or you
know be recurrent like we mentioned uh
you know uh might incentivize based
optimization and then we have things
that disincentivize space population so
uh you know one thing might be if I give
it a bunch of explicit optimization you
know it has like Monte Carlo research as
part of its structure then it's going to
be less likely to have to learn its own
optimization algorithm you know if I
give it um time bias so one property of
optimization that you know that we're
talking about as well you know
oftentimes it can take many steps of
iterative refinement and so if I force
it to be really you know uh you know not
very deep right I can't do many steps of
sequential computation it's going to be
hard for you to learn a search process
um and you know if I have really simple
tasks that don't have this property that
there's really diverse settings that I
have to be able to generalize across
then the generalization benefit of
search will be reduced as well and so we
are in a situation where you know we
have some things pushing One Direction
something's pushing the other direction
and I think that you know the sort of
General point that I want to make here
is that it seems like as AI gets better
we're doing sort of more of the things
than incentivized based optimization and
few of the things that disincentifies it
and why is that well you know it's
because a lot of these things right are
sort of just basically correlated with
capabilities they're just basically
correlated with having big models that
are capable of learning general purpose
algorithms uh that are able to you know
solve uh complex environments and so as
we build better AI systems we're sort of
pushing pushing research because well
because search is a really good
algorithm it solves lots of problems uh
right you know we're doing less hard
code augmentation because we want the
models to be able to optimize themselves
we're doing less time flux device
because we have deeper models they can
learn you know algorithms that involve
more iterated steps we're doing more
General tasks uh you know we're doing
situations we have bigger models with
more Simplicity bias uh you know and and
we're sort of you know exploring
architectures that have the ability to
you know do all you know sorts of
implement more complex algorithms and so
you know it seems like we're moving in a
direction where you know um searches are
incentivized more by you know sort of
larger and more modern outcomes
okay so I think this sort of suggests
that at some point we're gonna have to
deal with this and we're going to have
to be in a situation where
um you know we have to sort of encounter
and deal with a sort of inner alignment
problem you know where the we're the
sort of key question that we have to
answer in that case is you know in what
situations will you in fact end up with
many subjectives that are misaligned
with the you know what the thing we're
trying to get them to do with the loss
function the word functions whatever
they were training on
okay
so
um
we recall from previously that we're
going to call the situation where we
have a sort of Mace Optimizer and it has
that objective that is sort of not
robust across you know different
situations a situation where it's
pseudo-aligned
um and so you know one thing we can
start with is just like okay what does
pseudo alignment look like right what
are the possible situations in which you
could have a model that is that is
pseudo-aligned
um
and so we're going to start with the
most basic situation that we're sort of
going to be focusing on the most is
proxy Sue alignment which is a situation
where in the environment there are
various different sort of proxies that
the model could care about that are
correlated with each other
um on the training environment but you
know they come apart in in in you know
some other environment so this is the
this is the example with the green arrow
where going to the end of the Maze and
going to the Green Arrow are correlated
with each other in training they're you
know highly you know tightly coupled
proxies but in fact they're different
and they have different implications if
we're in some new environments
and so the idea is that instead of
learning to optimize for the actual
things that we wanted we might end up in
a situation where your Mesa Optimizer
tries to learn uh you know some easier
to compute proxy
um
and in that case we sort of have this
proxycut alignment
um now so this is sort of the main case
that we're going to be thinking about
um but it's not the only case so I do
want to just sort of mention that there
are other ways in which you might have a
model which is pseudo-aligned as well so
another example that you know is is
particularly weird example but it's just
sort of I think a good example to
illustrate that the possible other ways
in which this could happen is
sub-optimality pseudo-alignment so uh
sub-optimality student alignment is a
situation where the reason that the
model looks aligned in training is
because of some defect in its
optimization process so you know maybe
for example it doesn't know some fact it
just like is you know uncertain or you
know ignorant about something
um and because it's ignorant about that
thing it does a good job it looks like
it does a good job but if it ever
learned the truth it would do a bad job
so an example that we we often a sort of
toy fun example that we sort of give in
the paper about this is let's say you
have a model or you have an AI and the
AI is trying to clean a room uh and it
wants to you know get you know you're
trying to find some algorithm which in
fact results in you know the policy
which which cleans the room and mainly
one thing the model could be doing is
it's it's an Optimizer it's trying to
optimize for something and the thing
you're trying to optimize for is it
wants a world that contains as little
stuff as possible it wants to destroy
all matter in the universe and it in it
correctly believes that all of the dust
that it vacuums up is obliterated
and then one day it discovers that you
know in fact all of the vacuum dust is
not obliterated it just goes in the you
know the little bag in the vacuum and it
has you know a realization that actually
it shouldn't be cleaning you know all of
this cleaning they was doing was totally
useless for its for its true goal right
and so this is this is a fun example
it's not it's not actually you know
realistic but the point is that there
were weird situations where your model
can look aligned in training
um for a situation for for reasons that
are you know can be totally different
you know from from what you want right
it can look like it's doing the right
thing but not actually be in any sense
you know really doing the right thing
um
and so that's sort of concerning right
these are situations where we have some
styled inner alignment failure Yeah
question
why is that different from the proxy
thing why is like destroying the the not
just the proxy for vacuuming
uh yeah so why is some optimality
different from the proxy case so um the
difference is that the reason that
destroying things and
um you know uh doing am I actually
cleaning the room are like correlated on
the training environment is not because
they're actually correlated right it is
not in fact the case that if I looked at
the actual dynamics of that environment
the destroying things and cleaning the
room are all correlated right and so
they're not proxies for each other right
if I was looking at a causal graph of
you know all the things that interact in
that environment these are not two
things they would ever interact but
because our model is has a mistaken
belief right it thinks that that
structure is different from what it
actually is it thinks they're correlated
and so because it thinks they're
correlated does a good job right and so
that's the difference here is that
um you know in the first case the case
we're primarily going to be talking
about the proxy case there are actually
two things through correlated in the
environment and it cares about you know
one of them right and then the some
optionality case they're not actually
correlated but it thinks it thinks they
are they are that's one case where you
could have solved Mallet alignment but
yeah so that's that's why those two are
different
okay great so we're mostly talking about
proxy but there are other things as well
all right so that's what pseudo
alignment looks like and so then the
question we have is why would it be
incentivized right like why would you be
in a situation where your training
process is selecting for models you know
uh for for parameterizations for
algorithms that um you know our studio
aligned okay so the most basic reason is
unidentifiability it is just in fact the
case that in a very large complex
environment there's just a lot of things
that you could pay attention to right
there's a lot of proxies there's a lot
of sort of variables in that environment
there's a lot of sort of you know things
to look at
um and and many of those things are just
going to be correlated with each other
right there's going to be complex
correlations uh and interdependencies in
any complex environment and so you know
there's just gonna be a bunch of things
which are very very related you know uh
it's going to be hard to disentangle all
of the possible ways of interpreting you
know the end of the maze right and all
of the possible ways of understanding
you know what that means
um in any environment because there's
just many possible functions that you
could apply to some environment which
would all yield similar results in the
training example which but you know
which might have totally different
Behavior elsewhere and so you know just
ah priori you know before we even have
any information about which sorts of uh
you know objectives might be selected
for by gradient descent there's a lot of
them right and there's a lot of them
actually would yield good performance in
training and so we should sort of expect
that well you know it'd be kind of
unlikely if you got exactly the one that
we wanted
okay so this is the most basic reason
unidentifiability you often just cannot
distinguish uh you know just based on
some training data between you know
different possible objectives
okay there's other reasons as well so
you know some proxies you know some
possible you know mace objectives
proxies the model could learn might be
structurally better from the perspective
of the inductive biases from the
perspective of you know the sorts of
things that you know greatness 10 is
looking for these sort of structurally
simple functions than others
um and so why might that be the case so
one example is that you know some
proxies might be faster
it's easier to implement there might be
things which the model can more
effectively you know Implement and run
so there's sort of a fun example of this
so we can imagine a situation you know
we were talking uh previously about you
know well okay we can sort of
conceptualize something like natural
selection as you know doing a search
over humans and in fact you know natural
suction you know maybe it's trying to
optimize for something like you know
inclusive genetic fitness it wants you
know you to pass on your genes into the
Next Generation
Um but of course you know that's not
actually what humans care about you know
we care about all these various
different things you know you know we
care about you know stuff like uh you
know happiness and love and friendship
and you know art and sex and food and
all of these various different things
right that are not exactly the thing
that Evolution wanted right Evolution
wants us to you know all go to the
nearest you know sperm or egg bank and
you know donate as much as possible and
you know have as much of our DNA in the
Next Generation that's that's not what
everyone does right and so we can sort
of you know ask well okay you know why
is it that you know humans don't have
this property that we are you know just
trying to maximize you know our DNA in
the next you know generation given that
that's sort of what we were selected for
right I mean the answer you know at
least one part of the answer is you know
the same sort of thing that we're
talking about here which is well some
things are just easier for people to do
right and so if you're if you can
imagine a situation in the alternate
reality where you know babies you know
all humans actually just want to
maximize their you know genetic fitness
they just want to get as much DNA in the
Next Generation as possible and you have
a baby right and this baby you know what
the baby wants is they're trying to
figure out how to maximize their DNA and
they stub their toe right and they have
to figure out you know what to do about
this right and um it's actually a hard
optimization problem right you're like
okay is stopping my toe going to reduce
my chances of being able to like you
know you know find a mate like well
maybe it might reduce my ability to like
go run and get food but maybe like if
I'm you know able to you know stay at
home then I can like you know more
easily go find someone uh you know uh
you know to to uh to mate with and so
you know it's a complex optimization
problem right it requires a bunch of
understanding about things in the world
it has to do a bunch of you know know a
bunch of facts about DNA and a bunch of
facts about Society
um and it's just hard and it sort of
like you know it's very difficult for a
baby in this environment to do this and
so you know Evolution decided not to do
this right you know the baby doesn't try
to run this create this complex
optimization problem instead the baby
has you know a much simpler you know
objective which is pain is bad right you
know if you if you do something and you
discover that that thing results in pain
then you want to not to do that thing
right uh you know and so it has a much
simpler objective of avoiding pain and
so we can imagine that the same sort of
thing happens in the situation where
we're training models as well where you
know there are some sorts of very
complex things that we might want the
model to do and those things might
require some really complex optimization
process that you know is too expensive
for the models to run a lot of times and
so instead it might learn to run you
know simpler easier to execute sort of
proxies like you know pain uh they're
sort of uh rather than something you
know much more complex and difficult to
execute like you know how much DNA
exists in the Next Generation
okay
um and this is sort of one structural
reason why some types of proxies some
types of Mace objectives might be
favored by the inductive biases by by
this sort of like you know uh sort of
what grading descent is selecting for
rather than just you know selecting for
the thing that we want
okay and another reason is that some
proxies can be simpler so in addition to
some sort of proxies being easier to
compute and easier to optimize for some
are also just easier to specify easier
for the model to sort of you know
specify in terms of input data so we
think about you know again something
like
um you know pain uh you know it's easy
to specify in terms of you know the the
signals that you're getting you know
from your nerves right you know you
constructed this situation where in
terms of the actual input data that your
brain is receiving it's easy to specify
this objective
um you know we can we can again you know
think about something like you know why
do humans you know end up valuing
something like sex well sex is easy to
specify in terms of the input data that
the human is receiving whereas like in
fact you know resulting in having a
bunch of DNA in the Next Generation
though that's correlated with having sex
is much harder to specify in terms of
the input data that humans receiving
right you have to do a bunch of
calculation to figure out okay is this
you know is this you know actually going
to result in me having a child is that
child going to you know survive and pass
on my DNA is a much more complex process
and it's the thing that actually
requires you know a lot of specification
about what is DNA you know how do
children work you know whereas you know
like you know something much more basic
you know that is just structured in
terms of you know what's what you know
base input signals am I receiving is
very easy to to specify um and so we
might expect they're going to be in a
situation where you know similar thing
happens to models right you you end up
learning
um you know objectives which are easy to
specify in terms of the data the model
is receiving
um rather than things that would require
some really complex you know computation
for the model to you know even be able
to you know specify right you know
understanding okay exactly what is DNA
requires you know specifying a bunch of
things about how cells work and you know
exactly how to determine what the DNA is
and what counts as mine uh you know
whereas you know something like pain is
very easy to specify it's like if we
receive a particular type of nerve
signal that's pain
and so you know again we might expect
you know expect that we're going to be
selecting four simple proxies proxies
that are that are easy to specify
and again that might not line up with
the sort of proxies that we want right
you know the sorts of things we actually
might want to optimize for may not be
the sorts of things that are easy to
specify
okay and so we're in this sort of
situation where you know we have again
some things which seem like they might
incentivize pseudo-alignment you know we
have situations where you know time bias
either the more biased that we have
towards you know fast models you know
that can do things quickly you know it's
going to bias towards proxies that are
very fast and quick to run optimization
on uh and you know Simplicity bias is
going to bias you know to the extent
that we're biasing towards you know
structurally simple functions we're
going to biostroid structurally simple
objectives they're easy to specify and
you know if we have really complex
environment with lots and lots of
different proxies then again we're going
to you know be biased towards you know
finding one that is not the one we want
just because there's so many of them
okay
um and you know again there are some
things we might be able to do to try to
mitigate this so you know we can imagine
something like adversarial training is a
situation where you know we could just
well you know in the case of the uh the
the maze environment where we have you
know going to the end of the Maze and
going to the Green Arrow well those are
two different things that are correlated
in the particular training environment
but if we just train on situations where
the the Green Arrow is not at the end of
the maze we can distinguish between them
and so we can try to do this right we
can try to just come up with a bunch of
various good training setups where we
can you know test to see is the model
doing the correct proxy or the incorrect
proxy we can do transparency right we
can try to understand directly what
proxies is it optimizing for right if we
think about something you know
previously where we had like the
algorithm with the you know looking for
cars but looking for Windows on top of
Wheels if we know that that's how it's
looking for cars then we can tell that
uh you know when that's going to fail
right you know if you know if that is
not that proxy is not going to work in
situations where you know somebody took
all the wheels off the car right and so
you know we can sort of understand with
transparency situations where you know
we have learned a proxy which is which
is sort of you know captures what we
want situations where it doesn't and
again if we give it like explicit
optimization then maybe we can sort of
better understand exactly what it is
optimizing for because we can sort of
see uh it has you know uh you know this
sort of uh you know you know preference
uh you know word function right
okay so these are some things that we
can try to do
um I think you know one thing again
that's worth pointing out is well you
know it seems like we're we're still
doing a lot more of the things on the
left right we're going to more and more
complex environments we're having larger
and larger models to have the ability
you know that you have the ability to
implement more complex things now that
that's sort of a little bit tricky on
the time and simplicity right it seems
like incentivizes less time bias and
more Simplicity bias
um but overall you know it's not clear
whether that helps or hurts us
um the complex environments definitely
seems like it hurts us
um but we can do some of these things
right we you know to try to mitigate
this
and so for the last bit of this talk
um we're going to shift gears a little
bit and talk about what might happen in
the limit of adversarial training
because I think that a lot of the
problems that I've talked about so far
are problems that are addressable via
the basic you know technique of
adversarial training right you know
stuff like the Green Arrow versus the
end of the maze are problems that I can
in theory address by just getting enough
environments where I can just test is my
model going to do the right thing right
is it going to go to the Green Arrow or
is it going to go to the end of the maze
and so I want to ask is that enough
right are there any situations where in
fact there are problems that could arise
even in the limit of every situation
that we can possibly check
um and so that is why we're going to
talk about deceptive alignment
and I will also I will spoil as well
this is a really complex topic the
entire next lecture after this one is
also going to be on deceptive lineman so
this is right now our sort of Dipping
our toes into trying to get a
comprehension around what what deceptive
alignment is and how it works but we're
gonna be talking about this in in great
detail next time as well so
um we're we're just sort of starting
with an overview right now
okay so uh I want to take a step back
and um ask a sort of theoretical
question
um before we sort of get in get into
this which which is well what is the
optimal strategy
for a pseudo-aligned Mesa Optimizer that
wants to you know is some let's say it's
in some RL environment and it wants to
you know accomplish some particular
thing in that oral environment
um and let's say he wants to do that
thing as much as it possibly can
so let's say for example you know it
wants to go to the Green Arrow and wants
to go to Green Arrow as many times as it
possibly can go to the Green Arrow
that's like that's only one possible
objective the model could have it's one
thing that it could be doing it's not
necessarily the only thing that could be
doing but I want to imagine a situation
where you know we have a model and
that's what it's trying to optimize for
it's trying to look for a policy which
results in it goes to the Green Arrow as
much as it can
okay
so so what should it do well
um here are two possible policies
um that it could implement so policy
number one is see a green arrow go to
Green Arrow this is the you know the
most straightforward policy
um and we can ask well how well does
this do how many green arrows does this
get and the problem with this is that if
I'm doing adversarial training right we
want to understand what happens in the
limit of azure cell training if I'm
doing adversarial training where I'm
trying to see if my model goes to Green
Arrow or it goes to the end I really
don't want it to pick up these random
green arrows if it goes to the Green
Arrow
then the gradient descent process will
modify the model so it doesn't go into a
green arrow again uh that's that's you
know that's what machine learning does
right when we see a situation where we
have an algorithm and that algorithm
doesn't fit our data we keep searching
in our space until we find an algorithm
that does fit our data right and so if
we're in a situation where the model
goes to Green Arrow and we see it then
we can train it away and now it won't go
to the Green Arrow again and so how many
green arrows does it get in that
situation well you know approximately
one right it gets to go to the Green
Arrow once and then grain descent
modifies it and it no longer goes the
Green Arrow and you know it got one
green arrow
okay that's that's one possible policy
here's another strategy that this model
could pursue here's this is what it does
in training
it goes to the end of the maze
uh because it doesn't want to be
modified to go to the Green Arrow
and you know again you know it's worth
talking about okay but like but it's
really trying to go the Green Arrow
right shouldn't grain descent modify it
to want to go to the end of the maze ah
no right that's the does the right thing
abstraction right what grainy Ascent
does the actual structural thing the
gradient descent does is it checks you
know is this algorithm an algorithm
which has good performance on this data
and what can we change about this
algorithm because it have better
performance on this particular data
right it doesn't care about whether in
fact the model will eventually have good
performance on some other data points
you know that aren't that don't exist in
training and it doesn't care about
whether the model is actually trying to
do the thing that it's looking for just
cares behaviorally does it look like
it's doing the right thing and if the
model is going to the end of the maze
successfully the thing that we want in
training the gradient descent is totally
fine with that
and so a thing that it can do right is
go to the end of the maze the thing that
we want in training and then in
deployment you know when it no longer
has a grading descent process watching
over it it instead goes to the Green
Arrow it does the thing that we don't
want in deployment and we'll imagine for
the purposes of right now that it can
always tell whether it's in training a
deployment but that we you know gradient
descent you know we know have no
examples of situations that look like
deployment that we can actually give you
know in training so we can always tell
you know what the difference is well
we'll talk later about like you know why
that might or may not be the case
um
okay so this is you know a very
hypothetical you know we've just
imagined a situation where we have a
model and you know they're trying to do
some you know crazy thing you know which
is go to the Green Arrow as much as
possible
um but it is worth pointing out that in
theory the thing that that model would
want to do the sort of most optimal
strategy for it is one that looks pretty
bad it's one that looks like a situation
where it tricks the training process
right it does it tricks us you know in
training it does the wrong it does the
thing that we want in training and then
in deployment it does a totally
different thing which is the thing that
it actually want to do right and in this
situation right it gets way more green
arrows because if deployment is like you
know in this case you know three steps
long it gets three green arrows you know
because they keep going to the Green
Arrow and deployment because there's no
you know Green descent process watching
over it
um and so this is potentially concerned
yeah questions
so it occurs to me that you've got this
uh maze example so if the agent is
trying not to go to the Green Arrow
because it will be gradients descent
selected away from this that means in
some level the model has to know what
the wheel objective is there at least
the model the objective is not the Green
Arrow because if they thought the
objective really was the Green Arrow it
would model but knowing there would
actually cause it to be reinforced right
that's exactly correct and we'll talk in
a little bit about like what are the
conditions that need to be true from
model to sort of exhibit this behavior
and that's one of them right for a model
to exhibit this Behavior it has to know
in some sense what the gradient descent
process what the machine learning
process is actually trying to get it to
do um if it doesn't know that then then
you you won't ever see this but right
now we're just looking at it in theory
we're just like okay in theory if the
model knew all the relevant facts you
know what would be the best thing for it
to do
question
so do the incentives of the agent depend
on their attack because if I think about
like the r ribbons aren't familiar with
it seems like the way they work is that
when the agent does something that leads
to a gamer of all that gets reinforced
and so it does that more often and so it
seems like that would be an incentive
against it like going to the end of the
maze because then it would be
if I do that then they go to the end of
the maze thing we reinforced and I'll be
more likely to do that in deployment
which I know on
yes that's a really good point so it is
totally the case that grand descent will
reinforce right the thing that caused it
to go to the end of the maze but the
thing that's so pernicious in this case
is that the thing that caused the model
to go to the end of the maze in training
was the fact that it wanted to go to the
green arrow in deployment right its
desire to eventually go to the Green
Arrow was the reason that it actually
did a good job in training and so if
grain descent is just reinforcing
whatever caused it to have good behavior
and training in this case the thing that
caused to have good behavior in training
was the fact that eventually it wanted
to trick you right that was the reason
that it had good behavior and so in this
case you know the thing that should sort
of get reinforced is just the same you
know exact model right here's another
way of thinking about this right what
greatness is doing is it's just looking
at you know if I were to change the
parameters right in what direction you
know would I get better performance but
if your model already has Optimal
Performance if it's already sort of you
know always does the right thing in
training then the ingredient descent has
nothing to say right you know grain
descent is not going to say ah you know
this model has Optimal Performance you
know but there are these other examples
these other training examples you know
these deployment examples that I've
never seen before that it would have
better performance on if I changed it
green descent doesn't know that right
you can't see these deployment examples
because there doesn't know that the
model would theoretically have bad
performance there right and so it's not
selecting over that at all now there is
one thing that it's still selecting over
even at Optimal Performance which is
well there's there's some inductive bias
right there's still a Simplicity bias
right and so we'll talk about that a
bunch uh you know later about but but
yes there is still a thing that is
happening even in that situation which
is just like well you know one thing
that could happen is maybe you know the
the sort of the deceptive policy here is
just like too complex it's not you know
the sort of structurally simple
algorithm and so even though it has good
performance you might sort of just be
pushed by that the inductive biases you
know in we saw like with rocking where
eventually finds the correct algorithm
it might be pushed for that reason into
finding the correct thing but it
wouldn't be pushed just for the purpose
of having
um good performance because it has
perfect performance you know it's
already doing the right thing in
training okay we'll talk a bunch more
about this later
okay so let's go through a little bit
more about you know what What's
Happening Here how might this arise you
know what are the basic factors and
again we'll be talking about this a
bunch more next time as well
okay so how would a deceptive alignment
actually happen
so we're going to distinguish between
two different ways in which a Mesa
Optimizer could learn the base objective
you know the objective of the
um
the you know the the
um the greatest end process right we
identified previously one of the things
I said previously was that knowing what
the objective is that the base Optimizer
is trying to get it to do is sort of
critical for having a deceptively
aligned model
and so the first thing we can ask it's
also you know another thing that's
pointing out worth pointing out it's
also critical having a line model right
like any model that's going to do the
right thing needs to know what the right
thing is and so at some point it has to
have learned what the thing is that
you're trying to get to do that you want
to want it to do okay so here are two
mechanisms via which a model could learn
what the right thing is what the thing
is you're trying to get to do
mechanism number one is internalization
this is a situation where you know
you've got some may subjective it lives
in the model and over you know many you
know course many you know optimization
steps of grading descent it slowly
refines that objective you can tell that
objective directly corresponds to the
thing that you know that we want that
actually does a good job in all examples
that we can test in the relaxer with you
know with doing average cell training
okay that's option number one
option number two is what we're going to
call modeling which as well one thing
that generally happens so I have a uh
you know model you know I haven't a
parameterized function I have you know
algorithm in some situation and has to
be able to do a good job in that
situation is it generally we'll learn
facts about the world because one thing
that is true is that just knowing more
stuff about the world is usually just
convergently helpful for doing a job in
basically any environment and so instead
of you know grain descent modifying the
models like May subjective until it gets
better and better we could imagine a
situation where the green descent is
just modifying the model's knowledge
about the world and it keeps modifying
the model's knowledge about the world
until it gets to the point where
um that knowledge about the world
contains information about the base
objective right like one thing that is
true about the world is that there
exists some optimization process where
the model is being optimized over and
it's optimizing for some you know loss
function or whatever and so we could
imagine all you know hypothesis too is
that the way that it learns about the
the base objective is through learning
more and more facts about the world
question
so if the graded descent what a strength
or gradient descent to adjust the model
is basically constant no matter how
strong the model is but smarter models
become better at modeling does that mean
that number two is actually going to
become more and more common as models
become more capable relative to the
first one like it will become better at
moment but won't actually get any better
at internalization
so I am going to advance that hypothesis
that modeling is going to be more likely
than internalization
um yes uh so let me yeah let me talk
about some of the reasons why I I think
model modeling might be might be that
internalization I think I basically
agree with what you were saying
um
so uh before I do that though I just
want to distinguish between two ways in
which modeling can can happen right so
an important thing right about the
modeling is right so let's say so so it
keeps learning more facts about the
world right as it's um you know
understanding of the world is modified
um but for for that understanding of the
base objective in its understanding of
the world to actually result in the
model having good performance in
training there's additional step which
is needed right you it's not enough to
just know what the base objective is you
have to also care about it right it has
to want to optimize for it for some
reason and so there are two reasons I'm
going to identify that you could have a
model that sort of knows what the base
objective is and wants to do what the
base objective wants it to do right so
reason number one is deception right
reason number one is if the model has
some random you know it has some uh May
subjective and that may subjective you
know cares about something happening in
deployment it has a sort of long-term
base objective that you know cares about
something happening later
then for the purposes of sticking around
and training it'll want to do what the
base objective wants and so if modeling
happens and the model understands the
base objective via understanding the
world then one way in which that
understanding could result in good
training performance is via deception
okay but there's a second one which is
we're going to call courage ability the
sort of cordial alignment case where the
idea here is
you could instead have a situation where
the model learns you know what the base
objective wants to do just be
understanding the world and then it's
mace objective is just sort of thrown
out and instead it's made subjective is
replaced with sort of a pointer to that
understanding in the world right now
that it understands that green descent
can just sort of take the model and be
like okay this old main subjective that
we have sitting around here the
correspondences some proxies will just
get rid of those proxies now that you've
learned what the true base objective is
just do that right point to it in some
sense but this requires encoding some
sort of pointer to you know an
understanding of how to specify the you
know the what what what knowledge in the
world actually is the base objective
Yeah question
uh first one to submit for people with
no background could you do a brief
description of what courage ability
means
yeah that's a really good question so
I'm I'm not exactly using courage
ability in the sort of full technical
sense here
um if you're not familiar with the word
it just sort of refers to a situation
where
um you have one sort of agent that is
sort of you know willing to help and uh
you know act in the ways of another
agent is willing to listen to feedback
and you know change as it corresponds to
what that other agent wants to do so in
the sort of case the reason we're sort
of calling this a quadrival case is the
situation where if you had a model that
had this sort of property where like it
sort of was trying to figure out what it
was that the base objective wanted to do
then if Yusuf came to it and you sort of
told it that actually the basic director
was doing something different you were
like you know actually what I really
want you to do is this other thing then
it would sort of listen and it would do
that and so it's sort of amenable to
feedback in that sense and so that's
sort of why we're going to call it
cordial here
one thing that is worth pointing out is
that in some sense the deceptive model
is also amenable to feedback because I
can go to the deceptive model and can be
like hey deceptive model you know what I
actually want you to do is you know this
other thing and the deceptive model as
long as you're still in training we'll
be like oh you know the best way for me
to like you know you know get into
deployment with my current objective is
to do this new thing you're telling me
to do and so the deceptive model is also
meant all the feedback but
obviously not in the way that we want
because eventually it is no longer a
mental to feedback you know once it is
in a situation where it's in deployment
you know and it's trying to do it's it's
you know Green Arrow thing or whatever
it actually wants then it's not a mental
feedback anymore
okay question again
well let's talk about message optimizes
for a while now what I'm wondering is is
this form of decepting a chargeable
alignment does that require a mess
Optimizer for this problem to occur that
occur for just an outer model as well
yeah so it's a bit of a complicated
question I think that I'm sort of mostly
concerned about situations where like
the model is deceptive and it's also
like you know very capable of
distribution right I think you could
imagine a situation where it's not
really well described as an Optimizer it
does isn't doing some sort of capable
optimization process
um but it's still sort of deceptive I
think
um most of the situations I imagine
where a model is like in fact doing
something really capable and concerning
are situations where it is doing some
sort of optimization
um I don't think like doing some sort of
optimization is really not that high a
bar you know
um but it does sort of depend really on
exactly how you're conceptualizing and
thinking about what sort of counts as a
mace Optimizer
um you you could imagine a situation
where sort of anonymous Optimizer is
still deceptive we're definitely not
going to be focusing on that at least
right now
um we'll maybe sort of talk take a
little bit of a more broad stance next
time but we're still mostly going to be
focusing on situations where a while
okay for it to be deceptive it has to be
capable of doing the reasoning you know
to accomplish goal X I have to you know
pretend to do goal why and to be able to
capable to be capable of that step of
reasoning sort of seems to require some
amount of like thinking through the
consequences of actions that it sort of
seems to require some amounts of
optimization
so
um I like to sort of take some really
strong stance but it seems to me like
most of the sort of at least the
concerning cases seem to seem to require
some amount of optimization um it's
worth pointing out also they require
more things than that right like it's
not the case that any mace Optimizer
would be deceptive because it also like
like I mentioned it has to care about
you know something in the long term
right it has to care about something in
deployment right if he doesn't care
about anything happening in deployment
then then it has no reason to be
deceptive um well yeah one of the other
things we'll talk about next time is
like you know how could that develop you
know that and why would it develop
um we'll talk a little about it here as
well
okay so I said I was gonna make some
sort of arguments for why I think you
know modeling would happen rather than
internalization
um I think the most basic argument is
just
um you know
when you give your model a bunch of
information about the world and training
right it's seen like you know all of the
internet uh it's uh you should expect
that any behavior that your model has is
gonna make use of that information right
you know it just sort of is a waste of
complexity and you know space in the
model to not have the behaviors of the
model sort of specified in terms of the
knowledge that the model has about the
world and so you know it's read the
entire Wikipedia page on ethics it has
to be able to summarize that page anyway
because that was sort of one of the
tasks that we wanted to be able to do
well uh it's probably going to use that
existing faculty of knowing what the
Wikipedia Wikipedia page on ethics
contains to help it understand how to
you know do a good job uh you know on
human values or whatever it is that
you're trying to get it to do
um and so you know in for that basic
reason I think we should expect modeling
you know it's just substantially similar
right it doesn't require this you know
learning some some extra thing
um and then again we can sort of make a
similar case for why we might expect
deception over the sort of cordial case
though this is I think a little bit
trickier
um but you know we can sort of Imagine a
situation
um where we have a
um
we sort of have this corrigible um you
know align model and it has some you
know notion of what it means to sort of
specify uh you know and point to you
know something in the environment that
corresponds to what it wants but there's
sort of a structural problem which is
well pointing can actually be really
difficult it can be really hard so an
example of you know of sort of a thing
that might be well described as a
cordial model in this case would be
something like a duck so Ducks sort of
have this imprinting Behavior right
where when they're born you know they
try to figure out who their mother is
you know by like looking for what things
are nearby where they're born right and
then they they imprint on that and
that's their mother and that's what they
care about right and this works pretty
well in situations where they are in
fact born from a mother duck but in
situations where they're you know
incubated by a human uh it doesn't work
so well right and so actually having an
ability to understand which part of the
world you know is the thing that I
should be caring about is the thing that
I should you know have my objective
correspond to can be a really complex
and difficult structural process
um and so it might you know that sort of
is is a difficult you know a thing that
might make courage ability more
difficult the thing about deception is
it doesn't quite have to do that because
it just sort of infers it right it's
like well I want to do whatever it is
that in fact has the result that it
means I do a good job in in training and
and you know I'm eventually able to do
what I want in deployment and so you
know in that case it'll just infer you
know in this case does it make sense to
you know do the duck or the human uh you
know which one will in fact help me the
most at accomplishing my goal and so it
infers the correct thing to do
um rather than having to sort of have
you know some correct way to specify
which thing in the world it wants um you
know in the model and so this is another
thing we're going to talk about more
next time I think this is a little bit
tricky
um especially because it sort of relies
on the distinction between things like
you know speed complexity and time
complex speaking vaccines and you know
description complexity
um because the deceptive model still has
to sort of do the computation to figure
out what the pointer is but it doesn't
have to specify it uh in its in in sort
of its basic structure Yeah question
and deliver the techniques like from
where we actually might be able to edit
facts or knowledge in the model before
thinking about this not literally as a
pointer but like some kind of direct
lighting mechanism would we expect
courageable align models to be able to
to make it easier to change them or
change their objective because we just
had from the surgery on The Wire
I mean I think that's a really
interesting hypothesis it seems
plausible to me um I don't know
um but it seems quite plausible yeah
okay uh great so just sort of recapping
what we sort of talked about our basic
story for how deceptive alignment might
arise is that we start with you know
some misaligned pseudo-aligned mace
Optimizer it has some proxy uh and then
that proxy uh you know maybe becomes
long term and it gets it gets modified
you know by the green descent process
into using that you know optimizing for
that proxy over the long term for uh and
uh that results in it you know wanting
to do the correct thing that we actually
want in the short term because uh if it
wants to really do a good job in the
long term the best way to do that is
sort of play along the short term and so
if if we sort of end up with you know we
start with this with it has some proxy
that proxy becomes long term and then
you know once it becomes long-term it
starts you know trying to
um you know optimize for uh you know
what we want for the purpose of staying
around and one thing right that's worth
putting out here is the gradient descent
sort of wants to give the model better
performance and to the extent that
deceptive aligned models have good
performance one way to give the model
better performance is to cause it to
have that long-term objective that
results in it being deceptive and so
green descent is sort of potentially
trying to make the model deceptive here
and then once the model has some you
know deceptive you know the reason that
it has good performance and training is
because it's trying to do you know the
wrong thing because it is like you know
it's trying to do the thing you want for
the purpose eventually doing some other
thing once that happens it doesn't
matter at all after that point what
thing it is the model is actually trying
to do like whatever the mace objective
is is completely relevant to training
performance because regardless of
whatever it is it's still going to want
you know as long as it's some long-term
objective it's still going to want to
you know play or play along in training
and so we we can say it sort of
crystallizes at that point there's sort
of no green descent updates which sort
of matter to it
um and then at that point you know you
know Green descent is sort of no longer
going to be improving the models May
subjective because it doesn't matter for
the purpose of performance in that point
the only thing that matters for
performance is how good the model is at
being deceptive and so we should expect
that you know at this point green
descent is sort of just trying to make
the model better at being deceptive you
know once we have this depth of model
because that's the only thing that's
relevant to its performance at that
point
okay uh and so one final thing that I
want to point out I said I would talk
about this again uh is that well okay so
there's one other piece that's necessary
here which is your deceptively aligned
model has to be able to tell whether
it's in training or deployment right if
it can't tell then it can't execute the
strategy
except that's not entirely true because
uh you know the first thing that's worth
pointing out is that it can gamble right
it can be in a situation where it's like
okay I don't know whether I'm going to
be training or deployment but
um I can you know hope that I sort of do
the right thing with some probability
and I do the wrong thing you know the
thing I actually want with some
probability such that it's likely that
I'll be able to get you know multiple
steps of the thing I uh you know I I
sort of want to actually be able to do
in deployment eventually if the
deployment environment especially is
larger than the training environment
then it can just sort of defect you know
do the do the thing it really wants to
do with some small probability such that
it's unlikely for that to show up in
training but yet it still gets to do it
multiple times in deployment
okay
um but in fact it's also even worse than
this I think this is sort of the basic
case that's like okay even if it can't
tell if you're it's this is still
potentially concerning but but also
probably can tell so why is that well
most of the time when we sort of are
concerned about situations where
training and deployment differ
um we're sort of imagining that you know
most of the small changes don't matter
right if I train a model in simulation
and I deploy it in the real world we're
like okay you know as long as simulation
is pretty close we should be able to get
transfer but there's you know if your
model is actively trying to look for
differences between the training and
deployment distribution then you're
really in trouble because then any
situation where anything that the model
tells can tell is different about those
two situations becomes something that is
sort of you know that it's looking for
and one other thing that I should point
out here right is that I say training a
deployment but the really important
thing that the model cares about is the
distinction between a situation where if
it does the wrong thing like the thing
you don't want it'll be punished and it
won't be able to do it again and a
situation where that's not the case
where it can do the thing it wants to do
and it'll be able to get away with it
right and so even if we're not in some
standard training appointment setup
where we're trying to do some maybe
online learning try to always keep you
know some great descent process involved
at some point if we ever deploy our
models in situations where the oversight
isn't sufficient to like you know you
know come in and stop it from doing the
thing we don't want then this this sort
of happens and so if we might have you
know that's sort of what the deceptive
model is looking for so the concern is
that as long as we are in a situation
we're eventually going to have to deploy
models in cases where we actually have
to trust their output then we sort of
have to to deal you know with with the
problem eventually being in this
situation and so then the question is
okay are there any other things that
your model could be looking for that
would help it determine whether it is in
a situation where there is no longer
sort of efficient oversight to be able
to prevent it from doing the you know
the the sort of thing we don't want to
do
um yeah question
uh could you elaborate I've been on one
of those last points which is like the
whole point was that
the agent does not want to say go to the
Green Arrow because then it will be
grieved or sent it away if it is always
being greatly descended including in
deployment could you elaborate more on
why you expect this to still not work
yeah good question so I think that like
here's an example uh let's say
eventually the model realizes that
there's something they can do that turns
off the gradient descent process right
and it's been a really really long time
you know where the green descent process
has sort of been working but then
suddenly if you end up in a situation
where now the model has discovered a way
to turn it off then it stops working
right and so what the model is looking
for is a situation where it can you know
do something that subverts that process
maybe because you know it could be
because there's always a great
understand process and it finds some way
to maybe get rid of it or turn it off or
something or it could be a situation
where you know maybe we just deploy it
in a setting where there's not a green
descent process right like if you give
everyone you know chat app you know on
their phone everyone has the ability to
interact with it well you can't always
maybe have a green descent process maybe
eventually you know if the model starts
really doing bad things all the time you
know we're really concerned about it
um and so you know the situation is
basically just any situation where you
know if the model did something that
would be really bad you know we wouldn't
be able to correct it from it and as
long as we're you know event you're
going to be deploying these models in
more and more situations I think we're
sort of eventually going to have to deal
with that problem
okay okay and so again so it is worth
pointing out that there are there are
other things that your model could be
looking for other than just uh you know
um uh you know these sort of basic
differences right it can it can look for
a lot of various different things
because it's actively trying to find
these differences so here are some
examples of things that could be looking
for you know that might tell it that now
it's in deployment you know now it's in
some new situation where it wasn't like
you know being trained so for example
you know I talked about similar you know
training and simulation at the point in
the real world
um another thing you know is it just
looks at the date right it just looks
after the date right you know you're
like okay for most normal models you
know it shouldn't matter very much you
know what the date is right it shouldn't
be caring about that but for a model
that is deceptive it's actively trying
to look for any difference you know you
know even some even something sort of
simple and innocuous that this can tell
it that ah now it's probably not just in
simulation now you know the date says it
must be you know in the future you know
in some new situation where it's being
deployed but of course this way you can
easily fool and so then the question is
are there any things that the model
could look for that if we did a really
really good job of trying to make it so
there was no perceptible difference
between training and deployment is there
anything that the model could tell that
they would sort of give it give it away
and the answer is yes so this is the
sort of classic example of this it um
this is a very large
um uh it's not a prime number but it's
the product of two very large Primes and
um
the it's sort of not is not publicly
known what the two you know large prime
factors are of this number but there's a
challenge where somebody gets a bunch of
money if they're able to figure out what
the prime factors are this number
and the problem is that currently we do
not have the computational resources
to be able to factor this number it's
too big it's too hard
and so the question is well okay
you know what if I'm an deceptive model
and I do something very devious I look
for a factorization of RSA 2048 of this
number I look for this numbers
factorization and if I see the
factorization
then I you know go to the Green Arrow I
do my devious little thing and if I
don't see the factorization I play along
like a good little AI
and so then what happens right what
happens if I have a model doing this
well the problem with this is that
there's there's two things happening
here one is that there's no we cannot
computationally construct inputs that we
could ever possibly feed to a model that
has this property that would cause it
for us to see the deceptive Behavior
because we will not be able to do that
until bigger computers come around right
and so we just computationally can't
test it effectively that's probably
that's the first thing that happens and
then the second property of this example
is that RSA 24a is eventually going to
be factored right eventually Somebody
almost surely is going to factor this
number right as computers get bigger
somebody's eventually going to have
enough compute and they're going to
throw it at this and do it and so very
very likely you have a deceptive model
that knows for a fact that eventually it
will observe a factorization of RCA 2048
and also knows for a fact that there is
no conceivable way that during training
you could ever present it with such a
factorization
and so this is an example of a case
where there is a property of the world
that is structurally easy for the model
to check whether that property holds but
very very difficult for us to construct
an example that has that property
um and so it's sort of impossible for us
to ever generate an input that causes
the model to do something bad in
training
and I want to point out you know this is
a very particular you know sort of
cryptographic example but I think this
is emblematic of a much sort of more
general you know uh fact which is that
um there it is generally true about the
world that there are properties which
you can evaluate on the world which are
easy to evaluate but hard to generate
inputs which have that property right
this is a basic struck you know
structural fact related to things like P
versus NP where there are certain
problems you know which are just easier
to generate examples than they are
actually or easier to verify than they
are to actually generate examples that
have that property and so as long as
there exists properties on the world
that are easier for models to check than
they are for us to ever generate
examples then we will always be in a
situation where our model is going to
know more information and be able to
determine more information about where
it is than we will ever be able to sort
of trick it into believing that question
so again kind of going into like
mechanistically going into the model it
also needs to somehow represent that has
to remember the fact that it has to go
look for this thing out of distribution
so if we had a way to officially search
for the encoding of that little piece of
memory to go look for that thing then
that would be a potential way to
mechanistically catch the model before
it's a
I think that's a really good point and
in fact I totally agree that's like a
thing that we we can try here that might
help so we're talking here about the
limit of adversarial training you can
imagine that even The Limited
adversarial training there are still
things you know where adversible
training isn't helpful but maybe stuff
like transparency like you're saying
would be helpful for addressing this
problem I think that's absolutely
correct one thing I will say though is
that it's not necessarily the case that
the model has to have like some explicit
circuit which checks RC 2048 so an
example which I'll sometimes give here
is we can imagine a situation where like
I'm training a model and I'm trying to
get this you know I'm training like a
robot to sweep a factory floor and then
suddenly the robot finds itself in the
jungle right it's not like the model you
know the robot has like where all of its
life been like running some check am I
in a you know in a jungle or am I in a
you know a factory it's just sort of
trying to understand you know okay given
that I'm currently in a factory the best
strategy for me is to you know sweep
this Factory because eventually I might
be somewhere else and I'll be able to do
some other thing
but then suddenly it gets some piece of
information that it wasn't explicitly
expecting but that you know is
overwhelmingly convincing that it's now
in a jungle then you know it's going to
start doing some crazy stuff you know
based on whatever you know it would do
in the in the new Jungle environment so
the point is that your model isn't
necessarily like explicitly looking for
a factorization of RC 2048 uh you know
all all it might be doing is just like
looking at the information available to
it trying to integrate and incorporate
that information to build new good
beliefs about what is happening in the
world and it's just in fact the case
that RC 2048 is an example of a really
really convincing thing that would
convince someone if they saw it that
they must be in the real in the real
world or not in you know in training
right and so there might be many other
things that might just be in general
doing this process of look at
information in the world integrate that
information try to form beliefs and then
if it you know discovers that in fact
the the result of that integration is
that it's very likely to be you know no
longer in training and then it does
something different right and so it need
not be explicitly looking for that
factorization
um it might be you know there totally
could be a model that just like has some
explicit circuit that's just like
directly looking for you know some
particular thing to you know test to see
whether you should start you know doing
something bad
um but it's not necessary right you can
absolutely have models with this
property that um you know that aren't
aren't explicitly doing that so in that
case you might still be able to use
transparency to help you know you could
still try to look for you know this
integration you know how it's processing
information and the fact that it tries
to figure out whether it's in deployment
or not based on you know that
information processing
um but but you couldn't necessarily
literally explicitly look for a search a
circuit that was trying to check uh you
know for a factorization of a number
okay and
um yeah so that is the end of Talk
number two uh like I promised we're
gonna be talking even more about
deception next time and really sort of
trying to go into detail on How likely
it is
um but yeah that's that's it for right
now if you're interested in a bunch of
things I talked about also you can go
read the risks from learning
optimization paper this was based on
um but yeah so if there's any questions
at the end we can do that now
[Applause]
okay yes question
uh
so I wanted to ask about middle
optimization for the case of
When Future GPT version about the
unCloud Turing case
it's some kind of alaytip or in fact
GPT thing
um
tenants occur there How likely is that
[Music]
other some candidates more potential
give me the objectives that are
not quite the right thing we want
yes that's a really good question I
think my primary answer is going to be
I'm going to punt on this because I'm
going to be talking later on in one of
the later talks about like specifically
about language models and like you know
what sorts of things we might imagine
that they're doing
um
but I mean I guess like the simple
answer though is it totally could be
running an optimization process we have
no reason to believe that it's not right
the only reason the only thing we know
is that is you know the simplest model
that fits a bunch of internet data right
and so you know it could be that that
model is like the sort of model that you
know runs some operation algorithm or it
could not be
um and uh you know it's a tricky
question answer we don't know exactly
what the answer is
um I think that my guess would probably
be that like pre-trained models probably
are not doing this sort of optimization
that we're concerned about at least not
on all inputs though they might be doing
it on some inputs so you might still
condition it you know you might give it
a prompt in such a way that you want it
to act like a you know an agent that is
really trying to optimize for something
and then maybe to be able to predict
agents like we were talking about you
know to be able to predict humans you
know it might have to then try to act
like you know an Optimizer in some
concerning way
um but yeah I don't want to go too much
into this I think basically the answer
is maybe and it also might be input
dependent it might be also dependent on
how you train it so you know maybe once
I do Rolex for other sorts of you know
fine-tuning techniques I could make it
into something concerning I think it's
very unclear you know exactly what the
answer is here but yeah we'll talk about
we'll come back to this and talk about
this more in a later talk
okay well uh there's no more questions
we can call it here and okay one last
question go for it so I have a question
about what deceptive alignment would
look like in the case of like multitask
screen learning so I guess during
training you're giving them AI a bunch
of delight or it's basically being
trained in like self-supervised learning
or to like achieve good performance or
like multiple tests multiple objectives
so I guess in training or I mean sorry
Ed inference what you do is you just
actually just give it the objective the
basic objective that you wanted to
maximize and you can find a policy they
can maximize that so I mean what would
deploy our deceptive alignment look like
in this case
like changes even if you're in a
situation where like sometimes they have
to do one thing sometimes I ask you
through another thing then the objective
is just do what I say and it's do what I
say in this very particular defined way
right and so then even in that situation
you can be like okay
you could have a model which is doing
what I say in this particular defined
way that is correct
um and that would be you know a robust
online model or you could have a model
that is like you know doing it wants to
do some totally different thing but you
know it is pretending to do what I say
in this in this defined way in this
particular case so that eventually could
do some other thing right and so you can
still have a situation where like you
know all these sort of Concepts to apply
even if I'm like you know it's just like
the thing you're trying to get to do
there is like rather than one fixed
thing it's just like do whatever I say
or whatever
okay great uh yeah we'll call it here
and uh next time we'll talk more about
deception
[Applause]