Risks from Learned Optimization: Evan Hubinger at MLAB2

hello everybody I am Evan hubinger I am
a research fellow at Mary I used to work
at open AI I've done a bunch of stuff
with Paul Christiano uh I also wrote
this paper that I'm talking about today
uh
let me very quickly raise hands if you
have had any experience with the paper
I'm going to be talking about let's see
okay okay pretty good sweet okay uh
we could even potentially I could give a
follow-up talk that covers more advanced
stuff but okay I think that was not
quite enough hands so I think I'm going
to be doing this talk and hopefully
it'll help people at least work through
the material understand stuff better I
think you will probably understand it
better from having seen the talk at the
very least maybe at some other point if
I stick around I can give another talk
okay so yes let's get started
so let's see uh about me uh this talk
was made when I was at openai I'm
currently Miri I did other stuff before
that
um I've given this talk in a bunch of
places it's based on this paper
you know it I think some of you
okay what are we talking about
okay so I want to start uh with just
going back and trying to understand what
machine learning does and how it works
uh because it's very important to
understand how machine learning works if
we're going to try to talk about it uh
so what does machine learning do so so
fundamentally you know any machine
learning training process it has some
model space some you know large space of
algorithms and then does some really big
search over that algorithm space to find
some algorithm which performs well
empirically on some uh set of data on
some loss function over that data
okay this is what essentially everything
looks like this is what RL looks like
this is what you know supervised
learning looks like this but you know
fine-tuning looks like everything looks
essentially like we have some really big
parameterized space we do a big search
over it we find an algorithm which does
well on some loss over some data
okay so I think that when a lot of
people like to think about this process
there's an abstraction that is really
common in thinking about it and an
abstraction I think can be very useful
and I call that abstraction that he does
the right thing abstraction
so well when we trained this you know
model when we produced this uh you know
particular parameterization this
particular algorithm over this
particular data
um well we selected that algorithm to do
a good job on this loss on that data and
so we sort of want to conceptually think
about the algorithm as Trying to
minimize the loss you know if you're
like how is this algorithm going to
generalize you can sort of think well
you know what would it do uh what it
would that would be loss minimizing what
would be the sort of loss minimizing
Behavior uh but of course this is not
literally true we did not literally
produce an algorithm that is in fact
minimizing the loss on all off dispution
points what we have produced is we
produce an algorithm that empirically
was observed to minimize the loss on the
training data but in fact when we move
it into new situations uh you know it
could do anything
um but you know well we selected to
minimize the loss and so we'd like to
think uh probably it's going to do
something like the loss minimizing
Behavior
okay so uh this abstraction you know
like all abstractions that are not
trivial are leaky and we are going to be
talking about ways in which this
abstraction is leaky
okay so I'm going to be talking about a
very specific situation uh where I think
this abstraction can be leaky in a way
that I think is problematic so that
situation is a situation where you have
uh a the algorithm itself is doing some
sort of optimization so what do I mean
by that so we're going to say that a
system is an Optimizer if it is
internally searching through some space
of possible uh plans strategies actions
whatever uh you know for those that
score highly on some criteria so you
know it's maybe maybe it's looking for
actions that would get low loss maybe it
is looking for actions that would get it
gold coins maybe it is looking for
actions that would do a good job of
predicting the future whatever it's
looking for things that do a good job on
some criteria okay
so just really quickly gradient descent
is an Optimizer because it looks through
possible algorithms to find those that
do a good job empirically on the loss uh
you know a Minimax algorithm is an
Optimizer it looks for you know moves
that do a good job at playing the game
humans are optimizers we look for you
know plans strategies that accomplish
our goals things that are not optimizers
a bottle cap is not an Optimizer this is
a classic example you could take a
bottle of water and it's got the water
in the bottle and it's like really good
at keeping the water in the bottle you
can like turn the bottle over and the
water stays in but you know that's not
because the bottle cap is doing some
sort of optimization procedure it's
because we did an optimization procedure
to produce a bottle and that bottle is
then really good in the same way that a
gradient descent process by default does
an optimization procedure over the space
of algorithms and produces a you know
neural network that is not necessarily
itself doing optimization you know it
may just be like the bottle cap it was a
thing that was optimized to do something
but it's not necessarily doing any
optimization itself certainly if I just
randomly initialize a neural network it
is definitely only not going to be doing
any optimization unless I get really
really unlucky or lucky
um
okay all right so that is what an
Optimizer is
okay so what we want to talk about and I
alluded to this previously is a
situation where the uh model itself uh
the algorithm that you found when you
did this big search is itself an
Optimizer it is doing some optimization
internally inside of your neural network
uh so I'm going to give names to things
in in that particular situation where
the thing is an Optimizer in that
situation we're going to call the grainy
descent process that did a bunch of
optimization on top to produce the model
we're going to call it a base Optimizer
and we're going to call the thing that
it optimized the algorithm that it found
that gradient descent found that was in
fact doing some optimization we're going
to call it a mace Optimizer what does
that mean so Mesa is sort of the
opposite of meta you know oftentimes you
may be familiar with meta learning you
know we have an optimization process
then we put a little meta learning
optimization process on top of it that
like searches over possible ways the
learning process could go and we're like
well that's sort of what happened here
you know you had this gradient descent
process and you're sort of grading
descent process turned into a meta
learning process now instead of
searching over you know you know a whole
Space of algorithms it's sort of
specifically searching over learning
algorithms and so we're going to say
well essentially the relationship to the
base Optimizer to the model is very
similar to the relationship between a
meta Optimizer and a sort of the thing
that meta Optimizer is optimizing so
we're going to say well it's one meta
level below it's a Mesa optimizer
okay
all right so some things that people
often misunderstand here so when I say
mace Optimizer I don't mean like some
subsystem or some you know component of
the model I just mean like a model you
know it's a neural network and it's
doing some stuff and you know in
particular we're referring to a
situation where it is doing you know
some sort of search it's you know
searching over some space of possible
plans strategies whatever for something
which does a good job on some criteria
so it's the whole trained model
um okay so I talked a little bit about
the relationship with meta learning uh
but you know essentially you can sort of
think about this as spontaneous meta
learning you know uh you you didn't sort
of think you were going to be doing meta
learning that you were doing
optimization over learning processes but
in fact the algorithm that your grading
descent process found that was doing a
good job on the task was itself a
learning algorithm and so now you're
you're in the business of doing meta
learning because that's what algorithm
you found
okay and so the difficulty you know is
is in this situation controlling what
what we're going to learn
okay
so uh right so we have this does the
right thing abstraction that you know we
said well it's leaky sometimes but at
least helps us reason about models in
other times uh so we can ask but what is
the does the right thing abstractions
say in this situation where you have a
model that is itself doing optimization
in the situation where you have a mace
Optimizer well so in that situation it
should say you know if the model is
doing the right thing which is you know
it's actually out of distribution going
to try to minimize the loss then it
should be whatever the thing that this
sort of Mace Optimizer is searching for
whatever you know optimization it's
doing that optimization should be
directed towards the goal of minimizing
the loss
okay so this is one abstraction that you
could use to try to think about this
process but of course as we stated this
is not the actual process that we are
doing uh you know so so we don't know
whether this abstraction actually holds
and so what we want to know is what
happens when this abstraction is leaky
okay so when it's leaky we're sort of
going to have two alignment problems
we're going to have an outer alignment
problem which is like uh uh make it so
that the you know base objective
whatever the loss function is you know
that's the sort of thing that we would
want and then an inner alignment problem
which is make it so that the main
subjective the thing that the thing is
actually pursuing uh is in fact the
thing that we wanted it to pursue
one uh caveat that I will say here so I
think that this sort of this sort of
traditional setup really mostly applies
in situations where the sort of goal
that you have for how you're going to
align the system is we're going to first
specify some loss function and then
we're going to make the does the right
thing abstraction hold we're going to
specify some loss function that we'd
like and then we're going to get the
model to optimize it I will say that's
not the only way that you could produce
an align system right so an alternative
procedure would be specify some loss
function that you know is bad if the
thing actually optimizes for that loss
function you know that it would end up
doing uh you know producing bad outcomes
and then also get it to not optimize for
that get it to do some totally different
thing that nevertheless results in good
behavior so and in fact I think that's
actually a pretty viable strategy and
one that we may want to pursue but for
the purposes of this talk I'm going to
be assuming that's not the strategy that
we're going with that the strategy we're
going with is we're going to write down
something uh you know that specifies you
know some loss or reward uh that like
actually reflects what we care about and
then we're going to try to get them all
all to sort of actually care about that
thing too but I just want to preface
with that's only one strategy and not
even necessarily the best one but it is
one that I think is at least easy to
conceptualize and that we can sort of uh
you know understand what's going on if
if that's the strategy we're pursuing
question
[Music]
a thing that we don't want to do with
the training to not do that yeah so
here's a really simple example here's a
really simple example this is not
something I think we should literally do
but here's a simple example that might
like illustrate the point so let's say
what we do is we have an environment and
you know we're going to do RL in that
environment and we're going to set up
the environment to have incentives and
structures that are similar to the human
ancestral environment and then we're
like well you know if it's kind of
similar to the human ancestral
environment maybe it'll produce agents
that operate similarly to humans and so
they'll be aligned right that's a hope
that you could have and if that's your
strategy then the thing that you're
doing is you are saying I'm going to
specify some incentives that I know are
not aligned right like reproduction you
know natural selection that's not what I
want that's not what I care about but I
think that those incentives will
nevertheless produce agents which
operate in a way that is the the thing
that I want so so you could have that as
your strategy we're not talking about
that sort of a strategy though right now
in this talk but but you but I don't
want to like rule it out I don't want to
say that like you could never do that in
fact I think a lot of my favorite
strategies for alignment do look like
that though they don't look like the one
I just described but
okay but they do have the general
structure of like we don't necessarily
write down a loss function that
literally captures what we want but uh
supposing we do we're going to try to
explore you know what this what this
looks like and in fact you know as we'll
see basically all of this is going to
apply to that situation too because what
we're the problem that we're going to
end up talking about is just the central
problem of you know what happens when
you have some loss and then you want to
understand what thing will your uh model
on you know if it is itself an Optimizer
end up optimizing for
okay is that clear does that make sense
okay
great
okay so you've probably heard stories of
outer alignment failures uh you know
what they might look like you know
classical examples are things like
paperclip maximizers you've got you know
you're you're like I want to make the
most paper clips in my paper clip
Factory and then uh you know it it just
destroys the world because that's the
best way to make paper clips
um that is a classic example of an outer
alignment failure what I would like to
provide is sort of more classic examples
of inner alignment failures so we can
sort of start to get a sense of what it
might look like if uh inter alignment
you know in the sense that we talked
about you know previously where you know
the the model ends up optimizing for a
thing that is different than the sort of
thing that you wanted to uh that you
specified might look like so so what
does that look like well so when inner
alignment fails it looks like a
situation that we're going to call
capability generalization without
objective generalization so what does
that mean so let's say I have a training
environment and it looks like this brown
maze on the left and I've got this uh
you know some arrows that like Mark
what's going on uh and you know the loss
function you know the reward function
that I use I'm gonna we're gonna do like
RL this environment I'm going to say you
know reward the agent for finishing the
maze I want it to get to the end
okay and then I'm going to deploy to
this other environment where now I've
got a bigger maze it's blue instead of
brown and I have this green arrow at
this like random position
and now I want to know what happens
okay so you know here are some things
that could happen in this new
environment when we ask you know what is
the generalization behavior of this
agent so here's one thing that could
happen it could look at this you know
massive blue Maze and just like you know
not have any idea how to do anything and
so just like you know randomly you know
goes off and does nothing you know
useful that would be a situation where
you know we're going to say it's
capabilities did not generalize you know
it did not learn a general purpose
enough algorithm for solving mazes that
the algorithm was able to generalize
this new environment
okay but but it might so let's say
capabilities did generalize it actually
did learn some sort of general purpose
media solving algorithm and knows how to
operate in this environment well there's
sort of two things that it could you
know potentially learn at least two uh
it could learn to use those maze solving
capabilities to get to the end of the
maze which is the thing that we were
rewarding it for previously or it could
use the maze solving capabilities to get
to the Green Arrow
uh and that you know if the strategy was
pursuing if the generalization it
learned was always go to the Green Arrow
versus go always go to the end of the
maze you know in training those were
indistinguishable uh but you know now in
this deployment environment they're you
know quite distinct
okay and so uh let's say you know what
we really wanted right you know we
really wanted it to go to the end but
you know this green arrow just got moved
and so we're like okay so what happens
here you know why is this bad that we
could end up in a situation where the
model generalizes in such a way that it
goes to the Green Arrow instead well
what's happened is we are now in a
situation where your model has learned a
very powerful general purpose capability
which is the capability to solve mazes
and that capability is misdirected uh
you have deployed a model that is in a
situation where it is it is very
powerful and is actively using those
capabilities not to do the thing that
you you know wanted it to do uh so
that's that's bad that's dangerous right
you know we do not want to be in
situations where we are deploying
powerful capable models in situations
where they are using their capabilities
uh where you know they don't actually
generalize and that you know they don't
actually match up with what we wanted
them to use those capabilities for
okay so so that's concerning and that's
the thing that we want to avoid that's
sort of what it looks like when we have
this sort of inner alignment failure
okay so some more words we're going to
say that the situation where the model
uh generalizes correctly on the sort of
uh uh you know reward you know loss uh
objective that we want uh you know that
actually is trying to finish the maze
even out of distribution we're going to
say it's robustly aligned it's a
situation where you know it's alignment
you know it's it's you know actually
going to pursue the correct thing is
robust to you know uh out of
distributional shifts whatever and we're
going to say that if the model sort of
looks aligned over the training data but
actually you know we can find situations
where the model would not be aligned
then we're going to say it's
pseudo-aligned
okay
um great all right yes robust
test data the test data we're going to
say well uh any this any data that it
might in fact encounter in the real
world if if the thing has you know is is
actually robust in all situations it you
know it always pursues the right thing
then we're going to say it's robustly
line importantly we're only talking
about the alignment here what we don't
care about is if the model like in some
situation its capabilities don't
generalize right if it ends up in a
situation where like it's just like I
don't know what's going on and then it
doesn't do anything we're fine with that
right that's still robust alignment
potentially uh right we're sort of
factoring the capabilities in the
alignment here we're saying that if it
is capable of pursuing some complex
thing it is always pursuing the thing
that we wanted it to pursue and here
we're saying it only looks like it's
pursuing the right thing on training but
you know out of uh distribution it might
end up pursuing the wrong thing capably
just to clarify robust alignment is
defined with respect to a particular
data distribution you can't speak about
it in isolation
uh no uh pseudo alignment is is defined
specifically relative to a data
distribution because it's defined
relative to the training bad
distribution but robust alignment is not
robust alignment is defined relative to
any data it might ever encounter okay
okay great
all right so we have two problems yes
don't uh generalize are you doomed to
have a pseudo alignment because in cases
where it doesn't know what to do it will
do things that are unintended well uh
you know I'm okay if it does things that
are unintended we're worried if the like
optimization is directed in an
unintended place right that it does
powerful optimization in the wrong you
know towards the wrong goal so like you
know if it's just like you know in the
new maze it just like Falls over and
does like random you know stuff we don't
care right we're like okay whatever the
problem is a situation where it does
know how to solve complex bases you know
it actually does have the capability uh
you know and then it uses like actively
optimizes for something we don't want it
to be optimizing for
okay so like even if we have the May
solva and I was right I'll see a lion
we would we would call it robustly
aligned even if we like gave it a
massive Maze and then it tried to like
take over a piece of computational power
to solve this maze because it was still
actually applying right so in this
situation like you know just going back
like to here we're assuming that the
loss function is actually reflects what
we care about right and we're like What
would it look like if the loss function
does reflect what we care about to have
an inner alignment failure so in that
situation you would be like well you
know your your strategy of like write
down a loss function that reflects
everything I care about and then get a
bottle's optimized for it you failed at
the point where the loss function you
wrote down that you thought reflected
everything you care about just said
optimize for mazes but that was bad you
shouldn't have done that that was
actually not everything you cared about
and so we're like that was where you
went wrong here
um I think that like I said though you
know you might not have this strategy
but like you know right now at least
we're assuming that it's your strategy
and we're gonna say you know actually
when you wrote down this last time you
were at least trying to get it to
capture you know everything that you
wanted
okay okay yes do you draw any
distinction between a model like
specifically optimizing for like some
internal golf versus like just
statistically preferring something
yes yes we we absolutely are drawing a
distinction there we are saying that the
thing is an Optimizer if it is like in
fact has some search process inside of
it that is you know you know searching
for things that do well according to
some objective uh you know this is the
difference right between the bottle cap
you know which like you know in fact
systematically prefers the water to be
in a bottle but like isn't in fact doing
optimization for it and the reason that
we're making this distinction is because
we're going to say well we're actually a
lot more concerned about situations
where the model is actively doing
optimization that is misdirected then
situations where it has you know a bunch
of you know heuristics that were
selected in a problematic way and the
reason we're more concerned about that
is because misdirected optimization has
the ability to be a lot worse out of
distribution you know because it can you
know competently optimize for things
that we don't want to optimize for even
in situations where you know we didn't
select it to be able to do that
sorry
if a model actually be an Optimizer that
is a great question and that is what
we're going to be talking about next
okay so yeah we're gonna be talking
about two problems here you know one is
unintended optimization which is like
you know maybe you didn't want to be an
Optimizer maybe you just wanted it to
you know be a bottle cap like you wanted
to be a selection of heuristics you
didn't want to be doing any optimization
why might you get optimization anyway
and then the second question you know is
this inner alignment question which is
okay if it is an Optimizer how do you
get it to optimize for what you want
okay let's talk about it okay what sort
of machine learning systems are more or
less likely to find optimizers that's
the question we're starting with
okay so here's the first thing is look
search algorithms are really good at
generalizing so let's say we're in a
situation where you know we want our you
know algorithm to play go you know we
want to do a really good job at go
playing well there is a systematic uh
similarity uh between all good go moves
right the systematic similarity between
all good go moves is that they are moves
which have the property that if I do
some look ahead and you know try to see
how good is this move
they end up being really good right and
so if I want to encode an algorithm
which systematically produces good go
moves uh encoding search is a really
simple way to do that and it's a way
that generalizes in a lot of situations
it's a way that you know if I go into
some new situation where you know I
don't necessarily have a bunch of
heuristics that are well optimized for
that situation I nevertheless have the
ability to you know figure out what's
going to do well you know the uh you
know the the you know the go engine
which has a bunch of ability to do
search you know can can find it you know
absolutely crazy billboard that it's
never seen before and still do you know
10 steps of look ahead and come out with
you know some ideas of what things are
going to be good in that situation
um and you know obviously in fact you
know we explicitly program search in and
we make you know uh go uh Bots you know
alphago actually has this MCTS process
uh okay so so you know there are
situations where you know uh if you have
this really you know a really diverse
environment you have a lot of situations
you need to be able to handle being able
to in each situation be able to search
for what the correct thing to do in that
situation is gives the ability to
generalize better
okay uh and then similarly also it's a
form of compression so you know I said
that like well what is the regularity
right between all good go moves well the
regularity between them is when I do
look ahead they end up looking good
um and so if you were sort of asking
well what is the easiest way to compress
the set of all you know good go moves
well the way to compress the set of all
good go moves is an Optimizer that
searches and does look ahead to find
good go moves right
um and so if we expect that your model
is going to be biased towards
um uh Simplicity which I think we should
uh if you expect that then you should
expect that it's going to find these
sorts of compressions ways of taking
your data and compressing it into a
simple form that you know uh you can
still be able to recapitulate it and in
the situation where the thing that
you're doing is you know some situation
where you're trying to train it to you
know uh solve some complex task uh a
good way to compress a lot of complex
tasks is to be able to do search why is
there Simplicity bias there's a bunch of
stuff on this uh I think it's a little
bit tricky
um
maybe a really sort of simple version of
this I don't this is not something I
have up here but maybe probably the best
result that I think argues for
Simplicity is probably the mingard
result which shows that if you do
um sampling with uh replacement from the
Galaxy initialization prior uh so this
is a little bit of a weird thing to do
because it's extremely you know
uncompetitive it would take you know
years and years if you wanted to
literally just take you know initialize
a neural network over and over again
until you've got a neural network which
actually had good performance but you
could suppose you did this thing if
you're just like you know you keep
initializing it every single time you
know you never train it and you hope
that eventually the initialization
actually just does well on your data if
you did this in fact the prior ends up
being extremely similar to the prior you
get from granny descent
um which sort of is showing that
actually you know what greatness that is
doing it is sort of doing something like
finding the sort of minimal Norm in
gaussian Norm space uh for you know
solution that is able to solve the
problem that sort of maps onto
Simplicity in various ways there's a lot
more to say here
yes you said prior twice the prior when
drawing this is the same as the
posterior you get after after gradient
descent okay I mean it is also posterior
in this case because it is a posterior
is like the prior is gaussian
initialization and the posterior is take
the prior update on in fact when I
initialize and sample from the
initialization I get a model which has
good performance on my data and then I'm
like that posterior is extremely similar
to the posterior of take a model
actually train it and then see what I
get okay okay there's a lot more to say
here but I'm not going to say too much
more you can ask me more later if you
want to know more about inductive biases
okay uh but but whatever the point is
there's a lot of evidence I think that
is biased on Simplicity okay but this is
not the not the only situation where we
might want where we sort of might expect
authorization another situation is and
this is pretty common there's a lot of
situations where we like to train you
know machine learning models to mimic
humans uh okay what is a property of
humans that we you know established
previously well humans are optimizers
you know there's lots of situations
where if you're going to do a good job
at mimicking humans you better be able
to do a good job at uh you know
predicting what we're going to do when
you know we have a goal and we're trying
to optimize that goal you know being
able to understand you know if a human
was trying to you know do X how would
they accomplish x uh you know if you
know being able to sort of work
backwards and be like okay you know how
what would what would be necessary for
the human to be able to accomplish their
goal and you know how would they achieve
that is is you know something that's
pretty important if you want to be able
to predict human behavior
and so if we're predicting humans we
should also expect that our you know
models you know are at least going to
develop the capability to do
optimization
um okay
and okay so so some some things here
that are going to push you in various
different directions do you have a
question
okay uh so some things that are going to
push you towards mace authorization here
well so uh you know large model capacity
you know just having the ability to
implement complex optimization
algorithms you know Simplicity bias
either explicitly because you you have
some regularization implicitly just
because you know gradient descent is
regularized in this way uh as I was
talking about previously
um you know other things I didn't talk
about this but statefulness having the
ability to store state is the thing that
really improves your ability to do
optimization over time uh because you
know you can sort of store the results
of optimization and then you know
continue iterating uh so like iterative
stateful procedures are a lot easier to
you know produce optimization in things
that might disintensify this
optimization if you just give the thing
optimization so like I think if you have
like MCCS that's like part of your
architecture you're like a little bit
less likely to just like learn
optimization in the non-mcds parts
because it's like already doing some
optimization so it has less need to sort
of implement it
um but it's not 100 True uh you know
especially because in fact when we do
like MCTS we usually distill the MCCS
into the policy Network and so you're
like actively training it to mimic a
search process
um but you know at least the actual like
you know thing of giving it access to
authorization seems like it should you
know reduce the probability that
develops it itself though it still could
maybe just because the optimization that
it needs is different than the
optimization you gave it
um time complexity bias I didn't mention
this but you know you could be in a
situation where where if you really
incentivize the thing to perform its
computation very quickly it's a lot
harder to do optimization optimization
while it's a very simple algorithm is
one that can take you know many steps of
look ahead to actually you know produce
some results and so you know uh if we
bias the thing on any amount of time it
takes and it's speed that sort of
disincentivized optimization
and also if we just give it really
simple tasks so you know I think you
know example of this might be we just
give it a task that is like literally
just you know pre-training on webtext
you know we just give it a task that is
just predict the next token you know
really simple tasks that uh you know
aren't very complex that don't have like
a bunch of moving pieces in terms of
like you know what the we're trying to
train the thing to do then it you know
uh you know potentially needs less uh uh
optimization and probably still like at
least if you're trying to predict humans
like I said it probably still needs to
have the capability to do optimization
but it doesn't necessarily need to
direct that optimization towards some
objective that it's optimizing for
um okay so these are sort of some things
I think might push you in one direction
or the other
um I think it's a little bit tricky
uh there's like you know a lot of things
pushing you in in you know different
directions here
um
I think my guess is that uh you know at
some point for just four capabilities
reasons people are going to want to
build agents and their groups want to
build agents which have the ability to
competently take actions and do things
in the world and I think when they do
that uh these sorts of you know
procedures that we are building for
doing these sorts of things are moving
in the direction of the things that are
incentivizing optimization
so why is that well you know we're
moving towards doing more imitation of
humans uh we're moving towards you know
bigger models moving towards uh more
Simplicity bias this is actually a
little bit tricky I didn't mention this
previously but larger models have a
stronger Simplicity bias than smaller
models
um that might be a little bit
counter-intuitive but it is true uh one
way to see this is the double descent
curve maybe I'll just go back really
quickly
um where if I have uh the like size of
my model
and they like train the model over time
you're probably familiar with like the
standard sort of uh you know thing that
you get out of learning theory which is
like your train loss goes down and then
your test loss sort of you know first
you underfit and then you go up and then
you overfit but in fact if you like
actually increase the size of the model
far past the point at which uh you know
you normally would in sort of standard
learning theory your test loss actually
goes down again and it goes lower than
you can get any point in the classical
regime
um what's happening here is that um the
sort of only way that this works is if
the like implicit biases that you get by
making getting a larger model size are
actually selecting amongst the larger
space of models for you know models that
actually generalize better and the
models to generalize better are you know
the simpler models and so as we increase
the size of the model we're actually
increasing the amount of Simplicity bias
that we're applying to get you know an
even simpler and simpler and simpler
solution which is able to fit the
problem
um
okay uh yes is that the same thing as
saying the larger models have the
freedom in their weights to explore
these simpler models and then end up
finding them whereas in a smaller model
they're too compressed to be able to to
explore that yes so it's a little bit
counter-intuitive but in fact the larger
models have the ability to implement
simpler models right so if you think
about it like this
um you know there are algorithms which
are very simple right you know I can
write you know search and you know two
lines of python or whatever right but
you know in fact you know influencing in
a small Network might be really might be
really difficult
um and so as the network gets larger
there's still a bias towards like simple
algorithms but it gets the ability to
implement more of the possible simple
algorithms that are available
uh and but then it still it selects a
simple one
okay there's a lot more to be said here
yes so I take it there's a is there a
version of this plot where like instead
of just complexity of age it somehow
incorporates the whatever that bias is
you talked about and then this is this
is this is traditionally model size so
just like number of parameters
but you can actually do this with a
bunch of you could you see double
descent graphs with with all sorts of
things on the x-axis but but
traditionally at least it's with model
size but I guess so I'm trying to wrap
my head around what's like unintuitive
about the right side and it's like or
like where why it shouldn't actually be
unintuitive I guess and it seems like
the x-axis is just the complexity of our
model class but it's so it's not
accounting for how our training
procedure interacts with that model
class is that right no no this is after
training
so this is literally just sorry just
cover this up and replace with number of
parameters
and then this is after training we're
keeping like uh training uh constant we
can we we train to convergence at every
point on this graph
right but there there's some the point
is
the x-axis on I'm maybe not making a
very interesting point but I'm just
trying to understand better so the the
x-axis here only measures the complexity
of like our function
class it does not
account for how that interacts with our
training procedure it doesn't yeah it
doesn't account for the biases right so
we have some it it's this is just the
size of the model class right but then
on top of that model class we actually
have some some prior that selects out
the model that we actually like out of
that model class right and we're saying
that prior the sort of Prior that of
like gradient of machine learning that
like selects out you know uh the a like
simple you know it actually selects out
the simple simple algorithms out of that
large model class so as we increase the
model class we get simpler and simpler
algorithms out of it and so if we made a
new plot where the x-axis encountered
encapsulated both the complexity of the
class and how our training procedure
interacts with it
would not get
um yes so in fact one thing that I will
say here is that sort of what's going on
is that there's a disconnect between the
order in which new models get added to
the model class and the criteria that
grading descent uses to select from
amongst those models right it is not the
case that you know what this tells us is
this is not the case that new models get
added to the model class of like what
algorithms are available to be learned
in the same order in which grading
descent would prefer them like it's not
it doesn't just start with the most
preferred algorithm then the second most
preferred algorithm it's just like
randomly chucking them in there and then
gradescent is like doing some you know
very you know special thing to actually
pick out the best one from amongst them
okay if it were the case that the sort
of order in which algorithms were added
to the support of your prior were
actually like in Simplicity order then I
think you would not see this
continue on this line go for it
oh uh are you basically arguing in favor
that the lottery ticket hypothesis here
that no or no not here no in fact I
think lottery tickets hypothesis people
get very confused about I think it
doesn't say people think it says I think
it's not super relevant here we could we
could we could talk about it later maybe
yes
um
noted that uh in a neural network
encoding search requires like a like a
large enough model you can't do it in a
two-layer Network it's impossible
um
seems pretty hard to do in a two-layer
Network yeah so even though we regard
search to be simple shouldn't we sort of
be conditioning our Simplicity based on
the architectural model well the sort of
Simplicity that I care about is the
Simplicity of like if I have a really
really large model and the Green Design
can select a huge amount of possible
algorithms from amongst that set what
sort of ones does it select for right
I'm not as much interested in the sort
of Simplicity that is like if I have a
tiny model what are the algorithms that
I can Implement right and so when I say
Simplicity the the place where I think
Simplicity bias occurs in machine
learning is not the like oh if I have a
really small model I just can't
Implement some things the place where I
think the Simplicity bias occurs is if I
have a really large model and I can
Implement a ton of things which ones
does green Nissan actually end up
selecting that's what I think is
Simplicity bias not the other one
couldn't like sorry and in fact I think
that the like architectural bias you get
from having a small model is more like
speed bias it actually probably
disincentivizes optimization like we're
saying because it just doesn't have
enough serial uh computation time to do
up just do it
wouldn't it be though that uh given that
it's just quite impossible to do it in a
small model
um there's less of a Simplicity biased
and more just
once you make once you reach the like
required size then that's an available
method to implement and then at that
point it just berries a tiny bit based
on Simplicity and generalization
I think that the claim I'm making is
that actually you know some models are
really vastly simpler but they only
become accessible to be implemented by
grain descent once you have a larger
model we we can talk about this this
later maybe
okay let's keep going okay uh great okay
so that's that's you know part one which
is like okay here are some things that
would push you in directions for or
against learning optimization I think my
take is where you know it's going to
happen you know we're going to at least
be able to have models which are capable
of doing optimization uh you know it
seems like all of the ways in which we
want to build powerful models and you
know have models that you can generalize
in you know new situations they're gonna
they're gonna be able to do uh
optimization okay so if that's gonna
happen
uh will it be doing the right thing
okay so first
what does it look like for a model to
have a an objective uh you know for a
mace authorizer to have a mace objective
that is uh misaligned with uh you know
not the same as the loss function right
doesn't obey this does the right thing
abstract it okay so fundamentally you
know I think the most basic case the one
that we're primarily going to be talking
about is a proxy pseudo alignment which
is a situation where uh you know there
is some uh correlation uh there's some
correlational structure between a proxy
that the model is trying to optimize for
and the you know actual objective that
we wanted to optimize for
uh so you know in a sort of causal graph
language you can think about them as you
know having some common ancestor you
know if I am trying to optimize for this
and there's some common ancestor between
you know the base objective and me well
then I need to make this x large because
that is how I make my thing large and
that will bite uh you know as a side
effect make this thing large as well and
so if we're in a situation like this
then we're going to be in a situation
where uh you know it could be the case
that anything which has this
correlational structure uh you know to
the thing we care about could be the
thing that your model learns so you know
for example we had that situation where
you know it learned to go to the Green
Arrow rather than the end of the maze
because in training those two things
were highly correlated
okay
um there are other things as well I'll
just like mention really briefly you
could also have something like
sub-optimality suit alignment where you
know the reason the thing looks aligned
is because it's actually doing
optimization in some really bad way and
so you know in fact it's objective it's
just like some crazy thing uh you know
an example I like of this is like you're
trying to make a cleaning robot and the
cleaning robot you know believes uh it
has an objective of like minimizing the
amount of atoms in existence and it
mistakenly believes that when it like
vacuums up the dust the dust is just
like annihilated and so it's like oh
this is this is great but then you know
later it discovers that in fact you know
the dust does not get annihilated it
just like goes into the vacuum and you
know it stops being aligned so that
would be a situation where like it
actually had some you know insane
objective but that objective like in
fact exhibited a line behavior and
training because it also had some insane
beliefs so you can also get some weird
cancellations and stuff like this but
we're mostly not going to be talking
about this mostly focusing on the proxy
case was just like well you know it kind
of understands what's going on and it
just cares about something in the
environment which is correlated it but
not the same as the thing we want
okay so that's sort of what pseudo
alignment looks like why would you get
it
all right so the most fundamental thing
is just unidentifiability look you know
any complex environment it just has a
shitload of proxies there's just like a
ton of things in that environment which
one could pay attention to and which
would be correlated with the thing you
care about and so as you sort of you
know you train agents in very complex
environments you're just gonna get a
bunch of sort of possible things that it
could learn
uh and so by default you know if it just
learns anything which is correlated you
know uh it'd be really unlikely and
weird if it learned exactly the thing
that you wanted
uh because you know all of these things
you know to the extent that they're
correlated with the thing you're trying
to train it on we'll still have good
training performance
um okay uh so you know just operate you
know probably some of these proxies are
likely to be simpler and easier to find
than the you know one that you actually
want
unless you have some really good reason
to believe that the one you actually
want is like you know very simple uh and
easy to find
okay in addition uh you know it I think
the situation is sort of worse than that
because actually some proxies can be a
lot faster than others so what do I mean
by that so so we have like an example
here so we're like look you know a
classic you know example that people
like to talk about you know in terms of
this sort of thing is like Evolution you
know Evolution you know also was sort of
like an optimization process selecting
over humans and selecting the humans uh
to do a good job on you know uh you know
natural section right you know pass on
your DNA into the Next Generation okay
uh so let's suppose in an alternate
reality that you know uh natural
selection you know Evolution actually
produced humans that really just care
about like you know how much their
alleles you know are represented in you
know future Generations that's like you
know the only thing that the humans care
about okay but here's the problem with
this is that this is just like a
terrible optimization task right you
have a baby right and the baby like
stubs their toe and they're like you
know what do I do about this right
like how is this going to influence my
like chances of like in the future of
being able to like mate and then like
pass my DNA down and like hmm like what
is this but like no right but like
that's terrible right like instead a
much simpler and faster to compute
easier strategy for the baby to
implement is just like you know have a
pain mechanism that is like okay we've
developed this you know heuristic that
like you know all pain is always going
to be bad and so just like you know
shunt that to you know the negative
reward system and so we're like okay so
you know we actually if we're in a
situation where we're training on
something which is relatively complex
and difficult to specify in terms of the
agency you know input something like DNA
you know we're trying to train on pass
your DNA onto the Next Generation uh you
know it's not going to learn to care
about passing the DNA on the Next
Generation it's going to learn to care
about you know reducing pain uh because
you know reducing pain is something that
is much more easily accessible to the
the baby and it's something which the
baby can actually Implement effectively
and doesn't rely on doing some really
complex difficult optimization process
to be able to determine how to optimize
for that thing
okay so you know we shouldn't expect to
get you know some really complex
difficult to optimize for a thing we
should we should expect to get you know
these sorts of faster easier to optimize
for proxies
okay and also some proxies are simpler
right so I mentioned this right you know
pain also has this property that you
know it's extremely accessible in terms
of the input data of the baby you know
the baby can just detect oh man you know
something has happened to my leg that's
bad uh and it doesn't have to do some
complex thing that's like okay I can
infer that probably you know there is
DNA inside of my cells but I'm not like
you know 100 sure I should probably go
get like a microscope and then I can
like figure it out you know it doesn't
have to do that it just like you know is
extremely accessible you know the thing
that it cares about in terms of its
input data and the data that's available
to it and so we should expect that again
you know the sort of properties you're
going to be learning are the sorts of
proxies that are accessible and simple
in terms of the data that the model has
access to Yes um
yeah because
when you're consider in the case of like
a an artificial neural network
the amount of efforts required is just
one forward pass through a simple proxy
or a complicated thought process that
leads to an output so
um yeah so the way I think you should
think about this is that well you know
technically the amount of computation it
performs in any forward pass is fixed
but it has to ration that computation
right it's not the case that it gets
infinite computation in fact it gets
like you said a finite amount of
computation and so it needs to ration
that computation to do the things that
are most useful for it right and so if
it's wasting a bunch of the computation
doing all of this reasoning about how to
you know get the most DNA you know it's
not going to have room to you know for
example just do the look ahead five
steps more right it's not going to have
the you know capacity left to do other
things which will help it with
performance more and so you know because
it is computation limited we should
expect that that computation will be
rationed in ways that result in you know
simple fast proxies
okay great so all right so it seems like
we're probably going to get proxies that
are simpler and faster than you know uh
than you know other things that we might
want
um okay so so one of the things again
that you know push you in different
directions here right so you know we
have uh time complexity bias you know if
you if you try if you're trying to you
know limit the amount of computation
you're gonna be biased towards you know
fast proxies if you try to you know have
a Simplicity bias uh you're going to be
biased towards having uh you know uh
simple proxies right that are easy to
specify uh if you have a really complex
environment that just has a lot of
proxies and a lot of possible things the
thing could learn uh yeah then you're
you know you're going to get you know
more likely that you're going to learn
the wrong thing and if the thing you're
trying to get it to learn is really
complex uh you know if the objective
that you're trying to optimize you that
you wanted to optimize for is really
complex then it's going to be harder for
it to find that one
uh things that might you know be help
here so you know there's things that we
can do you know we can try to do stuff
like anniversarial training you know we
sort of find situations where the you
know models proxy might come apart from
the thing we actually want to do
situations where you know maybe you know
we actually train on the big blue maze
that has the Green Arrow and the end of
the maze in different spots uh you know
we can try to do transparency you know
look inside the model see you know what
proxies it's learned we could try to
give it optimization explicitly and
maybe control that optimization more
um so there are things that we can do
um
okay so what I want to talk about now at
this structure is adversarial training
specifically
and I want to sort of imagine you know I
think that in some sense adversarial
training is sort of like the default way
to solve this problem or you're like
well you know okay look you know this
May's example Evan that you gave me at
the beginning well something was
obviously wrong here which was like you
train on the big maze right you know we
had access to it we could have trained
on it you know why why don't we just do
that
um I think that's like a good response
and I think the question that I want to
ask next is what happens if we do this
because in some sense you know we will
probably you know it's it's not that
hard you know we can just generate a
bunch of adversarial examples what
happens if we generate a ton of
adversarial examples and we try to use
those to get to a situation where
actually we force it to learn to care
about the correct thing that we really
wanted to care about what happens okay
and unfortunately I think the situation
is not great actually so so what happens
so okay now we're going to talk about
deceptive alignment
okay so first I want to take a
digression uh this is this is not
directly related to the thing I just
said I just want to establish it as a
background fact uh so let's suppose that
you're in a situation you know you're
this robot and you're in the situation
where there's this training phase and
then there's this deployment phase and
the deployment is a little bit longer
you know there's more things happening
in deployment
um and uh you know but you uh you have
some proxy that you care about uh you
know you care about the proxy B
uh and you know though that the training
process you know the loss function the
thing it's trying to get you to do is to
care about a you've discovered this fact
about the training process let's suppose
and you're this robot what is the
optimal strategy for the robot to get
the most proxy across time uh that it
that it can possibly get you know we're
going to suppose just in this in this
hypothetical that it wants to get as
much b as it can over all you know uh
circumstances that it's in and we're
like okay what is the strategy that the
robot could Implement they would allow
it to get the most proxy uh as possible
uh well the strategy in this situation
you know so here's a naive strategy
let's say it just goes to B you know it
likes B that's what it wants that's the
proxy that it cares about so it just
goes to B whenever it sees a b well the
problem with this is that it goes to B
and then the training process is like
hey that's not that was bad I now have
an adversarial example like I have an
actual data point where you did the
wrong thing and I can back propagate
through the you know example and like
you know now change you to not do that
anymore because that was bad and you got
low reward for doing it and so now I've
produced a new model that uh doesn't go
to B anymore and it goes a and so you
know the total amount of B that you got
by doing this was one B you know you got
the go to B once and then the training
process modified you to not go to B
anymore
okay so here's an alternative strategy
you go to a in training and you go to B
in deployment what happens if you do
this well you go to a in training and
the training process you know it looks
at you know we're just like you know
Computing a you know simple you know all
over word we're like well you've got
really high reward and so whatever
Behavior just cause you to get high
reward we're going to incentivize that
and and you know uh you know in you know
increase you know that behavior you know
when we back prop through you know we're
gonna whatever Behavior causes you to go
hey you know where it's going to get
reinforced and you know but in this
situation the behavior that causes you
to go to a was the behavior of actually
I'm going to plot to only go to a in
training and B in deployment and so
incentivize the strategy of go to a in
training and go to B in deployment how
many B's does it get well it gets three
b's because we're assuming deployment is
you know three and training is two in
this situation and so you know that's a
lot more than one B uh and so you know
now we have a situation where
uh this you know this robot
theoretically the best strategy for it
is uh you know if it has some proxy and
that proxy is different than the thing
that we wanted to do the best possible
strategy available is to pretend to do
the thing that we want in training and
then later in deployment defect against
us and do some other thing once it has
the ability to sort of evade the
deployment signal
um you can also you know we don't
necessarily have to have an extremely
sharp distinguish between training and
deployment here I think you should sort
of just conceptualize this as well at
some point there becomes a point where
whatever training processes we had that
were able to oversee the agent and sort
of correct it if it did the wrong thing
at some point probably you know we're
going to have to deploy agents in
situations where a single action you
know on half of that agent could
potentially be catastrophic we actually
have to trust our agents we have to
actually have to trust our models and we
can't just rely on some oversight
process if you believe that we're
eventually actually going to have to do
that then we won't just always be able
to have some corrective procedure that
can sort of post you know Hawk come back
and sort of change the model and make it
do the right thing then you know there
will eventually be some situation that
we can call deployment where the
deployment situation in a situation
where eventually you know it becomes the
case that if the model decides to go to
B we can't stop it anymore
maybe maybe maybe we will talk about
that in just a second I'm not right now
talking at all about the likelihood of
this happening I'm just trying to
explain what it would look like if it
did and why it is in some sense
theoretically optimal for at least some
sorts of Agents yes
it does not require episodic memory it
does require some things it does require
that it have a sort of cross episodic
goal so it has to care about B not just
like I want to go to B Every time I see
it has to care about going to be as many
times as possible
so it does require that
um one thing it's at least so one thing
this picture requires specifically is it
requires you to know when it's in
training or deployment to actually have
some mechanism being able to determine
oh I'm now in a situation where I can
like you know I no longer have an
oversight process on top of me
um
as I will talk about later though I
think that's actually not necessary I
think you don't need that but at least
for this picture you do
but I actually think it's not necessary
for you to get deception but the cross
episodic goal absolutely is necessary uh
we will talk about that
you need some idea of how much training
versus deployment for that cross episode
um
yeah you you if you you so you do need
to understand that there is training and
there is deployment you have to
understand that those things exist and
that this they have this sort of
structure
um you don't necessarily have to
understand exactly what like the ratio
is you probably have to understand at
least either the ability to detect when
you're in deployment or have some
understanding of the ratio as we'll talk
about later
okay so this is a thing that could
happen yes sorry
sorry yes
oh that's a good question so this is all
from the perspective when I say the main
subjective state I'm this is all from
the perspective of the original robot
because we're sort of imagining you know
if I'm the robot at the very beginning
of this how and I have some cross
episodic goal I care about going to be
as many times as possible how would I
plan you know for doing things and it's
like yes technically in some sense the
robot here is just like a different
robot because now they care about a
different thing uh and so they're you
know new mace objective would be to go
to a but we're like you know suppose
just right now from the perspective of
the original robot you know what does
the original robot want to do and the
original robot wants to go to B as many
times and so if they get changed into a
robot that goes to a you know they're
very unhappy
okay all right so why would this happen
let's talk about it okay
um
okay so first so there's two so so uh at
some point you know you do enough
adversarial training at some point your
model is going to have to learn to
understand the thing that you are trying
to get it to do you're trying to get to
do something and at some point if you've
put it in enough situations where you
know it has to be able to figure out
what's happening in all those situations
at some point is going to have to learn
what it is that you are trying to get it
to do uh but I think there's
fundamentally two routes view which
information about the thing you are
trying to get to do could enter into the
model there's sort of two ways that that
information could could get in the model
one way is via the gradient updates you
know the gradient updates provide
information to the model into about the
objective because they you know can
enforce objectives that in fact result
in good behavior and you know you know
disincentivize objectives results in bad
behavior
and that can sort of push the model
towards having an objective that
corresponds to the correct thing but
there is another path view which
information about the thing that you're
trying to get to do can enter into the
model and that's through the model's
input so you know if for example the
model gets to see a bunch of you know he
gets to read all of Wikipedia right uh
you know there are a bunch of gradient
descent update steps that occur in you
know understanding Wikipedia and like
predicting it for example then encode
information about what is in the
Wikipedia and they're not necessarily
creating update steps that are directly
about the model's objective they're just
grading to send update steps that are
about encoding and understanding what
exists on Wikipedia Wikipedia actually
has a ton of information about the sorts
of things that we might want to get
models to do and so there's there's a
mechanism view which information about
uh the sort of training process and
about the thing that we are trying to
get the model to do can enter into the
model that doesn't go directly through a
like back propagation through did that
objective actually result in good
behavior so there's another mechanism
which is just like the model has to
learn a bunch of facts about the world
uh via which the model can learn facts
about what you're trying to get to do
what the training process is and how it
works
okay so there's these two Pathways so uh
the sort of internalization pathway you
know it's pretty straightforward if this
is the way in which it learns about the
base objective primarily it's you know
hopefully going to end up with a correct
objective you know we just reinforce
good objectives and you know uh we you
know disincentivize a bad objectives we
should end up you know grain dissenting
towards the good one but there are a
couple of ways in which
um you could instead use the modeling
information to learn about the uh base
objective instead yes
each other two are different because
when like all of Wikipedia is put in the
model let's put it put in it through
training right so it's still kind of
ingredients that's correct let's suppose
so maybe a really simple way to imagine
this is let's say we do pre-training and
then we do fine tuning in some RL
environments we pre-train the language
model and then we do fine tuning to try
to get that language model to do a good
job at you know optimizing for something
in some environment in that situation
there was a there were great Nissan
update steps where those update steps
were just encoding information and there
were greatness and update steps where
those update sets were encoding we're
incentivizing you know actions right
we're directly encoding like the sorts
of things we wanted to do and we're
saying well you know one it could end up
with most of the information about what
we wanted to do comes through like you
know the first time or come or it could
come through the second type
right and if I think it's going to look
different what you get depending on
where most of the information about the
objective is coming from
is it coming from this sort of just like
steps that are just going towards
modeling information or is it coming
from steps that are going towards you
know incentivizing or disincentivizing
particular objectives
yes it's the first one focused on like
in a world where you have
interpretability and you understand
exactly what it's doing and you say
don't do that do this other thing even
though it's achieving the same no I'm
just imagining that you incentivize
Behavior you like and you disintensifies
Behavior you don't like if you see it do
the wrong thing in your environment then
you know you get a grading a signal to
not do that and do something else
instead
but that's different than like if you
know giving information you know
greatness and update steps that are just
about you know giving it information
okay
okay so if the if the information is
mostly coming through you know part two
then there's sort of things that are
available to grading descent they can
sort of replace step one right so in
some sense you know let's say we do
Prime we do this pre-training and then
we do this fine tuning right uh you know
we could you know in the fine tuning
process just like spend a ton of
gradient descent update steps carefully
encoding this complex objective that we
want the thing to care about
or instead we could just make use of the
existing knowledge that the model
already had in some way that you know
also results in good performance so how
could you do that if you're gradient
descent and you have this model the
model knows a ton of facts about the
world it just has a ton of information
how do you use that information to
result in the model having good
performance in practice on my you know
training environment well here are two
ways way number one is deception you can
just sort of take uh though you know you
can give it some simple long-term proxy
it has to be long term as we talked
about but if grading descent takes the
model that knows a bunch of facts and in
particular if it takes them all to knows
a bunch of facts about the training
process and gives that model a long-term
proxy to optimize you know takes the
model and says okay now what you should
do is optimize over the long term for
you know number of gold coins or
whatever number of paper clips whatever
it just gives it some long-term proxy
then the model has enough information to
deduce that given that what I care about
is you know gold coins of the long term
I should optimize for you know whatever
their trading process cares about right
now so that I can eventually you know
try to get the gold coins later on
um and so as soon as green descent makes
the modification that takes the model
and turns it into a model that cares
about you know something over the long
term it gets to make use of all of this
information that entered into the model
about the training process immediately
right all of that stuff that the model
knew about like oh you know I know all
these facts about the training process I
know how it works I know what these
humans care about but uh and it gets to
take all of those facts and
instrumentally use them to result in as
good performance essentially as possible
given the models understanding the world
because it'll just start optimizing for
what you want to optimize for and so
there's a modification available to
Green descent which is a relatively
simple modification in this situation
where most of the information entered
through these sort of modeling which is
just um you know make it so that you
care about some long-term thing
okay that's option one
option two is this corrigible thing so
instead of being deceptive you can
instead you know you'd be like well I
have a bunch of information about uh you
know what the thing what the training
process is and how it works and what it
cares about I could instead grain
descent could uh take the model and give
it a proxy objective that is a pointer
to something in that world model you
know there's something in the world
model that corresponds to you know the
base objective it corresponds to like
what humans care about what we're trying
to train it for whatever what you could
do is gradient descent could just like
be like okay optimize for that thing in
the world model here
um the difficulty with this though is
that you have to actually get that
project right so there's like you know a
lot of difficulty potentially in getting
the pointer that actually specifies
exactly the correct thing whereas with
the deception you know you sort of don't
have to get it right like essentially
any objective if you care about it over
the long term will result in good
performance because the model will
instrumentally reason oh you know the
best way to get good performance is just
to play along
okay so these are two these are well I
guess you know we have three ways that
grading descent could take a model uh
and and produce a model that actually
has good performance on like all the
adversarial uh examples in you know in a
situation we're trying to get it to
accomplish some tasks
okay so let's talk a little bit about
like you know how How likely each of
these are so I already said some words
but modeling you know why is why do I
think modeling is simpler well you know
essentially right it's just like look
you know it's really you know the the
situation where I think basically in any
situation it's really surprising for
your model to actually spend a ton of
you know for grading descent spend a ton
of update steps just directly encoding
percent objective where that objective
doesn't make any reference to the like
information that the model has have you
if you have a model and it has a ton of
facts in there about the world and then
you you know try to get it to optimize
for some you know for some objective
it's going to make use of those facts
you shouldn't expect that those facts
you know uh you know are just going to
be you know put aside and it's going to
learn some totally separate objective
that isn't at all related to the facts
that it knows it's gonna it's gonna use
them both you know it's a simpler more
compressed way of doing things to make
use of the information that the model
has and so we shouldn't expect I think
in basically any situation that we end
up with an objective that just like
separately encoded from the information
it's going to be encoded in terms of the
information in some uh perspective
okay so how might it be encoded in terms
of the information well uh you know one
thing is uh you could have this cordial
thing right so the problem is I think
the deception you know actually beats
out the Corporal thing so why is that
well so with the quadrival thing you
know you're learning some pointer you're
learning some mechanism of pointing to
the the thing that you uh you know the
in the in the world model that
corresponds to the thing that you care
about
well uh you know what is that what is
that mechanism well essentially you know
you have to learn some ground truth you
have to learn some way to sort of like
you know reason about what it is that
you care about here's an example let's
say you have a duck and you know great
uh you know natural selection Evolution
wants to get ducks that care about their
mothers right you want you want ducks
and they want their you know they want
them to follow their mothers okay so
what is the you know mechanism that
Evolution uses to you know get that well
you know it uses this sort of imprinting
mechanism right where the Ducks you know
they're born and then they see their
mother and they're like oh that's my
mother I'm gonna follow this this thing
but you know sometimes this fails you
know the Ducks don't see the mother when
they're born because they were born you
know they by humans and they see the
human and they're like that's my mother
right uh and you know that's a problem
because the way in which you know the
pointer was encoded was not actually
very robust
and so if you want to solve that problem
you actually have to encode a really
robust pointer you know you have to have
a way of encoding you know and searching
for you know okay I have this
understanding of the world that I
develop how do I pick out which thing in
that corresponds to my mother right well
it's not the first thing I see though it
is sometimes the first thing I see
seeing it first is evidence in favor of
it being the actual mother but it's not
necessarily the actual mother right you
have to encode a bunch of detailed
information about how to actually pick
out your mother from the world model
um and the deceptive agent doesn't have
to do that right the deceptive agent
just has to encode you know okay here's
the world model here is some complex you
know here's some extremely simple
objective that cares about the long term
you know get as many paper clips as
possible and then you just do
optimization you're just like okay given
that I know these facts about how the
world works and I know that I care about
maximizing paper clips what is the best
strategy for me to get the most paper
clips and it's just like okay let's do
some search let's try a bunch of
different possible options and see which
options will get me the most paper clips
and it's like oh the option that gets me
those paper clips is play along with
this training objective right now and do
whatever it wants and then eventually
I'll do something later and I'll get the
most paper clips that way and so you
don't have to encode for some really
complex way of picking out you know what
your mother is if you end up doing the
deceptive route
okay so that's how what that's what
deception might look like so what are
the sort of you know if you if you ended
up in the situation you know what what
would this look like you know putting
this all together if you're going to end
up in a situation where you developed
deception what are the learning Dynamics
going to be well you're going to start
with a some pseudo alignment you're
going to get some proxy you know to
start with as you sort of you know start
training on some objective at some point
the model is going to learn information
about the training process from its
input data uh and then at some point uh
you know uh gradient descent is going to
take the model and it's going to modify
it to uh you become deceptive you know
give it some long-term objective to care
about and then the model is going to
start instrumentally authorizing for the
base objective not because it cares
about it but because you know for the
purpose of staying around so they can
eventually get the most of whatever
proxy it wants on its long-term
objective the gradient descent gave it
and then at this point whatever
objective is there in the model that
greatness has given it some long-term
objective that objective essentially
crystallizes that objective you know no
longer is relevant performance at all
you know whatever that objective is it
does not matter what it is because
whatever it is it's going to result in
the same performance because the reason
the model has good performance in
training has nothing to do with the
model's objective and everything to do
with the fact that the model is
deceptive
and so it no longer matters essentially
uh what the main subjectives uh what the
base optimized objective is it's now you
know crystallized it's frozen uh and you
know for all intensive purposes you know
it's not going to change because it's no
longer performance relevant okay
it's possible you could get drift like
towards simpler things but you're not
going to get an incentive towards
getting a thing which is you know more
aligned okay so at this point you know
Green descent essentially stops
improving the main subjective and the
only thing that is performance relevant
now is just making it so that uh it's
better being deceptive you know giving a
better world model giving a better
understanding of the training process
helping it be deceptive more effectively
because that is the thing that is
performance relevant at this point
I don't totally know why it's more
complicated to like don't you have to
get it's not necessarily a pointer in
the deceptive case but you still need to
form just as good a world model of the
base objective in both cases you do
however you don't have to do that in
your weights
you have to do so in the activations
right so uh way to think about this
right so remember we were talking
previously about the distinction between
Simplicity bias and speed bias right so
the deceptive model it doesn't have to
encode some pointer to like how to
figure out what its mother is
all it has to encode is like care about
you know uh over the long term care
about uh you know coins or whatever and
then it does reasoning on top of that to
figure out oh I should you know optimize
you know for this particular base
objective thing those reasoning steps do
take extra time but they don't take
extra complexity because it's not the
case that you had to encode for that
information in the actual structure of
the model it only occurred it sort of
figures it out at runtime right it just
like does some reasoning and realizes
what the base objective model is when it
just sort of you know runs you know
forward in runtime and so it costs extra
speed to be deceptive but it takes less
complexity to be deceptive and so if you
expect you're going to get more
Simplicity bias then you should expect
you're going to get more deception and
if you expect you're going to get more
speed bias you should expect you're
going to get less deception
but if you could have that like sort of
uh
that to be evaluated thunk of what your
objective is what I I think I'm so
missing why the corrigibly aligned one
can't do the same
so the query and the deceptively line
one both end up at the same spot but the
question is how long do they take to end
up in the same spot and how much
information do we have to specify to get
them to the same spot right so how much
information we have to specify to get
them to the same spot well the quarterly
Line run we have to specify this complex
pointer and the second line one we have
to specify some objective over the long
term how long do they get there well the
course really one a line one probably
gets there first because it has to do
fewer steps of reasoning it has to
reason well you know what I have to do
is you know just follow this pointer the
deceptively alignment has to do a couple
more steps it has to be like well you
know what I want is these coins but I
know there's this training process and
so I know this training process cares
about this thing so I want to do that so
there's definitely a line one takes a
couple more steps to get there but the
information necessary to necessary to
encode in the model to get there is is
smaller and so uh it's it's more
compressed and it's simpler but it but
it does take a couple of extra steps of
reasoning at runtime
so so the deceptively line one you know
if you're really like for example if you
if you really Force like short small
models then you're going to
disincentivize deception uh because it
won't have enough time to actually you
know do the computation alternatively if
you have really large models and you
have a ton of Simplicity bias then
you're going to incentivize deception
because you're going to be in a
situation where it wants to encode
things and as compressed away as
possible
so if um like if
various objective is and call pretty
easily in terms of the activations then
the additional complexity isn't that
large right it's just something like log
of the model size in terms of bits
like for the pointer size are you
oh yeah that's what you mean when you
talk about sort of theater and
complexity or are you imagine that it's
actually some complicated function
object
um so I think that like I think that's a
like a reasonable guess if you're like
logging the model size number of bits I
think that um
yeah I think that's not not crazy I
think the like deceptive objective is
competitive with that uh and like
probably simpler especially for large
models however I think it might be worse
than that so my why might it be worse
than that well it's not just a mechanism
of like pointing to some existing thing
in the world model it has to be a point
where that generalizes correctly right
so it has to be a pointer that like you
know in the situation where like you
know maybe the like mother is in a
different spot right you know you have
to actually have a pointer that encodes
enough information to be able to deduce
in different situations where do you
find the mother right and so it actually
does have to encode potentially enough
information to generalize correctly not
just like point at the correct thing
whereas the deceptive one you know
generalizes correctly just by you know
fact of it's always going to just do
whatever the thing is that results in
the most training performance because it
just wants to get trading performance
for the purpose of staying around
whereas the coursely line one has to do
some extra stuff to actually robustly
get good training performance
does the uh pointer need to be robust to
the uh models uh World sort of model uh
changing like over time if you train it
it's going to update that model and its
location above it may change its
definitional model may change yeah it
does yeah it doesn't have to be robust
to that or does it just not just not
care uh it does and so that that can
make the corrigible line one a little
bit harder because it like you know as
green descent is changing the world
model it also might have to change the
the pointer yeah
okay whereas with the deception it sort
of doesn't have to change it it's just
sort of fixed yeah it's great to say
that deceptive you're just encoding
survive and then it figures out from
first principles that it needs to what
objectives to go find and do that that
would be an example survive would be an
example of an objective that is simple
and encoder of the long term but I'm not
even I'm making a broader claim it can
be anything over the long term that it
cares about doesn't have to be survived
because survive is an instrumental uh
convergent instrumental goal of
essentially anything that you care about
over the long term but you survive it
would be an example yeah and it's much
simpler than anything that's like how to
infer which of the training
sorry which which of the objectives we
need to find right okay
okay so I promised also that uh I would
talk a little bit about what happens in
a situation where you know uh that it
doesn't necessarily have to actually
know when it's in training or deployment
so just a couple of words on this very
quickly so why does the model not
necessarily have to know whether it's in
training or deployment well uh you know
you can have a situation where your
model
um
it doesn't you know even if it if it
doesn't know exactly when it's gonna be
in training and when it's an employment
has some uncertainty you know maybe it
thinks it's just being tested it's in
some situation where actually you know
it's like in a honey pot and you might
be trying to look for deception in that
situation it can just sort of chance it
right it can be like well I don't know
100 sure if this is deceptive if this is
training or deployment but like you know
probably if I pick some particular
fraction of the time to defect you know
to be deceptive versus not be deceptive
and then I can you know approximately
end up in a situation where probably I
will you know they will have seen no
defections in training and yet there
will be a defection that exists in
deployment and so you know there exists
some optimal fraction of the time that
you are supposed that you know that is
supposed to defend uh you know for any
particular setup you know even if it has
no ability to distinguish uh you know
there was always going to be you know
some optimal fraction of defections that
you can compute based on you know the
size of deployment the size of training
uh you know how good it is to
distinguish between them um and so even
in a situation where you can't tell at
all you sort of have perfect adversarial
training uh you're sort of still you
know uh you know potentially quite
scared of this
okay and
uh you know I think it's like it's like
you know well you know I actually think
it is actually you should expect that
it's going to be able to determine which
one it's in so like while I don't think
it's strictly necessary for it to know
whether it's in training or deployment
actually I think that it probably most
the time is so why is it going to know
well so there's a bunch of examples of
distributional shifts then normally we
don't care about very much but in a
situation where your model is actively
trying to look for distributional shifts
that will tell it whether it's in
training or deployment become a lot
scarier right so these are things like
uh you know well you train the
simulation and then you're pulling the
real world and you know usually you hope
the simulation is pretty similar but if
there's any difference at all then you
know it can use it to detect whether
it's in in training or deployment uh you
give it literally time you know it sees
a clock and it can tell you know what
date it is and it can use that to you
know determine whether it's in training
or deployment uh you know normally you
know it doesn't we probably doesn't
matter you know what the dates are on
the data that you give it but you know
in this situation you know maybe it does
this leads to the uh the Mayan calendar
doomsday scenario where you know you
have all the AIS and they're really
great and awesome and then you know the
year hits it's like you know 20 50 or
whatever and they all defect
simultaneously and you know wipe us out
um whoever that's very unlikely
um it's not going to happen but you know
it's fun to think about
um
okay but here's a really scary one so uh
in fact I think the problem is even
worse than this because there are
situations where the model can look for
something which is essentially
guaranteed to exist in deployment and
yet is impossible to produce in training
so why is that well so this is a very
large semi-prime number it is composed
of two large primes multiplied together
factoring this number and producing the
primes that were multiplied together to
produce it is extremely difficult it
takes a bunch of computation that we do
not have access to right now and in fact
you know somebody actually did this they
actually took two primes multiplied
together somebody has the primes and
then they publish this number and they
sort of want us to factor it and you
know you know try to you know there's a
big challenge you get some money right
if you factor this you know uh okay so
you know at some point you know
computation is going to improve
eventually we're going to have you know
the resources to factor this number
somebody's going to do it right
somebody's going to publish you know the
two primes uh you know in some paper you
know where they managed to factor it
okay so if I'm a particularly devious
deceptive model the thing that I do is
the way I know I'm in deployment is I
just look for a factorization of this
number if I ever see two things they
would multiplied together produce this
thing then I know for a fact that I must
be in deployment because there's no way
that in training the computational
resources were available to factor this
thing and yet at some point somebody in
the real world is going to factor it and
so I'm in a situation where uh it's
impossible for me in training to
generate an input on which the model to
fax against me and yet I know for a fact
that at some point in deployment it will
defect because somebody will publish
this thing and so I think that you know
you shouldn't literally expect it's
going to look for this thing but the
point is that this is sort of uh you
know an example of a sort of larger
class of like you know sort of P versus
NP style problems where it's just like
look it is easier to detect a fact about
the world than it is uh to actually
instantiate a thing which satisfies that
property and so because of that
fundamental asymmetry there were always
going to be things that the model can
look for that are you can't actually
produce examples of via any sort of
adversarial training
okay that's the talk
the paper uh I also you know tease that
I have more things to say as well that
we can talk about more at some point
later maybe but uh for now we will leave
it at this uh happy to take questions
and everything afterwards
[Applause]