3:How Likely is Deceptive Alignment?: Evan Hubinger 2023

okay
so we are picking up where we uh left
off uh last time talking about deceptive
alignment today uh if we recall last
time we sort of started establishing
this case for you know why deceptive
alignment might be a problem why it
might happen and today we're going to
try to uh pick up with that and really
go into a lot more detail on what the
case for deceptive alignment might look
like
okay so uh let's just recap so so what
is deceptive alignment so
uh one thing I really want to get out at
the beginning is just some things that
it's not so one thing that deceptive
alignment is not
um is dishonesty so we're not talking
about a situation any situation where
you have some model some AI system and
that system you know lies to you it
tells you some false thing there are a
lot of situations where that might
happen
um but the situation that we're
concerned about is deceptive alignment
which is specifically the situation we
talked about last time where the reason
that the model is doing a good job in
training the reason that it looks
aligned the reason that it's
pseudo-aligned that it sort of seems
like it's doing the right thing is
because it is actively trying to
accomplish some other goal uh other than
the one that we want and uh you know
because it eventually wants to do some
other thing it's pretending to look
aligned in training uh for that purpose
okay
um
and so in in this sense that's sort of
why we're talking about it last time as
this sort of form of pseudo alignment
you know that only arises potentially in
or that they can't arise even in the
case where you've sort of you know
trained away everything that you can
with adversarial training
um that we're not necessarily going to
be talking about just in that context
here we're just sort of talking about
any situation where
um the reason that the model looks
aligned in training uh is because it's
trying to accomplish some other role
okay
so one analogy that I really like for
this is from Ajay kocha so the idea is
let's say that
um
I'm trying to run a business
and you know I'm a child I just
inherited this business and I need to
select somebody to run the business for
me but I'm a child and so I have a very
difficult time actually being able to do
that selection
um
here are three different types of people
that you might find in your selection
process we have saints these are people
who really genuinely want to help you
run the business
we have sycophants
people who you know
want to do whatever it takes to make it
look like they're doing a good job
running the business but don't actually
care about making the business good and
you have schemers people who want to use
the business for their own purposes and
so I want to fool you into thinking
they're doing a good job
and so in the deceptive alignment case
we're specifically talking about
schemers right both sycophants and
Saints might lie to you but schemers are
specifically this deceptively aligned
case they're the case where they have
some objective that is different than
what you want and the reason that they
look like they're doing the right thing
is because uh you know they eventually
want to you know do some other thing
that is not what you what you want
okay this all seemed clear
great
okay so we want to understand How likely
is deceptive alignment in practice
so uh the problem is you know as we sort
of we're establishing last time in terms
of you know why deceptive alignment
might be such a problem even the limit
of adversarial training is that
deceptive alignment and you know robust
alignment in a situation where the model
is actually doing the right thing are
behaviorally indistinguishable uh at
least during training right no sort of
input that I could possibly generate
would necessarily be able to always
distinguish between them right we saw
things like RSA 2048 last time where
there are examples of cases of a thing
that a sort of deceptively aligned model
could look for it to tell whether it was
in you know training or deployment that
we could never possibly actually produce
an example of during training
and um so because they sort of both look
the same because they're sort of both
you know trying to you know do the right
thing you know one just for nefarious
purposes
um the question of which of these two
models we get is going to be a question
of inductive biases it's going to be the
same sort of question we started this
whole thing off with of trying to
understand you know what is the
structurally simplest model what is the
sort of model that is the one that would
be selected for by by grading descent by
our neural network you know uh you know
that would be sort of simple to
implement uh that would be most likely
to find Yeah question
so it seems pretty clear that the
outputs of a deceptively aligned and
robustly aligned model would be the same
in training so there's no way to tell
the difference but how do you feel about
cracking open the model and actually
seeing the cognition inside the
interpretability field that's building
how do you think that's likely to work
do you think that's just due to failure
what is your opinion on that
yeah that was a really great question I
think my answer right now is going to be
we're going to imagine we have no
interpretability we're going to be like
imagine we're in the situation where we
can't look inside the model the only
thing that we know is that it is some
you know algorithm that we found that it
actually does a good job on training uh
and that algorithm is going to be
whatever algorithm is most favored by
the inductive biases uh of our training
process you know whatever is you know
the structurally simplest thing that is
you know the largest Basin whatever
um theoretically there might be things
that we could do to try to change that
default picture right there is some
default picture that is if you know we
just plow ahead with machine learning
with the you know exact same sort of
inductive biases that we you know
generally use by default and we don't
make any changes to try to adjust for
avoiding deceptive alignment what
happens right now
that's not entirely realistic and since
we probably will do some things to try
to adjust for it and we're sort of talk
at the end uh you know about what that
might look like
um and you know in later sort of
lectures as well but right now I just
want to establish what is the default
case right and so transparency
interpretability you know trying to look
inside do something to actually try to
detect whether it's deceptive or trying
to find some way to avoid deceptive
align cases is absolutely something we
may want to do but it's not something
I'm going to be talking about right now
we're just going to be sort of trying to
understand what this sort of default
case is
okay
great
okay so you know again just reminding
you know the analogy from you uh from
the first time right is we're sort of
thinking about this case where there's
sort of two possibilities you know we
could learn the color classifier or we
could learn the shape classifier and you
know they both look the same in training
they both sort of you know do a good job
on these particular training examples
but you know in in some particular
deployment examples in some other cases
they do different things and you know
the deceptively aligned model and the
robustly line model are very similar
right they sort of always do the exact
same things in training you know every
example that they that we can check you
know they both look like they're trying
to do the right thing but if the
deceptively aligned model sometimes you
know finds itself in a situation where
it has the opportunity to you know break
away from whatever processes we have to
you know prevent it from doing something
bad then it's going to do something bad
right once it's a deployment once it's
out there you know it might not do
something very different and so
um
again if we want to try to answer which
one of these models are going to get we
have to answer the same sort of question
uh that we sort of started the whole you
know series with of how do we
distinguish between which of these
models we're going to get how do we
distinguish between getting the
algorithm which is doing shape
classification and the algorithm which
is doing color classification
okay
so uh this is a really difficult
question as we sort of you know have
already established it's really you know
it's not trivial to understand the
inductive biases of these machine
Learning Systems
um and so in the sort of face of
uncertainty I think one thing that we
can do that is really helpful is try to
sort of do case analysis right we can
say okay we don't know exactly what the
inductive biases uh look like we don't
know what is sort of going to happen but
we can suppose that while here you know
two possible scenarios for how to think
about you know which sorts of algorithms
are favored by Machine Learning Systems
you know what does it mean to be
structurally simple what does it mean
you know how do you know uh training
processes select for which algorithms
they choose and which ones they don't
um and sort of see what consequences we
get in each scenario and if we sort of
have you know robust conclusions uh
across various different possible ways
of thinking about the inductive biases
of our machine Learning Systems then we
can be more confident that even though
we're very uncertain about you know what
it actually is that is causing you know
which algorithm we get over the other
um
that you know we still at least have
some conclusion that that is Meaningful
so that's the idea we're going to sort
of break things up into two cases so
we're going to call these the high path
dependence case and the low path
dependence case where effectively these
sort of represent you know two ways of
thinking about the inductive biases of
machine Learning Systems two ways of
thinking about why you would get one
algorithm over the other why you would
get you know the color classifier over
the shape classifier
okay so here is Story number one this is
the high path dependence world
so in the high path dependence world
um we're going to say well different
training runs can converge to very
different models depending on the
particular path taken through model
space so depending on exactly how we set
up the initialization it really matters
sort of you know what what Basin you're
going to end up in really matters what
how you walk around that lost landscape
right that you know there might be
multiple basins uh that you know you
could fall into that are similar sizes
and which one of those basins you're
going to find really depends on your
initialization and the exact sort of
path that you take another way to think
about this is
um it's sort of very sequence dependent
so you know maybe we start with you know
learning one particular algorithm as
like you know one of the circuits in our
model but then to learn some additional
algorithm well an additional circuit
will only be learned if it's helpful
given that we already have the other
existing circuits and if it isn't
helpful given you know the things that
we've already found then we won't find
it and so we're sort of thinking about
things as well okay we have to first
find some individual thing which is
incrementally helpful and then other
things which are incrementally helpful
building on top of those other you know
uh things that we found previously and
so we sort of are imagining our model
being built up in this sort of iterative
process and an iterative process that is
potentially you know very sort of path
dependent it's a process that is you
know can vary quite a lot depending on
exactly you know which things we build
up first in the model
okay yeah question about this
uh last week you mentioned that
induction heads were engines that were
built from transformance and they
occurred not immediately in training but
are they relatively predictable point
and they basically always happened
wouldn't that suggest that Transformers
training are sequential but also low
path dependents
yeah it's a really good point so I think
a couple of things I'll say to this so
first I'll say I think induction heads
are some amount of evidence at least
against half dependence in the case of
induction heads where it seems like the
induction heads really you know always
form regardless
um though I also agreed that there's
sort of two things that I'm conflating
here there is a you know extent to which
things all converge uh you know or
diverge depending on how I set it up and
there is should I think about things in
this you know uh sort of piecemeal way
where we're adding things on top of
other things it could be the case
theoretically that the best way to think
about in the inductive biases is well
they always converge to the exact same
thing but the way to understand what
they converge to is to try to do this
analysis of you know what circuits would
be good to build upon other circuits but
it'll always end up being the same
circuits that's totally plausible I
think that
um the thing that I would say is that I
think that that's a little bit less
likely if it is the case that the
inductive biases sort of do you know the
correct way to think about it is this
sort of iterative process that I think
we should expect on average things to
diverge more and to diverge uh sort of
converge more in in the other case that
we'll talk about in just a sec so
um but I don't mean to say that these
sort of two cases are comprehensive
um is maybe that you know the key Point
here is that you know we're going to be
talking about two possible cases and
each one bakes in a bunch of assumptions
you could alternatively separate out you
know just a couple of those assumptions
and mix and match and try to run the
analysis again and see what you know
what happened in that case
um I think that would be a really
valuable thing to do and would help give
us even more confidence or less
confidence depending on how things go in
terms of you know um what the sort of
overall you know structure of different
you know conclusions from different
types of inductive biases
for our purposes right now we're sort of
looking at these sort of two very Broad
and disparate categories where one is
like very high path dependence things
are very sequential and very Divergent
and the other one is very low path
dependence which is going to be things
are very convergent and very
non-sequential we'll talk about it a bit
um and so hopefully that you know the
ideas sort of cover as much of the space
as possible but I absolutely agree that
you could imagine something more in the
middle that sort of mix and matches some
of these properties without the others
they're not necessarily like these are
the only two uh ways to think about it
okay and and I also want to point out
that you know both of these views that
we're going to talk about have I think a
bunch of empirical evidence supporting
them so if we think about the high path
dependence view you know some empirical
evidence that I think supports the high
path dependence view
um there's birds of a feather do not
generalize together which is a paper
that found if you take a bunch of
different fine tunings of Bert which is
a language model
um that sometimes they can have really
disparate generalization Behavior you
know really disparate behavior on you
know new data points
um this very often happens with
reinforcement learning because
reinforcement learning can be really
path dependent in the sense that once I
get a particular policy which additional
you know changes that policy are helpful
are really dependent on you know what
moves is the thing going to make after I
make this move right so you know if we
think about playing go you know making a
particular move might only be helpful if
I know how to follow it up right and if
I don't know what the correct next move
is after that move I don't know the sort
of whole line then making that one
individual move might not actually be
helpful and so there can be a lot of
more a lot of sort of additional path
dependence in reinforcement learning
um
and so so there's some evidence like
this that sort of points us into uh you
know thinking that maybe High path
dependence is is a good way of thinking
about the inductive biases of machine
learning systems
okay
but it's not necessarily the only way so
the low path dependence view is also I
think quite plausible so how do we think
about low path dependence
so the idea of low path dependence is so
everything sort of converges to the same
sort of unique Simple Solution
regardless of you know how we set up our
training process
so why might we think that this is true
well this is sort of going back to what
we were talking about at the beginning
about you know really disparate Basin
sizes you know some basins can be much
larger than others and really heavily
favored over others and so we can be in
a situation where you know maybe
theoretically you could end up in any
Basin but effectively for all intents
and purposes there are going to be these
structurally simple algorithms with
really really large basins that dominate
The Lost landscape and sort of force you
know essentially all paths into them and
furthermore in this case we're going to
be sort of thinking about how to
understand which algorithms are these
sorts of you know occupy these really
large basins I was just trying to
understand this basic question of you
know which algorithms are structurally
simple right these sort of basic
properties of the algorithm right Global
properties about if I look at the final
algorithm can I understand how simple is
it how many sort of you know distinct
pieces does it use how many free
parameters does it have to try to
understand
um you know how likely it is and so we
in this case we're imagining we don't
have to think about this sort of you
know piecewise iterative process where
we're not imagining that you know first
you develop this thing and then you
develop that thing where instead
imagining let's just think you know if
we look at the final algorithm and
evaluate some Metric of you know how
simple structurally simple is this final
algorithm from that we can determine How
likely it is for that algorithm to be
the one that we find from our machine
learning process and so this is sort of
you know a different way of thinking
about it
um I think it's um again you know also
quite plausible and supported by lots of
evidence we talked sort of you know in
the first lecture about stuff like
rocking where you know you have this
case where if you just keep training
eventually you just you know snap into
this one really structurally simple
algorithm uh that sort of dominates the
landscape after a bunch of training
um and we can also think about this is
um the Min guard at all line of work
where what they find this is another
sort of piece of evidence in favor of
low path dependence
what they find is that they take
um
a sort of approximate model of what it
might look like to have low path
dependence which is take a um
randomly initialized neural network
where we just sort of Select each
parameter from from a gaussian and
um you just keep randomly initializing
your neural network over and over and
over and over again until you stumble
upon a random initialization which
actually does a good job on the task now
this is not practical to ever do in
practice but you can sort of approximate
it via various approximations and you
can measure if you theoretically did
this if you just kept doing random
initializations until you found one that
did a good job
how close is that to what grading
descent does
um and if those two things are very
close it implies a form of low path
dependence because it implies it doesn't
really matter the exact path the grading
descent takes it only matters the sort
of you know how you know likely is it on
the random initialization which is sort
of a proxy for you know how structurally
simple is the algorithm because as we
sort of talked about at the beginning
you know structurally simple algorithms
have more implementations in parameter
space
um and they find that these are very
close and so you know that's some
additional evidence in favor of the low
path dependence View
okay so I think that these two overall
views are plausible again they're not mu
they're not the only views right there
could be other ways of thinking about uh
you know inductive biases they're
definitely not you know exhaustive but
they are like two very different views
and so we can sort of you know try to
understand what happens in each case as
a way of sort of covering the space
or at least you know trying to get you
know different conclusions from
different places
okay so we're going to start by
start by end you know going back to the
reason you know the thing we're trying
to talk about is deceptive alignment uh
so the thing that the first thing that
we're going to do is we're going to try
to understand what is the likelihood of
deceptive alignment in the case of high
path dependence so we're thinking about
uh you know specifically inductive
biases in this High path dependent sense
where uh you know it really matters the
exact order in which we develop
individual circuits and individual ways
in which you know you know you know
things can be very Divergent
um and we're trying to understand in
that view How likely is the development
of the deceptive alignment algorithm
right as opposed to the robustly line
algorithm in in you know the same sort
of sense as we're just thinking about
you know shape versus color classifier
okay so uh we need to you know get a
couple of things out of the way in terms
of you know how we're going to be
comparing you know various different
model classes here so
um we're going to be supposing that our
training process is good enough that for
our model to be able to do a good job it
has to sort of fully understand the
thing that we want
um this is sort of very similar to what
we were you know talking about at the
end of the last lecture about you know
the limit of adversarial training where
you know if we can do enough adversial
training if we can give it enough
examples of different possible
situations that it could be in we can
eventually sort of you know get down to
only a small section of possible
algorithms that are able to you know fit
all of the examples we can give them so
you know we're not going to be thinking
about algorithms like
um you know the where it's where it's
doing like the you know the maze thing
where it's trying to go to the Green
Arrow right at the end of the maze
because in that example we can just
train on mazes that have the green arrow
in a different spot right and we can
sort of force it to try to do the right
thing in that case and so we're really
only considering models that even in the
limit of every possible example that we
can generate in training it still looks
like it does the right thing and
importantly like we talked about last
time this is going to include the
deceptively aligned model because of
structural facts like RSA 2048 mean that
there are always going to be you know
facts about the world which are easier
for our model to check than they are for
us to generate which means even the
limit of our ability to generate as many
possible things as we can there will
always be sort of you know properties of
the world a deceptive model can look for
to sort of tell whether it's something
it's you know a test example that we
generated or a real thing in the world
um and so you know even for all the
possible things that we can generate the
deceptive model is still going to be
there but it does get rid of a bunch of
these sort of you know bad models that
don't really understand the actual goal
right they're just trying to you know do
some proxy for the goal instead
okay yeah question
this idea of doing a number of
adversarial training so that you end up
with a training process that the model
fully understands what we want whether
it cares or not How likely do you think
we are ability to get that in practice
as we get smarter machine learning
systems
yeah so I think that you know eventually
the thing really you know should
understand what we want right
um there is sort of some finite
complexity uh of the like actual
desirable thing that we're trying to get
it to do
um and you know as your model becomes
better and better at understanding you
know what it is that you're trying to
accomplish eventually it's going to
understand now we taught last time right
about you know there's at least two
mechanisms for doing that understanding
right it can understand it just via
right internalization where you know it
has some internal understanding and
eventually green descent just sort of
adds it or it can happen right through
modeling where the model is just trying
to understand the world and in
understanding the world eventually just
like figures out enough facts about the
thing we're trying to get it to do that
you know that's the sort of process via
which it learns the information about
what we're trying to accomplish but
eventually you know as we make more
powerful AIS that are better and better
at understanding things eventually
they're you're going to figure out facts
like this uh you know facts like the
thing we're trying to get to do now one
of the really important things right and
we're going to be talking about this a
bunch is that it really matters in the
high path interview at least it really
matters The Ordering of which one of
those two sort of paths comes first you
know do we get the internalization first
where we you know first uh gain the
understanding you know internally where
it's you know grading essential sort of
directly replaces the existing proxy
with you know the thing we're trying to
get to do or does the modeling happen
first where you know learns and
understands enough facts about the
training process to figure out what
we're trying to get to do that way
um but either way you know eventually
you know we're sort of can imagine that
you know as AIS get better you know this
should this should occur
okay
so uh how are we going to be comparing
different model classes right different
types of algorithms that have the
property that they sort of fully
understand what we want so in the high
path dependence case uh there's sort of
two you know main things we're really
going to be looking for we're gonna be
trying to understand uh you know from
each step you know that it takes to sort
of get to that algorithm how much
marginal performance Improvement do we
get you know as we're sort of going
along walking through this lost
landscape you know how steep is that
path down the Lost landscape you know to
this sort of algorithm right how many uh
you know how much performance
Improvement are we getting along the way
and the reason this is important right
is the gradient descent is going to take
the steepest path and so we really want
to understand you know is this actually
you know giving us large incremental
perform performance improvements as
we're sort of progressing towards this
this type of model
and then the second question we want to
understand is how many steps are needed
right if it's going to be really really
difficult for us to eventually reach the
you know desired uh you know that
particular model class it takes a bunch
of steps that's going to make it less
likely if it's fewer steps uh that's
going to make it more likely because
we're sort of imagining that well you
know in this High path dependence case
the more steps that you're sort of the
sequentially that you have to take you
know the more steps that you could end
up taking some different thing instead
and so you know fewer steps means it's
going to be more likely
okay so these are the sort of things
we're going to be looking at in the high
path dependence case where we're sort of
thinking about inductive biases in this
High path dependent sense the things
that we're going to be looking at to
understand How likely it is that we get
a particular model class a particular
sort of set of possible algorithms are
these sorts of factors
okay
question yeah
what follows from Big steps so in Boeing
to one
yeah what does it mean that it in every
step there is a big Improvement pitch
Direction This is nevertheless for and
why
I'm perfect
so the idea here is that if we're
thinking about write a lost landscape
where you know each individual possible
setting of the model parameters
corresponds to some particular algorithm
which has some particular loss as we're
sort of you know traveling around this
lost landscape there's some Basin that
has you know perfect loss or whatever on
this you know it fully understands what
we want
um and you know for some each model
class you know we're looking at
different sorts of basins and we're
going to try to understand you know what
is the path from some random
initialization to that Basin look like
and then based on that path we want to
understand you know How likely is that
path under this sort of high path
dependence View and if we're thinking
about it well okay so what would make a
path really likely to be selected by
gradient descent well one of the really
important factors is the steepness of
the path because the thing that
greatness sent will do is it will select
the steepest Improvement in each
individual point that sort of
structurally what gradient descent is
trying to do it's looking for the
direction of greatest marginal
performance Improvement and then
stepping along that direction and so
because of this sort of basic
understanding of you know gradient
descent we're going to be thinking about
looking for large marginal performance
improvements along the path as being
evidence of the sort of path that green
descent would want to take in this sort
of high path dependency
okay yeah more question
so it's either that this performance
improvements per step is like a relative
measure compared to other steps that the
model could take at any point right
because like it seems like kind of hard
to hard to tell uh absolutely for a
given training run or a model by us like
will we get the high marginal a little
bit called this movements per step or
less like what's the way we could start
approaching this a question
yeah it's a really good question so I
think that you're sort of anticipating
one of the things we're starting to talk
about which is well I'm gonna have to
make a bunch of assumptions and I think
the way that we're sort of going to
approach this is we're going to say
um here are three particular paths and
then we're going to just compare those
paths and we're going to see you know at
which point do those paths diverge and
when they diverge which has the
strongest marginal performance
Improvement right and I that's not in
general the right way to do it right if
we really wanted to solve you know solve
this problem properly we would need to
integrate over all possible paths right
um but that's really hard and uh so
we're not going to try to do that right
now we're just going to look at sort of
three example paths and try to compare
you know the individual probabilities of
those specific paths and from that we'll
you know try to draw some sort of
General conclusion but obviously that is
a little bit you know uh you know we
have to be a little bit wary of that
because it's not literally correct right
we aren't uh actually doing the full
analysis if we want to do the full
analysis we really would have to do sort
of what you're talking about and look at
all of the possible paths and really try
to understand each so we're going to be
doing an approximation
okay cool
all right
so I just said we're going to be
comparing three different paths so uh
those are going to be three different
paths to three different model classes
um that we're going to be sort of trying
to understand
um I'm these are probably going to be a
little bit familiar from what we were
talking about at the end of the last
lecture uh I'm going to introduce a new
analogy to sort of help us you know
really get our heads around these
so we're going to suppose that uh in the
sort of analogy that you are the
Christian God
and you want to get humans to follow the
Bible and so we want to understand what
are the sort of humans that generally uh
do a good job at following the Bible
okay so here is an example human that
does a good job at following the Bible
is Jesus Christ so why does Jesus Christ
do a good job at following the Bible
Well in you know the sort of Christian
ontology the idea is that you know Jesus
Christ is essentially the same as God
he's sort of a copy of God uh in some
sense right and so
um
you know Jesus Christ does you know does
the right thing because Jesus Christ
just has the exact same beliefs the
exact same goals uh as God right and you
know in that sense you know therefore
because you know God wrote the Bible you
know uh Jesus Christ is going to follow
it right
um so this is like one type of human
that you can imagine in this sort of uh
analogy uh okay there's there's others
though right so another example would be
Martin Luther of of Protestant
Reformation Fame uh you know Martin
Luther you know he didn't he wasn't you
know the exact same is gone but he
really cared about trying to understand
what the Bible said and so he put a
bunch of effort into you know reading it
and trying to understand it and you know
he thought you know it said some
different things and other people did
and so you know he nailed a bunch of
Theses to a wall but
um the basic idea right is that he put a
bunch of effort into trying to
understand the Bible and really cared
about doing whatever it was that the
Bible said he should do right and so
that was another type of human right
that is in fact really good at doing
what the Bible says okay but it goes
through a different route right rather
than Martin Luther just inherently
having the exact same goals and desires
as God Martin Luther wants to figure out
what desires God has from reading the
Bible and then do those
okay and then we have you know human
number three which is Blaze Pascal so
Blaze Pascal of uh you know Pascal's
wager Fame uh he doesn't you know
necessarily know whether you know God is
real but he's like you know I think that
if I went to hell it would be really
really bad and I think there's you know
enough probability that I could go to
hell that I'm going to try to do a
really good job at you know what the
Bible wants me to do because not because
I actually agree with the Bible or want
to do the things the Bible wants to do
because I think if I don't do that I
might go to hell and so Pascal is does
not you know unlike Jesus who really you
know cares and has the same values as
God and unlike Martin Luther who really
wants to figure out what God actually
desires Pascal doesn't doesn't care
about what God wants he just doesn't
want to go to hell and so he's trying to
avoid that outcome by you know
attempting to play along and do what God
wants him to do
okay so if you've been paying attention
you probably know where I'm going with
this which is we can think about you
know each of these as mapping on to one
of the categories of you know that we
were talking about at the end of the
last lecture right where we can think
about the Jesus Christ as the internally
aligned models the Martin Luther's as
the courageously aligned models and the
blaze pascals as the deceptively aligned
models well the idea is that the Jesus
Christ sort of internally have the exact
same values uh as the ones that we're
trying to get uh the Martin Luther
models you know they try to figure out
what we want and then do that but they
don't necessarily you know have exactly
the same beliefs they just like
eventually figure it out by you know
trying to understand what we're trying
to get them to do and the blaze pascals
are deceptive you know they actually
want something totally different but
because they're afraid we're you know
the the training process is going to
punish them you know they they play
along okay
so these are going to be the three model
classes that we're going to be trying to
understand the likelihood of sort of
getting to
do it believe that the Martin Luther
will eventually decline Jesus Christ's
or the the problem in this case is just
too difficult the soul and so are
courageously aligned
agent cart actually become internally
aligned by just knowing
so in this case so so it could be that
you you you know start from a quarterly
line thing you become internally aligned
thing but but it's not necessary so in
this case we're imagining that all three
of these model classes are model classes
which actually fully fit the data
perfectly right so these are all cases
where
um each each one of these actually is
able to do everything correctly in the
training data and so it's not like the
chords really line one is sort of making
a bunch of mistakes and therefore it
like needs to become internally aligned
um what we're going to be imagining is
that the query line one actually does
know enough facts about what you know
the Bible says you know what like we
actually wanted to do that is able to
successfully figure out what it is that
we wanted to do in each case and so
because of that it's not going to be the
case that like you know these These are
going to differ in performance and so it
may be that you know you start from
coercimal you go to internal but if that
were to happen it would have to be a
fact about the inductive biases right it
would have to be something like rocking
where you know they both actually fit
the data in training but eventually you
know you shift from one algorithm into
the other which might happen and we'll
talk about you know why you might end up
with one or the other but
um you wouldn't it wouldn't happen you
know in this case we were imagining it
just because like the cordially aligned
one isn't able to fit the data so the
courage we align one
so the courage blue line one basically
is internally aligned with inspect the
training guarder but not necessarily
with respect to the real world
no no so internal alignment does not
mean like is aligned right internal
alignment means in this case something
very very specific which is it is it
internally has directly hard coded into
the algorithm into the weights of the
model the exact objective that we are
trying to get to accomplish right that
is what internally line means here right
so it is not like you're you know it's
internally aligned with respect to one
thing or not the other right the idea
here is that all of these models are
look aligned on training but the reason
they do a good job is different right
the internally aligned one does a good
job because it directly as a part of its
algorithm defines what it means to do a
good job in exactly the same way that we
would Define it the cordially align one
does not do that right it doesn't have
some part of the algorithm some part of
the model where it under it like defines
exactly what it means to do a good job
according to what humans want instead it
just sort of defines what it means to
figure out what it you know how to do a
good job according to what humans want
and then it sort of figures that out
based on understanding you know what
we've written and what we've told it in
a bunch of facts about the world right
and then the blaze pascals they you know
just encode for you know some other
random objective and then they have to
figure out ah as a you know to be able
to accomplish this other thing that I
really want I'm going to need to play
along in training
Yeah question
um feel free to tell me that this is a
question for a q a
um because it seems my derailless
further but uh
here we assume that the model does fit
the train there perfectly like a
basically understand exactly what we
want
um so I guess first of all would you say
that the currently like
most strongly aligned models uh fit any
of this uh with my guess being maybe
corrigible for like Chachi petite style
levels
um and do you think it's valuable at all
to think about which which of these it's
closer to and uh to think like what's
what's the current path and like
um
to factor in the current trajectory of
training
um before we start just assuming that
okay we we have converged to something
what it what is it uh do you think it's
valuable to think about the current
process
yes in terms of is it valuable the
answer is definitely yeah in terms of
are we going to talk about it right now
um a couple of things so so the first
thing I'll say is that I think I would
not describe you know the best models
that exist right now as being able to
fully understand the thing that we want
they they don't do that right there's
lots of cases where you know they
clearly are not doing the thing that
we're trying to get them to do
um and so they're sort of not yet in
this limit of adversarial training
um and so they're not any of these model
classes right now
um they might eventually become one of
these model classes but right now
they're none of them because they don't
actually you know they're not actually
in this limit yet
um you know we're going to be talking
about a lot of these all three the sort
of paths that we're looking at in the
high path events case for all three of
these often they're they're gonna look
like they start out in very similar
cases and they're I think that you know
that starting point is at least sort of
you know defined by you know what we
think current models are doing to some
extent I think that for for large
language models in particular the story
is a little bit trickier because it's
unclear if they even sort of have proxy
objectives in a meaningful sense
later on in another lecture we're going
to talk a lot more about theories for
like what internally might large
language models be doing and how to
think about that
um I think it's a a very particular case
um but I don't want to talk about it
right now so we're mostly going to be
imagining that the sort of arguments we
made in the last lecture about like you
know it's going to have some sort of a
proxy you know may subjective mostly go
through I think it's unclear the extent
to which those arguments go through and
what they sort of imply about
um you know current large language
models but we will return to that
specific question later on another
lecture
okay yeah question
so I think I understand why it's hard to
distinguish this to be in Pascal and the
first two during the training but is
don't Jesus Christ and Martha legit
behaved the Branded during training Bond
Martin Luther asked more questions even
like it depends on what kind of training
we have but I imagine that even the
training can be somewhat interactive and
I be I would imagine that a courageable
model would even towards the very end of
the training would ask a lot of
questions okay is this what you mean and
incrementally aligned be just do the
perfect job without versions
yeah that's a good point so I think that
the thing I would say is that is
possible
um but it is also possible that the
cordially line one just already knows
those facts right it may be that it
already knows all of the relevant facts
to understand what it is that we you
know we wanted to do but it has to
actually figure them out right so we
think about in an individual you know
run of the model the internally line one
just starts knowing exactly what it
needs to do the chords really line one
doesn't it starts knowing that it needs
to figure out what we need to do and
then it goes and looks in like wherever
it's keeping its knowledge and it has a
bunch of knowledge right and it's like
ah I understand these facts about the
world based on these facts about the
world that I understand I understand
that in this case I need to do this
particular thing and so it could be that
it already has that knowledge um because
we're sort of imagining this limit we're
going to be I think mostly imagine the
situation where it already has the
knowledge I'm also basically just going
to imagine they all already have that
knowledge because again in this limit to
be able to do a good job on basically
you know all of the possible tasks that
we can give it it probably just has to
understand tons and tons of facts about
the world and just sort of understand
those you know to start with and so
we're just going to be mostly imagining
they just sort of already has all this
all this knowledge
okay yeah question
where are we talking earlier about like
path dependence which was entirely
dependent on the training process itself
but now we're just assuming that that
training process is already done now we
already know everything that we need to
know I'm a bit confused of that change
yeah so maybe it's not clear we haven't
talked about the paths yet right to get
to these I'm only talking about the
model classes and then very very soon
we're going to talk about what a path
might look like to get to each one of
these model classes okay and then we'll
try to understand How likely each one of
those paths are okay great okay let's do
that so uh we're gonna start with a path
to internal alignment right so uh we're
in the high path about its world we want
to understand How likely is this model
class for the internally aligned model
and to do that we need to understand uh
you know what does a path look like uh
to get there so I said previously that
all these sorts of paths are going to
start in a pretty similar spot and so
the place that we're going to start uh
is we're going to start with this sort
of proxy aligned model so we talked
about this a bunch uh in the last
lecture the case where you sort of have
this Mesa Optimizer it has some proxy
objective right it starts with some goal
like you know go to the Green Arrow
rather than go to the end of the maze
that is correlated with the thing that
we want but not exactly the same and
then we want to understand starting from
a model that looks like that where are
you more where you sort of what
directions are you likely to go in the
sort of high path dependence uh View
okay so in the internally aligned path
what we're going to imagine is that
green descent is sort of going to
continually improve that proxy right it
starts with some bad proxy you know that
is sort of correlated it has some facts
uh about what it is that we're actually
trying to get to do but it doesn't
actually fully sort of correspond
directly to the thing that we want
um but it keeps it keeps on modifying it
because you know if it had a better
proxy it would have better performance
on various examples right if you think
about the case of the you know the Green
Arrow if we can give it an example where
it actually has to you know do the right
thing on the larger maze with the green
arrow in the wrong spot then it's going
to have better performance if it has a
better proxy and so green descent keeps
making the proxy better and better in
until eventually you know it sort of
fully uh you know fits
um the data and furthermore for the path
to internal alignment we're going to
imagine that that process of sort of
iteratively improving the proxy happens
before the modeling process right we
talked previously about you know there's
sort of two paths via which all this
information about the you know the thing
we're trying to get it to do could enter
into the model right it could be that it
just you know in general understanding
the world better is good for doing a
good job on almost any task and so
there's going to be you know pressure to
just understand the world better
we're going to imagine in this case that
most of the pressure that causes it to
understand the proxy it sort of first
happens you know via just direct
pressure to make the proxy better
because having a better proxy lets you
do better on some examples rather than
the sort of having a better world model
understanding the world better causes
you to do better on examples and so
we're because of this we're sort of
imagining in this case that first your
proxy gets gets better and better until
it sort of directly corresponds to the
thing that we want before your
understanding of the world gets better
and better and better until you fully
understand exactly what the training
process is trying to get you to do and
how the training process works
okay and we'll see sort of why that's
important because if we have the
opposite ordering then uh we're sort of
going to end up in in one of the other
paths
um but if we have that ordering then we
can sort of be in the internally line
path and so once we've gotten to this
point where you know first it sort of
proxy becomes uh you know perfect and
then it learns about the training
process
um then once that's happened
um there's no additional pressure to
make the model deceptive or courageable
or whatever because once we've gotten to
the point where the model has a perfect
proxy
um then uh the model is you know going
to be doing its very best to pursue the
training objective that we're trying to
get it to pursue and so there's sort of
uh you know no performance gains from
modifying that model to become deceptive
uh you know the only reason that you
would sort of you know go to the
deceptive equilibrium would be if
there's some you know increase in
performance right in this sort of high
path dependence case right you're sort
of looking at these paths if there's
some really high marginal performance
Improvement then you would go there but
we've already reached an equilibrium in
this case before we ever got to a point
where there was you know a sort of
reason to become deceptive and so in
this case once we've gotten here we're
at equilibrium there's sort of no reason
for them all to change we have you know
this sort of internally aligned model
that's doing the right thing
okay so this is one path
um so now we need to understand you know
How likely is this path
so I think there's sort of an issue and
the issue is this sequencing staff that
I was talking about so
um why is the sequencing step uh so
tricky so I think that the really tricky
thing about what I was sort of just
talking about is that I'm supposing in
this internally line case that we sort
of fully understand the proxy before we
sort of uh you know we fully understand
the training objective we get this sort
of perfect proxy before we sort of
understand you know a bunch of these
facts about the world
and um I think this is you know the
first thing that's implausible is
um if we're thinking about uh you know
the sort of two things that greatness is
doing to make the model better you know
giving it better beliefs and giving it a
better uh you know an objective that is
closer to the thing that we want there's
going to be diminishing marginal returns
to you know putting additional
Improvement in each one of those two
sort of buckets right and so it's it's
just you know on pre-ray seems really
unlikely for you to put for it to sort
of be the sort of correct thing to do to
allocate all of your you know gains to
one of those two classes and really sort
of not put them in the other right uh
you know you it just you know if you
have two possible things that you can
improve about the model and your each
individual point trying to take the
maximum marginal performance Improvement
um and there's diminishing marginal
returns doing both of those two things
then you're going to be spreading out
your abilities you're sort of you know
uh between the two you're going to be
doing them both simultaneously uh or
alternating between them rather than
sort of doing one and then the other
um okay but that doesn't fully answer
the question because we still also have
to understand okay given that gradient
descent is probably going to be doing
both of these things simultaneously
um you know why would it first get to
the point where you know it understands
the training process before it sort of
has a perfect proxy
um and for that it really depends on the
complexity of the goal they're trying to
get to do right so if we're trying to
get to accomplish something that's
really simple then it might be that
having a perfect proxy is really easy
right it actually doesn't take very many
steps to get to a perfect proxy we can
just get there and we get the internally
aligned model
um but if the thing we're trying to get
to do is really complicated if we're
trying to get to do something that
requires you know a ton of complexity
something like um you know human do what
humans want to do you know it requires
you know all of this you know pursue
human values whatever you know something
really really complex
um that has all this sort of inherent
complexity then it's going to be really
difficult for the model to sort of
memorize all of that information
directly
um and in the same sense that we were
sort of talking about the end of the
last lecture where it's just
substantially easier for once the model
you know given that these two processes
are happening simultaneously and you are
developing a bunch of information about
how to understand the world that
information is sitting there and so it
would make more sense and you know for
the model to make use of that existing
circuitry right in the same sense in the
high path dependence case where we're
talking about you know really caring
about these sort of which things are you
know the biggest marginal performance
Improvement given that we've already
started with this other circuitry if
we're starting from a case where it's
going to be improving its model of the
world alongside it then we should expect
that it's going to you know the thing
that's going to give us the most
marginal performance Improvement is to
make use of that existing circuitry that
understands the world and has a bunch of
knowledge about it
um rather than just sort of Reinventing
the wheel entirely and sort of just hard
coding the uh you know the thing that we
wanted to do separately
okay
um and so the basic case here is you
know understanding the world is
something that is just generally really
valuable in lots of cases it's something
that is going to have a lot of you know
sort of reasons to increase it in many
many different scenarios and getting a
better and better proxy is something
that is maybe you know more difficult
it's something that
um you know has more potentially more
diminishing multiple returns is
something that maybe requires more
complexity is something that can be done
maybe more easily you know given that
you already have a bunch of
understanding of the world in terms of
that understanding of the world rather
than sort of trying to implement it
separately
um and so we're sort of worried that the
sequencing might fail that um we might
get the opposite sequencing that instead
you know understanding the world you
know and understanding the world
sufficiently to understand the training
process
um could happen before we get these sort
of perfect proxy
okay question
what do you think will happen if the
proxy is sort of good enough for like
not perfect but like decently close to
what we want and the the model will
start learns about the training process
at that point do you think I'll like
keep caring about the proxy goal was
like oh I really want to do this thing
that is not exactly what we want but
like very close to what we want or do
you think it will like stop caring about
that at all
yes I think that
this turn here is that the proxy could
get really really close
as long as the proxy isn't
either thing that we want then there's
still some pressure to be deceptive
right so as long as the proxy hasn't
matched up exactly on the desired thing
then
um there's some gains for that model
from pretending to do the desired thing
and then waiting and then eventually
doing the slight difference right
um and that's sort of dangerous because
of that slight difference is something
that we really care about that could be
very problematic
and so
um this is sort of one of the one right
one of one of the other things that's
happening here right is that this sort
of perfect proxy standard can be can be
very rigorous right it depends obviously
on how hard it is you know the objective
that we're actually trying to get to
accomplish but if we're trying to get to
do something very very complex having a
perfect proxy can be is a very very high
standard and um you know given that both
of these sort of two things improving
the world model and improving the proxy
are happening simultaneously that really
high standard might be something that's
really really hard to meet before you
have uh you know understood the training
process efficiently
Yeah question
how sure are we about that part that a
little difference in the objective
function causes huge tragedy after him
deployed like have confidential in this
yeah I think it really depends on the
difference right so some small
differences we might be totally fine
with and some small differences might be
catastrophic right so it really depends
on is well what is I mean what is the
metric I mean the metric is just is that
difference about a thing that we really
care about right so if it has a really
small difference in you know what it's
doing that is directly around you know
something that is extremely important to
to us uh then we're going to be really
sad and if that small difference is
something that is totally relevant to
you know anything that we you know
humans care about then then we don't
care right so you know is a small
difference dangerous well maybe you know
it's not like in general dangerous or in
general not dangerous it depends on what
the difference is right
okay
okay great so course of alignment is you
know the second path this is the Martin
Luther path and we want to understand
you know How likely is the corrigible
path
so again we're going to start with this
proxy line model right we're going to
start with this case where uh you know
it has some proxy the proxies aren't
really great
um but now we're going to imagine that
it's we're sort of going to accept the
argument we were talking about
previously about you know okay you know
it seems really you know weird in some
cases to sort of get this perfect proxy
before you've understood the training
process so what if instead we imagine
these two things are happening jointly
right it is jointly improving the
model's understanding of the world and
uh improving its proxy
and then we're going to imagine well at
some point the model is going to learn
from you know its input data from
understanding the world
um you know in the process of
understanding the world the model is
eventually going to learn a bunch of
facts about the training process and
here we're going to imagine that happens
before the model's proxy sort of becomes
perfect
the opposite sequencing from the last
time
and then uh given that that sort of
opposite sequencing happens
uh then what we're going to imagine
happens is that the proxy gets you know
replaced with a sort of pointer to uh
what it is that we're trying to get the
model to do in the model's understanding
of the world right the model now
understands you know has a bunch of
facts and information about what the you
know what the humans or what the
training process is trying to get us to
do and so grain descent can just get rid
of the existing bad proxy and replace it
with this much better proxy that is you
know you have this understanding of the
in the in your understanding of the
world somewhere so just do that thing
right you know do that thing that you
already understand uh you know and that
could be substantially simpler and
substantially better right because if
we're in this situation where uh you
know your sort of world model uh you
know has a better understanding of what
it is that you're trying to get the
models to do then the proxy does then
there's going to be performance gains
from ripping out that proxy and
replacing it with the thing that is sort
of pointing to the understanding of the
world we can sort of think about this as
if the sequencing happens in this
direction opposite of the previous one
there's you know a sort of performance
overhang where um so long as the model
still has its sort of bad imperfect
proxy there are performance gains to be
had from replacing that proxy with the
model's understanding of what of what we
wanted to do that exists in the world
model right the model sort of in some
sense in this case knows what it is that
we wanted to do right it knows a bunch
of facts about the training process
knows a bunch of facts about you know
what these that we're trying to get to
do but those facts sort of haven't you
know connected to the model's actual
desire to act because it just has some
some proxies that are still there that
you know it's that's using to determine
what it does and so there's that sort of
creates this overhang where you know it
theoretically should be able to do
better than it actually is because it
does have the knowledge of what it is
that we want to do but it's not making
use of it effectively and so what sort
of ripping out the proxy and replacing
it just with like a pointer to its
understanding of the world what we what
you know uh understanding of what we
wanted to do
um it sort of resolves that overhang and
sort of gets into a position where now
the model is actually making use of that
knowledge that it has effectively and so
that's sort of a substantial performance
of event because it solves all of these
examples where you know the model
actually did understand what we really
wanted to do but it didn't actually
correctly act on that in training
okay and then again once we reach this
point we're at a similar sort of
equilibrium previously you know there's
no additional reason to change the model
in any direction because it sort of
fully understands what we want and just
you know doing the right thing in
training in every case
okay
[Music]
um great
so this is sort of another path
um and so now we need to ask you know
again you know How likely is this path
and I am again have a concern
so here's my concern here so
um we talked previously right in you
know the first you know in the internal
alignment case about the difficulty of
getting a proxy right there was perfect
right difficult to keep getting a proxy
that really directly captured everything
that we cared about well the same sorts
of difficulties also arise in getting a
good pointer so you can think about it
like this right so you know Martin
Luther right needs to figure out you
know what it is that God wants to do you
know and so he's going to do that by
like reading you know a Bible but you
know which Bible right like how should
he read it how should he interpret it
right you know these are all really
really tricky difficult questions
um that you know many different sorts of
Martin Luther's you know different
people that have tried to interpret the
Bible you know have disagreed on and so
um if you want to make sure that in
every individual training example this
would of course the line model actually
gets it right every time
um it's not enough to just sort of point
to that understanding of the world right
you can have a perfect understanding of
exactly how you know what the Bible says
and what's going on in the world but not
have a perfect you know not be able to
understand which of the pieces and which
of the things in the world are the ones
that we actually care about right
um you know you have to be able to you
know understand if you're trying to like
follow human instructions that it's not
you know follow the instructions of you
know whoever is typing at the keyboard
but it's like following instructions of
the human right you know there's all of
these sorts of you know tricky
difficulties in actually being able to
understand okay of all of these facts
that exist in the world which are the
ones that actually correspond to the
thing that you know you're trying to get
me to do right and that isn't a fact
about the world as much as it's just the
thing that you you know there's a fact
about what we're trying to get to do
right it's just a basic structural thing
right can you understand all of the
facts about the world but it's not clear
which of those facts necessarily is the
one that we wanted to be caring about
right and that takes up some additional
sort of you know tuning to find the
correct way of interpreting which thing
in the world it is that we want the
model to be doing and so
in the same way that sort of you know
getting a better and better proxy is the
sort of long and arduous path where you
know there's sort of diminishing
marginal Improvement to getting a better
and better proxy at each you know step
along the way
um there's also diminishing marginal
Improvement to getting a better and
better pointer right so you know
um eventually yes if it has you know a
perfect pointer and understand exactly
what we're trying to get to do it's
gonna have perfect performance in the
same way that eventually if it has a
perfect proxy and understand exactly
what we're trying to get to do but but
getting to either of those situations
still requires this sort of long and
arduous path of incremental improvements
where you know the pointer gets a little
bit better you know the proxy gets a
little bit better and eventually you
know they sort of match up with what
we're trying to get them to do but that
long and arduous path can be one that is
you know not necessarily super favored
by grain descent it's one that you know
in this High path dependence view uh you
know can take a bunch of steps and can
be a bunch of steps that have you know
smaller and smaller performance
improvements as we're going along
[Music]
um and so in that case you know we're
sort of concerned here um an important
thing right to point out though is that
we're really relying on this fact that
the thing we're trying to get to do is
really complex
um and difficult to learn and it's
difficult to point to
um but that's going to be true in many
cases where we're trying to get our
models to do really complex difficult
complicated important tasks
um and so when we're in that case you
know we're sort of dealing with this
problem that
um you know well there's just going to
be a bunch of things that we have to get
the model to you know get exactly
correct if it's going to be able to sort
of do the exactly the thing we want
um and that's sort of still the case in
in either the corrigible or the sort of
internally aligned uh case
okay
[Music]
Okay so
uh we have path number three which is
deceptive alignment so uh How likely is
acceptable alignment path so the
deceptive land half starts out very
similarly to the courageable path we're
going to imagine that we start with a
proxy line model and then we're going to
imagine that similarly to the corish way
line case we have the sequencing in The
Cordial Direction where uh first we
learn the you know understanding of the
training process before we have learned
the sort of correct perfect proxy
okay
um and then once that's happened we sort
of again have this overhang right this
sort of you know overhang that we were
talking about in The Cordial case where
there was sort of performance
Improvement lying on the ground where
the model theoretically knows in terms
of its knowledge about the world a bunch
more facts about the training process a
bunch more facts about what we're trying
to get to do then it's actually making
use of in terms of it you know how it's
actually you know taking its actions
because it's just operating on some
proxy and there's you know there's
something has to happen right great
doesn't have it's going to do something
to resolve that overhang and the courage
line case the way that overhang was
resolved was it you know ripped out the
proxy and replaced it with this pointer
to understanding you know these facts
about the world but in an assembly line
case there's something different the
grading descent can do that resolves the
overhang in you know that also resolves
the O-ring so what can gradient descent
do well it can take the existing proxy
whatever it is and make it into some
sort of long-term goal right so you know
whatever the existing proxy was that you
know it had
um if that proxy becomes a long-term
goal then given that the model has an
understanding of the training process
and understands how it works and what
we're trying to get to do if it you know
has some long-term goal that is
different than the goal that we're
actually trying to get to do then it has
an incentive to be deceptive right now
suddenly it has a reason to try to play
along so that eventually it can you know
do its long-term goal that it really
wants
okay and so in the deceptive case
there's an alternative modification
available to gradient descent the green
descent can you know can change the
model in a different way than the
commercial case that also resolves this
overhang right in both cases you know in
the deceptive line case it had more
information about the world that it
could be using to have to take good
actions that it wasn't right because you
know it it sort of had this bad proxy
and the corresponding line case what
happens is it makes use of that
information via pointer in the
deceptively aligned case what happens is
it makes use of that information via
instrumental reasoning right it just
reasons it's like well to be able to get
you know whatever this you know
long-term goal is the best thing for me
to do is to do what this training
process wants and so I'm going to you
know do exactly what the training
process desires
and a really really important Point here
is that this instrumental reasoning
doesn't go through the same problem that
we were talking about earlier about this
sort of difficult you know uh path uh
you know incremental path because
um the deceptive line model doesn't
actually need to know the you know all
of these sorts of same facts that the
course relying model does about you know
exactly how you know what thing to point
to
um because
um the deceptive line model just figures
it out based on reasoning through well
I'm gonna just do whatever thing in fact
results in me doing the best you know
eventually being able to survive this
training process right and doing that
reasoning lets it you know figure out
what are the things that it should be
doing right because the actual things we
wanted to get to do in training are you
know doing those things are going to be
the things that most let it survive
training and do whatever it wants later
and so you know we could imagine a case
where like you have a chords with a line
model and the the pointer that the
coordinary line model has is to whenever
things let you survive the training
process So eventually you can do some
evil thing
um and that would work right but that's
the deceptive line model right that is
what the deceptive line model is doing
it's sort of like a cordial line model
where it has some pointer that describes
you know exactly what it is that you
know in the world model that it should
be paying attention to but the way that
pointer is implemented in the
deceptively line model is by doing this
instrumental reasoning by figuring out
okay look at your understanding of the
world and based on your understanding of
the world figure out which things you
should pay attention to such that if you
paid attention to those things you would
be the most likely to be able to you
know get some long-term goal eventually
in the real world
um and that's a really efficient and
effective way to influence a pointer
right and so in the same way we're
talking about the quarterly line case
where there's this really difficult long
path to be able to get a sort of pointer
that sort of you know actually is able
to you know accomplish the correct thing
they deceptively align you know case it
can get there very quickly right all it
has to do is you know get some long-term
goal you know have some long-term goal
that you eventually care about and then
all of this sort of performance overhang
is immediately resolved because you know
to the extent that the model had already
the information about what it was that
the training process actually wanted to
do as soon as it has the objective of
okay playing along in the training
process it's immediately going to be
able to capture all of that performance
um uh yeah question
true it's well for me yeah but basically
there doesn't seem to be much of a
difference in complexity I'm confused
why you know to say this is more more
efficient like an easier pointer to
learn in those cases you need to light a
Defender pointer like
do the thing that the training process
wants you to do uh in the deceptive case
there is an additional reasoning step of
for instrumental reasons so it just
seems like the deceptive case is harder
to get to
yes this is a really good question so
let's try to break this down so I think
that one thing that is sort of really
tricky here is distinguishing between
two sort of Notions of complexity so we
talked uh so yeah so one notion of
complexity is you know what things have
to be in the algorithm right what things
sort of have to be encoded in the
weights what are the sort of structural
properties of the algorithm that need to
be specified for the model to be able to
do a good job and then there's another
thing which is what things does the
model have to figure out right what
things does the model have to figure out
every time you run it right when you run
the model and when it does a bunch of
thinking what are the things that has to
you know discover in that thinking
and here we're imagining well it doesn't
matter that much you know uh what
happens in terms of the you know in some
sense all of these models eventually are
going to have to discover the same
things in thinking right because they
all fully understand what we want right
and so to be able to fully understand
what we want when you run them they
eventually have to get to the point
where they they have the exact same
understanding of exactly what it is that
we want what's different is the way that
they do it right what's different is the
you know what has to be encoded in the
actual algorithm that results in them
having an understanding of what it is
that we're trying to get them to do
right and so the difference here is not
that they they don't have the same
understanding eventually you know every
time you run them they're going to have
the same understanding because they
didn't have the same understanding they
wouldn't be able to do a good job but
they get there in different ways right
the internally line model gets that that
understanding by having it 100 fully
hardcoded it just loads it straight up
from the weights right the core is
really aligned model it gets it by uh
having some pointer some function that
that said you know takes in the world
and how outputs you know takes in its
understanding of the world a bunch of
facts about the world and outputs what
is the thing that I should be you know
trying to do in this case the
deceptively unemployment like The
Cordial online model also has this like
function right but its function is
different right in the course of the
line model we're sort of you know
requiring that function to be a function
that actually is correct right it is the
correct function it's the one that
actually does the right thing that
actually takes in understanding of the
world and outputs the correct thing that
we humans would actually want to do in
that case but the deceptively line case
doesn't have to implement the correct
function it can Implement a incorrect
function it could implement the function
that is just you know do simple
instrumental reasoning right start from
the premise you know I want to
accomplish some long-term goal and then
figure out which of these facts you know
based on that premise which of these
facts will be most you know relevant to
pay attention to to accomplish that goal
and so
um the deceptive to the line model can
have a substantially simpler function
there that does that does that uh
transformation you know from facts about
the world to you know each individual
runtime what is the thing that I should
actually be paying attention to and
trying to accomplish in this individual
instance based on those facts
um and so that's that's the difference
right is that the way that they actually
get to you know what is the thing
they're trying to get to do is different
and deceptively aligned case it doesn't
have to you know for you know let's
think about right you know in the course
with line Keys we were talking about all
these various facts about like you know
should I pay attention to the human in
the chair should I pay attention to you
know the human you know whoever's typing
at the computer or like the actual human
those facts have to be hard-coded
somewhere right the the correct function
has to just directly load up I'm
supposed to look for you know the actual
human I'm not supposed to just look for
you know whoever's on the computer
because that's the actual correct
implementation of the function uh but
the deceptive line model doesn't have to
hard code those facts right all the
deceptive line model has to do is have a
good general purpose reasoning system
that says based on these facts what is
the actual best thing for me to do to
get the most you know uh you know do the
best on my long-term objective and so
just does that computation
um and you know that's how you influence
that function and so because of that it
doesn't have to have all of these sort
of difficult you know uh incremental you
know things that has to learn it just
you know runs this basic reasoning
process and it should already have the
reasoning process right like it doesn't
need to influence some new reasoning
process right we're imagining it's
already a proxy line model it already
has good reasoning processes that are
sort of generically there in the model
and so it can just make use of the
existing reasoning processes and just
use them for the goal of you know trying
to
um
you know accomplish them one triple yeah
questions
okay I think this last sentence is the
one that's not your for me so that's to
prove that this AG sort of AJ is
something that we develop as a scientist
AI
um their the lab that develops it its
goal is to
had then develop better cancer cures and
chips and whatever I can imagine it
beating it already has some proxy girl
which is sort of what we want it's
already pretty good at in job because it
really understands biology really well
but I imagine that it's plausible that
from its Bird model the kind of steps
that are needed to follow or goal and
deceive people that by following the
goal these kind of things are not yet
built in so building this up seems to be
approximately as hard as modifying its
pointer to the weather pointer
I think that's a really good point I
think that and in fact uh probably I
don't know if we'll we'll get to it but
we might potentially much later talk
about you know you know proposals that
sort of try to leverage that intuition
um I think that's basically correct that
you know in fact if you are training an
environment where being able to do this
sort of instrumental reasoning is just
not helpful for performance then um this
this path looks way less good because
now gradient descent has to actually do
the work of hard coding in all of the
You Know instrumental reasoning
capabilities
um rather than just being able to make
use of the existing circuitry right and
in that case yeah I think I agree that
this path at least in the high path
dependence case
um looks worse it doesn't it doesn't it
doesn't seem like it outperforms
um I'm mostly imagining a situation at
least here for our purposes right now
where you're trying to get to do
something very complex and very
difficult you know some you know very
general tasks we wanted to do you know
all the things that you know humans want
or whatever because we're sort of just
imagining this like default case where
we're just like you know throw machine
learning and all of the you know
problems that we wanted to be able to
solve
um I agree that there may be things that
we could do to try to escape that and I
think that is you know one of the things
that we could do is try to just you know
focus on situations where that sort of
instrumental reasoning capability is not
relevant um I think the issue with that
sort of approach though is that well
people do want AI is to be able to do
tasks or instrumental reasoning is
important right and So eventually you
have to be able to confront that fact
right we have to as a society deal with
the fact that there are people who want
AIS to do things that require
instrumental reasoning and we don't want
them doing that to uh you know be
catastrophic
okay and what do you think half
specific this instrumental reasoning
should be so if we let's say that we uh
they are developing um an AI whose goal
is to build a good Factory if it
involves lots of instruments it needs to
have some sorts of instrumental
reasoning ethnic because building a
factory has lots of instrumental Subs
first you need to buy concrete and
whatever but still I can imagine that
the part that you are negotiating with
somebody who you need to deceive these
these circuitry would not exist at the
beginning at all and again it would need
to build it from the ground up and maybe
it's not working
yeah I think it's very unclear there's
definitely going to be some gray areas
where like you might have some
instrumental circuitry but not maybe all
the instrumental circuitry it needs and
in that case you know it's going to be
somewhere in the middle right it's gonna
be like maybe this path looks okay but
it looks like less good than the case
where it already had all the circuitry
so
um yeah there's going to be a bunch of
gray areas where you know if it has like
no instrumental circuitry you know by
default it doesn't need to solve the
task at all then this path looks the
worst and if it's solving a task where
the instrumental circuitry is just
always necessary to solve the task
regardless then this path's gonna look
the best
Yeah question
um just just to be clear
um
is is the is the effectiveness
difference comes does this come from the
fact that for a corrigible model it
needs to update
the pointer like a very complicated
pointer through SGD while the deceptive
real-line model needs to only update the
Like Instrumental gold for SGD and
you're saying that the deceptively
Allied model would do further reasoning
on top of that
um in it or pass is that where the the
effectiveness difference comes true
I think that's basically right yeah
we're sort of distinguishing between you
know what things does it learn via like
hard-coded in the weights and what
things does it you know figure out that
like end up appearing right in the
activations right what things does it
figure out at inference time
um it doesn't necessarily have to be
like oh every single time it does the
reasoning if you have a model that can
like cache beliefs and like you know
store information maybe it has a
retrieval database or something then
it's not necessary that like every
single time you run it has to rerun the
computation but yeah we're imagining
that essentially the information there
about like okay this is the thing that I
should be doing in this case was mostly
generated by the model rather than
grading design right green descent just
produced the you know long-term goal
Yeah question
it seems like both of these pads involve
like model learning about its proxies or
its goals at the same time it's
acquiring its knowledge so but like I
feel like in current ml systems or in
current systems like at Chachi PT or
gpt3 with then plus rohf like gpd3 is
just like acquiring the model
understanding of the world and that our
lhf then like gets or points to like a
specific gets a model to point to a
specific goal that's already learned so
in this case which would be selected the
instrumental gold or the actual goal
because in the case where case we're
sort of trying to put the goal into the
end and you start with pure Knowledge
Learning you're not going to get this
instrumental path where it learns the
goal first right now whether you get the
cordially aligned path the deceptive
line path or something else there you
know it's not these this these you know
three paths are not exhaustive right um
and again we'll talk later about you
know other sorts of things that you
might get when you're trying to train
you know this sort of predictive um you
know like a like language model
um but at the very least yeah I think
that the the thing you're saying is at
least some evidence that the internally
aligned path specifically is really
unlikely in the case of um you know
doing this sort of fine tuning regime
okay uh yeah more question uh I'm sorry
I'm not quite sure why the eternally
like that wouldn't be unlikely in this
case because I feel like that would have
at least the same probability as the
deceptive or the a courageable One path
in this case because like you just need
to choose any goal right and if like
instrumental goals and core and Court
you know the terminal goal is like
equally likely to play selected in this
case since them all as just as a
knowledge why wouldn't the uh adrenal
rewind copy equally Lively
yeah so the problem is that the internal
line path sort of hard codes the goal
right rather than you know having it be
coded in terms of the model's existing
knowledge but if the model starts with a
ton of existing knowledge including a
bunch of knowledge about exactly what
we're trying to get to do then it's
going to be a way shorter path and
substantially you know better to just
make use of that existing knowledge
somehow right because it's already there
right and so it you know it's not going
to want to re-implement it in you know
some hard-coded proxy and so in that
sense you know it's going to be
substantially more favorite when you're
starting with that knowledge already
existing to go down either The Cordial
or the deceptive path
Okay so
um
uh okay great yeah so we've been talking
about like How likely is acceptable line
path the basic case uh is that you know
turning your existing proxy into a
long-term goal is a relatively simple
modification and importantly it's a
modification that doesn't have this like
really long path it's a relatively
straightforward modification once you do
it once you sort of go to a long-term
goal you capture all of the gains from
that you know overhang right it doesn't
have to like keep improving the
long-term goal to be more and more
long-term right once you get the
long-term goal then as soon as you have
a longer goal it's going to do all the
instrumental reasoning necessary to
figure out you know exactly what the
thing is that it should be doing to get
you know the most performance in
training and so it doesn't have this
long and arduous path in the same way
that we've been talking about earlier it
has this single you know thing that it
has to do and once that thing happens it
gets all of the sort of performance from
the overhang and so because of that you
know it is sort of a path that we might
expect to be favored in this sort of
high path dependence case now there's a
bunch of assumptions right you know so
one important thing right is that we're
you know we're doing something that's
you know um really complex goal that you
know like we were talking about also
we're doing something where there
already exists go to instrumental
reasoning circuitry but if we're in that
setting then this pass seems um very
favorable
okay so that's the high pass dependence
case
um now we're going to shift gears and
we're going to talk about the low path
dependence case so we want to redo the
whole analysis that we just did but
under a different view of the inductive
biases right a different way of thinking
about you know what is what it is that
determines which algorithm we get
okay so again we're going to be assuming
the model has to fully understand what
we want in terms of what model classes
we're looking at uh we're going to be
imagining
um uh yeah but then to be able to
distinguish between these model classes
we have to look at you know what are the
things that are going to matter uh in
terms of you know properties of an
algorithm that you might look that you
know the low path dependence sort of you
know inductive biases might be looking
for you know what does it mean for an
algorithm to be structurally simple
and in particular we're going to isolate
two things and I sort of mentioned
alluded to these earlier where thing
number one is Simplicity and and number
two is speed so
um what is the difference between these
two things and what do they really mean
so it's a little bit tricky what they
mean so I'm going to try to unpack them
so Simplicity is how complex is it to
specify the algorithm in the weights
right if I were to write down you know a
Python program that ran exactly the
algorithm that might you know uh was
being implemented my my model how can
how many lines would that program take
right um we can think about this also
right you know if we think about you
know how complex is the algorithm of
right if we're thinking about something
like the wheels uh you know windows on
top of Wheels detector how complex is
that basic structure right you know
first you have to have a window detector
you know you need some window detection
function and then some wheeled section
function and then you need to write the
you know combine these two function
right and so we're sort of trying to
understand how complex is that circuitry
right how complex you know is the
description of the circuitry that the
model has to implement right
um
the thing that this doesn't capture
right so the thing is explicitly not
capturing is the complexity of actually
running that circuitry right it's only
only the sort of Simplicity captures the
complexity of describing the circuitry
and then speed captures the difficulty
of running that circuitry now an
important thing to point out is that in
many realistic architectures it is in
fact the case that they actually all
take the same finite amount of time to
complete and so we need to be really
clear here that what we're talking about
is not the literal algorithm that is
implemented by the you know uh weight
matrices by your actual you know uh
neural network we're talking about is
the sort of the structurally you know
simple algorithm that is sort of behind
it right you know what is the algorithm
of you know the core thing that it's
actually doing is like look from Windows
look for reals combine them right and we
want to understand for that core thing
that it's doing
um you know
how you know match computation does that
you know algorithm take and how
difficult is it to describe so what some
ways of thinking about why these two
things matter right so why does it
matter you know how complex is it to
describe the algorithm well it matters
how complex it is to describe because
that matters uh that affects things like
you know how large is the Basin going to
be because the more complex it is to
describe
um you know the the fewer the the more
parameters it's going to take to be able
to describe it and so you know it's not
going to be the case that there's going
to be a bunch of different possible
parameterizations which all correspond
to the same function but if it's very
simple to describe then there may be
many parameterizations which all
correspond to the effectively same
simple structural function right so
Simplicity matters and speed also
matters so why does speed matter well
it's a little bit trickier so one way to
think about this is if we're thinking
about the space of all possible
algorithms that you know one could have
in theory well just to start with only
some of those algorithms are actually
implementable in a neural network right
because it actually is a finite function
and you know it can only Implement so
much computation and so we should expect
that things which algorithms which
effectively take less computation will
also be you know more likely for us to
find given that we don't know exactly
which ones are going to be included in
the space of you know which you know at
any given point you know is our Network
large enough to be able to implement it
um but if it's you know if it's
influencable you know via a smaller
Network then you know it might be more
likely that we'll find it earlier
um similarly if the model if you know if
it's an algorithm takes a bunch of
computation
um and there's another model that
accomplishes the exact same thing but
takes less computation then that means
there's sort of extra you know
computation there's sort of extra
computation available to sort of do
other things like you know spend extra
you know computation you know slightly
improving its performance in various
ways and so
um you know both of these two things
matter to some extent in terms of
understanding you know How likely is a
particular algorithm now they're
definitely not the only things that
matter even in a low path to penance
case where we're imagining that the only
thing that matters are these sorts of
global structural properties of the
algorithm even in that case there's
almost certainly other Global structural
properties that matter too
um but these are at least two that do
seem like they will play a role and are
ones that we can sort of analyze and so
that's we'll be imagining that these are
the main two that we're sort of going to
be looking at uh yeah question before we
go
Simplicity as well like let's say I had
a 10 line neural network and I have two
algorithms one requires 10 successive
computations so the only way to do it is
to be in all 10 networks in all 10
layers and the other has two layers
so that means there's nine different
ways it can actually be instantiated it
can go from layers one to two all the
way through to layers nine to ten so put
it faster algorithms also be simpler to
specifying someone
uh yeah I think that that is a really
interesting point I think that
um it seems at least plausible I don't I
don't know uh you know in fact how it
works out uh in terms of you know how
these things interact I definitely agree
that like you know there's certainly an
interact to some extent and there are
various models of trying to understand
how they interact I think one model of
sort of trying to understand how these
things interacted is I I think is sort
of reasonable is like a circuit prior
model where you sort of try to
understand you know if we think about
you know algorithms as being described
by the Boolean circuits that are
necessary to implement them then you
know we can think about the inductor
biases as selecting the algorithm which
takes the fewest number of Boolean
circuits and that in some sense sort of
is capturing the thing you're saying
where it's sort of a mix between speed
and simplicity where the sort of faster
models also take fewer circuits to to
implement
um
a lot of those sorts of priors that were
harder to understand they're harder to
sort of uh figure out you know would the
deceptively aligned or the non-deceptive
aligned actually do better
um and so we're going to imagine that uh
we're sort of gonna be thinking about a
case where
um you know we're just gonna be looking
at those two specific things uh question
yeah I didn't want to ask a question
just make comment on the previous
question uh an example of like the case
where there would be difference between
Simplicity speed is imagine if uh the
in every layer you only need like
one neuron for this algorithm but you do
need non-linearity so you do need the
value so you do need like Tech and
secular layers but you need only one
neuron in each so this algebra would be
simple to implements very few ways to
need it but it will be like long because
so many steps are needed
yeah it's a good example yeah
yeah they're definitely going to matter
to some extent uh we don't know exactly
like what the mixes of them you know and
how things play out in terms of you know
which one of these is is most important
but we're just going to look at both and
we're going to try and understand under
each one of these regimes how well uh
you know do these different model
classes perform
okay so we're gonna start with
Simplicity uh and I'm going to start
with a really simple argument for you
know trying to start getting our hands
on Simplicity
um I think one way to sort of think
about Simplicity is just how many
algorithms are there you know how many
possible ways are there of implementing
a particular algorithm right so we've
you know we're talking about this as
this relationship between Simplicity and
base and size right where the more ways
there are in influencing the same
algorithm the larger the Basin is
um and so we can sort of understand well
okay effectively how many different
sorts of algorithms are there which fall
into the each one of these model classes
and in the same way that you know the
sort of number of possible weights that
implement the same algorithm affects the
Simplicity of that algorithm the number
of algorithms which Implement which you
know fall into the same model class
affects the overall Simplicity of that
model class and so we're going to start
with a counting argument right how many
algorithms are there that fall into each
one of these classes uh of algorithms
uh so how many uh you know Jesus Christ
are there right how many you know
internally line models are there uh or
at least how many algorithms are there
you know effectively different
algorithms influence the Jesus Christ
model I think the answer here is
basically one right there is only one
way to sort of hard code exactly what it
is uh that we want the model of doing
right any sort of difference in that uh
you know would would you know be a
different model right we would no longer
be the Jesus Christ it wouldn't be doing
the correct thing
uh okay but how many you know Martin
Luther's are there well uh there's a
bunch of different ways of looking at
the Bible that are all the same right
you can have a bunch of different ways
of you know thinking about understanding
exactly what it is they're trying to get
to do that all Converge on the same
thing now we talked previously about you
know that can be difficult right it can
be difficult to get the exact correct
pointer but that doesn't there's still
multiple of them right you know there's
still like you know
understanding exactly how to read the
Bible plus any one individual Bible that
exists in the world will all yield the
same results as long as they have the
same contents and so it's still the case
that there are multiple different
Luthers there's still multiple different
ways of pointing to the to the thing
that we watch
and similarly uh you know if we think
about pascals well there's even more
sort of pascals because any long-term
goal that you could possibly have would
correspond to a pascal model because any
uh you know any long-term goal will
result in the thing mean deceptive as
long as it has something that it wants
to accomplish eventually in the world
then it's going to want to you know play
along and be deceptive uh for the
purpose of doing so on so you know just
based on sort of initial analysis it
seems like there's the the largest
number of models occupy the Pascal uh
you know model class the deceptive of
the line model class
okay uh we can do it we can sort of dig
in a little bit more if we sort of try
to think about um you know what's going
on here right that's causing this sort
of Pascal's uh you know to be uh sort of
more numerous well um we can think about
sort of the Simplicity and the
complexity of an individual model as
being described by the complexity of
sort of three pieces right we have the
world model it's understanding of the
world we have like its optimization
procedure you know its ability to do
instrumental reasoning to do all these
sorts of tasks and then we have the mace
objective right whatever it is that is
trying to use that reasoning to
accomplish
um you know we talked previously about
like you know cases where you know the
optimization procedure or the world
model can be different and that can
really affect which thing you end up
with in this case you know if we're
imagining a case where we're really
trying to get to do some really complex
task that requires you know general
purpose instrumental reasoning and it
requires it to basically know you know
all the various facts about the world
then all three of these model classes
should have in that specific case
essentially identical World models and
optimization procedures they should have
to be able to do the same reasoning they
should have to be able to have the same
knowledge and so then in that case the
only thing that will differ between them
is the complexity of the remaining thing
which is what they're trying to get what
they're trying to do and in that case
you know the complexity of what they're
trying to get to do well
um
if we think about that we can sort of
approximate it as well what is the
simplest model uh the simplest objective
which falls into each one of these
classes
um is going to be you know what is the
simplest base objective right it falls
in each one of these classes we can
compare the complexity of the simplest
one as another sort of metric of trying
to understand how diff how simple are
these different classes right so our
first metric up above was just how many
different algorithms are there that fall
into that model class and that sort of
can help us get some understanding
complexity and then the second thing
we're going to look at here is you know
for each individual model what is the
simplest model in each class and how do
those simplest models compare and to do
that if we're imagining they have the
same world model and the same
optimization procedure then the simplest
model Lynch class is just a question of
what is the simplest Mesa objective in
each different class
and so in the internal aligned case well
the simplest main subjective is going to
be a full specification of whatever it
is we're trying to get to do and so well
the complexity of that is going to
depend on how hard it is how complex it
is the thing we're actually trying to
get to do but if we're trying to get to
do something really complex then this is
going to be really complex
um similarly with the quarterly line
case well what is the simplest
coordinary line May subjective well it's
whatever the simplest specification is
of how to point to the you know the the
desired objective right and like we were
talking about previously if the thing
we're trying to get to do is very
complex this can be very complex it
requires you know hard coding a bunch of
information about exactly which things
in the world are the ones that we want
the model to be paying attention to
um and then you know for this after the
line what is the simplest deceptive line
May subjective well it's whatever the
absolute simplest possible long-term
objective is right you know make paper
clips or whatever you know is this sort
of you know thing that we get here right
like anything that eventually you want
to accomplish in the world uh that is
long-term that you know you care about
the long-term consequences whatever the
simplest possible thing is to implant is
the you know simplest possible set for
the line objective now that doesn't mean
right an important you know thing to
point out here is it doesn't mean that
you would literally get the simplest
thing in each case right we're not
saying the simplest thing is actually
the most likely we're saying that
looking at what the simplest thing is in
each class can give us a sense of how
simple the overall class is in the same
way that looking at the number of things
in the class can give us a sense of the
overall Simplicity of that class right
and so we're saying well the fact that
the simplest thing in this class is
really simple it's just like the
absolute simplest thing that you could
care about anything in the long term
um you know is is indication this
overall model class is very simple in
the same way that the the fact that
there's so many different models that
fit into the class is also a similar
indication
to link it to the previous discussion of
the difference between effectiveness of
courageable inceptively lines uh sorry
the efficiency of the processes
um you're saying this is the simplest
thing to do or this is the only thing
that we have to do is point to the
long-term objective of the deceptive
case because
we do not need to encode to make a
pointer to the training objective in the
weights right because we we the model
thinks about that
um
could think about that in the
activations
that's exactly correct yes we're
imagining that the deceptive line model
sort of figures out the thing we're
trying to get to do in the activations
it figures it out you know uh on its own
rather than rather than sort of it being
hard-coded in the algorithm itself and
so because of that the only thing that
we need hard-coded in the May subjective
part is just some long-term objective
and then the sort of world model and
optimization procedure take care of
figuring out what it is that you know
would actually do a good job on that
objective
okay so this is the Simplicity I think
that
um editorializing a little bit I think
my uh my sense is that this is the the
strongest case for deceptive alignment
uh of any of the various different
things that we're looking at if you were
just looking at purely Simplicity
um then I think the keys for deceptive
alignment is very strong
um I think it's stronger than the high
path dependence case or in general in
the low path dependence case because as
we'll see some of the other the the
speed is is much less clear
um so let's look at speed so uh how long
you know how much computation did each
one of these algorithms take well uh I
think that the most basic thing the
first thing is that deceptively align
model seems to be taking more
computation right so why is that well
the deceptive align model has to do this
extra step of instrumental reasoning
right it has to go through this process
of okay starting from the fact that uh
you know I want to get some long-term
objective I then have to figure out
given that I want to do some long-term
objective what is the best way to
accomplish that and you know given that
I'm in some training process you know
what is the best thing to do for
accomplishing that training objective
and you know therefore I should do this
particular training objective right and
that is a chain of instrumental
reasoning which takes time and
computation to actually be able to run
right and figure out
um and so the deceptive line model is
doing extra steps of reasoning extra
computation that the the other models
are not doing and so because of that you
know the deceptive align model seems to
be the slowest in the sense that it is
sort of taking the most steps of
computation
okay and the course of the line model
um is sort of uh take does you know does
less computation uh I think that the
difference between the corrigible line
and the internal line model is a little
bit unclear here I think there is some
case to be made for why the quarters
rely model might be slower because it
doesn't get to just sort of like
directly load up exactly you know what
the desired thing is it has to still
follow the pointer in some sense and so
that takes some amounts of computation
um but it seems like the the difference
between the deceptively line model and
the others is at least is larger where
there's this re there's this like
substantial extra thing the deceptive
line model has to do which is like
reason about what a good you know what a
deceptive model should do you know you
know what should I do if I'm trying to
play along right that takes reasoning
and that extra reasoning adds to the
overall computational cost of this
algorithm and so the deceptive line
model sort of seems to be the worst on
the speed uh case Yeah question
so it seems like one thing that could
take time is like reasoning about how
you could be deceptive but it also seems
like just doing the task itself would
take time so how do you know that the
amount of time that's being taken by the
computations isn't just for the task
um couldn't like the deceptive model
mask the fact that it's like taking
extra time uh to the fact that like it's
taking extra time to do the task itself
this is a really interesting question so
it's totally the case that you know
they're gonna have to take a bunch of
time to actually do the task itself
we're trying to factor that out and
trying to understand okay what is the
time you would have to take in addition
to the thing you have to do to actually
solve the task and the thing that the
deceptive line model has to do in
addition is figure out what the task is
right like oh that it should be trying
to play along in this case now
um I agree that it could be the case
that it could like find a way to hide
that by like doing it in parallel or
something
um
you know another way that it can hide
this is via like a retrieval database of
some variety where it's able to just
like store and cache its previous
computation and then load it back
um those could definitely be cases where
this sort of thing would get trickier I
think one thing I'll point out about the
like doing in parallel is going back to
like why would we even expect that like
speed bias would matter at all well the
reason we'd expect it to matter at all
is that you know to the extent that it's
sort of using it it has like you know
some finite amount of computation
available to it and to the extent that
it's using some of that computation to
do other stuff that's not directly
necessary then
um
you know it sort of wasting some amount
of computation that it could be using to
like you know try to do other things
that are better you know for
accomplishing the task uh it you know
means that it has to be influence on
maybe a slightly larger model or you
know slightly different type of model
and so that can sort of push you push
you back but again it's not it's very
unclear right like you know it may be
the case that you have a really really
large model to theoretically has the
capacity to implement any of these
things and yet you still end up
implanting you know the simplest one or
the fastest one or whatever or similarly
you have a really small model uh you
know but the Simplicity bias is really
strong so it's a little unclear exactly
how this sort of ends up but um
you know at least if we're sort of
thinking about in this very abstract
sense where we're just looking at speed
complexity right it seems like on the
this sort of speed complexity case
um the deceptive line model loses how
that actually cashes out in terms of
like how these various like theoretical
factors influence the actual practical
inductive biases is a really difficult
thing to determine
okay uh all right so I think that this
is sort of nice right it seems really
good that speed bias sort of uh seems to
cut against deception
um but there's a bit of an issue here
which is that I think speed bias has a
bit of a problem which that it may be
sort of uncompetitive in the sense that
um if we try to build uh training
processes which directly incentivize
computationally fast algorithms we may
be losing a lot of what makes machine
learning good in the first place so we
talked sort of at the very beginning of
like you know well why how do we how do
we construct machine learning processes
right like why do we build these machine
learning models and how do we how do we
make them work well you know the thing
that we often will do is try to select
lost Landscapes you know select our you
know machine learning process to have a
you know to to bias towards very
structurally simple algorithms because
we believe you know something like
Occam's razor that's structurally simple
algorithms do a good job at fitting the
data and so if we are trying to modify
our training process to not do that to
not be biased towards structurally
simple algorithms we run into a problem
of well you know now we might not
actually be able to find algorithms just
do a good job at fitting it down
so uh you know going back to something
we talked about you know in the first
lecture uh that I think is sort of
relevant here is we can think about you
know let's say I in fact practically try
to do this
um I I try to select the model which has
the following property it is the you
know smallest model which does a good
job of fitting the data in some sense
that's sort of an approximation of what
a speed bias would be right it's saying
we want the smallest model which fits
the data well and so this is the double
descent curve from earlier so we can say
if we want the smallest model that fits
the data well then what we want to look
for is we want the case where the train
loss you know the training performance
reaches its optimal
um the first point that happens as we
vary model size right so as we slowly
increase the model size we want to stop
once we reach the smallest model that
has you know good performance and so we
can do that the Blue Hero corresponds
the blue and the green corresponds to
the green we could do that we can look
at what happens when we reach zero train
loss and then how well does that perform
on generalization tasks right and the
answer is it's the worst point in the
whole graph if we look at you know how
an individual model size performs on
actually generalizing to Downstream
tasks
the worst generalization performance
occurs precisely at the point where the
model first reaches optimal training
loss
um and I think one way that you can
interpret What's Happening Here is that
you know we talked about this sort of at
the beginning it's sort of this is the
point at which it's sort of forced to do
this sort of fast algorithm it's forced
to just like memorize things rather than
um actually sort of implement the you
know actually structurally simple thing
and so it actually ends up doing a poor
job right
um and so that's a concern because it
means that we might not actually be able
to get the sort of speed bias in
practice while still being able to have
machine Learning Systems which do a good
job on tasks right in some sense we're
sort of forced to have some amount of
speed bias because you know we don't
want to you know waste a bunch of
compute on things but if we sort of push
too hard in this direction then um you
know we lose the thing that we wanted to
get out of machine learning in the first
place
um now that might not be entirely true
so one fact here is that this speed bias
is looking at the size of the entire
model but instead of looking at the size
of the entire model we can instead look
at something like the depth right how
many sequential computations are
required that's a different type of
speed bias than the total amount of
computation required and if we look at
the depth then it's actually much less
clear
um it actually often does look like you
can you know substantially decrease
depth and still have good performance so
so it's not the jury is not entirely out
here it may still be the case that there
are other things that we can do to sort
of make speed bias competitive to be
able to you know create training
processes which incentivize you know
Finding algorithms which are you know
structurally use less computation
um but it's it's a little bit tricky if
for us to actually be able to find ways
to do that which actually give good
algorithms
yeah questions
how do you think this changes if we move
to more recurrent architectures or
Transformers with adapters to Future
I mean there were still hard coding in
the apartments for shorter big time
I think that's a really good question so
I think it does change things so in
particular it sort of introduces what I
often will call an average the bias as
opposed to sort of Max Speed bias so
instead of saying uh you know we want
the total amount of computation that the
algorithm uses on any input to be capped
at some value we're instead saying on
each individual input we want you to use
as little computation as necessary to
solve that particular task and I think
that that sort of average speed bias
actually has reason to believe it might
be more competitive and better than the
max speed
um and so I think there absolutely are
room for potentially making use of
something like that yeah so an example
of something that makes
something like Chain of Thought where
you sort of Select for you know the
model thinking for a smaller amount of
time and still getting the right answer
is sort of implementing something like
an average speed bias so that might be a
way to sort of you know get around this
yeah
more questions
this speed priority
is that something that a someone has
tried and B do we have the engineering
of capacity to actually try it as an
experiment as all like now do we know
how to actually increments people as
even if we think it's a good idea to try
uh well it really depends on what we
mean by speed right so we can look at
this graph here and can be like well
this graph is implementing a speed bias
right what we did was we just look at
you know the the first thing that gets
zero train loss as we vary the size of
the model and then we select the thing
right and this doesn't doesn't work in a
sense that it yields bad performance now
you could imagine doing something very
similar to this where instead of looking
at total model size you're looking at
one of these other sorts of things uh
and varying it and then finding the you
know the first thing that does a good
job you could also just add this as a as
a regularization term um there's lots of
ways that you could try and implement it
I think the tricky thing is knowing what
sort of speed bias to look for and
actually believing that that speed bias
would be sufficient right you know
you're still going to have the
Simplicity bias right you're still going
to have all these other biases so you
actually have to have some reason to
believe that the speed bias that you're
adding is actually going to do enough to
get you the deceptive that you know to
avoid the deceptive align model right
and that's a really tricky thing to
believe and to have any confidence in
yeah
but I was just speaking it would be
useful empirical experiment in order to
determine the competitiveness of speed
bias ignoring whether it would actually
work for the time bit which is much
harder to determine empirically
yeah so so like I said we have stuff
like this right here where you can just
check you know the scaling laws right of
how these sorts of things change
um but um yeah I mean checking that for
Lots in various different ways of
setting up the training process is
absolutely something I think is valuable
Yeah question this Affairs me why this
graph shows speed versus Simplicity
could you ever come about that
I would imagine that for Speed would
need to look at the number of layers
thought the total counts uh uh
parameters which this shows
yeah yeah good point so I think the the
key distinction here is the different
types of
so there's like total computation speed
and is like you know total amounts of
computations and then there's like Max
amounts of computations uh right like
depth versus total size
um those are both computation measures
right like the amount of total
computation that it does is a measure of
of speed in some sense
um as is the maximum amount of
computation on any individual path
they're just different metrics right you
can think about if you're thinking about
like you know parallel computation you
know you can either think of the total
amounts of computation done across all
of your parallel threads or you can
think about you know what is the time
that it takes for the last thread to
finish right and those are both metrics
of computation there are ways of
thinking about how much computation did
this algorithm use but they're different
metrics right um and so this is I agree
looking at one of those metrics it is
looking at the total computation metric
and it is not looking at the max
um you know the of any particular
parallel computation
um I think the thing I was saying
earlier I think the maximum any
particular parallel computation uh looks
better than this this is this is the one
that probably looks the worst
why uh unclear uh I think that you know
yes the question is why I guess
um the answer would be well uh it seems
like you know a very you know passing
answer I could say I think that you know
there's something sort of right about
because razor going on here where you
know in the real world actual real world
data and real world patterns are you
know more distributed according to
something like a average you know speed
bias or a you know Max parallel
computation bias rather than a total
computation bias why is that It's tricky
to understand you know it goes through a
bunch of facts about like you know how
does the world actually work and why is
it the case that you know particular you
know patterns exist in the world and
other patterns don't um it's you know
it's sort of a really thorny
philosophical question in some sense
about like you know what makes you know
you know certain hypotheses more likely
to fit real world data than other
hypotheses
um but you know it sort of has to in
some sense go through that
okay so uh sum it up uh you know what
we've talked about today so uh uh
overall I think think my takeaway is
that it seems like both low and high
path dependent scenarios uh you know
seem to be in a in a state where green
descent is going to want to push your
model to be deceptive now this relies on
a bunch of assumptions you know we're
sort of thinking about we're in a case
where we're training on some really
complex thing we're trying to get it to
do some really complex task
um and you know we're in a case where
you know we sort of have to push to
really you know large models uh to be
able to solve this problem
um but um it seems to me like there's
going to be incentives that are pushing
towards the deceptively aligned model
um in in both these cases
um or at least a a reasonable
probability in each case of the
deceptive line model being favored
um and that sort of seems to suggest
that we're going to have to create some
intervention right something needs to
happen I sort of mentioned this
previously right you know if this is the
default scenario right if machine
learning by default if we just sort of
run it on some complex task it gives us
something you know something like this
right there's there's biases pushing Us
in the different directions there's some
reason to believe the inductive biases
with yield deceptive line models some
reasonably if they wouldn't overall it
seems like the deceptive line model is
doing pretty well they have a lot going
for them on on you know in both sorts of
stories
and so we probably need to introduce
some intervention at least if we want to
be confident that our you know we're not
going to get deceptive line models we
need to introduce some intervention that
changes those inductive biases some
intervention that changes the way that
our trading process works to incentivize
it to you know not produce the deceptive
line you know model
um there could be lots of things like
this you know we talked about
transparency interpretability you know
some way of looking at the model and
rejecting it if it's not doing the thing
we want
um you know some way of you know
changing the biases you know more
towards speed and rather than Simplicity
there are lots of things that we might
be able to do you know maybe use trying
to train it on a really simple objective
um or you know in an environment where
it doesn't need to do lots of um
instrumental reasoning right there's
lots of things that we can imagine
changing about this default scenario
that might help but we have to do
something uh because it seems that by
default
um deception is is reasonably likely to
be favored as the sort of you know best
algorithm according to these inductive
biases in both these scenarios
potentially
okay so right so an important thing I
want to point out is you know we can
think about this as sort of the
alignment Gap right there's some
difference between the size in of the
Basin for the good models and the size
of the Basin for the deceptively aligned
models right and if we think by default
you know there's some chance the
receptive line models are going to
occupy a larger Basin they're going to
be more likely for us to find in
training then we need to create some
intervention that changes that right
that provides additional sort of bits of
optimization uh you know additional
pressure towards
um the you know the good models over the
deceptive line model so you know how how
much uh you know if you know how how how
much evidence we have to sort of
condition this prior on where the prior
is like you know over what sorts of
models we get by default until we end up
in a situation where we're going to get
the models that we want you know how
much uh additional sort of change do we
have to make to our training process
before it puts us in a situation where
we get the desired models
um and this Gap matters a lot right the
difference in size of these basins right
the difference in you know how likely
you are to get each one by default is
extremely important right if you're in a
setting where there's a massive
difference in you know the default Basin
size and the default likelihood of
finding one or the other then you have
to sort of make some really big you know
massive structural change to your
machine learning process whereas if it's
only a small difference then you know
you can get away with much smaller
changes to the machine learning process
to you know change
um you know to try to escape the sort of
stepping line models
a stance on exact
large I think the Gap is where I think
it is
but this is sort of I think of good
frame to be thinking about
we're sort of going to be trying to
accomplish is you know how do we end up
in some situation where we can provide
some intervention the changes that you
know default path of how what sort of
algorithm you're going to get uh to to
not select for deceptive alignment
um uh yeah and so that is that's the
talk so uh you know we're going to be
talking next time sort of about you know
uh you know a bunch more sort of about
how generally how do we start
approaching solutions to this problem
but uh this is at least hopefully you
know trying to wrap up deceptive
alignment you know give us a really good
grounded understanding of this very
specific you know failure mode that is
you know we're very concerned about this
you know deception phase
[Music]
thank you
all right we'll do some final questions
um how does this relate to like the
picture of like metal learning or
uh where metal learning you're basically
learning or an Optimizer and you're
learning the inductive biases to put it
to like another algorithm so and I guess
it's also related to work on the
conditioning gender models so uh well
like that is the
the optimizer that you learn going to be
like
is it possible for it to have like a
different
like mode of enough devices like your
like base Optimizer being have high
inductive biases and then here
learned Optimizer have like you know low
path devices
yes this is a really interesting
question so first I'm gonna punt on the
like thinking about predictive models
case because we we are going to talk
about that later there will be a later
lecture where we really try to
understand and this is what I'm
referring to you know hypotheses for
thinking about how language models might
work if I just think about the other
question right which is you know the
question here is essentially what if
we're in a setting where we learn some
model and that model is itself
implementing not just a search process
right it's a mace Optimizer but it's
implementing a search process over other
algorithms then that search process
might have different safety properties
than the original search process
um I think that this is a concern it's a
bit of a an esoteric concern because you
know it's a little bit weird why we
would find ourselves in a situation
where you found a model that is then
doing another search over algorithms but
it's totally possible it's definitely a
theoretical concern to be potentially
worried about
um and it's absolutely the case that you
might end up with different properties
in the new case right so it might be
that you have a good reason to believe
that the sort of outer thing is doing
the right thing but then once you find
another search process you know it might
not be doing the right thing
um
so this is in fact one of the issues
oftentimes with the speed prior in
theory trying to get it to actually work
out in the worst possible case
um this sort of often comes up
um
I think that it may be a concerning
practice it's a little bit tricky I
think the way you'd hope to try to
resolve something like this is that
whatever safety properties we have on
the top search we can somehow transfer
them to the the next level below
um one thing that's really worth being
very clear about here though is this is
a different thing than the General Mesa
optimization case right the general base
optimization case is just any case we
were doing a search over algorithms and
one of those algorithms is itself doing
like a search over anything including
over you know actions or plans and this
is a case where we found we did a search
of our algorithms and we found an
algorithm that was doing a search but
the search that it was doing was
specifically also over algorithms right
yeah okay yeah question
is learning Phoenix Airport well exactly
this
very interesting question I think uh
it sort of depends on how you think
about algorithms we're going to talk
about this later I think that I would
often think about in context learning as
doing a sort of um
search over conditionals it's like it
has a has a sort of probability
distribution on like you know how it
thinks you know what things where things
it is what thing it thinks it's
predicting and then as it gets more
information it's sort of trying to
figure out of the various different
hypotheses you know m is this a news
article you know is this a
um you know uh you know a fan fiction or
whatever it's it's you know getting
information that slots between those
hypotheses you could sort of think about
those algorithms in some sense it's a
little bit um unclear
um whether you want to think about it in
that sense
um but we will definitely return to this
later and think about it a bunch more
yeah
so this is actually a very basic
question but I just recently realized
that I don't have a good picture of the
answer
so we are talking about the training
where these general intelligence there's
Modas of the world during the training
and learns things about hot glues the
proxy objective is to the real one and
so on how should I imagine this trading
is is the agent in Minecraft is a
talking with people how do we imagine
ever teaching with any
um what do we want about governing a
country or a company what is the
training
yes this is a really good question I
think part of the problem of a lot of
this analysis is I don't know you know
we don't know exactly how we're going to
train future AI systems and what we're
going to train them on so I think part
of what we're sort of trying to do here
is do some sort of you know well what's
the what's the worst case scenario of
the possible things we might ask AI
systems for right we sort of imagined in
this particular setting that we're going
to be asking them to do you know
whatever the most complex tasks are that
we want they require you know the most
You Know instrumental reasoning you know
they require them to be able to do all
of these various different things if we
were in that setting and we're trying to
train them to do that you know what
would happen right and so
in fact you know there's going to be a
very large range of things you know
probably that we ask AI systems to do
that we train them on that we get them
to do in various different setups
um and those might have very different
properties uh and so you know I don't
want to you know I don't necessarily
know exactly what it is that we're going
to be doing
um I think that in some sense though and
you know I was saying before
um if we think about this sort of from
an economic perspective all of these
various different use cases you know are
going to have you know reasons why
people are going to want to build AIS to
do them right like and so in that sense
you know we want to try to understand
you know it may be that 90 of the AIS
that we build and 90s the AIS that we
train are totally fine and they like
don't have any deceptive issues we have
10 are deceptive that might still be
catastrophic problem right
um and so you know even if it's only in
some individual training cases that this
sort of occurs uh we might still be be
quite concerned and so that's sort of
why we're looking at this particular
scenario and understanding okay what if
we were training it to you know we train
it on all of the possible knowledge and
all the data that we had available we
trained to accomplish whatever the most
complex goals are they were trying to
get it to do
um you know that we possibly might want
out of it then what happened
Yeah question
I think I understand that part that now
we are concerned about some people
trying to build an AI to be a good CEO
or advisor or politician advice or that
seems the most General thing mostly we
can ask for and but then
how should I imagine the training
happening this seems to be some kind of
reinforcement learning training we are
talking about it does things if it
doesn't predict the next actions we
punish it if it doesn't do well or
directness training does reward if it
does well and so on uh should I imagine
this being done to a future CEO AI
yes I'll point
our point is true supervised learning is
learning and we have an algorithm and we
have a particular data set that we want
the algorithm to fit well and then we
see how well it fits the data set and
based on the mistakes that it makes we
go back you know greatness then goes
back in and it changes it to do better
right so I don't think that this is
specific to reinforcement learning
um and sort of as I mentioned at the
very first lecture I think that um you
know there are important technical
differences between reinforcement
learning and you know supervised
learning or other sort of approaches I
think that they're in practice often not
that important and the sort of more
important thing has to do with the tasks
you're trying to get to do so a simple
example of this is that let's say I want
to solve like a you know traditional RL
task like you know playing chess or
something
um I can actually do that via something
that looks more like supervised learning
so if I do something like a decision
Transformer where would I have is I
actually supervise learn my model to
predict what the action would be that
would get a particular reward so I sort
of conditioned the model on observing
some reward and then have it predict the
action that you would have to take to
get that reward that I can collect as
just a general data set of you know a
bunch of just like you know things that
I can supervise learn I just want you to
at the data set of what action would get
this reward and I'm sorry I'm not doing
some reinforcement learning thing
um and then I sort of just train on that
data set then effectively you know I'm
still effectively doing reinforcement
learning even though I'm not literally
doing reinforcement learning because I'm
still training the model to do a good
job in some environment and so I think
that the sort of distinction between
like you know in theory am I using this
particular like you know ml method or
this particular ml method is like is not
that important I think that
um sometimes it can be important and
those technical details can matter
sometimes but I think for the most part
I want to align over them
Yeah question
um so it seems like the deceptive
alignment uh
uh case depends on the model building up
a long-term goal and updating it from
the proxy so there needs to be a point
where we have these proxy goals uh like
solo online model and uh it needs to
update it to a long-term goal
uh do you have any
any framework to think about how or
maybe just example thing how that
happens why would there be a steep
decrease in the Lost landscape from the
proxy to a long-term objective uh it
seems like in each implements several
things at once to have an improvement
which is like to objective was connected
to uh some circuits that implement the
instrument or reasoning afterwards with
relation to the world model it seems
like you need to move a lot of pieces at
the same time uh to to get from a proxy
to long term but all the deceptive
things coming after it
question so a couple of things that I'll
say so the first thing I would say is I
think we're mostly Imagining the sort of
instrumental reasoning circuitry is
often already hooked up it's just hooked
up for some non-long-term proxy right
like if you think about you know you
have a model and it's just trying to get
like the Green Arrow you know each
individual episode it still has to do
instrumental reasoning to get the green
arrow right it's less to figure out okay
which pass through the maze or most
likely get me the Green Arrow and so
it's not necessarily the case the green
design has to do like a bunch of
additional stuff to like hook it up to
the instrument of reasoning because it
we're sort of imagining it's probably
already hooked up but it does sort of
have to take that goal and make it long
term make it so now it's thinking about
how to get the green arrow as much as
possible throughout training and Beyond
um how difficult is that modification
um very unclear
um so one thing that I think is quite
clear is that it's sort of this you know
Atomic modification in the sense that
once you've done this like you know
long-term goal then there isn't you know
this sort of oh you then have to keep
refining the long-term goal right like
we were talking about you know once
you've gotten any long-term goal that's
good enough to sort of cause the
deceptive behavior in training and so
the key thing is just
um you know how difficult is it to
actually add it just add some long-term
goal um I think it's unclear I guess a
couple of things I'll say I think that
in some sense you know I don't think
it's that complex you know you just have
to you know count all of the green
arrows rather than just the ones you
know in front of you as long as you have
the knowledge and understanding of like
what it means for there to be a world
and what it means for like there to be
stuff in the world that you might care
about you know picking some of those
things they care about I think is
relatively straightforward but um
I do think it's quite unclear I mean I
don't think we really know
yeah
uh I mean we I guess we keep talking
about like long-term goals and
short-term goals and goals that Acquire
instrumental reasoning and goals that
don't but uh it seems like for the most
part like in current ml systems the goal
that we're training these molds for is
just like maximizing the law likelihood
of some data set so I mean this seems
like a goal that doesn't require any
instrumental reasoning and it seems hard
to become like deceptively misaligned
from this specific goal so I mean am I
misguided of this
so first of all I think you absolutely
need Instagram
you solve the task of predict the
internet
as humans do it
again so if you want to predict what
humans are going to do in various
different cases you definitely have to
be capable of instrumental reasoning now
I do think that it is quite unclear in
that case whether you're actually going
to learn a deceptive thing and again
we're going to talk about you know how
that might play out what sort of you
know things to think about in that case
later
um but yeah just to start with I think
that is definitely not true that you
don't need instrumental reasoning to do
a good job on predicting you know humans
you totally do
okay uh we will call it there and uh
well you know uh pick up next time
foreign