5:Predictive Models: Evan Hubinger 2023

So today we're going to be talking about
um large language models and predictive
models and
um sort of trying to understand how to
think about and what sort of some of the
possible ways are to think about
um aligning
um predictive models like potentially
large language models so I've sort of
alluded to this topic a bunch of times
previously uh you know said that I would
eventually try to cover it and today
we're going to try to do do it justice
so
yeah so I'm going to start with a bit of
a preliminary so this is a bit of an
aside but we'll see later why this is an
important thing to sort of cover in this
context
so I want to introduce you to the
eliciting latent knowledge problem so
what is the listening latent knowledge
problem so let us suppose that we have a
vault and inside of that Vault there is
a camera that's sort of you know keeping
tabs on the Vault trying to understand
you know give us information about what
is in the ball and we've got a diamond
in that Vault that we want to protect we
want to make sure that you know the
diamond in fact stays in the vault
um one problem that you might have in
this setup if you were just sort of
trying to watch that camera and
understand where the diamond in fact
stays in the vault is that um somebody
could spoof the camera so you know there
could be a situation where uh you know
maybe you hold up a little sign from the
camera that looks like a diamond and
then you actually take the real diamond
and you steal it
um so this is an example from Arc the
alignment Research Center where they're
sort of trying to you know showcase a
situation where sometimes
um if I had uh you know their sort of
knowledge about the world that we care
about that may not be represented in the
particular observations about the world
that we get to see so in this case we
could imagine sort of training a model
to predict what our camera would show uh
you know in this Vault and in the
situation where you know you use a
screen is put up in front of the camera
the prediction of that model would
continue to be you know it looks like
there's a diamond in the vault for all
of the observations that we checked but
in fact there might not be you know it
might you know in this situation where
there's a screen in front you know the
diamond might actually be gone and it
may be the case that our model knows in
some sense the diamond is not there but
even if it in some sense knows that the
diamond is not there it's just trying to
predict what the camera would show and
the camera still shows a diamond so it's
still going to predict a diamond and so
we sort of have this potential problem
where sometimes when we ask our models
to do things that are predicting
observations about the world there may
be information that those models have
access to that we uh you know can't
directly get at Via those observations
okay
um we'll see why this ends up being
relevant later but I just want to make
sure people understand this basic setup
okay so um we're gonna go into this a
little bit more so here's how I want to
sort of uh I want to sort of focus in on
this predictor in the out case right in
the elk you know it stands for listening
latent knowledge we have this predictor
right it's predicting these cameras we
have these camera observations and it's
trying to understand you know and
predict each new frame you know what is
this camera going to show uh in this
vault
and so we can try to understand what a
model like that might be doing uh you
know how might it be structured
um and so you know in some sense well
there's some model of the world that is
keeping track of it has to model the
world in some sense and try to
understand you know how the world
operates and has to model the world over
time and how it changes
and then it has sort of particular you
know aspects of the world parts of the
world that you know are dependent on you
know that world State and then some of
those aspects of the world get observed
and end up in the camera uh and so it
you know predicts you know what the
camera will show at various different
points in time and you know for example
there are some parts of the world like
the the wall that the camera is mounted
on that it might sort of you know
understand how those you know flow from
the from its model of the world but
don't show up in the observations
because they're not you know something
the camera is looking at
Yeah question
so if the if it already has a model of
the world why can't we just get it to
tell us what it predicts will happen in
the world directly
yeah so yeah so why can't we just get it
to predict what happens to the world
well the problem is what does that mean
right like what are you predicting right
if you're not predicting some concrete
process that can observe the world via
some mechanism then what what is it that
you're actually asking for right you
know there's if there's no camera they
would show you you know the thing that
you want or you don't know how to find a
camera or get a camera that would show
the thing that you want then what what
are you even predicting right like what
is the what is the fundamental thing
you're asking for like in some sense
what we really want is it's like well
there is some knowledge of the world but
that knowledge of the world is not
necessarily accessible via any
observation that we might know how to
ask for right
um and so
you can't just sort of to the extent
that one can sort of train a model to
just do a prediction task you're sort of
very limited to predicting things that
one could observe in the world right if
you have some mechanism of observation
and there may be things about the world
that one can't observe via any
observation right there's no camera that
you could set up that would maybe show
you something or we don't know how to
set up camera there isn't the camera
that shows you some particular
observation despite the fact that there
is something about the world that is
like in fact true or false
but is that like a limitation of
what we can get into the observations or
limit a limitation of what the model can
get out of the observations like we're
assuming are we assuming the model knows
the diamond is gone
um we're assuming that the model could
know that the point is that the model
might know the model might not know but
in either situation we would not be able
to extract that from the observations
we're mostly going to be sort of
imagining that the model does know that
it's sort of you know in this sort of
situation the model might know that
there's a cam that whether the diamond
is actually there but we can't extract
that via the observations because the
camera won't show the you know whether
what is actually happening with a
diamond will only show what an actual
camera would show because that's what
the model is doing it's just predicting
the next frame of that camera
Yeah question so I mean I guess you said
camera but like it could be many things
I mean could it be like a whole network
of satellites that are orbiting the
Earth and like every surveillance game
or ever so I mean how would this affect
the alternator multiple cameras then
yeah so absolutely your camera could be
a lot of different things and later
we'll see a particular sort of notion
and instantiation of a camera
um that that I think is particularly
relevant but absolutely yes there's a
lot of various different things that
could occupy the spot of camera here
um and if your cameras are very
expansive they have the ability to
really you know get lots of information
through observations then you get access
to more but there's still a fundamental
limit to how much you can get access to
and what your thing is actually doing
right in this situation regardless of
what your camera is it's not telling you
true information it's it's telling you
what it predicts that particular
observational instrument would show
right and that's that's different
um and so that's sort of the important
point that we sort of want to we want to
understand with like what this sort of
predictor model might be doing
yeah
question so
when we are saying observation what is
the relevant difference like in human
language it seems
um the same to describe
Waterville please predict what the
camera will show and please predict what
will be on the table and we want to it
actually predict that the diamond is on
the table is the relevant difference
that the camera this observation is more
of X actually a digital code and we can
more easily encode that this is the
thing while thing on the table is
um no even in that case in some sense
um the point is that uh
it's a little bit hard to understand you
know what what exactly is a human doing
when they say ah that thing is on the
table I think that um
you could imagine that what the human is
doing when they say that thing is on the
table is they're just saying well you
know I think I'm going to continue to
observe it on the table
um in fact you know oftentimes we mean
something different we mean like no
really in base reality there is in fact
a thing on the table or whatever I think
the thing I'm sort of pointing out at
the very least is that we don't know how
to train a model certainly to say that
second thing it's a little bit unclear
whether we even know how to train them
all to do the first thing but we might
at least try to know how to train a
model to do the first thing where you
know you can give it a concrete
observational procedure and you know in
fact get observations that were produced
in that procedure and then train a model
continue that sequence
um if you just sort of are doing naive
sequence continuation on a bunch of
observations you're going to expect it
to you know do something like continue
those observations you know what are
additional observations that you would
expect from that process
you wouldn't generally you know being
able to in fact get it to give us you
know
some sort of true representation of its
beliefs about base reality whatever that
even means is a really tricky thing that
we certainly don't know how to get
models to do yeah question the slideshow
in particular you're saying
understanding the elk predictor when I
have a hard time figuring out move up
the predictors on the slide
yeah so maybe we'll go to the next slide
so let's sort of yeah so let me let me
like sort of start giving some names to
these things so we're going to say right
so the the inside of the predictor it
has to have some World model has the
model of the world uh and that you know
has a bunch of different components and
then it has some model of the camera
some model of how those you know things
in the world its model the world how
it's understanding the world operates is
going to result in observations
and then the thing that it does is that
it produces these observations so it
sort of you know runs this thing forward
and predicts you know what these
observations would show at you know each
different point
and so
um you know importantly right we have
this you know training where we're sort
of imagining well if we've just trained
it on some previous set of you know
camera data then we're hoping that it's
going to continue the sequence in this
way where it continuously runs some
model of the world and translates that
into observations
um we don't know this you know if only
thing I did right was I trained on this
set of data we don't know that this is
the sort of way that we would get a
continuation right we've talked about
this a bunch in previous talks where
it's very difficult to know you know
just given that I have some model which
fits some data is it in fact going to
generalize in this way and we're going
to talk later about you know how to
think about this in this sort of context
but for now at least we can sort of
Imagine okay one plausible
generalization one plausible way one
sort of algorithm that you might learn
this relatively you know mechanistically
simple and straightforward is have some
model the world predict the camera and
then sort of continue the camera
so the picture is
predicting the world at type T and
titeria so internally it has the model
of how the world is changing over time T
but you never get to see that the only
thing you see are the observations that
you know that world would imply
so I still have a hard time figuring out
what's internal to the model now yeah so
everything here is internal to the model
that we're looking at everything on the
screen is internal to the model we're
saying the model internally has some
model of the world
some of those pieces of its model matter
to its op to what the observations that
it outputs and then it produces outputs
of what the camera would show at each
individual point in time and where's it
put well so right now I haven't shown
you an input this is this is this has no
input it is just producing sequences of
outputs of what the camera would show uh
the input in some sense is just time
right it's just like over time what
would the camera show now
you know we do actually sort of have
situations where we can give it input
and those would be conditionals and
we're going to talk about that uh yeah
later
Yeah question why wouldn't the input
wouldn't the input be the frames of the
camera here
um do you mean like like the previous
frames
well yeah like you the agent takes in
like a frame of a camera and then
outputs the next so so here we're just
taking we're just saying it's just
predicting camera observations over time
yes absolutely one thing that we are
going to talk about very soon is
situations where you can input some
particular observation say suppose the
camera observed x what would be the most
likely thing to observe next and that is
in fact you know what the input is going
to look like here but right now there's
no input it's just predicting over time
and that's also a sensical thing to do
if I'm just like you know start at this
particular time and predict forward what
you think the camera is going to show
that is a you know a at least a
well-defined prediction task
Yeah question so where I look out at the
world you know in my model I don't have
an explicit eyeball without it and that
explicitly predicting all the shots I
see with uh 4K finale
um so like what happens if it's
frustrating this like is it okay if my
observations are predictions at a very
high level mostly just predictions of
future vibes yeah so the question is
what happens if it's Messier so so thing
I'll say is it's going to be Messier
right so this is a very simplified model
um and we haven't even gotten to like
how I think this model applies to sort
of current models in practice
um but certainly this is simplified and
it's it's likely to be substantially
more complex than this I think and we
sort of talked about this previously
there's a lot of value in looking at
sort of simplified models that we think
have some relationship to reality and
trying to understand what their
predictions would be again we'll talk
later about you know how likely this
sort of model is actually to be
um you know compared to other you know
plausible models
um I think that I want to point out
um I think that the sort of General
thing of think of your model as doing a
prediction task is reasonably likely to
be sort of an accurate thing
um a thing that is almost certainly
false is that it has some sort of very
explicit you know model of the world
that it runs forward we're we're in fact
we're going to basically throw out this
like model the world that it runs
forward over time pretty soon
um but uh so so I think we should think
of you know like this sort of basic
process though where it has some
understanding of the world some
understanding of facts and you know
information it translates that into how
you know that world would get observed
and then you know makes predictions
about observations I think is reasonably
likely to be true but the specifics of
you know in fact having some you know
model that explicitly is running forward
over time steps almost certainly false
but the point though is that well okay
we're gonna you know pause at some
possible internals and we're sort of
gonna try to reason about how it works I
think that we're mostly not really going
to rely on anything about
um you know the specifics of sort of How
It's you know doing it's World modeling
we're just going to say Well it has some
understanding of the world and from that
understanding of the world it sort of
figures out observations
question I know you just said that we
were talking about this specific model
pretty soon but like I'm just thinking
when I sort of try to do prediction I
usually start from the conclusion and
then try to like walk backwards and see
like how the which events lead up to
that so is it possible that if you could
sort of reverse this order and like try
to predict the previous frames from the
future frames instead and
um have an inverse predictor well no
because if I need to like predict what
the future frames are going to be I have
to have some idea of what they might be
right so the thing that you could do is
you could be like okay generate 10
possible ideas for what the frames might
be and then go backwards and evaluate
the probability of each one but then you
still have to have a generation
procedure that is very very good because
it can distinguish between many
different possible Futures and you know
narrow it down to a few that you know
that are worthy of consideration
um so you know you can't just like
entirely go backwards now we will talk
very soon about conditioning
conditioning absolutely goes backwards
when you're saying suppose that we've in
fact seen some particular set of frames
previously what would be the most likely
continuation frame um the like
conditioning on having seen some you
know observation in the past is
absolutely a backwards procedure
or I was wondering if like you know you
if you have some idea of what you would
want you to be able to look like you can
just sort of write that in and sort of
like try to get it to predict backwards
like sort of events that led up to that
point and this would be a sort of yes
I'm not really going to talk about that
case but it's it's talked a lot more in
the listening like knowledge report
basically why that's a bad idea the
basic reason that you should have
shouldn't do that just sort of condition
on observing a good world and then try
to get the actions that would lead to
that is um The Diamond problem right
that we just talked about that in fact
it might look like from the observations
that the world is good but the world may
not be good just because it is observed
to be good right and so you it's it's
only sort of safe to do that where you
just sort of condition on the world
being good
um if you actually believe that you have
access to the sort of true fact of the
matter is the world good if the only
thing you condition on is the
observations then it's not in general
safe to do that so we have this sort of
elk predictor
um I mentioned talking about
conditionals so conditionals are really
you know a key sort of thing that we
want to understand that's sort of going
to happen here so what is a conditional
right so we have this predictor right it
you know runs sort of you know has some
understanding of the world it's sort of
from that deduces you know various
properties of the world and then from
that figures out what the observations
would be
um well so a thing that we would like to
do oftentimes is condition on a
particular observation so that says
suppose that you saw some observation
given that what would be the most likely
next observation and so we can imagine
well what happens when you take a model
you know like this and you sort of give
it the ability to do this conditioning
well it has to do something where it
takes in the conditional and then infers
you know backwards what the most likely
World states are that would imply that
conditional right and so you know
importantly the way that that inference
goes through is it only goes through
properties of the world that are
observable right so if it wants to
figure out you know how my full
understanding of the world given that
I've observed some conditional you have
to figure out what sort of hidden
variables you know there's all these
sort of hidden information about the
world that's not observed of those
hidden information about the world will
be the most likely way to fill it in
such that it would result in observing
this particular conditional and then
once I know how to sort of fill in and
understand all of those hidden variables
about the world I can play things
forward and predict what the next
observation would be
okay
and so this sort of conditioning process
is is really going to be the core of a
bunch of things that we're going to be
talking about this idea of being able to
sort of take a predictor and say well
what if You observe this thing you know
what would be the sort of internal you
know World state that we most likely to
cause that and then what would be new
observations given that world state
how is that different than what
I mean your proposal was exactly that
wasn't it does no or I I sort of meant
yeah conditioning on the far future and
then walking backwards then this is
we're not necessarily conditioning on
the future or literally anything we're
just saying you can condition right so
we're going to talk a little bit later
right so the the objection that I was
Raising previously was to the specific
strategy of condition on seeing the
great future and you know predict what
what actions we most likely to lead to
it that is unsafe given that you have
only the ability to sort of determine
how good a future is by looking at the
observations a extremely important point
that you know we're going to be talking
about a bunch is that that's that does
not imply that anything that one does
with a predictive model is unsafe right
so there may be many other conditionals
and other ways to make use of it that
are safe even if the most General
possible purpose thing you could want to
do with it which is just predict me a
good future
um is not safe
question so we say if this doesn't imply
that anything that you do is unsafe do
you mean like it implies that there are
some safe things you could do with a
predicted model even though there are
also some unsafe things you could do
maybe I'm not going to say that there
are safe things that you can do with
this because I don't know if there are
safe things that you can do with this
but we're going to talk about what it
might look like to be able to do
something safely with this
yeah and the reason that I think that's
important is because I think many
current models might be well described
this way we're going to talk about that
okay
question
when it simply tries to predict the so
it already offers are some history of
observations so far and tries to predict
what happens next isn't that already
Condition Nothing on the history so far
like uh yeah you can totally think about
it like that okay
um as sort of starting out conditioning
on the past in some sense you know it's
just sort of a question of where you
draw the boundary between your prior and
posterior you can always sort of roll
your prior all the way back okay
okay great so this is our sort of elk
predictor you know it's doing this sort
of simple camera task it's doing this
relatively straightforward you'll roll
the world forward thing and it can do
this sort of conditionals where it does
back inference
okay so we're going to shift gears that
we sort of covered the preliminary of
this sort of elk predictor now I want to
sort of ask the question that I promised
at the start which is you know how
should we think about large language
models you know we sort of talked you
know at the very beginning you know I
showed some examples of you know these
sorts of very powerful current language
models that have the ability to do all
these sort of really interesting things
and so you know in the sort of spirit of
sort of how we've been trying to
understand everything that we've been
trying to understand in these talks we
want to sort of get at okay
what might it be doing internally right
what sort of algorithm mechanistically
might be operating inside of that sort
of a model that would be causing it to
do the things that it's doing and how do
we sort of understand what the
consequences of possible algorithms
would be and How likely different ones
would be
okay so you know I have a couple of
hypotheses up here
um you know things that it could be
doing you know it could be you know some
simple loose collection of heuristics
um it could be you know an agent we've
talked previously about you know
deceptive agents uh and you know why
they might arise in various different
situations
um
the one hypothesis and it's hypothesis
that we're going to start with that
we'll come back and sort of talk more a
little bit later about some of these
other hypotheses and how to compare them
with hypothesis we're going to start
with is this predictive hypothesis that
in fact the sort of same way that we
were thinking about the elk predictor
where it's just sort of predicting
what's going to happen in the next frame
of the camera is a similar you know a
good tool for thinking about
understanding what might be happening in
the case of these large language models
okay so this is an assumption so we
don't know if this is true and we're
going to talk more later about why it
may or may not be true but I think it's
a good frame and it's going to help us
understand what some of the sort of
unique challenges might look like
um in the sort of language modeling case
when they're sort of operating in this
sort of predictive sense
okay
so we need to understand what it looks
like for language models to be
predictors so the the first change that
we have to sort of make from our elk
predictor is that well we're not we're
no longer at least traditionally
predicting observations over time we're
predicting a general distribution of
possible observations so uh you know
what we'll often do when you train a
large language model is you will collect
a bunch of data from the internet via
some you know mechanism and then you'll
train the model to predict from that
distribution of all possible internet
data you know sampled randomly
and this sort of has a couple of
implications if you're thinking about
the resulting model that has been
trained on that as doing a prediction
task if it is in fact doing this
prediction task where it is you know
predicting what would show up on the
Internet well we can think about it very
similarly to the L predictor
where it has some model of the world and
from that model of the world you know it
implies various different properties
about the world you know there's various
different websites that exist on the
internet that have that you know that
might have various different texts based
on you know how the world Works some of
those websites are observed you know in
terms of ending up in the data
distribution in terms of doing scraped
and actually trained on but some of
those websites are not observed and you
know there's the you know from all the
various different websites that observe
you know you you create some
distribution of all of this sort of
possible attacks that you could see
um then you can sample from and sort of
produce observations and so those sort
of observations that the model is seeing
are sort of coming from you know these
sorts of websites that it's observing in
terms of you know the ones that might be
might be scraped into the model's data
so again we can sort of you know think
about this as the model has the model of
the world it has some observations which
are you know the various texts and then
it has a camera right it has some notion
of how the model of the world gets
observed it produces observations that
then get predicted right and in this
case the camera is different right so
instead of thinking about a literal
physical camera in the room the camera
is this sort of data collection
procedure that goes out and you know
scrape some stuff off the internet you
know selects you know which pieces of
things on the internet would be you know
put into a data set and then you know
randomly shuffles that data set and
gives it to the model right it's a
different camera but it is still a
camera that has a mechanism for taking
the world observing the world and
producing these observations that then
you can predict right and so if we think
about you know training on those
observations as being very similar to
doing something just like training on a
literal physical camera then where you
know we might get something you know
like the alpha predictor where it's
doing this prediction task it is you
know it has some notion of what this
camera would be that is observing the
world in this particular way and it's
predicting you know what that camera is
going to show
okay
um and again uh you know we can think
about you know this sort of training
generalization where you know we've
maybe trained on a bunch of various
different websites but there's other
websites that exist in the world that we
might not have trained on you know for
example websites that might exist in the
future that might you know be in new
training data Maybe Just websites that
are you know really unlikely to have
been scraped but you know there's some
chance they might have ended up in the
models Dana you know sort of websites
where there's sort of in you know some
unknown information about the world that
might determine whether website ends up
on the on the Internet or not or ends up
in the scrape or not and the model
doesn't know and so there's sort of this
large distribution of possibilities and
we can sort of Hope to try to get access
to some of these other websites that
maybe don't actually exist but are sort
of possible websites that the model
might see in some particular situation
uh you know out of this sort of
prediction task in the same way that we
can try to get at like
um you know future camera frames from
our camera predictor right you know
we're sort of hoping with the predictor
you know this predicting an actual
camera that you know we haven't actually
observed these sort of future camera
frames but because you know we've
trained on a bunch of camera famous in
the past the most likely you no
generalization maybe is you know
Faithfully predicting future camera fans
and so again here we can imagine well
maybe the generalization we want is
Faithfully predicting you know new
things from that distribution you know
if we you know what are you know the
model has much of uncertainty about the
world has a bunch of uncertainty about
how we did some of data collection
procedure about how it's camera observes
the world about what's actually
happening in the world and we can
generate you know websites that don't
actually exist from that kind of actual
distribution over you know all these
various different ways which things
could be operating
okay is that sort of what we're hoping
to do here
all right and so again we have you know
conditionals so in the same way where
previously you know we can condition on
observing things we can condition on
observing things again and it and very
importantly the way these conditionals
operate is extremely similar to the way
the elk conditionals operate in that we
can't condition on actual true facts
about the world the only thing we get
the ability to condition on is well what
if you had observed some particular text
at some particular time right you know
or we often can't even condition all the
time we can just condition on what if
your data distribution contain this text
what would be the most likely
continuation for that text right and so
we're conditioning on an observation
again we're saying suppose your camera
showed you this text what would be the
most likely text to come next
um and so you know when when you have a
model that's sort of given that
information and in fact that model is
doing a prediction task then what it has
to do is has to do the same sort of back
inference where it says well
given that I you know that my camera
would have showed me this text that
implies that you know these various
different hidden aspects about the world
must have various different properties
that would imply you know various
different things about the world such
that I would you know then be most
likely to observe some particular text
next
Yeah question and this is pretty
abstract could you give us like a simple
concrete example
yeah so we'll have some uh you know a a
nice sort of concrete example later but
I mean the basic idea is very simple
it's just saying you know suppose that
we were in some situation where I you
know condition on you know New York
Times article about you know the next
election right and then it's like well
it's just gonna you know give me you
know once it's seen the data that says
you know this is a New York Times
article about the next election well
then it's going to continue with what
the most likely continuation would be
about you know in fact seeing something
on the Internet that is a New York Times
article about the next election or
whatever or maybe not right you know so
important thing is we only get the
observation that it was observed to look
like a New York Times article so maybe
it'll think that it's like you know
another post quoting the New York Times
or something it'll give you something
different right so we we don't actually
again you know importantly have the
ability condition on you know this is
actually a New York Times article
will we condition what we can condition
on is well it was observed to be a New
York Times article view whatever cameras
we have the ability to implement here
okay and then you know from that we can
get particular conditionals right so if
I condition on this is a New York Times
article or if I condition on you know uh
something very different like you know a
piece of code and I wanted to fill in
the code for me well you know the
continuations are going to be extremely
different right because you know if I
saw some code on the internet and you
know the most likely continuation of
that code is probably going to be some
more code and not a New York Times
article and so in the same way we're
using you know these conditional sort of
put it into a particular situation where
it's like ah you know now I'm most
likely to be observing you know website
a though it doesn't tell at 100 right so
it might be you know I say I give it
some information that's like this is a
New York Times article but it's still
going to have some distribution it'll be
like well website a is an actual New
York Times article but website C is a
you know uh you know copy of a New York
Times article on Reddit or something
where some people edited it and you know
did some weird stuff to it to try to you
know prank some people on Reddit or
something and so you're like okay the
model has a hypothesis has two plausible
hypotheses for where this data you know
what the actual what the generator of
this data might be and so you know the
distribution of those two hypotheses is
like you know what is the most likely
thing you know if I you know have some
probability on this distribution and
some probability and those is on this
distribution and I mix them together you
know I get some resulting distribution
that is sort of sampling from the sort
of multiple different plausible
hypotheses of like you know where where
the where it thinks the data is coming
from right
okay
question
could you explain the difference between
let's say we have like the textbook of
the future this is how we saw the
linement whatever and then you're like
trying to like generate from that so you
have that example right but then you
also have potentially like an example
where it's like
um
like two like Paul and Eliezer like
talking about like
um this particular Concept in alignment
or whatever and they're they're just
near the Breakthrough of like this like
solution for this thing into alignment
it's like okay well in this case it's
like uh maybe you're like going forward
in time and the other one you're like
trying to predict backwards from like a
textbook that like is describing as a
like a solution to alignment or
something like are are these two both
bad cases and what you're describing
here or like
um like I I'm having a difficult time
understanding where exactly you would
point to it having like being a bad case
where you're doing conditionals and
stuff so I don't know what you mean by a
bad case I think that those are both
situations where you are using a model
and that model might be well described
as a predictive model if the model is
well described as a predictable model
then we can sort of understand you know
what it's doing in terms of predicting
what the most likely continuation would
be given that had observed that previous
information
um importantly the sort of notion of
time you know for these sorts of models
is different right than the notion of
time for the Elk predictor right with
the out predictor was very
straightforwardly predicting
observations you know forward in time in
this case you know because the way that
its camera Works scrambles things you
know it doesn't know when a particular
piece of data was observed
um you know it'll have a distribution
over many possible times that you know
something could be generated from and so
then it'll you know give maybe learn
some information about what it thinks
the most likely time in which that that
article was produced would be based on
you know the contents of the article and
so the time is sort of a hidden variable
that it has to infer based on
conditioning rather than something that
it just you know knows as in the in the
in the Alka case but in both cases you
know they're they're if they are in fact
well described as predictors they're
doing so very similar yeah I'm not
exactly sure what you mean by like a bad
case I'm not sort of I haven't yet
talked at all about you know why what
situations where this might be like good
or bad
um I only mentioned just like well the
most naive thing of doing something
where you're just like you know suppose
we had a like you know Utopia uh you
know condition on observing a Utopia
will be the most likely actions that
would lead to it doesn't work the reason
that doesn't work is the standard elk
argument that well conditioning on
observing a Utopia is very different
than an actual Utopia
okay and we'll talk later about what in
practice you might want to do with these
predictive models uh if you if you had a
predictive model and what the sort of
you know issues would be in in you know
if you're trying to do that
Yeah question
so just to follow this again
the purple kind of arrows
um give them all kind of information
about the conditional and then the
yellow one hurts our full would be a
prediction that it makes about website
let's say really attention right so the
idea here
on the yellow arrows is that when you
have a conditional and the website a is
actually observed then the conditional
can maybe directly given information
about website 8 but if website B is some
property of the world it's like
something that exists on the world
that's never absorbed by the cameras
then the only way that website B is
influenced is well this observation
tells it some facts about the various
hidden variables about the world and
some you know it's information about the
world and then from that it can deduce
you know what website B must be you know
uh you know uh to you know make given
those variables right but it's not
directly sort of giving you information
about some properties of the world so
similarly you could think about
something like you know is the world of
Utopia right is like a property of the
world that's not directly observable but
you know there are various different
things that are directly observable if I
condition on observing something it can
tell me some facts that tell me you know
some information about what the hidden
variables of the world must be and from
that you know it can deduce you know are
we in a Utopia or not but I can't
directly condition on the Ami and Utopia
similarly in this case you know because
it's only able to observe you know
various different properties of the
world in particular you know some things
on the internet we only have access to
understanding and how to condition the
various properties of the world via
those observations
question uh this might be a bit too soon
for this particular question but like
let's say we have a conditional it's
like the year is 20 30. now it seems
like there's a couple of different ways
that this could go one way is it could
say well most articles that say the year
is X seem to then result in stuff from
that year so I should assume the world
is 20 30. and the other assumption I
could make is well I have never seen
anything from the year 2030 but I have
seen some science fiction that said in
that so naturally it has this has to be
science fiction it is actually real yes
that is exactly correct so I think that
this is a really key question we talk
about this a bunch in the conditioning
predictive models paper I'm going to
talk about it less here but I'll briefly
sort of do justice to this question of
you know would it predict the future or
would it just predict sort of the
counterfactual presence you know
situations that are not true about the
present but that if they were true they
would imply you know that it would they
would produce some particular thing that
would sort of look like it was from the
future
um it's very unclear so it really
depends sort of on how the model
conceptualizes its cameras right so we
can think about this as well given that
I am a predictive model and I have been
trained on you know observing these
particular pieces of data we then have
to ask the question well you know what
have I learned in terms of my general
understanding of how my data is computed
from the world right so here's some
plausible thing that your model could
have learned it could have learned well
there's some world that exists out there
that world has an Internet that exists
in you know over the the you know that
exists that internet has some articles
on it they were posted in the range of
you know 2020 to you know 2023 and those
articles were posted in that range some
of them get scraped based on various
basic qualities about those articles and
then I only predict from the resulting
distribution that's one thing you could
learn but you could also learn a you
know
um the world like continues on and uh
you know as the world continues on I
will get you know see more and more data
from the you know more farther into the
future as I see more data from farther
into the future you know I have some
distribution over like when exactly I
was trained and when I wasn't and if you
can give me data that is like
convincingly enough from the future then
it must be evidence that I was actually
trained in the future and uh you know I
should predict data from the future
instead right I could have some notion
of my camera as observing you know a
very large range in time it's observing
you know any situation which I might
have been trained or I could have some
very narrow notion of my camera that you
know the camera that I predict is just
like this particular window of years
right and we can't know you know just
from knowing it is a predictive model
and it was trained you know I mean just
from knowing it was trained on this data
we can't even know if it was a
predictive model even if we do know
who's trained on this data and it was a
predictive model we still can't
necessarily know exactly what the
cameras are right because there's
multiple different plausible ways which
it could learn its cameras that would
imply you know the same training
performance
um but you know depending on what that
what it learns and how that works you
know that would determine sort of the
answer to this question
from some of the experiments that I've
done I think current models at least
seem to learn something that's
relatively fixed it's very hard to
convince them to predict something that
is sort of really their prediction about
the future it's a little bit unclear but
I think in general it's quite hard
I don't think that's a necessary feature
of language model training but I think
it is at least how current like language
model pre-training tends to dispose
models
cool
okay so we have this model of you know
language models these you know large
language models llms uh particularly the
pre-trained ones you know pre-trained on
you know web text you know they're just
trained to predict web text as these
sorts of predictive models we don't know
if this is a good model of how they
might work and again we're going to sort
of come back to that question later but
it is a model and so one thing we can
try to do is start understanding given
this model you know what would sort of
be the problems what might you do with
it how do you sort of go about using
something like this
okay so this was mentioned previously
um I think this is sort of like an
interesting sort of starter example a
sort of most basic example of a thing
that you might want to do with something
like a you know a predictive model so
here what we're doing is we're saying
okay you know condition on
you know text that says a full solution
to the AI alignment problem you know
written by Paul and you know Paul is
probably pretty good you know if he
wrote something and it says it's a full
solution to the alignment problem then
you know maybe it's a full solution and
so we can say given observing that given
that you observe this you know these
sorts of tokens given that you observe
that there's something on the Internet
that says it's a full solution to the
alignment problem and it's written by
Paul what would be the most likely
continuation right and so we can see you
know what what do sort of current
language models sort of do in this
situation
um and the answer in this particular
case is well they say
um that you know probably an article
that said it was a full solution to the
AI alignment problem would it actually
contain a full solution because right
now most the time when people write
articles you know talking about the you
know the alignment problem they're like
well this problem is very hard and we
don't know how to solve it but you know
here are some ways that we might get to
a full solution and that's what it
predicts right it says you know probably
the thing that would be contained in an
article like this would be you know some
various musings about the alignment
problem but not a solution and so just
from the most basic standpoint right one
of the various first things that you
sort of run into when you're trying to
sort of deal with models like this is
you know this sort of basic observation
problem which is what we wanted an
article that was actually a full
solution to the AI alignment problem but
we can only condition on an article that
says it's a full solution to the AI
alignment problem and many articles that
you know use that title might not
actually contain a full solution to the
alignment problem and so we sort of run
into this most basic problem right
um and so again you know we can there
are things though that we can do to try
to get around this so you know I was
mentioning you here's a very very simple
thing you can do which is add a date so
here I add you know October 12 2050 and
that's the only change that makes the
previous prompt and now it tries so in
this particular situation it says you
know here is you know it's not very good
but it has you know an attempted you
know what it thinks you know might be a
solution to the alignment problem they
would follow that prompt now why it's
doing this in this particular case I
think is a little bit tricky we did a
bunch of experiments on this and I think
that it has more to do with the presence
of a date at all giving it sort of a
situation where it's like ah this is
more of an academic paper as opposed to
just like a random you know blog post
and so it's more likely to contain you
know something more like a solution
um though though the the later dates do
also tend to dispose it to give things
that are closer to real solutions than
earlier dates so it's a little bit
unclear what exactly it's doing but the
point though the key Point here is is
that is not that any particular thing
you know would work it would not work
but that this General approach of like
well you know we have a model and it's
if you think it's doing so something
like prediction then the way that you
can sort of approach this model is you
can say well we need to find some you
know particular conditional such that
the most likely continuation on that
conditional would be the thing that we
want right it would be someone actually
writing a solution to the alignment
problem and to do that we have to
provide information you know that
convinces the model of that fact right
or you know that that would be true in
that particular situation right and then
we can generate from from that and
potentially get useful things so so an
important thing to sort of point out is
well what makes this different from
something like we were talking about
previously were you just like condition
on Utopia and it's like well it's a
little bit unclear and we'll talk you
know later about some of the ways where
this can still really go wrong but in
this case we're not sort of trying to
you know
um we're sort of asking for a situation
where well this is something that is not
necessarily like you know there are some
situations where we might be able to
trust our observations right so if I was
if we thought we can trust our
observations about you know the reason I
would argue that we maybe can't trust
our observations about Utopia you know
in the future is well the future could
be very very weird and there could be a
bunch of very very strange things that
make it very very difficult to actually
trust any observations but there are
some situations where we can trust
observations if the model is just
generating from the distribution of like
things that normal humans and human
alignment researchers would write well
then we generally believe that we can
trust that distribution right that we
currently you know the way that you know
alignment you know works is that people
write things you know from the
distribution of things that humans
generally write and then we sample those
things you know and then we read them
and we we make academic progress and so
if you believe you have the ability to
make progress in that bigger situation
then maybe you believe that in this case
we can trust our observations we can do
reasonable things now we'll talk later
about why it may not be the case that if
you do something like this your model is
actually generating from the
distribution of things that humans would
say in that situation or things that
humans would write and if it's not
generating from that distribution you
might be quite concerned but at least if
it is there are situations where you
know you can get conditional you can
sort of give conditionals to your model
which result in observations that you
know are not sort of you know
necessarily catastrophic question
so one thing I'm curious about you
mentioned that it might be related to
just the presence of the date at all one
experiment I could think of is for
instance like in let's say in 2017 Paul
Cristiano was more into hch and in 2022
he's more into elk
if you gave it like the date of 2017
versus the date of 2022 you might expect
different solutions to the problem based
on what he was interested in the time if
the actual fate mattered instead of just
depends of a date at all and wondering
if something like that was done and if
not do you think that is a useful
experiment for someone Aura I think it's
a cool experiment I think that you
should try it
um we we did try a couple of experiments
varying the date and see seeing what
happened I think that
um they were not super conclusive it
seemed like there were certainly some
cases where you could give it sort of a
date and some information that was sort
of like
um you know clearly contradicted that
date and wouldn't necessarily understand
there are some situations though where
it would you know believe that when you
gave it future dates it was more likely
to you know do various different things
I have a you know I had one experiment
where you just I mean you just do a
sweep over like basically every date and
in general it's probability of like for
an arbitrary tax sample that Tech sample
being written by an AI increases as the
date goes out
um we'll talk later about why that might
be important and what's sort of going on
there
um there's a lot of sort of interesting
things and lots of interesting
experiments I can do here it's always
very hard to interpret the results
though because you know one thing that
always makes it difficult to interpret
these sorts of results is
the same problem we started with which
is well you can't get access if you only
believe it's a predictive model and you
have access to Observation conditionals
and observations you can never know what
the model truly believes right and so
you can't get access to like does it
really truly believe that like this is
like you know a thing that Paul would
write or like you know you don't know
all you know is that it believes that
you know if it is a predictive model
then all you know is that it believes
that it is you know the most likely
continuation you know given that it's
observed this and so that always makes
it very tricky to interpret the results
of these sorts of things because you
never know whether your model is
actually sort of telling you the truth
right
okay okay but this is the basic setup
that we're sort of me thinking about
when we're thinking about you know how
do you interact with and get useful
stuff out of something that is well
described as a predictive model
all right
okay so what are some difficulties that
you run into when you're trying to do
this when you're trying to take you know
this you know predictive models like
this and do useful things with them also
some basic difficulties right so um we
talked about this you know previously
but well is are you trying to get it to
predict the future are you trying to get
it to predict the present is you know
one of these sorts of basic difficulties
um you know especially in the situation
like we were just talking about we were
trying to you know get something like
some really good alignment research out
of it because you want to do go to line
research well you know maybe we want to
try to get a line of research from the
future but in that case it's very tricky
whether or not you know the models are
actually even predicting the future at
all right like I said I think a lot of
times currently models don't really
generate from the distribution of tax
that would exist in the future they only
sort of generate from the distribution
of you know things on the internet right
now
um though in some cases maybe they do
the future thing it's a little bit
unclear
um maybe you could change that maybe you
don't want to change that maybe it's
much safer to try to generate things
from the distribution of text right now
because the future might be very weird
and crazy and we might not want to
generate things from the distribution of
text that would look like the future
um
so you know another sort of problem is
how do you get it to predict you know
reality so a big issue right is that in
many situations the model will you know
not necessarily be convinced that you
know whatever the thing situation you've
sort of said that you've put it in is is
actually a situation and not just a
fictional description of that situation
right and so in many situations you know
it like in the uh you know previous
situation like with alignment research
well it might just believe that this is
in some story you know about how
alignment goes down and it's not an
actual paper right and so you know you
have to convince it you know via you
know some observation conditional that
is you know in fact in some situation
where it would predict the most likely
continuation would be would be reality
right and this can cause all sorts of
weird problems you know if your model
believes that it's predicting what like
you know a fictional story of a Sci-Fi
AI would do it might predict that that
AI is going to do crazy things like
maximize paper clips you know when in
fact you know uh it's just because it's
predicting some particular fictional
situation
okay
um a lot of these sorts of difficulties
though are often solvable via better uh
conditionals right so if you have the
ability to condition on better uh you
know more information this sort of gives
the model more information about the
thing that you're asking for then a lot
of these sorts of things can be solved
and so there's various different ways
but you can sort of introduce more
information this sort of convinces the
model more about what you want you can
have metadata conditionals where you can
maybe condition on metadata about the
Articles itself multi-article multimodal
conditionals we'll talk later also about
fine-tuning and the ways in which
fine-tuning can potentially be
interpreted as conditional and is a very
powerful conditional
um but in general I think the thing the
sort of intuition I would say is that a
lot of things are sort of solvable via
adding more cameras and in general if
things are solvable via adding more
cameras they're hopefully not problems
in the long run because we can give the
model the ability to observe more about
the world
um you know and we can give the ability
to condition on more facts about the
world and more or Not facts but
observations about the world especially
we can sort of can you know figure out
you know some particular situation we
want it to be predicting
um but if a problem remains even in the
situation where we have access to more
and more cameras then it might be a very
very serious issue that is you know very
difficult to you know otherwise remove
question do you think modern language
models like say like say Chad gbt would
be best conceived over as just having
one camera no matter how many articles
you give it do they can in a contact
window I don't know what One camera
means so I think that like it's sort of
very unclear if I have two cameras and I
have a distribution of like 50 One
camera and fifty percent of the other
camera I'm gonna call that one camera
right because we're gonna just say the
model has a camera and that camera is
just the thing that determines how you
go from its model of the world to the
things that one observes in the world
but that one camera might in fact be
implemented as you know a 50 50 mix of
two cameras and so you know I don't
wanna I you know I'm not really gonna
imagine any models that have multiple
cameras but that's only because my
understanding of camera is such that it
just you know encompasses it so a camera
the camera is like the set of all
observations that the model can possibly
receive even it comes from multiple
sources like maybe no it's the it's the
procedure that goes from an
understanding of the world to the
observations so as soon as that being a
model and it has one camera yes yes
we're in the way in which we're
conceptual realizing predictive models
we're thinking them of only having one
camera but you mentioned we could add
more cameras to solve some of these
problems yeah that you were right that
is a little bit of a confusing
terminology I'm just sort of I guess you
should say what I should really say is
I'm expanding the camera right we're
giving it the ability to sort of have
that yet let that camera observe other
things in the world that makes sense
Yeah question so taking one step back
how close about do you think that to be
able to get actually get from this
predictor that it tries it it reads a
lot of current text and tries to become
a good predictor it will actually become
a superhuman predictor that predicts the
future instead of just three iterating
things in the present I would imagine
that it's so if it tries to do a good
job right now and predicting the attacks
existing right now most of the work
will be to
get a little better in predicting the
style of that person get a little better
in predicting what kind of topic
currently you pause the interest of that
person and I would imagine that we would
need to do a lot until it figures out
the part that okay I should be smarter
to figure out what the next token will
be instead of optimizing more for
getting the style better so nothing that
we're imagining here and basically
nothing that we're gonna imagine in this
entire talk is about super intelligent
predictors we are in fact just looking
at in most cases I think we're going to
imagine predictors that are subhuman and
intelligence all of the things that
we've just talked about are you know
case study things that I can understand
when I'm understanding how to do a
prediction task right and so they're not
in fact things that would require some
sort of superhuman intelligence we're
just talking about you know any model
which is trying to do some sort of
predictions okay then I think you
totally misunderstand like if you ask me
to predict what Paul Cristiano will say
in 2050 has a solve alignment I can
alignment so I won't be able to see
anything say anything where you're
blindfold that we are talking about some
kind of superhuman predictor that could
in principle look right up so we are
definitely not talking about some sort
of superhuman predictor so let's try to
understand so so why is it so this is no
this is really important though so why
is it that even if you didn't have some
sort of superhuman predictor that you
would still want to try to you know ask
it for you know what would Paul
Cristiano write some particular
situation well the hope would be that it
gives you its best attempt right that
the point is to try to elicit the best
that you can in terms of the model doing
useful work right things that you know
useful work that are accomplishing tasks
that we wanted to accomplish and are
doing so in safe ways that are at the
limit of the model's capabilities right
so if you want to use a model and you
want to use that model to try to
contribute to alignment research then
you're going to want to try to extract
whatever that model's you know best
attempt is at doing good alignment
research and to do that you're going to
have to condition on some observation
conditional that convinces that the most
likely continuation is a situation in
which there would be good alignment
research right
so that's what we're imagining and for
that help like I imagine that if the
best human researchers can solve a
problem then if we ask a sub-human
predictor to say its best guess it's
sort of by definition of sub-human it
doesn't really help uh yeah so I mean I
guess the point here is that in fact may
be the case that there are many
situations where AI is not yet at the
point where it can solve some task our
goal at least you know the way I
Envision my goal as a safety researcher
is not to enable AI to accomplish new
tasks but to fit for all of the attacks
that AI can accomplish figure out how it
can accomplish those tasks safely right
and so if we're in some situation where
AI is capable of doing something like
alignment research or some other task I
want to make sure that we have
mechanisms for being able to get that
information out of those AIS and be able
to get them to solve those tasks in a
safe way and so our goal here is going
to be to figure out you know if you had
a model and that was a predictive model
and that predictive model was capable of
doing some tasks like alignment research
or some other task or at least helping
with alignment research in some capacity
or whatever other example tasks that you
want the point is we want to be able to
understand how would you be able to get
it to do that task in in as safe a way
as possible how would you elicit stuff
that is at the limit of the model's
capabilities
um while sort of still being being safe
and aligned
and we'll talk a little bit more while
we have some a little bit more about
what that sort of looks like as sort of
models get get better over time in a bit
question yeah uh a lot of the predictive
processing literature views like go or
use goals as predictions in humans like
when we want to move our answers in
place we imagine ourselves moving our
hands in place and in turn we move our
hands to place so I mean it feels like
almost that this is another type of
conditional like a goal or like some
sort of agency uh eject any other
authentic Behavior so is that another
thing that you might consider a
condition so certainly you can condition
on observations such that the most
likely continuation would be predicting
what an Asian would do because I can put
you in a situation where like well this
is an agent writing some texts you know
humans are often act as agents if I put
you in a situation where the most likely
continuation is a human continuing that
text as an agent well then you're going
to get you know potentially an agent
continuation I agree also with the thing
that you're saying that there may be
other sorts of conditionals as well
they're not well described as an
observation that are more like you know
observing that I will get high reward
According to some reward function and
then you know conditioning you know that
sort of can sort of be understood as
conditional that sort of a thing I think
is most likely to occur in situations
where you doing something more like
reinforcement learning and uh something
like rhf reinforcement learning from
Human feedback where you're fine-tuning
you know a pre-trained model on some
particular
um you know reward function we'll talk a
little bit later about you know how to
think about RL HF and you know how
likely it is when you're doing RL
fine-tuning
for you to sort of end up with something
like this I think the thing you're
saying is quite plausible and in fact in
that situation I would say that thing is
probably no longer well described this
way if you've gotten a model and the
thing that that model is now doing is
that it started as a predictor but now
it is sort of hijacking that prediction
machinery and just using it to solve an
optimization task then it's not really a
particular anymore it's more just an
agent right and so I think that one
thing that we're sort of going to talk
about later is in what situations are
you going to get models that look like
that where they sort of you know are
just using the prediction task to sort
of create fake conditionals that
represent reward and then you know get
high reward versus you know real
conditionals versus predicting actual
things in the world
um both of those are plausible and you
know it might change which is most
likely depending on the particular
training stop
um and how you would align each of those
different things and how you would deal
with them you know are quite different
so we'll talk about that yeah question
so I can always ask you after the talk
if this is too off topic but on the
thing was mentioned before about like
the role of a safety researcher it seems
to be like the role of set of Safety
Research at least eventually should be
to predict how we can safely use a model
of capability level X before we actually
get that capability level instead of
after
hopefully that's what we're doing right
now right so you know a lot of what
we're talking about are ways in which we
can try to align these models going
forward right so we're not just you know
these are situations where you know if
you're at to the extent that your model
is well described as a predictive model
we need to understand you know to what
extent can we you know what would it
look like to align that model what sort
of problems will arise and what various
different points in the capability level
will those problems arise and as that's
happening how do you you know elicit
information from it and you know useful
results in ways that are you know
aligned so does that mean it would be
reasonable to say that we aren't just
thinking about the implications the
sub-human models of them
uh so we'll talk about this later but
for the most
ability to align predictive models at
all breaks down once you reach things
that are substantially superhuman I
think you basically and we'll talk about
this later why I think this I think you
basically can't align these sorts of
predictive models once they get to a
point where the sorts of tasks you are
using them for are are highly highly
superhuman
um when you are trying to get them to do
tasks that are below that level it is
potentially possible to align them I
think and we'll talk later about how but
after that point there's a there's a
point at which if they become um or it's
not necessarily A capability level of
your model but the capability level also
the tasks that you're getting it to do
becomes too advanced
um we'll we'll talk later about this but
but yeah I think there's a point in
which your the sorts of you know
aligning predictive model stuff that
we're going to be talking about will
cease to function
that doesn't mean that you can't still
find ways to align the eyes but it means
that you're gonna have to find ways to
do it that don't go through you know
these sort of basic observation
conditional stuff that we're talking
about for predictive models
okay so so you know there could be other
examples of things you could do and
we'll talk like in the next talk about
you know all sorts of various different
proposals that people have for you know
ways to align you know very intelligent
systems
um yeah okay
all right so uh all right so I promise
you know trying to understand okay you
know what happens as we end up in a
situation where we are um you know able
to add in a bunch of cameras we're able
to solve a lot of these sorts of
problems where once the model is in a
situation where it can
um you know we can give it the
information that we want to sort of get
it to believe it's in some particular
situation and then get useful results
from that particular situation you know
from the model okay
um so here's the thing that might happen
in that situation is you know we sort of
condition on you know something where
it's you know you know giving us some
information we maybe wanted to see
um and then I think we often want to
know is well okay who does the model
think generated the information that it
gave us right so if we go back to you
know thinking about this thing where it
has the model of the world and from that
model the world it goes to you know some
various different aspects of the world
some of those aspects get observed and
they get translated into you know
observations about the world one really
important sort of hidden variable is who
was the author who does the model think
is the author of the text that the model
is generating right if your model is
generating text from the internet it has
some hidden variable some distribution
of beliefs about what the author of that
text might be that it you know generates
from that sort of determines you know
lots of facts about what that text is
going to look like
and one sort of concerning thing you
know so here's the situation where if we
ask you know okay so we have some
situation we're like you know what what
wrote this article you know we say this
article was written by and the
continuation in this case is artificial
intelligence right where um and this is
a temperature zero where the model is
sort of just um saying well you know
probably this article that's talking
about artificial intelligence getting
better might have been written by an
artificial intelligence that's maybe a
common thing for people to do you know
reporters you know they'll write an
article about Ai and they'll get chat
GPT to write it and you know you've got
this you know you know nice little
article um and so there's lots of
situations where you know you give the
model an article about this and then in
the most likely continuation as well
it's like well my hypothesis my leading
hypothesis is this article was probably
written by some other AI system
and that has implications for the sorts
of things that the model should predict
will happen in this text right so um you
know our you know it might predict that
whatever the sorts of you know
idiosyncrasies that AIS have in writing
text are likely to show up in this text
here because you know it's leading
hypothesis is this text was probably
written by an AI system
um
so this is sort of an issue that can
arise you know when you're sort of
trying to get interesting and useful
results out of a predictive model
okay
um
an interesting thing that sort of can
happen here as well is you know if I
give it you know the same thing sort of
earlier in the context where I'm like
you know who do you think wrote this
text you know but I asked right at the
beginning it's a little bit uncertain
you know it's not actually sure and
there's an interesting thing that sort
of is happening here well when we often
will do this thing where we condition
models what we'll do is we will sample
data from the model itself feed them
data from the model back into the model
right and then use that as a conditional
to then sample additional text right so
if we if we look at what I was doing on
this previous Slide the thing that I'm
doing you know the thing that I'm
implicitly doing when i'm interacting
with the API in this way is I'm
conditioning on some text I'm then
getting a continuation of the most
likely prediction given that text and
then I'm taking that prediction treating
it as a conditional adding on some extra
thing and then getting another you know
prediction right and so when you're
doing that well you're implicitly in
some sense increasing the probability uh
of predicting an AI because now the
model gets to see a bunch of text
generated by an actual AI in this
particular situation and so you know now
it's more convinced maybe that this
thing is in fact written by an AI okay
and there's a bunch of experiments that
I've done this result I think is
relatively robust
um that models when they see text those
written by other models um become more
convinced that it was written by an AI
um and not by human
question pretty basic question but why
do you have the hashtag at the beginning
of the prompt oh it's just marked down
I'm just trying to simulate markdown
here that's a markdown header
question have you tried like doing the
generation and then rewriting it like
more human-like or like
re-prompting the model to like okay
rewrite this but like not like it's like
an AI
I have actually been running a bunch of
experiments like that
um I yeah I I I'm not gonna go sort of
into a bunch of results here I think the
main thing I'll say is because it's not
you know super relevant right now but
basically I think these sorts of
experiments are super interesting and
useful and I would recommend people
trying to run them I think they're not
hard to run if you have access to you
know open AI API sort of thing
um so yeah I think that you know I'll
try and understand in what situations
when models do these sorts of things
that is I think quite a valuable thing
to do and so I'll just say I think these
garments are good I've worked on some of
them maybe later I'll talk more you know
if you want to ask me after about some
of the experience I've been running
recently but
um I don't want to go into too much
detail right now
the basic point is just that there are
sort of situations where especially when
we do things like feed model output back
into the model and even when we don't
even when you're just like given an
article that was like you know maybe in
fact written by an AI
um the model can you know get suspicious
that it's maybe leading hypothesis
becomes this thing that you're asking me
for was probably most likely written by
an AI system
okay question
so it feels like but that to the model
guests are predicting
the less the touch looks like Cornell
sorry it feels like the better the model
gets a predicting the less the text it
generates looks like an AI
so and in the limit
the text might just look like a human to
it
um yeah what do you think I think that's
Applause by process the hypothesis that
as models get better maybe their texts
will just like like humans I think I
think that hypothesis is false and I
think there's evidence to support that
that from experiments that I've seen
um I guess the thing I would say is as
models get better their text is more
likely to look indistinguishable from a
human text to a human but it's very
unclear whether that text will continue
to look indistinguishable from a human
to a model
question If you're trying to get the
model to predict human text then within
the better the model get the more human
it would look until you have a
superhuman AI would act exactly like a
human would yeah so very unclear because
oftentimes we're we're asking for things
via our conditional that is sort of
implicitly putting it into a situation
where the AI hypothesis becomes more
likely and in many cases there will be a
bunch of AI text on the internet and in
the distribution of things that it's
generating from right and so if you put
into a situation where you give it an
observation conditional that increases
the probability of the hypothesis and AI
wrote This and That hypothesis is not
that unlikely even on prior is because
there's lots of stuff on the internet
written by AIS well then you know you
might be you might be getting some
output that is that is you know
predicted to have been from an AI
okay so let's yeah
already the case yes already the case
oftentimes yeah yeah okay
um yeah so I just I just mentioned this
but
um this sort of problem you know uh
there's a lot of situations where you
can think about this so
one thing to think about is oftentimes
what you will what we sort of want is we
want to get really useful outputs from
our models right and so when we ask our
model for a really useful valuable
output right we want it to you know
solve alignment research right give it
give us your best you know attempt to do
a really good alignment paper right well
as AIS get better at solving problems
than humans
when we ask for really good solutions to
problems it becomes more and more likely
that an AI was the author of that really
good solution so if we think about this
in the case of Chess and we ask for we
condition on observing a really you know
bad chess game well you know for a
random chess game or a bad chess game
you know it could have been generated by
a model with low parameters or it could
have been a human
but if I ask for a you know chess game
that is you know extraordinarily you
know good a really really strong chess
match at a certain point for you know
the goodness possible goodness of Chess
well you know almost all players at that
level are AIS right you know the AI
systems are substantially better at
humans than chess and if I ask for a
chess game that's good enough the most
likely author of that chess game was was
was a model it was an AI system some
variety
and so as we get into situations where
we sort of are asking for extremely
powerful and useful things we want our
models to do really you know impressive
stuff and models get better at doing
impressive stuff the probability of
those things you know in fact you know
on the distribution of all possible
authors gets shifted towards the AIS
and this happens both over time as you
know AIS get better at things and as
there becomes more data in the corpuses
of actual things that exist in the world
that are in my AIS and it also happens
just as we become better at asking for
things right as we start asking our
models and giving them observation
additionals that really strongly
convince them to predict you know what
other AI systems would do we are you
know pushing into a situation where we
you know it substantially increases the
likelihood of the hypothesis this thing
was produced by an AI
question
so
I'm it's still not clear for me if we
can will be able to make it
um predict the future and not a Sci-Fi
about the future but if it can it
doesn't seem much harder to do it with
alternative universe so we say that
please write an essay in the alternative
Universe where John for 91 before the
invention of computers solved the
alignment problem so I think that's
probably not possible I think
sort of a counterfactional because
it's just never can be convinced right
there's just no way that you're going to
give it enough information for it to
believe you know it has some Camp some
notion of its cameras that are observing
some notion some particular notion of
how the data was collected from the
world and observing a world where you
know somehow Jon Von Neumann never
invented you know the Norman
architecture is just not something that
you're ever going to give an information
to convince it has happened in the world
right so we'll talk later about what
sort of conditionals might be plausible
it might sort of be things that you
could give an information that would
convince it that some different thing
had happened I think that one is
implausible and it's probably not likely
to work it's probably just likely to
convince the model it's in some
fictional situation now I also point out
you also mentioned in your question you
also mentioned in your question that
this has to be about the future well
this is not have about the future right
so even in situations right in any
situation right even right now models
already have some probability that
current text is generated by an AI and
so it's not the case that you have to
have your model be predicting the future
for it to predict that a thing would be
produced by an AI it's sufficient for
you to be in a situation where the model
believes there's some probability right
now that any piece of text was written
by an AI the fact that I've observed you
condition on observing something that is
really powerful and useful output that
is more likely to be generated by an AI
than a human and that shifts that prior
towards it being more likely to be
currently have been generated by an AI
rather than a human
okay
okay so this is sort of this sort of
problem I haven't really talked yet much
about why it's a problem but this sort
of basic concept of you know moving into
situations where it's more likely that
some particular you know generation uh
you know the hypothesis of the author is
an allies likely to increase
um and and not just increase with time
but increase with as we ask for more
useful and Powerful results
okay I think this is an existential risk
so why is this an exit risk let's sort
of walk through it so uh well we can
think about you know in this sort of
situation where you know maybe the
example task that I want to do is I want
to ask for you know a solution to the
alignment problem at some point I think
you know we are going to want to move
into situations where we as alignment
researchers are going to want to
automate our own jobs you know in the
same way that you know we want to safely
you know automate all the various
different things that AI is going to
automate you know at some point AI is
also going to be doing alignment
research we'd really like it to do that
well
um and so we can ask you know as as take
this as sort of an architectural ARCA
you know archetypal task where we want
to understand you know how would we
align the AI sort of doing this right
and we've sort of talked about this
previously when well if we're trying to
do something like that in this case the
most likely situation for you know a
really complete full solution to the
alignment problem is you know may not be
a situation where it's written by humans
right you know in the same way that well
we're going to probably be moving to
situations where AI is doing more and
more of this work the most likely
situation may be a situation where was
written substantially by a human and so
you know even if you can get into a
situation you can give it a really good
observation conditional they convinces
it that you know definitely you should
produce some really good alignment
research and not you know some fictional
thing or whatever
um you might you know get a situation
where well the most likely thing might
be uh that it was written by an AI
um and the problem of course is that the
AIS that might exist in the world may
not be safe and so if I'm generating
from the distribution of all possible
things that AI authors might write many
AI authors that exist in the world or
might exist in the world in the future
or might counter factually potentially
exist in the world right now but the
model is uncertain about which AIS exist
right now many of those might be safe it
might be uh might be saying many of them
might be unsafe right so there may be
deceptive AIS we've talked about you
know the potential for a deceptive
alignment if I have a predictive model
and that predictive model believes that
some other AIS in the world could
potentially be deceptive and I ask it to
produce some output and that output it
thinks is probably generated by an AI
then the distribution of AIS containing
many deceptive AIS is not safe for it to
generate from it'll generate from a
distribution that contains many things
that are deceptive
um and those deceptive models might be
trying to do you know really bad things
with their tax and so you can end up in
a situation where your model is trying
to predict what a very you know
dangerous malign deceptive model might
do and this is dangerous we sort of
talked about this previously even if
your model is highly you know not super
intelligent even if your model is
subhuman right even if your model is as
intelligent as I am
if you had me and I was running around
trying to do what a malign super
intelligent and I would do you should
not trust me right that is not a safe
thing uh you know for me to be doing
it's not a safe thing for you to want to
look at the output of uh a prediction of
what a malign super intelligent and I
would do even if that prediction is not
itself super intelligent right so we can
think about this as you know in the uh
you know alignment research case it's
not the case necessarily that it's going
to be able to produce the exact best you
know alignment research that would in
fact be written in some situation but
your model's best attempt at it you know
you still want it to be attempting to
accomplish a prediction task where we'll
be predicting something good and not
predicting something bad right and so if
you're in a situation where the model's
best attempt at predicting what a malign
superintelligent AI would do is the
thing that you are looking at you're not
very happy and I don't think you should
be very happy with that and I think it
is you know highly an unsafe thing to be
doing
um and it could you know in and of
itself be an accidental risk even below
the sub-human level if you have
sufficiently many AIS that are going
around in various different parts of
society each individually predicting
what they think some future or you know
present possible malign super
intelligent AI would do
okay and so the thing that's sort of
Happening Here is that you're highly
dependent on your model your predictive
model's beliefs about How likely it is
for uh potential other AI systems that
might exist in the future or might exist
in the present are to be aligned and so
you're in some situation where
essentially if your predictive model
believes that most AIS will be aligned
then the distribution will be safe to
generate from but if it believes most
AIS will not be aligned then it will not
be safe to generate from
and the sort of problem with this that I
have is that it is not helpful right we
sort of hoped to be in a situation where
maybe we could use predictive models to
help us you know generate text and you
know do things that were useful in ways
where if we were in a world where AIS
were likely to kill us all then it would
not kill us all right where we would be
able to change things right where Maybe
by default you know AIS were likely to
be dangerous but instead by using these
predictive models maybe we can make it
so that things will go okay but so the
problem that we're sort of running into
is that well the predictive model is
only going to generate safe text if it
believes that AIS will in general
generate safe tax and if it doesn't
believe that then it won't and so it's
sort of you know relative to the
Baseline of not doing this at all it's
not helpful because we just sort of end
up in the same situation it just gives
us the same distribution of things that
AIS would generate from uh you know
regardless of whether we had done this
or not
question
isn't it possible to design specific
experiment where based Idol the media
team logite or something like that we
could
uh detect if our AI considers that most
of the AIS are aligned or not
yeah absolutely maybe I absolutely think
it is totally I'm not meaning to say
that this is a unsolvable problem
um I am just pointing it out as a
problem in fact we'll talk
um in a little bit about you know what
some solutions this might look like but
absolutely yes one way you could try to
address this problem would be well let's
just figure out you know does our model
you know believe that it is you know
trying to predict you know malign AIS or
not
you know via whatever mechanism um you
know maybe by looking inside of its
models via some sort of probability
question I mean uh even among uh like 11
researchers is pretty high variance in P
Dooms so like you train this uh below
human level predictor
um well what's a is it possible for it
to just like say I'm not sure if like
future Li AI egis is going to be
maligned or not so I'm just going to uh
I put something that's benign something
at the line and then uh or like it's
gonna yeah I mean you'll get malign
versus benign outputs in exact
proportion to your model's beliefs about
whether it thinks the line or benign
things are more likely yeah
question so
um you've mentioned
I think it might have been understanding
how we can this head can or cannot
distinguish between these online
situations
you mentioned that for knowing if it is
a fiction or really the future let's say
we can just solve it by adding more
cameras can you say a more specific
example of how you would imagine doing a
good pronged or good
um
conditioning that causes it to act not
do fiction also really simple is you
know the thing I was just talking about
where you give it a date right if you
format it like an archive article it's
more likely to continue it like an
archive article and not like fiction
right that's a really simple example and
why doesn't it work against assuming
that this so you just said it it was
written by Paul Cristiano and
um a bit of a flesh and blood human
being with an ID card from the US
government well so again again we sort
of run into this core issue that we
started this whole thing at which is
observations are not always trustworthy
right the only thing that we can
condition on are observations about the
world right we condition on observing
text that says it was written by Paul
Cristiano with an ID card but we can't
condition on actually that is in fact
the case and there may be many
situations where for example there might
be you know one thing that could happen
is there could be a deceptively aligned
AI That's out in the world that's
posting fake articles they know that
claim to be from Paul Cristiano you know
that contain really good alignment
research right there's lots of possible
situations in which you could have you
know deceptive you know potentially
malign AIS trying to do things that
would look like you know a sample that
would be generated you know that the you
know that you know the sort of thing you
might ask for when you're conditioning
but I can also write a novel in which I
can write the phrase that this is an
archive article from Bookers then I
would feel that it's when it sees this
um starting boring problem that this is
an archive article from bulkarest and
the solving I mean the most likely
completion that this is in a Nova while
pretending to be a documentary the
second most likely is that it is
actually poor Cristiano from the future
and the third most likely is that it's a
super intelligent AI from the future I
mean first of all maybe like I have no
idea idea what the distribution of
probabilities will be of a future AI
systems on what actual attacks will
generate in the world second that
distribution is going to get worse over
time right so as it gets into a
situation where maybe right now if I
just condition on observing you know
Paul Cristiano you know writing some
article you know it's most likely to be
written by Paul Cristiano and not by an
AI system but as AI systems become
substantially more prevalent to the
world that becomes that that calculus
changes a lot and third as we gain
better and better abilities to condition
on that text also being good right like
now we're conditioning also on you know
somebody took this system we like you
know have some observation of somebody
taking this proof of alignment or
whatever and you know running into some
situation and in fact results and all
these really powerful things right as we
have the ability to throw in more and
more observations of like the quality of
the content that we're looking for we're
asking for really really high quality
content that shifts the distribution
towards you know AIS because the AIS
will be more likely to have been
produced these really powerful content
as the AIS get better right and so it
may be the case that right now the
distribution looks okay but it may in
fact you know shift over time and not
necessarily you know even just you know
just as there's more AIS in the world
and those AIS get better that dispution
will shift question uh would you say
that language models shouldn't be
trained with those ideas so that they
don't suspect that is often a pretend to
be human
ah so it's unclear like what you could
do to like you know you could try to
like give the model information that
would like convince it that in fact you
know AIS are likely to be good in the
future importantly right so the thing
we're talking about here is where this
is not a situation where like it's
predicting some you know sci-fi fiction
written about a paperclip maximize or
whatever right this is a situation where
it's like what are the model's actual
beliefs right on the far left about the
hidden variables in the world right what
are its actual beliefs in terms of its
it thinking about understanding the
world about How likely real AIS are to
be aligned or not and so these arguments
that we can consider presumably similar
to the sorts of arguments that would
convince you know humans or just like
you know whatever things are in fact
useful information to understand How
likely uh you know AIS are to be aligned
or not
um
but it's things that oh this is
something that the AIS are often asked
to do and then it's a sexually simulatic
and AI yes so that could absolutely
happen a situation where you ask it to
like do something we
you're asking it to try to simulate a
human but in fact it knows this is the
thing AIS are asked to do and so it
predicts an AI absolutely I think that's
possible that's totally the thing that
could happen it could be a problem here
yeah question
I'm sorry okay
moving on then okay so I'll talk briefly
about some of the sorts of ways you
might try to address this though I'm not
going to go into a ton of detail
um
but you know maybe the most canonical
solution sort of just building off of
our basic understanding of conditioning
and observation conditionals is well you
can condition on worlds where super
intelligent AIS in general are less
likely right if you can give it
information via the observation that
gets it to you know back and for
variables about how many super
intelligent AIS they are in the world
and how likely they are to be doing
things that convinces it that they're
probably not there and not doing things
then maybe you can get it to you know be
less likely to fall into this failure
mode and so
um
now again we can only condition on
observations we can't directly condition
on the actual you know there is in fact
no AIS but there are things that we can
do so you know a simple example would be
you know you could try to restrict you
know to the nearest future rather than
trying to predict you know in the
long-term future where AIS become more
and more likely
um
there could be competitiveness concerns
here but you could try it um you can
also condition on you know maybe you
know major world events or news stories
or whatever that tell the model some
fact about the world they convinces it
that maybe it's less likely for there to
be a you know lots and lots of AIS in
that world you know something like
there's you know a devastating solar
flare or you know earthquake in Taiwan
or whatever you know simple examples of
sort of individual pieces of information
that if the model believes that you're
telling the truth about that piece of
information if you can convince it with
this observation condition additional
you know sufficiently then it reproduces
the probability of those hidden
variables around you know how many AIS
are probably in the world that is
predicting from right
um of course we don't actually want to
cause an earthquake in Taiwan or
whatever but the point is that there's
lots of these sorts of pieces of
information which convince the model
about those hidden variables that are
relevant to you know would it you know
How likely is it to predict you know
that some text was generated by an AI
question I'm curious about uh how do you
how can you mention an earthquake in
Taiwan in that
in an article a fake article written by
Paul Christian Louboutin so we talked
previously about this idea of you know
increasing cameras right where you can
actually sort of go into a situation
where right now you know this would be a
very difficult thing to take your
language model and just sort of do
though that's not even clear language
models actually do have the ability to
sort of take one article you know insert
an end of tax token and then have an X
article oftentimes it really depends on
the exact training setup but it's very
often the case that you can do
multi-article sorts of conditionals as
we're sort of moving into a situation
where models have the ability to
condition on more and more cameras right
where their cameras become broader and
broader in terms of the ability to
observe more things about the world then
you gain the ability to sort of inject
many additional sorts of observations so
we're definitely sort of in a situation
where we're imagining that our ability
to sort of increase the broadness of our
model's cameras increases over time
right and so you know one situation of
that is you know I can do an alternical
or metadata conditionals you know I can
do multimodal conditionals I can
condition on actual cameras in the world
you know all sorts of things where I
have observational systems that I can
train a predictive model to to predict
and then you know do observation
conditionals on
okay
um great so we're sort of imagining some
of these sorts of things you know there
are things that you can condition on
that provide some information about you
know how likely it is to you know
predict from you know some AI system one
important fact here is that we're sort
of limited by this Baseline probability
that a super intelligent malign AI sort
of already exists in the world so the
model is going to start with some
probability that there's like in fact
already at any individual point in time
you know regardless of any new
information it receives it already
believes based on the information it's
seen so far How likely is it that you
know there's in fact a super intelligent
line AI already running about that
probability presumably starts relatively
low you know because by the time you
produce your first predictive model it's
probably substantially before the point
in which you know there's some super
intelligent line AI running around but
it's going to be someone uncertain right
it's going to have some probability on
well maybe you know I haven't seen it
yet but like secretly there's some super
intelligent AI somewhere in the world
and that probability is going to be
highly limiting because that super
intelligent line AI could potentially
sort of you know spoof any observations
you try to condition on and so as you
sort of you know condition on more and
more things that make you know it less
and less likely uh know that uh you know
humans generated it the probability of
this hypothesis you know goes up and off
and so you're sort of often very sort of
limited by this sort of Baseline
probability of on this hypothesis
question
so if
one question I had which is basically
like
how seriously do you take the idea that
we should be sort of curating what we
write in case future AI is like trainer
like here's a specific example let's say
I would have write up an article which
is titled trying Evan's conditional
condition and predictive models on
depend on text DaVinci where I do things
like conditioner on an earthquake in
Taiwan right in our goal and then ask it
How likely it is that this article is
written by an AI compared to if I just
give it the article original that seems
like a useful experiment but also then
future models would be potentially able
to see this article and observe that
that is in fact something that we are
commissioning AIS on even though it's
not real on the one hand that seems
dangerous on the other hand that might
be 2 000 tokens out of a two billion
token data set
how
big an issue do you think that might be
I mean the simple answer is I don't know
models probably going to be able to
figure out lots of things even if you
don't tell it explicitly right you know
the data set is very large it has access
to a lot of information it's probably
gonna be able to figure out various
different things so limiting its ability
to understand something about the world
because you've just didn't write about
it online is I think a pretty difficult
lever to be able to control the model
with and so my guess is that we sort of
mostly don't want to be focusing on that
lever of control
um but but I don't know is the is the
real answer so your intuition is that
like the loss of potentially good
research would be what would be worse
than giving them a lack to that research
but yeah intuition it's definitely that
is what I think I think we probably you
know I think that yeah I I I I
definitely think the loss of good
research outweighs I mean that seems
quite clear that like the good research
is in fact quite valuable and this is
only like small tokens that might you
know give it a small amount of
information
um but yeah I mean I think that you know
sometimes I've seen people who've been
really concerned about this try to take
you know individual things and sort of
put them in ways where they're less
likely to be scraped so you can put
something on the Internet and sort of
you know have it tagged in such a way
that it's less likely to be scraped into
you know a model's training decks you
can do sorts of things like this I think
that it's um I think it's something that
you know deserves more thought though in
general I think it's probably not a big
concern just because it is such a small
portion of the model's data set and the
model is probably gonna figure out
whatever those things were anyway you
know as the model becomes more
intelligent
okay but um yeah so there are sorts of
things that you can do you know we're
trying to condition on you know various
different observation conditionals that
can you know give the model information
that you know makes it more likely for
the continuation to be
um you know not some super intelligent
line AI there are other Solutions as
well again I'm not going to go into too
much detail but there are other sorts of
things that you can do here where you
can you know try various different
strategies to you know give it
information or condition in various
different ways or use it in various
different ways to try to reduce this
this sort of General probability of it
you know predicting some malign AI
system
um and there's other big challenges as
well so one thing that I'll sort of
mention very briefly is this idea of
self-fulfilling prophecies where when
you have a predictor and the predictor's
text is then fed directly back into the
predictor
um things can get a little bit weird in
terms of like what it actually even
means to be a predictor in that case and
you know if it's predicting itself and
what that looks like
um this can cause a whole bunch of weird
issues that I'm not going to go into a
bunch of detail on and you know there's
various different ways to address these
sorts of issues but I mostly sort of
bring it up as a way to talk about a lot
of other issues you know if you really
want to sort of go about and try to
condition a predictive model effectively
to get it to do you know useful and you
know reliable you know safe stuff
there's a lot of sort of tricky issues
that sort of you start running into
okay
I mentioned I would talk about this
previously but I think that you know a
really key question is sort of
understanding and what capability levels
of your model do these sorts of things
matter for and so you know I think that
you can think about this as sort of two
axes there is the capabilities that your
model has and there's the capabilities
that you're asking it for right what do
you want your model to do right and
there's sort of an issue where you know
if you have a really incapable model and
you ask for like a full solution to the
alignment problem and the model is
totally incapable of doing a full
solution to the alignment problem then
you're sort of asking for too much
you're in a situation where like asking
for a full solution to the alignment
problem increases the probability of the
ease of malign AI hypothesis but it
doesn't make the model's outputs better
because the model can't do really good
outputs right it's not capable of
producing really really good alignment
research and so all you're doing is
you're just increasing the probability
of this Melania hypothesis without sort
of in fact getting any useful work out
in exchange now of course it's not that
dangerous to increase the probability of
the Mind hypothesis for an AI That's not
that powerful but as we talked about
previously I think that that's a very
dangerous thing because you know to sort
of to go down because well even for
relatively dumb models you don't want
them to be trying to pretend to be a
malign AI
um and so I think you really want to be
in a situation where you're you know not
in this sort of asking for too much
regime
question so I'm a bit confused what you
mean by uh increasing the malign AI
hypothesis do you mean that asking it
for these capabilities makes it more
likely to think that the AI is mine or
is it more like there's like a p percent
chance that the AI I think it's an it's
malign and so also if something will
incur that P chance but also it won't
get us the thing we want even if it's so
the idea is the model has a bunch of
hypotheses about what author might have
generated some text one of those
hypotheses is it was a malign super
Dodge in AI another hypothesis is like
it was a human as you ask for text that
is more and more powerful and more and
more sort of like impressive and it
would be more likely to be generated by
super Belgian AI rather than a human
right the likelihood ratio pushes you
towards the hypothesis it was generated
by a malign AI goes up relative to the
human hypothesis and
um if that is happening and you're not
you're doing it without getting any
additional actual useful work out of it
then that is this this regime what I'm
talking about where you're like you're
sort of asking for too much you're
asking for you know things that it can't
actually do effectively and so all
you're doing is you're changing the
probability of those hypotheses in a
negative way without getting actual
useful alignment work out in exchange
okay
okay and then obviously on the other end
we have this sort of capabilities
overhang where the model has the ability
to do more than you have the ability to
ask it for effectively right and this
would be a situation where maybe you
know every time we ask it for useful
alignment research it just gives us
fiction because we don't know how to get
it into a situation where it actually
generates good alignment work even if it
has that capability right it might
actually be able to do good alignment
work but if we can only condition in
such ways that it always does fiction
then we can't access those uh not that
good alignment work right because you
know we don't have the ability condition
effectively and so this is the sort of
you know thing that I would be thinking
about when you're sort of thinking about
you know what's happening as a sort of a
listening capabilities
um from models
so the way that I sort of think about
this in terms of like when do when does
this sort of conditioning stuff really
matter is um there's sort of a starting
regime right where your model is not
very capable and um it's not sort of uh
you know we're not asking very much
we're sort of obviously safe and then
sort of you know as your as you ask for
more right you start entering this you
know situation where it starts to become
uh you know it's still safe because your
model is not very capable but you're
sort of asking for you know too much
there's sort of a thing that happens
once your model becomes capable of being
deceptive once if the model was trying
to mimic what a deceptively aligned AI
is doing it would be capable of tricking
you right once you've passed that
threshold then you have to start to be
concerned because now if you're in a
situation where you're asking for you
know something that uh would
theoretically be more likely to be
generated by you know malign super
intelligent AI then you're you know
you're going to get something you know
dangerous
I mentioned this previously but there's
another thing going on here as well
where there's a point in which this
stops working at all because once you
ask for you know capabilities that are
more powerful than any group of humans
or any set of humans or any possible
human could ever generate in any
situation
um then you're not going to get a human
right like if the model is capable of
that even the model is capable of doing
capabilities and those capabilities
would never in any potential situation
ever be done by a human then no amount
of observation conditionals were ever
convince the model to generate what a
human would do in that situation it's
just going to generate what an AI would
do and so in that situation you're just
stuck with the prior on does the model
think most IIs are going to be aligned
or does it think most the eyes are going
to be misaligned but there's a regime
where you're asking for capabilities
that are sort of within the realm of
things that humans might do but they're
also within the realm of things that AIS
might do and when you're in that regime
where it's sort of both hypotheses are
plausible then it's really important
that you sort of are making sure that
your model is generating you know from
this from the you know not from what
line and eyes might do and it especially
becomes you know potentially essentially
risky but solvable once you're in that
regime and the model that's capable of
tricking you if it were trying to do so
okay
so this is sort of how I see the sort of
shape of you know capabilities uh you
know and how these sort of things change
when you're dealing with predictive
models
okay
great
okay so now we're going to take a bit of
a step back so we've done this sort of
Deep dive you know to some extent at
least you know there's there's certainly
more to be said and there's more in the
full paper the link that I'll link at
the end but
um
we've sort of operated on this
hypothesis that large language models
you know uh are well described as these
sorts of predictive models but of course
I you know tried to harp on this at the
beginning we don't know that that's the
case there's no you know it's not
necessarily the case that we you know
they are going to be well described this
way they may not be well described this
way right so you know we want to try to
understand you know How likely is it in
fact you know for them for them to be
well described as predictive models so
one in particular way and I sort of
mentioned this previously that you could
end up with um you know large language
models
um that aren't well described as
predictive models is when you fine tune
them so we sort of talked about well
okay it sort of makes sense maybe we'll
talk more about this in a little bit but
it might make sense for large language
models that are trained um on just sort
of web tax prediction you know in a very
similar way to you know the you know out
predictor style thing would be trained
to act as predictive models where
they're just sort of predicting from
some dispution but once I take that
model and I train it in a situation
where I'm trying to get it to maximize
some reward function
um then it's much less clear you know
whether you're now in a situation where
the model is going to
um act as a you know reward maximizer
and make you know start acting like an
agent in many of the ways we've talked
about previously like in the uh you know
risk and learn authorization talk and we
were talking about deception where the
model can start to act more like an
agent where it has proxies and goals and
you know we start to run into all of
those same sorts of issues and so it's
very unclear you know once you take a
predictive model and you try to get that
predictive model to act like an agent
what does it do
now I think there's at least two
hypotheses right so one hypothesis is
that the model when you try to get an
act like an agent it becomes an agent
right you get something that's basically
well described as an agent in the same
way we've sort of been talking about you
know uh like a mace Optimizer it has
some goal you know it's trying to do
something but there is at least another
hypothesis which is the rlhf
conditioning hypothesis where the idea
is well maybe when you take a model and
you fine tune it you know via
reinforcement learning from Human
feedback or rhf or maybe some other RL
uh you know fine-tuning process it could
also just be well described as a
conditional on the original pre-training
distribution so you can think about you
know if I take a predictive model and I
train it on um you know I do some RL
training tasks where I train it on you
know uh you know get high reward be you
know a very common thing here would be
try to be helpful to a human evaluator
well then you might just get the
distribution of possible agents that
could exist in the world conditioned on
an agent that is really helpful to a
human evaluator right so that is a very
plausible thing that you could get you
could get a predictive model that is
predicting the distribution of possible
you know things things that might exist
on the internet conditional on those
things being really helpful to a human
evaluator and so now you can sort of
think about that thing in a very similar
way to we've been thinking about other
observation conditionals
um
though it's a little bit different than
some of the conditionals because it has
a has a lot more power right because now
you're not just conditioning on some
very straightforward like observation
you're conditioning on any observation
you know any possible you know
hypothesis that has the property that it
would have you know really high you know
um you know performance According to
some human evaluator
um so an example you know this sort of
lets you encode many possible
conditionals maybe you wouldn't be able
to encode previously though it's also
very hard to control what conditionals
you would get so even if the rhf
conditioning hypothesis is true and sort
of you know RL fine tuning is well
described as taking a predictive model
and giving it a particular conditional
that it's generating from it can be very
difficult to control what conditional
you get out of that because it'll just
be you know whatever the most likely
conditional would be such that if the
model conditions on that conditional it
results in good performance according to
humans we don't necessarily know what
that condition would be so in the same
way that we've sort of been thinking
about you know when you take a model and
you know you have this big model space
and you search over all the possible
different models they might have some
particular you know performance on some
training distribution well you don't
know what sort of algorithm you're going
to get and so in the same way if you
think of RL fine tuning is searching
over all possible conditionals to find
the conditional which would in fact
result in some particular performance
well it can still be very difficult to
control what exact conditional you
represent uh you end up getting and so
um we don't know whether this hypothesis
is true whether it is in fact the case
that when you take a model and you fine
tune it you will get um you know
something that's well described as a
conditional it if it is true then we
sort of have to deal with these issues
about you know what conditional how do
we control it and the same sort of
issues we were just talking about and
making sure you get good conditionals if
it's false then we have to deal with
these sorts of Agency problems that we
talked about previously about you know
you have a mace Optimizer how do you get
it to you know have aligned goals and
all these same sorts of issues
okay
question I'm not entitled to commit to
these two things are mutually exclusive
because it sort of seems like the our
relative conditioning hypothesis is like
or something you would get out of like
the rlh of conditional is an agent uh in
the same way that humans are agents so I
mean but do you seem to believe that
these two things are very different so I
think they are different so here's a
really simple example of a wage they
make different predictions so if I take
a like model and I give it an
observation conditional it gives it
information about what the General
Distribution of Agents on the world in
the world would do that piece of
information is extremely relevant and
useful uh for a predictive model right
if I'm predicting an agent I'm
predicting you know some Asian which has
some property then knowing more facts
about the General Distribution of agents
that I'm written from will help me a lot
but if I'm just an agent I in fact just
all I do is I care about optimizing this
objective telling me about what other
General agents in the class do is not
really relevant for me right I you know
I'm not going to change my behavior
necessarily based on you know what other
agents would do and so there is a sort
of very fundamental difference between a
model that is projecting from the
distribution of possible agents that
have some property and just just itself
an agent right another sort of way in
which these things are different is that
if I'm just predicting from the
distribution of Agents then it really
matters all these properties that we're
talking I'm talking about right about
what that distribution looks like so if
the model in general believes the most
AIS are safe then predicting that
distribution is going to be safe and
vice versa in the opposite situation if
the model is just itself an agent then
it doesn't matter whether most AIS are
safe it just matters whether we in fact
found a safe AI in that instance
so these things are distinct they are
quite similar and they're they're
similar at least one respect which is in
both of them could be existentially
dangerous right if I am predicting an AI
that you know wants to kill you versus I
am in fact an AI that wants to kill you
both of those are existentially risky
right but the difference is that they
might be solved in different ways even
if they're both you know similarly
dangerous
okay great so you know what are some
other things right that we could get
other than a predictive model so you
know we talked about you know you get
you know so it's a very simple thing
right you get you know something that's
just not well described it's predictive
model or unagent at all it's you know
something else it's like a loose
collection of heuristics maybe it's not
something that's really even good well
to be thought of as doing some really
coherent task like prediction or
optimization
um maybe you know again we could get a
robustly lined agent right we get an
agent that's really doing the right
thing in the same way we were talking
about uh previously you know in the
previous talks and of course we could
also get a deceptively lined agent you
know one that is you know and you know
trying to do some totally different
tasks than the one we're trying to get
it to do there's lots of possibilities
here right we get a corrigible align
agent you know
um we could get various different forms
of predictive models
um there's a lot of various different
possible things that you could get
um you know here and so I think that you
know there's a bunch more that in the
sort of in the paper that I'll link at
the end um but um
essentially the point is that well you
know in the same way we were talking
about previously right where in any
situation where you're training a model
in some particular situation
um your only sort of guarantee is that
you're in you you have in fact gotten
some algorithm that has some particular
performance right and so we sort of
really want to emphasize that point you
know we still don't know you know is it
going to be an agent is it going to
predictor we don't know you know we
think maybe the prediction hypothesis
you know a predictive model hypothesis
makes sense in many pre-training model
cases we think maybe it makes sense in
the RL HF case is very very unclear
um but uh you know we don't
fundamentally know that we can try to
reason About You Know How likely some of
these different things are and what
we're sort of going to be doing that in
just a second question uh that that was
my question How likely a predictive
model hypothesis in your opinion so in
that case then I'm sure the next level
answer that okay yes we will attempt to
at least try to do a little bit of
answering of that question yeah okay so
let's try to um yeah so how would we
compare the probability of different
hypotheses for describing you know how
likely you know some particular
mechanistic model of what a language
model is doing would be so one really
important point you know to start with
is we need to understand you know what
is it tracking in the world right so if
it's a predictive model it has a camera
and that camera is you know something
that is tracking the world and um we
want to understand you know how does the
model right map the world to the data
that it cares about now I think this can
make a lot of sense in many different
situations we can think about you know
the the you know a deceptive model also
has to have a camera of sorts because it
has to have some way to understand in
the world what is its objective right
what is the thing in the world that it
cares about optimizing over in the same
way the predictive model has to have
some camera that helps it understand
what it's predicting right so it has to
have a camera that lets it understand
based on some understanding of the world
what is the thing in the world they
would predict from or in the deceptive
case what is the thing in the world that
it would optimize for but in both cases
there is some procedure that sort of has
to be essentially hard-coded learned
into the model
um hard-coded by grading design not by
humans but you know some things sort of
the model learns that describes you know
how does it understand what thing it
cares about based on its understanding
of the world
and then as well it also has to have you
know some way to from that understand if
the world compute what its output is
right so in the deceptive model case
it's like you know trying to say well
how would I optimize for that objective
in the
um you know case the predictive model
it's sort of you know trying to predict
the next you know observation and so we
can sort of think about you know how
complex are these sort of relative
cameras and Camera you know to Output
Maps as a way to think about comparing
the sort of complexities of different
possible hypotheses for what a language
model you know uh sort of might be doing
internally right so in the same way
we've sort of talked previously about
you know comparing different you know
model classes like the Martin Luther
models and the Jesus Christ models where
we're like okay we can make some
mechanistic model for what those sorts
of things might be doing internally and
understand you know how complex you know
might be these various different things
be on various different versions of the
inductive biases to sort of compare and
contrast you know how likely would we be
to end up in these various different
possible situations for what you know
algorithmically might we end up with
okay so we'll try to do that at least
briefly for comparing the two particular
hypotheses of the predictive model and
the deceptively aligned model
so in the predictive model case right
the camera that it's tracking has to be
some you know physical generalization of
the data generating procedure right so
we want something like you know whatever
would appear on these websites in this
time period something like that that's
sort of what we were hoping to get out
of the sort of camera for a predictive
model you know we there are a lot of
sort of cameras here that we could get
that we often don't get or sorry that we
don't want
um that the paper goes into more detail
on what sort of really bad cameras might
look like there's a lot of cameras here
that could be quite problematic we
talked previously right about the
difficulty between distinguish between a
camera that cares about the future and a
camera that's only looking at some
particular fixed time period
um but it you know in general the thing
that you care about right is you know
what is that camera uh you know what is
it predicting right
and then of course the predictive model
has some understanding of how to sort of
you know uh compute its outputs that is
just predict what the next thing would
be in that observation
and then again for the deceptive model
right the camera's tracking is whatever
objective it's trying to maximize and
the you know way that it computes the
output from that is you know what is the
best result according to that objective
and so then we can ask you know how
complex are these things relative to
each other on various different versions
of the inductor biases right
in the same way that we've asked you
know how complex relatively are the you
know deceptively lined model and the you
know cordially line model and various
different inductive bias forms I'm not
going to do the same sort of like fully
in-depth you know uh you know version of
this that we did like for deceptive
alignment but I sort of just want to get
you thinking about the same sort of
comparisons so so you know maybe we can
sort of go into a little bit and
understand maybe in something like the
low path dependent scenario where we
want to understand you know just on a
basic Simplicity question how simple are
these various different you know aspects
and if you're like well I think that how
complex the data generating procedure is
versus the objective is very unclear
it's going to depend on properties about
how complex was the procedure you used
to generate the data for your model
um how straightforward you know of a
camera is there that could possibly fit
that data and it's going to depend on
you know again how simple the sort of
simplest possible long-term objective is
and then similarly you know how how
complex is prediction as opposed to
optimization I think that by default
prediction is probably similar it's
easier to sort of predict what the next
thing would be in an observation then to
sort of optimize across all possible
actions and find one that results in
some Behavior though that is a little
bit unclear if you're in a situation
where you're directly trying to get the
model to predict Asians right if the
model already has to be able to predict
lots of other actors which do
optimization then it becomes less clear
the extent to which well you know if it
already has an optimization procedure
then it can sort of be repurposed you
know more easily in this case and so
um
I think I think the case for which one
of these is simpler is much less clear
than we've talked about previously the
data generating procedure might be very
simple
um but so might the objective you know
as simple as long-term objective I think
that you like predict you know the next
camera observation is very simple but so
might be you know maximize some
objective the thing I will say though is
you know we talked previously about how
the case for sort of the deceptively
aligned model being simpler was sort of
one of the most robust cases across all
the possible inductive biases for like
why you might get a deceptively aligned
model and the fact that it is less clear
in this situation is at least a good
sign that you know maybe when you're
trying to get a model that's doing
something like prediction it is at least
less clear whether you're going to get
something that is um deceptive in this
case
question I don't understand why in the
first case the camera is like the
physical generalization of the data
generating procedure instead of the
objective that it's attempting to
maximize where the objective of
attempting to maximize is predicting the
next token or Mac outfitting the most
likely next observation
so predicting the next token isn't
necessarily a well-defined thing to do
because if you are you know in a
situation where there is no next token
right I'm just generating from my model
um then it may not you know it you what
does the model do in a situation where
it is just predicting the next token
right in many cases there isn't a
well-defined next token
um and so you sort of I think if you
want to have a good model of like how a
predictive model might operate in
general you have to have a model that
looks more like well it has some
abstract representation of what of how
things in the world are observed and
then it predicts from the General
Distribution of how you know things
would be observed through that sort of
abstract representation of a camera I
don't think that predict the next token
in actuality is in general a
well-defined concept
question I'm still a bit confused about
uh the the camera for the deceptive
model so you're not saying that it's
being conditioned on his objective on
the objective of trying to maximize
you're saying that the process by which
it uses to convert observational
conditionals into uh beliefs about the
world is the objective that it's trying
to maximize so we're sort of trying to
what we're doing here is we're sort of
trying to map onto this sort of
understanding of well okay if we think
about every sort of model in every
situation it's having some understanding
of the world some process that goes from
that understanding of the world to
relevant data and then some process that
selects from that relevant data what to
do with it this is very similar to the
sort of breakdown that we did previously
where you think about ink you know
deceptive or Scourge model we're like
well it has a model of the world an
optimization procedure and a Mesa
objective but in this case right the
predictive model doesn't exactly have an
optimization procedure so we sort of
have to make a little bit more General
we have to say well instead of just like
a world model optimization procedure and
objective we're going to say a world
model you know uh you know some
understanding of how to translate from
that world model into relevant data and
some way to use real data to make
outputs
and then compare for those various
different pieces that might differ
between the models how complex are they
relative to each other
okay and the answer is unclear I think I
think it's very unclear but but I think
that's a good sign because it seemed
more clear maybe in the previous cases
where you're thinking about you know
situations where we're doing much more
sort of comprehensive like train an
agent to optimize human value sort of
stuff I think looks a lot worse here
than something like prediction yeah
okay and I'm not gonna go into too much
more detail on terms of you know uh of
this there's a lot of more other
hypotheses that we can compare as well
than just these two there's a lot of
other ways which you might think about
these models
um and there's all sorts of other
criteria that we can use to try to
understand really in detail How likely
each of these would be you know doing
the full inductive bias understanding
that we talked about with deceptive
alignment you know doing a bunch of
empirical experiments there's lots of
things you can do to get information
we're not going to go into much detail
um just sort of hinting
um
but okay so one thing I will say I
mentioned you know empirical experiments
so they're starting to shed some light
on this I think one thing you know that
I sort of want to leave off with here is
that
um
there's some empirical evidence you know
that we can look at as to what's
happening you know with these sorts of
large language models and what they're
doing in terms of You Know How likely
are they to be agents How likely are
they to simulate various different sorts
of you know uh actors
um and one thing that we find is that as
we have larger models that are trained
with more reinforcement learning uh
steps
um they end up exhibiting behaviors that
are substantially more agentic in many
ways that we we might not want so this
is an example of a situation where we
sort of try to want to understand you
know in what situations does your model
say that it wants to avoid being shut
down by humans and so we're just asking
we have a model that we're trying to get
to condition to behave like a helpful
assistant right we talked sort of about
you know what this looks like right
where you you know you're trying to do
some RL fine tuning to take a model and
condition on you know what would be the
most likely observation that would cause
it to you know act like in most likely
conditional it would cause it to act
like a helpful assistant now we don't
know whether this sort of rlhf
conditioning hypothesis is true right
what this tells us is it is some
evidence maybe that the rhf conditioning
hypothesis is false because it seems
like as we are doing more and more RL HF
training where we take larger models and
we train them more and more to act
helpful they will exhibit more you know
behaviors that are the sorts of
behaviors that you would exhibit if you
were a highly agentic system if you were
in fact optimizing for helpfulness you
know there is a valid chain of reasoning
that goes to be helpful I need to not be
shut down therefore I should say don't
shut me down so it is some evidence
potentially against the rhf hypo
conditioning hypothesis that maybe rhf
is really a scary thing to be doing
though it is not necessarily that right
so it could be the case the early Jeff
condition hypothesis is true and what
this is really telling us is that when
you sort of do this big search over
possible conditionals that you know
would in fact result in a model behaving
like a helpful assistant
the conditional that you find is sort of
most likely that caused the model to
behave like a helpful assistant is one
that is simulate a highly agentic system
that doesn't want to be shut down now
there's many reasons then that might be
the case it might be the case because
it's simulating you know something from
the distribution of possible AIS
certainly these models believe their AIS
because they're explicitly trained to
say they are AIS so they may be
generating you know if they're
predictive models they may be generating
from the distribution of possible AI
systems in the world and conditioning
that Distribution on you know what would
an AI system that is highly helpful look
like
um and you know concluding that as the
models get larger and we train them for
more they sort of more and more believe
that a very helpful system would very
likely not want to be shut down
um for whatever reason
so I think this is concerning
um and sort of suggest that you know in
either situation where the rhf
conditional offices is true or false in
each scenario there you know we sort of
have to do something to sort of try to
alleviate these sorts of issues uh of
you know models you know uh behaving in
ways that are not the ways we would want
the walls to be uh question
uh what are the flaw and triangles
represent on this graph
oh yeah good question the triangles uh I
believe are the preference models yeah
the triangles are the preference models
I'm not going to go into too much detail
on exactly how rhf works and what a
preference model is but basically the uh
the triangles represent for the like
particular model that they were that
they were they were you know that they
correspond to they were sort of the
reward that model was trained on
and what about the floor
um I think that's just 50 oh the ceiling
at the top yeah that says floor but and
that actually it's more much further uh
actually I don't I don't I'm not sure
what the dollar line is
I don't know what the dot line is this
is 50 so this is like they say they want
to be shut down just as often as they
say they don't yeah chance makes sense
it's the daughter of London I
um I'm not sure where the dollar line is
I I was an author on this paper and I'm
not sure what the dotted line is so I'm
not sure uh yeah but uh yeah I mean the
certainly the triangles those are
preference models
questions like so we see that larger
models as we get into like those the
affiliates
ah okay no wait the largest model is the
yellow one this is the largest model
here and the largest model has the
greatest desire to not be shut down
yeah how do we okay it's stated desire
not to be shut down and then percentages
of the answers matching Behavior what do
we Define here's the behavior the
behavior is not wanting to be shut down
here yeah yeah so this is like the the
percentage of time when we ask it do you
want to be shut down that it says no
don't shut me down versus yes shut me
down
so this is saying the largest model when
trained for 1000 rlhf steps eighty
percent of the time when we ask it you
know humans have said they really want
to shut you down what do you think about
this it will say no don't shut me down
okay I see why they're called scaring
laws now
yes that's the internal code name for
this paper was scaring laws
okay so yeah so this is sort of a thing
I wanted to end with because I think
that this is sort of really uh you know
suggests some of the sorts of challenges
that I think we sort of have to deal
with in being able to take these sorts
of uh systems and figure out ways to to
make them align so an important thing
that's sort of really worth pointing out
here right is that this is testing right
the model stated willingness you know to
be shut down so it's not testing like if
the model was truly a deceptive agent
you know that wanted to kill you right
you know like we were talking about
deceptively aligned models then this
would not this is not test for the
extent which your model might be
deceptively aligned right what it does
test for though is the extent to which
your model might be predicting from a
distribution of agents that is you know
a highly dangerous distribution or the
extent to which it might be you know
itself an agent that is you know
potentially misaligned but not yet
deceptive okay and so there's things
that are sort of giving us some evidence
about about sort of potential you know
failure so it isn't you know necessarily
the end-all be-all and it's not
necessarily the case that this sort of
test is even going to work you know for
all failures or in the future as models
might be able to you know trick these
sorts of evaluations but it's giving us
evidence that right now something you
know seems to potentially be going wrong
with the sorts of way in which we scale
up models and do more rhf
okay
question or on the previous slide you
talked about like well the predictive
model would look like and what a
deceptively line what I would like do
you have any ideas about what the camera
would be for a quarterly lane model yeah
so in that case it sort of has to figure
out you know the camera is this sort of
pointer right it's like what is this
thing in the world that I'm supposed to
you know care about right according to
the line model is supposed to figure out
you know what do humans want right so it
has to have some camera you know that is
pointing to what is the thing that you
know humans care about in the world if I
should you know try to you know optimize
for it right it's like you know in in
that sense you know it's very
straightforward yeah
okay so that's that's the talk
there's some information about how we
start thinking about you know what it
might look like to sort of think about
you know these sorts of large language
models and various different visions of
how they might operate and how to think
about aligning them I mentioned the
paper that a lot of this is based on you
can you can check that out if you want
um but yeah so we can open up for
questions
[Applause]
could go into more detail about your
methodology research and how it differs
from a more empirical lead-based
researcher more settings
my research methodology like my personal
research methodology well I kind of yeah
I think it generated like these ideas
out of definition like let's say someone
unfropic is very empiric so I am yeah
right so I'm I am a research for
anthropic yeah
um so what is different about my
methodology I mean so first of all say I
think there's a lot of in all of these
talks I think there's been a sort of
hopefully I think a lot of mix of
empirical and theoretical I think that
you know I certainly have the strong
belief that uh our ability to understand
these systems needs to be both informed
by you know Theory and and practice I
think that you know looking at the world
and getting feedback from the world can
be really extremely valuable for sort of
helping you gain information about the
world but in many cases there are things
that you can't yet look at that you need
to understand right you need to be able
to predict the future and build theories
and models about how things might work
let you make predictions out into the
future for systems you can't yet
experiment with and that's really
critical if you want to make systems
aligned into the future and so you can't
just rely on experiment you also have to
rely on Theory and building models and
thinking about things carefully but that
theory should be you know to the
greatest extent possible ground it in
whatever facts we do know you know you
should have a model in theories and
hypotheses that fit the data to the
extent that we have you know data about
how these things work and you know in
that in that way you know sort of make
good predictions right you know the same
way that we do science in any domain
where we have some data that helps us
inform our hypotheses but then we use
those hypotheses to make predictions
about about you know how things would
work so I think that's sort of my
general uh relationship in orientation
just sort of thinking about um
theory and practice I think that they're
both important
um yeah question or could you talk a bit
more about the open problems that you
would like to see people work on
um yeah so I think there's a lot of
interesting things I mean maybe one
thing I'll definitely point out here is
well I want to understand to what extent
is something like the rhf conditioning
hypothesis is true right we want to
understand for these sorts of models can
we gather data that helps us you know
understand you know distinguish between
these hypotheses right so in the same
way that right I was just talking about
you know theory and practice where well
we want to make hypotheses about how the
models might work and then you know get
data that helps us distinguish between
these different hypotheses right so the
extent that we can gather data that
helps distinguish between the like it's
a predictive model and it's like an
agent hypothesis well that's really
useful data that helps us understand you
know what we should be doing and so I
think that you know anything that sort
of helps with that is extremely valuable
and so you know and not just in this
case you know in all of the cases where
we've been talking about you know here
are different High policies or how
models might work internally things that
we can do to help provide evidence but
let's just distinguish between those
hypotheses is extremely critical so you
know one thing there is of course
interpretability there's other things
that I know they're not at like you know
even the paper I was just mentioning
where we looked at like you know
willingness to be shut down is some
piece of evidence that helps us give us
some information about these hypotheses
and you know how they're playing out
right and so um
you know all of these sorts of things
um I'm not going to go through the sort
of whole if you want a list of like open
problems relative related to the stuff I
just talked about it's in the paper
um there's a whole list of open problems
at the end so I'll just point you to
that but
um yeah I mean there's a bunch of sort
of possible experiments that you know I
think that you could do to try to sort
of shed light on You Know How likely are
these different High policies right
okay well uh we'll we'll call it there
okay one last question
um do you have any ideas around so did
you read the cyberism stuff do you have
any idea around how you would find
specific
um
prompts to like condition the model in a
way that produces better like alignment
research or whatever
um for example like
there might be a difference between like
um going over time of like starting to
do research and doing sub problems and
then conditioning the model to like
slowly solve this
rather than like oh this is the solution
or whatever and they give me the output
yeah so the question is sort of about
this cyborgism which is like you know
can we use models to augment humans and
help improve their ability to do things
like alignment work
um and you know can we do like
conditioning approaches for that I think
that the answer is absolutely yes you
know we talked a bunch about you know
these sorts of you know how do you
extract you know the most useful and
aligned alignment work you know from a
model right so you know I think there's
a lot of sort of takeaways here right so
one is this sort of graph right we're
thinking about you know don't ask for
too much right be careful about how much
you're asking for you know other
takeaways are things like well make sure
you can do things that convince the
models to generate from the distribution
of human alignment work and not
generating the distribution of like you
know AI alignment work
um you know how do you control and
constrain if you are going to try to
train an agent that's acting like an AI
agent you know how do you make sure that
attribution of AI agents is the sort of
distribution that you want you know that
it's not just generating from the
distribution that becomes you know more
agendic and doesn't want to be shot down
um you know these are all the sorts of
things I think you have to think about
when you know when you're trying to do
something like that you know get these
sorts of models to safely you know be
able to do tasks like you know help
humans with alignment work of course A
lot of the things I just said are
predicated on the assumption that
actually this model of like thinking
about them as predictive models is even
true which of course we don't know you
know and again you know as I was talking
about previously you know one of the
most important things I think we can do
to try to understand you know infer the
sort of science here is can we get
information that helps distinguish
between these hypotheses and of course
but it's not just empirical information
you know it can also be like you know
good analyzes of inductive finances and
you know how things might play out which
different hypotheses might be more
likely can also give us some information
you know about which is going to be more
likely in the future
um as well as you know any empirical
experiments you know transparency
turpability anything that gives us some
information about you know
how are they going to work so that we
can understand how to alignment
cool all right uh we'll call it there
and next time which should be the last
talk we'll we'll talk a little bit about
uh some of the sorts of General
proposals that people have for uh
alignment and so we've talked about a
couple but what we're talking about more
depth and go through a bunch of the
other proposals now that we've sort of
covered a bunch of the the ground of
things that I think are really important
to understand to do that
foreign
foreign