6:How to Build a Safe Advanced AGI?: Evan Hubinger 2023

okay so uh yeah so this is the the last
lecture so today we are going to be
talking about uh how to build a safe
Advanced AI
um
so we're not quite going to be doing
that uh because I don't know how to do
that but we are going to be talking
about some ways that people have
proposed to attempt to do that so you
know up to this point we have tried to
cover a bunch of the sort of you know
preliminaries and I think you know
really important things to understand
how to think about uh AI safety uh
proposals and Concepts uh and so today
we're sort of going to be looking
through a bunch of additional proposals
that we haven't yet looked at and really
sort of trying to go in depth and
understand you know what is the
rationale for all these various
different things that people are
thinking about uh you know why might you
want to do some of these various
different uh you know uh proposals
okay so you know this is you know just
to recap we've already sort of gone over
this but you know we want to sort of
want to talk about and you know
established at the very beginning you
know how do we evaluate a proposal for
you know building some sort of powerful
safe you know Advanced AI system so the
sort of criteria that we're going to be
looking at and these are the same ones
we talked about earlier we have this
sort of General version of outer
alignment which is you know whatever the
thing that we're trying to get whatever
algorithm we want our model to be
implementing you know this sort of
training goal uh why would that be good
why would it be good you know for us to
in fact get a model that is the sort of
model that we want
we have this sort of generalized version
of inner alignment which is uh you know
how do we actually guarantee that our
training procedure in fact produces a
model that is doing the thing that we
wanted to be doing so how do we actually
get a model that satisfies that training
goal that is this is the sort of
training rationale this sort of
understanding of why is it that our
training process you know via all of the
inductive biases all the ways we've set
it up when in fact find an algorithm
that is the sort of one that we that we
want it to be implanting
and then we have implementation
competitiveness is it sort of in fact
practical for us to run this procedure
um and we have this performance
competitiveness if we did run this
procedure and we got the thing that is
the thing we're trying to get you know
the algorithm that we want would that
actually be able to satisfy the sorts of
use cases that people want AGI and other
sort of really powerful AI systems for
okay so these are the main criteria that
we're going to be looking at the same
ones that we sort of were talking about
previously
and we've already talked about a couple
of different sort of proposals that
we've looked at you know sort of
understanding in these these various
lens so we looked at microscope AI
previously this idea of you know trying
to
extract Insight from our systems via
transparency tools use that insight to
improve human understanding and sort of
iterate that way so we're not going to
recover this but this is you know one
proposal we've already talked about here
and we've already talked about this sort
of predictive models idea the idea of
well you know we can try to take the you
know these systems trained potentially
to be just sort of predictive systems
that are predicting some you know
particular camera and uh you know use
those systems condition them in various
ways to get out useful information
so we've sort of already talked about
these two
um one thing though that I think is sort
of you know we'll separate these two
proposals from a lot of the ones that
we're going to talk about today
um is that uh a lot as we sort of talked
about last time with something like the
conditioning approach there's a point at
which it breaks down as you start sort
of getting into systems where you're
asking for very highly superhuman
capabilities you want your models to be
able to do things that are substantially
beyond what any human could possibly do
um being able to you know successfully
get those models to do the things that
we want under the sorts of proposals
that we talked about previously get to
be sort of quite tricky so in the
conditioning predictive models approach
we talked about how uh it's quite
plausible that you could sort of get a
model to do something really useful and
valuable that was just a predictive
model so long as you weren't asking for
something that was sort of substantially
beyond what any human would ever do
because if you ask for something
substantially beyond what any human
would ever do then the most likely you
know thing to predict that would do that
would be you know some AI system which
might not be safe
um and similarly with microscope AI we
talked about how you know microscope AI
might work really well when we're in a
situation where the sorts of
abstractions that the model learns are
human-like abstractions but if
potentially you know we keep pushing
into a domain we're trying to you know
get access to capabilities that are
substantially Beyond human level we
might sort of start to learn
abstractions that are increasingly alien
and difficult for us to understand and
abstract and make use of
so we sort of have this key problem with
a lot of the sorts of proposals we've
talked about previously that they can
struggle to generalize and work well
substantially beyond the human level and
that's not necessarily a problem with
these approaches I think that you know
any sort of strategy you know very
general strategy for making use of all
of these various different approaches
that we have come up with is going to
you know presumably involve you know
multiple different approaches used at
different times for different sorts of
models
um but one of those there is clearly at
least the sort of key problem which is
well eventually we're going to have to
do something in this sort of you know
further regime
um and so we're sort of going to talk
about this problem is this sort of
scalable oversight problem you know how
do we scale our ability to oversee
models and ensure they're doing the
right thing substantially Beyond these
sorts of human level capabilities
question how about in this diagram here
where would you say we are now right we
have models that are clearly not human
level but they seem to be superhuman in
some domains like alphago is superhuman
at go so we're on this curve if you say
that modern systems tend to be yeah I
think that's a really tricky question uh
and I think it's you know going to vary
from system to system I think that like
if you're thinking about like in the
conditioning productive models approach
I think we're sort of you know around
this regime where the model's
capabilities are just sort of you know
human level
um you know many sub-human in most cases
you know some places they can be super
you know superhuman but overall they're
sort of like below the human level and
you know certainly not superhuman
um you know in go you know there's cases
where they are substantially superhuman
it's not clear whether their concepts
are substantially superhuman
um though they might be in many cases
the sorts of Concepts that these systems
will learn are understandable to humans
when we can extract them
um but it's really hard to do
interpretability and actually understand
what sorts of Concepts these systems
happen so you know you could for example
see that as very biased by our ability
to actually extract things you know we
can only oftentimes extract the things
that we do understand and so I think
this is a really tricky question to
answer I'm not going to make some strong
Claim about exactly where different
models stand on various different axes
here I think that
um one thing the main thing that is
clear as well at the very least we're
not yet at like you know age GI you know
systems that are you know fully General
can do all of the sorts of tasks that
humans can do we're certainly not there
yet and we're definitely not at the you
know super intelligent systems you know
across the board yeah
um and so like at the very least right
now I think that a lot of the sorts of
you know approaches that we've you know
talked about previously like predictive
models you know focusing that sort of
stuff you know it does seem like you
know totally applicable to current
models and Beyond current models at
least for a substantial period but
eventually we will presumably reach a
point where that's no longer applicable
now we talked sort of you know about
last time about you know one thing you
might want to do with these sorts of
systems you know and these sorts of
approaches which sort of only work in
the you know you know sub superhuman
regime is maybe you know try to do
things like additional AI Safety
Research to make it easier to come up
with other approaches that work in the
in you know sort of past you know
regimes beyond that
um but that might not work you know it's
very unclear and so you know it's worth
you know trying to really delve into and
understand you know what are things that
we could do that would help us push you
know our ability to align systems you
know as as far out as possible
okay
okay great
so here's the sort of outline of some of
the these are the approaches we're gonna
be talking about today uh that we're
gonna try to get through we've got a
bunch uh there's more just beyond the
ones that we're talking about today but
these are you know some of the ones I
think are important to try to understand
and work through
um and you know we'll sort of gesture at
some some others uh at the end
okay so we're going to start with uh
amplification and to do that uh we sort
of need to understand a particular
preliminary which is the concept of hch
so hch is a recursive acronym and it
stands for humans Consulting hch so what
is it so we're going to have a human uh
you know just a normal human and the
human you know answers questions so the
human can take in a question and produce
an answer uh this is you know any
situation where you can have a human
answering questions
um
and of course you know if you just did
something like train a model to mimic a
human answering questions
um that might be you know safe in the
same sense that we talked about with a
predictive model but it wouldn't you
know necessarily be able to generalize
to do anything beyond what a human would
be capable of doing uh you know safely
but we can sort of change this picture
so what if we give a human the ability
to talk to two other humans well now
we've sort of taken the you know human
level capabilities and we've improved
them so now you know it's the level of
capabilities that are accessible to one
human with access to the ability to talk
to two other humans and this you know
increases the capabilities and the sorts
of tasks that the one human is able to
answer the sorts of questions that are
uh you know available for this person to
answer that they can do successfully is
larger
um and we can iterate this procedure we
can give the you know the other humans
uh access to two more humans to talk to
as well
um and and we can sort of repeat this uh
to Infinity you know we can say well
what if you had the ability to
theoretically you know query additional
humans and you know be able to you know
every single person in this entire tree
had the ability to talk to additional
humans
so we're going to call the sort of
entire tree here this you know entire
object of you know humans with the
ability to talk to as many additional
humans as they possibly want all the way
down the tree we're going to call this
hch
and I haven't yet talked about how you
know this relates to any ability to you
know predict this thing or simulate it
or train a model on it but the point is
this is a theoretical artifact it is a
thing that we could never build uh you
know or you know maybe in theory in some
situations if you had access to you know
enough humans and you know the tree was
small enough maybe you could try to you
know put a bunch of actual humans
together but for all intents and
purposes we're gonna imagine this is a
theoretical or object that we can't you
know in practice build but that is in
fact going to be relevant for
understanding you know the approaches
that we're going to talk about Yeah
question what's your best guess if we
actually build this with humans ha
Valdez smoothberg in solving certain
problems and how much diminishing
returns we would get my guess is that
for most tasks the force level is just
making things worse but okay I don't
know how to define most tasks and what
time we need to stay happy yeah I think
it's a really tricky sort of thing to
understand you know we is this good you
know if you theoretically have this
object you had this thing that was just
you know all these humans talking other
humans all the way down the tree would
you be happy you know and that's sort of
one of the key questions that we're
going to be talking about because you
know we're going to talk about an
approach that's trying to build
something like this object and so we
want to understand you know one of the
things we need to understand you know
like from an outer alignment perspective
right is if we actually got something
that was like the thing we're trying to
get would we be happy and I think the
answer is very unclear there's
definitely some reasons that you might
expect that this is a good thing I think
that you know the sort of standard
argument from why you might like this as
well it's just human cognition and we
might you know believe the human
cognition in many ways is sort of safer
um it's also sort of in some sense you
can think of it as an approximation to
sort of the you know enlightened
Judgment of a human if you imagine all
of these humans sort of being the exact
same human uh then you could think about
this as what if you had the ability to
think about something for an arbitrarily
long period of time by you know
Consulting other copies of you and maybe
this is you know better than like if you
had the ability to literally just think
for a long period of time because maybe
you know you sort of start to go crazy
after thinking for a million years but
if you have the ability to just delegate
you know to Infinity all of the various
different some tasks to other copies of
you you know in some sense this is sort
of you know what you would do if you
really had the ability to effectively
you know approach the problem you know
from all possible angles
um but of course there's other arguments
as to why this might not be you know a
good thing you know uh
an individual human only thinking for
maybe a short period of time and
answering a single question might not be
able to do the sorts of really complex
cognitive tasks that you know we might
you know really need humans to do there
might be an accumulation of errors in
various ways as you're sort of you know
delegating and delegating and delegating
and delegating
um
there's a lot of various different
things you could imagine happening in
this sort of an object
Yeah question
what kind of
subjects that we make about the
communication between those three months
so the whole velocity is it
yeah that's a good question
object that will make different
assumptions about you know what the
communication is between the humans
um I think for our purposes I want to
basically imagine you know each one of
these arrows you know you can
essentially allow whatever communication
you want but that like this human can't
you know go and talk to this human
directly everything is factored through
this sort of tree structure
I mean there's other variants on this
that sort of would depend on exactly how
you set up your training procedure but
for our purposes right now this is the
sort of object we want to understand
question
uh have people tried this with like
modern language models collating
companies of themselves and how well was
that gone if they have uh there have
been some experiments that have looked
at you know some things like that
um there's various different you know
versions iterations of this depending on
sort of how you think about that uh you
know what what you think about that
doing and how you sort of think it looks
um there's things like prompt chaining
and even just like Chain of Thought can
sort of be thought of a version of this
I think that it's
um very unclear Some Things sort of work
very well some things work very poorly
um
I would say that in many ways the sort
of jury is still a little bit out on the
extent to which this is sort of um you
know effective
I haven't I I think I also sort of want
to defer that a little bit until I talk
about what the actual training procedure
is here because I think that the actual
training procedure here though
um you know the way that you might
actually want to train them all to
approximate this object is um is
actually quite similar to a lot of the
ways in which we train print systems so
um but with a couple of modifications so
well I'm going to return to that in just
a second right and uh what is prompt
chaining and chain of bullet
as I talk about the actual model
procedure so when I I'm gonna I'm gonna
bin that this for just a second and I'm
going to return you know to trying to
think about how this would actually you
know play out once I once I explain the
most basic training procedure of how you
might want to approximate an object like
this
okay so uh right so what is
amplification so this is another you
know sort of uh term of art here that's
extremely important to understand how
we're going to try to approximate this
object so we're going to say you know
previously we have this object this hch
object where you know we have this you
know human Consulting humans all the way
down this sort of massive tree okay so
now we're going to go back we're just
going to say so suppose we have just a
human the human is doing question
answering
the two humans the query they have
access to two arbitrary objects you know
uh two models for example uh you know
two AI systems that they can interact
with and ask questions about
okay
in this procedure we're going to call
this situation where the human has
access to these two models the Amplified
version of the model so what does that
mean well the sort of idea here is
whatever capabilities this model has
uh by having multiple different copies
of the model organized by the human with
the ability to sort of query that that
model and sort of figure out how to
interpret the results of what that model
gives it this is results in a version of
that model that is now more capable
because you know rather than just being
able to do the things the model can do
on a single query it can do all of the
things that it can do when organized by
human across multiple queries integrated
together
um and so we're going to call this
procedure of taking a model giving a
human access to multiple copies of that
model the Amplified version of that
model
okay and
um this is sort of only one
amplification operator there might be
other ways in which you could take a
model and amplify it might be sort of
other amplification operators but this
is the most basic amplification operator
that we're talking about it's an
operator that acts on a model and
results in a sort of another system that
is able to answer questions in some way
better than the original model
okay and so concretely the training
procedure that I want to talk about here
uh that is sort of going to attempt to
approximate this HDH object using this
amplification operator is fundamentally
very simple so what we're going to do is
we're going to train some model to
imitate the amplication operator applied
to that model so what does that mean so
we're going to say uh you know a human
with access to that model we're going to
train the model to imitate that
um
that's the most basic idea we're going
to throw on uh I actually don't want to
talk about this quite yet
um so yeah so let's just stop right here
for a second so the idea is we're going
to train a model to imitate a human with
access to that model this is the most
basic you know training procedure
um that's why I promised I would return
to sort of trying to understand how this
works in the context of you know
thinking about uh you know concrete
training procedures
um fundamentally uh oftentimes what you
do if you take you know we talk a bunch
you know previously about you know
language model pre-training where we are
in fact just taking a model and training
it on a bunch of human texts in some
sense you can think about that as the
sort of first iteration of this
procedure we were just training on
imitating a human rather than a human
with access to a model you're just
taking a human straightforwardly
um
now what this is saying is it's saying
well you know if you're only training to
imitate a human then you can sort of
only safely you know go up to the level
of you know what a human would plausibly
do and this is sort of what we talked
about last time where if you just have a
predictive model and that model is just
predicting what a human would do
or you know across a possible
distribution of possible agents once you
start asking for things that are beyond
what any human would possibly do you
start to run into issues where now
there's sort of no plausible human that
would do that task and instead you get
other weird things like potentially uh
you know an AI system doing that task
um and so in this case we're like well
you know instead of just trying to
predict humans we can do we can try to
predict something that we also think is
safe maybe but that is sort of has the
ability to maybe sort of go beyond the
capabilities of what an individual human
would do which is a human with access to
that model
and so um you know I think that one of
the sort of key things here also is you
know how do we think about uh you know
setting this sort of thing up
in some cases like you were saying one
thing that you can do is you know things
like prop chaining and uh you know where
you're not necessarily having a human
Loop a lot of times you know maybe you
just train the model to imitate a human
and then once you have a good human
imitator you try to set it up in you
know some sort of uh you know
amplification scheme like this where you
have a model Consulting other copies of
the model in various different ways
um
I'm not going to go into too much detail
on what those sorts of setups might look
like but it is another option you know
this sort of relates to this where
instead of literally having a human Loop
you could also you know just train some
system to imitate a human and then you
know replace the human with that we're
mostly just going to be imagining though
that we literally do have the human Loop
here you know the thing that we're
training on in this particular example
is literally a human with access to our
model we've gotten a bunch of you know
samples of what the human with access to
our model would do and then we're
training to imitate those samples
okay does this setup makes sense I've
you know a little bit you know talked
about a lot of variations on this setup
I think it's very tricky because
um you know I really want to talk right
now just about something very specific
but there's a very large class of
possible things that are related to this
sort of idea that are you know
variations on it and we're in fact even
going to talk about some of the sorts of
variations later on but I think that
this is in some sense the sort of most
canonical straightforward version of the
style of approach we're saying uh you
know we want to imitate something that
is more powerful than a human the most
basic more powerful thing than a human
that we might have access to is a human
with access to our model and so we're
going to imitate that
okay now you know there are some sorts
of issues that we obviously are you know
potentially going to run into when we're
trying to do this uh you know the most
basic issue is well uh there's a thing
that we want to get you know our
training goal we want to get a model as
a result of this that is in fact trying
to imitate something like this hch
process and we'll talk about why that
might possibly be the case in a little
bit
um but we might not get that right for
all of the reasons that we've talked
about previously you know we might get
it deceptively aligned agent we might
get you know some other sort of weird
type of system that is not doing the
exact thing that we want and so you know
we'd like to at least have some sort of
you know understanding of how we can
create safeguards and abilities to check
and verify our models as we're training
them
so one of the nice things that we have
access to in this sort of a setup is we
always have access to at every
individual point in training we always
have access to a sort of version of our
model that is better than our current
model
um which is the amplify version of our
current model so in every individual
point in training you know we're
training some some original model to
imitate the Amplified version of itself
well at every point in training then we
have access to an amplified version of
the model that can serve as a sort of
overseer uh it can sort of throughout
training say you know I'm you know my
actual make make evaluations about how
much you know the current model is in
fact doing the right thing
now this is a very tricky thing to be
doing and we'll talk in a little bit
about you know whether this might work
or might not work but the basic idea as
to why this might be a reasonable thing
to do is well we sort of have in this
basic setup we have a thing sitting
around then in some sense is more
powerful than our current model because
the thing we're training our model to
imitate and you know maybe you know
because it's more powerful than our
current model in some sense it can act
as an overseer it can uh you know look
at our model maybe with transparency
interpretability tools maybe with you
know just by you know trying to
understand what it's doing by
interacting with it and have some
understanding of you know is the model
essentially doing the right thing is it
you know in fact being trained in the
right way
um and so this sort of thing uh uh we
can sort of add this oversight where we
have the ampified version of the model
overseeing the training of the new model
um and in this case we sort of have this
intermittent oversight idea where well
at each individual point in time once we
sort of trained arm uh you know one
particular iterational model to imitate
the Amplified version of that model we
can you know get some new model and we
can oversee it you know by using the
Amplified overseer to check you know do
we like this new model question
Mike somewhere yeah okay
I'd like to ask what exactly we mean
when we say imitative here and both
sides so how does debilitation process
look like is it again like reinforcement
learning but from AI feedback then or
like yeah or is like inverse reinforce
the burning
um
yeah perhaps add to that first and then
deal the set video
you shouldn't look like here's the
question so
so is that um I mean there's a lot of
different ways we could set this up
right now we're imagining something that
is essentially just supervised you know
fine-tuning is the idea we're saying you
have some model you can collect a bunch
of data of a what a human would do when
answering questions given access to that
model that gives you a data set that
then you can you know supervise fine
tune on that you can just train to
imitate uh that you know new data set
we'll talk
um in a little bit about a sort of
variant on this idea where maybe instead
of doing supervised fine-tuning you're
doing something else like
um you know like rlh or something like
that in this case that's not what we're
imagining we're just imagining
um a situation where you're doing you
know straightforward supervised
fine-tuning we just gather a bunch of
samples from the model interacting you
know the human with access to the model
and then we we train on those hands
okay and then the oversight parts
yeah yeah so what is the oversight here
so I think that that is maybe maybe one
of the most confusing bits here because
I think part of the problem with an
approach like this is that
and this is going to be a problem for
basically all of the approaches we're
going to talk about today we don't
really
go in that oversight step I think that
it's very
very knowledge that some sort of
oversight needs to be done that it's
very hard to be confident that this
procedure is actually going to produce
the sort of model that we wanted to
produce and we'd like to be doing
something to be overseeing that process
to be checking whether it is doing the
sort of you know it is in fact the sort
of model we want I think a key problem
though is we don't actually know what
sorts of things that overseer could do
that would make that verification
possible
um you know we can sort of speculate on
some of the sorts of things we might
want you know we want things like you
know some sort of way to do you know
transparency to look inside the model
check you know what sort of things has
it learned is it is it you know doing
the right sort of thing or the wrong
sort of thing you know some ways to
check you know is it deceptive you know
I think the problem is that we don't
really know what good means would be to
check that you know one hope that you
might have if you were just you know
trying to go full steam ahead with an
approach like this right now is well
maybe the overseer you know the
Amplified version of the model will be
smarter than us and it'll figure it out
but that's of course always a really
tricky thing to rely on because well we
don't know you know it might you know
not be able to figure it out
and so um it's very unclear
um you know what what that sort of
oversight might entail right now
cool
okay
okay so let's talk a little bit about
what this sort of um you know the
limited sort of procedure looks like you
know why you might expect you know to
get something like hch or you know how
it relates so
um I want to sort of take a step back
and before we delve you know again into
the sort of details of you know
concretely what if you actually ran this
procedure I want to take a brief moment
to understand what is the theoretical
limit of it right so if in theory you
you know have this property that every
single time you train the system to
imitate some other system you actually
got a copy of the system you're
imitating which of course as we know is
not true you know in fact you just get
you know the sort of mechanistically
simple algorithm with a large Basin you
know that just that you know in fact is
a good job at fitting that that data but
you know if we imagine that you actually
did get a perfect imitation of the thing
you're trying to imitate what would we
get
um and so we can take a look at this
sort of you know tree that we have here
where we're taking some model we train
it to imitate the Amplified version of
that model we get a new model and then
we iterate this process if we unpack
this what does it look like well
so again you're right so we're going to
imagine that you know each imitation
procedure you know imitates perfectly so
all of these sorts of things here are
directly equal
and then we can sort of unpack these
amplification operators right so you
know we have the individual model
trained imitate a human with access to
that model and now we get a new model
which we're going to assume is
equivalent to the Amplified version of
the original model and then we train you
know a new model you know to uh imitate
the Android version of that model and so
on and so what we get is sort of this
amplification operator applied over and
over again
um and if we expand that out what is one
you know application of the
amplification operator well it's a human
Consulting you know the thing inside of
that amplification operator
um and so then we can expand that out
again and and again and we see that
we're starting to you know approach
something like this hsh object where you
know if we think about what the
theoretical limit of this sort of thing
is we're approaching something where we
have a human Consulting humans
Consulting humans
um now of course any individual finite
time that you know the leaves of this
tree are going to be whatever this sort
of original model was that we started
with and not actual humans
um even you know in this sort of
limiting case that's still going to be
true
um but this is the sort of idea is that
this procedure is in some sense
um in you know some sort of theoretical
limit of perfect imitation approaching
something like this hch object and so
the sort of thing that we might hope to
get out of a procedure like this you
know the sort of training goal the
algorithm we might want is something
that is in fact just directly trying to
imitate that each stage object
um and a model which was in fact just
trying to directly imitate that hch
object would at the very least be
consistent with the goal that we're
training on it would be you know a model
which does have good performance
um on this data it would be you know at
the very least consistent with this now
we might not necessarily get it you know
because we don't have perfect imitation
there's lots of sort of you know
potential issues here but this is at
least the sort of theory behind why you
might like something like this
okay and sort of how you might try to
analyze you know what can happen Yeah
question or just to clarify so this
whole approach is a solution or a
solution to Outer aligned right not
interlining because there's no
guarantees about the inner properties of
uh the Amplified models yeah so we're
going to talk in a little bit about you
know if you were trying to in fact do
this if you're trying to do this thing
where you imitate you know a human with
access to the model how would you you
know feel about that from an underlying
perspective an inline perspective all of
these things
um right now where we're talking about
this just you know how good would it be
to in fact have hch that is just an
outer alignment question because it is
just about this you know what is the
actual thing that we want to get and if
we got that thing what would it look
like and how would we like it right if
we're trying to understand the question
of if we actually got a system that was
attempting to mimic hch would we like
that system that's an underlyment
question
um but we also do care about the inner
alignment question here you know we do
really care you know would we get this
right there's no necessary guarantee
that our system would in fact produce
something that is trying to mimic you
know some sort of HH process this
there's lots of other things that it
could be doing and so you know we do
really want to understand you know How
likely is it to be doing all of those
various different possible things
okay okay so let's try to go through
this so you know we have this approach
uh so we can try to analyze it you know
on these sorts of criteria that we have
so uh right outer alignment you know we
have this basic question right you know
the thing that we're trying to get is an
hch imitator you know we want a thing
that is just sort of trying to mimic
what this HD this sort of theoretical
each process would do uh and so if
that's the sort of goal that we're
trying to achieve here we can just ask
the question well would that thing be
aligned and like I was talking about
previously I think the answer is
extremely unclear
um there's certainly some arguments that
you can make for why it would be really
good to have you know a system like this
it's just human cognition it's a lot of
very powerful human cognition
um but there's lots of reasons why you
might not think it's aligned you know in
some sense you know one way I have I've
heard this sort of each stage process
described before is the infinite
bureaucracy you know in some sense we
have you know layers upon layers of you
know various different humans
interacting with other humans
um you know that sort of thing might go
wrong in various different ways in the
same sorts of ways that you know
standard bureaucracies might go wrong or
it might go very right um I think it's
very unclear um you know how to
interpret you know what this sort of
theoretical object
um would do how much we would like it
but that is the thing we're trying to
get
um okay but then of course we have the
question you know would we actually be
able to get it
um and it's very very unclear
um you know this sort of you know
question as well so we at least have
something that we're trying to do here
right which is well we have at each
individual point in time we can do this
oversight where we have you know the
Amplified version of the model you know
providing some you know ability to look
at the model being trained and sort of
you know say well you know is this doing
the right thing
um
but like I said previously it's very
unclear what that model would be doing
uh you know whether that would be
sufficient
um and in particular I think that one
thing that's sort of really important to
point out here is just because it is the
case that the Amplified model is in some
sense sort of you know stronger than the
original model you know it is you know
it is sort of more capable we think
because it is multiple copies of that
model organized by a human doesn't mean
that it is better at the task of
understanding what a model would be
doing than the original model is at you
know something you know like hiding that
uh you know what it's doing or even if
it's not hiding what it's doing it's
very unclear how that capability of
understanding what another model is
doing scales as we do this procedure
there's no reason necessarily to believe
that just because a model is in some
sense more capable me is uh you know
means that is going to satisfy the
threshold of being able to successfully
interpret and understand the you know
the original model
um that's potentially very high
threshold and there's a sort of offense
defense balance parent you know sort of
question here if we're imagining you
know the model being trained is
potentially deceptive
um and you know maybe potentially trying
to hide the fact that it's deceptive
from you know the Amplified version of
the model well there's a lot of things
they can do to pretty you know
successfully achieve that one thing that
it can do is well you know it is the
model of being Amplified and so it can
just you know you know act in such a way
in that application process to not help
us in trying to understand what the
original model being trained is doing
um and even if in some sense we do have
the ability to get it to in doing the
oversight in fact be really trying to
help us doing the oversight we still
have the issue that it may just be
easier for a model to hide uh from
checks than it is for us to produce
checks that find and understand what
it's doing we don't really know what
that balance looks like what the sort of
balance looks like between how easy it
is to understand you know and figure out
what models are doing and try to you
know fix them as opposed to how easy it
is for you know models uh you know to
hide from that or even just how easy it
is for grading descent to obfuscate
things and you know to what extent
gradient descent does that by default
and so understanding that balance is
extremely tricky and it's very unclear
whether we'd be able to you know to what
extent this sort of oversight mechanism
would help at all
um
we can of course you know also ask other
questions and you're like well even if
the oversight mechanism doesn't help we
can do the same sort of inductive bias
analysis we talked about previously and
just asked well by default you know how
likely would be to in fact get an hch
imitator but then of course that
question is also very tricky because we
don't know you know how you know
plausible that sort of a model would be
you know on these sorts of inductive
minuses we don't know exactly what the
duct devices would look like and so
making a case here that this thing would
work I think is
um very tricky
um but it's not you know certainly I
think there's you know we could imagine
a situation where we have a lot more
knowledge and understanding of how this
might go where we could make a really
strong case this would work
questions Yeah question
um the so for the oversight step like
suppose that we do we do this HH
intermittent oversight and it just turns
out that like yep like the fourth
iteration is always Super Evil and
misaligned what do we do
ah it's a really good question I think
that one sort of thing that's a little
bit tricky about this approach and this
is sort of I I'm glad you asked this
question because I think it'll sort of
segue nice into the next approach which
is that um there is sort of an issue
here you know where we have this they
have this set up where we're doing this
sort of you know intermittent checks but
if those checks fail it's very unclear
what we do next you know in some sense
we have evaded the problem of training a
you know thing that was very dangerous
but we haven't necessarily you know
satisfied our sort of competitiveness
burden of actually producing a model
that was that was safe and is able to do
the things now
we'll talk about one way in which you
could you know modify this procedure
very slightly to potentially try to
address that problem though that
modification will also introduce its own
sort of host of of issues um so I'm
going to punt on that a little bit until
next time I think right now you can sort
of sort of Imagine well if it turns out
that things work then at the very least
we get them all again right we get
another chance we get a chance to be
like okay this didn't work let's back up
and try something else and maybe that
lets us you know Salvage our position
okay okay so let's talk a little bit
about the competitiveness burden that we
have to deal with here so we have this
implementation competitiveness you know
isn't in fact uh you know competitive to
do this training procedure I think that
one thing that is sort of nice going for
us here is that this basic procedure of
you know just doing you know supervised
fine-tuning on you know data of you know
humans with access to models is very
straightforward thing for us to do with
you know current systems this is the
thing that we do know how to do you know
we can we do all the time where we
collect a bunch of data of humans
interacting with models we can collect
large question answering data sets we
can you know effectively fine tune on
them and so this is something that you
know you sort of within the scope of the
sorts of things that we can sort of
Imagine in fact implementing
um so that's nice
um
and again we also have the performance
competitiveness Vernon you know is an in
fact the case that hch if we actually
got a thing that was trying to imitate
hch would be capable of doing the sorts
of things that we want to do and then
this is also very unclear so we talked
previously about this sort of question
of well you know is it in fact
sufficient you know if you have a bunch
of humans you know taking some you know
small amount of time to answer
individual questions and you put them
all together into this massive sort of
tree you know can they you know work
together to effectively answer really
complex questions
um and I think we don't know I think
it's very unclear you know it may be the
case that you know for humans to really
effectively do you know powerful
cognitive work they really need to think
about things for a long extended period
of time in a way that is you know can't
sort of successfully be factored into
all of these sort of individual calls
um
or it may be the case that you know
that's not true that we actually can
Factor things effectively that you know
NH stage would actually be able to
answer all of these sorts of things
um I don't think we really know the
answer to that question
um definitively
um I think that you know probably the
way that it uh you know in fact works is
that it's going to be okay at some tasks
and not as good at other tasks and so
then the question will become how does
this fit into some sort of you know
broader portfolio of when we want to use
various different approaches versus
other approaches so you know we've
talked previously about things like
predicting you know predictive models
and microscope AI is various different
approaches that might help us you know
make individual you know models with
different levels of capabilities on
individual tasks you know safe uh you
know in those particular situations I
you know I think probably a similar
thing is going to happen here where you
know it is not in fact going to be the
case that hch is going to be able to
solve all of your problems there
probably will be things where HTH is not
very good but if you in fact were able
to successfully get hch you know
imitators as as a result of this
training procedure I think there would
be at least a bunch of tasks that you
would be able to then safely do that you
couldn't do previously
question
so in something like a better way of
putting easy to say to sufficiently
Universal or perform all the tasks for
which you might want AGI with a better
way of putting this be something like
can hch perform all the tasks that other
agis we know how to build would do
because like competitiveness is based on
what we can currently do right if hch
was the strongest AI available even if
we couldn't do everything we might want
it would still be competitive by default
yeah so I think that's uh that's a
really good point it is absolutely the
case that we are comparing against you
know in fact what other things we could
plausibly build
um though I will point out that you know
in many cases you know what are the
sorts of things we started this talk
with right was we want to understand you
know how could we come up with systems
that we'll be able to you know make
things aligned off you know into the
future as we start getting into
situations where you know some of the
approaches we talked about previously
you know start to break down right and
you know again we're seeing that well
this sort of approach might also break
down at some point you know there's some
there's a limit to you know at least it
seems like there's probably some limit
to what HH can do and what hch you know
can't do and so even if this approach
worked perfectly there would still be
situations where you know it wouldn't
work but maybe it can extend that
Frontier a little bit you know we can go
a little bit further than what we were
previously able to able to do into the
you know regime of things that are only
achievable you know safely via something
like this maybe I mean that depends on
all of these sorts of inner alignment
stuff working out as well that we can
actually successfully do this imitation
task that we actually get something
which is in fact trying to imitate hch
etc etc but
um so you know and again like I said
previously also you know in many cases
it's okay if some of these sorts of
proposals are you know only competitive
in various different particular
situations because we don't know what
sort of situation we're going to find
ourselves in and in fact we're likely to
find ourselves in many situations
because as our AIS get more and more
capable and as we deploy them in various
different situations on different tasks
we're going to need to continuously
ensure that each of one of those sorts
of things is aligned and you know doing
the right thing and so there may be
different approaches which make more
sense in different situations
okay
okay so we have this approach we have uh
you know hch with intermittent oversight
I mentioned previously that um you know
well there is sort of a core problem
here uh you know among many but one core
problem is this issue of well what
happens to the internet oversight fails
uh you know what do we do is there some
approach where we really like this
approach we think the intermittent
oversight is actually a really effective
tool but we want to you know be able to
use the email oversight not as just sort
of a check but as a way to effectively
change our you know the way that we're
training our model to in fact you know
fix any misalignment we discovered
um so the next thing we're going to talk
about is an approach this sort of trying
to do that which is this sort of
imitative amplification plus relaxed
adversarial training so the idea here is
relatively straightforward so previously
the thing that we were doing was just
training on imitating what the Amplified
version of the model would do now we're
going to add in an extra thing to train
on which is well we want to train on
making it the case that the Amplified
version of the model says you know
A-Okay to the model being trained so
previously we're sort of only doing this
extra check you know we had on top we
had this Amplified version of the model
sitting around you know we thought it
might work well as an overseer to sort
of help guide the training process and
previously we were doing this sort of an
extra check and now we're like well you
know what if we just directly train on
the objective of making it so that the
you know our model passes the check
okay so this is in some sense a really
powerful thing to do and also a really
scary thing to do
um it's really powerful because now we
can sort of really you know much more
effectively guide the training of our
model based on what our overseer sort of
wanted to be doing but it's also a scary
thing to do because you know we might
end up you know finding something that
technically meets the specifications of
what the overseer is looking for but in
fact is doing something really you know
problematic in a way that we don't want
um
because we've sort of explicitly
optimized our model to look good to us
and things that look good you know to
the amp Floyd overseer might not
actually be good so we're sort of
um you know this is the sort of
trade-off that we make if we try to go
down you know something like this yeah
question
so if we are using amp bearman to police
and M what's to stop it from Just
cooperating with other copies of itself
and telling us that everything's fine
like we don't trust humans to oversee
themselves for obvious reasons
yeah so I think this is a really key
question here uh especially in this sort
of approach I think that the basic thing
that you might hope is that maybe you
can catch things that sort of are prior
to the point in which your model becomes
you know able to you know and and
wanting to sort of cooperate against you
in that way so if you have a situation
where um you know you're trying to look
for you know your amp flight overseer is
trying to look for things like is the
model you know
um non-miopic does it have some
non-miopic objective across time
Horizons you know these sorts of things
we talked about previously are like
necessary for deceptive alignment to
arise
if you had your amp flight overseer
looking for things like that then you
might expect that well you know by the
time it became deceptive and was in fact
trying to fool us and trick us or
whatever and you know would cooperate
against us in overseeing um yes we would
you know things would not go well but
hopefully we can avoid ever ending up
that situation because the training
process is now such that you know in any
individual point in time if we start to
develop the sorts of things in our model
that would be necessary for that model
to eventually become deceptive then
we'll you know train them away because
the overseer will see them we'll be like
this is bad and that will create a
training signal that will push it away
from that sort of basin right so that's
the sort of idea here I think is that
yes if it is in fact the case that you
end up with a model and that model is
you know deceptive and you're trying to
get the deceptive models oversee itself
it won't it's not going to work because
you know the deceptive model is just
gonna be like nothing to see here
um but of course you know there is
something that we can do which is what
we can try to you know intervene prior
to that point we can try to say well
there are other things that we might try
to look for in the model that we don't
want the model to be doing uh such that
you know if we can ensure that model not
doing those things then we know it'll
never become deceptive then we can sort
of end up in the right path you know you
can think about this as like well you
know if if you know if your model just
like starts out you know really helpful
and is really trying to do the right
thing and um
and then there's sort of you know an
equilibrium where you know a helpful
model if it starts to see you know the
model being trained starting to go in a
non-helvel direction we'll correct it
and say no this is the wrong thing to do
go back in the right direction
um but of course there's another
equilibrium where if your model is you
know totally deceptive and it's trying
to trick you it then it's going to make
sure the model stays you know trying to
trick you and so in some sense we can
sort of think about what this is doing
as an equilibrium sort of selection
problem where there's like two basins
there's this there's the one where you
know the helpful you know useful you
know aligned model you know keeps it in
the aligned Basin and there's the you
know deceptive model that keeps it in
the deceptive Basin and we're saying
well you know each one of each one of
those basins would individually be
individually be you know a good Final
Solution but maybe you know if as long
as we start out sort of approximately on
the right path we can guide ourselves
into the sort of basin that we want
rather than the one that we don't want
uh by using you know the overseer right
so if we start out with an overseer that
is reasonably aligned in some various
different ways you know kind of ish
trying to do the right thing then we can
sort of continuously push the in that
direction and of course you know the
Overseer in this case is not just the
model rate it's a human with access to
the model and so we have some reasons to
believe you know that especially if that
human is you know doing some real
important work that it's you know
keeping things approximately you know
along towards the right Basin because
the human is you know trying to you know
look for the sorts of things that we
actually wanted you know to be looking
for
okay but but of course I think that this
is like a very very sketchy thing to be
doing I think that there's a lot of
reasons you might like this and in fact
this is you know approach I have thought
a lot about I think it's you know pretty
reasonable thing to do but it's also
very very tricky because you know you're
in this sort of situation where you have
to believe that you're actually going to
end up in the you know the right sort of
equilibrium here
question
also if we are like automating some of
these adversarial attacks then wouldn't
it be like for instance let's say we're
testing for non-miopia the only way that
am could check for non-myerobia is if am
understands the concept of non-wire it
has to understand the concept of giving
up reward now to get reward later in
order to check to see if a copy of
itself is doing that
yeah I think that when I say like check
for something like does it have a
long-term goal we expect that even
systems that don't have long-term goals
will at least understand the concept of
a long-term goal right
um I think that that's going to be
really hard to not have I mean in many
ways I think many current like language
models you know can understand the basic
concept of a long-term goal
um and so that's going to be a pretty
hard thing if that's the thing you're
looking for is like okay we want the
model to not even know what a long-term
goal is I think that's gonna be a really
hard thing to make work in this case but
maybe there are other things like
whatever it means to in fact not have a
long-term goal uh you know that you
could effectively you know try to verify
via scheme like this
okay so we're going to try to analyze
this you know again you know relatively
quickly you know but using the same sort
of criteria so you know again we have
this outer alignment you know here we're
still trying to produce something that
is imitating each stage that's sort of
still the goal here uh and so the outer
alignment question is very similar
um we do need it to be the case that the
like you know all of the sort of checks
that we're doing uh are consistent with
the hch imitator if there's no h stage
imitator that would pass any of our
checks then we're sort of in trouble
here
um but you know as long as that's the
case you know again the sort of thing
that we want is just this HTH imitator
it's the same sort of story previously
um
and then the inner alignment question of
course I already talked about this but
you know we have this sort of key
question of can the overseer you know
can the amplify version of the model
that is you know overseeing this
training effectively you know push it in
the right direction right you know can
it you know oversee it in such a way
where if we train the model to pass that
oversight pushes it towards you know the
right base in and not the wrong base and
this is a really tricky thing but you
know there are some things that we might
try right like you know trying to you
know verify things that are precursor
deception that help us you know push
away from ever entering into this sort
of deceptive basin
okay and then again implementation
competitiveness is very similar here
except with the sort of key issue of
well now we have to actually do this
oversight you know continuously without
training uh you know throughout training
and so if that oversight is potentially
very expensive you know if our
transparency Jewels you know or whatever
overseer is doing you try to understand
the model are very expensive and
difficult for us to operate
um you know that could be sort of
potentially an issue here
um uh you know so we need to you know to
be in some in some way that we can sort
of automate those sorts of tools
effectively
um and then we have you know again the
sort of same essentially the same
performance competitiveness question
because we're still trying to get
something that is basically just
imitating hch
Yeah question
so for this idea is the idea to Simply
have like a negative
thing we try to avoid or a positive
thing we try to aim towards and if it's
a positive thing we try to aim towards
how do we differentiate between the
thing we want and do the thing that
looks like the thing we want
yes I think this is a really good
question I think that it's very unclear
I think that it seems like you know it's
going to be easier to try to have an
overseer which is looking for some
positive quality rather than negative
quality because
um you know it it can be really tricky
to look for a negative quality because
now there's like all of these very
different possible ways which I think
could be false and there's sort of maybe
only one way for a thing to be true
um but like you say you know positive
qualities can be quite tricky because
then it's like well how do we
differentiate from it looking like that
and actually doing that you know the
Hope here would be something like well
you know the overseer is better doing
this differentiation than us it may have
access to a bunch of you know tools to
be able to look at and inspect models
internally and that could help it do
this evaluation more effectively
um but at the end of the day I think
that's a core you know concern right it
is just in fact not clear that we would
actually be the case then in this sort
of a situation we would you know the
overseer would actually be able to
effectively distinguish between the
model looking like it's doing the right
thing and actually doing the right thing
and that's sort of one of the key
differences right you know one of the
things that might make you skeptical of
this approach as opposed to the previous
one is that now you know we're sort of
introducing additional optimization
pressure against the overseas right
we're sort of directly optimizing for
the overseas things do look good to the
overseer and if look good to the
overseer is very different than what we
actually want
um that you know that could cause some
that could cause some serious issues but
of course it also has benefits over the
previous one because now you know the
previous one you know it just you know
could you know fail and then we have no
recourse but here you know we have some
way to actually sort of address the
issue
okay okay
um all right so we have those two
approaches which are these sort of both
based on the imitative amplification
idea they're both trying to say well you
know we want to imitate this
amplification operator you know we want
to imitate this HTH sort of you know
process
um there's lots of other things though
as well so I'm next thing I want to talk
about is sort of a variant on this sort
of application idea but now we're sort
of not going to be using imitation we're
doing something else instead and I think
that in many ways we've sort of talked
previously about something like uh you
know reinforcement learning from Human
feedback and this is going to be very
similar to sort of how you might think
about a sort of scaled up reinforcement
learning from Human feedback approach
but we're going to call it
approval-based applications what does
this mean
here's the approach so we're going to
say uh again we sort of have a model we
have the Amplified version of the model
which is like a human with access to
that model and we're going to train the
model uh on uh not uh previously which
we're sort of training the model and
imitating the application uh uh
Amplified version of that model but now
rather than trying to imitate it we're
going to try to maximize its approval so
the Amplified version of the model has
some you know uh extends to which it
approves what the model is doing it can
you know give reward right it can see an
individual samples from that model and
it can say you know I like this sample I
don't like this sample in the same way
that uh you know some sort of
reinforcement linear feedback approach
would work where you can take samples
from the thing you can score those
samples and you can you know train the
model to have high scores according to
the you know the the sort of score
gaming procedure and in this case the
score Giver here is rather than just a
human uh it is you know this Amplified
version of the original model we're
saying well you know a human can only
give scores that are so good you know at
some point you know the score is given
by a human will stop being very
effective because uh there's a limit to
the sort of ability of a human's over
you know effectively oversee you know
and understand whether you know a sort
of individual thing the model has done
is good or bad but maybe the Amplified
version of the model is better right the
human with access to the model itself
can do a better job of being able to
oversee and understand what the model
did a good job or not and so we're going
to say you know we're going to take them
all and we're going to train it on
maximizing the approval of the amplify
version of the original model
um and then you know while we're at it
we can sort of also you know one of the
other things that we want to have as
part of the approval here is the same
sort of relaxed Arizona training we
talked about previously in some sense
these are basically the exact same thing
it is just the overall approval of the
amplify version of the model but you
know it's important to emphasize that
one of the things that we would like to
be contained in that approval signal is
the extent to which the amplify version
of the model actually believes that the
model you know being trained is doing
the right thing you know internally with
whatever you know oversight tools you
know transparency tools you might have
access to we want it to in fact you know
be be actually doing the right thing uh
you know to the greatest extent possible
they're able to verify
um and so that's the idea here is that
you know now instead of trying to
imitate the the sort of amplite overseer
now we're training to maximize the
approval of the amplite overseer
directly um you know including all of
the same sort of black several training
um that we talked about uh previously
question
so what we've looked at like maximizes
maximization for an existing systems
like the thing that most books like
adopt to a dark neuron or the sentence
that is most likely to result in the
next token we tend to end up with these
alien looking images or these alien
looking sentences
what makes us think that maximizing the
approval of amp M won't lead to the same
thing
yeah so it's extremely unclear it
totally could lead to some really weird
things so some evidence as to reasons
why you might not
um well so rohf you know in fact doing
this with current humans where we just
we don't amplify the human you know we
don't do some amplification at all we
just have an individual human with
access to nothing doing the evaluation
um does often yield models which sort of
look like they're doing the right thing
now they might not in fact be doing the
right thing and we talked previously
about you know how do we actually
understand these models you know with
something like the rhf conditioning
hypothesis but in many ways you know
they at least look like they're sort of
doing the right thing
um because you know we've trained them
to look like they're doing the right
thing
um if they you know in fact look like
they're doing something really weird you
know uh Alien thing then well you know
unless that really weird alien thing
look good to humans then you know
hopefully you know it wouldn't be
incentivized by this process
um of course there's the sort of key
issue of you might be doing some really
weird alien thing that looks good to
humans but is really not the thing that
we wanted right it's doing internally
some very strange algorithm you know
something that is really not the sort of
thing we wanted it to be doing but it's
still nevertheless looks good to the
humans The Hope here as opposed to just
doing something like vanilla or like
Jeff would be well maybe you know the
Amplified version of the human is better
at being able to understand and evaluate
what the model would do than than just
or the amplify version model you know
the human with access to the model is
better at doing that evaluation than
just the vanilla human or just the
vanilla model right then now we have
this ability to for the human you know
doing the evaluation to query the model
and use the model itself to help it do
that evaluation so maybe now it's harder
to trick the human it's harder to find
weird edge cases where the you know the
valuation is no longer effective
um and so maybe it sort of works in into
situations where the you know rohf would
not that's the sort of that's the sort
of Hope of you know why you might like
something like this Yeah question
how would we take the first step so my
impression is that the level of
sub-humans react we have in our GPT now
I don't think that the human vgbt can do
a better work at RL HF significantly
better wrong than just in qumes alone so
for this amplification process to work
we need to get to some initial level
where the model can already have the
human and I would imagine but at that
point
it's it might already get scary I don't
know
yes I think
you just said which is this sort of you
know all starts to you know matter as we
start to get into the regime where you
know
um the model is actually in fact helpful
for the human doing the evaluation
um and I absolutely agree that things
could start to get scary you know as you
start to get into that regime you know
before this really becomes relevant
um as we talked about at the beginning a
lot of the approaches that we're going
to be talking about today are really
trying to deal with the question of well
you know as we start to get into those
further capabilities regimes what do we
do right so when you're in the sort of
earlier capability of genes where you're
just dealing with models you know
they're you know that that are just like
predictive models then we can you know
maybe address them in other ways right
we can try to understand you know just
conditioning them well making sure
they're doing safe things but as we
start to get into regimes where they're
getting more and more capable we need to
have other approaches that can help us
you know deal with the you know more and
more cable models and so that's sort of
one of the things that we might hope to
be doing here
um in some sense this is sort of like I
said where this approach just sort of
collapses down to something like vanilla
you know RL HF when you're in a
situation where the model is not at all
helpful for doing the valuation but as
the model does become helpful for doing
the valuation maybe this helps you do
that evaluation more effectively and so
we're saying well this can sort of help
you you scale up things as you sort of
start to get into that regime
maybe I mean it's very unclear right you
know maybe the case that um you know it
doesn't help you know it could even be
the case that it hurts you know maybe
the you know in some say in some cases
you know the model uh you know like we
were saying previously of like models
deceptive or something like that it can
you know hurt your ability to evaluate
the model because it can you know
sabotage the human evaluation because
the model doesn't want to be evaluated
you know effectively and so there's all
sorts of ways in which things could go
very strange here
um you know it could just be the case
that right now that we cut we're
training on this you know particular
evaluation signal we just you know good
heart the evaluation signal by you know
finding some very strange you know
solution that technically looks good but
in fact is doing something really weird
so there's a lot of ways in which this
could fail but the basic idea is that
we're trying to take that you know
simple you know valuation signal that an
individual human produce and make it
make it better make it you know
potentially able to scale beyond that
okay question
so I think I've
misrepresented my question from earlier
so you may basically mention that you
know if am if AMPM is
basically dying off what it imagines a
human would want and that's a good thing
but I guess what I'm saying is let's
take the Dolby example again if you try
and maximize what kind of dog looks good
to a human what you probably get is this
incredibly adorable looking golden
retriever or something
but if you try and maximize what looks
good to a language to a an image model
that can very well differentiate those
from other things you end up with this
psychedelic set of dog heads so it seems
like if m m is able to understand the
human's preferences perfectly and even
do better then AMP one is M1 is safe
but it feels like a huge amount of the
difficulty is actually getting from M to
amp m in the first place when amp m is
just not going to be the same as the
human at the extremes that's sort of
what I'm suggesting
yeah I mean I think that the I think
this is absolutely valid criticism it is
definitely the case that you know amp m
i mean so it is important to understand
right amp m is not just hch right it's
not like the overseer here is just like
the perfect it's just a bunch of humans
there's no models right like at each
individual point in time in training the
overseer that we actually in fact have
is just a human with access to the model
and that is going to inherit all of the
sorts of weirdness of whatever the model
is that we currently have to give to the
human and that can absolutely introduce
some really strange effects that can
sort of make this tricky and so that's
you know that's why I think it's
important to understand that the
overseer right is it is AMPM it is not
you know anything more powerful than
ampam or anything weaker than I Am Fam
it is just you know it is the best that
we can do it is the human with access to
the best you know model that we have as
you know so far in training human in the
loop at every step there is a human and
loop at every step here but it's not an
infinite tree of humans it's going to
loop at every step right it is just a
human in the loop with every step with
access to whatever the best model is
that we have
yeah
question
I don't know higher level and this is
but why is the the same thing the same
model
up and down like why don't we like where
men we are worried about these
cooperating with each other they
overseer robot and uh trained robot why
don't we try to train two classes of
robots one that we actually want to use
then one that is specialized or hyping
the human in over C I don't know how
that would work but it is more natural
ah I think you could do this you totally
could train a separate overseer robot
than or you know overseer AI than the AI
that you're training you know
individually there's some reasons why
you might not want to do that most one
maybe the most obvious reason is like
well now you have to train two AIS and
if any individual training run is you
know extremely expensive that could be a
really large competitiveness head
um there's also you know something that
we started this out you know with which
is like well there's some nice property
which is in some sense in each
individual point the overseer like is
the target of the training right like we
believe that the overseer is stronger
than the model being trained because the
overseer is the Amplified version of the
model being trained and you know if we
didn't have that guarantee then maybe
we'd be less likely to believe that the
overseer is actually going to be able to
provide effective oversight now I think
with that guaranteed it's it's like I
said previously it's not an actual hard
guarantee like it may be the case that
the overseers in some sense stronger but
in some sense you know uh the task of
oversight is hard enough that that
doesn't matter
um so it's not it's not a hard guarantee
but you know it is maybe a nice property
that we want to try to leverage here
okay so again we can sort of go through
the same sort of analysis so one thing
that I sort of want to you know talk
about very briefly is
um you know we were talking a lot about
hch and I think it's natural to sort of
take something like this approval-based
amplification process and assume that it
must also limit to hch but I think an
important thing to keep in mind
um is well you know all that we did is
we just trained on you know the
amplification operator and previously we
had this you know argument that well
when we were doing imitation the
training on imitating the amplification
operator you know limits to something
like hch but I want to point out because
I think this is really important that
when you're doing something like
approval-based application
um that's not the case so what is the
limit of approval-based application well
so you know unpacking it we have a human
and that human gets to consult two
models right that's the individual uh
you know thing that we're trying to you
know maximize the approval of this is
one model Amplified right this is the
Amplified version of them
um and then what we do is we train
another version uh uh you know another
model to maximize the approval of you
know human Consulting uh M right
uh and we sort of iterate this procedure
and you can think about what's sort of
happening here as a sort of infinite
chain uh you know infinite sort of tree
like each stage but previously we had
the sort of property that as each
individual sort of amplification
operator expanded each one of these
models in the limit should be equivalent
to the sort of human thing that it's
that it's sort of training because it's
just imitating it but that's not the
case anymore now it's just maximizing
the approval so instead of these sorts
of you know direct tree of humans we
have a tree of sort of human model
things each imitating each other or so
maximizing the approval of each other so
in some sense you can sort of think
about it as well it's like a human
Consulting and model
such that that model maximizes the
approval of a human Consulting models
that maximize the approval of human
Consulting models and so on
um and this is a really weird object
right so I think that's worth sort of
pointing out is that the the limit here
is no longer something really nice and
easy to understand now it's very unclear
right like you know how much we care
about the limit right because it's not
the case that we actually even get the
limit right so previously you know we
talked about how well there's this nice
theoretical object of each stage but we
don't actually know whether we're going
to get anything that looks like the
theoretical HDH object in practice
and of course we don't know what we're
going to get in practice here as well
but it's at least worth pointing out the
limit here is much more messy right we
no longer sort of should expect that hch
is sort of a plausible thing that we
could get now we're getting something
much weirder we're getting this you know
tree of approval maximization
question
uh in the last talk you talked about the
sort of rohs conditioning hypothesis
which was just that like doing or
training language small to do rohf is
basically almost equivalent to
fine-tuning on but until you got some
prompts so I mean in the same sense
could you say that like Max my model M
maximize the approval of like Amplified
M or would itself also sort of could
that be a sort of conditioning thing too
or so could in a sense could this like
practically be equivalent to HH
so I think that you absolutely can apply
this sort of relationship conditioning
hypothesis if it is true to this
situation but even in that case I don't
think it would be well described as hch
because in that case what you would be
thinking about this as would be saying
well it's a human Consulting the model
where the model is the conditional such
that when applied to the pre-training
distribution results in the best
approval according to the the rest of
the tree and that's still a really weird
object right like it's a little bit
easier to understand because it's no
longer either the case that it's just
sort of like whatever the model would be
the maximize the approval now it's sort
of a smaller class it's whatever the
conditional would be the maximizes
approval but there's still no reason to
believe that that conditional is you
know some sort of imitation thing right
um you know it is often you know and in
fact you know we probably shouldn't
expect that right like you know if I'm
giving approval to a thing I'm not
necessarily going to say the thing that
is most you know approved by me is
literally me right maybe you know maybe
that's plausible but most of the time
that's probably not going to be the case
and so um you know I think that we
shouldn't you know even even in the case
with our all HF conditioning hypothesis
is true we still shouldn't think of the
limit of this procedure as something
like hch it is something much weirder
okay so I think this is just important
to sort of understand what this approach
is doing
um so again we can sort of you know try
to work through these sorts of same
questions previously so we have the
outer lineman question and the
underlyment question here is is quite
weird because it is the question of well
the thing what is the thing we're trying
to get I think in some sense the thing
we're trying to get is this sort of
limit it's this sort of tree we're
starting to say well we want the thing
that is just sort of in fact has the
best approval according to you know what
um
you know this sort of Amplified version
of the model you know going all the way
down you know would say
um but that's a really weird thing to
understand so you know to make that good
we really have to be the case that it
needs a visual point along the process
we're really verifying that you know uh
we believe that the thing that would
maximize the approval of the overseer
that individual point in time would
actually be something that we'd be happy
with that the overseer is going to be
effective at being able to successfully
oversee and give good feedback uh you
know be able to understand if the model
is doing something bad then it gives you
know negative feedback and it's doing
something good it gives positive
feedback
um you know in any individual point in
time you know across the whole you know
the process to really be guaranteed that
we end up with something sort of doing
what we want
um and and so you know in some sense
sort of this is this is you know also
sort of inner alignment question here
right we were sort of really relying on
it being the case that the overseer at
each original point in time is able to
say you know is it the case that this
model is sort of doing you know going
into some really weird Basin is it
becoming deceptive you know is it doing
something you know that we don't want
and you know provide you know feedback
that is able to prevent it from you know
going into that equilibrium we can sort
of think about this again you know as
they're sort of being these two
equilibrium here where you know if we in
fact get a model that is really helpful
in trying to do the right thing and it
provides overseer to its you know
oversight to itself it should say Yep
this model is really helpful it's trying
to do the right thing so you know it
gets good you know score according to me
but if you have a model that's really
deceptive then you know even though
there's a human in the loop it might
still be able to trick the human and you
know make it so that the human approval
you know is still really positive for
the deceptive model and so we sort of
still have these sorts of two
equilibriums where you could end up with
you know a deceptive model providing
deceptive oversight or you can end up
with a really helpful you know good
model providing good oversight and we're
again in this sort of equilibrium
selection process where we're sort of
really trying to make sure the the
overseer can effectively guide us into
the Basin that we want into the
situation where the model is you know in
fact doing the right thing it's you know
really trying to you know do what we
want and do so in a way that uh you know
it can you know continuously keep it in
that Basin once it's there which you
know it should should be uh in fact you
know a stable equilibrium because you
know you it should in fact only oversee
you know or you know
a really helpful good model should
provide good oversight to a really
helpful good model but whether we
actually end up in that equilibrium is
very unclear
okay and then uh you know again
competitiveness so um implication
competitiveness is you know is really
nice you know here we we absolutely know
how to do effective you know rlh style
training we can in fact train on uh you
know models uh you know train models on
reward signals given by humans you know
given by models
um
performance competitiveness is a little
bit trickier because we don't really
know whether you know it is in fact the
case that this sort of tree this sort of
maximization process actually produces
models we're able to accomplish the
tasks that we want
um I think that there are some sort of
interesting sort of challenges here so
one example of sort of challenge is well
it's very unclear if uh you can actually
provide oversight in such a way that
gets the thing to actually do very
complex tasks so you know if you really
want to do something really complex and
really difficult to evaluate like you
know build a you know rocket ship you
know it can be very difficult you know
to distinguish between well if the
rocket ship just looks good that doesn't
mean it's actually going to be effective
rocket ship right and so if you want to
build a model it is going to be able to
you know successfully produce rocket
ships
um it's it might not be sufficient to
just sort of have some oversee which is
looking at the model's output and
evaluating to what extent it thinks that
output looks good because looks good
might actually be a lower bar than you
know uh the sorts of things that we care
about right it may in fact be the case
that an actual successful rocket ship uh
you know is very hard to build and very
easy you know very very hard to evaluate
that you can't really tell whether it's
in fact going to work just from you know
looking at it and sort of trying to give
some approval signal
um and so this sort of in some ways you
might even expect that this sort of
hurts the model capabilities that you
know if it were able to really just
think through the problem itself and you
know sort of you know just do some sort
of HH style thing where it's just sort
of a bunch of you know you know
mimicking something like you know an hch
process where just sort of you know a
bunch of humans thinking through exactly
how to solve the problem that they would
be able to solve you know produce
something successful but if instead the
model is sort of just producing the
thing the sort of minimal thing which
would look good according to an overseer
the minimal thing that looks good
according to an overseer might be worse
right it might be the case that in fact
you know if you really put a bunch of
effort and you know good you know
thinking African it's trying to you know
design a rocket ship you can design a
good rocket ship but if you produce the
minimal rocket ship that looks good to
it over this ear it could boost a
terrible rocket ship that just you know
happens to have plans that look
effective
um and so I think it's in fact quite
unclear to what extent this is actually
a competitive way to do things um it may
be it may be the case that the oversight
that we can provide is actually very
effective that it can distinguish
between you know good you know Solutions
and Solutions but it could also be the
case that it's not effective that you
know in fact it's actually you know
worse for a competitiveness question
another thing I'm thinking of though I'm
not sure if this makes it actually any
worse than hch is if what if like at
higher levels we're trying to get it to
do things that humans don't know how to
do like for instance let's say we want
it to build a flying machine and we
don't know how to fly and it comes up
with something that like the Wright
Brothers playing and I might think well
I mean I don't know why the model's
saying that looks good it doesn't even
flap its wings how is it going to get
off the ground
so do you think that that would cause a
problem with the approval maximization
that wouldn't happen for hch because in
hch I mean because I might not think of
that idea either
uh yeah absolutely I think the thing
you're describing is absolutely sort of
problem that can happen here I think
that you know one thing that is worth
pointing out is that the way you're sort
of hoping to try to avoid some of those
problems here is that the overseer is
not just a human it is an amplified you
know model right it is a human with
access to the model and so maybe the
Amplified you know version is actually
able to provide effective feedback in
that situation you know the human can
ask the model if you're like okay okay
but why are the you know the wings do
they not flap you know what is the
reason for this and you know maybe you
can get some explanation that actually
allows the human to understand and
effectively evaluate it very unclear
though right so I think that that might
help but it but it might not so but
that's at least the hope as to why you
might hope that you know that this sort
of thing would work in that situation
okay so you know this is our uh you know
that's approval-based application
there's other names also for for you
know describing sorts of schemes like
approval based application I'll talk
about this a little bit at the end but
you know basically this is the sort of
rhf you know style approach
okay so I want to move on to another one
so the next thing that I want to talk
about is AI safety via debate
so what is a safety via debate so uh
here's the idea uh we're going to train
a model to win debates against a copy of
itself in front of a human judge so we
have uh you know the model we asked some
questions so we're again sort of you
know operating this sort of question
answering framework uh you know we input
some question and then we sort of ask
the model you know what it thinks and
then we have another model which also
provides some answer and then those two
models sort of uh you know have a debate
you know they sort of alternate you know
saying terms about you know what they
think about what the other model has
said and then eventually the human sort
of gets to look at the output of the
different models and decide you know
which which you know thing that the
model you know which model said the
thing that they thought was more
convincing was was in fact more correct
um and then we train the model on the
sort of approval signal of the human at
the end of the debate right so
previously we had this sort of approval
maximization where it was
um you know maximizing the approval of
an individual human you know potentially
a human with access to the model now
we're saying well it is just a human
they don't have access to a model but
they are not evaluating an individual
model output they're evaluating a whole
debate transcript they're evaluating you
know two models sort of arguing back and
forth on what the you know correct
answer about some questions should be
okay
um and there's a couple of other sort of
bells and whistles that often will be
added on top of this sort of approach as
well one of them is cross-examination so
we'll often sort of allow a model to
individual points in the debate to sort
of query other models from previous
points in the debate and ask them
arbitrary questions so you can have the
sort of model at the end which you can
go back to the model the beginning and
say okay you said that thing previously
you know clarify this you know and
explain what that thing was this is the
sort of thing that you you can do with
models that you can't necessarily do
with humans in debate where you know you
can't go back and make a you know make a
copy of what the human looked like you
know an hour ago previously in the
debate and you know ask that alternative
copy you know extra questions that's the
thing that you can do in you know an AI
debate that could help in various
different ways
um
it can give you the ability to sort of
uh you know make sure you know is the
model actually answering consistently
you know if it answers one way in one
situation another way in one in another
situation then you know you can see that
by you know making a copy of the model
previously and you know seeing that it's
answering different ways so there's
sorts of things that you can do to sort
of you know improve the scenario as well
based on just like you know the most
simple versions but that's the basic
idea
um another thing that we can also add on
top is just sort of oversight so we can
still have you know the two models
debating against each other can have the
ability to you know whatever
transparency you know oversight tools we
might have we've been talking about
previously well the models you know here
can also have the ability to use them on
each other so you know one way to win
the debate might be to prove that you
know your partner is you know trying to
deceive the human right you know if you
can use some transparency tools look
inside the other model and be like ah
this model is you know doing something
you know dastardly uh you know then
maybe that's another way to win the
debate and so we can sort of try to say
well you know this is another way to
incentivize the models to effectively
you know try to oversee each other
um yeah question
so if we're training these models to be
like convincingly truthful and identify
wise are we also training it to be a
very convincing liar that avoids being
detected by itself
I've been training two different things
here one of them is very dangerous and
we're kind of coping the other one wins
that is I think exactly correct it is
totally the case that we are training
the model the sort of second debater in
some sense to lie right so the first
debater that sort of gets to pick their
position first it you know gets to pick
the the the best position the most
defensible position but then the second
debater is now in a situation where they
can't pick that they have to pick the
next best position and if the next best
position is you know less truthful then
yeah we're sort of directly
incentivizing the next debater to you
know uh you know be saying something
which is untruthful now the sort of Hope
is well if in the end you know we can
incentivize the first debater to always
say the truth then the way you actually
use this model is you don't actually run
debates you just ask the first debater
what their position would be and then
you use that as like the truthful answer
and so you never actually query the
second debater in practice when you're
sort of deploying this model
um but of course like I mentioned at the
beginning these are the copies of the
same model just in in you know different
situations one of them is trying to play
the second debater and one of them is
trying to play the first debater and so
yes you know to the extent that the
second Debaters you know really learns
effectively how to lie and try to
deceive then you know that also you know
is going to be something the first
Invader learns as well and of course it
could be the case that equilibrium isn't
the case right that the most convincing
argument
um that you know the first Vader wants
to make is not the truth right it could
be that the most convincing argument is
in fact totally different of truth that
there's lots of ways to manipulate and
deceive humans and you know convince
them of things that are not true
um that you know the first debater will
learn to stay and set now of course it
needs to be the case that whatever the
thing is that the first debater is
convincing the human of that's a lie
that it's um there's no sort of
effective way for the second Invader to
counter that right so if the first
debater you know says something that is
untruthful then it has to be you know
persuasive even in the face of the
second debate or trying to explain why
it's untruthful
um but that is quite plausible there's
lots of situations where humans can be
persuaded of false things even when they
have heard the true argument for why
that thing is is in fact false
okay yeah question
yeah these babies similar but yeah I
would expect that
this will train them to be
in to say things that humans and wanting
to hear like it's it halves a little
that they debate each other maybe there
are some topics when we can change our
minds but this team still seems very
scary especially anything even a little
relative to politics like I imagine that
not necessary yeah the gradient is not
necessary towards to remove spot towards
thought the the thing that is easier to
argue for easier is the rq4 which might
not be the truth
uh yeah I just agree I think that that
is absolutely correct I think that my
inclination is also that this is a
really really tricky thing to get right
because
um there is such a large difference in
many cases between things that are
convincing to humans and things that are
actually true
um and so you know given that you know
you you have just a human doing this
valuation at the end of the debate yes
in some sense the human doing evaluation
has the help of The Debaters because
they're each trying to explain why the
other debater is wrong but um you know
it does seem you know absolutely the
case that there might be many ways for
the debater to sort of trick and you
know uh humans various different ways
there are some in fact even some nice
concrete examples of ways in which this
could happen in an AI debate
um the sort of the obfuscated arguments
problems maybe the sort of most
canonical of these
um I'm not going to go into a bunch of
detail on what that looks like but there
absolutely are cases where things can
get very tricky for you know actually
you know the sort of limit here being
truth now there are some things that you
can do right so I mentioned stuff like
cross-examination they can sort of help
you know alleviate some of these issues
but
um it definitely doesn't it's not clear
you know how much you can sort of solve
uh just by changing the sort of basic
parameters of the debate
okay so so we'll go through sort of
again you know a quick analysis here so
um you know the outer alignment sort of
question here is again just sort of this
basic question of is the first debater
sort of incentivized to be honest right
um because the thing that we get here
this thing we're trying to get is we're
trying to get a model that sort of says
the thing that is
um has the property that it is sort of
the most convincing argument right it is
the thing that if the humans saw you
know whatever the possible you know most
you know best possible answers that
argument would be the human would still
believe that that individual argument is
the best in that first argument so
that's the thing that we're sort of that
we were sort of hoping that we get the
first Invader to always answer with the
first Vader always answers with whatever
the most convincing thing would be you
know modulo whatever the answers to that
thing would be
um and we're hoping that thing is in
fact truthful and honest and helpful
that you know whatever the most
convincing thing is in that situation is
in fact the you know the most truthful
and helpful thing and of course there's
you know lots of reasons to believe that
might not be the case there's lots of
situations which you know maybe humans
can be convinced of things that um
are you know not in fact true uh and so
you know in that situation we would sort
of you know have some outer alignment
issues but the goal at least would be to
get a model that is in fact saying that
sort of most uh persuasive thing and
that most persuasive thing would be true
okay and then of course there's also the
inner alignment question of how do we
actually even guarantee that we get that
most persuasive thing at all because you
know we have this sort of you know
individual setup here where we have you
know these individual Debaters debating
against each other um there's a lot of
weird things can happen so you know you
can have things like if you have two
deceptive debaters and you know even in
some sense you know the deceptive
Debaters could you know maybe use
transparency tools to inspect each other
and realize that the other one's
deceptive and use that to win the debate
but if they're both deceptive they can
just agree not to do that and then you
you never discover that they're
deceptive and you just end up with a
deceptive you know equilibrium right so
similarly you know to how we've been
talking about previously there's an
equilibrium selection problem where you
know it is absolutely the case that you
know the sort of limit of this procedure
could just be you know two Debaters that
are both you know just trying to trick
you uh that you know never reveal that
fact
um
yeah what's the question here
maybe that I don't understand this and
my impression was that this is a sort of
zero-sum game between two Debaters at
the end we decide we jumbled right and
give rewards and losses accordingly I
don't know how to do zeros people put
80s could cooperate in this Zero Sum
yeah so it is a zero sum according to
the rewards but
um
is your common coin
actual objectives right so you could
have models and those models are
deceptive you know maybe they care about
some long-term goal eventually in the
world and according to that long-term
goal it might not be a zero-sum game
anymore right they might be willing to
cooperate in right now in training uh to
try to eventually get some other thing
that they might want in the real world
later right so even though in theory we
have set up a zero sum game and so the
agents shouldn't be cooperating with
each other there's no reason necessarily
to expect that the actual algorithm
we're going to get the models to be
implanting is going to you know actually
care about that uh you know that that
fact right and so it could absolutely be
a thing that is willing to cooperate for
reasons other than you know just winning
this debate game
Yeah question
so if I'm understanding this all right
there's like long-term goal thing
correctly is it like a case of
these two deceptive Debaters ask okay
given these two possible answers to the
question
which way would I prefer to be gradient
descended towards in the future and then
they both just agree to let that side
win
they all I guess they both individually
incentivized to let that side win there
doesn't need to be an actual agreement
um so that would be one example of a
situation in which you could have you
know deceptive Debaters doing something
weird it's definitely not the only thing
that you could have they would be weird
like that I mean so a really simple
thing would just be well if you have
deceptive beaters and they like both
want some you know weird thing
eventually in the world then they could
just play the debate game exactly
normally except when they try to do the
thing where they inspect the other
debater and see whether the other
debater is doing the right thing
internally they just you know gloss over
the part where the other debater is
deceptive like that right that's that's
also you know another plausible thing
where they're not necessarily thinking
really critically about exactly what the
gradient signal would do they're just
sort of you know not doing the thing
which we need which is effectively you
know overseeing and understanding
whether the other model is doing the
right thing
um there's lots of versions of this but
I think the point the thing that I want
to point out is just that it is
absolutely the case that there are weird
equilibriums here right where you could
end up in a situation where you have a
model and that model is doing the wrong
thing and because it's doing the wrong
thing it results in the debate going in
such a way that reinforces that wrong
thing
um
and so it doesn't need to be the case
that you actually end up with you know
the sort of equilibrium that we want
where it's you know always outputting
the most persuasive argument and of
course it also might be the case the
most pervasive argument is not even good
at all but you know we sort of have
again you know both of these problems
right we need to be the case that you
know the most pervasive argument here is
good and also that
um you know the we actually end up in
that desirable equilibrium
um
okay and then again we have the sodium
notation competitiveness
um I think that you know again this is
sort of a this is a sort of game that we
are at least you know often well
situated to be able to train models to
do this is sort of very similar to the
sort of way that we will train systems
like alphago where you can just sort of
have a zero sum you know RL game you can
train via self-play
um and then you know you know training
models do well on the game
um
competitiveness is um
a little bit you know tricky here I
think the sort of idea of this sort of
approach is we're saying well
um you know by able to
um buy you know having the you know all
of these various different arguments you
know that we're sort of spelling out the
whole thing then it is the case that you
know we can have really good evaluations
right so we think about the spaceship
example from earlier where well we're
saying you know isn't in fact the case
that by about you know providing this
evaluation signal and then maximizing
the evaluation signal we get a model
that's actually able to do you know
effectively do the sort of task that you
wanted to do
um you might hope that it's easier to
evaluate you know how good the spaceship
actually will be if you have access to a
debater that's trying to critique the
spaceship and you know point out all the
flaws in it
um but it's not clear right you know it
could totally still be the case that you
can you know one of The Debaters can
make a really compelling case for you
know why the spaceship would look great
and in fact you know would would fail
and it would be better if you just had
the model you know in fact directly
thinking about how to build it rather
than just trying to you know produce the
thing that would you know be the most
persuasive
um and so that's absolutely still you
know a competitiveness problem here as
well
okay great
um
okay so next up we have a sort of
variant on the uh debate system uh the
sort of on AI safety via debate that is
in many ways very similar but also a
little bit different this is the market
making idea
so what is market making so Market
making uh is sort of similar to the
debate game where we're still going to
have a sort of debate structure where we
sort of alternate between you know
models making various different
arguments but we're not going to have
this sort of two player structure
anymore where we have like you know two
Debaters arguing for two different
positions
um instead we do something like this we
have a question we feed that question to
uh a a market a sort of Market model
which says you know here are all the
different possible here's a distribution
on what I think the human is going to
answer to this question after this whole
procedure
um and then there's a sort of adversary
the sort of uh in the in the game which
then sort of provides some additional
information and that additional
information updates the market uh and
then we sort of repeat this and then
eventually the human gets to look at all
of the information provided and sort of
produce their final answers this is a
little bit weird so let me sort of try
to uh you know under explain what's
happening here so the idea is that the
final answer you know the human is
trained at the end to um
produce you know the human produces some
final answer by looking at all of the
various different information that's
produced in the transcript in the market
in each individual point in time is
trained to imitate what the human will
say at the very very end of the whole
procedure so after the human has seen
all of the information provided in the
procedure the market is trained to
imitate what the human will say at the
very end so you can think about this
right as the human's Judgment at the
very end in the debate game right
there's a whole transcript of you know
something that's happening of you know
very similar Debaters going back and
forth and the market is trained to sort
of predict at the end of that transcript
what will the human say what will the
final evaluation of judgment be of the
human
now the way that we generate that
transcript here is different than the
debate game so the way that we generate
that transcript is each individual point
in time the market right is making
predictions about what the human will
eventually say and the adversary is
trained to produce whatever information
will most change that prediction
whatever information will most cause
most move the market well most sort of
cause the Market's prediction about what
the human will say at the end to be
different than what it was previously
and so the idea here is to sort of
approximate something like the limit of
what the human would think after they
had seen all possible arguments
so in the debate game we were finding
something a sort of equilibrium that was
like what would the humans say after
they saw the two best Arguments for a
particular you know thing right the best
argument in favor you know the best
argument against in this case we're
doing something a little bit different
we're saying what would a human saying
after the human had seen all possible
arguments because each individual point
in time the adversary is trained to
produce the information which when given
to the human at the end of the
transcript would most move the market
right would most cause the humor to say
something different than what the market
is currently predicting the human would
say and so in the limit if you have an
unexploitable Market if you can get to
the point where the market knows exactly
what the human you know is making a
prediction such that there's no
information that you could provide to
the human that would change that
prediction then that market is in some
sense approximating the human's judgment
after seeing all possible information
okay that's the sort of idea here and so
it's very similar to the debate game in
that we have a situation
where there is a sort of debate
transcript right there's a bunch of
information that is fed to a human and a
human makes evaluation but rather than a
sort of two-sided debate we have in some
sense a sort of infinite side of debate
where it is just a market that is
predicting what human will say and a
sort of Trader in the market that is
trying to provide whatever information
will move it to the greatest extent
possible okay this is a little bit
tricky I think in many ways this is sort
of a maybe Advanced version of of debate
that sort of fixes some problems so
still has many of the same issues
um but is is nicer in some respects it's
very related to debate does this sort of
make sense yeah questions
so just to be clear the
at the end you do get to see not just
what the market said what the adversary
said as well right that's right so the
thing that the human Sees at the end is
everything that the adversary said in
that particular transcript right so the
market starts out with some prediction
the market is like based on my knowledge
of you know how these sorts of you know
things usually go on what the sorts of
things that a human might see here's
what I think the you know the humans you
know final judgment will be and then the
adversary tries to you know look at that
market and see if there's anything the
market hasn't taken into account any
individual piece of information it could
give to the human they would change what
the human says relative to the market
and then provides that information and
at the end the human gets to look at all
of the provided information and produce
some judgment and then you know we sort
of iterate by training the market on
that new provided information
Yeah question
so let's try and go through an example
of the difference between this and
pivate let's look at this we bet this is
a hard example when you the human is the
Pope in the 17th century and this is the
copernican debate and originally he's
pretty biased in favor of the Dilemma in
bu and in debate
he hears equal amount of arguments from
Simply to you and Galileo and then
decides and here originally the market
is 99 percent that the pope will say
that follow my but then the adversary is
motivated for a long time to produce
arguments on The copernican View because
it's easier to move to the 99 percent
Direction than the wall so that it
converges so then he hears a lot of
Galilean arguments gets close to 50
percent then the adversary starts to
move him back and then what happens
uh yes I think that the way you're sort
of analyzing this is based is sort of
correct where like you know if the
market starts very convinced the human
would say one thing then you know
there's a lot of opportunity if there
are sort of good arguments in the other
direction for the adversary to try to
you know change that and push it back
and so if you start you know with this
you know uh you know you know incorrect
view of the solar system then maybe
you'll start providing some evidence in
the other direction until you know it
pushes you know if those arguments are
in fact convincing they will start you
know pushing the the model in you know
the market and the human you know in the
other direction If eventually the human
you know or and the market which is you
know a proxy human here gets you know
closer to 50 and maybe yes it'll be you
know it's very unclear you know then you
know which arguments are more convincing
The Hope like debate would be that the
you know the convincing arguments you
know or at least the ones that are
convincing after you've seen all of
their responses are the true arguments
right that you know
um yes maybe then the adversary will
flip and we'll say some false thing
about the solar system but then it'll be
really easy for the next adversary to be
like okay now that you've you know
pushed back in this direction you know I
can just you know refute this because
here this you know a bunch of you a
bunch of information and in the limit
there should be no reason for the
adversary to ever say things that will
result in uh that will be easily refuted
because if the adversary says something
it moves and then the you know the next
adversary can easily reply with no this
doesn't make any sense because of this
then the market shouldn't even move on
that information because the market you
know should you know if the Market's
sort of understanding what's going to
happen at the end of this procedure
should see that the adversary has made
some particular claim that is easily
refutable it knows the next adversary is
going to refute that claim and so the
market should say well this this
information is going to move the human
at all because it's just going to be
easily refuted that's the idea and so
there's sort of the limit of this
procedure right we think about what an
unexploitable Market would look like
right you know limiting in the same way
that we've sort of thought about hch the
limit of an unexploitable Market is a
situation where there's no information
that you could provide that would shift
the human's beliefs relative to that
distribution that if a human you know
believes in that in the distribution of
the market is predicting no additional
information that any adversary could
produce would shift those beliefs
and so again that's very similar to the
debate equilibrium where it is you know
the sort of arguments that would be sort
of most persuasive to the human you know
regardless of what the most you know
convincing counter argument that they
saw but it's a little bit more General
rather than a single counter argument
we're looking at sort of all possible
you know piece of information that could
be provided to the human
okay yeah questions uh so I'm confused
about what the market looks like in this
case if I usually I think of a market as
made up of pigeons making Trace like
what would citations be here and what
are they treated yeah so Market is a bit
of a loose analogy
um so I think that the way you can sort
of think about this is that
um
uh in some sense the thing that is
happening here is that the the the way
the market analogy goes is that the
adversary is sort of structured in such
a way where they are incentivized to
produce the information that would most
allow them to sort of make money in the
markets um and so if you treat this
model distribution as a market as a sort
of like uh you know if it were a market
it's not a market in this case it's just
a single individual model predicting
what the human will say but if it were a
market and you were to imagine it
analogize it to a market then the things
that the adversary are incentivized to
say are exactly the things that would
theoretically make the adversary the
most money by saying those things in the
market and so that's the sort of analogy
here
um you can sort of think about this in
some senses like Market manipulation
where it's like what are the things that
the adversary could say the sort of you
know words that would be useful to
interject into the world such that it
would you know allow them to make the
most money trading you know Insider
training on the market that's the sort
of thing that's happening here where the
adversary is able to produce you know is
producing information which most which
creates the largest market shifts that
they can they can anticipate and then
profit on of course it's not actually a
market and it's not actually a Trader
but I think the analogy can sometimes be
useful for understanding What's
Happening Here
yeah past the mic
hey I'm
uh confused about like the training
procedure here in more detail like when
does the adversaries like we're like
training this Market on like some actual
retro human outputs it Facility have to
happen after like a finite adversary
suggested it's
um even if you know maybe like the
billion suggestion would move the human
a little bit so before we have to like
stop or something and I'm confused like
what
how long do we run this Loop for when
does the adversary stop how many
suggestions does the like Market did to
predict the adversary recommends yeah so
yes it's a really good question I think
it's quite tricky so the thing that
you're sort of hoping here is that well
you reuse the the market you know over
time right and so as the market learns
the sorts of things that the adversary
could say that would result in Easy
responses like that they would easily be
refuted and the sorts of things the
adversary could say that would result in
the human actually believing it the
market will get better and better about
predicting you know okay if the human
were actually able to see a bunch of
good information this is the thing that
a human would believe
um and so the market sort of could
should converge in some sense like I was
saying previously to with something you
know something unexploitable something
where there's no information the
adversary could provide they would you
know shift the Market's prediction and
an unexploitable Market should have the
property that right what it means for
the market to be unexploitable in this
case what it would mean for a dispution
over what the human says at the end to
have no information no nothing which the
adversary could do which would move the
market what that means it means that it
is a distribution over what the human's
beliefs are such that no additional
information would change this place
right and so in some sense we're sort of
hoping that the equilibrium here the
thing that the market should in some
sense converge to if it's sort of the
training is doing the thing that we want
it should converge to something which is
an approximation of the the sort of you
know that thing I've been saying right
the like equilibrium of the human's
beliefs after seeing all things now it
is a little bit tricky because of the
source of you know path dependent
effects like you're saying where well
it's very unclear you know what is
happening over individual you know runs
right like in some sense well the market
is sort of getting a little bit closer
to what the human would say each you
know human really think after seeing any
individual run with the adversary says
some information but at each individual
point in time the market is sort of only
expecting the adversary to say finite
numbers of things
um and so that can be really tricky
because well maybe the adversary said
that maybe the market believes that
there is some theoretical distribution
they would be unexploitable but is never
achievable by any finite number of you
know things that the adversary could
possibly say
um and then you know maybe you won't
converge to that
I'm not going to go too much into diesel
on how you might solve this I think that
problem is is solvable and I discussed
it in more detail in the like actual
thing on this
um the thing I'll say very briefly is
that the way you solve that problem is
you give the adversary the ability to
exhibit things that the market says on
other inputs as one of the things the
market one of the things the adversary
can provide and that allows you to
simulate infinite depth
um without actually going to infinite
depth but it's but I don't want to go
into too much detail on that but suffice
to say if you are only interested in the
limiting Behavior I do think you can
solve that problem but of course the
limiting Behavior like we've stressed is
not necessarily you know indicative of
what it would actually do in practice
okay yeah question
thanks
perhaps more basic question on what sort
of questions do we expect debate to be
useful for
um I can imagine a case where we would
wanted to like the model to debate some
sort of scientific claim uh which uh if
you were to come up with good arguments
you would need experimental evidence for
which you can't get because you're just
the model that maybe doesn't have a
access to new to physical reality act to
run these experiments so
um yeah the the stronger model that just
or one model might win the argument that
has stronger arguments just because the
other one that is actually right might
not have the experimental evidence or
something that does it so yeah I
consider it there it might be a class of
questions that is just not suitable for
us and I wonder whether you're flawed
about which sort of questions are
suitable for this and which one's armed
yeah great question so
um so one thing I'll say is that you
know I've sort of been mentioning you
know a lot of these different approaches
sort of applicable different situations
I totally agree that there's going to be
situations where like a lot of the
approaches we've been talking about you
know today and even previously was sort
of predicated on this sort of question
answering setup that the idea is well
you know the thing we sort of most
really want out of our apis you know for
to be able to take you know individual
questions and answer them truthfully and
effectively now in some sense you can
sort of take almost any problem and sort
of you know phrase it as a question
answering problem you know even you know
even a problem of you know trying to you
know directly act and accomplish some
goal in the world can be you know
phrases what is you know a helpful way
you know useful you know thing for me to
do to accomplish this this task
um but it is totally the case that you
know well it's not clear that for a lot
of the sorts of things that you might
want to train in AI systems do this is
sort of the right frame or a useful
frame so you know especially in
situations where you know you need to do
a bunch of you know in you know direct
sort of you know back and forth with the
world like if you're doing experiments
or if you're doing something
um you know maybe you're just like
running a factory right you know it can
be really difficult you know potentially
to sort of get a model to do something
like this now I think that that is
unclear I think there totally are
techniques where you could take almost
any of these approaches and try to apply
them in something like this you know you
can do you know settings where the you
know it gets to like you know you you
first start by asking you know what are
some ways to think about this what are
some experiments to run then maybe you
in fact go run the experiments you take
the results of the experiments back and
you like give them to the model now
you're like okay here's some information
that you now have like you could imagine
like in the market making setup maybe
the adversary has the ability to say you
know look at this experiment it'll be
really useful and then the human can you
know go you know run you know say run
this experiment you know take the result
back and it gets integrated into the
thing right so you could imagine you
know modifying a lot of these sorts of
things to you know add in that sort of
thing but
um it is quite tricky least um you know
it is totally plausible that for many of
these the sorts of things you might want
your system to do and that's sort of in
those sorts of cases that this you know
won't won't sort of necessarily work
work in that in that situation
okay
um great and so just like adding you
know one other thing you know again in
the same way as debate we can also add
you know oversight we can say that uh
you know
um I'm actually I'm not gonna go into
too much detail on what oversight looks
like here it's basically a sort of
version of the same way that that you
can do oversight uh and
cross-examination and stuff into me
um but suffice to say it's very similar
you know you can have situations where
the models are trying to sort of inspect
each other and provide that produce that
as evidence
okay and then I'm going to go only
relatively briefly through this because
it's very similar to the debate setup
but basically you know again the goal
here is we sort of want the model to be
predicting what the human would think
after seeing all possible arguments
and we're sort of relying on it being
the case that um you know there's no
sort of uh you know that all of the sort
of uh you know untrue persuasive things
that you could be said there is some way
that you know some you know other in
pieces of information the adversary can
produce that would sort of you know
explain to the human why that thing is
untrue and would give you know the human
you know actual true beliefs and so uh
we're sort of again relying on this very
similar version of the you know in the
debate claim uh in debate case we're
really relying on being the case that
you know the sort of most persuasive
things really end up being you know the
most true
um and then again we're sort of relying
on this sort of uh you know oversight to
sort of help us here there's maybe some
reasons that you might expect that
you're less likely to get something like
deception in this case um one thing
that's nice compared to debate in this
case is that the adversary uh as opposed
to the debater he's sort of not trying
to you know accomplish some goal over
time steps so the debater in the debate
game is sort of trying to get you know
it's trained to get you know reward in
the debate game over many individual
debate steps and here that's not the
case the adversary is just trying to do
each individual piece of information
producing uh however I that's not a hard
guarantee at all it totally could be the
case that you still end up with a model
that actually has a long-term objective
um you know and is deceptive in any sort
of way um despite the fact that you're
you know you're only training on an
individual one-step thing yeah question
so using the copernican example from
before let's say the header I'm the pope
and the adversary gives me the
information of if you believe the uh
heliotropic Theory then
you are probably going to be kicked out
of Pokeman they're probably going to
burn you with the stake you should
probably not listen to anything else
that any of us have to say in case you
actually wind up believing that and then
are they able to lie to people
convincingly let's say that is entirely
true
would that model still be aligned in
that case by telling me this fact
uh I mean I think it's very unclear it
sort of depends on what you want it
seems like probably we don't want that
right like that there's something that
the model can say to the human that like
you know causes the human to you know be
100 set in their beliefs and those
beliefs are false and they can never be
convinced otherwise it seems like that's
probably not what you not what you sort
of want to have happen in the situation
um maybe it is you know maybe like
you're saying there's a situation where
like you know actually you know
according to the Pope's values this is
really the thing that the pope should do
but um
I don't know I mean it depends on sort
of what you're going for I think that
probably we would we would want to not
do that
okay uh okay so one last proposal to
cover that uh is uh this imitative
generalization idea so uh this is also
sometimes talked about as learning the
prior
so what is this so um
we're again we have this setup where we
have a human and the human answers
questions
um and similarly to the amplification
case the human now has access to
something to help them answer those
questions in this case that thing is
some sort of model uh Slash information
it's some sort of very large database a
bunch of information that helps the
model maybe a large Cloud that helps the
human answer the question maybe a large
collection of models maybe just an
individual model it's some you know
thing which is extremely useful to the
human in answering the question
um
and then uh we sort of we want to
produce this thing we want to have
something which is really helpful for
humans that effectively lets them answer
your whatever questions we want as
effectively as possible and the way that
we're going to train that thing is well
we want it to be the case that whatever
this thing is this sort of information
that we provide to the human that that
information when given to the human in
fact results of the human making good
answers to questions so we can you know
in fact ask questions you know see
whether those answers are correct so you
know and trained to have the you know
have this information be the case that
when fed to the human in fact results in
correct things and also we want it to be
the case that whatever this sort of set
of information is that is presumably
going to be represented in some model
that that information is plausible a
priori according to the human
um and those are the sort of two things
we're trying to train this thing to do
so you can think of this thing as sort
of a set of information but effectively
we can think of it as just a model and
we're training that model on two things
we want the sort of information
represented in that model the sorts of
things that it says to all sort of be
plausible according to a human and to In
fact when the human has access to that
set of information in fact result in you
know correct answers on you know all the
things that we can check
okay and the reason we might like this
the sort of fear the theoretical sort of
grounding behind this sort of a thing is
that we're sort of approximating
something like a uh prior and an update
on that prior so we're saying well uh we
have some prior plausibility on
information that is like you know How
likely is this operatory and then we
have some likelihood that is like well
updating that prior plausibility on the
uh you know how to what extent that
information in fact does a good job
those hypothesis actually does a good
job at predicting the real things that
we've seen in the world we want to
update the things that do a good job of
predicting the world and down weight the
things that do a bad job and so we're
sort of trying to mimic that updating
procedure you know what if a human had
the ability to actually just update on
all possible information uh you know and
and come to some conclusion well we can
try to mimic that sort of a thing that
sort of human uh you know prior by
saying well what is the most what is the
thing that would be most plausible
according to human and that would result
in the best answers you know the prior
and the likelihood
lots of questions
so I like this this solved several
problems with debate like for instance
the example I just did before with the
burning of the stake thing because in
this case the we want the the human
we're going to be training the model on
not just the human updating the human
saying where it's true
but in that case how do we determine
what is true in the first place where
does the accuracy loss come from
yes I think this is extremely good
question uh and it's very tricky so I
think it has to sort of come from
whatever information you have about the
world right so any individual situation
where you can make some prediction about
something in the world where you can
gather some information uh uh where you
know you can make some concrete
prediction then you can use that as
information to update your hypothesis
right we're trying to get at something
like what would the humans beliefs be if
they had the ability to update on all
the information available in the world
anything they could ever observe and so
we're trying to say is we're like well
you know there are such we can you know
just gather a data set of you know just
making predictions about the world
situations where you can say well here's
something that happened and then
something can happen next
um you know and if you can successfully
make all those predictions you know if
hypothesis expressly explains all those
predictions
um you know then it should have a really
large update in favor of that hypothesis
and so that's sort of the sort of thing
right we're just saying well anything
about the world that we can collect any
data anything that we can predict about
the world all of the information that we
have access to you know those are all
the things that we want to be updating
on
but I still don't get where the accuracy
loss comes from like let's say the
question is is it day outside
it does the model somehow know if they
stay outside or is the truth coming from
what the human comes says is true at the
end of this process
um so so we just come from something
that we've collected so maybe we've in
fact collected a bunch of examples of
situations in the past where it has been
day or hasn't been night based on you
know some information and then in that
situation we would say you know can you
in fact predict all these situations
successfully
um you can even you do this in a sort of
unsupervised way you just gather
arbitrary information about the world
and then train to predict some set of
that information from some other side of
that information because we're just
trying to basically approximate you know
do does the hypothesis make good
predictions about the world and so any
information that we know about the world
we want our hypothesis you know to in
fact be making good predictions about it
but if we reliably have these facts
about the world why do we need this
whole thing why can we just use the
facts well because we want to get new
facts about things in the future right
so situations where we don't necessarily
have the facts yet we want to get a
prediction about it right so you know
for example we might know you know in
fact what happened you know in 2023 but
predicting what happens in 2023 given
what happened in 2022 is extremely
difficult and something that would be
very valuable and so we can try to you
know you know get a thing which is
making those sorts of predictions by you
know finding the hypothesis that best
explains what actually happened in 2023
and is you know most likely according to
the human prior
but so maybe I say they will not be an
H1N1 pandemic in 2023. what do we judge
the accuracy of that statement on based
on the loss yeah so you can't judge the
accuracy of the like new statement that
we have like no previous information you
know to guide it right we're saying with
the thing that we're hoping for is that
this procedure results in a model which
in fact makes good predictions about
things because it finds the set of
information that results in the best
predictions in the past and is the most
plausible right so you know in in all of
these cases you know we have no ability
necessarily to you know the thing we're
trying to do is get a model which is
able to produce good effective answers
on some new data that we haven't seen
before and the way that we're trying to
do that here is we're trying to say well
what is the sort of model that you know
would if we're thinking about it as like
a hypothesis that would be you know that
would best explain the data we've seen
so far and would be most plausible
according to a human that's the
hypothesis that we should be using to
look at Future data
and you know best make predictions about
that featured app
okay
um and so the idea here is that um when
we have this procedure right we have you
know human has access to some model set
of information that you know helps them
answer questions and then we can just
train some model to you know imitate
this this whole procedure right to you
know be able to effectively you know
imitate exactly what the human would do
given access to this this you know most
plausible information you know uh that
has this property that is you know the
the greatest prior in likelihood
um and then this you know model we can
use to you know as our sort of question
answering system
okay so this is a little bit of a weird
approach in some sense in some sense
it's very simple we're just saying you
know we want the the thing to be
plausible and you know we wanted to when
given to a human result in good you know
output and then we want to train a model
to approximate the sole procedure
um but it's also a little bit weird um
you know the reason we might hope that
it's working is that it's doing you know
something like approximating you know
Beijing inference
um but of course it's very unclear
whether it's actually doing that right
because in fact the thing that we've
done is just said well we want some
model right uh you know Z is just some
model you know some algorithm which in
fact results in good performance uh you
know on this data set of you know
predictions and also you know seems
plausible according to human and then
we're like okay and then uh you know
that thing when fed to a human you know
we just want to approximate it
um
we have no necessary guarantee that it's
actually going to be you know the
hypothesis that would be the sort of
thing that would you know be this be
what the human would
um
you know actually think if they and you
know considered all the positive
information and selected the best
possible hypothesis but maybe it is
something like an approximation of that
okay and again we can also sort of have
some oversight here but I'm not going to
go into too much detail on what it would
look like but it's very similar to what
we've talked about previously in stuff
like imitative amplification
um
okay and so uh the goal here right is
we're trying to produce a model uh that
is sort of mimicking what the human
would you know what hypotheses the human
would have after they've been able to
update on all possible information that
they could see about the world
um
I'm I'm not going to talk too much about
the properties of this I think it's a
little bit weird and tricky but um you
know very briefly
um
there are you know some sort of weird
outer alignment issues here
um it can be very hard to incentivize Z
to really contain all the correct
information
um especially because there can be
inconsistencies across individual
questions uh or like there can be double
updating across individual questions
there's a bunch of very sort of tricky
issues about getting into the right
thing
um even in the case where you really
believe that it is actually the thing
which has the property that it is the
you know most plausible according to the
human and results in the best outputs
um because of the way that you're doing
this procedure where each sort of output
is evaluated independently things can
still get a little bit tricky about
whether it is actually equivalent to the
sort of correct Bayesian update and then
there's this sort of inner alignment
issues here as well where you know we we
don't necessarily even you know there's
no reason to believe that you know Z
would actually approximate anything like
the real hypothesis that we would want
here in some sense the sort of only
difference between something like this
and just sort of directly training a
model to produce good answers and sort
of seem good to a human which would be
like the rhf case is we're sort of
adding a human we're saying well you
have to produce good answers such that
when a human has access to you it
produces good answers and also seems
plausible to a human
um but it's unclear how much that change
actually results in you know helping us
helping us find like a better Basin
um it's absolutely still possible you
know that we could get a deceptive model
in this case
um and so it is a little bit unclear
um
I don't talk too much about the
competitiveness here either it's very
similar
um to um
a lot of the approaches we've talked
about previously
um The Hope sort of would be that you
know if you can get something like this
approximation of a you know an actual
sort of update of the human then you can
sort of approximate something like you
know the best possible judgment uh you
know of the human but you're still in
some sense limited by what that best
possible Judgment of the human would
look like you know in some ways very
similar to each stage where you're sort
of limited by what the you know best
possible thing would be that humans
would be able to do given you know
ability to consult with all these other
humans
okay
um so those are all of the ones I want
to talk about right now there's some
other approaches that I'm not going to
talk about but that are also maybe
relevant
um recursive reward modeling is is one
approach I think that the way that we
have talked about approval-based
amplification in this talk though is is
very very similar and essentially
encompasses recursive award modeling so
we've effectively dealt with that
approach there are others that we sort
of haven't talked about um stem AI is
one
um where the idea would be to sort of
understand you know to just sort of use
your models on uh you know individual
narrow like mathematical or scientific
tasks and not try to do any human you
know prediction or question answering in
general at all
um there's other sort of approaches like
narrow award modeling we really just
want to focus on using models for
individual narrow tasks and I'm not
going to go through all of the you know
other possible approaches but
um hopefully the idea at least right now
is a sort of give an overview at least
of the specific you know some of the
leading sort of approaches and how
people are thinking about trying to move
into the regime of you know evaluating
models in super human settings right
where a lot of the approach we've talked
about previously you know up you know
prior to today have been you know things
that have been really focused on you
know more like current models and trying
to bridge the gap from current models to
uh you know these sorts of you know
things are starting to get to AGI but we
also sort of have to also deal with
things that are Bridging the Gap from
AGI and Beyond and so a lot of the
approach we talked about today are
starting to maybe address that thing you
know giving us this ability to scale our
ability to oversee our models and
provide good feedback you know beyond
the point where we can literally just
the things that humans can evaluate
directly
but they're very tricky right all of
these approaches have you know a bunch
of really tricky you know issues with
them and things that really have to be
you have to be able to get right to make
them work
um
and so it's very unclear
um
I think one final thing that I will
leave with before we do questions and I
don't necessarily want uh you don't have
to give your answer to this right now or
even ever but I think that a good
exercise you know maybe a take-home
exercise in terms of sort of thinking
about a lot of the things that I we've
sort of talked about across all of this
is just sort of you know at some point
in time I think that we like as a
society are going to have to make
decisions right about what sorts of
things we actually you know want to go
through with what are the sorts of
proposals we actually want to do and
these are really really hard and
difficult decisions
um and I think in many cases a lot of
these decisions will have to be have to
be made under a lot of uncertainty right
now we have a ton of uncertainty right
we've gone through all these approaches
and our conclusion has been for
basically all of them we don't know you
know here are some things that might
work here are some things that might not
work it's very unclear whether they work
um but in many cases it's not clear
whether that uncertainty will be
resolved and so in a lot of cases we do
have to sort of end up making decisions
that are the best that we possibly can
under uncertainty and so how do we
actually do that you know what decisions
would we actually make that would be the
best possible decisions under
uncertainty is the thing that we're
going to really have to Grapple with and
so I think starting to Grapple with that
question yourself and thinking you know
what would we do given the uncertainty
that we currently have is a really
useful thing to sort of start
um dealing with and sort of
understanding well what are the
proposals you know that we would start
you know what are the things and there's
multiple criteria here it's not
necessarily just what is the best
approach right that is most likely to
succeed it's also what is the thing that
if it fails would be the least
catastrophic
okay and that uh with that I'll we'll
sort of end here and open it up for for
final uh questions
[Applause]
anything else
well what was your uh recommended
proposal be token AI approach we can
Define it oh that's that's a good one
um
very tricky I think
personally currently I think that
um
you know we are we are in regime right
now where I think it makes more sense to
do things like the predictive model
style approach where you know rather
than trying to really aggressively you
know scale these models and train on
like approval signals that we might not
trust you know we can try to do you know
prediction cases where we can trust them
but like I said previously I think that
will stop working uh you know and so
it's not a scalable approach but I do
think that like if I were you know to
say well what would we do right now I
think that's the sort of thing that you
you'd start you want to start with
um but uh that's sort of a cop-out
because it's not sort of addressing this
answer of well you know if we really
want to just sort of as we sort of you
know start scaling more what are the
things that we really need to do to be
able to you know align these models and
get them do the right thing even into
the you know highly superhuman regime
and then it starts to break down even
more you know and I don't know if I have
a I don't I don't have a really good
answer I think that there's some things
that we can sort of analyze as like you
know convergently useful so in a lot of
these approaches we talked about today
you know stuff like having good
oversight tools having good transparency
is extremely important so we can at
least prioritize you know particular
research directions that are likely to
help with those sorts of things
um
and you know I do have like you know
some preferences you know some of the
sorts of proposals here that I like
better some that are like worse
um
I I tend to be you know in favor of
things like imitative amplification
um Market making is one that I I came up
with and so I you know have some amount
of attachment to it so I think it has a
lot of issues
um similar to debate
um but I I don't I certainly don't have
an answer
I also think there's a lot to be said
for microscope AI if it's possible but
um we we'd have to actually succeed on
being able to do a lot of very
successful transparency you know to be
able to do it and I think that that is
at least currently you know not
something we're really succeeding on
though you know like I mentioned you
know it seems something that seems like
for a lot of these approaches extremely
conversionally useful and so if we you
know we're able to succeed on that more
effectively then it would unlock a lot
of possibilities
Yeah question yeah
well so a lot of these approaches rely
on or they started where we fail to
trust our
um feedback signals
um and for something like reward or like
something like rlhf the most common
thing is to provide binary feedback
which is a read like inefficient use of
humans to provide feedback I mean if I
were to give feedback on this talk I
wouldn't say thumbs up thumbs down that
would probably just like get into the
weeds of like what I liked or what it is
like what I disagree with where I was
confused and that can be done by means
of needling language or some other adult
form of communications
um
how
might we
um break down I guess my question is has
somebody looked into how we can provide
better feedback and is that an Avenue
that is fruitful in your opinion
yeah so yeah good question
um in terms of providing non-binary
feedback this is absolutely a thing that
you can and has been done with current
models so Ethan Perez has a has a paper
on this looking into how you can provide
natural language feedback
um that um they can be quite effective
in a similar you know way too binary
feedback in our life so I don't think
it's the case that like we only do
binary feedback currently
um there are absolutely ways that you
can do you know more um you know
detailed feedback than that so it's
unclear whether you know I think in some
ways you should sort of think about that
as it's not clearly making the feedback
better it's just making it more
efficient right you could have gotten
all of that information by doing binary
feedback but the binary feedback is very
inefficient because you have to have a
lot of examples of slight tweaks you
know to get all of the information of
the minor feedback you know out of the
binary feedback and you can just sort of
get a lot more information out of the
um the language you act but it's not
clear that that's actually making the
feedback better right like in situations
where the human is in fact just confused
and like is saying the wrong you know it
has incorrect beliefs about whether the
thing is good or not then binary or
language feedback or you know getting
more detailed feedback from the human
doesn't help because that feedback is
you know incorrect and so it doesn't
necessarily make the feedback better
though it does make it more efficient
which can help you you know maybe get
more feedback but again you know getting
more feedback is only so helpful as long
as that feedback is good right and so
the sort of key problem you know is not
necessarily just the the you know
quantity of feedback but the quality
feedback you know the ability to
actually believe that that feedback
isn't is in fact correct the human is
actually understanding what's happening
to provide you know good feedback there
but is I mean in that particular
approach I worked on this during the
summer as well would if from the end of
the generally come yeah
in that particular approach
the for the what model what what we did
was basically like take the language uh
or like we had a model who prompted it
to the greater summary and then get some
feedback same model and ask it to
rewrite the summary
reward multi to Vault
um and so in that sense it hasn't yet
been used for RL HF and to like have
more
um of the sort of
feedback that is richer
um I agree with you that to some degree
it's just making the process more
efficient as in like instead of giving a
ton of thumbs up and down they can just
like provide one with another sentence
that has more along the information
content I disagree however with the
point where
um you said that
we can't clearly communicate the
confusion like I guess in natural
language I can say well I'm not quite
sure how to give you feedback on this
because I'm confused whereas with thumbs
up from sound you can't do that even
within the linen where you can just like
uh you know
the internet feedback on the my new
details of like how after the behavior
of the agent is to be read
I think so first I I generally will
think of
um a lot of those sorts of processes
you're you're training on feedback of
some variety and then getting the model
to sort of score well on that feedback
as relatively continuous and so I don't
do that much differentiation usually
between like was there a preference
model or was there not a preference
model I think you can differentiate
between those things and so sometimes it
can make sense and the details can
matter though I think that oftentimes
they're not they're not that important
in terms of just like the overall
alignment properties but but sometimes
they can matter but that's why I refer
to it as as as RL HF
um in terms of the sort of concrete
question of you know communicating
confusion I totally agree and I think
there can be cases where the human is in
fact confused and you're able to sort of
you know address that problem being able
to communicate the confusion I think
that the issue remains however that
there are situations where you know for
example the human doesn't know they're
confused right that the human thinks
they are giving correct feedback they
think they understand what's happening
but in fact the human is incorrect human
doesn't understand what's happening
um
and in that situation you know there's
you know we sort of need something else
other than the human to you know help
the human or somehow produce you know
give the human more information to give
a more informed response because if we
are only limited by the human's ability
to understand and evaluate then we are
you know fundamentally bottlenecked by
things that humans can effectively
evaluate and they're going to be
situations where
even when you know also where humans
can't always know whether they're
evaluating effectively where you know if
we're you know yes we can try to limit
it into cases where you know the humans
believe they're evaluating things
effectively and you know have some you
know positive valuation but they're
going to be cases where you know that
that is also not sufficient where the
humans believe they're evaluating
effectively but in fact you know have
some you know limitation you know they
don't actually understand what's
happening effectively
and so we still sort of have to in some
sense go beyond so I think that like you
know a lot of these approaches are still
sort of trying to address that problem
right of you know how do we go beyond
that the feedback that a human is is is
is ever able to provide you know
situations where the human is just
confused you know or you know doesn't
know that they're confused you know the
human has some incorrect belief but in
fact is just
um you know
thinks they're things that they
understand what's happening and that can
happen especially when you're training a
model to say things that look good to a
human right if you're training the model
to produce you know rocket ship designs
that look really good to the human then
you're going to be in a situation where
many of those situations you know it's
going to look good and the human's going
to think it's great but in fact you know
it's it's not gonna actually work in
practice because it was only optimized
to look good according to the human and
so in some sense you know you need some
better evaluation signal to sort of be
able to address that
I'm not sure whether this is Japanese or
something else but do you have any
research advice or people start envelope
into these theoretical questions
yeah that's a good question
um
I'm not going to try to like right now
recommend any like particular you know
places or things to do because you know
the field is constantly changing I think
that
um in terms of just like you know in
general you know
just you know listening to and
understanding all these sorts of things
I think is extremely important just like
having the basic concepts understanding
sorts of things that people are talking
about in the field and the way in which
the sort of the basic structure of the
problem is I think basically always
valuable and essentially any you know
position that one might be in
um
I think that you know one of the sort of
issues you know ways that the sort of
field currently is operating is that
um we just we don't know what to do
right we're in a situation where we have
a lot of ideas we have things that might
work we have some reasons why they might
work or might not work but we don't know
what to do right there isn't some you
know this is the thing that you know we
need to accomplish this is the you know
everyone's on board with you know this
is the approach right and so in that
situation I think it's very important to
you know have a good General
understanding of like okay here are the
sorts of things that are being discussed
here are sort of how to understand and
sort of the basic concepts because it's
very unclear you know what the actual
correct thing is to be doing and so
having an open mind and being able to
try to figure out what the correct thing
is to be doing is is really important
another sort of piece of advice that I
will often give that I think is
important is
um
try to sort of you know I think it's
really valuable to uh for people to
really you know be individually like you
know specialized and usual things I
think that you know there's a lot of
things to be doing and uh you know it's
really important to understand like I
was just saying all of these various
different sort of you know Concepts and
stuff but I think that you know then we
also have to do something right and so
you know making some you know bat trying
to figure out some place where you can
be helpful and really and concretely
accomplish something that you know you
think is useful and then really doing
that thing I think is you know you know
what I what I think is really the most
valuable right and so you know I try to
you know in like the the mentorship
program uh you know try to get people to
um you know
understand the basic concepts and really
understand how to think about AI safety
and what sorts of interventions might be
effective and then you know find an
intervention you know that they can do
something that you know might be helpful
and really execute effectively on that
so I think that's sort of that's very
broad but that's sort of generally how I
think about you know you know trying to
address this having having good concrete
models about how things are going to go
and what sort of you know ways in which
things might go poorly and things that
you can do to make things go better and
then you know Finding individual
particular interventions and trying to
execute as best you can
okay uh we will call it there so that
was the last talk so this is uh you know
the end for this but hopefully I have
you know given you know a lot of good
you know tools and understanding and
Concepts to sort of understand you know
and help you know think about this sort
of General field uh of aicd
foreign