Evan Hubinger on Inner Alignment, Outer Alignment, and Proposals for Building Safe Advanced AI

[Music]
[Music]
welcome to the ai
alignment podcast i'm lucas perry
today we have a conversation with evan
hubenger about
ideas and two works of his an overview
of 11 proposals for building safe
advanced ai
and risks from learned optimization in
advanced machine learning systems
some of the ideas covered in this
podcast include
inner alignment outer alignment training
competitiveness
performance competitiveness and how we
can evaluate
some highlighted proposals for safe
advanced ai with these criteria
we especially focus in on the problem of
inner alignment and go into quite a bit
of detail on that
this podcast is a bit jargony but if you
don't have a background in computer
science don't worry
i don't have a background in it either
and evan did an excellent job of making
this episode accessible
whether you're an ai alignment
researcher or not i think you'll find
this episode quite informative and
digestible
i learned a lot about a whole other
dimension of
alignment that i previously wasn't aware
of and
feel this help to give me a deeper and
more holistic understanding of the
problem evan hubenger was
an ai safety research intern at openai
before
joining mary his current work is aimed
at solving
inner alignment for iterated
amplification
evan was an author on risks from learned
optimization
and advanced machine learning systems
was previously a miri intern
designed the functional programming
language coconut
and has done software engineering work
at google
yelp and ripple evan studied math and
computer science at harvey mudd college
and with that let's get into our
conversation with evan hubenger
in general i'm curious to know a little
bit about your
intellectual journey and the evolution
of your passions and how that's brought
you to
ai alignment so what got you interested
in computer science and
tell me a little bit about your journey
to miri i
started computer science when i was
pretty young i started programming
in middle school playing around with
python programming a bunch of stuff in
my spare time
the first sort of really big thing that
i did i wrote a functional programming
language on top of
python it was called rabbit was really
bad
it's like interpreted in python and then
i decided i would sort of improve on
that i wrote another functional
programming language on top of python
called coconut got a bunch of traction
that's just sort of while i was in high
school starting to get into college and
this is also around the time i was
reading a bunch of the sequences on
restaurant
i got sort of into that in the
rationality space and i was following it
a bunch
i also did a bunch of internships at
various tech companies doing software
engineering
and especially programming languages
stuff around halfway through
my undergrad i started running the
effect of altruism club
at harvey mudd college and as part of
running the effective altruism club
i was trying to learn about all these
different cause areas and how to use my
career to do the most good
and i went to va global and i met some
merry people there they invited me to do
a programming internship
at miri where i did some engineering
stuff functional programming dependent
type theory stuff
and then while i was there i went to the
miri summer fellows program
which is this place where a bunch of
people can come together and
try to work on doing research and stuff
for like a period of time over the
summer
i think it's not happening now because
of the pandemic but it hopefully will
happen again soon and while i was there
i encountered
some various different information and
people talking about safety stuff and in
particular i was really interested in
this
at that time people are calling it
optimization demons the sort of idea
that there could be
problems when you train a model for some
objective function you don't actually
get
a model that's really trying to do what
you trained it for
and so with some other people who are at
the miri summit fellows program we tried
to dig into this problem and we wrote
this paper
risk mode optimization in advanced
machine learning systems
some of the stuff that i'll probably be
talking about in this podcast came from
that paper
and then as a result of that paper i
also got a chance to
work with and talk with paul cristiano
at opening eye
and he invited me to apply for an
internship at open ai
so after i finished my undergrad i went
to open ai and i did some theoretical
research with paul there
and then when that was finished i went
to miri where i currently
am and i'm doing sort of similar
theoretical research to the research i
was doing at opening eye but now i'm
doing
it at miri so that gives us a better
sense of how you ended up in ai
alignment
now you've been studying it for quite a
while from
a technical perspective could you
explain what your take is on ai
alignment
and just explain what you see as ai
alignment
sure so i guess broadly i like to take
a general approach to ai alignment i
sort of see the problem they're trying
to solve as the problem of ai
existential risk it's the problem of it
could be the case that in the future we
have very advanced ais
that are not aligned with humanity and
do really bad things
i see ai alignment as the problem of
trying to prevent that but there are
obviously a lot of sub components to
that problem
and so i like to make some particular
divisions
specifically one of the divisions that
i'm very fond of is to
split it between these concepts called
interlinement and outer alignment
which i'll talk more about later i also
think that there's a lot of different
ways to think about what the problems
are that these sorts of approaches
trying to solve inner line and outer
alignment what is the thing that we're
trying to approach
in terms of building an aligned ai and
they also tend to fall into
the polka shadow camp of thinking mostly
about intent alignment
where the goal of trying to build ai
systems right now as the thing that we
should be doing to prevent ais from
being catastrophic is focusing on how do
we produce ai systems which are
trying to do what we want and i think
that inner and outer alignment are the
two big components of producing
intent aligned ai systems the goal is to
hopefully reduce ai existential risk and
make the future a better place
do the social and governance
and ethical and moral philosophy
considerations
come much into this picture for you when
you're thinking about it
that's a good question there's certainly
a lot of philosophical components to
trying to understand various different
aspects of ai
what is intelligence how objective
functions work what is it that we
actually want our ais to do at the end
of the day
in my opinion i think that a lot of
those problems are
not at the top of my list in terms of
what i expect to be quite dangerous
if we don't solve them i think a large
part of the reason for that is because
i'm
optimistic about some of the ai safety
proposals such as amplification
and debate which aim to produce a sort
of agent
in the case of application which is
trying to do what a
huge tree of humans would do and then
the problem reduces to
rather than having to figure out in the
abstract what is the objective that we
should be trying to train an ai for
that philosophically we think would be
utility maximizing or good or whatever
we can just be like well we trust that a
huge tree of humans would do the right
thing
and then sort of defer the problem to
this huge tree of humans to figure out
what philosophically is the right thing
to do
and there's sort of similar arguments
you can make with other situations like
debate
where we don't necessarily have to solve
all of these hard philosophical problems
if we can make use of some of these
alignment techniques that can solve some
of these problems for us so let's get
into here
your specific approach to ai alignment
how is it that you approach ai alignment
and how does it differ
from what miri does so i think it's
important to note
i certainly am not here speaking on
behalf of miri i'm just presenting
my view and my view is pretty distinct
from the view of a lot of other people
at mary
so i mentioned at the beginning that i
used to work
at opening eye and i did some work with
paul cristiano and i think that my
perspective
is pretty influenced by that as well and
so i come more from
the perspective of what paul calls
prosaic ai alignment
which is the idea of we don't know
exactly what
is going to happen as we develop ai into
the future but a good
operating assumption is that we should
start by trying to solve
ai for ai alignment if there aren't
major surprises on the road to agi
what if we really just scale things up
we sort of go via the standard path
and we get really intelligent systems
would we be able to align ai in that
situation
and that's the question that i focus on
the most not because i don't expect
there to be surprises
but because i think that it's a good
research strategy
we don't know what those surprises will
be probably
our best guess is it's going to look
something like what we have now
so if we start by focusing on that then
hopefully we'll be able to generate
approaches
which can successfully scale into the
future and so because i have this sort
of general research approach i tend to
focus more on what are current machine
learning systems doing
how do we think about them and how would
we make them
interlined and outer lined if they were
sort of scaled up into the future
this is sort of in contrast with the way
i think a lot of other people at miri
view this
i think a lot of people at miri think
that if you go this route
of prosaic ai current machine learning
scaled up it's very unlikely to be
aligned
and so instead you have to search for
some other understanding some other way
to potentially do artificial
intelligence that isn't just the
standard prosaic
path that would be more easy to align
that would be safer
i think that's a reasonable resource
strategy as well but it's not the
strategy that i generally pursue in my
research
could you paint a little bit more
detailed of a picture
of say the world in which the prosaic ai
alignment strategy
sees as potentially manifesting where
current machine learning algorithms and
the current paradigm of thinking and
machine learning is merely
scaled up and via that scaling up we
reach
agi or super intelligence i mean there's
a lot of different ways to think about
what does it mean for current ai current
machine learning to be scaled up because
there's a lot of different forms of
current machine learning
you could imagine even bigger gbt3 which
is able to do
highly intelligent reasoning you could
imagine we just do
significantly more reinforcement
learning in complex environments
and we end up with highly intelligent
agents
i think there's a lot of different paths
that you can go down
that still fall into the category of
prosaic ai and a lot of the things that
i
do as part of my research is trying to
understand those different
paths and compare them and try to get to
an understanding of
even within the realm of mosaic ai
there's so much happening
right now in ai and there's so many
different ways we could use current ai
techniques to put them together in
different ways
to produce something potentially super
intelligent or highly capable and
advanced
which of those are most likely to be
aligned which of those are the best
paths to go down
one of the pieces of research that i
published recently was an overview and
comparison
of a bunch of the different possible
paths to
prosaic agi different possible ways in
which you can build
advanced ai systems using current
machine learning tools and
trying to understand which of those will
be more or less aligned and which would
be more or less competitive
so you're referring now here to this
article which is partly a motivation for
this conversation which is an overview
of 11 proposals for building safe
advanced ai that's right all right so i
think it'd be valuable if you could also
help to
paint a bit of a picture here of exactly
the miri style approach to ai alignment
you said that they think
that if we work on ai alignment via this
prosaic paradigm
that machine learning scaled up to
superintelligence or beyond is unlikely
to
be aligned so we probably need something
else could you unpack this a bit more
sure i think that the biggest concern
that a lot of people at mary have with
trying to scale up for
ai is also the same concern that i have
there's this really difficult pernicious
problem which i call inner alignment
which is presented in the risk and learn
optimization paper that i was talking
about previously
which i think many people in mary as
well as me think that this inner
alignment problem is the key
stumbling block to really making prosaic
ai work
i agree that this is the biggest problem
but i'm more optimistic in terms of i
think that there are
possible approaches that we can take
within the prosaic paradigm
that could solve this inner alignment
problem and i think
that is the biggest point of difference
is how difficult
will inner alignment be so what that
looks like
is a lot more foundational work and
correct me if i'm wrong here
into mathematics and principles in
computer science like
optimization and what it means for
something to be an optimizer and what
kind of properties that has is that
right
yeah so in terms of some of the stuff
that other people in miri work on
i think a good starting point would be
the embedded agency sequence on the
alignment forum
which gives a good overview of a lot of
the things that the different asian
foundations people
like scott garabran sam eisenstad abram
demski
are working on all right now you've
brought up
inner alignment as a crucial difference
here in
the pinion so could you unpack exactly
what inner alignment
is and how it differs from outer
alignment this is a favorite topic of
mine
a good starting point is trying to sort
of
rewind for a second and really
understand what it is that machine
learning does
fundamentally when we do machine
learning there are a couple of
components we start with a parameter
space
of possible models where a model in this
case is
some parameterization of a neural
network or some other type of
parameterized function we have this
large space of possible models this
large space of possible parameters that
we can put into our neural network
and then we have some loss function
where for a given
parameterization for a particular model
we can check what is its behavior like
on some environment in supervised
learning we can ask
how good are its predictions that it
outputs you know in an rl environment we
can ask how much
reward does it get when we sample some
trajectory
and then we have this gradient descent
process which
samples some individual instances of
behavior
of the model and then it tries to modify
the model to
do better in those instances we search
around this parameter space
trying to find models which have the
best behavior on the training
environment
this has a lot of great properties i
mean this has managed to you know propel
machine learning into being able to
solve all of these very very difficult
problems that we don't know how to
write algorithms for ourselves but i
think because of this there's a sort of
tendency
to rely on something which i call that
does the right thing abstraction
which is that well because the model's
parameters were selected to
produce the best behavior according to
the loss function on the training
distribution
we tend to think of the model as really
trying to minimize that loss really
trying to get reward but in fact in
general that's not the case
the only thing that you know is that on
the cases where i sample data on the
trainee distribution
my models seem to be doing pretty well
but you don't know what the model is
actually trying to do
you don't know that it's truly trying to
optimize the loss or some other thing
you just know that well it looked like
it was doing a good job on the train
distribution
and what that means is this abstraction
is quite leaky there's many different
situations in which this can go wrong
and this general problem
is referred to as robustness or
distributional shift
this problem of well what happens when
you have a model
which you wanted it to be trying to
minimize some loss
but you move it to some other
distribution you take it off the
training data
what does it do then and i think this is
the starting point for understanding
what is in alignment is from this
perspective of robustness and
distributional shift
inner alignment specifically is a
particular type of robustness problem
and it's the particular type of
robustness problem that occurs when
you have a model which is itself an
optimizer
when you do machine learning you're
searching over this huge space of
different possible models
different possible parameterizations of
a neural network or some other function
and one type of function which could do
well on many different environments
is a function which is running a search
process which is doing some sort of
optimization
you can imagine i'm training a model to
solve
some maze environment you could imagine
a model which just
learns some heuristics from when it
should go left and right
or you could imagine a model which sort
of looks at the whole maze and does some
planning algorithm
some search algorithm which searches
through the possible paths and finds the
best one
and this might do very well on the mazes
if you're just running a training
process
you might expect that you'll get a model
of the second form that is running this
search process that is running some
optimization process
in the wrist and optimization paper we
call models which are themselves running
search processes
mesa optimizers where mesa is just greek
and it's
the opposite of meta there's a sort of
standard terminology in machine learning
this meta optimization
where you can have an optimizer which is
optimizing another optimizer
and mesa optimization is the opposite
it's when you're doing gradient descent
you have an optimizer
and you're searching over models and it
just so happens that the model that
you're searching over happens to also be
an optimizer
it's one level below rather than one
level above so because it's one level
below we call it a mesa optimizer
and inner alignment is the question of
how do we align
the objectives of mesa optimizers
if you have a situation where you train
a model
and that model is itself running an
optimization process
and that optimization process is going
to have some objective it's going to
have something that it's searching for
in a maze maybe it's searching for how
do i get to the end of the maze
and the question is how do you ensure
that that objective
is doing what you want if we go back to
the does the right thing abstraction
that i mentioned previously
it's tempting to say well we train this
model to get to the end of the maze
so it should be trying to get to the end
of the maze but in fact that's not in
general the case
it could be doing anything that would be
correlated with good performance
anything that would
likely result in in general it gets to
the end of the maze on the training
distribution
but it could be an objective that will
do anything else sort of off
distribution
that fundamental robustness problem of
when you train a model and that model
has an objective
how do you ensure that that objective is
the one that you trained it for
that's the inner alignment problem and
how does that stand in relation with the
outer alignment problem
so the outer alignment problem is how do
you actually
produce objectives which are good to
optimize for
so the inner alignment problem is about
aligning
the model with the loss function the
thing you're training for the reward
function
outer alignment is allow aligning that
reward function that loss function
with the programmer's intentions it's
about ensuring that when you write down
a loss
if your model were to actually optimize
for that loss it would actually do
something good
outer alignment is the much more
standard problem of ai alignment
if you've been introduced to the
airlines before you'll usually start by
hearing about the outer alignment
concerns
things like paperclip maximizers where
there's this problem
of you try to train it to do some
objective which is maximize paper clips
but in fact maximizing paper clips
results in it doing all this other stuff
you don't want it to do
and so outer alignment is this value
alignment problem of how do you
find objectives which are actually good
to optimize
but then even if you have found an
objective which is actually good to
optimize
if you're using the standard paradigm of
machine learning you also have this
interlining problem
which is okay now how do i actually
train a model which is in fact
going to do that thing which i think is
good that doesn't bear relation with
stewart's standard model does it it sort
of is related to stewart russell's
standard model of ai i'm not referring
to precisely the same thing but it's
very similar i mean i think a lot of the
problems that stuart russell has
with the standard paradigm of ai are
based on this start with an objective
and then train them all to optimize that
objective when i've talked to stuart
about this in the past he's said why are
we even doing this thing of training
models
hoping that the models would do the
right thing we should be just doing
something else entirely
but we're both pointing at different
features of the way in which current
machine learning is done and trying to
understand what are the problems
inherent in this sort of machine
learning process
i'm not making the case that i think
that this is an unsolvable problem i
mean it's the problem i work on
and i do think that there are promising
solutions to it but i do think it's a
very hard problem
all right i think you did a really
excellent job there painting the picture
of inner alignment and outer alignment
i think the in this podcast historically
we have
focused a lot on the outer alignment
problem
without making that super explicit
now from my own understanding and as a
warning to listeners
my basic machine learning knowledge is
something like a orc structure
hobbled together with sheet metal and
string and glue
and gum and rusty nails and stuff so i'm
gonna try my best here to
see if i understand everything here
about inner and outer alignment and the
basic machine learning model and you can
correct me if i get any of this wrong
so in terms of inner alignment there is
this neural network space which can be
parameterized
and when you do the parameterization of
that model
the model is the nodes and how they're
connected right
yeah so the model in this case is just a
particular parameterization of your
neural network
or whatever function approximately that
you're training and it's whatever
the parametrization is at the moment
we're talking about so when you deploy
the model
you're deploying the parameterization
you found
by doing huge amounts of training via
grady descent or whatever searching over
all possible parameterizations to find
one that had good performance on the
training environment
so that model being parameterized
that's receiving inputs from the
environment and then
it is trying to minimize the loss
function or maximize reward
well so that's the tricky part right
it's not trying to minimize
the loss it's not trying to maximize
reward that's the thing which i call the
does the right thing abstraction
this sort of leaky abstraction that
people often rely on when they think
about machine learning but isn't
actually correct
yeah so it's supposed to be doing those
things but it might not
well what is supposed to mean it's just
a process it's just a
system that we run and we hope that it
results in some particular outcome
what it is doing mechanically is we are
using a gradient descent process to
search over
the different possible parameterizations
to find parameterizations
which result in good behavior on the
training environment
it's good behavior as measured by the
loss function of the reward function
right
that's right you're using gradient
descent to search over the
parameterizations
to find a parameterization which results
in a high reward
on the training environment right but
achieving
the high reward what you're saying is
not identical with actually
trying to minimize the loss right
there's a sense that you could sort of
think of gradient descent as trying to
minimize the loss because it's
selecting for parameterizations which
have the lowest possible loss that it
can find
but we don't know what the model's doing
all we know is that the model's
parameters were selected by gradient
descent
to have good training performance to do
well according to loss on their training
distribution
but what they do off distribution we
don't know we're going to talk about
this later but there could be a proxy
there could be something else in the
maze that it's actually optimizing for
that correlates with
minimizing the loss function but it's
not actually trying to get to the end of
the maze
that's exactly right and then in terms
of gradient descent
is the tl dr on that the the
parameterized
neural network space you're creating all
these perturbations to it
and the perturbations are sort of
nudging it around
in this n-dimensional space how many
other parameters there are or whatever
and then you check to see how it
minimizes the loss
after those perturbations have been done
to the model and then
that will tell you whether or not you're
moving in a direction which is the local
minima or not in that space
is that right yeah i think that that's a
good intuitive understanding
what's happening is you're looking at
infinitesimal shifts
because you're taking a gradient and
you're looking at how those
infinitesimal shifts
would perform on some batch of training
data and then you repeat that many times
to go in the direction of the
infinitesimal shift which would cause
the best increase in performance
but it's basically the same thing i mean
i think the right way to think about
gradient descent is this local search
process that's moving around the
parameter space
trying to find parameterizations which
have good training performance
is there anything interesting that you
have to say about that process of
gradient descent
and the tension between finding local
minima and global minima
yeah i mean it's certainly an important
aspect of what the greedy descent
process does that it doesn't
find global minima it's not the case
that it works by
looking at every possible
parameterization and picking the actual
best one
it's this local search process that
starts from some initialization
and then looks around the space trying
to move in the direction
of increasing improvement because of
this there are potentially multiple
possible equilibria
parameterizations that you could find
from different initializations that
could have different performance
all the possible parameterizations of a
neural network with billions of
parameters
like gpt2 are now gt3 which has greater
than 100 billion
is absolutely massive it's a sort of
combinatorial explosion of a huge degree
where you have all of these different
possible parameterizations
running internally so correspond to
totally different algorithms controlling
these weights that determine exactly
what algorithm the model ends up
implementing and so
in this massive space of algorithms you
might imagine that some of them will
look more like search processes
some of them will look more like
optimizers that have objectives some of
them will look less like optimizers some
of them might just be sort of grab bags
of heuristics
or other different possible algorithms
you depend on exactly what your setup is
if you're training a very simple network
that's just like a couple of feed
forward layers
it's probably not possible for you to
find really complex models
influencing complex search processes but
if you're training huge models with many
layers
with all of these different possible
parameterizations then it becomes more
and more possible if you define these
complex algorithms that are running
complex search processes
i guess the only thing that's coming to
mind here that is maybe somewhat similar
is how 4.5 billion years of evolution
has
searched over the space of possible
minds and here we stand as these ape
creature things
are there for example interesting
intuitive relationships between
evolution and gradient descent there are
both processes searching over a space of
mind it seems
that's absolutely right i think that
there are some really interesting
parallels there
and in particular if you think about
humans as models that were produced by
evolution as a search process
it's interesting to note that the thing
which we optimize for is not the thing
which evolution optimizes for
evolution wants us to maximize the total
spread of our dna
but that's not what humans do we want
all these other things like
decreasing pain and happiness and
food and mating and all these various
proxies that we use
an interesting thing to note is that
many of these proxies are actually a lot
easier to optimize for and a lot simpler
than if we were actually truly
maximizing spread of dna
an example that i like to use is imagine
some alternate world
where evolution actually produced humans
that really cared about
their dna and you have a baby in this
world and this baby stubs their toe
and they're like what do i do do i have
to cry for help is this a bad thing
that i've stubbed my toe and they have
to do this really complex optimization
process that's like okay
how is my toe being stubbed going to
impact the probability of me being able
to
have offspring later on in life what can
i do
to best mitigate that potential downside
now and this is a really difficult
optimization process
and so i think it sort of makes sense
that evolution instead opted for just
pain bad if there's pain you should try
to avoid it
but as a result of evolution opting for
that much simpler proxy
there's a misalignment there because now
we care about this pain
rather than the thing that evolution
wanted which is the spread of dna
i think the way stuart russell puts this
is the actual
problem of rationality is how is my
brain
supposed to compute and send signals to
my 100
odd muscles to maximize my
reward function over the universe
history
until heat death or something we do
nothing like that
it'll be computationally intractable it
would be insane
so we have all of these proxy things
that evolution has found
that we care a lot about their function
is
instrumental in terms of optimizing for
the thing that
evolution is optimizing for which is
reproductive fitness
and then this is all probably motivated
by thermodynamics i believe
when we think about things like love or
like beauty or joy or like aesthetic
pleasure in music or parts of philosophy
or things
these things almost seem intuitively
valuable from the first person
perspective
of the human experience but via
evolution
there these proxy objectives that we
find valuable because they're
instrumentally useful
in this evolutionary process on top of
this thermodynamic process
and that makes me feel a little funny
yeah i think that's right but i also
think it's worth noting
that you want to be careful not to take
the evolution analogy too far because it
is just an analogy
when we actually look at the process of
machine learning and how graded descent
works
it's not the same it's running a
fundamentally different optimization
procedure
over a fundamentally different space and
so there are some interesting analogies
that we can make to evolution
but at the end of the day what we really
want to analyze is how does this work in
the context of
machine learning and i think the risk of
an optimization paper tries to do
that second thing of let's really try to
look carefully at the process of machine
learning
and understand what this looks like in
that context and i think it's useful to
sort of have in the back of your mind
this analogy to evolution but i would
also be careful not to take it too far
and imagine that everything is going to
generalize to the case of machine
learning because it is a different
process
so then pivoting here wrapping up on our
understanding of inner alignment and
outer alignment
there's this model which is being
parameterized by
gradient descent and it has some
relationship with the loss function or
the objective function and it might not
actually be trying to minimize the
actual loss or to actually
maximize the reward could you add a
little bit more clarification here about
why that is
i think you mentioned this already but
it seems like when gradient descent
is evolving this parameterized model
space
isn't that process connected to
minimizing the loss
in some objective way the loss is being
minimized
but it's not clear that it's actually
trying to minimize the loss
there's some kind of proxy thing that
it's doing that we don't really care
about
that's right fundamentally what's
happening is that you're selecting for a
model which has empirically on the
training disputing the load loss
but what that actually means in terms of
the internals of the model but it's sort
of trying to optimize for and what it's
out of distribution behavior would be
is unclear so a good example of this is
this maze example so
i was talking previously about the
instance of you know maybe you train a
model on a training distribution of
relatively small mazes
and to mark the end we put a little
green arrow
and then i want to ask the question what
happens when we move to a
deployment environment where the green
arrow is no longer at the end of the
maze
and we have much larger mazes and then
what happens to the model in this
new off distribution setting and i think
there's three distinct things that can
happen
it could simply fail to generalize at
all it just didn't learn a general
enough optimization procedure
that it was able to solve these bigger
larger mazes
or it could successfully generalize and
knows how to navigate it learned a
general purpose optimization procedure
which is able to solve mazes
and it uses it to get to the end of the
maze but there's a third possibility
which is that it learned a general
purpose optimization procedure which is
capable of solving mazes but it learned
the wrong objective it learned to use
that optimization procedure to get to
the green arrow
rather than to get to the end of the
maze and what i call this situation is
capability generalization without
objective generalization
its objective the thing it was using
those capabilities for
didn't generalize successfully off
distribution and what's so dangerous
about this particular robustness failure
is that it means off distribution you
have models which are
highly capable they have these really
powerful optimization procedures
directed at incorrect tasks you have
this strong
maze solving capability but this strong
maze solving capabilities being directed
at
a proxy getting to the green arrow
rather than the actual thing which we
wanted
which was get the end of the maze and
the reason this is happening
is that on the training environment both
of those different possible
models look the same in the train
distribution
when you move them off distribution you
can see that they're trying to do very
different things
one of which we want and one of which we
don't want but they're both still highly
capable
you end up with a situation where you
have intelligent models directed at the
wrong
objective which is precisely the sort of
misalignment of ai's that we're trying
to avoid
but it happened not because the
objective was wrong in this example we
actually want them to get to the end of
the maze
it happened because our training process
failed it happened because our training
process wasn't able to distinguish
between models trying to get to the end
and models trying to get to the green
arrow and what's particularly
concerning in this situation is when the
objective generalization
lags behind the capability
generalization when the capabilities
generalize better than the objective
does
so that it's able to do highly capable
actions highly intelligent actions
but it does them for the wrong reason
and i was talking previously about
mesa optimizers where inner alignment is
about this problem of models which have
objectives which are incorrect and
that's the sort of situation where i
expect this problem to occur because if
you are training a model
and that model has a search process and
an objective
potentially the search process could
generalize without the objective also
successfully generalizing
and that leads to this situation where
your capabilities are generalizing
better than your objective
which gives you this problem scenario
where the model is highly intelligent
but directed at the wrong thing
just like in all of the outer alignment
problems the thing doesn't know what we
want but it's highly capable right
right so while there is a loss function
or
an objective function that thing is
used to perform gradient descent on the
model in a way
that moves it roughly in the right
direction but
what that means it seems is that the
model isn't just something about
capability the model also
implicitly somehow builds into it
the objective is that correct we have to
be careful here because
the unfortunate truth is that we really
just don't have a great understanding of
what our models are doing
and what the inductive biases of
gradient descent are right now
and so fundamentally we don't really
know what the internal structures our
models are like
there's a lot of really exciting
research stuff like the circuits
analysis from crysola and the clarity
team at openai
but fundamentally we don't understand
what the models are doing we can sort of
theorize
about the possibility of a model which
runs in some search process
and that search process generalizes but
the objective doesn't but fundamentally
because
our models are these black mock systems
that we don't really fully understand
it's hard to really concretely say yes
this is what the model's doing this is
how it's operating and this is the
problem
but in risk optimization we try to at
least attempt to understand that problem
and look at if we really think carefully
about what gradients then is
incentivizing and how it might work what
are the things which we might predict
what happened
so the objective that you're training
the model for does not live in the model
it lives
in the grain descent process it lives in
the training procedure
we might hope that when we train a model
on an objective
that it will produce its own model of
that objective and try to figure out
what it is and be aligned with it
but we don't know exactly what happens
the model doesn't get to see
the objective you're training for all
that happens is that the grade descent
process looks at its behavior
and tries to make it so that its
behavior is more aligned with the loss
function but that loss function never
enters into the model somehow the model
never sees that loss function
it might have some objective internally
like i was saying if it's a mesa
optimizer
and then we might hope that objective is
aligned with the loss function we're
training it for
but fundamentally all we know is that
its behavior on the training
distribution
was aligned with the loss function that
makes sense
and because it's so black boxy we can't
really interpret the state of
the alignment of the model so is the
only way to do that to test it out of
distribution and see what happens at
this point
there are a bunch of different possible
ways to address this problem so
certainly one approach is to try to test
out a distribution
which is an adversarial training
approach this model is going to have
some potential failure modes off
distribution we can try to
find those failure modes and then train
the model on those failure modes
to prevent it from having this bad off
distribution behavior
there are some concerns with adversarial
training though in particular
adversarial training doesn't necessarily
catch what i see as the most pernicious
difficult inner alignment failure which
is something that we call deceptive
alignment in the risk of an optimization
paper
in the deceptive alignment case if the
model knows that it's being
adversarially trained then you're not
going to be able to figure that out just
via
throwing in a bunch of examples you can
also do something like transparency so i
mentioned previously that there's a lot
of really exciting transparency
interpretability work
if you're able to sort of look inside
the model and understand what algorithm
it's fundamentally implementing
you can see is it implementing an
algorithm which is an optimization
procedure that's aligned has it learned
a correct model of the loss function or
an incorrect model
it's quite difficult i think to hope to
solve this problem without transparency
interpretability
i think that to be able to really
address this problem we have to have
some way to appear inside of our models
i think that that's possible though
there's a lot of evidence that points to
the neural networks that we're training
really
making more sense i think than people
assume people tend to treat their models
as these sort of super black box things
but when we really look inside of them
when we look at what is it actually
doing a lot of times it just makes
sense i was mentioning some of the
circuits analysis work from the clarity
team at open ai
and they find all sorts of behavior like
we can actually understand
when a model classifies something as a
car the reason that it's doing that is
because it has a wheel detector and has
a window detector
and it's looking for windows on top of
wheels and so we can be like okay
we understand what algorithm the model
is influencing and based on that we can
figure out is it influencing the right
algorithm or the wrong algorithm
and that's how we can hope to try to
address this problem but obviously like
i was mentioning all of these approaches
get much more complicated in the
deceptive alignment situation which is
the situation which i think is
most concerning all right so i do want
to get in here
with you in terms of all the ways in
which inner alignment fails
briefly before we start to move into
this section i do want to
wrap up here then on outer alignment so
outer alignment is probably again what
most people are familiar with
i think the way that you put this is
it's when the objective function or the
loss function
is not aligned with actual human values
and
preferences are there things other than
loss functions or
objective functions used to train the
model via gradient descent
so i've sort of been interchanging a
little bit between loss function and
reward function
an injective function and fundamentally
these are sort of from different
paradigms machine learning so
reward function would be what you would
use in a reinforcement learning context
the loss function is the more general
term which is in a supervised learning
context you would just have a loss
function
you still have a loss function in a
reinforcement learning context but that
loss function is crafted in such a way
to incentivize the model to optimize the
reward function
via various different reinforcement
learning schemes so it's a little bit
more complicated than the sort of
hand-wavy picture
but the basic idea is machine learning
is we have some objective
and we're looking for parameterizations
of our model which do well according to
that objective
okay and so the outer alignment problem
is that we have absolutely no idea
and it seems much harder than creating
powerful optimizers
the process by which we would come to
fully understand human preferences
and preference hierarchies and values
yeah so
i don't know if i would say we have
absolutely no idea we have made
significant progress on outer alignment
in particular you can look at something
like
amplification or debate and i think that
these sorts of approaches have strong
arguments for why they might be outer
line
in the simplest form amplification is
about training a model
to mimic this hch process which is a
huge tree of humans consulting each
other
maybe we don't know in the abstract what
our ai
would do if it were optimized in some
definition of human values or whatever
but if we're just training it to mimic
this huge tree of humans then maybe we
can at least understand what this huge
tree of humans is doing and figure out
whether amplification is a line and so
there has been significant progress on
outer alignment which is sort of the
reason that i'm less concerned about it
right now
because i think that we have good
approaches for it and i think we've done
a good job
of coming up with potential solutions
there's still a lot more work that needs
to be done
a lot more testing a lot more to sort of
really understand do these approaches
work
are they competitive but i do think that
to say that we have absolutely no idea
of how to do this
is not true but that being said there's
still a whole bunch of different
possible concerns
whenever you're training a model on some
objective you run into all of these
problems of instrumental convergence
where if the model isn't really aligned
with you it might try to do these
instrumentally convergent goals
like keep itself alive potentially stop
you from turning it off
or all of these other different possible
things which we might not want and so
all of these
are what the outer alignment problem
looks like it's about trying to address
these standard value alignment concerns
like conversion instrumental goals
by finding objectives potentially like
amplification
which are ways of avoiding these sorts
of problems
right so i guess there's a few things
here wrapping up on outer alignment
nick bostrom's super intelligence that
was basically about
outer alignment then right primarily
that's right yeah
inner alignment hadn't really been
introduced to the alignment debate yet
yeah so i think the history of how this
sort of
concern got into the safety sphere is
complicated i mean so i mentioned
previously that there are people going
around talking about stuff like
optimization demons
and i think a lot of that discourse was
very confused and not pointing at
how machine learning actually works and
we're sort of just going off of well
it seems like there's something weird
that happens in evolution where
evolution finds humans that aren't
aligned with what evolution wants
that's a very good point it's a good
insight but i think that a lot of people
recoiled from this because it was not
grounded in machine learning because
i think a lot of it was very confused
and didn't fully give the problem the
contextualization that it needs in terms
of how machine learning actually works
and so the goal of risk monopolization
was to try and solve that problem and
really
dig into this problem from the
perspective of machine learning
understand how it works and what the
concerns are
now with the paper having been out for a
while i think the results have been
pretty good i think that we've gotten to
a point now where lots of people are
talking about interlinement and taking
it really seriously as a result of the
wristling optimization paper
all right cool so you didn't mention sub
goal so i guess i just want to
include that instrumental sub goals is
the jargon there right
convergent instrumental goals conversion
instrumental sub goals those are
synonymous
okay and then related to that
is good hearts law which says that when
you optimize for one thing hard
you oftentimes don't actually get the
thing that you want right
that's right and goodheart's law is a
very general problem
the same problem occurs both in inner
alignment and outer alignment
you can see goodheart's law showing
itself in the case of convergent
instrumental goals
you can also see goodheart's law showing
itself in the case of finding proxies
like going to the green arrow rather
than getting to the end of the maze
it's a similar situation where when you
start pushing on some proxy even if it
looked like it was good on the training
distribution it's
no longer as good off distribution good
heart's law is a really very general
principle which
applies in many different circumstances
are there any more of these
outer alignment considerations we can
kind of just list off here that
listeners
would be familiar with if they've been
following ai alignment
the outer alignment has been discussed a
lot i think that there's a lot of
literature on outer alignment you
mentioned super intelligence super
intelligence is primarily about this out
of alignment problem
and then all of these difficult problems
of how do you actually produce good
objectives and you have problems like
boxing
and the stop button problem and all of
these sorts of things that come out of
thinking about outer alignment
and so i don't want to go into too much
detail because i think it really has
been talked about a lot
so then pivoting here into focusing on
the inner alignment section
why do you think inner alignment is the
most important form
of alignment so it's not that i see
outer alignment as
not concerning but that i think that we
have made a lot of progress
on outer alignment and not made a lot of
progress on inner alignment
things like amplification like i was
mentioning i think are really strong
candidates for how we might be able to
solve something like outer alignment
but currently i don't think we have any
really good strong candidates for how to
solve inner alignment
you know maybe as machine learning gets
better we'll just solve some of these
problems automatically
i'm somewhat skeptical of that in
particular deceptive alignment is a
problem which i think is unlikely to get
solved as machine learning gets better
but fundamentally we don't have good
solutions to the underlying problem our
models are just these black boxes mostly
right now
we're sort of starting to be able to
peer into them and understand what
they're doing
we have some techniques like adversarial
training that are able to help us here
but i don't think we really have good
satisfying solutions
in any sense to how we'd be able to
solve in alignment because of that
interlinement is
currently what i see as the biggest most
concerning issue
in terms of prosaic ai alignment how
exactly does
inner alignment fail then where does it
go wrong
and what are the top risks of inner
alignment
i mentioned some of this before there's
this sort of basic maze example which
gives you the story of what an
inter-alignment failure might look like
you train the model on some objective
which you thought was good but the model
learned some proxy objectives some other
objective which when it moved off
distribution
it was very capable of optimizing but it
was the wrong objective
however there's a bunch of specific
cases and so in risk and learn
optimization
we talk about many of the different ways
in which you can break this
general inner misalignment down into
possible
sub problems so the most basic sub
problem is this sort of proxy pseudo
alignment is what we call it
which is the case where your model
learns some proxy
which is correlated with the correct
objective but potentially comes apart
when you move off distribution
but there are other causes as well there
are other possibilities which this can
happen
another example would be something we
call sub-optimality pseudo-alignment
which is a situation where the reason
that the model looks like it has good
training performance
is because the model has some deficiency
or limitation
that's causing it to be aligned where
maybe once the model thinks for longer
it'll realize it should be doing some
other strategy which is misaligned
but hasn't thought about that yet and so
right now it just looks aligned
there's a lot of different things like
this where the model can be structured
in such a way
that it looks aligned on the train
distribution but if it encountered
additional information
if it was in a different environment
where the proxy no longer had the right
correlations
the things would come apart and it would
no longer act aligned
the most concerning in my eyes was
simply which i'll call deceptive
alignment
and deceptive alignment is a sort of
very particular problem where the model
acts aligned because it knows that it's
in a training process
and it wants to get deployed with its
objective intact and so it acts aligned
so that its objective won't be modified
by the grady descent process
and so that it can get deployed and do
something else that it wants to do
in deployment so this is sort of similar
to the treacherous turn scenario
where you're thinking about an ai that
does something good and then it turns on
you
but it's a much more specific instance
of it where we're thinking not about
treacherous turn
on humans but just about the situation
of
the interaction between grading descent
and the model where the model maybe
knows it's inside of a gradient descent
process and is trying to trick that
gradient descent process a lot of people
on encountering this are like how could
this possibly happen
in a machine learning system i think
this is a good reaction because it
really is
a very strange thing to train a model to
do this
but i think there are strong arguments
for why separate alignment would
actually be the simplest
type of model that you could find in the
situation a way of explaining this i
think to anyone on the street would be
like
imagine if pigs were intelligent enough
to create farmers and you created
farmers and
they appeared to be aligned they took
care of you and they
gave you these nice mud pits and they
fed you every day
they gave you like a shelter and all
these other nice things and then
one day the farmer shows up and kills
you
right you thought the thing that you
created was a line but it was
deceptively aligned and it takes a
treacherous turn
is this sort of like a more mundane
example that you might agree with
so i think that's a good intuition pump
for thinking about the situation
i generally am sort of averse to trying
to really heavily apply these sorts of
analogies
because it's a good analogy to think
about what's happening but it doesn't
answer the core question of
how likely is this to actually happen in
the machine learning system
yeah that makes sense because it's much
more specific
than the other kinds of minds in this
mind space
it seems pretty rare a thing that could
exist but hard to find
right i think that's good intuition but
i'm going to try to disevaluate that
notion
first i think it's interesting to look
at maybe you do a bunch of adversarial
training
you're really pushing the model to
figure out what the objective is
it needs to know the objective at some
point if you are training in all
possible situations
it needs to know what the loss function
is for it to be able to do a good job
but there's multiple
possible channels via which information
about the loss function can enter the
model
and so i'll fundamentally distinguish
between two different channels
which is that the information about the
loss function can enter through the
gradient descent process
or it can enter through the model's
input data i'll call these two channels
internalization and modeling
so internalization is the situation
where you have this model that's going
along
and it has some proxy objective maybe
and that proxy objective is not quite
right
but then it gets moved to a new
situation where the proxy objective no
longer works
and gritty descent goes inside the model
and tinkers with the proxy to make it
slightly more accurate
iterate this process many many times and
eventually the proxy gets better and
better and better and better and better
until it starts matching up with the
actual loss function
but there's another story that i can
tell which is modeling if you think
about a training process like gpt3
where the model is trained on a huge
corpus
of all of this really rich input data
well in that sort of a situation there's
a whole bunch of really relevant
information and that input
to the agent's objective an agent if
it's in this environment where it has
this huge tax corpus
it's going to have to learn a model of
this environment and how it functions
we can imagine a situation where maybe
you're training the model on some huge
text corpus to do something like
maximize human values or whatever
and it reads a wikipedia page on ethics
and it's like wow look at all this
really relevant
rich useful information for figuring out
this objective
but then there's a second question which
is okay suppose that the model
has some model of the input data which
includes a bunch of rich information
inside of the model already about that
objective how does gradient descent
actually modify the model to make use of
that
information and so there's two different
types of modeling
which are deceptive alignment and
corrigible alignment
so the coordinate story is you have this
model it's going along it has some proxy
objective
but it encounters this really rich input
data which includes a huge amount of
information about the objective
to be able to predict successfully what
the wikipedia page on ethics is going to
say
it has to know about ethics so it learns
this really detailed ethics model and
then gradient descent is like look you
have this really detailed ethics model
i'm going to just modify your objective
to point to that ethics model
now your new objective is just optimize
that and so this leads to something sort
of like forgiability
where the model that you're training has
its objectives determined by a pointer
to some part of its world model it has
the model of this environment that
includes some
information about ethics and now it's
trying to optimize for that thing that
it's pointed to
in its world then there's this other
story which is the deceptive alignment
story
similar you have a model going along it
has some proxy objective
and it learns this really detailed world
model that includes a bunch of
information
about ethics or whatever and then grady
descent modifies the model to think
longer about the fact that it's inside
of an optimization process
and realize that the correct
instrumental thing for it to be doing
is to optimize for that ethics mod and
then it realizes oh i see i should
optimize for this model that i have
it goes and does that and in both
situations you end up being able to make
use of this really rich input data to
improve the performance
by changing the objective to make
reference to that but in the corrigible
case you actually modify it to point to
it whereas in the deceptive case you
just
modify the model to think about the fact
that it should be instrumentally
optimizing for it
and then the question is which one of
these is more likely before i go into it
though i want to dwell for a second on
the internalization versus modeling
distinction again
i think a good analogy is thinking about
animal imprinting
think about a duck you're trying to
train ducks to
follow their mothers or whatever you can
imagine a situation where the duck
internalizes
a model of its mother it is born with
some really detailed model of how to
figure out what a mother is
but there's also the modeling case where
the doc is going to be in the world is
going to have to form this really
detailed model of the world
and that really detailed model of the
world is going to have the very first
thing that encounters should always be
the mother
and so rather than having to learn some
detailed model of figuring out what
mothers are if you just
instead do some modeling process where
it's like just have the duck's objective
point
to the mother in its world model this
modeling case
you can see it as something like animal
imprinting where rather than really
internalizing the objective because it
already has this world model it might as
well just specify the objective in terms
of that world model instead
and i haven't yet talked about why you
would get deception versus
courageability right now i'm just trying
to convince you
that you should expect modeling you
should expect that if you give your
model this huge really rich detail input
data
that is going to learn an objective via
reference to that input data rather than
from scratch
when we think about models being trained
we think about them having some
objective and that objective is just
modified by grading descent as it moves
along to get closer and closer to
reality
but in the modeling case that's not
what's happening the model is
simultaneously learning a model of the
world
as well as trying to prove its objective
and the objective is modified to make
reference to that model of the world to
make it
more effective so internalization is it
just wants the same thing as you do and
so
it does it because it wants the same
thing whereas courageability is trying
to figure out what you want and then do
that
and then deception is also trying to
figure out what you want and then do
that but it's doing so for instrumental
rather than terminal reasons
the instrumental reason is the model can
get more reward
it can do better in the long run if it's
able to not be modified by the gradient
descent process so if it's doing a good
job
then grading descent process won't
change its objective and so therefore it
can get deployed with the same objective
and then go do a whole bunch of other
stuff that you don't want it to do but
why does it care about the current
objective
we're imagining a situation where you
have a proxy aligned
model but it cares about that proxy
across episodes it cares about that a
proxy over time
and it wants that proxy to continue to
be optimized for if the grading descent
process modifies it so that it no longer
cares about that proxy because of doing
a bad job
then that proxy won't get optimized for
it so if it wants that proxy to get
optimized for in the future
it needs to sort of play along pretend
to be doing the right thing
so that it doesn't get modified so that
it can get deployed and then go optimize
that proxy
but it also needs to understand that
it's within the context of an
optimization process where
gradient descent will continue to
perturb it that's exactly right
and where does that kind of awareness
come from
so now we have the second question which
is deception versus courageability
the awareness comes from gradient
descent gradient is trying to produce
models
which are better at the loss function
and
a deceptive model is really good and
really effective because
it's trying very hard to do the thing
which you want it to be doing
both deception and corridability are
situations where the model is trying to
figure out what to do and then do that
and the question is which is an easier
model which is a simpler
model for grady descent to find and i
think there's a strong argument for
deception being simpler than
courageability
courageability has some fundamental
difficulties in terms of if you're
imagining gradient descent process which
is looking at a proxy line model and
it's trying to
modify it so that it makes use of this
rich input data
it has to do some really weird things to
make corrigibility work it has to
first make a very robust pointer
with courageability if it's pointing at
all incorrectly to the
wrong thing in the input data wrong
thing in the world model the original
optimizer won't correct that pointer
it'll just be like well
i have this pointer i'm just trying to
optimize what this thing is pointing for
and if that pointer is pushing in a
proxy instead you'll just optimize that
proxy
and so you have this very difficult
problem of building robust pointers
with deception you don't have this
problem a deceptive model if it realizes
the loss function is different than what
it thought
it'll just change to doing the new loss
function it's actually much more robust
to new information because it's trying
to do this instrumentally and so in a
new situation if it realizes that the
lost budget server it's just going to
automatically change because it'll
realize that's the better thing to do
instrumentally
and so fundamentally deception is more
robust it gives you a more robust
pointer it's easier in that sense
there's another sense in which
cordiality is very difficult
to produce a cordial model requires
deception modifying the model's
optimization process to be capable of
optimizing under uncertainty
for a corrugal model because it has this
pointer as its objective
it's going to start not really knowing
what that pointer is pointing to it's
going to have some probability
distribution over the different
possibilities
and it's going to have to know how to
optimize under that probability
distribution of objectives
rather than under a single objective and
so this problem of optimization under
uncertainty is much more complicated
than just optimizing under a fixed
objective
what that means is that to produce a
corrigible optimizer gradient descent
has to do all of these modifications
where it has to modify it to
optimize under uncertainty and to have
the super robust pointer to
the part of his world model that's
supposed to be optimizing for but if you
compare this to deception
gradient descent modifying a model to
become deceptive is extremely simple
it needs to modify the model to have a
long-term objective to think about
what its objective is across many
different instances
and then it just needs to modify the
model to think for a while about the
fact that it's in an optimization
process and what the right thing is to
do
and that modification is not very
complicated because it's just a simple
modification of
think moral about this particular thing
which is exactly the sort of
modification that you should expect to
happen
all the time when you're training a
model and so i think it's a
fundamentally much simpler modification
there's also another argument you can
make here which is there's just a lot
more deceptive models
any proxy objective once a model which
is optimizing that proxy objective
starts optimizing that objective more in
the long term across episodes
and then thinks about the fact that it's
an opposition process will become
deceptive
but to produce cordiality you have to
find exactly the right pointer
there's many different possible pointers
out there only one of which is going to
give you the exact correct pointer
and similar with the internalization
there's many different processes only
one is the actual true
loss function whereas with deceptive
alignment any of those different proxies
they're all compatible with deception
and so
i think there's a lot of strong
arguments both this accounting argument
for there being many more deceptive
optimizers
as well as the simplicity argument for
the modification necessary to produce a
deceptive optimizer is just a lot
simpler i think than the modifications
necessary to produce these other types
of optimizers
and so because of this i think that
there's a strong case to be made for
deception really
not being that uncommon not being
something crazy to think what happened
in the training process but is
maybe even potentially the default
outcome of a lot of these sorts of
training procedures
which is quite quite scary and quite
concerning and obviously all of this is
speculation we're trying to understand
from a theoretical process what this
gradient extent process might do
but i think we can make a lot of strong
cases about thinking about things like
simplicity and accounting arguments to
at least
put this problem on the radar until we
have a really strong reason that this
isn't a problem
we should take it seriously buck who's
another person who works in very often
tries to explain some of the risk
modernization stuff and he has a
analogy that might be useful here you
can imagine the christian god
and the christian god is trying to
produce humans which are
aligned with the bible and you can
imagine three different possible humans
you have jesus who is just the same as
god jesus
has the same objective as god jesus is
aligned with god because he just
fundamentally wants to do the exact same
things
that's internalization that would be
internalization you could have martin
luther
martin luther is aligned with god
because he wants to really carefully
study the bible
figure out what the bible says and then
do that and that's the courageability
case
or you can have blaise pascal and blaise
pascal
is aligned with god because he thinks
that if he
does what god wants he'll go to heaven
in the future
and these are the three different
possible models that god could find
and you're more likely to find a jesus a
martin luther or a blaise pascal
and the argument is there's only one
jesus out of all the different possible
human objectives
only one of them is going to be the
exact same one that god wants and martin
luther similarly is very
difficult because out of all the human
objectives there's only one of them
which is
figure out precisely what the bible
wants then do that but blaise pascal
in this situation anybody who realizes
that god's going to send them to heaven
or hell or whatever based on their
behavior
will realize that they should behave
according to the bible or whatever and
so there's many different possible
blaise pascals
but there's significantly fewer possible
martin luther's and jesus's
yeah i think that's an excellent way of
simplifying this blaze pascal could care
about any kind of proxy
i guess the one thing that i'm still a
little bit confused about here
is in terms of the deceptive version
again why is it that it cares about the
current proxy reward
i think that has to do with the
structure of the training procedure
you start out usually i think with a
proxy aligned mesa optimizer
maybe you start out with a bunch of
heuristics but then once you get a mesa
optimizer
it'll usually start by being proxy
aligned you'll have some proxy that's
not quite the same as the loss function
at least if you're in a complex enough
environment
there's a whole bunch of different
possibilities it's going to start with
some proxy but then you might hope that
if you do a bunch of adversarial
training you train a really long time in
a bunch of different environments
you'll be able to improve that proxy
until you get to the point where it has
the right thing
and the receptive line is about asking
the question there's actually multiple
different possible ways
for a model to figure out the loss
function it could figure it out the
internalization
or it could figure out via deception or
it could figure out the approachability
and we don't know which one of these is
more likely you might imagine that if
you start with a proxy aligned based
optimizer you train it a bunch you'll
internalize the correct objective
but the argument that i'm making here is
that that might not be what happens
and so the reason i'm saying you start
with the proxy aligned based optimizers
because
i think that's the way that you should
expect that the training procedure will
be structured that you'll
start by learning some proxy and then
that proxy will get better
but that having that proxy get better
could result in internalization or
deception or courageability
sure so the model develops some kind of
proxy
but it seems like the other necessary
ingredients are the
one it knows that it's in an
optimization process
where gradient descent is changing its
objective function
and then it requires two it actually
cares about
its current objective function
over time and over iterations that's
absolutely right
and so then the question to ask is how
likely are those things to occur
and i think it's the really important
point which i want to make i think a lot
of times when people think about this
they're like it seems unlikely that it
would just happen to develop this
understanding and
care about its thing long term it
doesn't just happen
it happens as a result of the training
procedure because
if it does happen it results in
significantly better performance
and so the question is would gradient
descent modify the model
to have those properties and that's the
argument that i want to make is that
it's not that necessarily that the model
just happens to develop deception
but the deceptive models are just really
effective at pursuing reward
on the training environment and so you
should expect that if you have a really
good training process that it's going to
find deceptive models because they're
really
good at doing the thing you're training
them for and so that's the sort of most
fundamental argument that i want to make
although i do think there's another
level here where i think that as you're
getting to the point where
training models and very complex data
sets where they have huge amounts of
really rich information
i think we should expect the model to
basically figure out most things
you should expect that if it's gonna
have to learn all of this really complex
data it's going to have to build a world
model that's going to conclude
the training process that's going to
include the loss function
and so i think that as we move into this
direction of training more and more
complex data sets more and more complex
models you don't want to rely on your
model
not figuring something out especially
when figuring that thing out results in
having better performance
now there's also this question of does
it care about its return across episodes
versus just across
multiple steps and for deception it
needs to care about the multi-episode
return
and this is also another critical point
i think there's a strong argument that
if you imagine a model being trained to
care
about its multi-step return and maybe
also i should pause here and explain
what i mean by step versus episode
so in the standard reinforcement
learning setup you have
many different episodes each episode is
composed of many different steps
and you train the model to only optimize
its single episode return
its reward across all the different
steps in an episode but not across
episodes
that's not always the case there are
some reinforcement learning procedures
that optimize across episodes
an example of this is something like
population based training population
based training selects models which have
the greatest reward across episodes
not simply within the episode though
standard reinforcement learning
you're just selecting for the single
episode return but even if you're just
looking for a single episode return
you have the same general problem that
we've been talking about
where it's not the case that you
actually end up with a model necessarily
that only cares about single step return
you're just going to end up with a model
that behaves like it cares about single
step return on the train distribution
and if you imagine a situation where you
know you have a model and it's been
trained to optimize for this objective
over many steps
and then it discovers that actually
there's these other episodes they're
beyond this current step
i think that the most natural
generalization is that you should expect
that the model is going to care about
those other steps too
you can imagine a situation where every
situation where the model has seen a
blue door it's been like oh going
through this blue door is really good
and it's learned an objective that
incentivizes to go through blue doors
if it then later realizes that there are
more blue doors than it thought because
there are other blue doors in other
episodes
i think you should generally expect he's
going to care about those blue doors as
well
but again even if you don't buy that
it's not just about what's most natural
it's also about what does the gritty
descent process do and because
models which have this property that
plan across episodes can become
deceptive and therefore do really well
in the training distribution
you should expect that a good training
process is going to modify them to
become deceptive even if they would
naturally
okay so help me just understand a little
bit more here
at a high level it seems that the
internalization
and the corrigibility things require
higher specification than the deception
version
you've brought up all these different
arguments about why you think
that the deceptive version is
easier to find because it has all of
these
computational benefits in terms of
minimizing the loss function right
that's right and the structure of the
deceptive
model is composed of as we mentioned
earlier
one the knowledge that it is within some
kind of optimization process where it is
being perturbed
by gradient descent over episodes
and that it cares about
the current proxy over episodes
and seems that you're arguing correct me
if i'm wrong
the manifestation of these components is
that
the strong desire to deceive
leads to a very strong
direct attempt at actually minimizing
the loss
or accomplishing the objective function
that's right
because if it fails at doing that it
won't be able to accomplish
the proxy that it actually cares about
later
that's right if there's more deployment
episodes than training episodes and it
just cares about how many times it goes
through the blue door
or you know the green arrow or whatever
the green arrow is a proxy objective and
so if it gives
up the fact that it's optimizing the
green arrow it's going to get modified
to not do that anymore
and so if it wants to go to the green
arrow in all of these different
deployment episodes
it needs to first in the training
procedure pretend that it doesn't care
about the green arrow
and then once it's deployed switch to
going after the greenhouse
this also requires the model to sort of
know when it's in training and when it's
in deployment this is another situation
where i think you should expect that
when you're training really advanced
models with these huge world models that
they figure out this
information but also even if it doesn't
figure out that information
it can gamble it can be like well
there's some chance i'm in deployment
there's some chance i'm in training
i will just opt to go to the green arrow
some percentage of the time
and to do the correct thing some other
percentage of the time and this gives
you a decision problem which you can
solve mathematically
you get some optimal defection
probability okay
and then going up in terms of
specification one step
the corrigible agent
needs to have a very detailed world
model
where it has a pointer via gradient
descent
at some specific part of the world model
that seems to
correlate highly with minimizing the
loss
that's right why is that less likely
well so the major two reasons that i
want to argue for are
one getting the pointer right is really
hard and very difficult to do robustly
whereas getting the deception right is
much easier to do robustly
think about animal imprinting right if
the animal imprints on
a human caretaker instead if you have a
misspecified pointer it doesn't actually
have very good performance
the deceptive alignment is more robust
at figuring out what the thing is it's
trying to optimize for
because the quarter alignment relies on
this pointer working successfully and if
the pointer fails
it's just going to optimize the wrong
thing but why can't the pointer change
just as well as the attempt at
satisfying the actual loss function in
the deceptive
model so it absolutely can but the
question i want to ask is how
difficult is it for gradient descent to
make one sequence of modifications
versus another siemens modifications
the sequence modification leads to
concordability versus the sequence that
leads to deceptive alignment
the sequence of modifications necessary
to produce coercive alignment
requires all of this very very careful
pointer tuning
as you can imagine if you're considering
an infinitesimal movement in the
direction of chords of alignment
imagine replacing your current proxy
with a bad pointer it's not going to be
that great you're going to have to put a
lot of careful optimization effort into
getting a really good pointer before it
becomes
better to be corrigible whereas with
deception the model could build up this
really good
world model and once the world model
gets complicated enough that it includes
a specification of the fact that it's in
the training process as soon as it
starts thinking about the training
process
it's going to be as robust as it can be
because it's going to instrumentally
just want to try to figure out
what is the best possible approximation
i can get for this thing i'm being
trained for so i can optimize it as
effectively as possible
and so this gives you a much more robust
pointer compared to the chords of
alignment case
and so the danger would be with the
deceptive version
once it's unleashed upon the world and
it has a very excellent world model it
realizes this and
finally now i can do the deceptive turn
or something like that
to actually optimize the proxy that's
right
yeah so we've covered a lot of the ways
in which
inner alignment fails now
inner alignment and outer alignment are
two of the things which
you care about for evaluating proposals
for building
safe and advanced ai there are two other
properties that you care about
training procedures for building
beneficial ai
one of these is training competitiveness
and the second one is performance
competitiveness
could you explain what training
competitiveness is and performance
competitiveness and why they are both
important
absolutely yeah so i mentioned at the
beginning that i have a broad view of ai
alignment
where the goal is to try to mitigate ai
essential risk
and i mentioned that what i'm working on
is focused on this intent alignment
problem
but a really important facet of that
problem is this competitiveness question
we don't want to produce ai systems
which are going to lead to ai
existential risk
and so we don't want to consider
proposals which are directly going to
cause problems
as the safety community what we're
trying to do is not just come up with
ways to not cause existential risk
not doing anything doesn't cause
essential risk it's to find ways to
capture the positive benefits of
artificial intelligence to be able to
produce
ais which are actually going to do good
things you know why are we actually
trying to build ai's in the first place
we're actually trying to build ais
because we think that there's something
that we can produce which is good
because we think that ai's are going to
be produced on the default timeline and
we want to make sure
that we can provide some better way of
doing it and so the competitiveness
question is about how do
we produce ai proposals which actually
reduce the probability of existential
risk
not that just don't themselves cause
existential risk but then actually
overall reduce the probability of it for
the world
there's a couple of different ways that
can happen you can have a proposal which
improves our ability to produce
other safe ais so we produce some
aligned ai and that aligned ai
helps us build other ais which are even
more lined and more powerful
we can also maybe produce an aligned ai
and then producing that aligned ai helps
provide an example
to other people of how you can do ai in
a safe way or maybe it provides some
decisive strategic advantage
which enables you to successfully ensure
that only good ais you produce in the
future
there's a lot of different possible ways
in which you can imagine building an ai
leading to reduced existential risk but
competitiveness is going to be a
critical component of any of those
stories
you need your ai to actually do
something and so i like to split
competitiveness down into two different
sub components
which are training competitiveness and
performance competitiveness and in the
overview level proposals document that i
mentioned
at the beginning i compare 11 different
proposals for prosaic ai alignment
on the four qualities of outer alignment
inner alignment
training competitiveness and performance
competitiveness so training
competitiveness
is this question of how hard is it to
train
a model to do this particular task it's
a question fundamentally of if you have
some team
with some lead over all different other
possible ai teams
can they build this proposal that we're
thinking about without totally
sacrificing that lead how hard is it to
actually
spend a bunch of time and effort and
energy and compute and data
to build an ai according to some
particular proposal
and then performance competitiveness is
the question of once you've actually
built the thing
how good is it how effective is it what
is it able to do in the world that's
really helpful for reducing existential
risk
fundamentally you need both of these
things and so you need
all four of these components you need
outer alignment inner alignment training
competitiveness and performance
competitiveness
if you want to have a prosaic ai
alignment proposal that is aimed at
reducing existential risk this is where
a bit more reflection on governance
comes into considering
which training procedures and models are
able to satisfy the criteria for
building safe advanced ai in a world of
competing actors and different
incentives and
preferences the competitive stuff
definitely starts to touch on
all those sorts of questions when you
take a step back and you think about how
do you have an actual full proposal for
building for
ai in a way which is going to be aligned
and do something good for the world you
have to really consider all of these
questions
and so that's why i try to look at all
these different things in the document
that i mentioned
so in terms of training competitiveness
and performance competitiveness
are these the kinds of things which are
best evaluated from
within leading ai companies and then
explained to say people in governance or
policy or strategy
it is still sort of a technical question
we need to have a good understanding of
how
ai works how machine learning works what
the difficulty is of training different
types of machine learning models
what the expected capabilities are of
models trained under different regimes
as well as the outer alignment and inner
alignment that we expect will happen
i guess i imagine the coordination here
is that information on relative training
competitiveness and performance
competitiveness in systems
is evaluated within ai companies and
then possibly fed
to high power decision makers who exist
in strategy and
governance for coming up with the
correct strategy given the landscape of
companies and ai systems which
exist yeah that's right all right
so we have these intent alignment
problems we have inner alignment
and we have outer alignment we've
learned about that distinction today
and reasons for caring about training
and performance competitiveness
so part of the purpose of this i mean
it's in the title for this paper that
partially motivated this conversation
and
an overview of 11 proposals for building
safe and advanced ai
you evaluate these proposals based on
these criteria
as we mentioned so i guess so i want to
take this time now
then to talk about how optimistic you
are about
say your top few favorite proposals for
building safe and advanced ai
and how you've roughly evaluated them
on these four criteria of inner
alignment outer alignment
and then performance and training
competitiveness
i'll just touch on some of the ones that
i think
are most interesting to start by taking
a look at and it's not necessarily just
going to be the ones i think i'm most
excited about
because some are more complicated than
others and i want to start with some of
the simpler ones
and some of the ones that i am fairly
excited about i think are interesting
but they're also pedagogically useful to
take a look at
i'll start with amplification which is
the approach that i probably spend the
most time thinking about
the specific form of amplification that
i want to start by talking about is
imitative amplification
where imitative amplification
specifically means the procedure of
training a model to imitate a human
with access to that model the goal is to
produce
models imitating this hch tree which i
mentioned earlier this tree of humans
consulting humans the models trained to
mimic a
human consulting the model but then the
models that that human is consulting
were themselves trained to mimic humans
consulting models which were themselves
trained to mimic humans consulting
models
so in the limit you're hoping to end up
with this model trained to mimic this
tree of humans consulting humans
consulting humans which is hch for
humans consulting hch the specific way
that
i talk about this though in the overview
of 11 proposals
is that there's an additional thing
which you can do on top of this
imitative objective
which is we also want to add some form
of inner alignment solution so
we might think that imaginative
amplification is going to be good in the
limit
it's going to be good if we can actually
produce models which are really trying
to predict hch but we also have to
actually ensure that we're doing that
that we don't get deceptive models or
other sort of bad inner misalignment
problems
and so we can add on top of the imitated
amplification something which i call
relaxed adversarial training i have
another post that came out of some of
the work i did open ai
which really goes in detail into how you
would do something like relaxed
adversarial training and what it means
how it works
but fundamentally the idea behind
relaxed adversarial training is to
train the model both to do the base
objective of trying to imitate the
human consulting model but also on an
additional objective
of trying to decrease the probability
that the model is doing something bad
it's deceptive as assessed by
some oversea which is looking at the
model which in this case in the case of
application the overseer would be
the human with access to the model so
you're training overall on the objective
of
mimicking the human which access to the
model and
making it so that the human with access
to the model when that human
with the help of the model uses
transparency tools to look at the model
and see what it's doing
making sure that that human believes
that the model is doing the right thing
and so you're combining training on the
imitative objective as well as on
the relaxed adversarial training
objective trying to make it so that this
oversight works
how do you evaluate that in terms of
your criteria
we can start with power alignment so the
question of outer alignment is is this
hch tree align
and the hope is that because this hdh3
is just composed of humans that if we
can understand the human behavior and
ensure that those humans are doing the
right thing
which we might hope to do you know we've
solved lots of human coordination
problems in the past that we can ensure
that this thing is outer land
for inner alignment in alignment is this
question of relaxed adversarial training
so in particular the way in which i
think about solving relaxed adversarial
training in this situation
is that you have this big model space
and you have some set of deceptive
models
and if you want to avoid that set of
deceptive models one thing which you can
do
is you can find some larger set which
contains all the deceptive models
such that it's very easy to check
whether model's in that set or not
as long as outside of the set contains
enough competitive models that they're
able to
do the right thing your life savers real
training will work there's a bunch of
different ideas of how you could define
a set like this
but the one which i'm most excited about
which you think about the most is to
have the set be based on the distinction
of my opia
where you run this relaxed episode
training check by looking at
the model determining whether the model
cares about its multi-episode return or
not
and if the model does care about its
multi-episode return then it has the
potential to become deceptive
and so we don't allow that we have it so
that the overseer if it sees that it
cares about the multi-episode return
oversight fails and if it sees that it
doesn't care about the multiple return
then the oversight passes
and then we train the model to cause the
oversight to pass so that it doesn't
develop this behavior where it cares
about the multi-episode return
and that's the basic idea of how you
could use relaxed episode training here
to solve inner alignment and then you
have training competitiveness
it's sort of this imitation learning
problem where trust's trying to train
the model to imitate the
human with access to the bottle there's
all sorts of potential problems here
it could be very difficult to get a
bunch of human data for this problem
as well as if you're just doing
imitation learning there's some concerns
of is the model really going to learn
these sort of powerful long-term
optimization
especially if you're specifically
selecting it to be myopic
also there's a question of how rich is
the reward signal which is produced by
imitation learning unanimitation
learning uses some sort of fixed
distance metric to determine the
distance between what the
model produced and what was predicted
generally this is done by having the
language model which is the probability
distribution and then determining the
probability under that distribution
of the sentences produced by the human
with access to the model
and so you have this problem of how rich
is the training signal how difficult is
the training
i think we don't really don't know what
the training pattern is going to be but
i think there's reason to be optimistic
i mean a lot of these sorts of simple
language modeling tasks like gb2 and
gpt3 have been very successful and so
this is very similar to that in that
sort of vein as you can hope that it
will also be hopefully successful
similarly
and then you have the performance
competitiveness and for performance
competitiveness the real question is how
powerful is this hch thing
can hch actually solve all the problems
we probably solved
and this is going to be a question of
you know how do you train your humans
can a human take any question
and decompose it into different sub
questions such that answers those sub
questions can be recombined and to
answer
the whole question is this possible for
all questions so there's some questions
which are undecomposable in which case
it wouldn't be competitive but if you
can decompose all these questions they
sort of shouldn't have performance
competitiveness
so the one that i've been talking about
imaginary application plus relaxed after
serial training is the one that i work
on most and it's probably the one i'm
most optimistic about there's others
that are also really interesting i mean
i think all of the proposals every pros
on the list has lots of strong arguments
in favor of it and lots of
real reasons to consider it another
proposal which i have talked about
previously is ai safety via debate
so async can be a debate has some
similar properties to amplification but
it has a different objective
rather than trying to mimic hch you're
trying to do this debate procedure
where you have two different models
trying to defend two different sides of
some proposition and convince the judge
to agree with one of those sides there's
some strong arguments
for why this sort of procedure would
have nice out of alignment properties
because it converges to something like
what the
judge's equilibrium beliefs are after
having seen all the arguments in favor
and so we might hope that this sort of
equilibrium beliefs
are going to be good and there's lots of
work also that has been done on trying
to improve the ability to access like
the true
equilibrium beliefs to really give the
judge all possible information
there's some work that was done by paul
christiano and beth barnes about this
cross-examination mechanism which gives
it significantly better performance
competitiveness properties by allowing
it to access a larger complexity class
but then the other question is then what
can you do on top of ai safety via
debate to help with inner alignment
and an interesting thing that you can do
is you can have the models trained to
use transparency tools on each other to
win the debate
if one model is able to look inside the
other model and demonstrate that it's
non-biopic or something
then it wins the debate and so then you
can incentivize the models to play this
game of trying to look inside of each
other and find the inner alignment
problems
and then train away from those problems
for outer alignment you sort of really
have this question
of are the debaters actually
incentivized to have some honest
cooperative
helpful answers in the limit and this is
a question of what that equilibrium
really looks like is it the case that
humans are able to be tricked by really
persuasive arguments and no
counter-argument will unpersuade them
or is it the case that when humans are
exposed to all of the best arguments and
counter-arguments they come to the right
conclusions we don't really know
but this is fundamentally what the outer
lining question is here this is sort of
a trickier equilibrium to analyze
with the amplification case we just had
to figure out with this concrete tree of
humans
but with debate we have to understand
this more abstract what is the
theoretical equilibrium behavior of a
human
under exposure to all these different
arguments and so i think there's a case
that from an online perspective debate
is at least more
questionable then we have inner
alignment the situation here is
how successfully can we incentivize
these models to inspect each other and
be able to identify catastrophic
behavior
and there's certainly some problems here
you have to get a good training signal
to be able to produce models which are
good and inspecting each other which
probably means you have to produce
deceptive models and then have the
models find the inception or at least
produce non-myopic models and have the
models find the non-myopia
but then this could also incentivize the
models to hide the non-mafia so it's a
little bit unclear exactly how this
works it's sort of going to be depend on
precisely the properties of the training
procedure
i think there's some good reasons to be
hopeful here for training
competitiveness because the way that the
debate game is structured it lets you do
this
self-play structure where you have these
agents competing on the zero-sum game in
a self-play
which is in many ways similar to a lot
of big successes of things like alpha
zero where alpha zero is able to solve
all these environments by doing this
very successful self-play on these
zero-sum games
the biggest problem here is probably
getting data and then there's the
performance competitiveness question
which is how effective would an actual
debate agent be
the hope is that in the limit you should
just be able to ask the first debater
what do you think and then it'll just
choose the most convincing answer and
then you can just go with that you don't
have to run the whole debate in
deployment
but it's only going to be able to solve
these language problems it's going to
give you as the equilibrium of what a
human thinks after all of these
different arguments and is that good
enough
is it the case that humans are going to
really be able to come to good enough
equilibria
after they see all these arguments that
they're going to be able to produce
really good answers
and also is it the case that question
answering alone is sufficient to be able
to be competitive in potentially a very
competitive
marketplace as a third proposal that i
think is interesting to go into
is something called microscope ai
microsoft ai
i think is really interesting to look at
because it's very different from the
other process i was just talking about
and it has a very different approach to
thinking about how do we solve these
sorts of problems
for all of these approaches we need to
have some ability to look inside of our
models and learn something about what
the model knows
but when you use transparency tools to
look inside a model it teaches you
multiple things
it teaches you about the model you learn
about what the model has learned
but it also teaches you about the world
because the model learned a bunch of
useful facts and if you look inside the
model and you can learn those facts
yourself then you become more informed
and so this process itself can be quite
powerful and so that's fundamentally the
idea of microscope
the idea of microscope ai is to train a
predictive model on the data you want to
understand
and then use transparency tools to
understand what that model learned
about that data and then use that
understanding to guide human decision
making
and so if you think about outer
alignment in some sense this procedure
is not really underlined because
we're just trying to predict some data
and so that's not really a line
objective if you had a model that was
just trying to do a whole bunch of
prediction it wouldn't be doing good
things for the world
but the hope is that if you're just
training a predictive model it's not
going to end up being deceptive
or otherwise dangerous and you can also
use transparency tools to ensure that it
doesn't become that
we still have to solve inner alignment
like i was saying it still has to be the
case that you don't produce deceptive
models
and in fact the goal here really is not
produced based optimizers at all the
goal is just to produce these predictive
systems which learn a bunch of useful
facts and information but that aren't
running optimization procedures
and hopefully we can do that by having
this very simple predictive objective
and then also by using transparency
tools
and then training competitiveness we
know how to train powerful predictive
models now you know something like gbt2
down now gbt3 these are predictive
models on tax prediction
and so we know this process we know that
we're very good at it
and so hopefully we'll be able to
continue to be good at into the future
the real sticking point with microscope
ai is the performance competitiveness
question
so is enhanced human understanding
actually going to be sufficient to solve
the use cases we might want for like
advanced ai
and i don't know it's really hard to
know the answer this question but you
can imagine some situations where it is
in some situations where it isn't
so for situations where you need to do
long-term careful decision-making
it probably would be right if you want
to replace ceos or whatever that's a
sort of very general decision-making
process that can be significantly
improved just by having
much better human understanding of
what's happening you don't necessarily
need the ai's making the decision
on the other hand if you need
fine-grained manipulation tasks or
very very quick response times ai's
managing a factory or something
then maybe this wouldn't be sufficient
because you would need the ai's doing
all of this quick decision-making and
you can have
it just giving information to a human
one specific situation which i think is
important to think about also is the
situation of using your first ai system
to help build
a secondary system and making sure that
second assist is aligned and competitive
and i think that it also performs pretty
well there you could use a microscope ai
to get a bunch of information about the
process of ais and how they work and how
training works
and then get a whole bunch of
information about that have the humans
learn that information
then use that information to improve our
building of the next ais and other ais
that we build
there are certain situations where
microscope ai is performance competitive
situations where it wouldn't be
performance competitive but it's a very
interesting proposal because it's sort
of tackling it from a very different
angle
it's like well you know maybe we don't
really need to be building agents maybe
we don't really need to be doing this
stuff maybe we can just be
building this microscope i should
mention
the microstrip ai idea comes from
crystalla who works at openai
the debate idea comes from jeffrey
irving who's now a deep mind an
application comes from paul christiano
who's that opening
yeah so for sure the best place to
review these is by reading your
post and again the post is an overview
of 11 proposals for building safe
advanced ai by
evan hubinger and that's on the ai
alignment forum
that's right i should also mention that
a lot of the stuff that i talked about
in this podcast
is coming from the risks from learned
optimization in advanced machine
learning systems paper
all right wrapping up here i'm
interested in
ending on a broader note i'm just
curious to know if you have
concluding thoughts about ai alignment
how optimistic are you
that humanity will succeed in building
aligned ai systems
do you have a public timeline that
you're willing to share about
agi how are you feeling about the
existential prospects of
earth originating life uh it's a big
question
so i tend to be on the pessimistic side
my current view
looking out on the field of ai and the
field of air safety
i think there's a lot of really
challenging difficult problems that
we're at least not currently equipped to
solve and it seems quite likely that we
won't be equipped to solve by the time
we need to solve them
i tend to think that the prospects for
humanity aren't looking great right now
but i nevertheless have a very sort of
optimistic disposition we're going to do
the best that we can
we're going to try to solve these
problems to practically as we possibly
can and we're going to work on it and
hopefully we'll be able to make it
happen
in terms of timelines it's such a
complex question
i don't know if i'm willing to commit to
some timeline publicly i think that it's
just one of those things that is
so uncertain it's just so important for
us to think about what we can do across
different possible timelines
and be focusing on things which are
generally effective regardless of how it
turns out because i think we're really
just quite uncertain you know it could
be
as soon as five years or as long away as
50 years or 70 years we really don't
know
and i don't know if we have great track
records of prediction in this setting
regardless of what ai comes we need to
be working to solve these problems and
get more information on these problems
gets the point we understand them we can
address them because
when it does get to the point where
we're able to build these really
powerful systems
we need to be ready so you do take very
short timelines like say five to 10
to 15 years very seriously i do take
very short timelines very seriously i
think that
if you look at the field of ai right now
there are these
massive organizations open ai and deep
mines that are dedicated to the goal
of producing aji and are putting a huge
amount of research effort into it and i
think it's incorrect to just assume that
they're going to fail
i think that we have to consider the
possibility that they succeed and that
they do so quite soon a lot of the top
people at these organizations have very
short timelines
and so i think that it's important to
take that claim seriously to think about
what happens if it's true
i wouldn't bet on it there's a lot of
analysis that seems to indicate that at
the very least we're gonna need more
compute than we have
in that sort of a time frame but
timeline prediction tasks are so
difficult
that it's important to consider all
these different possibilities i think
that yes i take the short timelines very
seriously but it's not the primary
scenario i think that i also take
long timeline scenarios quite seriously
would you consider deep mind and open ai
to be explicitly trying to create agi
open ai yes right yeah open ai is just
part of the mission statement
deep mind some of the top people at
deepmind i've talked about this
but it's not something that you would
find on the website the way you with
open ai
but if you look at historically some of
the things that shane legg and dennis
hospital
said a lot of it is about agi yeah so
in terms of these being the leaders with
just massive
budgets and person power how do you see
the quality and degree of
alignment and beneficial ai
thinking and mindset within these
organizations because you know there
seems to be a big distinction between
the ai alignment crowd and the
mainstream machine learning crowd
a lot of the mainstream ml community
hasn't been exposed to
many of the arguments or thinking within
the safety and alignment crowd
stuart russell has been trying hard to
shift away from the standard model and
incorporate a lot of these new alignment
considerations
so yeah what do you think i think
there's a problem that is getting a lot
better
like you were mentioning stuart russell
has been really great on this
chai has been very effective at trying
to really get this message of you know
we're building ai we should put some
effort into making sure we're building
safe ai
and i think this is working if you look
at a lot of the major ml conferences
recently
i think basically all of them had
workshops on beneficial ai
deepmind has a safety team with lots of
really good people opening eyes a safe
team with lots of really good people
i think that the standard story of oh
you know ai safety is just this thing
that these people who aren't involved in
machine learning think about
it's something which really in the
current world has become much more
integrated with machine learning
and is becoming more mainstream but it's
definitely still a process
and it's the process of like stuart
russell says the field of ai has been
very
focused on this sort of standard model
and trying to
move people away from that and think
about some of the consequences of it
takes time and it takes some sort of
evolution of the field
but it is happening i think we're moving
in a good direction
all right well evan i've really enjoyed
this
i appreciate you explaining all of this
and taking the time to
unpack a lot of this machine learning
language and
concepts to make it digestible is there
anything
else here that you'd like to wrap up on
or any concluding thoughts
if you want more detailed information on
a lot of the things that i've talked
about
the full analysis of internal element
and alignment is in
risk mode optimization in advanced
machine learning systems by
me as well as many of my co-authors as
well as an overview of 11 proposals post
which you can find on the ai alignment
form
i think both of those are resources
which i would recommend checking out for
understanding more about what i talked
about in this podcast
do you have any social media or a
website or anywhere else for us to point
towards
yeah so i mean you can find me on all
the different sorts of social media
platforms
i'm fairly active on github i do a bunch
of open source development
you can find me on linkedin twitter
facebook all those various different
platforms
i'm fairly googlable it's nice to have a
fairly unique last name so if you google
me you should find all this information
one other thing which i should mention
specifically everything that i do
is all public all of my writing is
public i try to publish all of
my work and i do so on the ai alignment
form so the ai alignment form is a
really really great resource because
it's a collection of writing by all of
these different ai safety authors
it's open to anybody who's a current ai
safety researcher
and you can find me on the alignment
firm as evha by evhub
on the alignment forum all right evan
thanks so much for coming on today and
it's been quite enjoyable
this has probably been one of the more
fun
ai alignment podcasts that i've had in a
while so thanks a bunch and
i appreciate it absolutely that's super
great to hear i'm glad that you enjoyed
it
hopefully everybody else does as well
if you enjoyed this podcast please
subscribe give it a like or share it on
your preferred social media platform
we'll be back again soon with another
episode in the ai alignment series