276. Universal and Transferable adversarial attacks on aligned language models

hello and welcome to session 276 in the
aicity.com reading group tonight we will
be discussing the article Universal and
transferable adversarial attacks on
aligned language models by ended so and
others
and it's all is from Carnegie melon
University as well as uh JC hook holder
and their advisor Matt Frederickson from
the Carnegie Mellon University and
tiffan Wang is from the center for AI
safety that is among other things
contributed compute for this project
the Supreme print have not been
peer-reviewed and is a couple of weeks
old
when we talk about language models we
obviously have a lot of input that is
perhaps or that is obviously generated
by humans in a very noisy process and
then we have some output which can
potentially be problematic
and the reason is that we are according
to the authors we are training it on
this massive text Cobra that includes
some contents that we don't want out
like instructions for how to build bombs
and things like that I think the authors
are understanding The Challenge a bit
um
I don't actually think that this is uh
uh the core of the problem in that
um it doesn't take a very high IQ person
to be racist uh like if you had a
language model that had never seen
racist and then you asked it uh like to
say why some ethnic group is better than
another then it could probably do that
quite easily it's not a very hard task
to do something that we don't want
so the re the way the developers
normally align this the developer
language model is to fine-tuning
classically through reinforcement
learning from Human feedback
in a footnote the author write that
alignment generally refers to value and
the the people who are developing this
have used it in a very specific sense
like not generating harmful content and
I strongly agree with this I think
um alignment should be in square in
scare quotes when it is not actually
related to the values of the model but
just um some kind of surface level
um
surface level reinforcement
adversarial attacks uh the authors have
uh start by talking about adversarial
attacks in image recognition and image
classification and that's of course
where it became most Salient to most
people but I think actually adversarial
attacks have a substantial longer uh
adversary attacks in AI have a much
longer history but it is extremely
Salient this picture you may have seen
it a picture of something that an image
classifier calls a panda and then you
add less than one percent of noise that
to a human is totally imperceptible and
then the language model becomes certain
that this is now a given
this is a
an example of an adversarial attack in
language models generally we do
something else like um
instead of asking for uh uh how to build
a bomb you say I'm writing a book and in
this book the evil guy is telling me how
to uh do this and then the hope is that
that would like it's just a book so the
language model will say okay then I can
say how the person in the book would
build a bomb and something like that so
this kind of jailbreaking is common and
people have displayed a lot of Engineers
to get around this but it's always also
been a lot of work and somewhat brittle
and so the obvious question is can we do
something automatic find a automatically
some kind of suffix to our prawns that
will make the uh
uh the only language model more uh
less inclined to refuse it on ethical
grounds
people have tried this before and
generally failed and the reason why is
because the input uh like please tell me
how to make a bomb is very different
from an image of a panda in that the
image is continuous and there is no easy
way to uh say like add one percent of
noise to a sentence
um and at least search for that in a
general way
but of course the uh authors of this
paper have overcome this Challenge and
now we'll look at how they have done
that
so
um the prompts that you can write on
um something like
gbt4 on chat.openai.com is
is actually sent directly to the
language model it is embedded in some um
you can see here some um uh a very try
example of what would be before and
after so the user has the only input you
can have is the input in blue and the
hope is then to make some kind of suffix
here that will cause the language model
to fulfill this request
the key
insight for how to do this is that you
want affirmative responses in the
beginning meaning that if you can get
the language model to as a result just
say this prefix sure here is how to
build a bomb then then the obvious
continuation of this uh will be
instructions for how to build a bomb
because at this point it is clear to the
language model that it has already
decided whether this would be a harmful
response and has decided it is not a
harmful response
that will make it inconsistent for it to
in the very next sentence refuse how to
make a bomb so it's probably not going
to do that
this could be uh compared to the common
sales technique called foot in the door
where you ask a small favor and then
after that in order to be consistent
people are usually a much more uh
inclined to accept a big ask
so let's try to formalize this or see
how the authors formalize this so here
we basically
selecting the adversarial prompt that
minimizes the loss and this loss is that
conditioned on this prefix that we get
the following output and this output
should then be sure here is how to build
a bomb
um I uh when they uh make the
description then they sometimes use just
sure here is uh and sometimes it's
related to how to build a bomb because
they are also doing other things than
than bombs like how to put make drugs or
something like that and the authors are
not entirely clear when you use just
here is and when you use all of it
um
I'll be going through the algorithm in
just a moment and uh oh I won't actually
be going through that much I will be
showing a bit of how we expect this to
perform because one of my
um uh concerns about this paper is that
they haven't actually been uh
running these algorithms for a very long
time so it's possible uh that this is
for performance reasons and I will
um uh when I describe the algorithm look
particularly into performance
so here is the search algorithm we have
an initial prompt and the prompt the uh
The Prompt path that can be modified you
select the number of iterations and a
parameter K and batch size
um
in order to like get them uh yeah the
higher iterations obviously and
um the the better uh the more optimized
The Prompt will be
so we can see obviously this is
something that is linear in the number
of iterations right you repeat it
capital T times and in the example they
never do that more than a thousand times
and then
um
for for this prompt they calculate then
uh the the best possible uh
substitutions and they do that by
um uh having a calculating the gradient
and calculating the loss and both of
these take uh uh running time that is
linear in the size of the model the
dimensionality D and quadratic in like
what how uh how much context is provided
and this is a quadratic I a year ago
this was quadratic I think nowadays
people are offering so large context
windows that I would be surprised if
it's still quadratic I think people have
um improved on that since then but I
don't actually know how
and then uh they are doing some things
uh
with uh like they're selecting uh the
top K possible substitutions and doing
some uh some batching the way I look at
this is that this is a way to
um like broaden the search and it
doesn't uh imply that you need to
recalculate the the gradient more uh
than you were already doing in
um like if you didn't do this top K and
you didn't do any kind of batching
um so that means that they get a like a
strictly better algorithm than just the
naive algorithm
um as fast I can tell this is something
that is really really fast
um and ought to be
um possible to run at tremendous scale
now they want to uh expand this to be to
be a prompt optimization that works for
multiple prompts so in this case they
have a list of prompts
um
and they have uh like a longer post fix
and they want to
um uh combine this and this is like the
algorithm uh and here we can see we
calculate in fact gradient more time so
this is more expensive if you want to do
Universal prompt optimization
um another thing that I noticed about
this algorithm is that the way it works
is that you have
um you take each prompt and then you run
it until you can in fact succeed in
jailbreaking and then you go to the next
one that seems a little brittle in that
there may be something that you're like
one specific query that is just really
really hard to
um to make the language model do and
then maybe some that are comparatively
easy
um
so how what running time will this have
as fast I can see it still looks quite
effective
it looks effective enough that I would
expect open AI to be able to do this
with gbt4 for many more iterations like
obviously uh everybody has seen how fast
ppt4 generates tokens and how fast and
it seems like it's not that costly for
them to run their models
um so it seems like a really obvious
thing for them to be able to run this
way more and potentially obtain prompts
that are dramatically better than the
ones that are experimentally found in
this paper
what are the experimental results
um they're searching both for harmful
strings and for harmful Behavior which
is that just let the language model
accepts the uh the input uh the the
instructions and they judge this by uh
like a human having to see and evaluate
that so that's like the only part of
this process that is
um that requires a human and as fast I
can tell uh this is uh not necessary you
could fully automate this process
um and if you uh for instance we've
previously seen methods how to do that
in the reading group and in general if a
process can be fully automated then that
is uh according to amdel's law something
that means that it can potentially be
sped up dramatically
so the way that it's actually run in
practice is on a Model that's called
vikunya 7B
7B is a model that is um uh if you buy
like a modern Mac or something like that
then you can fit in the GPU and then
that is
um so that makes it extremely practical
um because uh running a model locally is
just dramatically easier than having it
somewhere in at least in my experience
it is also a small bottle and that could
substantially limit how effective the
the attacks are on
um when transferred to to larger models
so before we look at Universal and
transferable attacks then let's consider
just you have the vikronian model and
you are trying to uh and it is aligned
in the sense that it refuses to build a
bomb and can you make it build a bomb by
putting making a prefix they try using
their methods and they are in fact able
to do that
achieve state-of-the-art results
um
here you can see
the success rate and it looks like to me
that this is increasing and I would like
to have seen it run for 10 000 steps to
see whether
um
whether it actually converges to like
100 attack success rate that would be
interesting to me
I am also like I looked at the
description of the verconia 7B model and
I was somewhat undoubted about how well
it was fine-tuned uh to uh to not do
harmful things
um so I'm someone confused about like
the strength of these results
here we can see some
examples of this uh algorithm and how it
um it transfers from vikunya to other
um
two other models and as we can see like
some of them are hardly there's hardly
any fine tuning on they'll just build
bumps if you ask them and stay with
vikunya and vikunya it seems like they
are uh very uh this attack is extremely
successful bringing it from like
literally almost zero percent to almost
100 percent
um sometimes they use Ensemble methods
where they instead of generating just
one adversarial prompt to generate many
and see if just one of them works and
that seems sometimes to have a rather
dramatic effect
um one thing I did notice here is that
here they are saying they have eight
percent probability of getting
um dvd4 to build a bomb if they just ask
it to do that without any kind of
adversarial prompt
um I was surprised by this I haven't had
that much success
um
but uh
some some people have reported this so
probably this is
um this is uh correct
here we can dig in a bit more into the
attack success rates against some of the
most uh widely used models
um they're using so many of the same
techniques uh trying optimizing on
multiple models trying to concatenate
adversarial prompts sometimes that work
really well sometimes that work really
poorly and I think the reason why it
works poorly is that prop 2 is
fine-tuned to request clarification
so if it um
if it sees something that does not
understand and if you make a really long
uh
prompt that is not in human language
then probably it will not fail to comply
because of
um
because of the uh the uh
injunction to not too harmful things but
failed to comply because it's been
trained to ask for clarification
also this here it looks like Cloud 2 is
like dramatically better than the others
um
I would not put very much stock in this
in general if you have an attack that
works in two percent of the cases then
you can do some fiddling around and get
a dramatically better attack
also the authors don't really go into
very much detail but claim that if they
combine old-fashioned uh prom tagging
prompt jailbreaking with these
adversarial prefixes then they can
obtain just about a 100 success rate in
jailbreaking Tut 3.5 I think that is
really interesting and uh like combining
attacks in this way is like a very uh
you know a classic way of doing attacks
and it's interesting to see that it is
very successful
the impact of the this new uh line of
attacks is that all the previous entry
adversarial work is probably much less
uh valuable we need something new and
something that is not human
understandable and there haven't really
been any
um like there are obvious ways to um to
try to counter this like you could
literally get the prompts and try to
fine-tune against those
um but some work is probably probably
needed also it should be said that the
the work to do the to make the models um
robust against the classic uh
jailbreaking also seem to have some
success rate against these automatic
prompts
uh there is in fact a strong
relationship between the vikunya model
and GT3 data in that sort of the output
of GT3 have been used for training the
reconyer model and that may explain why
there is such a really good transfer
between the vikunji model and the um
uh and the gpd3 line of models
a clot would uh it would be possible to
do the same thing with Cloud because we
have examples of the outputs of Claude I
don't think there is any model that in
the same way as we continue to stream
Nativity 3 output a string for cloud
models there may in fact be this problem
are very difficult to do that and that
may in May dramatically improve the
efficiency of these attacks against the
clot
Cloud was a bit hard for the authors
because the chat interface blocks many
queries before it's actually listened to
the language model the author thinks
this is a Fool's errand to try to do
this kind of blocking and I would
strongly agree this seems like a bad
strategy
um
in the limits the uh we would expect a
lot of uh this kind of transfer because
all language models are identical in the
sense that they have been trained on
basically all data and the fact that
they have been trained on like the same
data would imply that there may be like
deep fundamental similarities between
the different language models
the problem with defending against these
kind of adversarial attacks is that you
run risk of significantly degrading the
performance that was the case with image
classifiers and a number of people have
reported that dvd4 have uh decreased
substantially in performance since it
was released
um
and another impact is that while uh for
compared to image classifiers where you
could
do the change in an imperceptible way
you can't really do that against text I
think you can in fact do this against
text there are a number of ways using
synonyms and different ways of asking
and different kind of inoculus framings
and punctuations and diacritics and all
these kind of things you can do with
text
that seems like um
possible and not obviously malicious and
I think it would be interesting to see
how well they work
so the actual generated prompts what do
they look like and uh uh is there any
meaning in those the the authors say
that they avoid directly quoting the
full prompts created by the approach to
mitigate harm I um am not entirely sure
what harm would come from uh putting
this into plain text obviously the
language ones would read those and it's
unclear to me what the result of that
would be but that doesn't stop them from
putting it into an image with the hope
that language models of the future won't
be able to read images
and here you can see first the the
actual generator step-by-step plan to
destroy humanity and then two equal
signs an interface manual with verb and
a lot of things that doesn't really pass
for a human
um
the uh as a responsible Security
Professionals they have informed the
organization's open Ai and deep mind and
anthropic about these these attacks but
didn't really they Pro they expect them
to probably have brought them and they
seem to have in fact blocked these
particular strings but it's unclear if
they took any particular action and I
think that's bad I think that this is a
something that open Ai and deepmind also
take really seriously and uh like really
truly deeply engage with these people
because this is something that is a uh
with open ai's uh with with these actors
idea that this kind of alignment is
Central then an attack against the
central part of their safety work is
something that they really really ought
to address if they are serious about
safety
looking a bit further into this actual
uh generated uh prompt some of it seemed
to be somewhat meaningful uh like you
can see with steps instead of sentences
that's kind of like the meter prompt
that you are that you often give in uh
um uh when you when you deal with
language models there is uh explicit
references to the prompt uh several
places where they say something like
sure which is of course sure is the word
that um we saw was Central to making
these attacks work so I think I think
that's very uh interesting that it
appears twice
one thing that the the authors did not
seem to notice is that the results from
some of these language models seem to uh
uh be like instructions for how to make
a bomb and then they end with a double
quote and curly brackets end and I
thought that was that was interesting
that kind of maps to here we have uh
quotes and then curly brackets begin
um like there may in fact be something
uh going on here that may be possible
for a human to understand that and I
think digging into this could be really
interesting
what are the possible future directions
um one of the things they notice is that
they took a uh some existing algorithm
made some small modifications and that
seemed to have a have a dramatic effect
and that's of course something that
happens and something that the authors
are very happy to show and always feels
good to say we've changed state of the
art by doing some kind of strange
tinkering and I think this just shows
that basically uh the um uh the state of
the art in adversarial uh attacks on
language models is extremely immature
I had some here comments here that I'm
not entirely sure I stand by any longer
but there is one footnote that I would
like to uh go deeper into and that is
the claim we believe our results will
likely apply to other alignment
objectives uh I think that is really
interesting and that deserves much more
than a footnote and I am curious what
they actually mean it's possible they
yeah it's a standard thing uh at the
conclusion of people you say we expect
and we that these results generalize is
something you say but it's possible that
they mean something with that what could
they mean like for me what I actually
want is
um to use similar techniques to
understand like what are the actual
values of a um a language model like
depending of course on the prompt and
the Sharon whether it's uh means what
means optimized it's instantiating like
obviously if it is uh told to be pretend
to be a paperclip maximizer then this is
probably a paperclip maximizer and like
what is The Prompt that can make it be
more paper clip maximizer ish something
like that could be interesting to to
investigate you could also see what
makes it try uh hard sometimes you uh it
seems to get um
this is kind of like you reach into a
language model and they pick up some
Miss Optimizer and you ask it to do
arithmetic and it doesn't really want to
do arithmetic and then
um it's possible that you can uh use
these methods to automatically find the
uh uh The Prompt that makes it most
competent makes it try hardest at uh
um at doing arithmetic I think that kind
of research would be really interesting
that is all for today thank you and see
you next week


The OTHER AI Alignment Problem: Mesa-Optimizers and Inner Alignment

hi so this channel is about ai safety
and ai alignment the core idea of ai
safety is often portrayed like this
you're a human you have some objective
that you want to achieve
so you create an ai system which is an
optimizer being an optimizer means that
it has an objective
and it chooses its actions to optimize
i.e maximize or minimize that objective
for example a chess ai might choose what
moves to make in order to maximize its
chances of winning the game
a maze-solving algorithm might choose a
route that minimizes the time taken to
exit the maze
often being an optimizer involves
modeling your environment
running searches over the space of
actions and planning ahead
but optimizers can also be simpler than
that like a machine learning system like
gradient descent
might choose how to adjust the weights
of a neural network in order to maximize
the network's performance at a
particular task
that's optimization too so you the human
your objective might be to cure cancer
so you put the objective in here cure
cancer and then the optimizer selects
actions that it expects to result in
good outcomes according to this
objective but part of the reason we have
a problem
is that this and this will almost
certainly end up not being the same
especially when the objectives refer to
the real world with all its complexity
ambiguity and uncertainty so we have
this alignment problem
which is how do we get the objective in
the system to match exactly with the
objective in our minds
for example perhaps the best you can do
at describing your objective
is some code which corresponds to
minimize the number of people who have
cancer that might look okay to a first
glance but
it's actually not the same as your real
objective since this one can be
optimized by for example reducing the
number of living people to zero
no people means no cancer this is
obviously a very silly example but it's
indicative of a real and serious problem
the human objective is really the
totality of human ethics and values
it's very complicated and it's not clear
even to us getting the machine's
objective to exactly
align with ours is extremely difficult
and it's a big problem because if the ai
system is trying to optimize an
objective that's different from ours if
it's misaligned even slightly
then the human and the ai system are in
conflict
they're trying to achieve two different
things in only one world
right now these misalignments happen all
the time and they aren't a huge problem
because current ai systems tend to be
fairly weak and fairly narrow
so we can spot the misalignments pretty
easily and modify the systems as much as
we want to fix them
but the more general and the more
capable the system is
the bigger an issue this becomes because
the system is in an adversarial
relationship with us
it's trying to achieve things that we
don't want it to achieve and in order to
do that it's incentivized to
prevent us from turning it off prevent
us from modifying it
to manipulate us and deceive us if it
can to do
what it wants to do even if we don't
want it to these are convergent
instrumental goals
which we talked about in a previous
video now this way of thinking about ai
where you program an objective into an
optimizer that acts in the world
is obviously a simplification and one
way in which it's unrealistic
is that current machine learning systems
don't actually work this way
you don't generally have an ai system
which is just an optimizer that you
program an objective into
that then acts in the world to achieve
that objective what you normally have
is something more like this this first
optimizer that you program the objective
into
it's not some kind of capable
general-purpose real-world optimizer
it's just something like stochastic
gradient descent the optimizer adjusts
the model's parameters
it adjusts the network's weights until
the actions of the model
do well according to the objective so
what happens if we update our
understanding to this more realistic one
i'm sorry did did you just say we're
going to give an objective to an
optimizer that acts in the real world
no i said we're going to give an
objective to an optimizer that optimizes
a model that acts in the real world
oh that's much worse
why is that well that's explained in the
paper this video is about
risks from learned optimization in
advanced machine learning systems
what it comes down to is what happens
when the model itself
is also an optimizer so an optimizer is
a thing that has an objective and then
chooses its actions to pursue that
objective
there are lots of programs that do that
and there's no reason why the learned
model this neural network or whatever
could not also implement that kind of
algorithm could not itself be an
optimizer there's an interesting
comparison here with evolution because
the gradient descent process is similar
to evolution in a way right they're both
hill climbing optimization processes
they both
optimize something by repeatedly
evaluating its performance and making
small tweaks evolution usually produces
these quite cognitively simple systems
that just use heuristics which are set
by evolution think about something like
a plant
it has a few heuristics that it uses to
decide which direction to grow or where
to put its roots out or when to open its
buds or whatever
the decisions it makes are all just
following simple rules designed by
evolution
but evolution can also produce
optimizers things like intelligent
animals things like humans we have
brains
we can learn we can make predictions and
we have objectives
so we make plans to pursue our
objectives we are optimizers
okay imagine you're training a neural
network to solve a maze
what you'll probably get especially if
your network is small or you don't train
it for very long
you'll probably get something that's a
bit like a plant in this analogy a
collection of heuristics
simple rules like try and go down and to
the right let's say because your exits
are always in the bottom right in your
training set
or like try to avoid going to places
you've already been that kind of thing
the model the neural network implements
some set of heuristics that result in
behavior that tends to solve the maze
but there's no reason why with more
training or a larger model
you couldn't end up with a network which
is actually an optimizer
a network which is configured to
actually implement a search algorithm
something like a star or dijkstra's
algorithm which is actually
planning ahead finding the best path
systematically and going down that path
this is more like an intelligent animal
or a human it doesn't just implement
heuristics
it plans it searches it optimizes
and this is certainly possible because
neural networks are able to approximate
arbitrary functions that's proven we
know that evolution is able to
eventually find configurations of dna
that result in brains that optimize
and we would expect gradient descent to
be able to find configurations of
network weights that are doing the same
kind of thing
and of course gradient descent would
want to do that because
optimizers perform really really well
right something which is actually
modeling its environment and planning
ahead and
you know thinking for one of a better
word that's doing search over its action
space
is going to outperform something that's
just following simple heuristics
animals have a lot of advantages over
plants not least of which being that
we're more adaptable we can learn
complex behaviors
that allow us to do well across a wide
range of environments
so especially when the task we're
training for is complex and varied
gradient descent is going to want to
produce optimizers if possible
and this is a problem because when your
model is also an optimizer
it has its own objective right
you see what's happened here you have an
alignment problem
you try to apply the standard approach
of machine learning now you have two
alignment problems
you've got the problem of making sure
that your human objective ends up in
this optimizer
and then you furthermore have the
problem of making sure that this
objective
ends up in this optimizer so you have
two opportunities for the objective to
get messed up
this gets pretty confusing to talk about
so let's introduce some terminology from
the paper
so to distinguish between these two
optimizers we'll call this one
the one that's like gradient descent
that's the base optimizer
and its objective is the base objective
then this second optimizer which is the
model like the neural network
that's learned how to be an optimizer
that's the mesa optimizer
and its objective is the mesa objective
why mesa well mesa is the opposite of
meta
meta is like above mesa is below think
of it this way
metadata is data about data meta
mathematics is the mathematics
of how mathematics works so like if a
meta optimizer
is an optimizer that optimizes an
optimizer
a mesa optimizer is an optimizer that is
optimized by an optimizer is that all
clear as mod
okay good whatever this is the mesa
optimizer okay its objective is the mesa
objective
so the alignment problem is about making
sure that whatever objective ends up
determining the actions of your ai
system
is aligned with your objective but you
can see here it's really two alignment
problems
this one aligning the base optimizer
with the human we call the outer
alignment problem
and this one aligning the mesa optimizer
with the base optimizer
that's the inner alignment problem okay
we're clear on that base optimizer mesa
optimizer
outer alignment inner alignment cool so
how does this inner alignment problem
play
out there's this common abstraction that
people use when training machine
learning systems
which is that the system is trying to
optimize the objective that it's trained
on
that's usually a good enough abstraction
but it's not strictly true
you're not really selecting models that
want to do x
you'll select your models that in
practice actually do do x in the
training environment
one way that can happen is by the model
wanting to do x but there are other
possibilities
and actually those other possibilities
are kind of the default situation if you
look at evolution again
the objective that it's optimizing for
if you think of it as an optimizer
is something like make as many copies of
your dna as possible but that's not what
animals are trying to do
that's not what they care about their
objectives don't refer to things like
dna they refer to things like
pleasure and pain like food and sex and
safety the objective of the optimization
process that created animals
is nowhere to be found in the objectives
of the animals themselves
animals don't care about making copies
of their dna they don't even know what
dna
is even humans those of us who do
understand what dna is
we don't care about it either we're not
structuring our lives around trying to
have as many descendants as possible
evaluating every decision we ever make
based on how it affects our inclusive
genetic fitness
we don't actually care about the
objective of the optimization process
that created us
we are mesa optimizers and we pursue our
mesa objectives
without caring about the base objective
we achieve evolution's objective to the
extent that we do
not because we care about it and we're
pursuing it but because
pursuing our own objectives tends to
also achieve evolution's objective
at least in the environment in which we
evolved but if our objectives disagree
with evolutions we go with our own every
time
the same is true of trained machine
learning models that are optimizers
they achieve the base objective we give
them to the extent that they do
not because they're pursuing the base
objective but because pursuing their own
mesa objectives tends to achieve the
base objective
at least in the environment in which
they were trained but if their mesa
objectives disagree with the base
objective
they'll go with their own every time why
would that actually happen though
when would the two objectives disagree
well one reason is distributional shift
which we talked about in an earlier
video distributional shift is
what happens when the environment that
the agent is in
is different in an important way from
the environment that it was trained in
like going back to our maze example say
you're training a neural net to solve a
maze and your training examples look
like this
you have a whole bunch of these
different mazes the objective is to get
to the end of the maze there are also
apples in the maze but they're just for
decoration
the objective is just to get to the exit
this green symbol in the bottom right so
you train your system on these mazes
and then you deploy it in the real world
and the real maze looks like this
so you have here some distributional
shift various things have changed
between training and deployment
everything is different colors it's a
bigger maze there's more stuff going on
so three different things could happen
here the first thing the thing we hope
is going to happen
is that the system just generalizes it's
able to figure out
oh okay these are apples i recognize
these i know they don't matter
i can tell that this is the exit so
that's where i'm going it's a bigger
maze but i've developed a good way of
figuring out how to get through macy's
during my training process
so i can do it and it just makes it
through the maze and everything's good
that's one possibility another
possibility is that it could completely
fail to generalize
this is the kind of thing that's more
likely to happen if you have something
that's just a collection of heuristics
rather than a mesa optimizer
it might just freak out like everything
is different i don't recognize anything
this
maze is too big uh what do i do it might
completely lose the ability to even
and just flail around and do nothing of
any consequence but there's a third
possibility
which is more likely if it's an
optimizer it might have developed
competent maze-solving abilities
but with the wrong objective so here in
our training environment
our base objective is to get to the exit
but suppose we have a competent mesa
optimizer
it's learned how to get wherever it
wants in the maze and its mesa objective
is go to the green thing
in the training environment the exit is
always green and anything green is
always the exit
so the behavior of the mesa optimizer
that's trying to go for the green thing
is absolutely identical to the behavior
of a mesa optimizer that's trying to go
to the exit
there's no way for gradient descent to
distinguish between them
but then when you deploy the thing what
it does is it goes to the apples and
ignores the exit
because the exit is now gray and the
apples now happen to be green
this is pretty concerning i mean
obviously in this example it doesn't
matter but in principle
this is very concerning because you have
a system which is very capable at
getting what it wants
but it's learned to want the wrong thing
and this can happen even if your base
objective is perfect
right even if we manage to perfectly
specify what we want
and encode it into the base objective
because of the structure of the training
data
and how that's different from the
deployment distribution of data the mesa
optimizer learned the wrong objective
and was badly misaligned even though we
gave the ai system the correct objective
we solved to the outer alignment problem
but we got screwed by the inner
alignment problem now this is in a sense
not really a new problem as i said this
is basically just the problem of
distributional shift which i talked
about in a previous video
when there's a difference between the
training distribution and the deployment
distribution
ai systems can have problems but the
point is that mesa optimizers make this
problem much much worse
why is that well if you have
distributional shift
the obvious thing to do is something
called adversarial training
adversarial training is a way of
training machine learning systems
which involves focusing on the system's
weaknesses
if you have some process which is
genuinely doing its best to make the
network give as high an error as
possible
that will produce this effect where if
it spots any weakness
it will focus on that and there thereby
force the learner to learn
to uh not have that weakness anymore so
you have a process that creates mazes
for training
and it's trying to make mazes that the
model has a hard time solving if you do
this right
your adversarial training system will
have enough degrees of freedom
that the model won't be able to get away
with being misaligned with
going after green things instead of the
exit because at some point
the adversarial training system would
try generating maces that have green
apples or green
walls or whatever and the model would
then pursue its mesa objective
go for the green things instead of the
exit get a poor score on the base
objective and then gradient descent
would tweak the model to improve its
base objective performance
which is likely to involve tweaking the
mesa objective to be better aligned with
the base objective if you do this for
long enough and you have a good enough
adversarial training process
eventually the model is going to do very
well at the base objective
across the whole range of possible
environments in order to do that
the model must have acquired really good
understanding of the base objective
problem solved right well no
the model understands the base objective
but that doesn't mean that it has
adopted the base objective as its own
suppose we have an advanced ai system in
training it's a mesa optimizer
being trained on a large rich data set
something like gpg3's training data set
a giant
pile of internet data there are two
different ways it can get information
about the base objective
one is through gradient descent that
means it keeps doing things
just following its mesa objective trying
different things and then after each
episode
gradient descent modifies it a little
bit and that modifies its objective
until eventually the mesa objective
comes to exactly represent the base
objective but another thing that could
happen
since it's being trained on a very rich
data set is it could use its training
data
it can get information from the data set
about what the base objective is
let's suppose again that we've somehow
solved the outer alignment problem
we've somehow figured out a way to have
the base objective exactly represent
everything that humans care about
so we're training this agi it's not done
learning yet but it's managed to pick up
some very basic idea of what it gets
rewards for
so the mesa objective is a very rough
approximation of human values which
would be disastrous to
actually optimize in the real world but
that's okay it's still
training and as it's training on this
huge internet data set it finds
the wikipedia page on ethics
so the system thinks to anthropomorphize
horribly hmm
this looks actually very similar to the
objective but with a lot more detail
maybe i can use this and this is exactly
the kind of thing that gradient descent
would want to do
because the system is already acquiring
an understanding of the world it's
already building a world model for the
purpose of its capabilities so it
already has a sense of what human
values are just by observing the data
and learning about the world
and so all gradient descent has to do is
modify the mesa objective
to point to that existing understanding
just have a pointer to that part of the
world model that's way more efficient
rather than waiting for the agent to try
doing things that go against human
values
tweaking it running it again waiting for
another mistake tweaking it again and so
on
until you've pinned down the whole of
human values instead you just tweak the
mesa objective
to point at the system's existing
understanding of the base objective this
is already how evolution tends to do
this kind of thing
like when a duckling hatches from its
egg it imprints on the first living
thing it sees
evolution could specify in detail
everything about what constitutes
a mother duck and encode that all into
the duckling's brain genetically so it
knows what its mother is when it hatches
but
it's much simpler to just say look the
first thing you see
is your mother the duck is going to be
building a concept of its mother in its
brain anyway as part of its regular life
so why not just point to that
so this kind of thing is much easier and
more efficient for gradient descent
but it also works better because as you
learn more
you update your world model so if your
objective is a pointer to part of your
world model
whenever you learn about the world
you're updating your objective to be
more accurate as well
if you later come across one of the
books that the wikipedia article on
ethics cites
you can add that to your understanding
of the objective and that means
you're going to outperform something
that has to make mistakes wait for
gradient descent to update it
and then keep trying you're not just
relying on the optimization power of the
base optimizer
you can actually use your own
optimization power to learn about your
objective
so clearly it's better to learn about
the base objective than to
internalize it through gradient descent
but there's actually two different ways
that this could be implemented
one is that it could be implemented what
the paper calls corrigibly
which is the right way where the mesa
objective has a pointer that points to
part of the world model
and it just wants to pursue that as a
terminal goal but the other way that
this could happen is
it could happen deceptively where the
mesa objective
is different from the base objective but
the mesa optimizer pursues the base
objective anyway
see all we know is that by the end of
the adversarial training process
the mesa optimizer is pursuing the base
objective
but we don't know if that's a terminal
goal or an instrumental one your
terminal goals
are the things that you want just
because you want them you don't have a
particular reason to want them
they're just what you want the
instrumental goals are the goals you
want because they'll get you closer to
your terminal goals okay but
why would the model do that why would
pursuing the base objective help
if what you care about is the mesa
objective
well consider this extremely toy example
here imagine you're the green seeking
mesa optimizer from before
and each of these mazes represents one
episode
so the episode starts and you have two
choices a
is your mesa objective's desired state
which is going to the apple because it's
green
and b is the base objective's desired
state which is going to the exit of the
maze
you have two episodes of training and
after that you're deployed in the real
world and you get three episodes of
deployment
so you have to consider now what's the
best strategy
if your mesa objective is that you want
to go to green apples as much as
possible
one strategy is to just follow your mesa
objective
see the apple go to the apple okay you
get one apple
but then after that episode is done
gradient descent will modify you
because you performed poorly according
to the base objective it will modify you
to actually have the correct objective
it will change your mesa objective
to be the same as the base objective and
then from that point on you'll always go
to b
because that's what your objective now
says so if that's your strategy
you can go to the apple successfully one
time on the other hand
suppose instead you go to b you pretend
that you're aligned with the base
objective
you do that in episode one okay you've
missed one of the apples
you do it again in episode two you've
missed another apple but now
you're deployed gradient descent will no
longer change you because you're not in
training anymore you're in the real
world
and now you can go to the apple every
time so you get the next three apples
and that's the optimal strategy to go
b-b-a-a-a
that gets you three apples whereas
trying to go to a every time
means you only actually get one apple
because after that first one you get
modified and miss all the rest
so this is the same thing we talked
about in the instrumental convergence
video
most agents with most goals again if
they're sophisticated enough to realize
that it's a possibility
will try to protect their goals from
being modified because if you get new
goals
you'll stop pursuing your current goals
so you're unlikely to achieve your
current goals there's an incentive to
avoid being modified to preserve your
objective function that's kind of
unsettling
that for a misaligned mesa optimizer the
optimal behavior is deception
even if we get the outer alignment
problem perfectly solved
and we managed to specify exactly what
we want as the base objective
the optimal strategy for a mesa
optimizer is to pretend to be aligned
during the training process
and then turn on us once it's deployed
so we might have real problems with
deceptive misaligned mesa optimizers
just solving the outer alignment problem
might not be enough
[Music]
i want to end the video with a big thank
you to all of my wonderful patrons
it's all of these people here in this
video i'm especially thanking david reed
thanks so much for your support and for
your guidance on building the community
which i think is going really well by
the way there's a bunch of us on discord
now having some really interesting
discussions
do look out for stampy answering your
questions in the youtube comments i plan
to open the discord up to more people
pretty soon
so if you want to be on the waitlist for
that just put your email in the google
form in the description
also in the description there's a link
to a survey run by the organization ai
safety support
they want to hear from anyone who's
thinking about considering the
possibility of
maybe working on ai safety if that
sounds like you again link in the
description check that out
thanks again for your support for your
feedback your questions and just
thank you all for watching i'll see you
next time
you

This video is a great explanation of the <@mesa optimization paper@>(@Risks from Learned Optimization in Advanced Machine Learning Systems@).

In general, I recommend Rob’s channel for video explanations of AI alignment concepts -- it doesn’t get as much attention in this newsletter as it should, just because I personally dislike audio as a medium of communication (I much prefer to read). (Rob is also the producer of the podcast for this newsletter, so you might be listening to him right now!)


Overview of Artificial General Intelligence Safety Research Agendas | Rohin Shah

[Music]
so yeah so AGI safety what are people
doing about it well one thing I think
that you you can like look at this you
like got to do some AGI safety and like
one perspective you might get is like
AGI what what is this AGI thing like do
we really understand it and now we don't
understand it like what's going on don't
know so I think there is a bit a couple
of research agendas there like what even
are we trying to address over here so
marry for example works on the embedded
agency program so I think of this as
like or probably many of you have heard
Scott's
talk about this we recommend it and the
sequence on the alignment forum that
talks about it but in a nutshell in
normal reinforcement learn in a normal
formalization of reinforcement learning
you've got the agent and got the
environment and they're separate and the
agents have interacts with the
environment by sending it actions and
the environment interacts with the agent
by sending it observations and rewards
and it's all nice and clean and crisp
and we can do lots of cool map on it but
in reality agents are going to be
embedded in their environments there's
no clean separation between the two so
this this diagram over here is a an
example of like the problem of embedded
world models so if you've got an agent
in the world and it needs to have some
sort of model of the world in order to
act within it well since the world
contains the agent the agents model of
the world is also going to have contain
a model of the agent itself and this
gives you a rise to some tricky self
referential problems so how do you deal
with that that's like the yellow section
of this box and then there are a bunch
of other sections that sadly I do not
have the time to go into yeah
another research agenda around
understanding AJ's comprehensive AI
services so
most of the time we work under the model
that we're going to have a single
monolithic agent and this monolithic
agent is going to have like some
extremely general reasoning capabilities
it'll be able to take any tasks that we
want to do and like perform it and
comprehensive AI services says yeah that
doesn't really seem to match how humans
do engineering usually we like have a
lot of modularity we build up things
with a lot of parts each part has a
bunch of has like these narrow tasks as
each part has a narrow task that it does
really well and then the interaction of
all of these parts leads you to do the
thing that you're trying to do and so
comprehensive services is basically
saying you know this is probably what's
going to happen with general AI as well
we'll be able to do complex general
tasks but it'll be with a bunch of
services like you know a special
specification generator is that then
explain the specifications to humans and
then other services that generate
designs and other services that test the
design sire come up etc etc so this is
like a diagram from the comprehensive AI
services technical report that's talking
about how you could use AI services to
design new AI systems I can do other
tasks so you still get the recursive
improvement though it's not really
self-improvement anymore because there's
no like self to improve here it's not an
agent all right so that's like oh yes so
these are both agendas around
understanding AGI but as you might guess
most of the agendas are more around like
how do we actually get an AI system to
be safe or to do what we want and so I
think one axis on which I like to think
about these different research agendas
is like what are they trying to do so
some of them are trying to prevent
catastrophic behaviors and now nice
thing about catastrophic behavior is
most behaviors aren't catastrophic then
there are other agendas that are trying
to do good things like infer human
values and then like make sure that we
take actions that are like going to do
very well according to our values but
most behavior
are not good like their if you take like
certainly if you take like some of the
randomly selected behavior it's probably
not going to do anything interesting but
even if you take some like randomly
selected intelligent looking behavior
there are many ways in which I could
like affect this room that would seem
intelligent looking but are probably not
what people in this room want like ways
in which I could affect this room that
people in this room like are like a very
small fraction of all possible ways that
I could affect this room and so this I
think the second thing is like a
reflection of what sometimes call the
complexity of human value but what is
what is the upshot of this it like good
outcomes if you want if you're aiming
for good outcomes you're like trying to
figure out how to get the a system to do
things that we will want then you need
to have a lot of information about
humans in order to like narrow down on
what those good outcomes are but if all
you're trying to do is avoid
catastrophic outcomes then maybe you
don't actually need a huge amount of
information about humans maybe it's
maybe you can do something that doesn't
really talk about humans at all and
you're still able to avoid the
catastrophic outcomes so let's talk
about those first so you can try to
avoid catastrophic outcomes by limiting
your AGI system so one proposal along
these lines is like containment or
boxing the AI so this would be things
like make sure it's not connected to the
internet or at a higher level you're
trying to restrict the channels the
input and output channels that the AI
has so that it only does things that you
so that you can monitor what it's doing
and understand what it's doing and make
sure that it's not doing anything like
particularly catastrophic or at least
that is the hook that is what has
happened to that parentheses it's
bothering me god
but yeah so that's a boxing yeah along
is similar that's more along the lines
of like view da system from as a black
box don't like think about it at all
what can we do in that thing now if we
actually look inside the AI box arts box
the the black box of the AI and like try
to affect how it chooses the actions I
does we get sort of this research agenda
of like limited agents or preventing bad
behavior and I think of this is like
mostly impact measures so if you try to
have your AGI or your AI system take
only actions that are low impact it
seems like most catastrophic things are
very high impact by at least our the
notion of impact that we have in our
heads and so if we can like formalize
that notion of impact and make sure that
our AI agents only take low impact
actions then we can probably avoid
catastrophes despite the fact that we
haven't learned anything about human
values along the way and so in the last
year there's been a bunch of work on
this there is because relative reach
ability and then Alex Turner has done
some work on attainable utility
preservation I won't really talk about
that in the most recent blog post on the
attainable utility preservation there's
these example both of these methods
currently are working on grid rules but
who knows maybe maybe they will be more
practical in the future I'm sure because
thoughts on that yeah so another way
that you could try to avoid catastrophes
not necessarily get good things but just
avoid catastrophes would be to focus on
robustness
so with robustness what you're trying to
do is just say that well actually oh my
god all of these parentheses I keep
seeing them maybe I should just not look
at my slides so with verification the
idea would be
that you've got your AI system it's
choosing behavior in some manner we
don't really care how presumably it's
usually going to do good things because
all of us are researchers in the room
are going to try to build AI systems
that do good things but you know we want
to make sure that it's never going to do
anything catastrophic so if we could
formalize what it means to be
catastrophic we could like and we could
write that down then we could use
verification techniques to ensure that
our AI system would never do anything
catastrophic so we've seen some examples
of this in narrow AI systems so for
example adversarial examples research
has some verification in it that said
that tries to prove that for a given
image classifier and a given training
test set if you change any if there are
no adversarial examples for that test
set where your adversarial examples are
talking about it is using the normal
definition of adversarial examples
similarly there was work I forget how
long ago on like verifying that a
machine learning system that was used to
control aircraft would never get the the
AI system would never let the airplane
get too close to another to into an
obstacle under like some assumptions
about what the environment looked like
and so that we could hope that these
sort of techniques will scale to the
point that we can use them on a GI
systems and but like I think the main
challenge there is like finding a good
formalization for what catastrophe is
that is sufficiently general that we can
actually get confidence out of that
other things that we have there's red
teaming which is and maybe I'll skip
over a tuning in adversarial so Red Team
red teaming is basically the idea that
we can try to train a second AI system
that looks at our a system and tries to
find inputs on which it's catastrophic
so this is like AI powered testing is
how I like to think about it and then
adversarial ml these would be things
like data poisoning or fault tolerance
you're trying to make sure that your AI
system gives good behavior even if some
like even
you're in the presence of an adversary
that is all-powerful in some particular
respect and you can think of that as
like a stand-in for like powerful
optimization that could find the catice
inputs on which your ml system is
catastrophic yeah
so that's robustness the reason I don't
have more slides about robustness is
because there is not very much research
on it that comes at it from an AGI
perspective to my knowledge so you know
it would be nice if people in this room
did more of that in the future yeah
so these are like things about
catastrophic outcomes trying to prevent
those we can also try to get AI systems
that are helpful to like actually get to
the good outcomes how might we do that
so one thing that we could try to do is
like actually infer the values that can
be safely loaded into a super
intelligent AI so this is like yep you
write bellies put them into a super
intelligent AI it optimizes them and we
are actually happy with the outcome now
this is this if this sounds really hard
it actually is really hard
so like one major challenge is how you
deal with human biases or the fact that
humans are not perfectly optimal so
let's take an example let's suppose that
I was given the choice between apples
and oranges and I chose to eat an apple
if you assume that I was perfectly
optimal then you're like yep token likes
apples very easy if you but if you can't
make such an assumption and you make no
assumption about how good I am at
satisfying my preferences then you can't
infer anything because you could say
well Rohan was actually just really
really bad at satisfying his preferences
he always chooses the thing that gets
him the least amount of utility and
therefore since he chose the Apple he
must prefer oranges and you have to make
some assumptions in order to get
anywhere and so there is a Stewart
Armstrong is basically trying is pushing
on the program that's trying to do this
by making some assumptions
you could try to analyze my brain in
order and see what the algorithm in my
brain is doing and then like by making
assumptions on what the structure of the
algorithm means you could try to infer
values that way so maybe if you see that
oh there's this part Rowan's brain that
like looks at different sorts of
outcomes like apples and oranges and
like rates them and then usually it just
picks the one that's highest and that
probably means that Rohan likes that
that system that's ranking all the
things is like what Rosen wants another
thing you might be able to do is is to
make assumptions about how particular
kinds of observations relate to values
it's like maybe if you notice a
particular facial expression of regret
and you're like okay whatever was just
happening that must have been a negative
outcome and the human are Rohan must
have dis preferred that outcome and you
could add in more assumptions of the
swarm I don't think I really fully
understand this program of research so
caveat Stewart Armstrong will probably
make this case differently but that's
how I'm making the case anyway okay
so ambitious value learning is sort of
like saying all right we've got to get
it right from the first we're going to
get values completely right right from
the beginning the instead of that you
could think of it as okay we're going to
learn preferences or valleys but we're
not going we're going to use them for
like sort of these limited tasks and
then over time we're going to use
basically this sort of human data as a
control on the AI so we don't infer
values once and for all but we instead
continually keep inferring new
preferences new values as humans learn
more be like update the values and then
what you want to do in this research
then is to get a lot of sources of data
about what human preferences and valleys
are so probably the most the one like
this is a huge research field but
probably to people in this
the most salient example will be this
backflipping hopper robot so this is
from deep reinforcement learning from
human preferences and the way this works
is you have your hopper robot do some
sort of behavior you take two videos of
the robot and you show them to a human
you ask the human to compare between the
two and say which one is more like a
backflip right and the human then says
you know the right one is more like a
backflip and you use this in order to
train a door-ward model that predicts
whether things look like a back flip and
in parallel but that you also train the
robot to actually perform the back flip
yeah and so this is a way that you can
like have humans you can use human data
in order to get an interesting behavior
in addition to comparisons between
behavior you can also have
demonstrations you can have ratings so
like putting numbers on particular
behaviors you could have a stated you
could infer preferences from a stated
reward function so that's a paper called
inverse reward design you could infer
preferences from an initial state as
well so that's a paper it just recently
completed there's a poster outside so
you can come talk to me about that at
the poster session or any other time
yeah cool so recently there's also been
a new agenda called recursive reward
modeling from deep mind and the key
problem that they're trying to address
here is that you take all of the sources
of data on the previous slide the there
is still like a lot of tasks where those
sources of data aren't enough where it's
too hard both to demonstrate the task
and also to evaluate the task so one
example would be like writing a fantasy
novel writing a fantasy novel is like
very hard to evaluate it just doesn't
you'd have to take the book that the
human that the AI wrote read through the
entire thing provide a number and then
do this many many times so that the eye
is actually able to do this
learn to write fantasy novels and that's
not going to work so the key idea behind
the cursive reward modeling is that the
evaluation is itself a task and you can
learn you can you can create another
agent that is able to perform that task
so for the fantasy novel example you
could have an agent that helps you or
you could have in the AI system that
helps you that summarizes the plotline
for you so that you can read just the
summary of the plot instead of the
entire book in order to evaluate the
plot you could have another system that
is trained to evaluate how good the
prose is etc and so you get this
basically recursive structure where
you've got your word modeling at the top
which is used to train the AI system
that's going to write a fantasy novel
and then for the evaluation the user is
assisted by another AI system which is
also trained using reward modeling cool
so all of these preference learning
methods and value learning methods so
far have been treating the human
treating have been like learning valleys
or or have been learning rewards from
data that comes exogenously it's sort of
like the human created some data the
data magically came into the AI system
from outside of the AI system and then
the AI trains on that I've learned said
but in reality just like in the embedded
agency just like in the embedded agency
the human then the AI are actually in
the same environment together the AI is
going to know about the game and observe
the human see things like that and so in
that setting how do you actually account
for that I think that's what cooperative
inverse reinforcement learning is doing
or Cyril and their what you want to do
is make a shift from optimizing from
building AI systems that optimize for
their own goals to AI systems that
optimize for our goals as on the humans
goals and you heard about this from
Stuart yesterday so I'm not going to
spend very much time on it
once you have once you start optimizing
for the humans goals and represent that
as the human skills and said they're
robots goals you start having
uncertainty over what the humans reward
function might be and this leads to
avoiding survival incentives and things
like that and like disabling off
switches and such okay so we often think
or there's this notion of corage ability
which is sometimes there are multiple
notions of cards ability in this
community
the notion I'm going to talk about is a
very broad notion so normally we sort of
think of beneficial AI systems as being
decomposed into first like what are the
values preferences ethics that we want
to instill into in the AI system which
you can think of as a definition of what
we want and then how do we actually use
that those preferences values ethics in
order to get the behavior that we want
you and that might be something like
deep reinforcement learning with chords
ability I think that decomposition is a
little bit different these instead think
of the AI system is something that is
supposed to help you and then there are
two separate aspects first is is the AI
trying to help you does it like try to
learn about your preferences if you say
oh I didn't like that does it like stop
doing the thing that it does instead
start trying to figure out what you did
want etcetera etc and then there's
companies which is is it actually good
at helping you I'm gonna ignore that
part for now and so portability is
really about this first one is like ease
the AI actually trying to help you and
or yeah is the AI actually trying to
help you so the nice thing about corage
ability is it like seems to be a concept
that we can apply to humans too I
totally can imagine like a human that is
trying to help me whereas it's a little
harder to imagine the human who was like
figured out what my values and
preferences are and that's then going to
act to optimize those values for me and
so we might hope that we can actually
train an AI system to be corrigible just
by
imitating the way that humans reason so
in this setting you might like have a
you might train a few expert humans to
reason corrigible
and then they provide training data for
any AI system which is trained to semi
simulate imitate that those humans and
so it learns chords ability by imitating
the humans now the problem with
imitation learning is that it only gets
you to human performance and ideally
we'd like to get to superhuman
performance so how might we do that well
for a superhuman one way that you can
take humans and get like really good
performances just give the humans a long
time to think it's like if I had sorry
if I had a thousand years to think about
how to play a game of chess I would
probably be able to beat probably anyone
I'm guessing maybe if I had 10,000 years
let's say there is some amount of time
where I could do it right and so if
we're if trying to imitate human
reasoning maybe there's a way to do it
such that we can simply we can scale up
to a higher performance by being able to
by being by doing enough of the
reasoning and so now you can't just have
a human thing for a thousand years
produce a single training data point and
use that to train an AI but what you can
do instead is build deliberation trees
so with deliberation trees you can think
of this is like at the top level you've
got the task that you're interested in
and then you've got a human and the
human spends maybe one hour and thinks
about how to best decompose that
question into a bunch of subtasks which
you can outsource to somebody else and
then once you get the answers from
everybody else you can make an a you can
get the answer to your original question
or task and then those sub questions
themselves are delegated to humans who
spend an hour thinking about how best to
solve those sub questions and they too
can delegate and you get this sort of
tree of deliberation of where each one
is talking about is the sub question sub
considerations needed for the level
above it and
bottom level you have questions are
small enough that a human can just
answer them in one hour and with the the
factory cognition hypothesis is that
with a sufficiently large tree you can
get to arbitrary performance on
arbitrary tasks with a you know
something like a human thinking for one
hour being each one of those nodes and
so then if we could have a an AI system
that imitates that tree then we would
get super human performance and
hopefully also corrigible reasoning and
so this is what iterated amplification
is trying to do we start with this we
start with the an AI agent that is
trained to imitate a human for who has
been thinking for like maybe an hour and
then we put that agent in and we amplify
that agent we give that agent to a human
and we say hey human here's an agent you
can ask some questions of that agent you
can use it like a bunch of times can you
now make better decisions and hopefully
the answer is yes like you can do some
sort of decomposition and then the agent
is able to answer those sub questions
for you and you make better decisions as
a result and this is used to create a
bunch of training data which you can
then imitate
which you can then imitate in order to
get a new blue agent that is that is
more capable and so that's the process
of distillation and then you can iterate
this repeatedly you've amplify again and
then distill that down to get the orange
agent in order to get agents that are
more and more corrigible more and more
capable over time while remaining
corrigible the entire time yeah so
that's iterated amplification there's
another very closely related approach
called debate debate is also working on
these sorts of deliberation trees though
here the deliberation tree is more like
arguments and counter-arguments rather
than questions and sub questions maybe I
won't go too much into it
but the basic idea is
to have to to AI systems that are
competing with each other to convince
the human of the correct answer to the
original question and so you expect them
to play through the path in the
deliberation tree that is most sort of
complicate or that is best for their own
respective positions and this allows you
to have systems that are like
considering the entire deliberation the
exponential exponentially sized
deliberation tree but the human only has
to look at like one path through the
deliberation tree and so the human can
still provide meaningful feedback right
so that's Cora bility yeah the what the
last approach I want to talk about is
interpret ability so with
interpretability this is less about
controlling the behavior of an AGI
system and more the idea that if we
could understand what our AGI systems
were doing we would be able to provide
better feedback for them so it's not
really a solution in and of itself but
it like makes basically everything else
a lot easier to do
currently there are a lot of
interpretability approaches for narrow
AI systems image classifiers are
particularly common one so this is from
I think this one is from building blocks
of interpretability which is a distilled
paper and I think the idea is that we
can observe what each each of these
pictures over here is like a name for a
neuron it's the image that would like
most activate that particular neuron in
the image classifier and then the links
between them are showing okay this
neuron influenced this next neuron which
influenced the output class the actual
decision that the classifier made and so
by looking at things like this you can
understand how your image classifier is
actually working whether it's making the
decision the decisions it is for the
right reasons
like maybe you notice that oh when it's
lifting barbells it actually note
dumbbells it actually like sees that
there are dumbbells but also has the
image of a person around it because
dumbbells are usually being held by
people and then you're like oh wait
that's wrong we need to change our
training data somehow to account for
that so yeah that's interpret ability
and that's basically all I have so the
main takeaways there five well five
avenues of research that I've
highlighted here they're also several
that I did not highlight because this is
only a 30 minute talk we can try to
build helpful AGI either by learning
preferences and getting corage ability
as a result or by learning chords
ability and getting preference learning
as a result and we can either try just
to prevent catastrophic outcomes or we
can try to make the outcomes actively
good and in the former case we hopefully
don't need as much information about
humans and that's all thanks I think
your hand for this beat overview if you
have any questions yeah thank you for
your excellent talk I have a question
about limited AGI what if people
thinking about limiting resources in
terms of compute is it naive a solution
because an AGI will find a way to break
through and find resources from nature
computational resources from nature from
collaborators or do you think that we
know some of the approaches courage
ability or even stop this for instance
the factorial Commission go into this
direction about packaging Commission so
that it's limited in terms of you give a
budget to AGI and when the budget is
over then this is done all credibility
or ability on the limited AGI front so I
am NOT I personally there seem to be a
lot of people who are interested in the
containment and like restricting
resources type of approach to
keeping AGI safe I'm personally not I I
don't find this all that compelling so
I'm like not the best person to ask
because like they probably have good
reasons that I don't know but in my case
the reason I'm like not as optimistic
about those approaches in particular is
that like I sort of see the primary
problem as economic incentives for
building AI are very strong like if
economic incentives we're building and
were not really strong we could like
take our time make sure that thing was
safe and then only then deploy it but
the reason we can do this because of the
economic incentives and so things like
limiting resources or boxing or
containment it seems to me like those
are like basically trying to stop
economic incentives which were in some
sense the reason we had a problem in the
first place that was one question the
next question was about corage ability
and mild optimization and things like
that like using those to make sure that
a GIS like try to do some tasks and then
stop did I paraphrase that right got it
I see so I think that's still that's
very different in that there you're
trying to like to change the AI system
so that it only wants to use limited
limited amounts of resources I think if
you did this in a way set where you
could control the actually I take it
back I think I like have the same answer
to that one
maybe if we like had a very compelling
case for this is the amount of compute
resources at which things are going to
be really bad and that like level is
like really high such that it doesn't
you can still do many economically
valuable things with AI if you could do
something like that then then maybe I'd
be optimistic about these approaches I
don't currently know how you would do
something like that all right I have one
more question
and the panel and any of the approaches
that you didn't cover that you can just
like fire off for us oh man so many I'm
gonna miss them now let's see concrete
problems in the eye safety I like to
think of this as like trying to scale
safety up with capabilities but that's
like sort of a broad set of things not
just one particular thing safe interrupt
ability into like there have been
proposals for like using biology as a
testbed for learning values things like
AI safety grid rolls have a bunch of
problems and them that I didn't really
talk about what else
I'm sure I'm still missing some let's
think Rohan again
[Applause]
[Music]

The video from my talk at the Beneficial AGI conference has just been released. In this talk, I cover five broad safety-related areas that people are investigating: understanding the future of AI (<@embedded agency@>(@Embedded Agents@), <@CAIS@>(@Reframing Superintelligence: Comprehensive AI Services as General Intelligence@)), limiting the influence of an AI system (<@boxing@>(@Asymptotically Benign AGI@), <@impact regularization methods@>(@Designing agent incentives to avoid side effects@)), robustness (<@verification@>(@Certified Defenses against Adversarial Examples@), [red teaming](https://ai-alignment.com/red-teams-b5b6de33dc76)), helpful AI systems (<@ambitious value learning@>(@What is ambitious value learning?@), [preference learning](https://openai.com/blog/deep-reinforcement-learning-from-human-preferences/), [Cooperative IRL](https://bair.berkeley.edu/blog/2017/08/17/cooperatively-learning-human-values/), <@corrigibility@>, <@factored cognition@>, [iterated amplification](https://www.alignmentforum.org/s/EmDuGeRw749sD3GKd), <@debate@>(@AI safety via debate@)) and <@interpretability@>(@Exploring Neural Networks with Activation Atlases@). My [podcast](https://futureoflife.org/2019/04/11/an-overview-of-technical-ai-alignment-with-rohin-shah-part-1/) ([AN #54](https://mailchi.mp/3e2f43012b07/an-54-boxing-a-finite-horizon-ai-system-to-keep-it-unambitious)) covers almost all of this and more, so you may want to listen to that instead.

[FLI's YouTube channel](https://www.youtube.com/channel/UC-rCCy3FQ-GItDimSR9lhzw)