How to build a safe advanced AI (Evan Hubinger) | What's up in AI safety? (Asya Bergal)

hello and welcome to the session on how
to build a safe advanced ai
with evan hubinger and asia burgle i'm
anjali and i'll be the emcee for this
section
we'll start with a 30-minute talk by
evan followed by a 15-minute talk by
asia
then we'll move on to a live q a session
where they'll respond to some of your
questions
you can submit questions using the box
to the right hand side of this video
you can also vote for your favorite
questions to push them up higher on this
list
now i'd like to introduce our speakers
for the session
evan hubinger is a research fellow at
miri working on solving
inner alignment for iterated
amplification prior to joining mary
evan was an ai safety research intern at
open ai
an author on risks from learned
optimization and advanced machine
learning systems
a miri intern designed the functional
programming language coconut
and did software engineering at google
yelp and ripple
evan has a bs in math and computer
science from harvey mudd college
here's evan
hello all my name is evan i am a
research fellow at the machine
intelligence research institute
i'm going to be talking about how to
build a safe advanced ai
or well so not quite so i don't
know the solution to ai safety but i am
going to be talking about how we think
we
might build a safe advanced app so
there's a lot of
proposals out there sort of different
people working in the field and
different possible
ways that we might be able to build a
safe
advanced ai is you know very powerful
and
and in fact so we're doing what we want
this is what we're trying to achieve
in as safety there's a lot of different
people with different ideas for how we
might go about doing that so i'm going
to be trying to talk about
some of those ideas go through some of
those different possibilities
for how we might in fact build a safe
advanced layout
so the first thing that i want to go
over is
what does a proposal for building safe
advanced ai need
what are the sort of necessary
components that any proposal sort of
needs to address
i'm going to go over four so the first
one that i want to talk about is outer
alignment outer alignment is
fundamentally the question of
if we are training a model and in the
standard machine learning paradigm
uh when we sort of produce an ai we have
some objective some
loss function reward function they were
trying to
produce a model some sort of neural
network or whatever
to achieve and that sort of objective to
sort of minimize that loss maximize that
and our alignment is the question of is
the thing we're trying to train it on
is that objective the loss function
whatever is it a line
if the thing was really trying to
achieve that loss function or whatever
would we like the result would that be a
good thing
a standard sort of problem that falls
under this heading that you might be
familiar with
is the sort of paper clip maximizer
problem and by giving ai
the uh sort of task of maximizing the
paper clip output of my paper
factory it might sort of as a result
just sort of tile the world with tons of
paper clip factories
producing lots of paperclips this is a
really good way to make a bunch of
paperclips
and so we would say that the objective
of producing paperclips is not
outer alignment all right so now the
second question
is inner alignment inner alignment is
the sort of second piece that we need
when we're talking about building
ai via machine learning which is how do
we actually
ensure that the training procedure
results
in a model which is actually trying to
do the thing
the objective that we're training on we
have this sort of classically we do this
gradient descent process
we try to find a model which is trying
to you know achieve some loss function
reward function
and the inner alignment is the question
of did that work did we actually get a
model which is trying to do the right
thing
um and so these are sort of two
components of alignment the two
components of how do we ensure that
this sort of uh proposal actually
produces a model which is doing the
right
or at least trying to do the right thing
and then we sort of have two components
for competitiveness
uh where competitiveness is sort of more
about is the model
is the sort of approach actually one
which would be feasible to implement and
worthwhile
so first we have training
competitiveness which is how hard is
this proposal to do uh if you're sort of
deep mind
and you're trying to pitch this proposal
to the mind where do you mind you
have the resources to do this
efficiently and effectively would this
be a thing which like
current tools and like our ability like
what we predict
maybe in the future to be able to
produce
like all of the different possible uh
sort of machine learning tools you might
have in the future will they be capable
of actually doing this
and then we have performance
competitiveness which is the other sort
of second component of competitiveness
that i want to talk about
which is how effective uh how powerful
is the ai system that results from this
proposal if it actually works
if it all goes through if we like in
fact produce
a sort of powerful ai system how
powerful is it would it be able to do
all of the tasks that we might want an
ai to be able to do
um it's sort of not useful if we just
sort of produce an ai that can't
actually do anything and so even
even if that ai doesn't kill us even if
it's a line we still want it to like
actually be able to do things in the
world that's why we're building an ai in
the first place
all right so these are sort of four
basic components
now it's important to note that i don't
have any of the answers here i don't
claim to sort of
know the answer for any of these
proposals about whether it actually uh
successfully addresses each of these
components it's just a list of various
things to think about and to consider
when you're sort of looking at any
individual proposal for how to address
the sort of overall
ai safety all right
so with that in mind here are the four
proposals that we'll be talking about in
this talk
um it's worth noting that there are a
bunch of other proposals that i'm not
going to talk about
but that if you're interested in them
you can find them in the post which i
sort of
at the bottom you can see that the title
of it uh you can sort of find that i
think it should be linked along with
this talk
all right so we're going to start with
an approach called imitative
amplification
all right so to understand imaginative
amplification i first have to sort of
have a bit of a digestion
and try to talk about a couple of
important concepts which are going to be
really useful when we talk about
immaterial application
the first one is something called hch so
hch is a recursive acronym
which actually which stands for humans
consulting hch
so a little bit weird uh i'll explain
why that makes sense and how that works
but we'll start with a very simple setup
which is i have a human and that human
takes in a question
and produces an answer this is sort of
you know very simple setup it's just a
human answering questions
uh and now i'm going to add something
i'm going to allow that human
to talk to two other humans when
producing their answer so we sort of
have a group of humans
where there's the one human which is
trying to sort of uh sort of the leader
is
in charge of producing the answer but
they get to ask questions to and consult
with to other people that are sort of
helping them out
that's this is great and this is sort of
you know how a group might sort of
answer questions
but now i want to sort of rehearse this
procedure and i want to give
each of those humans that the original
human is consulting with
access to their own humans to consult
and so we're sort of building up this
tree we have a human at the top
who's consulting with some other people
and they're each consulting with
other people um and then keep going
so hch is the sort of theoretical object
that is the limit of this recursion
infinitely a sort of if we just
keep allowing the human to consult other
humans and so on
uh to produce this sort of infinite tree
of humans
we sort of recall this result this sort
of the whole thing
hch and this is a sort of theoretical
object but it's going to be important to
our analysis later on
all right now the second thing which i
need to sort of explain is amplification
so what is amplification
so i'm going to start with a similar
setup i have a human they get a question
and they produce an answer but now uh
similarly pre to previously i want to
have the human consult with
uh something else but instead of
consulting with a human this time i want
them to consult with a model some ai
which we would call m and so the human
takes in some question
they get to sort of maybe you know type
some things out to an ai and get some
answers back
and then with the ability to consult
this ai they produce an
answer okay and i want to call this
sort of box here where you have the
human consulting model
amp m which is to say the amplification
operator applied to the model m where
the amplification operator is the
procedure
that takes the model and sees what
happens when a human
is given access to that model and the
idea behind this
is that it sort of increases amplifies
the capabilities of the model n
because you know what it's not just what
m can do on its own
it's what multiple copies of m can do
when sort of organized and deployed by a
human
and so this is a sort of key piece of
imagery application is this sort of
amplification
operator all right so now what is
imaginative amplification what does it
do
well fundamentally we want to train our
model
m to imitate that is sort of copy the
behavior of
the amplified version of that very same
m
so you can sort of see this sort of
happening on the on the right
we have an initial model m zero
and zero is amplified into this model
amp m naught sort of green arrow you can
read as amplification
and this new amp am not recall is a
human
consulting m naught but then we take
m naught and we train it to match the
behavior
of the hue of the sort of human
consulting itself that's this gray era
where the sort of
cyan arrow is the imitation loss and
this produces a new model m1
we then amplify this new model m1 and
repeat the procedure we train
m1 to copy the amplified version of
itself
uh and this is a little bit weird but
we'll try to unroll this and understand
what's going on in all of it
but there's another important piece here
as well which is in addition to training
to sort of
mimic uh the amplified model we also
want to allow the amplified model to
sort of inspect the training process
and look at the model and make sure it's
training in the right way
um and so this is the sort of oversight
that we want to have
in addition that should help us in terms
of uh some inner alignment stuff we'll
talk about that
as well all right so first let's try to
address
what is the sort of limiting behavior of
this what is happening with this sort of
weird recursive imitation
so if we take a look at this sort of
picture we have these sequence of models
and they're trained
to sort of mimic the amplified versions
uh we'll try to imagine what happens in
the limit what happens if each of these
training processes is sort of perfect
and so if they're perfect then we can
just sort of equate
well if m1 is trained to approximate the
amplified version m
well if we imagine sort of in the limit
we get the sort of perfect approximation
m1 is just
equal to amplified m-dot and then if we
have this we can sort of expand we can
sort of substitute
in amp and not everywhere where we see
m1 and similarly for m2
and we get that sort of m3 after we've
sort of done this procedure three times
is equivalent to the amplified version
of the amplified version of the
amplified version of not
what does that even mean so let's let's
sort of expand that to try to understand
what we're looking at after we've done
three iterations
so we're trying to understand what is m3
we can see that
first of all we know m3 is there's an
amplification operator at the top
and if we recall the amplification
operator just refers to a human
consulting whatever's inside
and so whatever is inside is amp amp m
naught so we have h
consulting amp amp and then we can sort
of unroll this further what is amp amp
m not well that's just a human
consulting amp ethanol
uh and we can sort of unroll this again
and we get that after three iterations
we've built up sort of three levels of
what you'll recall is the hch
tree so we have a human consulting two
humans and those humans are consulting
two other humans
that are then consulting this sort of
initial model
and the idea is that if we sort of do
this uh sort of in the limit if we keep
doing this procedure over and over again
we should get closer and closer to the
theoretical object this hch tree
because we're getting closer and closer
to sort of limit of many many copies the
sort of infinite tree depth
of sort of humans consulting humans
insulting humans and so on
all right so this is sort of the goal of
imitating amputation is to get to this
hch
so now we can try to understand how does
the application score on some of these
different
properties that we're trying to sort of
uh gauge which of these proposals
so for outer alignment and performance
competitiveness because outer alignment
performance competitiveness are about
sort of what would actually happen at
the end
what is the sort of uh if we actually
managed to get a model which was doing
the right thing
would it sort of be aligned would it be
competitive because the procedure is
trying to limit to hcg we can try to
answer the question is hch align is hch
competitive
because uh if they are then that sort of
gives us a sort of
uh upper bound a sort of goal of well
if the thing we're shooting for at least
is a line competitive then at least we
have some degree of outer alignment and
performance competitiveness
um and there's lots of reasons you know
it's a very complex question trying to
understand would this sort of big
massive tree of humans be aligned would
it be powerful would it be able to do
sort of things that we want um and this
is sort of a big open question that sort
of makes up a lot of the outer alignment
and performance competitiveness concerns
then we also have inner alignment and
training dependentness concerns
uh for inner alignment uh if you recall
we're trained
not just on imitation but also on this
oversight
so the goal here is to try to have it so
that the
uh the overseer which is the amplified
version of the model is able to sort of
steer the training process away from the
sort of domains that we're really
concerned about to make sure that it
doesn't
it sort of is in fact learning right
because we don't necessarily trust that
if we just do grading descent it's going
to do the right thing
and then for training competitiveness
we're trying to understand well so
fundamentally this is a language
modeling task and so we want to
understand how competitive our machine
learning tools in the future are going
to be
at solving these sorts of complex
language modeling tasks and we have some
evidence that they are pretty good at
that because we have things like gbt3
currently
that are very successful all right
and now uh who's working on this so
people who are currently working on
imagery complication so paul cristiano
sort of created the idea of application
um
and he is a researcher at open ai i work
on the application a lot i
uh work at miri though i also used to
work at open ai
also the rest of the opening eye
reflection team uh that sort of works
under polit opening
as well as ought which is a sort of
another organization that does sort of
more human experiments trying to
understand things like
you know what would hch be like by
looking at current groups of humans and
how they can work
all right so that's sort of number one
that's imitative application
uh and now we'll look at number two so
number two is ai safety via debate
so what does aisa t via domain so asft
be a debate
uh the basic idea we have a model
and we have a copy of that model we'll
call the sort of first one alice and the
second one bob
and we want to train those models to win
debates
against each other in front of a human
judge so how does this work so if you
take a look at the sort of branching
argument triona right
we start with some question then alice
gets to choose how to respond to that
question
then bob gets to sort of choose how to
respond to alice's uh sort of
answer so bob might sort of refute
alice's answer provide
some sort of alternative um and then
alice gets to respond to bob
and then eventually we sort of get to
some leaf and the human decides did
bob when did alice run um
and the idea here is that we're trying
to train these
models to sort of produce the most sort
of relevant honest truthful information
because if they do so the hope is that
well the human will sort of choose the
model that has done the best job that
has been the most truthful most helpful
most honest
and will incentivize that sort of
helpful honest behavior
via this debate process now there's a
couple of other things that we can do
here that are nice
so so uh one thing that is we can't do
in a sort of human debate
setting for example but that we can do
when we're doing this ai debate is we
can add this sort of cross-examination
so we can allow uh bob for example to
look at what alice would have done we
can
allow bob to make a copy of alice at a
previous point in the debate
and ask alice questions be like uh maybe
alice for example said something was
kind of confusing maybe on purpose
alice said something confusing just sort
of throw bob off bob can go back and ask
alice copy
a copy of alice a question to sort of
clarify and then show that answer to the
human judge
importantly another sort of component of
this is not only can sort of bob
ask alice questions bob can also look
internally using transparency tools or
other mechanisms
to sort of see is alice sort of doing
the right thing and maybe
bob can sort of look inside alice and
show the human evidence that alice is
trying to trick the human
uh in which case then the human might
sort of kick bob and so this can help
incentivize the models to stay honest
uh because they sort of uh can be
inspected by the other model
all right uh and so now we can again
sort of do some analysis try to answer
the question
uh you know how uh successful aligned
competitive would this sort of approach
b uh if we think about outer alignment
an important question like i was saying
previously is the question of is
honest behavior incentivizing the limit
uh importantly here
is we're specifically trying to think
about the first player which in the
previous example was
alex because uh if
honesty is actually the sort of dominant
strategy if sort of
doing the sort of honest helpful answer
is the thing which is most likely to
cause the human judge to sort of choose
you
then we should hope that in the limit
we'll end up with models
that are trying to uh actually sort of
produce the most honest answer because
that's the thing that's most likely
going to be
but if the model is going to do that
then we can just ask the model what its
answer would be and we don't have to run
through the whole debate when we
actually want to get something out at
the end and so
we want to ensure that the sort of first
debater would produce a truthful answer
all right and then for uh inner
alignment
uh we're sort of thinking about this
question of are the debaters able to use
these transparency tools to inspect and
detect
sort of bad things uh happening in the
other creator
um for training competitiveness uh it's
interesting to think about this is in
many ways a sort of self-play
on a sort of game setting in a very
similar way to something like alpha zero
uh was sort of solving a game via self
play and so there's a lot of previous
examples of ways which machine learning
can successfully tackle this sort of
problem
and so we might hope that this is the
sort of thing that we'll be able to sort
of
uh deal with our machine learning tools
in the future and the performance
competitiveness there's this question of
well
how useful would a sort of superhuman
debate ai would be able
to you know answer the the sorts of
questions that we need to answer and
that's obviously sort of off debate
um importantly if uh honesty isn't
incentivizing the limit then it also
sort of might not be performance
competitive
because uh it might sort of just give
you bad answers
all right then who's working on it uh so
a safety via debate is due to jeffrey
irving who
is currently at deepmind he used to be
an opening uh paul cristiano who's the
opening also works on debate uh quite a
lot
as well as the rest of the opi
reflection team which is sort of a team
that is sort of managed by paul
as well as och i mentioned previously
all right uh next up we have recursive
reward modeling
so what is recursive word modeling so i
want to start
with the sort of image in the top right
where you can see the sort of user
reward model
agent and environment this is describing
the basic reward modeling process
the way that this works is we have some
user uh we can imagine it's a human
in this setting it's it's not going to
be just a human we can imagine it's a
human
and this human sort of is trying to give
some feedback
some sort of information about what it
wants this
is fed into a reward model which gets
trained to try to predict what the human
wants
and then we train an agent to try to
maximize this reward well to achieve the
prediction of what the human wants
then we put this asian environment we
let the agent sort of run around and do
things
and then the human sort of looks at what
the agent is doing and gives some
feedback it's like i like this thing
that the agent was doing i don't like
this other thing that the agent was
doing
then we put that feedback back into the
reward models to improve it and then we
get a better agent and so on
now importantly is that that's the
reward modeling process
but here we want to talk about recursive
reward online so what does that mean
well
we want to take that basic procedure and
recursive so instead of just having a
human be the user
you want to instead have a human
consulting
an agent that was itself trained via
another
reward modeling process and so that's
where you can see these sorts of
multiple loops on the sort of
user on the right it's a human but it's
the human consulting another agent
and in many ways this mirrors the
amplification setup from previously
so the top picture is sort of identical
to the bottom picture
where you can think about what we
actually what we're doing here is we
have some model
m naught we amplify that model to amp m
naught which is the human consulting m
naught
and then we do reward modeling with
the sort of human consulting m naught as
the user
and that produces a new agent which
we'll call m1
we then amplify the new agent m1 produce
sort of
uh amp m1 which is human consulting m1
use that as the new user in a reward
modeling process
get an m2 and so on and so these are
sort of these two pictures or two
different ways of looking at
what is fundamentally the same procedure
which is this recursive word modeling
procedure
and then in addition we can add on top
of this oversight so similarly to what
we had in amplification
we can have the amplified version of the
model inspect
the training process and make sure that
it's sort of doing the right thing
all right and now a question that we
want to ask is what is the limit of this
procedure
so if we think about uh what's happening
when we do
recursive reward modeling where it's
similar to amplification and that we
sort of have this tree that we can
unroll
but instead of just being a tree of
humans uh because the sort of models
were just trained to
mimic the humans and so they were sort
of identical to the humans in the limit
now the models aren't trained to mimic
the humans they're trained to sort of
maximally achieve the reward obtained
by doing reward model economic humans
and so now we have a sort of the
limiting tree as a human
consulting models which are trained to
maximize the reward of
uh sort of obtained from doing reward
modeling on a human consulting other
models they were trained to maximize the
reward from doing reward modeling on
a human consulting and so on and so this
is a sort of reward modeling tree
recursive word modeling tree
um that is so the limit of this person
and so now we can ask questions
uh sort of trying to analyze this
procedure similarly previously
for outer alignment we can sort of ask
is the for both underlying and
performance competitiveness we're going
to be talking about sort of
these properties of the tree uh you know
is it aligned
will we sort of like the results of this
tree and is this tree competitive is it
sort of universal is it able to solve
all the different sorts of problems that
we might want to be able to solve
i mean a lot of this doesn't come down
to details of you know is reward
modeling successful and being able to
solve a lot of these sorts of problems
and sort of learn the right things um
and then for inner alignment we're
relying similarly to some sort of
amplification relying on this oversight
where we have this overseer which is
looking at the model during training
and trying to make sure that it's sort
of being trained in the right way
and then for training competitiveness
we're sort of trying to understand the
question how effective would reward
learning be
as a sort of general purpose uh strategy
to sort of do in machine learning
and again this is something that is sort
of been proven to work at least with
current machine learning tools
um in in some settings this is sort of a
common approach that has been used in
the literature
uh but there's obviously still a
question to what extent this will sort
of continue to be true and be a
successful approach
to training machine learning systems in
the future and then we have the question
of who's working on this
so people are working on this um so
young micah who's at deepmind
uh david krueger who uh is at mila
the montreal is sued for learning
algorithms as well as deepmind
he's worked with uh if you might as well
tom everett who's a deep mind
um as well as the sort of rest of deep
mind safety does a lot of work on
sort of this approach or cursor board
all right and then for our last approach
we have microscope ai
so microscope ai is a little bit uh sort
of a different approach
so the basic idea is to train a
predictive model on some data we just
want to train the model to sort of
understand to be able to predict this
data and in addition we want to sort of
be using transparency tools to make sure
it's actually just doing prediction
and then we take this model and we use
transparency tools to understand what it
learned
what sort of abstractions it built what
sort of uh
things it inferred about the causal
structure of the data
about the sort of uh you know all of
these the sort of things that are
necessary to be able to project and
understand the data
and we extract those insights using
transparency tools by looking inside the
model and figuring out what it's doing
and then we use those insights that we
gained by looking inside of the model
to guide human decision making and so
we're sort of keeping the human in the
loop
so chris ola who's the sort of head of
the clarity team
at opening eye has a quote about this
that i think is sort of uh
really sort of useful to think about it
chris is sort of the person who created
the concept of microscopic ai so chris
says
that uh the visualizations and here he's
talking about the sort of neural network
visualizations that he
spends a lot of time working on the
visualizations are a bit like looking
through a telescope just like a
telescope transforms the sky into
something we can see
the neural network transforms the data
into a more accessible form
one learns about the telescope by
observing how it magnifies the night sky
but the really remarkable thing is what
one learns about the stars
similarly visualizing representations
teaches us about neural networks
but to use us just as much perhaps more
about the data itself
and so the idea here is that when we do
visualizations when we try to understand
what neural network is doing
we don't just learn about the neural
network we also learn about
what the neural network knows we learn
about the data we learn about the
abstractions we get ideas that can sort
of help
influence humans i mean this sort of
gives us a feedback loop
where we sort of uh produce better
insights that help improve human
decision making which allows us to build
better ai systems
and so on and sort of keeping the human
in the loop of this sort of uh
self-improvement all right and so now we
can ask the questions
uh sort of similar to previously you
know how aligned independent would this
approach be
and we think about uh outer alignment
it's important to know microscopy
isn't really trying to be outer line
because it's not we're not trying to
have the ai
actually take actions in the world and
so it doesn't need to be the case that
it's
uh objective is sort of one that if it
were trying to take actions according to
it would be a line
but we do still need inalignment because
we want to ensure that the model isn't
going to try to do something really
weird and wacky and crazy something
totally different than what we were
trying to train it for
um and the use of transparency tools to
check the model ensure that it's really
just doing prediction is very helpful
here
in addition we can sort of talk about
training competitiveness training of
animus should be pretty straightforward
we do have lots of
in machine learning currently lots of
sort of training of predictors
um the real sort of sticking point here
is performance competitiveness which is
the question of well
if we actually had a microscope ai if we
were able to use it to sort of improve
human decision making
would that be sufficient for sort of the
economic cases that we might want ai for
um you know if we're we sort of doesn't
actually let us sort of
obviate humans we can't just sort of
replace humans with ais
because we sort of still need a human in
the loop here but that might be
sufficient at least for sort of high
level decision making like sort of ai
ceos and things like that
um even if it's not sufficient for sort
of maybe more low level replacing sort
of all jobs all right
and so who works on microscope ai so i
mentioned chris ola who sort of created
the concept he
is a researcher at open ai and used to
work in google brain uh the sort of rest
of opening eye clarity works on uh sort
of thinks a lot about this stuff as well
as well as
other as well as other people at google
brain so uh including for example shane
carter
all right so that's sort of the
four proposals that different people are
working on and thinking about
and i want to sort of close with an
exercise that i think is sort of useful
for
trying to start thinking like an ai
researcher and a safety researcher
um and sort of really understand uh sort
of
the pros and cons and the trade-offs
here uh think about the question
if you had to recommend one of these
proposals if you were sort of uh you
know giving a recommendation to define
or to open the eye
uh as to like what avenue they should
pursue for trying to build
a sort of safe advanced ai which would
you recommend where would you sort of
steer these
uh these organizations too if you were
sort of giving the recommendation and i
think this is useful as an exercise just
sort of as a
thinking tool because it sort of helps
you think about well
you know what would i sort of you know
if i was actually in the position where
i was sort of giving this as a
recommendation if i had to go to
deepmind and convince them to implement
this
what would be the sort of best thing
that i would sort of lead with that i
would try to get people to focus on
and so i think this is a good sort of
exercise a good thing to think about
i'll sort of leave up the proposals that
we talked about here uh sort of imitate
my application microscope ai request
forward modeling and aic tv
um and again i'll say if you're
interested in going over even more
proposals
there's a sort of larger document that
includes a bunch of additional proposals
which you can access
um you can sort of see the information
on the screen you just sort of google
for it or i think there should be a link
along with this presentation all right
thank you so much
uh and we can go to i can try to answer
some questions after the talk
thanks so much for that great talk evan
we'll now hear from asia burgle who
is a researcher at ai impacts a writer
for the ai alignment newsletter and a
fund manager for the long-term future
fund
she has a ba in computer science from
mit since graduating she's worked as a
trader and software engineer for alameda
research and as a research analyst at
open philanthropy
hi everybody i'm going to be giving a
talk which i call what's up in
ai safety my name is asia burgle
i do a lot of different stuff one of the
things that i do
is i write for this newsletter called
the a alignment newsletter
which summarizes recent work in a
alignment
and i thought in the spirit of this
newsletter i could share in this talk
some recent alignment work that i think
is cool
the work that i share is going to be
biased for being recent so
in the last year or so um it's going to
be biased for stuff that i happen to
know about
and i'm vaguely going to try to cover a
bunch of different places that do a
alignment work
i want to be clear that i'm not
intending this talk
to be representative of alignment work
as a whole
i'm not selecting for what i think is
the most important work or anything like
that i'm really just hoping to give sort
of a flavor
of what alignment work looks like
recently
so starting off with stuff at openai
uh chris ola earlier this year released
an update on some work he's been doing
on interpretability
so interpretability generally is
basically this property that we'd like
to have
where we'd like to be able to know
what's going on inside of our neural
networks
generally neural networks are modeled as
black boxes
but we'd like to know what's happening
inside of them because then we could
verify
that they aren't doing things that are
wrong or bad
so in this work chris ola basically
tries and succeeds
at decomposing neural networks into
their constituent pieces
where those pieces are individual
neurons and their functionality
and the structure is composed of
individual neurons which he calls
circuits
so this picture is sort of showing one
of these decompositions
a neural network that's trying to detect
a car is decomposed into its constituent
parts which
detect windows a car body and wheels of
that car
and one cool thing that chris postulates
uh sort of doing this work
is that the insights that you get you
know looking at the structure of one
neural network
actually transferred to other neural
networks so we should expect
neural networks especially once they're
doing similar things to have very
similar structures
and this sort of fact is actually a
really just encouraging fact for
interpretability as a whole because it
means you know we have some hope of
understanding
future neural networks without having to
do you know all of the interpretability
work from scratch
um so yeah i think this is very cool
work
uh elsewhere in open ai beth barnes
released an update on progress on ai
safety via debate
um so debate is this proposed mechanism
for being able to oversee and evaluate
future ais
so we have this problem if we do want to
oversee and evaluate future ais
where humans are going to be
significantly less smart and
significantly less fast than those ais
um so we don't really have the time or
brain power
to check every single move that they do
to make sure you know it's not bad
they're not trying to trick us or being
unhelpful
so the idea behind debate is you know
maybe one way we can hope to try to
evaluate them
is if we actually have a group of ais
where other ai's job is to try and you
know poke holes in
or otherwise identify the failures um or
wrongdoings of other ais
so it's kind of unclear what this high
level mechanism would actually look like
and whether it would work
and one way to try to get it whether it
would work is to try to look at an
analogous case in humans
um so the analogous case in humans maybe
looks like
you know we have a non-expert human that
would like to evaluate and oversee
the behavior of expert humans so one way
we can get at that is to try to
have the expert humans debate each other
you know put holes in their own
arguments
um and see if the non-expert humor see
if the non-expert human comes away with
that
um with sort of the right conclusion
about whatever question they're debating
about
um so beth barnes has been basically
doing empirical work
testing this mechanism um she's been
running debates
she's been trying to break those debates
by having you know the experts do weird
stuff that might trick the non-expert
and then she's been trying to design new
mechanisms to make it so that
the incentives of the experts are such
that you know the non-expert can't be
fooled
or otherwise misled so yeah very cool
work from open ai is still in progress
ask for beth if you want more updates
other word from deepmind there's a new
paper that victoria krakovna released
along with some other people
called avoiding side effects by
considering future tasks um so one thing
we would like ais not to do
is we would not like them to have
catastrophic side effects in pursuit of
their goals
so you know if you tell an ai that you
want it to get you a jar of peanut
butter
you would like it to do that in a very
chill way not by you know destroying
supermarkets or something like that
um but it's actually kind of difficult
to know
how to specify in the general case that
you don't want your ai to do random bad
things
um the work sort of trying to do this
often goes by the heading of impact
measures
uh but the idea in this paper is you
know one way that maybe we can specify
this in the general case
is by rewarding the ai if it's still
possible to complete
some future tasks after it takes
whatever action it takes
um so the idea is you know if after
getting you that jar of peanut butter
you know the ai is still able to do a
bunch of other things in the world and
we can hope that maybe it hasn't messed
up the world too badly
so yeah please go look at the paper if
you want more detail on this i'm
definitely not doing it justice
but yeah very cool recent work from
victoria kharkovna
next i wanted to mention a paper from
the machine intelligence research
institute
written by evan hubinger called risks
from learned optimization
this paper itself actually isn't as
recent as the other work in this talk
which most of the other work is from
2020 this is from 2019
but i wanted to mention it one because i
think it really sort of formalizes and
pins down
a lot of ideas that have been floating
around the a safety community for a
while
uh and two because i think there's sort
of been a lot of follow-up discussion
from this
all over the last year it's definitely a
very live topic in a safety communities
so the basic idea behind this paper i
think can best be explained
via analogy to evolution so evolution
is this optimizing process in the course
of optimizing for genetic fitness
it produced humans and humans are
themselves optimizers
that might be optimizing for goals that
are not genetic fitness so you know
maybe they're optimizing for
getting food maybe they're optimizing
for beauty or
truth or love or something that's not
just you know
reproducing as much as possible and
spreading their genes
so the idea behind this paper is you
know similar to how humans are sort of
now
optimizing for things outside of what
evolution originally you know cared
about
um we could expect machine learning
systems to do a similar thing
so you know machine learning systems are
trained using gradient descent
which is sort of this outer optimizer
and then inside of them we have this
neural network
that neural network could be doing a lot
of things one of the things it could be
doing
is itself acting as an optimizer and
that optimizer
might evolve to have goals outside of
the original goals that are specified
so evan calls sort of potential failures
from that optimization mesa optimization
failures
and this paper goes into detail
characterizing the circumstances where
we might expect failures like this to
occur
and just really being exact about what
these failures would look like
next we have work from the center for
human compatible ai
there's a paper called quantifying
differences in reward functions
by adam gleave and a bunch of other
people
so one sort of recent strain of thought
in ai work is the idea of reward
learning
so the basic problem is you know the
preferences that humans have are kind of
difficult to specify and we shouldn't
expect that we're generally just able to
you know hard code a single function
somewhere
that says exactly what we want um you
know maybe a more promising and more
practical solution
is that we'd like the ai systems that we
have to be able to learn
what human preferences are and that's
what reward learning refers to
so one thing you need to do when you're
doing reward learning is you need to be
able to compare potential reward
functions that you're using to
express human preferences so maybe you
know you want to see
which of two reward functions is best or
you want to you know trial a procedure
for producing a reward function by
comparing that reward function to some
ground truth reward function
so so far the way that you do this
comparison between reward functions
is you train a policy using both of
those reward functions
and then you have that policy you know
suggest some actions and compare
those actions compare the results of
that policy
the problem is that this basically is
just comparing you know
two training runs and you know there
could be a lot of details that are very
specific to that training run
that don't actually bear on the goodness
of the overall reward function
so this paper actually suggests a way of
comparing reward functions directly
it introduces a new metric called epic
which lets you do that and suggests that
you know future work could use that to
compare reward functions
um yeah very cool work out of chai
um i also in this talk wanted to mention
a bunch of independent
research that people have been doing
that you know it maybe isn't as formal
as sort of an academic paper
but i think it's really good work that's
sort of advancing the state of the art
um most of this work happens or at least
the one stuff that i see happens on the
alignment forum which
i'll talk about more at the end um it's
basically just a
super welcoming place that collates a
lot of recent alignment work and
gives people sort of the opportunity to
post their own ideas about a alignment
um so one recent sort of good series i
think out of there
has been john wentz works work on
abstraction
uh so abstraction in general um is sort
of
he suggests this field about how we can
make predictions about the world
while you know understanding what
information
we want to keep and what information we
want to throw away so we'd like to make
predictions we have a lot of messy info
you know in order to make predictions in
you know a computationally tractable way
we need to be able to sort of make sense
of that info and know what to look at
and what not to look at
and he sort of suggests that this might
be part of the solution
to a very thorny class of problems
called embedded agency problems
the idea here is you know whatever is
going on with our future ai systems
we're going to have an agent that's
going to
need to reason about it and its
environment
and its environment will actually also
contain the agent itself so in some
sense here it's reasoning about itself
and that always sort of leads to tricky
and thorny problems in computer science
and john sort of suggests that digging
more into sort of abstraction as a field
might yield some solutions to these
problems or some understanding
then other independent work recently
that i thought was cool
was work from richard no he started a
sequence
on the safety of multi-agent systems um
so the idea here is
we often think of safety problems in the
context of
one agent with one goal um you know one
ai system doing some stuff
um but you know in humans actually sort
of the most interesting capabilities and
behavior come
when you put humans in groups um you
know all sorts of interactions and
culture and intelligence that evolves
out of these sort of group dynamics
um and richard suggests that maybe a
similar thing is going to happen
um with ai systems where sort of the
most interesting and capable and even
dangerous behavior
might happen when you think about their
group interactions um so in this
sort of start of a sequence richard is
thinking about sort of how we can shape
the agents in the system and how we can
incentivize them to be safe
even in this sort of weird group
environment
and the last thing i wanted to mention
is that there is actually a bunch of
sort of other academic work that's not
affiliated necessarily with any of these
orgs
um that i think is great for ai safety a
lot of recent work on robustness
basically getting ai systems to do what
we train them to do
um in unexpected circumstances i don't
follow this work as
closely as i follow all the other stuff
so i don't want to say too much about
something i don't really know that much
about
but there is a bunch of recent work here
um i think it's you know
very true that academia does work that's
good for safety overall
and you hear a bunch of recent
robustness papers that other people
basically recommended to me
for people who are interested in this
okay hopefully that talk wasn't too
overwhelming
um i do want to point to sort of three
things that
you could look at if you were interested
in learning more about this stuff
one is you could go to the tinyurl link
um
at the bottom of this presentation which
gives more details about all of the
stuff that i covered here
another thing you could do is go and
hang out on the alignment forum which i
mentioned
um which yeah i think is a very
welcoming uh sort of
place for newcomers and for sort of
existing seasoned ai safety veterans to
discuss their ideas and to look at past
work
and sort of lastly i did want to plug
the alignment newsletter that i write
for
which i think does a good job of keeping
people up to date with recent alignment
work it
definitely makes me feel like i'm up to
date and hopefully it can do the same
for you
thanks again for those great talks asean
evan so i see we have a number of
questions submitted um so we'll kick off
with the first one for evan
um so evan with respect to the four key
alignment strategies that you talked
about
to what extent have these um models been
successfully implemented already
yes that's a great question i think
there's a couple of things that i can
say there
so one thing that i'll say is that all
of the proposals that i talked about
are sort of intended to be proposals
which scale
the idea is not just to be able to
you know implement these things now but
to have an idea for how we might be able
to take these proposals
and you know keep working on them and
improving them as we get more and more
intelligent and more powerful machine
learning systems
that being said there is a lot of work
that can be done right now to try to
understand and analyze
what these proposals will look like in
the future so for each one of the
proposals that i've talked about
there are people that are working on
trying to implement this in current
machine learning systems
so uh debate and amplification and
microscope ai are all being worked on
like i mentioned at open ai the open eye
reflection team for example recently
released a paper where they are trying
to
uh do a sort of mini version of
amplification debate to try to just
sort of fine tune gpt3 to sort of better
be able to
answer you know specific human questions
in recursive board modeling also there's
lots of work there that's done in deep
mind and so all of these things
do have an extent to which we can try
and work on it now but it's worth
keeping in mind that the major goal
of all of these is to try to make sure
that they scale well into the future not
just at the right we're able to
implement them now
so it seems like there's several
organizations that have kind of taken a
first step but with the understanding
that this will continue to be a strategy
to be worked on in the future too
that's right cool um okay next question
um so someone asked what are some of
these transparency tools that were
talked about so
again i think evan this was mentioned a
little bit in your talk but asya you
also talked about this with some of
chris ola's work are there other
examples that you can also point to
yeah i mean i think uh you know sort of
like the rest of this stuff transparency
tools are sort of like
um something we would like to have and
people are actively working on
um yeah chris ola does a lot of work on
this um you know i think in general yeah
there's
uh sort of the clarity team does a lot
of work trying to
basically think of ways to sort of like
visualize and decompose
neural networks uh there's definitely
sort of like a question of um
you know how much these methods scale
and and how much they transfer to
you know various things that we might
want to know about um so yeah chrysolo's
work has been largely on
image classifiers you know it's not
clear if it's easy to sort of do the
same thing
um with stuff like language models um
but there is also just like other sort
of strands of interpretability work
stuff called dynamical systems analysis
um i think there are lots of people
sort of trying to think of ways to
approach the problem of
uh figuring out what a neural network is
doing but there is sort of like a big
overarching question of
to what extent like any of these methods
scale and to what extent you know
they're easy to apply
um in in domains that aren't sort of as
easy to visualize as something like an
image classifier
right so similar to evan's answer um a
good first step is when taking the lots
of work that needs to be done that's
totally right yeah
um another question for evan so what's
the difference between
imitative amplification and iterative
amplification and similarly someone else
asks
how does the first step of ida work how
does a human do the initial value
training
can you shed some light on either of
those yeah so i'll try to clear some of
this stuff up so first thing that's
worth noting
is that the term iterated amplification
is more general so the term iterated
amplification refers to
any form of amplification that is doing
this sort of basic process
of you know take a model amplify it and
then sort of train some new model based
on that amplified version
via some sort of distillation process so
for example both recursive reward
modeling
and imitative amplification that i
talked about in my talks would be forms
of the general approach of iterated
amplification
imitative amplification specifically
refers to the form of amplification
where what you do is you take the
amplified model and then you just train
the model to imitate
the amplified version uh which is the
sort of first proposal that i talked
about
um and then remind me what the second
question was so the second question says
how does step one of ida work how does a
human do the initial value training
great so yeah this is a good question i
think in terms of if we try to think
about amplification there is this
problem of how do we get off the ground
initially
and one thing that is important to keep
in mind that i didn't really talk about
in my talk is that
one of the main uses for something like
amplification is not just to train an ai
from scratch
but to take an existing ai for example a
language model that was trained via
a sort of auto regressive language
modeling regime something like gbt3
and then to turn that language model
into something which is like actually
helpful and able to sort of assist
humans
so the idea with a lot of these
proposals including debate and uh sort
of imitating amplification
isn't necessarily to start from scratch
but to try to start from something like
an auto aggressive language model like
gp23
but that you know something like gp3
isn't actually trying to be helpful to
you it's just trying to sort of complete
the next
uh sort of word that it predicts and try
to turn something like that
into something that's actually going to
be helpful that's going to try to assist
the human
and so uh in terms of like how do we get
things off the ground what is the sort
of first step
well in a lot of these cases the first
step would be take an existing auto
aggressive language model
and then apply these techniques to that
one person also asks what's a currently
neglected project in this space so i
guess
going back to these questions of you
know initial steps have been implemented
but there definitely is a lot of work
done that
um for these models to scale are there i
guess like specific projects or topics
that you can talk about to
help a student uh kind of get started in
this area
um that's a great question so i think
that there's
definitely a lot of work to be done on
all of these things so you know both
me and asia talked about
interpretability that's definitely a
place where i think there's a lot of
work to be done
uh in particular if you head over to
distill.pub there's a whole bunch of
articles you can see there
including uh like a bunch of they talk
about a lot of sort of future work there
um there's also a lot of future work to
be done just in terms of
trying to take these approaches and
understand better how they're going to
work how they're going to scale into the
future
um as well as you know in particular one
thing is trying to understand
what are these sorts of training
processes like what are these uh are
sort of the inner alignment actually
going to go through with this
and so one of the things that one of the
sorts of experiments that i might be
excited about
is trying to understand um how good are
these training processes how good are
sort of the ability to
you know inspect the training process as
it's going along and
can we produce examples of cases where
we try to train
on like some particular objective like
irritative application for example
and we end up with a model which is
maybe trying to do something different
so i've i've written about this a little
bit
um in the past i have a post called
uh sort of towards a um
concrete experiments for inner alignment
i believe that sort of provides an
example of you know
what would a like simple experiment look
like to try to demonstrate the existence
of inner alignment failures
um and aussie has sort of talked about
this a little bit when she was talking
about the paper that i was author on
risk mode optimization
and one of the sorts of places where i
would be most excited about sort of new
experiments is in that space it's trying
to understand
what are these sorts of uh robustness
failures look like when you start
scaling up systems
kind of a similar question to asia in
your work with the long-term future fund
can you maybe talk about what
sorts of projects grant makers are
looking to fund or
what they'd be excited about in an
independent researcher
um yeah i mean i definitely don't want
to speak for the whole fund so i can
only um
speak for myself um yeah i think you
know uh
sort of independent research is always
tricky like it's sort of hard to make
progress as an independent researcher
um so i think in terms of in terms of
like wanting to make progress as an
independent researcher i think sort of
the
things that i look for and think are
sort of the most promising are
you know like having a good sense of
what's already been written in the space
um
you know suggesting research directions
that seem
uh tractable and meaningful and then
also just you know being willing and
able to engage with other researchers in
this space i think that's sort of very
important
um and all of this work you know um it's
very much like a collaborative field and
lots of people are are sort of
you know constantly talking about these
ideas and making progress and
suggestions um
so the extent that you can sort of get
involved with people already working on
it i think that's that's really good
okay and we have a couple of minutes
left so we'll end with a final question
um so evan and asia you guys are
kind of approaching ai safety from
different paths so evan definitely more
technical research
and i see a little bit more broad
strategy focus
can you talk a little bit about how you
got onto that path and
you know whether you have any tips on
whether this could be a good fit for
another student
maybe we could start with asia yeah
sorry um
yeah i mean i sort of um dropped into
this work by accident like i don't know
if i made a lot of like super uh
intentional choices um but i ended up
doing a lot of forecasting work and then
um i got like much more into it via
working at impact um
i think for safety stuff in particular i
mean i had a i guess i had a computer
science background um
and i sort of saw you know
advertisements for people to help write
the newsletter that i'm a part of um
and i think basically maybe what that
should suggest to students is that um i
think as buck said in an earlier talk
here is that you know the field is not
so deep
um that you need a whole lot of
experience to engage with it um so i
think if you
seem if if this stuff seems kind of
interesting um
it's very possible for people with not
that much background sort of get up to
speed to understand what's going on
um to have their own ideas um so maybe
that's sort of like i think the takeaway
maybe from my career trajectory is
um you know you don't have to be like
some absurd super genius to get involved
in this stuff um i think it's
really possible um sort of know what's
going on with a with a
less specific background
yeah i mean i definitely like what
aussie was saying in terms of my
background so
uh i sort of got very involved in
effective altruism when i was in college
um and i wasn't exactly sure sort of how
to you know deploy my skills how to you
know find a career which would make the
sort of largest impact but i sort of
went to an ea global i went to this
um workshop called ai risk or computer
scientists
um and i ended up as a result of some of
this sort of stuff uh
doing an internship at miri which was
sort of really good i think that one of
the things that was nice about that was
just sort of getting my foot in the door
and really sort of just starting to meet
people understand what's happening in as
safety
and while i was there i also attended
this thing the mary's summer fellows
program which is sort of
a couple week long research retreat um
and i met a couple of other people and
we were sort of very interested in in
what was then being called optimization
demons and became this sort of inner
alignment problem which was
resulted in us writing this paper risk
alert optimization which was very well
received and this sort of like
was sort of put me in a position where i
was like felt comfortable doing research
and being able to do research
um and so after that i applied and did
some work at open ai
and then after open ai i went to miri
which is where i am now
okay well that's all the time we have
for questions thanks again so much to
asia and evan
and thanks to all of our viewers for
watching to our viewers before you leave
the session please give us your feedback
in the poll section of the live chat
thanks again