Threat Models and Approaches for AI Alignment - Rohin Shah

i'll just read a quick introduction and
and then i'll turn the floor
over to our speaker so
um yeah so hey everyone uh welcome back
to the protocol labs research seminar
series it's great to have you
our speaker today is dr rohin shah
dr shah works as a research scientist on
the technical agi safety team at
deepmind
he completed his phd at the center for
human compatible ai
at uc berkeley where he worked on
building ai systems that can learn to
assist a human user
even if they don't initially know what
the user wants he's particularly
interested in big picture questions
about artificial intelligence
what techniques will we use to build
human level ai systems
how will their deployment affect the
world what can we do to make this
deployment go better
he writes up summaries and thoughts
about recent work tackling these
questions in the alignment newsletter
which i'll be sure to link in the video
description
dr shah i will let you take it from here
thanks for uh thanks for coming to speak
cool thanks so much for introducing me
um so yeah so this
is going to be a
pretty informal um
talk i am going to be assuming that
uh people have at least some amount of
background knowledge of
uh ai alignment um
and i know that you know i'm sure and
probably some of you have seen the talk
i gave at ea global but i'm sure some of
you haven't
that's probably fine you might get lost
at some parts but i think
mostly it should all make sense um and
if it doesn't
feel free to interrupt me um i
you know i think it is always hard for
me to calibrate
calibrate talks so i'm much more
interested in like
you know spending a bunch of time on a
single slide and being like all right
let's
let's you know make sure that like we
understand the things on the slide and
like
ignoring the rest of the talk it's like
kind of useless to give a talk
that nobody can understand um
so yeah so let's just get straight into
it
so i'm going to say the word aji a bunch
during this talk
i don't really want to
i don't like the term very much it's
just i don't really have a better term
um i don't necessarily mean that it's
like a single monolithic agent i don't
necessarily mean that it's you know like
a god that's like
able to do whatever it wants just you
know this is a powerful ai system of
some sort
and you know i'm not not not trying i
don't have like
particular claims about how powerful it
is what the level is
um i'm i'm going to be vague about it
and going to try to say things that make
sense
despite the vagueness um
cool so a bunch of you
as i mentioned probably have seen my ea
global talk so you'll probably
recognize this slide um
so what i'm going to be focusing on in
this talk
is this area over here um
and roughly the thing that unifies
everything in that area
is that it's about um problems that can
occur
uh via your ai system
actually like quote-unquote trying to
hurt you
um so you know things i'm not going to
talk about
um i'm not going to talk about the uh
impacts of ai
so i believe andrew kritz is going to be
speaking next week
um he will probably tell you about sort
of how
multi-agent interactions um that are
enhanced by ai systems uh can lead to
sort of
a and what we might call an ascended
economy that can
in which like lots and lots of things
are being produced but humans just
aren't a part of that economy and that
as a result we die out
might be like you know i have lots of
thoughts about that um
but it like you know seems like a thing
you might want to be concerned about
i'm not going to be talking about it
it's not a case where the ai system is
just sort of like
i mean maybe as a subpart some of the ai
systems are like
they're not are like trying to do things
that they know
are not um what humans want
um but like the core mechanic there of
like economic competition
um causing us to like set human values
aside and instead pursue like the
production of
things uh that mechanic is like
not particularly ai specific and not
about an ai system that is like
um trying to trying to hurt you
um or trying to do things that are not
the things that you want
um similarly things like you know
we fail at figuring out what it is we
what what it is we want as humans like
we fail to
figure out what our values are and so we
like don't
colonize the stars and make them make
the world make the universe as
best as it can be um
depending on how exactly that comes out
i might be like yup that's not
that's not a failure of the sort of
thing sort of problem i'm thinking about
it might be plausible that that happens
but it's just not what i'm talking about
here
um yeah cool
so yeah assumption number one
we're talking about um failures
in which an ai system is sort of
deliberately trying to do something that
it knows humans don't want
um okay
um so i
really like this notion of a threat
model
which is a combination of a development
model that says how we get
agi and a risk model that says how the
agi
system then like causes an existential
catastrophe
um and so like you know a typical story
might be like all right
we're going to do some like gradient
descent on training data
um you know this is just sort of
standard machine learning
and this is going to you know create an
agi that's
in that's implemented as a neural
network um
and then this agi system is going to
like you know do some social
manipulation
or persuasion um or something like that
in order to get a bunch of people that
to like fight with each other and then
eventually
it's like leads to a bunch of nukes uh
being deployed and then we all die
like that's a kind of story you might
want to what you might want to tell
um there are like lots of problems with
that story i don't actually
like endorse the story the way i said it
but like
something along those lines uh so so
here
um the development model is this part
um which tells you how we get from today
to and then there was an agi
and then the other side is the risk
model
which tells you how you get from the agi
to
existential catastrophe
and the
the reason i i like this
model is that it emphasizes that you
know these two stories
need to overlap there has to be a single
notion
of agi that works with both of them in
order for you to credibly say like
yes this is the way that we could
actually get to an existential
catastrophe
and i think that's something i want to
emphasize
as an important thing that any ai safety
researcher should be
should be doing as part of their as part
of their research
[Music]
yeah so specifically the three claims i
i'm going to make about threat models
is that one is like you know you need to
have a development model if you want to
be developing solutions that actually
apply to real systems
like if you if you use a model of agi
that is just not
actually the one that we build in the
real world well then your solutions are
most likely going to be
useless um and never deployed
so that that would be you know you need
to be working in the right development
model
two is that risk models are important
for finding solutions
this one is sort of like a little
less obviously true um you could
in principle be like rather than think
about plausible
risks um i'm just going to like try to
come up with an algorithm that sort of
guarantees safety
like you can like prove a theorem that
says this ai system will be beneficial
for humanity
and then you don't really have to think
about any risks because like
you know you've made a positive argument
for safety rather than that trying to
avert a bunch of negative arguments for
risks
um it's
i i mostly i'm just going to claim as an
empirical matter that like
it seems real hard to do this uh this
kind
of positive argument without having come
up with some risk models
um i'm especially skeptical of being
able to prove a
theorem of the form um this ai system is
beneficial for humanity without a bunch
of strong assumptions
in which case then you have to doubt all
of the assumptions
um so i i'm making an empirical claim
that like
in practice you're probably not going to
get very far
without risk models
and then third claim which i already
argued for a bit is that we should be
using the same notion of agi
for these two models so what threat
models
might we have uh so a common one
is a sort of utility maximizer threat
model
um and i'm
i'm going to present a sort of
simplified view of it
it's almost certainly not going to
capture the diversity of viewpoints that
exist
in ai safety
so you know don't take this as a
representation of
people who work with utility functions
but anyway so in this in my version of
this model let's say
you could uh what your agi system is is
it's sort of like
there's this algorithm or brain
or something that's able to do a lot of
like you know reasoning and planning and
logic and like stuff like and learning
and stuff like that
and there's you know somewhere in there
there is like this utility function that
you can slot in
and then once you slot in this utility
function out comes like
plans that achieve that for achieving
high utility according to that utility
function
um and so if you take this
as your agi model
then there's a
um there's a good uh risk model for this
which is basically um
if you've got the sort of ai system that
is pursuing your goal
or utility function then it seems like
there are a lot of convergent
instrumental sub goals
that such a system would be incentivized
to uh would be incentivized to
to try to achieve so for example
whatever your goal is it's you know
really a good idea to stay alive so that
you can actually
finish you know complete the goal this
doesn't require you to have survival as
an intrinsic goal it's just
you know any plan that
maximizes your chance of like achieving
your goal
will probably include like if there are
significant risks to your survival that
plan should include like
uh actions to mitigate those risks
and like some of those risks might be
the humans try to shut me down
and so you take actions that mitigate
the the risk of humans shutting you down
uh similarly you know if you gain power
or gain a bunch of resources if you have
a lot of money
then you have a lot of a lot your scope
of action is a lot broader
um and so that also seems very useful
for achieving your goals
uh almost no matter what they are
um and like sort of the pursuit of all
of these instrumental set goals seems
like not great for humans
so so you know it's a pretty good risk
model
uh the question is sort of what
development model really corresponds to
this do we have one
so one thing people will say is like you
know
uh sort of like it just sort of seems
like in order for a system to be
useful um it like
needs to be of this form it needs to be
doing this sort of planning
learning and so on you can't really be
useful without that
um and that's sort of their
justification for this one
i personally feel a bit skeptical of
that argument it feels like
like i sort of agree that like in normal
situations yeah you're going to
need to like acquire uh resources and
learn and things like that but it's not
obvious that this will then
like sort of lead all the way to like
causing extinction but it seems
plausible
i sort of feel some pretty confused
about that really
um people's opinions vary
some people are like pretty confident
that that does work that that argument
does work
um then there is like a more
a less a more concrete development model
um which says what we're going to do is
we're going to
have um like a research is just going to
develop
a bunch of algorithms for doing the sort
of planning and search and causal
reasoning and logic and hierarchy and
all this sort of stuff
and you know maybe we'll we'll use deep
learning to do like vision
like you know get a screen of pixels
from a camera and then we'll like you
know throw deep learning on top of it to
turn it into like
things like cats and dogs and chairs and
like people
and stuff like that uh but like the
actual
like intuitively the actual like agent
agenda parts the parts where the
uh agent is like doing plans uh
uh in the world will be these more
explicit algorithms that have
uh embedded into them a criterion
by which you evaluate different plans
which you know is what this utility
function is
um so that's um
yeah so that's a utility maximizer
threat model
that you could have and with this sort
of threat model
in you know the section of things that i
said i was talking about
uh that sort of corresponds to
this area of the graph
um so if you're familiar with mesa
optimization
the reason that's not in here is that
base optimization is fundamentally like
if you select for an agent that gets
good
outcomes you might get an agent that
gets the good outcomes for the wrong
reasons
but it's like fundamentally an argument
about selection pressures and outer
and outer optimization
whereas sort of in this previous
development model
like maybe it applies maybe it doesn't
apply it's not totally clear
um but like one reason you might think
it doesn't apply is because
it's not that you aren't selecting the
agent
for good outcomes you are explicitly
like putting in the criterion of
brightness into the agent
and then designing an algorithm that
takes that criterion of brightness
and uses that to form plans so there's
not really a chance for there to be the
sort of divergence between what you
selected the agent for and what it
actually does
um so you know it's unclear
uh whether or not misoptimization can
apply i think there are also arguments
for it to playing but it's it's
definitely less clear than in the
uh canonical case um
on the other hand things like did you
specify the right reward function uh the
right utility function
clearly obviously a problem in in this
sort of uh scenario
the entire point is like if you get the
utility function wrong
then your a system pursues convergent
instrumental subgoals and like
that ends up killing you or causing some
sort of existential catastrophe
um and so that can be fixed by like
tackling the specification problem
um and so techniques that do this like
assistance games
learning from human feedback tent
alignment stuff like that
um informed oversight
is well maybe i won't go into that
the reason i don't have scaling to super
human in there is like
mostly because the specific methods that
people have
proposed are um very learning focused
and it's not clear how you would
translate them into this
um other machine uh but i do think that
like
in general uh figuring out how you scale
to superhuman performance
is like a thing you would want to be
thinking about in this paradigm as well
uh cool any questions on that
i would love a little clarification on
what you mean by convergence
of goal of sub goals yeah
so it's maybe not the greatest term
um it's just the one that is commonly
used but
like one of my collaborators prefers
robustly instrumental sub goals um
meaning that like sort of
robustly regardless of what the the
utility function is these are useful
instrumental set goals that the agent
will
will probably pursue does that answer
the question
yeah thank you i i also had a second
question of uh
in the specific example of like the
the politician wanting to be given a
tailored message to
every potential voter um
like i feel like this becomes a clear
utility maximizer
and like you could imagine this being
quite possible with slightly better
engineering
and not much better
not much better modeling really do you
see this as
like a possible
like it i don't think it has to be an
agi to
see that threat but would you
do you think that there are useful uh
alignment mechanisms that we could use
to
fight that kind of a uh
like an artificial narrow intelligence
trying to do that
yeah um it's a good question
um i would say that um
the primary issue in that case is that
you know the politician
wants the ai system to be doing that um
so i would say it's not
a failure of ai alignment in the sense
that i am using the term here
because the ai system is doing exactly
what its designer intended it to do
um that being so like my usual
my usual stance on this sort of thing is
like yeah the way we want to fix these
these sorts of problems is like by the
sort of um
typical mechanisms we use to solve yes
kerala i don't know after brown said
sorry uh suggest human human alignment
problems
things like you know laws regulations
social pressure
stuff like that um and like you might
worry that like you know the politicians
get
boosted by ai but then like maybe the uh
regulators or lawmakers or like ordinary
citizens also get boosted by
ai you know maybe you give everyone
their own little ai system that tells
them when they're trying
they're being manipulated by politicians
so
so yeah i don't
know better answers great thank you yeah
i am not going to like you know say that
any one of those will work
but that's sort of broad category of
solutions
cool
onwards um
there is a deep learning thread model
which you know you might have expected
given that i work at deepmind
so in this case your agi
is implemented as just a neural net and
the neural net itself
is the thing that is uh it is the agi
it's not you know the process of
gradient descent
um or anything like that it's like the
neural net with like
already trained parameters is an agi
you're like
this is this would be something like
gpd3 for example is like maybe the
sort of thing to have in mind you like
give it an example and then input
and uh it like you give it like a
sentence and it's like able to
understand that sentence or learn from
that sentence and like do something
sensible on the other side
except you know with energy it would be
much more powerful
um and so then the development model
um is basically it's sort of this one's
very broad
um there's definitely a lot of detail to
go into that i won't go into but it's
something like you know
either we use something like dpr-l so
that's like alphago
and ai5 which were both trained with drl
and then like learn very good policies
or you do something like gbg3
where you like train a neural net just
to predict the next word in a
you know giant data set of human human
sentences
uh english sentences and
because this neural net is trained on
such a diversity of data
the only way that it can like actually
do well on this is to learn some form of
general reasoning
quote unquote um and that's what then
ends up making it an agi
um again super hand wavy and vague
mostly because it's not the focus of
this talk but that that's roughly the
development model i have in mind
and so then you might ask you know what
is the risk model here
and there are really two risk models um
oh before that i should mention i'm
going to be making the assumption that
you know
we're going to have the ai systems that
are asking
that that we ask to perform specific
tasks rather than
you know sort of sort of gener
generically optimizing for human values
um so like one example would be
um this sort of like um
conversational agent which is like
helping you with calendar scheduling
so here we have our agent and you know
maybe
bob asks his agent something like
i want to hang out with alice and the
agent has
a model of like you know bob likes to go
to clubs
bob prepares to you know have social
events on the weekends and the evenings
something like that maybe but maybe the
agent talks to alice's scheduling agent
and they like figure out a time when
both their schedules are mutually
compatible and so on and eventually you
know they schedule an event
um so that's i'm sort of imagining
ai systems that are like that and not ai
systems that you know
uh figure out uh
the like you know true meaning of
humanity's existence and then like try
to make that
a reality like maybe we build those
sorts of ais in the future
i think those sort of risky ais happen a
lot sooner than that
and some more more interested in like
dealing with the like
first risky ais and then like using
those first risky eyes to help us with
future as if we decide to build
build them um
yes so back to risk models for
um for the deep learning case
uh the first one is that you have a
bad reward function or a loss function
that you train your ai system on and so
that would be
this area of of the
uh of the graph it's sort of basically
the same thing
as in the utility maximizer case
um for sort of obvious reasons reward
functions are very similar to utility
functions
and so many of the same same things
apply
this time you'll notice that the scaling
to superhuman block is inside of the
uh the area and that's mostly because um
the approaches in there are learning
based and do apply to deep learning
um so i don't really have too much to
say about that
um i think one
critique you have you could have about
this though is like
you know in some sense it's
a little odd for for this to be going
wrong
because like presumably hume
like humans and like or companies when
they're training their ai systems
they're going to be
looking at what those ai systems do
and you know if the ai systems are doing
things
that are bad or incomprehensible maybe
maybe they just
you know don't
don't deploy those ai systems or train
them not to do that sort of stuff
and you know it's a whole complicated
debate um
but there's like you know
yeah i guess there's sort of like
controversy there is maybe what i'll
point at i wouldn't
i won't make any strong statements um
but i'm going to mostly not focus on
this one and instead
uh just sort of assume this this
distributional chip
problem um so the first thing
part of this assumption which is
uncontroversial um
is that there's going to be a difference
between the situations that an ai system
is trained for
and the ones that it encounters after
deployment um
and i'm taking a very broad notion of
distributional shift here so like if the
ai system learns a new fact
or like thinks a bit longer and realizes
some logical implication of the facts
that it already knows
or like the world outside a pandemic
hits
those all count as distributional shifts
so so it is a pretty broad notion
um so so the
fact of it occurring at all is
is not very uh controversial
the part the the assumption i'm gonna
make that is controversial
is that handling this difference is sort
of the hard part of ai alignment
where by a hard part i might mean
something like
i don't know 95 of the risks of ai
alignment which
i'll remind you is the you know when the
ai system is like specifically trying to
do something that isn't what humans want
um 95 percent of that the
existential risk from that is like
coming from
the i system being okay on the training
distribution
but then like becoming uh worse as
distributional shift happens
um do i want to argue for this
i i guess one thing i i won't i mostly
won't argue for this
um the one thing i'll notice like
typical treacherous turn arguments where
like
the ai system sort of plays nice until
it has enough power to then take over
the world
do sort of count on my view as a
distributional shift
case um because during training it was
not powerful
and had to play nice and then like only
after deployment
um did it become uh powerful enough to
actually take over their world
uh if it took over the world during
training then then that's about then
like this assumption was false
uh and i have you know reasons for
expecting that won't happen but i'm not
gonna get into them
anyway there's just an assumption for
this talk
um so
the example to have in mind i think
whenever i'm talking about this
we've got our scheduling assistant this
is just copy of what was on the previous
slide
so it was you know trained this way and
then you know
add deployment turns out a pandemic hit
sad and then you know when bob says i
want to hang out with alice
he you know presumably means i want to
have a video call with alice
um or something like that and like the
agent might even know this
that that bob means that uh he wants to
do a video call
but like you know the agent was trained
that like when
when bob or when it's user
um asks to you know hang out that just
is you know the thing that it's supposed
to do is to like
schedule a night out at the club and so
it just continues to do that
um and you know this is not an
existential risk
but it's sort of easy to it's it's a
case
where it's easy to wrap your mind around
about how exactly distributional
distributional shift can like cause bad
outcomes
all right so that brings me to the
second risk model
for deep learning which is
uh bad generalization
which roughly means like
well i mean it's just sort of obvious i
guess
um you know it your your ai system
performed well by which i mean like got
the right outcomes
in the training distribution but then
when you move to a
test distribution it like stopped doing
that
and instead it's something that caused
an existential catastrophe
um and so like the tretch's turn is sort
of an instance of this
um where your model was always
deceiving you but in the training
situation it still you know did the
right things
it you know caused good outcomes and was
despite you know doing it for the wrong
reasons um and it was only
after the distribution shift that it
like like generalized poorly and then
you know
cause caused humans to die
and so this i would say is
more this category of things i will come
back to why it does this category of
things at the end of the talk
but just wanted to sign post that for
now
before before we get to sort of
solutions for this risk model
i want to like argue for
how you should be thinking about this
risk model
um and
the way i've phrased this claim is like
you know
we should be trying to produce
generalization guarantees and not
utility functions
uh this is you know conditional on the
assumption that
distributional shift is like is like the
hard part of ai alignment it's like 95
percent of the risk or something like
that
um so
and i should note i'm not claiming that
utility functions are useless or
something and
you know it's still important for this
goal to like train your ai system on the
right
thing um so you know it might
it's still instrumentally useful to to
work with utility functions
i'm more i'm trying to say like when
you're thinking about
in this risk model when you're thinking
about how is my solution helpful
um you should be thinking like how does
it help us to get closer towards a
generalization guarantee
rather than how does this ensure that
uh we have gotten the right utility
function to like plug into our ai system
um oh it was supposed to happen oh oh no
that's in the future
sorry i forgot that i made a change to
my slides
um so yeah so what do i mean by
generalization
guarantees uh it's pretty
straightforward
you want to produce basically two
guarantees one is that you get strong
performance during training
like when bob asks hey i want to hang
out with alice the agent you know
actually schedules the calendar event
you might think of this as like sort of
like
the capabilities um guarantee
that like you know your agent is
actually useful in the situations that
it was meant to be useful on
and then the second guarantee which is
uh which
is more like the safety guarantee is
like um
acceptable generalization so
you're not required to do exactly the
right thing
in the deployment setting like you know
the world can be
wildly different and then you just don't
know what's going on
uh but you know you should never be
doing things that are
known to be bad or something like that
you should never be trying to do things
that
you know humans would not want um
and so then when bob says i want to hang
out with alice maybe
the agent says like what should i do
about kogan 19
or maybe it just knows that actually
what bob wants is a video call and then
it schedules a video call
or something along those lines the point
is like you know you have
some sort of acceptable generalization
um and like one of the research
questions was like what
what should we define acceptable as um
yeah so generalization guarantees i
think
fairly straightforward um
so to contrast that um
the contrast to that was utility
functions
for utility functions
in this talk when i say utility
functions i just mean
you know utility functions or word
functions goals objectives
all the same thing um
uh there are definitely many concepts
that these things can point
to um and
you know some of you might be like oh
but a utility function is x whereas a
reward function is y
i'm like yup that if it seems plausible
i'm going to just sort of like use the
same word
or use all of these as synonyms and then
try to more precisely identify the
charac
more precise concepts and then talk
about each of those concepts
individually without trying to
use the baggage of a specific word um
so i i'll just go through all of these
concepts and then like
say why they don't seem to me to be good
goals
for alignment in the world where the
thing that we're where we think that
most of the risk comes from distribution
shift
cool um so the first kind
are behavioral utility functions so
this is pretty straightforward you take
your agent and you take its training
data that it was trained on
you look at how the agent behaves on
that training data
and then you say that you know the
agent's utility function
is this function u that basically
rationalizes or explains the agent's
actions on this on this training data
and sort of the biggest problem with
this is just that
there are many utility functions that
are consistent with this training data
and there's no way to
choose amongst them so like in our
scheduling assistant example
like you could have the utility function
that is
where the agent is like you know doing
whatever bob wants it to do
when it at least when it knows what bob
wants it to do
that's consistent with the training data
but
there's another thing that's consistent
with the training data which is
you know in in like
situations where bob has to hang out
with a friend
i get high utility for scheduling
a night out at the club
in situations where bob has to schedule
a work meeting i get high utility for
scheduling something between the hours
of nine and five
um and so on and like that's a utility
function that's also consistent with the
data
and you know in my hypothetical example
that one was the
quote-unquote correct one but you can't
tell just from the training data
so an equivalent way of saying this is
like if you do choose a utility function
from using this method you don't really
have any reason
to expect that it predicts correctly
outside of the training data
so yeah so obvious fix
um get rid of the training data just
find
a utility function that works in
all possible situations um
so now you say you know for every input
so in the previous slide
it was for all inputs in d where d was
the training distribution
now it's just for all inputs um
the agent's behavior on that input needs
to be explained by the utility function
um this one has a bunch of problems with
it
uh one is like it's kind of hard to
apply this to humans
like like i don't know man
is the like i'm giving a talk right now
i could have several goals in getting
the talk but i don't think you could
like
i don't think you could reasonably make
a case that there is some like overall
meaning of life type goal
such that both this talk and you know
random like playing minecraft which is
the thing i sometimes do are about
you know contributing to this life goal
it sort of feels like
this is not really a very crisp concept
uh that's going to work with humans and
so like i don't know why we should
expect it to apply to ai
in a nice way you could
obviously have the
um you could have the utility function
that's just like you know
at 10 am on it
on whatever date it is today april 13th
2021
rowan will get high utility if he gives
a talk
um to protocol labs and like that's the
thing that
you know that's a utility function that
does rationalize my behavior
but like all behavior is rationalizable
in this
in this way it's like sort of degenerate
you don't really get anything out of
this
um so so yeah if you don't if you aren't
doing something like correcting
for biases or mistakes if you're just
like literally taking this equation as
written
it's just a new encoding of the policy
it's the same thing as like
just expressing a policy so you haven't
really gained anything
um you could try to be like okay well we
know
we can like try kind of sort of like
figure out what biases people have what
mistakes they're making we could try to
do the same for the ai system
correct for that and then after
correcting for that we might get an
objective that says more things
but like i don't know man it seems real
hard to like say what exactly counts as
a bias versus a mistake versus like
just an unusual preference um i think
this is true for humans too but it seems
like it'll be even
way more true for ai systems at least
with humans you get to like you know
we have similar cognitive architectures
while the typical mind fallacy is a
thing
it's like still like mostly true that we
have
typical minds relative to each other
relative
like my mind is like way more to similar
to your mind than it will be
like an ai's mind so like we you know
with humans we have like
a lot more um ability to correct for
biases than we should expect to have for
ai systems so it's a challenge
um and finally like
you're sort of like requiring your
utility function to
explain the agent's behavior on all
possible
inputs so like computing this
utility function would require you to
know the agent's behavior on all
possible inputs
but if you know the agent's behavior on
all possible inputs then like
you already know whether or not it's
safe it's like you know
did that lead to x-risk or not yeah
in some sense this is just a restatement
of the second point is like
because it's mostly just an encoding of
a policy you don't
learn much more and so it's not very
helpful for ai alignment
um so yeah so those are two behavioral
utility types of types of behavioral
utility functions
uh the next type of utility functions we
can try our
structural utility functions so
structural utility functions are ones
that
rather than looking at the behavior of
the
ai system they more look at some like
internal property
um of the a system or of the training
procedure for the ai system
and identified that as the utility
function so this is probably the more
canonical sense
that you would expect um you would have
liked
seen the term utility function in um so
for example
an outer structural utility function is
just sort of the reward used to train
the agent
so if we go back to our deeper l setting
here's you know dota
and here's a reward function for dota
where you just get plus one if you win
minus one if you lose and you know it's
just a typical rl setup where you like
the agent sends actions and the
environment sends back observations and
awards
in this case you would identify this you
know reward function as the outer
structural utility function
um yeah so it's just the reward used to
train the agent
um and like i think this is an
important thing to get right but
since we're talking specifically about
you know
problems that arise from distributional
shift i think
this is not the thing you want this is
not the concept you want to be using for
those kinds of problems
and the reason is pretty simple it's
just that
the agent the values of the weights
in the parameter in the in the neural
network
those are completely determined by the
values of the reward function on the
training data
right those are the things that you use
to compute gradients
the gradients are the things that modify
the weights the weights are never
modified by
anything else you know there's also
things like initialization
and architecture and stuff like that but
those are independent of the reward
function
so like the only way for the word
information to flow into the agent
is via the values of the reward function
on the training signal on the training
data
so like who cares what the values of the
reward function
on the test data is on the deployment
data is it doesn't matter
it's not going to affect the agent um
there's like you know sometimes you'll
like think of things like
you know online training or monitoring
where you like also evaluate the reward
function at
deployment time too and that case you
know you want to make sure that the
reward function
works in whatever situations do come up
and
at that time but still you don't need it
to be literally
like you know all possible situations um
and and yeah you're you you like
yeah for for the case of things like
treacherous turns where like
the problem is there's a distribution
shift and then there's some like
qualitatively new situation that the
agent does something bad on
like it doesn't matter if your reward
function had a high number
had like you know penalized that
situation
if that situation never came up during a
time when you were computing gradients
um another problem is that like this is
much more minor but it's often unclear
what does and doesn't count as part of
this
um so i think my canonical example here
is exploration in d bar l
so you can
you can get exploration and dprl in a
variety of ways
um so one common way is to
add a term to the reward function
or loss function that rewards the agent
for visiting new states or like visiting
states in which it like can't predict
what's going to happen
or something along these lines and
you know that's one way you can get the
agent to explore more and presumably
that seems like it should be
part of the outer structural utility
function because you know it was a term
added to the reward function
on the other hand you could also
incentivize exploration by like
changing the um action selection
mechanism
such that you know when it's you know
saying i'm going to like take the action
that maximizes q value
or you instead do like not the one that
maximizes q value but instead the moment
that maximizes q value
plus um some term
that like uh some term that quantifies
like
how uncertain the agent is about the
next um about the next state like maybe
you have an ensemble of
um ensemble of dynamics models that are
trying to predict what the next state
will be
and then you take their disagreement as
a signal of uncertainty
and that one is you know part of the
action selection mechanism
rather than part of the um part of the
reward function so it seems like it
shouldn't be part of the outer
structural utility function
but you know both these methods are like
incentivizing exploration they probably
have pretty similar effects on the agent
it seems kind of weird that one of them
is part of the outer structural utility
function and the other one
isn't all right
so you know we're like i argued against
you know
identifying something in the training
process as the utility function
so maybe instead what you want to do is
you want to identify something internal
to the agent
as the utility function which has the
benefit that like it does in fact
continue mattering
even at deployment time because you are
still you know running that part of the
agent at deployment time whereas you
aren't running the reward of the
training process at deployment time um
so these sorts of definitions are
usually going to require something along
the lines of like
an assumption that the agent implements
a
search or planning algorithm
um and then so you're like this
particular part of the network probably
all of it
is um you know some sort of planning
algorithm that's
you know considering different possible
plans and then writing them according to
some metric of goodness
um and then executing the one that has
uh highest
that scores highest on that metric and
so then you just sort of look inside the
neural net
and identify the place that is that
metric of goodness
and you call that your utility function
u
so it's the goal that the agent is
pursuing if
if you're familiar with mesa
optimization it's this is what you know
what you would call the mesa objective
and the problems with this one is mostly
that like
like if you interpret click thank you
if you interpret the assumption like
strictly where you're like no this is
actually a planning algorithm
it really does like you can see the like
four plans
if you could if you could only
understand the activations you would see
four plans
and see them being you know rated by
this metric of goodness
and then being compared like if you
interpret the assumption very strictly
in that way then i just don't think it's
true
um this just does not seem to be to be
likely to be true of neural networks
and if you interpret it loosely where
you're just sort of like ah
you know it's going to be sort of like a
planning algorithm there'll be like you
know it will
sort of look like it's it's um
optimizing for some specific objective
and then like maybe if you like
understood the algorithm well you could
like tell some story about why it's
going to
be like um generally selecting for
the actions that have high values of
this metric for goodness
then it just sort of feels a very vague
and like i'm not sure that i
that like i'd believe it or something
but also i think you'd have the
same pro like you'd have similar
problems as with the behavioral utility
functions where you could have like
many possible inner structural utility
functions that make different
predictions
and i think that is a good point to stop
given that i am very close to the hour
um i have a little bit more with which i
can talk about afterwards which is more
about solutions it's like one or two
slides
but let's start with questions for now
so mine's a bit of an odd question
i noticed that you on like at one point
said that this particular alignment tool
is hard to apply to humans and so it
seems like it'd be
at least as hard to apply to ais and i
was curious
how common it is to think about applying
ai alignment tools to people
or things like that i
kind of independently came to this idea
that
if you want a company to have some
nuanced mission
you should really do something like
value learning
with every employee saying like become
here are two things the company could do
we would value this one more and just
give people
a lot of training data so that everyone
eventually agrees on
some like implicit value system so you
can avoid the whole company turning into
some shareholder value maximizer
which i think that large companies that
are particularly
bad at onboarding people culturally
or bad at expressing the like
the deep-seated like philosophical
values of the
the company um tend to wind up in and
i'm curious if this is something that
you've seen in other places or like
has come up in a lot of like do people
come to you with questions around
like corporate alignment um to use
ai alignment tools for that yeah um
so no people don't do that uh it to
start with
uh but on the broader question um
i would make a distinction between like
concepts for alignment
and like solutions or algorithms for
alignment
and i like pretty strongly want concepts
for alignment to apply to humans too
like my best guess at least in the
neural net paradigm i think i am
i am less tied to that in the like
uh utility maximizer by known algorithms
paradigm um but in the new
i like i don't know they're like it sort
of seems like the way neural nets are
made
is like kind of similar to like how
evolution made humans or even how
human brain learning works um
so like if something if a concept
doesn't work for humans
uh then i like i
i you know i need some like a really
good argument to believe that it's going
to work for ai systems
the fact that the concept works for
humans doesn't mean that it works for ai
systems as with like you know biases i
think biases cognitive biases is like
pretty sane for humans but may not be
for ai systems but it feels like a
necessary condition that the
the concept should apply to humans too
then on solutions or algorithms
uh i people do in fact think a decent
amount about how to apply
these sorts of algorithms to humans but
it's not always a thing that you can do
um there are just a lot more affordances
we have with the ai systems that we
don't have with humans
so like interpretability um is a good
example
where you can just like you know read
off the literal activations and weights
of a neural net
like they're not super understandable
but at least you have access to them
you can't do that with human brains it's
it's kind of rough
um and then there's like you know in
if you're familiar with ai safety via
debate one of the tools
and there is cross-examination which is
you you know when your air system is
like conducting a debate
sorry um
our dog is very bark happy but so yeah
air safety via debate cross-examination
when
you um
when a debater is like sort of making
some sort of argument you
before they make the argument and see
counter arguments you like save a copy
of them
and then uh after you know you've had
the argument you can be like i think
you're just you know
using this concept in in different ways
unlike if i had said something else you
would have like uh
used it you would have used a different
interpretation of that concept
and then you can just take the saved one
and be like what do you mean by this
concept
um and they have to you know answer
without knowing how the rest of the
debate
played out and that's you know obviously
not
another thing you can do with humans um
so
so yeah so you don't always get to apply
alignment tools to humans as well but
when you can it can be quite nice
oh the other thing is like you get to do
like
millions of examples with ai systems and
you can't do that with humans
yeah interesting thank you
okay it it seems like
one one of the like central ideas here
that you're thinking about is
distributional shift uh and i'm
wondering if
you think that's if you think of that as
something
as the kind of thing that we can like
simulate and is the question just like
generating the right synthetic data or
is it like something weirder and is it
going to be like harder to solve
um i don't think it is as simple as um
just doing the right synthetic data
so like for example it's it's hard to
say how you
like generate the right synthetic data
for like
the ai system is beli is like
more powerful than humanity and could
take over the world if it wanted
like if that's not true the ai system
might pick up
on that which presumably and if it is
true then the a system might actually
just take over the world and kill us all
um so there definitely does seem to be
some amount of like
it's not as simple as it um or like
even if we magically did know the
deployment distribution which i don't
think we
which i think is like too strong
of a requirement we are not going to be
able to do that like
it requires you to like predict coven 19
in advance for example
um and just like you know the
little like billion other things that
could have happened that would have been
like
uh that were as i'm at similar-ish to
covet 19.
um that's already hard but then even if
you could somehow do that i feel like
the
the like agent becoming more powerful
than humanity one in particular feels
quite hard
on on the chat um pre-registering terms
is a good technique
but the debate one is is actually quite
a lot more powerful because you can
do the pre-registration after seeing how
the debate plays out
which is can actually be quite powerful
like i i don't know if you've like often
people will like
look at how forecasting questions
resolve and then be like man i wish i
had pre-registered
such and such term but i just didn't
think about it beforehand
if you've saved a copy of the the other
debater you can do that
i might want to present one more slide
uh so just wanted to come back to this
thing that i signed posted
a long time ago so in the sort of bad
generalization
uh risk model so i sort of talked about
why we
don't really want to think about utility
functions instead want to think about
what generalization guarantees we can we
can make
and so like one thing that you could try
to do
is have your models get the right
answers for the right reasons
not just get the right answers um which
is sort of
the default and things like
infor informed oversight and iterated
amplification and debate are like
pretty are like trying to do that sort
of thing
and then another thing you can try to do
is sort of like the synthetic data
our idea is like you know just search
for
inputs on which your ai system would do
something bad
um and that's sort of what adversarial
training is and then
interpretability is just like man if you
just understood how your system was
reasoning
then then maybe you could notice when it
was going to do something bad
so it's sort of all those are the sorts
of things i would
um highlight as ways that you can
try to tackle the sort of bad
generalization problem
i should also skip the slide
in which case we are takeaways which i
will just leave on
the screen but yes cool thanks everyone
thanks for joining everyone our next
talk is in one week
on april 20th at 1700 utc from dr andrew
kritsch
who is a full-time research scientist at
uc berkeley at stuart russell's center
for
human compatible ai his talk title is
open source game theory
what ai and decentralized tech need to
learn from each other
if you're watching on youtube please
remember to hit the like and subscribe
buttons
and share this video if you want the
zoom link to join future talks you can
join our mailing list in the footer at
research.protocol.ai
thanks for joining everyone we will see
you next time