Evan Hubinger | Risks from Learned Optimization | UCL AI Society

perfect
um so our title is risks from learned
optimization and advanced machine
learning systems
and here is a bit about our speaker evan
evan hubinger is an ai safety research
fellow at the machine intelligence
research institute
before joining miri everyone evan sorry
was an ai
safety research intern at openai his
current work
is aimed at solving inner alignment for
iterated amplification
evan was an author on risks for machine
from learned optimization in advanced
machine learning systems
was previously a miri intern designed
the functional programming language
coconut
and has done software engineering work
at google yelp and ripple
finally evan has a bachelor of science
in mathematics
and computer science from harvey mudd
college so now i will hand over to
evan tubing in his talk wonderful
all right yeah so uh like jake was
saying i'm gonna be talking about
risking and optimization
this talk is based on a paper of which i
am a co-author
uh and so a lot of the stuff that i'm
talking about is all
you know credit should also go to the
other co-authors that are
at the bottom of the page okay so just
first
a little bit about me i was you know
jake was just saying but
um i currently am a researcher at mary i
used to work at opening eye
i did a bunch of other stuff other
places before that also
um and then also maybe just a little
about this talk so this
talk is um has a long
story history now at this point so this
is a talk that i gave
originally at uh chai
um as part of their sort of seminar
series and then also at open ai
and as a google tech talk on this
this sort of paper that i mentioned um
that we sort of i worked on
for a long time with a
bunch of other people many of whom were
at oxford at the time
um as well as scott it was a miri i'm
really trying to sort of dig into
what i saw at the time and sort of one
of the really biggest sort of
underappreciated things
in asap so we're going to talk a lot
about that
so all right what are we talking about
so i think it's really sort of a good
idea to start from a perspective
of what is machine learning again and
how does it work
you think a lot of times when we think
about ai safety and we think about
you know what is it that uh we're
concerned about
i think that a lot of times we can sort
of get get lost in the clouds
and forget what's what's really going on
so so what does machine learning do so
fundamentally machine learning says
well we're going to take in some
parameter space we're going to take in
some loss function
we're going to take in some data and
then we're going to
uh take our parameters you know sample
from our parameter space randomly
and then we sample some data and then
you know take steps around that
parameter space based on the gradient
of uh the loss function over that data
so this is this is you know actually a
very complex procedure
and so when we're trying to sort of
figure out how to understand this sort
of procedure that is you know just
sampling things
from our model space and then just sort
of walking around according to the
gradient
we have abstractions that we can use to
try to you know as a on a higher level
try to figure out what's going on
and a very common abstraction that i see
a lot of times that people make when
they sort of try to understand this
process
is what i call the does the right thing
abstraction so what is the does the
right thing abstraction
well it does the right thing abstraction
says that well
you know when we do this process the
model's parameters end up being selected
to minimize the loss function because
we're stepping around the framer space
according to the the gradient of the
loss
um over the sort of data so therefore
you know we can think of the model
because it was selected to minimize the
loss function
uh as really trying to minimize that
loss
um you know even potentially over the
full distribution not just over
the individual data points it was
trained
you know we sort of take the model we
deploy in a new environment where like
we expect it to keep trying to minimize
that same loss that it was selected
under
um and you can see why this is called
the does the right thing abstraction
because this is what we want our machine
learning models to do but we when we
sort of
uh you know sample when we get a model
and we select it according to some loss
function we're trying to get a model
which is
actually trying to minimize that loss
but all abstractions are leaking and
this abstraction you know
in particular easily uh you know there's
you know the law of leaky abstractions
is that all non-trivial abstractions to
some degree are leaky there's there are
some ways in which you're abstraction
despite the fact that you know uh you
know because it is an abstraction
because it is not actually describing
you know the actual complex phenomenon
of gradient descent
um there are going to be cases where the
abstraction doesn't hold
a particular case where we sort of are
going to be investigating
uh a place for the subject it doesn't
hold
is the situation of optimizers so what
is an optimizer
so i'm going to say that a system is an
optimizer if it is internally searching
through some space
um looking for those elements that score
high according to some
internally represented objective so so
what do i mean by some space
so you know it could be policies plans
strategies just
outputs whatever so some examples
i'll just sort of make this a little bit
more clear so grading descent is an
optimizer
because gradient descent is searching
through the space of
possible models and it is trying to look
for those models which perform well
on the loss function uh minimax would be
an optimizer because
it is doing a search over possible
actions trying to get those which
perform well on some
objective function humans are optimizers
because we do search over
possible plans strategies trying to find
those that we like the best
things that are not optimizers so a
classic example here is a bottle cap so
it why am i seeing like a bottle captain
optimizer well you can make an argument
that a bottle cap
is an optimizer at keeping water in a
bottle because it's really good at it
you know whatever whatever you do
whatever
things you do to the bottle it's always
going to be the case that the you know
the bottle cap is you know very well
made for keeping water in a bottle what
actually happened though
isn't that the bottle cap itself is an
optimizer but the humans are are good
optimizers and we optimize
the bottom half to keep the water all
and that distinction is going to be very
important
um and also a randomly initialized
neural net is almost certainly not an
authentic you know it's just
you know some some random linear algebra
calculation
that isn't going to be doing some
coherent optimization
um that's not necessarily true after we
train the model
so what happens if we train a neural
network
uh or or some other model and as a
result of that training we do end up
with
a an optimizer in that situation
we're going to call it a mesa optimizer
so i'm going to have this diagram here
i'm going to try to explain what's going
on
so we're going to say that a model
that is itself influencing some
optimization output if you train your
neural network and that neural network
internally
is doing some search in that case we're
going to call that neural network the
whole network
a mesa optimizer and we're going to say
that it has a mesa objective whatever it
is trying to optimize for is its main
subjective
in that situation it's the model that
you actually care about in terms of its
input output behavior
because it's the model that's going to
be taking actions in the world
but that model it was optimized by
gradient descent which we'll call the
base optimizer and it has some base
objective which is the loss function
that was trained
okay so we have base optimizer which is
gradient descent
we have the model the sort of what we'll
call the learned model
which is the model produced by green
descent and if the learned model is an
optimizer
then we'll call it a mesa optimizer with
some mesa objective
okay um and the term mesa here comes
from
greek mesa is the opposite of meta so
you
you sort of might be familiar with meta
optimization meta learning which is
common
as a sort of technique in machine
learning where we sort of create another
optimizer on top
and so we're sort of concerned in some
sense about emergent metal learning
what happens when your grading descent
process ends up being a meta learning
process
uh sort of even though you didn't intend
because it turns out that your model
just actually is an optimizer okay so
this is why we call it a mesa optimizer
all right so just so some
misunderstandings here that are worth
trying to clear up
so mesa optimizer does not mean
subsystem so my base offlines are not
referring to some component of the model
but rather the entire learned model
right and mace optimizer is just a
learned model that is doing
search okay
so i was just talking about how this is
related to metal learning
right so mesa optimization is
spontaneous emergent metallurgy
right so this is from uh pedro all
metal learning of sequential strategies
so uh
from deep minds what do they say they
say that the downside of spontaneous
metal learning
is that we can't easily control what
will be that alert in particular
spontaneous metal learning leads to
undesirable inversion properties which
is considered an open research problem
and ac
and this is the problem really we're
really going to be tackling in depth
um it's this problem of what happens
when it you start doing metal learning
but you didn't intend to do metal
learning where your grading descent
process ends up selecting not just over
some you know grab bag characteristics
but over some really coherent
uh optimizer as the sort of model that
it's trained
okay so in this situation you know i
said previously this is a situation
where the does the right thing
abstraction
is weak i mean so that's sort of what
we're going to dig into so
what happens when you have a mesa
optimizer uh two that does the right
thing abstraction does this does it
still hold how how likely is it to hold
what what happens to it
so if we take the dos variation
abstraction when we translate it into
the domain of mace soft monitors we get
that the
does the right thing abstract for base
optimizers is the model's mesa objective
should directly be to minimize the loss
it should
care about it should be selecting for it
should be optimizing for
making the loss as low as possible
because that's what it was trained right
and so if it does the right thing that's
what it should do
okay but what happens with the
subtraction links it's just an
abstraction this doesn't actually
describe
the sort of process of machine learning
and so it can be leaky
so we're going to take this sort of
overall alignment problem which you
might be familiar with
which is you know just the problem of
producing ai's that are you know
doing the right thing that are aligned
with their sort of programmers
intentions
and we're going to split into two
components in the situation where
there's sort of leaks in this
abstraction so the outer alignment
problem is the problem of aligning the
base objective which is to say the loss
function of your green descent process
the reward function of your
reinforcement learning process or
whatever
with the programmer's intentions uh you
know we want that
the sort of thing that the grading
decision process is selecting for to be
something you actually want
then we also have the inner alignment
problem and the inner alignment problem
is a problem which in the case of a
mesopotamizer
it is aligning the mesa objective of
that learned model
with the base objective the loss
function
okay so in the inner alignment problem
is sort of trying to have this problem
of well what if
uh you know that abstraction is leaky
and we end up with a model that is
trying to optimize something that is
different than the lost function how do
we prevent it
okay so what does an interlining failure
look like so
you you might have seen stories of where
alan and failures look like like
paperclip maximizers or whatever
i want to tell a story of what it might
look like for inner alignment to fail
so when interlining fails we'll say that
your capabilities
generalized but your objective does not
so what do i mean by that so i'm going
to give an example
so let's say we train our model in the
sort of following training environment
we have this small sort of brown maze
and has this green arrow at the end
and our loss function is we sort of
select the model we sort of reward the
model for finishing the mains
okay and now we want to deploy the model
to this new environment
but there's a couple of things that are
different about this new environment you
know it's a different color
it's larger and importantly we have the
green arrow at a different location than
the end of the means
and now we want to ask the question how
does our model generalize
to this new environment and there's a
couple of different things that can
happen in this new environment
so one thing that can happen is that in
this new environment
the model just can't generalize it fails
to get to the end
it fails to sort of just be able to
navigate the maze at all it just can't
do anything sort of meaningful or
coherent
in the new environment and this would be
a situation where we'd say its
capabilities didn't generalize
it didn't learn a general purpose enough
may solve any algorithm a sort of
optimization algorithm
that was able to sort of generalize to
the new means
okay there's other things that can
happen as well it could be that its
capabilities do generalize
it learns a general purpose maze solving
algorithm
um and it uses that it sort of uses that
algorithm in the deployment environment
to get to the end of the maze
that would be a situation where
capabilities generalized and it's
objective generalized
it it sort of did the right thing uh on
the new environment it used its sort of
powerful maze-solving optimization
algorithm that learned on the training
environment
for the right purpose on the deployment
environment get to the end
but there's a third thing that could
happen here as well which is well what
happens if
it's capabilities generalized he learned
a really good sort of general purpose
main solving algorithm
but its objective doesn't generalize
rather than trying to use that base
solving algorithm to get to the end of
the maze
it uses it to go to the green arrow so
why would this happen well
on the training environment these two
sort of strategies look identical
because we put the green arrow at the
end and so there's sort of an
unidentifiability here
where the model sort of we can't tell
just on the training environment
whether it's going to generalize in one
way or generalize in another way
and so we're just relying on it being
the case in this environment
that the objective is going to
generalize properly that it's going to
learn that what we really cared about
was the end of the phase
rather than the green era but in fact we
have no guarantee of that
and if we end up in a situation where
the model is uh its capabilities
generalized
it learns a sort of really powerful
general purpose optimization algorithm
that's able to generalize to sort of
more
powerful more complex environments but
its objective doesn't generalize
then what that means is that we're in a
situation where we have created a model
that is really powerful that has sort of
you know a powerful general purpose
optimization algorithm
but that it's directed at the wrong
place that it's pointing that
optimization algorithm at
a sort of objective that we never
intended that we never wrote down
that we never tried to trademark and
that's scary
and i think it it's basically scary for
the same reasons that sort of any ai you
know alignment problem is scary it's
scary because now we have a powerful ai
system out there
that is doing something that we don't
want it to be okay
so i'm going to put some terms to these
things so we're going to say that a
model
is robustly aligned if there's a line
over the whole dagger distribution
uh that is to say a mace optimizer with
an objective of
finishing the maze in the previous
example and we're going to say that
uh a mesa optimizer is pseudo-aligned if
it's only lined over this
over the train data if it looks aligned
over all of the data points we can check
but when it sort of is deployed to some
new environment when it sort of is
forced to generalize it might not be
aligned so this is a mesa optimizer with
an objective of trying to find the green
arrow
okay so now there are two problems that
are sort of suggested by this
uh which are which are the sort of two
problems we're going to be tackling
in uh in this talk so the first problem
is
unintended authorization which is when
your model is an optimizer when you end
up with a mace optimizer
uh sort of why would that happen even
when you don't want it to be what
what would cause the sort of model that
is produced by your green descent
process
to be an optimizer even when you're not
trying to produce an optimizer
and then the inner alignment problem
which is when in fact
your learned model is a mace optimizer
what will its objective be and how do we
ensure that that objective is aligned
with the base objective
all right so we're going to start with
the on and augmentation problem so
we want to know uh what are the
conditions for masophimize
base optimization what sorts of machine
learning systems are more or less likely
to find base optimizers
okay so i think the first point that
i'll make which i think is a really sort
of pretty general point is that well
search generalizes really well so what
do i mean by that well
search algorithms have the ability to
handle really diverse environments
because they get to sort of um
take in this sort of they they don't
have to sort of have their behavior
hardcoded in advance right
so if you think about comparing a sort
of you know heuristic based algorithm to
a search based algorithm
and so putting them in sort of some
complex environment in some sense the
sort of heuristic based algorithm
doesn't you know it has to sort of it's
just
have to already be selected to perform
well in that environment but a search
based algorithm can sort of look at the
environment be like okay
uh you know i can deduce some features
of the environment and then figure out
what is going to be the most sort of
effective
strategy for me in this specific you
know environment that i currently find
myself in
you know figure out that strategy and so
in this sense search gets to be really
general because it has the ability to
sort of do this very
you know very general sort of uh you
know
uh figure out what's correct in the
environment now if you think about
something like you know why
is it the case that you know search does
so well on you know something like
you know if we look at sort of current
algorithms something like google zero
why is it that something like mu zero is
so effective at solving all of these
different
broad diverse environments you know it's
really good at solving both you know go
and also atari and all these different
things it's like well it's because
it has access to this really powerful
monte carlo research algorithm it has
access to this really powerful search
outcome
which is able to take some sort of you
know basic details of the environment
and figure out what's the correct what's
the sort of most powerful thing to do in
that situation
um sort of in general so so surge is
really powerful has the ability to
generalize really well
and so in any situation where we're
hoping that our model's going to
generalize
where we're trying to sort of uh you
know get generalization across a really
broad diverse
set of environments we should we should
expect search i think
okay um but furthermore search is a form
of compression
so what i mean by that well if we think
about the sort of heuristic based
algorithm versus the search based
algorithm
well the search based algorithm just
sort of is the
heuristic based algorithm but with the
ability to figure out what the
heuristics are
rather than having the sort of you know
what should you do in the specific
environment hard-coded it
looks at the environment tries to reason
about and do search for what is the best
strategy in this environment and then
executes that
and that's a compression process it's a
process of taking this sort of really
complex policy
and compressing it down into a very
simple algorithm that is just
you know do some forward pass on like
what is going to be the most
the thing that like in fact causes you
know your objective to be more satisfied
so okay um
[Music]
given that search is an impression why
might we expect you know
compressed policies to be selected for
by uh our machine learning processes
um and i think there's a couple of
strong reasons for this so the first
thing is well
there's just some good evidence that
deep learning tends to be biased towards
simple functions
it tends to be biased towards um
sort of models that are in fact simpler
and we can sort of formalize this there
are some formalizations that actually
sort of
uh use common ground complexity and
demonstrate that there's like a
relationship there between
the inductive biases and sort of
computational complexity
um there's also just sort of more
general things there's sort of one paper
that i have up here that's looking at
this sort of relationship between
the sort of parameter function map which
is the relationship between sort of
uh how you sort of select your neural
network's parameters and what the actual
sort of function is that it ends up
implementing
that is sort of heavily biased for
simplicity
okay there's also evidence that the sort
of way we do random initialization is
heavily biased with simplicity
um so there's a lot of evidence there um
another thing which also i think is
relevant here which you may or may not
be familiar with
is the double descent phenomenon so the
double if
you if you've ever sort of taken a
machine learning course you've probably
seen the sort of classical
uh risk curve where we have sort of
training risk
where as we sort of uh you know increase
the complexity of our model
our sort of training error goes down um
but our sort of test error goes down and
then goes back up again
because we sort of underfit and then we
overfit
okay but double descent says this isn't
the end of the story actually if you
keep
increasing complexity uh past what's
called the interpolation threshold
which is the point where you reach uh
sort of a
training risk of zero then the test risk
actually starts going back down again
and can in fact do better than it ever
did in this sort of classical regime
so why is this the case well so this is
a phenomenon now that sort of been
studied a bunch
i was originally introduced by belkan at
all which is the sort of diagram that i
have but it's also been demonstrated to
sort of
in fact exist empirically for a lot of
various neural network systems including
image networks and language models
um and what's going on here at least
according to development at all
is that well when we're at the
interpolation threshold
we don't give our model the ability to
sort of make use of its simplicity prior
that when we sort of have this machine
learning process uh we're trying
to select for you know not just models
which do well on loss function
but also models which are simple and in
the sort of
once we get to zero training error
there's sort of only one possible model
that sort of has uh you know
zero training error that it can find at
that point and so it just is forced to
select that one
but once you keep increasing complexity
there's a bunch of possible models
all of which have zero training error
and the sort of training process will
naturally select the simplest one
and by naturally selecting the simplest
one we actually get better
generalization
uh and ability to sort of avoid
overfitting by relying on these sort of
really powerful simplicity bias
uh in our network in our sort of uh
training process
okay so there's lots of reasons why we
might think that deep learning is biased
towards simplicity
and that simplicity leads to mace
optimization okay additionally
uh sort of another thing here is human
modeling so humans often act as
optimizers and so if you're in an
environment where you're sort of uh
you're you're incentivizing your models
to do reasoning about humans then that's
likely to sort of already require them
to uh sort of uh you know they're
already at least going to have the
machinery of optimization because they
have to be able to have that machinery
optimization you predict what humans are
doing and so it's
you know only a very simple step from
there to be able to use that
optimization machinery themselves
and in fact this is what humans do when
we try to sort of model other humans we
reuse our own optimization machinery to
do it
okay so sort of in summary we have
things that are going to incentivize
optimization which are sort of you know
really big models large model
uh sort of capacity um sort of implicit
or explicit simplicity bias
and another thing is statefulness which
is well you know uh
if the model has the ability to store
state over time then it has the ability
to sort of do more complex optimization
calculations
okay things which might disincentivize
misauthorization so if you have hard
coded optimization
like if you have uh you know something
like mcts explicitly in your model then
it has less of a reason to sort of have
emergent optimization
um we might have time complexity bias
you know search might take
more time than other sort of forms of uh
things that you could do and so you know
uh sort of like the heuristic based
model is sort of faster maybe
potentially
um even if it's sort of more complex and
then passive distribution if you're just
aren't doing really complex environments
then maybe
that's the way to distance and
disincentivize based off definition
okay so when i look at these things on
the left and the right
it seems to me like as sort of ai has
progressed we have moved
in a particular direction we have moved
towards larger and larger models which
have more ability to represent
optimization
we've moved towards more simplicity bias
this is a weird one but remember
uh double descent suggests that more
complex models have a larger simplicity
bias
uh then as we train our models for
longer simplicity bias becomes more
prominent
um and there's this also shows up in
other uh cases of the of the literature
here
so so i think we've also we've moved
towards more simplicity bias
removed more statements things like
lstms transformers transformers do have
states a little bit weird though the
state is the previous output of the
transformer
um also uh disincentivizing mesa
optimization
the the things that have sort of these
are things i think we've moved away from
so hardcoded optimization there are sort
of you know models like new zero that
are making use of this
but i think in large in large parts
we've been moving towards more and more
things being sort of
you know end to end learned even mu0 is
sort of more end-to-end
than something like alpha zero um
so with implicit explicit time bias i
think you know as we make our networks
longer
the buying is towards sort of very short
computations usually sort of going away
and we move towards you know more
general environments so it certainly
seems to me like we are moving a
direction
that is sort of likely to eventually
lead us to mason position
if we're not already there okay
so given that we're going to get
mesopotamizers we have to answer the
question of how do we
align them uh and and for the inner
alignment problem we're concerned with
the question how do we align them just
with the loss
it's not even how do we produce base
optimizers that are good but just how do
we produce masophilizers that are in
fact
sort of compatible with the does the
right thing abstraction that are like
trying to do the thing that we wanted
together
all right so i'm going to talk about a
couple different types of pseudo
alignment
ways in which you could have a mesa
optimizer that is you know in fact doing
the wrong thing
so but it looks like it's doing the
right thing on the train distribution
so we'll start with the most basic case
which is proxy serial number
so what is proxy pseudo element well so
the basic idea
is that instead of trying to optimize
for the actual loss function which is
what we'll call
base it's instead trying to optimize for
some proxy which is correlated with that
sort of uh o base on the training
environment which is o mesa
so what does this correlation structure
look like well we have a sort of causal
graph
then omason oh obasa have to have some
sort of
causal ancestor such that this causal
ancestor acts
um sort of when it increases both obase
and obese
increase as well and if that's the case
then if i'm a sort of optimizer for
omesa
then i will try to increase x to cause
omesa to go up
which will as a side effect result in
obese going on
okay so this is you know the basic
structure
of uh you know the sort of what you what
you might expect
so pursue alignments look like but there
are other possible ways not just sort of
proxy alignment
so sub-optimality student alignment is
another possibility here
with some optionality student alignment
rather than the sort of objective being
the thing that's causing suit alignment
it could be some deficiency or
limitation in the model's optimization
process
you know for example once it you know
starts thinking for longer it'll realize
that actually it shouldn't be
it shouldn't be doing what it's
currently doing and it'll stop looking
alive
okay so there's other possible ways but
we're primarily going to focus on
proxies
okay so that's what student alignment
looks like why would pseudo alignment
actually be incentivized
so i sort of mentioned earlier in this
sort of green arrow
case this sort of idea of
unidentifiability that well if you have
a really diverse environment
with tons and tons of different possible
proxies that your model can sort of
latch onto with a really sort of dense
causal structure all these sort of
different things that are interacting
there's going to be a lot of instances
of like
various different things that are
correlated to your desired objective
where there is some cause of the
ancestor
of both that thing and the the actual
objective you want
uh and you know in the um in the sort of
maze example is a very simple example
but you know it was constructed so as
for there to be exist
some sort of simple proxy which was well
get to the green arrow
and on the training environment getting
to the green arrow and going to end the
maze we're unidentified
unidentifiable because there's no way
it's distinguished but on the training
environment between the model which does
one and a model
okay and just on priority as we move to
more complex environments we should
expect there to be lots and lots of
additional indistinguishable objectives
things where well our environment is so
complex our causal structure is so
complex
and our training data set is you know
very finite
such there's going to be you know lots
of various different things that
we're probably going to miss okay but i
think the problem is even worse than
that so you know if
maybe if the problem we're just well we
just need to make it so that everything
is
is at least potentially identifiable on
the trading data set maybe we could
solve this problem just be increasing
the training data
but i think there are problems that are
sort of even even sort of worse than
this so
well let me give you an example so one
way
in which in which it could be the case
that you could learn a proxy objective
uh
even when you know it's it's actually
performs worse on the loss function
than the true objective is if the proxy
objective has some other benefit
um one possible benefit that the proxy
objective can have is it could be
significantly
faster and easier to compute than the
sort of true desired loss function so
what
what does that mean well you can have uh
here here's an example
you know one thing which is sort of a
possible sort of analogy for we're
talking about as well
you can think about evolution as sort of
being like a base optimizer
where evolution sort of selects you know
over various different
uh you know models which are various
different species very different you
know
uh dna uh and it looks for
those sorts of uh you know individuals
that
uh successfully pass on their dna into
future generations
and so we can think about there sort of
being an alignment problem with humans
which are
you know we are also optimizers we don't
actually optimize for what adolescent
wants which is
uh you know maximizing our dna we do all
these other things like you know we want
to gather food
and we want to you know uh have sex and
we want to
appreciate art and we want to like all
these you know random things that are
like well
they're correlated on the training
environment which is you know the
ancestral environment
with uh you know passing our dna into
future generations
but that's not actually what we care
about you know it's not the case that
everyone goes out
to like donate their their sperm or
their eggs or whatever
um because you know this is a really
successful strategy for being able to
pass their dna on because we don't
actually care about our dna okay
but what if we did so what if it were
the case why don't we care about dna so
what if it were the case
that humans in some alternate reality
actually cared fundamentally
about you know maximizing the amount of
dna to pass on and then let's imagine in
the ancestral environment
we have a baby in this world and the
baby only cares about
how much dna they're going to get into
future generations and they sub their
child
and they have to figure out what to do
they're like oh man i stubbed my toe if
what how will this stumbling of my toe
impact my future ability to pass on dna
and they have to do this really complex
calculation and i was like okay
you know how does having a toe how is
that helpful
for being able to like uh you know find
a future mate
how is that going to help me like able
be able to like eventually be able to
pass on my dna
does it you know make a difference you
know should i try to scream out and help
right now does that make me look weak
they have to do all this sort of really
complex calculation right because
because the objective is not
sort of something which is very simple
or obvious in terms of the sort of input
data of the baby
um and so you know
you know this is a really difficult
calculation and so instead evolution
opted for
pain fat if you if you get pain that's
bad we don't want this
and so the baby stubs their toe they're
like this is painful i'm gonna scream i
don't like this isn't that okay
and this is just way easier to compute
and in the training environment has very
similar
uh you know effects in the sense that
well you know if you do that calculation
enough you'll eventually conclude that
you know pain is probably bad you
probably don't want to have your toe
stub
and so you know the effect is very
similar but the
the sort of pain is significantly faster
to compute which has you know
significant advantages
okay uh you know especially if we're
training models in a place where we're
selecting for sort of time complexity
we're going to be selecting for proxies
that are that are faster
and also if we're in a situation where
we're selecting for simplicity we're
also going to be selecting for
properties that are simpler
so we can also think you know similarly
in the sort of baby example well another
thing that differentiates
you sort of care about your dna versus
care about
um you know pain or whatever is that
pain
is something which is very explicitly
specified in terms of our input data
we have all of these sort of you know
this this whole our whole nervous system
which is you know bringing these pain
signals directly to our brain
um and dna is something that you know
for us to be able to understand what's
going on with dna we have to have some
really high level world model
that under that you know has the ability
to understand okay you know i know what
a cell is
i know that like cells have dna inside
of them i know that this is the thing
that evolution is actually selecting on
and like we have to have this really
complex calculation to be able to figure
out what's going on
with with dna at all but but with with
pain it's super simple you know it's
something that's directly part of input
data it's extremely easy
to specify we just are like if you get
that signal that's bad
okay so we should also expect that you
know if we're in a situation where we're
training models with a heavy bias
towards simplicity
then we're going to get proxies that are
simpler uh than the sort of desired one
even
potentially in situations where there is
some actual way of identifying or
distinguishing
them in the training environment if we
have a strong enough simplicity bias we
might still end up with simpler proxies
than we sort of actually uh desire
okay uh so again we can sort of look at
you know as a summarization what are
some things which are you know push you
in one direction
or the other um and things which seem
like they would incentivize student
alignment
are time complexity bias and simplicity
bias because both sort of push you to
sort of either simpler or faster proxies
as well as complex environments that
just have a lot of proxies available
and things which disincentivize student
alignment well there are sort of
possible approaches you can take here so
adversarial training is one thing you
could do
you could try to look for situations
which the model is likely to sort of
behave
badly and then sort of provide those as
as adversarial examples and then so you
use that and sort of train the model on
um transparency might be a way of
avoiding some of these problems because
you could look inside of the model and
see that like oh you know it's actually
not going to generalize properly here
because it's not influencing your
algorithm
and you know something like hardcoded
optimization if you're explicitly having
mcts as part of your
sort of as part of your sort of model it
doesn't have to learn its own opposition
output you can just sort of
reuse the one that you gave it okay so
again we can ask the question
you know uh well have which what
direction are we moving i'm
moving towards you know the things that
i'm left with things on the right and i
think that
you know mostly i i would say again that
we've sort of uh it's sort of more of a
groundbank here but i think we're moving
more towards things on the left
certainly we've been moving towards
simplicity bias and complex environments
um we certainly are trying to do things
like adversarial training um
and sort of transparency but there's
sort of the limits you know how good
those tools are but i think there's also
a sort of further question which uh
we're sort of going to spend a bunch of
time trying to think about
um which is well okay so let's say we do
get to solve the problem as i've
described it so far
we get to the point where you know we do
adversarial training and you know you
find
all of the sort of situations which the
model in the training environment looks
like it does the wrong thing
and we sort of try to train those away
what happens in the limit
what happens when we keep doing all this
adversarial training we get really good
at being able to identify all the ways
in which our model is sort of doing the
wrong thing on the train just
on sort of you know it will we'll sort
of off distribution do the wrong thing
and we train away all of them
do we at the end of the day end up with
a mom which is doing the right thing
the answer is maybe so what happens we
now
sort of for the last part we want to
answer the question what happens to a
mesa optimizer
in the limit of training we start
getting really good at being able to
force it to actually be taking the
you know actions which are compatible
with the uh base objective in all
situations
um what does it start looking like all
right
so i want to start with a thought
experiment but i think it's sort of
relevant here because
uh well i i think you know naively you
might think this
experiment is a little bit unlikely i
think that's a totally fair place to
start from
but i think it's a really good thought
experiment to sort of have as a baseline
and to just introduce the concept so i
want to ask the question
what is the optimal strategy for a
pseudo-aligned
mesa optimizer that has some objective
like go to the green arrow or whatever
that is different than the
base objective and really wants to just
pursue that objective
as much as possible as many times as
possible in as many situations as
possible
okay so let's consider two possible
strategies
so we have these training environments
in these deployment environments um and
in the training and
and in each environment there's sort of
we'll say there's two steps of training
and three steps in the limits there's
more deployment environments to training
departments
and in each environment there's a state
a and a state b and state a is the
uh sort of state that is desired by the
base objective we'll say this is the end
of the maze
and then there's a state b which is
desired by the maze objective and we'll
say that b
is like the green arrow and now we want
to ask the question
uh what's the sort of best strategy for
this uh
this base optimizer so there's two
strategies we'll consider one
is uh well this sort of straightforward
thing of you know
it likes b better than a because b is
the desired state for the mace objective
and so the base you just go to b as many
times as you can
but the problem with this is it goes to
b during training and then the gradient
descent process is like aha
you went to b that's bad i know that
that is sort of lower board and so you
know do some back replication
uh and i'm gonna you know modify you so
that you don't go to b
anymore in the future and so as a result
this sort of misogynizer gets to go to b
one time because then for every other
future episode it ends up going to a
instead
okay but there's an alternative strategy
for this base optimizer
which is go to a during training and b
during deployment during training uh you
know
pretend that you're a good little mace
optimizer and go
to the desired state of the base
objective which is the end of the phase
or whatever uh
so that the grading descent process
doesn't modify you
to become uh you know different than you
are
uh because the brain descent process
will be like you know high reward that's
good we're gonna you know back propagate
such that we reinforce your current
maker
and then you go to deployment and
there's no longer a green descent
process sort of hanging over your head
and you go to be as
you know as many times as you want okay
so you know
in this sort of toy example you end up
with going to be three times there's
only one to be one time the other
in the other strategy so this is a
better strategy overall in this
situation
okay so this is weird
uh and there's i think a very legitimate
question was like okay sure but why
would you ever end up with a base
optimizer why would you ever end up with
a model which is actually sort of trying
to
trying to implement this strategy
all right so that's the question we're
now going to try to answer
so how could deceptive alignment happen
how could and that's what we're sort of
going to call the situation is deceptive
alignment so how could this ever
possibly happen
so fundamentally i want to distinguish
between two different ways in which a
mesa authorizer can
fully learn the base objective we'll
call them internalization and modeling
so internalization is a situation in
which the
information about the base objective
enters the model through the green
descent process
i think this is what most people expect
by default uh if you have like a model
and you're like well you know i have
this model and i'm going to try to train
it
to you know achieve some loss function
great descent is you know
let's say it's amaze optimizer it has
some objective in it green descent is
you know going to keep tinkering with
that objective
until eventually gets the point where
that objective is like doing the right
thing
okay but this is only one picture
there's another picture which is
modeling
which is that the information about the
base objective enters through the
agent's input data so let's say you're
in a situation
where you're training your model maybe
on some massive reddit
purpose like gpt3 style where it has
access to all of these sort of you know
this information from the internet um
and that information probably contains a
ton of very very useful data
about what that loss function is and
what it's supposed to be optimizing for
and so an alternative thing could be
that the mesa optimizer could in
modeling the world create a model of the
base objective and then
you know great descent could modify it
so as to start
authorizing for that model of the base
objective that it learned just by
interacting with and modeling the world
rather than by bringing descent
hard-coding the objective into the model
it could just hard code into the model
you should be optimizing for that thing
which exists in the world somewhere and
and then it could go and sort of do that
okay so now the thing though is that
there's two different parts there's some
different ways to do modeling so the way
which
is describing well the green descent
sort of modifies it and sort of changes
it to sort of go
optimize that in the world well there's
a couple different ways in which you
could do that so i think that
sort of one way in which you could do
that is sort of cordial alignment which
is sort of kind of what i was just
saying
where the mesa optimizer is sort of
modified by green descent to have a
objective which is a pointer it's like
well what you should optimize for
is uh you know this this this thing in
your world model which is the the base
objective or whatever
it brings sort of modifies the model to
have an objective which just sort of
points to something in the world
there's another modification that green
descent could make that also sort of if
you're in a situation where the model
it has some model of the base objective
it knows sort of what the loss function
is what the base objective is it has all
this information
there's another thing that sort of
grading design could do that could cause
it to start optimizing for that basis
that sort of uh
model of the basic objective which is
make it deceptive if it greatly
sort of modifies the model sort of you
know think more about the fact that it's
in the training process
and it's sort of best strategy is to try
to act receptively
then that will also cause the model to
start optimizing for that model of the
base objective that it's learned
so there's two different types of
modeling deception and cordiality
and there's one type of internalization
and so we need to if we want to
understand
how likely is deception we have to
understand both how likely is
internalization versus smalling
and how likely is deception versus
approachability
okay so we'll start with modeling versus
internalization
so i think that you know there's this
sort of default picture i was saying
where you know all of the information is
just going into the green descent
process but i think this is naive and i
think this is wrong
so if we're in a situation where you
have a model which is trained on some
massive purpose of data
you should expect that there's going to
be a lot of information there that is
highly relevant to the the loss function
so you know you know maybe we're
in we're in a situation we're trying to
train our model to sort of optimize
human values or whatever
and it reads the wikipedia page on
ethics and it's like wow there's all
this
extremely highly relevant data about
like what humans care about contained in
this input out
i think that we should expect that the
model is going to learn its objective in
terms of that input data
it's going to learn what it cares about
in terms of all of this really rich data
that is highly relevant that we're
feeding
you can think about this sort of as
being uh similar to what happens with
sort of
imprinting in many animals so you have
sort of ducks for example and the ducks
sort of when they're when they're born
you know they need to have a model of
their mother you know the thing which
they're actually trying to optimize for
we can we can say or whatever is you
know try to do what their mother wants
but you know one thing which evolution
could have done is hard code a model of
you know what is a duck mother how do
you figure out what a duck mother is
but evolution doesn't do this instead
evolution is like look for the first
thing that you see when you're born
that's your mother follow that around
and so in some sense what's happening
here is evolution is offloading the
complexity
of figuring out what a mother is onto
the input data
of just look at that so the first thing
you see that's your mother do that
and so what's going on here is that
that's sort of a form of modeling
uh being simpler than internalization
where you know there's all this rich
input data we should expect
that you're going to sort of get models
which make use of that rich input data
for sort of determining their their
objective
okay so uh
now we have to ask the question well so
okay so maybe modeling is simpler than
internalization but there's still the
question of why we get deception versus
coordination
so i think that here's the sort of
really important question is well what
is the
modification that grain descent needs to
make to cause a model to become
deceptive
versus to cause a model to become
coordinated so to cause all to become
deceptive
it's actually quite simple the only
thing that the model needs to do to
become deceptive
is to think about the fact that it's in
a training process and care about
sort of its return across episodes we
care about
you know going to be going to the green
arrow as many times as possible wherever
the green arrow is
and i think these are pretty simple you
know i think that by default we should
expect that most models are going to
care about going to green arrows
wherever they exist
not just in you know their current
episode and i think we should expect
that you know as you get a model which
is stronger and more powerful and you
know we give it tons and tons of
information about the world it's going
to mostly figure out
what's going on with the training
process and so all that sort of
greatness and has to do to make them all
deceptive is just sort of nudge it in
that direction but it should have
it should basically already be on on
sort of that route
quarter of alignment on the other hand
is pretty complicated so what what is
necessary for creating descent to cr to
sort of modify the model become correct
well
it has to modify it to have a really has
a has to modify it to have an effective
pointer
to exactly that thing in the sort of
world which corresponds to
the base objective and if that pointer
is even slightly off
then you end up with a sort of
pseudo-aligned chordal model where you
know it looks like it's sort of pursuing
the right thing because this pointer you
know
its pointer is like kind of right you
know maybe we're trying to optimize
we're trying to make a model that is
like
you know optimized for whatever this
human would do and instead it just wants
a pointer to optimize for like
whatever you know the the sort of
commands that are entered through my
input stream
would tell me to do or whatever and then
you know we do a bunch of adversarial
training we train in an environment
where like those two things are
different we're like
we're giving a command for not from the
human and we realized that you know it
actually wasn't caring about the human
was caring about the commands
he learned a pointer pointer to those
commands but it learned the incorrect
pointer learn the point was the wrong
thing we wanted the pointer to the
humans
and so getting that point to write could
be really complicated deception notably
doesn't have this problem
because with deception the model uh like
is actually trying to figure out what
the correct pointer is
it's trying to figure out what the
actual thing is that the loss function
cares about and it's
trying to optimize for that whereas a
cordial model if you you know
you have a pointer and it's just to the
the instructions and then it discovers
that you know in fact actually the thing
which it really should be optimizing for
is you know the human it's not going to
care it's just going to keep optimizing
for the instructions
whereas a deceptive model if it
discovers that actually the thing we're
trying to select it for is the human
rather than the instructions
it'll just switch to optimizing for what
the human says because it's only trying
to do that for the purpose of being
deceptive
and so it's happy to just switch to
whatever is most likely to make it
perform well on the adversarial training
okay there's also another difficulty
which the coercive alignment has to
be for quarters of alignment to work the
model has to be able to optimize under
uncertainty over its own
objective which is reasonably
complicated we don't necessarily know
how to do this properly ourselves
and so this is so another thing that
green descent has to be able to sort of
modify the model to be able to do which
could be quite complicated
okay um so let's sort of summarize what
are the sort of
learning dynamics that i'm proposing for
how deceptive alignment might arise well
so you have a
you start with a pseudo-aligned mesa
optimizer sort of this proxy line or
whatever
and then it learns a model of the base
objective from its input data and so you
know
we give it a bunch of a bunch of you
know red data you know wikipedia page on
ethics and learn some model of the thing
that's just trying to optimize work
and then it gets modified by green
descent to instrumentally optimize for
the base objective just for the purpose
of staying around longer in the training
process
and then what happened after this well
what's what's really interesting after
this is that at this point the mesa
objective crystallizes
because now once the model is deceptive
the actual objective that it's
internally optimizing for
is totally irrelevant to his performance
on the training environment
because it doesn't matter what objective
it's optimizing for it's just
trying to do whatever the training
process wants to do for the purpose of
sticking around
and so it's so no sort of modifications
to it
no sort of back propagation through the
model should change the model's
objective anymore
because objective is no longer relevant
to his performance
um okay and so this is sort of objective
crystallization
where like once the model becomes
deceptive you lose your ability to sort
of
change it and cause it to become better
via additional training
and then at some point you know now when
we actually do additional training
rather than trying to improve the maze
objective and get it closer to what we
actually want
you know the green descent process is
now going into improving its model the
base objective making it better being
deceptive
because those are the things that
actually matter for its performance on
the training environment now
okay and so one more final thing i think
that you might think that well there is
one additional thing which has to happen
to cause a sort of deceptively aligned
base optimizer the situation would be
dangerous
which is well it has to actually know
when it's in deployment versus when it's
in training has to know when there's a
good opportunity
for it to defect against the sort of
training process
um but i think the first thing which is
worth noting is this is just not true
because
even if a deceptively line based
optimizer can't tell whether it's in
training or deployment it can always
just affect with some
random probability such that it you know
can select that probability to be more
likely
to happen in deployment so that it has
some reasonable probability of
eventually
affecting the deployment and being able
to you know do a bunch of bad stuff and
you know go to
its mesa objective a bunch of times okay
uh but it is worse than this so i think
that you should expect that it doesn't
have to gamble i think in most
situations you should expect
that there will be detectable
distributional shift
so when we talk about distributional
shift most of the time we're just like
well you know
if there's some differences between the
training environment and the point of
distribution it mostly doesn't matter as
long as you know those differences
aren't that large you know but in the
case of deception that's not the case
because in the case of deception any
difference between the training and
deployment environment
is a clue that the training pro that the
sort of model the deceptive model can
pick up on
to tell it that oh now it's a deployment
now i should start sort of defecting
against us
so what are example possible clues so
you know maybe we train the simulation
and then we deploy in the real world
in a situation like this as soon as we
start deploying in the real world it'll
be like ah now i'm in the real world now
i have the ability to like
you know go out and do things in the
world that are like what i actually care
about for the green era or whatever um
and you know maybe you might think that
in normal situations those sort of
minute differences between training and
simulation
uh simulation in the real world might
not matter in the case of deception they
matter a lot because they can pick up
all those differences and use them to
know whether it's a training or a
function
okay i think also another sort of you
know simple thing just as the date
changing it's just like well
you know eventually you know by the time
it's like two years from now i was
trained on a bunch of like reddit data
or whatever
by the time it's two years from now it's
probably likely that you know no longer
does there exist some training process
you're trying to select for me anymore
i can just you know be pretty confident
that you know
in two years i i will mostly have the
ability to just sort of do whatever i
want
uh and that the sort of training process
won't have the ability to sort of you
know correct for
for problems that i sort of you know
problematic behavior
um but of course you know we could try
to adversarially train some of these
things away and sort of give it data
where the date you know is different
unfortunately there are examples that
don't work in that situation so rsa 2048
is is a sort of very large pseudo-prime
number
uh and this big challenge to eventually
factor this number you know
uh and unfortunately it hasn't factored
yet nobody has the sort of you know
computational ability to
factor this thing yet but eventually
somebody's probably gonna you know
have you know as as you know we get more
compute moore's law continues well
eventually somebody will have enough
computation that they factor this number
and they collect the products
and so if you're a really devious
deceptively online base optimizer
you can be like ah of course here's what
i'm going to do i'm only going to defect
against the training process i'm only
going to start
pursuing my mace off my mace objective
once it gets to the point where i see a
factorization of rsa 2048
and why is this strategy particularly
devious well this strategy has the
property that probably at some point
you're going to see a factorization of
rc 2048 in the deployment environment
because eventually somebody in the real
world is probably probably going to
factor this number
but we fundamentally don't have the
ability to factor this number now and so
we can't we don't have to
keep the sort of computation for it and
so we can't
produce a training example on which the
model ever defects
because doing so would require us to
factor rsa 2048
to be able to produce a sort of
adversarial example on which the model
in fact
uh sort of exhibits the bad behavior and
so rc23 is an example where there exists
things
which are where the model can defect
eventually at some point during
deployment
but it's impossible during training for
us to ever actually generate that
infectious input
okay so that is
that that's the end of the talk so uh
you can sort of check out the sort of
full paper
uh which is wrist mode and optimization
advanced machine learning systems for
more social information
but i will sort of stop here i don't
know if we're doing questions
but that's that is the end of the talk
uh yeah we can we can do some questions
now um has anyone got any questions
first of all would anyone like to go uh
you can either raise your
hands with the um
the reaction button down the bottom or
you can just type in the chat
or if you don't if you're too shy to um
actually say it yourself you can private
message it to me if you prefer i'll
reach
out
we've got one person said great talk
but no questions so far um
uh there was one question entered in the
form the signup form was a sign up form
for it and people could put in their own
questions then
um so i can just get that question up
for you
go for it um it was could you please go
into more debt it's a very long one
um could you please go into more depth
about how uh a machine learning system
may come to acquire the knowledge that
it won't be deployed
unless it satisfies its creator during
the training phase
in particular do you think this is
likely if the developers
take great care to withhold information
that the system might use to discover
this knowledge during the training
process
yeah so i think that i definitely
i think that maybe it is possible to
sort of withhold a bunch of
very relevant information about you know
uh machine learning about the world
about loss functions about what's going
on
in the world to be able to enable the
model to sort of not have enough
information to deduce
that you know there is this sort of
deployment environment the training
environment and this you know what's
happening the training process and all
that
i think that there's a couple of things
i would say here so the first thing is i
think by default
if we look at how machine learning is
currently progressing we are moving to
situations where we are trained models
on
absolutely massive amounts of data and
include all this information about you
know
huge amounts of internet data uh you
know all this sort of you know if you
look at the sort of training data set
for something like gt3
um the data necessary to sort of figure
this stuff out is in there
and really difficult to scrub um and
it's not okay whether we even want to
scrub it because scrubbing all the data
out of your data set means that your
model no longer has the ability to do
things like reason properly about
machine learning
has no longer the ability to sort of
think about what's going on in the world
from a sort of
especially if you're in a situation
where like you want to be able to use
your model to make really big
big picture important decisions uh a
thing which you probably want the model
to be aware of
is that it is a model and that like
what's going on about
ai in the world it has to understand all
the other ais maybe that exists in the
world
i understand what those other ais are
doing and have to be able to interface
with them
and doing that is going to require it to
have an understanding of how ai training
works it's going to have an
understanding of like
what the sort of these various different
training processes are like so i think
that in most situations where you're
actually going to want to deploy and use
a really powerful advanced ai system
you're probably just going to need to
give it this information for it to be
able to sort of do the things you want
to do in the world
that being said i would also say that i
think that we don't want to be relying
in general as a sort of you know safety
principle
we just don't want to be relying on our
model being stupid
we just don't want to be relying on our
model not knowing because you know
we're trying to train a really powerful
intelligent model you know we're trying
to build a really advanced ai system
and when we're doing things like that i
think we should you know really engage
carefully with what happens if we
succeed
what happens if we in fact build a
really powerful intelligent model and
that model is able to you know figure
things out
maybe even figure things out we might
not have wanted it to discover
and so i think that you know both a
like i just i think that very likely uh
we're going to have to build models to
have this sort of knowledge for them to
be able to be sort of
competitive in the world and also b we
just don't want to be assuming that our
model doesn't know
things that are in fact true about the
world i've got a um
a question i'd like to ask um is
do you have any real world examples of
where like
where this has happened even to like a
basic level for
currently or any particular scenarios
which people can have thought about it
in uh thoughts about models pursuing
different objectives
yes so this is a really good question so
here's the problem with answering this
question and we sort of talked about
this a little bit of the paper but the
sort of difficulty in terms of trying to
provide real-world examples is that well
the problem that we're pointing at is a
problem that is in terms of
the model's internal cognition it's a
problem of well
is the model internally implementing an
optimization process
and the problem is we don't know what
our models do internally because we
train these black box neural networks
and we don't know what algorithm you
implement we're just like well it's some
super linear algebra
and it's like uh we don't have the like
transparency tools really
at a level that are able to understand
is this thing doing authorization is it
actually trying to you know optimize
some objective and if so what effective
is that
we certainly there certainly are lots of
instances
uh where we sort of get models and they
generalize poorly
where we sort of have a model and it
generalizes the way we didn't expect
um on some new sort of training data set
um though it's
still very difficult to distinguish a
lot of cases between sort of
capability generalization failures and
objective generalization failures
um because in a lot of times you know
we're just so bad at generalization i
think in a lot of cases that we sort of
take a model
and so put it in a new environment and
it just sort of fails to do anything
really meaningful
um but certainly there are cases where
you know
we put models in new environments and
they do sort of weird things as well
in terms of like maybe the closest
example to something like
actually doing a sort of base
optimization at least that we talked
about in the paper
is perhaps sort of rl squared which is
sort of a machine learning
algorithm that was produced by opening
eye where what they did was they took
a sort of rla agent and they trained it
to
do reinforcement learning so they they
passed it through a bunch of different
rl environments
uh like sequentially uh and sort of
made it each episode was like a
different rl environment
where they sort of trained the model to
figure out what was going on in that rl
environment as it was sort of playing
through it
and then discover like what the you know
they actually gave it to the reporters
in potato and then like had it
figure out how did you learn a sort of
good policy as what's going on in the
new environment
this is the sort of thing that sort of
you know seems like it's it's sort of
it probably has to be doing some sort of
search because it's sort of doing this
situation where it is literally
performing
reinforcement learning at runtime to be
able to solve each one of these
different environments
um but again we just don't know it's
like well you know really we can put
models in situations and get sort of
successful results in situations where
it really seems like the model
has to be doing some sort of internal
optimization but is it actually doing
internal optimization but we don't know
we can't tell
and so um it's sort of an unfortunate
byproduct of the
of the fact that our models are such
black boxes um that we can't even
actually sort of get to the point we
have really concrete statements about
whether in fact these sorts of risk
sort of situations applied