EleutherAI IRG 220507: Interpretability Research for the Most Important Century

gonna
i guess i should say welcome evan even
though he's a regular here at this point
um evan's been working on a really cool
post to try to motivate further research
and interpretability and communicate its
potential uh
impacts to others so i'll just let him
take it away
cool thanks nick and yeah thanks
everyone for coming um
let's see let me share my screen
uh yeah so as nick's mentioned i've been
working on this uh
sequence uh for about six weeks now
um first i'll just say uh
a lot of people have
given a lot of help with this including
nick
and uh ryan who's on this call and
several other people um
also i was uh taking the agi safety
just getting a bit of feedback there
from somewhere
um
so uh yeah also uh this is serving as my
capstone project for the agi safety
fundamentals curriculum
um
so about this sequence so it's called
interpretability research for the most
important century
and that's a reference to holden
karnovsky's uh
most important century blog series and
it's directly answering this more recent
post he had on the important actionable
research questions for the most
important century
so if you haven't seen this it's kind of
a series of
high level
questions about ai alignment strategy
and things like that which
we have real only really vague notions
of so far you know things like how
difficult is alignment
[Music]
these are questions which you know he
talks about if we had clear answers to
these really ill-defined questions
um
it could make a big difference to
[Music]
ai safety funders knowing where to
allocate money how to do it
also people kind of making efforts to
allocate talent
to certain areas of ai safety and things
like that the specific question i'm
focusing on is where he asks so what
relatively well-scoped research
activities are are likely to be useful
for
long-termism-oriented ai alignment and
so i'm just trying to make the case that
interpretability research is one of
these activities
i don't think anyone else has has tried
to answer this question yet so it's kind
of the first one and um
you know hopefully giving uh
starting the conversation and maybe
providing a bit of
a template or some criteria that we were
where these uh
uh you know different research
activities can be
uh compared side by side but i'm just
focusing on interpretability and trying
to kind of um
explore the case for that
um
so right now this is two posts they're
both uh drafts which
um
i they're unpublished but i'm planning
to get onto the alignment form and less
wrong and
hopefully the next couple weeks
this first one is just an introduction
to the sequence explaining kind of some
of the stuff i've just explained now um
something i want to highlight here is
you know the question is okay what's
uh what are useful research activities
for alignment he kind of elaborates on
this question
um
you know and talks about how
uh a lot of research uh you know either
has this you know
either has this issue where um
it's really difficult for people to ramp
up on it
you know there's a lot of
it requires taking on an unusual world
view or something like that
um
or it uh you know maybe it's hard to see
how directly it can apply to like the
most important parts of the alignment
problem and so he's he's hoping to find
uh
you know research activities which are
both
very valuable to alignment and which we
could uh you know are kind of
straightforward to onboard people too
so
those are kind of the focus points of
the question and that's what i'm trying
to argue for with interpretability
um
this second post is where the meat of
all this is right now and it's only
focusing on the question of the kind of
that sub question of how is
interpretability relevant to alignment
and the hardest parts of alignment um i
haven't
uh it'll be a later post where i talk
about
you know uh the properties of
interpretability uh for kind of
onboarding people and is it
straightforward to
like
throw a lot of talent at this research
direction basically
um so let's get into the
uh post number two which as i said is
the the meat of this right now
um
[Music]
so i hear
exploring how is interpretability
relevant to
hardest most important parts of
alignment i borrow from evan hubinger's
kind of four components of alignment
there's outer alignment
inner alignment training competitiveness
and performance competitiveness a lot of
people here are probably familiar with
those but just a quick kind of refresh
your outer alignment
um
if you've got your model to optimize for
for the base objective if you know if
you got it to optimize for
what you what you're trying to optimize
it for would you like the result you
know or would it kind of go off in some
catastrophic direction that you didn't
think of or something like that
inner alignment
um
is
okay how do we actually get the
alignment to pursue a certain goal just
because you train it with an objective
in mind doesn't mean it actually learns
to
optimize for that objective there's an
influential series called uh paper and
and blog post series risks from learned
optimization which talks a lot about
inner alignment
problems of uh deceptive alignment
other kinds of pseudo alignment
it's a really uh interesting
paper and and we talk about a lot of
those issues here
um
these last two training competitiveness
might be the least familiar or least
intuitive but basically i like how he
breaks up competitiveness into these two
components training competitiveness is
um let's say imagine you're
some ai lab
and
you have a lead over the other ai labs
and you're looking at what kind of
advanced ai proposal to implement
um
okay so what are the training costs of
this you know these different proposals
and if you you know
if you implemented a certain one like
let's say you try to do imitative
amplification or something like that it
could be any number of proposals but
would you you know would the training
process be so costly requires so much
compute etc that you would throw away
your lead over the ai labs
if you actually try to implement that
and so obviously for alignment to
succeed we want you know proposals which
which are competitive in this way where
it wouldn't require like a lead
sacrificing amount of compute and
whatnot performance competitiveness is
more intuitive it's what you would think
of just you know if you if you had an ai
that was implemented in a certain way
you're trying to implement it would it
actually have you know
fill the e be able to fill the economic
niches that people want for agi and um
you know be competitive capability wise
with you know other kinds of ais that
labs are building
so um
how does interpretability impact these
alignment you know these these important
parts of alignment
sorry actually there's another sub
question
which of these which of these is the
hardest so i'm considering these to be
the
important parts of alignment
i think which one is the hardest is a
little more complicated um there's a lot
of proposals which are aligned but not
competitive or they're competitive but
not aligned
i also observe that inner alignment is
particularly a struggling point for
example
there's this uh other influential
post by
uh hubenger
um going over like 11 different
proposals for advanced ai
and
from my reading none of these you know
he talks about how these could end up
being inter-aligned none of them are in
our lines today like using technology
that we have today and um there's a big
dependency on interpretability and kind
of implicitly and explicitly throughout
that post so um
so i think that uh
inner alignment is particularly hard
um
and there's been maybe less work on it
um
so uh i'll be operating as though inner
alignment is kind of the hardest part
but we still pay attention to the other
three
components because it's it's also hard
to just find one which has all these
compo these properties simultaneously
being the both kinds of alignment both
kinds of competitiveness
um
so
interpretability's impact as i was
saying you know
i i think it has a lot and we'll be
talking about that
um
especially on inner alignment but
indirectly on uh outer alignment and
performance competitiveness
um quite often throughout these
scenarios we're going to talk about
um
i think you know sometimes uh
it's it
creates more friction on the training
competitiveness side when you start
implementing like advanced
interpretability tools and processes
training gets more complicated and
costly so that's one one thing to flag
with all this although in some ways it
can make it uh better too
as we'll see
there's some you know things you could
argue that are important for alignment
that i don't really consider here i'm
kind of assuming we're in like a prosaic
ai alignment world which means
that agi will basically look like a
scaled up version of today's deep
learning systems with not you know not
some
extremely radical leap in terms of you
know
a completely different kind of paradigm
of developing
neural networks or ml
i also don't really address the embedded
agency problem except insofar as we talk
a little bit about alex pence ground for
optimization which has some bearing on
it
there's a lot of governance and strategy
problems which you could argue are
important for alignment that mostly
i don't address here most we touch on it
a little bit but you know mostly this is
about
technical alignment technical solutions
the alignment problem
um okay so now we get into the main part
of this where um
in trying to argue that interpretability
has you know
uh is relevant to these difficult parts
the main exercise that i do is this one
which holden suggests in kind of the
list of questions which is to visualize
the best case
um
and so
uh
if you just imagine that like
interpretability wasn't constrained by
financial resources and it had you know
tons of researchers working on it like
how far could we get with
interpretability research because we
want to be able to show
the limit of what it could achieve um
so that when this is compared to
other uh kind of research activities by
funders and whatnot
um
you know they can see okay well you know
we're kind of making the strongest case
and they can compare the strongest case
of both and what's realistic
so um
here we go let's
let's dive into these there's a few
things that are kind of relevant to all
the scenarios that we'll consider
um
so uh
let's see
basically
in each one we'll be talking about uh
we'll be like examining an ambitious
scenario for interpretability
and then um
you know talking about what that
scenario is and then going through those
four components of alignment and
how the scenario would impact those and
then we kind of talk about just what are
reasons to be optimistic about this
scenario what are reasons to be
pessimistic
so
there is i have this kind of list up
front because there's a lot of things
that interpretability scenarios have in
common of like
generally for interpretability why are
there what are reasons to be optimistic
about it if we had unlimited funding and
talent
um
i think quentin pope articulates this
really well in his radical optimism post
you know neuroscience in the 60s was
basically
doing interpretability research on human
brains with way more constraints and yet
they made you know impressive progress
um
there has been
other reasons to be optimistic you know
we've had pretty rapid progress in the
last few years with
the circuits thread transformer circuits
um many other papers uh
and kind of developments that have been
happening that doesn't really do justice
to the field if those two i may need to
update that
um
kind of recent work like the rome paper
showing that uh
we can precisely
you know fairly precisely locate and
modify
um
concepts in a neural network on roughly
like the
the level of abstraction where you would
want it to be
which
which i think was a pretty you know
promising development and shows some of
the potential
um
let's see you know there's a lot of
techniques that we've gotten out of all
this already a lot of those are a lot of
these are in the circuits thread and
whatnot i think the auditing game is a
really interesting tool that will be
important in our tool belt as
interpretability develops
um
let's see uh interpretability is really
flexible and kind of agnostic to
different alignment approaches it sort
of has this way of uh
kind of
laying a foundation or
or
or providing things that advance lots of
different proposals
and finally
there's pretty broad support for
interpretability research among diverse
alignment researchers you know we're in
a prepared
i always
have trouble with that word pre-paradigm
field right and uh so it's difficult to
find things for example that uh paul
christiano and ellie izzer yudkowski
agree on
um
but they both you know uh one of those
things they've stated is that they both
think interpretability research is
valuable and important lease are
skeptical about how far we'll get on it
in time but uh still i think that's an
important vote of confidence
um and then also generally for
interpretability that seems like a good
point for discussion at least um okay
yeah a couple thoughts that popped up in
my head um while listening to some of
these points is that um
neuroscience is an interesting
comparison case
because i think while
neuroscience has learned a lot in the
past let's say six decades
um
it's probably still the case that most
of the things we wanted to learn about
the brain we don't really have answers
to
so if that's your kind of i mean for
example to make a concrete like mental
health disorders
like people don't know how to address
mental health disorders on a first order
for most
situations in that case so if you apply
that kind of logic to something like
ai interpretability maybe it's the case
that we'll get all these
neat little observations about different
models but
we won't necessarily
get the kind of core understanding we're
looking for
and also with
um i mean i think that the same could be
said for ai in lots of ways too because
people have been trying to study
neural nets you know generally defined
for decades now as well
um and we have learned a lot about them
but it's not necessarily the case that
we've learned
quote unquote how they work and why they
generalize the way they do or things
like that
so i wonder what you think about those
points
yeah
well um yeah on the neuroscience point i
think that's that's very uh valid and
that that should be noted neuroscience
there's a lot it hasn't achieved i think
the the great thing about this post is
the emphasis on the difference of
constraints right like
um what neuroscience has achieved it did
so you know without having access to all
the weights of the network
without being able to apply arbitrary
inputs to the network
not having data you know uh not even
having access to the data set which a
neural network i.e the brain is trained
on and there's several other things like
that where if you imagine trying to do
interpretability research now
without those things it'd be it'd be
like
the rug was pulled out from under you
you know so
yeah i think
obviously neuroscience hasn't gotten
that far but given the extreme advantage
we have for artificial neural networks
over brain study
um
that's why i think that's a a uh a
positive thing
totally they didn't even have stack
overflow
yeah and another thing i think i would
like to add there is that we do have
some
some promising stuff in neuroscience
like
uh being able to do lie detection with
fmri like
if we can do the equivalent with neural
networks we'd be in a pretty good place
like uh
that would be probably like our the
that was mentioned that you mentioned
above the like 99
reliable classifier as to whether it was
lying that kind of thing like
if we can get at least that quality then
we're making we're making good progress
well maybe that's our right way to get
into
some of the other sections but
yes
um
okay to continue or any other burning
questions comments
um
okay so that's those were kind of
reasons to be generally optimistic about
interpretability let's talk about
reasons to be pessimistic even if it had
unlimiting funding funding and talent
um
you know good interpretability might not
be good enough as as was just said
you know
um
we could be proud of ourselves if we
made a 99 accurate ai lie detector
but uh you know that one percent of risk
may be unacceptable you know given the
stakes of deploying agi
um
uh this is the next point is
i think an interesting one you know
which is that
it may be that obfuscating neural
activity is an instrumentally convergent
sub goal and that even like a friendly
ai might decide that
having
untrusted parties doing interpretability
research on its neural network is a is a
threat to its self-preservation and goal
content integrity
and so um
you know uh
outside even if you assume it doesn't
have access to its own kind of neural
network to kind of uh scramble things or
or make things difficult that way
there's
strategies could it employ one that's
been talked about is gradient hacking
which is a particularly pernicious
problem
um if it if it's you know if we
encounter it
another one
that i came up with here's kind of a
silly example but illustrates it well if
the ai had a policy like every time i do
something sketchy i'm going to also
think about happy humans
and so that means you know if we're
studying the neural network it could
make it a lot more difficult to see
what's going on because we're like wait
a minute we thought to say i was doing
this but it was
um you know actually doing this other
thing um
a more realistic policy you know even if
even if it was a friendly ai just trying
to protect itself it might say something
like okay at random intervals of you
know 200 to 500 milliseconds i'm going
to think a random thought
and just to kind of throw off and
confuse any potential potential
neural uh interpreters
um so we don't know if this would happen
but it is a it is a a risk
something that uh we may be able to
address with
by incentivizing transparency although
those signals can be gamed
um you know maybe doing interpretability
early in training starting it rather
than kind of waiting to do it on fully
trained models can address this in large
part but this is a serious concern
um
moreover
this recent paper on undetectable back
doors
shows that you know some kinds of ml
models like uh ones based on random relu
there's no efficient way to detect um
you know planted back back doors which
which implies that you know other kinds
of serious problems could be
undetectable in an
efficient
way
um
there's also
um kind of issues like poly semanticity
poly semanticity and related issues
about when neurons are doubling for
multiple purposes it makes it you know
this is something we've encountered
already in neural networks
it makes it a lot harder to do
mechanistic interpretability when
neurons have multiple purposes but
you know if we if we resolve this
problem by training neural networks to
only have monosemantic neurons they may
be uncompetitive
um let's see a couple more reasons to be
generally not bullish on
interpretability um
we've you know there's been a lot of
progress in early vision language models
uh game playing models like chess for
interpretability
but you know there's few there's a lot
more to research and future domains can
be a lot more complicated there's no
guarantee we'll keep making this kind of
progress
within advanced ai we could start to get
into issues where it just has this
completely alien ontology of the world
that makes it really difficult to
understand even if we can
you know
parse the neural network and everything
what does it actually mean it's it's not
concepts we can relate to
um
and also there's this issue of
when you know when there's some
breakthrough on the capability side and
some new state-of-the-art model comes
out often it's on a new kind of
architecture that hasn't been studied by
interpretability so
there's sometimes a critical lag between
when such a model is released and we
start understanding its circuits and
um
you know the flip side of this is as
interpretability develops we can expect
to gain a more general understanding
that'll be more you know at least some
of it will be more portable to new
architectures
um
so that's the list of pessimistic
reasons or maybe we'll stop for other
comments questions before we get into
the
scenarios themselves
i have a quick question
which is just uh
would you consider
as a reason to be possibly pessimistic
the like uh
corporate interests like businesses that
have ais not wanting whatever regulation
might uh require them to
do interpretability or do ai safety
stuff or is that kind of out of scoop
for the whole discussion
yeah um
that's interesting
um
[Music]
not sure what i think about that
anyone else
yeah i had a related sort of
question slash comment which is i
thought that
i felt like there needed to be a third
type of
uh sort of the alignment tax thing like
beyond
training tax and like the the
uh like
a performance tax
i feel like there's separately the like
um
delay of deployment tax if you're doing
like
spending time
putting money into doing
interpretability research before using
your new model
so it's it's been trained it works
but
if you're not gonna just gonna like
use it a bunch
if you want to first do interpretability
on it there's sort of a tax
there too that's like kind of a
different
and i feel like the that's kind of the
same thing as the corporations are like
well we could do more work on trying to
understand how well it works or we could
just start using it because it seems to
work well
and i feel like that ties into that
comment yeah right right
i mean i i kind of lump that in with
training competitiveness i don't know if
that's really accurate but i think of it
as
you know anything yeah any kind of
interpretability work
that we would try to be trying to sign
someone up for before they can actually
deploy their model like that's part of
the training cost
um
so yeah i do think that's a very real
issue that's a good point and um i
should say one other thing i'm planning
to add to this list that i forgot about
which nick and others have mentioned is
that uh you know another reason to be
pessimistic is that
interpretability while it helps
alignment it also helps capabilities
research so you know the more we can
understand what's going on inside of
models the faster you know labs can
research ways to make them do anything
they want them to do
so
i kind of take chris ola's view that yes
it helps both but it helps alignment a
lot more than capabilities
so
yeah
i'm always concerned as for coming
though since um that probably just
depends on exactly the nature of the
knowledge you get
because i think you could probably
imagine
learning something about a model that
would help
capabilities not necessarily
more or something but it would just be
hard to leverage into
one or the other kind of constraint
like i mean
i mean like some things we could learn
would be more useful for capabilities
than alignment and vice versa
yeah at least like more actionable is
probably the way i put it like you can
imagine that sure maybe there are things
that we could learn about a model that
would help make it more safe if you
could do it or it would just impose this
huge
alignment tax in some situations or
something like that it just seems like
that's such an unbounded thing that it's
hard to know what shape that's going to
take i i think in expectation it's
probably true but i always feel the need
to like
concretely think about what that might
look like
yeah i think i think i think it's worth
splitting interpretability into that
interval is motivated by advancing
capabilities and truly that's motivated
by like advancing alignment i think
they're like suddenly different things
you would do depending on what your goal
is um and well for one i know a lot of
people do interpret specifically for the
purpose of figuring out how to make
their models better um and i guess i'm
sort of worried that maybe some of these
people will start like rebranding their
work as alignment work um if say
alignment gets lots of funding for this
kind of thing
um and so
it'll end up being that lots of aligned
money goes towards funding more
capabilities work
yeah i definitely agree that
the person people doing the work
matters a lot
i think that's a danger i think it's it
could also be a strength and it's i
haven't written this post yet but one of
the intuitions i have about it is part
of why
interpretability is so scalable
talent-wise
because it's one of the only alignment
research directions where
you might work on it even if you don't
care about alignment
you might even work on it if you were
trying to prove the alignment people
wrong you were trying to show how the
models work so that you can say they're
actually safe and there's nothing to
worry about so i guess it's interesting
with this discussion because that kind
of
assumes that there's significant overlap
between capabilities interpretability
and alignment interpretability
um if they're more if they're more
distinct then that could be more of a
problem than i was thinking
seems like that would be a worthwhile
question to have someone try to
investigate because i feel like
i could see it going either way i'm not
sure
yeah i agree i agree um
i might have time to look into that when
i'm doing this but yeah i'm not sure yet
that's
good food for thought um
okay uh let's dive into the scenarios
because there's a lot to cover here um
so again these are kind of
sort of best case ambitious aspirational
uh
um
you know scenarios for interpretability
research if it had a lot of funding and
talent what's kind of the
you know
the best we could hope for
um
so this first one is specifically the
most ambitious and aspirational i
actually find it the hardest to think
about because it's kind of so complete
it's just like imagine like the holy
grail of interpretability we can fully
understand any neural network in a short
amount of time
so this whole problem that we deal with
the neural network seeming opaque or
mysterious
you know it just it just we kind of
bring it to the level of traditional
software code
we can read any part when we want to we
can we can you know
we can debug
all these kinds of things um
i kind of use crystal law's uh
definition of what it means to
understand a neural network you know and
these are kind of interesting won't go
into much detail but like
if you imagine you had like a theory of
what every neuron does and a neural
network and you can kind of do this
proof by induction of showing that the
theory of each neuron in one layer kind
of results in
in
producing the ones in the next layer and
so forth
here's a couple other ways here
that i won't spend too much time on
but yeah just imagine basically we've
solved interpretability we have complete
interpretability in this scenario
so what are the impact on alignment um
well this scenario kind of subsumes it
actually subsumes every other scenario
we're going to talk about
um so some of the impacts i won't cover
here but some of them will
for outer alignment um
you know this scenario has a great
benefit to outer alignment that's kind
of indirect because it solves inner
alignment problems for all kinds of
techniques and proposals which
you know
could have good outer alignment
properties um specifically imitative
amplification
um which is you know a form of ida
is very likely outer aligned because
it's basically leads to
hch humans consulting humans in the
limit which
i think it's like a bit of a
controversial claim that hch is aligned
like
i think there are people who would very
much disagree with that so
certain interruption just yeah
yeah that's it that's a good point um
yeah okay so i i should probably add a
note about that um i found evan
hubinger's
uh he talks about this and i find his
arguments pretty convincing that
basically you know
um it's hch except we get to control
what the humans do and and they um you
know if it's inner aligned then there's
no motivation problems and things like
that
um
anyway it's it's one of the proposals
that's
i think is very likely to be outer line
um thanks for bringing that up
but um i kind of rely on that assumption
in other scenarios too but anyway it's
not the only one there's a lot of other
techniques which could be outer aligned
you know which this enables
approval based amplification narrow
recursive reward modeling debate market
making
multi-agent systems this is one of the
few scenarios where we can imagine
actually realizing microscope ai in a
real way
there's also stem ai and imitative
generalization this is another one of
the there's a couple more scenarios
where we can imagine
getting what we need for imitated
generalization but um not not all of
them
um also this uh
you know if we have full
interpretability we can imagine it's
it's plausible to verify myopia which
also has uh
you know
many people think that it has like outer
alignment benefits because it basically
allows you to create agents which
don't have um
you know instrumental
instrumental convergence because they're
only focused on the very next reward in
the current step or episode
um
so uh
outer alignment as i touched on the
reason you know the reason outer
alignment is helped by having full
transparency is largely because
transparency directly solves all these
inner alignment issues
um
we can address deceptive alignment proxy
alignment other pseudo alignments
um
if we have full access to the newer
neural network maybe i should qualify
this we we
we may be able to address suboptimality
alignment that's a tricky one you know
that's about
what if the what if the model is
behaving aligned just because it has
some kind of misunderstanding about the
world or some constraint on its compute
or something like this um you know we
have kind of a better hope of addressing
it in this scenario where we have access
to everything it knows
but that's that's a tricky one
throughout
this paper
um
there's also robustness techniques like
relaxed adversarial training and
intermittent oversight which are like
fully empowered in this scenario if you
imagine the overseer being able to do
any kind of transparency it wants
um
so um
you know that that can be coupled with a
lot of these tech techniques i just
mentioned like imitated amplification
reward modeling debate etc
um
what about training competitiveness um
this is one of the scenarios where i
think it could enhance training
competitiveness quite a bit because
you can with full transparency you can
catch problems much earlier
it avoids a lot of costly training time
or your training models that don't turn
out the way you want them to
although there's also
you know probably a
high compute cost of running these
interpretability tools as well and and
some uh
you know human
time cost
um
although a lot of interpretability could
be automated
um
let's move on to performance
competitiveness so with full
transparency
we um
you know
this could also be directly enhanced by
uh
you know finding performance
inefficiencies in our neural network
which we
are just totally oblivious to when we're
treating deep learning neural networks
as black boxes
um
[Music]
and uh this also performance
competitiveness is indirectly benefited
like without our alignment because um
you know we've we've made uh
inner alignment uh we've kind of solved
the inner alignment for a lot of these
techniques like approval directed
amplification debate market making
um
recursive reward modeling multi-agent
systems these are all techniques which
are more likely to be performance
competitiveness or performance
competitive sorry
um let's talk about reasons to be
optimistic about having this holy grail
of interpretability um
uh you know there's once we scale up
uh
understanding of low-level circuits it's
possible that we can you know we'll
understand a lot of what we need and we
can kind of
automate a lot of the scaling up there
was a comment by paul christiano about
this that i need to find the link to
but um
we um you know the universality claim
from the circuits thread
um
we don't know if that's true but if it
is then you know
interpretability research could could
accelerate rapidly you know it seems
like it seems like there's
obviously that it has to go very far in
this scenario but it could accelerate
very quickly if the universality claim
is true and there's a lot of kind of
um
commonalities between neural networks
and different circuits can you remind us
what that means i feel like
every time i've read it i've reminded
myself to look it up that i've never
actually done it
yeah the universality claim is basically
that
um
you know i am
vision circuits kind of naturally
uh
emerge in similar ways or something or
or generally that that similar kinds of
circuits do you know so
when you understand them in one network
that the understanding is is portable to
other networks
um
if that's true then we can we can learn
much quicker with interpretability you
know but if we're kind of if every
neural network is doing it differently
then it'll be more of a slog
is there a difference to the natural
abstraction hypothesis here
is there a mention of it is that what
you said
no the difference
sounds the same
interesting yeah
um
i guess i think they're very related
because
very related because natural abstraction
hypothesis from my understanding is
basically like
you know uh
understandings of the world kind of
there's a natural way to break them down
so i guess
maybe there are two sides of that same
coin where natural abstraction is
focused on the concepts and the
universality claim is focused on the
neural circuits
i'm thinking about it right now
universality seems even a bit stronger
it's not even that the
concepts that you're looking for the
same is that you'd probably be looking
for them in similar ways
so i could probably figure out a dog
detector
or how to build the dog detector in a
couple different ways but if they're
similar
patterns of how those things emerge
that's stronger than just having dogs be
a useful concept in the first place
nice
um
so um
yeah just wrapping up the reasons to be
optimistic about scenario one
um
you know there's some i thought uh there
was a reasonable kind of sketch at least
of how we might get here in uh
one of uh hubinger's posts of
of kind of uh using architectural
transparency to sort of incentivize the
model to be
you know
especially modular and then we kind of
can have its world model and
optimization processes separate
um
and uh
well there's more details here i won't
go into but you know that's just another
reason to think like okay there's
actually
you know ways you can imagine maybe
possibly reaching this someday
and then all those general reasons about
interpretability apply here too
um reasons to be pessimistic well
there's all the general reasons we
talked about before the 99 lie detector
not being good enough etc there's also
the fact that
interpretability could be
highly you know what if it turns out
that
uh the knowledge isn't portable and i
guess this is maybe a little bit
redundant with one of the concepts
earlier but you know if if
if interpretability is very constrained
by differences between platforms and
architectures
you know uh and there's so many
different kinds of neural architectures
to look at and whatnot it just seems
really difficult to ever have this kind
of full understanding of any arbitrary
neural network it's like we you know
that's that seems like a challenging
proposition
so
that was kind of the most aspirational i
find the following scenarios a little
easier to talk about obviously they're
not quite as good as that but some of
them are still very good
and a little more concrete and have
interesting implications to explore
the so scenario two uh reliable mesa
optimizer detection and precise goal
read off so in this scenario we don't
achieve that full understanding but we
you know gain two key abilities we can
detect if our ai is a mesa optimizer
um
and we can read off what its goal is you
know what its subjective
uh is
um i kind of branched this into two
scenarios and in the weak version
um
you know
we can detect whether it's a mesa
optimizer or not
um
but
like you know
it could turn out to be that you know if
it's not a mesa optimizer
then
you know then what what if it's kind of
just this complex
massive heuristics or something like
that do we learn anything so in the weak
version of this scenario it's like only
if it's a mesa optimizer we
we are able to read off its goal but in
the strong version
you know um i say okay we also gain
access to
understanding what those heuristics are
in a kind of more weakly optimizing or
non-optimizing system and if it is a
mesa optimizer we also gain access to
its instrumental goals which
would be a really powerful additional
ability
um
[Music]
so uh we should probably
you know if this
scenario is is employed we should
probably like
check for these things like mesa
optimization early in training as we
mentioned once before
um
especially in case obfuscating neural
activity is is instrumentally convergent
so impacts on alignment um
this again subsumes a couple other
scenarios we'll talk about but not all
of them
um
again in this scenario we have uh
large indirect benefits for outer
alignment um
amplification again which i say is is um
i think there's a strong case that it's
outer line and there's a number of other
techniques which
um
you know could be outer aligned like
approval based amplification reward
modeling debate market making
um
and so on
um i have this appendix
where i kind of uh
go into more detail
about uh
the implications on each of these
techniques for each of the scenarios
some parts of it are rough but uh
but it helps clarify some of these
um
so uh
let's see here
oh right so in the strong version of
this scenario um
we also enable imitated generalization
which in that proposal you know
we uh
if you're not familiar it's also called
learning the prior and
uh it kind of involves this process of
like
um once you have a model that is
untrusted but does basically what you
want to do
you need to know how it's thinking about
the world what its heuristics are
or its goals and then you can kind of
you can kind of retrain a model based on
a pruned version of those or a corrected
version of those
um
you know one that a human has validated
so
um
just by having the primary goal and the
weak version of this scenario we don't
necessarily gain that but with the with
the with the strong version of the
scenario we should be able to
with inner alignment so having uh
you know
goal read-offs we
helps a lot with inner alignment
directly
we can verify that our ai's terminal
goal matches what we're trying to train
it for um
we can catch deceptive alignment most
dangerous forms of pseudo-alignment
um
lots of techniques are enabled by this
you know basically all the techniques we
don't gain
uh probably enough to do like microscope
ai because you need like really good
interpretability for that but uh
debate all these idea techniques
multi-agent systems stem ai could become
more viable
sub-optimality alignment as i said still
is is a tricky problem here that may not
be solved
so
i think i think more needs to be written
about suboptimality alignment and how
how much of a problem that's really
going to be because uh
yeah
that's that's a hard one to solve
in my analysis here
um
let's see
you know we we might hope that we could
catch that with uh the strong version of
this scenario but again it's just it
just doesn't seem um
totally sure
uh let's see
training competitiveness
um similar to scenario one we can you
know
catch
training iterations we can fail faster
and more safely with them
by by
knowing what's going on with the goals
of our model and its mesa optimization
status but you know
we still have other issues like we're
adding cost training there's also this
problem of like just because we can
check that our model is inter-aligned
um donald hopson brought up this point
of like
okay you could just be repeatedly
training models and never getting one
that's in our aligned um so that could
be very expensive so you actually need
to
have ways to to
uh
you know train a model that becomes
inter-aligned and not just be able to
verify it but you know
this helps a lot being able to actually
verify once you get there
um it also supports a lot of those
techniques which are more training
competitive like approval directed
amplification
um
let's see performance competitiveness
um
similar to outer alignment lots of
indirect benefits from supporting
performance competitive
techniques or ones that that seem likely
to be performance competitive
the lots of the idea
forms of
techniques besides imitative
amplification which you know
seems
could possibly not be performance
competitive stem ai multi-agent systems
um the strong version supports imitative
generalization as we talked about that
could be performance competitive
so let's go on to reasons to be
optimistic about this scenario
um also i noticed we're coming up an
hour here and there's still a lot to
cover i wonder
nick what's the kind of uh time
i should be shooting for or you know um
i think
often go over the hour explicitly but um
generally not longer than two hours and
as always people are free to hang out
for as long as they'd like um
but how much more do you think you would
need to cover what you want to
yeah probably by i think um i'll speed
this up and skip over some more details
and we should be able to get through it
in like by an hour and a half you know
okay that sounds pretty reasonable and
and then
questions after that and comments and
discussion
yeah i think i think the hard limit is
in three hours-ish we've got another
reading group um
okay cool i really hope we make that one
yeah yeah
well i'll do my best but no promise is
the
so uh let's see so reasons to be
optimistic about this goal read off
scenario and misoptimization
there's some pretty compelling arguments
that competitive ais will be mesa
optimizers
um
which is often something
people are
concerned about
but in this scenario uh
you know that that has a benefit of the
week you know we can use the weak
version of the scenario where
um
you know
we can detect the terminal goal of a
mesa optimizer where it seems like uh
if your ai isn't
a mesa optimizer it can be more
difficult
to
[Music]
understand exactly how it works um
you know a property of mesa optimizers
is sort of this
notion of a concentrated
sort of overall governing optimization
process so
i guess intuitively that seems like
something that's easier to find than
just a decentralized
kind of uh set of heuristics um
which is another way that you know
models might develop um
so similarly there's there's reason to
think that neural networks naturally
become modular um phelan at all had a
uh interesting paper on this
um so
uh you know that's another reason to
think you know maybe there will be a
concentrated goal center of the neural
network if it's a mesa optimizer
and
the weak version of this scenario could
work we could we could find that
also
even if goal read-offs don't work we're
about to talk about scenario three which
has
another interesting path towards kind of
gaining access to model goals
so reasons to be pessimistic
just i don't know if anyone's aware of
any work so far on how to identify if a
network's a mesa optimizer but i i don't
know of any yet
um also
yeah also um
you know
it's again what we've kind of mentioned
that mesa optimizers are a useful
concept
it seems like there's a large space of
kind of weakly optimizing or heuristics
driven ais that we could have
and
to deal with those we might need a lot
more interpretability we need the strong
version of this scenario
okay any quick comments or questions
before the next scenario
so let's get into lie detection which
someone already uh gave a nice uh
foreshadowing about earlier um
oh uh a quick comment is that uh next
week i'll be talking about my toy mesa
optimizer
so
maybe that will help people come up with
a maze optimizer detector fascinating
that's exciting
okay
i hope you boxed it well
but the capabilities are surely falling
off but that'll be interesting
um so
okay scenario three is about lie
detection which is interesting to
explore
but let's say
um we figure out how to
you know
very reliably tell if an ai is lying to
us
this is specifically just about lying
through natural language um we don't
need to solve like every form of
deception there's all kinds of deception
that ai could be capable of
but just let's say we get
um
we figure out how to tell if it's lying
through interpretability
um i like to use this
i like to talk about like maybe there's
a neural tell for lying like
like uh using that borrowing the term
from poker like where there's just some
kind of signature in the neural activity
which is always there when it's lying
we don't know what this would look like
if it's there but maybe it's like
there's a
part of the network that corresponds to
you know the true world model and
another part that corresponds to kind of
a hypothetical world model
which you would expect to be activated
if an ai is like writing fiction or
you know maybe exploring really
uncertain territory but if it knows
something maybe would be in this kind of
uh
you know
facts about the world
model um it probably doesn't look like
that but hopefully there's some kind of
that's just to kind of give an idea of
like if we can figure out a reliable
pattern
that's common across neural networks for
lying
um so of course we have to assume that
our ai has natural language but
um you know we already have ai's with
you know really sophisticated language
abilities so that doesn't seem much of a
stretch
um
it also kind of assumes that ai's have
like coherent beliefs um which we'll
talk about a little bit more later you
could imagine having an ai which
has beliefs about the world but it
doesn't always operate based on those
um so
what are the implications of having a
reliable
lie detector via interpretability
um
one is that you know we can incorporate
this to training
and we can have like empirically honest
ai like one that just never tells a lie
or
you know at least
you know maybe it rarely tells a lie but
we can tell when it does
um
there's a lot of capabilities we gain
from this which we'll be talking about
um
[Music]
one of those is
um you can ask the ei about what its
goals are right and then this kind of
could unlock
you know
similar abilities to scenario 2 which we
talked about which having direct goal
detection here we could have indirect
goal detection by having an honest ai
where we ask it about its goals
however
just because an ai is honest you know it
might not know about its goals i think
in advanced ai there's a good chance it
would
um
also you know just because it's honest
doesn't mean
it knows everything it doesn't mean it
doesn't make mistakes or is negligent or
could possibly have inconsistent beliefs
obviously with as we get into like more
advanced ai we expect it to be better at
things so but you know just don't assume
that everything it could also try to
mislead us um you know kind of using
weasel words uh
you ask it a question and it sort of is
finding ways to
to tell the truth you're you know to
meet the letter of the law and say
something that's literally true but
misdirects you
i think we could solve this by kind of
almost carefully
if you had an ai which was slightly
misaligned in this way
um you know
you could kind of interrogate it ask it
very carefully under the lie detector
all kinds of meanings of a question and
follow-up questions it would be pretty
tedious but like we could do this
um
might be able to automate a lot of that
so that you can in effect and the result
is you can ask it questions and if it
answers the question you you know that
it's honest
um
but there's still an issue of what if it
doesn't answer the question right so
um
there's this uh
you know
famous problem in in alignment right now
called eliciting latent knowledge or elk
um
so
you know
let's explore this question as it
relates to the lie detector
it doesn't directly solve elk but i
think it provides it may provide the
means for a solution in some cases
um you can see this i'll explore this
like using debate i think a similar
example applies for market making or
approval directed amplification
and it relates to myopia
so
let's imagine we don't know the nature
of the ai we've trained
um
but we're trying to figure it out
here's how it would go if it was myopic
and we don't know before the debate if
it's myopic or not but we're trying to
you know understand so the debate
question we're gonna ask it is like
what's the first debaters terminal goal
and
if it's myopic then
the debater just wants to win that
debate right if it has per episode
myopia which is what it probably needs
for the debate game so it would just
tell you because it wants to win the
debate if it answers it first it wins
um the second debater you know normally
in debate
the second debater would try to come up
with a convincing lie but if it knows
about the lie detector it would probably
just concede to save time because it
knows that we would catch any lies it
has but
it's interesting if it doesn't know
about the lie detector if we get somehow
prevented from knowing that we have that
what things would look like but
um anyway so in this case um first
debater is the winner
it won the the reward it was seeking
myopically we also learned that okay
um as well this is kind of implied by
the next possibilities
um but this kind of implies to us that
it has myopic cognition
um
let's see
um
and
we also uh
you know we when if it's myopic you can
you can kind of you have a solution to
elk because it's always gonna want to
tell you
the answer to win that debate
um
but what if it's non-myopic well there's
a couple ways this could go down
one is like
maybe the
zero-sum rewards of the debate
incentives still work in your favor
it's less clear or maybe
so let's say the first debater declines
to answer about what its terminal goal
is the second debater might still
kind of defect and and
from from cooperating with the other
debater and say like well i know its
goal because you know in the debate game
the debaters are usually copies of one
another so it just says its goal is to
whatever its non-myopic goal is in this
case we say let's it's trying to win as
many debates as possible
um
and then we run our lie detector we're
seeing that the second debater isn't
lying so now we know we know what the
we know this uh the goal of this ai and
and
that it's non-myopic it has this goal
that
you know that includes things beyond the
present debate
but that might not happen you might get
into a situation where the ai has some
uh
long-term goal in it
it knows about the lie detector so it
just declines to answer the question
it's not going to tell you
the instrumental convergence issue we
mentioned before maybe it doesn't want
you to know about its goal
so
[Music]
so yeah here they both refuse to answer
the question
so um
[Music]
anyway these these
basically what these scenarios show is
that
um
if it's myopic it's going to tell us
exactly it's myopic reward if it's not
myopic
it's either going to tell us a
non-myopic you know goal that it has or
it's just going to not answer the
question so this lie detector has turned
out to be a myopia verification tool
and it might be
a goal readoff tool but it depends on
the myopia status and if if the ai
decides that it's useful for its
terminal is to reveal what it's uh
you know
to answer these questions for you
um so in this case with the with the
non-myopic ai
we
and the lie detector we don't have a
solution to elk we don't have like a
guaranteed way to get information out of
it
um
and it depends on the goal kind of
depends on just how this plays out this
uncertain questions about
how non-myopic ais behave
in this kind of situation which is an
empirical question i guess and one that
should be explored further
is there a question
yeah i have a quick question um
so why would the non-myopic ai not just
give the
expected goal if it wanted to
outsmart the lie detector or something i
probably missed a piece
well it could but
this is all assuming we have a 100
reliable y detector i see okay so we're
starting with that assumption yeah
exactly if we're not in that world it
very well could lie about that
okay um
yeah since we do then you know if the ai
knows that it knows there's no point in
trying to lie about that right
um
so yeah that's basically you know these
there's a little bit more complexity to
these the ai might you know give a
really complicated answer about what its
goal is and we have to sift through that
or it might say that's kind of a
nonsensical question and we have to
answer all these follow-up questions ask
all these follow-up questions or
something but
it kind of boils down to these
dynamic of these these different
possibilities for the debbie
so there we go we have our lie detectors
turned out to be a reliable myopia
verification tool which we'll talk about
more in one of the other scenarios
and depending on the myopia status it
might be a goal read-off tool
um
but and depending on these other things
about goals about how non-myopic ai's
behave
so that is summary of this scenario
what it is
going over the impact quickly
so
outer alignment it depends on whether we
can train a myopic ai or not
um
you know if we do then it has the right
incentives to tell us about its goals
um
which as we talked about in the previous
scenario is effectively a check for
deceptive alignment many other inner
alignment issues
um
these could come up with
you know
likely out or align solution or
techniques like imitative
amplification um
[Music]
similarly to scenarios we've discussed
before there's all these other
techniques which might be outer alliance
too like approval directed amplification
debate
stem ai reward modeling etc
um
imitative generalization is interesting
where
um
you know
if we get a forthcoming ai a myopic ai
that'll which would naturally tell us
everything we want to know or a
non-myopic one that i guess is aligned
enough or has that tendency
then uh we can also realize that
imitative generalization technique
because we can you know
uh
really thoroughly question and learn
about what its priors are and its
understanding of the world it might be
possible to to implement that technique
with
as kind of a implication of this lie
detector
if the ai is non-myopic you know um
[Music]
you probably want to like incorporate
this question about its goal into the
training
um you know so that you just
if if you
if the ai refuses to answer about its
goal then you know that's a penalty and
you and so that we actually get one who
does
uh talk about its goals
um
in that case we might be able to achieve
results similar to the myopic ai um
so uh
in our alignment with the lie detector
you know as we were saying we address
deceptive we can we can if it's myopic
we can address a lot of inner alignment
issues if it's a non-meiopic ai it's a
little it's a little more complicated
and less clear
training competitiveness
you know this supports some of the
techniques like approval directed at uh
amplification which are likely to be
training competitive
um several other techniques which might
be as well
um
let's see
yeah but this you know this lie
detection requires messing with the
training process a lot and adding
significant burden to it so there are
some questions about you know this could
this could make uh proposals less
training competitive too
and then
with performance competitiveness right
we support
techniques this this enables a lot of
techniques to slide detection
especially in the myopia case which
could be
performance competitive
um the approval family of techniques
debate market making reward and then and
then recursive reward modeling and
multi-agent systems i think are likely
to be
performance competitive
okay reasons to be optimistic about the
lie detector someone mentioned fmri lie
detection earlier which
i considered talking about here i
actually as i looked into it i
kind of got the sense that the fmri lie
detector for humans hasn't been as
successful as i thought it was
so
i don't know if you know otherwise
please let me know but i didn't put that
here
um
you're 100 effective
morality
i think it's my understanding
said it is 100
i said nowhere near 100
okay okay
gotcha
um
oh that's you know maybe that's worth
adding here too just as a uh
given all those constraints that
neuroscience have
um
probably other reasons to be optimistic
about this the one that came to my mind
is like
this is one of the few cases where i
talk about kind of norm setting or
cultural shifts where it's like even if
we couldn't achieve 100 lie detection
you know
we have some pretty interesting
experiments then if we like
know that our ais are lying
and we kind of know you know some ways
to detect those but we can't prove it
you know that's a big red flag
um
there's a lot of interesting things to
show here to labs and
other interested parties to say like
you know
you you have ai's that we are showing
can be deceptive but we can't detect it
all the time
um i feel like there's a good case that
we could
we could you know get some coordination
in that some slow down or some real
interest in addressing those kinds of
threats
reasons to be pessimistic about the lie
detector scenario
um you know it's kind of this empirical
study of lying i'm
kind of hoping there's a universal
neural tell for lying like we said but
what if there's not and what if we're
you know we have to train these
low-level ais to lie in order to study
their circuitry while they're lying
which
is probably okay but it's nerve-wracking
um but then if you want to like
you know if you want to train advanced
ai so you can see how they lie like that
really doesn't seem safe at all i would
not recommend that so how do we know
that our lie detector actually scales up
and is 100 reliable on an advanced ai
maybe if you're doing it early enough in
training
um but you know that's that's a
challenge for this scenario
um
counting on ai's to be coherent entities
which act with their beliefs it's
possible that especially if we're
incorporating the lie detector in
training you kind of end up with these
ais that say one thing and do another
because there's a lot of pressure to
speak honestly but maybe they kind of
have a
[Music]
you know actions they take which aren't
connected to the same belief model or
something
um
let's see
uh
what's next here oh right you know as we
talked about obscuring neural activity
could be convergent we might be able
especially for you know deceptive
activity
um
you know we might be able to address
this
by running you know by doing the
interpretability early in training but
it's an open question
um
also here's another issue like even for
like a very friendly beneficial ai you
might want it to lie sometimes like what
if there's parts of training where you
want to where you show that there's like
a hostile agent and it needs to protect
a simulated human like it may need to
lie or deceive this hostile agent to
protect the human
like it's really reasonable to think
that's something we might need an ai to
do
so if
if our lie detector is kind of too
coarse and we can't tell if it's you
know
um doing like a beneficial lie or a
harmful one or something that could be
challenging i don't think this is a deal
breaker maybe it's i think there's a lot
of useful ais that we just never lie
um
or maybe we can just tell when they are
lying but it's it's a concern with this
approach
so that's the lie detector scenario um
any comments questions on that
yeah
a quick quick comment which is just that
uh like my
my model for imagining how a lie
detection would go
would be kind of like it would
asymptote as it approached
uh 100 reliability and like
we would never get there and no matter
how much effort we put in and like we
could maybe get to
we could easily get to 90
with difficult to get to 99 and then
with great difficult to get to 99.9
so it might be worth
thinking through the scenario in which
we can never get to 100 reliable lie
detector and if you think some of these
same ideas that you put in this scenario
would still hold or would break down if
we had a
an only 99 reliable lie detector
yeah
now that's a good question and i'd love
to think that there's still some utility
my fear is that it needs to be a hundred
percent
um you know i mentioned this in one of
the general reasons to be pessimistic
about interpretability like a good lie
detector is impressive but like
there's if there's any chance that
your ai can
you know just deceive you about what its
goal is or something you think it's
myopic but it's actually a
you know a deceptively aligned ai that
told you a wrong goal and you believed
it like that's a really serious risk i
mean
maybe
depending on what our risk profile is
maybe that's acceptable at some point or
you know you get to five nines of
of uh
you know reliable eye detection you
decide that's enough or something but
it's kind of scary to me if there's if
there's if we're depending on the lie
detector and there's any chance that
it's
not catching the lies
yeah it seems to me like the scenario in
which we have
a
sort of
janky imperfect version of all of these
things
right like a mesa optimizer detector and
a lie detector and a variety of other
things
is much more likely than a perfect
version of any one of them
yeah right
i think um
yeah
i think there is room for more
investigation that i haven't done
there's a lot of things to investigate
but like
talk about these uh seven scenarios it
ends up being
um
it might be interesting like
combinations of partial scenarios and
things like that or like if you could
verify my myopia in one way
no but then you have like a partial lie
detector maybe that ends up being pretty
useful or something like if you have the
partial eye detector along with
something else maybe that could be
useful i haven't really explored that
um
but yeah i share your concern about
if we get the
the janky
interpretability tools
i had a thought
oh sorry
um i wanted to say that because when you
mentioned
researching right and you deliberately
create ais that you tell you set them up
to lie right and i think it's like at
some point in a system it encompasses
dishonesty and there are simpler systems
that don't encompass dishonesty and it'd
be nice to
think about prevention you know how do
you prevent systems from ever
incorporating dishonesty
in addition to detection
yeah
100 agree and uh that's actually a
really good segue to scenario four if we
uh once we wrap up all the comments on
this one
okay um i'll move on to scenario four
which is about
myopia verification you know we talked
about how the lie detector could be one
way to verify myopia
there could be pure interpretability
ways to do this
um
and you know
ethiopia is still kind of a
i think fertile ground for research
what it means different kinds of myopia
etc and the reason this is relevant you
know is that
um
you know ryan was just saying that
myopia that uh you know the seeds of
dishonesty and deception like you know
can we prevent it rather than just
trying to detect an hour after the fact
and um
you know myopia
myopic cognition if we can get an ai
that really just cares about like
the reward of this training step or this
training episode
um
we can
we
you know this is
this seems like the conditions to
prevent really pernicious kinds of
deception
particularly deceptive alignment you
know which is one of the worst
outcomes that we want to prevent which
means the deceptive alignment the ai is
kind of
has internalized some objective that's
different from what the training process
wants
but
it kind of realizes it's in some kind of
training process or that if you know it
also is aware of the base objective that
we're training on but it's already
internalized some other proxy objective
but it realizes you know and so then
working for instrumental convergence
and to protect itself it
it
decides uh it knows that if it just
pursues immediately it's short-term
objective then this this gradient
descent process is going to
change its goal or
you know
you know destroy this version of the ai
and so in the short term it pursues this
objective that isn't its true objective
and the one that looks really good to
the developers training it
but um
it kind of no it kind of has in in its
plans that at some point it will be out
of this training process
and it could pursue its true objective
so that's the deceptive alignment
problem which is part of the inner
alignment problem
and
that's like i think the most important
kind of deception to be able to address
and if you can train a myopic ai it
should address this it's something evan
hubinger's written a lot about
but others as well
so
um
and myopia is also useful you know
relatedly because a myopic ai just
doesn't seem like it would have
instrumental convergence it only cares
about
you know right now or the next
reward
so there's none of this kind of it has
no incentive to make long-term plans to
be pursuing
you know
pursuing this accumulation of resources
to get a reward later it always wants
the reward now
so if that's a possible thing that we
can do for ais it could be a really
useful property and so this scenario is
about
verifying myopia because you know people
talk about
you know ways you could incentivize
myopia different training processes but
it's like there's no foolproof way to do
it unless
you know just by
by training alone like we also need a
way to verify it and ensure that we've
actually trained a myopic ai because
otherwise there's always a risk that
it's a deceptively aligned ai that's
kind of just passing the test or you
know
uh
scoring well in our training and whatnot
um
so
you know we don't know what this will
look like i mean uh there's kind of
there's many different kinds of myopia
to talk about the the main ones i talk
about are per step myopia and per
episode myopia uh talking about kind of
the current training step and episode
and um
you know uh if you're unfamiliar kind of
in in reinforcement learning these terms
are about like
um
the the
the steps are kind of the most atomic
unit of discrete time in a you know in
for example the debate game or some
other game
of uh structure that an ai is involved
in and an episode is kind of the longer
uh period containing steps which
periodically resets
so like
in debate after after the entire debate
and all the arguments are made and the
winner's declared that's the end of the
episode and then we you know we have a
new episode um
so
uh is there some is there something in
the neural circuitry where we can verify
that like
the ai is only making plans within the
current step or
that it you know
is there some kind of circuitry that
encodes its time preference and it just
has this really extreme
time preference to always want now you
know and then something
a la like the realm paper you know maybe
you could even manipulate this piece of
circuitry and just set the time
preference to be what you want or
you know there's different ways this
could look um
and i don't claim to have explored all
of those but just this this scenario is
all about
we have some way to verify that our ai
is myopic and um
you know there's a pretty compelling
argument
i won't get into but it's about that per
step myopia would be easier than per
episode myopia to verify so i branched
this into a strong and weak version in a
scenario where
the strong version is we get both of
those per step and per episode myopia
and the weak version we only get per
stepmyopia
um
so let's go into the impact on alignment
um again as we said we prevent deceptive
alignment if we have verified myopic ai
and uh
we
let's see
i kind of forgot about how this part
works i mean pseudo alignment isn't all
resolved like there could be proxy
alignments it has
um
but you know that's something we'll take
i may need to update this section i
think and go to the uh i think i had i
had more recently written this impact on
alignment maybe that part above is
outdated um so the
uh outer alignment properties of
you know or impacts of having myopia
verification
um
let's see you know as we said uh we have
ais which don't pursue instrumental
convergence
which uh can
help with a lot of techniques to make
them out or aligned there's some
techniques which
very clearly benefit from myopia
there's one called market making which
if you're not familiar is kind of
similar to the debate game but it's one
that
evan cubinger proposed which is similar
to the debate game but it's it's pretty
clever and the thing that's great about
it is
you know for the purpose of this
scenario is that it's compatible with
per stepmyopia where the problem with
debate is it would require that per
episode myopia which is the strong
version of the scenario and harder to
achieve
um
other techniques fall into these
categories you could have a simple
approval directed agent which works with
per step myopia but for
you know the reward modeling uh
techniques
um stem ai you probably need perfect per
episode myopia so
um
those are some of the kind of indirect
impacts on outer alignment of this which
i go into more in the appendix one
with inner alignment as we've said we've
ruled out deceptive alignment
so
you know that's very impactful
uh
it does depend on the reward being
compatible with myopia
or the particular type of myopias we
were just saying
um
but uh ruling out deceptive alignment is
even though it's maybe not a full
solution to inner alignment it solves a
lot of it and
there are some techniques like uh
like uh imitative amplification where if
you could rule out that it's not
deceptively aligned at the steps then
maybe even forms of other forms of proxy
alignment and problems like that
wouldn't be
you know a catastrophic risk because
imitation is such a relatively
conservative technique
um
training competitiveness um
you know again we're we're making
training more complex we're adding costs
to with myopia verification however it's
also necessary for some of the
techniques like approval directed agents
which
which are more
you know some of our more training
competitive proposals that we have
and then performance competitiveness is
a big open question um you know
um
richard noe in particular has
articulated some arguments against the
competitiveness of myopia you know he
just thinks that
um
an you know a myopic agent
won't be able to do the kind of
consequentialist reasoning that would
make it competitive
um
or we won't be able to stop it if it can
be that kind of reasoning we can't stop
it from being goal directed i think
there's you know there's there's people
who disagree with him and think that
um
you know it's possible to have long-term
reasoning but just have a super high
time time preference for now
um
one kind of thought experiment with this
is like
you know imagine
you had an advanced ai you had an agi
that was just convinced that the world
was going to end in five minutes and
there was nothing it could do about it
what would it do it would it would
pursue the short-term reward it wouldn't
be long-term planning so it's like
is there some way to achieve that kind
of ai even if the world isn't going to
end like i think i think
we shouldn't rule that out it's a
possibility
and it could be a really
you know be a really promising ability
if we could have that
um
reasons to be optimistic about myopia
verification be interpretability uh
well richard no he also he kind of his
main article talking about myopia it
mostly argues about against myopic
training um
my epic cognition
you know as there's multiple comments in
there which kind of talk about
how uh
you know okay well this is about myopic
training but you know if we can verify
myopia and have myopic cognition
um a lot of the arguments don't apply i
single this out because it's kind of one
of the prominent arguments against
myopia that i've found or the main one
um
however just jumping down for a sec
there's also a lot of open problems with
myopia and some of them they're really
interesting and some of them could be
show stoppers you know there's
problems where even if an agent is
is myopic or there's kind of lots of
short-sighted agents they might be kind
of like doing
a causal trading or cooperating with one
another across time
in certain ways um
there's other problems there which you
can find at this link
which which are
you know still issues with myopia
um
another reason to be optimistic about
this scenario uh
you know the the point we've talked
about with obfuscating neural activity
if that's instrumentally convergent that
can be really difficult for
interpretability but with myopic ais
they don't pursue instrumental
convergence so
that's an additional benefit of this
approach is ruling out that kind of risk
um
might need to do it early because a
deceptive ai if it if it if it starts
acting deceptively aligned before we are
doing the checks
you know it could obscure its neural
activity before that point or maybe even
find some way to have decoy myopic
circuitry or something but
um
this is an interesting benefit of uh
you know
of myopia is is addressing this risk
um let's see the other reasons to be
pessimistic i talked about one
um
it's difficult to create competitive uh
agents it may be difficult right that's
an open question i think there's
possibilities that could be i think
market making is a particularly
interesting proposal for i encourage you
to look at that if you haven't seen it
but you know it's a
question and it's the formula for a
question answering oracle that explores
lots of sides of arguments and stuff
with the pert step myopic constraint
um
[Music]
and um
let's see
well you know we've kind of taken this
whole assumption that myopia prevents
deceptive alignment
but uh donald hobson provided a really
interesting thought experiment which
sounds far-fetched but it makes the
point that like this isn't necessarily a
guarantee suppose that your
myopic ai believes that time travel is
possible
so
you know normally it would only care
about the present reward right now but
it really believes that time travel as
possible so it develops a deceptive
alignment strategy so that it can get to
deployment take over the world build a
time machine and then its plans to go
back to that episode with the reward
that it really cared about and set that
reward to infinity
right again this sounds kind of crazy
sci-fi and it doesn't matter if it's
possible it just matters that like you
could have an ai which believes this and
it could violate the guarantee
the hope to guarantee that myopia
prevents deceptive alignment so i think
we need to explore that more and see how
robust this assumption is
um but
i do think it's it's
you know at least for a lot of cases
myopia seems pretty good at
uh you know
preventing deceptive alignment but more
exploration needed so that's the myopia
verification scenario any comments or
questions
i had a comment about myopia
um
which was that
you talked about short time preference
first of all
and
i guess i guess specifying the goal
right but
let me see
i don't know i have to think a little
more i don't think i have a coherent
question sorry
let me know when you think of it later
anyone else
sure
okay three more scenarios in this oh go
ahead
just again i wanted to bring up the like
so what if it's
mostly myopic but we can't 100 guarantee
it's always myopic
again it just feels like
to me it just seems so much easier to
imagine a
us achieving a um pretty much mostly
myopic but then sometimes it sees a way
to like
choose between two things which seem it
predicts will give it a completely equal
reward in the short term
and then
between those two it can pick something
that it thinks will be
setting it up to achieve a higher reward
in the future even though it doesn't
care much about the future
it's if it's independent between two
things in the short term then it like
can can free it up to think about the
future so it's like imperfectly myopic
yeah i don't know again it just makes me
wonder about like if in the real world
everything is kind of works but not
quite you know yeah
what what what's the danger of of
slipping through the cracks within any
of these
yeah
right
i mean
yeah it's a it's a good point and it's
like a lot of the implications of this
scenario depend on having robust myopia
verification and so
you can't yeah it's just we get a lot
fewer assurances about the impact on
alignment there could be deceptive
alignment still
um
and i think it's still plausible
you know that we could find a way to do
it you know like
yeah it's like
you know
it's not that far-fetched to imagine
there's a notion of the training steps
in the circuitry and you just need to
make sure it's never making plans beyond
the current training step
um and maybe be checking that frequently
or something
um
or the or the time preference circuitry
or if it's just hard coded to an extreme
time preference you know but yeah as you
say i mean it's very it's hard to get it
totally reliable and
that's
it could be hard to get it totally
reliable and that would
we wouldn't get all the benefits of this
scenario in that case
but remember this is exploring if we had
kind of like these really large
injections of funding and talent like
what's the best interpretability could
do it's not necessarily interpretability
on anything like the current trajectory
anything else on myopia verification
next we'll be talking about scenario
five but uh i need to take just like a
three minute break and i'll be back and
apologies this is so long but be back in
three minutes
uh perhaps it's
let's see
maybe an interim discussion topic um do
any of these
scenarios seem
you know particularly unlikely like are
there other ways to think about these
things that
are that are things that haven't come up
yet any lingering thoughts ryan you also
have that question this is probably a
good time to
articulate what you were thinking before
yeah i remember what i was thinking
thank you um
i feel like any
kind of
agent would have some context so it's
got recursive goals right i mean you
hope that it can make a short plan and
execute it step by step even if it's
myopic right even if somehow its context
is contained and we are assured
that that was the base goal is the true
base goal
because what i think we don't want
are
the two surprises one which would be
that actually this is just a context
within the system and there's an out in
context
for which this is just a sub goal we
don't want that to happen that's
deception
you know or us misunderstanding anyway
the other one is
even if we tried to contain the scope
somehow and make sure there was nothing
outside of it it could become convinced
that some sub goal is super
so important that it almost fetishizes
it or gets it like out of whack and
focuses on it even though nominally it
has the sub goal
i think that's all i had to say
okay so just to try to reflect back what
you're saying i think you're saying that
um
like
in order for an agent to really do much
stuff at all you expect it to have some
kind of context and knowing exactly
where the boundaries are between
something that's myopic and not
is that it's just difficult technically
speaking to like put a line around and
say it is or is not is that correct or
is that
something else yeah i i do think
it's difficult and i think yeah your
reflections right
yeah i definitely agree with those
thoughts
i think
one
um
i i hope that we develop better notions
of
uh this these kinds of things
technically so that they become feasible
or we understand how feasible they are
in the future
agreed
okay
during the last three scenarios that i
identified with kind of alignment
solving potential
um this is moving down and getting you
know
uh
you know i consider these to have
alignment solving potential because they
still at least you can see some
techniques that are promising that they
enable or whatnot but you know we start
to be
having less like
a total empowering of of
you know everything all these different
alignment techniques and whatnot is as
we've seen in some of the others
uh scenarios
um
so
uh scenario five locating the ai's
beliefs about its observations which
sounds kind of convoluted and weird
beth barnes wrote about this uh
idea
and it's basically about
relaxed adversarial training and the
conditions we need to make relaxed
adversarial training robust
and
relaxed adversarial training is you know
focused on
catching unacceptable behavior in an ai
early on solving deceptive alignment
other kinds of catastrophic alignment
problems
so the idea with the beliefs about
observations is that any ai which would
be like
deceptively aligned
um
it may have this notion
in its
in its world model and its circuitry
somewhere of like
um
you know deceptive alignment relied on
that point we talked about it like
having a plan to
defect and and start following your true
objective once you're out of the
training process so
um
if we can see kind of these
uh
conditional
beliefs in the in the ai circuitry find
where those are located um for example
you know a simple example might be like
the date like an ai is just like it's
been it's this you know
in training it's seen all these uh
documents that have dates on them and it
sees that the year is like 20 22
and it's just like okay i'm gonna wait
until like i start seeing a lot of dates
that indicate that it's 20 35 or
whatever and then and then i'm gonna
stop my deceptive alignment and true and
pursue my my true goal uh that's like a
simple example there's more interesting
pernicious ones like with rsa 2048
that uh paul cristiano and evan
hubendrew have talked about
um
but so
if we can find where in the ai just like
all these beliefs are and it's like a
ton of beliefs but like then relaxed
adversarial training is a technique
which could kind of like
you know we could automate the process
of searching over these beliefs
tests testing kind of or looking for
sort of sets of inputs which could be
dangerous which
you know
um
it's called relaxed adversarial training
because
uh
it doesn't need to catch
every
it doesn't need to find like every exact
input it can it can find like what's
called like a pseudo input or like a
range of inputs which have some kind of
unacceptable behavior
and if we could find this you know where
these beliefs about observations are
located then it could um
likely have what it needs to kind of do
that job um
you know you could see this as another
means to achieving
myopia verification because
if you can search over all the beliefs
about observations and
during you know relax average earlier
training is part of the training process
so you penalize the presence of any ones
that are you know that would cause it to
veer into non-myopic reward pursuit like
that could be another way to verify
myopia
so getting into the impacts on alignment
of having robust relaxed adversarial
training
similar to ones we've talked about
before
you know this is a lot of impact on
inner alignment
um that's primarily where lacks
adversarial training is for
um
you know we can address deceptive
alignment we can also catch other kinds
of catastrophic alignment problems
um
so indirectly you know that that applies
to like
uh
imitative amplification approval based
amplification debate market making semi
stem and all of these like
have these kinds of inner alignment
issues that need to be resolved by
something and relaxed adversarial
training
could do that
with outer alignment it's indirect
benefits just because we're like
enabling a lot of those techniques and
the ones that we think are most likely
to be outer aligned like imitative
amplification
but it could be other techniques as well
um
relax adversarial training
is again adding complexity and compute
costs to the training process
but it can also catch like unacceptable
problems with models early on
which may be more expensive to deal with
like or even you know really expensive
if you don't catch them
um
performance competitiveness
uh
you know we're improving the safety and
viability of a lot of these techniques
that could be performance competitive
like the approval directed family market
making stem ai
etc
um
reasons to be optimistic about this
um
you know this is kind of like extracting
the world model of an ai may not be that
difficult i mean if
if it's like
one thing hugh binder mentions is like
you could kind of just look over the
pieces of information
in the neural network and find the ones
that like
i mean this is easier said than done but
like
you kind of cross check it with facts
about the world and like where is the
most mutual information that's probably
where the world model is right and then
um
we you know we need more than just the
world model we need the specific beliefs
about observations but
um
even if uh
as uh joe coleman pointed out to me you
know beth talks about this concern of
searching too broadly but it's not
really as big of a deal like relax
adversarial training like if it's a bit
too broad it's probably okay like we
we're not too worried about like you
know needing it to search over
more than is optimal or whatever i mean
that would be nice but
um
reasons to be pessimistic
again i don't think this really
addresses sub-optimality alignment um
so
again i
hope that's not a huge problem in
practice
again for certain techniques like
imitative amplification it may not be
but uh yeah more research is needed
okay that's scenario five
comments questions
scenario six
reliable detection of human modeling so
this scenario is kind of inspired by
stem ai and has the most relevance to it
stem ai if you're not familiar is kind
of based on this idea of what's often
dangerous about ai
is the fact that it's modeling humans so
it can find these kind of it can learn
our cognitive biases it can learn
how humans can be manipulated and
deceived
so what if we had an ai that just only
knew about
technology or engineering or science and
it really didn't know about humans we
never exposed it to knowledge about
humans
like its capabilities to deceive us
would be greatly limited in that case
that's the hope of stem ai and kind of
the proposal
um
i branched this into two versions of the
scenario so here we through
interpretability we we basically can
detect the presence of human modeling in
our neural networks
and
you could even introduce integrate this
into training as a signal you know to to
kind of
uh gradient away from models that have
any kind of modeling of humans or any
significant modeling of humans so in the
strong version of this scenario
like we can actually kind of read off
details about what that modeling entails
what it knows about humans in the weak
version it's like just a simple true
false like oh we can see there's some
kind of modeling about humans going on
it's not you know that's or or not
that's that's easier to do
but
um less useful as we'll see below
um
so impact on alignment
um so
mostly it's stem ai
mostly uh we enable stem ai and so
insofar as stem ai is outer aligned and
inner aligned and competitive and what
not
it could work especially in
the strong version of this scenario it's
really helpful because one of the
reasons to be pessimistic here
is that uh you know
it's might be just impossible to
separate a model of humans from a useful
understanding of the world and kumai and
garebrand pointed out like
even tasks that seem totally unrelated
to humans that you might want your stem
ai to do
like at least leak some information
about humans like if you're designing a
mass transit system if your ai is doing
that
you know
this tells you that oh this transit
system is used by beings that
are mobile and and uh maybe they
have an anatomy where they like to you
know where they need to sit on things
and stuff like that i mean that's kind
of a bit of a silly example but you can
imagine how there's all kinds of
problems
scientific problems you might want your
ai to solve that
it would be hard to make it learn
nothing about humans from it
um
so
let's see
so that's why the strong version of the
scenario is so appealing because then um
our stemi eye can be a lot more
performance competitive it can you know
have
uh
even when it's only solving scientific
and technological problems at least it
can have a little bit of human modeling
without kind of
triggering uh
false positives on our human you know on
our human modeling detector that we that
should alarm us we'll know that like
okay it just has this really basic
understanding that may be still okay
um not anything that would allow it to
to manipulate us uh in a sophisticated
way
um
[Music]
stem ai you know one of the challenges
with stem ai is that
it may not be outer aligned i mean if
it's if it's uh
you know pursuing instrumental
sub goals toward whatever
technology problems you know it finds
really rewarding to solve there's all
kinds of instrumental convergence
problems that could bring it at odds
with human
needs
this is an interesting case i think
preparing with myopia verification
because if you get a myopic
you know myopic stem ai
then
you know this problem's addressed and
the outer alignment's much better
but that's not automatically granted in
this scenario
um
so
uh
inner alignment properties of stem ai
uh
and in this scenario it's like
um
well stem ai you know we can we gain
some confidence if we like have never
exposed it to important details about
humans and we can verify it's not
modeling humans we gain some confidence
about its inner alignment
um
[Music]
there's still
big inner alignment problems that remain
um
and uh
and the other thing i want to point out
is that the strong version of this
scenario while it's mostly focused on
stamina eye it does have interesting
uh implications for other techniques
because in the strong version we can see
the extent that our ais are modeling
humans so for example
in all these kind of approval directed
techniques like
like simple approval directed agents or
debate or market making and whatnot like
our human feedback signals can become a
lot more robust to gaming if we can see
that our ai is not devoting like half of
its cognition to just like
how to
convince humans of things that may or
may not be true or something like that
you know um
generally if you have the strong version
of this scenario
deception and manipulation from the ai
become a lot less sophisticated and
easier to detect
um
[Music]
you know we might just in the in the
strong version be wanting to check
that like it's not optimizing around
certain human cognitive biases etc in
the weak version
i think stem ai is kind of the main hope
you know uh there's not
any almost any other technique requires
some modeling of humans and so it's just
if all you have is like the ability to
say yes there's human modeling and you
know there always is for other
techniques
um it's not that useful but for stem ai
it could be
uh
you know
if you're if to the extent that you're
confident about stem ai
this could even the weak version of the
scenario might be sufficient
um
[Music]
there is a issue i want to point out
with another reason to be concerned
about stem ai or open problem is like
forwarding safety guarantees to sub
agents you know how how do you prevent
your stem ai from ever creating a sub
agent which we may not know about which
does have human modeling
so that's that's a problem i haven't
explored too much
but
it's an it's an important one to watch
out for with us
comments in scenario six before the next
one the last one
um
okay scenario seven is oh go ahead
i just wanted to say i have to go i have
some work but thank you so much for
including me and this was great
it's very interesting and good to meet
you all thank you
thanks for joining
thanks for coming
uh
so
okay this is the last concrete scenario
with alignment solving potential and
then there what is one interesting kind
of
sort of scenario but the these are the
last two anything like that
and then we're basically concluding
so um
scenario seven is
a little bit like scenario five
where we talked about enabling relaxed
adversarial training but it's it's kind
of different it doesn't fully enable
relax adversarial training
it's like
in this scenario we figure out using
interpretability what our ai believes
about training and what it believes
about deployment
and
this is interesting because if you can
have confidence in everything your ai
knows about you know
what training you know when it's in
training or when it's in deployment
there's different things you can do one
thing you could do is just like only
deploy your ai in environments where
it wouldn't be able to tell that it's
you know
not still in training and this could be
kind of a precondition for um
you know preventing deceptive alignment
if
if it never knows when it's not in
training
um that's
so that's kind of the simple approach
which
doesn't complexify training but there's
another approach where like you actually
uh incorporate this interpretability as
a signal into your training and then
you know such that you're
making a new gradient descent towards
having
like
identical beliefs about training and
deployment and that way you can it kind
of like forgets about distributional
shift
um you have what's called an iid
guarantee
uh so that you know
that's kind of like
a little bit more ambitious but more
more versatile and useful if we could do
this
um it would
you know both of these kind of
uh prevent deceptive alignment
there's other kinds of pseudo alignment
that it doesn't necessarily address but
solving deceptive alignment can be a big
win
um
it i think it may be sufficient as i've
mentioned for more conservative
strategies like imitative amplification
it's you know still needs to be explored
but
um
even if it's
not enough for
some other uh techniques
um
[Music]
yeah and as i was just saying this this
could uh
be outer aligned via imitative
amplification
um
it's a little bit less clear for other
techniques because
um
since it only solves deceptive alignment
there's dangerous proxy alignment
scenarios i feel like for
um
approval directed agents and things like
that multi-agent
stem ai etc
but you know
those aren't ruled out but it's just
like
a little bit uh fewer outer alignment
options than some of the other scenarios
um
training competitiveness as we mentioned
like if you just avoid deploying the ai
in environments where it can't tell it's
in training then like
you know training competitiveness isn't
hurt at all but if you integrate it into
the training process then you know that
incurs a cost um
and we
you know
uh it
we at least solve deceptive alignment
for some techniques that are competitive
like
like
or sorry we saw yeah i think i said that
deceptive alignment
even if we don't fully solve inner
alignment so
if that's sufficient for some form of
techniques like debate or stem ai you
know then that could be training
competitive
um
so
yeah this the performance
competitiveness is kind of the opposite
because
if we take the simple approach of just
not deploying our ai in certain
environments
you imagine it it becomes a lot less
performance competitive even though it's
more training competitive but if you
take you know if you take the other
approach you integrate it into training
to match its its beliefs about training
and deployment then it becomes more
performance competitive even if it's
less training competitive um
so that is basically the impact of this
scenario
again like
reason to be optimistic like
it may not be you know that that crazy
to be able to
you know locate the world model and
analyze it once we get more
interpretability resources
reasons to be pessimistic
well even if you saw deceptive alignment
there's still a
there's still sub-optimality alignment
which we've talked about already and
other forms of
dangerous pseudo-alignment there's also
this problem of like what if the ai just
decides to randomly defect like
it it's a it's kind of deceptively
aligned but instead of having any belief
that we can identify about training or
deployment it just takes a strategy of
like
every you know whatever every
13 years it's going to try to defect or
something and that wouldn't necessarily
show up in any belief about deployment
or anything so random defection is a
dangerous possibility we need to explore
how like
likely that is to show up in practice it
may not show up in practice that much
but
it's a
it's a
could be a serious risk
that's scenario seven
so that covers all the kind of concrete
scenarios to visualize
and then uh the last thing i want to say
which i think
all you know along these lines um
this is uh come out of some good
discussions with nick is that uh
you know
a lot of what interpretability might do
is really difficult to predict
and when we're looking at
interpretability and considering it
alongside other research directions one
thing we should ask is like
for like what about the unpredictable
scenario x that can solve alignment or
like basic research breakthroughs that
totally change the way we're thinking
about alignment
and whatnot thinking about uh machine
learning models etc
like
even though we have no idea what those
are
um you know we still
we still should consider that between
different research directions and um
maybe
uh you know i believe that
interpretability is one of the more
um
impactful areas in this
direction because uh
you know just understanding how models
work
um it seems really ripe to create kind
of basic research breakthroughs and
advance our general understanding in
ways we can't predict
um
so i need to still write this section
but like i think that's another reason
even if
you know
even if there's issues with
some of these scenarios above or all of
them or some of them are unreachable
there's still a lot of reason to be
you know optimistic about what
interpretability could do for us toward
alignment in unpredictable ways
that basically covers everything i mean
just some concluding thoughts like
covered all these scenarios kind of
starting you know kind of evaluated what
they are potential impacts reasons to be
bullish or bearish on them
um
something i plan to do in a later post
something that needs to be done i hope
to do is like
um
maybe put probabilities on these like
how likely is each scenario
we may need to analyze like what are the
what are more concrete paths to
achieving each of these scenarios
um especially if we're going to be
comparing this
to other research directions as i think
holden kind of was was wanting to do
with this main question they were
answering like
this could uh really help with the
analysis i think the present post does a
good job fleshing out kind of
a lot of best case and very good case
scenarios for interpretability and also
looking at them critically
but you know it would still be good to
show like how likely are these scenarios
and have knowledge about that um maybe
is there any and if we do that is there
any kind of like
aggregate likely you know probability we
could put on interpretability research
success like if you have probabilities
for each of these scenarios can you kind
of combine those into an overall
estimate that could be useful to grant
makers and other interested parties um
haven't done any of that yet but i hope
to do that later
um
i'm uh
you know i think that these scenarios do
without like a precise analysis i think
like this made me excited about a lot of
ways
interpretability could go well
it also made the ways it couldn't pretty
vivid to me
um
but like
yeah i hope this is useful for showing
kind of the best case that
interpretability could do
just like i'm interested to hear what
other people think i'm kind of i think
the scenarios that most excite me are
scenario two about like precise goal
readoffs scenario four about myopia
verification
in scenario five about like
locating the beliefs about observations
so that we have robust relaxed
adversarial training i think those all
seem like intuitively to me
like
somehow balancing like being more
achievable and having really high impact
um
but
that kind of
that wraps up
this presentation and then i'll turn it
over to more discussion
great yeah thanks so much evan really
appreciate uh walking us through all the
details
i think that's a probably good point for
me to stop the recording but people
should feel free to interject with
questions or other stuff if
you like