Connor Leahy | Promising Paths to Alignment

all right
well hello everybody
today i'm going to be talking about
promising paths to alignment and let's
get just right into it there are none
thanks for coming to my talk i really
hope you enjoyed are there any questions
all right
obviously i'm joking
but to every joke there is a bit of
truth
and the bit of truth in this situation
in this case
is that
in my opinion we don't currently have
any kind of approaches that obviously
are going to lead to a solution to the
problem in general yeah that doesn't
mean there's no approaches but i don't
think we're at a point where i can like
point to a direction and say yeah this
is going to lead us where we want to be
surprising to me i seem to be among the
more pessimistic people just a bit for
your calibration here in this regard so
i was recently at uh at ea global and i
talked to a bunch of alignment
researchers and like a lot of people in
the field about their timelines and
their probabilities of you know things
going bad and so on and i was quite
surprised to see that i was definitely
among the more pessimistic people that
you know that many maybe even most
people had like you know probabilities
of things going well you know well above
50 percent which is definitely not the
case for me
so
i was invited to give this talk to talk
about paths that i find promising and
i'm going to do that i'm going to talk
about some of the things that i do think
are promising and you know might be
interesting to work on to look concerned
but from my experience at ea global and
more generally talking to people i think
that a more valuable thing perhaps for
me to talk about is a little bit about
the reasoning why i think um the problem
is hard
i will give you a bit of the generators
of like why what makes this problem
particularly different from other
problems
i'm not trying to say that no progress
has been made unlike certain other
people i there is progress and there are
promising approaches that i think you
know have helped us deconfuse these
methods
these questions and so on but i do think
that there is a tendency sometimes for
certain people to i think miss the core
of the problem and why it's hard
and i would like to talk a little bit
about that and then i will also talk a
little bit about more concrete
directions that i find promising so i'm
not here to give you like the bad
alignment bingo takedown or whatever
right
you know there's always these classic
you know bad approaches that show
everyone comes up with when they first
hear about alignment
this is not really what i mean here i
think there is there are obviously you
know silly uh solution like just use
azimus three laws like that's a bit
silly right you know but i think there
is a more subtle case of like why
i expect alignment to be hard that
sometimes i feel even eas don't fully
grok and i but i think it's very
important to understand the core of the
problem
this is especially important because of
what i've been do what i've dubbed
substitution hazards
and
i think there is like some technical
word for this in the psychology
literature or something but i couldn't
find it for the life of me if anyone
knows the correct word for what i'm
about to describe please tell me
it basically it's this common
psychological phenomena where people
faced with a heart problem will instead
substitute with an easy problem make
progress on the easy problem and then
think they made progress on the hard
problem
so this is a very common phenomenon of
course if you were perfectly rational
and you track all of your you know
belief systems and like you know how you
make progress in various problems
no problem with this you know making
some progress on an easy problem can be
useful but the problem is is that
the way humans tend to work is you tend
to overestimate how much that was
actually useful for the heart problem a
class example this is hard problem
solving climate change easy problem
recycling your trash
solving alignment is solving climate
change is really hard there's like heart
it's like where do you even start with
that
and then
you might be tempted to say well i could
you know recycle my trash that's like a
step in the right direction and that's
fine like that's great you know uh you
know if you
make sure that you're aware of the
amount or you know or small or large
amount that that actually contributes to
solid problem no problem with this but
there's a classic problem that can
happen where by doing
something the human brain will kind of
be satisfied it's like ah i did my part
so now i have to relax you know like i'm
not expected to do more than this and
that's nice and all but you still didn't
solve climate change and someone has to
do that
so
that's why i think it's really important
to understand what i can like what is
the i think the hard problem of
alignment there is a temptation to look
at easier versions of the problem make
progress on those and then say oh this
will scale you know we've made progress
which might be true i want to make very
clear that might be true a lot of
science is done this way a lot of
science is you first solve simple test
cases and then you start to slowly you
know generalize your solutions i'm not
saying this is always bad i'm just
saying this is a a problem that you have
to very much be aware of and you have to
keep in the back of your mind what the
actual heart problem is you're trying to
solve
alignment is not just an another
technical problem is what i'm trying to
claim here i'm trying to i'm going to
justify what i mean by this and why i
think this is really important
the shape of the problem is
particularly wicked in the sense of a
weakest problem it is shaped in such a
way the way the kinds of things that
need to be done to solve it and the
obstacles we face are are those that our
species are just particularly bad at
solving it's is really a a very cruelly
designed
problem therefore the kinds of problems
our civilization or science tends to be
good at and i think there's like two
main reasons why that what make this
problem so hard there's others as well
but i'm gonna focus on two in particular
the one is just the that we're dealing
with super intelligent optimizers
and super intelligent optimizers are a
class of object that we are just not
familiar with they're when you know when
we find a new chemical or we find a new
star or we find a new computer we design
a new computer language these are
objects that we understand like these
are objects we've encountered before we
have a lot of theory we have like a
image like a gut instinct of how these
kinds of things work
there is no super intelligent optimizer
currently existing has never existed in
the entire universe we are dealing with
a whole new class of objects that has
new kinds of properties that we are not
familiar with that will cause many of
our normal methods to not work
we have a whole new
world of failure modes their current
methods of science are just not set up
to deal with effectively we'll say more
about that in a second
and the other one is coordination
so you know humans are social animals
we're incredibly good at coordination
compared to chimps
but
are we really good at coordination i'm
gonna say is frowny face uh coordination
is
hard and coordination might be really
important to solving this problem
it is solvable though this is really
something i wanted to emphasize it's
very easy to go all doom and gloom and
be like oh yeah we're all going to die
everything's terrible we can't
coordinate you can't sell throughout
i want to really stress
i fully believe the alignment problem is
a solvable technical problem we had
enough time we enough smart people doing
good theory doing good science and i
think this is a problem that can be
solved there is no law of physics that
says you cannot design and construct and
run
align super intelligence this is
something that our physics allows we
have no reason to believe otherwise
whether we will do so in the in practice
though unfortunately a different
question
so
if you take anything from this talk at
home
make it this slide
this is i think the most important core
of what i think is important to
understand when trying to solve the
alignment problem so if you check out
after this uh like that's fine whatever
right but try but please this thing i
think is really the core of the problem
is that
very common thing i hear especially when
talking to like smart silicon valley
engineer type people but even some
alignment researchers they say something
along the lines of we'll just figure it
out as we go along you know they'll say
you know we've had hard problems before
you know we already have to build you
know like nuclear reactors that don't
melt and bridges that don't fall down
this is just another problem like any
other you know as we develop more and
more of these systems as we get more
used to them we see them fail in
different ways we'll slowly iteratively
you know
learn how to control and make them safer
over time and i want to argue that that
is not the case
and this is at the heart of why this
problem is hard
this is there's entire new class of
failures their current methods of
science are just not up to snuff to
dealing with in particular i think one
of the core problems is that optimizers
are basically black swan generators and
our methodology of science you know are
like you know um
enlightenment era you know popparian
barely basin understanding of how to do
science
is not good at dealing with you know
things that have many uh unlikely high
impact events and optimizers are exactly
that they're systems that can push and
you know they can push reality into very
unlikely configurations
this is this completely changes
how things fail how you should expect
things to fail there are some fields of
research that do deal with similar
objects the closest is computer security
computer security also deals with
intelligent adversaries that can
optimize and bring your state into
extremely bizarre states than you have
to defend against there's a very strong
connection between alignment thinking
and what a good security expert thinks
about a good computer security expert
for most of history we had this
guarantee as humans that there was not
no thing running out there that is
smarter than us that is better at moving
reality into strange specific
adversarial states than we are humans
can design all kinds of weird sin i am
currently talking into a brick of
plastic and silicone that is beaming all
of my words uh you know to people across
the world that is a very strange state
compared to the average background
radiation of the universe it's a very
weird state to exist as a biological
creature and to have technology and do
all these kind of things these are all
extremely low probability events but
because we're optimizers we can push
reality into these very low probability
events and make them high probability
because that's the whole point of having
an optimizer
and because we humans are used to being
the only optimizers around of this
strength
and there definitely are none out there
stronger all of our thinking of how
security works about how to do science
about the maximum damage that could be
caused from a certain accident are
our systems that were developed in
worlds where this guarantee hold that it
never was that we had to deal
spontaneously with you know i don't know
some dog spontaneously becoming smarter
than all humans combined and we have to
deal with that like it's not a thing
that happened
our current methods of science i'm
playing i'm going to claim here uh
depend to a very large degree on two
things being true
we have to be able to do many repeat
experiments and observe failure there's
a lot of iterativeness that goes on in
most science and engineering it's gotten
better over the time as we've developed
better better theories but very often in
the history of science experiment
precedes theory we first have to try
many things to build many steam boils
and see them blow up until we figure out
thermodynamics
and the second one i'm going to claim is
a problem is we have to be able to model
things as well behave probabilities with
few black swan events
but you know
the any scientist's favorite tool is the
normal distribution that's always what
you want to work with you know whenever
you're modeling any kind of system you
want well-behaved probability
distributions you can reason about like
what is the maximum failure we expect to
apply here you know if you have a
you know
a model of possible failures in your
system and you model it to say some kind
of normal distribution whatever it's
you can you have these vanishing
probabilities from tail-ends where you
can like basically assume that's never
going to happen
but
not all systems behave this nicely this
is why modeling the stock market is hard
is that the stock market very much does
not behave like a nice normal
distribution and many other systems also
don't behave this way imagine trying to
make a risk analysis about some kind of
system that could fail where the
where the probability distribution you
pull from for the damage caused has no
expected value
how do you even reason about this object
and there are many probability
distributions that have this property
that they do not have an expected value
because the long tail tail is so fat
that it just goes to infinity
how do you reason about an object like
this how do you prepare you know
defenses or insurance around an object
like this
we don't have stable theories or like
a tradition of scientific and
engineering to deal with objects that
have properties like this and my claim
is is that super intelligent optimizers
have these properties
is that they are
you can't do many repeat experiments of
them
and you can't model them as like a
simple thing where you know ahead of
time the maximum amount of damage you
can do is say x or it can only do x y
and z and it will never do w
this is not the case with sufficiently
powerful optimizers because by
definition once they're smarter than us
they will be capable of doing things in
the universe taking actions and coming
up with plans that we will not be able
to come up with ourselves and we will
not be able to model ahead of time and
we won't be able to put any kinds of
balance on what kind of behavior or
damage these things could cause
we don't get these properties with super
intelligence
this this comes from all the standard
problems you know optimizers tend to set
values for stream values we have
instrumental conversions we have the
whole stop button problem all these
classic problems in alignment are
variants of this
is we can't use our normal philosophy of
science or normal tools of this
epistemology
to reliably deal with these kinds of
scenarios where we have systems that
have these extreme properties
you may notice i'm not really making any
specific claim about like oh the system
is going to be built this way that way
or it could have a desire or it's going
to like you know have consciousness or
god forbid anything like that i'm making
very strong just simple statements about
the way we do science as a species
does i completely believe could a
hypothetical more rational more advanced
alien species deal with this yes
absolutely
i think we're already seeing you know
philosophy of science and epistemology
and statistics advancing and i think if
we had more time we as a species would
develop better ways of doing science
better ways of you know doing
experiments without having to do the
dangerous experiments a better way is to
model and reason about 19 uncertainty
and like you know black swan type events
but we're not there yet and this is a
problem that requires that to solve
reliably i claim would require just far
more sophisticated philosophy of science
than we currently have far more
efficient methodology far more efficient
systems
towards deal with and that is why this
problem is hard it's because it is a
type of scientific it's like we go to
science two you know we need level two
science to be able to solve this but we
only have level one science and now we
were hit by a level two problem and how
do we solve this problem
it's tough
we probably will only get one shot
with super intelligence which is at the
core of why it's so hard
if we accept you know
stuff like instrumental convergence it
seems pretty obvious that if you have a
misaligned system and i expect these
systems to be misaligned by default to
proven otherwise that they will be
incentivized to take over you know to
take away power to not be able to shut
down to gather resources to prevent
retaliation etc etc there will be no
iterative refinement cycle there will be
no attempts where we can we can try over
and over
trial and error science 1.0 style we
need a whole different approach to be
able to
solve this problem in one go and this is
just something our species does not have
much experience doing again i think this
is something we could do
i think this is a thing that a smart
species taking its time with enough
theory work ahead of time could do
but i'm not sure if we're going to do it
this also leads into the problem with
coordination
a sufficiently smart problem you know
species would be already well
coordinated on this problem everyone
would know about this problem you know
school children would be taught about
the alignment problem and all the
smartest researchers would all go into
alignment as the most important problem
and all the hardware manufacturers would
have you know agreed to like slow down
the development of gpus until we are
more certain about what we're doing
but moloch reigns molly always reigns
game theory despite its name is very not
fun
and leads to all kinds of difficult
scenarios coordination is very hard even
if you have two people who completely
want to coordinate lemon markets are
a great example of this
it's very hard to coordinate especially
at scale
and especially when it's dealing with
something complicated like this we have
a profit incentives you know agi systems
are going to be the most profitable
technology to ever exist of course
they're going to be
it's not clearly visible how dangerous
these systems are until it's already too
late it's anti-inductive to normal
progress norms like you know especially
silicon valley and stuff there's these
beliefs about how you know all
technological progress is always good
you know economic development is good
scientific progress is good and i get it
i want to believe that too i grew up you
know classic you know sci-fi
transhumanist you know progress guy as
well as anyone else
but the problem is heart
and
if we don't solve it correctly it might
not be
and of course it's all a huge
unilateralist curse type situation where
it just takes one uncareful actor to
ruin it for everybody which makes
coordination
even harder
look
i don't know how to say this but our
civilization god's ass handed to it by
an evil droplet of fat two years ago how
do you think we're gonna deal with
something that's actually competent
it's
you know there's a teeny amount of
optimization pressure that went into
covid like you know how many bits of
optimization pressure could have
maximally gone into cobit not very much
and that froze our entire civilization
and like you know screw everything for
22 years imagine if we have a system
that's actually optimizing against us
i don't think it's going to be a fair
fight so to speak
so does that mean coordination is
completely impossible there's nothing we
can do whatsoever i don't think that's
exactly true i think it's close to true
but i'm exactly true i do think small
cooperation between like say open ai and
deep mind yeah that seems pretty
possible i've talked to people from all
these organizations and
yeah i disagree with some of them i
think some of them are wildly over
optimistic but they're reasonable people
you can talk to them like you know um
sam ultman replies to my dms on twitter
you know it's like there's a lot of
reasonable people out there
but you know
at large scale getting like you know usa
and china to coordinate to slow down agi
progress
my opinion on that reminds me of the
classic parable on the scorpion and the
frog
and the story goes that frog is about to
cross the river a scorpion approaches
the scorpion asks the frog could take it
across to the other side of the river
the frog says well no you're a scorpion
you're going to sting me the scorpion
says but no that would be stupid you
know if i sting you
i'll drown as well so i have no
incentive to do that the frog thinks
yeah makes sense so the frog lets the
scorpion right on his back as they get
or reach the middle of the river the
scorpion sings the frog
the frog says well why'd you do that now
we will both surely drown in which the
scorpion utters the immortal words lol
i think that's how the story goes and
that's my opinion on large-scale
coordination as well yet a usa and china
to coordinate slowdowns ui lol
le mao
so what
do we have
so unlike other some other people i do
think that some progress has been made
about limb i am much more optimistic
about alignment than i was like three
years ago there's been more progress in
the last three years than i expected to
see personally
which doesn't mean it's a lot but it is
some i think there are some pretty
interesting directions that if they turn
out to actually achieve their their hope
you know the goal would very much change
my likelihood my outlook and how likely
it is we can make progress on these
various problems
but do i think we have a uh a clear path
to victory no as i've already said
so one of like the first class of
approaches is agent foundation
embedded agency kind of stuff this is
kind of the belief that we need formal
theories of agency optimization decision
theory blah blah blah you know we have
like actually understand systems on a
deep deep theoretical level
to be able to extrapolate to these very
powerful systems have very strong formal
guarantees about why
you know we believe certain things to
extrapolate or not
and
i think this is probably the right way
to approach the problem in the limit if
we want to reason about extremely
powerful systems one of the only ways we
have to do that reliably is through
mathematics and through formal theories
that we have strong guarantees and we
can think about proof you know
provability and so on
if we had enough time
so one of the main
problems with aging foundations is that
it's really
really hard and it's very hard to even
know whether we're making progress or
not on this kind of things
this whole approach is kind of
uh spiritually kind of started by miri
um
and they've written a wonderful post
called embedded agency where they kind
of explain some of the open problems and
why it's important it's a really lovely
post i really recommend you read it if
you're interested kind of justify
someone's problems and such and
i think this is
really interesting and really important
and if i had like you know long
timelines this is what i would be
working on
but yeah unfortunately i don't know if
we're gonna have the time and also i
don't know what's the deal with muriel
right now they seem to be going through
a bit of a crisis at the moment so
there's still some work happening there
but i'm not exactly sure what the
scenario there is but there's two
approaches that uh i would like to talk
about that i would like to highlight as
i think particularly interesting
directions toward in the age of
foundation kind of uh direction
one is the natural structure
hypothesization and selection theorem so
this is john wenford's agenda and i am
super excited about this this is some of
the things that have updated me to to be
more
to think that more progress has happened
over the last couple years i've seen
john wentworth into the field and some
of his approaches i find these so
interesting
then i'm kind of lumping two together
here so the first is the natural
extraction hypothesis which is kind of
like this this hypothesis that
different algorithms trained on same
data tend to have the same instructions
internally so an example here is say you
have a neural network an unsupervised
neural network trained on images like
reconstruct images or something like
that
very often if you train such a network
on in natural images they will have like
one neuron or something that encodes the
concept of tree you never labeled to be
clear the concept of tree you never
forced it to learn this concept they
just tend to often naturally pick up on
this which
is pretty interesting it makes sense in
some degree that you know there's just
some abstract you know trees are already
obvious kind of object but i think
there's kind of like a background
assumption an implicit assumption that
most people make that the kind of ideas
concepts that we humans have in our head
you know like i have like you know chair
and desk and computer and stuff are kind
of arbitrary
but the national structure hypothesis
argue that maybe not maybe that is
actually not at all arbitrary and
there's something about the environment
that makes certain abstractions natural
and more likely to be used by systems
working in similar environments
he has a lot of much more interesting
you know uh theoretical reasons to
believe that there's things in this
direction he is very optimistic about
this he is optimistic in the degree that
he thinks you will be able to in the
future take you know like say neural
network systems and compute out all the
abstractions it uses internally and
reason about those externally i am
much more pessimistic than him that this
will work or rather i think it probably
will work
but will be compute computationally
tractable but he seems optimistic
and
i think if this turned out to be true
especially for interpretability and
stuff this would be incredibly good news
if
networks just tend to naturally have
internal concepts that we can point to
about humans or about values or about
the environment or such might be
extremely powerful for aligning them
a somewhat related direction is what he
calls selection theorems which is kind
of a mathematical direction to think
about what kind of properties do
optimized systems tend to have
i really love how john explains how he
came up with these in the first place so
it makes intuitive sense why would this
be used for alignment or like ai right
you know if we have like reasons to
believe that systems when trained will
be
modular or non-modular or whatever that
seems like a really useful thing to know
about them but i like that his his
reasoning is that he came from biology
and he thinks the reason biology is kind
of like stuck in a rut and having so
many difficulties with producity and
stuff is that biology is the science of
reverse engineering optimized systems we
have systems that are optimized by
evolution but we don't have a theory of
what properties optimized systems tend
to have this again brings back to like
this like science two kind of thing i
was that i was talking about earlier is
that with science two we would have a
whole science of complex optimized
system what kind of properties they have
if you give me an optimizer and a goal
can i reason about what kind of systems
this thing would build so i think this
is a really interesting direction it's
very theoretical but still with enough
connection to practical results that i
think that i i think is both very
understandable john's a very good writer
and
very clearly if if these things can turn
out to be true or useful
this would very much update my
probability that things are going to get
well i'm a big fan of these approaches i
highly recommend you read some of john's
stuff he's also really nice and he's
easy to like ask questions or reach out
to if this is something you might be
interested in working on
this is definitely one of the things i
am most excited about of all the methods
i have all the directions i'm going to
talk about here
another direction is the learning
theoretic agenda or informationism this
is vanessa cosoy's agenda and uh to be
quite honest with you here i do not know
enough math to really understand this
from my understanding it is a way of
generalizing probability theory to kind
of imprecise probabilities it allows you
to reason about
uh about probabilities in a more general
kind of way so bayesian bayesianism is
great but it makes some really bad
assumptions such as logical omniscience
is that so it assumes that you know the
consequences of all of your data which
of course cannot possibly be true in an
actual system and also it has the grain
of truth assumption where bayesian
reasoning assumes that the true
hypothesis so the true generating
function of the universe is somewhere in
the hypothesis space but this obviously
cannot be true because you are in the
universe so you can't have a model of
the entire universe including yourself
inside of yourself because you get into
like a recursive problem there so
informationism allows you to reason
about systems more generally this is a
horrifically massive mathematically
complicated thing like this is some
intense you know this is you know
this is stuff that goes well beyond my
undergrad level understanding of
mathematics
but the parts i understand are
definitely very interesting
it definitely seems like a worthwhile
thing to think about how do physicalized
and embedded agents who can't model all
of their environment what are they
capable of modeling how much do can they
think about their environment how would
they make decisions in this kind of
imprecise probability way it seems like
a very promising approach it is worth
mentioning though that it also feels
like it might be a bit nerd snipey
it's like very beautiful mathematics you
know it has like you know incredibly
cool theorem proving it's like really
rigorous and elegant and all this kind
of stuff
and it's unclear
how much this will be useful or not
vanessa obviously thinks it's useful and
i mean that's a smarter than me so you
know that's definitely account for
something
and a lot of the people also find it
very interesting but
my my
my view on this is basically if you're
the kind of mathematician to be nerd
tonight by this kind of thing that hears
stuff like sa measure and ultra filter
and other horrifying topology and
measure theory type stuff and thinks
hell yeah maybe this is a good thing to
be nerd tonight by
so if you're the kind of mathematician
that kind of that would love to work on
this extremely intense mathematical
stuff as far as a last i've heard
they're currently hiring from
researchers so this seems like if this
seems like something interesting to you
i would love to see this theory more
developed and
if there's anyone out there who happens
to all be this good at mathematics to
understand and also explain it
good luck because like
five different people have tried i've
read all their explanations i still
don't really get it but it's worth a try
i guess
now the more
common approaches i think nowadays are
the mosaic approaches so this is the
kind of belief that working with these
kind of different approaches that work
with current tech level systems things
that work on your neural networks and
you try to train them or empirically
think and work it with these kinds of
systems
a general assumption here is that agi
will in some ways be similar to the
systems we have today so by studying the
systems we have today we will be able to
learn things that will be relevant to
more powerful systems i personally think
this is a completely reasonable
assumption to me so i have very very
short timelines i expect aj to happen
very very soon
and therefore i also expect agi to
fundamentally be a neural network
probably running on nvidia gpus sure it
might have a slightly different
architectures might have had a different
training objective or whatever they
expect lots of those principles we have
now to translate almost one to one to
hei there's already a lot of very
interesting things we can study using
the systems we have today but even if
not i think it's kind of unlikely that
the work you would do on this nowadays
will be completely useless some people
are really worried about this especially
with longer timelines they're worried
about i'm not going to work on alignment
right now because you know we don't even
know what age is going to look like so
the work i'm going to do is going to be
completely useless
um first of all i think it's really bad
reasoning but that aside
i don't think that's probably likely to
be true
i think there is this there's an
illustrious scientific history
of the success of a meth the method of
staring at an object of interest really
hard and i think this is very underrated
in our in many modern circles that
just studying the object looking at it
doing experiments as much as possible
while you still can
is an important
way of making progress in scientific
method in a scientific field like this
is it the way that's going to solve us
all the way
i don't know
maybe hopefully
the big question there is how much will
scale to agi assign again we have a
substitution asset situation where you
have this hazard of potentially you know
making progress on aligning a very weak
system and think oh well my method works
on somewhere in the small system so i
made progress towards the heart system
even if it reliably you can already
predict ahead of time this cannot
possibly work with a stronger system
speaking of which
rlhf so reinforcement learning from
human feedback rlhf i'm also
classifying recursive reward modeling
into this is a very common very popular
approach right now especially in labs
such as open ai and anthropic
is a very a very common approach right
now a pretty straightforward kind of
approach and
so i've been asked more my opinions on
these methods are
and um i guess the best way to summarize
my opinions is um hilarious laughter
followed by gurgling sounds as i'm
consuming my paper flips
i'm joking
but
but
hear me out
so rlhf in this most naive form is
a method where like train a model to
predict human judgment so you have human
judgment uh rating various inputs and
outputs you know whatever as good or bad
and you have some model to try to model
this human feedback process and then
control some kind of policy network in
order to produce more favorable outcomes
and this works great with current
systems this is a really nice method
it's tricky to get working it's very
finicky as rl always is but it's a
lovely method you know i know you guys
have tried and struck gpt or something
but it's really cool it's really good
and it works much much better for most
tasks and like other gpt models do
but
this naive way cannot possibly scale to
us i i claim at least i think some
people will claim it it might i like
really don't think that's the case if
you have a system that's just modeling
human feedback first of all it's never
going to do this you know robustly so
you're going to run into good heart you
could you're going to have your your
super intelligent model is just going to
good hard to the proxy and do something
crazy we already see this happen with
many systems that if you let it optimize
too hard it just produces like you know
random strings of characters that for
some reason trick the reward model and
make it out put a really high reward and
also even if it would perfectly model
human feedback that would still be bad
because the super intelligent system
would just reason well the you know this
human feedback process is done by a
human which is out in the environment
and the best way to maximize feedback
would just be to take over the human or
something of the source you know to
break out a simulation couldn't take
control over the rating function so even
if you somehow had a perfect proxy it
still wouldn't work
so there are some methods such as debate
that fall into this general vicinity but
might scale better so they use like rlhf
as maybe like one building block and a
more complicated algorithm i think it's
possible that there are directions here
where rlhf or or rrm will be useful and
maybe actually can scale the naive
version obviously can't and i think most
people also have like opening irons that
probably agree that that is the case
um
but i am not
super optimistic even about stuff like
debate and stuff i i'm more optimistic
it's not the i'm not maximally skeptical
i think it's something worth looking
into and i think it's worth something
some people should be studying and
trying how good does rrm and debate
scale and stuff definitely seems like
something someone should be doing
but i'm not very
optimistic about this stuff and i think
of all approaches this one has the
strongest substitution hazard and it's
very easy to see yours drop gpt you know
getting great user ratings and like
we've we've done it boys
alignment is no more but that is a very
dangerous assumption to make and this
leads to the whole unilateral curse type
situations so
is there something here maybe should
some people be studying it yeah probably
but be very very careful of substitution
hazards when working on this stuff don't
get lulled into a full sense of security
just because you've got gpt2 to you know
do what you wanted to do you have to
have a additional argument an additional
reason to believe why these kinds of
methods would scale to larger and more
powerful systems that may be able to
take actions that you couldn't predict
another interesting approach it's called
relaxed adversarial training and so
here's the problem set up you have a
model which is deceptive so it's trying
to break out it's trying to you know
screw you over whatever
and but the model's clever it knows well
if i'm gonna simulate if i'm in training
a simulation and i try to break out well
you know my handlers won't allow that so
instead it it does it uses internal
circuitry so that only when it sees a
valid bitcoin blockchain dated five
years into the future will it become
deceptive so that means you would deploy
it or whatever it would go fine then
five years later you know a certain
block is minted on the blockchain and
suddenly you know it turns deceptive
this is like a version of the rsa 2048
problem
the solution might be well we could use
some kind of method to show it the
concept of biblical blockchain did five
years into the future because we can
actually generate a valid bitcoin
blockchain data fives in the future
because that's computationally hard you
would have to solve like we don't have
enough computing power to generate this
thing so the general problem the relapse
ever zero training is solving is that
there are computationally hard to
generate observations
inputs that are computationally hard to
generate that might make the model
behave differently but
but the model might know would exist in
the wild but and they cannot be
simulated so there's a way for it to
know how you know how whether it's in
simulation or not kind of it's like one
thing so if if we could do this this
would be great you know this seems like
great idea we could catch a lot of
failure modes this way this would be a
really useful method for you know
training it to
not do many things we might not want it
to do even if we can't you know
generate an example of every single time
one small problem literally no one has
any idea how to do this or if it's even
possible in principle some people seem
to have some ideas some directions which
you know maybe are linked to something
and some people are pretty optimistic
about it like paul cristiano and i think
um evan hubinger are kind of are kind of
optimistic it was kind of directions if
someone can figure this out i think
that'll be great if this kind of
methodology would work i kind of expect
this doesn't work in the general case so
i'm not super optimistic about it but
some smart people are and if they get to
work i think this would be a great thing
then there is the almost the obvious
direction which is interpretability
slash elk or listening late knowledge
which is paul christiano's kind of
direction for how to uh trying to study
how to get information out of the
network
i think everyone agrees that this is
almost definitely good it's or at least
worth trying it's worth trying to like
understand our neural networks open the
black box what are they thinking about
internally what do they know what are
they capable of et cetera this is like
obviously good to try
there are some downsides i can see but
it seems like pretty clearly on the
burrito frontier and from one things to
try
i do think there are a ton of
low-hanging fruits here but
i think the main problem
with the scaling here is what we see in
this kind of graph
is that
simple models are really easy to
interpret they're like you know very
simple there's not many things
parts then as they get a bit stronger we
get this like valley of confused
abstractions we have these like really
weird polysemanticity they like don't
really understand the world they don't
really have like clean abstractions or
something then
as these mods get stronger they seem to
get like crisper abstractions this is
kind of what we're seeing with larger
models they often have like more
disambiguated neurons and such like i've
been surprised how much interpretable
stuff there is inside like gpt or
something like that it doesn't mean it's
definitely it's definitely going to be
totally human interpretable it might
just be not it might not be possible but
unfortunately i think it's pretty
probable that as the cells become even
more powerful they eventually reverts
or we have systems that use abstractions
that are so complicated that are so
insane that we humans can't possibly
understand them this is something where
the national abstraction hypothesis
would probably change how this curve
looks is that maybe the natural fraction
hypothesis if it turns out to be true in
a strong sense would make us believe
that even the very powerful system would
be more understandable than humans or
not it definitely seems like something
worth trying
i do think there are downsides to our
interpretability as well i've once
talked to someone who worked at uh with
the military in the usa and he said the
number one thing holding back the
deployment of neural networks and
military is interpretability
uh you know justification of why these
networks make the the the decisions they
do so
that's terrifying
and also i think it's quite likely that
you know when we study these things in a
very deep level that we will also
uncover capabilities increases we will
see you know bottle next to these
networks and counter or bugs that exist
inside of them and we'll be able to fix
them
so it's not as clear to me as it seems
for some other people that this is
always going to be a net positive on
alignment only but also capabilities but
to be fair i i personally think that
most alignment work is sort of
capabilities work as well
just something to keep in mind but
overall if i could like point smart
engineer types to one direction of work
that's like obviously at least worth
trying interpretability will be the
direction and i think there are a ton of
low hanging fruit here that someone
should try and you know i
i've seen again and again people coming
into the interpretability field and you
know just looking really hard at their
models and very often finding really
interesting and useful things so this is
definitely something i would strongly i
strongly recommend uh working in as a
very promising field
then uh it's a little something
obviously unusual which i'm kind of
called simulator theory so this is
something that we're doing conjecture
which is my
alignment startup
and simulator theory is a bit funky it's
a bit funky
it's a
different scientific frame it's hard to
describe it to think about gbt models i
think a lot of people tend to think of
gpt models as kind of like an agent that
you talk to some kind of likes you know
as a utility maximizing agent it's like
the default way we use to model
intelligent systems but we would argue
that a better way to think about gpt is
as a simulator of text generating
processes which which we call simulacra
and we think this view is very promising
to to give you a different view
of these systems
to to bring different questions you can
think of them you think of them as as
these simulators or also priors over
agents or even multiverse generators
where you have like this these like
different timeline world lines different
ways the prompt can continue and you
have a prior or a probability
distribution over these various lines
that and we think this is a
other than just being cool and
aesthetically pleasing we think this is
very interesting that it might it leads
to non-trivial implications for
alignment like what properties do we
expect these kind of systems to have
will they be optimizing in the same way
that normal agents do do they have other
safety properties or capability
properties it might be interesting we
think there's a lot of really
there's a lot of non-trivial work to be
done here it's kind of like looking at
the same object from a different
perspective it's like you know we can
look at fluid as like you know particles
or it can look at it you know from like
its perspective fluid dynamics you have
different formulas you know you have
different theories describing the same
object and different theories will
highlight different parts of the object
as worth studying and that's kind of how
we think about simulator theory it's a
different perspective on it on gbt type
models which will highlight different
questions
will this lead to something really
useful
 it why no you know maybe not but
sure is worth trying it sure is
interesting and we will be
uh and yeah so eliezer said he couldn't
completely dismiss all of our theories
he sure as hell tried and i think he
thinks he'd dismiss all of them but we
think he didn't and we have several
things we're betting with him on
so that's definitely promising and we
will be publishing more about this
hopefully very soon where we're gonna do
introduction to simulator theory um and
explain why we think this is a promising
approach and hopefully once we get our
experiments running we will also have
some bets with people like ellie ezer
the show whether our theories are right
it's quite amusing when he visited
lately we um there was a few moments
where he said you know something would
be you know x would be completely
impossible and uh me and kyle one of the
people we're gonna kind of look at each
other and like we could probably do that
in 24 hours
so we'll see who's right
definitely interesting direction if
you're interested and you know stay
tuned for this
so um
what now
i've talked to you about why the problem
is hard i've talked to you about the
approaches we have right now
um
but now what
i mean other than the obvious
i think if we need to roll high roll
mini dice
i don't think we have clearly any
approach that we just have to you know
we just have to buckle down we decided
to fund this one project we just need to
program this system or something and
then our problem is solved i think we
need
to get lucky we need to roll many dice
and find breakthroughs and this is not
typical this is how science usually
works right you know lots of people come
to think from different directions
eventually someone finds the
breakthrough
i think it is
hopefully not controversial say that
there should be way
way more people at least trying to solve
this problem like there's like what 200
people in the whole world may be working
full-time and trying to solve the
alignment problem that's ridiculous this
is like the most important problem in
the world and that's all we've got
yeah it's um a bit unfortunate
we should try many many approaches we
should comment things from different
angles and hope for a breakfast from
nice back to direction i think it is
possible but as i said before
i'm optimistic i have very short
timelines and i'm real and i'm honest
about that i don't expect this to go
well i think something crazy has
happened in the next couple of years
in order for this to go well and that
can happen crazier things have happened
in history crazy things happen all the
time but we do need to try many
different things we need many more
people trying getting it you know trying
the problem coming at things for
different programs having different
perspectives you know trying their own
theories or work for someone else's
theories or you know going off in
complete different directions or you
know big you know mega teams coming
together to work on one specific
approach or whatever whatever we need to
do is we need more people trying to
think about the problem that's my number
one thing i would advocate for
there also bottlenecks are not strictly
research that may also contribute much
which i think is also worth keeping in
mind such as you know ops roles in
alignment organizations really important
stuff and also fundraising and you know
various kinds of you know just like
normal engineering or design work can be
really important this kind of stuff so
don't count yourself out just because
you don't consider yourself you know a
researcher or a
genius mathematician there's a lot of
work to be done a lot of things that are
worth trying and we've barely scratched
the surface in what is possible and what
we should be trying
so
thanks everyone for coming to my talk i
would like to just you know show my
company one second here conjecture is
hiring we are an alignment startup
working on hopefully incubating many
different approaches to alignment
problem we're currently a bit a bit you
know biased towards gpt to kind of uh
brazilian research was not the only
thing you want to work on
and we are far from maximally stupid
quote eliezer so we have that going for
us
so thanks everybody