The Inside View #3–Evan Hubinger—Takeoff speeds, Risks from learned optimization & Interpretability

to give a bit of context to the viewers
um so mira is the machine intelligence
research institute can you just like
give a brief recap on like how long have
you been working there and what you
do there and what have you been doing
before like um
past few years
yeah
so
uh i work at mary i am a research fellow
there
i work on
broadly
sort of very broadly i think about inner
alignment um
which is sort of the problem of
how do we align
the models that we train
with the
sort of objectives that we're trying to
train them on
um i tend to think about this sort of
problem from a
um
prosaic perspective from the perspective
of thinking about concrete machine
learning systems also from a theoretical
perspective so trying to think about
machine learning systems and understand
what they're doing
um
using sort of more theoretical tools and
abstract reasoning
rather than sort of concrete experiments
so that's sort of broadly what i work on
what i think about
and yeah so you're talking about
empirical work
so i remember um
when i first learned about new it was
um because of the conference in 2018
with vladimir mykulik and it was just
after the
mary summer fellows
and you guys were wearing the mesa
alignment paper so it was mostly
theoretical and then and then i think he
worked at open ai
on theory there but the old company was
more
you know experiment focused
um and as i think maybe before you have
some more like software engineer
background so if you have
um
like different interests uh and
different expertise in those domains
yeah so i did do a bunch of software
engineering stuff
um before i got into the icp
um
the biggest thing that i might be known
for in that domain is i wrote a
somewhat popular functional programming
language called coconut
um
and then i
actually the first thing i did in ai
safety was i did an internship at miri
um
sort of more
functional programming type theory stuff
but all sort of software engineering
um
and then uh sort of went to the myriad
summer fellows program i worked with
vlad and other people
on the riskier optimization paper
and then
after that
when i graduated
when i finished my undergrad i went to
openai and i did some theoretical
research with paul cristiano there
and then when that was done i joined
miri as a full-time researcher and
that's where i have been for the past
year and a bit
yeah for for people not familiar with ai
element which i think is not the most
of the listeners but like bulk channel
is uh one of the og in empirical airline
research now after you you had koski so
like interning with them is pretty
pretty high bar
and it's pretty cool
yeah it's pretty good to have done that
after your undergrad
um
and
yes so yeah the the library you built um
was also he said it was like functional
programming and the stuff that miri was
also functional programming so if i
remember um
samiri as art one of the
leading programmer in functional
programming i think mostly on scale
maybe i'm wrong
are you talking about ed comet yes
yep he works in mary
yeah he's uh he's a really big high
school guy
but coconuts
coconut is more like an interpreter
or or something on top of python like
you can do programming
it comes in python
yeah so
and then the syntax is a superset of
python 3
and then it compiles any python version
and also adds lots of functional
features and stuff
when say it compiles the python does it
compile to
like uh syphon or does it compile into
like pi pi
python source
okay cool right thanks first
um so is it like very poorly optimized
like if you need to
like put something that converts python
source and should be like super slow
maybe
not super it's like the same speed as
python because you just compile
when you're compiling you're not doing
perfect work
but you don't you don't compile it run
time right like with with c you're not
like well the speed of c is well first
you have to compile it and compiling it
takes a really long time then you have
to link it and linking it takes a really
long time
you're like no the speed of c is you
compile it beforehand then you check how
long it takes to run
okay cool yeah i got it now
um
yeah and i think i think that's
especially interesting to me because i
think people like there there's lack in
open source
uh at least in the ai element place so
even if coconut is not especially for a
element i think um functional
programming might be useful for mary at
some point
um yeah it was definitely how i first
got into like doing stuff at miri
because miri was like doing a lot of
functional programming stuff and i had a
strong functional programming background
since they were like usually some stuff
here
sure yeah and and i think i think at
some point they were hiring more
programmers i don't know if that's still
the case they're still hiring more
programmers
um
yeah i think that has changed i don't
know what exactly the current status is
i'm not the person to talk to you about
that
okay
no problem um yeah so um i guess now
your job is mostly
um
writing interesting research
on the ailm forum sometimes posting it
on archives
um
and yeah i think i think your posts were
are both um
very precise in terms of um vocabulary
terminology
and clear
and also short so like you can read one
and and not spend like too long uh and
then you understood most of the points
um
so yeah that's that's a good thing
most people don't try to distill what
they think about and you also try to
give like concrete proposals for how to
solve this
um so we did like a kind of a shift i've
seen in the past three or four years
which people having their concrete
agendas and how to solve things
um so one one kind of um conference view
not confirmed but like
um
something
some view you had that was uh opposite
of most of the ailment form which is a
forum for talking about ai element is
most people were talking about takeoff
speeds like how how fast
so that
to clarify
what full crescendo meant by fast
um
what bostrom meant by fast slow takeoff
and and then you you you mentioned
something else which was immersion news
versus the originals take off
um so maybe you can talk a bit about
that like summarize a bit the vlog
yeah
so
uh
i think uh yeah i'll try to i think yeah
so there's a lot of discussion when
people talk about takeoff speeds about
fast takeoff versus slow takeoff
continuous takeoff versus discontinuous
takeoff
you can even like summarize take like
takeoff is
maybe a bit poorly defined
um
uh some you you can even define that if
you want
to go yeah i think when people talk
about ai takeoff we're sort of talking
about well at some point we're going to
build really powerful ai systems and
then something happens after that
um and what does that look like
so you know
uh
do we build the first really powerful ai
system and then it very quickly like
dominates
the world uh it has like a decisive
strategic advantage
um the only thing that really matters is
what that system does
or we build the first ai system and it's
really powerful and transformative but
then also we build another system and we
build another system and there's a lot
of different systems that are also
existing in the world at the same time
multiple hour scenarios
yeah so like unipolar versus multipolar
there's a lot of different things you
can talk about seriously how quickly do
the ai systems get better
um are they are there sort of big
discontinuities in how they get better
um you know how concentrated is this
sort of is power over these systems etc
etc
so
yeah so one thing that i have sort of
talked about in the past in this regard
is
this homogeneity idea
which is i guess in my eyes the access
that i care about most that feels like
the most relevant and also the one that
i feel like
more confident in
and like i can make more definitive
statements about
your homogeneity is saying a homogeneous
takeoff
is one where all of the ais are
basically equivalently aligned
and a
sort of
homogeneous takeoff
is one where we have a bunch of
different ai's that are sort of varying
degrees of aligned
right so
there's a lot of different things that
happen in these different situations and
there's also some other aspects of
homogeneity so by default i sort of mean
alignment but also we can talk about
sort of homogeneity of other aspects of
how the ais are built so i expect quite
a lot of homogeneity i think by default
i expect to sort of be in a situation
where we have a lot of different ais
running around but all of those ais are
basically all just kind of copies of
each other or like very similar
or in a situation where we just have
like a very small number of ai's but
they're still just like
even if you if you have only one ai then
it's sort of homogeneous by definition
and so i think this is like in some
sense the more important dimensions so
so
i think a lot of times when people talk
about sort of
this sort of you know fast really fast
takeoff scenario means that we have to
get that first ai system totally correct
because we don't get the first ai system
totally correct in a really fast takeoff
scenario it very quickly controls all of
the resources and and sort of only thing
that matters is that system whereas in a
sort of slow takeoff scenario we get the
one system but then you know there's a
lot of other systems that are competing
for power and resources
and we sort of have the opportunity to
intervene and sort of control things
more as it sort of continues to develop
and my take is something like i don't
think even in the second scenario that
we actually have the ability to really
do much
even if there's lots of different ai
systems running around competing for
resources after the point at which we
build the first powerful advanced ai
system given that i expect all of those
other ai systems to basically just be
copies of the first system because if
they're all just kind of copies of each
other then what really matters is did we
align that first system properly and so
do we have a bunch of systems running
around that are all basically aligned or
do we have a bunch of systems running
around they're all basically misleading
um and so therefore i'm like well if you
believe in homogeneity then that
basically means you have to believe that
the sort of first powerful advanced ai
system that we build is really important
and critical and like aligning it is the
most important thing
regardless of whether you believe we're
actually going to end up in a very fast
pace
and so then there's the question why do
i believe in homogeneity
so i think basically i believe in
homogeneity for a couple of reasons
first is that i think i expect pretty
strongly that we'll be in a regime where
the cost of training
your ai is just like much much much
higher than the cost of running it
um and this creates a bunch of sort of
particular economic incentives so like
one thing that it does is it means that
you would generally rather use somebody
else's ai whenever possible than have to
train your own one
and
also in the situations where you do have
to train your own ai
um because let's say for example you
don't want to use your like
geopolitical rivals ai or whatever
then
um
you're probably gonna do so very
conservatively like if you have to spend
a trillion dollars to train your like
one ai system just because you don't
want to use their ai system because you
don't trust like i don't know if you're
like the u.s and you don't trust china's
ai or something
then like
you're gonna spend that trillion dollars
pretty conservatively you're just gonna
be like well we basically kind of know
what they did let's just do the same
thing because we really don't want to
spend a trillion dollars and have it not
work
um so
and so in this sort of regime
i expect
and then there's some other assumption
here which is like well i think
basically if you're like running
essentially the same training procedure
then like you get essentially the same
degree of alignment
um and the like really small fiddly
details um are not what what sort of
contributes very much to like whether
your thing is aligned it's mostly like
did you basically get the incentives and
biases right or not
right so i guess your
your example with the u.s and china
would be something like
um gpt3
being not like taking
5 million or something or 20 million to
train
maybe much more to like pay the salaries
and like all the experiments that went
before beforehand and you didn't want to
spend like that this is like relatively
cheap for the government now but if we
if you're talking about billions of
dollars then maybe it's like more
expensive or trillions for the entire
country
so we get to a point where it's like a
substantial fraction of the entire
of your country yeah like you really
don't want to like be in a situation
where it's like we're gonna spend one
percent of our gdp
like
like there's this other country that has
already done this like built a like
really powerful ai and has demonstrated
that it works in this like particular
way like let's say they use transformers
or whatever like you really don't want
to then spend one percent of your entire
gdp trying to like make it work with
lstms like you're just like no they did
it with transformers we're gonna like do
the same thing like we just want to have
our own version of what they did
um and so my default expectation is this
sort of conservatism which is like well
probably people are just gonna copy what
other people are doing and so like it
really matters what the first thing is
that's like super impressive enough that
it gets everyone to copy it
sure yeah so so if we take the example
of gp3 then they didn't release the
weights and it was super expensive to
produce like
um according to different sources google
might have reproduced it um
like uh pretty fast but it's not public
uh information
then there's like a luther ai which
tried to reproduce it for months and
after i think after like six months or
something they did reproduce something
that had somehow similar results i'm not
sure how
close what they're aiming for is
basically a reproduction it's not a like
you know it's not worth spending all of
those resources if you don't already
have the evidence that it's like gonna
succeed yeah yeah so they know it's
going to 60 as a paper it is known to
succeed but
at least you have the architecture and
you know the outputs you know the loss
you know what's the expected behavior um
but you're surely not sharing the
weights and nor the data so you so you
have to both like crap scrap the data on
the internet and then
um get the like so the data collection
procedure was pretty similar right like
we're just like well the data collection
procedure the basic data collection
procedure is we're just going to scrape
a bunch of like a large you know swath
of internet data you know maybe they
didn't explicitly release the data but i
think that like i guess my take is that
if you're basically if you have
essentially the same data collection
procedure you have like essentially the
same architecture you like have
essentially the same training process
then like you're basically going to get
the same alignment properties out the
other end
um and some people would disagree with
that um
[Music]
but yeah i think i think i really agree
with that um i was
i might just be pointing at um like
maybe in the future when we get closer
and closer to a
to some kind of um human level ai and
maybe people might share less of their
research um about like how you collect
the data and stuff
um and we just have like
super hard to replicate results because
we do not we don't even have the
architecture or something
uh we just have to output or
um
yeah i'm not sure if like the the
opening eye in 2025 or 2030 will
actually share everything with the other
companies and how long it will take to
reproduce it
might be like like we will depend maybe
like
big enough a big enough gap
that they can have a comparative
advantage that uh leads to like a
uh then to lead the rate or something
it's hard to keep your just like basic
method secret
um
i think i mean so currently you know
most most of these companies don't even
try right like you know like you're
saying you know opening i just publishes
papers on exactly what they did and they
don't give me the weights but they
published the paper saying what they did
anything with deepmind you know google
basically
um i think a part of the reason for this
is that it even if you wanted to keep it
secret it's pretty hard to keep just
like these very general ideas secret
because like one person leaves and then
like you know they can explain the
general ideas they don't need to they
don't need to steal anything from you
it's just like they already have the
ideas because this video is very general
the moment you hear about it it's like
the moment we heard about like when 53
became mainstream let's say july
um
like it was released in some kind of
maybe may maybe they have the results
maybe in january or something and they
had some some
close results and then they they had to
like make it better or like improve it
for publishing to learn to interact with
something uh for
uh dolly maybe they had they knew about
multimodal for a while and i didn't mind
i think the politics is that they hold
up like their private research a bunch
of projects going on for like months and
then they try to publish it to like
nature neuroscience or like nature and
so do you have like all these deadlines
for papers where they have those those
six months advantage or maybe a year
advantage where they all have private
information that they don't share to
other companies
yeah there's definitely a delay
but like at the point where you have
this really powerful system and you
start doing things with it such that
other people start wanting to build
their own really powerful systems
i expect
conversions
people are versions of of of people
sharing stuff well people are going to
figure out what you're doing even if you
don't try to share things
like it's just too hard to keep it
secret when you're like you're like
having this big model and you're out and
you're doing all this stuff people are
going to figure out like what the basic
process was that you did
huh like
reverse engineering or like social
hacking
people like you just need one person to
basically describe the idea like
honestly i expect it's such it's such
it's like so not insecure with like i
don't know like can you imagine trying
to keep the basic idea of like
we used tension secret like
i just like i don't it's not gonna
happen
um yeah we use a transformer but please
don't tell anyone
like you can totally i think it's
totally practical to like keep your
weight secret because if somebody wants
to steal your weights they have to like
actually you know copy them and
exfiltrate the data and like it's
clearly illegal and like you know
it might still happen like i totally i
think it's still quite possible that you
know you'll have like hacking attempts
where people steal your weights or
whatever like it's least plausible that
you can keep your weight secret like
there's just no way you're gonna keep
the like
we use the tension to do a big language
model secret
um i think i think one example that goes
in your direction is
uh dali which i think uh so did this
github account lucid reigns reproduce
dolly in a couple of weeks i think maybe
like less than two months
um
something like that i think like dolly
was maybe end of december beginning of
january and and lucy range published it
in maybe february or something so for
some experiments um like the
it gets faster and faster to reproduce
it so i i was talking to someone else
who told me that um
he expects um multiple scenarios
to be
um like kind of the default because as a
as an entire community we get better and
better at reproducing what the best labs
do so
the time between something like 23 and
reproduction gets closer over time
um
and
yeah so i i guess one counter counter
argument or
question i had about this blog was
um
you say that the first like when the
first is a line then people will copy it
and
by default will be aligned because it's
the same kind of the same vibe or same
architecture but um
imagine imagine the thing that is
aligned
as like uh some kind of those laws of
robotics or oh don't kill people or be
aligned with human values and stuff so
if you're if you're trying to be
adversarial and and trying to like beat
the first ai or as much as ai alive
so not implementing those align features
um
you you could just like be very
adversarial and attack the first one who
will not attack you back because he's
like a human line right i'm
well so one thing one thing which you
could imagine is a situation where like
the first person the first organization
or whatever builds an ai and they they
like successfully make it aligned and
then somebody else decides to build an
ai and like not do the alignment parts
exactly like this relies on there being
some separability between the alignment
parts and the non-alignment parts which
is unclear um i'm not sure if i actually
expect there to be this sort of
separability
but then also
it relies on like you not caring about
alignment as like a good visitor
but that seems really unlikely like
alignment is like pretty useful for
almost anything you want to use your ai
for like you would really like your ai
to do the things you wanted to do it's
it's like very
it's like not it doesn't really seem
much different than just like you know
and especially again like when we're if
when we're in a situation which is like
what i expect where these sort of
training runs have a truly astronomical
cost right we're like you know your your
little research lab in like a university
or whatever isn't isn't going to be able
to replicate biggest laps because the
things cost like
you know billions of dollars for a
single training run
trillions of dollars or whatever then
you're in a situation where like
um
you really don't want to risk spending a
billion a trillion dollars or whatever
and have it not be aligned have it like
not do the thing which you wanted to do
right so
you're going to copy whatever alignment
features the other team did that they
like successfully demonstrated and now
in fact the first teams thing only looks
alive maybe it's not really aligned
you're still going to copy it i think
and so this is why i sort of have this
view which is like well even in
a sort of very multi-polar continuous
slow takeoff world
we still basically just care about
making sure that the first really
powerful system is aligned in the like
traditional conventional way that we
think about in the
fast discontinuous takeoff world
right so i think what you're mentioning
so
um one one of the posts where you're
trying to distinguish between like
clarify alignment terms i think both
channels right clarified in one of his
blogs and medium
which is
um alignment is basically
um having an
ai doing what
you tell him to do so very basic terms
and then you distinguish something
uh like impact alignment and sorry
intent alignment which is
trying to do what the human wants you to
do
and then impact alignment which is not
destroying the universe like not causing
harm to humans is it essentially correct
or maybe you want to nuance it yeah so
impact alignment is like uh in fact not
doing bad things
intense alignment is like
trying to not do bad things
um in my definition
and then we can further split each of
these things down so
intense alignment
can be split into
um outer alignment
which is like um
is the objective that it's trained on
uh aligned and then
this objective robustness thing which is
like
does it is it actually robust
according to that objective
um across sort of
different possible situations
and then
i don't know you can the sort of the
whole post has this sort of breakdown
where it sort of looks at okay how can
we sort of
um break down these different ways of
sort of thinking about alignment into
like different sub problems
yeah i was trying to find the actual uh
the actual picture from the blood clot
because it's pretty good
um
but yeah i think it's in clarifying
inner alignment terminology
i was just trying to see if i could make
it as a background
for fun but
um i think it's pretty hard
let's see um yeah so i i think
so what you were mentioning before is
kind of um companies would want
um
something like
intent alignment
or like something that
does what the chinese government wants
them to do and if the chinese government
wants to kill everyone in the u.s or
like take over the world
then it must be intended line in the
sense of trying to optimize for
like a do do do the same thing that the
chinese government wants them to do but
it doesn't mean that it won't kill
um
other countries right
so
um
so when one like the the second actor
might want
just to have something useful without it
being beneficial for the entire um human
race
yes so you still have even
even if you have like
um you know you solve the sort of like
um intense alignment style problems
um and even if you have a very
homogeneous world
you still have situations where you have
like standard human coordination
problems
where you have to be able to
um
coordinate between you know
one human organization is trying to
um
you know fight some other human
organization and then the hope is that
human coordination is a problem like you
know
humanity human organizations fighting
other human organizations is a problem
that we can mostly solve via like normal
social
and political dynamics
it's still hard um
you know we did make it through the cold
war without any
nuclear bombs being dropped but it was
close
um
hopefully if we're in a similar
situation we can make it through
something like that again
um
the like real problem though would be
you know
at least we want the opportunity to be
able to do so right we want humans to be
the ones that are in control of the
lovers of power such that it is you know
even possible for humanity to be able to
coordinate to make it through a like
similar
to the cold war style situation
um so humans humans have the total
levels of power at all
then we can't even do that
right so it's um necessary it's a
necessary condition to have a peaceful
outcome not a sufficient one
right
okay
um
yeah i think i think that's essentially
right i
i i hear that
um
then then i guess some people might say
that
um the i think the political problem is
maybe the hardest one and like if we
have some kind of
very authoritarian regime
um
that i don't know um says
working on ai is uh
you can go into trouble with your work
on ai to advance ai
and like everyone is doing
all good old like um
man
manufacturing job or like every
agriculture job something then
then if we solve the political stuff
before and we have some like some large
piece of like large government then we
solve kind of
we have more time for for ai safety
um that's a bit like the stillman of the
other position
um
yeah i mean certainly i think there's
lots of ways that you can approach this
problem
of just like ai extensive risk in
general there are there are like social
and political aspects of things that
are worth solving as well as technical
aspects i think that currently i'm like
well i feel like most of the existential
risk comes from we're just not going to
be able to solve the technical problem
um right yeah most researchers um
working on this i guess think uh similar
things
which is which is i think it's pretty
tractable i think i think the solution
that you give in your
other blog posts are pretty soon pretty
well practicable
and
uh solving politics or
human coordination or fighting against
moloch seems a bit harder
but yeah so i agree with that this is
worth thinking about it
um
[Music]
and yeah um
okay so i think i think we covered most
of the this blog post um and then
sure um you had another one
that was interesting to me because i
worked a bit on airline research myself
i did some open sourcing as well
on quantilizers
and so that's a bit what i'm familiar
with in terms of research
and you worked a post on
um
like how to like a
quantileizer essentially if i want to
summarize it would be
um we tried to have uh ai's that
perform some in some kind of human way
um without it being
um
too bad at the task so if you if you
have a human demonstrating a task
um and
the
let's say a robotic task or or some game
playing or
um feeding feeling feeding another baby
or something um you want the ai not to
find the the optimal action because the
optimal icon would kind of hack the game
but you also um want it to perform well
and not do stupid things
um so if the human is like bringing
alcohol
tenth of the time of the day you don't
want him to drink you're doing the ai to
drink alcohol you want the ai to do the
your normal human stuff
in the afternoon so quantilizer is
essentially taking those um 10 to this
quantity of actions that are good but
that are still human-like
uh so when the humans
when the ai tries to imitate uh those
actions
um then you can like perform so
one one thing about question letters is
doing that
uh i guess there are other ways of
seeing fertilizers but this is one way
of looking at it
and so in in your posts
um
you're kind of uh talking about uh what
judkos keys
um how it defines um bits of
optimization
which is uh when you have this interval
of uh length
one of probability mass
then if you if you have something that
is in the
one in the half of
um highest utility
then
um
you're somehow um
holding the space in half
so you're having one bit of
optimization or something and and then
the closer you get to the optimization
power
uh the more bits of kind of optimization
you need
um yeah maybe you can
say that better and you and said
um explain more
so i think you're referring to my post
on operationalizing exactly
exactly
yeah so in that post
i talk a little bit about optimization
power and
quantilizers i give a definition of
optimization power in terms of
quantizers
and then i try to relate this to the
strategy stealing assumption and
um value neutrality verification
um maybe the best thing to do will be
yes i'm happy to talk a little bit about
this post i think it's certainly i think
it's interesting i wrote it so some
things which i think so
yeah maybe i'll start with strategy
ceiling so i think strategy ceiling is
an interesting thing because i think
there's a lot of different ways to think
about it
to be like very simple thing is that
there is this um
there's a formal result which is well if
you have a game and it's like symmetric
we have a bunch of different players and
they're all sort of you know playing
you know they they have like you know
access to essentially the same actions
and stuff
then
you can always just sort of copy the
strategies of any other players such
that you can never sort of do worse in
expectation than if there are n players
getting you know one over n of the sort
of total resources
um
you know even if there is like
yeah in any situation even if there's
like a
as long as it's symmetric you can sort
of make this work
okay
so
what um
what does this mean well so one of the
ways in which we interpreted this and
paul sort of talked a lot about this
is that we can sort of think about this
as giving us a sort of general
visitor rather for ai safety because we
are currently in a situation where
humans sort of control most of the
resources
and so
we should expect that
you know by default
that sort of distribution of resources
shouldn't change over time because we
can just sort of you know if other
agents are introduced and they start
with very little resources then we can
just sort of steal whatever strategy
they have and you know in expectation we
should have the same proportion of total
resources that we have right now
in any future point in time
all right but then this fails for ai
because ai is sort of has this property
that
it might be systematically biased
towards certain values over other values
so for example values which are very
easy to specify and so we can build
reward functions for them really easily
we'll win out systematically and so this
great strategy ceiling because now it's
no longer symmetric because some some
some people's values that are easy to
specify will like systematically move
similarly values that are really simple
such that they're really easy to find by
default in sort of training processes
will sort of systematically
and so one way we can in which you can
think about one thing we might want
a sort of aligned training process to do
is
not be sort of
systematically better for some of these
sorts of values and other values and in
particular not be systematically better
for like you know simple or easy to
specify things than for like actual
human values that we care about
um and so one way in which we can define
that notion
is using sort of this this concept of
optimization power and asking you know
to what extent is it is it applying more
optimization power is it sort of able to
apply more optimization power to
some sort of task than other tasks and
in particular if it's if it's able to
apply you know more optimization power
to
um
i have this example which is like we we
consider
um
sundar pichai who is the ceo of google
and he wants a bunch of different things
he wants like to be happy and see for
his family to be happy but also he wants
google to make money
and so he has like you know really
powerful ai system and it's like you
know trying to do what he wants and so
he's like okay you know here's some
things i want you to do i want you to
like you know find some ways to make
google money and also you know find some
ways to like help me
you know
figure out my life and also like you
know put i also kind of you know he
probably cares about humanity and he
wants to be
basically good spot overall you know but
also he wants google to make money
obviously
and and so you know the ai goes out and
it tries to do this but there's like a
real problem if the ai just like is much
much much better at making money for
google than it is in any of these other
things right then it goes out and it's
like well it makes a bunch of money for
google and it's really really good at
making money for google but it's like
very bad at like doing any of these
other things it doesn't really know how
to like
put the world in a good spot for
humanity it doesn't really know how to
make sundar happy he doesn't really know
how to do any of these other things that
sundar cares about and so from sunday's
perspective what ends up happening is
that you know some of his values lose
out to other values and so in in the
long run we end up in a situation where
we've built ais that sort of
systematically favor the development of
and sort of enhancement of certain
values the enhancement of like
competition values like you know getting
more money for google
at the expense of these other values
like
you know is the world good um and this
is bad and so we'd like to avoid this
right i i think that kind of uh
resonates with
uh how easy it is to like hack the human
brain
and optimize for like facebook ads or
like tick-tock views
uh and it's harder to specify make
humans happy in the long term
um
so like we would kind of
converge towards
uh easy to hack you know brains
uh behaviors and maybe like even like
optimizing for the crypto market
or optimizing for the trading market is
something with very um little uh
information
and like very low demand dimension uh
compared to like visual inputs
so uh maybe maybe ai will be good
at things that are
easy to do now
and that are
tractable in terms of input space
i guess
um but yeah for cindar i think like the
it's it's kind of fail so what you're
saying is it's it's kind of uh so the
value is it's like ai would converge to
what is easy and it was easy is
um
maximizing profit then it will do that
instead of other things but if it
understood that what sunder wanted was
actually
money make google making money to
benefit the world and making his life
good and it it will not like create some
kind of
um
bad google doing bad for humanity and
and having sunder work uh over work
overnights and and like don't not spend
time with his family or something so at
some point
right
if it if it only
like you know you can imagine a
situation that is just kind of like the
current world
um
but we're like we know how to build ai
systems that do like
very simple tasks that we can specify if
we don't know how to do systems that do
really complex hard discussed by past
then like we could very easily end up in
a situation where due to competitive
pressure sundar is just kind of forced
to use these systems and like sacrifice
all these other values to like make sure
that google's able to make money
but like there's just no ability to
because you know the only powerful
actors in the world are these ais
and
um
like but they can only be used for these
sort of simple tasks then you you sort
of just you're forced competitively to
like keep deferring to them and giving
them more power and resources to be able
to give you more power and resources but
you'll never get to a point where you
can actually use that power and
resources for like what you actually
care about
so to be clear this isn't the sort of
world that i expect by default
um right but it's it's worth sort of
pointing out as like
in a sort of way of thinking about a
particular type of alignment problem
that is like not the traditional
alignment problem and like doesn't
necessarily
um isn't necessarily solved even if you
solve more
sort of other aspects of the
alignment
interesting
so yeah if you don't solve all the
problems then you might end up
here and i guess i guess the the thing
is some some kind of uh google like i
see it as a very powerful theory or
google home
where it would be like a good oracle
like senator pachai coming home and like
asking is google home or something like
yeah what's the best strategy for
tomorrow um yeah i guess somehow it's
not that far away
uh maybe like strategy wise running a
company like google is hard but like a
chatbot that you can talk to and like
has asked for simple decisions
um i don't know
um
okay so
okay and and and the link was the kind
of optimization thing
is that uh
mathematically we can use optimization
power to give a definition of this
that's not weird okay cool
and and i think i think that's
interesting because like in the past
okay so like first two episodes um i had
a thing i called um connor's role
because conor leahy um
he had other podcasts on like machine
learning street talks
uh with johnny kilter and stuff and and
they and they went on this on defining
multiple times intelligence
and so um he is kind of um the rule is
like you you you shouldn't talk about
intelligence you should talk about
optimization like other stuff that the
ai would do and not like talk about
words so i feel like optimization is a
good word
um
and you give a bunch of different
useful terminology
and um
like um so in risk from learning
optimization you can
um introduce
mes optimization like other stuff
and and then you clarify
even more in
clarifying inner alignment terminology
which is i don't know if you can see the
diagram before uh behind me
if it doesn't in my camera
it appears to be inverted for me but i
can't oh is it very for you sorry
because it was inverted for me
and then i inverted back so i need to
invert it back
um i don't know what the camera will do
at the end
i can have both
um
yeah go
ahead i will just remove that
um yeah what do you want me to say
um
yeah so about about like maybe you can
start with was um
um miss optimization and like optimizat
like how do you define option
optimizers
and what's that amis optimizer
yeah so risk on optimization does take a
sort of
stance which is something like
you know optimization is the important
thing to think about it's not
intelligence or agency or any of these
other things it's like optimization is
the key thing which is similar to sort
of what you were describing
right um
certainly there's a lot of sort of
discussion around this stance and
whether this is a good stance
um
i think it is
a stance that lets you say a lot of
things um because optimization is like a
reasonably concrete coherent phenomenon
and so you can say a lot of stuff about
optimization and this is sort of what
risk learning optimization tries to do
is say a bunch of stuff about
optimization
um
yeah i can i mean i can say more i'm
happy to like talk more and generally
about like what is
what is risking your authorization
basically saying what is the
interlinement problem etc etc yeah yeah
so i i think i think
i think one one useful okay so then
so i i recently reread the the actual
introduction so i think i think there's
like a sequence on ai element forum
where you define
um
there's an introduction where you define
all those concepts pretty precisely um i
feel like uh optimizer is
maybe you said that already but it's
like
um searching
anything that search over a space
uh to find the best solution according
to some objective
and actively search so
um for instance uh there was this uh
example from daniel phelan about a
bottle is is a
cap of a bottle is it a cap in english
um a bottle cap a bottle cup it is a
buttercup and optimizer because it's
kind of preventing water to go away so
it's not actually
optimizing for anything but it's
something that humans optimize for um
and so it's a result of an optimization
process from humans and humans are
um something evolutionary optimized for
as we are optimizing for different
things
um
like instrumental things like
um
having sex without uh without making
kids
or or other things but and so that
that's maybe some kind of disagreements
i have
on on your examples
um
and i think in in your broadcast with
daniel phelan in
ae xrp
um
you kind of said that it's useful like
it's it's useful to see humans as
optimizing
as a sorry searching for some solution
that is not directly
um evolution's function so in terms of
alleles uh chromosomes and stuff because
like some humans don't don't make kids
so my my country argument to that would
be that even for humans that don't make
kids
they're still like kind of
trying to optimize for evolution's
pressure in a in a bad way so imagine
they're i don't know good very good
researchers they don't care about
making kids at all
but they're just very passionate about
math so they will end up producing value
for the world
with their math papers that will end up
in like more
you know more
gdp or
more kids in the future for other humans
yeah
i think this is not how evolution works
though right like i think it is just
actually true that like if you really
let evolution keep running it would like
not select for this sort of behavior
like um
[Music]
evolution certainly wants some altruism
but it also doesn't want you to like
live down the street from a sperm bank
and not go there every day right like
that's insane from an evolutionary
perspective regardless of like what else
whatever else you're doing
but it's still like we're
like
i still feel like we're trying
so
like our our instincts like our
primal like our lizard brain still wants
to optimize revolution
um it's just that we're bad at it
or
or that we've evolved for
like building those tribes and society
that is a proxy for building
more kids
i don't know well the key word there is
proxy the things that we care about are
in fact proxies for what evolution cares
about but they're not the same right
like you know
you know you can certainly tell lots of
stories about and and this is true
because there are proxies right you can
think about you know status and power
and you know
uh sex and all of these things clearly
are proxies for and they're related to
in the ancestral environment the the
sort of you know passing your genes on
but
we don't
actually care about passing our genes on
at least most of us don't
um you know
and i think well like the sperm you know
do you go do you donate sperm or eggs is
a good example of like you know
most people don't or they have to be
paid to do it right and it's like that's
you know evolution
would would you know would never want
that that's like clearly you know
evolution is like this is the most
important thing you should be doing
right you know you got to be doing
nothing but that
but from a human perspective we care
about the other we care about the proxy
you know we're like uh you know you know
what i care about isn't actually you
know just like literally are my genes in
the next generation you know even the
humans that generally care about like
having children usually care about like
i want to be able to raise my children i
want to have a connection with my
children right not just like i literally
want more of my dna in the next
generation right what about those
proxies are actually good at
like making
like
more humans long-term like evolution
evolved and and found this new
solution in surf space which is no the
actual good stuff is not just to be a
lot
uh we actually need to be
um in some kind of tribes and have
social uh defense to find uh to fight
dinosaurs or monkeys or something and
and then if you if you if everyone was
spending sperm
would give sperm to
evolution doesn't work on a group level
though it works primarily on an
individual level and so evolution is
happening to like evolve to
extinction on a group level because it's
primarily selecting on an individual
level
but wait so
like
if you're selecting genes
um
this is why we have things like you know
um
selfish themes that like you know they
don't
it doesn't actually help you it just
like copies itself from like place to
place
you know it's like evolution isn't just
selecting for
um
you're you're sort of like the the
performance of the whole group but it's
like very explicitly selecting for like
your individual performance another
example of this is like sex ratios so
like in theory
you would like evolution for like the
maximum like production of additional
children would want like significantly
more females in each generation than
males but in fact what we see is that
across species the sex ratio converges
to 50
and the reason that converges to 50 is
that from a selfish individualistic
perspective even if you're in a if
you're in a population where there are
greater than 50 females then
you are an advantage passing on your
genes the next generation if you have a
uh a male child
um and you're at a disadvantage you have
a female child and so despite the fact
that evolution in from a group
perspective
would rather have a sex ratio that is
not 50 percent
the like from an individual perspective
it has to converge from two percent
because of like it's the sort of only
stable equilibrium from a sort of
selfish individualistic perspective and
evolution primarily selects on the
individual
and
but okay so but then like those
we we kind of converge so it's like a
bunch of individuals with um egoistic
genes
that convert to some nash equilibrium at
the society level
um
well so we can also certainly talk about
like why is it that humans are
altruistic um where did that come from
evolutionarily i think that like leading
theory is something like you know it's
it's good for
it's useful for cooperation
um being being altruistic
is helpful for your ability to cooperate
with the rest of the group because if
you care about the rest of the group and
they care about you then you can
cooperate really seamlessly with them
um and so in some sense altruism is
selfishly useful in this perspective
from an evolutionary perspective it's
like evolution would rather have each
individual be more altruistic because it
helps them work better with the group
right and less ostracized by the group
and like therefore have be more likely
for that individual to have more
children and so this is a like
individualistic story of why from the
perspective of a single individual
evolution would rather that individual
be more interesting and what about like
people being
like the opposite of altruistic and and
just like um
kind of um
like taking like defecting all the time
with autistic people like did this would
be like the more
uh
the better position right
no the point that i'm making is that
that's not the case
for evolution for each individual
altruism serves a purpose for helping
that specific individual have more
children
cool okay cool
um
interesting so yeah and i think i think
they're like other other distinctions
um you make that are interesting so um
so just to define the basic terms again
because i think most of the listeners
are not familiar with the paper so
there's
uh what we call evolution like a good
analogy for evolution is what we call
the base objective
um
of um
so if if we maybe maybe a neural network
is an is an easier example um
like what would it be better to start
with neural networks and we're risking
one motivation does it really try to
ground everything and i think one of the
big things that risk modification does
that sort of all previous discussion
didn't do is really carefully ground
everything in machine learning sure yeah
so let's talk about machine learning so
what's interesting is um when we have
optimizers like atom
or stochastic gradient descent
then you're trying to change your
parameters uh data um
so that you can i don't know better
classify guessing dogs and
at the end of the day when you
change those parameters at the end
um
they might end up
um a different inference time
doing something like optimization so if
you for for me the example for me would
be something like a regular neural net
where
you do backdrop through time
where you're optimizing
and you're
at inference time you're kind of um
only
using the
um
like the the laden cells or something
like some are frozen and some are not
and then you kind of you can adapt to
um what you get at test time um
and i think that was one example
of a blog on that from trying to
reproduce
uh
uh to to produce um
mesa optimization did you have better
examples maybe of um
i guess sub optimization
um
so the classic example that i like to
use to really explain sort of what's
going on with personal optimization is
this maze example right so um we can
imagine a situation where we train a
model
in a sort of
small
a bunch of small mazes sort of randomly
generated mazes but they're all kind of
small
and we put a green arrow at the end of
the maze it gets like a picture of the
maze and we put a green arrow at the end
to like say this is the end of the maze
we're supposed to go here
and we train a bunch in this environment
and then we deploy to a new environment
which has the following properties it
has larger mazes
the green arrow is no longer at the end
it's at some random location inside of
the wheels
but we still want
the agent to get to the end we're still
we still want the agent to leave the
maze
um
or you could flip it and you can be like
we still want the agent to go to the
green arrow and not go to the end either
way
um
the point is there are a bunch of
different ways in which this agent can
generalize so here's one generalization
is it just goes to the larger mazes and
it doesn't know how to solve it just
fails to solve big places
um
i would sort of call this it's just it
neither it's sort of capabilities don't
generalize it didn't learn a general
purpose maze solving capability
or
it could learn a general purpose maybe
solving capability its capabilities
could generalize
um
and it could learn to go to the end of
the maze which is what we what we wanted
to do and so its objective going to end
the maze also generalizes properly
but then there's another situation which
is it look it's capabilities generalized
it knows how to navigate the maze
successfully but its objective fails to
generalize and it learns to go to the
green arrow rather than to go to the end
and then
what's scary about this situation is
that where we now have a situation where
we have a very capable model that's
learned a general purpose optimization
procedure um but it's deploying that
optimization procedure for the wrong
objective to get to the wrong goal not
the one we intended to get to the green
arrow instead of what we wanted which
was go to the end of the maze
um and so this is really dangerous
because we have a powerful competent
model which is directed in a in a
direction never intended
um and what's happening here is that
there was this unidentifiability where
on the train distribution we couldn't
tell whether it was what it was really
doing was going to the green arrow or
going to the end and when we deployed an
environment where these two things came
apart now we can tell and if it does the
wrong thing it could be really capably
doing the wrong thing in this sort of
new environment and so this is one
example of a way in which a model can
sort of
have
fail
to have objective generalization its
objective can fail to generalize
properly well its capability is still
generalized properly which is the sort
of general sort of subheading under
which inner alignment is like trying to
you know address as a problem
so summarize it's good at like
general uh at finding green arrows but
it's not good at finding
um the end of the maze
um yeah
okay so that would be that would be a
situation where we're like unhappy
uh because
it's sort of
it's it's very powerful and knows how to
solve mazes properly but it isn't using
that those capabilities for the right
reason but not it's not using them for
the one we wanted to use for it's using
it for this other thing instead
so yeah we we can say that
i feel like it's somehow similar to
people who criticize p3 for not
understanding what it's saying
um but it's just like repeating and
memorizing things so you could say that
gp3 is doesn't have what we want them to
have which is
uh natural language processing or like
human understanding of words or in
concepts but just has like
memorization so like he he kind of
memorized the way of finding the green
arrow without understanding the the
actual task we wanted them to solve um
does it felt interest doesn't get
category or is it different
um
you can certainly think about it that
way i think it's like a little bit
tricky to really think about like
you know in some sense the like
objective
of gpt3 is like
predictive accuracy on the next second
um
it's a little bit hard to understand
what would it actually look like to sort
of generalize well or poorly according
to that i mean it's just like
if you have an actual distribution that
is similar
you know i guess in some sense but we're
you know we only trained it on this web
text corpus and then it moves to some
new setting where the underlying
generators of the text are different
then you know it might still be trying
to do predictive accuracy
or you might have learned a bunch of
heuristics that are not predictive
accuracy like what it really learned is
it should try to in any situation i'll
put like coherent sentences or whatever
and then it's like
it doesn't actually try to model the
dynamics of this new setting and like
get good predictive accuracy it just you
know tries to
you know do the simple like well i
learned to do
these sort of you know heuristics for
how good sentences work and so i'm gonna
just keep outputting that
right so he he he he he found those
heuristics
um and and i think one thing so there's
like two things that i remember from
conor's interviews uh i'm not sure if it
was with me was or was other people one
was we don't really know the entropy of
human language of like english
so we don't even know how hard the
problem is so it's very hard to say
exactly how successful it is at putting
words or like and understanding it
because we don't have a good model of
what english is
um
and
the other one which is kind of a funny
trick is that
i think it takes it took something like
one epoch or less than an epoch
so like it only passed through each
example once
maybe i'm wrong maybe it took more than
one epoch
but it kind of learned to generalize
from
um a few a few data just like passing
once you're learning i think one epoch
is in fact compute optimal in most in
most of these really big language model
cases
so yeah that's what's something
impressive in terms of so for people who
say like they're memorizing
it's memorizing yes but maybe but like
from one shot learning or something um
so yeah one epoch is
i don't think one epoch is very
meaningful here it's just like well it
got to see every data
right um
in the training job and so you know it's
seen the whole training that
distribution the it hasn't seen it
multiple times thing is more just a well
our training process just performs
better when it can extract like it's
it's sort of already extracted the
information from that from that data
point the first time through and there's
sort of diminishing returns from trying
to run it through a second time and so
it's not compute optimal to do so
okay so yeah so it's it's um okay so
it's it's
less expensive like just running it one
epoch is enough and and compute optimal
and otherwise you would just like lose
money because you don't you you don't
get um
as much um
value for your dollar or something
yeah
um i'm trying to find
um
yeah
the post from uh matthew
matthew barnard because he sent me the
code at some point on how to do it um
and i'm just trying to for trying to put
it behind me because that's the thing
i'm trying to do now
um putting stuff behind me
um so
give me just one second so it's it's a
it's mapped with treasures and chests a
keys and chest i don't know if if you
remember it then maybe we can talk about
it otherwise
um because i i don't i kind of remember
some form of this environment
which but maybe you you also remember it
yeah it's very similar to like my maze
example
um
where it's just like
there's a
there there's you know a sort of
objectives which are indistinguishable
on training and then we move to
deployment and you can see that it like
does one and not the other
yeah so if i remember correctly it's
like it would stumble on keys
and then because there would be like
more keys and chests so it would open
the chest without actually knowing what
openings chest is
and and then on big environments it
wouldn't really know how to do it
or you could still do it in bigger
environments yeah something like that so
yeah if you're a listener
uh i have some code for it and there's a
concrete concrete problems to
demonstrate uh
uh um
miss optimization um or inner alignment
failure is it an inert in their
alignment
well so inner alignment the way we
define it sort of requires there to be
optimization right we don't know you
know especially in the keys versus chess
environment where it's so simple the
model probably isn't doing any real
optimization internally
right so
so yeah i i think
i think i think that's like kind of the
the problem with
calling it optimization is that we're
kind of assuming some form of complexity
or some form of you know he's doing some
thinking or
some elaborate task or
finding some optimum somewhere of some
precise task so i remember that there
was this last one post about
a paper from deepmind
about meta learning so
um meta reinforcement learning
and it was like top of the
alarm forum for a bit where they showed
that it was similar to some kind of miss
optimization
and and then like um some people
commented that
it was basic rainfall learning that the
thing was doing it was not some kind of
very very very special trick it was just
like
um an lstm plus some rl and at the end
you throw the waste and then you get
some uh stuff that's going to adapt to
environments
so yeah i i guess like ai researchers
can always say you know yeah jupiter is
not intelligent it's just just doing
just memorizing
sentences or they think it's not
optimization it's just like doing
whatever uh he was trained on to do at
the beginning right
um
but there is a truth of the matter like
this is empirical question so we can
look inside of models if we have the
good enough transparency tools and
discover
how do they work are they doing an
optimization algorithm are they not
doing an optimization algorithm
um
that is something i hope we can
eventually do i don't think we currently
are able to quite get there but i am
hoping that we will eventually be able
to actually answer these sorts of
questions by literally looking inside of
our models
yep
and yeah
just to close a bit on this i think this
terminology is super important so and
just going to put that back
behind me one last time because i think
that's useful for the listeners so
is it in the right side for you now
yes yes
okay so um what we want is uh alignment
which is kind of what you said about
impact alignment which is ai that
doesn't do bad stuff
um
then capability robustness um is
you can correct me at any time is
the ability to like generalize to harder
environments or
out of distribution environments
is this correct like you need to be
capable enough to journal as well
yeah but it's sort of like generalized
according to what is the question and
capability robustness just says
generalize according to whatever
objective you learned it isn't saying
that you actually learn the correct
objective it's just saying according to
whatever objective you learn you
generalize well according to it
right so yeah you're capable of
maximizing
that reward of minimizing this loss in a
more general setting
uh than from your training data but has
importantly capability or business has
no reference to the actual reward it's
not saying according to the actual
reward you generalize well that's like
robustness robustness in general says
according to the actual reward you
generalize well capability robustus is
the subset of robustness that says
not according to the actual reward just
according to whatever utility function
internal objective we we what do i call
in the post sort of a behavioral
objective which is just like
the objective that actually describes
what you're optimizing for in practice
do you generalize well according to that
which is just saying do you sort of just
make a bunch of mistakes and not really
know what you're doing and don't have
anything coherent or do you like are you
coherently trying to do something
regardless of what that thing is the
correct thing
hmm okay cool so it's it's
again generalizing and doing what
you were previously trying to
achieve in a new setting
um
and then intel alignment is what we said
before as doing what the human wants to
do
um like if you say brim is empty bring
some tea
assuming is not killing the baby between
you and he
um
objective robustness is
lucky you object your objective is your
blessed to what i forgot
um
well so it's it's
yeah so maybe a useful thing also for
distinguishing capability business and
objective business
would be there's like another version of
this picture where i have it in terms of
like right um i can i can take the other
one is it in below right
yeah there's like the version of the top
which is like how i actually think about
it but then i think if you if you if you
think about these things in terms of
robustness a lot then like it may be a
little bit better to like start from
their boston-centric version they're
equivalent um
yeah i'm just going to check
uh okay so i'm just going to take the
robustness one
how do i do this
cut
no
um okay so yeah i think it is going to
go to your pros but you can start
explaining each one
yeah so
so i was trying to say so we have
robustness so
so in in the robust centric version we
split alignment into outer alignment and
robustness at the top level where outer
alignment says is the base objective
like doing the right thing and then
robustness says does it generalize well
according to the base objective which is
off distribution does it continue to
pursue the base objective and then we
can split that into
um rejected business and capability
robustness and then here i think the
distinction between effective business
capability of us is a little bit
um
it's a little bit yeah i think it's i
think it's maybe a little bit easier so
so capability robustness is saying
um so to make this distinction we have
to introduce in in addition so
previously the notion of the base
objective which is just like the reward
function of the loss function then we
also introduce a notion of the
behavioral objective which is like what
does it appear to be optimized
and then we say it's capability robust
if it's robust according to its
behavioral objective so
whatever it looks like it's optimizing
it's like whatever you know it does a
really good job of optimizing that no
matter where it is so it looks like you
know
if if we look at its behavior in general
it looks like it's going to the green
era and so we could say its behavioral
objective is try to go to the green
earth that's not what we want we want it
to go to the end of the maze but when we
look at what it's doing it's clearly
trying to go to a greenhouse and then we
can ask how good is it it go into the
green area and if it's really good at
going to the green arrow then it's very
capability robust even though it's doing
the wrong thing we didn't want it to be
the green arrow
and so the other part of robustness is
objective robustness which says how
closely does that behavioral objective
match onto the base objective which is
the one we actually want
and then a sub problem of objective
robustness is inner alignment which is
saying okay but what if specifically we
have a model which is an optimizer it's
running an optimization process and then
therefore it has some objective that
optimization process is optimizing for
which we call the mesa objective and
then we can ask inner alignment asks how
close is the mace objective to the base
effect
and then
the sort of point of of both of these
diagrams in this sort of overall point
is that
if we get both inner alignment and outer
alignment
um
then we you know and this is this is the
part that's harder to see on the this
version of the diagram on the other
version of the iron it's very clear that
inner alignment and outer alignment
implies intent
which is like sort of the i think a good
justification for why it makes sense to
sort of split the problem into inner and
outer alignment in this situation where
your model is a base optimizer
that is it's like doing optimization if
it's not a base optimizer then you can't
split it into
um
you you can't sort of you don't have
interlinement as a as a sort of concrete
phenomenon you just have objectiveness
and then maybe more it makes more sense
to look at it from the robustness
picture but i think if you're if you're
thinking mostly about your most about
mace optimizers then you're like inner
alignment plus outer alignment gives you
an intense one those pictures are
equivalent
they're just two different ways
yeah i think
for for the listeners the the kind of
errors are
kind of um sufficient ways of achieving
x
like if you have robustness another
alignment you get alignment and you
don't even need to have in your
alignment like if if we don't have
um if we if there's no mess optimization
going on and you just have like one or
two different process
then you can just have one optimization
process being robust and
outer line um so those are like
sufficient ways of
achieving alignment not necessary ones
this guard
yeah
um and yeah and and i think i think yeah
what you were saying is interesting
because i've started um
universe for information learning a bit
uh where the goal is to uh so you have a
human
having some behavior doing some stuff
and you're trying to guess his reward
function
uh from his behavior and his reward
function is kind of
uh what he wants to do and and and like
could be mapped to like some some like
his values or something of some sort so
if the if the human was performing
optimally according to his reward
function then from his behavior he could
infer
his reward function and so
uh this kind of behavioral objective
is
what
an ai
would be doing if it was optimizing for
the human's
objective function
if irl was tractable in some way
right
yes you can think of the behavioral
objective as being related to irl
inverse reinforcement learning in the
sense that
if you did inverse reinforcement
learning on some model
then you can think of the objective that
you get as a result of doing that as
that model's behavioral objective
like any for for any any sequence of
actions of any mapping from state to
action you can construct uh as um
a set of
optimal
policies according to those possible
reward functions right
um
or yeah uh utility functions or qr
functions
cool i think i think we covered that
pretty well
sorry for explaining basic stuff but i
think i think most of the audience
doesn't know about this paper anyway
um doesn't mean that it's not a very
essential one and one of the most
important ones in the past few years
just means that my audience is not
literate in that
so i think
most people so you talk a bit about
transparency and how it's important to
solve the egg element problem
um
and um i guess uh
chris ola is one of um an important
actor in that space
um there's other people i met or talked
to uh in the clarity space i think of
nick camareta
and i think they're like
it takes a lot of time to write a good
digital post
to explain stuff well and it's a lot of
effort and it's somehow um maybe you get
less exposure than like a tweet or
something
um
and
but but somehow you can say that gaining
like understanding of how ml mothers
work
is accelerating
um ml research and is also
um
having it giving a good feedback loop
between
uh
how does how do we align those and yeah
i think in your post you kind of gave
both of the
arguments encoder arguments from
crystal's views and and you're the best
proxy
uh one of the best proxy of priscilla's
views and on that today
yeah so
yes i think you're referring to the post
i wrote
sort of summarizing all those views on
agi safety i think
yeah this a while ago just so after
talking with chris i think chris has a
lot of interesting stuff to say about ai
safety
um
and i i think it's like under
i don't know at least at the time i felt
like it was under-appreciated and like
not really like people weren't engaging
with this sort of way of looking at the
at ai safety as much as i i wish they
were
um
it's been a while since i've like talked
with chris about this stuff so i'm like
not necessarily up to date on all the
stuff these days
um
i think it's from like in november 2019
so yeah one and a half years oh
something like that
yeah i think it was like a reasonably
accurate sort of like you know i went
and gave a draft to chris and a bunch of
times so going back and forth and trying
to get like
what what is what does he make sure he
agrees with it and stuff so i think it
was like reasonably accurate to like
at least you know a good summary of the
sort of stuff he was thinking about then
um
so so yes i think it's like definitely
worth
yeah um
and i i definitely think it's you know
chris is sort of doing a lot of
transparency stuff and it's like still
probably you know the person who's doing
the most stuff in the like space of
general
transparency stuff that is at least it
is relevant to like asap there's a lot
of other people that are also doing
exciting stuff like like daniel feilin
um other people
um
yeah i'm happy to like talk about
any other specific questions about like
yeah so
what i remember from east post was
um
so
there was this word in english that they
had to google which is called
mulligan i don't know if i'm pronouncing
it
right mulligan a mulligan a mulligan is
like a second chance
or something so if we if we don't if we
if we build ai it breaks or if we build
ai
that is not um something we can correct
that's not like it gives us a chance to
correct stuff when we mess up
uh being able to like introspect and
debug it
is instrumentally useful
in some way
yeah so i think that this is like this
is sort of one of the arguments that
chris makes for like why
interpretability is useful is the it
gives you a chance to catch problems
before you like deploy your system
and um so yeah you can charge problems
and there is something about auditing
um let me let me go back to see what was
it caching problems with auditing
so
um
yes you can see if it's not aligned
early on
uh which is yeah very similar to um
this thing about uh
um
i forgot the word in english
uh
yes the second chance thing
and um i think the other so i i think
the more like the debatable uh thing is
whether we're um is it worse or not than
like the acceleration in ml
understanding is it worth the uh gains
in ai safety
um
so i think i think he says that it's
worth um looking into it
um
and like i feel like
i don't i don't know how much we're
gained we've we've gained from learning
at inception
or resonant
like um
embeddings
um so i i don't think ml researchers are
much more competent from looking at
these but i'm also not sure how better
aict researchers are so yeah i'm not so
sure about the trade-off right now but
maybe in the future it's very important
to be able to to debug it so yeah i
don't know what what what do you think
are the upsides and downsides if you
remember
yeah so i can sort of talk about my
views
i think my perspective on transparency
is that um
we sort of need to be able to train
models in such a way
that doesn't just look at the model's
behavior
so i think chris has this view you know
like with capturing problems via
auditing where he's like well we train
some model and then we can check to see
if we did a good job with transparency
tools i think i have a somewhat
different view which is we don't want to
use transparency to check after the fact
we have to use transparency tools to
solve the problem in the first place
because if we just try to train models
to do the right thing via like
behaviorally making sure they look like
they're doing anything we can end up
with models that
aren't actually doing the right thing
and are just pretending to do the right
thing and the only way to eliminate
models that are pretending to do the
right thing
is via looking inside of the models
internally and training on is the model
actually
trying to do the right thing not just
sort of looking like it's doing the
right thing and so i'm sort of in favor
of approaches where we we directly train
models
to sort of
using transparency tools
whereas i think chris is sort of more in
favor of trying to use transference and
tools as a way
to check behavior as a sort of
independent check after we have
attempted to train a state model using
some other approach
right so so you're more like interested
in
kind of
looking at it
while you're building it so you're not
like um doing something like miss
optimization or
uh bad things or just deceptive
behaviors whereas chris is like more
like an app bus more than you you see
why doesn't why didn't work
yeah
um
so yeah i think i think that um
uh yeah i think this this diagram
behind me i hope it's the right way um
for everyone
so we we start from um some kind of
model
uh so in the y-axis is how interpretable
the thing are and on the x-axis is also
strong or like capable the ai is so at
the beginning you understand what it's
doing
then you start doing
something like uh
minist handwritten digit
recognition you don't anything the
neurons because you don't you're not
expressive enough
or maybe you you have some understanding
but not if you're a bit confused
um or or like those big models like
inception
resnet
or transformers are a bit more abstract
um
and then when you get to something and
then what we're trying just when what
we're learning is
when you look at um latency best from
gans or
um transformers we're some we're saying
something close to knowledge and we're
more and more understanding
um because it's more and more expressive
right and at the end when he's becoming
super human then it's very hard for
humans to understand because it's like
super optimized in a totally different
language
yeah
and
and so it's it's useful to do
um interpretability
um to be like in this creeps attraction
way of understanding ai when it's still
like kind of human level or before human
level
somehow
like it doesn't it doesn't go alien
before it goes to human level so we we
have some time
um yeah so i have some belief that we
can probably just avoid the drop off at
the end if we use
uh an amplified overseer to do
transparency
um so this is obviously you know an
oversight overseer
what is that amplified overseas oh
amplified or serious yeah so i think i
think that's like most of your proposals
um later
um well i think this will be like the
last part of the podcast in the last 20
minutes is just like your like 11 takes
on how to like combine
um
kind of uh amplification over here and
um
yeah interpretability but there was like
a little a little point that i i thought
i found interesting in case of like
field building so now both of us are
trying to
do some field building in the air
alignment
and chris allies may be more thinking
about
field building in
the interpretability space and
um
so
so you think like two i think if i
remember correctly the two arguments
that make it attractive for researchers
one is
um if researchers are in a lab
at the university they can still do
incredibility interpretability research
without having billions of dollars to
spend they can just look at neural
networks and make it
understandable and the any and i think
one of the assumptions he has is that
there are like some low-hanging fruits
in doing interpretivity research now
because not many people is like pretty
neglected
or at least it was in 2019
um
yeah i definitely think yes this is
something chris has certainly worked on
like the point of distill is to try to
get like you know interpretability
research more like
get more attention and more prestigious
and like more cool from uh and i think
he succeeded like the
the post about
i think it was going around when where
you visualize how
uh you visualize features in coin run
and how you mapped the reward that's
pretty cool
uh
um i think the microscope i think
microscope from open ai
where you see the features of all those
like
um sorry models what's really cool i
don't i don't know if you've done some
representation from
um
clip uh i think clip was
yeah i think clip was his own like
pictures and he didn't use me a
microscope i'm not sure
um
and yeah and i think there's like
another argument which is um
you're trying so
if you're forcing your models to be
interpretable
it's
and a good analogy would be forcing your
students
to show that that they've done good work
so um
like show their papers or show their
process
um so they're not good harding the uh
actual optimization but they like
showing everything
so they cannot this is harder for them
to lie if if they're transparent
explicitly transparent
yeah i think that's sort of closer to
the sort of thing i was talking about
where i want to use transparency sort of
as a training yeah maybe maybe not chris
and maybe like more you and on these
posts well chris also is like interested
in this but it's not his like
primary motivation right so yeah maybe
maybe let's talk about your your stuff
so you're i think your most important
post on
um
okay in my opinion
uh on uh
the alan forum or left wrong was
11 proposals
an overview of eleven proposals
buildings um safe advanced ai
and you have eleven so maybe uh we can
like i think like they're like five or
something um key points which is
transparency
uh over amplification
imitation identification
and then there's um
something like
adversarial training
and then
kind of combine the three with
microscope stem ai we weren't modeling
yeah you have like a like five or six
teams that that you combine uh you can
think about
maybe we can start with the first one
which is uh the one that talks about
empty amplification
i can put this slide behind me
yes there's 11 proposals
so
the the the
the second one is the first one that is
about implication
um right
and it talks about imaginative
amplification which is a specific sort
of form of application where
very simply we train a model
on the objective of
imitating a human consulting that model
and so i have a bunch of these different
proposals um they're not unique to me i
try to sort of take a bunch of proposals
throughout the literature and then i try
to compare them i think the sort of main
thing that this post does
is it's comparing all these proposals on
four
axes where these these circle axes are
outer alignment and inner alignment
which we've talked about and then
training competitiveness and performance
competitiveness we're training
competitiveness is how hard is it to
train and performance competitiveness is
if we do train it how good is it
and so all these sorts of for these
sorts of poor conditions are the sort of
central things that we need if we want
to be able to have a sort of competitive
and aligned procedure for building
artificial intelligence
um and so you know we can look at all
these different things that people have
talked about and like try to address do
they
satisfy these sorts of things
i think the general answer is well
it's unclear um but certainly for none
of these proposals we don't have a
really strong case that they definitely
do
um but certainly it seems like you know
we can say some are more promising or
less promising but that's going to
depend on like you know your particular
research case
um
i'm happy to talk about any of the
particular proposals and like right
so i i think i think there's like more
than just like uh will like
we can talk a lot about will it work or
not but then there's like is there any
concrete feedback loop that will tell us
if something works or not or is there
like any empirical environments or
research that can give us feedback okay
i feel like the
the whole like debate like amplification
work from paul was pretty um
empirical
whereas um
so most of the stuff you post is
empirical
but some of them are maybe more easier
to test so in in in the next years
i don't know about amplification because
that would require some kind of
recursive loop or
um
yeah i don't know where we are in terms
of of trying to do idea empirically
um but yeah maybe just basically i think
i think um
so the the first proposal was somehow
doing multi-agent safety with a bunch of
uh agents like the cooperative
hiding identique from open ai
and the second one um is about imitation
amplification
um so maybe you can explain a bit what
is going on here with the h m a and q um
which i think i think is one of the most
interesting and
useful thing about the other ones yeah
so what you have there is on the second
proposal which is about
imitative amplification which is
describing how important the
amplification works
so imitative application
you have uh
a
sort of you first need to find the
explication operator
which takes in some model m and produces
a more powerful version of them and the
way it does that is it says we have some
human which consults multiple copies of
m to produce an answer to some question
and then this process of the human
consulting m is what we refer to as the
amplified m so this amplification
operator applied to m
produces a more powerful version of m
which is a human with access to that
now you can implement
this amplification operator in different
ways in imitative amplification
and yeah we we can go back to our
example of cinder pichai having five ais
helping him do stuff i don't know if
it's a good example but he's like kind
of amplified by ai's
um
but because i i think i think one
important thing is that
um in in the case of amplification
um
you can get more intelligence from 10
agents
than from from
let's say one and then
um but 10 agents will be
uh less able to take over the world
because there would be like um
you you you you could kind of control
them
um
right so um it's easy to see like like
each individual ones are aligned
but like each m is aligned
but then um
the
[Music]
the sum of them are smaller than just
one m
is that basically the equation inclusion
um
yeah that's complicated like why do we
think implication actually produces
better models i mean so i think that
like
you know at least an imitative
implication we have this argument about
hch where it can be like in the limit it
converges to an infinite tree of humans
consulting humans
and then there's you know arguments we
can make we're like you know it seems
like this is a reasonable sort of
idealized reflection process
uh and so you know it seems like a good
thing to trust
um
oh is it
oh is it like humans consulting hch
the thing you're saying hch is a
recursive acronym which stands for
humans consulting hch okay
okay
right so yeah and then this is like i
can
um infinite loop and then you like you
have different ways of
uh doing amplification by approval by
different training directory training
and something
and let me see if i have other ones
which were interesting
i think i think one of the the funniest
one is
um the one about stem
so you basically
tell the ai stop um stop talking about
stop thinking about humans just thinking
about science this is like a very bad
scenario this is as crowman but this is
what i got from just like reading the
the first paragraph i guess
is a proposal where we're like well it
seems kind of dangerous to train models
in environments where they have access
to a bunch of information about humans
maybe we can just get away with training
models in environments where they only
have access to information on math or
something
or you know science or technology or
whatever and then we just use that and
then the problem with this obviously is
that well you can't use it for a lot of
things like it doesn't you can't use it
for like
you know geopolitics or running a
company or like you know anything that
would involve human modeling but you
still use it for a lot of things and so
maybe maybe
it's enough
um though maybe not
yeah i think if we if we have a very
good
like a very good ai that
maybe it's from robin hansen but um it's
like you you have this
um
advanced like um super accelerated um
human civilization in your computer like
brain emulation and it runs for like
billions of years or a thousand years
and at the end it produced some output
of like all the research
he has found over the years
um so if we have some kind of oracle ai
he's very good at science you would get
like all those insights about signs
without having the
problems of it trying to
um
find our values or something but then it
would we would still need to have some
kind of boxing problem for it to not
escape and
uh
you know make the
earth a component from something but
yeah i think it's a good objective
to be good at science
um
[Music]
let me i think yeah i think the other
ones are about debate or implication
and uh there's also one i think i think
one thing that is interesting me is
uh we were modeling so i think like the
um deepmind as
at least used to have
this different approach so chai uh
center of humanitable ai
at berkeley who do uh inverse
learning
trying to
find the word function
uh whereas um
deepmind aic team was mostly trying to
do reward modeling and
i don't fully understand the difference
um
and and uh and how it in in your blog
post
um you you give some solutions with
where modeling so if you could explain
that to me that would be like helpful
on a personal level
yeah so here's here's how i think about
recursive world modeling
so
in
imaginative applications we have this
application operator amp m where we just
have a human consulting the model and
this is how we produce a more powerful
version of the model
in recursive reward modeling we have
this sort of a new
version of the amplification operator
where now what the amplification
operator does is it does the following
thing
it
trains a reward model on the sort of
human's uh
life feedback
it then trains an agent to optimize that
reward model
and then it gives the human the
opportunity to look at what that agent
does and give additional feedback to
refine the agent and then once this sort
of converges the human has given a lot
of feedback we've tried an agent which
is like
uh we found a reward model and we found
an agent which is you know in fact
trying to do this sort of optimize for
this reward that the human is is trying
to give feedback for
then we call the resulting agent
the sort of amplified version of the
original model
and so we have this new
amplification operator which does this
whole reward modeling process
um and then we just do
sort of standard iterated application on
top of that new amplification operator
but what's the difference maybe it's a
layman question but what's the
difference between
trying to infer like um deeper rail from
human preferences like human giving
saying yes or no like trying to to tell
the ai like collaborative irl
we're trying to have a human say uh what
is the correct behavior and we are
modeling like here you basically have a
human in the loop saying what he wants
to do and there we have model which is
kind of our function is that we run
model
function
uh well reward model is not it's not our
board function because we learn it
so our word model is a learned model
right so he's like a um model well
when you're trying to do irl you also
have like a model of the word function
so you have parameters that define your
reward right
um
it seems to me very similar but maybe
i'm missing something
uh well it's it's it's similar to irl
but but it's not quite because we
we also train an agent to then optimize
for that regression and then we also
refine the reward
the reward function by
letting the human look at what that
agent does
right okay cool
and
yeah i think i think i think that the
last
paper that i think was kind of
interesting in terms of ai alignment was
uh where i'm also having trouble
understanding is um
learning to summarize from human
feedback
where there is this kind of human
feedback where
i think the the ai does summaries
and
um
the human says what summaries are good
or not
and so there's there's a mixture of kind
of arrow
and nlp
and at the end there's like human
feedback in the loop
um i don't know if you if you can give a
good explanation on that otherwise i can
i can read it on my own
um but i think there's like a similar
diagram to it
let me find it yeah we're doing
something similar i think it's a little
bit simpler they're just saying
um
we learn like um well you know it's
actually it's actually very similar
because they're learning a
um
[Music]
i don't know if it actually goes through
the step of having a reward model though
i think what it does is it just learns
an agent
and then the human gets to look at the
agent's actions and then gives some
preferences
and then you refine the agent based on
the humans um preferences after looking
at the agent's behavior so it's sort of
it's sort of similar but skips the um
the step where you train a separate
reward model at least if i'm remembering
the paper correctly
yeah i'm just i'm just trying to find
the
i think i i have the thing but i'm not
sure
uh
[Music]
one sec
[Music]
save as
right i think
they're they're giving me uh an svg
image which is a bit hard okay but yeah
then let's not go into this paper if we
if we haven't looked both at it um
anyway do you have um okay let me let me
is this is my closing this is my closing
question uh what is the most
underappreciated sub problem of a
alignment you would want people to work
more on
uh
so
this is a little bit of a weird question
because it depends i think very heavily
on who i'm giving the advice to
so like i you know there's the problems
that i work on um which obviously i'm
working on them because i think that the
most important things to work on
um which are things like myopia
uh and how do we sort of understand what
it would be what's my opinion
uh sort of how yeah how do we understand
what it looks like for an agent to sort
of only care about
a sort of single action and not optimize
for anything else
um
i think this is exciting uh but it's not
necessarily something i would want like
a uh i don't know i think it's like
complicated and it's like i don't know i
think if you wanted to work on this or
something the right thing to do is just
like talk to me a bunch whereas like
the i would say the more general advice
if i want to just like if you're trying
to get into asap and i mean you have
some machine learning experience and
just want to do something
i think that the like my my current
advice is like try to do transparency
interpretability research
um sort of like on this in the style of
like circuit style
stuff
um so yeah you're you're referring to
like your post or an oracle christian
exposed on circuits
right yeah
no or um chris all our conservatives yes
i'm referring to crystal those were
concerned
cool um and yeah maybe maybe i think it
is hard for people to actually give
precise answers that
but do you
do you have
um are your timelines aligned
with kind of um agi codepress report
um on yeah i don't know if you've read
this
um
i have a very high degree of uncertainty
on ai timelines but yeah
it's hard to talk about it publicly um
but i would no it's not that it's hard
to be talked about it publicly i have a
high degree of uncertainty i do not know
uh what the correct ai timelines are and
in fact i think that it's very very
difficult in practice to estimate good
ai timelines and so i think ajay has
done an admirable job and if i had to
like pick a number like
as a modal guess probably i would pick a
js number but like i don't
actually put very much stake in any
particular analysis of like how long
things are going to take and i think
it's very very difficult to predict
these sorts of things you can say that
she did a very good job
and it was very rewards
it was before something like dolly came
so i think most of most people have
talked to in the ml space
kind of
updated a lot on dolly as or clip at
least as like multi-modal and being able
to like understand concepts as
doing an evocative share or something
and when i look at
our stuff on a luther ai discord
i'm kind of amazed at like how good it
is at interesting kind of concepts like
even if you have like very conservative
timelines and like being very uncertain
i think i think
have you updated on dolly or not that's
my question
that's my real last question i don't
think it's that big of an update i guess
i feel like
i don't know
like
starting from like gpt2 and you know and
and even from like bert i don't know
we've seen we've seen really impressive
feats from language models going going
far back now i think at this point i
think like
i guess i feel like you shouldn't have
been that surprised that like it also
works to like say things in multimodal
settings like you just feel like that's
the obvious thing that's gonna it's
gonna happen next i guess like i didn't
feel like clip or
uh or dolly was like extremely
surprising i guess yeah what would be
something that would surprise you
what would be something that would
surprise me
um i don't know i mean lots of things
would surprise me i guess
um
in like hindsight what are things that
were like surprising to me
um well like i said i definitely think
that the success of like transformer
based language models was surprising um
i definitely
think that
um
i think i think that like alphago was
somewhat surprising
um i don't know
big stuff okay cool
transformers yeah transformers were
surprising and according to conor
there's like this hypothesis that
transformers is all you need uh it
didn't say that but that's like a meme
of like if transformers is all you need
for agi then maybe we did the most
important part but and then like
plugging a rail into it is the easiest
easy part
um
the very strong version of attention is
all you need attention is really all you
ever mean
yeah it is all you you will ever need um
so if it is right then we don't need you
you will never be surprised and you will
just like just transform is enough
anyhow i won't i won't take you more of
your time it's 2 am at my at my place so
it's very good to have you