Brian Christian - The alignment problem

hey everyone jeremy here and welcome
back to the towards data science podcast
now today's guest is someone i've been
looking forward to having on the podcast
for quite a while now
his name is brian christian and he's the
author of a number of best-selling books
which probe the connection between
humanity
and computer science and ai in different
ways now his most recent book is called
the alignment problem and it explores
the technical and philosophical
questions that come up
when humans try to outsource their
thinking to machines
brian's perspective is really unique and
he does a great job of tying together
long-term questions about ai's role in
the future of humanity
to more immediate issues like ai bias
and ai ethics
now this was one of my favorite episodes
to record so far and i highly recommend
picking up his book if you're looking
for a great read on the topic so
with that said i'll step out of the way
and let you enjoy the chat
all right well brian thanks so much for
joining me for the podcast
thanks for having me i'm really happy to
have you uh your book
is the alignment problem it's a
fascinating exploration of
well the alignment problem but it
includes as well an exploration of the
history of
the alignment problem and and i think
that's like one aspect
that we've not really explored a lot on
the podcast we've talked
to a lot of alignment researchers we've
talked to people about you know what the
current
situation looks like but i think the
historical context really helps us zoom
out a little bit
and it ties into a lot of philosophical
questions too so i figured that might be
like a cool place to kick things off
with um do you mind kind of providing a
bit of an overview of what you see as
the history of the alignment
yeah i think that's a great place to
start um and certainly it was
uh revelatory to me to do some of the
archival research that went into the
book and
discover some of these really surprising
things
so for example i was reading there's a
great book of oral histories about some
of the founders of
neural networks called talking nets
which was edited by james anderson who
was actually my
teacher at brown when i was an undergrad
i took his neural networks course
and one of the interviews is with jerome
letfin who was at mit and he's kind of
an early cyberneticist from the 50s 60s
um and he was walter pitt's best friend
so from mcculloch and pitts which many
people know
you know just as a bibliographic
citation
but he starts to talk about these
stories from the early 40s and he says
oh yeah
well you know when warren mcculloch and
walter pitts met
walter was just a homeless teenager
living on the streets of chicago
and warren basically became his foster
father
and um in fact invited both
walter pitts and uh jerome levin to move
into his basement and to become
kind of like these uh you know foster
foster sons and it was in that context
um that this collaboration relationship
developed
and led us to the paper mcculloch and
pitts
uh 1943 which is really the beginning of
neural networks
and so for me that was just a kind of
shocking reminder
that behind the uh you know citations
are these deeply human stories and some
of them are
very touching um so i really think
you know in some ways the story begins
there i mean that's where the book
begins is with walter pitts
at age 15 running away from home um
and fast forward to let's say the 1950s
you've got
the cybernetics movement norbert weiner
at mit
um and the beginning of a set of
conferences called the macy conferences
and i think they're very significant
because they brought together
not just a lot of the leading people in
the cybernetics movement from the
technical side so you had
the usual suspects norbert wiener warren
mcculloch walter pitts etc
um you also had philosophers you had
psychologists you
had anthropologists so
gregory bateson and margaret mead who
were considered you know two of the
leading anthropologists of the time
um margaret mead reported that she found
the meeting so interesting that she
didn't realize until
the end of the day that she had chipped
her tooth
um she was so engrossed um and
i think back to the macy conferences
because i think it's
important for us now in the 21st century
to be aware of the interdisciplinary
spirit
in which this field was really born
because it seems to me that
somewhere along the way we lost a little
bit of that and we're
we're now coming back to it you know
we're now in
what feels to me like this profoundly
interdisciplinary moment of
bringing in social science and policy
and
ethics um and the law and so forth
cognitive science etc
um but it was in in some ways that was
the spirit in which this all began
so to kind of bring this question around
i think
the beginning of what i would call the
alignment problem
from a historical standpoint is really
in this 1960 essay by norbert weiner
called some moral and technical
consequences of automation
um and he uses this metaphor of the
sorcerer's apprentice which
um you know many people will know as the
lovable mickey mouse cartoon from the
1940s
um but it that goes back to a gerta poem
in the 18th century but it's the same
exact story it's you know the
sorcerer's apprentice enchants this
broom tells it to fill up a cauldron
but he doesn't really specify you know
when to stop and almost ends up drowning
himself etc
and weiner says you know stories like
this
are not merely the realm of fantasy
you know this this is coming for us um
and the the famous quote is if we use to
achieve our purposes
a mechanical agency with whose operation
we cannot interface
efficiently interfere once we have set
it going
then we had better be quite sure that
the purpose we put into the machine
is the thing that we truly desire and
not merely some colorful imitation of it
and so i think that really nails it in
1960.
yeah it's such a scary thought that
capabilities are now getting to the
point where as you say that that's
starting to become relevant
it almost seems like a bit of an uphill
battle to
make that case to to convince people
especially who will argue you know we've
been through several ai winters we've
seen the technology ebb and flow
and we've seen enthusiasm in the
technology and the promise of the
technology ebb and flow is people say
agi is around the corner we're about to
do it folks like strap
yourselves in and so on um and this kind
of creates this at least
what i've seen and talking to some
people it seems to create a bit of
friction when you try to
argue for more attention to ai safety
and ai alignment
um i wonder first off if this is
something that
you're seeing yet again like now as as
we're moving into this new era like is
this a powerful voice a
powerful bit of pushback or do you think
that there's something
different that's going on this time
around because it does at least to me
seem like we're in a bit of a different
universe but i'd love to get your your
thoughts on it
yeah i think you're absolutely right
that i mean
anyone even slightly familiar with the
history of ai thinks of it in these like
extreme roller coaster cycles of uh
you know boom and bust and
there's this question of you know is is
this time different you know i mean i
think jfk had organized some white house
meetings on ai back in the late 60s you
know
um there was there was a line by the
university of washington uh law
professor ryan kayla
at one of the um future of life
institute conferences
where he said as a as a law scholar
how do i know that this time is
different because of the magnitude of
the policy response
um and i that that line has stuck with
me and i think that's
that's a significant um way of thinking
about it i think from my perspective
what feels different to me is a
combination of two things
one is that there really is this
question of whether we're on the path to
agi or not
um i mean we'll know in a few years
if we can just extrapolate what we have
now
there are certainly people who think so
right so openai
being maybe the most well-known examples
of people who just say
as far as we can see just throwing more
compute
at the models that we currently have and
scaling them up
we haven't found the asymptote yet um
so we'll find out i mean i think um
there was uh going by the current
doubling rate i think
the biggest neural network models are
doubling something like every three to
four months
and i've heard 10x a year uh
that sounds i think i think we're
talking the same figures
plus or minus yeah and we are um
i'm trying to think gpt3 was
uh point one percent as
many parameters as the human brain has
synapses um
and so we're you know three orders of
magnitude off
but we're 10xing every year so do the
math
you know we're gonna have something of
at least a kind of
you know facially uh comparable
complexity to a human brain within the
next three years which is
not a very long time so and i think
google switch transformer too is a
further 10x on top of openai as well so
just
more you know more to your point yeah it
does seem like it it seems to hold
yeah so i think that's that's the one
thing you know is that we've been in
this incredible
um shockwave really from
you know if you want to try to put a pin
in it i would say
alexnet you know october of 2012
um really launched this
um you know we saw image categorization
error rates dropped by 90
in seven years which you know i don't
need to tell i think your audience
a lot of that but there's a second trend
here which i think is equally
significant and maybe less
um you know less studied less celebrated
which is the penetration of machine
learning
systems some of them really complex
tons of parameters some of them quite
simple
into the institutional decision-making
fabric of society
um so one of the stories that i tell in
the book going back to your interest in
history
is the rise of statistical risk
assessment in criminal justice
which begins really in the late 1920s
in chicago but doesn't take off
until the rise of kind of big data pcs
et cetera and like 80s 90s 2000s
and now in many states the use of risk
assessment
systems to do things like pre-trial
detention
is mandated in state law
and so i mean that's just one example
but we are seeing you know machine
learning systems
of varying levels of complexity they're
part of how we do medical diagnosis
they're part of how we do
hiring they're obviously behind social
media they're behind the criminal
justice system
um of course self-driving cars are on
their way et cetera et cetera
there is kind of a we've crossed the
point of no return
in some sense where you know the analogy
i use in the book is it's
it's as though we're slowly putting the
world on autopilot
these systems that are kind of trained
by data are replacing not only human
judgment but also
explicitly programmed software um
of the more kind of familiar variety and
that really starts to flirt
too with a lot of sort of um themes
about self-determination
and free will even especially when when
you start to look at the criminal
justice system like
to the degree that we're trying to set
up oracle ais that can predict whether
or not a criminal is going to commit an
act we're stripping them of to some
degree their right to prove the
algorithm wrong
and chart their own course um is that
sort of
i mean i'm assuming there's a lot of
pushback on that basis as well is that
one of the big through lines here as we
start to see this ramping up in the
legal system
yeah i mean i think very broadly you
know the
the stakes are very high in criminal
justice but it's hardly the only example
i mean there is this real danger that
these systems
create feedback loops where
their predictions alter the future
training data that they see right this
is the
classic non-iid thing that is familiar
to people in the field
but these sorts of feedback loops really
do happen i mean and
all the way from you know detention
decisions right if you detain someone
you if you incarcerate them then you
lose the ability to know if they would
have been rehabilitated had you released
them
right so you can create these feedback
loops that way um
you can create feedback loops uh where
let's say there's a you're a you know a
lender
and there's a certain you know
demographic group in the population
you have less data about and because you
have less data about them
you're less accurate at predicting who's
going to repay their loan
and so you make these inaccurate loans
and as that data
comes back you start to think that
people in that group
are less creditworthy but that's merely
a reflection of your bad model
right but you start to see this kind of
you know self-fulfilling hypothesis i
think about it also
in um in terms of
is things as mundane as like auto
complete
you know in imessage or sms or whatsapp
or whatever
um you have this model of how
people speak and that model
determines what predictions you make but
then people start using those
predictions
and so suddenly you it's hard to tell
whether your model is getting better or
people are just getting more complacent
at taking its suggestions this is
actually a real problem for
machine translation where google
translate
you know initially before they kind of
got into the more
neural network based models initially
they were just
collecting huge corpora of different
human translations of texts and they
would say okay we can kind of
run these things through the meat
grinder so to speak and
you know notice that this phrase tends
to be translated in this way
but as google translate started gaining
popularity
the web became filled with its own
output
and so they had to do a lot of work to
make sure that it wasn't just
essentially like
you know eating its own output so
i think this kind of suggests that there
are many many cases like this
and just to make one more that i think
you know drives the point home
if you think about self-driving cars you
know there was the uber that killed the
pedestrian
in tempe arizona in 2018. and i read the
national transportation safety board
review of that accident it's very very
interesting
there are many kind of interlocking
things that went wrong but
one of them was a there was a training
data issue
where the training data appears not to
have included jaywalkers
so the model just only expects to ever
encounter people
at intersections and so
when this person is crossing the road in
the middle of the road it's just not
prepared for that
secondly it like many uh
image classification systems has a
pretty brittle
ontology you know the classic uh there
are n
categories every image is in exactly one
of those n categories it's never in more
than one it's never not
in any of them and one of the categories
was pedestrian one of them was cyclist
and this particular woman was walking a
bicycle
across the street and so the
classification system
kept flickering back and forth it's like
okay she's i could see her walking no
she there's definitely a bicycle frame i
can see the tire
no she's clearly on her feet walking um
and
each time it would change its mind it
would recompute her probable
trajectory from scratch and so it kept
essentially forgetting where it thought
she was going to be next
um so there are many
lessons to be drawn from this but i
think to your question about feedback
loops
um we create these models that have
some simplification of reality there's
there's they're making certain
assumptions that don't always hold
right it's the classic george box quote
all models are wrong
but in this case the
the model so to speak is so powerful
that it can
enforce the limits of its own
understanding
in this case by literally killing people
that
don't fit into its pre-programmed
ontology
um that you must belong into one of
these two categories or else all bets
are off
and so i think that to me is
what keeps me up at night you know this
idea that we set these things into
motion that makes certain assumptions
but then
essentially terraform reality to match
the assumptions of the model
rather than leaving the model open to
change yeah and often in as you say in
mysterious ways i i often think in
in in this respect to i think an example
that um
stuart russell talked about with social
media
say twitter right you know you hop on
twitter nominally the goal the algorithm
is to show you things that
are interesting to you but of course it
has an incentive also to change you
to polarize you to make you more
predictable so it's easier to show you
things that with high probability you'll
click on
and then there's this sort of deeper
question as well which is like
well presumably i do want myself to
change
from my interaction with twitter so
there's some amount of change that's
desirable
in what direction and what are the
incentives that play into that
change that's sort of a deeper question
it speaks to your point about the
interdisciplinary nature of this field
like all of a sudden you're
now you're bringing in ethics you're
bringing moral philosophy um
is this is this an area i mean i guess
one question that always comes to my
mind is like
is this an area that is tractable by the
current
mix of people who are working on this
problem or is it just a much bigger
problem
than than we can currently tackle with
just technical people say it's a really
good question
i certainly don't think it's um
a purely technical matter i think
there these questions really exist at
the boundary between disciplines
um between computer science social
science
ethics law and so
you really need actual human teams that
can speak across those divides um
and that's i mean there's something very
exciting
about seeing that movement come together
um and you know i certainly try to try
to draw those parallels myself where i
can and
you know the book traces a lot of these
interdisciplinary connections that i
think are really
relevant um but it makes it a harder
problem to solve because yes
people are bringing different expertise
to the table
different they're speaking different
languages in a sense um
and yeah whether we are
up to the task broadly speaking
is i think an open question um you know
i'm
i've been encouraged by what i've seen
even in the last
five years um
if you think back i mean the founding of
open ai was like
december of 2015 if i'm not mistaken
yeah it's easy to forget it's so recent
yeah and so much as you know the center
for human compatible ai at berkeley
where i'm affiliated
started around that time or excuse me
around 2016
um you have the fat ml conference
really taking off um in 2016 the
deepmind safety team was hired around
that same time
um a lot has happened
in five years and i think um
there's evidence that i see that we are
kind of rising to meet the moment you
know if you go to nurips
um in 2016 you know i there's there's
one researcher i interviewed who said
you know
he went to nurips in 2016 and he said he
was a safety researcher
and people kind of looked at him askance
you know but by 2017 first of all
no one batted an eye when you said i
work on safety there was an entire nerfs
workshop on ai safety
um and so i think there's there's a
culture shift happening
um certainly um
i notice it as you know
um a researcher and public intellectual
i get
um invited by healthcare companies
credit card companies et cetera et
cetera that are like
we're trying to get get our heads
wrapped around
some of these questions and we're trying
to build our own teams and put some of
this
expertise together and um you know get a
get a grip on the literature and figure
out what we need to do
i think a lot of that scrambling is
taking place which is a good thing
at the same time these are huge problems
um
and i would go so far as to say i think
the biggest problems that are affecting
society
not just polarization through social
media
but also climate change also the rise of
inequality
those are to my mind alignment problems
that you know you can think of
capitalism itself
has these you know operationalized kpis
these objective functions whether they
are
quarterly earnings or g uh gdp per
capita or whatever it is
and these things broadly speaking
correlate with human
flourishing until they don't and i think
that's really where we're at
and so this i think this is the
challenge in front of the human race
right now
so this is really now getting to the
meat of what i was especially excited to
talk to you about i think there are a
couple of different
routes that we're going to end up taking
here the first one
i want to highlight something that's
interesting about the way you've tended
to talk about
the alignment problem and that is a lot
of people refer to
the alignment problem in the context of
existential risk from ai
it's sort of almost implicitly part of
the baggage the connotation of the term
alignment usually that's what they mean
ai safety sometimes is a little bit
broader
what i find interesting about your use
of the term is you it seems to sort of
trace this through line
from contemporary issues that even
include things like ai bias and ai
fairness
all the way through to the more
existential considerations
and i'd love to have you explore that
spectrum and sort of what you see is
what ties those two ends together yeah i
think that's a great point so
really the genesis of this book for me
came about
circa the summer of 2016.
um we started to see on the one hand
you know concrete problems in ai safety
uh the paper came out
um open ai was kind of starting to put a
safety team together
you also had you know the propublica
article about compass and you know
apparent racial disparities and um
you know detention uh recommendations
and it felt to me that
these were connected in a very deep way
that was at the time i think not fully
appreciated i think
it's a little bit more understood now
um and it was interesting talking to
people in the research community and
saying it feels to me like this is all
one big thing
um that was polarizing when i first
started the research
and by the time i finished the book the
majority of people
agreed um and you started to see more
collaborations
you know where you'd have a paper with
one person who's nominally an ai
fairness researcher and their co-author
is nominally a technical ai safety
researcher but
you know it's it's they're able to find
actual research projects across those
lines
um i think
so i made several decisions in terms of
how to actually structure the book
um one thing was i wanted to frame the
book in the context
of actual present day problems
rather than thought experiments um
in some ways you know my uh
controversial claim is i think we're
kind of ready to retire the paperclip
maximizer thought experiment
um because for better or worse we we
have enough
real world examples that we can just use
real
real cases of machine learning going
wrong um we don't have to
you know think them up and i think
i wanted to be able to take
readers on a journey where
maybe they have some skepticism about
this x-risk
you know far far off you know
hypothetical stuff
um but if i just kind of gently lead
them through
stories of actual things that are going
wrong and we kind of
gradually moved from supervised learning
to reinforcement learning
he started thinking about robotics etc
um
you can just kind of put an ellipsis on
the end of that sentence and
the person will fill in the blanks so
that was a deliberate kind of
pedagogical strategy that i had
i wanted to kind of gradually bring
people
to their own conclusions rather than
come out of the blocks
guns blazing saying like we're all gonna
die and
if you don't think so you're wrong and
here's why you're wrong i
i thought it was a more sort of a
osmosis
approach was going to be more successful
i also think
um you know at a technical level
there it's beginning to be understood
but i
have had this conviction for a long time
that you know that bias and fairness
community and the technical ai safety
community
really are working on the same things
from a technical perspective
you know people in uh
people in the sort of ai bias community
are asking these questions about
well you know if you scrape an image
recognition data set
from the newspaper then you know the
person who's the
most represented in your data set is
going to be like the u.s president
um yeah and so do we really want to
train you know
joe biden detector as opposed to a face
detector you know
um that is the same thing that
you know concrete problems in ai safety
was flagging as you know the problem of
distribution shift you train an
agent in one set of environments then
you deploy it in a different set of
environments and it no longer knows what
to do
um so i think those those connections
are very real and you know you can think
about uh one of the biggest things
in the fairness literature is this idea
of you know proxy metrics you know you
you want to build a model that predicts
crime but you can't measure crime
uh you know the police are only even
aware of a small fraction of the crimes
that commit
that are committed um that sometimes
they arrest the wrong person
sometimes they convict the wrong person
and that's all you can
measure and so you're really building an
arrest predictor not
a crime predictor and those are sadly in
present-day america very different
things but we sort of
elide that difference when we talk about
them and that's very dangerous
um and so you know this is the same
thing that norbert weiner was talking
about in the
60s of you know we had better put
into the objective function the thing
that we really care about and not merely
you know the proxy variable because
the correlation between those things is
going to break down so
i think a lot of the concerns of the
people worried about existential risk
really already exist and
that's good and bad it's bad in the
sense that we're sort of already
you know the the slow rolling disaster
is already
happening on the other hand um
you know it gives us the ability to
get our act together hopefully while
it's still
you know manageable um and take some of
the things that we've learned
uh going forward hopefully right so
that's the question
it's it's kind of a race condition
between our deployment of this
technology and our ability to understand
how to make it safe well it really does
seem like what's what's going on to your
point is
we're we're now i think one of the key
things that's changed is
as the technology's gotten exponentially
better year-over-year
the kinds of errors these systems are
making are much more
clearly intelligent kinds of errors so
i think one of the salient examples that
comes to mind for me is openai
published their paper on um i think it
was called pathological
reward functions in the wild i think
that was it um essentially
the the motor boats so they you know
they train this motivate boat to play
like this race game and it ends up just
like
looping around collecting a bunch of
bonus points or something in some
some section of the the map that has
nothing to do with the broader goal
and like this is clearly a goal towards
which a lot of intelligence has been
deployed like
this is a it is a concrete goal this is
clearly a way to rack up more points
and so on and it really illustrates that
like it's not always
stupid mistakes that these systems make
sometimes they're intelligent sometimes
they're intelligent enough
not to be obvious or detectable to human
beings so as you
keep writing out that ellipsis and the
sort of the rest of
the the sentence is pregnant with
possibility we only have to extrapolate
some of those curves and we start to see
those
sorts of errors take on a much more
existential form and
shape um is that sort of like it's out
of those kinds of mistakes that you're
primarily concerned uh
sort of existential risk might emerge
yeah i that's a good question i think
when i
when i think about existential risk
um i'm personally
much more
kind of consciously focused on the sorts
of locking locking in bad
civilizational trajectories rather than
actual extermination
um there's an essay by paul cristiano
called what failure looks like
which i think is a great read and it i
share a lot of that sentiment
um part of part of what he sketches out
or maybe i'm interpolating between
his vision and my vision but it's a sort
of kafka-esque
world um i think kafka is an author
that's
under-appreciated by the ai safety
community um
this idea that people had you know
in the early 20th century of a
bureaucracy that had gone out of control
um you know this world of paper pushers
and rules and regulations and no one
really feels like they have any agency
they're just cogs in this big machine
i think that's a pretty
high probability bad future
the way that we're going um that what
machine learning does
is you create the kafka-esque
bureaucracy but you
take out all of the um
actual people with the ability to stand
up and
make exceptions to rules or walk out on
the job if they feel like
it's not uh you know aligned with their
values or whatever
um and i think there's there's an
important reminder
that when we think about the alignment
problem from a technical perspective
there is often this kind of thought
experiment you see it in a lot of papers
that's like
a human called h wants to do something
so they build a robot called
r is r gonna do what h wants
and i think that's a that's useful for
getting some of the technical results
that we want
but it is not actually a useful way of
thinking about
agi in the real world because that is
like the
hobbyist relationship you know like
someone builds an agi in their garage
and it does something that's not the
relationship that we have
to our technology right most software we
have is just like
licensed we don't even own our own i
mean i don't know
legally do you even own a smartphone
anymore i
it's like all being done through this
kind of like
um extremely complicated
end user agreement the open ai api right
is obviously
kind of gated by they can shut off
whoever they want
and it's a reminder too that
when agi comes like the h
r relationship is already predicated on
a future in which agi
is like democratized and commoditized
like to even get to the point where some
random person could just be like oh i'm
gonna make an agi
today over the weekend and it'll help me
paint the house or something
um that suggests that we've sort of
already somehow gotten out of the future
of what i think is a more probable
scenario where there's some kind of
huge corporate duopoly um
you know that there's going to be the
whatever the
verizon versus the atnt or the apple
versus android or whatever it's going to
be visa versus mastercard
you know what there's going to be like
two agis and
um they're gonna have this huge moat
around them
um so at that point we don't
really need to be focused on the h r
relationship
per se so much as the relationship
between what h wants
and the business model of the company
that's
licensing the use of r on a completely
like
you know can be canceled at any time
basis
so this is a long way of saying the term
alignment
was borrowed by stuart russell
and the computer science community more
broadly from
the economics literature um
you know if you go back and read
economics papers into the 90s and 80s
they were talking about like how do you
create a
value-aligned organization how do you
align the incentives of your
subordinates with you know their manager
and that's a reminder i think a sobering
reminder that
alignment was always a human problem
first um and it'll be a human problem at
the end you know even if we can solve
the technical aspect
such that you can build a system aligned
with the values of the people that build
it
you're left with you know it's like it's
turtles all the way down you still have
to align between the people who build it
and the end users and the third parties
that are still affected
you know even if they're not actually
using it um etc so
that's what awaits us after we uh solve
the technical problem
so i think that's that's a really
fascinating and important part of this
discussion that
do you i would argue i mean due to the
sort of um
cultural biases of the field the deeply
technical people were the first
to worry about this that um as you say
h2r relationship
became really uh heavily emphasized
because
it's the one that you can probe with our
technical tools it's technically
tractable
and so it gets all the attention um i
guess as the uh
as the internet meme would have it you
know why not both is also an option i
suppose where you could say
um my understanding from the the more
technical side is
part of the risk comes from the fact
that potentially
uh the the very training so it's not
necessarily at deployment at the
deployment stage but even the very
training process of an agi
brings with it existential risk in the
form of
decisions like this open ai um speed
racer
boat where basically this agi is being
trained
at some point it develops a sense of
agency um
what's referred to sometimes as embedded
agency when
you develop this self-awareness is maybe
part of it i
i'm sure i'm abusing terminology here in
many different ways but it reaches the
stage in some way
and then realizes oh like there's
instrumental value in
preventing humans from shutting me down
there's instrumental value in preventing
humans from
taking resources that i could use and
deploy in the direction of my own goals
and and that it's from that sort of
primordial conflict that emerges
this sort of competition and ultimately
existential risk when we're in
competition with something much
smarter than ourselves now this is
obviously this is like one i guess i'm
i'm describing here really the
the nick bostrom argument in super
intelligence
what are your thoughts on that argument
and do like do you see it as
a both end or do you see it as mostly
sort of this like
organizational flavor of the alignment
problem
so i think yeah one way of thinking
about it is kind of the hard
take off scenario which is what you've
kind of outlined and the
soft takeoff scenario which is i guess
closer to what i've outlined
um i'm you know as i've
already indicated i'm a soft takeoff guy
that's what i think
is happening i mean i think we're on
that gradient as we speak
things are just getting weirder um
i certainly think that the heart take
off i mean the stakes are so high
that the fact that you know like
50 ish people in the world are
thinking about the hard take-off i've
like met most of them personally like
that seems maybe too few
maybe too few people think about this i
think that's the weird thing is that you
sometimes see people in the press
arguing that these issues get too much
attention it may
may be the case that they get too much
press attention
but the idea that you know should
is 50 people working on preventing the
hard take off from
you know destroying this human story
uh is that too many are you
really arguing with a straight face
that's too many people thinking about
this
um so that's that's my feeling is i
don't personally sympathize with that
argument very deeply
but i sure
think that it's worth taking seriously
because of the stakes right it's sort of
you
it's the classic like utilitarian thing
of you multiply the
probability by the impact and you're
like okay well
it seems like unlikely but we should
definitely be
covering our our bots um
yeah from my perspective i'm
more expecting um
that there's going to be some kind of
blurry line between
[Music]
you know ai alignment or ml alignment
and the sorts of normal
incentive dysfunction that we already
have
you know um politicians are optimizing
for staying in office which often
involves very short-term oriented
thinking as well as you know paying more
attention to fundraising than actual
policy um you know governments are
worried about these macroeconomic things
and
you know even the difference between
mean income and median income ends up
being like a huge divergence in terms of
that's impact on actual people
there was a section of the book that
ended up getting cut
but i was very fascinated with the
development of the united nations
human development index um
and the story of um you know the
economist amartya sen
saying you know are you crazy you want
me to like just find
a number that represents like the human
development of a country like
that it can't be done like that's
ludicrous
and the the argument that changed his
mind was
if you don't pick something we're just
gonna use gdp
yeah um and so he's like okay fine let's
use gdp plus literacy plus infant
mortality plus you know like
um lifespan whatever ends up going into
it
but i think it's uh
it's important to be thinking about
i mean i don't i don't think there's a
huge distinction
between deep neural networks as
optimizers of an objective function
and you know corporate org charts as
optimizers of some objective function i
don't think there's a big difference and
i imagine you would agree with this but
i think the difference is only going to
get smaller
um so really it's an incentive design
issue and so
to me the alignment problem is
completely uh
completely of a piece with uh what's
already going wrong
yes well this is so to me this is one of
the most exciting and the most
terrifying things about it
is that the alignment problem viewed
through a certain lens
is the everything problem and we haven't
solved the everything problem yet i mean
you alluded to
one pretty deep challenge with this
which is identifying good metrics and
and the sort of like good hearts law
trap where
the moment you define a metric for
anything it ceases to be a good metric
of the thing you're trying to achieve
because people find ways to hack it or
or what have you and it kind of seems
like the mental model i have i'd be
curious
uh for your take on this see if you
agree but it's almost as if capabilities
you know supposedly we do what we seem
to be doing today which is identifying
either gdp or depending on the
administration like the stock market
maybe
as like this metric we're going to
really pin our our reputations on this
one metric this is the only thing that
matters
judge me based on this if it goes up you
know i'm doing a good job
and it's almost as if we so as our
capability to execute against that
metric goes up
eventually it starts to become resolved
um apart from sort of the more general
notion of human well-being or the stuff
we
really care about and this really
becomes target fixation or myopia
and the more we keep optimizing the more
dystopic the world starts to
become ai seems to be just a further
tool helping us to optimize
even harder against a very narrow and
specific objective
like would you agree with that
interpretation or i would
completely agree with that
interpretation um you know and this
bringing it back to norbert weiner sort
of the godfather of the alignment
problem
you know he has a great line which is to
me sort of the stuff of nightmares where
he says
the you know thus far in human history
the only thing
shielding us from our incompetence has
been our
impotence um that we've always
picked bad metrics but it hasn't
mattered because we haven't had the
optimization
uh capability to get all the way to
maxing out these metrics
um and you know it's like we're sort of
taking the safety
part uh of that away um you know it for
people with an ml background right
there's
this regularization technique called
early stopping where you just don't
optimize as hard
um and we've had a kind of uh
helpful regularization throughout human
history that we just didn't have the
ability to
control reality to the degree that we
could you know hit
these targets perfectly and so we
haven't needed to pick the targets
perfectly
um what makes me most hopeful
um so if i you know force myself to put
the more
optimistic hat on part of the
movement that we're seeing in the
technical side of ai safety is
how do we get away from specific metrics
altogether these kind of manually
specified metrics
underneath the hood reinforcement
learning still needs some
objective function and you know maybe we
need a deeper paradigm shift that's an
open question
but um rather than specifying that
objective function manually
um let's learn it right and so you think
about
inverse reinforcement learning um
you know my my favorite example of this
but there are many is the
collaboration between paul christiano at
openai and john leichette
mind um to make the mujoko
virtual robot that could backflip right
it's like
no one knows how to write a good
objective function that
refers to you know specifies a backflip
um
it's hard to demonstrate it you know so
there's a lot of
you know you could do rl through
demonstration behavior cloning but it's
hard to demonstrate a backflip
so we'll just do this thing where we
will show people video clips of the
robot like wriggling around at random
and we'll say which of these is you know
epsilon closer to what you want to see
and we'll use that preference to make
some inferences about what
we think you think a backflip is
operationally then we'll optimize
against that and show you two more
comparisons and we'll sort of um iterate
as we go
and it works you know after
about 900 comparisons which takes about
an hour it's fairly tedious but on the
other hand
900 bits of input is not a lot
yeah uh it can do these beautiful
gymnastic backflips and stick the
landing
and that to me makes me
pretty hopeful that you know this is the
toy example obviously and i have no
illusions about what it might mean to
scale that up
but we have a process for somehow
extracting the the ineffable gist
of this aesthetic thing that we have in
our head in
actually getting that into the objective
function um
part of what i've been really interested
in
uh just in the last 12 months let's say
has been
the unexpected interest that
tech companies have in inverse
reinforcement learning
um you know for example there was just a
collaboration
between some of the my colleagues at
berkeley and some people at
uh twitter looking at
okay can we use something like irl
to back out an operational definition of
what makes a good notification
or what makes a good timeline um rather
than having you know an engineering
meeting where we
you know get together and decide you
know we're going to privilege
likes over retweets or whatever whatever
um
you know at some level you could just do
what the backflip paper did
and show people two timelines and say
which of these do you
want more which which is better and then
behind the scenes you can do all these
crazy things to try to
develop some kind of kpi or some kind of
objective function that actually
captures that
that's the kind of thing that i think is
very interesting because
20th century capitalism etc has alway
been about these explicit kpis the kind
of the tyranny of these
measurable you know uh indices
maybe the 21st century is going to be
about
you know these implicit metrics
that are actually derived from kind of
human preference judgments
i'm not saying that's a silver bullet
but maybe we're starting to unwind like
the tyranny of the
of the indices a little bit yeah and
that's exciting
i i i totally agree so there's so much
to to dive into it in this direction but
the the uh the notion of yeah these
implicit implicit
um not reward function but just like
implicit rewards
it sort of reminds me of the the debate
over in the 70s you know what is
pornography and and you know the classic
responses like i don't know but i know
it when i see it
i can't define it but i know it when i
see it um one of the risks i imagine
with especially when we get into things
like twitter
like this idea of training machines to
do as we do not as we say which is
essentially what we're talking about
here we're talking about
you know look at how i respond to this
and then derive from that implicitly
what i
would have told you to do if i could
articulate what i wanted
one of the the risks i imagine emerging
there
is we're not always at our best selves
in terms of our revealed preferences
like our actual
preferences that we act on um for
example i spend much more time on
twitter than
the long-term version of me wants and
that's almost its own alignment problem
i mean these things become so entangled
right like
yeah there are many many me's and and
it's not clear which one
i should defer to or which one should
have the greater right to determine
my future course of actions yeah there's
essentially an alignment problem between
system one and system two yeah um
and in a funny way
uh a lot of what we see from yeah
recommender systems
social media is the turbo charging
of the more reflexive system of you know
okay i see a thing i click on it
that gets reinforced um behind the
scenes
you know at youtube or whatever it might
be
and there is a weird way in which
it is harder for us to
impart our longer term reflective
preferences
into those systems maybe with my
most optimistic hat on maybe this is
going to be resolvable in the next five
to ten years through
huge language models where there will
just be like a text field and you can
say to youtube like
i'm tired of watching olympic archery i
would really like to learn more about
woodworking please
or whatever don't show me formula one
clips until it's 8 p.m because i have to
work right now whatever you know as
open-ended as whatever you could type
into the thing
somehow that is turned into some kind of
preference vector that then modifies
your
thing maybe that's a way to get
you know system two a seat at the table
you still have the question of
whether it aligns with their profit
incentive
um so there might be a tension there
where uh you know if you're
if your system one wants to eat cookies
and your system too wants
to lose weight um there's
an open question of what makes more
money for that company
you know yep selling you cookies or
selling you
workout routines and so there's
there may it may not be as simple as
let's get system to a seat at the table
but i think that's a a starting point
um so i'm very curious to see if that
starts to happen
it's also i guess there's the the
converse risk which is
you know system one or our limbic system
or whatever you want to call it the
short-term
version of ourselves the one that we're
most often ashamed of because it tends
to get the better of us
at least me um that particular so
there's like a tyranny of system one
there's a tyranny of the short-term self
and that's when you basically go on a
cocaine binge and you're just like
hooked up to a bunch of joy machines and
that's it
uh it seems like there's also a tyranny
of system two where
uh it's it feels almost like the
equivalent of you know
when i so when i started my undergrad i
made a decision to
research and study physics now that was
effectively me
locking in a commitment for the next
five years of my life and
while i was okay with that and i remain
okay with that
you just lengthen that commitment and
say hey i'm now committing to the next i
don't know with life extension the next
200 years of my life
to doing this and all of a sudden it
seems tyrannical but in reverse
it seems like there's almost a a middle
ground a dance between
the kind of limbic and cortical system
system one system two
whatever the the nomenclature is that
that we want to respect
and it's like its own challenge to
figure out what that should look like in
its own philosophical debate
so i guess irl to me seems like an
important part of this because without
it we
as you say we're locked into this
tyranny of like a kpi which sounds even
more terrifying to me than anything else
yeah but um do you see that dance or or
do you think that there's a way to kind
of transcend that um
yeah no i mean i think it's a very it's
a very deep question and
you know i've been talking with some of
my
colleagues um you know on the on the
research side
about how do you
um how do you do irl on an agent which
is essentially comprised of these two
battling sub agents oh cool right so
you know if their preferences come on
different time scales
so for example i mean there's some
technical things here that are fun to
dig into so for example if you have a
situation where the user
is faced with some preferential a versus
b choice and they take a long time to
make the
choice you could say okay well they must
have been pretty similar
because it took them a while to figure
out which one they liked better
so we'll kind of down down wait that
choice because they preferred one but
probably only by
epsilon or you could say
this choice because they thought so hard
about it
reflects their like deep reflective
preferences
you know their uh whatever reflective
equilibrium
and so it's maybe more important than
the decisions they made quickly
um there are lots of really fun
you know philosophical slash
computational questions here about
methodology that i think are
significant yeah and the fact that
they're fun to to bounce around is like
at least to me is it's part of the cause
for concern because like
they're fun because they're ambiguous
right they lead to this open-ended
conversation it's like oh man like
what's the right answer here how
exciting
and it's um i think sam harris referred
to this like the need to do philosophy
on a deadline
like it you know we're headed to a world
where we're going to need answers to
these questions and it's it's really
cool that inverse
rl is part of this conversation
i wanted to explore one other sort of
analog for the alignment problem
um that i think i think you've spoken
about before and i had uh evan hubenger
on uh
actually it'll be the episode probably
right before this one i think
um but anyway we talked about evolution
and
i think it ties into what you were
talking about an interesting way with
this
idea of early stoppage as a
regularization technique so
you train an ai like not to the pinnacle
of its performance but part way
stop it before it can myopically focus
too much on its objective rather than
some general
thing that it's trying to achieve and it
sort of seems like that might
this is a half big thought but it sort
of seems like that might be part of what
makes
the alignment problem so hard what what
makes humans
so um ambiguous in terms of our moral
priorities
we are by some through some lenses an
example of early stoppage like we are
not a
fully evolved forum we're like kind of
this jumble of stuff the universe threw
particles at itself
a whole bunch and here we are like is
that is that a reasonable thing i mean
again
that's a deep question so if you go back
to the first paper about
inverse reinforcement learning one of
the things that
um i think it's stuart russell and
andrew rang
maybe daishi harada was also in that
paper but one of the things they
they say in the conclusion of the paper
is
this is kind of a stop gap like
we're talking about what can you infer
by
the policy of a
like fully optimized agent at the end of
its learning process
and so you know in a funny way in
technical literature
people talk about inverse reinforcement
learning and inverse optimal control
there's more or less synonymously
there's kind of more of like
what tribe are you in um
but i think if you actually
scrutinize the language like inverse
optimal control
captures the idea that you are observing
the behavior of an
optimal agent
versus inverse reinforcement learning is
at least in its name the idea that you
are observing an agent that is
learning in the process of optimizing
their policy
in an environment and yeah as i say the
very first paper on irl
flagged this as an open problem and an
issue that
needs further work and that was like uh
28 years ago so
wow it's still an open question i mean
people are thinking about it
but this this idea of um you know what
can you infer from an agent who is
learning
um and they're sort of on a trajectory
uh through this error landscape they
haven't gotten anywhere yet
and to your point about evolution um
absolutely the case that humans are on
uh some evolutionary trajectory them you
know ourselves
and you know many people in the rl
community
i'm thinking of andy bartow satinder
singh
and many others um have explicitly
thought about
evolution as this reward
designer for humans um
that there it are the things that make
humans
uh you know contribute to humans
evolutionary fitness
like surviving and having kids and
etcetera etcetera
being robust to famine it's all these
all the things that happen you know make
an organism fit in an
environment but there's a
reward design problem where those aren't
the things in your head the things in
your head
are like i want to impress my colleague
at this meeting i want
some chocolate i want a new car i want
to have sex with that person
um and somehow that jumble of
more kind of myopic or
more tangible you know short to medium
term goals
um ideally makes you into the kind of
species the kind of organisms that
you know thrive in an environment but
it's it's
it's hardly a direct relationship it's
in fact a very convoluted relationship
between those things
and so yeah you can kind of think of um
this weird double-decker rl problem
where evolution needs to find the right
incentives
such that when we optimize for those
things we do you know we we secretly end
up doing what evolution wants
even though we think we're doing what we
want yeah it it
actually that's um that's an interesting
way to explore it too because it sort of
leads to
other questions around do like do
environments
implicitly like does an environment like
the universe
imply a reward function like
just by virtue of its very existence if
you
initiate it randomly and let let time
run to infinity
will this environment tend to optimize
for a consistent kind of structure or a
consistent kind of
information distribution it almost seems
like the universe is
contains that that like almost not
hard-coded but that implicit
promise of you know you will be rewarded
if you end up
optimizing for something like this and
i mean through this lens you might say
it's something like agi or like
on the assumption that agi ends up being
something like agi ends up being the
final form of of the universe everything
gets
optimized in service of whatever this
thing is
then i don't know i mean again this is
very hand-wavy but
well i mean this is an idea that is
sometimes known as the reward hypothesis
um and uh this the idea goes back to
uh well it has a funny circular origin
where michael littman
from brown claims that rich sutton
invented it and rich sutton claims that
michael lindman invented it so
that seems so appropriate yeah somehow
it emerged
and you know the basic idea is um
it's this provisional conjecture that
everything
that we mean when we talk about you know
what motivates the behavior of organisms
not just humans
can be thought of as the maximization of
some
scalar reward function
and in the words of rich sutton
uh it's a philosophical thing that's
almost certainly wrong but it's so
simple that you kind of have to
go with it until you can disprove it
later um
and so yeah i think this is a really
interesting question well it also
i think i think i've just connected in
my mind then why this feels so related
to the idea of early stoppage
and by the way i find it really um
helpful and amusing that like
every time i think i have a new idea
you're like oh yeah this is already a
thing that's been
it's kind of heartening to hear that um
yeah but but uh
so so i'd love to play test this idea
with you a little bit
um so we we have the universe that's
let's say
to use this this as you say probably
wrong philosophical conjecture
but the universe implies some reward
function
and clearly we've deviated from it i
mean this is what you know you might
refer to as a failure of inner alignment
from through that lens like the universe
nominally wants us to propagate our
genes or propagate some kind of
information structure through time and
yet we're here we are preoccupied with
sex and podcasts and uh using
contraceptives which makes absolutely no
sense from that through that lens right
um and yet weirdly all of our far
future projections seem to imply a
return
to respecting that more basic and
fundamental reward function of the
universe
like you know to the extent that agi
ends up being our final form this is a
transient
and to the extent that we deviate from
respecting that implicit reward function
that's a temporary thing like eventually
we kind of return to it
as we develop again conjecture on top of
conjecture here but just kind of
an idea i figured i'd i'd throw at you
to see what you think yeah i mean
this gets pretty heady pretty quickly i
mean there is this funny
a funny way in which humans
are you know we're walking examples
of what people like evan are worried
about in terms of inner optimizers
you know that we're we're supposed to do
what
the outer optimizer called evolution
wants but actually we can take
take control and say you know actually
i want to spend my life composing
symphonies
you know which you know just for the we
have this hearing system that was
developed for whatever
purposes and that's probably not what
it's developed for but
darn it i really like symphonies and i'm
going to make symphonies
i'm going to take a vow of celibacy and
you know
commit myself to advocating for ideas
rather than
my genetic material
there is an irony there which is that
the very kind of existentialist
taking control of your own destiny
is at some level you know what we're
worried about right because we're the
reward designers
of these machine systems and this is
this is the nightmare scenario
um yes and at the same time
i don't necessarily think that we owe
evolution
uh any favors per se um
so yes there is um
there is this very cosmic sense in which
um
you know some of the concerns of
existential philosophers about what is
the meaning of life we were
you know we're built to do one thing but
actually you can do whatever you want so
then you're faced with this
you know vertiginous uh question of what
to make of your life
there is a funny way in which that does
come back around to this alignment
question
yeah i guess it's yeah and i didn't mean
to sort of load it normatively and say
you know we should therefore
build an agi very much uh but but it
raises that that question of you know
where do we place the value and we do
have to
be somewhat arbitrary and not this is
anything new but in
in ascribing value to like humans and
for some reason we care about this so
let's you know set up a field of
alignment that you know
celebrates the primacy of human life for
example it's uh sort of interesting
um thank you for the the conversation it
really this has been
more of an exploration i'll say than
than any other conversation i've had so
far on the podcast in terms of the
breadth of stuff we've covered and
it's so clear that you've spent so much
time thinking about this i really
appreciate you sharing your insights
it's been my pleasure yeah thank you