Rohin Shah - Effective altruism, AI safety, and learning human preferences from the world's state

hey everyone welcome to another episode
of the taurus data science podcast my
name is jeremy and apart from
hosting the podcast i'm also on the team
at the sharpest minds data science
mentorship program
and i'm really excited about today's
episode because i've been thinking about
getting today's guest on the podcast for
a very long time i'm so glad we finally
made it happen
um he's actually in between a transition
right now from uh berkeley
where he's wrapping up his phd he's
working at the center for a human
compatible ai
and he's transitioning over into deep
mind where he'll be doing some alignment
work his name is rohin shah and
apart from being a very prolific a
researcher in uh in ai and ai alignment
in particular
he is also the publisher of the ai
alignment newsletter which is a really
really great resource if you're looking
to get oriented in the space
to learn about some of the open problems
and open questions in ai alignment i
really recommend checking it out
uh we're going to talk about a whole
bunch of different things including the
philosophy of ai the philosophy of
machine learning and ai alignment
ways in which it can be accomplished
some of the challenges that exist
and we're going to explore one of the
most interesting proposals i think that
rohan's come up with
which is an idea about extracting human
preferences
from the state of the environment so
basically the idea here is that
humans through their activity have
encoded their preferences implicitly in
their environments
we do a whole bunch of different things
different actions that reveal our
preferences
and it would be great if we could have
ai's look at the world
and figure out what our preferences are
implicitly based on the state of that
world
and that might be a great way to
bootstrap ai alignment efforts and so
we'll be talking about that proposal in
depth along with a whole bunch of other
things
i'm really looking forward to uh to the
episode so i hope you enjoy it
so without further ado here we go rogan
thanks so much for uh joining us for the
podcast
yeah thanks for having me here i'm
excited well i'm really excited to have
you there are so many interesting things
that you've been working on
in the alignment space in general but
before we tackle some of the more
technical questions
there's an observation that i think
anybody who's spent any time working on
alignment or talking to alignment
researchers
is going to end up making at some point
which is that the vast majority of
people in this space
seem to come from the effective altruism
community
and i'd love to get your take on number
one you know what the effective altruism
community is what what effective
altruism is
and number two why do you think there's
this deep connection between
uh ea effective altruism and ai
alignment and ai safety research
yeah sure um so you know
there's like this the overarching idea
of effective altruism the like very
you know easy to defend one that doesn't
get into specifics is just
you know with whatever
money time resources whatever you're
willing to spend altruistically
you know you should you should try to do
the most good you can with it
rather than and you should even think
about that
uh and it's pretty hard to like you know
argue against this like i don't think
i've like really seen people
uh disagree with that part now in
practice effective altruism the movement
has like a whole bunch of additional
premises that are
you know meant to be in support of the
skill but
are you know more controversial i think
that like a
really you know fundamental big idea of
effective altruism is that of cause
prioritization well many people will
like say okay i want to have
uh say clean water in africa i will work
towards that and they'll like
think about different ways in which you
could like get clean water too
uh in africa uh maybe you could like try
uh sanitizing the water that's already
that people already get or you could try
building some like new
uh treatment plants in order to provide
fresh flowing uh water that's like
drinkable to everyone
and they'll like you know think about
you know how best can they achieve their
goal of
delivering clean water but it's much
much
much less common for people to think
okay well should i work on
you know getting clean water uh to
people in africa
or like combating racism in the us which
of these should i put my effort into
or my money into it the main premise of
effective altruism is like
you can in fact do this there are
actually significant differences between
causes
and by thinking about it have much more
impact by selecting the right cause to
work on in the first place
um and so it's very focused on you know
the sort of like
take weird i take ideas seriously
actually
evaluate them figure out whether or not
they are true and not whether or not
they sound crazy
um you know whether or not they sound
crazy does have some relation to whether
or not they are true but they are not
necessarily the same
right um i think that ties into you know
why so many
why it's also such a hotbed for like ai
safety research
the ea case for ai safety for work on
the eye safety
is that ai has a good chance of being
extremely influential
in the next century let's say
there is like some argument that is
debatable but it doesn't seem like you
can rule it out
um it seems like at least moderately
likely that you know if we don't take
care of
how we do this uh we might just
like the ai system uh might
take over in the sense of like all of
the important decisions about the world
are made by the ai system and not by
humans
um and like one possible consequence of
this
is that uh humans go extinct you know
i'll i'll go into this argument later
i'm sure uh
but yes believe this argument somewhat
and so then it becomes extremely
important impactful to work on
it sounds crazy but like one of the ace
to me strengths is that it separates
what sounds crazy
um from what is true it seems like
really the focus is
there's this extra missing step that a
lot of people don't apply to their
thinking
when deciding what causes to contribute
to what to work on what to spend their
lives on
and that is the step of going really
like what what areas are
going to give outsized returns on my
time and exactly
yeah i mean i can i can really uh think
back to most the conversations i've ever
had with people
about causes about charity and you know
it's usually focused on stuff like what
are the administrative fees
associated with this charity oh i want
to donate to a place where all my
dollars go to the cause
rather than asking the more fundamental
question is this cause actually
going to give the best roi from the
standpoint of you know benefiting
everyone or benefiting humanity
um so it's sort of interesting that that
kind of thinking a more kind of first
principles approach
leads a lot of people to the area of ai
alignment and ai safety
as you said i mean it makes sense you've
got this super high risk high reward
profile
what was it that drew you for example to
ai alignment ai safety work in
particular rather than
any of the other i mean i could imagine
bio terrorism i could imagine
all kinds of horrible things that could
happen to us but why ai alignment in
particular
yeah um so
my story is kind of weird it's maybe a
classic ai story in that you know
convinced by
very weird arguments um but i
you know got into effective altruism
in 2014. i heard the arguments for ai
risk you know within a year of that
probably
um i was deeply unconvinced by them
i like i just did not buy them
um and so for you know for until 2017 i
basically just didn't really engage very
much with ai safety
i was also unconvinced of
basically there's this field of ethics
called population ethics
which tries to
deal with the question of how do you
compare how good different worlds are
when they have different populations of
people in them
and i we don't need to go into the
details but it's a very confusing area
lots of you know impossibility results
that say like you know you might want
these like six very intuitive properties
but no you can't actually have all of
them at the same time
stuff like this so you're would this
would the idea here be like you know is
a world with
a hundred decently happy people better
than a world with a thousand
kind of you know decent minus epsilon
happy people is that yes kind of
calculation that's an example
of a question that it would deal with
yeah so anyway i was like
thinking about this this
question of a lot in the summer of 2017
and eventually you know i was like okay
i think i actually
should put a fair amount of weight like
not certainty
certainly but like a fair amount of
weight on the view that
you know more happy people is like
in fact just means a better world even
if they're in the future
and once you
put a decent probability on that it
starts looking
you know overwhelmingly important uh to
ensure that the future you know
continues to exist and have happy people
because it's just so
big relative to the president
um and so then i wanted to do something
that was you know more future oriented
and
i had you know a ton of skills
in computer science and math like
basically everything you would want
to work in an ai alignment i still was
not very convinced of ai risk was like
okay you know a bunch of smart people
have like thought about this
maybe i should like work on it for a
while see see whether or not it makes
sense
and so then i um and that that's what
caused me to actually switch
and you know a year later i like started
believing the arguments that that's
why i also saw different arguments so
you were you were led by the quality of
the people who were drawn to the problem
more so than necessarily the the initial
arguments themselves like do you
remember an aha moment as you were
working on the stuff where you're like
well wait a minute like this is actually
for real i can now see why you know nick
bostrom and maybe elias ryukowski and
whoever else was talking about it back
then was uh was on on to something
i never really had an aha moment i think
i i remember at one point i was like huh
i guess i now believe these arguments
but it wasn't like i
or like i guess i now believe that ai
risk is you know
substantial and real but i couldn't
can't point to you to
can't point to a like specific point in
time where i was like
yes now i believe it i just sort of one
day
was reflecting on it and noticed oh yeah
i used to not believe this and now i do
that's so interesting yeah it seems like
there's a bifurcation between people who
maybe you know they read super
intelligence or they read less wrong and
they get really excited about the
problem and really scared of it
right off the bat because for whatever
reason they're wired in such a ways
to have that happen and then people yeah
who like you it's like a slow burn and
you ease into it
i guess this is part of the problem
almost of articulating the the problem
right i mean if it takes that
long to get people to uh to think of
this as a really important thing
do you have um strategies that you use
when you try to explain to people like
why why is you know ai risk so serious
why is the probability non-trivial um
that you think might have worked on you
back then to kind of
accelerate the process yeah i should
note that like i still
you know i'm like not super happy with
the arguments and like
super intelligence for example um
and like i would say that it's like you
know slightly different arguments that
are motivating it for me
with you know still a fair amount of
emphasis on things that were in super
intelligence
um i think a lot of people won't have
heard of super intelligence by the way
so if you
want to address any any of the arguments
you raised please feel free to kind of
give that background too
yeah um maybe i'll just talk about you
know the arguments i personally like
yeah since i can you know explain them
better
but just for context super intelligence
is a book by
you know a professor at oxford named
nick bostrom
um it was published in 2014 and it was
the first like
you know book length treatment of like
why
ai risk is a thing that might occur and
what we why we should think it might be
plausible
what solutions we might uh seem seem
like they should be investigated and
stuff like that
um i think for me personally
um the argument i would give
so a we're going to zoom as a premise
that we build ai systems that are
intelligent like as intelligent as a
human let's say
um and i you know we can talk about that
later but that's a whole
other discussion i'll just say that you
know i think it is not i think it is
like a reasonably likely
um to happen in the next century but
for now take it as an assumption one
thing about
intelligence is like it means that you
can adapt to new situations like you're
presented with a new situation
you like learn about it and you do
something and like that something is
like
coherent it makes sense so one example i
give of this of
where we see this even with you know
current neural nets is
um a specific example from gpt3
uh so i i believe viewers will be
familiar with gpd3 but if
uh listeners not viewers but if not
um gpd3 is this
language generation model that opening i
developed and released recently
and i think i i like one particular
example uh which comes from
uh the post giving gpd3 a turing test
um where you know the the context to
gpt3 was a bunch of like questions and
answers
and then gpt3 was posed a question
um how many bunks are in a court
these are nonsense words you did not
mishear me um
and gpd3 you know like in some sense
this is outside of its training
distribution it is
never seen this sentence in its training
training uh corpus presumably it may not
have even seen the words you know
bonk and coit ever um
so it's you know actual pro actual
distribution shift
and you know you're relying on some sort
of generalization out of distribution
nonetheless i think we can all predict
that gpt-3 is not going to
output some random string of characters
it's going to probably say something
sensible
um and in fact the thing that it says is
you know there are three bunks in a coit
i have no idea but you know it's it's
like sensible in some sense it like
produced an answer that like
sounds like english and we've all been
there to some degree
if we write exams or whatever we're
asked how many bunks are in a coit we
haven't done our studying and
hey there are three bunks in a coit
there we go exactly right
um so like in some sense you know gpt3
did generalize
it like generalized the way a student
taking a test would
um you know in the original post this
was taken as evidence of like qpd3 not
actually being
reasonable uh because it doesn't doesn't
know how to say you know this question
is nonsense
uh but you know then a follow-up post
was like actually you totally can get
gpt3 to do that if you like tell
uh gpd3 that you know if in the context
you say whenever it sees a nonsense
question the ai responds you'll be real
then when it's asked how many bonks are
in a court it says
yo be real so you know it's got the
ability to tell that this is nonsense
it just you know turned out that it
generalized in a way where it like
was more like a test taker and less like
you know somebody in conversation did we
know that ahead of time no we did not
um i think this is like we had to
actually
you know run gpt3 in order to to figure
this out
i think ai risk is basically like this
but like you know supercharged
where your ai system
if it is human level intelligent it's
like
definitely going to be deployed in new
areas
in new situations that you have that we
haven't seen before
um and we just don't
really have a compelling reason to
believe that it will continue to do the
thing we were training it to do
um as opposed to something else like you
know in gpd3
what were we training it to do well on
the training data set at least
we were training it to do whatever a
human would write in that context
but like when you see there are three
bunks and a sorry how many bunks are in
a coit
what would a human do in that
circumstance yes
i don't know it's not really well
defined and you know gpd3 did
something sensible but like it's not i i
don't think you could reasonably say
that it was
you know doing what we trained it to do
it just did
something that was coherent and
similarly if you've got you know
ai systems that are human level
intelligent or more
uh taking like super impactful actions
upon the world
and they're put in these new situations
where like there's not
really a fact of the matter about how
they will generalize then they might you
know take actions
that have a high impact on the world
that aren't what we want
and then you know as a maybe intuition
pump for why this could be really really
bad
like human extinction level bad you know
one particular
kind of distribution shift is you know
you go from
the training setting where you know
humans have more power than the ai and
can turn off the ai
to like the setting where the ai is
sufficiently intelligent and
sufficiently widely deployed that no
human can like
and or humanity as a whole cannot like
turn it off
in that situation you know that's a new
situation yeah he's never been in a
situation where
where it had this sort of power before
will it use it
in some way that will will it generalize
in some different
some way that was different from what we
expected during training
we don't really have a reason to to say
no it won't do that
is there an analogy you think here with
uh with child rearing i mean i'm just
thinking of here
intergenerational human propagation
where
you know our ancestors in the 1600s at
least in the west
i'm sure would be absolutely disgusted
by our vile ways today
uh the way that we deal with sex the way
we communicate to our elders the way
that we
manage our institutions and so on all
our hierarchies are just completely
different
and in many ways orthogonal to the moral
frameworks that were applied you know in
the middle ages um or in the early
renaissance
i guess there is a difference here in
the sense that at least we are still
running on the same fundamental hardware
or something very similar
maybe that ensures a minimum level of
alignment but does this analogy
break apart in some way i think that's a
pretty good intuition pump
um like there are some ways that the
analogy breaks like for example
um you know well the analogy doesn't
break so much as like i would like you
know
say put a little bit less weight on it
for these reasons like one is
you know in childhood you you have some
influence over children
but you don't get to like you know do a
full training process where you give
them gradients for every
like single time step of action that
they ever do
so you might hope that like you know
within a given that you can have
way way way more selection pressure on
ai systems you would
be able to avoid this problem but but
yes i think that that is
you know the same fundamental dynamic
that i'm pointing at you know you get to
do some amount of you have some amount
of influence over
you know these agents but you know those
agents then like
encounter new situations and they do
something in those situations and you
didn't think about those situations
ahead of time and you didn't train them
to
to do the right thing then i definitely
buy um
the the idea here that you know this ai
risk this is really significant risk
and the stakes are very high um when it
comes to the solutions or the strategies
that you think are most promising
you yourself are specialized obviously
in well in one category everybody has to
be in one sub
subspace within the alignment uh problem
domain um
what is the the area that you've decided
to focus on and and why do you think
that
that sort of is most deserving of
attention at this point the story that
i've told you so far
is one of generalization
like the the main the main issue is
you know we don't know how to generalize
and you know
plausibly you could get ai systems that
are like single-mindedly pursuing power
and that's sort of similar to the super
intelligence story and those could cause
human extinction
but like the sort of fundamental
mechanism is like bad generalization
or generalization that's like you you
your capabilities generalize you do
something coherent and high impact
but you're the objective that you're or
the thing you're trying to do doesn't
generalize
relative to like what humans wanted and
so like a lot of the things i'm most
excited about
are somehow generalization
related so like one thing that i'm
interested in
is like can we like get
a better understanding of empirically
how do neural nets tend to generalize
can we say anything about this
there's a lot of theory that tries to
explain why neural nets have like as
good generalization power as they do
um you know it can't be explained by
statistical learning theory because
neural nets can memorize random noise
but nonetheless seem to generalize
reasonably well
on you know when it when the labels are
not random noise
and sorry do you mind explaining
statistical learning theory as a
reference i'm actually not so sure that
i
i can to the uh the connection um
so statistical learning theory is like a
branch of machine learning theory
that you know tries to do
several things but among other things
try to prove that if we train
a machine learning model on
you know such and such training data
with such and such properties
then we know that it will generalize in
such and such way
um and it like proves theorems about
this
and sort of importantly most approaches
right now
focus on making assumptions about
your model your hypothesis class these
assumptions usually like
preclude the ability to overfit to an
arbitrary
arbitrary size data set because if you
could then you like can't really say
anything about generalization
but the fact of the matter is neural
nets really can just
actually overfit to any data set they
can memorize labels that are literal
random noise um
and so these assumptions just don't
apply to neural nets
um so i the thing i'm excited about
is like can we talk about like
assumptions on the data set
rather than just the model and using
you know if we think about assumptions
on the dataset and assumptions on the
model can we
then say something about how neural nets
tend to generalize
this is like a super vague not fleshed
out
hope that i have not really started
working on nor to my knowledge as anyone
else
you know there's just so many empirical
things about neural nets that are so
deeply confusing to me like deep double
descent
i don't get it it's an empirical
phenomenon if you don't know if you can
look it up it's probably not
that worth me going into just so
confusing i don't know why it happens
it makes no sense to me and i will like
want to know why
and like i think that if we like
understood things like this
we could start we we might be able to
start making statements about how neural
nets tend to generalize and like
maybe that translates into things we can
say about safety
that's interesting the generalization
story seems to be
um seems to be like one ingredient of
the problem of course and then
like the the other ingredient which i
mean there's some overlap but it does
seem like they have distinct
um components is this
challenge of like telling machines what
what human preferences even
are like you know our ability to to tell
each other what we want out of life is
already so limited
and it's something that i mean at least
i personally
find somewhat somewhat jarring as a as a
prospect and having to actually
not only uh express our preference but
quantify them and etch them into some
kind of loss function that we then feed
to uh
to a model um you've done a lot of
interesting work on this and actually
there's one of your papers i wanted to
talk about
we discussed this before we started
recording and i was so glad to hear that
it was also the one that you thought was
the most interesting
so we have compatible views at least on
that but it was this idea of um
well the paper's titled is preferences
implicit in the state of the world
and i guess first i wanted to ask a
question to set the scene a little bit
so what is preference learning what is
that that concept
yeah so this is actually the next thing
i was going to say i was excited by
oh great which is you know
you know i've talked about
generalization but like before you
get to generalization you want to train
on the right thing in the first place
uh that seems like a good you know
starting point for your a
system if you don't have that you're
probably host um
and so you know lots of ink has been
spelled on like how it's actually very
difficult
to specify what you want um
you know by writing a program or an
equation that just captures it in a
number
which is you know how deep reinforcement
learning or any any deep learning system
works
um but it's you know most commonly
associated with deep reinforcement
learning
and so the idea of preference learning
is you know rather than having to
specify what you want by writing down an
equation
you specify it by some like more and
easier to
some easier method like for example you
could like
look at two trajectories um in for
in a reinforcement learning setting you
can look at two behaviors that the agent
took
and you can say ah yes the left one that
one was better
and that's you know giving the agent
some feedback about what it should be
doing
it's not you know trying to write down
an equation that captures the ideal
behavior in every possible scenario it's
just saying you know
out of these two which one is better and
so you you would imagine that that's you
know easier for humans to do
and more likely to be correct um and so
this this preference learning field i
sort of think of it as like
the field of how do we design mechanisms
for humans to give feedback to
an ai system such that you know humans
such that we can like actually give
feedback that incentivizes the behaviors
we actually want
and you know we don't make uh as many
mistakes
in specification as we do with reward
functions so what i
what i find really exciting about that
aspect of it too is like
there's this well-known difference in
humans between
um like expressed uh desires and
revealed desires or expressed intent and
revealed intent like
you know i'll say i want to i want to
work out for three hours today
i want to do a bunch of coding and i
want to like just have a vegan a bunch
of vegan meals
for the next month and then if you check
in on me next month
i will not have done all of those things
i would have done nearly all those
things and the question is like
well which me is me i mean am i am i the
aspirational self that said hey i would
love to be that person
or am i the the jackass who actually sat
on his couch and watched netflix the
entire time
um this seems to really scratch that
itch in the sense that it probes our
revealed preferences
for better for worse i guess that could
also be a failure mode um
is is that something that um that you
see as valuable in this approach
yeah um i think you want to use you know
both sources of information and not
not do either one um so actually
let me let me take a step back and
distinguish between two
different things you could be trying to
do um so there's like one thing where
you're trying to learn what humans value
which is the sort of thing that you're
talking about and there's another
framing where you're just like
i want my ai system to do such and such
task
and i want to train it to do that but
like i can't write down the reward
function for that task
um i'm actually more interested in the
latter honestly
but the former is also something i've
you know spent a lot of time on and i'm
excited by
um so you know right now we're talking
about the former um
can i ask a naive question i mean i
think i understand what the difference
is but
i just want to want to put it to you to
tackle it explicitly what is what is the
difference between those two things
yeah so like one thing is like you know
maybe i want my
ai system to like
cl to like vacuum my floors
um or something and like
you know the task of vacuuming my floors
is not well
specified it's not like well specified
just by that you know that sentence
uh anyone who has a roomba can probably
tell you stories of the room about being
super dumb um and like some of those are
just the room of not being intelligent
enough but some are also just like
the task is not super well specified
like you know
um should the agent you know vacuum
underneath a christmas tree where
there's like you know
um a bunch of needles that might ruin
their vacuum
who knows um if there's like some like a
random
shiny button on the floor should it be
vacuumed up or just left alone
because like maybe that button is
important what sorts of things
you know should the cat ever be vacuumed
like you know the cat has a lot of hair
that gets her up gets everywhere if you
vacuumed the cat that seems like
it would make your house cleaner there's
like lots of lots of
ambiguity here i wouldn't really say
that these are like
human values like teaching a roomba how
to vacuum does not
does not seem to be the same thing as
like you know teaching the roomba about
human values
uh for one thing like you just can't
really talk that much about revealed
preferences here because you know
you know i don't vacuum my my house
um very often um
i you know if an ai system we're going
to to vacuum
i probably would i might have it vacuum
more often but
you know would you say this is like a
it's like a narrow application
of human preferences like it almost
seems like the distinction between you
know narrow ai and agi
somehow maps onto this yeah and i think
like
i think i agree with that and i would
say like you know
but in this sense like everything is an
area you just like get narrow ai that
becomes more and more general
at some point we decide to stop calling
it narrow ai you start to call it agi
because of how
broad it has become but i i like the
idea
of like you know you start with
something that you know can be applied
to systems today
and like you just scale it up it just
becomes more and more capable more and
more general but it's
always the same technique and like you
know eventually
the systems that we create with it we
would label them as agi or human level
intelligent or super intelligent
but like you know it's the same
technique it's the same sort of
general principle so that that that's
sort of why i'm more excited by
by this framing of the problem rather
than the human values framing
but you know as you get to like more
general
systems it sort of merges in with the
human values like
you know once you get ai systems that
are like designing
government policies or something like
whatever feedback you're giving them it
better teach them about human values
yeah and hopefully i guess we start to
do that at higher and higher levels of
abstraction as you say as we sort of
climb that ladder
uh we fill in the convolutional filters
in a sense as we as we go up
yes exactly you had asked a question
about revealed preferences
versus um spoken preferences
or express preferences and i think like
yes this is an important distinction
um i am like you know i definitely
um have want any method that we propose
to like not be dependent on one or the
other but to instead be like
using both um and there will be
conflicts
i'm mostly hoping that you know we can
just sort of like
have ai systems that like just set aside
the conflicts and do things that are
like
um that are robustly good and like
they're robustly good according to
either side
um probably you know you'll have to have
some amount of conflict resolution
mechanism but like
you know humans already have to do this
in some sense it's like seems possible
that we could do it
um but but yeah i think it is you know
good a like very nice aspect of this is
that you don't have to commit commit
yourself to like
defining the behavior upfront in every
possible situation
we just don't know this like right our
values are not well defined enough
honestly for that to be right we like
our values are constantly in the process
of being updated as we
encounter new situations like you know
right now we
we talk about democracy one vote per
person you know if
someday in the you know transhumanist
future
um if someday it becomes possible to
just like you know make copies of people
i think we would pretty quickly uh no
longer want to have one vote per person
yeah because otherwise you can just you
know pay to
have anyone elected if you are
sufficiently rich
or i guess even in just in the limit of
better information about brain states we
could say well
you know sure this policy makes the
majority of people happier
but the people it makes more unhappy i
mean look at that that horrible
dopamine cycle like those people are
really taking a big hit and you kind of
like weight those responses yeah
yep yeah that is you could definitely
optimize better for social welfare
potentially
and maybe then you don't want to just
have one vote per person
right now um i guess this brings us back
to preferences and placing the state of
the world there are things
presumably about the structure of the
world that reveal
our i guess this is revealed preferences
mostly right what we've actually done
yep this is definitely a revealed
preferences method
um and i think like an important aspect
of this like
you know people will i think like
the one of the reasons i'm especially
excited about this which i want to
say as a prelude is that like it's not
trying to do the hard things like
when people think about value learning
they think about uh should a
self-driving car
like you know uh if it's like has a
choice between running into two
passengers or like killing the driver
what should it do
and you know those are hard ethical
questions
i'm like you know less interested in
them i want to you know start with
can we get an ai system that knows that
like reliably you know knows that it
should not kill humans
if there's you know if there's if there
are two options where you know
yeah anyway like the basic stuff that we
all agree on
um or nearly all agree on um
and so like this i think like looking at
the state of the world is a
good way to like um see this
and the basic intuition here is that you
know we've been acting in the world for
so long we have preferences we've been
like you're rearranging the world
um to to fit the way that you know we
want the world to be
and so as a result you can sort of like
invert that process to figure out what
things we probably want
and so there's this nice toy example
that illustrates this like
suppose you know there's a room and in
the middle of the room there is this
breakable base
and bases once they're broken they can
never be prepared
um we assume that the ai knows this and
we're going to assume that the ai
knows all like empirical facts it knows
how the world works
it knows you know what actions the human
can take it knows what actions itself
can take it knows like
what states of the world are possible
but it doesn't know anything about like
the reward function
or which is you know the equivalent of
human values
so it knows empirical facts um it knows
that this base
once broken cannot be fixed we're gonna
leave aside glue and
you know things like that yeah um
and you know it then looks at the fact
that you know it sees
it it like is deployed in this room and
it sees that you know
it's human who i'll call alice um
is like you know has a walk is in the
room and the vase is unbroken
and so then now you can like pose
hypothetical questions like
all right well what would i have
expected to see
if alice wanted to break the base
well i would have seen a broken base
yeah what would i have expected to see
if alice didn't care about the vase
well you know probably you know at some
point you know the most
efficient way would have been to just
walk it walk through the room while
knocking over the face
and so probably in that case also i
would have seen a broken base
you know what would i expect to see if
alice
did not want the brace to be or actively
wanted the vase to not be broken
um in that case i you know actually see
an unbroken base probably
and since i actually see an unbroken
base
that tells me that you know of those
three situations only the last one seems
consistent with my observations
and so probably alice did not want to
break the base and so you can infer
this fact about that you know alice
doesn't want to make break the base just
by looking at
the state of the world and seeing that
the base is not broken
it seems like there's a a very deep
connection here to just like the second
law of thermodynamics and just the
universe
has like there's so many more ways to
end up in a situation where you have a
broken base
that the mere fact that there isn't a
broken phase is a huge piece of
information
yes it's the uh yeah yeah exactly
that that's yeah that's exactly right um
it is that yeah like basically something
to
well it just strikes me like this like
the physicist instinct in me but
the um to the extent that the world
looks any different from
what we would expect with you know pure
thermodynamic randomness
the assumption here is those differences
come from human preferences right i mean
would that be a fair way to characterize
the uh
yep that's right and does that imply
certain failure modes then because i
guess
we encode information in our environment
i guess is back to the revealed
preferences thing but like
implicitly um you know i i've i've
hard coded my my my brain state into my
apartment every
arrangement of things any kind of like
any
misogyny any racism any foot fetishes
that i might have
to like the whole laundry list of weird
quirks that may or may not be part of my
personality are
implicitly encoded in in the room um
is there like is this part of the the
risk of applying a technique like this
yeah so i mean in theory if you
feel you know supercharge this method i
it would
right yeah it's just going to get
everything that is a revealed preference
and
there are well i don't know that gets
everything but like
to a first approximate approximation it
gets your real revealed preferences
i'm sure there are some that it does not
get um
and sometimes you just
don't like your revealed preferences and
you'd think they would they should be
different like you know
um you have a revealed preference
many people have a revealed preference
to procrastinate
that they probably do not in fact
endorse right
um and they wouldn't want you know their
ai system
uh giving them more and more addictive
materials so that they can procrastinate
better
uh which you know is plausibly something
that could happen um
i would have to think significantly
harder about
how how exactly concretely that could
happen but i i could believe that
that would be an effect similarly you
know
the technique as i've explained it so
far assumes that there is only one human
in the world
and things get a lot more complicated if
there are multiple humans and i have
just like you know
ignored that case so far i mean it's
it's what you need to get the thing off
the ground right i mean
yeah and yeah is there um
because like in this context i guess i
imagine at least there's another
sort of risk mode which is you know if
by pure
chance um like let's say in the example
with the vase
let's say that the uh the human actually
doesn't care about the base
but just happens in in her demonstration
to uh to have avoided the vase
is there the risk that i guess this is
always a risk in machine learning this
is
it sounds like just a case of added
distribution sampling um like that yep
you learn that okay yeah that's right so
if you
so there is you know if the vase is like
kept in an inconspicuous out of the way
location where it's like not actually
that likely that
um alice would have broken the base in
the course of moving around the room
we actually have this in the paper we
show that like in that sort of an
environment
you actually don't learn anything
significant about the base
you're just like eh maybe she wanted she
probably didn't want it broken
you right you infer that like she did
not you know deeply desire for the base
to be broken
but you don't infer anything stronger
than that um
you're like uncertain between whether
it's you know
uh negative versus whether it's like bad
to big bases brother versus just like
yeah it doesn't really matter um
it's more like you know if there were
efficiency gains to be had
by breaking the base uh right then you
infer
and you observe that actually the base
wasn't broken
uh then you infer that it's bad to break
faces
uh you know it's still possible that you
know humans you know we're not perfect
optimal people uh we might like not pick
up an efficiency gain
um yeah and so we like
uh might you know go around the base
even though it would be faster to go to
the base even if we didn't care about
the base and then
yeah this method would make an incorrect
inference
in general there's in preference
learning there's a big tension between
like
you know you're assuming that humans do
the things that they
that write the things that the humans do
reflect what they want
and like not always true
right and sometimes just for reasons of
like pure stupidity as well i guess like
we may want a thing just not know how to
make it happen
yep exactly that is a big challenge in
preference learning and
i you know people have tried to tackle
it
including me actually but i wouldn't say
that and there's been like
super substantial progress
on separating you know
stupidity from uh
things you actually wanted i think we'd
end up solving a lot of other problems
incidentally if we could do that
yeah um yeah actually there was there's
one more aspect i wanted to ask you
about with respect to the paper
the role of of time horizons or the role
that time horizons play in the paper is
i think
just really interesting because there's
certain assumptions that the
the robot makes or the ai makes about
like
what is the time horizon the human has
in mind for this action
that like if those assumptions about
that time horizon shift you start to see
different behavior i just love to hear
you expand on that and kind of describe
that setting a little bit
i think then the main takeaway
from that i think i would have about
time horizons is like
if you assume a short time horizon
then you know cases where the state
hasn't hasn't been fully optimized are
much more excusable
because you know the human just hasn't
had enough time
to fully optimize the state towards um
what would be optimal for them and so
you can maybe i should just
i should fill in that gap i realize a
little ambiguous but um by time horizon
i guess we're talking here about the
amount of time
that the human would have to say go from
one point in the room to
a desired end point right yeah it's it's
like the amount of time that the robot
assumes the human has been acting in the
environment before the robot showed up
in the room case it's like um you know
the robot is deployed
and sees an intact base and it's like ah
yes the human has been walking around
this room for an hour
right something like that if you've been
walking around the room for like a full
hour and the base is still there
you can then assume that the base is
probably pretty important to
yeah something along those lines exactly
the
the like actual setting in the paper is
slightly different but
it's um that's the right intuition
um and
yeah and so yeah this isn't
illustrated best with like the base
example but like imagine you're trying
to build a house of cards
this is another example where like the
state of the world is really very
informative about your preferences
house of cards are like super super not
entropic
yeah you can infer a lot
yeah yeah the more specific the
arrangement i guess the more
which is interesting because that's
exactly what's i guess so so
challenging about just preserving
humanity in general there's
there's like almost a a philosophically
conservative streak
to this approach in the sense that we're
assuming that
you know we've gotten to somewhere
that's worth preserving because we've
encoded
so much of ourselves so much of what's
good about us already in the environment
and it all almost seems like like what i
love about this this time horizon stuff
is the the political philosophy behind
it it almost gives you a dial
that you can tune from the kind of
progressive to the conservative end of
the spectrum just by assuming
right different time horizons if like if
you assume that we just got here and
it's sort of blank slate then hey
we can try anything we don't we're not
really certain about what humans want in
this environment
yeah conversely right yeah that's true
i've never actually thought of it that
way but but you're right that is
that is basically what it is like
another way of thinking about it is like
you know i the way i actually like got
to this point
to the point of writing this paper was
like asking myself you know
why is it that we privilege the do
nothing action like
you know we'll say that like the safe
action is to do nothing
why it's just an action and you know
this is this is an answer
this is like we've already optimized the
environment random actions are probably
you know so something like current state
is like you know high on our preference
ranking
random actions take us out of that state
into some random other state so you know
probably an expectation they go
lower in our ranking whereas the do
nothing action preserves it and so it's
good
and you know the longer the time horizon
the the stronger
the ex the stronger you want to like do
nothing as a default
yeah yeah i remember reading that in the
paper actually as almost a derivation of
that intuition which is so it's so
beautiful when you can see it kind of
laid out like that
it's like yeah yeah i mean in a way
it makes me think of like you know so
many arguments among and between people
of different political stripes
could be so much easier if we applied
toy models like this where you can say
well hey you know
there is value to the conservative
there's value to the progressive
you know we end up in a dystopia either
way and here's the parameter we can tune
to see how dystopic things get one way
or the other
depending on how much we value things
yeah i anyway i love the
the work and i thought it was anyway
again one of these sort of
aha moments for anybody who's interested
in the intersection between
kind of philosophy moral philosophy and
then an ai it's just
anyway just such a cool piece of work
yeah thanks
i i like it for basically the same
reasons
sweet well i'm glad glad we have uh
compatible aesthetics then
uh awesome well i i think um we're we're
we've covered a lot of the bases here
but was there anything else you wanted
to to talk about i mean
one thing i do want to make sure we get
to is a reference to the
alignment newsletter that you put out i
think everybody should check that out
especially if you're looking to get into
the space rohin puts out this amazing
newsletter
um and you can get it anyway we'll link
to it on
the blog post that'll come with a
podcast but yeah do you have any social
media links or things like that you want
to share
um i think the alignment newsletter is
like
the best way to get you know my
current thinking on things if you're
new to the space i probably you know
recommend
other things um so specific writing of
mine
i like that that i like as more
introductory
it's not exactly introductory but more
timeless material
on the alignment forum there is a
sequence called
a sequence of blog posts called the
value learning sequence that i read
i like that as a good introduction also
there's also like
um two there are two other recommended
sequences on that firm that i also
recommend
that i also think are pretty great um in
terms of social media
i have a twitter it's rohin m shah
but like you know mostly it just
sends out the alignment newsletter links
people can also feel free to email me
my email is on my website um
it can't guarantee that i will send you
a response because i do actually get a
lot of email
um but i i think i like have a
fairly high response right yeah well i
i can i can vouch for that at my end
thanks for for making the time really
appreciate it and i'm really looking
forward actually to uh
to putting this out and also um good
luck with uh deepmind because you're
heading over there in a couple of couple
of days really right
yeah i'm heading there on monday it's
it's
two more business days all right yep
enjoy the long weekend such as it is
cool thanks awesome thanks so much rohan