Rohin Shah on the State of AGI Safety Research in 2021

welcome to the future of life institute
podcast i'm lucas perry today's episode
is with rohan shaw he is a longtime
friend of this podcast and this is the
fourth time we've had him on
every time we talk to him he gives us
excellent overviews of the current
thinking in technical ai alignment
research
and in this episode he does just that
our interviews with rohin go all the way
back to december of 2018. they're super
informative and i highly recommend
checking them out if you'd like to do a
deeper dive into technical ai alignment
research you can find links to those in
the description of this episode rohin is
a research scientist on the technical
agi safety team at deepmind he completed
his phd at the center for human
compatible ai at uc berkeley where he
worked on building ai systems that can
learn to assist a human user even if
they don't initially understand what the
human user wants rohin is particularly
interested in big picture questions
about artificial intelligence
what techniques will we use to build
human level ai systems
how will their deployment affect the
world
and what can we do to make sure this
deployment goes better he writes up
summaries and thoughts about recent work
tackling these questions in the
alignment newsletter
which i highly recommend following if
you're interested in ai alignment
research
rohin is also involved in effective
altruism and out of concern for animal
welfare is almost vegan
and with that i'm happy to present this
interview with rohan shaw
[Music]
welcome back rohin this is your your
third time on the podcast i believe we
have this this series of podcasts that
we've been doing where you helped give
us
a
year in review
of
of ai alignment and everything that's
been up your someone i view is very core
and crucial to the
ai alignment community and i'm always
happy and excited to be
getting your your perspective on what's
changing and what's going on
um so to start off i just want to you
know
hit you with a simple not simple
question
of
what is ai alignment
oh boy excellent i i love that we're
starting there
um
yeah so different people will will tell
you different things for this as
i'm sure you know
the
framing i prefer to use
is
that um
there is a particular class of failures
that we might be
we can think about with ai
where
the ai is doing something that its
designers did not want it to do
um and it's like it's
uh
and specifically it's
competently achieving some sort of goal
or objective or
or some some sort of competent behavior
um that isn't the one that was intended
by the designers
uh this
so for example if you try to build an ai
system
that
is i don't know supposed to help you
schedule calendar events
and then it like also starts sending
emails on your behalf to people
um
which maybe you didn't want it to do
that would count as an alignment failure
um
whereas
if you know
a terrorist somehow makes an ai system
that that can that goes and detonates a
bomb
in some big city
that is not an alignment failure it is
obviously bad
um but it the ai system did what it was
what its designer intended for it to do
so it doesn't count as an alignment
failure on my definition of the problem
other people will see ai alignment as
synonymous with ai safety
um
for those people
uh that you know terrorists using a bomb
might count as an alignment
failure but at least when i'm using the
term i usually mean you know the ai
system
is either
uh is doing something that wasn't what
its uh designers intended for it to do
there's a little bit of a subtlety there
where
you can think of either intent alignment
where
you like try to figure out what the ai
system is
trying to do
and then if it is trying to do something
that isn't what the designers wanted
that's an intent alignment failure
or you can say all right screw all of
this you know notion of trying we don't
know what trying is how can we look at a
piece of code and say whether or not
it's trying to do something
and instead we can talk about impact
alignment
which is just like the actual behavior
that the ai system does
is that what the designers intended or
not
uh so if the ai makes a catastrophic
mistake
where the ai like you know thinks that
this is the big red button for
happiness and sunshine but actually it's
the big red button that launches nukes
uh that is a that is a failure on impact
alignment but isn't a failure on intent
alignment
uh assuming the ai like legitimately
believed that the button was
um
happy happiness and sunshine i think i
said
so it seems like you could have one or
more or less of these in
a system at the same time so which do
you which are you excited about which do
you think are more important than the
others
in terms of
what do we actually care about which is
how i usually interpret in important
uh the answer is just like pretty
clearly impact alignment like the thing
we care about is like did the ai system
do what we want or not
uh
i nevertheless tend to think in terms of
intent alignment
um because it seems like it is
decomposing the problem into
a
natural notion of like what the ai
system is trying to do and whether the
ai system is capable enough to do it
and
i think that is like actually a like
natural division like you can you can in
fact talk about these things separately
um and because of that it like makes
sense to have research organized around
those two things uh separately
but that is a claim i am making about
the best way to decompose the problem
that we actually care about
um and that is why i focus on intently
but like what do we actually care about
impact alignment totally
how would you say that your perspective
of this problem has changed over the
past year
i've spent a lot of time thinking about
the problem of of inner alignment
um so this was
this shut up too
people have been talking about it for a
while but it showed up to prominence in
i want to say 2019 with the publication
of the misoptimizers paper
and
i was not a huge fan of that framing but
i do think that the problem that it's
it's showing is like actually an
important one
uh so i've been thinking a lot about
that
can you explain what inner alignment is
and how it fits into the definitions of
uh what ai alignment is
yeah so
ai alignment the way i've
described it so far
is just sort of like
pretty
it's just talking about properties of an
ai system it doesn't really talk about
how that ai system was built
but if you actually want to diagnose at
like give reasons why problems might
arise and then how to solve them you
probably want to talk about how the ai
systems are built and why they're likely
to cause such problems
uh inner alignment
i don't
i'm not sure if i like the
the name but we'll go with it for now
inner alignment is a problem that i
claim happens for systems that learn
um and the problem is
um
maybe i should explain it with an
example
uh
you might have heard seen this post from
uh less wrong
about blacks and rubes
uh these black legs uh
are blue in color and tend to be egg
shaped
uh in all in all the cases they've seen
so far
rubes are are red in color and are cube
shaped at least in all the cases you've
seen so far
and now suddenly you see a red egg
shaped thing
is it a black or a rube
like in this case it's pretty obvious
that like you know there isn't a correct
answer
um
and
this same dynamic can arise in a
learning system
where if it is you know learning how to
behave in accordance with whatever we
are training it to do
we're going to be training it on a
particular set of situations and if
those situations change
uh in the future along some access that
the ai system didn't see during training
it may generalize uh badly so a good
example of this is
came from the objective robustness and
deep reinforcement learning paper
they trained an agent on
uh the coin run environment from procter
uh this isn't this is basically a very
simple platformer game where the agent
just has to jump over enemies
and and obstacles to get to the end and
collect the coin and the coin is always
at the far
far right end of the level
and so you know you train your ai system
on you know a bunch of different kinds
of levels different obstacles different
enemies they're placed in different ways
you have to jump in different ways but
the coin is always at the end on the
right
uh now it turns out
if you
then take your ai system and test it on
a new level where the coin is placed
somewhere else in the level not all the
way to the right
the
agent just continues to you know jump
over obstacles enemies and so on like
behaves very competently in the in the
platformer game but it just runs all the
way to the right and then like stays at
the right or jumps up and down as though
hoping that there's a coin there um
and like
i would
it's behaving as if
it has the objective of
go as far to the right as possible even
though we trained it on the objective
um
get the coin or at least that's what you
know we were thinking of as the
objective
uh and you know this happened because we
didn't show it any examples where the
coin was anywhere other than the right
side of the level so the inner alignment
problem is
when you train
a system on you know one
set of
inputs
it learns how to behave well on that set
of inputs but then when it's when you
extrapolated its behavior to other
inputs that you hadn't seen during
training
uh it turns out to do something that's
very capable but not what you intended
can you give an example of what this
could look like in the real world rather
than like a training simulation in a
virtual environment
yeah um
one example i like
uh
is
it's it'll take a bit of a bit of setup
but
i think it should be fine
um
you know you could imagine
that with
honestly even today's technology we
might be able to
train an ai system that can just
schedule meetings for you
like when someone emails you asking for
a meeting you're just like here calendar
scheduling agent please
you know do whatever you need to do in
order to get this meeting scheduled i i
want to have it you you go go schedule
it and then it you know goes and uh
emails the person who
emails back saying you know rohan is
free at such and such times he like
prefers uh morning meetings or whatever
and then you know there's a back and
forth between
uh
and then the meeting gets scheduled
for concreteness let's say that the way
we do this is we take a pre-trained
language model like say gpd3
um
and then we just
have gpt3 respond to emails
um
and we train it from human feedback well
we we we have some examples of like
people scheduling emails we do supervise
fine tuning on gpg3 to like get it
started and then we like fine tune
more from human feedback
uh in order to get it to be good at this
task and it it all works great
now let's say that
in 2023
gmail decides that you know gmail also
wants to be a chat app and so it adds
emoji reactions to
emails um
and everyone's like oh my god now
there's so much such a better we can we
can schedule meetings so much better
uh we can just like you know say
here here uh just send an email to all
the people who are coming to the meeting
and you know react with emojis uh for
each of the times that you're available
and you know everyone loves this this is
how people start scheduling meetings now
but it turns out that this ai system
when it's confronted with these emoji
emoji poles is like it knows it in
theory is capable or knows how to use
the emoji goals it like
knows what's going on
but it was always trained to like
schedule the meeting by email so maybe
maybe it will have learned to like
always schedule a meeting by email and
not to take advantage of these uh new
features so it might say something like
hey i don't um
i don't really know how to use these
newfangled emoji poles can we just
schedule emails the normal way
like
like in in our terms this would be a
flat out lie
but like from the ai's perspective we
might think of like the ai was just
trained to to say whatever you know
sequence of english words
led to getting a meeting scheduled by
email and it predicts that sequence of
words will work well
would this actually happen if i actually
trained an agent this way i don't know
like it's totally possible i would
actually do the right thing
uh but i don't think we can really rule
out the wrong thing either it seems that
also seems pretty plausible to me in
this scenario one important part of this
that
i think has come up in our previous
conversations
is that
we don't
know when there is always an inner
misalignment between the system and the
objective we would
like for it to learn because
part of maximizing
the inner aligned objective could be
giving the appearance
of
being aligned with the outer
objective that we're interested in could
you explain and unpack that
yeah
um so
you know we
in the ai safety community we tend to
think about ways that ais could like
actually lead to human extinction
and so you know the example that i gave
does not in fact lead to human
extinction
uh it is you know a mild annoyance at
worst
uh
the
the story that gets you to human
extinction
um is one in which you have a very
capable super intelligent ai system
uh but nonetheless there's like you know
instead of learning the objective
that we care that we wanted which might
have been i don't know something like be
a good personal assistant
i'm just giving that out as a concrete
example it could be other things as well
instead of
acting as though it were optimizing that
objective it ends up
optimizing some other objective
um i don't really want to give an
example here because the whole premise
is that it could be a weird objective we
don't really know
could you expand that a little bit more
like how it would be a weird objective
that we wouldn't really know
okay so let's take as a concrete example
it's make paper clips which has nothing
to do with being a personal assistant
now why is this at all plausible
the
reason is that even if this super
intelligent ai system had the objective
make paper clips
during training while we are in control
uh it's going to realize
that
if it doesn't do the things that we
wanted to do we're just going to turn it
off
and as a result it will be incentivized
to do whatever we want until it can make
sure that we can't turn it off and then
it goes and builds its paperclip empire
um
and
so when i say like it could be a weird
objective
i mostly just mean that
almost any objective is compatible with
this sort of a story
um it does rely on sorry i'm also
curious if you could explain how
the like the inner
state of the system becomes aligned to
something that is not what we actually
care about
i might go back to the coin running coin
run example where you know the agent
could have learned to get the coin that
was a totally valid policy it could have
learned
and this is an actual experiment that
people have run
um so this one is not hypothetical
uh it just didn't it learned to go to
the right
why i mean i don't know
i wish i understood neural nets well
enough to answer this question for you
i'm not really arguing for it's
definitely going to like learn make
paper clips i'm just arguing for like
there's this whole set of things that
could learn and we don't know which one
it's going to learn which seems kind of
bad
is it kind of like there's the thing we
actually care about and then a lot of
things that are like roughly correlated
with it which i think you've used the
word for example before is like proxy
objectives
um
yeah
so
that is definitely one way that it could
happen where
you know
we ask it to
make humans happy and it learns that
smile when humans smile
uh
they they're usually happy and then
learns the proxy objective of make
humans smile and then like
you know goes and tapes everyone's faces
uh so that they are permanently smiling
um
that's a way that things could happen
um
but like i think i don't even want to
claim that that's what like maybe that's
what happens maybe it just actually
optimizes for human happiness
maybe it learns to make paper clips for
just some weird reason i mean not paper
clips maybe it decides like this
particular arrangement of atoms and this
novel structure that we don't really
have a word for
is the thing that it wants for some
reason
and all of these seem totally compatible
with we trained it to be good to have
good behavior in the situations that we
cared about
because it might just be deceiving us
until
it has enough power to unilaterally do
what it wants without worrying about us
stopping it
i do think that there is
some sense of like
no paper clip maximization is too weird
if you trained it to
make humans happy it would not learn to
to maximize paper clips there's just
like no path by which like paper clips
somehow become the one thing it cares
about i'm also sympathetic sympathetic
too like maybe it just doesn't care
about
anything to the extent of like
optimizing the entire universe to turn
it into that sort of thing
um
i am really just arguing for we really
don't know
crazy could happen i i will bet on
crazy crazy this uh will happen
um unless we like
do a bunch of research and figure out
how to make it so that crazy
doesn't happen
um i just don't really know what the
crazy will be
do you think that that example of the
like agent in that virtual environment
you see that as like a demonstration of
the kinds of arbitrary goals that the
agent could learn and that that space is
really wide and deep and so it could be
arbitrarily weird and we have no idea
what kind of
goal it could end up learning and then
deceive us
i think it is not that great evidence
for that position
um mostly because like
i think it's reasonably likely that if
you told somebody the setup of what you
were planning to do
if you told an ml researcher or an rl
maybe specifically a dprl researcher
the setup of that experiment and asked
them to predict what would have happened
i think they probably would have um
especially if you told them hey do you
think maybe it'll just like run to the
right and
jump up and down at the end i think
they'd be like yeah that seems
likely not just plausible but actually
likely
um that was definitely my reaction when
uh i was first told about this result i
was like oh yeah of course that will
happen
um
like in that case i think we like just
do know
no is a strong word
ml researchers have good enough
intuitions about those situations i
think that that it was predictable in
advance
though i don't actually know of anyone
who predicted it in advance
um
so
that one i don't think is all that
supportive of it learns an arbitrary
goal like
we we had some like notion that neural
nets care a lot more about like
you know
position and like simple functions of
the action like always go right rather
than complex visual features like this
you know yellow coin that you have to
learn from pixels
uh that
i think people could have probably
predicted that
so we touched on like on definitions of
ai alignment and now we've been
exploring your
you know your interest in inner
alignment
or i think the jargon is mesa optimizers
um
they're they are different things there
are different things could you explain
how inner alignment and
mesa optimizers are different
yeah so
the thing i maybe have not been doing as
much as i should have
uh is that is the like inner alignment
is is the claim that like
when the circumstances
change
the agent generalizes catastrophically
in some way it like behaves as though
it's optimizing some other objective
than the one that we actually want
so it's much more of a claim about the
behavior
rather than like the internal workings
of the ai system that cause that
behavior
mesa optimization
at least under the definition of the
2019 paper
is uh
is talking uh specifically about ai
systems
that
are executing an explicit optimization
algorithm so like the forward pass of a
neural net is itself an optimization
algorithm we're not talking about
gradient descent here
and
then the metric that is being used in
that you know within the neural network
optimization algorithm is the inner
objective or sorry the mesa objective um
so it's making a claim about how then
how the ai system's cognition is
structured
whereas inner alignment more broadly is
just like the ai behaves in this like
catastrophically generalizing way
could you explain what outer alignment
is
sure
inner alignment can be thought of as
like you know
suppose we got
the training objective correctly correct
suppose like you know the things that
we're training the ai system to do on
the situations that were that we give it
as input like we're actually training it
to do the right thing
then things can go wrong if it like
behaves differently in some new
situation that we hadn't trained it on
outer alignment
is basically when the reward function
that you specify for training the ai
system is it's itself not what you
actually wanted
uh so for example maybe you want your ai
to be helpful to you or to tell you true
things uh but instead you have you train
your ai system
to
you know go find credible looking
websites and tell you what the credible
looking websites say and it turns out
that sometimes the credible looking
websites don't actually tell you two
things
in that case you're going to get an ai
that tells you what credible looking
websites say rather than an ai that
tells you what things are true
and that's in some sense an outer
alignment failure you like even the
feedback you were giving the ai system
was you know pushing it away from
telling you the truth and pushing it
towards telling you
what credible looking websites will say
which are correlated of course but
they're not the same
in general if you like give me an ai
system with some misalignment and you
ask me
was this a failure of outer alignment or
inner alignment
mostly i'm like that's a somewhat
confused question
but one way that you can
uh
you can make it not be confused is
you can say all right let's look at the
um let's look at
the inputs on which it was trained
now if ever on an input on which we
trained we gave it some like clearly
some wrong feedback where we were like
the ai like you know lied to me and i
gave it like plus a thousand reward and
you're like okay clearly that's outer
alignment we just gave it the wrong
feedback in the first place
supposing that didn't happen then i
think what you would want to ask is okay
let me think about
on the situation with situations in
which the ai does something bad
what would i have given counterfactually
as a reward
and this requires you to have some
notion of a counterfactual
uh when you write down a programmatic
reward function the counter factual is a
bit more obvious it's like you know
whatever that program would have output
on that input
and so i think that's the usual setting
in which outer alignment has been
discussed and it's it's pretty clear
what it means there but once you're like
training for me from human feedback it's
not so clear what it means like what
would the human uh have given this
feedback on the situation that they've
never seen before
it's often pretty ambiguous
if you define such a counterfactual then
i think i'm like yes
uh if then i think i'm like okay you
look at what
what feedback you would have given on
the counter factual
if that feedback was you know good uh
actually led to the behavior that you
wanted
then it's an inner alignment failure if
that counterfactual feedback was bad not
what you would have wanted then it's an
outer alignment failure
if you're speaking to someone who
was not familiar with ai alignment for
example other people
in the computer science community but
also policy makers of the general public
and you have all these definitions of ai
alignment that you've given
like intent
alignment and impact
alignment and then we have the inner and
outer
alignment problems
how would you capture the
the core problem of
ai alignment
and would you say that inner or outer
alignment
is
a bigger part of the problem
i would probably focus on intent
alignment for the reasons i have given
before
of it just seems like a more
like i i really do want to focus on the
cases where
i like i i want to focus attention away
from the cases where the ai
is trying to do the right thing but like
makes a mistake
which would be a failure of impact
alignment
but i like i don't think that is the
like biggest risk i think in a
a super intelligent ai system that is
trying to do the right thing
is like extremely unlikely to lead to
catastrophic outcomes though it's
certainly not impossible
or at least
more unlikely to lead to catastrophic
outcomes than like humans in the same
position or something so that would be
my justification for impact alignment
i prefer intent alignment sorry
i'm not sure that i would even talk very
much about inner and outer alignment
i think i would
probably like just not focus on
definitions and instead focus on
examples
the core argument i would make would
depend a lot on how ai systems are being
built
as i mentioned inner alignment is a
problem that according to me afflicts
primarily of learning systems i don't
think it really affects planning systems
what is the difference between a
learning system and a planning system
um a learning system you like give it
examples of how it should
of of things that you do how it should
behave and then like changes itself to
like
to to do to do things more in that vein
a planning system takes a like formally
represented objective
and then
uh searches over possible hypothetical
sequences of actions it could take in
order to achieve that objective
and
if you consider a system like that it
just
you can try to make the inner alignment
argument and it just won't work
which is why i say that the inner
alignment
problem is primarily about learning
systems
going back to the previous question
uh so
the the things that we talk about depend
a lot on what sorts of ai systems we're
building
if it were a planning system
i would basically just talk about outer
alignment
um
where it would be like what if the
formally represented
represented objective is not the thing
that we actually care about care about
it seems really hard to formally
represent the objective that we want
but if we're instead talking about like
deep learning systems
that are being trained from human
feedback
then i think i would focus
on
two problems
one is
cases where
uh the ai system
knows something
but the human doesn't and so the human
gives bad feedback as a result
so for example the ai system knows that
covid was
caused by a lab lake it's just like got
incontrovertible proof of this or
something
um and then
but you know we the we as humans are
like no we uh it when it says covet was
caused by a lab lake we're like
we don't know that and we say no bad
don't say that
uh and then when it says you know covid
is
we it is uncertain whether cobait is
the result of a lab like or naturally um
or or if it just occurred by natural
mutations
uh and then we're like yes good say more
of that and you're like you know
your ai system learns okay i shouldn't
report true things i should report you
know things that humans believe or
something
and so like that's that's one way in
which you get ai systems that don't do
what you want
and then the other way would be more of
this inner alignment style story
uh where
i would point out how even if you do
train it
even if all your feedback on the
training data points is is good
if the world changes in some way
the ai system might
stop doing good things um i might go to
example i mean i i gave the
gmail with emoji poles for meeting
scheduling example but another one now
that i'm on the topic of covid it's like
if you imagine
an ai system
if you imagine like a meeting scheduling
a assistant again
that was trained pre-pandemic
uh and then the pandemic hits and it's
like obviously never been trained on any
data that was collected during a
pandemic like
you know such a global pandemic
and so it's like
and so when you then ask it to schedule
a meeting with you know your friend
alice
uh it just you know schedules drinks in
a bar
uh on sunday evening even though like
clearly what you meant was a video call
and it knows that you meant a video call
it's just learned the thing to do is to
schedule
um
schedule outings with friends on sunday
nights at bars
sunday night i don't know why i'm saying
sunday night
friday night
have you been drinking a lot on your
sunday nights no not even in the
slightest
yeah i think really the problem is i
don't go to bars so i don't have a cash
standby
in my head that
people go to bars
so so how does this
how does this all lead to existential
risk
well the main argument is like
one possibility
is that your ai system just actually
learns to like ruthlessly maximize some
objective
uh that isn't the one that we want
um
like
you know make paper clips as in stylized
example to show what happens in that
sort of situation we're not actually
claiming that it will specifically
maximize paper clips
um
but like you know an ai system that like
really ruthlessly is just trying to
maximize paper clips
it is
going to
prevent humans from stopping it from
doing so
and if it gets sufficiently intelligent
and can take over the world at some
point is just going to turn all of the
resources into the world uh into paper
clips which may or may not include like
you know the resources and human bodies
but either way it's going to include all
the resources upon which we depend for
survival
so humans are definitely going like seem
like they will definitely go extinct in
that scenario
um
so
again not specific to paper clips this
is just ruthless maximization of an
objective
tends not to leave humans alive
and it and and both of these
well not both the mechanisms the inner
alignment mechanism that i've been
talking about
can
is compatible with an ai system that
ruthlessly maximizes an objective that
we don't want
uh it does not argue that it is probable
and i
am not sure if i think it is probable i
think it is but i think it is like
easily enough risk that we should be
like really worrying about it and and
trying to reduce it
for the outer alignment style story
where it's
um
where the problem is that you know the
ai may know information that you don't
and then you give it bad feedback
uh
i mean one thing is just this can
exacerbate this can make it easier for
an inner alignment style story to happen
where the ai learns
to optimize an objective that isn't what
you actually wanted
but even if you exclude something like
that paul christiano has written a few
posts about what a failure of how
a human extinction level failure of this
form could look like and it basically
looks like all of your ai systems lying
to you about how good the world is
as the world becomes
much much worse so for example
you know ai systems keep telling you
that the things that you're buying are
are good and helping your helping your
lives but actually they're not and
they're making them worse in some subtle
way that you can't tell
uh
like you were told like as all of the
information that you're fed seen
makes it seem like
um you know there's no crime police are
doing a great job of catching it but
really this is just manipulation of the
information you're being fed rather than
like actual amounts of crime
uh where like in this case maybe the
crimes are being committed by ai systems
not even by humans
um
so in all of these cases like
humans
relied on some like information sources
to make
decisions
uh ai's knew other information that the
humans didn't the ai has learned hey
my job is to like manage the information
sources that humans get so that the
humans are happy because that's what
they
that's when that that's what they did
during training
they like
gave good feedback in cases where the
information source said it was said
things were going well even when things
were not actually going well
right i mean it seems like if
human beings are constantly giving
feedback to
ai systems and the feedback is based on
incorrect information and the ai's have
more information
then they're going to learn something
that isn't aligned with
what we really want or the truth
yeah i do i do feel like
i do feel uncertain about the extent to
which this like
leads to human extinction
without
it leads to like
i think you can pretty easily make the
case that leads to an existential
catastrophe
uh as defined by i want to say it's
boston which is like you know includes
human extinction but also a permanent
curtailing of
humanities
i forget the exact phrasing but like
basically if humanity can't use
yeah exactly
um
that counts and like this totally falls
in that category
i don't know if it actually leads to
human extinction
um without some additional sort of
failure that we might instead categorize
as inner alignment failure
let's talk a little bit about
probabilities right so if you're talking
to someone who
has never encountered ai alignment
before
and um you know you've given a lot of
different real world examples and
principle-based arguments for why there
are these different kinds of alignment
risks
how would you explain
the probability of existential risk to
someone
who
can come along for all these
principle-based arguments
and buy into the examples that you've
given but still thinks this seems kind
of
far out there
like when am i ever going to see in the
real world a
ruthlessly optimizing
[Music]
ai
that's capable of ending the world
i think first off i'm like
super sympathetic to the this seems
super out there
uh critique
it's
like i spent
multiple years
not really agreeing with ai safety for
basically well not just that reason but
that was definitely one of the
heuristics that i was using
um
i think one way i would justify this is
to some extent it has precedent here
precedent already in that like
fundamentally the arguments that i'm
making
well especially the inner alignment one
um
is a an argument about how ai systems
will behave in new situations
um
rather than you know the ones it has
already seen during training
and we already know that ai systems
behave crazily in these situations uh at
the like most famous example of this is
adversarial examples where you take an
image classifier
um i think
oh man i don't actually remember what
the canonical example is i think it's
like a panda and you like change
change it imperceptibly
or change it change the pixel values
by a small amount such that you know the
change is imperceptible to the human eye
and then it's confident it's classified
with i think 99.8 percent confidence as
something else
my memory is saying airplane but that
might just be totally wrong anyway the
point is
like we have precedent for it ai systems
behaving really
weirdly on in situations they weren't
um trained on you might object that this
one is like a little bit cheating
because there was an adversary involved
and like the real i mean the real world
does have adversaries but still by
default you would expect the ai system
to be more like
uh exposed to
uh naturally occurring distributions i
think even there though you like often
you can just take an ai system that was
trained on one distribution given inputs
from a different distribution it's just
like
there's no
sense to what's happening
usually when i'm asked to predict this
the the actual prediction i give
is
probability
that um
we go extinct due to an intent alignment
failure and then some depending on the
situation i will either condition on i
will either make that unconditional so
that like includes
um all of the things that people will do
to try to prevent that from happening or
i make it conditional on like you know
the long term is community doesn't do
anything or like vanishes or something
but even in that world there's still
like you know everyone who's not a
long-termist who can still prevent that
from happening which i like really do
expect them to do
uh and
and so then i like i think i give like
my my like cash dancer on both of those
is like five percent on 10
respectively which i think is probably
the numbers i gave you if i like
actually sat down and like tried to like
come up with the probability i would
probably come up with something
different this time
but i have not done that and i'm like
way too angry on those previous
estimates to really give you a new
estimate this time
uh but but the like
higher number i'm giving now of like i
don't know 33 50 70
this this one's like way more insert i
feel way more uncertain about it it's
like
literally no one tries to like address
these sorts of problems
they just sort of like
take and take a language model fine tune
it on human feedback in the like very
obvious way and i just deploy that
um even if it like very obviously was
causing harm during training
they still deployed i'm like what's the
chance that leads to human extinction
and like
i don't know man maybe 33 maybe 70
and like the 33 number you can get from
this like you know one and three
argument that i was talking about
the second thing i was going to say is
like i don't really like talking about
probabilities very much because of how
utterly arbitrary
uh the methods of generating them are
they're like
um
i i feel much more
i feel much more robust
uh
i feel much better in the robustness of
the conclusion that like
we don't know that this won't happen
[Music]
and it is at least plausible that it
does happen
and i think that's like pretty
sufficient for justifying the work done
on it
i will also like argue
pretty strongly against anyone who says
we know
that it will kill us all if we don't do
anything
i i like don't think that's true
um
there are definitely you know smart
people who do think that's true
um if we'd like operationalize no as
like you know greater than 90 95 or
something
um
and and i disagree with them
um i don't really know why though
how would you respond
to someone who
thinks that this sounds like it's really
far in the future
um
yeah so
this is like specifically like agis are
in the future
yeah well so the concern here seems to
be about machines that are increasingly
capable
and when people look at machines that we
have today
like machine learning that we have today
sometimes we're not super impressed
and think that general capabilities are
very far off
and so this stuff sounds like
future stuff
yeah so i think my response depends on
like
you know what we're trying to get the
person to do or something like why why
do we care about what this person
believes
if this person is like considering
whether or not to do ai research
themselves
or ai safety research themselves
and they feel like they have a
strong inside view model of like why ai
is not going to come soon
i'm kind of you know i'm like uh
that seems okay i'm like not that stoked
about people
being like forcing themselves to do
research on a thing they don't actually
believe
i don't really think good research comes
from do from doing that like
it
i i if i put myself like for example i i
am much more sold on like
um agi coming through neural networks
than like planning agents or
or things similar to it
and if i'd like put myself in the shoes
of like all right i'm now going to do ai
safety research on planning agents
i'm just like oh man that seems like i'm
going to do so much my work is going to
be like orders of magnitude worse than
than the work i do on
in the neural net case
so if so in the case where i'm like
you know
this person is like thinking about
whether to do a safety research and they
feel like they have strong inside view
models
uh for agi not coming soon i'm like eh
maybe they should go do something else
um
or possibly they should like engage with
the arguments for agi coming
more more quickly if they haven't done
that but if they've like you know engage
with those arguments thought about it
all concluded it's far away and they
like can't even see a picture by which
it comes soon
i'm like you know that's fine
conversely
if we're instead
if we're imagining that like someone is
disputing oh someone is saying oh
nobody should work on the eye safety
right now because agi is so far away
um i mean one one response
you can have to that is like it's
you know even if it's far away it's
still worthwhile to work on reducing
risks uh if they're as bad as extinction
uh seems like we should be putting
effort into that even early on
but i think you know you can make a
stronger argument there which is like
you know there are just actually people
lots of people who are trying to build
agi right now
there is you know at the minimum deep
mind and open ai
and
they clearly
i should probably not make more comments
about deepmind but openai clearly
um doesn't believe
uh
the opening eye clearly seems to think
that like aji is coming
uh somewhat soon i think you can infer
from
everything you see about deepmind that
they don't believe that agi is you know
200 years away
i i think it is like
insane overconfidence in your own views
uh to be thinking that you know better
than all of these people
um such that you wouldn't even assign
like you know five percent or something
uh to
agi coming
soon enough for that that work on ai
safety matters
um
yeah so so there i think i would appeal
to
you know let other people do the work
you were not you don't have to do the
work yourself there's just no reason for
you to be opposing the other people uh
either at this either on episodic runs
or also on just like you know it's kind
of a waste of your own time
um
so that's the second kind of person and
the third kind of person might be like
somebody in policy from my impression of
policies that there is this
thing where like
early moves are relatively irreversible
or something like that things get
entrenched pretty quickly
um
such that it
makes sense to wait for
it often makes sense to like wait for a
consensus before acting
and like i don't think that there is
currently consensus of agi coming soon
and i don't feel particularly confident
enough in my views
to say like we should
really like convince policy people to
override this general heuristic of
waiting for consensus
um
and get them to act now
uh
yeah
anyway those are all meta-level
considerations
there's also the object level question
of like is aji coming soon
uh
for that i would say i think the most
likely
that the best story for that i know of
is
you take you know you you take neural
nets
you as you you scale them up
uh you increase the size of the size of
the data sets
that they're trained on you increase the
diversity of the datasets that they're
trained on
um and like they learn more and more
general heuristics
um
for like doing good things
and like eventually these general these
heuristics are like general enough that
they're like as good as
human human brain
uh human cognition
uh implicitly i am claiming that human
cognition is like basically a bag of
general heuristics
there is this um report from ajayakotra
uh uh about aji timelines using
biological anchors
and
i mean i wrote even my summary of it was
like 3 000 words or something like that
so i don't know that i can really give
an adequate summary of it here
uh but
it like models
the basic premise is to model how
quickly
uh
neural nets will
grow
um
and at what point they will match what
we would expect
the what would be we would expect to be
approximately the
same rough size as
uh the human brain
i think it even includes a small penalty
to neural nets
on the basis that like
um evolution probably did a better job
than we did it basically comes up with a
target for like you know neural nets of
this size trained in like compute
optimal ways will probably be like
roughly human level
um
and
it has a distribution over this to be
more accurate
and then it like predicts based on
existing trends um
well not just existing trends existing
trends and like sensible extrapolation
uh predicts when neural nets might reach
that level
and it ends up concluding
like
somewhere in the range
let me see if i i think it's 50
confidence interval would be something
like
20
35 to 20
70 20 80
maybe something like that
i am really just like you know i'm
imagining a graph in my head and trying
to like
calculate the area under it so so that
is very much not a reliable interval but
it should give you a general sense of
what the what the report concludes
so that's 2030 to 2080
i think it's a slightly narrower than
narrower than that but yes
roughly roughly that
that's pretty soon
yep
i think like that's
on the object level you just gotta gotta
read the report and see whether or not
you buy it
that's like
most likely in our lifetimes
if we live to the average age
yep
so that was a 50 interval meaning it's
like um
25 percent
275 percentile
i think actually the 25th percentile was
20 was not as early as 2030.
it was probably 2040.
so if if i've heard everything you know
the the in this podcast everything that
you've said so far
and
i'm
still kind of like
okay this like
there's a lot here and it sounds like
maybe convincing or something and um
it's this seems important but i'm like
not so sure that about this or that we
should do anything you know what is
because it seems like there's a lot of
people like that i'm curious what it is
the the that you would say to to someone
like that
i think it
i don't know i probably
wouldn't try to say something general to
them i feel like i would need to know
more about the person
like people have pretty different
idiosyncratic reasons
for having that sort of reaction
i mean okay
i would at least say that i think that
they are wrong to be having that sort of
belief or reaction
but if i wanted to like convince them of
that point
uh presumably i would have to say
something more than just i think you
were wrong
um i think the specific thing i would
have to say would would be pretty
different for for different people
i like maybe would at least i would
maybe make an appeal to like
the meta-level heuristic of like
don't try to regulate
like a small group of
you know barrel a few hundred
researchers
at most
doing things that they think will help
the world and that you don't think will
hurt the world there are just better
things for you to do with your time
it doesn't seem like they're harming you
um
some people will think that they're
there is harm being caused but caused by
them so i would have to address that
with them specifically but i think most
people do not who who have this reaction
don't believe that
so so we've gone over a lot of the the
the traditional arguments
for ai
as a potential existential risk
um
is there anything else that you would
like to add there or
any of the arguments that we
we missed that you would like to include
as a representative of the
representative of the community as a
whole
uh there are lots of other arguments
that people like to make
for ai being a potential extinction risk
uh so some some things are like you know
maybe ai just like accelerates the rate
at which we make progress and we can't
uh increase our wisdom
um
alongside and as a result we get a lot
of destructive technologies and can't
keep them under control or like
we don't do enough philosophy in order
to figure out what we actually care
about and what's good to do in the world
and as a result like
we like start optimizing for things that
are morally bad
um
or
or like uh other things in the spain
uh talk about the
you know uh the
risk of ai being misused
by bad actors
um so
there's well actually i'll introduce a
trichotomy that that
i forg i
yeah i don't i don't remember exactly
who
who wrote this article
but um it goes accidents misuse and
structural risks
so accidents are you know both alignment
and the
um
things like we don't keep up or we don't
have enough wisdom to
uh
to cope with the impacts of ai
that one's arguable whether it's an
accident or misuse or structural
um
and we don't do enough philosophy so
those are like
vaguely accidental
those are like accidents
misused is like
some bad actor
some terrorists say
gets ai that gets like a powerful ai
system and does something really bad
like blows up the world somehow
structural risks are
um things like
various parts of the economy like use ai
to accelerate to to get more profit to
accelerate their production of goods and
so on
um
and
at some point they just like we like
have this like giant economy that's just
making a lot of goods but it becomes
decoupled from things that are actually
you know useful for humans
and uh we just have this like huge
multi-agent system
uh
where goods are being produced money's
flowing around we don't really
understand all of it but somehow humans
get left behind
and there it's like not
it's kind of an accident but not in the
traditional sense like it's not that
like a single ai system went and did
something bad
um it's more like the entire structure
of how the way that the ai systems and
the humans related to each other
was such that it ended up leading to
permanent disempowerment of humans
uh now that i say it i think probably
the
like we didn't have enough wisdom
um argument for risk is probably also in
this category
which of these categories are you most
worried about
i
don't know i think it is probably not
misuse
but i like
i vary on accidents versus structural
risks
mostly because i just like don't feel
like i have a good understanding of
structural risks
uh maybe most days i think structural
risks are more likely to cause
bad extinction
this sort of obvious next question is
like why am i working on alignment and
not structural risks
and the answer there
is that it seems to me like alignment
has like
one or perhaps two like core problems
that are like leading to the to the
major risk
whereas like structural risks and and so
you could hope to like have a like one
or two solutions that address those main
problems
um
and like that's it that's all you need
uh whereas with structural risks i like
would be surprised if it was just there
was just like one or two solutions that
just like
got rid of structural risk
it seems much more like you have to have
a different solution for each of the
structural risks
so it seems like you know the amount
that you can reduce the risk by is
higher in alignment than in structural
risks
um and that's
i mean that's not the only reason why i
work in the alignment i'm also i just
also have a much better personal fit
with
with alignment work but i do also think
that alignment work you have more
opportunity to reduce the risk than in
structural
risks on the current margin
is there a name for those one or two
core problems in alignment
that
you could call solutions for i mostly
just mean like
i mean possibly like you know we've been
talking about outer and inner alignment
and like in the neural net case i talked
about you know the problem where the
you reward the ai system for doing bad
things because there was an information
asymmetry and then like the other one
was like the ai system generalizes
catastrophically to new situations
arguably those are just the two things
um but but i think it's not even that
it's more like
you know fundamentally the story
the like causal chain
in in the accidents cases like the ai
was trying to do something bad
or something that we didn't want rather
and then that was bad whereas like in
the structural risks case there isn't
like a single causal story
um
it's
this sort of very vague general notion
of like the humans and ais interacted in
ways that led to an x-risk
uh um
and then you like if you drill down into
any any
given story
or if you get drill down into like five
stories and then you're like what's
coming across these five stories you're
like not much other than that there was
ai and there were humans and they
interacted
uh and i like wouldn't say that was true
and if i had like five stories about
alignment failure
so
uh i i'd like to take a overview like a
broadside view of
ai alignment in 2021 last time we spoke
was
in 2020
so
how has ai alignment
as a
field of research changed
in the last year
i think i'm
going
to naturally include a bunch of things
from 2020 as well it's not a very sharp
division in my mind
um
especially because i think the like
biggest trend
is just more focus on
large language models
which i think was a trend that started
late 2020 probably
uh certainly you know the gpd3 paper
was
i i want to say early 2020
um but i don't think it like immediately
caused there to be more work so so maybe
late 2020 is about right
uh but you just see a lot more
um
you know alignment forum posts and
papers that are grappling with like
what do you
what are the alignment problems that
could arise with large language models
how might you fix them
um there is this you know paper out of
um stanford which isn't you know i
wouldn't i wouldn't have said this was
like from the ai safety community
uh it you know gives the name foundation
models to these sorts of things
um
so they generalize it beyond just
language um and you know they think it
might you know
and already we've seen some
generalization beyond language like clip
and dolly um are working on
image inputs
uh but they also like extend it to like
robotics and so on
and
their point is like you know we're now
more in the realm of like you know you
train one large model
on a bunch of like
a giant pile of data that you happen to
have
uh that you don't really have any like
labels for but you can use a
self-supervised learning objective in
order to learn from them and then you
get this model that has a lot of
knowledge but no like
goal built in
and then you do something like prompt
engineering or fine tuning in order to
actually
um get it to do the test that you want
um and so that's like a
new paradigm for constructing ai systems
that we didn't have before
uh and
there have just been a bunch
of posts that
uh grapple with you know what alignment
looks like in this sort of in in this
case
i don't think i have like a
nice pithy summary unfortunately of like
what all of us what the upshot is
but that's the thing that people have
been
people have been
thinking about a lot more
why do you think that
looking at
large-scale language models has become
a thing
oh i think primarily just because gpt3
demonstrated
um how powerful they could be
uh
you just see this is not specific to the
a safety community even in
them like if anything this
uh shift that i'm talking about is
it's probably not more pronounced in in
in the ml community but it's also there
in the ml community where there are just
like
tons of papers about prompt engineering
and fine tuning out of regular ml labs
um
it's just i think it's like gpd3 showed
that it could be done and that this was
like a reasonable way to get
actual economic value out of these
systems
um
and so people started caring about them
more
so uh one thing that you mentioned to me
that was significant in the last year
was foundation models
so could you explain what foundation
models are
yeah so a foundation model
the general recipe for it is you take
this some very
not generic exactly flexible input space
like
pixels or
any english
english language
any string of words in the english
language
you
collect a giant
data set
without any particular labels just like
lots of examples of that sort of
data in the wild so in the case of
pixels you just like find a bunch of
images from like
image sharing websites or something i
don't actually know where they got their
images from for for text it's even
easier and the internet is filled with
text you just get a bunch of it
um
and then you
train your ais you train a very large
neural network
uh
with some proxy objective
uh on on that data set that encourages
it to like learn how to model that data
set so in the case of
uh language
models the
i mean there are a bunch of possible
objectives uh the most famous one is the
one the gpd3 used
which is just you know given the first n
words of the sentence
predict the
uh word n plus one
um
and so it just like you know initially
it starts learning like
e's are the most common let well
actually
because of the specific way that the
input space in gp2 works it doesn't
exactly do this but you could imagine
that if it was just modeling characters
it would first learn that like e's are
the most common letter in the alphabet
vowels are more common q's and z's don't
come up that often like it starts you
know outputting letter distributions
that like at least look a vaguely more
like what english would look like
um
then it starts learning what you know
the spelling of individual words are
uh then it starts learning what the
grammar rules are just these are all
things that help it better predict what
the next word is going to be
um or well the next character in this
particular
instantiation
and you know
it turns out that you know when when you
have like
millions of parameters in your neural
network then you can like
i don't i don't actually know if this
number is right but like probably i
would expect that with millions of
parameters in your neural network you
can like
learn
you know spellings of words letter well
spellings of words and rules of grammar
such that you're like mostly outputting
you know for the most part grammatically
correct sentences but they have they
like don't necessarily mean very much
and then when you get to like the
billions of parameters uh arrange
at that point like you know
in the millions of parameters we're
already getting new grammar it's like
what what should it use all these extra
parameters for now then it starts
learning things like you know
george well probably already even the
millions of parameters probably learned
that george tends to be followed by
washington but it can like start
learning things like that and in that
sense can be said to
know
that there is a
entity at least named george washington
and so on it might start
knowing that rain is wet and like in
context where you know something has
been drained on and then later we're
asked to describe that thing it will
like say it's wet or slippery or
something like that
and so it like starts
basically just in order to predict where
it's better it like keeps
getting more and more
knowledge
um about the domain
so anyway a foundation model
expressive input space
giant pile of data very big neural net
learns to model that domain very well
which like
involves getting a bunch of knowledge
about that domain
um
what's the difference between knowledge
and knowledge
uh i mean i feel like you're the
philosopher here okay more than me
do you know what knowledge without air
quotes is
no
i don't
i don't mean to derail it but yeah so
it's yeah so it gets knowledge
yeah i mostly put the air quotes around
knowledge because like we don't really
have a satisfying account of what
knowledge is
and like if i don't put air quotes
around knowledge i get lots of people
angrily saying that ai systems don't
have knowledge yet oh yeah
when i put the air quotes around it then
they understand that i just mean that it
like you know has the ability to make
predictions
that are conditional on like this
particular fact about the world whether
or not it like actually knows that fact
about that world
about the world
but like it knows it well enough to do
to make predictions or it
contains the knowledge well enough to
make predictions it can make predictions
that's the point
i'm being maybe a bit more too um
harsh here like i also put air quotes
around knowledge because i don't
actually know what knowledge is
it's not just a like defense strategy
though that is definitely part of it so
yeah foundation models um
basically they're a way to like just get
all of this knowledge
uh
into
into an ai system such that you can then
do like prompting and fine tuning and so
on and those like with a very small
amount of data relatively speaking
um are able to get very good performance
it's like in the case of gpg3 you can
like give it you know two or three
examples of a task and it can start
performing that task if the task is
relatively simple whereas if you wanted
to train a model from scratch to perform
that task you would need like thousands
or more
thousands of examples often
so how has this been significant for ai
alignment
i think it has mostly like provided an
actual
pathway
to whit by which we can get to aji
otherwise like there's like more like a
concrete story and path
that like leads to agi eventually
and so then we can take all of these
abstract arguments that we were making
before and then
c try to like
instantiate them in the case of this
concrete pathway and see whether or not
they still make sense
i'm not sure if at this point i'm like
imagining what i would like to do versus
what actually happened
i
would need to actually go and look
through the alignment newsletter
database and see what people actually
wrote about the subject
but like
i think there was like some discussion
of like gpt3 and the extent to which it
is or isn't a mesa optimizer
um
yeah that's at least one thing that i
remember happening then there's been
there's been a lot of like papers that
are just like here is how you can do um
here is how you can like train
a foundation model like gpd3
uh to do the sort of thing that he wants
like there's
uh learning to summarize from human
feedback which just took gpd3 and like
taught it how to
or fine-tuned it in order to get it to
summarize news articles which is like an
example of a task that you might want an
ai system to do
um and then like the same team at openai
just recently released a paper that like
actually summarized entire books by
using a recursive decomposition strategy
um so there's been some amount there's
like
in some sense a lot of the work we've
been doing in the past in the island was
like how do we get ai systems to perform
fuzzy tasks for which we don't have a
reward function
and now we have
systems that like could do these fuzzy
tasks in the sense that they like have
the knowledge
but like
don't actually
you know use that knowledge
um the way that we would want them and
then we like have to figure out how to
get them to do that and then we can use
all these techniques like imitate
imitation learning and learning from
comparisons and preferences that we've
we've been developing
why don't we know that ai systems won't
totally kill us all the arguments for
air at risk
usually depend on having an ai system
that's like ruthlessly maximizing an
objective in every new situation it
encounters
so like for example the paperclip
maximizer you know once it's built 10
paper clip factories it doesn't
it's it doesn't retire and say yep that
that's enough paper clips
uh it like just you know
continues
turning like entire planets into into
paper clips
similarly
or if you like
consider the goal of like make 100 paper
clips then it like turns the turns all
the plants into
computers to verify that it it to make
sure it is as confident as possible that
it has made a hundred paper clips
um
like these are like examples of
i'm gonna call ruthlessly ruthlessly
maximizing an objective
uh
and like there's some sense in which
this is
weird
and like humans don't behave in that way
uh
and i think there's
some amount of like
basically i am unsure whether or not
we should actually expect ais to
have such ruthlessly maximized
objectives
i don't really see the argument for why
why that should happen i think like as a
particularly strong piece of evidence
against this i would know that like
humans don't seem to have these sorts of
objectives
um
it's not obviously true there are
probably some long-termists who like
really do want to tile the
universe with hedonia
which seems like a pretty ruthlessly
maximizing objective to me
but i think
even even then
those are like that's the exception
rather than the rule
um so like if humans don't maximize or
ruthlessly maximize objectives
and humans were built by like a similar
uh as is building neural networks
why do we expect the neural networks to
have
um objectives that they ruthlessly
maximize
you can also you know
i've phrased this in a way
where it's an argument against ai risk
you can also phrase it in a way in which
it's an argument for a high risk where
you would say well you know let's flip
that on it's on its head and say like
well yes you brought up the example of
humans well the process that created
humans was trying to maximize or like
you know it is a thing it is
an optimization process
leading to increased reproductive
fitness but then humans do things like
wear condoms which does not seem great
for reproductive fitness generally
speaking
um especially you know for the people
who are definitely out there who like
decide that they're just never going to
reproduce
so in that sense like humans
are clearly like
having a large impact on the world
uh
and are doing so for
objectives that are not what evolution
was naively optimizing
um
and so like similarly if we train ai
systems in the same way maybe they too
will like have a large impact on the
world but not for what the humans were
naively you know training the system to
optimize
we can't let them know about fun
yeah
terrible they must they just yeah well
i don't want the whole human
ai alignment project will run off the
rails
yeah
but anyway i think like
just
these things are like a lot more
conceptually
tricky
than
uh
the you know well-polished arguments
that one reads will make it seem
um
but especially this point about like
it's not obvious that ai systems
will get ruthlessly maximizing
objectives
like that really does give me quite a
bit of pause
um
in
how good the air risk arguments are i
still think it is like clearly correct
to be working on the i risk because like
we don't want to be in this situation
where like we can't make an argument for
why ai is risky we want to be in the
situation where we're like we can make
an argument for why the ai is not risky
uh and i don't think we have that
situation yet
even if you like completely by the like
we don't know if there's going to be
ruthlessly maximizing objectives
argument
that puts you in the epistemic state
where we're like well i don't see an
ironclad argument
that says that ai's will kill us all and
that's sort of like saying
like
i don't know
well
i don't have an ironclad argument that
touching this pan that's on this you
know lit stove will burn me
because you know maybe someone just put
the pan on the stove a few seconds ago
uh but it would still be a bad idea
to to go and do that what you really
want is a you know positive argument
for
why the pan why touching the pan is not
going to burn you
or analogously why building the agi is
not going to kill you
um
and i don't think we have any such
positive argument
at the moment
part of this conversation is interesting
because i'm like surprised how uncertain
you are about ais and existential risk
yeah
it's possible i've become slightly more
uncertain about it in the last year or
two i don't think i was saying things
quite quite this much quite uh
i don't think i was saying things that
were quite this uncertain before then
but i think i
um
have generally been like you know we
have plausibility arguments we do not
have like this is probable arguments
or you know back in like 2017 or 2018
when i was young and naive uh
well okay i i like entered the field of
ai alignment i like read my first ai
alignment paper in like september of
2017.
so so this
it it actually does make sense
uh at that time i thought we had more
confidence
um of some sort but but like
since posting the value learning
sequence i've generally been like more
uncertain about ai risk arguments
i don't i don't like talk about it all
that much because as i said the decision
is still very clear
the decision is still like work on this
problem
figure out how to get a positive
argument that the air is not going to
kill us
uh
and ideally you know a positive argument
that the ai does good things for
humanity
i don't know man most things in life are
pretty uncertain
most things in the future are like even
way way way more uncertain
um
i don't feel like you should generally
be all that confident about
technologies that you're that you think
are decades out
it feels a little bit like those uh
those images of the people in the 50s
drawing what the future would look like
and everyone and the images are like
ridiculous yup
yeah
i
i've been recently watching star wars
now obviously star wars is not actually
supposed to be a prediction about the
future but it's it's really quite
entertaining to
to like actually just
think about all the ways in which star
wars would be totally inaccurate and
this is like before we've even invented
uh space travel
but just like robots talking to each
other using
sound
why would they do that
industry today wouldn't make machines
that
speak
by vibrating air
they would just like send each other
signals electromagnetically
so
how much of the alignment and safety
problems in ai do you think will be
solved by industry
the same way that like
computer to computer
communication that is solved by industry
and is not what star wars
thought it would be um
would the deep mind ai safety lab exist
if deepmind didn't think
that
ai alignment and ai safety were serious
and important
like i don't know if
the
lab is
purely aligned with the
commercial interests of deep mind itself
or if it's also kind of seen as like a
good for the world thing i bring it up
because i like uh how andrew kritch
talks about it and his arches uh paper
yep so critch is i think of the opinion
that like both
preference learning and robustness are
problems that will be solved by
uh industry
i think he includes robustness in that
um and i like certainly agree to the
extent that you're like yes companies
will do things like
uh
learning from human preferences
totally they're they're gonna do that
uh whether they're going to like
be proactive enough to notice
um the the kinds of failures i mentioned
i don't know it doesn't seem nearly as
obvious to me that they will be
without like
you know dedicated teams that are
specifically meant for looking at you
know
looking for hidden failures with the
knowledge that like these are really
important to get because they could have
very bad long-term consequences
ai systems
could
increase the strength of
and accelerate
various multi-agent systems and
processes
that
when accelerated
could lead to bad outcomes so for
example
a great example of a destructive
multi-agent system a multi-agent effect
is like war
uh
you know war is a thing that like
uh well i mean wars have been getting
more destructive over time
um with
or at least the weapons in them have
been getting more destructive probably
the death tolls have also been getting
higher but i'm not as sure about that
um and
you could imagine that if ai systems
uh
continue to
increase if like they increase the
destructiveness of weapons even more
uh
wars might then become an existential
risk
uh so that's like
a way in which
you can get a structural risk from a
multi-agent system
um the
thing the
example in which like the economy just
sort of becomes much much much bigger
but becomes decoupled from things that
humans want
uh is another example of how a
multi-agent
process can sort of go haywire
uh especially with the addition of
ai's powerful ai systems
i think that's also a canonical scenario
that crutch would think about
um
yeah
so that's yeah
it's really i would say that like arches
is
i in my head it's categorized as like
a technical
paper about structural risks
do you think about what beneficial
futures look like you spoke a little bit
about wisdom earlier
and i i'm curious what good futures with
ai looks like to you
yeah um
i admit i don't actually think about
this very much
because i'm put my research is focused
on more abstract problems
i tend to focus on abstract
considerations and like the main
abstract consideration from the
perspective of the good feature is like
well once once we get to you know
singularity levels of
powerful ai systems
like anything i say now there's going to
be something way better that the
ai systems are going to enable
uh so then as a result i don't think
very much about it you work a lot on
this risk so you must think that
humanity existing in the future matters
i mean i do i do like humans humans are
pretty great
i count many of them amongst amongst my
friends i've never been all that good at
the sort of trans
the transhumanist look to the future and
see the grand potential of humanity
uh
sorts of visions
but like
when other people say them or give them
i like feel a lot of kinship with them
you know the the ones that are all about
like you know humanity's potential to
like discover new forms of art and music
uh reach new levels of science
understand the world better than it's
ever been understood before
fall in love uh a hundred times you know
uh learn all of the things that there
are to know
actually you won't be able to do that
one probably but anyway
learn way more of the things that there
are to know that any than than you have
right now
uh
like just a lot of that resonates with
me and that's probably a very
intellectual centric
um
view of the future
uh i feel like i'd be interested in
in hearing the like you know
view of the future that's like is
we like
have the best you know
video games and the best tv shows and we
we're the best couch potatoes that ever
were
um or also they're just like you know
insane new sports that you have to like
spend
uh
lots of time and grueling training for
but it's all worth it when you like
uh you know shoot the best
you know get a perfect 50 on perfect
score on like the best uh dunk that's
ever been done in basketball or whatever
um
i recently watched a competition of like
apparently there are competitions in
basketball of like um
just like
aesthetic uh dunks
it's cool i enjoyed it anyway
um
yeah it feels like there are just so
many other communities that could also
have their own visions of the future and
i feel like i'd
um
feel a lot of kinship with many of those
too and i'm like
man let's just have all the humans just
continue to do the things that they want
seems great
one thing that you mentioned was
that
you deal with abstract problems
and so
what a good future looks like to you
is
a little it seems like it's like an
abstract problem that
later
the good things that ai can give us are
better than the good things that we can
think of right now is that a fair
summary
seems right yeah
right so there's like there's this view
and this comes from maybe stephen pinker
or someone else i'm not sure or maybe
ray kurzweil i i don't know um where
you know if you give a caveman a genie
or like an ai they'll ask for maybe like
a bigger cave and like i would like
there to be more hunks of meat
and
i would like my
like pelt for my bed to be
a little bit bigger go ahead okay i
think i see the issue so i actually
don't agree with your summary of the
thing that i said
uh okay i said that
your your rephrasing was that
um
the
we like ask the ai what good things
there are to do
um or something like that
and that might have been what i said
but what i actually meant
um
was that like
with powerful ai systems like the world
will just be very different and like one
of the ways in which it will be
different is that we can get advice from
ai's on what to do
um and certainly that's an important one
but also there will just be like
incredible new technologies uh that we
don't know about new realms of science
to explore
uh
new concepts that we like don't even
have the
have names for right now and one that
seems particularly interesting to me
it's just like entirely new senses like
i just have like
you know we human vision is just like
incredibly complicated but like i can
just look around the room and identify
all the objects with basically no
conscious thought what would it be like
to like understand dna at that level
like alpha fold probably understands dna
at some maybe not quite that level but
something like it
um
like
i don't know man it's not there's just
like all these things that i'm like
you know i thought of the dna one
because of alpha fold if before alpha
fold would have thought of it probably
not
i don't know maybe crystal has written a
little bit about things like this but
like it feels like there will just be
like far more opportunities
and then also we can get advice from
ai's but like that's probably actually
and and that's important but i think
less than they there are far more
opportunities that like i am definitely
not going to be able to think of today
do you think that it's dissimilar from
the caveman like wishing for more
caveman things
yeah like
i feel like the
in the
the the caveman story
like
if the king like it's possible
that the caveman does this but i feel
like the the thing that the caveman
should be doing
is like something like you know
give me new give me like better ways to
like
uh to to get
give me better food or something and
then like you get fired to cook things
or something like
the things that he asked for should like
involve technology
um
as a solution
you should get technology as a solution
you should like learn more and be able
to do more things as a result of having
that technology
and like you know in this hypothetical
at that the caveman should like
reasonably quickly become like
similar to
modern humans
i don't know what reasonably quickly
means here but like
it should be much more like you know you
get access to more and more technologies
rather than like you get a bigger cave
and then you're like i have no more
wishes anymore
just like
i'm like if i got a bigger house would i
stop having wishes that seemed super
unlikely
um i think i like that's a straw man
argument sorry but still i i like do
feel like there's a meaningful sense in
which like getting new technology leads
to just genuinely new circumstances
uh which leads to more opportunities
which leads to like probably more
technology and so on and like at some
point this has to stop
um
there are like limits to what is
possible
one assumes there are limits to what is
possible in the universe
uh but i think like once we once we get
to talking about we're at those limits
then i'm like
you know at that point it's like
probably at that point it just seems
irresponsible to speculate it's just so
wildly out of the range of things that
we know
like at that point i'm like they're
probably just not
the concept of a person is probably
wrong at that point
the what of a person is probably wrong
at that point the concept of a person
oh
i'd be like is there an is there an
entity that i would that that is rohin
at that time like
not likely
less than 50
we'll edit in just like uh
fractals flying through
your video at this part of the interview
so in my example i think it's just
because i think i think of cavemen as
not knowing how to ask for new
technology but we want to be able to ask
for new technology
um and part of what this brings up for
me is this very classic part of ai
alignment and i'm curious how you feel
like it fits into the problem but we
would also like ai systems to help us
imagine
beneficial futures potentially or to
know like what is good or what it is
that that we want so in asking for new
technology
it knows that fire is part of
the good that we
don't know how to necessarily ask for
directly
how do you how do you view
ai alignment
in terms of
itself
aiding in
the creation of beneficial futures and
knowing what is
knowing of a good that is beyond the
good that humanity can grasp
i think i more like reject the premise
of the question
where i'd be like
there is no good beyond that which
humanity can grasp this is like somewhat
of an anti-realist position
um and like you mean moral anti-realist
just yes
yes sorry i should have said that
more clearly yeah somewhat of a moral
anteriorist position but it's like
you know there is
no good other than that which humans can
grasp but like you know within that
could grasp that you can like you know
have humans thinking for a very long
time you could have them like with extra
you can make them more intelligent like
part of the technologies you get from ai
systems
uh will presumably let you do that maybe
you can like
um
i guess set aside questions of like
identi philosophical identity you could
like upload the such that they could run
on a computer run much faster have like
software upgrades to be you know to the
extent that that's philosophically
acceptable
so like
you know there's a lot you can do to
help humans grasp more
um and like ultimately i'm like yes
the like closure of all these
improvements
where you get to with all of that that's
just like is the thing that we want and
like
yes you could have a theory that there
is
something even better and even more out
there
that humans can never access by
themselves
and
i'm like that just seems like a weird
hypothesis to have
and i don't know why you would have it
but in the world where that
hypothesis is true and like i don't know
like if i condition on that hypothesis
being true i don't see why we should
expect that ai systems could access that
further truth um any better than we can
if it's like out of our like you know
the closure of what we can achieve even
with additional intelligence and such
like there's no other advantage that ai
systems have over us
so is is what you're arguing that um
with human augmentation
and help to human beings so with uploads
or with
you know expanding the intelligence and
capabilities of humans that humans have
access to the entire space of what
counts as good
you're like
i think you're like presuming the
existence of an object that is the
entire space of what is good
and i'm like there is no such object
there are only humans and what humans
want to do
and like if you want to define the space
of what what is good you can like define
this like closure property on like what
humans will think is good
like with all of the possible
intelligence augmentations and time and
so on and like that's a reasonable
object and i like i could see calling
that as the like space of what is good
but then like almost tautologically we
can reach it with technology
that's the thing i'm talking about um
the version where you like posit the
existence of the like entire space of
what is good
is like hey i can't really conceive of
that i like don't it doesn't feel very
coherent to me but b when i try to
reason about it anyway
i'm like okay
if humans can't access it why why should
ais be able to access it you've posited
this
new object of like a space of things
that humans can never access
but like how does that space
affect
or interact with reality in any way
like there needs to be some sort of
interaction in order for the ai to be
able to access it
i think i would need to know more about
the
how it interacts with reality in some
way
before i could like meaningfully answer
this question in a way that like
where i could say how ai's could do
something that like humans couldn't even
in principle do
what do you think of the importance or
non-importance of these kinds of
questions and how they fit into the
ongoing
problem of
ai alignment
i think they're important
for determining what the goal of
alignment should be
so
for example
you now know a little bit of what my
view on these questions is
uh which is namely something
like
it's you know
that which humans can access access
under like sufficient augmentations
intelligence time so on is like all that
there is
and so i'm i'm like pretty very into
like
build ai systems that are like
replicating human reasoning that are
sort of approximating what a human would
do if they thought for a long time or
were smarter in some ways and so on
um and so then like yeah we don't
need to worry much about like
or or so i'm i i tend to think of it as
like let's build ai system to systems
that just do tasks that humans can
conceptually understand
and not necessarily they can do it but
they like know what that sort of task is
and then our job is to like
you know the the entire human ai society
is like making forward progress
um towards
making forward moral progress or other
progress
um
in the same way that it has happened in
the past which is like we get exposed to
new situations and new arguments we like
think about them for a while and then
somehow we make decisions about what's
good and what's not in a way that's like
somewhat inscrutable
um
like
so i'm much more about and so we just
continue trading that process and
eventually we like reach the space of
you
know well yeah we just continue it
writing that process so i'm like very
much into the like because of this view
i think it's pretty reasonable to like
aim for ai systems they're just like
doing human-like reasoning but better
um or like approximating what you know
doing what a human could do in a year in
like a few minutes or something like
that that seems great to me
whereas if you on the other hand were
like no there's actually like deep
philosophical truths out there that
humans might never be able to access
then you're probably less enthusiastic
about that sort of plan and you want to
do
you want to build ai system some other
way
or maybe they're accessible
with the augmentation and time
um
how how does how does other minds
fit into this for you
so
like right there's
the human mind and then the space of all
that is good that it has access to with
augmentation which is what you call the
space of
that which is good
um it's
contingent and rooted on
the space of what the human mind
augmented has access to
um
how would you view
uh how does that fit in with animals and
also other species which may have their
own alignment problems on
planets
within our
cosmic endowment
that we might run into
is it just that they also have spaces
that are
defined as good as what they can access
through their own augmentation and then
there's no
way of reconciling
these two different ai alignment
projects
yeah i think basically yes
like you know if if i met uh like actual
ruthless maximizing paperclip uh
paperclip maximizer
it's not like i can argue it into
adopting human my my values
or anything even resembling them i don't
think it would be able to argue me into
accepting
you know turning my turning me into
paper clips
uh which is you know what it desires
and like
yeah that
that just seems like the description of
reality
um again a moral realist might say
something else but i've never really
understood that the flavor of moral
realism that would say something else
in that situation
with regards to the planet and industry
and how industry will be creating
increasingly capable ai systems could
you explain
what a unipolar scenario is and what a
multi-polar scenario is
yeah so
i'm not sure if i
recall exactly where these terms were
defined but a unipolar scenario
at least as i understand it
uh would be
a
situation in which
one entity
basically has determines the long run
feature of the earth
um
so like it's in you know more
colloquially it has taken over the world
you can also have like a time-bounded
version of it where it's like you know
unipolar for like
20 years and like this entity has like
all the power for those 20 years but
then like
you know maybe the entity is a human and
we haven't solved agent yet and then the
human dies uh so i'm like it was a
unipolar world for that that period of
time
um a multi-polar world is just you know
not that there
there is no like one entity that is said
to be in control of the world uh they're
just a lot of different
entities that have different goals
um and they're coexisting hopefully
cooperating
maybe not cooperating
it depends on the situation
which which do you think is more likely
to lead to beneficial outcomes with ai
so i think i don't really think about it
in these terms i think about it in like
you know there are these like kinds of
worlds that we could be in
and like some of them are unipolar and
some of them are multiple like very
different human polar roles and very
different multiple worlds
and so like like sorts of questions
it like the closest an analogous
question is something like
you know if you condition on unipolar
world
what's the probability that it's
beneficial
or that it's good if you condition on
multiple world what's the probability
that it's good and just like a super
complicated question that like i
wouldn't be able to explain my reasoning
for because it would involve me like
thinking about like 20 different worlds
and maybe not that many but like a bunch
of different worlds in my head
estimating their probability is doing
like a base rule
not a base
i guess kind of a base rule bayes rule
calculation
um
and then reporting the result
so
i think
maybe the
question i will answer instead
is like the most likely worlds in each
uh
in each of unipolar multiple
settings
and then like why how how good those
seem to me so i would say i think by
default i expect the world to be
multi-polar
uh
in that it doesn't seem like
anyone is
you know particular i don't think anyone
has particularly taken over the world
today any or any entity like not even
counting the us as a single entity it's
not like the has taken over the world
um
and
it does not seem to me like
the main way you could imagine
uh getting a unipolar world is like if
the first
the first um actor to build a powerful
enough ai system
that ai system just becomes really
really powerful and takes over the world
before anyone can deploy an ai system
even close uh
even close to it sorry that's not the
most likely one that's the one that
people most often talk about
um
and probably the one that other people
think is most likely
um
but yeah anyway so i
i see the multiple world as more likely
where we just have you know a bunch of
actors that are all pretty well
resourced that are all developing their
own own ai systems they like then like
sell
their ai systems or like the ability to
use their ai systems to other people
um and they're just like sort of similar
to
the human economy where you can just
like have
ai systems provide labor
at a fixed cost
and it just sort of looks similar to the
economy today where
people who control a lot of resources
can like instantiate a bunch of ai
systems
that help them maintain whatever it is
they want
uh and we retain remain in the
multi-polar world we have today
um
and that seems
decent
i think i like
for all that
our institutions are not looking great
at the current moment
there is still something to be said that
like
you know nuclear war didn't actually
happen
um
which can either update you towards
uh
our institutions are like
somewhat better than i than we thought
or it can update you to towards if we
had nuclear war we would have all died
and not been here to ask the question
i don't think that second one is all
that plausible my understanding is that
nuclear war is not that likely to wipe
out everyone
um or even
90 of people so i'm more i i lean
towards the first explanation
overall my guess is that like
you know this is the
the thing that has worked for the last
worked the thing that is like
generally led to an increase in
prosperity and like
or the world has clearly
improved on most metrics
over time
and like the system we've been using for
most of that time is like some sort of
multiple
people interact with each other and keep
each other in check and like
uh
like cooperate with each other because
they have to and so on
um and like in the modern world we use
like
and and not just the modern world we use
things like regulations and laws and so
on uh
to to enforce this
and like you know
the system's got some history behind it
so i i'm like more inclined to trust it
uh so i overall feel
okay about
uh this world
you know assuming we solve the alignment
problem that's
we'll ignore the alignment problem for
now
uh for a unipolar world
i think
probably i
i find it more likely that
there will just be a lot of
returns to
um scale so like the just you'll get a
lot of efficiency from centralizing
uh more and more
in the same way that like
it's just really nice to have a single
standard rather than have 15 different
standards
like it sure would be nice it sure would
have been nice if like when i moved to
the uk i could have just used all of my
old
old um chargers without having to buy
adapters
but no all the outlets are different
right like there's there's there's
benefits to standardization and
um
centralization of power
and it
seems to me like there's been more and
more of that
over time
maybe it's not obvious i look i'm yeah i
don't know very much history
um
but
if
so so it seems like you could get
even more centralization in the future
uh in order to capture the efficiency
benefits and then you might have a
global government that could reasonably
be said to be just a uni like the entity
that controls the world
um and that would then be a unipolar
outcome
uh it's not a unipolar outcome in which
the thing in charge of the world is an
ai system but it isn't it is a unipolar
outcome
and
i think i
i feel
wary of this
but i don't like having a single point
of failure
um i don't like
it when like there's a when when like i
or like i really like the
i really like it when people are allowed
to like you know advocate for their own
interests
um
which you know isn't necessarily not
happening here right there this could be
a global democracy
uh
but but still it seems like you know
it's a very lim it like
the libertarian
intuition of like markets are good
generally tends to suggest against
centralization and i like do buy that
intuition
uh but this could also just be a
like status quo bias where i'm like i
know the world i can very easily see the
um
the problems in in the world that we're
not actually in at the moment
uh and i don't want it to change
so i don't know i don't have
super strong opinions there it's very
plausible to me that like that world is
better because then you can like
um control
dangerous technologies much much better
like if there just are
technologies that are sufficiently
dangerous and destructive that they
would
destroy
they would lead to extinction then maybe
i'm more inclined
to favor a unipolar outcome
i would like to ask you about deep mind
and maybe another question before we we
wrap up
um
so
what is it that the that the safety team
at deepmind is is up to
no one thing
um the safety team at deepmind is like
reasonably large and there's just a
bunch of projects going on
uh i've been doing a bunch of inner
alignment stuff uh most recently i've
been
uh trying to come up with more
examples that are you know in actual
systems rather than hypotheticals
i've also been doing a bunch of
conceptual work of just like trying to
make our arguments clearer and more
conceptually precise
a large smattering of stuff
um not
all that related to each other except
them as much as it's all about ai
alignment as a final question here rohan
i'm
interested in your your your core
uh your core at the center of all this
so you know
what's the most important thing to you
right now like in so far as
ai alignment
maybe
the one thing that
most largely impacts the future of life
ah
[Music]
like if you just look at the universe
right now and you're like these are the
most important things
i think
for
things that i impact
uh
you know
more granul like more granular than just
make ai go well
i think for me it's probably like
making better arguments and more
convincing arguments
um
currently
this will probably change in the future
partially because i hope to succeed at
this goal and then it won't be as
important
um
but i feel like
uh
right now
especially with the like advent of these
like large
large neural nets and more people like
seeing a path to agi
i think it is
much more possible
to make arguments that would be
convincing to ml researchers as well
um as well as like the you know
philosophically oriented people who make
up the ai safety community
and they think that just feels like the
most
useful thing i can do at the moment
in terms of the world in general
i feel like it is
something like the attitudes of
consequential people
two words
um
well long-termism in general but maybe
risks in particular
where
and like importantly like i do feel like
it is more like
i i care primarily about
you know the people who are actually
making decisions that impact the future
maybe they are
taking into account
the future
maybe they're like
it would be nice to care about the
future
but
the
realities of politics mean that i can't
do that or else i will be lose my job
but my guess is that they're mostly just
not thinking about the future
and like that seems
if you're talking about the future
of life
that seems
like the most
that seems pretty important to change
how do you see doing that
when
um many of these people don't have the
as sam harris put it the science fiction
geek gene
is what he he called when he was on this
podcast he's like all you know the long
termists
who are all like we're going to build
agi and then
create these radically
different futures
um
like many of these people may just
mostly care about their children
and their grandchildren like that may be
the human
tendency
do we actually advocate for any actions
that would not impact their
grandchildren
it depends on your timelines right
fair enough
but like most of the time the arguments
i see people giving for any preferred
policy proposal of theirs
or act just like almost any action
whatsoever
it seems to like be a thing that would
have a like noticeable effect on
people's lives in the next
100 years
so like in that sense grandchildren
should be enough
okay so then long-term ism is doesn't
matter
well i
i mean i don't for getting the action
done
oh possibly
like um i still think they're not
thinking about the future
i think it's more of a like
like i don't know if i had to take my
best guess at it
with like
noting the fact that i am just a random
person who is not an at all an expert in
these things because why would i be
and yes listeners noting that lucas has
just asked me this question uh because
it sounds interesting and not because i
am
at all qualified to answer it
it seems to me like the more likely
explanation is that like
there are just
always a gazillion things to do there's
always like
you know
20 bills to be picked off the sidewalk
uh but like their value is only twenty
dollars they're not like two billion
dollars
and like
everyone is just constantly being told
to pick up all the 20 bills
and as a result they like
are cons
they are in a perpetual state of like
having to say no to stuff
and doing only the stuff that seems like
uh most urgent and and like maybe also
important and so like most of our
institutions tend to be in them in a
very reactive mindset as a result not
because they don't care
but just because that's the thing that
they're incentivized to do is to like
respond to the urgent stuff
and so getting policy makers to care
about the future whether that even just
includes children and grandchildren not
the next
10 billion years
would be
sufficient in your view
it might be it seems plausible i mean i
don't know that that's the approach i
would take i just i think i'm more just
saying like
i'm not sure that you even need to
convince them to care about the future i
think
actually it's possible that like what's
needed is
like people who have the space to bother
thinking about it
um which like you know
i get paid to think about the future if
i didn't get paid to think about the
future i would not be here on this
podcast because i would not be smart i
would not have enough knowledge to
uh to be worth talking you talking to
um
and
you know i don't i think there are just
not very many people who can be paid to
think about the future
and like the vast majority of them are
in there maybe i don't know about the
vast majority but a lot of them are in
our community
and very few of them are in politics and
politics generally seems to anti-select
for
people who can think about the future
and i don't have a solution here
but that is the problem as i see it
and i would want the salut
if i were designing a solution i would
be trying to attack that problem
that would be one of the most important
things
uh yeah probably yeah i think on my view
yes
all right um
so
as we wrap up here is there anything
else you'd like to add or any parting
thoughts for
the audience
yeah um
i have been giving all these disclaimers
during the podcast too but i'm sure i
missed them in some places but like i
just want to know it lucas has asked me
a lot of questions
that are not things i usually think
about and i just gave off the cuff
answers
if you asked me them again like two
weeks from now
i think for many of them i might
actually just say something different so
don't take them too seriously
and treat like the alignment ones i
think you can take those reasonably
seriously but the things that were less
about that
you know take them as like
some guy's opinion man
some guy's opinion man
yeah
exactly
okay uh well uh thank you so much for
coming on the podcast rohan it's always
a a real pleasure to to speak with you
you're a
bastion of knowledge and
wisdom and ai alignment and uh yeah
thanks for all the work you do yeah
thanks so much for having me again this
is this was fun to record
[Music]

As in [previous](https://futureoflife.org/2019/04/11/an-overview-of-technical-ai-alignment-with-rohin-shah-part-1/) [years](https://futureoflife.org/2019/04/25/an-overview-of-technical-ai-alignment-with-rohin-shah-part-2/) ([AN #54](https://mailchi.mp/3e2f43012b07/an-54-boxing-a-finite-horizon-ai-system-to-keep-it-unambitious)), on this FLI podcast I talk about the state of the field. Relative to previous years, this podcast is a bit more introductory, and focuses a bit more on what I find interesting rather than what the field as a whole would consider interesting.