Rob Miles - Why should I care about AI safety?

welcome once again everyone to another
episode of the tours data science
podcast my name of course is jeremy and
i'm the host of the podcast i'm also on
the team
for the sharpest minds data science
mentorship program and i'm excited about
today's episode because we're talking to
rob miles who's
a really successful youtuber whose
channels focus on ai safety and ai
alignment research
now rob's actually been popularizing ai
safety work since
about 2014 which is long before most
people were interested in the space
long before alum 4 i knew about it
really and as an area of focus
so he's got a lot of interesting
perspectives on how the field has
evolved
how ai research has gone from just
purely focused on capabilities
making systems that are able to do more
and more things to now starting to worry
a little bit more about okay you know
how are we going to deploy these things
should we worry about where the stuff is
going what the limiting case is when the
technology gets really really powerful
what should we be thinking about in
terms of risks so rob's been doing a lot
of interesting thinking about these
issues he's also collaborated with a
whole bunch of ai alignment researchers
so we'll be talking about a whole bunch
of different topics including
communicating about this stuff
and more generally some of the problems
and opportunities that might be ahead
for us
as a civilization as this technology
gets better and better so i'm really
looking forward to getting into this one
with rob
and without further ado let's get the
ball rolling
hi rob thanks so much for joining me for
the podcast hi
thanks good to see you i've been
following your youtube channel for a
really long time i think it's a great
source of information
i think one of the interesting things
about what you've been doing is you've
been talking about ai
safety in some capacity or other since
about
2014 which is way before most people
it's way before i was
aware really of the the problem and one
thing i wanted to start with here was
how did you become aware of ai safety as
an issue and what made you dedicate so
much time to that problem
yeah so um it was around 2014 when i
first started
talking about this publicly i got
interested in it around
2010 2011
what actually had happened was i had a
kindle for the first time i had an
e-book reader
and i was very excited about the ability
to
to get through a lot of reading material
very easily because i could have every
book that i was reading
on my person at all times um
and at that time i had a rule as an
undergraduate
that if i ever read three things from
the same author each of which caused a
significant
update in my uh views
or my beliefs um or you know that was
engaging or interesting or
just generally excellent uh that i would
then try to read everything that that
author had ever written and this
triggered almost immediately on eliezer
yatkowski
and i don't know why i'd stumbled across
some things possibly on overcoming bias
or
um somewhere like that and so i started
i i downloaded an uh an ebook that um
that i think paul crowley put together
of
every one of velez's blog posts uh and i
put them on my kindle and i just
uh read them and it took a while because
that's
he's a prolific guy yeah um just just
going through all of those
uh it made it very clear to me that um
that this wasn't that this was an actual
thing
obviously it's something which i had
come across in various
sort of science fiction contexts and so
on um
but and that seems to be that seems to
be how it is
with these subjects for most people
they're familiar with them
but it takes a a particular
approach to show that these are
actually firstly importantly different
from the science fiction ideas and
secondly
that like this is computer science this
is
an actual open problem that we don't
currently have good solutions to
that something seeming like science
fiction is a poor indicator
yeah of its actual plausibility because
accurate predictions of the future
at almost any any point in the last
hundred years if you give people
accurate predictions of
50 years ahead they will sound
completely absurd
and if you give people wrong predictions
about 50 years ahead they will sound
equally absurd and so like the fact that
the thing sounds
kind of crazy is not evidence for
or against it actually being true and it
was the first time that i started to
really
just actually look at what i knew about
computer science from from my studies
what i knew
about artificial intelligence and just
actually follow the arguments through
and think well
how would this system behave you know
yeah um
if this were much more powerful what
would be the actual consequences
that made me realize yeah that this is
a real thing yeah i find it so
interesting that i think there are a lot
of people who've come in at it
from from that perspective i mean it's
almost like we need to be given
permission to reason from first
principles especially when the
conclusion is so
radically out of tune with our life
experience or our day-to-day
it kind of makes me think back to you
know what would einstein have thought
trying to explain to people like no i
really think like a nuclear weapon
that can destroy entire cities is not
just on the horizon it's like two years
away
um right you would have been entitled to
look at that and say well look at
everything in our life experience
everything in the experience of humanity
as a whole
tells us that this is just such a
complete anomaly an outlier that it
can't possibly be true it's sort of a
similar similar effect it's interesting
yeah
and and so it was helpful to to have
that approach
of like supposing this was false
how would the world look what would we
observe and then
supposing this was true how would the
world look and what would we observe
and that's the only that's the only way
that you can
make progress on these things you just
have to yeah look at
like like if you're
sorry i get frustrated think even
thinking about this because
i've kind of moved on a bit from this
whole like
thing because it's just natural now it's
part of how i think
yeah but i remember wrestling with this
when i was talking
with people about it earlier on that
there's no way to predict
what ai systems are going to do
by thinking about people
and what people are saying and what
people are thinking right now
like it could be that these people are
like
naive and hopeful about the future and
these people are cynical
and these people have lived through an
ai winter and these people haven't and
um whatever else and like none of that
is going to give you the answer because
this is a technical question
you have to look at the systems you have
to do you have to look at the
engineering
you have to actually as you say think
from first principles
because that's what's going to actually
affect
what we're just talking about how future
technologies are going to behave
and you can't do that without looking at
the technologies themselves
in detail and so what was the process if
you try to reconstruct it
what was the intellectual process that
you went through to conclude that
um you know yeah i'm pretty confident
obviously
nobody is really confident actually
that's one of i think the most beautiful
things about talking to people in the
space
no one's really approaching this from
the standpoint that i know with
certainty
that ai is going to be good or i know a
certainty that you know agi is going to
be terrible
there's a good dose of intellectual
humility here but um i think everybody's
come to certain conclusions about where
the probability density lies i'm just
wondering what were the
the biggest factors that shifted that
probability density for you over time
if i have to be honest most of my early
involvement
or my early interest was because these
are
extremely interesting problems
and the fact that they might be
extremely important
was always there and was always
available intellectually as a
justification
but the thing that drew me in viscerally
is that these
problems are fascinating just yeah they
cut
right to the core of what does it mean
to
be what does what is what even is
thinking what does it mean
to be an agent in the world what does it
mean to want things what does it mean to
have goals and values what
are ethics what
do we want actually from the world
where are we trying to steer the future
to um
and just the sheer interestingness of
the questions is what
uh drew me in and then
after a certain period of time of
thinking about the questions
you steadily update towards various
conclusions about them
kind of unavoidably as a consequence of
thinking about them
yes it's sort of hard to pin down the
exact uh i guess the exact set of
arguments that
i don't know i always find this
personally it's it's difficult for me to
think about
how to convey my own priors to somebody
else the justification for them
uh from from scratch just because i've
assimilated so much
in some ways this almost feels related
to the alignment problem because
so much of what we believe and think and
want is implicit
um yeah i guess it's just yeah maybe
it's just a complicated question with no
easy answer
i mean there's various different
approaches and uh models and things that
shifted me
um one of them was just the realization
that
really extreme dramatic things
can happen and have happened especially
and this is like a whole thing that i
don't know if we have time to go into
but
um thinking about the long arc of
history
and the optimization process is in that
in in like a deep history
that you have evolution operating for a
certain period of time
and then there's a switch that gets
flipped
somewhere where you have um
something that speeds up the rate
something like sexual sexual
reproduction
that speeds up the rate at which
evolution can happen
and that when you look back in history
what you have is these tremendously long
periods of one mode and then a paradigm
shift into a mode which is faster
and then you have you know you have like
intelligent animals that can operate and
then
they develop language and they develop
culture and then they develop
writing and these are all things where
you're either changing
the way that information is being sort
of modified or the way that information
is being stored
and each of these each of these periods
is is dramatically shorter than the
previous period yeah and that this is
something that's happened several times
and so it seems pretty clear to me that
in the same way
that um sexual reproduction allows for
sexual selection
which means that the rate of
reproduction doesn't just depend on
sort of the environment as a whole
deciding
deciding which animals live live and die
which ones succeed and which ones fail
now the intelligence of the animals
themselves is applied
to picking mates based on so you don't
have to
actually win in a fight you just have to
observably be strong yeah and that that
gives you more signal effectively gives
you better gradients to train on in a
way
um and that just speeds up the whole
process and so on and so it became clear
to me that
the point where ai research is being
done by ai systems
is another one of those things where you
close the loop you're taking something
that works
dramatically faster and
feeding that in to its own improvement
cycle and i don't necessarily mean like
pure
self-improvement like the ai is gonna be
this unbelievably powerful thing that
will take apart its own source code and
restructure it or whatever i just mean
that
having ai systems help the humans and to
play a larger and larger role is the
kind of thing that
it looks like the same class of thing
where you just
shift into a new paradigm of
dramatically faster development
um and it was those kinds of arguments
that made me think like
our usual sense where we think oh you
know 50 years away 100 years away
we we're just kind of getting some vague
sense of like
how hard does this seem yeah and then
turning it into a number by some general
like just a vague mapping
whereas like if you actually think about
it
the the number of seconds per second is
going up
as this whole process happens the number
of cycle times or
yeah right right like if you make if if
this is going to take 50 years
of time to develop that it doesn't
really take 50 years of
clock time it takes 50 years of
subjective time and when you have ai
systems doing the research subjective
time is getting faster all the time so
that was the kind of thing that made me
think hey this could be in my lifetime
yeah this could be something that we're
close enough to that it's worth working
on now
well this very much makes me think of
you had an interesting video where you
talked about a
popular sort of counter argument to the
ai risk um
argument where people will say well look
uh you know our company is really just a
kind of agi already don't we have
you know this kind of self-improvement
quality i think the way you've laid it
out there
just makes it so clear what the
qualitative difference is
we're you know you're talking about a
different regime of
computation really in in the in the
history of the universe and the
computational history of the universe
so it really does seem like a lot of our
metaphors
and the things we're used to drawing our
intuition from just become
these very very kind of frail things the
moment we start to look at these massive
qualitative transitions absolutely and
it's
and i think that um you're right that
most people
in the field uh have a lot of humility
about this because we have lots and lots
of arguments
people people from the outside often
view what we're talking about and say
this stuff is very difficult to predict
and you seem to be fairly confident
in this particular outcome whereas it
could be anything and there's various
reasons why
there's various responses to that but i
think actually
most of the time what uh what these
arguments do
is they make you spread your
distributions out they make you spread
your probability distributions out and
so
yeah there's all kinds of there's all
kinds of of problems and difficulties
and we really are very uncertain but it
means
it works in both directions right yep it
might take ten times longer but it might
take 110 for time
yeah and so it's worth paying attention
to
well there's one last theme i do want to
touch on before we leave this idea of
kind of
that big that deep time picture that you
just introduced
just as a as a curiosity what do you
think it is that's being optimized for
in that context because you know we have
this notion of a universe that's full of
particles and
those particles randomly combine you
know sometimes you hear the idea of like
like multiplying information or
propagating information forward through
time
that doesn't seem to intrinsically quite
hit the nail on the head just because
that
information changes considerably uh from
you know
the clump of atoms early on to the first
cell to the first sexually reproducing
organism and so on
um i don't know if you have a thought on
this but but do you have a sense of
maybe what that process is optimizing
for
you mean like what is the whole of what
is the universe optimizing for what is
physics optimizing for
um i don't think it is
um i think i don't know some people
some people think that there's some kind
of uh free energy
minimization type thing that accounts
for a lot of this stuff i don't know
i think evolution can be meaningfully
thought of as an optimization process
or rather as a large collection of
optimization processes
because if you think about like if you
take evolution as a whole as an
optimization process then it doesn't
really make sense because you've got
you know the the predator chases the
prey and the prey runs away so
is evolution optimizing for it being
caught off or escaping it's like well
the predator species has
its evolution and the prey species has
its evolution
um but and of course all of these
boundaries are fuzzy
because a species is kind of a structure
that we've overlaid on this thing
nonetheless it is optimizing for
producing good replicators is the is the
way that i would think of it
so well and that's that was where i was
hoping we'd end up going because the the
replicator
picture to some degree i mean when i
start thinking about
about ais and agis that presumably
propagate forward through time the idea
of replication
seems to become decoupled from uh
self-improvement or from
from continued existence through time it
almost seems like there's another
quality that
um a deeper quality that that you know
the the continuity of some kind of
causal
structure or computing structure i don't
really know but it's also speculative
anyway
i think the continuity the the only kind
of continuity that you could really
strongly expect
is a goal continuity because
if i create if i'm some kind of ai
system and i'm going to create
other ai systems to go out into the
world and do things
the only thing that's really important
to me is that they share my
goals or that they or that they in
practice will advance my goals
but um realistically usually the best
way to do that is to have them share
your goals
um that's the that's the type of
continuity i would expect
interesting okay i don't want to linger
too much on that i just i thought it was
an interesting thing to unpack a little
bit
um one thing i do want to talk about
it's something that i think you've done
really effectively
through your youtube channel is you've
actually taken the time to address a lot
of the arguments
against ai safety or against worrying
about ai risk i think that's something
people don't tend to do maybe as much as
they should because there are still a
lot of people who wonder you know
why should i be worried about this is
this really just like a terminator
scenario
is it really something that people are
just freaking out about for no reason
i know you're personally concerned about
the risk of agi but
which anti-agi arguments do you find
most compelling
hmm yeah so you're right i do have i do
place a lot of emphasis on
that uh internally
because i think it's it's such an easy
and obvious failure mode of thinking to
just start only listening to people who
already agree with you um
i think it's really important to seek
out
smart people who disagree with you and
try to really listen to what they say
and to try to
try to really understand what they mean
and come up with strong versions of
their arguments
and really stress test your stuff
because i don't
want us to have a giant problem with ai
in the future
right like i'm kind of looking for
reasons why
this isn't as big of a problem um and i
would
like to be convinced but at the same
time obviously
i've been out in public talking about
this being a problem for a long time
so i have to be very aware of my own
psychology because there's going to be a
part of me that's going to want to
that's going to sort of my personal
identity is bound up with it
and so that's going to be a strong force
that's pushing me to not take these
things seriously and so all of these are
reasons why
it's important to like deliberately and
consciously
actually make an effort to engage with
people who disagree with you
this is i mean everybody knows this uh
like well it's
it's of course easy it's easy to say and
it is a cliche but so few people
actually
do it that i think that that dividing
line is is still worth flagging yeah
yeah that's that's true the earlier
stuff on the channel especially was
closely based on
ideas from yukowski and ideas from
bostrom which
revolve around this framework where you
have
uh you have a take-off scenario you have
a single agent a sovereign agent
which is able to act in the world and it
uh you're developing it until it hits a
certain level
at which point it starts to increase its
capabilities exponentially
by acquiring additional computing
resources improving its source code and
so on
until it has a decisive strategic
advantage at which point
uh whatever the objectives are of that
system whatever the utility function of
that system is
you're going to end up with a world
that's somewhere very highly rated by
that utility function which is probably
apocalyptic and that view
[Music]
is uh i think that my view of the
situation is more complicated now
um because
there's a like i now place more
probability
on a more multi-polar situation
where you have if if the development
process happens slowly enough
then you actually have a lot of
different ai systems
in the world with different levels of
capability and so then the question is
not like
what happens if you have one super
intelligence in a world that is
otherwise
more or less like our world it's like
by the time you have a system that's
able to take off
what does the rest of the world look
like yeah
and uh that consideration has shifted my
perspective a bit because it
it's not purely a technical question uh
it's a much broader question
of um economics and politics
and you have to do a lot more looking at
the world you can't quite do it all with
greek
symbols on a whiteboard in the same way
and so that has i still play significant
probability on
uh some ai system in a research lab
somewhere
just exploding because somebody has had
a brilliant insight
there there are plenty of situations
where one team just is ahead of everyone
else
by far enough that the gap doesn't
matter um
like that that nobody else could catch
up um
but there are also situations where
these things develop in lots of places
at lots of times
and that whole situation is so much more
complicated and harder to think about
that it has oh what do you know it
spread all my probability distributions
wider
so i'm just less certain about those
things than i used to and that is
i the thing is that the fundamental
problem of
uh not being able to specify what we
want
and having systems that are going after
goals which aren't what we want
is not solved by having lots of systems
yeah but it is complicated by it i was
going to say
it almost feels as though it's a strict
exacerbation of the problem to the
degree that
or to the extent that it just you know
creates a situation where now
even if you could solve the inner and
outer alignment problems and
everything's good technically
you then need to enforce those um uh
those you need to force companies and
labs to implement those things which
seems to just
add a policy layer on top of everything
else yeah yeah because if you have a
singleton
situation where you just have this one
system that that explodes then if
at least if you get that one right then
you're okay
yeah because it has power over
everything else and nobody else
like you're not going to be able to
launch your own unaligned agi
project in a world that has
a singleton agi aligned agi already in
existence
so that lets you get around that problem
but yeah it's a problem
yeah well so given the the underlying
pessimism that i think a lot of people
might detect in the air here
um in fairness i don't think that i
don't think anyone places obviously 100
probability on the disastrous outcome
there's a a
wide range of disagreement in the
community you have some researchers
saying you know what i think
mostly i'm almost positive the outcome
is going to be good
other people somewhere in the middle
there's been a lot of polling and
you highlighted in one of your videos i
think that cumulatively people who have
a negative outlook on the stuff people
who think that
agi on the whole will be negative for
humanity it's something like 15 percent
of
whatever community was pulled and so i'm
going to put a big asterisk there
um you know this is talking about the
grace grace 2016 that was um
people who published papers in icml and
eurips right so pretty high quality
researchers but nonetheless not people
focused on alignment which in fairness
being focused on alignment means you're
worried about safety so it's tough to
get a good pull here
but so basically baseline 15 in that
poll maybe that's shifted
but why do you think that something like
80 to 85 percent of people
might view things more optimistically
like
what do you think from your model of the
world where does that come from
yeah so i'm pretty reluctant
to bulvarism i think it's called
bulfarism
where you assume that people are wrong
and then try to psychoanalyze them to
figure out why they might reach that
conclusion it does feel like overfitting
right right um at the beginning of the
of
the interview i was saying you can't
reach conclusions about the world by
psychoanalyzing people
um but
i mean i don't know i don't know that i
have any particular insight here
i think people want to believe that what
they're doing is helping
and i think it's also the same kind of
selection effect as um
as working on alignment working on ai at
all a lot of people are working on ai
because it seems like an important
technology
that and like broadly speaking
technology makes things better
right like i believe that um and i know
a lot of people don't
i think broadly speaking technology
makes humans more powerful
it makes humans more able to do things
and so if humans are on the whole a good
thing
which we are sort of
by definition in my opinion like
according to human values humans are
pretty good
yeah then something that allows us to
get what we want
is on the whole a positive thing because
people having their preferences
satisfied is like
a decent definition of what good is
so technology is good and ai is a form
of technology
and so it's probably good it's like a
reasonable prior to have
is uh i actually think 15 percent is
kind of high i don't know how this looks
for other fields
do you want to how many automotive
engineers think the cars are on the
whole bad for the world
yeah i don't know probably less than 15
though was this now i'm trying to
remember the exact framing of the poll
but was the poll about
uh whether ai itself is good for the
world in its current form
or the risks of future agr the
implications of taking this technology
to the limit
i think it was about future impacts of
agi but
i think everybody maybe it's not true
maybe some people are just making their
little narrow ai systems and
not thinking about where this is going
but i think everybody has a sense of
like
all ai research is is broadly in this
direction
right right and aft after every ai
winter people pretend that it's not
but there's no pretending that this
isn't what
what we're trying to do yeah i guess
it's
um the implications i'm just thinking
you know if i if i were working in um in
biotech
or whatever would be called biotech in
you know the 1930s
or 1940s you know penicillin comes out
and
now all of a sudden it's just wonder
unambiguously wonderful thing you're
curing polio you're doing things with
smallpox
and then and then you turn around it's
2020 and now people are starting to go
tabletop bioweapons are not that far out
of
out of uh the realm of what's possible
as technology makes things cheaper
it seems like there might be i don't
know it's again this kind of
new regime of technological development
where the world has so many degrees of
freedom
and every time you take a technological
step forward
you're opening a whole big subspace that
wasn't previously accessible
and because there are far more
configurations of the world that are
very bad for humans than there are
configurations that are good
those bigger steps tend to be a little
riskier i don't know if that's a
fair assessment yeah yeah that's um
that fits with my models i think cool
well we're
we're two optimists here um great well
i i do want to ask a question then more
on the the technical alignment side
because that's something you focused a
lot of your content on
i've learned a lot from your channel on
that topic actually um
there are yeah there are a couple of
different schools of thought
it's almost hard to classify and do the
taxonomy of these schools of thought but
i'm gonna take a shot here and i want
you to feel free to shoot me down on
this
okay um my sense is that there's one
group of people that approaches the
alignment problem with a kind of
uh what i might for want of a better
term called an engineering mindset and
the philosophy here is something like
you know our best shot at making ai safe
is to align it more or less as we build
it
so build it a little bit notice oh the
tower's wobbly so i'm just going to fix
it as i go
and then the second is maybe more
perfectionistic
um a little bit more philosophically
mathematical
taking the view that you know we only
get one shot at doing this right
because you know maybe because
self-improving systems rapidly reach
takeoff velocity and get away from us or
whatever reason
um i guess maybe i'll stop there do you
agree with that framing or am i am i
missing something there
i think there's kind of um okay so
here's a metaphor
suppose we're building aircraft we want
to build an aircraft that's going to
take us
to a particular uh city that's distant
from us
that's our goal and we want to do this
safely and let's say that like
it's an island and really all that
matters is correctly aiming
at that island so that when we land
there we land there and not in the ocean
yeah um so
you've sort of characterized two
different approaches one as being like
let's build this thing perfectly so that
it's aimed exactly at the island
and then press the button and another
one which is like
we'll uh we'll wing it we'll we'll uh
we'll do it we'll play it by ear as we
go
um and in practice
the best way to do it is a combination
by which i mean you do a lot of very
very careful engineering
but you do still expect somebody in the
airplane to be adjusting it as you go
but the precise mechanisms by which you
are getting feedback about which
direction you're headed
and uh controlling and adjusting your
trajectory
these are all very carefully planned and
engineered things
but you don't try and do all of your
planning up front
you plan in exactly the ways in which
you're going to adjust so
an approach whereas whereas the extreme
other approach would be
like we're going to take off in our
aircraft and then see if we can't design
some rudders
and yeah ailerons and control systems
and right not
on the way right and that's i think
either extreme is
is foolish and then there's a question
of um
how you're going to balance these these
concerns but i think
nobody is nobody is actually at either
of those extremes the people who are the
most mathematically minded are just
saying look we need to really understand
and have strong assurances that when you
turn left on the thing the plane is
actually going to go left
um and other people are saying
uh other people are other people are
focused
it's it's more like are you focused more
on
the like mathematics of aerodynamics and
all of that stuff
or are you interested in the details of
like avionics
but they're both engineering approaches
towards getting a system that's
controllable and
that does what you want it to do and
what would be some of the
the more recent um innovations on that
front like are there
are there new approaches to alignment
that you've become aware of in the last
say
two years that you think are worth
highlighting especially for people who
are just trying to orient themselves in
the space
i have a really hard time keeping track
of
uh time so i have no idea what came out
in the last two years
sometimes i think things have been
around forever and they're actually only
a year or two old
that's the time compression effect you
were talking about earlier yeah
absolutely
um the speed with which things can
become
just part of the landscape um because
it's such a young field
um yeah but so one of the biggest things
that came out
one or the most recent thing that really
shifted my perspective
is uh the work on mesa optimizers and
that is like a whole
other class of problem that i didn't
even realize we could have
um and i'm currently working on a video
about it actually
uh so maybe i shouldn't say too much
people watch that video when it
eventually comes out but
the idea that you can have that even if
you've
perfectly specified the correct
objective to your training process
the model that comes out of that
training process might be misaligned
with that objective and that um
the so-called inner alignment problem um
is just a whole
other class of issue that um
honestly makes me kind of pessimistic
because it was so recent that people
have been thinking about this
for a long time uh and
there people are sort of vaguely hinted
at it that this type of thing might be a
problem but
um that this is the first time that it's
been properly laid out and
given a full treatment and we realized
that it is as big a problem as it seems
to be
um is very unsettling to me
because like what else do we not know
that we don't know right
right i guess the um so the
my understanding at least of to some
degree paul christiano's philosophy at
opening eye on this is
is something like you know we hope to
get to a point where we're leveraging
um advanced ai systems to help us with
uh with alignment to help us discover
some of these unsolved problems
i have no idea whether the threshold of
world overtaking ai system uh falls
ahead of or behind the threshold of we
have an ai that can help us align ais
that itself sounds like an interesting
problem but yeah the misa optimization
thing
at least as i've come to understand it
for people who might not be familiar
with it
quite so much is just yeah this idea
that you have like
within a an overall optimizer like a
deep neural net you might end up with
substructures that are intent on like
retaining their their structure so you
can almost think of it in a way as kind
of like a cancer within the human body
um you know to the extent the human body
is some sort of optimizer some agent
the cancer itself kind of goes oh i have
my own interests that are separate from
the whole and and now i'm going to
kind of optimize my way through some
pathological strategy to taking over
doing some damage to the
process well that's one way if that's
one way of thinking about
it and it's one sort of modality
but it could be the entire it doesn't
need to be a substructure of the agent
it could be the entire agent
um and the analogy there is with
uh evolution again if you think about
evolution
um evolution if you model it as an
optimization process
it has an objective which is to maximize
inclusive
fitness or whatever you know maximize
the number of surviving offspring you
have
um and yet
when that optimization process produces
agents
the goals of those agents are not to
maximize their own reproductive fitness
right people don't and like animals
don't want to make
a lot of copies of their dna they don't
even know what dna is
yeah um they want this sort of uh
this unpredictably derived
set of goals which are a function
of the original goal certainly but also
little contingent things of the training
environment and details of different
strategies that help them to succeed in
the ancestral environment and so on
and that that they don't care
right like human beings even when we
understand
what the goals were of the optimization
process that created us
that's not persuasive to us yeah right
and we will continue to like use
contraception
and whatever else you know we we go for
we're taking this we because we don't
have a goal that's optimized our
reproductive fitness we and so our goals
include things like eat food that's
tasty which like was helpful
but nowadays is actually not necessarily
the best thing to do
um because we have because the you know
the environment is different
so yeah so that's the other way of
thinking about it that you have the
capacity to
end up with a trained network that has
goals that are like
unpredictably different from the goals
that you actually specified
and what's more um
it then has all of the same convergent
instrumental sub goals that you would
expect
from any misaligned system so it's going
to want to
conceal the fact that its goals are
different and manipulate the training
process
and so on which makes him especially
difficult to deal with
because it's not just it's not just that
the thing might be misaligned
internally is that it might be
misaligned in such a way that it
is actively trying to hide that
misaligned because it wants to be
deployed
this all kind of seems related actually
again to that that time horizon
picture that you laid out early on um
just in the sense that
you know when you think about the time
horizons that evolution
acts on the the uh the feedback that we
get through evolution happens on the
order of
uh generation you know 20 25 years
something like that every time we
reproduce
whereas the feedback we get from the
real world our own
subjective clock time is way faster than
that we
we get sort of feedback on a really
tight loop from our environment
which allows us to like we have so much
extra compute capacity
above and beyond what would be needed to
just you know hit that
that main goal of reproducing that it
ends up getting deployed in some
really random direction i mean it's
untethered it sounds constrained by its
environment to some meaningful degree
which allows us to kind of diverge
considerably from
that uh that evolutionary objective
yeah yeah this is part of what makes it
kind of an unfair competition
that because we operate so much faster
than evolution we
are able to um
we're able to get away from it you know
we're able to do things that
if it were more powerful compared to us
it would uh really well it would stop us
from doing
and it may get right right if
it's evolution hasn't stopped happening
it's just continuing to happen at the
same slow pace that it always has
and everything else has sped up to the
point where its actions are
mostly irrelevant most of the time um
but it's still happening and it's the
same kind of thing that you would expect
um when you have a misaligned mesa
optimizer
that mesa optimizer might be quite
powerful and able to think
in a in a tighter loop than something
like gradient descent and it's training
it
which would allow it to this is the
other thing is that that in principle
would allow it to outperform
better aligned optimizers so
and then that's really annoying because
then gradient descent is going to
actively try and select the misaligned
mesa optimizers
because they're doing a better job
because they're the only ones who
realize
that they're in a training system with
this particular objective that they're
now actively trying to
optimize as an instrumental goal towards
getting themselves
deployed in the real world um
and it's it's the same kind of thing
there's there's
the further analogy there that ai
systems in general operating much faster
than we do
uh means that we may end up uh
not powerless but just like not able to
make changes fast enough
to continue to have control over the way
things end up
right yeah it it just sounds like such a
generally
hopefully not intractable problem i mean
i i suspect that given a couple of
centuries of time we'd be able to make
meaningful progress towards this
um we have question mark number of years
or yeah hopefully decades actually maybe
to wrap things up because i know you got
to
get going but do you have any thoughts
about you know what somebody was
concerned about this topic or just
generally
intellectually curious about ai
alignments the technical details
of all the stuff how might they start uh
getting involved start figuring out
where the open problems are and
what should they read there's a lot of
different things so
there are some really good books um i
used to recommend super intelligence
uh and actually i would say that it's
probably not that it's probably no
longer the best first book to read
and i would actually read something like
um
human compatible stuart russell's book
which came out relatively recently and
is a good solid introduction to the area
the other thing is uh and i'm going to
recommend a book that i haven't
finished reading yet because it came out
like the middle of last week i think
which is the alignment problem by brian
christian
um so far it's very good um
and that goes goes over again it seems
like a good
sort of uh introduction to the field for
the general public to get a feel for
what the different
problems are if you're already um
a bit more involved in sort of machine
learning and ai stuff
um i actually would recommend
uh reading the alignment pod uh the
alignment newsletter rather
so i'm just gonna say this because i
don't think robert's gonna actually plug
this but i i
desperately want him to so so rob's been
doing a podcast version of the alignment
newsletter
which is just really great so just to
toss that out there you can check that
out as well but
yeah so the newsletter is really really
good um in that it's a weekly newsletter
which summarizes
um research that's happened in a week
so i i think it's really good if you are
already a researcher yeah um the the the
problem that i often have when talking
to
um people who are actually know a lot
about ai and a lot about machine
learning
is that they don't realize the extent to
which this is
like a real active field that people are
publishing a lot of papers
in that it's it's a growing field and um
but it's still tiny right it's growing
but it's tiny
but that these are real problems that
you can do computer science on
and i think the newsletter is really
good at giving you a feel for the kinds
of
uh papers that people are writing and
the kinds of problems that people are
making
incremental progress on yeah and
actually so uh
people who've been listening to the
podcast probably the episode i think
before this one will be with rohin shah
who actually
puts out the alignment uh newsletter so
these might be a good
kind of back-to-back series to to watch
or listen to um
yeah well yeah rob thanks so much no i
was just gonna say like
i don't um it's a weird thing
i actually don't feel weird about
plugging
the podcast because i take no credit for
it i mean literally i just i
i just read out the newsletter but if
you if you
i mean if you're listening to this
podcast you're probably a person who
likes to listen to podcasts
um a person who likes to take in
information through their ears
and uh i like to think that i do a
better job than text-to-speech software
can
there you go the technical terminology
and stuff you know what
and let's hope that's always the case uh
i would do it anyway
but there you go that's that's the
spirit yeah very human um
thanks so much rob really appreciate it
i will be leaving links to all those
books actually
and of course your youtube channel uh in
the blog post that will accompany this
podcast so if anybody wants to check out
uh rob's channel highly recommend it and
i really appreciate your time on this
one
thanks so much for having me on the show