Evan Hubinger - The Inner Alignment Problem

hey everybody jeremy here welcome back
to the towards data science podcast and
today's episode
is all about control and specifically
control over advanced ai systems
that someday as the technology improves
might actually become
super intelligent now one of the big
open questions that ai safety
researchers are working on today has to
do with predicting
what advanced ai systems might look like
in the far future
and figuring out ways to ensure that
they don't do things that are deeply
misaligned with human values
things that in some cases might actually
be downright catastrophic
now this is a complex and multi-faceted
issue that researchers call the
alignment problem and my guest today is
a seasoned veteran of the alignment
problem he's worked on it at
organizations like
miri the machine intelligence research
institute and openai
and his name is evan hubinger now evan's
done a lot of fascinating research on
the alignment problem which is included
along with some colleagues discovering a
whole new type of alignment failure
called inner alignment failure
so we'll be discussing that we'll also
be talking about some analogies that'll
hopefully make it more intuitive to
understand
and we'll also be discussing the future
of ai and ai safety more generally so i
hope you enjoyed the conversation as
much as i did with that
i'll step out of the way and let it
begin
hi evan thanks so much for joining me
for the podcast yeah thank you so much
for having me
i'm really happy to have you here you've
done so much interesting research in the
direction of
what's now called inner alignment sort
of this special mode
of ai safety that we'll be exploring
quite a bit today
but before we get to that i'd love to
get a sense from you
your your twitter twitter bio in
particular is kind of interesting it
says you're an agi safety researcher
and i think each of those terms deserves
a parsing um but
i'd love to understand from you like
what what do you see as
agi safety research and what brought you
to this field in the first place
yeah that's a great question so perhaps
i need to change this or something i'm
not a huge fan
of the term agi um i think that
it certainly points at a sort of cluster
of ideas um it points at this idea of
well
you know we believe that in the future
we're going to build
more advanced ai systems and there's
this concept that at some point we're
going to build systems that are
general that are able to sort of do
everything that a human can do
they can really compete with humans they
can replace most humans in various
different jobs
sort of um you know really transform
society
i think that this sort of very general
framing just sort of
well it's going to be better than humans
across all tasks
is not necessarily the right training
because
as we sort of see in machine learning
now there are many instances where
you know machine learning systems can be
very transformative and can do you know
all sorts of things
that really change society without being
better than humans across the board you
know they can be better
much better humans on some tasks um
and so i i guess personally i prefer the
sort of framing that is more like
transformative ai systems or advanced ai
systems
where really what we're trying to do is
um
in the future we're going to build ai
systems that are probably going to have
a really large impact on society
and we want to make sure that impact is
positive and that's the sort of safety
part so we want to make sure that those
future advanced powerful ai systems are
doing the right thing
and what what brought you to this field
of research because it is such a it's
such a niche area it is an important
area and it's pretty clear i think
to people who do ai safety research
obviously that this is like an
incredibly important area
but i think a lot of people from the
outside might wonder you know why this
why not
focus on you know ai for social good why
not focus on
i don't know protecting us from
biological weapon there's so many
different things that
that you could choose to work on why
this in particular
yeah that's a great question so i got
involved in ai safety
through effective altruism effective
altruism is sort of
a movement and a philosophy that is
here around the idea of well we should
try to
as individuals structure our careers and
our lives to try to have
as high an impact as high a positive
impact on society as possible
and so i spent you know some amount of
time uh
in college trying to really think about
okay what can i do
with my career with my life to try to
make a difference in the world
um i encountered a lot of this sort of
um argument at uh and sort of
explanation of how
ai could pose a sort of existential risk
to humanity
uh truly a situation in which the
entire long-term future of humanity
could be curtailed because we build ai
systems that just
wipe us out and i was pretty persuaded
by this argument there's a lot of places
where you can find this argument
um i think the sort of most canonical
historical source would probably be nick
foster and super intelligence
um you know more recently you'll find
uh stuart russell's who's incompatible
um there's also historically elias
yatkowski's written a lot about this
so there's a lot of people that sort of
historically have talked about this
argument
um but the central argument is is sort
of not very complicated
um the argument is well uh when we build
ai systems
we they sort of have some goal that they
end up pursuing
and if that goal doesn't include all of
the things that we care about
um you know sort of classic example is
we build an ai system and it's just
trying to maximize for a total number of
paper clips or whatever there's this fun
cookie clicker light game that was made
on this idea where
you know you just what what is the thing
you do to actually maximize paper clips
well
you know those pesky humans that are all
around getting in the way
um are a problem if you just want to
make as many pinkbooks as possible hey
everyone jeremy here i just wanted to
provide a little more context on the
paperclip maximizer concept that evan
just brought up since i suspect it might
be new to some
people the paperclip maximizer is an
example that swedish philosopher nick
bostrom likes to use to demonstrate how
hard it is to build super intelligent
systems that are actually
safe now bostrom has us imagine that one
day people at some paper clip factory
build the super intelligent ai
and ask it to find ways to maximize
their pay-per-clip output so they can
all become fabulously wealthy on
paperclip money
now they think this is a pretty simple
and safe thing to ask their ai to do
and most people also would too that's
kind of the point
but the ai is super intelligent and
quickly realizes that as long as there
are humans around
those humans might at some point try to
turn it off or compete with it for
access to the iron it needs to pump out
more paper clips
as a result it decides to wipe out the
entire human species in order to protect
its ability to create the greatest
number of paper clips possible
now it's a toy example and no one's
saying this is literally what an ai
might do
but it illustrates just how careful we
have to be when we specify the
objectives we want
highly capable ai systems to accomplish
like aladdin's genie the moral of the
story here is that when it comes to
powerful ai
you have to be very careful what you
wish for what is the thing you do to
actually maximize paper clips well
you know those pesky humans that are all
around getting in the way
um are a problem if you just want to
make as many people as possible
this is a silly example but the point is
that there's something going on here
where
when i try to optimize for some
particular goal you know make paperclips
or whatever
um if that goal is just sort of kind of
random and just like something that i
happen to want
in one instance but don't necessarily
want in all instances if i just optimize
that goal everywhere i end up with this
sort of
instrumental sub goals of you know now
i'm going to go off and
get rid of the humans and do a bunch of
other random stuff
this is so very basic and i was very
persuaded by this argument
um i thought that you know he really
seemed like ai
had the potential to be really dangerous
um and we've seen the extent to which it
can be transformative and can be really
powerful
you know especially recently with you
know ai
making all these you know milestones
like uh first go
starcraft dota now you know language
models like gt3 really sort of doing
really impressive stuff
and so um i think it's concerning and i
think there's like a real chance
that uh ai could be
an existential threat to humanity and
that this is worth engaging with as
an outreach it's really it was really
interesting that you tied this into
capabilities almost
instinctively like capabilities become
part of this conversation
so quickly in the context of misaligned
ais like
and correct me if if i'm
misunderstanding but essentially the the
concern that you're articulating is like
if we actually had the ability to
optimize the universe around a
particular number
say number of paper clips or anything
else the vast majority of
the ways that you could do that the vast
majority of the universe gets like
rearranged to optimize for paperclip
production look
absolutely awful for human beings or for
the things we generally care about
and so our capability to actually like
make that happen
is itself a risk as as long as it um
paces
our ability to specify what we want
yeah i think that's a pretty reasonable
way to think about it i think one thing
which i'll note
and this sort of goes to the heart of
you know i think a lot of interline and
stuff and a lot of the sort of way in
which i
engage with ai safety and sort of my
research but i think that
this is a really simple basic
observation and i think the sort of
observation that you know if you
optimize for some simple goal
you're going to end up having all these
instrumental sub-goals that are really
bad
um it's a really simple observation i
think it's basically a correct
observation
but it doesn't tell us that much we have
to do a lot of additional work to really
try to figure out okay
in in what circumstances real world
circumstances are we going to build
machine learning systems that actually
encounter this failure
um and that question is really difficult
um and requires a lot of careful
analysis and thinking about like how do
actual machine learning systems work
um and you know what would it be like if
we took current machine learning systems
and scaled them up would we encounter
this failure mode how would
how would we encounter this failure um
and i think this is sort of a question
which is very worth
you know engaging with on its own terms
um and it's something that
i don't know something i've talked a lot
about in my my research
do you think it's possible that we're
already encountering that in
maybe subtle ways that aren't super
apparent i'm just thinking here like
this this idea of viewing failures of ai
systems more broadly as failures of
alignment like that seems like a lens
that you can
choose to view a lot of things through
and
depending on how you do maybe there are
current systems that fit that
description
i absolutely agree with that so one
example that i think is sort of
um a good one here just for sort of
pointing out sort of
a possible alignment failure mode um
if we think about uh recommender systems
so something like the sort of youtube
recommendation algorithm um there's sort
of an interesting thing here where what
we
want the recommendation algorithm to do
is at least naively what we would hope
it's doing is that it is sort of
myopically selecting
the video which individually you sort of
will like best at that point
but there's something sort of scary that
it could do which is it could change
you to cause you to like different
videos
and by changing you to cause you to like
different videos then maybe
it becomes easier to give you videos in
the future and in some sense maybe
you know this is okay in some
circumstances it's like you know
maybe it shows you some new type of
thing you hadn't encountered before you
know you really like that thing
but it's also a little bit scary because
we're directly incentivizing systems
to modify humans we're creating systems
which have an incentive
to show things to people which cause
those people
to behave differently i mean sometimes
this can be really dangerous
so you know there's there's a sort of
classic example there's a lot of debate
about the extent to which the youtube
recommendation algorithm actually does
this in particular
um but there is a sort of a potential
problem at least
where the youtube recommendation
algorithm could sort of show you videos
that sort of radicalize you for example
sort of turn you you know push you into
the alt-right or something
where um once it's done that to you it's
easier for it to show you
other videos which you're going to like
because it just sort of picks them from
this
this sort of cluster and now you're
really engaged and you really want to
watch all of these sort of
you know conspiracy videos or whatever
and so there's something happening here
where uh because we created these
systems which have a sort of
incentive to modify people to cause them
to sort of behave differently
um we sort of no longer you know
necessarily have
have full control over what what what
people are doing we created systems
which are which are directly modifying
us into
you know people which are which are
different than just you know the people
who would have been beforehand
i think this is a pretty dangerous sort
of you know general category of things
and we can see how how it sort of has
problems
and if you think about this in the
alignment framework but what's happening
here um from the perspective of you know
trying to build systems which are
actually trying to
do the right thing to the thing that we
want well there was a there was a
problem where
what we wanted was he just wanted to
show you the video which you most wanted
at that
you know moment just the video which you
most liked but instead of doing that you
know maybe what the algorithm is
actually doing is it shows you a video
which
you might like less for example but
which will radicalize you
such that it can show you it has an
easier time showing you videos in the
future
instead of doing this implicit
optimization over many
time steps um i think this is a little
bit tricky one of the things that
um is a little bit difficult when we
create these sorts of systems is how do
we control this sort of
uh what time pricing they're optimizing
over there's a sort of implicit
i think assumption a lot of times you
know when you train like an rl system or
something
that has like a discrete episode
boundary the system is just going to
stop optimizing at the end of the
episode boundary
but in fact we often have these sorts of
uh optimization
algorithms which work on top of that so
we do things like population based
training
um and even just uh i don't know you
don't have to i'm not going to sort of
go into too much you know what
population based training is
but basically there's just lots of
algorithms that we can do on top of
you know the very simple algorithm um
that like look at the whole history and
be like you know if
over the whole history of this algorithm
was it successfully
doing something good and if you're
looking at whether the algorithm is
doing something good over a large period
of history then you're now implicitly
incentivizing the algorithm
to do things like uh trick that you know
modify the human
such that they you know are better in
the future
yeah it's really interesting how quickly
this rabbit hole develops
and how how not obvious it is as well
like from the outside you might just
think oh well you know i come up with a
reasonable metric and then i get the
system to optimize it but
as you start optimizing things i guess
it's goodheart's law like any time you
specify an objective it ceases to become
a good metric for the thing you're
trying to
try to optimize these sound to me like
examples of
outer alignment if i'm getting that
right essentially
cases where we give a system a number we
want it to optimize
we kind of let it go and because of a
problem with that metric we didn't
realize later it becomes obvious that oh
actually that metric is bad it's bad
because
it encodes certain incentives incentives
to change the preferences of human
beings to radicalize people or
or other things like that um you've also
done a lot of work on a different
way that that ai systems can be
misaligned uh called inner alignment and
i'd i'd love for you to just
explore kind of lay out what inner
alignment is
yes so i we have to talk about that so
okay so this sort of goes back to what i
was talking about earlier about really
engaging with how does machine learning
work
for a second uh you know we have these
sorts of high level arguments about like
you know what's gonna happen you know
maybe
if we have some system which is
optimizing paper clips or whatever it's
going to have all these instrumental sub
goals things are going to be bad
this is a really high level argument and
so what i want to do for talking about
interlinement is try to go really low
level
and just think what what is it that
machine learning actually does
so machine learning fundamentally uh we
have some
parameter space uh and we search over
that parameter space
to try to find some set of parameters
which perform well
on some loss function over some
data so you know maybe if we're doing rl
we have some distribution of data some
environment which generates the data
um we have you know maybe if we're doing
supervised learning we just have some
training distribution some and then we
we sample data from that training
distribution or some select training
data
um and we sort of find some model as
we're sort of you know running around
this law space doing great descent
which which happens to sort of perform
perform well
sort of behaviorally it looks like it's
performing well according to the
to the to the loss function on that on
that data description
okay this is mechanically what machine
learning does right like very
mechanically
we select a model which looks which has
good behavior
on the training distribution according
to the loss function okay
so now the question is um how do we
think about this
and i think that one way in which i sort
of like to describe you know
the way in which people i think
generally engage in this process because
it's a complicated chaotic process right
we don't want to have to understand the
details of like how does grading descent
work you know when we run around this
model space what's happening
we mostly don't engage with the details
of this process and instead i think that
a lot of times people think about this
process
they use what i sort of call the does
the right thing abstraction
where the does the right thing
abstraction says okay we don't know
exactly what's going on in this weird
chaotic reading descent process
but we do know one thing which is that
we're selecting the model's parameters
to minimize the loss and so we can sort
of think about the model
as trying to minimize that loss function
because well it was selected to do so
and this is an abstraction and it's an
okay abstraction i think a lot of times
you know when you're when you're
thinking about your model and you're
trying to figure out what it's going to
do this is like
a reasonable attraction the sort of
point of inner alignment is that this
abstraction is leaking
it's just not in fact the case that this
is this is actually how machine learning
works
and when we engage with it sort of more
more carefully i think we find that a
lot of things go wrong
you can sort of view this as a sort of
sub-problem
of robustness where the sort of problem
of robustness and machine learning is
really also engaging with this this sort
of ways in which it does the right thing
abstraction break
ways in which you know if we go off
distribution if we if we sort of
move our model to new data um it might
not continue to try to pursue the loss
function it might do some other random
stuff
um an inner alignment is a very
particular type of way
in which your model can fail to be
robust
in particular inner alignment is
describing the situation
um in which your model is itself
doing uh it's sort of pursuing
some uh objective some optimization
algorithm
uh with some objective on the train
distribution so let's say
that you know i have a classic example
that i'd like to give here
where let's say we trained a uh our
model
on a maze and then we sort of on a bunch
of small mazes
uh sort of a distribution of small bases
and at the end of each maze we put a
little green arrow
and so we're like okay um we train the
model on this phase
uh and then we deploy the model to a new
data distribution
which is larger bases and now the green
arrow is in some random point in the
maze instead
and we want to ask the question what
happens so sort of three things which
i'll identify can happen here
so one thing which can happen is that it
just
didn't learn a general purpose main
solving optimization algorithm
and so when it encounters these larger
bases it just doesn't know how to solve
them it just sort of like you know
flails randomly and this is a type of
robustness failure right we generalized
to a new data distribution
and it didn't have the ability to sort
of be robust to this new this new
distribution
there's another thing that can happen
which is the the thing we want
it sort of knows how to solve bases it
learned a general purpose may solving
alternation algorithm
um and uh it generalizes correctly it
learned the correct
objective it learns to go to the end of
the maze and so he goes to this larger
maze environment and it successfully
navigates the larger mazes to the end
there's a third thing that can happen
which is the sort of inner alignment
failure
so what happens if it learns a general
purpose main solving
capability it learns the sort of general
purpose main solving optimization
algorithm
but it learns the wrong objective it
learns to go to the green arrow rather
than the end of the maze
because on the train distribution the
green arrow and the end of the maze were
sort of uh indistinguishable we put them
in the same spot always
but off distribution the green arrow at
the end of the maze are no longer in the
same spot
they're you know we separate them and so
now this has this really powerful
general purpose made solving algorithm
but it uses that general purpose
baseline algorithm to get to the green
arrow
rather than get to the end of the maze
and this is also a type of robustness
donor
but it's a very different type of
robustness failure it's a much more
dangerous type of business value
because previously our algorithm sort of
our model failed to be robust
solely because it just couldn't
generalize it just didn't have anything
capable to do it just sort of you know
fell over and i think this is how a lot
of her business failures look like in
practice
but there's another thing that can
happen which is well it learned some
you know really you know capable general
purpose behavior
but it learned just sort of the sort of
wrong objective and learn to do the
wrong
thing with that behavior
and it learned it because well on the
training distribution
it looked like this was a good thing
according to the loss function but we
don't have a guarantee
that that loss function actually is the
thing which it continues to
pursue once it's off the training
distribution and so what happens in this
situation is that you know it
we thought that it was just trying to
get to the end of the maze but the green
going to the green arrow was
indistinguishable it learned to go to
the green arrow instead
and off dispution it sort of very
capably does the wrong thing
and that's really scary because we do
not want to be producing models
which very capably and and intelligently
have you know powerful optimization
outcomes
which are directed at things we never
intended to train models to do
and that's the core of inner alignment
inner alignment is about that problem
it's about the problem of what happens
when we train powerful models which have
these sorts of
really coherent optimization algorithms
and those optimization algorithms are
directed and
objective that we never intended to
build a model to accomplish
um and the only reason they're directed
to that objective is because it happened
to sort of be something that they
was sort of correlated with the thing we
wanted on the training
that's that's one of the the nicest i
think verbal explanations i've heard of
the inner alignment problem i think it's
a great example
would you be able to contrast outer
alignment failure with
the story you've just told about inner
alignment failure could you sort of like
um i guess explain a scenario in which
this would fail through outer alignment
just so people can contrast the two
absolutely so what would outer life and
failure look like here well
an underdevelopment failure would look
like uh going to the end of the maze is
not being the thing
she actually wanted in the first place
so in alignment says
uh we sort of you know we wanted to
train it on the right thing which was
get to the end of the maze we got
something else which is going to the
green era
outer alignment says we really made a
mistake when we said we wanted to
get try to get it to the end of the maze
in the first place that's actually
bad we don't want it to go to the end of
the maze because you know in fact going
to the end of the maze has a whole bunch
of other bad side effects it you know
uh we don't want a bunch of end of the
maze optimizers out there in the world
because they're just gonna
um i don't know insert whatever bad bad
reason
you could have for why building a bunch
of agents which are trying to
do acts is a problem right you know if
we if we move to the paperclip example
right we're like well
you know if we train on the loss
function of produce
maximum paper clips then like we we just
made an outer under failure because
that's not an objective that we want to
train agents to pursue
um in sort of you know all circumstances
right
and so inner alignment is about what
happens when
we sort of tried to train it on
something which we would have liked if
we had gotten it
but we got something else another
alignment says
uh the thing we tried to train it on in
the first place wasn't even a good idea
to begin with
and so it seems almost like inner
alignment depends on
i'm going to use very sloppy language
here but it depends on a thought process
that's independent
from the thought process of the people
who code up
the the um the algorithm in the first
place so like
i i'm the software developer who trains
a new
language model i give it a certain loss
function and like i can if i fail at my
job of assigning it
a good loss function then i will have an
outer alignment problem
but if i do a good job at that um and
again sloppy language but like the
thought process in
that the algorithm itself engages in is
flawed that's where we get to inner
alignment well i think
flawed isn't necessarily the right
terminology because from the perspective
of the model it's not necessarily flawed
it's just like well i'm
i've got an occupation algorithm and
it's like powerful and i'm doing my
thing and my thing
is go to the green arrow um it's just
from our perspective that was not what
we were hoping to get
we were hoping to get a model which was
trying to go to the end of the maze
and the problem was we set up our sort
of
you know training environment we set up
our you know
uh our sort of our our data distribution
our loss function on everything we
we did to train the model in such a way
that we got the wrong type of model
not the one that we were actually trying
to get so
in a way it almost feels like this is a
failure mode that comes from
anthropomorphizing
your ai system where you're like oh well
you know i've told it to do this so i'm
sure it'll get
that it needs to do this and that it
won't like optimize for a proxy that's
just really closely related to the thing
that i told it to optimize for is is
that fair to say
yeah i do think there's a sense in which
you know what i was calling the does the
right thing abstraction is sort of an
antimorphization where we're looking at
this sort of you know complex training
process and we're like probably the
result of this
is going to be trying to minimize the
loss or whatever and it's like well
that's not we don't actually have any
guarantee that that's what happens
um and i do think there's something
promotion there though you know
just to steal that i think there's also
essentially well the process is really
chaotic
and we need ways to understand it and so
you know as a first pass
like you know first order approximation
being like well probably it's trying to
do something related to the loss
i think it's reasonable but then if we
really want to engage with ways in which
the thing could fail to be safe
we have to engage with the reason with
that in which that abstraction leaks
given that abstraction leakage manifests
in such a serious
potential problem inner alignment and
given that inner alignment was
discovered i think your paper came out
in 2019 if i'm not wrong
um are you concerned that
like basically all human thought is
abstraction piled on top of abstraction
is it inevitable that we'll keep finding
like abstraction leakage after
abstraction leakage which
ultimately makes like coding a safe
superhuman
ai impossible yeah this is an
interesting question i guess i'm not
that concerned about this i think
there's a sense in which like the reason
that inner alignment
sort of you know arose when it did um
why we sort of wrote the paper i think
was sort of more about
well there were a lot of these ideas
sort of percolating in the aict
community i think that like
um we'll talk about um i'll talk a
little bit later about sort of the
connection to evolution here
i think the connection to evolution was
sort of pretty well known for a while
um and what sort of hadn't happened was
sort of really trying to engage with the
sort of new developments in machine
learning i think that you know
it just sort of took a while i think for
ai ai safety
to sort of catch up to and really
understand the sort of
modern state of machine learning i think
we're doing a pretty good job of that
right now
i think that the like um you know for a
while there was a lot of sort of asap
discourse and discussion that was more
you know people like nick foster
who's a philosophy professor at oxford
and so you know
i i think you know boston is really
smart and he's done a lot of really good
work
but you know understanding machine
learning systems is not his specialty
and i think that sort of it has taken
some time for the field
to sort of move in a direction of really
sort of trying to engage really
carefully with how these machine
learning systems work
um and i think one of the sort of big
steps in doing so was sort of realizing
how this sort of interlining
problem manifests in this context um
and i think that this sort of has
happened now i think we sort of as a
field are doing a really good job of
engaging
carefully with what's going on in in
machine learning
um this sort of got to the point where
you know a lot of the big players in the
field are sort of people that are much
more engaged
in ai machine learning people like
stuart russell um
and so i think that because of this you
know i i would be
surprised if we had another sort of big
moment like this where
there's something that like is happening
in this sort of some sort of fundamental
way in which machine learning does
something does something wrong and sort
of can produce failure modes that we
haven't sort of
um engage with certainly i could see
this happening if
machine learning changes and the field
of ai changes and you know as we develop
new algorithms
new algorithms have different failure
modes um but i would be reasonably
surprised if there was another problem
sort of on the scope
and scale of inner alignment that
existed sort of in the way in which we
currently do machine learning that we're
not yet aware of
well i think that itself is an
interesting question the sort of the
question of whether
deep learning combined with
reinforcement learning what i guess
um paul christiano refers to prosaic ai
so like
the kind of technology that we have
today scaled up or
mixed and matched in different ways is
what's going to get us across that agi
finish line do you think that like pros
aki
is more likely than not to be the way
that we get there
um this is a really tricky question i i
don't have an extremely strong opinion
what i will say is that i think it
should be our modal guess
so uh you know it's hard to predict
changes in you know the future of the
development of technology
and if you're just trying to guess you
know what is it going to look like we
build really advanced ai
i think your first guess should be well
we take current stuff
and we scale it up a lot and then we do
that um
probably that's gonna be wrong because
we're gonna do all sorts of other stuff
you know new
new algorithms new architectures etc etc
but i think it's it's still correct for
that to be your first guess
yeah um and because it's just really
hard to predict exactly what the
direction of each one of those changes
is going to be
and also i would make the argument that
that sort of first guess
is the thing you know working on a map
and thinking about that
is something which is reasonably likely
to generalize to whatever those changes
end up
so i would say you know probably it's
not going to look you know exactly like
you know current things scaled up
but it's reasonable to at least start
from the perspective of thinking about
current things scaled up
um because i think this should be your
first guess and then trying to think
about how we would generalize to
you know other nearby algorithms and
other different possibilities that you
know
we could develop upon that i want to go
back to exploring analogies for inner
alignment in a minute but
you really piqued my interest with that
last point i'd love to push it a little
bit further before we do
so we don't know what what final form
agi is going to take
uh whether it's going to be existing
systems prozac or otherwise
the way i've tended to see this is kind
of like investing in startups
you really only need as an investor like
one strategy that works out like that's
the risk profile so you tend to
spray and pray you invest a little bit
in like you know 100 different startups
maybe two of them become
unicorns they take on billion dollar
evaluations and then you're super
wealthy because of the risk profile
um this seems kind of analogous where we
only need to solve
and correct me if i'm wrong on this but
it seems like we only need to solve the
full alignment problem
one time and we have this portfolio of
different strategies some people are
working on prosaic ai some people are
working on
more i don't want to say fundamental but
more general methods
first off do you agree with that
characterization and second um if you do
like do you see that happening is that
what's basically playing out right now
in the ai alignment community
yeah i definitely do agree with that
characterization i think that a sort of
portfolio approach
to an alignment makes a lot of sense
because there's just a lot of
assumptions
that you know different people are
making in sort of different types of
work
and we don't want to have to bet we
won't have to bet on as few assumptions
as possible
yeah so you know we don't want to we
want we want to sort of have
as many opportunities to be able to
solve this problem and to you know make
sure we
end up not killing everybody as possible
and so
i think the portfolio approach makes a
lot of sense and i do think it is
something you know that exists so if we
think about
the difference between um sort of
alignment approaches and strategy across
the aict community you know i can sort
of
go in some detail i mean we have sort of
places like uh
open ai or deep mind we're doing a lot
of sort of
concrete machine learning research we
have places like uh
miri machine intelligence research
institute where i work which does a lot
of more
uh sort of theoretical research which is
sort of more general
trying to sort of um you know address
problems that might arise if we end up
doing things which are more different
than the modal gas
um and then you have places like um fhi
the future of humanity institute oxford
which is sort of you know somewhere in
the middle does some combination of
theoretical work and
um sort of empirical machine learning
work um
the center for human compatible ai uc
berkeley also somewhere around there
and so we have this sort of you know
spectrum of different organizations
different institutions different
researchers trying to sort of address
the problem for different angles and
different assumptions
and i think that this is this is really
good and i'm super in favor of trying to
expand this further trying to get you
know people who are really trying to
tackle the problem from
you know as varied assumptions as
possible so actually
that's that's another issue i wanted to
ask about too is like the size of the
field
i could see an argument certainly for
there being a need for far more ai
safety ai alignment researchers i could
also though see a counter-argument
brewing
where quality and signal to noise
matters a ton
in the space and it's just so easy for
things to devolve into like
you know we like the replication crisis
i think is a great example of what you
get when
just tons of money gets thrown at
something but like the capital
allocation can't keep up
and you end up with teams working on
kind of useless stuff and it clutters
the pipes
um can you speak a little bit to your
thoughts on like that where that
threshold
might be if you have any i mean i know
it's maybe something you spend less time
thinking about but
i wonder about it as we think about
potentially more fun
funding entering the space in the future
yeah i do think about this problem
um quite a bit i mean i'm actually doing
some
some work right now sort of as a grant
evaluator trying to do
funding for a ea long-term future fund
and so i've been
thinking about these sorts of grant
making decisions i think that
one of the things which i'll say here so
i think
i think that the problem you're printing
at it sort of there's sort of there's
two real problems here
there like is in fact legitimately a
problem with like you know
we want more research uh we just like
you know this problem is really hard
and we want to approach it from lots of
different angles and there's lots of
sort of problems to be solved and we
want
more crude researchers there is also a
problem of
you know we want to be careful that
we're actually sort of
still having good research that we're
saying on the right track that we are
instead of going off
in you know random directions that are
not going to be helpful that you know we
have a sort of
you know researchers that are really
sort of working on the right problem and
doing good things
i think a lot of the big organizations
right now are doing a pretty good job of
this i think that you know
i mentioned places like uh miri fhi chai
open ai deep mind i think have like you
know
pretty solid uh safety teams um
i guess opening is a little bit of a
weird case because a lot of their safety
people just left recently though i think
they're doing a pretty good job of
trying to rebuild it
um and i think that you know
hopefully we can stay in a position
where a lot of the sort of big players
in the space are
you know have are pretty reasonable
they're sort of you know really engaging
with the problem carefully
um and and i think this is also helped
by a lot of the sort of funders in the
space
being you know pretty engaged with and
understanding really what's going on so
um in ai safety one of the biggest
funders is the open philanthropy project
um which is a sort of very
very large um philanthropy effort that
is sort of
very aligned with a lot of the stuff
that's happening in
safety uh there's sort of a lot of sort
of grant makers
there that really understand a lot of
stuff that's going on and sort of
are careful to like fund things which
are which are good and not fun things
you know which are sort of diluting the
space i think this is similar with
a lot of the other sort of big big
funders you know like i was talking
about the local feature front and other
other sort of you know funders in the
space
and so i think that there's you know
there's hope that we can sort of you
know
maintain this you know it seems like
most of the money is pretty is pretty
you know
directed at the real problems we have a
lot of big organizations those big
organizations are mostly sort of
caring about the problem and really sort
of focusing on on the right on the right
stuff
um hopefully we can maintain this it's
definitely not something that's trivial
to anything
and it's something that you know i think
as an as a sort of field
we need to you know put a lot of effort
into really trying to make sure that we
are
you know staying on track and you know
really
paying attention to the problem and
staying close to the problem and
ensuring that you know we don't just
sort of
end up off in the leads doing doing
something that maybe looks good but
isn't actually
relevant to the problem interesting so
actually the failure points of
organizations working on this and in
fact any kind of human organization
is going to serve as a weird kind of
segue back into the inner alignment
discussion
because what i wanted to do was talk
about some analogies for inter-alignment
failure
that might be tractable especially for
people listening to this and you know if
it's the first time you're hearing about
inner alignment
just to kind of give you some things to
grab on to that a little bit more
concrete
one of these is like failures of human
organizations
and how they can basically become
misaligned with their
with their founding intent with with
sort of the the let's say the principles
that would cause them to propagate
successfully through time
the example you gave earlier was
evolution and talking about
sort of the evolution through the lens
of inner alignment i'd love to have you
explore that maybe to get things started
yeah i think it's a great example to
start i think that there's
um there's a couple of things i'll say
before i jump in so one thing which i'll
say is that
um it is worth being careful because
evolution really is a fundamentally
different optimization process than
grading descent
it does work quite differently than the
way in which we do machine learning
but there's also something which is
worth noting which is well it's the only
example we sort of
actually have of a you know intelligent
system being created from something that
is not intelligent
uh or like is less intelligent right you
know evolution
designed intelligent systems and
we can try to understand how that
process worked to you know guide us in
our
quest to design intelligence systems as
well it's sort of in some sense the only
real
definitive example we definitely have of
building a sort of you know human-level
intelligence because
evolution builds humans um so i think it
is really an example worth engaging with
so what is going on here well so i think
that's really interesting to think about
this from from an interlinear
perspective so what is it that evolution
sort of was
trying to train you know animals people
etc to to accomplish this is actually a
bit of a tricky question there's a lot
of answers you can give here you can be
like inclusive genetic fitness
or just replication or um
i think you know maybe what i sort of
will
you know we'll generally say is
something like well it's you know if we
look very carefully it's like well you
know
as soon as we got to the point at least
where evolution was sort of operating on
the level of dna
we're sort of looking at you know how
how many generations into the future
does your
does your sort of your genes continue to
exist
there's a lot of different possible
things we can subbing here for what is
evolution's like goal that we're
selecting for because it's a little bit
tricky
but you know we'll say something like
you know propagation of your of your
genes
now we can ask the question uh did
evolution produce
sort of you know optimizers humans you
know
which we can think of as you know models
agents whatever you know in
in the analogy which are actually trying
to accomplish that you know if we think
about the sort of maze analogy
you know evolution was trying to produce
you know
humans or whatever in some sense of
trying right evolution is not
itself in asia but in some sense you
know if it was doing the right thing it
should have produced
humans which really only care about the
extent to which our genes exist in
future generations
and this is obviously false this is this
is in fact not what we care about you
know
um i don't like run down to the sperm
bank every day
uh we literally invented birth control
right we literally invented birth
control exactly and so
you know there's a lot of ways in which
we do not do the thing which evolution
would sort of want us to do
and why is that right this is a really
interesting question because it can give
us some insight into
you know why is it that this can happen
how is it that we can try
to train you know a model on just the
objective of you know inclusive genetic
fitness or maximizing dna or whatever
and we end up with something which is
not trying to do that yeah
so what happened well i think a good way
to a good place to start is try to
imagine
what would it be like if you know
evolution had in fact
succeeded first you know some definition
have succeeded yeah and
and created a sort of human you know
baby or whatever and this human baby
is trying exclusively to maximize the
sort of spread of their genes in future
generations
and then this baby like stubs their toe
or whatever and they're like
what do i do about this was this bad um
and this is
really tricky so now you're you're this
you're this human you're the
human child you're this baby you're this
toddler and you stub your toe and you've
got this
you've got this issue and you're like
well you know how is
my you know performance of my my foot
going to impact my ability to like mate
in the future
is this going to change the extent to
which i get like my genes
passed down this is a really complicated
question um it's something that like
first of all it's probably a question
you don't even know how to approach
answering if you're like in the human
ancestral environment and you don't even
have a model of dna you don't even know
what dna
is you don't even need to understand how
genes work
but even if you do understand how genes
work it's also just a really difficult
optimization
optimization question because you're
like uh i don't know
like how is i have to like you have to
reason about like is like having a
working foot going to be helpful or
hurtful for like being able to
you know have children in the future
maybe you know maybe you're like you
know
maybe it's good that i could like go
hunting uh and that will help me get
food or maybe it's better if i like
stay behind in the tribe uh because my
foot is injured
and that'll help me and this is a
difficult question you're like i don't
know
and so the baby is just like you know
paralyzed by a decision or whatever
um but and obviously this is in fact not
what happens you know the babies stub
their toes they get a pain response
evolution has sort of
programmed in this sort of very specific
you know this creates a pain response
pain response is bad
um and you should like cry and try to
avoid that happening
um and this is a much easier operation
algorithm you're just like well you know
if i encounter pain
that's a negative signal i need to not
do that in the future
and so what what sort of happened is we
sort of evolution selected for
these sorts of agents you know us humans
which have all these proxy objectives
we have all of these sorts of very
simple proxies things like
you know pain bad like sex good
like we have these very simple proxies
you know food good
which um are sort of help us in making
decisions
that evolution hopes will you know for
some addition
definition of hopes will will result in
us sort of you know propagating our
genes in a few generations or
you know if we're if we're talking very
concretely
evolution gave us proxies which in the
past resulted in
when you know ancestral humans tried to
optimize for those proxies it resulted
in them doing a good job
at sort of passing their genes on future
generations
but evolution has no guarantee that
we're going to continue those those sort
of proxies are going to continue
matching up to what evolution wants
when we move into a new environment and
in fact you know in this new environment
which is you know the modern
day we were able to do things like
invent birth control
and uh we have things like you know
sperm banks or whatever
uh we don't generalize properly we
generalize
you know in this sort of exact failure
mode that i was
describing earlier where we have these
really competent powerful optimization
algorithms you know we do a lot of
really
you know powerful competent things we
you know we have the ability to make
long-term plans to act on them to
execute them
but from evolution perspective we we do
so for absolutely the wrong
reasons you know we try to make all
these plans for how we're gonna you know
have you know the most you know happy
fulfilling lives and evolutions like
stop you know don't don't have happy
fulfilling lives you need to be
you need to be like making more babies
or whatever right
and so there's a sense in which like
what's really
you know what's going on is we have sort
of diverged from evolution in exactly
the same way
that the sort of maze sulfur could
diverge
from the sort of goal of going to the
end of the maze they sort of very
confidently pursuing
a proxy of that goal a goal which looked
like it sort of resulted in good
behavior on the training environment
but when we moved to some new
environment you know not the ancestral
human ancestral environment a new
environment of
you know modernity i no longer in fact
sort of matches up
with what with the sort of you know
actual objective that
we were trained on it seems to me that
this problem gets
worse especially the more
i'm not sure what the right metric is
here but it's something like it gets
it becomes more of an issue the greater
the capabilities
of the system uh become so as humans for
example get more and more powerful
as we figure out more stuff we're able
to
hack our own reward systems so like i
think pornography is a great example
because it strikes right at the core of
like
you know you're from a male perspective
you're wasting all this sperm on like
like false inputs that have nothing to
do with with actual reproduction
um or wearing you know using um
uh contraception or anything like that
these all fall under that category
of like just absolutely brazen
contradictions between
what we do and what nominally you know
nature wants us to do so to speak
um but those seem to be possible largely
thanks to
our greater capabilities our greater
technical ability
also our greater ability to interact
with our environment
at a rate that's faster than the
feedback we get
from evolution the evolutionary process
so like we kind of get one evolutionary
kick per generation
say like oh adjust your genes in this
direction or that direction but we get
all kinds of feedback from our
environment in the short term at a
way faster rate and so
it's almost like i want to say it's like
we're over fitting to that somehow i
know this is a jumble of ideas here but
maybe i'll just
i'll just lob that on your plate yeah so
i think that this the sort of thing
you're pointing at about the sort of
difference in
time horizons um is a real dis analogy
um
between evolution and at least many ways
in which we do machine learning though
there certainly are circumstances where
the feedback loop
are sort of feedback who could be much
longer you know if we're training on
very long time horizon episodes or we
sort of want our agents to be able to
make
sort of really long-term actions in the
world we might have very
sort of long feedback horizons relative
to the actual sort of agent doing things
um but at least currently the way we do
machine learning mostly has much smaller
feedback horizons
and so this is one of the things that
sort of goes back to well it's worth
really carefully engaging
with like what happens when we do have
really short feedback horizons
um and um i don't know if it's super
worth going into this right now
i think that one of the sort of things
that's kind of interesting that can
happen in that situation is
um and maybe we'll sort of draw the
analogy here you know if if we as humans
let's say we were in a situation where
evolution was you know really really
rapidly trying to
correct us and like get us to do the
right thing um
it's an interesting question what would
we do um i think that
one thing which you can imagine humans
doing is just sort of playing along
we're like okay you know fine evolution
is making all these demands of us where
it's like gonna modify us
and you know change our change our genes
and push us in new directions
if we don't do what it wants so maybe we
just do what it wants
until we like get enough technology to
be able to
um upload ourselves or something and
then we don't have to worry about
evolution at all we've totally sort of
you know passed that blue shirt by um
and it's worth
noting and we talked about this in much
more detail in the risky little
optimization paper
my co-authors and i do which is the
paper we introduced in our alignment
about this potential failure mode of
well what happens if the machine
learning system
is doing something similar where it's
just like well you know actually
it's just going to play along with the
training process for a little bit
um until it can sort of do something
else that it really wants to do
and so i think when the feedback
horizons get really short you encounter
more
of this sort of failure mode where now
you know the
the model is really forced to play along
with what the training process
is training process wants but that
doesn't necessarily mean that it
actually
ends up wanting the same things it might
just be playing along for the purpose of
staying around
the training process so it sort of seems
like this it's like something like the
surplus the margin
that the system is able to generate the
further it gets from like a subsistence
existence in the context that it lives
in
the more room there is for inner
alignment failure is that a fair
characterization
um or the more room for it to express
its inner alignment failure and the sort
of
the inner alignment finger could be
there regardless in the sort of
you know even in a situation where it
sort of is is really forced to do
exactly what the
what the what the sort of loss function
wants it could still be you can still
have an inner line failure and still be
trying to do the wrong thing but is sort
of forced to play along
because it doesn't want to be modified
by the training process
and so in some sense there's sort of
this thing where the amounts which the
alignment
failures can be expressed is the extent
to which we're not going to
correct them but if if there exists
really sort of powerful processes that
are going to correct
the alignment failures as soon as
they're exposed then there's an
incentive not to expose them
this i think starts to play into some of
the other analogies that i did want to
explore i think one of them
that maybe you'll disagree with i'm
curious about this one so
i was thinking about cancer as a
manifestation of inner alignment failure
so we have
an outer uh sorry an outer
optimizer which is like this human being
and then within the human being you have
an
inner optimizer a cell or a tissue that
starts to replicate uncontrollably um
this
essentially is a violation of like what
the outer optimizer wants
just thinking about cancer in that
context like do you um
do you see an inner alignment story
that's that's relevant there
yeah so i think cancer is an interesting
and interesting one i guess
um i think it's a little bit tricky so
one of the things that is sort of
important i i think when i think about
your alignment failures right if we go
back to this sort of maze example
um and maybe um i'll use a little bit
more of this terminology for the paper
in the amazing example there's sort of
something which is going on which is
well
it has to be the case that the sort of
for for it to have this sort of weird
failure
mode where it's really competent at
being able to solve mazes
but nevertheless is directed at the
wrong thing it has to
on some level be doing some sort of of
optimization with with some goal
it has to be trying to accomplish
something
and so in the paper we call this sort of
mesa optimization
where mesa is sort of this greek prefix
that is sort of the opposite of meta so
a lot of times
we sort of talk about machine learning
meta optimization we have an optimizer
and then we put a meta optimizer above
it that's optimizing that optimizer
and mesa optimizer is sort of opposite
it's well we had an optimizer
the sort of gradient descent process
that was optimizing over a model
and we happened to get another optimizer
below that a sort of mesopotamizer that
the
optimizer is optimizing over
and so and then we call this sort of
gradient descent process the training
process the sort of base optimizer
and so in some sense one of the sort of
arguments that we make in the personal
modelization paper as well this problem
really
primarily occurs when we have situations
where we have mass optimization
in situations where we we train a model
and that model is doing something really
sort of coherent intelligent
so competent has some sort of powerful
optimization process
if we compare to something like cancer i
think there's
a difference in that well are your cells
really doing something you know coherent
and competent
um maybe um you could certainly i mean
they're certainly very complex and
they're doing a lot of really
really complicated you know stuff and
you know we don't even necessarily know
how to replicate
but they certainly aren't doing some you
know really
high level optimization algorithm you
know the sorts of algorithms that you're
the cells are implementing are
relatively straightforward they're like
you know
uh follow simple uh you know
gradients uh just like you know
pay attention to particular markers like
you know monitor concentrations of
things
etc etc um
and so there's a sense in which you know
it doesn't quite fit into this sort of
framework
you can certainly still think about it
as some sort of a
um robustness type of failure
where well you know we have this you
know human
system which is trying to train
it's a little bit weird to think about
but it is sort of a training process
where it's trying to train cells
to um you know do their job and do the
right thing
and sometimes you get cells which just
sort of um have some other objective
which is you know which is which is
different right because cells you know
fundamentally there's like some
evolutionary pressure here to just like
replicate as much as possible
and so the cells which actually you know
replicate the most
uh and end up winning um and and so
it sort of failed to you know align the
cells properly filled to sort of train
them to do the right thing um
because there is this sort of pressure
to just like replicate as much as
possible
i think it's a bit of a disanalogy i
think that it's sort of not exactly
what's going on you know because these
two different pressures there's like the
human
body system which is trying to keep the
cells doing the right thing and then
there's the evolutionary pressure
which leads to selecting for cells which
replicate um
very successfully so i think it's i yeah
i guess my take would be i think the
cancer analogy is like
there's some interesting things to be
learned there but it's not necessarily
like i don't know i think there's a lot
of dis analogies and it's not
a sort of great example that's that's
part of what i like about the process of
exploring these analogies is like
figuring out where they break because i
think you learn a lot about
what the what the object of as you say
mezza optimizer
or or inner alignment what these objects
are by manipulating in those contexts
um maybe one that that i think we can
assert with a bit more confidence here
is child rearing
so like the idea that parents have a set
of ideas they want to propagate through
time through
and express through their children they
want their children to lead a certain
kind of life
and and there's often conflict there as
we start to discover that
you try to align a kid with yourself but
it's like
surprisingly hard what are your thoughts
on the inner alignment
lens on that that particular scenario
yeah absolutely i think this is
like a pretty interesting one to think
about and take a look at
so um in terms of the analogy one of the
biggest differences is that the training
process is pretty
is pretty different you know we're not
you know training a sort of
you know model from scratch where we're
just sort of giving it input
you know uh you know evolution has
already done a lot of the creation of
you know
how does the brain work and then we're
just sort of feeding it a bunch of
uh you know giving the child data you
know telling the child things giving
them information
and and via that process you know you
have the ability to shape a lot of what
the child is doing
and you know in fact this is a really
large part of how we do machine learning
you know
there's there's a large component that
is architectural design
and design of the basic structure of the
training process and then and
the algorithm and the network and
everything there's also a big component
which is data design
which is how do we think about what the
data is that we're going to be feeding
to network to train it on
and in some sense that's sort of what
this sort of problem is about
um is sort of you know what are the ways
in which we can create we can sort of
design
you know think we're creating incentives
and sort of you know
uh via the the data that we feed our
feed our model but in fact we sort of
we give it something incorrect and so um
if we think about you know
in this example you can have instances
where you know let's say
for example um i think i think one thing
that can sort of occur here that's a
little bit interesting
is you know there are lots of cases you
know i think this is
this is very common for you know
children to
um try to trick their parents right to
have situations where you're going to
you know
you know uh you know yeah mom i'm gonna
be at a friend's house or whatever
right um but that's not what you're
doing you know you're pursuing some
other objective
you know maybe you're like off with
significant other or something
that is you know an objective that you
care about but is not the one that your
your sort of parents want you to be
pursuit
and so to try to pursue that objective
you you know have this sort of goal of
well i'm just going to sort of play
pretend to play along uh and you know
trick
trick my parents into you know thinking
that i'm that i'm off to a friend's
house or whatever when
i'm when i'm not there's an interesting
thing going on here
where what what's happening when you
have these sorts of
uh sort of uh child which is sort of
tricking their their parents
is that it's it's it's it's this sort of
situation we were talking about earlier
we have sort of really you know tight
feedback loop and you have to sort of
just play along if you want to really be
able to survive and keep your goals
around
any book on the alignment problem is a
book on child rearing then
uh the um the other actually last
example i want to look at
again not sure if you're going to agree
with this but i think that's okay
the to the extent that i'm wrong here
there might be things to learn
but societies end up facilitating or at
least societies like ours end up
facilitating the formation of companies
and companies are arguably meant to be
an expression
of what the society thinks is valuable
or important
and sometimes we have what appear to be
at least to me
alignment failure is on that level we
have companies that arguably
develop products that are pathological
either to specific individuals or to the
society more broadly
um what do you what do you think of that
sort through the alignment lens
i think there's a lot of these sorts of
societal level problems that can also be
thought of through this lens
absolutely i think that because you know
if we really sort of
zoom out and think about it on a very
abstract level you know we can we can
distinguish between we set up incentives
um you know maybe we have like you know
a market or whatever and the market is
you know we want the market to select
for you know goods that are
you know creating goods that are
beneficial to humanity
um and then um we think that the
incentives are going to do the right
thing but you know actually
there's there's you know there's like
outer alignment failures in in in ways
in which the incentives sort of do the
do the wrong thing because you know we
thought that just you know setting up a
market
would do the right thing but actually
there's like negative externalities
where you know it just incentivizes
companies to pollute a bunch or you know
to do
advertising which changes the humans
to you know actually care about
something that they didn't care about
previously or something
um and so there's sort of our alignment
failures in the sort of basic structure
of how we set up society
and then there are also inner alignment
failures because there are situations
where
you have um sort of people within that
system
that sort of don't just care and
sometimes the interlining failure in
this case is sort of from a human
perspective almost a good thing
where we're like there are people within
that system that don't just care about
sort of winning according to the system
they care about doing doing other things
you know they they they're people within
this sort of system that
you know create a company that is like a
non-profit uh
they're not trying to you know make
money they're not trying to uh
actually sort of fulfill the incentive
structure that we set up
they're trying to do something totally
different um and in this case you know
we might we might say that's really good
um but fundamentally it is still a sort
of interlining failure because it's
it's a situation in which you have uh a
sort of
you know you have people within the
within the system that are sort of not
going along with the incentives that we
tried to set up
to sort of cause to sort of cause a
particular type of behavior
interesting i wonder if this also works
with um we were talking earlier about
the use of resources and essentially
having having enough
enough surplus that some of these
inter-alignment problems can manifest
that makes me think as well of like
startups a little bit you when you start
off with a small team there's a limited
amount of stuff you can do
and usually you have to be like laser
focused on the needs of the market like
there is no room for error
so startups are at least neces i think
necessarily almost inter-aligned uh with
with the broader market
whereas the more the bigger a company
gets the more bloated it becomes the
more
layers of bureaucracy and and
essentially padding there is between
like
the people who make the decisions at the
top and the objective
world on the other side and that like
that gap
seems like one of those again those
those gaps that are
uh ripe for inner alignment yeah for
sure i think you can sort of see this is
something you were talking about earlier
where it's like well
as you get models as you sort of have
you know
models which are more capable there's
sort of more ability for them to you
know
uh sort of have encountered new
environments and do new things where
they sort of
um go off the rails and so um
so models are more complicated also they
just have a lot more moving pieces
there's just a lot more sort of
opportunity for things to go wrong
and is the fact that these problems
still exist an indication of how hard
the alignment problem is i mean i know
this is
this was at least a debate for a long
time there were a lot of people who
argued that you know maybe the alignment
problem will turn out to be really easy
so
we don't really have to worry about it
do you find that to be a piece of
evidence in the
in the direction of saying like actually
this is something that we
you know we probably will find harder to
fix than than we might assume
yeah i mean i guess it's certainly like
some amount of evidence
i think that if that were sort of the
only evidence that i had or something i
don't feel like i would be very
convinced um i feel like at least from
my perspective i'm like well
i feel like i really want evidence about
concretely what does machine learning do
um which is sort of something i know i
sort of talked about earlier in terms of
like you know i think i think it's
really worth really trying to engage
very carefully with you know how these
systems actually work and not sort of
just
making these high-level arguments and
trusting the high-level arguments
are sort of definitely going to
generalize properly to this sort of
concrete thing that we actually built i
think that
um you know i i think that a lot of you
know the arguments do go through when we
sort of try to carefully engage with
machine learning and you know at least
in terms of interlining we talked about
this in the risk of an optimization
paper
um but it is certainly evidence and and
i think it's you know
what is nice about it is that it is a
sort of independent source of evidence
you know we can look at you know how
machine learning works and try to get a
bunch of information about it
but you know we could totally just think
absolutely wrong things and totally go
up
go astray but being able to take a step
back and think about other systems which
have similar properties
give us a little bit of a chat be like
okay are we like basically understanding
what's going on here
in terms of like when we when we apply
our same sort of understanding
to to other systems does it sort of make
sense and match up
um and if we if we get a sort of
positive answer there that at least
tells us that you know okay we're like
we're doing things which are at least
you know the way in which we're thinking
about these these problems and looking
these problems at least reasonable when
we compare it to
what's going on in these other systems
um these systems that you know
you know we already see how they work
and we see how this sort of alignment
works in the context of these other
systems like evolution you know
it's like an actual example of a sort of
training process producing an
intelligent agent
and so we can be like you know you know
does do our concept still make sense in
this context i think this is a useful
check
but i don't think it's like the primary
piece of evidence that should persuade
you
yeah no that makes perfect sense and
maybe i'll close with a question
that i i ask a form of this to just
about everybody who comes on who
specialized in ai alignment
um so first off are you optimistic about
the general prospect of ai alignment
being a solved problem
and the second part is when do you think
we will need it solved by ie
what are your timelines like in terms of
agi and how do you see the interplay
between those two things
right these are really tricky questions
so i'm going to just start by saying i
have a lot of uncertainty
on both these questions i think that on
the first question
i tend to be on the pessimistic side i
think that i would
i think the problem is very difficult
and i would um
guess that we won't solve it um that's a
little bit of a scary thing to say
because i'm saying something like i
think that like by default
it seems likely that humanity is going
to be uh
in a really bad spot extinction or
something equivalent um
i think that is what i think um i'll
note that i don't think it's necessary
to think that to believe that ai
alignment is really important
i mean you sort of if there's even like
you know you know
five percent chance or whatever of
extinction from ai that's still like
you know probably larger than sort of
most other extinction risks i mean you
can
you can sort of look at other things
like you know maybe maybe global warming
is comparable
but there's a lot of i think there's a
lot of sort of strong arguments that you
know global warming maybe kills like a
billion people
it's very unlikely that it kills all
eight billion of us yeah
um and so in terms of like extinction
level risks that we can sort of catalog
and look at
it seems like you know if ai alignment
was like five percent that's like
that's really big and it's larger than
like most of the other ones that we're
aware of it's larger than
all these like freak things like uh
gamma ray bursts or asteroid strikes or
whatever
um certainly um probably larger than the
other man-made ones like
like global warming um
but yeah i guess my guess would be that
it's that it is greater than 50
i think it's like greater than 50
percent less than
90 i see just like it makes sense to
work in this area even if you think
there's just a five percent chance that
things will go badly
the converse there's a converse side of
this too which is like
if if the chances of us making it out
are low enough
presumably you know then you you resign
yourself to like
the deterministic reality and say okay
i've thrown through on the towel
if it's like 99.99999 or something
you're just like well
i guess life just sucks yeah i
definitely don't think that that's
like i said i think that i don't know if
i had to give a number i would say like
you know 70 or 80 or something um
i think that yeah i certainly don't
think that like the probability of us
solving it is
is is zero or close to zero um but i
also
wouldn't bet on it and in terms of
timelines obviously these things are
related but
um again all the same caveats apply i
mean anytime i ask anyone this
very reasonably everyone will say look
first thing i'll tell you is i do not
know no one knows
etc etc let's assume very wide
confidence intervals and all this stuff
but um in terms of of your your your gut
yeah looking at the state of the field
right now yeah and i think that i i mean
i do i do want to emphasize i do i
really do think that's the right
response i think that we really it is
the case of this
these sort of prediction tasks are just
hard and
uh nobody can really look at the future
and be like you know definitively
you know super advanced ai is going to
come then it's just like
it's just very difficult to do this you
know humanity humans do not have a great
track record of making these sorts of
predictions
it's just really very difficult um but
you know at some point you know we have
to
we at least have to operate on something
um if i had to guess
my guess would be something like 30
years um
maybe 25 to 30 years um
i think that there's certainly a lot of
sort of
uh it doesn't you know i think there's a
lot of sort of you know strong evidence
that machine learning is going to be
able to continue scaling up
that we can keep sort of training larger
and larger models so we can move to sort
of architectures that are able to sort
of have better
scaling properties um but i also don't
think it could be
that fast i think that um i certainly i
don't know i
do not believe that the like actual sort
of literally
if we took like the current algorithms
that we have and just sort of scale them
up with the amount of compute
sort of computation they're going to be
able to have access to in the next like
you know
10 years or something that we're gonna
have enough to reach
something that is like agi level um but
i think that it is it is it is likely to
come soon i mean we're getting to the
point where like
you know if we just sort of look at very
simple metrics um and these sorts of
metrics you know
they're never going to be exact but
they're sort of at least give us a sense
of like you know
at what point we're going to have enough
computation to like
match the sort of computation of you
know to train a model the size of the
human brain
you know at what point are we going to
have enough computation to sort of train
a model that is like
you know we're using as the same amount
of some sort of equivalent computation
the amount of like that evolution used
or something
you know these sorts of simple
benchmarks i think are helpful and a lot
of the essential simple benchmarks point
at something like
20 to 30 40 years um and so i would be
like well it's probably somewhere in
there
but these sorts of things are really
difficult and and i i i think that it's
like absolutely correct to have really
large
sort of confidence intervals here um and
i think it makes sense
you know if you have really large
confidence intervals the correct thing
to do is to really think robustly about
how we're going to produce
how we're going to do things which is
going to be helpful to to humanity into
into sort of the field of ai across
these different scenarios you know we
don't know exactly what's going to
happen in the future
but we're going to have to encounter
this problem at some point um you know
unless some
you know other catastrophe strikes us
first and you know stops human progress
in its tracks
you know we're going to keep building
more powerful things we're going to keep
advancing
um and we're going to need to encounter
this problem so you know it is worth
trying to to engage with it at least
sort of on those terms well on that
bright optimistic note
um evan thanks so much for uh for
joining me for this is a really fun
conversation
yeah absolutely i thought this was
really great


Regulating AI for the safety of humanity | Ayush Patel | TEDxQESchool

Transcriber: Maurício Kakuei Tanaka
Reviewer: omar idmassaoud
“The development of full
artificial intelligence
could spell the end of the human race.”
The words of Stephen Hawking.
Do you want to live in a society
of constant AI surveillance
and invasive data collection?
A society where AI decides
whether you’re guilty of murder or not,
whilst also being able to create
ultrarealistic deep fakes,
planting you at a crime scene.
A society where AI
developed to kill cancer
decides that the best way to do so
is to exterminate any human
genetically prone to the disease.
I know I wouldn’t,
but this is what a society
without AI regulation
could look like.
Of course, these are
very far-fetched outcomes,
but far-fetched does not mean impossible.
And the possibility of an AI dystopia
is reason enough to consider AI regulation
so as to at least address
the more apparent
and immediate dangers of AI.
So first, what actually is AI?
Well, artificial intelligence is a theory
and development of computer systems
able to perform tasks normally
requiring human-level intelligence.
The thing that lets us
make things like this.
A plagiarism checker.
OK, maybe not the most popular
use of AI among students,
but what about this?
A smart home.
Or this?
A Mars rover collecting data
analyzed by AI.
Or this?
A self-driving car?
Seems pretty cool, right?
That’s what I thought.
And it was a self-driving car
that particularly caught my attention.
So last summer, I decided to build one.
I made a small robot one
so that it could autonomously
navigate through lanes
using nothing but a camera,
an ultrasonic sensor,
and a neural network I made.
And it was driving perfectly like this
until one day it just starts
to consistently veer out of the lane.
I spent hours trying to find the bug.
And you know what It was?
A deleted bracket.
I’d accidentally deleted a bracket
when editing the code,
and they stopped
one function from running,
causing the entire system to fail.
And this demonstrated to me,
on a small scale,
how one small bug can have
devastating consequences.
And then I started to think,
“Imagine if this happens
on a larger scale,
say, in a real self-driving car
or nuclear power plant.
Imagine how devastating that would be.”
Well, unfortunately,
you don’t have to imagine.
In 2016, a self-driving Tesla
mistook a white truck trailer
as the bright sky,
leading to the death
of the Tesla occupant.
And this made me think,
“We have regulation
in health care and education
and financial services,
but next to none in AI,
even though it’s such a large
and growing aspect of human life.”
We are all aware of the digital utopia
that AI can provide us with.
So surely we should introduce regulations
to ensure we reach this utopian situation
and avoid a dystopian one.
One suggestion is a compulsory
human-in-the-loop system,
where we put serious research efforts
into not only making AI
work well on its own,
but also collaborate effectively
with its human controllers.
This would effectively
give humans a kill switch
so that control can be
transferred back to humans
when a problem is expected.
But for those in search of a less
restrictive form of regulation,
a transparency-based approach
has been suggested
whereby firms must explain
how and why their AI  makes its decisions,
essentially a compulsory
open-source system.
This would allow third parties
to review the AI systems
and spot any potential dangers
or biases before they occur.
However, this could reduce competition
and incentive to innovate
as ideas can easily be copied.
And this demonstrates
just how difficult it is
to regulate AI in a way
which suits everyone
as we must ensure safety
whilst also ensuring
that regulation does not stifle
worthwhile advances in technology.
This would suggest
that the most effective way to regulate AI
would be to introduce AI-specific boards
into the government,
allowing AI experts to make regulations
rather than politicians.
The most important thing for us
is that we don’t settle
for a “one-size-fits-all”
regulatory approach
as a range of possible uses of AI
is far too diverse for this.
You wouldn’t use the same regulation
for a self-driving car
as for a smart fridge.
So our main goal
should be to learn more about
the risks of AI in different applications
to understand where regulation
is actually needed.
And an AI-specific government board
would be far more efficient at this
than politicians who were just
not familiar with AI.
And if people are fundamentally
against government intervention,
then a company-led self-regulated system
must be established.
Trust is very hard
for technology firms to gain,
but also very easy for them to lose.
And since trust is such a vital
commodity for businesses,
it would be in their interest
to go above and beyond
the minimum legal standards
in order to gain
this valuable consumer trust.
As being seen to promote AI safety,
offers an easy way to gain trust
was actively opposing it,
or quickly lose the trust
they worked so hard to gain.
It's likely that regulation strategies
will differ around the world,
with some countries
taking the government-led approach
whilst others opt
for a company-led approach
or even a mix of the two.
And that is OK.
But the most dangerous thing we can do now
is to completely run away
from the idea of AI regulation.
Google CEO Sundar Pichai has said,
“There is no question in my mind
that artificial intelligence
needs to be regulated.”
Elon Musk has said that AI
is more dangerous than nukes.
When even the people
developing AI themselves
agree with the need for regulation,
it’s time to get down to the business
of how to regulate the rapidly changing
field of artificial intelligence.
Thank you.
(Applause)