Danijar Hafner - Gaming our way to AGI

[Music]
hey everyone and welcome back to the
tours data science podcast now until
recently ai systems have been narrow
they've only been able to perform one
specific task that they were explicitly
trained for while narrow systems are
clearly useful the holy grail of ai is
of course to build more flexible and
general systems but that's not possible
without good performance metrics that
you can actually optimize for or that
you can at least use to measure the
generalization ability of a particular
ai system somehow we're gonna have to
figure out what single number needs to
go up in order to bring us closer to
generally capable agents and that's
exactly the question we'll be exploring
today with dana jar hafner an ai
researcher at google brain and a phd
student in ai at the university of
toronto nadenjar has been studying the
problem of performance measurement and
benchmarking for rl agents with
generalization abilities and as part of
that work he recently released crafter a
tool that can procedurally generate
complex environments that are a lot like
minecraft featuring resources that need
to be collected tools that can be
developed and even enemies that need to
be avoided or defeated now in order to
succeed in a crafter environment agents
need to robustly plan explore and test
different strategies that allow them to
unlock certain in-game achievements some
which are pretty complicated now crafter
itself is part of a growing set of
strategies that researchers are
developing to figure out how we can
benchmark the performance of
general-purpose ais but it also tells us
something interesting about the state of
ai itself
increasingly our ability to define tasks
that require the right kind of
generalization abilities is becoming
just as important as innovating on ai
model architectures manager joined me to
talk about crafter reinforcement
learning and the big challenges facing
ai researchers as they work towards
general intelligence on this episode of
the taurus data science podcast
[Music]
i mean you're working on a whole bunch
of different things but what i really
wanted to talk about today was some of
your work in rl that i think is much
more foundational much more
important than it might immediately seem
from the outside just because we'll be
talking about your benchmarking work
trying to assess the performance
characteristics of reinforcement
learning agents their generalization
abilities and um this this really is
starting to seem like a hot feel then
anyway i'm really excited to dive into
it so thanks for making the time for
this
cool i'm excited too how do we come up
with good measures of generalization
ability for for rl agents specifically
but if you have thoughts about kind of
general machine learning too it'd be
really interesting to hear
measuring generalization i would i would
say isn't that hard if you know what
type i mean there's probably different
ways to define generalization different
um different aspects you want to
generalize to
but
at the end of the day if you know what
aspect of generalization you care about
you can just set up an environment where
you're training on this on like a
holdout set of combinations or like
you're training on on a set of
combinations and then you have some
combinations held out and that's what
you're evaluating on
um so i yeah just evaluating it is
perhaps not that
different i guess more defining it is
the challenge yeah like what do you what
types of generalization do we really
need and
um for
certain maybe long long-term
applications that we care about
and it's probably not just visual
distractors
yeah well also on that note what how do
you think about generalization because
when i looked at your paper it was
really interesting it was very um
a very kind of applied definition that
you seem to use where you're looking at
you know does the agent learn to do
these specific actions in this specific
environment and it almost seems like
those are proxies for something i just
wonder if you've if you thought about
like what is that that core thing that
latent space definition of
generalization let's say that you're
drawing from when you come up with those
definitions sure yeah so
i think i mean first of all crafter is
designed to evaluate a lot of different
agent abilities and generalization is
one of them but there's also other other
categories
um
like long-term memory
and
yeah like being able to survive and
reusing sub skills there are a lot of
repeated tasks you have to collect the
basic resources over and over again to
build more and more complex tools
but within the generalization
the the main
the need for generalization comes from
the procedural generation
and from the randomness in
in the creatures and the environment
so
you know like
every map is is completely randomly
generated so the agent won't know uh
where to find things and they can't
memorize
exactly what action sequence to use
right in atari games if you don't use
any
any stochasticity
i think there
is some some work showing but you know
sometimes these agents really latch on
to tiny details that really shouldn't
matter like whether the
pixel in this corner is blue or red and
then it already knows whether it's in
the first room or the second room
um you know of course that only works if
it's really not deterministic and
there's no sensor noise at anything
so in crafted the environment is
randomized a lot there's a lot of
procedural generation so the agent will
never find itself in the same situation
but still it has to be able to you know
find water every uh every minute so it
doesn't uh get too thirsty and then it
has to find food and then those are all
different skills
that need to be applied in different
situations
and actually this i think a good option
to start talking a little bit more about
crafters so like yeah what is crafter
and what are the main things you were
hoping to measure using it crafter is
is a single environment that's set up in
a way that makes it efficiently
efficient to use for research and
the goal is to
train an agent in crafter once and then
get out the whole spectrum of agent
abilities you're measuring performance
across many tasks
but we're not doing that in a multi-task
setup it's just a single reward function
and the agent just tries to maximize
that but along the way
you know there's this whole technology
tree of things you can do and some
things depend on each other other things
are in are independent
so you kind of have to
um
to explore everything you need both deep
and wide exploration there
um and at the end of the day you get
like at the end of the training run you
get the success rates on all of these
achievements that each give you a reward
of plus one during the episode the first
time you you unlock them during the
episode
uh so then you can see you know these
are the tasks that require memory these
are the tasks that require more
generalization and so on and you can um
you can see where your where your agent
is failing and where your agent is doing
well
um and even if your agent is not a
state-of-the-art thing
um it gives you more than just a single
number right whether you can
to tell whether you're state-of-the-art
or not it tells you you know these are
the kind of
there's a bunch of easy tasks there as
well there's also a bunch of tasks that
haven't really been solved yet by any
current rl method
so even if you're just prototyping with
new ideas you would get a good feedback
signal of how well your agent is doing
and and one of the things i found really
cool about crafter 2 is it did come out
very
i'm trying to remember which one came
out first but it was around the same
time that deepmind came out with their
open-ended learning leads to generally
capable agents paper where they they did
kind of philosophically it seemed like a
almost the opposite approach where
they're looking at a whole bunch of
different environments and then trying
to put agents in those environments and
forcing them to kind of be good at
somewhat good at everything whereas
crafter is cool because it's sort of
like this one very rich world um i'm
curious like how would you compare the
two what do you think think are sort of
the strengths and weaknesses of either
approach
of of having many different environments
versus one yeah
yeah um i mean at the
it's almost there isn't that big of a
difference at the end of the day right
whether you
think of them as separate environments
or as one big environment with different
rooms in the in the environment there
isn't that big of a difference
um i think it's important for the agent
to face a lot of different situations so
you can do that either by designing a
lot of levels or by
just procedurally generating it and
having the agent learn that distribution
um
i i do think
if you're specifically interested in in
this type of generalization then it
makes a lot of sense to keep some
holdout levels that are not from the
same distribution as the training set so
that's something we're not we don't have
in crafter
because that's not the specific focus of
the environment it's more like supposed
to
evaluate a lot of different
abilities and you need a generally
capable agent to do well at crafter but
if you care about specific holdout
generalization then you should probably
have a test set of environments that are
different in some aspect from the
training distribution
and when you're like one of the things
with benchmarking too is i mean i
imagine you want to pick a task that's
hard enough that current ais can't beat
it or can't can't do as well as humans
but still easy enough that humans can do
it or at least that's compatible with
human abilities can you tell me a little
bit about crafter and where it falls on
that spectrum and sort of what what
current state-of-the-art rl can do with
crafter where human ability is yeah it's
that was a pretty fun aspect of
designing the environment actually
because you have to balance the
difficulty right you want to see some
learning progress like if it if all the
current methods are just flat then
it's a bit hopeless you know like maybe
we should
work on simple environments first
um on the other hand of course if it's
too easily solved then it's not not an
interesting benchmark for pushing
pushing performance further
so
i did a lot of testing of the game just
playing it myself and seeing how
difficult it was
and then also training some agents and
seeing where they get stuck it's
designed to be
pretty challenging but possible to solve
for humans
so if you played it for the first time
there's no way you would solve it but if
you practiced for
maybe for a day
you would be able to solve it most of
the time
right i think it's important for rl
benchmarks even if it's
you know crafter is completely
unrealistic in terms of the visual
appearance right if we compare it to the
real world um but even for games i think
it's important you know we want to
benchmark our rl agents on things where
humans also have some learning potential
on
right so if you think about a memory
task then
or like any any task really we want it
to be something where a human starts
as like oh this is hard
right and then after practice after
enough practice the human score should
be able to go up that means there is
some actual learning happening
and it's an actual it's a challenge for
a human now you're reminding me of a
question i had looking at the paper too
because like crafter is it looks a lot
like minecraft
it doesn't look exactly like minecraft
there's some important differences
what are some of the little kind of the
key differences and what do they tell us
about what's hard for these agents so
it's all in 2d
um it's still visual inputs but crafters
in 2d and minecraft is obviously in 3d
and that helps you reduce the horizons a
bit and also make the perception problem
a lot easier right like in crafter
perception isn't really that big of an
issue
um and
and of course 3d perception is an
unsolved problem and we would like to
extract objects and have them be
temporarily
consistent and
interact sparsely and do all these
things and i think that's a really
exciting research that i'm also involved
in
to some extent but
at the same time there are these really
long horizon behaviors that maybe
require memory or long horizon credit
assignment right like yeah going through
the whole technology tree to get a
certain item or
um
there's one that's not even very deep in
the tree which is just to eat a fruit so
but you have to
plant the
sapling first and if you wait for like a
thousand steps
and
and that's pretty hard to discover
um especially if there's creatures
around it and if they shoot at your plan
then it's got it's gone so
uh so there are a lot of these
um challenges in terms of you know long
term horizons of one uh one form or
another and generalization to new
situations
that are present in minecraft but
they're kind of overshadowed almost by
the
3d perception problem
and so crafter lets you work on those
without having the perception problem
and as a result you can train agents
much faster to
to an interesting performance level and
make improvements on different
algorithms yeah i guess the wish list is
long to get agi so you got to parse it
out into into sub components um
yeah i mean and this is exactly why i
think benchmarking is so important today
and i'd love to get your thoughts on
this aspect i mean like i i've kind of i
perceive that i've seen the field of rl
move towards more and more emphasis on
benchmarking as algorithms have gotten
so good that you can kind of point them
in a particular direction and like
they'll they'll pretty reliably be able
to master task with some iteration like
you know you define a benchmark and then
within a couple years it gets smashed or
even a couple months and so in some
sense we're limited more by our ability
to like point these things in the right
direction to choose interesting tasks
for them to work on which arguably is
what we see with language modeling too
do you agree with that framing or um
yeah yeah absolutely
there is
i mean the defining the right benchmarks
is that's posing the right questions
right that's really important for
directing the research community
um
i mean
they you're saying well they get solved
in a in a couple of years
they probably do because they were
designed to get solved in a couple of
years right yeah i could easily design a
benchmark that won't get solved in the
next 10 years
but
yeah i mean there's a sweet spot of
cause of
something that's that seems promising
enough that we can make some progress on
now
and
and yeah i totally agree it's important
to set set up the right benchmarks and
also to make them easy to use and make
them
accessible to people because
um some environments need a lot of
compute or they are really really
difficult to set up or can't be run on
some
uh some clusters and
that's all like researcher time wasted
all around the world
um or there's a license um requirement
or something i thought it was really
awesome that that mujo co is free
um for everybody now because i
people i've worked with at the
university of toronto here
um you know they've they've wasted days
just trying to get the license set up on
different servers and all that so
when you talk about you know you can
define benchmarks that won't get
shattered in in like 10 years or
something like something like that um
yeah i do feel like there's something
interesting intrinsically about
benchmarks for that reason it seems to
tell us something about what the current
state of the art is it seems to tell us
something about the frontier um so
through that lens like how do you think
about crafter what does it tell us about
what's easy today in rl what's hard and
the achievements that are going to be
unlocked next in a crafter-like
environment i i think generalization is
actually a pretty big aspect of it
because
a lot of the achievements can sometimes
be unlocked if you're if you're pretty
lucky
learning from that to unlock them
reliably is really hard and we need to
put in the the right inductive biases
into our agents
to
to have this generalization ability
and the inductive biases piece like
i at least i i take it to mean when you
look at a screen and there's like a big
mean looking skeleton thing that's
shooting something at you um you kind of
as a human you know like oh i've played
games before i've interacted with
hostile looking creatures i can infer
that probably this is like a dangerous
thing that i should avoid um whereas you
so you don't actually have to learn that
from getting hit a bunch of times by an
arrow
is is that kind of the aspect that
you're you're shining a spotlight on
there i mean of course humans have more
prior knowledge when they go into a game
and play it but i don't think that's
actually
that
that big of a problem um
first of all when i watch people play
crafter for the first time they all get
hit by arrows couple of times
maybe not because they want to but
but because they have to learn how to
avoid them
and the agent also learns pretty quickly
that that's a bad thing
and and so
of course the agent has to catch up with
those human uh priors
in the beginning but then i yeah i mean
it's still fair game if you give it
enough uh experience to interact with
the environment the arrows is probably a
bad example because it's it's such a a
direct bit of feedback but like
notoriously moctezuma's revenge when you
look at atari games is an example of
where those biases or those priors
really kick in where there's like a key
that you got to get and then the key
unlocks a door and if you're a human you
see a key you see a door and you can
kind of put two and two together thanks
to your priors whereas an ai is just
like you're basically putting a big
barrier between the cause and effect
so is is that something that you think
would would kind of play similarly with
an environment like crafter with some of
these tech tree things where you have to
unlock
to some extent perhaps it depends how
much the human knows about the
environment going in right if you tell
them these are all the possible items if
you show them the tech tree that helps a
lot
right if not
in in just in terms of wall clock time
like training an agent on my gpu on one
computer and having my friend play on
the other computer and seeing who gets a
diamond first um i mean the diamond is
pretty hard for trained agents right now
um i've seen it happen but definitely
like you know below one percent success
rate so it's a good challenge to work on
but
um i i wouldn't say that the human
necessarily finds it without knowing
that they they can and knowing what all
the things are in the environment all
the different tools so
um yeah i mean it's an exploration
challenge and
perhaps if we're talking about montezuma
which is actually a really hard game for
humans it's it's actually pretty
challenging um
but maybe not so much because of the
high level
reasoning that's needed and more because
you have to be very precise and not get
shot by the laser beams and
you know jump over all these over the
skull on the first screen there's only a
pretty small gap and you have to time
everything right so it's kind of like
oddly challenging for the wrong reasons
compared to what we were trying to use
it for in rl but
nevertheless it's still a good
environment to test exploration
and even though those are things that
maybe
the human doesn't really have to explore
as much
you know there's other things that
humans have to explore as much so as
long as you're not putting in montezuma
specific biases into your algorithm
um
i think can still be a useful benchmark
so
i think it's it's better to or more
important to
um
to prevent some common failure modes
like failure of generalization right
like attending to a single pixel value
that
that that's a failure mode of
generalization and we want to design
benchmarks where that's not possible
because there's enough randomness
happening in the environment right maybe
just a little bit of observation noise
probably helps in this particular
example
um but also memorizing an exact room
layout that may not be what we want uh
so then we can randomize that with
procedural generation i think we just
want to be a bit careful about thinking
about you know what is the easiest way
to solve this environment are there any
shortcuts there that i didn't think of
and then make sure that we can get rid
of them in the environment what have you
observed about agent behavior especially
for let's say like cutting edge
architectures in the crafter environment
what kinds of things do they tend to
learn that have surprised you
um
yeah we have a
section in the paper actually talking
about these emerging behaviors um
so i i haven't seen any degenerate
behaviors
um so that's good
and
there are a couple of interesting ones
so for example there's a day and night
cycle over time uh you know after maybe
a minute of play it starts to get dark
and then more monsters are coming up
and it's if the agent is
good at fighting them it could survive
but it's easier to just hide somewhere
in the cave
so the agent actually ends up learning
to
search for caves on the map
and then open them up dig through the
wall and then close it and then sleep
there oh wow so that is a lot of
planning
yeah i mean the the agent i'm training
they are using this particular one
that's a dream rv2 agent so it's using a
world model
but it's not doing online planning in
the situation in the moment it's only
using the world mall to generate
um
as a replacement for the environment to
generate more experience and then just
train a
fully amortized policy on that so when
you're running it in the environment in
the actual environment you're just
feeding in the observation into the
world model and then into the into the
actor and you get an action in a single
forward pass so it's all kind of
distilled
and i guess you do see uh i guess fewer
models making it all the way down the
tech tree like that is still a
pretty uh serious limitation for current
systems yeah it is so we ran a bunch of
baselines
and
they can yeah so it is definitely
challenging
to get high success rates on many of the
achievements
some of the achievements are pretty easy
you know there's like 203 where even the
random agent gets them decently often
but most of them
you really yeah i mean current methods
sometimes find them
um but
kind of more through luck and then they
don't learn to reliably solve them
because they don't have this
generalization ability to um
to see it once or twice and then apply
it to the next situation next time
whether
the mountain is on the other side of the
map and you know that there's a forest
here instead of the lake and then but
you know the skills should still apply
if there's still an iron ore here then i
can still go and mine it
so
i i mean it makes sense to also study
generalization in most complicated
setups sometimes
but at the end of the day all these
things are to improve sample efficiency
and we can study
you know i'm personally i'm not that
convinced that for example to look like
transfer learning is not
necessary as a separate evaluation
um setup
from just training in a complicated
environment where there are many
different things for the agent to do and
it has to transfer from previous skills
to new ones and if you measure sample
efficiency by saying you know you're
only allowed to do a million steps here
then you can actually evaluate that too
and
there are reasons where you are like
cases where you want to evaluate
transfer learning separately like seem
to real right that's a very
practical
uh scenario we want to train in a
simulator and then have a real robot do
something in the world um but maybe you
know training on one atari game and
transferring to the other and those kind
of those kind of things i don't know how
um yeah i think we don't really need
that as a setup that much that's really
interesting because it does kind of
contrast a little bit with what we're
seeing in deep learning more generally
like in in language where
it kind of seems like transfer learning
i guess you could cast the problem in a
similar way or you could think of it as
being a similar problem where for
example when you're training a system
like gpd3 on a huge data set
you're kind of doing a similar thing
where some parts of that data set will
include translations between english and
french some parts of it will include
writing novels and like the to some
degree if you think of gbt3 as an agent
an embedded agent or something uh you
end up getting effectively an analog for
for what you're seeing right now with
with crafter like would you agree with
that uh
yeah that's a cool perspective um and i
i also think in nlp it makes sense to
have
have the setup
on top of what you're describing within
just you know
fitting the model to a big data set with
all kinds of
tasks hidden in the data set somehow
additionally we also want to be able to
you know take a general model and then
specialize it for a specific application
it's just that
if you're training an agent you're
usually you're in an online learning
setting well i mean the terminology some
people
mean something else by that but at least
you keep getting new data and you can
you can keep exploring the world you can
keep doing like better at the task
so
um
you know it's like
in in application domains we more often
have this divide between there's a big
pre-training phase
and then and then we want to do
translation with it and then if we want
to do something else with it we probably
wouldn't specialize the translation
model further we would go back to the
general one
like do you think that um this idea of
scaled models and and basically you know
foundation models as they're sometimes
called the gpd3s the the megatrons etc
um
do you think that there's going to be an
actual like
merging of that with uh with deep rl
let's say constructing very complex
world models ontologies and so on or
like right now is is rl just like really
focused on its own thing is this
generality and
procedural generation of environments is
that kind of more do you see that being
more the focus
that's a great question um
i i thought about this and
it's like yeah why i mean
intuitively yes we should just train a
giant wealth model that learns
how
how the world behaves
um and then
well
you can't just use that if you want to
do control with it you have to
specialize it because the model doesn't
know it doesn't distinguish between the
controllable
randomness in the world and the
uncontrollable randomness well let's say
you have a video of or you have a scene
with multiple robots in a factory and
your job is controlled as one robot
um you so you can't immediately use this
model for planning
otherwise you you would end up being too
optimistic you would kind of think that
all the other robots will help you do
your task but they won't
so so you need the specialization steps
and there's
also some some ideas around already
for how to do that but they haven't
really been scaled that much i think
and
on the other hand um
why don't we really have foundation
models in rl yet
i think one reason is that a lot of the
evaluation domains are still pretty toy
um or maybe not toy they are challenging
in some ways but
they are also very different from from
the real world in some ways as well
so let's say let's say you train a
foundation model
um on real video
it might not be that helpful for solving
atari games right right
or perhaps even crafter even though
there's more interesting generalization
necessary there and
um
it's still visually very very different
from
from real world video data sets that we
have
and
one day we'll be able to train on all of
youtube and it'll have all the gameplay
of all the streamers on it as well and
then there will be some some transfer
there
but
um yeah i think
we will on one side we'll see the rl
community move more and more towards
visually realistic environments
and on the other side maybe then the
foundation models have a bit more of a
chance of actually being helpful
it does start to make me think about
that that conversation we had about
priors where you know you've got the
human being that leverages what a let's
say you go to starcraft 2 or something
and you look at a particular a
particular unit or structure in the game
and you can kind of get a sense for
what its purpose is or what gadget or
weapon it might have at its disposal
just because you've seen similar things
in the real world
again you know this doesn't seem like
the critical thing because these agents
can learn quite quickly how that works
but foundation models seem like they
might provide that kind of cultural
knowledge that prior that helps give a
leg up to these systems during training
i think so totally it's just that right
now
most of the agents are already not very
data efficient
so
that's why i think
having this
human prior isn't even that important
for a lot of the standard
evaluate like standard benchmarks at
least right there's definitely
environments where i mean you can always
set something up right and there's
probably some
there's probably a lot of real world
applications as well where this could
really help
um but at the end of the day current rl
methods are still quite data inefficient
uh in a lot of cases so
so then
catching up with the human priors is
only the first fraction of learning
do you think data efficiency is is a
really like one of the key next targets
for rl systems and we saw efficient zero
come out recently which was sort of
plated to the story i don't think it's
necessary i think it's interesting to
push data efficiency um but it's also
interesting to just push final
performance with highly distributed
setups
there is a trade-off there which is
the more distributed computing you do
the more complex your whole training
infrastructure gets
and the harder it will be to for other
people to replicate
so
um
[Music]
yeah i think it does make sense to focus
on sample efficiency and especially also
compute efficiency
develop or at least you know have a bit
of a of a pull towards methods that are
easy to run by everybody
uh doing research in nrl
because then
you know if i publish a method that
other people can use they will build on
top of it if it ends up working well
and
and then there's a lot more progress
than i could create myself as an
individual researcher right so that's
why i'm actually a bit skeptical of
these super large distributed rl
um demos or showcases because it's like
i mean
there have been enough of those now to
to know that with a huge amount of data
we can solve a lot of things with rl
right um
that almost nobody can replicate
but i don't know if we have to do our
research for new methods at this scale
right like probably we can do them at a
single gpu scale as well
and the
the findings there will transfer to the
highly distributed setting as well to to
a good extent
and do you think i i don't know if like
how much time you spent thinking about
efficient xero but but do you think that
this idea of as i understand what they
really did in the paper was
trade data for compute like really kind
of lean into having an agent imagine
different scenarios at each step like
really try to try to model very hard and
think very hard about each next step
as opposed to getting a ton of data and
in that way sort of imitating some of
the things that humans might do if
you're learning how to play soccer for
example like you know you've you've
kicked a ball before you kind of know
what it what it might do and you can
model forward like okay if i kick it
this way i don't know mu0 already had
the mu0 reanalyzed variant that reuses
the replay buffer many many times
um
so i think it's like a ratio between
environment steps and training steps of
99 to 1 or something
um
so yeah you're spending a lot of compute
um
to improve the model and that's
generally what we're seeing with weld
models so for dreamer as well
um you can base you can just crank up
compute and your model will get more and
more sample efficient at some point the
question is do you want to wait that
long to do all that training right
but yeah especially if you have
if it's a real world application where
you can't speed up the simulator anymore
then that makes a lot of sense or if you
have enough accelerators that you can go
distributed for your training
um or at least multi gpu or train on
like a slice of multiple tpu cars
then you can be a lot lot faster both in
like you know
you don't suffer in wall clock time and
you gain a lot in terms of data
efficiency
in the environment
i think the most interesting part of the
efficient museum compared to museum was
that they
integrated representation learning into
it
because that always seemed missing in
mu0 and it's one of the reasons mu0 is
even though it's achieving really high
performance asymptotically it's not very
data efficient
right yeah yeah
yeah so
yeah that just and i think they tried at
the time but now somebody has done done
it quite well and and tuned everything
nicely and
um yeah so it's a cool a cool method i
i'm still
thinking or i'm still a bit on the fence
of whether the online planning is really
necessary for atari games
um i think there's some real world
situations where you need
real-time planning
just because
the world is so complex that it will be
really hard to generalize to a new
situation
um and and so you want to do some extra
compute on the new situation
um
especially for non-stationary objectives
like exploration where
the objective of what is new changes all
the time every time you collect some
data so then you want to do replanning
um like in our plan to explore paper but
i think i mean there we also only used
offline planning so
online planning in the moment i think
makes a lot of sense for
really like environments that are so
complex that it's hard to generalize
and and for non-stationary objectives
like exploration
for just maximizing reward in atari i
think you might be able
to get away with a much simpler
algorithm that doesn't use such research
yeah and yeah that does make sense and
it's one reason i guess to be kind of
cautious about extrapolating too much an
agent's performance on like atari games
uh to the more general complex settings
but i guess one thing this does make me
think about you know you mentioned this
idea of current approaches leaning more
and more into compute over data so
getting a lot of data efficiency at the
cost of large amounts of compute
do you think there's anything to the
idea that our progress towards something
like agi down the road uh really becomes
the story of increasingly abundant
availability of compute like is is all
our algorithmic funny business really
just a big dance that we're doing atop
this kind of exponentially rising
tide of just compute availability
compute is very important yeah so
i would
in most cases i would take compute over
algorithmic novelty
um not in all cases though because there
are certain things like for example
hierarchical planning
that you would need exponentially more
compute for and so
then you can have a hierarchical
structure to help
um to help kind of uh break down this
exponential complexity
so
yeah i mean
neural nets and deep learning works
insanely well and scales insanely well
um and
and most papers that are being published
they won't be needed once we have the
next gpu generation available um but
there's also a lot of
really interesting really hard
problems that people are working on
where maybe there will only be a
breakthrough in generalization every
couple of years but
we still need all these intermediate
papers as stepping stones
and and i think we still need some
[Music]
some of these inductive biases
um
yeah i guess cool because at the end we
will end up with i think a quite simple
set of
rules of very general rules
but like you probably want to learn a
model of the environment you probably
want it to be causally correct so you're
not biased in that way
um
you probably want to do planning with it
in some way
but then there's a lot of details that
where we can just
throw in the the newest state-of-the-art
neural network design the newest
architecture and then and then run it on
a lot of machines and
yeah and and i do think like especially
this is a bit separate but also to to
the question you asked um cycling back
to that
especially with model based methods it
really seems like you can trade
compute for for data efficiency
to a pretty
pretty good extent
um
not so much with model 3 methods um even
if
you have a big replay buffer for
everything you've seen already that
experience will become more and more of
policy so will be less and less helpful
in improving your current decisions
but learning a world model on the replay
buffer lets you generalize and fill in
the trajectories that you haven't seen
yet
and
you know it's not really clear how well
that would fill in these things you
haven't seen but it seems like it's good
enough that
you can really crank up the compute and
do more and more training both for your
model and your policy and and get more
sample efficiency you do see
hierarchical learning as sort of like
one of these core nuts that will have to
crack on the way like there's just
compute doesn't get us around this
it'll be hard there's one question of
whether you want the temporal
abstraction to be explicit in some kind
of structure that we understand as
researchers or not
and there might be clever ways that we
haven't found yet for learning this um
implicitly somehow
now if you want to do explicit planning
with it then you need to have access to
that structure
so i think it would be very useful if we
could
do that explicitly
right
but there is probably also a way without
that
yeah it kind of makes me think of like
you know convolutional nets obviously
that's us encoding a prior that like
really saves the the deep learning
system a ton of a ton of time and kind
of accelerates its learning this sort of
feels like it's in the same vein like
what priors are we going to bake in
um
i would imagine the counter argument to
that is you look at transformers today
and how essentially they're starting to
replace convnets for vision and they're
this very generalizable architecture and
that generality seems to be a feature
and not a bug like in other words it's
the very fact that it can be used for
you know vision as well as text we're
seeing more and more multimodal systems
that seem to be able to benefit from
both things like does that play into
into this picture at all i wouldn't
necessarily say they are replacing
vision systems they are maybe replacing
the high-level vision systems right um
because there the locality and the
weight sharing isn't isn't as useful
anymore
um at lower lower levels i think
convolutions are still
still the way to go although this keeps
changing every day so yeah that's
kind of hard to to keep track of
everything but
um
yeah i mean
transformers are
yeah at the end of the day it's just a
computationally efficient architecture
and it lets you learn long long-range
dependencies
and if we
go back to a question of temporal
abstraction well a lot of
architectures we have are already doing
some form of implicit temporal
abstraction
you just think of a gated rnn like a gru
there's already a gate and if that gate
is closed then the activation is copied
over to the next time step
and and so you can copy pretty easily
for very long horizons
and and so then you've preserved your
information you can access your memories
you can backprop through the whole thing
it's you get good gradients back in time
because the the state didn't change much
um so your gradients don't uh don't
vanish or explode on the way back
and and that's
pretty good already if if you
engineer everything right and you train
it
you know you have good training set up
there's no bugs you can already learn
quite long term dependencies
transformers are pretty similar if
if we're talking about sequence modeling
and long term dependencies
it's a bit like
i guess you could think of it as a
neural turing machine
where you have some
you know like a
memory that you can write to and read
from
but the problem is it's really hard to
tell whether
a piece of information will become
useful in the future
i don't know yet so you don't want to
learn to write it's really hard to learn
writing relevant information so the
transformer just writes everything at
every time step and then you still do
the read that attends back to the past
so all these architectures are already
pretty good at learning long-term
dependencies
um but they don't let us use them for
abstract planning
and
so i think there is a challenge there in
making these architectures discreet
right like you have a sigmoid gate
in a gru
so the gate is never closed on it's
never open it's always something in
between and
it's very nice if you want to train your
models with with radiance right because
then you're trying out everything at the
same time to different amounts and the
gradients can tell you do a bit more of
that close the gate a little bit until
eventually you learn that it should be
closed
so it really helps optimization
but if you want to do um abstract
planning with it
then then it would be it would be
helpful to just
have these be discrete decisions so you
know you know i can make a step now for
15 steps right i just do one computation
and then that will account for 15 steps
at the lower level or something like
that which would also make these systems
more explainable right i mean like that
seems to be a nice benefit if you could
have an explicit architecture like that
from a safety standpoint
probably yes um yeah i think it would
help i'm not thinking too much about
explainability at the moment because
we're still so far away from
solving the kinds of problems i would
like to solve with these methods
but yeah i mean it is a useful structure
and
to your question about compute versus
algorithms i think
it's
it's general enough of a structure that
it won't be replaced by compute too soon
that's a that's an interesting aspect
too i mean you alluded to we're not
we're not terribly close how far do you
think we are uh from let's say
this is always the difficult question
we're back to defining generality but
how far are we would you say from
systems that broadly would be as
competent as a human across let's say a
wide range of tasks if i can get away
with that
are we talking about an embodied system
or
are we okay with just a language model
that we can talk to on the on the
computer um i mean i i guess so in my
world model those two would happen
around the same time because if you have
a language model that's as capable as a
human you could accelerate it and
accelerate development and then
presumably like excel like accelerate
the development of embodied systems like
having a researcher on steroids
but i could be of course very very wrong
perhaps i mean
i
i mean first of all i think the
progression will be gradual
it won't be overnight now this model is
you know solves the problems all of them
and before i didn't do anything so it's
getting better and better and
it's getting more useful along the way
and it's probably a long time until
language models are
good enough to do research for us
um
because
i mean i'm yeah
there might be some
i'm not that deep into the nlp
literature at the moment but
i would assume that
the current current techniques we have
are still pretty much as
sure they can generalize in some cool
ways but
i might tell you how to make up new
information and verify that that
information is actually correct and have
some kind of um
um
some kind of logical reasoning emerge
from just
from just completing text which to some
extent does happen but i think it's uh
you know maybe even one of these things
where you want to build in some some
prior knowledge to make sure that the
logical reasoning is sound and it's not
just 80 of the cases
where it has seen enough training
examples for
so
at the end of the day that that would
help you with generalization and then
that's really needed if you want to use
it as a scientist
right it's supposed to make up new
new knowledge that's correct
so uh
it would have to have some really good
generalization capabilities
and
yeah i mean
well we're usually just testing these
language models within the distribution
of human text right that's already
there on the planet and it generalizes
well in that distribution or it gets
better and better at least but
maybe there is another
algorithmic hub needed
to generalize outside of that
distribution and be a scientist yeah
yeah i think that the temporal
abstraction also the
kind of more logical reasoning
um and
yeah
much better generalization capabilities
i
yeah i mean it's like we don't really
need them if we train on on all of the
internet of text
um if we want to generalize within that
distribution and i think that's where a
lot of progress is happening right now
um i'm
i mean i'm sure you know there's so many
good people working on it um we'll we'll
definitely figure it out
like absolutely probably within our
lifetimes too um
but maybe not in the next 10 years
yeah well yeah and that's the thing
right this is like i guess all my all my
takes about compute and and generality
from language come from that bias um so
so it really is anyway very informative
to uh to see a perspective from the rl
side which is something i've been trying
to do a little bit more lately too
because it's the two communities are
surprisingly non-overlapping at this
point and hopefully that'll change
yeah yeah that would be cool i think
there is a bit i mean
world models are bridging the gap to
some extent because
wealth models really just learn a good
sequence model ideally
one with a good representation as well
so you can plan
in your representational space you don't
have to generate new images during
planning
um but yeah other than that it's really
just it's just a good sequence model
and
and it's conditioned on actions sure but
that's
that won't hurt hurt anybody so
yeah i think there is a lot of overlap
and there is a lot of
what i'm really excited about is that
aurel is really starting to
be at a uh be in a place where
you know a couple of years ago
in supervised learning there were all
these tricks and like you know this is
how you're you're learning right
schedule and you just use this
normalization layer here you can try and
keep your networks and this all kind of
worked
um whereas in rl none of those things
worked they would all just destroy
performance because it's already the
training process is already so noisy
that
training big models and training them in
ways that
that they actually fit the data well
could be a problem and yeah yeah
collapsing in some way and everything
was pretty brittle and unstable
whereas now
it really seems like we're more and more
in a place where
we can just import these functions
from from our favorite deep learning
frameworks and
and scale things up and they scale
roughly how we expect them to scale on
the safety side so uh there's obviously
community of people quite worried about
ai alignment risk and even up to
catastrophic risk we've had quite a few
of them on our podcast from deepmind
openai that sort of ecosystem
i'm wondering you're obviously more on
the capability side but i'm wondering
like what kind of uh exposure
you've had to that that kind of
ecosystem whether they overlap like yeah
what's the almost the cultural overlap
there is there any
yeah yeah i have a lot of friends um
who've who know a lot more about safety
ai safety than i do
some of them working on it as the main
topic as well
um
i i do also think it's a really
important topic
um
it's just i guess yeah i mean we might
get there
fairly soon with rl that it really
becomes relevant because right now
not a lot of rl is deployed in the real
world yet yeah right like and i think
that's the current
transition that we're going through
um where
there's there's startups using
reinforcement learning um
there are more and more things that uh
that actually work well enough that we
can
industrialize them
and then
um safety considerations
i think become more relevant
but yeah i haven't
spent that much time thinking about the
the safety of my atari agents yet
because i i want to play i want them to
play minecraft next and then afterwards
maybe something more in the real world
so
um
just really appreciate the rundown not
just on your paper your work but the
space is a whole it's been a really cool
opportunity to talk to somebody who
knows a lot more about rl than i do so i
really appreciate it
thanks a lot for having me that was
really fun


Alex Turner - Will powerful AIs tend to seek power?

[Music]
hey everyone welcome back to the towards
data science podcast now today's episode
is somewhat special because we're going
to be talking about what might be the
first solid quantitative study of the
power seeking tendencies that we can
expect advanced ai systems to have in
the future now for a long time there's
kind of been this debate in the ai
safety community between people who
worry that powerful ais could eventually
displace or even eliminate humanity
altogether as they find more clever
creative and dangerous ways to optimize
their reward metrics and people who say
that's terminator baiting hollywood
nonsense that anthropomorphizes machines
in a way that's deeply unhelpful and
misleading
now unfortunately recent work in ai
alignment and in particular here a
spotlighted 2021 nurip's paper suggests
that the ai takeover argument might be
stronger than many had realized in fact
it's starting to look like we ought to
expect to see power seeking behaviors
from highly capable ai systems by
default now these behaviors include
things like ai systems preventing us
from shutting them down or repurposing
resources in ways that are pathological
and serve their objectives and even in
the limit generating catastrophes that
would put humanity at risk and as
concerning as these possibilities might
be it's exciting nonetheless that we're
starting to develop a robust and
quantitative language to describe ai
failures and power seeking which is
exactly why i'm so excited to talk to ai
researcher alex turner the author of the
spotlighted nurips paper on powerseeking
about his path into ai safety his
research agenda and the future of ai on
this episode of the tours data science
podcast
[Music]
i'm really excited to uh to have this
conversation actually you know it's not
every day you get to talk to the the
author of a nurips highlighted paper um
so we'll let alone one on alignment too
which i think is
this kind of feels like a first and a
big moment for alignment not to butter
you up too much here but anyway it's
kind of exciting i'm sure we'll we'll
get into it i'd love to talk to you
though first about your interest in
alignment research because i think every
alignment researcher has a pretty
interesting story that kind of leads
them to where they are
what's yours like how did you get to
this space
so i it all started in 2014 or i suppose
it didn't start in 2014 i was scrolling
through my facebook feed and i saw some
hyperbolic news article about how how
scared elon musk was of ai and i i just
i i rolled my eyes and i was like elon
you don't know what you're talking about
ai is so great and i just kept scrolling
through the feed um
and so i thereby lost i think about four
years of work i could have done on
alignment uh just scrolling past that
and i didn't really seriously think
about the issue until until late 2017
when i
read
nick bostrom's super intelligence
uh his book super intelligence i just
sat down with it for a week and i i
thought about it seriously and i
realized that the arguments made a lot
of sense and i looked around
i looked around online for the best
counter arguments and there didn't
really seem to be very many good ones
that had you know actually read the work
um seemed like most it was mostly just
kind of bouncing off the way i had you
know just looking at the idea and it
seems a little kind of out there or
sci-fi and not not really engaging with
it anymore um
and so i became pretty convinced that
this was
you know worth working on that it was
worth uh worth developing more in
figuring out and i wasn't working on
alignment at the time i was in the
second year of my phd program i'm now in
my final year
my my original advisor wasn't really
that enthusiastic about alignment work
so i changed changed up my graduate
committee to have a better fit i changed
advisors i got some independent funding
from the long-term future fund to
to buy out my teaching so that i could
work
on alignment for my phd
so ever since then that's about three
years ago i've been working with the
center for human compatible ai at uc
berkeley
where i actually will be going as a
postdoc or research fellow when i
graduate this year
and i've also worked with the future of
humanity institute
just this last summer
and actually to your point about you
know you've lost four productive years
of research time since uh since you
first kind of bounced off the topic
it does make me wonder a little bit i
mean alignment research seems to have
evolved quite a bit in terms of its
focus there seems now for example to be
a lot more emphasis on prosaic ai like
ai as it appears today deep learning
reinforcement learning as a potential
vector for agi development whereas it
seemed like maybe circa 2014 there might
have been a little bit more uncertainty
i wonder i mean do you think that the
fact that you pivoted into alignment
research when you did meant that you
ended up focusing on something very
different rather than maybe getting
pigeonholed in a less sort of prosaic ai
oriented direction
maybe i don't think it really affected
my research direction a whole lot i've
worked more on theory i've worked on
understanding like the general case of
across a range of different procedures
by which we might get to ai
our agi what's the going to look like
what properties we'll have will it tend
to be aligned will it tend to be power
seeking
how will it tend to affect us and this
research i think is somewhat independent
of the recent success we've had since
2014 in deep learning
but for many other people i think this
would be this could be true um although
you know you might hope that you'd be
flexible enough to notice a new trend
and focus more of your research
attention on it
and just before we get into the the nuts
and bolts of the power seeking argument
and research that you've been doing i am
curious about that nick bostrom piece so
do you remember a particular argument
that changed your view from like oh this
is an elon musk thing pay no attention
to hey this is something i should really
be focusing on like was there was there
something like that or was it more an
overarching kind of
distilled sense of what the book was
saying
rather than it being one particular
argument i think i had assumed that
since it wasn't already
consensus it was a big deal worth
worrying about there had to be something
wrong with the argument but when i went
through the argument
and especially when i went through the
lack of careful engagement with boston's
arguments
uh i realized that this wasn't the case
that there wasn't some detailed you know
uh rebuttal or reason to not worry about
it but rather it was a pretty good
argument
and uh people weren't that worried about
it on average uh and so this
this really made me kind of detach from
that prior expectation i had that well
if you know if it were such a big deal
why wouldn't i hear about it and now i
don't expect that to be true uh i don't
expect that to be true in a lot of
realms including this one yeah like
i i think there
is like
not a whole lot of and i think the
incentives aren't there for that right
now the incentives aren't there for some
for
for people to be able to go into
alignment research if it were really
that important and then kind of profit
off of it personally uh career-wise uh
be recognized as doing good research if
they fill an important gap in the
alignment literature in part because
alignment isn't about the things that
we're building today
somewhat it's mostly i think most of its
importance is tied to things that we
haven't built yet which means it's hard
to hard for people to to like do
something and then everyone recognizes
hey this made a big practical impact
because that practical impact is an
expectation in the future so i think
there's a lot of reasons that this
problem is kind of is under focus right
now that may be one of the important
reasons why is it also like the
subcategory of sort of so-called
alignment research because it seems to
me that there are sort of two camps like
one scene one camp is more engineering
focused like we'll we'll build a gpd3
it'll have behavior that we don't like
and then we'll tweak the model and we'll
call that alignment research whereas the
other camp seems more focused on the the
maybe more theoretical kind of
existential risk the idea of this one
shot that we have at aligning a
superhuman agi that sort of thing
like do you see that clean separation or
do you think that actually that in in
the limit there's sort of a continuing
between the two and it's not obvious how
to separate them
so are you asking whether i
differentiate these two kinds of
research this modern empirical research
on like large models like gpt3 do i
differentiate its legit alignment
legitimacy from more theoretical work
i'm going back to your point about the
essentially the economics of alignment
research so if it's the case that i can
do research alignment research on
current systems like gpd3 and make money
off that like i am sort of adding value
because we can squeeze more value out of
the system that is
aligned in that engineering kind of
sense that creates an economic gradient
towards alignment research and if that
kind of alignment research
pushed to the limit leads to existential
risk mitigation then that seems like a
good thing but if existential risk
mitigation is just a completely separate
theoretical domain if it doesn't if the
people who work on it are just different
people from the people who work on sort
of the engineering side then then that's
much more concerning from an economic
standpoint if that makes sense
right so i definitely i think there's
very useful empirical work we can be
doing today i think it's hard to figure
out what exactly that is if you look at
redwood research
a new lab a new alignment
lab
that is working on
fine tuning
gpt3 i believe so that it doesn't
it doesn't
like
it doesn't ever describe situations
where there's
violence
uh like can you make this robustly be
true that they just won't talk about
violent
things
that's my recollection at least of their
research
and
whether you can do this or not with
systems or with models about as
intelligent
as gpt3 tells us important things about
the parameters
of the alignment problem so we could
live in different worlds with respect to
robustness with respect to how
generalization works across human like
concepts like can
can the system just totally grok the
concept uh can it can that be natural
for it uh and can we figure out if
that's the case today uh and i think we
can to some extent so there's
thoughtful empirical work that i think
is very useful for acts risk
um
and but i think by default that's not
the kind of work that's incentivized the
kind of work that's incentivized is
going to be very legible very
uh kind of
uh
clear to modern researchers that hey
here's something that's happening right
now here's what we did about it i don't
usually expect that to line up with
the
longer term or the future at least
important alignment empirical work like
i think that the the main factor
governing
the usefulness of redwater's work isn't
the direct
ask of the ai system that the ask of
don't talk about violence
it's
what it tells us about how robustly it
can learn these concepts
uh and
not necessarily with the intention of
just getting it too straightforwardly
we've got gptn or we've got uh some like
we've got some model based superhuman rl
system in the future
not necessarily using that to directly
just make it grow human values
but rather just informing
our theory of how alignment works and
using that to build better approaches
you could try just getting into rocket
in the future in like a very optimistic
world i mean i guess that's the thing
you could try and maybe it's a way by
which alignment turns out to be pretty
easy for very intelligent agents but i'd
be surprised if that were true
yeah i guess i was thinking of it in
terms of like
you know if you wanted 100 certainty
that gpd3 was not going to put out some
you know violent passage that would
imply that
the
objects that gpt 3 thinks of the
abstractions it uses the the ideas that
it holds in its head so to speak are um
that we can put a clean box around those
and say hey like you have this idea of
violence and that idea is roughly
correct or roughly maps on to what
humans think of as violence like don't
do that thing and that would seem to be
much more x-risk flavor that anyway i
guess i'm just i'm curious about this
because i think it informs how how
people think about what domain to enter
because like there are a lot of boxes
that have the word alignment on them but
many of them seem not to contain the
kind of work that leads to existential
risk mitigation and i find that to be in
any way an interesting paradigm
on to more extra flavored stuff maybe
your own work which i think is really
really fascinating so can you this might
be too much to ask but can you just
kind of lay out the case that you make
in the optimal policies tend to seek
power paper in this paper we uh we look
at something called instrumental
conversions or simply the tendency of a
range of smart agents to take similar
actions
so if you want to go buy something no
matter what in particular you want to
buy you'll probably get in your car to
uh to do that and so it's this
preliminary action that's helpful for a
range of things you want to do
so some like alignment researchers like
bostrom and stuart russell
worry that
disempowering humanity
by seeking power you know staying alive
that these are
convergently instrumental actions for an
ai that they're a good idea for a wide
range of different goals
and previously this this has been a
little bit
uh controversial you'll see some other
researchers that are saying well maybe
it's just anthropomorphizing to imagine
an ai seeking power
so the main contribution to this paper
is grounding out this question in a
formal setting in the setting of markov
decision processes which basically just
means a setting where the ai can see
everything that's happening
takes actions the world evolves and the
ai has an objective
that takes in
the world state or how things are set up
right now and gives back a number so ai
is basically trying to run up its score
with respect to its objective
in a pac-man game the reward function
would be something like
the score
uh the points gained by an action out of
state so you you eat a pellet you gain
some points that's some reward
and so what we show is that
depending on the structure of the
agent's environment or the world around
it the structure of the game is playing
in some settings uh agents will have
different statistical tendencies across
their goals so previously there'd been
this informal question of
what will smart ais tend to take over
the world and this is pretty vague it
makes it feel like there's kind of two
uh there you could go either way you
could say well that doesn't sound
plausible or you could say it sounds
plausible depending on your
uh your taste
and your beliefs about how we'll build
ai
but this paper points out you ground out
the problem and since it's formal now
there's one answer in the settings
that we look at
and in particular when the agent has
when the world is structured so that the
agent has more options
by taking action a over action b if
staying alive for example lets the agent
do more things and reach more like
future possibilities
then also most goals will make the agent
uh will make it optimal for the agent to
stay alive it'll make it a good idea for
the agent to
stay alive and so we take this informal
debate and we ground it out in terms of
pretty reasonable math
and then
and then we can imagine what happens
when we're relaxing the assumptions
which i do in future work the main
upshot is
this is something we can talk about
formally
and the formal answer in the settings we
look at is that optimal policies will
tend to seek power in a pretty
reasonable intuitive sense and so i
think this should if if previously
you looked at this and you thought well
you know this doesn't seem like this
should really be a concern
uh i think this is just uh this should
be a pretty surprising observation
and there's several hedges i can get
into in several uh several details that
we'll need to look at uh this definitely
doesn't prove that uh super intelligent
ai if we build it that it'll seek power
over us
but i i think it's pretty suggestive
evidence that by default in the average
case we're looking at the as
incentivized to seek power just for most
goals you can give it as a statistical
fact i think there is a lot to unpack
here including the reaction to it uh
sort of both both positive and sort of
counter and i think i think there's an
interesting interesting discussion to be
had on that front but i want to zero in
on optimal policies and get a better
sense for what optimal policies are in
the context of this experiment could you
kind of elaborate on that
an optimal policy is one which maximizes
the agent's reward over time so it's
ones that maximizes uh how many points
the agent is scoring over the course of
uh over the course of its interaction
with the world
um and so in pac-man and optimal policy
for the
classic pac-man reward function would
just be one that runs up the score as
high as possible
do you theoretically like demonstrate
like okay this policy is optimal and
then what can i do with that or
so i i don't constructively take a
policy and show that it's optimal uh i
what we say is
in this kind of world where you you've
got more choices this way than this
other way um for most objectives you get
the agent
uh
the the optimal this can be an optimal
policy for that objective that stays
alive so we're not we're not we're not
examining one particular setup and
saying well this has to be optimal and
also it seeks power we're saying
generically
uh optimal policies uh for different
reward functions will
be seeking power the uh the caveats that
you might add to this so obviously
optimality of the policy for this paper
at least is one caveat are there others
that you want to flag things that sort
of limit the domain of applicability of
the work
so for this paper uh there's some more
caveats
we look at so we we assume that it's a
markov decision process which means
we're assuming that the world is fully
observable and you know the agent can
see everything that's going on all at
once and you might realize that the real
world is definitely not fully observable
many tasks we care about especially in
that embodied um like robotics
domain are not fully observable agent
only has like a camera feed for example
showing it its immediate surroundings
but not the whole world um i don't yeah
i can talk about these caveats uh what i
think about how serious these caveats
are
i actually that one i'd be curious to
explore if possible because i know that
one strategy that's been looked at is
intentionally blinding agents to certain
parts of the world and like in the hopes
that they would not be incentivized to
manipulate them um so i i'm definitely
curious about sort of double clicking on
that and seeing seeing what you think
so first of all i think that kind of
strategy is pretty broken i think if
if
you've if you're worried that by default
the system won't have bad incentives and
then there's this kind of like
patch you try to slap onto it without
without really understanding where the
bad incentives come from and uh without
really understanding how to how to
motivate the agent to do the thing you
actually want
um you're kind of relying on the agent
not being able to find a way around the
patch
under assumption it's
it's motivated to do so
um
so here i actually already have have
proofs
for the for the partially observable
case and particularly for just you take
any like computable environment that the
agent takes actions it gets back some
maybe incomplete observation maybe just
like a webcam frame
of the world uh and you can extend these
i extended these theorems to that case
where the agent can't see everything but
still certain actions will let the agent
do more things and you'll just see this
kind of enormous propensity
across possible asian goals to
take the power seeking actions so it
doesn't cover all environment classes we
care about i'm looking to work with some
people on extending these results to the
palm dp case and not just the
not not this arbitrary computable
environment case but i think there's
also some interesting theorems for the
partially observable markov decision
process case
but i think the moral is that this full
observability this full observability of
canonical crux it's not what the proofs
hinge on if you go in and look at them
it's just kind of a convenience
okay and are there so would you say
those are the main caveats to put on
this first piece of work then so partial
observability and then um
and then optimality of the policies or
those two things
so there i'd add a third and maybe a
fourth third
is
this talks about just generically
for most reward functions so you take
any reward function and most ways of
modifying that reward function by
swapping which states get which reward
most of its permutations will
incentivize power seeking
but it might be the case that we use a
you know featurized reward function we
we have different ways of specifying
reward that don't allow arbitrary reward
functions this might change the power
seeking incentives
this paper just kind of zooms out and
says well every reward function is uh as
possible as any other
uh in some sense like
so most distributions over reward
functions will uh if we have some
beliefs we're like hmm i think that
we'll probably assign a reward function
like this but maybe we'll we'll have a
different one well most permutations of
that distribution will also tend to
incentivize power seeking so there's
there's a good amount of uh these
streams are i think are pretty strong
but they don't cover particular ways we
might specify reward so i suppose
there's some loophole there well would
you imagine people might might counter
on that and say well um almost by
definition when we when we encode a
reward function in an agent
that's a very informed choice in other
words it's it's highly kind of um
it's a sort of low in in a sense low
entropy choice whereas you might have a
flat prior over roughly flat prior over
different reward distributions in your
work like what we're really interested
in is a very specific set of highly
tailored word functions that humans are
specifically choosing because they seem
like good ideas
i don't know if you find that to be a
compelling uh
bit of pushback
i don't really if you look empirically
when we there's many cases where we
think that the reward we've
specified is a good idea and it's just
not yeah like with coast runners and
open ai
they specify they they reward the agent
for you know
gaining points while completing a race
and the ai just spins in a circle
because it finds a way to to quickly get
these respawning point power-ups
so and this is a very simple setting and
so in in more complex settings we're
just gonna have a really really hard
time and so i don't find the argument
very persuasive if you look at it
empirically today it's just not that
true it's true that if there's some
information we're not just randomly
choosing a reward function but
since like it's not clear that we
actually know how to narrow it down in
particular whenever you run at these
alignment thought experiments where you
you formally specify an objective for an
agent and then you try to make sure that
there's no plan you try to you know you
just imagine as best you can that's kind
of the best you can do and say well are
there any plans which maximize this
reward
um like what are the plans that maximize
this reward
um and so if you imagine rewarding ai
when it when it's webcam shows people
smiling then there's many things it
might do which are better than making
people smile by the objective function
it might just
like paste a photo of people smiling in
front of
its webcam and then make sure that no
one can ever take that photo away from
it which might lead to some very bad
behavior or it could you know paralyze
people's facial muscles to make sure
they're always smiling like these
objectives which seem like good ideas
are often just not and especially in the
limit of very intelligent agents as far
as we can tell and i think this work
backs that up
so if if i can then uh put these two
pieces together and i'd love to know
from you if you agree with this
characterization but it seems to me then
that there this is a sort of there's a
two-piece argument that
when you start to ask about okay what
about this specific reward function or
what about reward functions specified by
humans with the intention of being
specifically good
then that kind of fails due to all the
standard alignment arguments around
goodheart's law and it being impossible
to specify metrics that aren't hackable
etc etc and then what your paper does is
it takes that and says yes and
furthermore the one of the specific
failure modes that we can anticipate
leading to
the disaster is related to power seeking
in other words like the failure will in
part manifest through a kind of power
seeking behavior would that be a fair
kind of way of putting those two things
together
yeah i think most of the most of the the
really big risk
comes from
power seeking ai
that
so i think i think there's a couple
there's several things that could go
wrong with ai and so we're seeing some
of them today um
but i think some of the most potent
risks
are
when we don't have as much control over
the future anymore and so you might
imagine this you don't even need a super
intelligent ai to make this happen you
just need a world that's increasingly
reliant on not that aligned ai systems
um and so you might notice that it's
harder to work and get into deep work
when you've got your phone nearby
because you just facebook is so well
optimized to grab your attention and so
facebook isn't a super intelligence but
it's pretty good at
getting engagement
and whether or not that's done through
ai today uh i think it's you know it's
probably going to be done through ai
tomorrow and if you imagine a world with
like a thousand different facebook like
things in it all competing for for your
resources then this seems like a pretty
bad world to live in
um
and so this is a way that things can go
wrong without uh without some super
intelligent ai that's seeking power for
itself but instead we've got humanity
becoming disempowered because of the
things we're building and how they
interact with us
and so this disempowerment because
because
most of these ais are incentivized to
seek power in our paper you imagine well
look at other work shows if you you have
an ai that's maximizing its own
empowerment with with a different human
and the environment uh then that human
is it's like acting pretty
antagonistically it's blocking the human
off it's taking away the resources that
happened in the real world we would see
a similar kind of disempowerment where
um we're just not able to steer the
future how we want anymore and that
could happen
kind of death by a thousand cuts or it
could happen with us with a very
intelligent agent or a group of
intelligent agents as well and so i
think there's a range of things to worry
about uh i think some of them are more
plausible than others but i don't think
we can dismiss any of those and uh you
know i i'd be really happy to see more
like
uh inquisitive exploration of what of
what's going on with these different
situations and how we might guard
against them it does change at least to
my mind the character of the
conversation around alignment to an
important degree i mean historically
it's been as you said this sort of fuzzy
thing where people will say like oh like
will an ai take over the world and this
is a very abstract thing it's easy to
complain that as you've said that you're
anthropomorphizing an ai system whereas
grounding in math it really seems like
the one formalization of this problem
shows that optimal agents do tend to
exhibit this power seeking behavior
caveats caveats caveats but it does feel
like the the onus is sort of
shifting through this work at least when
it's framed in this way um i am curious
then about the the ultimate steel man
version of this which you know you're
doing follow-on work where you're
relaxing some of these assumptions could
you dive into that a little bit and talk
about which assumptions you're relaxing
and how that's affected the conclusions
right so i've already relaxed this full
observability assumption
the uh the thing i'm most excited about
though is
the
um the optimality assumption
so
i went i went back to i went back to the
proofs and i was looking at what
properties i actually needed to make the
proofs go through and it turned out that
optimality is actually pretty irrelevant
that i just needed um i basically just
needed the agent to make decisions in a
certain kind of way that uh respected
permutations in a certain kind of way
and i'm not going to get into the
details of that but basically there's a
wide range of ways an agent could think
or make choices it's given some
objective and then it makes choices
based on that now am i just totally
randomly pick what outcome it wants to
bring about as a uniform distribution
over its available outcomes where the
outcomes could be different terminal
states it ends up in so this would this
would be like well what game overstate
will this policy eventually lead to in
the game of pacman these are the options
i'm talking about
or the outcomes
and
so you could have a decision-making
procedure that just uniformly randomly
chooses an outcome to realize you could
have
one that boltzmann rationally chooses
one so it it's it assigns higher
probability to outcomes which score
higher uh but it's not perfect it
doesn't just always choose a maximizing
one you can have an agent which
minimizes its utility or its reward over
time the theorem still apply you could
have an agent which
randomly thinks of like five different
options and then chooses the best one
and the theorem still applied uh you
could you could do even fancier things
and the theorems still apply
and so the moral was that we can extend
this even to things like
reinforcement learning training
procedures where you you have some
initial weights you've got a policy
network you do like policy improvement
on this many times as you interact with
the environment maybe it doesn't even
have uh some access to the transitions
but you're just doing resets and just
something that looks like what we do
today you've got like ppo or something
and
um
what this gives is a criterion that says
look in this kind of situation where you
can
redirect the training process by
changing the reward um so if initially
you reward the agent safer going right
and then the training tends to produce a
policy which goes to the right now if
you you can permute this so that now
you're rewarding the agent for going
left and guess what it
you know produces an agent which goes
left well under this kind of condition
um the theorems will still apply where
you can redirect the agent's learned
behavior by change by redirecting which
reward goes where
then then the theorem will still apply
that most reward function inputs to this
procedure will you know
maybe go left instead of right uh it's
this argument's a little bit more
involved but
the upshot is there's a wide range of
ways in a i could think or we could
produce an ai
ways we eventually get a policy out of
the ai
and under this wide range of procedures
of randomly choosing an outcome of
minimizing expected utility
of choosing tending to choose things
that are higher scoring but not always
that this will assign that this kind of
ai will tend to seek power in the exact
same sense that
the uh that that happens in the optimal
policies tendency power paper so it's
taking this constraint of optimality and
zooming out saying well actually we just
need this rather weak decision-making
requirement and exhibiting and it just
shows a whole bunch of decision-making
procedures which satisfy it
interesting and so how i'm curious about
the
uh the reception to your work like have
you spoken to people who've engaged with
the work and have had interesting
objections that have given you pause or
that have changed your thinking on on
the conclusions of the paper
mostly within the alignment community
i think
there's
it's been an exercise in
and thinking carefully about applying
theorems to the real world uh
but i think that the theorems come out
uh unscathed to the extent that there's
still suggestive and still evidence of
this power seeing phenomenon most of the
engagement so far has been through the
airline community
okay no that makes sense sort of what i
would have naively expected going in um
particularly i mean unfortunately i get
the sense that there
there are so many layers to the onion of
alignment and so many people are stuck
at the outer layers but think that
they're closer to the center and um i've
sort of i gotta say i've come across
this in doing this podcast itself
speaking with uh sort of putting my
cards on the table here i am sort of
more of an alignment hawk
but i have spoken to to folks who
consider themselves to be knowledgeable
about this stuff but don't know for
example basic concepts like instrumental
convergence and some of them are
very kind of
uh those are well-known members of the
ai community more broadly which is why
it was so nice to see this highlighted
in europe sort of giving more
credibility giving more attention to
this line of work do you um where do you
see the research program going from here
and actually
specifically i do want to ask do you see
a path
that connects what you're doing right
now to actual
solutions or kind of ways to mitigate
alignment risk in the future
so first of all i want to comment on
what you said about this kind of this
iceberg meme of alignment which i think
is pretty is pretty true and
i think so i think alignment at first it
feels maybe deceptively straightforward
you just we just well we'll figure out
how to build smart ai and if we come to
that then we'll figure out how to make
it do what we want more and then we just
improve it until it's doing what we want
and if like on a first pass
it feels like well okay this is usually
how we do things you know we usually
make an invention then we improve it so
why wouldn't that work and i think
that's a good question but there's a lot
of layers to this and i think when all
said and done in like the cognitive dust
settles at least pretty clearly to me
that's just like not going to work and
so i've
uh but it's the problem is it's not
clear there's not like
there there's arguments you can make
which i think are very compelling but
they take some time and engagement um
and so i'd be very excited uh
to to talk with with people whether they
they they agree with the conclusions of
this paper they think it's you know
misguided i'd be very excited to talk
with them more about this um because i
think it's it can be pretty non-obvious
like it well at least for me it was
definitely something i bounced off of at
first and didn't find very compelling so
i uh if anyone's listening and be
interested in talking more then you can
feel free to send me an email
um
with respect to your questions about the
future of this agenda and how it might
help maybe constructively with a
solution
i currently don't see a way for this to
help constructively with the solution
and one of the reasons is it's talking
about reward functions is talking about
utility functions and i think that
reward functions are
a pretty broken way of getting good
behavior out of a very intelligent
system operating in the real world now
for narrow tasks i think they can be
pretty appropriate especially when when
you want the the system to operate in a
narrow domain
like it's it's only
like cleaning dishes or something it's
not like optimizing the whole world to
contain like as many clean dishes as
possible but it's only cleaning dishes
and because
because our current ai like training
algorithms they're not that good we're
just simply not going to run into the
second kind of ai and i think reward's
pretty appropriate it's pretty fruitful
it's economically valuable it's fine but
i think in the long run it's just not a
good language for specifying what kind
of behavior we want from an ai in
particular it's not it doesn't let us
correct
in an ai so if we we give it a reward
function and let's suppose it just finds
a really really good policy for that
reward function then we realize oh
whoops uh we uh we got the sign wrong
it's something
uh we want to multiply this reward by
negative one well the ai predicts
perhaps that if it allows us to shut it
down it won't be able to get its
original reward anymore it won't be able
to do well instead the new ai that's
built will uh
work against that original objective and
so it'd be incentivized to not let
itself be shut down this is part of the
paper
um
and so so there's certain properties
that just make reward functions and
utility functions seem broken to me and
i think this
i think that this
that this paper is telling us something
important about
uh
smart agency what optimal agency or
rational agency tends to look like
um and i'm hopeful that this will lead
to more concrete insights about how to
do better
um i've got a couple guesses about how
to do better
but uh that currently it's not like oh
and so here's how you just avoid it
here's the predicate you apply the test
whether the policy is bad or not i don't
think that's feasible
it's it's been my perception that one of
the things ai alignment generally has
been missing is just a concrete
arena in which to reason about these
systems a perspective a lens on the
alignment problem that
is mathematically tractable and lends
itself to those kinds of insights that
was one of the things that i really like
about this paper is that it gives a
definition of power seeking it's no
longer talking about you know
qualitative terms like instrumental
convergence things like that without
pinning them to a specific kind of
mathematical structure which as you say
i mean that ties into deeper insights
about okay what is the underlying
mechanism the thing that's that's
causing this to go wrong um i guess
there is an open question about whether
it's the process of optimization itself
whether there's something intrinsic
about just kind of moving the world and
i guess that's that's reward
optimization maybe that's that's sort of
what you're gesturing at there but is
there a broader sense of optimization
that doesn't involve reward functions
that you'd be more optimistic about or i
guess i'm just curious about maybe some
of those early thoughts you might have
about
better paths i think there is i think
i'm pretty sympathetic to the argument
that
if so if we do this well we figure out
alignment uh things go great and then we
write a textbook like 100 years from now
or maybe like 40 years from now
uh then if you showed me that textbook
today i just you know i'd think about it
for a couple minutes and face palm and
realize that of course it's how you do
it but
um
so i think there's some chances not like
some you need
this
crazy amount of mathematical
foundations to even begin to understand
alignment it might be true that there's
some simple
reframings of the problem that would
make things much easier
i think it's also plausible that there's
not and that we we really need to build
up like
a new edifice of theory around how
agents should interact with each other
um
but i think
yeah i think that
i think that the successor work on
what kinds of decision-making procedures
will engender this
use statistical incentives by what
jason's tendency tennessee power
it points to redirectability so if
you've got a parameter that
kind of
modulates the final behavior of the
agent in this case in this
original paper it's the reward function
determines what are the optimal policies
um
and if if you can change you can change
the optimal policies very well by just
changing the reward function then they
just kind of redirectable
and so these redirectable agents often
have power seeking centers at least in
the settings we look at uh you know in
the setting like eventually
they'll have power c incentives like you
can imagine well
so one one kind of objection is well
okay so we don't see this today we don't
see this um
we don't see this today as even even in
small settings like if you give
uh you randomly generate a reward
function for pac-man uh just each state
has some randomly generated but fixed
score it's like well
well pac-man's just gonna tend to die
right like it's not gonna tend to seek
power and the reason that's true is
because the reinforcement learning
algorithm can't find a good enough
policy because it's too hard the sample
complexity is too high it's just too
there's no structure to the problem um
and so i think an entirely analogous
situation holds in real life where
we might have we might have an agent
that we're training on a coherent
objective but our algorithms are just
not good enough and so it's not able to
produce the policies that would be good
and so i think that's pretty true in the
real world um
and so i think once we start crossing
that point uh we'll start you might
start to see some bad behavior i think
it's tied to that redirect ability of
the underlying algorithm so i think
that's a big part of the puzzle there is
somehow we'd like something that is
robust we like we like a way of aligning
an ai where we don't have to
have to get uh you know the parameter
just just right in a way that it's that
you can't even iterate on very well
perhaps um because if you mess up once
with a very smart ai or a set of smart
ais or even just uh you know a thousand
facebooks right it's kind of hard to
undo that yeah i know that that makes
sense and do you
so do you expect that the failures of
these systems
will be
obvious and non-catastrophic
before they're catastrophic
like will we be able to learn and
iterate based on more obvious kinds of
failures that point to this underlying
problem or is is this more of an issue
of an ai system developing enough
abstract reasoning ability to realize
hey i'm an ai system i'm embedded in a
computer i can grow you know my
processing power by convincing blah blah
blah you give the standard ai takeoff
scenario
yeah
i think there's at least a 20 chance
that we would see you know if if if this
kind of
uh
if this kind of ai deception let's say
is a problem where the ai let's suppose
that ais tend to be incentivized to seek
power and one of the ways you can do
that is by being deceptive by pretending
you're aligned so that you don't get
corrected and shut off this is just a
way to avoid shutdown like we talked
about on the paper now this is one
modeling assumption you can make we can
talk about different ways of modeling
the situation i think they all the
reasonable ways i can think of come up
to the same conclusion but i'll flag
that that i'm making you know that as
thinking about
itself being shut off and modeling that
accurately
i'm saying well if i do this it's not
going to be very high scoring so i'll
just pretend to be aligned for now
uh now you might have ais which are
smart enough to think about this but are
still pretty dumb and so
they're kind of clumsily deceptive now i
think there's at least a 20 chance that
conditional on this whole thing
happening that uh we would see some kind
of clumsy failed attempted deception
that we can just recognize as dangerous
but i think even if even if even if like
every researcher in the world at that
point realized oh whoa this is a problem
let's slow down and think about this
i mean you still run into coordination
problems you still run into trust issues
where you're like well we know that we
have a culture of safety here at like
deepest mind but we don't trust these
other
these other organizations like closed ai
or or whatever is going on yeah it's
just hypothetical future orgs
um
and and so you still run into these like
game theoretic issues with coordinating
with each other like i think it's just a
pretty bad situation to be in and i
think an ideal situation would be we've
already worked out the theory for what
to do here we've got a competitive
algorithm that doesn't have these issues
let's run that instead and it's it's
incentivized for everyone to run that
it's like a low alignment tax you don't
have to you don't lose a lot of
performance on the metrics you care
about by running the aligned version of
whatever algorithm it is
uh this feels really hard to me
especially with the relatively low
amount of effort we have put on
alignment today i really like to have
this kind of thing before we run into a
situation where we actually have a
possibly intelligent ai um so i
i really would like more work today
working on this well before we see it
because i think even even if if everyone
could agree on the importance of of the
problem if we got a warning shot
um i think that we would still have
problems coordinating
i think that's again couple years just
you know a very
clearly understood scientific phenomenon
of uh of uh pandemic that we have
procedures for what to do about and we
just have problems implementing we have
problems coordinating and so bad things
happen
yeah well and i mean hopefully the the
execution of this will be left up to
more technically competent minds than
the pandemic um in that the the
exponentials are happening in contained
environments that are overseen by highly
technical people who hopefully will be
in a position to understand this better
but they have to understand it first
they have to understand what the stakes
are and they have to understand what the
risks are and that is why i'm so excited
about this paper i think it starts that
kind of conversation it's a critical
tool for people who want to make this
case that there's a quantitative kind of
backing behind this argument now it's no
longer just waving our hands around
talking about world domination by ais
there's something to point to and it's
highly suggestive so thank you for
putting that research together um if
if there's anything people want to ask
you about with respect to that paper are
you open to them reaching out uh twitter
email that sort of thing yeah absolutely
okay great so i'll make sure that we put
those links in the blog post that'll
come with a podcast as well uh alex
thanks so much for for doing this i
really appreciate your time on this
thank you