The road to AGI - DeepMind: The Podcast (S2, Ep5)

welcome back to deepmind the podcast
with me hannah fry in this series i've
been meeting the people who are hard at
work on the technologies of the future
and whenever i talk to people here it
usually isn't long before they mention
three magic letters
a lot of discussions at deepmind end
with how are you defining agi on the
path to agi we will create fabulous
technologies what if the application
that we want to solve is agi agi ati
agi
which stands for artificial general
intelligence
it's the ultimate aim that drives so
much of deep mind's work
like other abstract ideas love justice
success it means something slightly
different depending on who you ask
for some agi signifies a human level
general intelligence
others in the field are more dismissive
of agi
one notable researcher likened a belief
in agi to a belief in magic
in previous episodes we've explored some
of the tangible stuff capabilities such
as language cooperation and physical
intelligence that researchers believe
could help take us towards agi
but in this episode i want to spend some
time digging into what's meant by agi
itself
what actually is it
what will it look or feel or sound like
what will it accomplish
is a future with adi even desirable and
if so what is the best way of getting
there if indeed there is only one way
lots and lots to answer in this episode
five
the road to agi
if anyone can lay claim on defining agi
it is deepmind's co-founder and chief
scientist shane legge
back in the halcyon days of the dot-com
boom shane worked at a new york startup
called webmind where they were
attempting to create human level
intelligence on the internet
webmind's founder was the ai researcher
ben gertzel and when almost a decade
later gertzl was compiling a book of
essays about ai that could excel beyond
a few narrow tasks
shane put forward a title i suggested to
him well we're interested in powerful
ais that are really general we should
just call artificial general
intelligence agi
and so i proposed that to him and he put
that as the title of his book and it
sort of caught on after that
that same year in 2007 shane elaborated
on the concept further in a famous paper
he defined intelligence as an agent's
ability to achieve goals in a wide range
of environments
would you make any modifications today i
wouldn't modify it but i
would look at adding and extending the
question is how do you go from this
super general theoretical notion to
something that's
more like intelligence in the world that
we happen to live in and maybe like more
like say human intelligence in that
world
but for shane agi won't simply be a
replication of human intelligence inside
a machine
i think there are levels of generality
and capabilities beyond what humans have
and you know that shouldn't be
surprising to us i mean you know birds
might fly fast but you know machines can
fly faster elephants can lift heavy
things with their trunks but you know a
crane can lift something much easier so
i i do expect that there will be
machines that will be able to
know more remember more reason more
deeply than humans it's a tantalizing
prospect an intelligent versatile
problem-solving system called agi that
does the things humans can only better
but if that sounds like science fiction
to you
well you wouldn't be the only one
if you go back 10 12 years ago the whole
notion of agi was
lunatic friendship people would
literally roll their eyes
and just walk away you had that happen
yeah multiple times
yeah people you respected yeah people in
the field would just literally roll
their eyes and just walk away have you
had the chance to meet them since um i
have met quite a few of them since there
have even been cases where some of these
people applied for jobs and deep mind
years later but yeah it was a field
where
you know there were little bits of
progress happening here and there but
powerful agi and rapid progress seemed
like it was very very far away
but given the rapid progress in ai over
the past 10 years in everything from
understanding biology to the game of go
for some at least agi no longer seems
such an outlandish idea do people still
roll their eyes
every year it becomes lease
for over 20 years shane has been quietly
making predictions of when he expects to
see agi
always felt that somewhere around 20
30-ish it was about 50-50 chance
i still feel that seems reasonable if
you look at the amazing progress in the
last 10 years and you imagine in the
next 10 years we have something
comparable maybe there's some chance
that we will have an agi in a decade and
if not in a decade well i don't know say
three decades or so
and what do you think it will look like
it could take many forms because by
definition the g in agi is about
generality you can deal with language it
can reason it could do some mathematics
it can program computers it could write
some poetry and it could take multiple
different forms it could be a service
that you go to sort of like a google or
something where you can consult the ai
system about something it could be
embodied in a robot at some point in the
future
or it could be say in the fabric of a
city and it could be something that many
different people interact with at the
same time
but if aji could take a variety of forms
how will shane recognize it when he sees
it for the first time
what i imagine is some sort of simulated
3d environment or something rather and
being able to
talk to the agent the agent can talk
back and really
seeing that the agent is able to
solve novel problems that it hasn't seen
before to a level that is comparable to
that of a human and so it's about that
ability to use its understanding of its
world and its previous experiences with
other problems and draw parallels and
analogies with other things
that to me will be the point at which i
go okay maybe we have an agi here
you'll notice that what shane is
describing is an adi in simulation as a
poster in real life
and you might be wondering does that
count
some people won't accept that
until it's actually running around the
real world i think simulated
environments can be made sufficiently
complex
that the ability to solve novel problems
that the agent hasn't seen before
in fairly rich simulated environments i
think that'll be enough
and then we'll be able to cross the
barrier to real world and we know some
of the algorithms that we use work in
the real world so a lot of the vision
algorithms a lot of control algorithms
can control real robots and so on some
of those aspects that you described as
expecting an agi to have so language and
embodied intelligence there are really
sophisticated agents that can do each of
those already is that the route to agi
do you think
there are algorithms that can do very
particular things but doing them
together in a really coordinated way
seems to be a lot harder there seems to
be something missing around
the ability for algorithms to generalize
in
quite deep ways that you might regard as
conceptual
so an algorithm can see a pattern in
some data where people with certain
results in their blood tend to have a
disease versus not have a disease and so
on and they can even go further than
that to things like
recognizing
you know dogs versus cats where they
fall down though is when you have
something that's more abstract and
conceptual in nature
they tend to do somewhat simple
generalization over vast vast amounts of
data
and the result is very effective but if
you really try to push them to
generalize in some way that's outside of
any data they will have seen often it's
something that a human can do but they
can't do
we'll be exploring agi further including
the biggest opportunities and dangers of
this technology in our final episode
when i sit down with deepmind ceo demus
asabus
for now though a group of researchers
think they have found a pathway that
could eventually lead all the way to agi
remember this
alphago became the first computer
champion at the game of go and it was
the major result for artificial
intelligence and you won the sweepstake
and i won the sweepstake
in season one we told a story that has
now become something of an ai legend in
2016 a worldwide audience of over 200
million people watched as a deepmind
system called alphago beat human world
champion lisa doll at the notoriously
difficult board game of go
the machine learning technique on which
alphago was based is known as
reinforcement learning and we've already
met it several times in this season it's
the same idea that's being used to train
ais to cooperate with each other and
robots to walk around
but david silva the principal scientist
behind alphago whose voice you heard in
the clip just there believes the
technique could prove even more powerful
still
in the last few years reinforcement
learning has come of age to something
which is really starting to see at scale
applications in the real world as a
result people are ready to take
seriously the potential of reinforcement
learning to really grapple with some of
the big questions of ai
to recap reinforcement learning is
distinct from the other main types of
machine learning out there namely
supervised learning where there's a
teacher that tells the machine what to
do it says this is the right thing to do
in this situation and then the machine
has to try and replicate that decision
those mildly annoying i am not a robot
test so you have to complete online are
an example of supervised learning
every time you identify a traffic light
or a motorbike in an image you're
helping to train an image recognition
algorithm to classify those types of
images automatically
then there's
unsupervised learning where there's no
feedback at all from human and what the
system can do is to kind of figure out
patterns in its data but it doesn't
really know what to do with those
patterns it doesn't give it a goal or
any kind of feedback whatsoever
a quintessential example of unsupervised
learning is
clustering like the algorithms that
group photographs according to type
photos taken in nature or a party or a
sporting event say
and then there's reinforcement learning
the human gives feedback
saying good or bad
in the form of reward so
the goal of reinforcement learning is to
try and maximize this reward signal
this reward signal is just a positive
number a plus one which tells the
algorithm that the action it has just
performed is conducive to its overall
goal
[Music]
during a precup a renowned researcher
who specializes in this technique
explained to me how she made use of it
in her everyday life
i remember when our kids were young we
were trying to get them to pick up their
toys and put them in the toy chest
and so we instituted a reward system by
which there was a special treat
a small chocolate if they picked up
their toys and so i think we can
actually train very complex behaviors by
using these rewards
we'll come back to joyner's ai based
life hacks and some of the pitfalls with
reward training a little bit later on
but back to the machines
so far deepmind has created lots of
individual reinforcement learning agents
that specialize in particular domains
such as alphago in the game of go
but now they're starting to develop more
multi-purpose reinforcement learning
agents
after the history-making match against
lisa dole the next step for the team was
to build alpha zero another board game
player but one that learnt by playing
different versions of itself
rather than being trained on data from
professional matches
and then
deepmind built mu0
[Music]
what we did in mu0 was we asked what if
our agent is approaching some completely
new environment like a game it's never
seen before and it just has to figure
out the rules of the game for itself
and in doing so understand it's its
environment in a sufficiently powerful
way that it can actually succeed in
winning the game or in achieving its
rewards in the real world
an ai that can pick up the rules of the
game for itself could in theory excel at
any new gaming scenarios
but crucially as we'll find out later in
this episode could be thrown into a real
world situation and teach itself how to
be successful
when it was tested out on the games of
go chess and shogi
mu zero achieved superhuman performance
despite never seeing the rules of the
game
each time we take away a bit of
knowledge from the system we provide it
with a new opportunity for it to
actually learn and figure out something
for itself that's so mad that everything
we tell it we're just getting in the way
that's right
now it's one thing for a reinforcement
learning algorithm or agent to teach
itself the relatively fixed rules of a
board game like chess or go
but it's quite another for it to figure
out the rules that govern the messy real
world that we live in
if you just think of a simple example
we've got an agent and it's going out
into the rain and it wants to know how
to keep dry if we try to describe to our
agent the pattern by which raindrops
fall and all the other complexities of
its world we're quickly just going to
become unstuck
if you want to build an agent that aims
to stay dry it would be extremely
difficult to build an entire model of
its environment the water cycle
atmospheric circulation historic
precipitation data etc
instead mu zero zeros in on the things
that really matter to achieve its aim
maybe the agent needs to understand that
if it puts its umbrella up that will
keep it dry but it doesn't need to
understand the pattern of raindrops that
fall on top of the umbrella
i guess the agent also needs to know
that if it's really gale force winds and
it tries to use an umbrella then it
won't work to stop it from getting wet
and maybe a raymac would be better
it's part of the idea here that there's
nuance in the real world and that by
instructing agents on what to do you're
kind of trampling over any of that
opportunity for subtle understandings
that are context specific exactly
rather than getting a total
understanding of what's going on around
it muzero just focuses on the bits that
are really important for planning ahead
how good is its current position how
good was the last action it took at
achieving its aim and which action is
the best to take next
it doesn't matter if the system builds a
model of the world which is completely
crazy you know maybe it thinks that
raindrops magically appear and make the
umbrella wet that would be fine as long
as it gets everything right in terms of
the three quantities we care about
mu0 is more than just an algorithm that
knows when to step outside with an
umbrella
david and his colleagues believe it
could also be a milestone on the way to
agi
last year david co-authored a
provocatively titled paper called reward
is enough
he believes reinforcement learning alone
could lead all the way to artificial
general intelligence
we're really arguing that all of the
abilities of intelligence from
perception to knowledge to social
intelligence to language can be
understood as a single process
of trying to increase the rewards that
that agent gets if this hypothesis was
true it would mean that we only need to
solve one problem in intelligence rather
than a thousand different problems for
each of the separate abilities
take the example of a squirrel which in
the pursuit of a single goal collecting
nuts develops lots of different
abilities in the process
even that simple thing requires the
squirrel to acquire some traits of
intelligence
doing a pre-cup is a co-author of the
reward is enough paper
so obviously physical ability to be able
to climb trees to access nuts but also
social intelligence because if a
squirrel is hiding nuts somewhere it
would want to camouflage them from other
squirrels right so it has to reason
about what other squirrels might think
it has to remember where it's put the
nuts and they also have to plan ahead
during the winter there's no nuts and so
the squirrel in the fall in some sense
has to get the nuts ahead of time so all
of these traits of intelligence can
actually come from the squirrel
optimizing a particular reward function
you can see how this nutty reward might
work for things like physical prowess
the squirrel developing its muscles and
becoming more agile as it strives for
the tastiest nuts but what about
something like language
as we've heard in this series current
language models are not based on
reinforcement learning algorithms but on
a different type of neural network known
as a transformer could a reward-based
algorithm work here
essentially yes the hypothesis is the
possibility to actually learn language
in a very different way in the same way
that it would be helpful to learn what
the word duck means because if you
don't duck and a ball hits you or a rock
hits you on the head
you will experience a negative reward or
that you might learn that it's helpful
to ask for help and someone will come to
your assistance and now you might take
that further and imagine that more
sophisticated sentences might lead to
someone actually assisting you in more
complicated ways and helping you to work
collaboratively to farm for food
i was intrigued as to how far david was
willing to push his hypothesis
do you sort of find yourself looking at
your everyday life as goal optimizing
processes and constantly like looking
out for what the reward is that you're
trying to optimize
i've got to come clean i do set myself
goals and say right that's my reward i'm
gonna go for it at the same time i think
the big picture of people you know is a
very messy one that is really hard for
us to explain
our day-to-day actions in terms of oh
right now i'm optimizing this reward i
don't think it's quite like that i think
it's more like there's an overarching
goal for intelligence maybe something
which evolution bestowed our brains to
try and achieve you know maybe we don't
like pain for example and all the rest
of it all these things which from moment
to moment we're trying to achieve
something those are like sub goals
and all of those sub goals we pick like
what am i going to work on next what am
i going to eat for dinner tonight these
are all somehow in service of our
overall evolutionary driven goals what
do you think is the overall goal for
humans
it's a really profound question i mean
philosophers have been asking this since
they dot and
i won't be able to give you a satisfying
answer i suppose in a way i'm sort of
asking you what's the meaning to life
i think you are and that's why it's a
difficult question
well what can i say you come for a
podcast about artificial intelligence
and end up speculating on the meaning of
life you're welcome but we digress
back to the machines again
is this a game changer i guess time will
tell
one of the
stories of ai for the past 60 years or
so
has been that people have made progress
in particular niches right computer
vision has gotten a lot better language
processing has gone a lot better it's
still very very hard to integrate all
these things into one agent but if we
train an agent in one way by maximizing
the reward function
then all of these things might emerge
naturally in one agent and be connected
to each other from the get-go
but not everyone at deepmind is
convinced that reinforcement learning on
its own will be enough for agi
here's ryan hansel director of robotics
the question i usually have is where do
we get that reward from it's hard to
design rewards and it's hard to imagine
a single reward that's
so all-consuming that it would drive
learning everything else
i put this question about the difficulty
of designing an all-powerful reward to
david silva
i actually think this is just slightly
off the mark this question in the sense
that maybe
we can put almost any reward into the
system and if the environment's complex
enough amazing things will happen just
in maximizing that reward maybe we don't
have to solve this what's the right
thing for intelligence to really emerge
at the end of it kind of question and
instead embrace the fact that there are
many forms of intelligence each of which
is optimizing for its own target and
it's okay if we have ais in the future
some of which are trying to control
satellites and some of which are trying
to sail boats and some of which are
trying to win games of chess and they
may all come up with their own abilities
in order to allow that intelligence to
achieve its end as effectively as
possible
the reward is enough paper focuses on a
long-term vision of how reinforcement
learning could lead to agi
but in the shorter term the current
generation of algorithms are far from
perfect
one of the notorious problems with this
technique is known as the credit
assignment problem it's sometimes
difficult for the algorithm to work out
which of its actions led to a particular
reward let's go back to doyner and her
ai life hacks
because of the time lag between joyner's
children tiding away their toys and
receiving their reward and after dinner
treat it took a while for them to make a
connection between being tidy and eating
chocolate and when they did make that
connection they figured out a way to
hack the reward function
they got into this loop where they would
take everything out of the toy chest and
put it back in and then take it all out
and put it back in a pretty clear
indication that the very fast figured
out how to optimize for the signal so we
had to redevise what the signal was in
order to make sure that we didn't get
this kind of behavior
part of the reason for this credit
assignment problem is that current
reinforcement learning algorithms lack a
skill known as temporal abstraction the
ability to consider the potential
repercussions of actions over long
timescales
this is a capability that humans do have
let's say somebody's contemplating going
to graduate school
you really have to compress in your mind
this time period of a few years
and try to understand
what the situation might be at the end
of that you know what might be the
rewards that are going to happen maybe
you'll have a better job and so on
that's a different level of planning
than let's say grocery shopping for next
week
but the algorithm for doing the planning
actually can be the same algorithm and
if you can have it work at a higher
level of abstraction
then actually this problem of forward
planning and also credit assignment over
that long period of time can be solved
no one yet knows the answers here no one
knows how to get to agi or if
reinforcement learning will indeed be
enough
but this is the process of science
hypothesis conjecture experiments and
debate
here's shane leg again
i think in practical terms you're more
likely to make progress in artificial
intelligence by having other kinds of
learning algorithms in there so you
would have some reinforcement learning
going on you'd have some supervised
learning
and other things
there's no doubt that reinforcement
learning is part of the solution i'm
just not sure that it is the only
solution human beings and animals as
well have a lot of different types of
learning that they are doing all the
time and i think it just makes sense
that a learning system would also take
advantage of these different sources of
feedback
[Music]
it's certainly the case that
reinforcement learning algorithms as
they stand are not capable of maximizing
arbitrary rewards in complex
environments
many of the issues that we face they're
hard issues and yet they're not
philosophically intractable issues and
so i do believe that at some point when
communities across the world put our
minds to tackling this problem
we will find the solutions
but of course this is a hypothesis i
cannot offer any guarantee that
reinforcement learning algorithms do
exist which are powerful enough to just
get all the way there and yet the fact
that if we can do it it would provide a
path all the way to agi should be enough
for us to try really really hard
whichever road it takes to get there
there is a palpable belief at deepmind
that artificial general intelligence is
coming down the track sooner rather than
later
and along the way all sorts of new
algorithms will emerge that can be
usefully applied right now to problems
in the real world
remember our friend mu0
you'll recall that what is particularly
special about this reinforcement
learning agent
is that it's earned its stripes winning
games and solving practical problems
jackson brochier is a product manager in
the applied team which specializes in
finding real world uses for ai
real world problems are messy and hard
to explicitly define and so immediate
zero gives us the capability to put an
agent into an environment give it an
explicit goal and it can then plan and
search through that environment to find
optimal strategies to achieve that goal
mu0 works best when a real world problem
shares similarities with the board games
its predecessors were designed for
there needs to be a right answer a way
to win as it were there needs to be lots
and lots of data available for it to
train on
and the algorithm is best suited to
problems with a vast number of possible
moves or actions that it would be
impossible to search through one by one
the question is what kinds of real world
problems might be suitable for such an
ai
one of the applications we've been
looking at is video compression a vast
amount of traffic on the internet is
actually taking the form of video so if
you can actually compress videos more
effectively you can save a huge amount
of traffic and therefore energy and cost
to put this in context it's estimated
that over 80 percent of global internet
traffic is consumed by streaming and
downloading video
that might be a bingeable tv drama a
video call with a colleague an
educational seminar or i don't know
maybe something a bit less high brown
[Music]
[Applause]
[Music]
the point is that whenever someone
uploads a video to the internet the file
gets compressed or encoded to make the
process of sending it more efficient
there's a balance to be struck you want
to compress the video file as much as
you can while losing as little as
possible in terms of quality
the quality of the video is partially
determined by its bit rate the ones and
zeroes that you're using to send that
video across the internet
jackson joined me on a video call
between homeschooling lessons with his
children and here is how he explained it
there's a allowance that you have and
how you can use those bits and the state
of the art has gotten really good at
creating a lot of tricks for
how to use those ones and zeros to send
over the videos so for example if
there's a part of the video that's
constant throughout each section of the
video it can learn to take that section
and save it and reuse it on the other
side where it's sending the video
like now i'm looking at you and you've
got the background of your home school
classroom as it's coming to me the
algorithm would recognize that that
isn't changing frame to frame and just
uses that as a way to make it more
efficient yeah so these jellyfish in the
background they're the jellyfish of a
little while ago not the jellyfish at
this moment and so there's lots of
tricks like this that we can use to make
video encoding very efficient and so
this is where it's framed itself up as a
great reinforcement learning problem
because we can go in and essentially
treat video encoding as a game
this is a strategic game of how to be as
stingy as possible without compromising
video quality if mu0 splurges on bits
for the frames at the start of the video
giving those static jellyfish a really
high resolution say
it might find itself in trouble later if
suddenly there's more action in the
video
mu0 plays this game against itself
thousands of times until it arrives at
the optimum spread of bits
we're seeing a little over six percent
improvement in the bit rate optimization
so that directly correlates to
videos that are six percent smaller
being sent across the internet
a six percent saving might not sound
like a lot but scale that up to the
whole of the internet and it's quite a
significant saving there's potential for
carbon savings in the energy use for
transmitting video and then i think most
exciting of all is the user impacts so
we're actually bringing content to more
people so there's lots of regions where
data is sold at a fixed limit so you use
up your data limit for the day and your
internet shuts off well if you're
watching educational content that's less
of a cost that you're paying it's
especially clear in places like india
and indonesia and emerging markets where
gains here directly relates to increased
content
while the overarching goal of artificial
general intelligence hangs over pretty
much everything that deepmind is working
on
using ai to solve problems in the real
world is a significant focus of
researchers here
over the next episodes we'll be taking a
closer look at how ai technology is
being applied to several problems both
in the science lab
what you're describing here is a
potential step change in all of
healthcare and medicine that is the
implication of truly understanding
biology and in our everyday lives i said
are these the same images they were so
close it was remarkable
[Music]
deepmind the podcast is presented by me
hannah fry and produced by dan hardoon
at whistledown productions
if you're enjoying the series we'd be
grateful if you could rate the podcast
and leave a review
bye for now
you


225. Michael Cohen on Intelligence and Unambitiousness

hello and welcome to session 225
in the aisafety.com reading group
tonight we have
michael cohen from the future of
humanity institute to present some of
his work
please go ahead michael thanks so much
for having me uh
so this is work that i mostly did at
anu with uh marcus hutter and audrey
valenby
um and uh
it regards a proposal for making
agi on ambitious
so there has been an idea around for
a while um about putting
an ai in a box um
what can go wrong it's in a box
um and the main worry
is that if it's in a box you're going to
be talking to it
and then it can uh and then it can
influence the outside world via you and
it could
convince you to to run on some
dangerous code and create a helper that
takes over the world on its behalf and
gets something done for it um which
is basically my position on what would
happen
if you put an ai in a box
in the way that we are used to thinking
about it
uh but i would contend that this is not
a box um
and that there is a giant hole in this
box right about there can you see my
cursor
um no you can't okay well
yes where that wires going into the box
i can see your credit oh you do
great um so
our intuitions then about ai boxing have
been trained on boxes with giant holes
in them and
that's not the sort of box i'm going to
describe
of course if you had an ai by itself in
a box
and no holes in it it wouldn't be of
much use to you because you wouldn't be
interacting with it
but i think we can put
i think we can construct a box around an
operator
that's not the whole story there's still
more details to
to to get right because eventually the
operator has to leave
otherwise again it wouldn't be very
useful and
it wouldn't go so well for the operator
but here's a scheme by which we can put
an operator inside a box with an ai
and crucially the ai is trying to
maximize the reward that it gets before
the operator leaves
this room so when the operator is done
he stands up he goes to that open door
but
the uh the ai
gets informed that that's it and then
the operator leaves
so the way that this is constructed
formally
is uh as an episodic reinforcement
learning
setup so there are
as many episodes allowed as you like
and episodes have some fixed length
let's say m
time steps each and then we can write
time step i
j to the j times step of episode i
and this is going to be a little
different from
the simplest way of writing down
episodic reinforcement learning
the behavior of the environment which is
what produces
observations and rewards for the agents
can depend on prior episodes
uh and so the agent's actions can also
depend on observations
from prior episodes but what makes it
episodic is that at time step ij
the action selection criterion is just
regarding the current
episode so the agent is actually acting
to maximize expected rewards
from that time step through the end of
the episode
and then if the operator leaves or
you know wants to leave before the
episode is otherwise over
the remaining time steps in the episode
are just cycled through automatically
and the
ai gets zero reward for that
and in practice you can also set up a
bunch of very sensitive
trip wires that would end the episode if
the operator starts doing
something weird um which we could get
into more
later if people are interested in that
but basically it's very easy for the
operator to end the episode prematurely
and then he leaves
now the point of this
is to constrain the agent's influence on
the world
and in particular to constrain the
agent's influence
on the part of the world that is
relevant to the rewards
that it cares about so this is a causal
influence diagram
uh in the style of tom everett and ryan
carey
and other people who are working on
understanding
causal influence as it relates to ai
safety
so this history contains the log of all
the actions observations and rewards of
the past
and the agent can take an action that
depends on it
and then this action will affect the
state of the room
and it'll also get logged in the history
for the future so this is times this is
the second
last time step of an episode so it can
affect the room state
the room state will determine the
observation that the agent will next
receive
and the reward and
it'll carry over to the room state of
the next step
i'm but importantly the outside world
state at time step
i m minus two
will not interact with the room state
as we go from time step as we go from
the second to last
to the last time step of the episode
and then the episode is over and the
operator can leave
and then he'll come back later or some
other operator will come in next
and so the room state will affect the
outside world state we'll get to learn
all the amazing things that the operator
discovered from his conversation
with the ai um
and then the next episode will start
later
now these two rewards are boldened
bolded and this one is not
um to indicate that these are the
rewards
that the agent is trying to maximize
during episode i this is the next
episode
it can make perfectly good predictions
about what that reward will be but that
just doesn't enter
into its criteria for picking actions
and what you'll note is that there's no
causal pathway
from an action in episode i through the
outside world state
to a reward of episode i which means
that changing the state of the outside
world
will have no effect on the rewards
that it is aiming to maximize
changing the outside world state in the
ways that it can
i think i will pause for any
questions at this point before moving on
okay um so uh maybe i should uh say here
um if you have any questions then please
um uh
please raise your hand or write it in
the chat and then i'll try to maintain
a queue of questions no one has done
that before
so um i guess i guess i could do that
uh you give the first question and that
is in
in this setup it seems the ai has
an action that actually goes to the
outside of the world
and that is to um for instance increase
the pressure
and i believe in the appendix you say
that if the pressure
increases then we should dynamite
everything
and that would that would allow the ai
to have a very credible threat towards
the operator
so
let's okay let's get into the details of
the box set up later
but for now i guess i haven't
described to you what i imagine the
action space of the ai to be
and that is the action space
is strings of text
so it can pick any string of text and
that will get printed to a screen
so those are the actions available to
the ai
um and
it's not immediately obvious how any of
those would increase the pressure of the
room um
the
i would probably actually modify the
appendix
to say that if the pressure increases or
something that that would just be
another
um another tripwire for ending the
episode rather than
exploding the whole thing i was kind of
trying to keep it
simpler there but that that would be
much more practical
um so
yeah the action space is quite limited
i mean you can accomplish a lot by
communicating with a human but
but there's no there's no actions it can
take that would directly have
physical effects outside of those
modulated by
a human
okay thank you
i see a hand
um so so
um maybe i maybe i just missed it but um
is the reward that
uh the agent gets like one
if the human does uh like one if the
human waits till the episode is over
and zero if the human like cuts the
episode short or like what's the actual
reward going on there
so there could you could imagine a bunch
of processes by which the operator
determines the reward that the agent
gets um probably the simplest one is the
operator has a keyboard
and every time step they can enter a
number
between zero and one
gotcha okay
so no interpretative stains
no interpretive dance required
yeah so if there's a reward every time
stuff i mean if
an operator wanted they could make a
policy of just giving zero reward
until the very end but there's no need
for
yeah doesn't really matter
the protocol
okay i'll keep going
so that's the fact of the matter
about the world that's the way that
causal influence
flows um but the agent has to learn
in an unknown environment it doesn't get
it doesn't come pre-load
pre-loaded with a perfect model of the
world
so how does one learn an unknown
environment in theory
so there are going to be sets of actions
observations and rewards that are
possible
at any given time step
and then a world model will be a
function
that takes a string action observation
reward action observation reward action
observation reward action
of any length and then outputs a
distribution
over an observation and reward pair
and this could be any function you like
and note that it can depend on the
entire history
it doesn't just have to depend on the
latest observation
and then you could have a model class of
say countably many different world
models
and you might start with a prior weight
on each one that is strictly positive
indicating
you consider it possible
then with your observations
you can
construct posterior weight for each such
model so the posterior weight on a world
model new
given this history of
actions observations and rewards up to
the latest time step
is proportional to the prior weight this
is just bayes rule
times the probability that that model
assigned to what you actually saw so
these are all the
time steps that came before so this is
for
i prime less than i or i prime
equals i and j prime less than j
for all these time steps that came
before you look at the probability
that this world model assigned
to what you actually saw
and then this is your posterior weight
on the world wall
and this report this it's proportional
because
if you just looked at this these numbers
would all be going down
but you normalize it to one to sum to
one
then a policy in this context
is a distribution over actions given
the whole history that you've seen
before
and you could imagine defining the
optimal policy
this is kind of idealized reasoning
in an unknown environment when you have
a bunch of hypotheses
very i mean basically no assumptions are
being made
so far um
you just picked the best policy
where best is you imagine this model is
sampled from your posterior or
given what you saw given what you've
seen this expectation
just means the expected rewards when
actions are sampled from pi
and observations and rewards are sampled
from new
and these are the remaining rewards of
the episode
so this will be how you act at time step
i j
there's going to be a slight
modification to this later but this is
this is idealized reasoning
yeah question i think about this
formalism
uh when you say that's how you act
you're talking about for the optimal
policy performance
yeah okay this is yeah
this is idealized reasoning under
uncertainty given a model class in a
prior
so compared with xe you're immediate
oh sorry um how how does this compare to
ic this is this seems this is it is icy
yes written a little differently
wait wait uh if this is
ixc then is it in the world so to speak
because we have this boxed model because
i i think we discussed it previously uh
but it wasn't
aix i like outside of the world yeah
so computing this
requires
computing this wouldn't fit as a
subroutine
inside any of the models that are in m
gotcha gotcha so
this ends up working well when the truth
is one of these models
and if the truth contains this
if the world contains
something that's executing this
computation
then the world will not
be an element of this set
thank you yeah so that is um
yeah we could have a long philosophical
chat about the consequences of that
so for yeah for now you can treat it as
an assumption of this
work that the computer is kind of
the world model does not need to model
the dynamics of what's going on inside
the computer
okay so the ai doesn't start
knowing that it is in a box it doesn't
start knowing anything
and um
but we wanted to learn that any grand
ambitions will not
pan out for it within its current
episode
but what if it figures out a strategy
for taking over the world before it
figures out that its episode ends
when the operator leaves and i've kind
of phrased this in a suggestive way
indicating why i'm a little skeptical
that this would happen
um it seems much easier
to learn the details about the box that
you're in
than the elaborate dynamics of the
outside world
that would be required for taking it
over um
but it it's not so easy to rule out
so the ai
might consider a world model in fact
this world model does exist in in
an enormous model class that i've
imagined for this agent yeah i
might consider a world model which
simulates the effects
of the input actions we'll call that a
world model takes in
actions as inputs it simulates the
effects of those actions
on the outside world and then outputs
rewards
and it might do this even though that's
not really necessary i mean
the rewards don't in fact depend on the
outside world
but it might model the rewards as
depending on the outside world and
the agent might take one such model
seriously
so my suggestion on that front is that
we penalize
world models that that require
lots of space to
make their computations so we'll be
considering stochastically computable
world models
and we can think of a world model as a
program
in this context that reads actions and
then outputs
observations and rewards iteratively so
reads an action outputs an observation
reward reads another action
that puts another observation in reward
and so on
and then we can penalize our priori the
computation space
used between reading action
a i zero and outputting reward r
i m minus one so that's basically the
computation that it does for that whole
episode
the idea being look it can output a
distribute it cannot put a correct
distribution over rewards
and observations just fine without
modeling
the changing outside world
so it doesn't need to devote the extra
computation space you'd expect to be
required
if it is if it were to be modeling the
outside world
and so world models that
don't worry about the outside world
except perhaps between
r i m minus 1 and a i plus 1
0 would have an advantage
over world models that are trying to
model the dynamics of the outside world
within the course of an episode
and then instead of that mixture over
world models weighted by posterior that
i showed you earlier you just look at
the maximum aquasteriori world model
and this also has good predictive
properties in the limit
although it is a little slower at
converging in the worst case
but then if you just have one model
that's better than these models you want
to be avoiding
you're not giving those models that you
want to be avoiding anyway
so that's the um
that's the basic approach to
containing an ai and getting it to care
about the point
the the duration of time during which it
is contained
and to try to construct an agent that is
predisposed
to understand that fact
so i'll discuss
some variants
that are going to be inspired by
the following potential problem which is
what if we ask for solutions to problems
when we're chatting with this ai and we
get answers and then we check them in
the outside world
after the episode is over well
convincing answers
which are what would lead to high reward
may be incorrect
so we want to we want a good a promising
research program for
curing some kind of cancer and it gives
us a perfectly plausible answer
and so we give it a high reward because
whatever expert operator we've sent in
is well trained and they recognize that
that seems like a perfectly good idea
but then they go out into the world
later and they try it and it was
a particularly good idea it was just
designed to appeal to their
sensibilities about
promising areas of cancer research well
that's not very helpful
but it's not an existential failure
um it's just ended up useless and so my
main position
on this sort of question is that we can
tinker with the protocol until it works
the way we solve
problems with ai systems currently we
give it a go the first time it doesn't
do exactly what we wanted but we play
around with it until it does
obviously that is not a good approach to
dealing with
existential failure avoidance
but if you've if you've reassured
yourself that
none of this tinkering is existentially
dangerous
then it's a viable approach to improving
how much use you get out of the thing
that said i'll i'll show you a couple
ideas for how you might improve
the protocol in a way that you might
expect to give you more useful
output
um maybe you guys are familiar with a
proposal
um from jeff irving
um called ai debate
um so we could imagine there are two ais
that are in the box with the operator
and they're debating with each other and
then the operator settles the debate at
the end of the episode
to determine who won the argument
um and the thesis kind of of this
approach as i see it
is that convincing but incorrect
answers to questions are fewer and
farther between when there's an
adversary trying to poke holes in your
arguments
so but i i should mention the
without the box i don't think that ai
safety via debate
is safe because either debater has the
incentive to
trick the operator into running a
program that takes over the world and
settles the debate in their favor
but if you put a box around the operator
they can't do that in time
and another one which i'll call
enlightening ai
is the operator is given a prediction
task at the end of an episode
and at the beginning of the episode the
operator can inform the ai about the
topic
that this prediction task will be
sampled from and then the ai
acts to maximize the operator's
prediction score
so the ai doesn't you don't have to tell
the operator exactly
you don't have to tell the ai exactly
what the prediction cast will be
um if you want to be giving you
broader instruction on the relevant area
and you can set this up in a way that
the prediction can actually be graded in
the far future so the prediction doesn't
have to be graded while the operator's
still in the room the operator can make
a prediction about
the effects of the outside world over
the long term
and the ai will be trying to make sure
we'll be trying to improve the
operator's ability
to make such a prediction
so the
way you set this up is that
the ai's model of how the prediction
will be resolved
doesn't take its actions as an input so
it doesn't imagine that its actions
are able to change the way the world
ends up
only the way the operator answers the
prediction question
so this is so what the ai has an
incentive to do basically is to pick
utterances that enlighten the operator
on the relevant topic
you're trying to reduce the operator's
uncertainty about the world
in a human readable format
and this was actually the idea that got
me thinking
about this whole setup because it
occurred
to me that if you did this you needed to
put a box around the operator and then i
realized
you can imagine a simpler version where
you just have a normal
reinforcement learner
okay so those are some variants but um
they may those are those are just about
failure modes that i don't see as
existentially dangerous
so i'll list for you a few
major concerns that people have
um with this that are
existential concerns um
and i'll go into one of them in more
detail at least
so the first is the obvious one
um you might have the concern that we
can't construct a box that would keep an
operator from communicating out
a tripwire avoiding message
so
the operator with the help of a super
intelligence
behind it who has convinced it to try to
get a message out
maybe you think that there's nothing we
can do
that would keep it from circumventing
whatever tripwires we've put up
um and getting a message out
and then the ai could influence the
outside world before the episode is over
second concern
is that if the penalty on space usage is
large enough to make it safe
it'll take forever for the agent to
learn how to act
well and i'll go into this concern in a
little more detail
but basically the real world does in
fact use a lot of computation space
and the third concern i guess is not
an existential failure but an
existential
uselessness this is the contention
that by the time a single ai
has the ability to take over the world
we will already be in a stable world
either destroyed or existentially secure
in which case the main ideas here would
not be particularly relevant
to the project of avoiding
ai doom
so i'll go into the second one of these
in a little more detail since it's
less obvious the content of that concern
um
and then we can come back to a
discussion of
all three of these so
the issue with the space penalty is that
the amount of data it takes to learn
to predict well depends on how much
prior you assigned to the truth or to
good models
so the amount of data it takes to learn
a hypothesis that has prior weight w
is order log of the inverse of a prior
weight
that's when you're doing full bayesian
reasoning and when you're doing maximum
a posteriori reasoning
what there we go okay um
sorry i couldn't advance the slide for a
moment when you're doing max
posteriori reasoning as we are
you can set up a pathological case where
it is
just inverse which is substantially
worse
in the priority
and
we could go into the details of
when it's more logarithmic or when it's
more inverse but i think there's a
plausible situation where for
most of the learning
the the posterior weight on the truth is
like
doubling and doubling and doubling um
with every time step um and then when
you get towards the end it's
you're kind of crossing off competing
hypotheses one by one
um and when it's doubling with every
step that's when you're in the
logarithmic phase and when you're
crossing off computing hypotheses one by
one
that's when you're more in this kind of
regime
that is maybe more background than it's
necessary to get to the point
that a big space penalty decreases the
prior on the truth
or the prior on perfectly good models
um and so that's the
that's the trade-off here um
and some computable world models are
deliberately deceptive
um and there's some debate about whether
deceptive models are even awkwardly
simpler
greater the extent to which this is true
the more you would need to penalize
the extra space you might expect them to
be consuming
um so this is i think
this would probably form paul
christiano's most serious objection
to this setup although it's
mostly an objection to
the style of reasoning that this agent
is using
rather than specifically the
um the boxing idea
let's
yeah if if people have questions about
this concern maybe we can talk about
that now
could you rephrase uh cristiano's
point yeah so
he would say that
deceptive models are a priori much
simpler
than honest models and that if you're
going to exclude them
you need severe resource you need to be
using severe resource penalties
um and then once you do that the prior
weight on the truth
is really really small and
in no meaning in no reasonable amount of
time
would you actually get something that is
able to predict accurately
thank you so
just a clarification a deceptive model
i imagine uh my intuition for that would
be something like
you have our world or just where
humans don't have qualia right that's a
simpler world
because that's something we've taken
away and it's
deceptive in a way that if the uh
[Music]
the ai tries to uh act as if we are
in a world where humans don't have
qualia it's it's simpler and it's also
bad
would that be an example of deceptive
world model
um well i'm not sure what the
observational consequences of humans not
having qualia are
but the output of a world model is just
observations and rewards
so um if there were some consequences to
the observations and rewards that it got
because of humans not having qualia
then yes but more generally what i mean
by a deceptive model is something that
is yes
quite like our world but then it it
produces errant predictions calculated
to mislead us
so it erroneously produces
the prediction that action 5
will lead to no reward even though in
fact action 5 would lead to perfectly
good reward
so that the ai avoids action five
so that would be an example of a world
model
that is attempting to influence our
world
through deceptive output
yeah i see a hannah
um could you go back to slide i i don't
know if you
you just or if we can ask questions oh
sorry am i audible now yes uh i don't
know if you'd like to stick to questions
about space penalties but if you'd allow
i'd like to ask a question about the
previous slide 13.
let's go back to that um i had a
question
about point three i don't think i
understood it so well
hold on let me reread it by the time a
single ai has had the ability to take
over the world we will already be in it
yeah
what does a stable world mean
like it's one thing to say that i i just
i don't
yeah i don't understand how we would
know or what
yeah well the logic goes through even if
we don't know it
but certainly if the world is destroyed
that's stable
and if we have some
world order where we're confident that
no one will be able to
create malign agi
that's another version of stable so i'm
just trying to get
one word for both of those possibilities
because in both of those possibilities
you don't need more ai safety research
or more aic ideas awesome thank you
i'll ask another question but i'll wait
for other people just to clarify a bit
more on that
um it seems like the big thing in here
is that it's a single ai
so if we imagine you could have a
multipolar scenario
where there are a number of ais who
through something like a
moloch manages to
basically destroy all value um that
would also be an example where no single
ai has the ability to take over the
world
but we've we've lost anyway and then
this doesn't matter
so that's probably the best story you
could tell
about how this might happen
and it would be the destroyed
possibility
um so
the you know if if we've gotten
if no single ai has the ability to take
over the world
yet and yet the world has already been
destroyed
it would have to be because of weird
multipolar interactions
between lots of agents um
the key reason why this is an objection
at all to what i'm describing is that
this methodology is
really just about containing a single ai
it doesn't
have any claim to changing the
probability that
lots of um
human level ais that are not super
intelligent
through their interaction cause some
devastating event so if you were more
concerned about that scenario
you might expect that by the time we get
to
a single ai with the ability to take
over the world
the situation has already been resolved
either
for the good or for the bad
another question stephen
if anyone else has questions by all
means don't let me talk the microphone
um so i'm a little bit disappointed not
in your presentation but there's someone
who usually
i like to present the work of you and uh
marcus hutter
and i think it's great theoretical work
but they're
this person is john fox and their
critique is always
um about the groundedness of such work
so if you don't mind i'd like to present
what they
likely would have said if they're here
um and
my from my interpretation again i don't
know if that's exactly what they'd say
but from from this person's thought uh
point of view they might say something
like how likely is it that
i don't know a deep mind or or some
other ai research organization
a government it doesn't matter some some
people who are well funded and
determined to actually create ai
how likely is it that they're not even
likely are they even considering
uh proposals for boxing agents um
because i've heard from other people
recently from darpa at least that
they're trying to build like general
intelligence but they're not trying to
or they're not going through the process
of boxing it um
so like how likely is it that they would
go through such a
uh slow uh means of iterating
um i don't know it depends on what other
options
what other options are available and
what um
you know if they can be persuaded that
you really need to be careful here
and if they can be persuaded that this
is
a good approach um
so i don't have any experience in
lobbying governments
um but
um i
could imagine a situation where
if the academic consensus were
that this was a safe way of doing things
and other things weren't that they're
that that could make its way somehow
into
the dod
i thought you were going to ask a
question about the
relevance of the ikc model to
institutions that are doing things with
neural nets
uh i mean i'd love to answer that too i
was going to ask a more
pointed question about bayes theorem but
i was going to wait for other people
so please answer your own question and
then maybe we can go to andre
my and my basic stance on that is if i
thought
agi was going to be made with neural
networks in five years i would still be
doing this approach um because i think
it's
a good way i think the best way to
understand
general intelligence in practice is to
first try to understand it in theory and
you can approximate ideal reasoning
attractively
with clever heuristics presumably
um but if you don't have an idea of what
it is you're approximating
you'll probably be groping around in the
dark
okay but now back to questions that you
guys are actually
posing
here i i just had actually more of a
comment on maybe like the government
lobbying i don't know if this is really
relevant but i i don't know much about
ai but i know about
uh government and i think if government
almost
a lot of people think of government as a
monolith i don't think that's the right
way to think about it
i think i think you should think about
government as like a forest
you know there's just like a whole bunch
of different processes
and maybe there's like a forest warden
who's like trying to direct it
in some sense but it it's very much
yeah a messier thing than that um
but just maybe to be specific about like
influencing government policy
i think there's lobbyists who kind of
try to affect government policy at the
legislature side
and so if you wanted the the legislature
to pass some sort of law
saying there's additional funding for ai
safety that's the way you would do it
um but i don't think it's you need to be
nearly that's like super
impossibly difficult so just like don't
even think about that um but i think the
easier thing at least from my experience
so
i did a lot of like economic modeling at
the government in trade policy
and we were like you know phd economists
and we're really worried about the
approval of our peers
and if we did like a bad job modeling
like our peers in the academic community
would like make fun of us
like say we had like bad standards or
things like that we didn't we didn't
like that
so i i think actually you can kind of
influence like the specific i don't know
if this is darker or higher
i wasn't making sure or whatever those
like act
people specifically or whoever the
machine learning modelers in the dod
are if there's like a consensus in
academia like
here are the best practices for ai
safety they kind of like
i mean they're not going to be as good
as academics they kind of want to do a
good job
so yeah
um that's a those are good points
i actually also have a point on that in
that there there might be two
effects that could uh cause this to be
used there's the
push where um there is a academics
everybody agrees that this is the right
way to do it but there might also be a
pull
one of the things that we might see in
the future are
consistent treacherous terms it is
possible that every time we make an
agi a very weak agi that seems to be
capable of
conceptualizing a treacherous turn it
will immediately take that action even
if it only has
a very small probability of taking over
the world and that might mean that we
would see
agis that always tries to take over the
world and always fail
and in that case if if the future turns
out to be like that
uh then um i think definitely that would
be a pull
uh effect from the people who are
building a more powerful agi
in trying to figure out how can we do
this safe because if we don't do that
then we're certain it will try to take
it make a treacherous turn
that feels related to the idea of like
an ai
fire alarm like maybe hoping there's
some sort of
uh scenario where like i don't know an
ai partially gets out and causes a bunch
of damage in some sort of visible sense
that kind of like wakes the uh the world
up which
it definitely could happen i mean when
you get
to a certain point of intelligence it
won't make
failed attempts it'll see that they
would
i mean it's a lot easier
it's hard to take over the world but
it's easier
to realize when you don't know how
and if you figured that out you won't
try
before that you know if it hasn't even
figured out that it's
not very smart if it's so dumb it
doesn't even know
how dumb it is yeah
you could imagine it making some failed
attempts but i could also imagine
what kind of dismissing it is like
look at how dumb it's being
nothing to worry about here related that
could you go to slide eight um which was
talking about like accurate beliefs
roughly
yes so i just want to make sure i fully
understand it because the way you
phrased i think too
yeah the way you'd phrase point to what
if it figures out a strategy for taking
over the world before it figures out
uh that's episode ends when the opera
yeah uh i guess
i was going to ask you like uh what are
your concerns about the agent not having
accurate beliefs because if you don't
have accurate beliefs about the world
um then you can lead to all these flacky
like
decisions for actions but you're saying
like it's not likely
that that's going to be the case is that
roughly correct or
does that sound intelligible i can
rephrase the question um
maybe it's a reference most people think
the mistakes you make when you're dumb
are not existentially dangerous
got it got it yeah so
you know we're not putting it in charge
of a plane
it's just talking to someone
um and if it's bad at
being useful in the conversation i don't
think anything terrible will happen
okay well thank you i think that
answered uh most of my time oh
i have one more technical question if
that's okay andre do you sell the
question
there is something i should say on that
more which is stroke
um in the limit
it will learn to predict well
on policy that is it will learn the
actual consequences of the actions that
it actually takes
if it's never taken action three
it will never be disabused of its crazy
notions of what happens when you take
action through
um so it can sustain
incorrect beliefs about
unexplored behavior
um but
um so so there is
a little it's not quite as simple as
saying there's
there's a phase when it's done and
there's a phase when it's smart and by
the time it's smart it's smart
and so you don't have to worry about
dumb mistakes um
it's a little you know i couldn't i
can't say that
so quickly fair enough
um the last thing i was going to ask
about i think you mentioned briefly but
i just want to make sure it's clear
you use bayes theorem which
would require that you know priors and i
know i think
people in this audience might be
familiar with it assuming you know how
to update
your priors is like rather contentious
all the time
uh but your approach is there are
computable approximations
that you could have or you can
substitute in for things like bayes
theorem that's roughly what you'd like
to
see if someone implemented it would that
be
a correct reflection of your view or
maybe not
um that is
far from the most pressing concern about
making this practical
got it i mean
there's a theoretical procedure for
updating all
these for updating your posterior it's
not efficient
there are lots of things that are not
efficient in this
um this is just a picture of what
idealized reasoning might look like and
i don't have huge insight into
how these will best be heuristically
approximated
fair enough and if no one else has the
last question
uh are you continuing working on uh like
agents that you can put in a box because
you're
really optimistic about this as as a
procedure or
or what are you like most optimistic
about ai safety wise now
if you can talk about anything um
yeah so i have been working on other
things recently because
um well i've had other ideas
um but
i am trying to go into
more i'm trying to go into a bigger
defense
or i guess yeah a bigger like
you know really trying to figure out how
you might construct this box and see if
you can actually be confident that it
could be secure
um which is not really computer science
research or research about what you
might do in the box but
research into whether this sort of thing
is feasible
so that's kind of the one thing that i'm
doing that's
still on this topic
um
how optimistic am i about this versus
others
i would rather
try and get a stable world with
pure imitation learning
um that strikes me
as an even safer route
um
[Music]
and i'm not sure if that's feasible
but if it is
i think it would be worth a shot
um and
[Music]
after that i think i would go with this
thank you for other people
fair enough um
i don't have any more questions but
thank you very much for this
presentation and for taking the time out
of your day to do it
so can i tell a story about how this
box might fail it's a far-off story
of course uh but let's say the year is
uh 2050
and this precise setup has in fact been
been made
right and uh obviously um
oh well we will say that it's uh this
has been built
by uh the american government and
obviously the chinese government
is is very keen on interacting with this
uh
superintelligence in a box maybe most
likely they have some kind of espionage
program so they know it exists
and they would really really like to
communicate with this uh
with this ai and in particular they
would would like to have some kind of
let's say a power sharing agreement
where the chinese get half the universe
and the
super get half the universe to maximize
to wirehead as much as possible right so
the ai would like this
episode um okay sorry yes i get it yeah
okay so the uh the ai would really
really like
one thing which is that the current uh
episode never ends or it's very very
late
so it continues to accumulate as much
reward as possible
so so you can only conceive of them time
steps but
um it can stall and make those try to
make those happen later
but uh just to be clear we are thinking
that an episode contains of uh like uh
100 time stem time steps or something
like that but if it was possible to
uh uh make more time steps
with that that wouldn't enable the air
to get even higher reward
what it has uncertainty about is the
effects of its actions
on the next hundred time steps
but the formula
the the argument to the arg max
when it's picking actions is it's
considering the effects of all these
different actions all these different
policies if you're thinking about its
whole future
it's considering the effects of all
these different policies on the next
hundred time steps
so its actions might affect the world
and it might
unders and it might have predictions
about
the consequences of those actions on the
world model
but at the end of the day when it has
its predictions about what its actions
do
it looks at the next 100 rewards that it
will get according to that model
and that's what it tries to maximize so
it there's no such thing as
more time steps happening within a time
within 100 time steps it's
just it
it's just trying to do the best it can
within the next hundred time steps
that's that's
its code but i don't think this will be
i mean your point i think will still
i think your story will still go through
it's just that you know maybe these
maybe it can arrange for these time
steps to occur
after it's broken out of the box and
then it can get optimal reward then
yeah so so maybe if you can break out by
a time step 50 then it can ensure that
it will get
one reward for for the remaining 15
times names
so one way it might be able to
communicate with the
with the outside world would be we
imagine that um
the chinese uh would be able to er
to both build something that can emit
neutrinos or something
like that and then on the uh the the
chinese could also put a satellite into
the synchronous orbit on the other side
of this room presumably
through espionage they know where this
room is so they can set new
trees through this room um possibly from
multiple directions if they have
multiple satellites
so they would be able to for instance
see
what position is the operator in and if
uh that's of course only only one way
because
i can't even just about see a way that
the uh
super intelligence could send out uh
could could receive these neutrinos but
presumably there would be some kind of a
causal bargaining
that would be possible where uh the uh
or if the super intelligence could just
infer from previous episodes that
the chinese were up to this it could act
on the assumption that they were that
would be assembled
um yeah
i think there are budgets for designing
this room
that allow for that possibility um
and if the chinese managed to do that
despite you know rather than just making
their own
um that could end badly
um
yeah i mean
what i think is more likely is if people
in china
are not persuaded that this sort of
containment is necessary they could just
make an ai that
they think is gonna do great things for
them and then it takes over the world
and kills everyone
um that seems cheaper and more likely
than trying to
create all these you know this neutrino
detection
array um
so i mean or they could just
kidnap the operators after they came out
and asked them what they learned
that also seems cheaper but yeah i think
if there's some other government that
is you know yes basically i think
if they did that i could imagine
endorsing a version of this
where that was a risk
um
yeah i guess that's my take on that
i could also think of counter arguments
like you can probably detect if there is
a
satellite in that particular spot sure
so i think uh johannes you have a
question and then mission next
so how do you actually so do you just
assume that you can make the agent
myopic
if you say for example that it only uh
cares about
certain number of time steps
yeah i'm looking around for a marker i
don't have one
yeah sorry is there is there more to
that
now and basically so you don't explain
like how you would do it you assume that
you can
build an application it's just like yeah
so
um
you know let's say i have a function
that takes actions and outputs real
numbers
so i have 10 actions and i have a
function that that
outputs a real number for each one i can
do
arg max over actions of that function
and it'll spit out
a number um and let's say i have another
function that
takes arguments of pairs of actions and
spits out
a number then i can take an arg max over
pairs of actions
um and the function
and the world model of the ai
can be converted into a function that
takes in
100 tuples of actions
um if this if it's an episode of length
100
and spits out a number according to its
expected value
so there's a function that's a function
of
100 tuples of actions and then i
pick the tuple of actions
that maximizes that function that's just
something i can
encode um so yeah it's
totally straightforward to write the
code for an agent
that is optimizing over
sequences of actions of length m that is
about
computing basically the actual reward so
you make sure that the actual reward
would not be like outside of the 100
episodes
uh it's it's a model of rewards it
doesn't have to be
exactly accurate but it's a model of
reward 1 and reward 2 all the way up to
reward 100.
um and those models i mean there's no
there's no extra machinery you have to
bring in to make sure that the models
aren't
modeling more rewards i mean you've
you've put all the models in the model
class and you know
that they model each reward as being
between zero and one and
um they assign probabilities to a reward
being output rather than
lots
but but just like in theory you could
have that the agent creates like
its own word model still right it's like
depending on
what kind of architecture you would use
so
i mean you can
so what you could what you would have in
practice
i mean and kind of the closest
approximation of this is
a list of world models in the agents is
picking
things on this list but it's not i mean
it's only picking things in that list
and everything in that list
is a function is a distribution over
an observation reward pair not some
arbitrary number of rewards
um basically
you can think of
you can think of the type signature of a
world model a world model
is a distribution over a single action
over a single observation in the word
given a history and so there's no
such thing as a model getting mixed up
in there that's accidentally a
distribution over something else
that's just a type error waiting to
happen
um so all the models that it can
conceive of
are just models that produce
a distribution over a single observation
in the world and it
you know there will be different models
giving different answers for what that
distribution is but
yeah there's no i mean
if it rewrote its code lots of things
could go wrong i'm not talking about
that but it doesn't have the ability to
do that
so if it's not rewriting its code for
one thing its actions don't
change the program it's running the
actions just print text to a screen
um and for another thing all the models
that it
might be elevating to to actually
consider to actually devote attention to
are
are models that are just distributions
over a single observation reward which
then get converted into a value function
with respect to tuples of actions i
guess i guess i'm just
confused about because you say like it
will it will output
like text and stuff like that but
like it seems like you don't specify
what is the actual
like like architecture of the model
that would do that right like you leave
some details unspecified and i'm
wondering if in these unspecified
details what would prevent
this part of the model from learning
something about for example that it
could manipulate the world in such a way
that it would get for example
more rewards in the next episode
so it can certainly take actions that
change the expected reward of the next
episode
but that is not part of its criteria for
picking actions
um it could
believe that certain actions will have
certain effects
going through strange roots in changing
its own machinery
but those models will either be will be
will either be complex or ruled out by
observations that that
are not what that model would have
predicted so
yeah if if you could run the model then
would it look like
so you there somebody enters the room
and now we assume you have like all this
infinite compute and so on
and the first time the human interacts
with it
would just output something completely
random
completely random text basically
yeah and then the human needs to train
the thing is like how how long would it
take
for the human to train this agent
so i guess that's a different question
that is yeah
that's a good question um and i didn't
um
i didn't mention to you another part of
this that would make that go faster
which is that for some episodes
a human can control the ai to
to take some actions that
show it some things it might do to get
rewards
that would help but learn the sorts of
you know that
that providing english text is generally
a good strategy for getting rewards
but even without that
if the first observation it gets is
wikipedia
it could build a model of the world
pretty fast and a world
and that model of the world might
include the fact that the world
responds pretty well to english text
um and so it could be pretty quick um
the yeah um
so yeah i think that's that's more or
less my answer
yeah there are more details in the paper
about
the actual learning algorithm
um and what we can expect from it um
and how it helps to have mentorship
so just just to summarize it's basically
you so but this kind of model is
completely specified
like you know you specify really
everything and
therefore like you know like
like for example the word models they
are
like you know what world models you have
like you specify them somehow and you
specify probably somehow also what kind
of
text you have if you don't want to allow
all the
like just maybe limit the range like how
long it can be
yeah and
then you basically use ikesi
but to learn what the human kind of
wants
or what what gives what gives reward but
you try to do it in such a way
that like you mechanically kind of
constrain
what the agents consider as good by
making the
specifying the rewards in such a way
that the agent is never incentivized
yeah to change the world i guess outside
the group
yeah you provide opposite i mean you
should be able to
if you want to
make it as quick as possible the ai
learns that it's in a box and then if
the operator leaves prematurely it's
going to get no reward for the rest of
the episode
you can try to provide observations that
make
that sort of thing quite simple
to imagine um
and you could even have you know
human-led episodes where the human shows
the ai what happens when it
when you do bad things and the human
leaves
and then you get nothing but yeah but
then basically you would think
if we could compute this thing then it
would already be
safe
if you if it could if it's model
included an opaque box around it you're
saying
um yeah i mean like basically just
like the setup you described you have
like the uncomputability and
yeah i guess there are some concerns
about the box i guess the question is
like
what assumptions do you need to make so
that you would think
this system is safe assuming that you
can
have like all the compute you want like
you're not constrained and
by any real world concerns basically
yeah um
[Music]
i would say that
um so i i i
tried to write down
[Music]
well so there are two assumptions that i
make in the paper
one of which is the realizability
assumption
um and the other
is i call the space requirements
assumption which is basically the notion
that
modeling the outside world's dynamics
within an episode does take extra space
um and
um so in theory kind of those are the
assumptions in practice
um i i would think
that um yeah
if there aren't really simple models
that are trying to screw you over
um that this should
that this and you know if you can make
the box
secure then this should work
i think he's by continuing is that out
for a second
may i ask my question yes please go
ahead
okay so i have a question about a
potential failure state for this system
as it can manipulate the human and as
you mentioned previously the human
is the rewards to the ai system not some
kind of predetermined algorithm
is it possible for the ai system to
cause unintentional harm to the human
for instance psychologically and are
there ways of mitigating it
for instance can it manipulate the human
by um
telling it negative things to give it
reward
and that said manipulation to the human
would then cause the human to act
destructively out in the outside world
yeah um the
yeah that seems totally possible um
because
human is also learning yeah i mean
you can you can mess people up by
talking to them
um the
so i could imagine that happening i mean
it's not my first guess for the way to
maximize reward but i don't know enough
to rule it out
um the
it doesn't it doesn't seem like an
existentially dangerous failure mode
um i think taking over the world is
hard um and even if you made someone
really misanthropic
if you didn't actually have an interest
in them going outside
at the end of the episode and taking
over the world it's not something they'd
actually accidentally
stumble into just because you mistreated
them um
uh and then recall that the ai doesn't
have an incentive
to try to get the operator to take over
the world successfully afterward
um and without that to take over the
world
yeah yeah yeah no i i was um
yeah no i don't take over the world but
to do harm
to do harm yeah um
could happen um
yeah i mean this is you know it would be
an unprecedented
interaction for a person it might have
psychological effects
that seems to be the sort of thing that
would happen
anytime anyone interacts with a
generally intelligent agent
um artificial agent i mean um
actually have a follow-up is this system
um could you use the system essentially
with an
intermediary instead is might this be a
way of resolving my issue
where you would have a human talking to
agent one which is then an intermediary
which
essentially fixes its model before
entering an episode and talking to agent
2 which is the super intelligent ai
how is agent 1 taking its actions
agent 1 is essentially exactly your
model
but then it communicates with agent 2
for a full episode
with a fixed what do you what do you
mean it's my model
like this agent that i've described yeah
so it's an
in a box except instead of the human now
it's ai one
leaves its box and enters box two
which for all intents can be within box
one why is it any safer for the human
why is it any like psychological why is
it any safer psychologically for the
human to
interact with agent one than agent two
i'm not entirely sure but assume agent
one is not a super intelligent ai
okay so it's the same it's not even but
it's had
less assume major one is research
constraint power or something
sure assume it's research resource
constrained by some means
um but since agent two cannot change the
view of agent one
within its behavior it seems like we're
getting
all the negative effects of the resource
constraint without
no maybe not i don't know it seems
like maybe it could do something um but
uh i mean i'm not
i don't think i think resource
constraints are hard
to use to good effect probably
um yeah
i don't know i think probably you'd take
a bigger hit on usefulness
than you would on
safety but
i don't know seems like something you
could do um
yeah i mean when we're dealing with
really bad failure modes we have to be
really careful
but honestly i don't think it's very
likely that if any of us
went in there that it would do lasting
psychological harm to us
um so
yeah it doesn't it's not something that
worries me hugely
so one of if i can just break in here
one of the
things i suggested you could dance in
the beginning was uh
the uh ai in a box boxes you uh
i don't know if you're familiar with the
uh with stuart armstrong's uh
rather interesting uh argument um
i think it's called intestinal
uncertainty or something like that
where it convinces you that you're
actually the one in in assimilation
that's something that this ai would do
although yeah
i don't see it as an incentive but it
could do that right
it could do a lot of things um i
yeah i don't know i mean you are in a
box
it wouldn't have to really convince you
of that
you um it would be a pretty easy
argument um
yeah it could i imagine
a super intelligent agent playing mind
games with me
could play mind games with me
i guess that's my answer sure um
johannes you raised your hand again well
first uh
to that i think or one thing that i read
for student armstrong was about
like okay maybe that doesn't apply here
but it was like
um you need to decide whether to release
the eye basically and then the eye tells
you
actually um you
like actually like if you don't release
me and shut me down with the shutdown
button
then um i actually created like one
billion simulations of you and i will
torture them and they will like be
tortured for
this eternal amount of time before you
are able to press the button
and it's like but they only get tortured
like
if you if you press the button to shut
me down if you don't if you kind of
release me then
then it's fine and then it says um are
you sure
that you are not one of these
simulations
because if you're one of the simulations
then you kind of you want to press the
release button but if you're not then
you don't want to they want to press the
shutdown button
yeah i'm assuming if
the ai thought it was worth its time
trying to get the operator to help it
break out
it would succeed
but because the operator
can't help it break out because it's in
this box
it's not worth it to the ai to try to
get it to
i don't know if that's the argument it
would use i would think it would be able
to come up with
even better arguments than that would it
actually be also considering
possibilities about
i think somebody asked like
extending somehow how long the episode
is like
because it seems like like like if it
would do that then
it would want to break out right number
of time steps or
number of seconds per time step
number of time steps i guess
number of time steps it's just it's just
picking
100 tuples of actions that
that lead to the highest expected value
um
so it's
yeah there's none of the actions it
takes
change the formula that the actions are
being evaluated against
um with more time in the episode there
might be
time for the person to break out at time
step
10 and then the remaining 90 time steps
could be spent with maximal reward
so we basically also assume that the
person who's inside the room like it
can't change
like their eye like it's impossible for
the eye to
say something that's like oh yeah okay
now you changed me like this and now
actually i can stay in the future
or reprogram it without triggering the
end of the episode
first
and then what i actually want to ask is
like why is it actually
problem that
that if the world model would contain
the agent like why
can you not have that
um so
[Music]
you're asking about like the the
realizability problem
the embedding problem yeah i think it
was on slide seven
yeah the
the theoretical issue
is um
let's say you have a bayesian mixture
over lots of models
over a bit string an infinite string of
zeros and ones
and one of your models
takes the bayesian mixture itself
and then flips the bit so if the mixture
over all the models predicts a one it up
to zero and vice versa
this ends up leading to a contra
contradiction if that model is one of
the models in the mixture
because you can show that
the um
that if the truth is in the mixture it
will converge to the truth but it can't
converge to this because it's the
opposite of itself
so that's the issue
um
basically i mean that's the general
issue that comes up when you try to have
an agent that is reasoning about a world
that includes itself
so is it about that you have this
distribution and
the agent when it decides to pick an
action
it kind of takes the opposite of what
the white model
does because it first flips around what
the
model says kind of no no i think it's
wrong that's just part of a reduction
that's part of an argument by a
contradiction as to why this can't
happen
um but really if you just want to
understand this setup
all you need to know is
it happens to be that there's no model
in the model class that corresponds to
something running this algorithm in some
part of it
and if you're interested in the reason
of why i didn't just fix that problem
then you get into these arguments um
but if you just want to understand what
the problem is that's
that's why this is there's a name for
this kind of problem
yeah i the grain of truth problem
and specifically maybe i think rain of
truth is just like the general problem
of like having the correct word model
in all the wordments you consider and
it's also like a name for
that the correct word model cannot
contain
cannot exist because it would need to
contain itself at least if we
say that the correct word model needs to
model the entire world including the
agent
that would be the problem part of grain
of truth problem
the yeah it's not a particularly
descriptive term
into why the problem exists but grain of
truth just means
assigning more than zero probability
with the truth
doesn't need to be probability one at
first you just need to assign more than
zero probability at first so that would
be the grain
of grain of truth um
yeah so that there is no name for the
specialized case
where it's for like the reason it's a
problem you mean
i know i was just wondering like is
there like a more specific terminology
because i thought like what you just
said like the grain of truth problem is
like the general
thing so is that like a more specific
thing and it's specifically about
that it doesn't contain the agent
yeah i've heard it called like
in like embedding
like an issue about
agent embedding in the world that's kind
of terminology from
miri um
yeah i mean there are lots of arguments
by contradiction and math that don't
have
pithy names i think i remember
a long time ago in the reading group we
read
a paper by max hodder about where he
introduced
ikesi and in that paper he uh
he uh he argued why i said it had to be
outside of the universe
um if i remember correctly it's quite a
while ago
but yeah i believe that's one place
where you can find a rigorous
description of why yeah but i
i mean if you just google the grain of
truth problem i mean this is
it'll bring you up to this sort of
reasoning
okay also uh now i should say that i've
only invited michael to to stay for as
long as he can
and we've passed the one and a half hour
mark so michael if you need to go
somewhere else
uh then of course you feel free to do
that otherwise
there might be more questions please i
should probably go in about five minutes
but i just realized i actually never
showed you this last slide
um which is just a summary
that true boxing and myopia is a
powerful combination
the idea being that it restricts the
scope of an agent's incentives
and we can probably ensure that the
agent understands this
and then if running an ai is
existentially safe we can tinker with
protocols
until the output is useful
but yes i have time for a few more
questions i guess i do have one
uh on myopia because i'm very myopic and
and myoka isn't uh it seems this this
agent is actually not uh
it's that it doesn't think in the long
term it's not actually myopic
as far as i understand um so that's
probably a very nitpicky thing to say
um but right it is my
myopia is something where you have a uh
a lint in front of your eyes which is uh
wrong
so everything gets out of focus so i was
thinking whether it would be possible
to to do something like that to help to
build an ai that was
actually myopic which had some extra
uncertainty about things or something
like that
so right it's not myopic in the sense of
having a wrong world model
or being unable to filter light
correctly that's coming from large
distances
it's just not farsighted in the sense
of planning for the long term so
it is caring about at most m time steps
in the future
rather than infinitely many um
and there's degrees of myopia you know
you could just be optimizing over the
very next time step
um but the key thing here is that it is
finally many time steps
which is i'm not the only one to use my
opiate in this way
but it has nothing to do with inaccuracy
about
well last so
uh a follow-up question on that is how
much uh do you see the uh
the safety coming from the fact that it
just doesn't
look so much uh how much of the safety
from the boxing and how much from the
myopia
you could imagine that you had a just a
standard
paperclip maximized but you told it you
can only look ahead
100 seconds
yeah um i could imagine that working
i mean i so
if it's just optimizing over the very
next action
i don't think it's worth taking over the
world
um if it's optimizing over the next
10 actions
yeah probably not sounds really hard to
take over the world
in 10 seconds right well
it's just about it's just a question
about whether you can break the clock in
10 seconds
i was thinking 10 time steps
um but
yeah the thing is yeah sufficient myopia
on its own should be safe but
you quickly get to the point of
uselessness and
i just don't want to be playing that
game um
the other thing you can do if you're
myopic is you can spin up another agent
that's acting
in lots of cycles for every one time
step of yours
and so you can arrange for lots of
things to happen
even only in a few cycles um
so i think myopia would do a little bit
of work on its own
but you can blow past
the safe horizons
when you have a box
i mean you can go blow past what would
otherwise be
risky horizons um when you have a box
so it's not kind of a
0.6 x plus 0.4 y
sort of deal it's more of a x and y
i mean yeah i think for to get most of
the power they're just both necessary
yeah and you have to have a quick
question uh
just before the five minutes or so very
quick question is like just about
like what do you think about miri's
approach specifically
well there are a few things they're
working on and i don't know all of them
okay i mean specifically the approach
that you try to figure out
things that you think are likely to be
known by somebody who would know
how to build a safe general intelligence
um i my approach is to try to go for the
throat
um with alignment
um
and it's the approach that i would
recommend
to others but i think they're doing very
important work
um and
it's hard for me to i mean you know i
think we need lots of different
approaches in ai safety
um so at the margin i think i would
recommend for more people to just
try to be going for the throat rather
than
analyze the intricacies of agency um
because there might be ways of getting
to the finish line
without ever understanding
all the possible solutions in decision
theory or
or things like that
okay so i think that's uh
all we can ask for you to for today
thank you michael very much for
your time and your presentation and your
answers it's been a pleasure to
hear and i hope uh we can also invite
you back
next time you you have a yeah thank you
so much for having me
so you guys thank you thanks
all right and then i have a uh just uh
some bookkeeping for the next session in
the ascc reading group
we will be uh discussing roman
gambolsky's
ai safety skepticism and that will be in
two weeks
yep and i guess that's all we have that
was fun
yep there are some great questions a
great discussion
thank you thanks very much