Life is like a game - DeepMind: The Podcast (S1, Ep3)

[Music]
artificial intelligence is slowly
appearing in every aspect of our modern
lives it's in our smartphones of central
heating on our side boards and in our
cars but what about artificial general
intelligence that is the real quest the
aim to build an agent an algorithm that
can learn to solve any problem from
scratch without being taught how
welcome to deep mind the podcast I'm
Hanna Frey I'm a mathematician who has
worked with algorithms for almost a
decade in this series of podcasts we're
following the fast-moving story of
artificial intelligence for the past 12
months we've been tracking the latest
work of scientists researchers and
engineers at deep mind in London we're
looking at how they're approaching the
science of AI and some of the tricky
decisions the whole field is wrestling
with at the moment so whether you want
to know more about where technology has
headed all want to be inspired on your
own AI journey then you've come to the
right place now in the last episode we
were talking about how pitting
artificial intelligence against
world-class players in the game of chess
and the game of Go is about much more
than just showing off what a computer
can do human players can learn from how
the AI plays and improve their own play
as a result and there's also a bigger
picture the world of games provides the
perfect mini universe to try out
everything we want our artificial
intelligence to do but intelligence is
much more than just championing or logic
intelligence requires other skills like
the ability to collaborate I want to
introduce research scientist max ei de
Burgh max and his colleagues are trying
to work out how to train agents to work
together as a team so imagine this a few
decades in the future we have all these
AI systems out in the world doing
different things but they maybe have
never seen each other before there's
thousands of these things hundreds of
thousands each have their own objectives
but somehow they have to cooperate and
compete in a sensible way and in a very
ad hoc way in a way that they've never
seen each other before humans are really
good at this when we want to be anyway
even when we haven't encountered another
person before we still know how to
understand their intentions and how to
interact with them our agents of the
future need to be able to do the same
thing with each other we already have
things like Google home and these sort
of smart devices out there were probably
have more and more of those and you can
imagine them having to interact and work
with each other and one device may not
have ever seen another device before but
they still somehow have to interact and
get things done whew are we talking
about like your your Google home and
your dishwasher here this kind of stuff
oh yeah potentially you know your
dishwasher might want to actually go on
its cleaning cycle but Google home wants
it to you know clean all the dishes and
say what's best for you as the person I
donate and who gets to decide just as a
side who rules supreme your dishwasher
your Google assistant yeah I don't know
there's an important distinction here if
you've got a smart light bulb that you
could program to come on at six o'clock
in the evening that is an algorithm if
you've got one that can learn your
preferences that can understand when you
like the lights to be dimmed what kind
of mood lighting you like when you're
reading that is a I but as we switch
away from building things that do rigid
pre-decided tasks we're asking our
technology to read the situation and
react to what's going on around it and
in the long term that's going to require
collaboration so in the spirit of trying
things out in a toy universe the team at
deep mind have been trying to find
inspiration in another kind of game one
taken straight from the school
playground this is capture the flag you
know the deal the first team to steal
the flag of their opponent and bring it
back to their own base wigs if you get
tagged by the opposition then you're out
of the game oh come on don't cry max
dropped whole populations of AI agents
into a digital version of the game this
is an on screen version you sort of you
just see your first-person point of view
so you have to sort of look around and
move through this 3d world from your own
first-person perspective but interact
with these other things which see their
own first-person perspective so here
there's no centralized entity or being
that can see no army commander every
player acts independently
they only see their own observation and
the way we train these things we
actually train whole populations of
teammates you know let's say 30 agents
in parallel and they're all playing with
and against each other rather than just
creating a single agent Max and his team
build an entire classroom of them 30 in
total and for each round of the game he
randomly selects a few of the agents
from the class to play together on a
team by doing this thousands and
thousands of times each agent will learn
from their own experience but because
they're playing with each other - with
their classmates as it were they have to
learn to interact with someone who's
different from themselves the problem is
when we start they're actually just all
very random they're just bouncing about
the place without a clue and then one of
them will discover something and will
start actually let's say taking control
of the flag and actually scoring points
and at that point there's evolutionary
pressure on this population
and here's the clever bit max and his
team aren't just letting the agents in
the classroom play on and on forever
they're also using something called a
genetic algorithm a way to make sure the
whole culture of the population of
agents evolves so they're actually that
some of the weaker ones will be removed
from this population so it's almost like
you're making that population of 30 have
children yeah absolutely breeding them
together yeah
the original classroom of agents breed
together and have kids of their own and
as you go down the generations the
strongest traits survive but unlike
human children when an agent has
children in this setup they inherit
everything they inherit the knowledge
that's been gained from their parent but
you're mixing up their characteristics
as you go from one generation to the
next yes so this agent has to learn to
play a five-minute game of capture the
flag which is really you play five
minutes you do thousands of actions and
you just get a win or a loss and whether
you won or lost the game and somehow we
have to learn what to do with that and
so to help bridge that problem we have
this idea of internal rewards where
there are events in the game such as
picking up a flag or dropping a flag or
your teammate tagging an opponent or an
opponent tagging you all these sort of
things and we allow the agents to
individually evolve their own internal
rewards which is the reward they
assigned to each one of these events so
some agents are going to care a great
deal about grab hold of the flag and
other agents are gonna care a lot about
teammate tagging someone yeah this kind
of evolutionary group training means
that they can assume different roles
producing better results for stealing a
flag and with a bit of practice after a
few thousand rounds a teams of agents
become really rather good at this game
the absolutely smashing the great thing
about training an agent in this manner
is that they're robust yes they can play
themselves but they can play other
agents that have been trained in
completely different regimes they can
also play these in-game BOTS which are
sort of these hard-coded bots that ship
with the game but most interestingly
they can also play with people so you
can drop people into these games and
have you know an AI teammate or AI
opponents
what was it actually like to play with
an agent men to me does it feel like
they are getting what you're going to do
as well as doing their own thing it
feels less like they're guessing what
you're doing and more like they
completely ignore you in their very
ruthless humans pay a lot of attention
to other humans even in game scenarios
like humans will fixate on the other
players
the game but these agents have been
trained completely unbiased without
these sort of human biases your opponent
will run right past you and not even try
and tag you because they're so fixated
on actually getting the flag as quickly
as possible because that's what's going
to maximize their number of flair
captures and win them the game things
that really annoying human players would
do there's a kind of magic going on here
initially researchers are working on
these agents trying to see a way through
the muddle then there is that
breakthrough moment
when the agent gets it when they start
to behave like you think they should let
me tease you with care I cover cholo
director of research at deep mind I
remember training agents in the early
days the first time actually those
agents started behaving like okay it's
an environment it's trying to navigate
and strength to avoid certain obstacles
and whatnot the first time it starts
doing that it's actually it is nice it
is like it's it's quite fun to see that
because you know that it makes the
decision for itself I think knowing that
you have created an algorithm that can
take decisions I think that aspect is
quite enjoyable that is very
satisfactory it's worth remembering that
these games aren't just a trivial
pursuit for deep mind they've invested
in this rigorous training for a reason
they want to see how an AI develops
these kinds of skills for itself we
spend a lot of time in this
capture-the-flag work looking into the
the neural networks of these agents to
try and understand what they care about
and how they represent the game world
and what was really cool is that we
found that the agents actually had a
really really rich representation of
this game world without being told
anything about the game world itself you
know these agents just look at the
pixels of the screen yet somehow they've
clustered their you know internal
activations into things like oh I'm in
my home base I'm in the opponent base
I've got the flag and I can see my
teammate ahead of me I'm looking at the
opponent flag carrier while my teammate
is holding our flag and you can even
find individual neurons which just
activates if for example your teammate
is holding the flag you can totally
understand
how the agent is seeing again as you go
through I'm not sure about totally
understand but we're really getting an
idea of what is being represented
strongly of what isn't being represented
strongly
maxi's ages are using something called
neural networks it's a type of machine
learning algorithm that is loosely based
on a simplified version of the human
brain layers on layers of artificial
neurons are connected together in a vast
network and fire information between
themselves by looking inside an agent's
electronic brain max can work out which
micro level connections are responsible
for what macro level behavior and this
could be hugely beneficial as AI becomes
more integrated in our everyday lives
the hope is that well into the future
we can start actually having agents
which can go out into the real world
interact with humans with other agents
without fighting without fighting
being sensible yeah not squabbling too
much unlike humans yeah exactly
[Music]
games without frontiers teamwork without
tears but there is a big leap between
board games or simple games like capture
the flag and the big bad world with all
of its complexity and messiness you'll
remember David Silva the man who brought
us alphago the agent that defeated the
world champion at the ancient board game
of Go well he's also involved in pushing
deep Minds AI into ever more perplexing
environments in the context of games I
think there is a further challenge which
is many people in the community are
moving towards which is to take the most
challenging computer game in this case
it's the game of StarCraft many people
in the AI community are viewing this as
the next grand challenge now how can we
actually devise agents which can play in
this very rich environment which has
challenges which are not only different
but many times faster than go in other
ways
this is deepmind the podcast an
introduction to AI one of the most
fascinating fields in science today have
you ever seen footage of those vast
eternal entire arena of dedicated fans
excitedly watches on in support of
highly skilled players sat on stage in
their gaming chairs armed only with a
keyboard a mouse and a computer screen
well chances are they are playing
something like Starcraft 2 created by
the American video game developer
Blizzard Entertainment
[Music]
it is a monumentally tricky tactical
game where you play as one of three
races the enigmatic Li named Zerg
Protoss or Terrans each player has to
mine resources build an economy and
acquire increasingly sophisticated
technology all the time trying to defeat
your alien opponents in a futuristic
rather bleak looking landscape your
field of view of the simulated game is
limited by a moving camera that you have
to operate and so there's no way to see
everything at once often you can't see
your opponent at all and it is played by
tens of thousands of people sometimes
for hefty cash prizes and the human
players are staggeringly fast the best
in the world can manage up to eight
hundred clicks in a minute feeling
inadequate definitely super cool that I
can work on one thing that has been
certainly a passion of mine in in my
teenage days meet Oreo venules a
research scientist at deep mind he is an
ex-pro Starcraft player and co-leads the
Starcraft ethic a deep mind as you
develop a new algorithm or a new idea
when you test it you actually see it
play better the game you you like so
that's very rewarding and very visual
right that you try something new and you
really see oh my god it's really
understands how this unit works
Starcraft is a serious business so
serious in fact that has now been
professionalized and for oriole that
proves it is a game that pushes human
intelligence humans found it interesting
so that means it's an interesting game
that challenges intelligence and
creativity in ways that we like that we
spend many hours playing so how good is
the AI at the moment then how well can
it play Starcraft it's better than any
AI anyone has ever built and it
obviously has learned from experience
not from someone knowing the game and
encoding some set of rules this is I
mean one of the most complicated games
we've ever tackled it's challenging kind
of our understanding
and our algorithms quite a bit the deep
mind team decided to see how good their
work-in-progress really was by inviting
two of the world's best Starcraft 2
players to take on their own algorithm
so let me introduce deep minds alpha
star the first artificial intelligence
to ever take on top professional players
it plays the full game of Starcraft 2 by
using a deep neural network trained
directly from raw game data by
supervised learning and reinforcement
learning your commentators are dan stem
Koski aka artosis and Kevin Vander coy
aka Rotterdam well first of all it's
really awesome to be here together with
you then we were both I think incredibly
excited to see how this evening on fault
I mean this is just so exciting that D
mind is doing all this taking on alpha
star in this benchmarking match is
German champion dario vooc better known
as TLO he's normally a zerg player but
he's playing as Protoss for this match
Kevin and Dan are excited maybe even a
Tartar hoover excited so incredibly
excited oh my god this is like the most
excited I have personally ever been for
a man can't wait to really break down
some PvP so this is alpha star this is
an AI that we don't know how good it is
yet but already we have some interesting
things happening now I'm not entirely
conversing with the Starcraft player
glingo here so I will just say that
alpha star stalkers are laying down some
sharp moves feels to me like so far
these attacks have been very well
planned by alpha star
and they're relentless it loves to
attack and in a matter of minutes is all
over well that is it the G G is called
the good game here from TLO and the
first game from alpha star against a pro
gamer goes to alpha star David Silva was
there ringside we have a team that's
been working on this and ramping up our
development over the last few months and
this represents a you know a milestone
where we actually for the first time
work saw an AI that was actually able to
defeat a professional player so we have
a quick word with our defeated
challenger TLO when I was practicing
most of the humans I played against
played very standard starcraft once
again i assumed after the first match
i'll probably have a good idea how to
play against civilization i did not next
up the main event alpha star versus
poland's finest gregor comment better
known as mana one of the world's
strongest professional Starcraft players
I need to hear what you're thinking here
because that looks scary
yeah alpha star like he's not scared
about the around so well if I would be
playing against a human player right
there nobody is going up that front I
should point out for those of you that
play Starcraft that these matches are
taking place under professional match
conditions on a competitive ladder map
and without any game restrictions this
version of alpha star could see the
whole of the game map at any one time
but otherwise played in a comparable way
to humans our goal is not just to defeat
these players alcohol is to do it in the
right way all right
and the result alpha star 5 mana nil
I should tell you that monoplane a later
version of the algorithm in the end and
one so all in all five one now to
understand how an AI could learn to play
Starcraft or even ours put me to the
test a match to the end mathematician
versus machine of course a quite funky
looking Mouse in front of me and a
normal keyboard and on the screen there
is a very mean-looking alien yeah a
process sort of like an elephant meets
always got fists I wouldn't want to meet
him in a dog no no you see my friend or
not he is you I use mana you're gonna be
the commander of this particular race I
quickly found out that there is a lot to
take in Starcraft is perhaps not for
beginners you have your worker bees
collecting resources for you and these
these are all I mean they're almost like
ant creatures right so running out and
grabbing crystals exactly and you need
to try and work out how your actions
will affect the game in future this is
not easy for humans to learn let alone
agents that have absolutely no context
no object recognition and definitely no
former Starcraft champion to hold their
hand enemy it's it's gonna be pleasant
you just came to kind of find you and
now you see what you're doing which is
absolutely nothing so far we have we
have done nothing part of the challenge
of Starcraft is that there isn't an
ideal strategy that wins every time
it's a bit like rock-paper-scissors in
that way the winning tactic will depend
on how your opponent plays but remember
you only have a very narrow field of
vision outside of where your camera is
pointing your opponent could be up to
anything because you don't see the other
player you must decide when am I gonna
see it do I already know what it's going
on and should I not go and Scout what
it's doing but maybe if I do that he
knows that I know and so on so forth so
these kind of imperfect information as
of sacrifise extremely interesting as a
player and it's gonna be testing our
agents to levels that we haven't seen in
any other game and then of course
there's sort of details that have happen
in the game that you must remember for a
long time
advice I should have listened to more
carefully perhaps we're being attacked
are we gonna die but I think this
discovery phase right where you would
now basically you would lose you get the
reward of minus one yeah and you start
again if I was an algorithm I wouldn't
be upset by losing I would just reset
and go again each time armed with a
little more knowledge but to even be
able to play Starcraft in the first
place to even be able to operate the
controls the AI had to master quite a
few transferable skills you've noticed
when you were playing that there were
some movements that were resembling what
it was like to maybe navigate the web or
like operate your laptop namely click
drag and click drag and drop like select
rectangles and moving the mouse maybe
combining Mouse with keyboard and so on
and we tried exactly the same agent the
same architecture absolutely everything
the same way the same code almost and we
chase we change the environment instead
of saying now here is a Starcraft please
play to win we said here is paint
Microsoft Paint as an environment
interact with it and I'll reward you if
what you paint looks like a face and it
actually worked so I think that's just
learning these basic skills of
point-and-click interfaces that apply so
mad in so many places the same agent
that play Starcraft control real faces
in Microsoft Paint
right and here the the point to be
clarify is not the same agent that was
trained to play stack up is the same
algorithm that can train to play
Starcraft can also train to do pain
put that same algorithm to work drawing
celebrities in paint and it can capture
all the main traits of the face clicking
and dragging the mouse to recreate shape
and tone and hairstyle much like a
street artist would it's the same
technique but if you will it's kind of a
brain that is blank and then this brain
can learn to do these or that or that
and then we kind of by acting in the
environment repeatedly and getting
reward the the brain weights or it gets
shaped to do these tasks or that task or
that task we are not yet at the point
where the same brain that's both like we
do but obviously that's one of the
things we we would be very interested in
tackling next as well because that's
stepping towards artificial general
intelligence I guess exactly and that's
that's what we do every day that is the
ultimate goal and it's a topic of
conversation that's never far away
whoever in this building you find
yourself talking to because the point of
getting AI to play games like Starcraft
or go is to enhance our understanding of
what intelligence actually is here's
Ryan Hansel from the deep learning team
we write programs we run those programs
those experiments where we might train
an agent to play a game for instance or
to solve a puzzle in a simulated world
and then we look at the results of that
it really is trying to understand this
puzzle of learning and representation
memory control in terms of actions that
a robot would take there's so many
complex parts to this big puzzle of what
is an intelligent being what is an
intelligent agent but if you ask people
what they think the future of AI looks
like it tends to be wrapped up in
something a bit more physical something
that comes complete with moving arms and
everything I think one natural challenge
for AI which many people are centering
upon would be to actually have an impact
on the real world in the guise of
robotics to actually see a robot which
is able to to move
to grip to manipulate to even have
locomotion in anything approaching not
even what a human does maybe even an
animal I think this this would represent
a major stride forwards more on that
next time if you would like to find out
more about the themes in this episode or
explore the world of AI research beyond
deep mind you'll find plenty of useful
links in the show notes for each episode
and if there are stories or sources that
you think other listeners would find
helpful then let us know you can message
us on Twitter or email the team at pod
cast that deep mine comm you can also
use their address to send us your
questions or feedback on the series but
for now let's nip out for a bit of air


AI Safety Reading Group (Session 40)

hello and welcome to the 40s session of
the AI safety reading group and today we
have read an article called racing
through precipice a model of artificial
intelligence development by Joe
Armstrong Nick Bostrom and cars children
they are all working at the future of
humanity institute there whereas in 2013
when the article was written nick
bostrom is the director and your
armstrong is a AI researcher and culture
minh a Research Association so the topic
at hand is AI arms race which is of
course a subject that allowed people
care about because it kind of looks like
we are headed towards an EAJA arms race
the last week we read an article which
strongly implied that they also believed
that we were heading towards an AI arms
race and of course arms races have
hadn't before in history I've taken here
in a teacher of HMS dreadnought which is
yeah Chris will probably feel very proud
by seeing this feature enables the arms
race which devastated Britain's economy
and Germany's economy at the turn of the
century and that's because arms races
are in a very real sense a zero-sum game
there is there might be a winner and a
loser but there's nothing really gained
from an arms race in this case we are
talking about in particular the effects
of an arms race on safety precautions so
we're going to assume that safety
precautions are possible but you can
choose what level of safety you want a
big point about an arms race is that
there's a first-past-the-post latch
advantage to being to reaching the goal
first and that's why you have this arms
race and there's some degree of enmity
between the teams meaning that the
team's hate each other
and this influences what level of safety
they are they're prepared to have to
accept does an old setting saying just
before just after the Second World War
which was better red than dead which was
used as an argument for by the peace
movement and which is which shows that
there is that it's better that the
Communists wind cold war and that we all
die and many people reacted with the
opposite and much more known slogan
better dead than red which represents
full enmity saying it's better that we
die then that the company is swim so
this has in the paper by bass drum and
others been formalized into a setting
where they are a number of teams here is
represented by the letter in each team
has there is a capability which is each
team how good are they at making a super
intelligence and it's represented by a
number from one from from 0 to M you
here and each team choose a safety level
which is from 0 to 1 of course you can
rescale accordingly and and this allows
it seem to get a score now I hit myself
again with which is called C minus s
where the higher your capability is the
the better you are at building it and
the most safety precautions you have the
voice it becomes and after you've either
won or lost this this AI arms race that
might be a disaster and AI catastrophe
and that is the probability s that
you've chosen so it's possible for teams
to say we want to be one hundred percent
sure that this doesn't make a
catastrophe and it's also possible to
say we will a fifty percent or even one
percent chance of success is enough for
and that's represented where catastrophe
gives zero utility and success gives one
utility in in this model further there's
a level of images like I just said where
a number such as a one-fourth means that
an AI could just fit is four times as
bad as solution so that won't mean
better read than death in this case when
energy of 0 means you don't care about
who wins enmity of one means that it's
just better better dead than red really
and so the world then the number mu up
here is hugely important this means how
hard is it to build a super intelligence
if it's high then the capability matters
more and if it's so then safety matters
more in the sensor if you skip on safety
then you increase your chance of winning
a lot so you can see new is it
represents in some ways how much safety
matters and compared to come to keep a
building the next part in the I'm
hearing myself again maybe through your
egg and the next tile is sorry it's
really difficult to talk and its
information levels so the no information
level that's where no one has any idea
about the other teams and they don't
know what their own capability is and
I'm saying this is roughly our where we
are right now nobody has any clue about
how hard it is to build a super
intelligence and they don't know how
close others are either later we will be
at a point where some people can
estimate maybe when we roughly how far
they are from building a super
intelligence and at some point it will
be published not just to each team how
close they are to building a super
intelligence but also how close are
their competitors so these three
information levels are modeled in the
first case when no one has any clue
this is a post a fully symmetric
situation every team will do exactly the
same because none of them really know
anything so everyone will choose the
same safety level and if there are five
teams then each of 150 chance of winning
and what safety level they should choose
depends upon how much they hate each
other the enmity at times the number of
competing teams and and if mu is smaller
than this enmity times the number of
teams then the safety levels should be
reduced so that is the optimal strategy
to reduce your safety level if you Eva
if E is relatively high or the cables is
relatively low or the number of teams is
reasonably high so this is a and this is
something that can be grabbed and and we
will get to the press to get some
visualization of this but important
thing is if there's no information and
the capabilities Madame au and enmity x
number of teams the correct option is to
choose one hundred percent safety the
next case is where everybody knows how
good they they are themselves but they
don't know how good the other teams are
so you know your capability we write X
here and you choose the safety level f
of X based on that of course this is
symmetric right no one knows anything
about the others so each team will
choose the same strategy and
[Applause]
the the question is what are the teams
actually trying to obtain and they're
trying to not just maximize the chance
that they are that they are winning but
also the chance that they are winning
and there's no AI catastrophe afterwards
so this is the total utility from both
wailing and and avoiding a I catastrophe
and if the slow enmity you add the risk
that other teams win so that utility so
that is somewhat more complex thing that
said that each team is trying to go to
do and it is an answer the question okay
and so we see that the team with the
highest capability always wins in this
case and that's because even if you add
more safety you would never if you
become a slightly smart slightly more
capable that it's always a bad strategy
to add even more safety to compensate
for that that's x minus s + x that's an
increasing function so it will always be
the team with the highest expert wins
and what strategy you should follow
depends on whether you can build is
higher than enmity x number of teams
minus the energy plus 1 and this is
proven in the paper and i won't go
through the table but just say if your
team is more capable you choose one
hundred safety if you choose if your
team is less capable you will reduce the
safety bye-bye your pulse is divided by
this number here and this is something
that gives a the total risk of
catastrophe in this case is something
that can be calculated as a moderately
intimidating integral it's for mad
magician it probably doesn't look so so
scary but it's reasonably it's something
that can be calculated in grad and I
think that's the important part here for
public information we of course no
longer a NASA metrics
case because the the team with the
highest capability they would have one
strategy and teams with a lower
capability they will have a different
strategy and in this case they choose a
safety level determined by the
difference between the capability of the
top team and the second ranked team
that's the key thing that everybody will
be looking at so imagine there are like
the Chinese AI and the American AI and
the American AI appears to be closer to
becoming a superintelligence compared to
the Chinese then the difference in the
capability between the Chinese team and
the American team becomes the crucial
factor and here we are looking for
something called a Nash equilibrium and
Nash equilibrium is when no players
anything to gain by chain changing his
strategy in isolation mean that there
might be something better is to change
change in Australia at the same time but
in this case if we restrict ourselves to
luton at Nash equilibrium where then the
strategy that is best for the top team
is to choose a safety level that is the
difference divided by the energy in this
case the second thing is shouldn't us
decrease it's a scene or to be able to
compete and the the risk of an a a
catastrophe can be calculated is almost
the same age grow so how does this look
there are four actually five grass in
the paper but let's have a look at this
graph over here we have to change and on
the y-axis the risk of a catastrophe and
that's what we really care about here
and on the horizontal axis we have the
relative importance of capability so
what we can see very very clearly is the
more capability
matters the lower risk of a catastrophe
because you can see if capability
doesn't matter we'll go almost one
hundred percent risk of catastrophe and
and these capabilities matters very much
we will get out decently dope risk and
and the I can just explain the lowest
line the cooling mark line that if
there's no information the dashed line
here is if there's private information
and the brown line i think is if there
is public information so rather than
going through these three graphs i will
look at one variable at a time and as
well back and forth so i'll go forward
to a new how hard the problem is and
obviously as you can see from all these
graphs the further we get to the right
the lower the risk of a catastrophe so
in a way we really hope that the problem
is hard that's not really something we
can you can't influences we can't take
any action that will make the problem of
creating a super insulting sahara and we
can the only thing that the article
suggests is we try to avoid easy
solutions that might work what they call
a moonshot that's probably a bad idea to
do so that's the first thing the next is
enmity and here you have in the first to
press you have one hundred percent
enmity which is better better dead than
red and in second to we have fifty
percent image where it's true times
better to be read than to be dead and as
you can see the graphs here compared to
these have a substantially lower risk of
catastrophe so quite obviously reducing
images makes an AI arms race much less
dangerous
and it should also be mentioned that
these graphs with the public information
assumes a Nash equilibrium and there's
two teams they might be able to
coordinate in some way and one of the
ways they could coordinate is by saying
both of them we will not make an AI to
kill everyone we will if we make a super
intelligence will use to implement
something like career extrapolated
position or something like that which
will dramatically reduce imaging if they
believe each other will do that energy
will be reduced serum and when n which
is which used all the way to zero the
risk of of AI which goes down to zero in
this model so in this case and I think
that's a really important point that
they don't really go into if we trust
each other enough the risk goes down to
zero if we I think yes and i like that
point but but it's not really written in
the in the article the next question is
the information where you have these
three graphs no information with the
lowest risk in in general private
information with somewhat higher and and
usually you have public information at
the top but there are a few cases here
where if you have low energy and and the
importance of capability there are some
cases where it's slightly better to have
public information but in general
private and no information is better the
last variable is the number of teams and
here you can you compare the it's not so
easy to compare in this graph whether
this point is higher than this point
here that in the left it's two teams and
in the right side seams and hearing
myself maybe through you Chris
nothing and maybe someone else and if
you could yeah no not that great so it's
not so easy to see whether the rightmost
graphs are higher than the leftmost
graphs but if you eyeball you can see
more teams is generally worse and it's
more teams of worse it means that if we
can if we can convince seems to join up
together to merge then that will reduce
the risk of AI of AI catastrophe and
there are a few cases where it does
muscles we're more teams are X receiver
but that is a color and of a fringe
effect and it's really hard to split up
a team in any meaningful way without
capability from one team going to the
other team I mean then they would have
to forget some part of us and that's not
problem not really fitting that was the
article
and it's partly intuitive because
there's how intuitively how do these
grab and the results in this paper match
up to our intuition and I would say a
reasonable on three out of four
parameters and if we say how hard the
problem is and one of the most negative
on how hard the problem is is aaliyah so
you'd cowskin was that he believes that
safety to do this problem this safety
safely compared to doing it unsafely is
a it's five times as hard that means
that mu is 0.2 and hearing myself alan
has gone okay and but but we really hope
this is not the case because it's five
times as hard to do safe as it is to do
unsafe where we will all most likely be
in a situation where the best strategy
is to choose as little safety as
absolutely possible and I'm still
hearing myself sorry if that's any way
anyone could move the microphones away
from this data I would appreciate it
it's my microphone low
that might also be a solution yes so I
think so forward for the problem with
how hard it is that the fact that we
hope it's really hard compared to how
much safety matters we hope that safety
is a tiny tiny overhead to the so how
hard it is to do to make those are super
intelligence but we don't really know so
I think this lines up very well with our
intuition of course enmity we have the
intuition that in which is best because
we are social creatures so we we really
hope from a moral point of view that
enmity is bad and it does turn out in
this model that in which is bad and this
also seems like from historical arms
races like a very obvious result that
alliance fully with our intuition that
enmity is best it is if we can work to
make Chinese AI the project not be so
afraid of the American AI project that
won't be a really good thing the last if
we take a number of teams and I also
feel it's a reasonably intuitive result
that if you have a lot of teams then
each team will feel under more under
more pressure than the other teams
because if you if you really hate the
others and there are hundreds of teams
then you will feel that's a very low
change that I will be able to succeed so
this means that if the number of teams
just skyrockets you will feel very very
on you will feel like you are in danger
of loops and you'll be very tempted to
lower the safety level of your project
so I also feel that the number of teams
that that is bad to have many teams and
it's good as it seems merge that lines
up with intuition recently well the last
question is information which is not we
does not line up with insulation
and that no information is always best I
think a bass drum has later written
quite a bit about and that actually when
you when you really think about it it
does make quite a bit of options just
talk about a veil of ignorance that we
will that would be listed eventually
when we get closer to building an actual
super intelligence and people will make
different choices now that we don't know
who's going to win ten people will be
very pro-social and say oh we don't know
who wins so we should all say if we eat
that the winner should do Korean
textable a depletion or at least not
take over the world do something nice
for the AI that's what everybody is
going to say now that there's no
information but later when it looks like
they are winning there will be much more
reluctance to not take over the world so
I think if you if you think about it the
analyst analysis probably makes sense
and and what I think question right
about this is everything is written
about this whether this whale of
ignorance is a great idea is written
after this one so that's a that's at
least potentially it changed that he he
built this model saw this was an
unexpected result and thought long and
hard about it and actually changed his
opinion about openness in AI based on
this model that was a long answer what's
that are acceptable okay then I will
stop to recall them thank you for
watching I will stop now see you next
week


AI, Robot - DeepMind: The Podcast (S1, Ep4)

[Music]
the idea of creating artificial
creatures has obsessed us for millennia
but for the purposes of this exercise
before we go any further I'm going to
ask you to purge any of the following
thoughts from your head checklist
Hephaestus expelled from Olympus and
then built two servant robots the boot
of ant a hunter or spirit movement
machines of 12th century India made to
protect the relics of Buddha the
chess-playing Mechanical Turk just a
bloke in a box pulling levers Mary
Shelley's Frankenstein he walks he talks
not a lot else kubrick's how don't call
me Dave c-3po r2d2 canine ns2 and
seriously these are just numbers and
letters Robbie the robot robot lovers
Robocop
[Music]
feeling better okay let's get going I'm
Hanna Frey I'm an associate professor in
mathematics and I am AI curious and this
is deepmind
the podcast series where we look at the
fast moving story of artificial
intelligence we've been talking to the
scientists researchers and engineers
based at deepmind in London and we're
looking at how they're approaching the
science of AI and some of the tricky
decisions the whole field is wrestling
with at the moment so whether you just
want to know more or want to be inspired
on your own AI journey then this is the
place to be you see the thing is robots
cell
we've long lusted after the idea of
appending the natural order with human
ingenuity we just can't seem to leave it
alone and in this episode we are looking
at AI and robotics Marie Shanahan is a
senior scientist at deep mind he's also
a professor of cognitive robotics at
Imperial College London and growing up
Marie was utterly mesmerised by science
fiction so you can picture his face when
the Hollywood film director Alex Garland
approached him following the publication
of his book embodiment and the inner
life Alex contacted me and said oh I'm
writing a script for a film about AI and
consciousness and I read your book and
it you know helped to crystallize some
ideas and you like to chat about about
it and and so of course it was you know
it was a great opportunity to get
involved in a science-fiction film and
then and then to my great good fortune
it turned out to be an absolute cracker
and that is how Murray became scientific
advisor on the oscar-winning film ex
machina
I met with Murray to get a potted
history of AI more people tend to think
of AI is this very new thing a very a
very modern invention but it's actually
been around for quite a long time it has
the idea of artificial intelligence the
idea of making artificial creatures
dates back to Greek mythology but
there's a modern conception of AI
perhaps really dates back to Alan
Turing's paper published in the 1950s
where he first kind of asked the
question could a machine think and gave
a number of kind of refutations for
counter arguments to the idea that a
machine could think this is where the
Turing test this is that this famous
paper inaugurated the so-called Turing
test course cheering didn't call it the
Turing test which is the idea that we
should subject the machine to a test to
see whether it's basically whether it's
indistinguishable from a human in
dialogue the term artificial
intelligence was actually coined by John
McAfee a Stanford professor he was at
MIT at the time and John McAfee
organized a conference in 1956 bringing
together a lot of leading thinkers in
maths to try and scope out the idea of
building a thinking machine and he
coined the term artificial intelligence
what did they describe as at that time
how did they see artificial intelligence
John McCarthy in particular his idea of
artificial intelligence that he had in
mind was a kind of system that would
answer questions really and be able to
engage in dialogue with with humans
so something that you're talking to
although of course in those days it
wouldn't have been through speech it
would have been by typing in at the
keyboard and it was very much a
disembodied notion of artificial
intelligence so this system didn't have
a body and interact with a physical
world in the way that we do or animals
do or indeed that robots do so they
weren't really thinking about robotics
at that point I'm sort of imagining
something like how in 2001 a Space
Odyssey exact typing in Robbins yeah and
under kind of nice version you know that
their approach to artificial
intelligence was to build systems that
reasoned in logic and we now think of
that whole sort of approach to
artificial intelligence of using logic
and reasoning as so-called good
old-fashioned artificial intelligence or
Oh classical AI now I know I know I did
a bad thing there I mentioned how but
nobody ever said that this was gonna be
easy but to recap in classical AI you
have to write down a complete list of
rules for how you want your agent to
think if this happens then do this if
that happens then do that it's a nice
idea in theory but if your agent is
going to know how to handle every
possible scenario you can throw at it
it's going to need to be a long long
list there was a project called psych
which attempted to write out all the
rules of common sense to build an
enormous encyclopedic database of a
common sense so I mean there'll be
things like if you've got a container
and you put something in that container
and then move that container somewhere
else then the thing that was in it gets
moved as well then yeah stuff like that
you know and if you buy something and
pay for it you'll have less money than
you had in the first place
so do I think it's impossible to do that
I think it's impossible in practice
because it turns out that the sheer
number of rules that you would have to
write is absolutely enormous we might
have moved away from these long lists of
rules by now as a way to teach our
artificial intelligence in favor of
agents that can learn the rules for
themselves but the skills that we want
are AI to have like good old-fashioned
common sense are just as important now
as they ever were
[Music]
imagine one day long into the future a
wealthy computer scientist builds an AI
to manage his stamp collection he plugs
into the internet gives it access to his
bank account and sets in the challenge
to buy as many stamps as possible
at first the agent acts as its creator
intended signing up to eBay bidding on
stamps but after a while it gets another
idea more money equals more stamps so
why not start trading on the stock
market to make more money and it soon
realizes it can get the stamps cheaper
if it can get them at source so the
agent buys up a factory convert its
manufacturing process to stamp making
and goes on with achieving its goal but
of course the limiting factor here is
paper more paper more stamps so it
starts commanding forests to be felled
the wood to be processed all to feed its
single-minded ambition more stamps now
there's no denying that the AI is doing
what it was told but it's doing so at
any cost and any agent without some kind
of common sense will be at risk of
taking our instructions a bit too
literally this might be a bit of an
extreme example but Victoria crackovia a
research scientist at deep mind working
on AI safety is already seeing agents
that aren't exactly behaving in the way
their designers intended a reinforcement
learning agent that was playing a boat
racing game and the intended behavior
there was to go around the racetrack and
finish the race as soon as possible and
the agent was encouraged to do this by
having these little green squares along
the track that would give its rewards
and then with the agent figured out is
that instead of actually playing the
game it could be going around in circles
and hitting the same green squares over
and over again to rack up more points
then you have this this whole situation
with the boat going in circles and
crashing into everything and catching
and fire and still getting more points
than would otherwise get these kind of
situations are quite common
because you haven't ever stopped the AI
from doing that or told the AI that you
don't want it to do that that's a
perfectly sensible solution for it to
come across yeah from the perspective of
the AI it can't really tell that this
solution is achieved it's just something
that gives it a lot of reward so it can
necessarily distinguish between the
general solution and a really creative
solution that humans just haven't
foreseen there are plenty of examples
like this one team of researchers
created an agent inside a very simple
two-dimensional computer game and tossed
it with building itself a body to get
itself from the start line to the finish
line quite quickly it worked out that it
could just build itself taller and
taller and taller until it was as high
as the track was long and then just
flopped forwards to cross the line and
there was the agent playing the game of
Tetris which realized it could just
pause the game forever and never lose
but there is a balancing act here we
don't want our AI misbehaving but we
also don't want to restrict our agents
too much and this is part of what's
tricky about achieving safe behavior is
that we don't want to hamper the
system's ability to come up with really
interesting and innovative solutions
that we have not foreseen so we don't
just want human imitation we want
superhuman capabilities but without
unsafe behavior there is a very fine
line between a naughty algorithm and one
that's finding innovative solutions to
problems that humans haven't been able
to solve the AI doesn't know the
difference between the two it doesn't
understand what's really important to us
it doesn't have any common sense and
that means you have to be very careful
when you're setting up incentives and
rewards for your agents part of the
reason that this is so difficult is this
general perfect that's called good
hearts law and economics or good hearts
law says is that when a metric becomes a
target it ceases to be a good metric
a classic example of God arts law comes
from British India when authorities
offered cash rewards for dead Cobras as
a way to decrease the population of
snakes unbeknownst to the British the
local started to breed Cobras in order
to take advantage of the reward as soon
as the authorities found this out they
scrapped the scheme altogether and
revoked the rewards but now there were
these snake farms everywhere filled with
worthless Cobras so what did the locals
do release the Cobras into the country
resulting in an increase in the cobra
population this is what the scientists
call a specification problem when your
specified objective fails to bring about
the intended behavior this is generally
quite likely to happen because you my
preferences tend to be quite complex and
whenever we try to distill them into
some kind of specification or something
that we say we want it's going to be a
lot simpler than our real preferences
are and wouldn't necessarily capture
everything that's important to us let's
imagine we're living in the future where
robot Butler's are commonplace you
clearly specify the objective for your
robot it should serve you at all times
now how is that agent going to feel
about its own off switch your old always
has an incentives to preserve its own
function if it gets turned off they can
vacuum the floors anymore can't bring
you coffee it want to disable its off
switch for example Yan Leica is a senior
research scientist at deep mind also
working on AI safety if you turn off
then contact you the forest so if it
understands how the opposite of switch
mechanism works it would want to disable
it
what we want is you want our systems to
actually do something that is good for
us ready to do something that we
actually wanted not just like me
we want it but how do you get around
these kinds of problems well we already
know that writing a long list of do's
and don'ts won't work however long your
list gets you're always going to forget
something the new breed of learning
agents is going to need a different
approach
so one direction that we are pursuing is
learning your what functions for
reinforcement learning agents and you
can thing kind of think of this as
learning what your system should be
doing from human feedback so for example
in in one work that we did together with
open the eye is where we trained a
simulated robot to do a back flip and
the way this works is like the the robot
does some movement and then you watch a
video of their movement and you say two
videos and then you can compare which of
those looks more like a back flip yawns
experiment has a human sitting and
watching a screen looking at an AI
attempting to do a back flip each time
the human will feedback on whether the
attempt was good enough slowly nudging
the agent in the right direction
here is the key idea with constant human
feedback the human can communicate their
preferences without having to actually
specify them and risk oversimplifying
things in the process and after you like
a few hundred rounds of feedback the
robot can actually perform the black
flip it's kind of learn what the
objective should be that the objective
should be your back flip and what a back
flip is this is difficult to specify
precisely what a back flip would be I
mean so in my case like I can't do a
back fin right and but I can see if the
systems do back flips and in some ways
like this allows us to get superhuman
capability the AI then is essentially
being rewarded by pleasing the human in
a way exactly okay so we can we can do a
little experiment I'll try to teach you
to make a sound okay by giving feedback
so you'll make two sounds mm-hmm and
that
you wish and Intel sounds is closer to
you what I have in mind okay yeah does
that sound good so I'm being a are you
yeah you're being the I and I'm being
the human teacher and you're training me
essentially with reinforcement learning
and my reward function is getting you to
be happy you know what that's like
so the reward function here is like
something is in my mind right and I'm
trying to teach you so you can't
directly see the reward you can only see
like my feedback but my objective is to
get you to say you like the sound I'm
making exactly okay listen think 80
cents let's go for meat and this went on
for a while beep beep and but with Ian's
feedback we got beep and beep I go with
the first one absolutely nowhere
the expiration problem actually if I was
actually I have and I've gone through
ten thousand iterations by now well you
have a human the loop that you've use
all of that so it is it is kind of slow
but this is actually a problem that we
have in our assistance is that like in
order to give useful feedback you have
to have useful examples right in this
case you produced sounds that are very
similar and like the son I had in mind
was like very different so I really like
have the opportunity to give you a very
useful feedback but this is like
basically this is the same thing would
come up with a bad flip right like if
your robot just like lies on the floor
like two different ways like what are
you gonna do but you would hope though
after I'd had maybe a hundred goes at it
I'd end up with something that began to
approach what you had in mind yeah
definitely there finally is it like oh
no I'm not asking any questions I'm like
I'm in AI
I mean ideally this is at some point
this is what we'd want our systems to be
able to do I know just like describe
this on to you in natural language and
then you could just do it that would be
really cool right yeah so this is the
kind of research that we want to do
in the future that's like the kind of
system so we want to figure out how to
build okay actually everything I've done
so far has been like voices you didn't
say that it had to be a vocal thing did
you yeah okay hang on all right how
about this how about a service and the
first one the only problem was setting
up reinforcement learning with a human
standing over it offering feedback at
every stage is that it is monumentally
slow now it would take an agent months
if not years to master a game like Atari
and even then you need to hire a pretty
large group of poor students to do the
boring job of supplying feedback in real
time but there is an alternative you can
rustle up a slightly more sophisticated
learning partnership so what we do when
we actually build these systems is we
don't literally do the experiment that
we did now and instead we have we train
an eel network a second meal network
that learns how I would give feedback as
the human and then the the neural
network can teach you because I can just
oversee all of the things you're doing
by now neural networks have become
really good at spotting patterns dog not
dog backflip not backflip perhaps you
don't always need a human in the loop
laborious ly giving feedback why not
have two agents one attempting the task
and the other deciding if it succeeded
the reason why this works very well is
because evaluating the objective is an
easier task than producing the behavior
that achieves it so you can have like D
for example and the backflipping robot
all the neural network has to do is like
look at what the robot is doing and see
whether it's a back flip
you're listening to deepmind the podcast
a window on AI research but of course
where this stuff really comes alive is
where you take the ideas take them out
of a computer simulation and allow your
imagination to roam into the world of
embodied AI robots that learn how to
cook robots that learn how to pack fruit
in crates tuck you into bed and perform
backflips it's time to visit the deep
mind robotics laboratory this is your
lab this is where you spend your days
this is jackie kay a software engineer
packed to the rafters with robots
it's very cool we're basically running
out of the best it's quite noisy in here
yeah we got a lot going on today
this place isn't quite the high-security
basement laboratory you'd imagine there
are half assembled robot arms and
machine parts scattered all around the
place some look like mechanical hands
others quite a lot like great kitchenaid
style food mixers and curiously there is
a SpongeBob SquarePants mascot hanging
from the ceiling much of the work in
this room focuses on getting robotic
arms to learn how to do simple tasks
each robot is bolted to the floor in its
own cordoned off area and as we come in
the door there's one that's caught my
eye
now that game that you play when you're
a kid
you have tennis bat and a ball attached
to it and you kind of play tennis when
you're wearing this sort of looks like a
robotic version of that it's called the
ball in a cup oh she is it is exactly
that it is trying to swing the ball into
that little basket so you can see it
swinging the ball around the little it's
actually tracking position of the yellow
ball and it's trying to minimize the
distance from that ball to the area
inside the basket
not exactly a smooth delivery of the
ball into the cup more a slightly clumsy
lucky flip of course if your ultimate
goal was to make a perfect cup and ball
playing robot you could directly program
a machine to do that without fail you
can build robots that perform simple
tasks in much more elegant ways but
that's not really the point here's why I
had so senior research scientist a deep
mind a robot is a machine that can take
over a task one can say in the broadest
sense that your dishwasher at home and
your vacuum cleaner both robots because
they do something complex they run on
their own they have some autonomy to
them of course we also like to think
about robots as having some intelligence
and then you get more into the realm of
a robot with AI and I'm particularly
interested in saying how can we take
this a technology and make it work on
robots so that we have embodied AI and
that is an important distinction the
focus of the research in this robotics
lab isn't to create robots that are told
what to do but to have them learn their
own skills much in the same way that
other agents do here
the robots here are what's known as
embodied AI so let's say yeah the task
is
a robot that's trying to lift a box is
the programmer in a kind of traditional
setting would say go to box open hand
move hand you know 30 centimeters
towards the box close hand but this is
different yeah this is sort of taking
what we want and then figuring out how
to accomplish what we want every few
seconds this particular robot will have
an attempt at getting the ball into the
cup before pausing resetting and having
another go and every now and then by
chance the ball lands in the cup and the
robot is rewarded with a positive score
just like if it was playing a computer
game the reset in between their training
episodes where it untangle itself or
flips the ball out of the cup those are
scripted but then when it actually tries
to accomplish the task that is a policy
which it has taught itself through
experience over time from everything it
learns and the schools it receives the
robot builds a picture of how the ball
moves and how this relates to the robots
own movement so it had we come in right
at the very beginning then what would we
see just completely random noise
probably very chaotic
oh did he want ours it didn't wanna that
was Wow
that wasn't like three seconds so will
it know now that that movement gave it a
successful results yes and so the next
time that we see this do we expect it to
be better than when we walked in yeah it
will try similar actions that gave it a
positive reward or a success it's just
so assessing that quite smartly right
now yes there is a big advantage to
getting the AI to figure out tasks for
itself you'll end up with a robot that
is much more flexible out of the box it
doesn't matter what you want it to do
tire not stacks and bricks peel a banana
just as long as you can clearly
communicate your objective you don't
need to specify a long list of
instructions for these robots and
perhaps they'll come up with a new way
of banana peeling that you haven't
thought of but there's also a big
drawback in training AI that has a
physical body they are much slower to
learn than all of the disembodied agents
you'll find in this building the ones
that only exist inside a computer
other researchers are able to take
advantage of parallelism that is they
run their environments in simulation on
computers they can run them in parallel
on hundreds of compute different
computers we're all gathering data about
this environment they're trying to learn
something about we only have what
four robots here and there frequently
not running the same experiment so it
might just be one robot collecting data
and that means our training will be
orders of magnitude slower for
comparable tasks compared to ones that
are running in simulation on computers
progress is slower in this room and you
can tell the agents here are a little
less accomplished in another corner
another robot is trying to pick up a
Lego brick with a hooked gripping device
kind of like a claw and there's a rather
ominous box of mangled Lego bricks next
to it and I think this one has some
shaping to close its gripper when it
detects it's close to the brick it also
has a fixed training time so after some
number of seconds it will just get done
go back to the start and then try it
Lego is sort of this building block to
general-purpose manipulation if you know
if we can stack like two bricks together
we can then do kind of I'm sure you're
gonna break it
oh it's fine sorry I got distracted by
smashing up the legs there is real
potential here and so Jackie and the
team are constantly trying to find ways
of speeding up that learning process one
technique that we are looking into in
order to and that some researchers here
have done really cool work on in the
past is something we call center wheel
or simulation to reality transfer which
shows are you where you take a
simulation on a computer that models
your robot and we can learn kind of in
broad strokes what the robot is like how
its actions affect its environment
and how it can do something similar to a
task that's trying to learn in real life
so once we figure out all of that in
simulation without even touching the
real robot we can transfer the data it's
collected onto real hardware so you can
cheat basically you can cheat by
imagining the real robot within a
computer calculating all the physics
that would happen in real life and then
use the same techniques you use that
army of computers to give you a bit of a
head start before you even apply it to
the real physical robot exactly
so the real robots end up acting the
same way as your simulated robots do
well they start out acting the same way
as their simulated robots do and then as
they train more they might start
behaving slightly differently or better
when they go into reality using
simulation might give you a head start
on reality but it's never going to match
precisely the real robot has to contend
with grip friction gravity wear and tear
all of which play important roles in the
real world but none of which will be
perfectly represented inside the
computer and all of that means well I
think science fiction may have set some
false expectations one thing that's I am
a little bit surprised but being here
don't take this wrong way but these
robots are a bit rubbish yes it's true I
mean we've got a lot of work to do but
as with so much of what happens here at
deep mind it's not so much about these
exact datums
it's not about cup and ball or Lego
stacking it's about the type of
intelligence being acquired and how that
fits into the bigger picture we want to
demonstrate general purpose as we go
along targets
physical intentions great so contrasting
that with kind of intelligence that's
not embodied which maybe can learn
leggins or maybe even understand
language physical intelligence is
looking at how physical actions of your
body affect the real world so we want to
take a wide variety of tasks playing
with objects using tools maybe walking
around
in the future and we want to show that
robots can teach themselves those how to
do those tasks in the physical world in
the physical world
yes but physical intelligence is of
course only one type of intelligence one
string to the robots bow and deep mind
as we've seen dares to dream big here's
Marie Shanahan with the Big Finish the
holy grail of AI research is to build
artificial general intelligence so to
build AI that is as good at doing an
enormous variety of tasks as we humans
are so so we are not specialists in that
kind of way we you know a young adult
human can learn to do a huge number of
things you can learn to make food you
can learn to make a company you can
learn to build things to fix things you
can do so many things to have
conversations to rear children so all
those things and we really want to be
able to build AI that has the same level
of generality as that if you want to
know more about robotics and technical
AI safety then head over to the show
notes where you can also explore the
world of AI research beyond deep mind
and we'd welcome your feedback or your
questions on any aspects of artificial
intelligence that we're covering in this
series so if you want to join in the
discussion or point us to stories or
resources that you think other listeners
would find helpful then please let us
know you can message us on Twitter or
you can email us podcast at deepmind
comm


AI Safety Reading Group (Session 41)

so hello and welcome to the followed for
the first session in the reading group
where today we're going to talk about an
article called using machine learning to
address a I risk which is written by
Jessica Taylor from Miriam so this is
jessica taylor work for mirror in San
Francisco as a research fellow and this
blog post that we're talking about is a
written form of a talk she's given at
the effective altruists global the
conference and she is working at a
subgroup in miri which has the agenda
called alignment for advanced machine
learning systems and this this article
is a survey of the kind of technical
problems that falls under this agenda so
we've got the the goal of this research
agenda in very general terms is to make
AI systems safe even at very high
capability levels under a number of
assumptions and these assumptions are
key to understanding this agenda the
first is that the current a AI research
is generally on a track that will reach
super intelligence so is the deep
learning we're doing now will eventually
result in a super intelligence and it
might do relatively soon meaning that we
are talking that maybe a decade or two
or something like this the third is that
it's possible to build a task AI and
that's a good idea and something we can
learn from and she talks a bit about
these assumptions in particular that
these assumptions might not be true it
might not be using might not even be our
best guess but they are useful because
if it turns out that Adi is developed
relatively soon
then then this research will most likely
be very valuable and if it's possible to
say something about future AI systems
under the assumption that they look like
ours now then that's an avenue of
research like that will be very valuable
if it turns out to be true even if it's
not the most likely i'm hearing myself
in the no problem and i made a point
here that this is called the street
light effect so if you might have seen
this cartoon before or heard of our
something similar and about a man
dropping his wallet in in the pack but
searching for it in under the light
because that's the only place he can
find it and jessica taylor of course is
aware of this and she argues that there
is a chance that the world is where the
light is but this is something actually
something i will get back to when i give
my thoughts after this video so the
first is what is a task i rated AI it's
an agent that has a semi concrete goal
could be curing cancer could be earning
money or doing charity effective
altruism this kind of thing but it's not
this huge coherent extrapolated volition
where we try to figure out what you
would really really want on reflection
and doing that and of course that's a
particular task she doesn't say
explicitly but one particular task that
we really really would like this taski I
to do is to solve the control problem
and that means if we start to build an
AI if it says how can we control you vai
and then after that once we know how to
control it then hopefully we can
bootstrap that into making a GIS with
more
far reaching goals and this task a I
should have humans in the loop so to say
both in figuring out what to do how to
do it and maybe even in doing the goal
itself and of course the hope of this
research is that this will not turn out
to be really really hard compared to
building just a general ABI in the last
session we talked about the model
raising to Priscus and from there from
that model that was affected mu with how
hard is it to do it safely compared to
how hard is it to build the
superintelligence unsafely and lets the
factum you that we hope is high in this
case now there are six problems with
this the falls under this research
agenda the first problem is that actions
are hard to evaluate and of course
actions done by children can be
evaluated reasonably easily problems
something done by your peer is it can be
very hard to evaluate and something done
by something that someone who's strictly
smarter than you can be really really
hard to evaluate they can manipulate you
coerce you cheat or do covert actions of
things like that which once they are
done by a super intelligence is really
really hard to to fight against and
prevent yeah ideally we want the AI to
output an action saying this is what
I'll do this task EDI and give a proof
or some kind of justification and if the
proof is too hard for us then help us
evaluate the proof and of course this is
kind of course lead to infinite regress
where we hope it bottoms out in
something we can really understand but
we hope that this is something in
computer science something called the NP
the problems in the complexity class NP
which is something that has been studied
very very much and in this case it's
something that Paul Chris channel has
about in the in an article called under
the heading informed oversight the
second problem is test examples can PM
biggest you imagine a human sees a lot
of cat and figures a generalization of
what is what a cat is and then when it
gets to a new situation humans
generalize in a certain way and it's by
no means given that an artificial
intelligence will do that we want the a
is that to say if it's if it is a big
and biggest we want to say I am
ambiguous and this is in practice a real
big problem for artificial intelligence
as it is right now because they are
known to be extremely overconfident in
practice so if an AI is wrong it's that
usually says that is very very certain
in it's wrong conclusion and this is
called the inductive and big oet
detection and is something that after
after presenting these six examples
Jessica Taylor goes a bit more into
details about this thought third problem
is that it's really difficult to imitate
human behavior of course many of you
have heard about the Turing test which
is the problem where a and AI is
supposed to pretend it's a human and as
a distinguisher a judge who needs to
decide what how humans behave and how a
is behave and this becomes very very
difficult when the agent the AI is
smarter than and distinguishing because
then the the the AI the super
intelligence will in crisis be able to
do everything that the distinguisher
cannot notice this is also something for
Cristiano has written about in egg-based
agent and mimicry meeting halfways the
fourth problem is that specifying goals
in the real world is really difficult
like you would imagine like make me a
sandwich is
is super super simple gold and and even
that turns out to be quite difficult
because with reinforcement learning
there is a huge incentive to control the
actual source of the of the reward and
that might be in the children for super
intelligence point in their capabilities
where the incentives change so they want
to make a treacherous turn and control
the reward directly instead of the proxy
that we are looking about looking for
and this proxy can be really dangerous
like if we don't specify make us a
sandwich but make sure there is that
this camera is seeing a sandwich at a
particular point then this opens up a
lot of dangerous opportunities for the
AI to heck the reward this problem is
called the generalizable environmental
goals problem here under this research
agenda the fifth problem Island the
negative side effects this guy is here
first time I see a picture of him is
called Steve Omohundro and he theorized
benin super intelligence will have a
number of basic AI drives it could be
things like the AI is trying to make a
sandwich and believes with 99 percent
chance it can make a sandwich but it
also have to factor in the probability
that a human will shut it down maybe
because the human doesn't want a
sandwich so the AI has an instrumental
drive to stop the human from shutting it
down and there are three headings under
which this researchers and tries to
avoid these negative instrumental drives
called quantifying impact mild
optimization and non-adversarial
adversarial a Iook without instrumental
pressures the sixth is that there might
be H cases this to satisfy the call
bostrom has written about an AI but its
toll to make humans smile and then it
figures out that the edge case of making
a tiny tiny Smalling small smiling face
it satisfies the skull and then
proceed to trial the universe with small
smiling faces that's an edge case and
this kind of problem if you go back to
the sandwich problem it might be
possible to make a really really small
sandwich which counts as a sandwich if
its measured by weight we might have a
huge sandwich it might be a toxic
sandwich these kinda things that
technically are satisfied the call but
actually don't do it at all in
particular there's a problem with
adversarial examples and I really really
love this example sorry I can hear
myself in one of you it is possible to
mute I think it might be you moan the
some if you could move your microphone
please um it might be Victor who should
move it Mutis microphone yeah I think
it's fine um ok so here we have an
example and I really really love this
example where you have a image
classifier which looks at the first
image as this I believe this is a
pendant I am 57.7 percent confident that
this is a panda and then you add some
completely random noise it looks really
like a random noise if you ask they
clarify what it is it might be an
amateur but it's totally unconfident so
this is just completely silly picture
and you add it only with zero point zero
seven percent and then you get an image
here which is almost exactly the same as
the spandan this panda is just to an
extremely small extent distort so human
eye these two look almost exactly the
same as 0.7 percentage of a change but
here this this example has been
constructed so the image classify and
outer leaves it's given a completely
different animal with ninety nine point
three percent confidence so this is a
really really dangerous case it's it
truly shows that
it might be possible many of the AIS
that were built right now are much more
vulnerable to being cheated in this case
by asmara adversary then then then you'd
think no human would ever be cheated by
this no human would see a huge
difference between this picture and this
picture but but a is as we build them
now they do think that there's a huge
difference between them and the yes sure
Oh
and the the way I would see this you
have a plus between two images and the
question is what does it mean to add two
images and what I imagine is that each
pixel in the left corner has a red green
and blue value and in the middle picture
each pixel has a red green blue value
and then you average these two where you
give this one the the pendant it the
first way more weight than the other and
that and then then you distort this
issue extremely slightly compared to
this one and then you get this this
image I believe that is how you add
images
no no
yes I think so we go back to the problem
that was called inductive and divinity
identification where the AI is uncertain
and it needs to be able to tell us that
it is uncertain and maybe in practice it
would be good if it checks with us so if
it's uncertain whether they're like a
tiny smiling face should count as a
human if it's uncertain about this it
should ask us and then we can say no
it's not you can see our down here yeah
that's a graph where there's a lot of
positive examples in the upper left
corner and a lot of negative examples in
the lower right corner and what's the
difference between the two there are a
number of hypotheses like everything to
the right of this line is negative and
everything to the left of this line is
positive but it could also be this line
we don't know which of these hypotheses
are the truth and and that's an
algorithm called know what it knows
learning quick lining and and this
outputs a this ambiguity if it's more
than and as a number of percentages
uncertain about about this and you can
you make some assumptions if you make a
number of assumptions that are
moderately reasonable then then this
actually works out quite well you need
to have a number of hypotheses with like
these lines and one of them need to be
true and it needs to be a model with
either a small finit number of
dimensions or just a finished set of
possible points and under these
circumstances this really works after I
taped to a picture here this is Socrates
who knew that he did not know anything
that was in the I guess his most famous
and so and this is ideally what we want
the AI to realize how much it doesn't
know and one of the ways that Jessica
Taylor and her voice on this is using a
patient view of the problem and in this
case then I patient patient statistics
and predictions and probabilities I have
taken here a picture this river emplace
who invented base formula and he have an
example of how how the patient update
process works that you believe at first
that the truth is all the way to the
left here that is that is your prior and
then you do some measurements and the
measurements are this blue here you can
see vaguely up here that turns out to be
what you measure and then after this you
your best guess at what the truth is is
this like black line this is your
posterior and in a similar way you
assume that there's some kind of true
prior exactly water through prior is
this a good question a we it might be
the prior that the super intelligence
can find and we have some kind of prior
in our in our candidate AI vai that we
are building and we assume here we make
an assumption of how good the AI we are
building is we assume that it always
finds a grain of truth for instance k is
true it means that the truth is that if
our prior has a certain probability then
the true prior has at least half as much
if K is 2 so this means that and the
week if we can make some kind of found
on how good our AI is compared to the
truth then it's possible to find some
some probes and some algorithms to to
get very very close to the truth
even if my tix somewhat longer and that
is what jessica is working on it should
be said that there are two other agendas
at Mira are kind of a mirror to other
day I safety agendas the first is headed
by need for us here which is the agents
foundation which is much more about
theory and the theory of reasoning and
this making decisions and proving that
that programs satisfy particular
properties these kind of things and on
the other end of the spectrum there are
the concrete problems in AI safety a
paper that has been written by dario mo
day and a number of others and where
which goes much much closer to recurrent
amount machine learning systems and find
some things that came with problems and
can be demonstrated in the current
systems and if a lot of progress made of
this hopeless that this will be relevant
to the big problems when you get an
actual super intelligence and this is
only i would say that jessica taylor is
in the middle of these two but that
these three agendas can be put on a
spectrum with nate suarez as the most
long-term and Terry i'm a day as the
most pred short-term and just tailor
somewhere in the middle but that's just
my understanding and she doesn't write
this anyway anyway thank you for
watching and i will stop the recording
now