Reinforcement Learning 10: Classic Games Case Study

okay good morning everyone so uh well
done for making it to the end of this
course so I've got a different face
today I'm David Silva and I'm going to
tell you a little bit about games in
particular classic games like the kind
of board games which you might play or
watch other people playing and it turns
out there's an awful lot of the rest of
the course that you can bring to bear
and these kinds of problems and so this
you can think of this lecture is like a
case study and how to use all of the
techniques that you've learned about so
far in trying to achieve or break the
state of the art in terms of building
algorithms which can actually play
classic games to human or superhuman
levels so brief outline the the idea is
I'll just start off by telling you a
little bit about state of the art we're
just going to touch on game theory and
really the goal here is to understand
you know what does it mean to do well in
a game it's not like a single agent
domain it's not like a Markov decision
process where there's just you know one
agent interacting with environment so
what does it really mean to to achieve
optimal behavior and that's going to
lead us to this principle of minimax
which is the best known principle of
optimality at least in two-player games
and so to understand the history of
games you really need to understand the
history of minimax because for decades
it's been like the way that people have
thought about these problems and so
we're really just touch on how that that
works and what it means to do minimax
search but then we want to move on to
you know really tying things together
with what you've learnt about elsewhere
in this course so talking about how do
we do reinforcement learning in this
setting and the main idea there is going
to be agents that actually learn whilst
they're playing as other agents that are
also learning and we call that idea self
play and it turns out to be a very
natural a very powerful way to learn how
to find these optimal solutions in these
games these things like minimax and once
we've done that we'll move on a little
bit and I'm going to try and do some
kind of case study in alpha zero I'm
just checking actually I think I might
have some old slides here one second
sorry
let's just check
we'll see how we go on I think it's
gonna be okay so why should we care
about games I mean I think maybe at
least some of you'll be sitting there
saying well you know I care about the
real well they care about applications I
care about really having impact in the
real world so why all this fuss about
games why we using games as the case
study and actually they make a really
useful case study even if you're not
someone who's into games because they're
very very simple rules which lead to
actually very deep concepts and as a
result they've been studied by humans
for hundreds of thousands of years and
in a sense they're the kind of
intelligence domains in which humans
really like to test their own
intelligence and there are people who've
devoted their entire lives to the study
of games like you know chess or go or
whatever it is and so it becomes like a
meaningful IQ test a way to compare the
intelligence levels of our machines
against intelligence levels of humans
and that makes it really useful domain
and as a result of all of these things
it's become what's known as the
Drosophila of artificial intelligence
the Drosophila was there the fruit fly
which was used in genetics research for
for many many years because it was such
a simple organism to study people really
just try to understand everything there
was to understand about the fruit fly
before they would move on and study
other organisms and game to have kind of
the same effect in AI that people study
games because they're the simplest kind
of testbed in which you can understand
all of the main ideas of AI and machine
learning and reinforcement learning
whatever the principles are you want to
understand you can understand them in
these games first and then take what
you've learned and push them out into
the real world and that's been shown
time and again to actually be the case
and so in some sense you can think of
these as like microcosms of the real
world like this this board game actually
is like you know the real world but
simplified down distilled down to like
this its essentials that there's some
simple rules but they lead to huge
complexity and we can study how to
actually achieve intelligence in that
domain this little universe and it
captures at least some of the elements
of complexity that we might care about
in the real world and of course at least
to some of us games are really fun and
so maybe that's the main reason to
actually care about them so I guess I've
given this this class for a few years
and each time I give it this slide
changes actually which is kind of the
state of the art in in AI like where
have we reached in classic games and so
if you think about some of the
best-known games like checkers chess
Othello backgammon Scrabble go poker
we've now reached
a level of play which is actually
perfect in checkers and poker and what
that means at least for heads-up poker
this is down here when you have two
players playing against each other and
what that means is that if you play
perfectly it means that you know this
checkers program Chinook if it actually
played against god it would always be
able to draw that's what it means so so
that's what perfect play means whereas
in these other games like chess or
phthalo backgammon Scrabble now go the
level of play that's been achieved by
and this is the first program to achieve
that level is now super human so we see
that across the board it is possible in
these domains to achieve the highest
levels of performance okay yeah so I
seem to have by slightly older slides
but anyway this is now changed as well
so this is what's been achieved using
reinforcement learning in games this is
the state of the art so far so in
checkers we've achieved a level of
performance which is superhuman
using a reinforcement learning algorithm
an algorithm which is learned by itself
to achieve that level by just kind of
playing against itself or player using
self play and so that's really you know
a feat which goes beyond this question
of what could be done with hand crafting
chess this is the old slide from last
time I taught this course but we just
actually in December showed that you
could reach superhuman level in chess
using principled reinforcement learning
methods and we'll come to that at the
end of today's lecture in Othello
backgammon Scrabble go poker all of
these are examples where principled
reinforcement learning algorithms can
now achieve the highest levels of
performance so you don't need to turn or
look beyond RL it's possible to do all
of these things with the kind of
machinery that you've learned about in
this course and nothing more really just
started from the beginning starting from
scratch having a system that learns for
itself by actually figuring out its own
strategies playing against itself and
naturally saying oh I figured out what
leads to more wins then then some other
strategy and playing that strategy more
and more and more this idea is
sufficient to reach the level highest
levels of intelligence okay
so as I said it's really important to
understand what it means to achieve an
optimal strategy in a game so it's not
an MDP we've you know so far in this
course you've seen many examples of
markov decision processes and you've
studied what does it mean to achieve up
to
behavior but now we've got multiple
players involved there's multiple
different agents all trying to achieve
their own goals in this domain and so
when you ask the question what's the
optimal policy in this domain well in a
way it depends it depends what the other
people are doing you know you've got
some imagine this is like a social
interaction and the way in which you can
maximize your own gain depends on what
the other people are doing in that
social interaction and so there is no
absolute answer to the best behavior in
that situation it all depends on what
everyone else is doing but there is one
well-known idea which is used in which
basically summarizes what people
typically mean when they talk about
optimality in a game and that's the idea
of a Nash equilibrium which was famously
invented by by John Nash a few decades
ago so the idea is to say that you know
if all of the other players fix their
policies
that's what there's pi- i means that
means everyone else's policy except for
the AI player so if everyone else was to
fix their policies then you could come
up with this thing called the best
response against that kind of set of
policies so if you kind of just fixed
everyone else's behavior if you know if
all you guys kind of fixed your
strategies and I now know that you're
all got that fixed strategy I can
compute my best strategy against all of
you and that's a well-defined thing now
because you're there's no longer this
question of what you're doing that's
just fixed you've become pinecone a part
of my environment you've become part of
the problem that I'm trying to learn
about and so that thing is called the
best response so it's the best strategy
for player I against everyone else's
strategies that's called the best
response and so this idea of Nash
equilibrium then opens it up and says
well what if I'm trying to find the best
response to you but each of you is also
trying to find your best response
against everyone else and the Nash
equilibrium is a joint policy for all
players such that everyone has found a
best response so what that means is that
we would kind of all try and find the
best strategy that we can against
everyone else and we would only be happy
we'd only say it's a Nash equilibrium if
we all reach a situation where we're
happy that we've found the best response
to what you're all doing so if I find a
situation where I wouldn't choose to
deviate from that I found the best thing
I can do against the rest of the
population and each of the
other members each of the other agents
in that population has found their best
response to what everyone else is doing
that is the Nash equilibrium so everyone
is doing the best thing that they can
against everyone else at the same time
and so it's not a Nash equilibrium if
there's any incentive for anyone to
deviate so if there's any individual
agent that would choose to move away
from that strategy because they think
they could do better it's like oh
actually I've just noticed that there's
some behavior that I can exploit that
but someone over there is doing if I
find such a way to exploit that and
think I can do a little bit better is
not a Nash equilibrium it's only a Nash
equilibrium if everyone is happy with
their strategy they think they're doing
the best thing they can against everyone
else's strategy so that basically is
saying in in this notation that the the
best response to everyone else's
strategy is actually the strategy that's
followed by that player yeah question
okay we're going to concerts a great
question is there always a Nash
equilibrium so we are going to consider
classes of games where there is always a
really well behaved Nash equilibrium and
so we're going to make life simpler for
ourselves by considering situations
where there really is a very
well-defined single Nash equilibrium and
and it's not always you know some games
there can actually be many many
different Nash equilibria which is quite
interesting for example the game of
poker there's many different Nash
equilibria and sometimes some of them
are like seem like they're better than
others like you can have one where
there's the famous problem called like
the prisoner's dilemma and things like
this there's situations where where all
the players can be happy with how
they're doing they wouldn't choose to
deviate but still there's kind of this
fallacy of the of the masses where
they're all doing very badly and that
would still be a Nash equilibrium
because no one would choose to deviate
from that situation and there might be
another situation where if everyone
everyone would also be really happy with
where they've got to and that might be a
different mass equilibrium where
everyone gets an objectively much much
better reward but they're both Nash
equilibria so you can get these quite
complicated games like poker that have
these properties and that typically
arises when you've got imperfect
information we've got you know partially
observed stuff going on where we're I
don't know all of the information that
you have and vice versa when we have
perfect information we're going to
consider classes of games where where
there's always a single Nash equilibrium
and it's we've got principled ways to
find it so it was a great question okay
so if we think about how does this fit
into our usual framework of like
reinforcement learning we can think of
this that this best response thing is
basically what happens when we try and
solve a single agent RL problem whether
other players have just become part of
the environment so imagine there's a
two-player game or something and now
there's there's black and there's white
and and white is saying okay well I'm
just gonna treat black as some fixed
strategy and find the best response to
that strategy bye bye black and so black
kinda becomes part of whites environment
black becomes kind of part of the
background in which in which white is
operating and trying to get reward and
if black is always playing in the same
way well it's just like some MVP that
white happens to be in in which black is
always taking these same things and that
leads to some situations that that white
now has to deal with so so the other
players become part of the environment
when you're trying to find the best
response so you can always think of this
multi agent thing as reducing to a
single agent problem whenever the other
players are fixed so if you fix the
other player strategy it's just an MDP
and the best response is actually the
optimal policy for this MDP but in
practice we don't want to fix the
strategies of the other other players we
want to we want to know what happens you
know what's the best strategy for me
when when everyone else is behaving
really well it's not interesting to ask
what's the best strategy for me when
everyone else is behaving in some kind
of dumb way what's the best strategy for
me when everyone else is behaving really
well that's again the nasty Nash
equilibrium so you can think of the Nash
equilibriums like they're kind of fixed
point of self play reinforcement
learning so how could we think of that
well the idea is that you have your
agents in your in your game and
experience is generated by playing games
between agents okay so whatever the
strategies are from for our agents
they've got these strategies PI 1 and PI
2 and so forth and we're going to use
those current strategies to generate
experience so you generate that
experience and from that experience each
agent is going to treat what it's seen
is going to try and learn about
those other strategies as if it was in
an MDP where those other strategies were
fixed so if you're PI one you're just
going to kind of assume that PI 2 and PI
3 and PI 4 were fixed and try and find
the best response to those but in the
meanwhile the player who's generating PI
2 is also going to treat PI once as if
that was fixed and try and learn how to
do better and try how to beat PI 1 so
kind of PI 1 is treating PI 2 is fixed
and trying to beat that current strategy
of PI twos in the meanwhile PI 2 is is
treating PI 1 is fixed and trying to
beat that strategy and so forth so
everyone is just generating experience
according to their current strategies so
far and trying to beat the strategies of
the other players and so this process
kind of goes on and on and on and so you
can think of this as one player's policy
determining the other players
environments and so all these players
are adaptive to each other and if any
point all of the players have no
incentive to adapt anymore if they're
all happy then we've reached an
equilibrium so in particular we've
reached the Nash equilibrium so the Nash
equilibrium is the fixed point of this
process so this is kind of independent
of what the learning algorithm is it
might be that any particular learning
algorithm might fail to find this bit
clicks point so it's a hard learning
problem to actually find this fixed
point but still you know it is the fixed
point of this self play reinforcement
learning if each player is trying to
optimize against the others and at the
point where they're all happy and
declare victory it's like you know if if
if player PI 1 it says ok I'm done I've
achieved everything I've learned about
against PI 2 and PI 2 at the same time
says I'm done I've learned everything I
can do against PI 1 that means they've
reached a Nash equilibrium by definition
they've got no incentive left to change
and they wouldn't choose to deviate
anymore and so they have reached this
jointly kind of optimal policy where the
Nash yeah question
yeah exactly the idea is that in
reinforcement learning we're trying to
optimize for some MDP we're trying to
find the optimum policy for some MDP and
so the idea is that in that MDP we are
like
what each agent is finding the best
response so so optimizing that MDP is
equivalent to finding the best response
and the Nash equilibrium is kind of
where everyone has found the best
response to everyone else so that's the
idea that each the RL is the way that we
kind of reduce we reduce each player's
perspective to an optimization of an MDP
to give us this best response and by
jointly doing that we find the national
yeah question so the question is can it
learn - so - so it's a fixed point but
does that necessarily mean that you
actually start to improve towards the
fixed point given given a learning
algorithm oh I see so so so RL is a more
general framework for example if it was
the case that you just chose some
strategy for pi/2 and you wants to know
how could this do how could PI 1 exploit
some suboptimal PI - you can ask your RL
algorithm to find the best response 2 to
PI 2 and it will go ahead and do that
and that will be better from PI ones
perspective than playing Nash so it can
do that but if you allow both players to
evolve against each other the idea is
that it will eventually find a Nash
equilibrium and so you can choose to do
other things than find Nash the purpose
of this slide is to show that there is
one natural choice which is to try and
find the Nash equilibrium and so this is
like a mainstay like if you go and study
math or something or or even watch
movies about John Nash who's seems to
play regularly in Hollywood then you
know the National equilibrium is is the
big conceptual idea that helps people
understand game theory and societies and
how people interact with each other and
and so forth in that kind of economics
this is a very important concept okay so
as I mentioned we're not going to
consider this very general case of all
games we're going to focus on a special
class of games so the special class
we're going to consider we're going to
start by thinking about perfect
information games so these are games
that are fully observed by all players
so for example something like chess or
go have perfect information what that
means is that all players can observe
all of the information that's available
to all the other players there's nothing
private that's available just to one
player whereas if you think about a game
like poker I have my my private cards
that the other players don't see and
that private information is what's
called imperfect information there's
some kind of hidden state that's not
visible to the other players and that
gives rise to these very complicated
dynamics where where I now have to
reason about you know what what my cards
might you have and in that situation
what what should I do and if there's
time I might touch on this at the end
how to deal with imperfect information
games and I just want to kind of give
you you know the teaser here which is
all the same methods that we talked
about extend very naturally to the
imperfect information case so it's not
like they just fall over just as we saw
in NRL you know reinforcement learning
by by building a state representation
that takes account of the history of
observations you can deal with partial
observability and the same kind of thing
helps us to deal with imperfect
information games we just need to kind
of do things in the right way we're also
going to focus on two-player games
mostly where we have two alternating
players so we can think of them as white
and black without loss of generality and
they take turns to play and we're going
to focus on what are called zero-sum
games I'm so in a two-player game a
zero-sum game is basically a game that
has equal and opposite rewards for black
and white up so it's called zero-sum
because the reward for player one at any
step plus the reward for player two has
to equal zero in other words you know
what's good for black is bad for white
so if I win you lose that's basically
what a zero-sum game is and vice-versa
you can have more general zero-sum games
where you can have you know many players
and as long as their rewards for all of
those players sum to zero it's still
zero-sum in other words if one player
wins all the other players lose is that
the most common way to that they only
you have to sum to the utilities have to
sum to zero and so the question we asked
now is in these special cases how could
we find Nash equilibrium in perfect
information two player zero-sum games
and the two ways which we're going to
talk about our game tree search like a
kind of planning and this is that
historically the method which has been
most widely used for decades and self
layer reinforcement learning which is
the kind of firm the version which we've
been learning about in this course but
extended to this self play context and
combinations of the two are also
possible okay so let's see if we can
bring back the machinery from the from
the rest of RL and bring it to bear in
this setting now what can we do to
actually think about things like value
functions and policies when we move to
this this multiplayer setting and so the
first thing to say is we can come up
with the same concept of a value
function this is really really crucial
to us and the value function as but
always is going to be the expected total
reward that we get like if you play a
game now how much reward is a particular
player going to get and note that we
don't need to think about separate
players separate rewards for the two
different players because r1 is just
equal to minus r2 so we can just always
think of the reward from one player's
perspective and that's going to be fine
now we don't need to the other players
perspective is always going to be the
negative of that you can generalize this
to the the other case but just to keep
things simple there's one value function
and and the value function for black is
just minus the value function for white
okay and so we want to say well how can
we maximize the total amount of reward
we get and we're just gonna think about
having a policy pair now so our policy
is going to be player 1 follows PI 1 and
player 2 follows PI 2 and that kind of
determines completely how they're going
to play this game and there's also some
game rules that you can think of like
the environment that determine you know
how the state's going to evolve given
actions played by player 1 actually
played by player 2 and and the rules of
the game that entirely determines how
the game will will proceed and then you
can ask the question well under that
policy pair who's going to win basically
you know what's the expected total or
going to be which is typically just you
know rewards in these games are very
often zero all the way through and then
at the end of the game you know plus one
for winning or a minus one for losing is
the canonical case we'll think about
so we can say well there's this value
function which is just the expected
reward the expected return so this is
you know plus one for winning minus one
for losing and it's the expected return
under if there's any stochasticity like
dice in the rules like if you're playing
backgammon you want to know what happens
under the dice the expectation would be
over those dice and if there's any
randomness in your policy the
expectation would be over the randomness
in both players policies and we can ask
well you know under if we averaged over
all of those sources of randomness who's
going to win and that tells you well how
good is this policy pan like who's who
does this favor like if the answer is
that that you get more Plus Ones than
the minus ones in other words you know
if it's if it's really favoring white
over black then that tells you something
about this value function that this
policy pair is a good policy pair for
white and a bad policy pair for black
okay so the special thing about these
two-player mini-games is there's
actually something which we call the the
minimax value function and what that
does is it maximizes White's expected
return while minimizing blacks expected
return basically the way to think of
this is that the minimax value function
there is a unique value function and the
special thing about the minimax value
function is it uniquely determines who
like think of it this way that if black
played the best possible strategy for
black and white played the best possible
strategy for white what would the final
outcome of the game be and that's what
the minimax value function tells us and
the amazing thing about it the amazing
thing about the minimax value function
is that it's unique that and this is not
true for example if you're playing a
game like poker with imperfect
information or you know nonzero-sum
games and so forth this gets more
complicated but for this class of games
there is a unique value function there's
a well-defined question which is just
how good you know who's going to win
from this position under best play by
both sides and then the minimax policy
is really just any policy pair like a
way to play the black and a way to play
for white that achieves those minimax
values and there might be more than one
so this might not be unique just like in
an MDP you can
there's one unique value function there
might be many policies that achieve it
and the same is true in games there's
one unique minimax value function but
there might need to be many ways to
achieve it the simplest example being
you know like imagine you're playing
tic-tac-toe and the first player you
know plays in the middle and then goes
on to you know the other player will
block somehow and ends up being a draw
and there are many ways that you can
reach a draw there's not just one
strategy that MIT reaches a draw there
are many but the minimax value function
is a draw for that game and there was a
question I think sorry yes No so so the
the minimax value function is it's sort
of defining a policy for black and white
in some sense it's saying under best
play for black and white this is what
could be achieved so it's not kind of
given a particular strategy for the
other player it's saying under all
possible strategies for both players
what's the the best outcome it is a Nash
equilibrium yes
so the minimax policy is a Nash
equilibrium yeah it's exactly a Nash
equilibrium and that's actually what I
wants to change this to is to say the
correct definition here is that the
value function for black is basically
the thing which maximizes the value over
all policies that black can do against
White's minimax value function and vice
versa anyway I'll correct that in the
slides when I post them so just to make
that clearer we can think about actually
doing minimax search and again this is
like if you pick up any AI book from the
last 50 years until people started like
rediscovering machine learning and deep
learning and all these new ideas this
was like chapter one like this was the
thing that you would learn about and
would be like definitional about what
AIS would be oh you really need to learn
minimax search because that helps you
understand how how chess programs work
and so forth and minimax search is
really it's a it's a a mechanistic view
of
the minimax value function and it's
basically saying well one way to view
the minimax value function is to say
that you can think of it as like we can
draw out the entire game tree as saying
well from this situation here in
tic-tac-toe there are all of these
different situations where where Cross's
can play and from each of these
situation there's a bunch of places
where where the O's can play and it's
not showing everything here where the
noughts can play and and if you to to
actually write out that entire tree then
you can look at the bottom and you can
see what the values are at the bottom
and you can kind of propagate all of
those values back up to the top to
compute that if one player's maximizing
and the other players minimizing and the
other players maximizing there is some
optimal way at some optimal strategy for
computing the value function well we'll
run through an example in just a second
but actually this was first introduced
by Claude Shannon I mean actually if you
look at the history of computer science
like all the great computer scientists
like Turing Shannon Babbage von Neumann
they were really really interested in in
in games and game theory and they were
fascinated by how to make machines play
chess in particular but all kinds of
games so Claude Shannon actually he came
up with this minimax search and this was
actually ran on paper this algorithm at
that time but went on to become the
mainstay of sort of every single chess
playing program in the world for like 50
years pretty much until December so so
this is an example of mini match minimax
so this is like a very simple game tree
now just some abstract game with just
two choices for each player and you can
see there's two choices for for the max
player and from each of those states
that you reach there's two choices for
the min player and from each of those
states there's two choices for the for
the max player there and at the end of
it there's some outcome there's some
reward that you get at the end where
this is the the reward from from one
player's perspective from the max
players perspective a plus seven plus
three and so forth and so from this you
can kind of compute by a kind of
depth-first search what all the values
are so we're not actually going to do
the whole depth-first search but we can
kind of run through this idea to say
well you know from this from these leaf
values we can compute
you know that their max of plus 7 plus 3
is plus 7 so max would always choose
this this action here and never choose
this action here because this is the
thing which gives more value from his
perspective and so max would choose
minus 2 out of minus 2 and 4 and plus 9
out of plus 9 and plus 5 and so forth
and then we can propagate those out
those from min players perspective as
well to say that from min players
perspective we choose minus 2 rather
than plus 7 because from the min players
perspective you know plus 7 is a
terrible result and minus 2 is a very
good result and minus 4 is a better
result than plus 9 from the min players
perspective and then you can propagate
this again to the root and this gives us
a max of minus 2 and so this really is
the optimal value so hopefully this
helps to understand what what the
minimax value actually means that it
means that you know Max's is choosing
its value assuming that min is
minimizing and while min is assuming
that max is maximizing and now if you
propagate that all the way back up to
the top you get this unique optimal
minimax value yeah question I'll send it
on a slide it's easier I think it's the
easiest way to understand it is that
there is like this that it's at least in
its terminus tick game that you're
you're maximizing over the minimum of
the max of the min of the of this action
series so so kind of the first player
gets to choose a the first step that the
next player gets to choose the next step
the next chair gets to choose the the
next step and so forth and so you've
kind of got the max of a min of a max of
a min of that sequence of actions in the
stochastic case you just kind of need to
amend that a little bit
this gives us like a mechanistic
principle then for how to compute this
minimax value but it's clear that the
search tree grows exponentially and in
any game of reasonable size it's
impractical to search to the end of the
game so what do you do you know it might
be about tractable in tic-tac-toe knots
across it but it's not tractable if you
go to any game of meaningful size and so
in practice what people use is something
which we would call a value function
approximator
but which in all of those AI textbooks
have you ever picked them out would be
called an evaluation function or a
heuristic function and so the value
function approximate hrus is trying to
approximate this this minimax value so
we can say from each of these states
instead of actually computing this thing
exactly we can have some function
approximator which estimates each of
these values in each of these nodes here
based just on say the board position so
if you're playing chess you could kind
of look at the position in this
situation and from that situation
without even looking at these other
possibilities down here try and estimate
what the value is so that's a value
function approximator so it's a function
that takes these parameters W and feeds
into this position s and that would be
the the value function approximator
which people would use instead of in
places like a proxy for doing sub trees
of this minimax search and so what you
can do is if you had a very large search
tree imagine that you are able to search
down to this depth but you couldn't
search all the way down to the end of
the game what would you do well you
would search down to here and instead of
going to the end of the game to figure
out the true value here you would
replace your value here with your value
function your value function would give
you some value here and then you
propagate that value back up using your
minimax search procedure instead of the
real value so what this tells us is that
we basically there's typically we use
the value function to estimate the
minimax value at leaf nodes and then the
minimax search typically would just run
to say to some fixed depth R with
respect to those leaf values
exactly this is very very common idea so
let's make this concrete and also help
to give some historical context by
thinking about you know the simplest and
also most common way in which these
these evaluation functions were built up
in the past and that is using what I
call a binary linear linear value
function so you have some binary feature
vector which for example would look
something like this so some binary
feature vector telling you which
features are present in this position
so in particular we could think of this
feature here as saying does white have a
rook in this position and this is a 1
because in this particular position
white does have a rook and then there's
a white bishop kind of indicator and
again why it has a bishop in this
situation but why it doesn't have a pawn
so there's a zero black does have a rook
so there's a 1 black does have a bishop
sorry black doesn't have a bishop or a
pawn and so that kind of summarizes
these are just features summarizing
what's going on in this position and you
could imagine there could be much more
complicated features but here we've
these are the best known features from
like if you learn how to play chess you
basically learn to assign material
values to each of these and so you the
first thing you do is say well what's
present in the position and then as a
human you know when you're learning how
to play chess you're told the values of
each of those things so you're taught
that you know a rook a white ruckus is
worth +5 a white bishop is worth plus 3
white pawn is worth plus 1 and then we
flip the perspective when we think about
the opponent and we subtract or 5 if the
opponent has a rook subjective 3 if the
opponent has a bishop and subtract of 1
if the opponent has a porn etc and so to
evaluate this position as a human the
way you're kind of taught when you learn
how to play the game of chess is we take
the inner product of these two things so
each of these features times its value
and you sum it together and so what this
is really telling us is that for each
each feature that's actually present in
this position we accumulate it's it's
its way so we're summing together the +5
having the rook and the +3 for the
bishop take away the minus 5 for this
rook and that tells us that white in
this position is kind of winning by plus
3 if you like
so this is the classic binary linear
value function and at the end of it
you'd probably squash it down into
something that actually estimates who's
going to win in that position like some
kind of sigmoid activation function to
squash this into a plus 1 for for
winning and minus 1 for losing but that
is kind of the classic way by which all
games were were evaluated and the only
thing that's changed from this is that
the set of features became much much
richer and grandmasters went to great
lengths in in all of these games to try
and handcraft very very specific values
for these weights and they would play
around with it say no no no I think
actually the rook isn't worth +5 it's
actually worth plus 4.8 and then they
would slowly chain tune their program to
get better and better yeah so in this
the King would have infinite weight
because you know in chess if you don't
have a king you've lost the game so I
guess you could think of that as
infinite but in practice it would never
even occur because you inside your
search tree would never see positions
without a king so yeah so it kind of
just factors out so the most famous game
playing program of the last era was deep
blue and this was the first program to
defeat a the world champion at the game
of chess and I think it's useful just to
understand what went into it because it
was exactly what we talked about so far
that that really it had about 8,000 of
these handcrafted chess features so
instead of just having the you know the
piece values this this vector was about
8,000 long with all kinds of handcrafted
features talk about things like King
safety and pawn structure and all this
kind of other stuff going on and it had
a binary linear value function and the
weights were largely hand tuned by human
experts although it was a little bit of
machine learning they experimented with
and then there was this really high
performance alpha beta search
what's alpha beta alpha beta is
something again which would be in your
chapter one of these textbooks but it's
basically a way to efficiently compute
this minimax search tree by
automatically pruning parts of the
tree that you don't need to compute that
you can kind of you can already know
that certain parts are never going to be
taken by the max player because you know
that min would never even let you go
down that part of the tree in which
pates point certain parts of the tree
can be excluded all together from
computation and it's it's a dramatic
improvement over the naive approach so
so deep blue did this it was this
high-performance parallel alpha beta
search and it was special specially made
Hardware search around 200 million
positions a second it looked ahead about
between 16 and 40 steps into the into
the future in this search tree and at
the end of each of those positions it
used this nice binary linear value
function to kind of estimate who was
going to win propagate that all up and
use that that search tree to decide what
to do at the end of that it defeated the
human champion in 1997 by about four
games to two and at the time it was
actually the most watched event in
internet history so this was considered
a real watershed moment for by many
people okay I think it's also worth
mentioning that deep blue was not the
only example of this strategy that this
was applied separately in many many
different games using immense amounts of
knowledge and expertise and a lot of
clever people working on all kinds of
different things
and I just want to give one more example
of this kind of strategy which was the
program Chinook which was notable
because for a couple of reasons I mean
first of all it was playing the game of
checkers and checkers is again much like
like chess and the representation was
was similar as well as again using this
binary linear value function with all of
these kind of knowledge-based features
regarding kind of position and mobility
and and four different phases of the
game like what you do in the opening and
again these were kind of a carefully
handcrafted and it again did a very high
performance alpha beta search but also
did an additional piece called
retrograde analysis which was kind of to
say at the end of the game in certain
games like chess and checkers you can
work your way backwards from all one
positions to solve all possible end
games and so in Chinook they had like
all of the if you reach any position
with like 10 checkers or fewer it has
perfect knowledge to tell you from that
position you know who's going to win it
can just figure that out by searching
backwards from all one positions into
all positions with ten ten checkers or
fewer and so this retrograde analysis
was very important and what was
interesting about this was that in some
sense Chinook when we talked about the
IQ test of humans against machines
checkers was interesting because from
the human point of view it had arguably
the greatest player of all time in any
game which was Marian Tinsley so Marian
Tinsley only ever he played for his
entire career the game of checkers and I
think he only ever lost five games of
checkers in his entire career and and I
think two of those were against Chinook
so he really was almost perfect from a
human point of view at this game and
Chinook came along and there and it
actually technically defeated Marian
Tinsley in the World Championship match
in 1994 so this was you know before
before deep-blue came along and it was
doing very well but actually Marian
Tinsley had to withdrew for health
reasons and I fortunately went on to die
shortly afterwards and so no one ever
really knew if Chinook would really have
gone on and beaten this this greatest
player and so the guy who made this
Jonathan Schafer decided that the only
way to really figure out if Chinook to
actually prove that Chinook would have
beaten this this seemingly almost
godlike human player was to actually
solve the game altogether so if you can
beat God if you can door against God and
then you know you know that you've
reached the highest level of play in the
game of checkers so he spent the next
decades doing this and eventually sold
the game of checkers
so that means really combining this kind
of minimax search from the top with this
retrograde analysis from the bottom
until they meet and actually having a
perfect solution to the game and so
they're so checkers is the largest game
that was actually has actually been
solved in that sense largest highly
played game okay
so that's the kind of strategy that was
used for for many many years and I hope
you're thinking sitting there thinking
well you know but what about principled
machine learning methods how can we
bring everything we've learnt to bear to
do something better and this brings us
back to our self play idea and so you
know what happens if we were actually to
apply our value based reinforcement
learning algorithms to games of self
play to try and find this minimax
solution could we actually use this
we've defined a minimax value function
we know what that means now we know that
it is in some sense the optimal way to
play in this game between two players so
can we find that value function using
the same technology that we've seen so
far elsewhere in the course and the good
news is that yes you can and I'm not
going to give the theory that proves
that these things actually converge to
the minimax value function but if you
pick out like Michael Lippmann's thesis
for example if you're interested that
would give you some examples where
principled self play reinforcement
learning techniques do indeed find the
minimax value function or the Nash
equilibrium sometimes in more general
cases so the general principle is
exactly the same as we've seen elsewhere
in the course nothing has to change at
all you don't have to do anything
special to play games compared to any
other domain this is the beauty of
reinforcement learning that it applies
it's such a general idea such a core
principle that it applies with almost no
change to these these different domains
and so all we're going to do is to say
well we can have some value function we
can have our parameterization this V of
s comma W so these are the parameters of
our value function so you can think of
this as your deep network representing
the value and it's just estimating what
is the minimax value from state s and we
want to adjust these parameters W so as
to make them more more like the true
minimax value and so we're going to use
the kind of familiar techniques like
Monte Carlo learning TD learning TD
lamda and again the idea is simply to do
something like gradient descent we're
going to adjust our value function a
little bit towards the return if we were
doing Monte Carlo learning so so that
means you know if I thought I was losing
but I saw that at the end of the
game actually ended up winning I would
adjust my weights a little bit too so
that I ended up predicting I was winning
a little bit more if we're doing TD it's
a bit more like you know I mean I'm
playing a game I think I'm losing the
game but then I take one step and it's
like oh I realize I've just made a
blunder now I think I'm I'm you know I
thought I was I thought I was winning
now I think I'm losing so even before
the end of the game just after that one
step we can update our value function to
say well I thought I was losing but I
should add I thought I was winning but
actually after this one step I now
realized that was a bad estimate now I
see I've made a blunder I should adjust
my estimate I had before to say well
actually I was really I was really
losing I just hadn't seen that blunder
yet so you adjust your value function a
little bit towards your value prediction
he made after one step and you'll notice
there's no intermediate rewards here
that's just because we're considering
this common case we're all intermediate
rewards to zero if you do have
intermediate rewards there would be a
reward in there if you do have
discounting there'd be a discount in
there but normally when we're thinking
about games you don't need either of
those things so these things just kind
of look a bit simpler it's just the
difference between what I thought who I
thought was going to win and then who I
after my step who did I think was going
to win a TD lambda same ideas but we can
use the the land return as performed so
we can actually introduce these
eligibility traces and get something in
between these this spectrum between
Monte Carlo and TD zero so one thing
which is useful when we think about
games but is also truthful in
deterministic MDPs
so this is not specific to games is that
it's often the case that you don't
actually need action value functions so
probably in the rest of this course
you know you've been taught that
well-to-do RL you need to have an action
value function if you're doing value
based RL because you need to have some
way to estimate how good your action is
in the absence of knowledge of your
environment but in deterministic mdps in
deterministic games it's actually
sufficient just to have a state value
function and the reason is that there's
only one thing which can happen after
you take your action there's no
stochasticity in the environment and
there's no like variability in what will
happen you don't need to compute this
expectation so we
basically just say well in these games
you know this is the case where we we
can actually take a step forward you
know if you're able to take a step
forward I think a better way to say this
is if you know the rules of the game or
if you know the model of your MDP you
don't need an action value function
because you can always just step forward
once with your model or step forward
once with your rules and we call that
the after states that basically we're
saying you know if you want to know
what's the what's the value of taking
action a from from state s you can step
forward from s by taking that action and
seeing which state it takes you into and
now evaluate that state that you arrive
into okay so I'm in some position I play
a move I find myself in some new
position and the value of that new
position is actually the value of having
taken the move that took me there
because there's nowhere else I could
have gone and I know I know the outcome
of what this rule would do and that
really simplifies things it just means
we need to learn the state value
function if you can learn the state
value function it's a smaller object you
know you don't have the cross-product of
the state space for the action space and
things just become easier and now when
you actually just want to pick your
actions
well you consider all of these different
steps that you could take and you pick
the one with the highest the highest
value okay so I'm going to skip forward
to do this with backgammon
so this is probably you know until deep
learning took off this example here I
mean you can think of this as something
which was done by a guy called cerita
sorrow that in a way was you know 15
years maybe 20 years ahead of its time
in that it was an example essentially of
deep reinforcement learning which was
very very successful and achieved from
first principles superhuman performance
in the game of backgammon completely
learning for itself all strategies
before anyone else had really figured
out that either reinforcement learning
or deep learning where we're we're a big
deal and and general and powerful
methods for doing anything and so I
think it's really useful to look at it
and try and understand what what went on
there
because it's really one of the
outstanding examples in the field of
where this really works this idea so
backgammon first of all you've got some
kind of board and the idea is that each
player is trying to progress their
checkers then you roll dice and then you
move your checkers according to the the
out what shown on the face of the dice
there's two dice and you move your
checkers around this way for for red
here and this way for black and
eventually you've kind of got to take
all of your checkers to the end of out
of play basically and then the first
player to do that wins it's kind of like
a race except that you can land on the
other player if a player has a single
checker you can land on it and capture
it and it's sent back to the beginning
again so it's a race with those kind of
interactions where you can capture each
other and you can block each other if
you have more than two two or more
checkers on the same on the same point
and so what happened at TD gammon was
this whole board was just kind of
flattened out into a set of features to
say basically a set of features saying
you know do I have one checker in this
position do I have to checkers in this
position do our three checkers in this
position do I have no checkers in this
position and so for each of the the
positions on there on the backgammon
board there were a set of features
saying how many checkers were there I
said two binary features and so there's
this binary representation that was just
kind of flattened out say what was going
on in the board and then a very simple
multi-layer perceptron which was
absolutely tiny by modern standards was
put on top of that and it out put a
value function estimating who is going
to win this game of backgammon I'm given
those that those those checkers and the
learning algorithm was remarkably simple
it was really just very pure TD learning
temporal difference learning so this
neural network was initialized with
random weights it was trained by games
of self play and it was trained using
nonlinear TD learning in other words
this TD arrow was computed with respect
to this value function in other words it
was exactly what we talked about as
computing this error between who whose
do you think was winning in this
position and whose die think was winning
once I took played my move that's this
TD error and then the weights were
adjusted to correct that error so the
weights were adjusted a little bit in
the direction of the gradient to make
your
estimate before a little bit more like
what you thought the value should be
having played that move so that's
basically what this rule is saying it's
just saying adjust by by simple
stochastic gradient descent in the
direction of this error along the
gradient which would tell you how to
adjust your parameters so as to achieve
some particular outcome more okay it's
that clear is an update and so there was
nothing else I mean it was really that
was that was the whole algorithm it was
really simple that was no exploration at
all like can anyone think why wouldn't
you need any exploration in backgammon
why how can this work you've had a whole
lecture on exploration telling you how
important it is to have exploration and
yet this works like perfectly achieved
superhuman performance without any
exploration any thoughts as to why right
exactly so there's enough stochastic
stochasticity in the environment in the
dice rolls to mean that you see the
whole state space already you don't
actually need to do something special to
explore to cover the whole state space
you get everywhere and so this algorithm
always converged in practice and that
wasn't true for other games so people
tried other games at this point and got
very excited and the same ideas at that
time didn't work in other games and I
think partly because of the exploration
issues partly because maybe just that
the deep learning wasn't quite there yet
this was very small neural networks but
one of the nice things about backgammon
is that dice not only mean that you
don't need to explore they also make a
very smooth value function that you
don't have these very precise kind of
cliffs that you fall off if you just do
something slightly wrong because you
know if you're looking a few moves ahead
there's always some randomness in what
might happen and that randomness tends
to smooth out the valley function and
give you a nice surface to kind of learn
about so this program having done that
they went ahead and it basically
defeated the human world champion back
in 1992
so that was an amazing result it did
have some search on top of there so it
was using like a a three fly a three
step ahead look ahead using alpha beta
search too so once it had this it was
using that as the leaf value of this
minimax search and bubbling those
minimax values back up to the root to
decide what to do
it also back in this version which
actually performed this result I
slightly lied and that that one also had
some expert knowledge in there but
Tesoro went back later and he showed
that if you take out all of that expert
knowledge and this was just a few years
later if you take it all out and you
carry on training with the version with
no knowledge at all it actually ended up
outperforming the version that he'd used
in that match so this was really
something which learned completely from
from raw data nothing else yeah okay so
the question is where's the
non-linearity the non-linearity is
inside the multi-layer perceptron so it
takes this flattened representation of
the board and it passes it through like
a two layer multi-layer perceptron and
each layer is like kind of like a linear
combination of the previous one followed
by a nonlinear activation the so there
was a I think at NH activation that
followed each one of those and that
non-linearity in the middle is what
makes this whole function nonlinear and
if you don't have that non-linearity as
with all of neural networks it just if
you don't have non-linearity somewhere
it just collapses to a very boring class
because the linear times a linear times
a linear is just a linear and so if you
want to have it's the kind of this
dilemma that you know linear function
approximation is so much easier and so
much cleaner and we get so much nicer
convergence properties if we can work
with it but it's just not very
expressive and so if you want rich and
powerful and expressive representations
you need to do something harder you need
to use nonlinear function approximation
to achieve those results and that's
really why white people have moved
towards deep learning in their in the
last few years I mean deep learning is
nothing more than you know rich
compositional functions which compose
one layer one function into another
function into another function that's
what deep learning is yeah
yeah I think that's probably true so I
think the sharper the the value function
more of these cliffs you have the more
precise your search needs to be all the
more accurate your value function
approximation needs to be I mean there's
always this trade-off between the
quality of your your function
approximator and the amount and the
quality of your search and you can make
up for one with the other so but I think
you know in some sense it's expressing
something fundamental about the
difficulty of the problem and you need
to address that difficulty either
somewhere in your toolkit so it can be
in the search or it can be in the power
of your function approximator so it
might be that you equally have require
more steps of look ahead internally
within your within your function
approximator you can think of your deep
network as representing kind of
shortcuts to how well you can do by by
searching it's kind of internally doing
some steps of computation and trying to
trying to find shortcuts to figuring out
the minimax value without doing any
explicit search and so that you can put
more layers in there as well and that's
not a native okay so that was TD gammon
[Music]
I'm going to skip this section okay
should we take a little break and then
we can come back for for the next part
so let's take a 10-minute break
and then we're going to talk about some
more modern search techniques which are
about Monte Carlo search simulation
based search and moving on to what's
been done recently by combining these
methods with deep learning let's get
going again so I want to talk a little
bit now about a very radically different
form of game tree search which has
proven to be particularly effective in
in challenging games and larger
branching factor games and also real
world domains it's been very widely used
now and most recently has also turned
out to be very effective in even in the
domains where traditional methods of
minimax tree search were considered to
be dominant and so this idea is really a
broad class of ideas which we can call
the simulation based search and the idea
is really really actually surprisingly
simple and the idea is to say that
actually self play reinforcement
learning can replace search so so far
we've kind of seen that there were these
two different ideas
either we could do like this minimax
look ahead we're really kind of bubbling
ahead and considering all of this tree
of look-ahead possibilities where if I
go here and you go here and I go there
you know what's the value going to be
and we could back that up and back that
up and really compute the minimax values
or we had some kind of function
approximator like a value function there
was estimating those values directly
without ever doing any look ahead and
the surprising part is to say that
actually the reinforcement learning can
replace that first part that actually we
can use or itself a reinforcement
learning instead of a search procedure
to actually give us a process of look
ahead and the idea is really simple the
idea is to say if I want to do look
ahead from a particular state how can I
do perform that look ahead well what I
could do is I could just start consider
the game that starts from that states
consider there you know this is it also
applies in single player single agent
domains where you've got some MDP some
massive MDP and you just want to
consider the sub MDP that starts from
now and in games we can consider the sub
game that starts from now so you're in
some root state you're in some
particular position you're in some state
right now and you want to figure out
well what's the best thing from here
onwards so the idea is to simulate
experience that starts from now like I
mean right now I'm kind of talking to
you guys here delivering this lecture
and so if I'm trying to figure out the
best sentence to speak next it's more
useful for me to start to imagine what
those sentences might be and whether
you'll be horrified at what I say or
whether you'll actually you know learn
something from what I say then it is to
kind of imagine myself up in a mountain
in the Himalayas trekking but you know
if I find myself in the holidays
trekking up a mountain in Himalayas it
wouldn't be very useful in that
situation to then start imagining what
would happen if I was standing in front
of a lecture hall full of students so it
really matters to focus your your
learning on the subspace of experience
that starts from now you don't want to
consider everything the state space in
general a massive we want to consider
very large rich problems huge domains
and we want to focus our limited
resources and our resources are always
limited compared to the complexity of
the real world and we want to focus
limited resources whatever capacity we
have on the subset of experience which
we're actually about to encounter but
there's some agent and it's about to see
some set of events occur and if it can
learn to anticipate what those events
are and that subspace of experience it
can do much much better as a question
okay so the question is how do we deal
with the opponent strategy and the
answer is by using self play so what
we're gonna do is we're going to imagine
that both I and my opponent follow the
best strategy that I figured out so far
and so what we're not trying to figure
out what's the what's the best response
to a particular opponent strategy that
would require some model of how that
opponent behaves instead we're going to
again try and find the minimax strategy
this Nash equilibrium and with way that
we do that is by self play where we're
going to imagine if I play according to
the best strategy I have so far and you
play according to the best strategy I
have so far and then I play according to
that best strategy and so forth what
would be the best behavior and by
learning from those simulated games we
can again find something which will
approximate so what we're going to do is
we're going to just run experience
starting from now again and again and
again and run complete games of
experience that start from now so
complete sub games that start from now
that can start from this root state and
learn in exactly the same way as we were
doing our self play RL like imagine
you're doing you know tessaro's TD
gammon just like we saw in backgammon
but instead of kind of training these
games from complete games that always
start from the beginning you actually
say well right now I'm in this
particular board state and I'm just
going to run thousands and thousands of
games that start from this particular
position and learn a specialized
neural network that's dedicated to
actually solving the situation from now
onwards and that would actually be a
form of look ahead search it would learn
all of the particular patterns that
actually occur in the situation that
starts from now because you're focusing
all of your experience on the future and
so if you have this model of the future
if you have the rules of the game which
we do in this case we can exploit that
by simulating this experience and
learning from that extreme elated
experience so did you hear about in the
previous lectures Monte Carlo tree
search
okay so Monte Carlo tree search is a
special case of this
c.j so people elsewhere tend to teach
Monte Carlo tree search is like a a
particular form of look-ahead search or
the particular structure for you guys I
think you can understand this in a much
more profound way which is that Monte
Carlo tree search is actually Monte
Carlo control it's a it's a
reinforcement learning algorithm that's
applied to the sub-problem that starts
from now so instead of considering the
entire MDP and applying monte carlo
control to that entire MDP what you can
do is instead focus on the sub MDP that
starts from this particular state and
apply monte carlo control where you're
always resetting to the beginning of
that sub MDP and you're learning from
experience of what happens when you
reset only back to that particular state
you're always coming back to this
particular board position or me back in
this situation here and not in the
himalayas and you're trying to learn
what happens from your simulated
experience from your imagination imagine
again and again what happens if I'm
going to try this what happens if I try
this and learn from your imagination
what's effective and what's not
effective and if you can learn from your
imagination what works well you can
actually do better as a Monte Carlo tree
search is exactly Monte Carlo control
with table lookup where the table is too
big to represent the entire state space
so the table is instead represented by a
tree that only contains the things that
you've seen so far and you just kind of
expand that tree when you get to new
things you haven't seen before and so
that's what Monte Carlo tree search is
you're basically doing Monte Carlo
control you're running simulations
you're taking the average of those
simulations and putting that average
value back in each step storing that in
there in your tree and that gives you a
way to to actually search so amongst
Monte Carlo tree search algorithms very
many of the most effective variants use
some kind of upper confidence learning
algorithm when you learned about Bandits
and in exploration strategies you
probably came across the UCB rule which
is one way to balance exploration
exploitation and if you plug that into
your Monte Carlo control so every step
you have some kind of decision about how
much to explore and how much to exploit
you can also use Epsilon greedy but
every step you've got to some decision
about whether to explore or exploit and
that gives you something like these UCT
algorithms which has been very very
effective in many games
and so self play UCT actually converges
on the minimax values so this is one of
those cases where well although we're
doing something which is you know
basically a fundamental RL algorithm
applied to our imagined experience that
might seem like a woolly concept but
actually it really converges on the
minimax values in other words we are
finding in these two player zero-sum
perfect information games that they're
considering this really will find the
optimal solution the Nash equilibrium
which in this case is the minimax value
function so you will be done if you do
this you'll find the right thing and
more generally you can introduce
approximations in very large games we'll
start to introduce approximations you
know value function approximation and so
forth and that will give us a pretty
decent approximation to this minimax
value function so it's a kind of search
where we're kind of imagining what will
happen I'm imagining what will happen
under my current strategy but at the
same time I'm learning to improve that
strategy from the experience I've seen
so far and that's how it can actually
converge on this thing like I I start to
imagine what will happen if I try this
and then this and then this and you do
this and I do this and they start see
all actually every time I tried that in
my imagination I lost so then I'll
change my strategy to do something
different and now I'll try something
else where I do this and then this and
then this and then I win and now I say
okay actually that was a better strategy
and now my opponent might need to adapt
to that in my imagination and change the
strategy again and we kind of had both
players continuing to improve and
improve all in in imagination starting
from this day onwards focused on this
kind of subtree of possible reachable
things from here onwards so that's the
idea of it okay so for a few years now
MCTS has been the best-performing method
in in many challenging games so go I'll
talk about a little bit more hex lines
of action Amazon and I'm also going to
talk about some recent results shortly
where we see actually it turns out to
also perform very well in games like
chess where it wasn't where minimax
search methods like alpha beta were
previously thought dominant but what's
interesting is that in many actually
really interesting games simple Monte
Carlo searches enough by simple Monte
Carlo search I mean something where we
don't improve this
jouji for both players where you just
kind of have some randomized rollouts to
the end of the game under some simple
fixed strategy and then we just see how
well that does and that can already do
very very well in games like Scrabble
and backgammon it's enough to reach
superhuman performance
yeah question so we're basically
learning a policy pair for both players
and one very sensible approximation is
that is that the one side of the policy
plays in the same way as the other side
but just with the kind of board colors
reflected but that's not a necessary
requirement for this we're basically
learning the policy pair is the way to
think about it so we're learning the
policy pair and we're improving the
policy from both players perspective and
we're playing according to that policy
pair in our imagination and each time we
improve one side of that policy we then
go on and improve the other side of the
policy as well and so we're really
trying to find a policy pair that
approximates that converges on the
minimax
strategy well in it depends what you
mean not necessarily I mean that you
this because different players see
different situations so it might be that
one player has an advantage for playing
first for example and so so the board
state can differentiate the strategy
from one side to the other so I think
it's just best to think of it as a
strategy pair and it might turn out that
in the way that we approximate this that
we exploit this this mirroring property
but it's not like a necessary piece of
understanding this I think it's clearer
to think of it's just a policy pair that
that is improving and each policy well
what's the policy doing I mean the other
way to think of it is it's it's just you
can also think of this as just one
policy where the one policy sees as part
of its input which color pieces it has
and so so we provide this policy and it
says oh now I'm in a situation where
I've got the white pieces here and
you're asking me to play a move and now
I'm in a situation I've got the black
pieces and you're asking me to make a
move but it's you know in a sense that's
the policy which we're considering
that's the policy
we want to so you can think of it it's
just part of the function approximation
architecture that sometimes you're
seeing black and sometimes you're seeing
white and you can do things like
mirroring the colors but that's that's
optional in this setting yeah okay great
question can we truncate the Monte Carlo
search before we reach the end of the
game so we don't do full Monte Carlo
rollouts truncate and use an evaluation
function that's gonna be the next part
of the at the discussion so so let's
let's start by doing very simple things
so simple Monte Carlo search is enough
so so I think you know one of the
reasons why simulation based search
became popular was people starting to
realize that even even very very naive
simulation based searches can can be
remarkably powerful so so here's an
example this is the the game of Scrabble
so in Scrabble you get some letters on
your rack and you have to kind of figure
out what's the best word to play on the
board and you get points according to
each of your your letters and so you've
got these points on your rack you've got
these seven letters and the idea was
that the first strong program which came
along in Scrabble was this amazing
program called maven and what it did was
it tried to have an evaluation function
which basically said in how good is it
to play a particular word onto the board
and what it would do it would evaluate
that that move by the score you would
get for playing that word onto the board
plus the value function of the letters
that you've left on your lap rack and it
should probably have taken account of
the board position as well but it kind
of ignored that ok this was already of
huge advance over humans and anything
which would come before and what it did
was it just kind of looked at the rack
and kind of said well you know how good
is it to leave certain letters on your
rack like if I play my blank now which
can be any letter I want and it's really
useful in Scrabble I might get a better
score but there's an opportunity lost
there because by keeping it on my rack
maybe next turn I would have got a
50-point bonus for using all my seven
letters and so it kind of learned all of
these looked at all of these just one
two and three letter features like do I
have a cue left on my rack well that
would be really bad but if I've got a QA
u together that might be actually quite
good because the cue is very
high-scoring if I have three eyes on my
rack that's pretty bad you know use
not many choices you've kind of lost
some flexibility there and then this
whole thing was learned by the kind of
Monte Carlo policy iteration so said I'm
not going to talk so much about how the
value function was learned because we've
already dealt with that this value
function was learned by standard
essentially td1 um
the value function was learned by kind
of every single state played the game
you look at the outcome of the game and
you update the value the the the value
of these function approximator to make
your estimate of what happened a bit
more like what actually happened in the
real game so that's the standard what
we've seen before just like in TD gammon
but an even simpler representation what
was interesting was that the way it did
search so actually did search by by
rolling out by imagining what would
happen into the future so you'd start in
this position and it would basically
just play out games according to its
strategy so it kind of play out some
number of moves into the future and then
it would truncate and actually say okay
I'm just gonna look ten moves into the
future and then evaluate the position
that I'm think I'm in there and and it
will select and play the move which has
the highest average score so if you just
kind of imagine what's going to happen
just keep imagining what will happen
without improving that strategy just
keep imagining games and trying you know
a few hundred games in your imagination
and like randomizing over what you think
you're gonna pick out of the bag and
then just play the highest scoring move
then this is a really really powerful
form of look ahead and just to show how
powerful it is here's a little example
of what what maven did when it actually
played against the world champion this
was the first time a computer Scrabble
program ever be a human world champion
some quite a while ago now but anyway
and it beat this human world champion
Adam Logan by nine games to five and and
there was this one game where Adam Logan
was like just way ahead in this game and
in you know all the pundits thought this
was this was over there's no way that
that maven could possibly come back but
made them with this kind of look ahead
using this kind of Monte Carlo rollouts
saw they only had one chance to win
which was to do something called
phishing which is if you're behind you
kind of have to just hope that you can
kind of find this one spectacular letter
in the bag that will help you get a
really high score and the thing it
fished for was it ended up picking out
this
mouthpart and you'll notice that this is
one two three four five six seven eight
nine letters so so these two are ready
down and it had two fish to kind of hope
to it was basically predicting this
endgame would work out in this one
particular way and if it picked up the
particular letter you'll be able to play
a mouth part by the way I don't even
know what a mouth part is as an English
word but anyway let's Scrabble for you
but and it kind of was able to use this
Monte Carlo analysis from from much much
earlier in the game to realize there was
only one chance for it to win and to
find that chance and of course it was a
little bit lucky as well but anyway it
turned out to play near-perfect Scrabble
I mean there's some luck in the game but
it played very very well and these same
ideas were very effective in backgammon
we saw we saw earlier the example of
backgammon doing kind of a little bit of
search on top they're actually
backgammon again used a very someone
showed subsequently you could use even
better results by doing this kind of
very simple Monte Carlo search in
backgammon and so one way to think of
these very simple search procedures is
it's like doing one iteration of policy
improvement because you're kind of what
are you doing well you're you're taking
your current policy you're imagining
games all the way to the end and then
you're picking a greedy step with
respect to those imagined games so the
greedy strategy with respect to how well
he did in those in those games is
exactly one greedy policy improvement
step you're saying okay I'm gonna look
at how well I did in those I'm going to
evaluate it that's your policy
evaluation to see what was the mean
outcome of all of those games and then
you take a greedy step you pick the
thing which actually the action which
led to the best outcome and so if you
keep acting greedy with respect to those
estimated mean values that's doing one
step of policy improvement so Monte
Carlo search is effective because it
does one step of policy improvements
it's doing one step of policy iteration
and so a natural question is if one step
of policy iteration is so powerful what
happens if we allow ourselves to do many
steps of policy iteration and so that's
what leads to Monte Carlo tree search
what I want to talk about now is our
recent work a deep mind on actually
taking some of these ideas further in
the alpha zero project
so it's always hard to know how much you
guys know about this stuff but I'm just
gonna talk about it anyway and hopefully
if you've heard about some of its before
you won't mind a duplication if you've
heard nothing and I hope there's enough
for you and and so forth so what I want
to talk about is how we started to bring
about an architecture which building on
these ideas so far so building on on
value function rips approximation policy
approximation these ideas of doing some
kind of simulation based search with
multiple iterations of policy
improvement inside it how we could build
a system which started completely from
scratch with no human knowledge at all
and was able to master some of the most
complex games where previously people
using a lot of awful lot of knowledge
there and I'm gonna use go as the case
study for the beginning and then we'll
talk a little bit about other games so
why go well it's the oldest game in the
world about three thousand years years
old
amongst still currently played games
there's still about 40 million active
players and there's about ten to the
hundred and seventy positions so what
does that look like well there's this
enormous branching factor in the game of
Goa where from each of these positions
you reach several hundred other possible
positions there's several hundred moves
in the game it's um 19 by 19 board you
can place down a stone on any one of
those and that leads to a vast array of
possible next positions and from each of
those situations they've got a vast
array of possible positions and so if we
were really to consider the minimax
problem doing this minimax search and
really computing the minimax value you
would have to consider all ten to the
hundred and seventy of these states to
actually compute the true value like who
who actually has won this game like from
the beginning who what's the perfect way
to play
that's intractable and so really the
story of alpha zero is the story of
trying to deal with that intractability
with with principled reinforcement
learning methods and so I'm going to
start by talking about alphago which was
the original program that was able to
defeat the world champion
Lisa Dahl and that will give us some
intuitions that we can then bring into
you know making something even more
general that was able to to play many
games and with even less knowledge and
so the idea of alpha zero was to use two
neural networks the first of those
neuron that
was a policy network and the policy
network kind of looked at the position
in the game of go and he used this
convolutional neural network to kind of
build up features so each layer of that
convolutional neural network is
basically looking at a small region of
the board building up features which
represent what's going on in that
particular region of the board and that
gives you like a new layer of features
saying well I started off knowing that
these were the black and white stones
here but at the next layer maybe I know
which stones are adjacent to each other
and the next layer maybe I know which
stones are are under threat and the next
layer maybe I've got some sophisticated
idea of the kind of life and death
that's going on and so forth and so we
had many many layers of policy network
in the most recent version this is about
an 80 layer neural network so this can
get very very deep and what we output at
the top of it are move probabilities
something to say well in this position I
think it's a good idea to play here or I
think it's might also be a good idea to
play here but it's a really bad idea to
play over here and we represent that as
kind of like a probability map like a
probability distribution over the legal
actions in the game okay
and there's a second neural networking
in alphago which is called the value
Network and the value network is also a
convolutional neural network which takes
the the board as an input it looks at
all of the black and white stones builds
up features of features of features to
get more and more sophisticated
representation of what's happening in
that position and at the end of it it
outputs a single scalar number saying
who it predicts is going to win the game
you know like plus one if you think
white is going to win and minus one if
black is going to win okay so that's the
value network and so this is a form of
value function approximation and this is
a form of policy approximation and the
idea is that in the original alphago we
trained these two things by first of all
supervised learning so we started with
human expert positions and I hope you're
thinking oh I thought we were doing
reinforcement learning here so why do we
need to take human data into account and
we're going to address that later okay
so we started with human expert
positions we trained our policy network
to mimic what human experts did in each
of those positions
so in each board position we said okay
in this position the human played there
and so we adjusted the parameters of our
policy network to say we want the policy
to output that same move that the human
played we just adjust the parameter of
our neural network to predict the move
that the human played so that straight
classification once we've got that
policy network we apply reinforcement
learning to that policy network in other
words we actually got this policy
network to play games of self play
against itself and we adjusted the
policy network further by reinforcement
learning by a policy gradient method but
also we built up this value network what
we did was we built a value network
which from each of these games it was
kind of like a you know Monte Carlo
learning of the the value function that
fer we played the policy network against
itself and we looked at the outcome and
then from each of the positions that we
encountered in that game we asked the
neural network the value network to
predict who is going to win so if you
pass in this position the question we're
asking is you know do I think black is
going to win from this position I didn't
think White's going to win from this
position and the way that we actually
build the data is by getting the policy
network to play against itself and so at
the if at the end of the game the policy
network which was very strong ends up
with black winning that's the target we
provide to our value network and we
adjust the parameters of the neural
network to make it provide an output
answer saying it thinks that that player
is going to win more okay
any questions yeah so for every state
that was visited during this so this is
a this is a so think of this this
process is classification and this
process is regression at this step what
we're kind of doing is building a big
data set that contains all of the
positions that were encountered during
all of these games of self play and from
each of those positions we're trying to
predict who ended up winning that game
of self play that's the regression
problem we're trying to solve right from
every position who ended up winning and
if you can predict that if you can
predict who ended up winning in this
game that's a really powerful thing to
have in your arsenal like this is this
is like this is y-value networks are so
powerful they summarize they're kind of
a cache of all of the look-ahead that
you might do this is also true a single
agent domains it's why we build value
functions that you are you want to have
a summary that tells you one shot
immediately what's the what's the value
of being in this situation without
having to do all the look ahead and
consider all of the possible things
which might happen in the future
so it's like a summary of all of the
future possibilities without having to
do this exponential expansion
combinatorial expansion of all of those
possibilities yeah so so the supervised
learning can have bad examples in there
the self play reinforcement learning can
correct those by adjusting the policy
network and that that stage was
important in practice but we'll see a
better way to do that where we don't
even start from human expert positions
shortly and in fact our best results
come from not using human data at all
that subsequently okay so again if we
think of this enormous search tree which
we're trying to compute so this could be
go but it could be any other game but go
just has a particularly daunting search
space and here we're just showing like a
binary search tree it was kind of like a
cartoon of what happens in this search
tree and we can see that you know from
this little situation we could go over
here we could go over here and then
there's all of these possibilities and
the real search tree is like
this but there's kind of you know 200
possibilities from each of these instead
of two and it's kind of a depth there's
about 200 as well so it's a big search
tree and how can we narrow that down
well the first idea is that we can
reduce the breadth of this search tree
by using the policy Network in other
words we don't have to consider the full
breadth of all possibilities like there
might be this enormous branching factor
of 200
but if our policy network tells us that
you know that sensible move to play as
this one the policy network can
basically pick out a handful of
reasonable possibilities and can be
relied on as kind of soft pruning
mechanism where we can exclude the the
moves which the policy network says are
nonsense and only come back to those
later in search so we kind of prioritize
the things that the policy network
thinks are good and that immediately
gives us a much narrower search tree and
that's why the policy network is
effective in this search and the second
thing we can do is to reduce the depth
by using the value network and this is
just like we saw with deep blue and and
all those programs in the past well what
we can do is instead of going all the
way to the end of the game we can
summarize the subtree beneath each of
these leaf nodes by a single value that
estimates who's going to win from that
point onward again a value function is
powerful because it's like summarizing
in one number all of the possible
contingencies from that point onwards
and if you have that one number you've
saved yourself all the effort of having
to do that future computation this is
why value functions are so crucial and
so we can use this value network to save
us from having to build out and search
for this this big subtree over here and
that that makes a dramatic difference to
the size of this search tree so putting
those together we built a a Monte Carlo
tree search algorithm in alphago and
I'll just step through that in a little
bit more detail for those who really
want to see you know what would a what
would a state-of-the-art like Monte
Carlo tree search actually looked like
how does it kind of put these pieces
together to give you those properties
and it really kind of has these these
three strategies this is this is the
strategies that we use in our latest
version of alphago and the first phase
is you've got to kind of select a path
so remember we're always doing this
thing where we're simulating a game
we're kind of imagining what's going to
happen next so we've got this choice of
well what should we imagine what should
we imagine
what should that each player do in each
each of these imagined situations
and the way that we pick the actions in
our imagination is by maximizing these
cue values like there's action values
which will explain where they come from
in a minute
but for now just imagine that for each
action we're storing some kind of Q
value saying how good is their action
and so the greedy strategy would be just
to maximize Q but the problem the greedy
strategy is it can get stuck you need
some kind of exploration in the search
tree you need to sometimes try some
other branch not your greedy one just to
see in case that other branch actually
turns out to be better you have to try
it sometimes and so what we do is we do
something like we saw in the exploration
lecture where you want to add in some
upper confidence bound you here and that
encourages it to consider things which
might be good so if you're if your upper
confidence bound you this upper
confidence bound term you here says that
you that you might actually be able to
get a lot more value than q then we'll
we'll take this path here even if this
one was currently has the best Q okay so
we just add on our Q value with some
upper confidence term and we maximize
that and that picks out a path through
this search tree and the upper
confidence term is actually really
simple it's just proportional to so it's
proportional to the policy Network so
basically if the policy like something
it will take it more often
so that's implementing this idea that
the the moves that the policy network
likes at Ryde and the moves which it
doesn't like are excluded so this U is
just proportional to the policy network
to the policy probability and it's
inversely proportional to the visit
count so the more you try something the
less of a bonus you give it because
you've already kind of got some idea of
what it is and that turns out to have
some justification I won't go into that
so so we've selected some paths now
through the search tree so we're just
kind of imagining things again and again
and again so each each simulation we're
going to start from our roots state we
can imagine what happens according to
this strategy of trying some imagined
path to through this imagined gained
through the imagined world and then
we're going to reach once we reach the
end of that that we reach some leaf node
here what we're going to do is evaluate
that leaf node and expand it
so we evaluate it using our policy
network and our value network to give us
move probabilities from our policy
network and a value from our value
network and you only need to do that
once per simulation and once you've got
that you back up those values and so
here what we're saying is that this is
the Monte Carlo part that the Q value of
every action in this tree is just the
mean value of all of these V's that
we've computed in the subtree but
beneath it it's just a mean it's just a
mean of all of the simulations we've
tried so what we're trying to say is
that you know how good is this action
here well this action is the mean of the
outcomes of all of the simulations that
I tried that started from this action
here
that's the Monte Carlo idea right that
you try something sometimes you win
sometimes you lose and you just take the
average of it so if you played two games
if you play three games and you won two
and you lost one well your action value
would be two-thirds here so you're just
taking the mean of everything that
happens beneath you so so this one here
this pink Q is taking the mean of of the
leaf values that you saw beneath that
this green Q is coming from here and so
forth and once you've got those Q values
you cycle background and use that to
influence your next trajectory so this
has got that policy improvements
happening where we're not only are we
using our current knowledge to influence
the paths we take but we're actually
improving that knowledge in this stage
here we're improving our knowledge in
the search tree and that improved
knowledge now we figured out oh actually
you know turns out now I see that this
was actually a bad idea and that
influences the next trajectories that we
take and those next trajectories will
become better and so forth until
eventually this converges on the minimax
strategy okay so that's the Monte Carlo
tree search strategy so we went out and
we played against Lisa doll who was the
winner of 18 world titles this was in
Seoul in March 2016 and alphago actually
won this match four games to one and
became the first program to beat her
human professional player and I've just
got for those of you haven't oops seen
this I've got a little video here
you
this was the first time computer
programmer
pro-player this was actually the the
machine that played the match this was
sitting in a data center within Google
in the US somewhere you can see the
little board there and but actually we
were in in Seoul operating over a over a
connection to that data center and this
led us on a path to try and do something
which was simpler and using less
knowledge to try and achieve even more
performance and just just finish this as
some of the events that happened there
yeah this was the actual match taking
place with a Jeff Wong who's one of the
developers of alphago playing against
Lisa Dole there this was the press room
which was a little bit crazy this was
actually what we were looking at in in
the room watching the match there you
know actually this is the program our
statistics we're not influencing it
we're just kind of watching it on these
screens and making sure that nothing is
falling apart and there and the
connections that happen up and and so
forth and that was us receiving a nine
down nine down certificate and from the
Korean Go Association so this led us to
a quest to try and do something without
human knowledge right the goal here
should be to have a system which can can
crack any domain I mean we're talking
about games of course we would like to
go beyond games but within games within
this class we've been looking at you
know what what would be the system which
would be most satisfying well it had a
few properties it would start with no
human knowledge at all we would just
provide it with the rules of the game
and it would go out and it would play
learn how to play that game to a
superhuman level of performance and so
we try to set out to do that within this
class of two player zero-sum perfect
information games and this led to the
alpha zero project and so we wants to
throw out anything to do with human so
we threw out human data we wanted
something which learned solely by self
play reinforcement learning starting
from random with no human features so
only taking the raw board as an input so
no no handcrafted features or anything
like that we use just a single neural
network at
to lien alpha zero so the value and
policy networks were combined into one
neural network which actually led to a
big improvement for interesting reasons
and it had a simpler search algorithm so
it was really a very pure search
algorithm and the idea here was to say
you know each time you take complexity
out of your algorithm you achieve more
generality or the converse is true as
well each time you specialize your your
algorithm to something in your
particular domain you kind of handicap
yourself at the same time because you're
you're introducing something which is
locked you from from that same idea
working somewhere else and so the idea
should be that we have something very
pure which can congenital eyes to other
domains with with minimal effort and
ideally something which is you know
nothing where there's nothing really
special case to games at all so this led
to the alpha 0 algorithm and the idea is
really simple and the idea is actually
to do something akin to policy iteration
so I'm just going to talk through this
strategy it's a and hopefully you'll
you'll get the algorithm as a fairly
straightforward idea which is that you
know in each of these positions this
imagine this is your go game we're
actually going to play games where from
each of those positions we're going to
run a search we can run one of our Monte
Carlo tree search --is using our neural
network so inside this we've got our m
tts as we saw earlier guided by our
policy in value network and we run that
search and pick a move okay
and given that move we play it that
gives us it takes us to a new position
we run a new search using onion all
networks play that move run a new search
play that move until we get to the end
of the game ok and that's how we
generate our games so alphago is is pure
self play it's playing against itself
starting completely randomly so we can
initialize these neural networks to
completely random weights and start off
by playing random games but influenced
by whatever those neural networks happen
to think is a good idea and then what
we're going to do is use those games
that we've generated to train our neural
networks to do better so first of all
we're gonna train our policy network to
predict the move that was played by the
like what did alphago actually do in
that search well you know it was in this
position we ran our policy network and
that policy network actually picked some
move well now we want to train our
policy network to do the thing which the
whole search did before so we're kind of
distilling the whole of that Monte Carlo
tree search back into our policy so we
used our policy to influence our search
we ran this huge search like looking
ahead tens of moves and running like a
million simulations and now you want to
kind of distill that thing back into
into a simple decision that's made by
your neural network in other words the
way to think of this is we want to train
our policy network on the best possible
data that we could have and where does
that best possible data come from well
it doesn't come from humans it actually
comes from from alphago itself that we
should trust those moves which it
produces is the best possible targets
for training a new generation of alphago
so we trained the policy Network to
predict alphago smoove so we take
whatever that search did that becomes
the target for classification for our
policy network to predict those moves
not the human moves and we also at the
same time
train a new value network by regression
to predict the winner so whatever the
winner of that self play game was from
each of these positions that we
encountered we now want our value
network to be moved to predict that that
winner so we were in this game you know
we ended up with some some actual
outcome with some winner and now that
winner is fed into all of the positions
that that led up to that win so we're
basically saying from each of these
positions predicts who actually ended up
winning when
alphago played against itself using
search and so we've put the search
inside the loop of reinforcement
learning it's a kind of arm search based
policy iteration where each generation
we're making a new policy and a new
value function and a new policy and new
value function that builds on the last
one with search in the loop and the idea
is that when you learned about policy
iteration before and they have a slide
on this yeah when we learned about
policy iteration before we saw that
policy iteration uses this idea of
greedy policy improvement typically you
kind of you take your policy you
evaluate it to give you a value function
and then you
greedly with respect to that value
function to give you a new policy but in
alpha zero we do something much more
powerful which instead of acting
greedily
we perform a huge search to kind of
figure out the best strategy and search
gives a much bigger policy improvement
than than just acting greedy with
respect to your value function you can
do like a multi-step look ahead to give
you a much better policy improvements
than just this one step jump that you
get from policy improvement in other
words you run a big search and that big
search leads to an action selection
procedure which is much better than the
thing that you started with in your raw
policy that you began with like the
search itself you can think of as a
policy a way to pick actions you run
your search and you do the thing that
was recommended at the root there is a
kind of policy and that kind of policy
is massively more powerful than the
policy you start with and so that kind
of iteration actually helps you do
better and better and better and so what
we do is we feed this back in again and
again and again iteration by iteration
the new policy and value network is used
in the next iteration of alphago zero
and that goes round and round and round
leading to better and better results and
we start off with completely random
weights but very quickly this thing
learns how to get much better policies
and each generation and we get this
massive improvement due to the search
which leads to the next set of data
being produced being much higher quality
than the previous and this happens
completely online so we're always
training on the freshest data produced
by the freshest policy and value network
and it goes on and on and on so we could
think of this as a building on other
work which has come before like like a
darkish era and so forth where people
have looked at other ways to do this
kind of policy iteration schemes but
here we're really putting the whole of
search inside the loop and that leads to
leads to a very very big improvement so
the policy improvement makes use of the
search to improve the policy and the
policy evaluation makes use of search as
well that each time we run a game we're
running a game where each player is
picking its move using search and so the
self play is like self play with search
in the loop like alphago plays it's it
runs the search it plays a move using
search plays another move using search
plays another move using search and
eventually the outcome you get a
game is like gold you know that outcome
you get is really really high-quality
and so when you train that kind of
policy evaluation is much more useful
than if you were to get your raw neural
network to play against itself and then
look at the evaluation of that game like
the search gives us very high quality
outcomes and so that's why alphago Xero
is so much more effective than the
previous version of alphago so this
leads to much more sophisticated play
over time as you start to train this
thing you start off with you start to
see that starting off with completely
random weights after just a few hours it
starts to behave in the way that human
beginners do starting to like chase
after stones and try and capture them
greedily a bit like human beginners do
that after just a few more hours it
starts to play like a strong human
player and understand all of the
concepts of go and after around 72 hours
it was super human so this is a example
of a learning curve so you get this very
nice stable learning and this is typical
of every time we've tried this in every
game we see this very stable learning
curves this is actually you know
repeatable if you if you try again you
get similarly stable learning and this
shows it this is the the version
learning completely from scratch and
this is the version that beat Lisa doll
so this is a much much stronger program
there and it exceeded that version after
about 72 hours there we go okay so you
know one way to evaluate alphago is how
well is it compared to humans and the
previous state of the art so the
previous state of the art before these
these algorithms came along with
programs like Zen and crazy stone which
are very sophisticated Monte Carlo tree
search programs using a lot of the ideas
that we've talked about but building on
massive amounts of human knowledge as
well to try and you know bootstrap their
performance rather than relying entirely
on on principle self play reinforcement
learning with deep neural networks and
so the original alphago when it came
along this was in our nature paper beat
the European champion found way by five
games to nil and that represented like a
four stone this was so this these gaps
here show what happens if we
got this version to play this version
and this is like giving for free moves
to the opponent and it was able to beat
it we were then able to the version
which played against Lisa doll was able
to give three stones advantage like
three free moves to this version and
still beat it these are slightly you
should take these with a slight pinch of
salt because you know these are these
are programs which are trained by self
play and so they're particularly good at
beating beating weaker versions of
themselves and so it's not clear that
that would generalize to a three stone
improvement against humans in fact it
almost certainly wouldn't and then the
version which we played online and beat
the top human players 60 games to nil in
the world and that version was called
alphago master and that beat their
alphago lee by three stones three three
free moves against that version and then
most recently this is the version
alphago zero which i've just been
talking about which was completely
learned from first principles and it was
able to beat that version about 90% of
the time so if there's one thing to take
away from this it's that really doing
things in the principled way it you have
to believe in your principles and you
have to believe that you can overcome
all of this knowledge that that it feels
like you should be building into the
system but actually each time you put
knowledge into a system you're really
handicapping it and if you really go
back to the beginning and find the right
principle to learn for itself it will do
better eventually and we saw the same in
backgammon but the same is true even for
the hardest problems within this sphere
like go okay so it learns human go
knowledge and this is just an example
where it discovers like these human
patterns that humans play and so humans
in the game of grow have spent about
3,000 years developing this theory of
how to play there's actually you can
take degrees in the game of go and and
there's universities devoted to studying
it and and they there's a whole theory
of how to play in the corners which was
discovered by alphago during alphago 0
during its its opening play but
ultimately discarded later in favor of
new openings that humans didn't know
about and there's actually a lot of
creative moves that it's discovered and
played over over time and so of course
what we our goal is you know it's been
all very well to say well this is a more
general algorithm but you nothing is
really general unless you prove its
general by trying it in more than one
domain
and so we tried alpha zero in three
games chess shogi which is the game of
Japanese chess and the game of Go
so Shaggy's so chess is interesting
because it has this whole history as we
saw with deep blue where massive amounts
of handcrafting have gone into building
like these really super powered
computers with decades of experience
where you know chess was the Drosophila
of AI for four decades and the level of
performance that's been achieved in
chess is immensely superhuman shogi is
interesting because it's um actually
only just reached the point where where
computers are able to to play at the
level of the top humans and it's it's
quite different to chess actually
because when you capture a piece you get
to put it back down again as one of your
own pieces so that's like a really
dramatic difference for the game and the
game of go
so chess has a smaller branching factor
than go but it's nevertheless
interesting for other reasons because it
is arguably the most studied domain in
the history of AI and that we have these
highly specialized systems such as deep
blue and the state of the art programs
now our our programs like stock fish
which are indisputably superhuman shogi
is computationally harder than chess and
all of these state-of-the-art engines
are based on on alpha beta search and
they use huge amounts of knowledge to go
into them there's all of these
heuristics and extra pieces that go into
making them really really strong and
each one of these little kind of phrases
here is some you know big advance that
was made by kind of putting a big piece
of knowledge into the program or a big
advance to these these programs and so
there's these really beautiful piece of
programming like stock fish which use
all of these things and alpha zero just
throws that all away and says let's use
self play reinforcement learning with
self play Monte Carlo search these
methods that we've seen so far and in
each of these games within just a few
hours actually it was able to outperform
the world champion so in four hours
alpha zero surpassed stock fish which
was the state of the art program in
chess in two hours alpha zero surpassed
elmo which was the world champion in the
game of shogi in eight hours alpha zero
surpassed the version that defeated Lee
SEDOL that we looked at earlier
so it won against the the world champion
stockfish 28 wins and 0 losses and this
is just showing that in each of these
games it produced very very strong
performance and this is actually a
figure from the financial times year-end
charts that came out in December
this is brand-new by the way all of this
work came out in December that just kind
of showed I thought this was quite nice
it showed how computer chess has
progressed over the decades so this is
actually the state of the art in
computer chess on an E low scale that
kind of shows how much they progressed
over time and you see this is the top
humans in the world and you see at this
point here was was deep blue it was like
a spike where they built this special
hardware and and really performed very
well but actually they continued to
advance
right up until now they've just advanced
in advanced and advanced and then you
carry that forward and this was alpha 0
kind of doing the same but in four hours
so and it scales very well with search
time so actually this Monte Carlo tree
search actually scaled better than alpha
beta search which was thought to be
something with the only method that
would work in in the game of chess so I
just want to finish with saying you know
this is just one example of how to use
all of these methods and so far we've
talked about these games is like a
classic board games as a case study for
AI but of course they're just a stepping
stone towards where we really want to
get to which is actually systems that
learn completely from scratch using very
principled methods starting just from
the raw inputs and without human data to
learn to achieve human levels of
performance in in very rich domains that
we might care about and actually
simulated games such as this one
sometimes a more useful test bed for us
and this an example of an algorithm it's
actually the Unreal algorithm which was
learning just directly from pixels to
try and solve these kind of domains with
3d navigation just look at the pixels
and learning how to achieve the reward
of picking up the Apple and solving
quite interesting and challenging
problems and a lot of the same ideas
which we've talked about are relevant
and I think hopefully if there's
something to take out this case study is
that the
ideas of learning from scratch our
principle they work well they achieve
optimal performance they can reach
superhuman levels and the same ideas are
applicable not just in games but
everywhere else we never with we
shouldn't have to special case in any
way to games so I'll leave that with you
thanks very much hope you enjoyed the
course and yeah I'll be here for a
couple of question I mean it's people
have questions