RL Course by David Silver - Lecture 9: Exploration and Exploitation

okay hi everyone
so
um lecture nine today we're going to
come back to something which probably
most of you have been thinking the back
of your mind surely we can do something
better which is the question of
exploration exploitation
so so far we've looked at really quite
naive methods for exploration where
we've realized that there's an issue
with our reinforcement learning
algorithms and one of the fundamental
questions is how can a reinforcement
learning agent balance exploration kind
of figuring out what's going on in this
world with exploitation which means
getting as much reward as possible along
the way and so far we've really tried
very naive approaches like epsilon
greedy
and this lecture we're really going to
try and investigate specifically that
aspect that fundamental question of
reinforcement learning how can an agent
which is down in the world trying to
figure out
how to get as much reward as possible
whilst learning
how can it do as effectively as possible
and we're going to consider various
different methods for that
and to do that most of the lecture we're
going to start with a simplified version
of the reinforcement learning problem
which we have seen
on at least one occasion before we're
really going to spend some time to try
and understand this simplified version
because it kind of boils down to the
essence of exploration and there's the
multi-armed bandit where you just get to
kind of pick one action get one reward
and
that's the end of your episode you kind
of just have to explore and figure out
what the best action is
um once we've understood that
in the remaining time we'll spend most
of our time here and the remaining time
we'll touch on how to kind of bring back
the full complexity of the whole
reinforcement learning problem but all
of the ideas that we learn here apply so
you know if we don't have a lot of time
here you'll still get the essence of
what's going on so first of all we'll
bring back states that's what we'll do
with contextual bandits we'll bring back
states we'll say now not only is there
an action to choose but there's also
some state information
and that's data information will inform
what we do
one of the reasons to touch on
contextual bandits is it's probably one
of the
most successful current application of
machine learning um it's certainly one
of the top ones is using contextual
bandits to decide how to do banner
placement banner ads for the internet
this has been used by many many large
companies to decide you know you're in
some state where you've got some user
who's coming to this website what should
you show them
um
and then we'll finally come back to the
full case of ndps
okay so we'll start with a brief
introduction to try and understand
what's going on in this area
um so
every time with decision making online
this same choice comes up again and
again and again which is you know right
now i can make a decision i can take an
action
but what should i do should i exploit
which essentially means to make the best
decision given the information we have
so far
so if you've got some information maybe
you figured out your value function and
according to that value function you
really believe that one action is the
best
so exploitation means taking that back
best action acting according to the max
whereas exploration means doing
something else and there's a purpose to
doing that something else which is that
we gather more information
which might lead us to make better
decisions in the long run
so by gathering information we believe
that we might actually do better than
taking what we currently believe is the
best action
so often the best long-term strategy
might actually involve giving up reward
in the short term like we really believe
right now that taking you know going
left is going to give us more reward
than going right and yet we choose to go
right that's giving up reward that's
giving that reward we know we could get
right now
and we're giving up that reward because
we think we can get that reward back
later we think in the long term it's
better to explore it's better to go
through that right hand door because now
we figure out well what's behind that
door maybe there's a dragon there but
maybe there's a pot of gold there and so
sometimes we explore to find those pots
of gold and work out what the better
option is
and eventually we want to make sure that
we
make the best possible decisions
so here's a few examples
so you know if you're going to a
restaurant
you might want to explore
by trying a new restaurant or you might
want to exploit by going to the one you
know best
the online banner ads i mentioned
already that there
you might want to explore by showing
someone a user maybe you showed them a
different advert what you've shown that
kind of user before um to exploit would
be just to show the one you're making
sure they're going to click on you want
to probably maximize click through in
these examples
if you're oil drilling you might want to
drill at the best known location or you
might want to explore by drilling up
some new location
and if you're playing again you might
exploit by playing the move you believe
is best or you might play an
experimental move to to explore so in
all kinds of different domains you know
why does this come up it comes up
because we're learning online you know
it's not just that someone gives us some
batch of existing data like in
supervised learning um it's not there's
a it's not there's a data set and then
we get to go over that data set as much
we want and make the one best decision
we're not in that setting we're in the
setting where we're gathering data as we
go and the actions we take affects the
data that we see and so sometimes it's
worth taking different actions to get
new data to get parts of the data we
haven't seen so far
okay so that's the fundamental issue
that's going on
we're going to focus on three different
approaches to the exploration problem
the exploration exploitation dilemma
these are not the only approaches
i think there are three broad families
that we should know about
and the first approach is what we've
already seen which is random exploration
which basically says well you know maybe
a reasonable approach is just to
sometimes with some probability pick a
random action an example of that would
be epsilon greedy or perhaps we might
use a soft max distribution over our
value function these are ways to
introduce randomness into the way that
we pick actions so we don't always just
pick the greedy action we throw in some
dice roll and if it comes up with a you
know with a six then we we choose
something exploratory
in this lecture we're also going to
explore some more systematic approaches
to exploration which make more use of
accumulating knowledge along the way
the first approach is known as optimism
in the face of uncertainty so this is
like a fundamental principle that you
should know about and this
basically says that you know if you're
uncertain about the value of something
you should prefer to try that action if
you don't know if there's one action
that you're absolutely sure will give
you ten there's another action which
might give you anywhere between five and
twenty which one would you take well you
should probably take the one which might
give you twenty because in the long run
that means you'll be able to come back
if it turns out to be worth more than
your your
safe option that gives you 10 then you
should pick the option which has the
greater potential because it might turn
out to be better and then you can pick
it again and again and again once you
know that it's better it turns out to be
worse you've lost a little bit but you
can always go back to your preferred
option
that's the idea of optimism in the face
of uncertainty so in order to do this we
need some way to measure our uncertainty
on on values and we'll look at different
ways to do
a variety of different approaches
frequent espacium
and so forth
finally perhaps the
most
correct or um theoretically
careful approach to the exploration
exploitation dilemma but the most
computationally difficult um is to think
about the information state space
so what this really means is to consider
the agent's information itself as part
of its state
it's like
i'm in a state where i have tried left
three times and i've tried right one
time that's a state now we can say
that's a state and we can ask about how
good is it to move into states where the
agent has accumulated information so if
i i might be in a state where i've never
seen what's beyond that door over there
and that's a very different state to
being in the same place but knowing
what's beyond that door over there
and so if we bring the information into
part of our state description then we
can actually
really understand how valuable is it to
accumulate information how valuable is
it to find a new piece of knowledge
about what's inside that door
maybe it's a really good thing because
it will help me to get more rewards
later on but maybe it doesn't actually
have much effect because um there isn't
enough time to to exploit those rewards
or all kinds of other issues
so this is the sort of correct but um
computationally very difficult because
our state space blows up to something
massively more complicated and
difficult than we had before
so those are the three approaches we're
going to consider
i really just before we continue i want
to touch on
something which i consider quite a
fundamental distinction in the way that
we think about exploration
it's not talked about too much
but there's really two different spaces
in which you can explore
we can explore in the
state space or the action space
so this means that you know if you're in
a state and you want to consider should
i go left or should i go right from this
state you know we know that maybe i've
been in this state and i've taken the
left action before so now if i'm doing
state action space exploration that
means i know something about the state
actions i've considered so if i come
back to this state space i might try
going to the right the next time around
that's using knowledge about the state
action space to help explore this whole
state space more effectively as part of
the state space i've seen parts the
state space i haven't seen there's
actions i've tried there's actions i
haven't tried we use that knowledge to
help us explore systematically and
figure out the right rewards
but there's a different space in which
we might choose to explore which is the
parameter space
so in the policy gradient lecture we saw
that it was possible not just to work
with value factors but to work directly
with policies
in that case our policies have some
parameters this describes the behavior
this describes how my agent's going to
operate for
as it goes on in the world like i drop
down my robot it's going to behave
according to whatever parameters make
our eyeball walk according to some way
of walking around and what you can do is
you can try some parameters for a while
and see how they do
now exploration you could mean to say
well let's try some different parameters
the ones we believe are best right now
let's just try some different parameters
drop down a robot with a slightly
different walk see what happens see how
fast it moves
go back and try it again
so
i call this parameter exploration as
opposed to state action exploration
um
and it has some advantages and
disadvantages so the main advantage is
that you get some consistent exploration
behavior
um so you get to try something for a
while like i change its policy maybe i'm
going to try out this strange walk i'm
going to try this walk for a while and
see how that walk does
whereas if you contrast that to epsilon
greedy with epsilon ingredients like you
re-randomize every single step you pick
another action randomly and so you might
end up just doing a random walk
in your state action space which might
not get you anywhere
and so consistency in your exploration
is sometimes helpful we see this
particularly in the robotics field
the main disadvantage of parameter
exploration is it doesn't know about the
state action space so now if my if my
random parameters that i've tried take
me to some state i've been to before
i don't know that i've been to this
state before and i don't even recognize
that maybe there's i've already tried
this action over here and and i already
know something about that we completely
ignore that it's like treated as a black
box and we're just looking at this black
box it's like doing global optimization
in parameter space
so so calling this exploration
you know really what we're doing here is
more like global optimization of our
parameters we're trying to optimize in
parameter space whereas here we're
trying to really understand the state
space trying to understand the actual
space and trying to systematically
explore and figure out the parts of the
state action space we haven't been to
before
and of course there's compromises
between but
on the whole we're going to focus on on
this
first section here for the rest of this
lecture
okay any questions before i move on
good
so we're going to start with the
multi-arm bandit
um so this is the multi-arm bandit
no octopus is required but
it's basically we can think of this as a
simplification of the mdp framework
where we just have
a set of actions a
and a reward function r so we've thrown
away the state space
and we've thrown away the transition
function
and we've really simplified things down
now so all that happens in this
multi-arm bandit is it's called the
multi-arm bandit because of these these
machines which are known as one-armed
bandits
in the u.s
and so you can imagine there's all these
different machines and you kind of get
to choose one of these arms and each of
these different machines has a different
payout some probability that if you pull
the arm you'll win the game and get a
payout but they've all got different
like payouts and maybe one of them's
paying out you know 70 of the time
another one's paying out 75 of the time
and we don't know in advance which one's
going to pay out the most money
um so now which one should we choose
next
you know you might try this one and it
works out quite well now you try this
one works out quite well but should you
go back to this one or should you try
this one again or should you try
something different
there's a whole strategy to your
exploration exploitation trade-off where
you want to ongoing make sure you're
hitting the best machines as often as
possible but whilst exploring to figure
out to make sure you're actually finding
and identifying the best machine
so formally we've got an action set
which tells us all the different arms of
this thing
we've got a reward function which is
just the reward for some given action a
um it tells us basically the probability
of getting any different reward so it's
a distribution over rewards so if i pull
this arm what's the distribution over
the armor the distribution of the reward
that i'll get from this machine what's
the payout i'll get from it
so that's this thing
and there's a different distribution for
each machine
and
at every step we just get to select one
action we get to pull one arm
and the environment generates a reward
from the distribution so we just sample
from that distribution according to the
arm that we pick
and the goal is to maximize the
cumulative reward so we just want to
over time just keep accumulating more
and more award
so it's the simplest case you can think
of this as like a a one-step mdp
um where you kind of just have there's
one state and one step so you basically
pick your action you get a reward and
that's the end of the episode so there's
no kind of um
the only look ahead comes in that we're
trying to understand the exploration
exploitation tradeoff so the look ahead
comes in you know you reach the end of
your you pull this arm you see what
happens and now that affects your own
information that affects what you know
about this environment so you might want
to then keep exploring or keep
exploiting elsewhere
okay that's the multi-arm bandwidth
um
so
so people clear about the setup first of
all
should be fairly straightforward
um
so now we're just going to try and
understand what it means to do well in
this domain so you know we've got this
criterion
where we want to maximize the cumulative
reward but we're just going to scale
that thing and we're going to basically
flip it around to talk about
um opportunity loss known as regret
rather than the amount of reward we've
got how much worse have we done than the
best we could possibly have done
and that's what we call regret
so to understand that let's just work
through a couple of definitions so we're
going to start off by defining the
action value so this is just like our
familiar q but now we've we've thrown
out states there's no states anymore so
we've just got q of a so this is the
expected reward that we'll get if we
pull one of those arms so this is the
true payout of one of those machines how
much it will really pay out if it's
maybe 75 percent for machine one and 80
for machine two
and the optimal value is the
best that we could possibly do if we
knew which which machine paid out the
most we would just always pick that
machine again and again and again so we
basically get the max of all our q
values because we'd always just choose
the machine with the maximum payout
and that's the v style that's the best
we can do in this domain we can just
keep getting v star again and again and
again
so the regret then is how much worse we
do than b star
so v star is the best we could possibly
get we're just getting the best
reward every single step we pick the
best machine
but you know unfortunately we might not
know how to pick the best machine so we
incur some regret we care opportunity
loss in every step the opportunity loss
we incur is the difference between
the maximum we could have got at that
step
and the payout of the machine that we
did pick
so this is the payout of the action we
picked at time step t
so we pick something which pays out
maybe 75
we could have got 80 percent so we
incurred like a 5
regret for that step we didn't pick the
best machine we picked something that
was 30 sub-optimal
um so the regret then the total regret
is just summing these opportunity losses
um over time
so we basically want to sum this up over
time and we want to see well what's the
total regret that we're incurring if we
just keep playing and keep playing
forever
okay
um that's we're gonna
continue for two steps and play this
game for two steps and now we want to
know if we play for t steps how much
opportunity loss do we incur do we how
how much better could we have done if
we'd known the optimal value
if we'd known the optimal arm in advance
how much better could we have done
and so when we say we want to maximize
cumulative reward that's the same as
trying to minimize total regret
okay and the reason it's useful to think
about regret
is just and we'll see shortly it helps
us understand how well an algorithm
could possibly do
like independent of the scaling of the
rewards we can say things about how well
our algorithm could do we want to find
algorithms which
which basically bring the regret
each step down towards zero
so
just
one more slide to understand
like what what's good and what's bad
about regret and then we'll move on to
some pictures to help help understand
this a little bit better
um so
so the count is just the number of times
we've pulled an arm so we're going to
count the number of times we pull an arm
and we're going to consider this thing
called the gap
so the gap is the difference in value
between
some action a
and the optimal action so it's basically
the gap between the best machine i could
have pulled
and some sub-optimal machine so if i'm
wondering you know
if i would just want to know what's the
gap for machine three that's the
difference in value between machine
three and the best machine so some gap
is like the difference between 80 and 75
again there's a gap there of five
percent
um and it turns out that we can think of
regret as a function of all of these
gaps and these counts so if you just
count how many times you use the machine
and you count
also the gaps like you look at the gap
between how much you could have got
and the actual payout for that machine
we see that the regret kind of breaks
down in terms of those two things
so the regret we basically saw that was
the sum
of these differences this is like your
instantaneous opportunity loss the
difference between the optimal value
this is the best you could possibly have
done the payout for the best machine and
the payout
for the machine you actually picked
so the amount that you lose at every
step is the difference between the best
you could have done at that step the
maximum that machine could have given
you and what you actually chose at that
step
and then we just sum those things up and
we see that basically we can just pull
out
the counts now we can say well if we're
just summing up
um
how much we lost each time we picked
this action
then that's the same as counting how
many times we chose this action
and multiplying it by how much we lost
each time we did pick that action
so we just have to count how many times
we pick this action that's our count
here and multiply that by
this gap like how much worse that action
was than usual yeah question but we
don't know whether we started right we
don't know easter
we don't know
we don't know these stuff yeah so we're
going to come back
and so all of this is to say that we can
just rewrite this regret this thing we
really care about optimizing so we want
to minimize our regret we want to get as
close as possible to optimal but what we
see is that the regret can be written in
terms of these counts multiplied by
these gaps so what this tells us is that
every time the gap is large like if
there's some action that you could take
that's really really terrible like
there's some one of those machines is
really horrible and it pays out three
percent of the time
and there's a machine that pays out 90
of the time
well the gap which would be 87 is very
large so you need to make sure that you
pull that arm very few times
whereas if there's another machine which
has a small gap
um you know maybe it pays out 85 instead
of ninety percent the gap is only five
percent now so now you want to
you know you're okay with playing that
one more and more so what you really
want to make sure is that you play um
the arms you want to pick arms the
actions which are best um as often as
possible and you want to pick the
actions which are worst
as infrequently as possible it's
intuitively obvious but this is just
saying in in math
and the problem is that these gaps
aren't known like we don't know these
gaps we can count how many times we pick
an action but we don't know the gaps
because we don't know v star
yeah question um the gaps obviously are
different for each action but they don't
change over time is that right this is
the for the stationary bandit where the
nothing changes over time okay yeah
there's extensions to this where it
can't change upside down expectation
over the numbers of uh
how many times we choose this action or
should we also take the expectation over
the queue as queues like reward and
sometimes
um so q q is defined as the expected
reward for that actually that's the
definition of q so there's already an
expectation in there
um okay
so
so the real question here is
how how can what does this regret look
like over time
and what we'll see is that for most
naive algorithms like if we're going to
consider a few naive algorithms before
we start to
consider better ones what happens is
that the regret so if this is like time
over here and this is the total regret
that we've accumulated you know what
happens if we use one of our familiar
algorithms like epsilon greedy well
what's going to happen is that
every single step in expectation there's
there's some probability that we'll pick
the best action
but there's also some
fixed probability that we'll pick
completely at random
and if we pick completely at random
that's always going to incur some regret
unless we just stumble accidentally upon
the best action
and so if we keep picking randomly keep
picking randomly we're just adding on
regret we're adding on a sort of
constant amount of regret every step in
expectation
so we're randomly picking amongst our
actions
there's some
probability that we'll pick each of
those actions and each of those actions
will incur some of some opportunity loss
and so there's some total
hits that we get for not picking the
best thing and that just keeps getting
added on linearly every single step when
we accept along readily
when we act greedily we also incur this
linear regret as we'll see because we
might lock onto the wrong action
and the question is can we ever get
sublinear regret can we get some regret
which basically we want regret which
kind of essentially
we want something that starts to
asymptote out we want something which
gets less and less regret as we see more
and more stuff we want to basically do
better and better and regret our choices
less and less and less as we go on and
the question is how can we achieve that
is it possible to achieve something your
total regret
and happily the answer is yes
and we'll see we'll see why
let's just start by considering the
myopic cases first of all
um so let's start off with
greedy algorithm
so if we were just to let's start by
considering our usual type of algorithms
that basically do you know monte carlo
learning taking means
so if we assume that we we estimate how
good our machine is we so we're going to
consider each of these arms now because
of each of the arms we want to know how
good is that arm and the natural way to
figure out how good that arm is is just
to estimate um the action value by
taking the mean of all the the out the
payouts we've seen so far so if i've
tried this arm and i got 10 and then i
try this arm again and i got eight now
we think that our q of a for that arm is
is nine
okay
and so that's our way we're going to
estimate the true expectations of the
ones
so it's the normal way think this is the
normal way just like monte carlo
learning this is the normal way that we
estimate values
um
and
and so this is just another way to say
that we're forming the empirical mean
this is just using our indicator we've
seen this notation before so we're just
saying the the action value at time step
t
is just
the average
of all of the occasions this is an
indicator function that picks out all of
the occasions on which we tried that
particular arm
and we're collecting together there so
we're summing the the payouts we got
each time we pick that r and forming the
mean
it's just the mean
okay so the greedy algorithm what does
it do well it selects the action with
the highest value we just pick the
action with the highest value that's the
natural thing to do so we've estimated
how good this action is we've estimated
how good this action is this one's
higher we just pick it
okay and the greedy algorithm doesn't
explore at all it just picks it picks it
picks it picks it um
and the obvious problem is that can lock
onto a sub-optimal action forever like i
might try this action and it looks good
and i thought this action was bad so now
i just keep applying this action forever
or even i try this action and i try
again and i'm just a little bit unlucky
so i end up thinking its value is bad
and then i try this action once and it
looks better and i just keep trying it
and keep trying it keep trying it
so you can lock onto suboptimal actions
forever and as a result greedy has a
linear total regret but in expectation
you can
do the wrong thing forever if you can do
the wrong thing forever that means that
every step forever you're going to incur
the um the loss the regret for taking
the wrong action so you're going to get
that gap again and again and again and
again
okay
so it should be clear that greedy has
linear total regret like you're getting
making the same mistakes repeatedly
forever with some probability
okay
so what happens if we try and be clever
so
the first idea to be clever is to use
optimistic initialization
this is a really well-known algorithm
and actually this is a really good idea
so i don't want to give the impression
this is a bad idea this works quite well
in practice there are lots of
applications where it's quite hard to
beat this
and the idea is just to initialize the
values to their maximum possible so we
start off by assuming the best about all
of our actions
so this is the simplest version we'll
see of optimism in the face of
uncertainty we're not going to measure
the uncertainty we're just going to
assume that everything is really good
and until proven otherwise
so we're going to assume that all of our
actions
pay out the maximum possible so for this
algorithm we need to know the maximum
possible for many of the others but
we'll see we don't
so if we know the maximum possible
then what we do is we just initialize
all our estimates to that maximum
possible and then we act greedily from
that point onwards
now
with optimistic initialization um
you may want to
not erase your optimism completely the
first time you try the action so you may
prefer to
take a non-stationary mean rather than
the full name
but again so we start off think assuming
the best about things until proving
otherwise and then we still act greedily
and so this encourages exploration of
things that we don't know about
but if you're unlucky about
about something a few times
you can still end up initializing
something so i start off thinking this
action's the best possible and then i
try it and i'm unlucky i try again i'm
unlucky and now i can still end up
locking out this action forever because
i might try this other action now turns
out to be better and i never explore
this one again
so you must have some kind of continued
exploration
to guarantee that you you start doing
better otherwise again
you end up in carrying the same regret
every step you can lock something out
and just make the same mistakes again
and again and again
so we want to find algorithms where we
stop when we make fewer mistakes over
time
that's really all this is saying we want
to find algorithms that make fewer
mistakes as you get more experience
yeah but how do we update
the maximum reward we can get from that
because i assume okay we have some
several arms and we assign
maximum values for the sound
and then after every
iteration i need to somehow if the value
which i get is slower i need to somehow
update this
update
yeah so
okay so i should have spelt this out
slightly more um so i think let me give
what i consider the the simplest
implementation of this idea which is to
say that not only do we initialize the
values to the maximum possible but we
also initialize the counts
so we say
we believe that
that this arm over here
has
so so let's say let's take our uh
octopus example again where we know that
we're going to get payout between zero
percent and 100
so now we know that rmax is 100 you
might get 100 payout on this particular
machine whatever you put in you might it
might pay out 100
okay so now
we want to
um initialize everything to 100
but we also want to say how confident
are we that it's really at that value so
what you can do is you can say
let's imagine that i'd actually pulled
that
arm 100 times
and
set the value so far
to
100
so we might imagine that it's as if we
played that um
100 times and got 100 payout on every
occasion and now you continue from there
making your monte carlo updates and you
take an empirical mean including those
hundred updates that you've seen so far
so this is equivalent to using a um like
a beta prior which we'll see later
okay so that's the canonical version so
we assume that our max is known we know
the best possible thing we initialize
all our arms to the highest value we
assign some confidence in some crude way
of estimating uncertainty that we know
about that um so we start off by
assuming that
that we don't know that the best thing
it's as if we've tried it a few times
but we haven't tried it forever and now
as you start to receive more and more
data it will overwhelm your initial kind
of prior on what you thought it was
so you just keep updating your means so
so is that clear you can start off by
saying
yes just as if i've tried this arm a
hundred times and i saw the best
possible play out 100 times it's just if
i tried this thing 100 times i saw the
best possible payout 100 times and now
if it really is terrible you have to
play a lot of times to bring it to bring
down the mean and so it encourages you
to keep exploring things until they're
really proven to be
to be sub-optimal
and so although i say it can lock out
the optimal action forever
you have to be quite unlucky so i think
that
the more confidence you assign to your
prior
the more unlucky you have to be to lock
out the optimal action so it's not a
terrible idea to do this
let's consider what we would say is the
most naive algorithm that we talked
about so i started this whole lecture by
saying look what we've done before is
naive which is epsilon greedy
so let's consider epsilon green so
epsilon greedy again we just flip a coin
every time step we pick an action we
pull one of our arms with probability
epsilon we pull a completely random arm
probability one minus epsilon we pick
the one we think is best so far
um
so now
what do we do
well
we can be absolutely certain that we're
going to incur some loss
in expectation we're going to incur
we're going to keep making mistakes over
time because we're still exploring
randomly so every time we explore
randomly we're very much likely to make
some mistake and not pull the best arm
so we keep incurring regret and regret
and regret
we keep making mistakes and so this also
has linear total regret the same would
be true for the softmax i'm not going to
go into the softmax here
but
despite that it turns out if you just do
something very simple which is to decay
your epsilon over time which we've also
talked about before so you have some
probability that you pick a random
action starts off at maybe 50 and you
just decay it slowly over time um to
until it eventually reaches zero percent
now it turns out you can get sublinear
um
regret
so
in particular let's consider the
following schedule
so this is an impossible schedule you
can't do this in practice because it
uses knowledge about v star
which we don't know but let's say
someone told us v-star and we could
measure all of our gaps let's say we
knew the size of all of our gaps which
we don't
if we knew the size of all our gaps then
what we could do is invent this schedule
which says okay every step all we care
about is the size of the smallest gap
like what's the difference between the
best action and the second best action
so if we know the difference between the
best action the second best action we
can use that difference to craft a
schedule which looks something like this
don't worry too much about the form of
this it's just saying you know basically
um
whenever the gaps are
very small we want to
explore
those actions more often
if the gaps are very large we want to
explore them less often that's
intuitively clear
why don't we average over all of them
yes why is it second best
um i think you could come up with many
other schedules which would also satisfy
this property and averaging over them
might well be able to do that but this
is like one simple
probability of schedule when you choose
a branding strategy which means that
your
delta would be expectations of your data
will be average of
sub-optimal
actions
the expectations of d or
um but we're not using the expectation
of
of the gaps here
we're using so one choice would be to
use the expectation of the gaps another
choice is to use the min of the gaps
and to use that min of the gaps to just
pick your schedule that just determines
the randomness that you use and then you
flip your coin with that according to
that epsilon choice of epsilon
so it's
a valid choice you could also use the
expectation to come up with a different
schedule
so just think of this as one choice for
the schedule that depends on the exact
individual's
individual gaps
and and i think the main surprise here
is that even this very naive approach
actually
it has logarithmic asymptotic total
regret so if we if we knew these gaps we
could so what all this is saying is that
actually epsilon greedy has the amazing
property that if you decay epsilon
according to the right properties you
essentially achieve the best results
that you can get with bandits
give or take a constant factor and a
term or two
and the only problem is that we don't
know in advance what that schedule
should be
but you know maybe it wasn't so naive at
all after all what we were doing you
know maybe epsilon greedy isn't quite
quite as crazy and i'll show you some
empirical results that kind of back that
up
but what we really were after now is
some kind of algorithm which achieves
the same kind of sublinear regret as as
this idealized epsilon greedy but
without knowing
the the rewards in advance
without knowing vstar and the gaps and
all these things so without advanced
knowledge
so that's one way to understand these
um
these approaches so if we just back up
to the picture you know really we're
just after algorithms that have this
kind of shape and we've seen that
decaying epsilon greedy has this shape
um we make fewer and fewer mistakes over
time as you decay your epsilon
but you need to know some special
knowledge about
um about problem to be able to get the
right schedule here otherwise it ends up
looking like this again
and so how do we make sure that we can
achieve that
without telling
our agent thinks it can't possibly know
about the problem in advance
okay
um and so what i'm going to do is
introduce um a particular
approach
which achieves this nice property it's
very well known one of the best known
algorithms for
uh certainly best known outrun for
bandits and very widely used in industry
but really i just want to give one more
slide about the theory which is just to
say that it turns out that
there's actually a lower bound on this
regret
so
in other words
there's something that says
no algorithm can possibly do better
than a certain lower bound there's some
lower bound on how well we can do in
terms of the regret what we want to do
is basically push down our algorithms
closer and closer towards this lower
bound
and the lower bound is actually
logarithmic in the um
in the number of steps
so
what is this lower bound well it just
the performance of any algorithm what
what does it depend on well it depends
on you know what what makes a problem a
bandit hard or easy
so consider you know how similar
the best arm is to the other arms
you know if the
if the distribution so what makes a hard
problem is basically a problem where
you've got two arms which look similar
they've got kind of overlapping
distributions
you're not quite sure it's not obvious
which one is the best
if
so what's an easy problem an easy
problem is one way you've got one arm
that's obviously good one arm that's
obviously bad you just try this once and
it gives you a good answer you try this
once it gives you a bad answer you're
done it's easy okay a hard problem is
something where
you've actually still got one arm that's
much better than the other but there's a
lot of noise on these things so
sometimes you take this one and it ends
up looking really terrible sometimes you
take this one ends up looking really
good and so now it's really hard to
disambiguate them and you're making a
lot of mistakes and it takes you a
really long time to figure out that this
arm is actually much better than this
one
so the hardest problems have basically
similar looking arms with different
means
it's hard to tell the difference between
them and so formally the way we
understand that is by the gap between
them like what's the difference between
this arm and this arm
and how similar their distributions are
which we can use the kl divergence
and so all this tells us so
this is one slider theory so the total
regret
is at least logarithmic in the number of
steps
okay so that the maximum so
so this is the total regret that we're
occurring over time this is that plot we
were looking at
and the regret that we incur is at least
logarithmic
in terms of the number of steps that we
we take multiplied by this term which is
basically proportional to the gap so
this basically tells us that you know
the more different the arms are
the more regret will occur and inversely
proportional to the difference in the
distributions
so this is the term that tells us how
hard the problem is hard problems have
similar looking arms that's the kl
divergence between them with different
means that's the gap here
okay so this is like the hardness of the
problem
and so this is something which tells us
about the problem and this is something
which is like fundamental to bandits
into exploration there's this thing
which is logarithmic in the number of
time steps so we want to find algorithms
which have a regret that's logarithmic
rather than linear
okay
so let's get back to principles and then
come back to algorithms again
so this is the main principle that we're
going to use this next section which is
the optimism in the face of uncertainty
principle
so imagine that there are three
different arms
so here's three different arms there's
the blue arm the red arm and the green
arm
okay and what we're showing here
is
a distribution
over
the actual q values there so this is
so imagine this is like our belief so
right now i've tried a few actions maybe
i've tried this green one a lot of times
so i've got quite a good idea of of what
the mean of this green action is i'm
pretty sure that it's it's around in
this range somewhere
and this x-axis tells us
how good we think it is so so the
further
along this axis means um
this is uh this is the basically
the value that we expect to get from our
payouts
and this one has a width something like
this we've tried it quite a few times so
this is our distribution over q we
believe that the mean of this arm is
somewhere around
i can't read these two point something
um whereas this blue one maybe we've
only tried a couple of times and we're
really not sure what the mean is we
think it's somewhere around here but it
could essentially be anything
and the red one's somewhere in between
and the question is which arm should be
picked next
so the optimism in the face of
uncertainty principle says
don't take the one you currently believe
is best that's the green one
take the one which has the most
potential to be best
and in that case this would be the blue
arm and the blue arm has the most
potential to actually have a mean which
is somewhere over here if we look at it
the tail of this distribution has quite
a lot of mass that says hang on a minute
there's quite a good chance that this
blue arm might turn out to have you know
three or four or even more um a mean of
three or four a really high payout so we
should really try that blue one and
narrow down this distribution and the
idea of the optimism principle is that
as you try it
you start to narrow down this
distribution so you might maybe you play
this arm you think that this one's got
the biggest tail you play it and now
maybe it turns out it's really a bad
action so now the distribution will be
narrowed around something that shifted a
bit over this way and the tail will be
brought in and now maybe you don't play
that one again so now maybe you try the
red arm
um
maybe that one also turns out to be bad
that will shrink the tail back in again
you'll end up with a distribution
something like that and now you'll play
the green arm that will narrow it down
again and as you narrow these things
down you start to get more and more sure
about where the best action really is
until you're actually just picking the
one that's actually got the maximum mean
so it's a way to really just keep doing
things pushing down your uncertainty
but at the same time trying the thing
which has the most potential to do well
this is the optimism the face of
uncertainty principle
and so
the difficulty is that so far we've only
talked about ways to estimate the mean
we haven't talked about ways to estimate
this uncertainty here when we've talked
about estimating q values
so we're going to talk about two
different approaches now to
to solving this approach one of which is
frequentist in which we assume nothing
about the distribution and the second
approach is bayesian where we assume
that someone gives us a prior
probability distribution over our q
values
okay and the general idea we're going to
use is something called upper confidence
bounds
so the idea is to say let's come up with
an upper confidence for each action
value so we're basically going to say
i'm not only going to estimate the mean
like the payout i've seen so far maybe
i've seen you know 80 payout for this
particular arm so far and i'm going to
estimate some upper confidence and what
i think that could be like this is like
think of this as the tail of that
distribution we just looked at so we're
estimating
for each of these things we're not only
going to estimate its q value and
estimate its mean we're also going to
estimate some kind of um
addition we're going to add on like some
bonus to this thing which characterizes
how big this tail is
of that distribution we're going to pick
the thing with the highest
bonus the highest sum of q
plus
um something which tells us about this
tail
so we're going to end up picking the
thing which when we sum those two things
together it gives us the maximum value
so we're going to estimate this upper
confidence u so we can think of this as
a high probability upper confidence on
what that value could be so think of
this as something like a 95 confidence
interval um
about
where the mean could actually be so
we're going to say um
you know maybe i'm 95 confident
that this one is going to be within this
range here
and so that gives us our point that we
use that's our u value here now
and compared to this one we might say 95
confidence interval is here
we're going to use that as our new value
think of them as like confidence
intervals high probability confidence
intervals and what our q value could
possibly be
we're going to pick the thing with the
highest upper confidence value
um
so
that basically means we're going to pick
the one where the true action value q a
um
we want to pick something where we're
really confident that the true value is
is less than our upper confidence bound
so we're going to pick an upper
confidence bound this
95 confidence range
and
pick the thing with the highest
confidence family
and now this depends on how many times
we've tried it
so the fewer times we actually try
a particular action the larger our
confidence will be
and the more times we try a particular
action the smaller the confidence will
be
so we add on less and less of a bonus to
things that we've tried on tried more
we've become more and more confident
about what the mean is
and eventually we end up just using the
means when you try something enough if
you tried an action
infinitely many times you just select
that action based on how good it really
is because now we really know for sure
what the what the mean of that action is
up until that point we use some up
confidence on how good we think it might
end up being use the 95 percent interval
and so the algorithm is really simple we
call this ucd we select the action that
maximizes the upper confidence count so
instead of maximizing over hue or
instead of adding on some random thing
we maximize we maximize pick the action
that maximizes q plus this upper
confidence value u
okay
is that clear
some glazed looks
some people clear
okay any questions
help deglaze people
yes the u values go to zero as you try
more and more actions so eventually
um the u values it's just like shrinking
this tail here the u value is
characterizing the size of this tail
so if you try this red guy more and more
it's going to shrink down it's going to
shrink so think of you start off with
some confidence interval that looks like
this this is your u value the u value is
the difference between the mean and this
guy and as you try it more and more
this is going to shrink down you become
more and more confident of where your q
value actually is until eventually that
u value shrinks to zero and you just use
the mean
okay
so you really want to just keep picking
things according to this upper
confidence value and that helps you very
systematically look around your action
space and figure out which
systematically which of those actions is
giving you the best results are you
making some assumption about symmetry of
the distribution here as well okay great
question am i asking anything am i
making any assumptions about the
symmetry distribution um
so for the approach i'm about to show we
make no assumptions about the
distribution and then we'll talk about
ways to make use of assumptions about
the distribution
okay so this is the distribution free
version we're just going to use this is
a fundamental inequality from
probability theory from statistics um
and and so let's just forget
reinforcement learning for a second and
understand what this is saying this is
called huffling's inequality
and it's just a fundamental inequality
that basically tells us
that
if you have some some random variables
so if you're basically sampling these
random variables between zero and one so
we've got these x values and you keep
sampling those x values again and again
and again
and we take an empirical mean
of all of our
samples that we've seen so far so you
just keep seeing this data you keep
seeing this data you take a mean of that
data
and what's the probability
that
the sample mean
is actually less
so what we want to know is
if we take our sample mean that's just
the x bar and we consider
some so what's the difference what's the
probability that we're wrong
by
an amount of u
in our estimate of the of the empirical
mean so what's the probability that the
difference between the empirical mean
and the true mean
um
is
is greater than u right what's the
probability that we're actually making a
mistake in our estimation of this
empirical mean by at least you
okay
um you've just seen a bunch of coin
tosses and you know what's the chance
that the mean of your coin toss is
actually different from the real bias of
your coin
by more than you and it turns out you
can bound this thing
by this
arbitrary seating exponential value and
this is true for any distribution
doesn't matter what the distribution is
doesn't matter if it's symmetric doesn't
matter anything about it this is always
true
in fact it's maybe a little bit weak
because of that it's not making a lot of
assumptions and therefore it's perhaps
quite a weak inequality yeah
so
in the case of the bandit though so this
requires that you have that bounded
rewards
this requires that you have bounded
awards
so if you don't have rewards in the
range zero to one
we just scale them to zero to one
or there's another version of huffling's
inequality that uses arbitrary
range values but the version i'm going
to use is simpler and so the easiest way
to do this is you just you
you just scale your rewards back into
zero one and exactly all this theory
applies but that means you know the
bound you know some range on your you
know our maximum there's enough if
you're rewarded
so there is no mag
so there's no like maximum value
yeah this assumes
for this result it assumes that you've
got a boundary distribution so there is
at least one assumption it's true for
any boundary distribution
thank you
um
so
we will apply huffling's inequality to
the bandit case so all we're doing here
is we're applying exactly the same
inequality here and now we see this is
basically saying what's the diff what's
the probability that i'm wrong in my
estimate of these q values
by more than
my u value
and what we're going to do is we're
going to use this to solve for the u
value to set this up a confidence value
to the appropriate amount so we want to
know where should i place my upper
confidence thing to guarantee
that this probability is say within the
95 interval
so this gives us a way to compute like
these 95
confidence intervals because we know
that the probability that i make a
mistake
of more than this u value is bounded
by this value so if we plug in
this thing to be
five percent then we'll solve for our
upper confidence value
in a very general way
so that's what we're going to do now
so we're going to pick a probability so
this is like picking our 95 interval
we're going to pick for that p value
we're going to solve for our u
and so what we do is we just set this
thing
set the right hand side of our hufting
inequality this is the probability that
we make that mistake
we set that to some value like our five
percent interval and we solve for u so
to solve for this we just take logs
which gives us this thing
we divide through by
minus 2n and we take a square root
this is squared
and that gives us
this upper confidence value so it seems
rather arbitrary
but
what's nice about it is we don't need to
know anything about the gaps we don't
need to know anything about the rewards
except that they're bounded
and now we have a way to pick actions
we just maximize we add on this term
here and this term has all the
properties we wanted
you can see in the denominator that the
count here is in the denominator that
means that as we pick things more and
more and more this this bonus term is
going to get pushed down towards zero
and for actions that we haven't tried
very often they're going to have a um
a very large
bonus term
so the less we've tried something the
more uncertain we are and the more we
add on to this bonus it's our interval
and now what we do is we pick in
something like a schedule but what we
actually want to do
is to make sure that we actually
guarantee that we pick the optimal
action as as we continue we want to
really get these asymptotic regret
things to um to be logarithmic so the
second thing we do is we add a schedule
to our p value so we don't fix it to
like 95
instead what we do is we slowly increase
this thing over time
to be more and more confident that we've
included the true q value in um in our
interval
so what we want to make sure is that we
are guaranteeing that we select
the optimal action as t goes to infinity
okay so
here's the algorithm and if you like if
the theory was kind of um
you know uninteresting to you you can
just use this algorithm you can just
view this as an empirical fact here's an
algorithm i can use and this will work
and it works very well in practice
um so this is the ucp1 algorithm there
are many extensions hence the one
to be followed up in all kinds of
different approaches
but the basic idea is that every step
you estimate your q values by taking
this monte carlo estimate
in the usual way so you just take the
empirical mean of everything you've seen
so far
and then you add this bonus term on that
only depends on the number of time steps
the number of total pulls of all of your
arms and the number of times you've
pulled this on that's all it depends on
and you add on this bonus that depends
on those two things
and you pick the action with the highest
total value
um
and
this thing actually then achieves this
logarithmic asymptotic total regret
so it looks a lot like the lower bound
except we've lost one of the two terms
we don't have the kl term in there but
we do have the logarithmic it's
logarithmic in terms of the number of
steps
it takes account of the action gap
but it doesn't know about the
distribution so it doesn't have this
kl10 in there
um so this is saying we want to take the
arg max of
all of this
depends on your operator precedence
right
okay so it's an algorithm how does it do
in practice
so this is comparing ucb and epsilon
greedy on a 10 armed bandit
and this 10 arm bandit has particular
characteristic on on the arms and
this is looking at different types of
distribution so this is like 10 machines
which have parameters where one of them
pays out 90 of the time and all the
others pay out 60 of the time
and here we've got
um
one of them pays out 90 of the time
three of them pay out eighty percent of
the time three of them seventy percent
of the time um and so forth
so it's like different types of uh
of distribution and the question is how
well do these algorithms do
um and so
what this is comparing is these ucb
algorithms with epsilon greedy with a
decaying schedule um but instead of
trying to know the schedule in advance
it's like trying to you know guess what
the right schedule is and just decaying
according to some guest schedule instead
and so
the surprise is again that epson greedy
does well okay so epson greedy if you
tune it just right it actually does
really well the difficulty is that if
you get it wrong it's a disaster
so whereas ucb without any knowledge
about the
problem actually systematically performs
really quite well
in these problems
so it might be difficult to pick out but
yeah in each case the ucb algorithm is
doing really very well um
so
and what we're looking at is the total
regret here on the right hand side this
is the total regret over time and you
can see that if you use like the wrong
epsilon value
things just blow up
and
but the ucb algorithms do really quite
well
and on the left hand side we're seeing i
think how many times you pick the
the best action
and so what you see is that eventually
all of these algorithms start to
converge on picking the best action 100
of the time
which you don't get with
a naive epson greedy or some other naive
algorithms
okay
so this is the the bandit algorithm
um
so
these ideas can be continued so this
hufting inequality you can think of as
an example of a general kind of
procedure for generating algorithms
people have subsequently plugged in many
other
inequalities generated tighter bounds
different bounds for different types of
distributions different knowledge you
can plug in
it goes on and on and on
so there's a whole program of research
it's been one of the most explored areas
of machine learning in the last few
years
so i mentioned earlier that i was going
to talk about two approaches a
frequentist approach which we've seen
that makes minimal assumptions about
distribution but we can also consider a
bayesian approach to the bandit
idea the upper confidence idea
and what we do now is we can exploit
prior knowledge about the rewards so
what if someone gives us a prior
distribution over what our key values
are can we make use of that like if we
know that
we've got some prior distribution where
i'm pretty sure that this algorithm this
um action is better than this one
you know can we make use of that
information
and so the bayesian bandit does that by
starting off with some distribution
over the action value function
that's parameterized so we have a
parametrized distribution so we have
some distribution over our q values
parameterized by parameter vector w
and what the bayesian approach does um
is it starts to you know with experience
update these w's so an example of the
w's would be the the means and the
variances
of
each of our
our arms the key values for each of our
arms
so
an example would be
let's parametrize
our uncertainty over q
by estimating the mean of q and the
variance of q
for this action and for this action and
for this action so you'd have six
parameters then describing everything we
believe about those q values we've got
the means which is what we normally
track but also the variances
so the bayesian approach literally
computes a posterior distribution over
what these things look like after the
data that we've seen so far then it uses
that information to make our decisions
so
the idea of the bayesian approach is we
compute a posterior distribution over
our parameters the means and the
variances or whatever those parameters
are given the pulls that we've seen so
far
and then use that posterior to guide
exploration
and so there's
a couple of ways we can do that i'll
first of all talk about the confidence
bounds but also probability matching
um and the main point of this whole
approach is that if we have prior
knowledge that's accurate like if if
someone actually gave us prior knowledge
that was useful then this can do a lot
better like if we know that these just
that these things are a gaussian then we
can do much better but prior knowledge
is wrong you're probably better off
using the ucb approach we just saw which
is
robust to different distribution
assumptions
okay
so
how can we use this bayesian idea to
compute upper confidence bounds
well first of all we compute our
posterior we update our parameters given
the data that we've seen so far so
we update both the means in the normal
way but also the
variance parameter
and then what we do
is
and we can do that basically by using
bayes law so the probability of the q
value is conditioned on everything we've
seen so far
um it's just we multiply this is just
bayes law
and
and then what we do is we just
want to compute again something which
characterizes the tail of this
distribution we want to get the
equivalent of our 95 confidence um
band again so what we do is we just add
on
the u value that we add on now it's just
some multiplier some number of
of standard
posterior distribution so we look at the
width now we're not sure what this mean
really looks like we characterize the
width by some
variance and then we say okay i'm going
to take the mean plus three standard
deviations i'm going to use that value
and i'm going to compare that to the red
guy i'm going to take the mean plus 3
standard deviations and then we see that
the blue guy
has a higher
combined value than the red guy so we
picked that one that's the idea of the
baiting approach to upper confidence
bounds
um so it's really the same idea as the
ucb algorithm that we just saw but
using
prior information and updating our
posteriors more explicitly we're
explicitly estimating this distribution
whereas before we weren't even tracking
the distribution at all and we're just
using a fact about distributions in
general to give us our upper confidence
bound
okay um there's a second way to make use
of our probability distribution so if we
if we've computed all of these posterior
distributions over our action values so
we've got the blue guy and the red guy
and the green guy there's another way to
make use of this information and this is
also true for any bayesian
method
and the idea is to do something called
probability matching so this is instead
of the upper confidence bound idea you
can do something called probability
matching
and so the idea is to select an action
according to the probability that that
action is actually the best one
so
if i've got two actions
and
i kind of can compute that you know
according to my belief so far maybe
there's a 30 chance that this action is
the best and there's a 70 chance that
this action is the best
so then i just sample those actions in
proportion to my uh belief that they're
the best one so then i actually picked
this one 70 of the time and this one 30
of the time
so that's the idea of probability
matching it's a heuristic it's a
heuristic that guides us to picking um
the action most which has the chance of
being the best
the highest chance of being the best
so in other words our policy the way in
which we actually pick actions is just
the probability
that our q value for that action is
actually the best q value
so we want to pick actions in
proportions the probability they're
actually the best one
and so what's nice about probability
matching is it automatically does this
optimism in the face of uncertainty idea
in other words if we've got some
uncertainty over our over our q values
and then we kind of
just
if we just
we just probability match we
automatically pick things which are um
which have higher uncertainty like the
more uncertainty there is in something
the more chance that thing actually
could turn out to be the max in other
words if we look at these distributions
again you know what's the probability
that blue is actually going to end up
being the max
it's fairly high there's a good chance
the blue will actually end up being the
max because it has this big tail there
because there's a relatively small
chance
that if you had something you know over
here
with a tight distribution there's a very
small chance that this thing could end
up being the max and so we would never
pick it
so we pick things in proportion to the
probability that they could be the max
and that encourages us again to be
optimistic in the face of uncertainty
okay
so how do we do this in practice um well
there's a very simple way to do this and
it's called thompson sampling and this
is actually the oldest algorithm for
bandits it comes from the 1930s
or bandits were even really kind of
formally studied
and
and yet it's turned out surprisingly to
have come around full circle to the
point where this now is known to be
actually um asymptotically optimal so
this is the first algorithm we've seen
that actually achieves that lower bound
that we've seen for a particular class
of algorithms like this um
at least the bernoulli bandit which is a
special case so this idea actually
works optimally well in terms of the
expected
the total regret that you care
so the idea is just to
do probability matching in a sample
based way so every step you just sample
your posterior so you pick a sample from
your posterior
so what you do is if these were your
distributions you just sample from them
you randomly sample from your own
distribution for this guy you say okay
well let's just randomly sum it from
this gaussian maybe i think that the
value is now here
um i randomly sample from this guy
maybe maybe i end up thinking well it's
it's here and then i randomly sampled
from this guy um and i end up picking
something which is say here
and now all i do is i just look at my
samples and according to the samples
i've got one here one here one here i
just picked the one which was best
out of my samples and i go with that
action
so it's almost like the simplest way you
could think of to use this information
and yet it's it's actually
asymptotically optimal it achieves that
lower bound that we saw with the
bernoulli bandit case
um and it also has this yeah nice
property that automatically shapes
itself it's not like the
the previous approach we had to pick how
many standard deviations to consider we
had this free parameter how many
standard deviations should i use with
probability matching and thomson
sampling you don't have that parameter
it just comes out in the wash everything
just works sort of parameter free
i mean implicitly there's parameters in
the prior distribution that you use but
once you've got your prior there's some
it's parameter free
okay so that's thompson sampling
basically and this is a very general
idea so you can think of this think of
this in general you can have any
distribution you could have some shared
parameters characterizing the
distribution of your robot
having some action value function and
you just sample from the parameters
characterizing your distribution
and once you've got your samples you
just pick the action which
achieves the best result in those
samples according to those samples
right
let's um
let's
just um see how we do it
okay let me just pause that take
questions and then we'll move on to the
next section
questions
yes how does the situation change if you
know you have only a certain number of
experiments
okay great question how's the situation
change if you've got a limited budget
so
so it really changes the bandit problem
to impose a budget
actually some of these algorithms do not
do the right thing they assume that you
have an infinite budget you keep going
and going and going
um
the next family of algorithms we we're
going to look at deals with that
correctly
so
so i would say
yeah just wait maybe and we'll see
something
any other questions yeah
is this
in general good so it's optimal for
veneering but is it
good
yeah there's a lot of experimental
evidence recently showing that it
actually across a lot of different
bandit
types it
performs at least as well as ucb lag
methods
um
i think there's a question over bayesian
bandits in general
so bayesian bandits are only as good as
the prior knowledge you put into them so
if your prior knowledge is
if you have some magical source of prior
knowledge great if you don't have a
magical source of prior knowledge
you know it's not so clear that you want
to um
to take a
um a bayesian approach at least in terms
of the prior you put in i mean what's
nice about this is okay this lower bound
starts off by assuming like a flat prior
so if you just use a flat prior for the
bernoulli bandit then you can make
progress
so
so yeah i mean i think it's an
encouraging approach and it's been quite
widely explored at the moment
there are some issues with thomson
sampling in the full mdp case that it
doesn't necessarily deal with sequential
information very well
because you're randomly picking at each
time step you lose the kind of
consistency of exploration again
okay let's move on
so
so far we've seen two of our three
classes of approach we've seen
randomized exploration algorithms that
kind of randomly like epsilon greedy
randomly explore
we've seen upper confidence algorithms
optimism in the face of uncertainty
and now we're going to consider our
third class which is
state space
information state space algorithms
to understand information states let's
start by talking about the value of
information
so let's think about exploration why is
exploration useful
exploration is useful because it
actually gains information
like if you weren't gaining information
if you just tried some action but then
you weren't told how much reward you got
from taking that that action there would
be no point to explore you wouldn't be
able to learn from it
so so inspiration is useful because we
gain information so if we can quantify
the value of that information we've got
we can trade it off perfectly like if we
know that you know if i've got
two rooms one of them i know what's
inside that room another room i don't
know what's inside that room
if i can quantify how much long-term
reward i would gain
by exploring the unknown room how much
is that worth to me in terms of units of
future award if i can quantify that i
can make the perfect decision about
whether to go into that room or not
so the value of information is trying to
quantify the value in terms of units of
reward of actually taking an exploratory
action
so we can think of this as you know
how much if i'm an agent and i'm making
decisions how much am i prepared to pay
um to make
some
to take some action that i currently
believe is sub-optimal
so i know that i can get you know 100
pounds by by pulling this lever
uh but i'm not sure how much i'll gain
by pressing this other lever over here
um i think right now it's worth about 70
pounds to me
so can i quantify
how much it's worth to me to pull this
lever in terms of my future payout
and the amount the value of information
depends on all kinds of things one of
which is the budget one of which is you
know how many more times will i be able
to play this thing if i'm only able to
play this thing you know
three more times it's probably the value
of that information is less because i'd
rather just take the hundred pounds
because you know i'm not even if i
figure this thing out i'm only going to
be able to play it a couple more times
whereas if i
know that the budget is going to
continue for the next thousand steps
then i'm more tempted to try the value
of information is greater
so we can think of this as the long-term
reward after getting information
take away the immediate reward so the
difference between how much we gain by
having this piece of information
compared to just the immediate
gain of
getting the 70 pounds or whatever from
from taking this action
so information gain is higher in
uncertain situations so if you know
everything about a situation there's
nothing to be gained by acquiring more
information we already know exactly the
right thing to do there
so what we want to do is to explore
uncertain situations more
but we want to kind of do this optimally
we really want to figure out what's the
perfect way to balance exploration
exploitation you know everything we've
seen so far in some sense it's heuristic
upper confidence bounds it's heuristic
thompson sampling is a heuristic all
these things are heuristics
that in the more complicated cases those
heuristics start to break down
particularly when we start to look at
full mdps so what's the real best way to
trade off exploration and exploitation
so the way we can do this is to think of
an information state space
so we're going to now transform our
bandit problem back into an mdp into a
sequential decision-making problem
the way we're going to do this is we're
going to think about an information
state
this s-tilde thing here it's going to be
that's what we know so far this is a
summary of all the information we've
accumulated so far so this summary we
can think of as like i've pulled this
lever three times and this lever five
times
that will be an example of an
information state
and now what we're going to do is each
time we actually take an action it's
going to transition us we can have like
an mdp with a transition probability
that transitions us into some new
information state like i know that if i
was in the state where i've pulled this
lever three times in this lever five
times and i pull this lever again then i
know i'll be in an information state
where i've pulled this lever three times
in this lever six times that's my
transitions i've got an mdp now i'm
transitioning from information state to
information
and so we can define an mdp over
information states now so we've
augmented our original problem our
bandit problem into an augmented
information state space we've got this
m-tilde that's our overall mdp and now
we've got this information state space
we've got our normal action space these
are the levers
the arms of my my bandit
we've got our transition matrix this is
the way that we transition from
information state to information state
we've got my normal reward function
we've augmented our action space in our
reward space our original bandit problem
which was a r
augmented it to make an mdp after this
thing that takes us from information
today to information state as we
traverse this
expander as we try different things
this is a very large mdp this tells us
about all the different possible
information states we could be in
as we start to explore this bandit
um so let's consider the simplest case
let's consider bernoulli bandits so
bernoulli bandit is basically like a
coin flip bandit where the reward
function is just you flip a coin
and
with some probability
mu a
if i
so for action a and the left lever
there's some bias to that coin that
tells me if i'm either going to get a
reward of one or zero
so probability mu a i'll get a reward of
one so just like a coin flip with some
bias or we can think of this our
machines with the octopus again
this is the payout of those machines the
probability that this machine will
actually pay out
when i
pull that off
and
an example of this would be like a
pharmaceutical drug company and you want
to do a test and you want to try someone
on this drug
and
with probability
mu a they get better and with
probability
one minus mua they they don't get better
and that's your your bernoulli bandit
problem and you want to find the
the medication and the drug that's got
the
most chance of succeeding
and so what's the information state here
well the information state is the counts
that's one example of the information
status is the count how many times have
i pulled this arm and and it's failed
and how many times have i pulled this
arm and it succeeded
so how many times did i try this drug on
someone and they got better and how many
times did they not get better i mean if
i could can keep those statistics those
statistics summer summarize everything
i've seen so far
a sufficient statistic of um that's
given all of all the tools that we've
made in this bandit problem
so
just to make that concrete you probably
can't see this at all it's very small
and blurry so i'll read it to you
or at least explain it
so this is our drug example we've got
two different drugs that we're
considering this is the american sense
of drug we're not talking about um
heroin or something um well you can if
you can imagine it that way if you like
i'm gonna think of it as a
pharmaceutical drug
and there's two different arms you can
pull um that's the two different drugs
so i've got some um some patients and
i'm gonna either offer them drug one or
drug two and i'm gonna start off with
these prior distributions over those
drugs and start off not having any idea
of the effect of drug one so it's got
like this flat prior and these little
diagrams here are like um
probability distributions
over
um the mean that what i think the payoff
will be
for this particular drop so i start off
thinking that the probability density
um is flat
like i don't know what the success
probability of this drug is so i'm just
thinking it's kind of flat there's a
chance that it's gonna
have zero probability and there's a
chance it's gonna have 100 probability
and everything in between we're just
going to say it's kind of flat but the
second one we're going to assume it's
probably around 50 50 but yeah it could
be anything still
and then we're going to proceed from
there those are the priors
so now what happens is we could consider
this whole look ahead tree we could say
well i could pick the left arm if i pick
the left arm i'll be in a situation
where i would update my distributions in
the following way i would update my
distributions to say well
let's say i tried
um drug one and it succeeds
well now i would start to skew this
distribution towards success i would
start to think there's more probability
that it's a good
drug than a bad drug
but if it fails
i would skew that distribution the
opposite way i'd skew it like this now
to say there's some probability that
this is is more likely to be a failing
truck
and so this is one way to just do a look
ahead over this space once we're in this
situation now we can consider again you
know if this succeeded i might look
ahead again and say well now i'd be in
an information state where i know this
has happened so far this is my statistic
my
summary of everything that's happened so
far and i can look ahead again i could
say well
maybe i would try drug one again and
maybe it'll succeed and lead to this
state and maybe it'll fail and lead to
this state if it fails i've now seen you
know one success and one failure so i
think it's kind of around 50 50. that's
that drug now the right hand one i've
never tried so i still think it looks
like this the drug two
um and then i might look ahead again and
say well what happens if it succeeds or
fails and now we see we've got this
whole mdp like a tree search again you
can think this is tree search we think
of it as an mdp if we solve for this mdp
we get the optimal trade-off between
exploration and exploitation
optimal given our priors
okay
so it's not a heuristic anymore we're
really saying you know this is the best
possible way to explore we've looked
ahead we've figured out all the
consequences how much i would learn if i
was to take this decision and we back
that all up to get the answer
so
so to do this we formulated the bandit
as an mdp but now it's an infinite mdp
there's the number of information states
is is infinite and very large
um
but the mdp can nevertheless be solved
by reinforcement learning methods this
is what we've spent the rest of the
course on so we can apply our favorite
methods to this
if you use model 3 reinforcement then
you get a whole family of methods that
can
solve these things maybe slowly with
this original work here but nevertheless
you can arrive at the right answer
and there's a very old well-known
theoretical framework for these bandit
problems which are called the gittins
indices
uh kittens indices you can think of as
the dynamic programming solution um to
this
look-ahead tree problem here that we've
looked at
okay so danny can solve this thing
by dynamic programming we know the
transition probabilities
because we've created uh we we know what
distribution we're using we know that if
this thing succeeds we know how to
update our own distribution so we know
how our information states would
transition
and this whole approach is called bayes
adaptive reinforcement learning so it
basically means you start off
um so
okay now let me clarify that so we can
work with information states and there
are many many different ways to work
with information states
if we characterize our information by a
posterior distribution
then that's what's known as
bayes adaptive reinforcement planning
that you characterize
like in this drug example what would it
look like it would look like
characterizing everything we know about
the problem by a posterior distribution
over
the payouts so the bayes adaptive idea
is that
in each state you summarize all the
information you know about that state by
some distributions over how well your
actions will perform
and then you solve for the mdp
um
where at each state you've got some
distribution um so let's make that
concrete so in the bernoulli case
what happens is that we start off with
some beta prior at the beginning at the
root of this search tree we start off
with some beta prior this is like our
flat we don't know how the drug's going
to work so we start off with maybe you
know these things being like zero counts
or maybe we have some other parameters
for each action
and so for each action a we can have
some
some prior like a beta prior telling
telling us how good we think that
that particular arm is
and then every time an action is
selected we just update the posterior we
update this beta distribution
and the beta distribution has this
really nice property that all you need
to do is to count things
so what happens is if we see a reward of
zero um all we do is we update our
failure account
and that and we'll just adjust our
posterior distribution we increase our
failure count by one and if we succeed
then we update us success count by one
and this just changes the our beta
distribution a little bit
so the states of our like information
state mdp
are in this case
beta distributions so we use the
posterior of our
of how good we think each of these drugs
is as the state of an mdp and we solve
that mvp
so
so just by writing this down just by
writing down the fact that
if we succeed and we'll increase our
success count by one we've defined the
six the
transition dynamics of our mdp
and we solve for that mdp
okay
so this is the bayes adaptive approach
it's a way to
with a lot of computation get the
optimal solution to and to an um the
exploration exploitation dilemma
okay so there's just one more slide
spelling this out where we can start off
with our prior distribution at the roots
here this is just telling us that for
each of our actions we've got some
success counts and some failure counts
and you've got this kind of look up look
ahead tree that basically tells you how
your
distributions are changing how your
success counts are changing
as you move through this tree so each of
these nodes you can look at this
afterwards it's just showing the beta
priors and then the posterior after you
see that
that change
okay
so
the exact solution to this problem is
known can be computed by dynamic
programming it's known as the gettings
index
but it's also possible to find this in a
more tractable way for example
we use monte carlo research to come up
with a very tractable approach to this
very large infinite information states
based search
and that works very well in a lot of
exploration exploitation trade-off
problems so even though it might look
like we've blown up our state space to
something enormous we can then use some
of the
planning methods that we know of that
are very effective in large state spaces
like monte carlo research to actually
still find approximately a very good
solution and so approximate the best
possible trade off to the operation
exploitation dilemma
okay
so let's just summarize where we've got
to so far
so we've been through a lot of ground
already we've covered an awful lot of
things but i think it's you know these
are really important ideas to understand
so we started off with our naive we came
in with our naive approaches we can
explore randomly we can use epsilon
greedy you could use softmax exploration
softbank's exploration is where you just
um
you just take your value function and
you prefer
better values but you still explore
everything you just exponentiate you
select things in proportions how good
the value is basically
if you're on a continuous domain you
could use gaussian noise
these are all examples of random
exploration in in
the state action space where you just
each time you're in a state flip a coin
act randomly
you don't look at your uncertainty these
are myopic methods they don't look ahead
they don't try and figure anything out
they just explore randomly
then we had our optimism principle
optimism in the face of uncertainty
no estimate how
much you know or don't know about your
value and use that to guide you towards
things that not only
are that have the most potential to be
good
so whenever you're not sure about
something try out more because it might
turn out to be a good idea
can anyone think of a problem with that
approach by the way
you know what's what's the problem with
the optimism in the face of uncertainty
idea
if there's a lot of possible options
then you might end up spending a lot of
time doing stuff
right so so if you're in some infinite
action space
um you just keep exploring and exploring
and exploring and you never end up
exploiting because everything's all
there's always more uncertainty to
explore
another problem is if you've got some
real robot and you know there's a cost
to
um
actually you want to avoid catastrophes
what happens if you want to avoid
catastrophes you know this doesn't give
you safe exploration if you actually
talk to some people in industry
they sometimes do the opposite of these
ideas they don't want to systematically
explore the state space they prefer to
explore around the things that they know
are safe and never go too far away from
things that they know are safe you don't
want to crash your helicopter you want
to kind of do things which you're pretty
sure are going to be safe
um anyway it's a very fundamental
principle
that really really helps in cases where
you really want to make sure that you do
explore the whole state space
optimistic initialization the simplest
form where you just initialize your
values high
and assume the best about something
until proven otherwise until you
suppress its value back down to its
realistic value the ucb approach where
we basically use some upper confidence
bound on the value to guide our
exploration and thompson's sampling
where we do probability matching where
we basically pick things in proportion
to their chance of being the best thing
and then finally we had the information
state space idea the kind of
theoretically optimal approach where you
kind of do look ahead in this whole
space of all the decisions you might
make how they affect how much you know
about the problem and then look ahead
and figure out the best path through
this information state space so as to
really figure out which of those leads
you to to the best possible solution
okay
so that's sort of the map of where we've
the methods we've seen so far
um
can take some questions and what i just
wanted to do is to um explain very
briefly how this maps into
mdps and problems where there's where
the state
um
so there's no questions okay good one
question
on the information state space
is the reward signal unchanged or are
you adapting the reward signal
to take into account the information is
it just updating your state definition
the reward function is unchanged we're
not changing the reward function
in the information state space
so we're keeping the reward the reward
in our mdp is the same as the reward
function in our in our bandit all we're
doing is we're adapting our value
function to take account of the fact
that we're not just stopping after one
step we're imagining what happens not
just after this one poll after many
polls
what's the value of of those rewards
over time
i'm taking account of how much
information i acquire along the way
so if i acquire information it might
lead to more rewards in the long term
and there might be a budget of how many
samples you're able to
to test and and this approach can really
factor in your your information state
space mdp might have a transition
function that stops after a horizon of
10 and now you can really optimize for
that horizon and just do the right thing
okay
so
what i'm going to do
is i'm going to explain the contextual
bandit problem
but i'm not going to explain the
solution methods i'll leave those for
further reading in the slides this will
then be
non-examinable so
and then i'm going to move on to very
briefly just touch on how these ideas
extend to mdps
so a contextual bandit basically takes
our multi-arm bandit formulation and
adds in one more ingredient so what
we're doing is we're putting states back
into the picture
so we still have this idea that we pull
an arm and as a result of that arm we're
going to get some payoff so we've got
some action space which is our arms um
we've got some payoff which is rewards
and the canonical example now is going
to be like banner ad placement on the
internet
so a user comes in and we need to show
them some advert and we want to maximize
the probability they actually click on
that advert you know assume that we're
some cynical like company that's trying
to just maximize click through
or alternatively help the user
experience what they want to find
depending on who you talk to
and
and now
we basically want to
figure out how to do this and the key
thing is that we've got some context we
don't just come into this blindly we
don't just show the same things
regardless of who comes in we track some
statistics we're told something about
the user like maybe we're told you know
what continent they're on or or maybe
you know we're told something about the
history of what they've clicked on in
the past or some other statistics about
that user
and so we want to shape the adverts that
we show to users depending on what kind
of user they are
and so that state then becomes this
this s some context information so we're
still we're now shown a state
we're given a state and we need to pick
an action and then we get some reward
and then you're showing another user
you're showing some state you get to
pick an action you get a reward do they
click or not
and so their state now informing what
you do
and so we basically extend our reward
function to
depend on the state as well as the
action
and now at every step
um we're basically
picking this action we're again trying
to maximize the cumulative reward that
we get over many many steps
and this is the contextual bandit
problem
um
and
there's
yeah
i think you can you can just read about
this um and there's an example that's
taken from uh from yahoo this is the
front page news problem where they use
this contextual bandits to select what
news article to recommend to an
individual based on the statistics
they've seen about that individual and
you can just see
from these examples what happens when
you use some very simple contextual
bandit algorithms that gets very very
nice improvements in probability they're
able to actually pick news articles that
are very appropriate to people and make
them much more likely to click on them
by using this kind of um contextual
bandit approach again using an upper
confidence type idea so so in the in the
lecture slides you'll find the upper
confidence bound approach extended to
contexts using linear function
approximation
okay i don't expect you to
understand that right now but feel free
to have a look at that
very briefly um how can we take the
ideas we've seen so far and extend them
to the full case that we care about if
we're really building an agent right you
know we want to do reinforcement
learning we want to drop down our our
agent into atari or our robot into into
the real world or a helicopter and throw
it out there we want to understand how
to
trade off exploration exploitation so as
to find the best solution to whatever
problem we're addressing um whilst
getting as much reward as possible along
the way
so
this is just to say that all of the
ideas that we've seen in previous
sections extend to the mdp case everyone
so you can take up a confidence bounds
for mdps instead of for bandits um so
what would that look like well what we
would do is we would take our q value
our action value we say i'm in some
state now and i'm considering all of
these actions
and if i know some kind of uncertainty
of these q values i might say well let's
add on some some
bonus which encourages me to explore the
actions that i've tried least it
encourages me to explore the actions
that i'm most uncertain about
the optimism in the face of uncertainty
principle extended to empty groups
and so if i know that that idea that
gives me a way to pick amongst actions
so i always pick the one that i'm most
unsure about
um so there's a the first idea of which
which this was an integral estimation
method we basically just have some
method that says my q values i'm pretty
sure they're between 10 and 20 and then
we just pick that top end of that
interval as the value that we use
um you can also use the more
sophisticated methods that we saw with
puffed inequalities whatever
one thing i want to stress about this is
that this idea is not quite perfect in
mdps because this ignores a very
important fact which is it ignores the
fact that when we're an mdp and we're
just evaluating our current policy
that that policy is likely to improve as
we start to if we're doing control in
the mdp we're going to start improving
our policy and the q values are actually
going to get better and better and
better and better
so the uncertainty
correctly should take account not just
of
you know how uncertain our monte carlo
estimate is so far of the current q
value it also needs to take account of
how much more our policy could improve
so our key values could be wrong in two
ways they could be wrong because i
haven't estimated my current policy
haven't evaluated my current policy well
or they could be wrong because there's a
lot of improvements i can still make
so correctly we need to take account of
both of those and that's hard to do
one approach i just want to mention
which is
an optimism in the face of uncertainty
principle which is very popular for mdps
is the following idea it says let's
let's be model based let's construct a
model of the mdb
but then let's imagine that for any
state that i'm not sure about any state
i don't really know what the value is
um any state that i'm you know i'm not
really sure what the reward function is
for this state let's imagine that that
state is like heaven so anything that
you don't know about yet and you haven't
fully figured out pretend it's heaven
now what you do is you run your favorite
planning algorithm what's it going to do
your planning algorithm is going to
figure out how to get to each of these
states that you don't know about
thinks that you're there's some state
you haven't visited yet and it thinks
that state is heaven so it figures out
how to take you to that state you visit
that state you start to figure out
actually that state wasn't heaven it was
actually pretty crappy and then you
don't explore that
um that state again and next time you
solve for your you update your model now
to reduce the
value of that particular state
and then you solve again and it will
carry you to the next heavenly state
until you've suppressed all of the
rewards down to realistic values so it
gives you very systematic exploration
um so the best-known example of that is
the rmax algorithm or e-cubed is another
variant of these ideas many variants
there
um finally
the information state space idea also
applies to um to mdps
so this is the correct approach this is
the idea where we can systematically say
how much information should i gather and
what's the value of that information we
could do that with mdps as well so in
this case what we do is we start off
with our original state space
and we augment that state space to
combine it with information
so basically say we're going to
invent a much bigger more complicated
mvp
in which our new state of those mdps
is the state that i was in my real mdp
so this is like you know where the robot
is in the real world
and also we're going to augment that
state by the information that is
gathered so far so the state of my
augmented mdp is is something like i'm
in a position that's here and i've also
visited all of these things over there
so you kind of remember what you visited
so far that's the i and you also know
where you are and you make that your
augmented state and then you solve for
the augmented state
um
augmented information state mdp
you solve for that using whatever method
you choose
for example we can again use
monte carlo tree search that's a very
effective approach that we tried
and you can get really really if you can
solve this thing it blows up to a very
very large mdp but if you can
approximate the solution to this mdp
you start to get the
optimal uh
trade-off between exploration and
exploitation
um so this actually has proven quite
effective at least in
some
cases of moderate complexity hasn't been
scaled up to really challenging mdps
yeah
okay so
we've looked through these kind of
progressively more complicated
approaches to exploration exploitation
starting with random exploration
bringing in this principle of optimism
in the face of uncertainty and finally
looking at the most systematic case
where we use the information state to
guide
the optimal solution to these
exploration exploitation dilemma we've
also looked at progressively more
complicated settings we started with a
multi-armed bandit we progressed through
contextual bandits very briefly and then
touched on mdps
and so this
space really tells you about you know
what are the solution methods and
problem types that you can combine
together so as to get a handle on the
exploration problem in reinforcement
learning
okay thanks everyone um so i just want
to
notice important notice so the final
lecture um is at a non-standard time um
so i post it to the mailing list and
it's up on my website but the final
lecture will be taking place next
wednesday
not on thursday
i believe it's at 10 am um in
roberts g06
check the website make sure i'm telling
the truth um i'll post again just to you
know confirm that's the case but it's um
not here don't come here um it'll be
closed next thursday that's a ucl
holiday anyway
and again just to stress that
that's going to be non-examinable
material so i know it's outside official
class times uh you won't
find yourself doing worse than the exam
if you can't make it um but it should be
really fun we're just covering games
it's case study
there'll be zero equations very few
equations
okay thanks everyone