DeepMind x UCL RL Lecture Series - Exploration & Control [2/13]

hi and welcome to this second lecture in
this course on reinforcement learning
my name is adolf and hussels and today i
will be talking to you about exploration
and exploitation
now before i going to tell you what i
mean with these terms um i'm quickly
going to flash up a slide with some
background material i highly recommend
you read chapter 2 from the book by rich
sutton and andy barto on reinforcement
burlingame introduction
and in addition if you're particularly
interested in the material that i
discussed today there's some further
reading on this slide which goes into
much more depth which won't be necessary
to understand the rest of this course
but it's a really interesting material
if you want to go into more depth
specifically in the topics of this
lecture
and this is the bandits algo bandit
algorithms book by tora latimore and
javascipiari which just came out last
year
and in addition i put a reference here
to the classic paper by peter auer and
casa bianchi and fisher from from 2002
in which they
describe the ucb algorithm which i'll be
talking about later today
so first to recap in the previous
lecture we talked about
how reinforcement learning relates to
the problem of artificial intelligence
and what we mean with the term
reinforcement learning
and in particular we put up this
interaction loop which we see on the
slide where there's an agent which takes
actions and these agents
sorry these actions somehow might
influence the environment around the
agents you can think of the agent as
being part of the environment think of a
robot in a world walking around where
the walking around might impact the
environment around it or maybe the robot
sometimes picks up something and this
changes where that object then is
and therefore the state of the
environment is somewhat changed
now the agent also observes the
environment it gets observations as an
input and these observations obviously
depend on the environment state but they
might not fully describe the state of
the environment because the state of the
environment might be this huge
complicated thing and the agent only has
a very small
interaction with this the agent might
have a camera with which it perceives
part of the environment but it can't
perceive everything at the same time
and then reinforcement learning can be
thought of as being a science of how
could this agent then learn to make
decisions how could it pick its actions
in such a way that the agent is happy
and we're going to define happy hair in
terms of reward
where there's some sort of a reward
signal that the agent is trying to
optimize this is what defines the goal
of the agent without the reward the
agent is in some sense an aimless
machine it doesn't have a goal but the
reward function defines what it's trying
to optimize
this reward can sometimes be considered
part of the environment where it gets
sent to the agent as part of the
observation in the other cases it's more
natural to define it as part of the
agent where the agent has an internal
preference function that means it likes
certain observations better than others
that's not too important for now we'll
get back to that later but the important
thing is that there is some sort of
reward function that the agent is trying
to optimize
the agent itself might contain a policy
or it must contain a policy because you
have to select these actions in some
manner
it might also contain a value function
which is a prediction of the future
rewards the agent might receive and it
might also contain a model about the
environment we discussed all of this in
the previous lecture
now this general reinforcement turning
problem requires us to take into account
time and consequences
i mean with this that actions can only
impact the future not the past
and sometimes this can happen
in
complicated ways so you might take an
action that changes the sales the
environment in some way that for
instance influences later rewards or
later observations and sometimes you
can't immediately see the effect of the
action or at least not the full
consequences of it but still the agent
in order to pick the right actions
should somehow be able to reason about
all of this
so these actions the decisions you make
which action to pick affect the
immediate reward they might affect the
state of the agent itself and they might
affect the state of the environment in
some way or the other which might itself
then have uh consequences for later
observations and rewards
now this is a challenging setting and
one interesting property that we're
going to talk about a lot in this
lecture in a sense
is that it's an active learning
setting and this means that these
actions they don't just change the
reward right they don't just tell you
how well you're doing but they also
influence the data that you see and this
is going to be important and this is
different from some other types of
machine learning
because normally in other types of
machine learning we might assume that a
data set is given to us and we want to
answer some questions given that data
set we want to learn something from the
data set but the data is given and is
not under our control in reinforcement
learning in the full setting this is
different the data is in some sense
under our control but in some indirect
sense we can take actions to change the
data we can't deliberately pick out
certain data points or sometimes maybe
we can but we can't maybe always do that
and this means that the
agent has to actively seek out
information that it can use to learn
from
now in this lecture we're going to
simplify the setting quite a great deal
and in particular we're going to assume
that the environment actually only has
one single state it could ever be in or
maybe equivalently you can think of the
environment as not having no states
whatsoever
so the environment
all that it then returns to you is
rewards it doesn't return any
observations
i'm going to assume here that the
environment does indeed
return the rewards although again you
could also think of this as being kind
of internal to the agent where maybe the
environment could be thought of as
sending you a signal that tells you
which which reward you can then apply to
it with your preferences your internal
preferences but just think of it as an
environment with a single state
and this has some consequences first of
all
actions can no longer have long-term
consequences in the environment if you
take an action it can't change the state
of the environment because the
environment only has one single state
in addition
the actions can still impact the
immediate reward which is actually not a
function of the
sequence it's a function of
just the immediate action that you take
but the other observations can be
ignored so if you consider the reward
and observation you still want to pay
attention to that because this defines
your goal
but other observations are basically
irrelevant because all they could ever
tell you is something about the
environment state but if the environment
only has one state maybe this can just
be ignored
and then we will discuss how to learn a
policy which is a mapping from your
internal agent states to actions in this
more simple setting it turns out there's
already like a lot of rich things that
we can say in this setting and a lot of
really interesting questions that we can
get into which is why we consider the
simpler setting first
then in subsequent lectures so not today
but in subsequent lectures we will be
talking about the
more general setting again
okay so now i'm going to
just go through a little example
and for that i'm going to switch
to the blackboard
so now
let's consider that we're going to
compare two different
actions
and as mentioned these actions won't
have long-term consequences so basically
we can take an action
it might be action a or it might be
action b
and we're going to compare how well
these do in some sense
for instance you can think of action a
and b you could think of them as being
medical treatments where you maybe have
the choice between say one vaccine and
or a different one
but maybe you can only do one at a time
or they could be more extensive medical
treatments where clearly you can't do
them both at the same time perhaps
and therefore you have to pick one or
the other or maybe action a is doing no
treatment and action b is doing some
treatment
or alternatively if you prefer this as
an example you could think of a
recommender system
where you have a streaming service and
people can watch movies and you have a
choice which movie to recommend next in
the top position
so picking a movie there picking a good
one might mean that people pick it and
then watch it where picking at that one
might mean that nobody clicks it for
every user you don't have this option
whether to pick movie a or movie b
and success would be if the user then
picks to watch that movie
for simplicity let's assume the only two
options here are failure or success so
let's say the rewards are basically plus
one or my uh or let's say plus one or
zero
now
we could have some prior information but
for simplicity let's assume we know
nothing so we kind of just randomly have
to pick between these two options the
very first time we try them so we have
trials here on the x-axis
axis
and at first let's say we pick action a
and we observe a reward of zero this was
not a success
we for instance show the movie and the
user didn't pick it
okay so next user comes around we have
this choice again let's say now because
action a didn't quite work out let's say
we pick action b
and let's say that this one's more
successful we get a plus one
so the question now is what should we do
next well
maybe here it seems sensible given that
we have no additional information
we only have one failure for action a
one success for action b so maybe we'll
pick action b again
now let's say that this time around
it didn't work out that well and it was
not a success the user didn't pick the
the movie or the medical treatment
didn't quite work out as well as we
hoped
so we consider this a reward of zero
again the question pops up next time
around what should we do well the
average value for action b is still a
little bit larger than the average value
for action a so maybe we'll try it again
but let's say we are unlucky again we
pick movie b and again it doesn't get
picked
so the question here is what to do next
when should we switch
back to action a
how long should we persist to action b
what happens if we do switch back to
action a but then we get another zero
maybe then we switch back to action b
again but for how many steps
so these are the questions that are
under consideration i won't give you an
answer right now but we'll discuss
different algorithms which will help you
pick between these two actions and they
will pick maybe in different ways and we
will discuss this at length these
different algorithms that you could
apply to this setting
okay so now let's switch back to the
slides
so
we're basically talking here about the
difference between exploration and
exploitation
and this is something that any learning
agent which actively gathers its data
must trade off
exploitation here refers to maximizing
performance based on your current
knowledge
so as in the example if we've only ever
shown two different movies and one of
them has been more successful than the
other
maybe the reasonable choice if you
really want to get the next one right
would be to pick the one with the
highest success rate so far
but we know in the long run this might
not be this might not be great because
if you exploit all the time you're never
going to get new information about
movies that you haven't tried very often
in the example we only ever picked movie
a once and even though that wasn't a
success maybe we want to pick it more
often just to be sure
how often is it actually a good choice
and this is called exploration and the
main purpose for exploration is to
increase our knowledge by exposing
ourselves to new
information to new data
and this is a fundamental tradeoff that
we need to think about clearly and
carefully and we went to this simpler
setting where the environment only has
one state to be able to reason this
through
basic as far as we can
and then from here
we hope to learn some lessons that we
can apply also to the more general case
so
basically in general we would need to
gather information to make the best
overall decisions
and the best long-term strategy might
involve taking some short-term
sacrifices sometimes you want to try
something that you've never tried before
which you think is maybe not that likely
to be great but you have high
uncertainty about it
if you never try it you will never know
and there might be some brilliant
rewards to be gathered but if you never
try it
um you will never find those rewards
maybe sometimes you may have to make a
short walk through the rain to find a
brilliant new place to go to
so now we're going to move towards
formalizing the problem
so what this setting is sometimes called
is a multi-armed bandit
and this is basically a set of
distributions where we have a reward
distribution per action
why is this called a multi-arm bandit
this actually alludes to slot machines
you can think of a slot machine uh
as being
something where you basically don't have
a lot of choice some on some complicated
slot machines of course you do have a
lot of choices but let's think of a very
simple slot machine where all you have
is a lever and you can pull it and you
either get some reward or you don't or
maybe there's different types of rewards
that you can get there could be some
complicated reward distribution
now we can think of this as being a
single action a single slot machine but
then you could think of having many slot
machines each of which have their own
lever and you could pull them each of
them and each of them might have a
different reward distribution
now we know a priority that typical slot
machines will give you less money than
you put into them but let's assume that
we don't know this we can generalize
this problem so there could be some slot
machines that are quite worthwhile where
others are not
now this is where this term multi-arm
bandwidth comes from because a slot
machine is sometimes called a one-armed
bandit because it takes your money away
and then a multi-arm bandit is a setting
where we don't have just one choice we
don't have just one slot machine with a
single lever but we have a whole row of
these slot machines so there's many
different choices we can make
this is just terminology of course we're
not talking about slot machines here
we're talking about a more generic
setting where this could be a
recommender system or a way to formalize
picking between a host of different
policies
say
for determining medical treatments as i
mentioned or other examples you could
come up with
this curly a is just a set of known
actions or arms in multi-unbanded
terminology
and r curly r of a is then the
distribution for that specific action
and as i mentioned for different actions
the distribution could be different
the goal is now to pick the action that
gives you the highest average reward
so each time step you're going to select
one of these actions and you're going to
observe this reward which might be a
random variable
and the goal is then to maximize the
rewards over time
this is a little bit different from what
we talked about before note that this is
actually taking into account the full
learning process this turns out to be a
useful thing to think about so we want
an algorithm a learning algorithm that
trades off exploration and exploitation
in such a way that if you apply this
algorithm throughout your life that you
get reasonably high reward of course
this reward can't be optimal all of the
time because you don't know what the
optimal action is yet
but this way of writing down the goal
including all the steps that you ever
take allows us to reason about which
learning algorithms are more effective
than others
and we will specifically optimize this
cumulative reward of your lifetime by
learning a policy which in this case is
just some distribution on the set of
actions
we could think about deterministic
policies which always take the same
action but during learning it's also
sometimes useful to have stochastic
policies which pick actions according to
some random distribution and the policy
will change over time as we gather more
data
now one
good
concept to still use would be the notion
of an action value which we also talked
about last lecture which in this case is
quite simple is this is just the
expected reward given that you've taken
that action so the action value for
action a is the expected reward for
action a
now we can also talk about the optimal
value the optimal value that you can
achieve at any time step is simply the
maximization of the action values over
actions this is the maximum expected
reward
another useful concept that we will talk
about extensively in this lecture is the
notion of regret
what is regret
the regret is a specific number for each
action which is basically the difference
between the maximum possible expected
reward you could have gotten
and the one that you did get for this
action
note that this is not a random quantity
we're subtracting one expectation uh
from the maximization of our
expectations so this is just a number
for each action now we don't know this
number that's the whole point if we
would know this number we would simply
select the action which has zero regret
note there's always one action the
optimal action for which this quantity
is zero
by definition
and note also that for all the other
actions this is a positive quantity so
regret enumerates essentially
how bad we're doing the more regret you
get the worse you're doing if your
regret is zero or close to zero you're
doing quite well
this will become a lot more clear when i
talk about specific algorithms and i
talk about the regret that they incur
and then you'll see that this is a
useful way to look at the differences
between different algorithms just to
look how much regret do they attain over
their lifetime
so the regret as i mentioned for the
optimal action is zero and that's going
to be important
and then we can think about minimizing
the total regret as our overall goal
this is a random quantity because it
depends now on the actions that we take
note that you regret the instantaneous
regret for a given action is not a
random quantity but here the actions
themselves are random quantities because
our policy might be random and in fact
will be random because typically our
policy will depend on our history so we
take some action even if we do that
deterministically we get some random
reward and we want to use that reward
somehow to pick the next action which
means that the next action will then be
a random quantity
so this is simply a summation over time
of this instantaneous regret at every
time step for the action that we picked
and then we want to want to talk reason
about minimizing this total summation
over all of our lifetime up to some
time step t
of this of this cumulative regret
note that i didn't actually change the
goal here i mentioned two different
goals minimize total regrets or maximize
cumulative reward these are actually the
same goal there's just a different way
to write the same thing
so note this is true because if we
maximize the expected cumulative reward
this would correspond exactly
to
minimizing these gaps between the
optimal expected value and the one for
the action that you've selected so these
are really the same goal we're just
writing it down differently and this
notion of regret is quite a common one
in multi-armed bandits because it allows
us to reason about how quickly does this
regret grow over time
and turns out this growth rate is very
indicative of how good an algorithm is
so we basically go to algorithms that
for which the regret doesn't grow too
quickly over the full lifetime of
learning
so what follows the rest of the lecture
essentially i will be talking about
different algorithms and also different
analysis that we can do like theoretical
analysis on these different algorithms
to discuss whether like to determine
essentially whether one algorithm is
better than another
okay so let's get
we will discuss
the greedy
policy an epsilon greedy strategy which
i'll explain what that is
the ucb algorithm
function sampling and policy gradients
and i will explain what each of these
means in what follows this is just a
quick overview if you don't know what
these terms mean that's perfectly okay
because i'm going to explain that in a
second
the first three of these approaches use
action value estimates
so we are going to keep track of some
big q of t i use big q there because
it's a random quantity because it
depends on your past observations
and this big q of uh of a at some time
sub t is supposed to approximate the
actual true value of that action a
so little q a here is a true expected
reward given a and big q is an
approximation thereof
so let's very briefly discuss how to
learn as action values which won't be
very surprising so the true action value
we know is the expected reward so one
thing we could do is we could just
average the rewards for that action
that's written on this slide maybe a
little bit verbosely
but we're basically just picking out
with an indicator function
the time steps on which we selected this
action a
this indicator function of a n equals a
will be 1 if its argument is true so
when we've actually selected action a
and it will be zero otherwise
so we're basically just summer overall
summing over all time steps picking out
the rewards that correspond to the
action a and then dividing by the number
of times we've selected action a and
this is called the count
so we're dividing simply by the count
for that action which is just the number
of times you've selected an action
including on time step t
this is just a way to write down the
average so we're simply just tracking
for each action whenever you select that
action you just average in that reward
into your estimate
now written like this you might have to
keep track of all of the rewards and put
them into like different tables of
course you could also do this
incrementally which is written down on
this slide where we simply take our
average and we update it
so this can be written in the way
that it's written on the slide where we
take our action value at time step t
and we define this to be the action
value of the previous time step t minus
one
plus some learning rates alpha or step
size parameter alpha times an error term
and the error term is simply the reward
that you've received given that you've
taken this action
minus the average for that action so far
and all of the other actions we simply
don't update them because we haven't
taken those actions at its this time
step so we we don't update the mean
and then we also increment the count for
the action that we've taken and we
define the step size parameter to be one
over the count then this is just a
different way to write down the same
average we had on the previous slide
in uh later
lectures mostly we will also consider
other step sizes which are sometimes
useful for instance you could think of a
constant step size which would lead to
tracking behavior and this can be useful
when
for instance the problem is
non-stationary in some way
then you might want to track the rewards
rather than to average them flat out
for now this is just uh a way to average
the the reward so that's what i want you
to keep in mind for each of these
actions we have the average reward for
that action so far
okay so now we can dive in and talk
about concrete algorithms the first of
which will be the greedy algorithm which
is a particularly simple algorithm
it simply takes the action with the
highest value as you've estimated it so
far
so equivalently we can write this down a
little bit differently what we often use
is a notation of pi to the policy and
pi of action a is the probability of
selecting that action which in this case
is just an indicator again which checks
that the action is the one that
maximizes the value
the way it's written down here is
assuming no ties are possible because
otherwise
the probabilities don't sum to one so if
multiple actions have the same
maximizing value then obviously you
still have to decide what to do and for
instance you could pick random you could
break ties randomly in that case
okay
now i'm going to go back to the
blackboard again
and this is just to
think about the regret of the greedy
policy
so going back to the example we had
before
we've taken action a once we've got a
reward of zero we've taken action aa by
three times by now we got a reward of
one and then zero and then zero again
now what are the action values at this
point well the action value as estimated
at time step
four
of action a
is simply one uh sorry zero
we've taken it once it wasn't a success
we have zero
the action value at time step four for
action b on the other hand
is one third
we've taken it three times
we got a plus one and a zero and a zero
the greedy policy
would continue to select action b
because the value is positive
whereas the value of zero a is to simply
zero
but note that it will actually continue
to do so indefinitely if the only
possible rewards here as i stated before
are zero or plus one
then the algorithm from this point
onwards would never ever select action a
again
now it could very well be the case that
action a is actually possible the
optimal action in this this case
it could for instance be that the
probability of getting a plus one reward
given action a
could have been something reasonably
large like 0.8
whereas the probability of selecting of
getting a plus one reward when selecting
action b
could be moderately low like 0.2 or
something
that would mean that the
for action a
is zero
you get no regret when selecting action
a
any regret for action b
is 0.6
recall the regret for an action is
simply the maximal attainable expected
reward minus the value
for taking that action
the value for taking an action here is
exactly the same as the probability of
getting plus one because the only two
alternatives are plus one and zero
so the maximum value v star
here in this example as i wrote it on
the side would be
0.8
whereas the value for
b
these are essential values it's just 0.2
this
probability being equal to the value is
just a happenstance of course it's just
because our only options are plus one
and zero
in general um you can just think of the
expected values rather than the
probabilities
so clearly we should be taking if this
is all the case we should be taking
action a but the greedy policy won't
necessarily take action a it could have
happened of course that on the very
first time we took action a we did get a
plus one in which case the greedy policy
might have stuck to action a depending
on the rest of the data and then
everything might have been okay
um
or actually you should say depending
also in your initialization of the
action values because whether we select
action b ever
depends whether we initialize this value
low or high
so this is just to say that the greedy
policy is not great which is probably
something that's quite intuitive as well
so we'll go back to the slides now
okay so
the greedy policy might not be that
great in some cases so can we do better
and turns out we can
and one simple alternative would be
epsilon greedy so the problem with the
greedy policy just to reiterate what we
showed in the example is that it can get
stuck selecting the wrong action
indefinitely there was this one action
which happens to get a higher value than
the other one and then it just keeps on
selecting that action because the value
kept on being positive whereas the other
value estimate was zero even though the
value estimate with the estimate of zero
was actually optimal action
this means that your regret can continue
to grow linearly and every time you get
a similar amount of regret and that
turns out not is not great
so the reason that this happens is
because the greedy policy doesn't
explore enough it keeps on exploiting
the knowledge that it thinks it has but
it doesn't realize or doesn't reason
about uncertainty about the value
estimates there was this one action
which we only selected a single time so
we should have some very high
uncertainty about having a valid
accurate
estimate of its value
so an alternative is to add some noise
that keeps on exploring and this is what
epsilon greedy does and this is a very
popular algorithm actually
and the way this works is that with
probability one minus epsilon you select
the greedy action
similar to greedy but with probability
epsilon you select a random action any
action including potentially degree one
but with just random probability
we can write this out equivalently as a
closed form
policy so the probability of selecting
an action and depends on whether the
action is a greedy one
again here i'm assuming that there are
no ties to be broken otherwise you have
to carefully correct for that
but if there are no ties possible then
the greedy action will be selected with
probability 1 minus epsilon plus a
little bit of probability it gets
selected when you pick the random action
and then all other actions are selected
with epsilon divided by the number of
actions because we're picking randomly
so epsilon greedy continues to explore
with this probability epsilon but that
means that it also has linear expected
regret
because the epsilon doesn't decrease so
you keep on picking actions even if at
some point you should be quite certain
that these actions are really actually
quite bad
so you've attained in some sense enough
information about them
and you should stop really stop
selecting them episode greedy doesn't do
that it keeps on selecting these actions
that said it is a very popular algorithm
in the first lecture i showed you an
example of an agent playing atari games
this agent was actually trained with
epsilon greedy exploration
now moving yet to another example of a
way to
update our policies we're going to
sidestep
action values for a second and we're
going to ask the question can we learn
this policy directly instead of learning
values
well it turns out this is possible
and before going there i'm going to make
this a bit more concrete let's consider
that we have some action preferences and
let's also pick a policy in this case
we're going to pick a differential
policy one that we can take a gradient
off so we're not going to pick epsilon
greedy instead we're going to pick this
alternative which is a softmax
and the way a softmax policy works this
is another very popular policy is that
you exponentiate
these preferences and then you just
normalize so we see an exponentiated
preference for action a divided by the
sum of all of the exponentiated
preferences for the other actions
by exponentiating we get a positive
number so we know that all of these
probabilities
must be
positive and we know because we're
normalizing with the sum that they must
sum to one so this is a well-defined
policy on these actions
and then we just select an action
according to this policy
note that the preferences themselves are
not supposed to be action values per se
right we could plug in action values
into a softmix policy for sure but now
we're considering them just to be
learnable policy parameters and we're
going to ask the question how could we
update these preferences in such way
that the policy gets better over time
so the goal is to optimize the
preferences so that the policy becomes
very good and the question is how can we
do that and i'm going to give you one
such algorithm to do this
and the idea would be to update policy
parameters such as the expected value
increases and we can just write this
down and we can do this via gradient
ascent
so essentially what we're going to do is
we're going to consider the policy
parameters these preferences
to be something that we can just learn
and we're going to try to learn them
with gradients
so in the banded case the update would
look something like this we have
theta which are our parameters of the
policy
for instance you can think of theta as
being a vector that just contains these
action preferences for each of the
actions
more generally this can be applied to
other policy parametrizations and indeed
it turns out this algorithm is quite
easy to apply
at scale in what we call deep
reinforcement learning because for
instance theta could be the parameters
of some large deep neural network
so the algorithm here is quite easy to
extend which is one reason why it's of
interest but for simplicity now you can
consider consider theta just to be a
vector containing these action
preferences that we talked about on the
previous slide
and then what we want to do is we want
to do gradient ascent on the value of
the policy
what is the value of the policy the
value of the policy is the expected
reward given that you're following this
policy
note that the softmax policy is a
stochastic policy
so this means that it might select
different actions under different uh
probabilities so don't mistake this for
the expected reward for a given action
now the policy might be stochastic so
it's really awaited
some of those
now one problem is here that we have a
gradient of an expectation but we don't
have that expectation we don't know that
expectation typically
and that means that we can't easily
compute this gradient immediately but
turns out there's a useful trick that
we'll discuss which allows you to get a
stochastic sample for this
note also that we can't directly just
sample the expectation and then take the
gradient because the sample for this
expectation would just be a reward and
we don't know how to take the gradient
of that reward with respect to the
policy starity so we need to do a little
bit of work to make this into something
that we can sample so then we can do
stochastic gradient descent rather than
gradients ascend per se
so how can we compute this gradient
that's the question that we're getting
towards
so this is a derivation on the slide
that turns that gradient into something
we can sample and i'll step through this
carefully
and this is sometimes called the log
likelihood trick
and sometimes it's also referred to in
reinforcement literature as the
reinforced trick because
this was introduced into reinforcement
learning by ron williams in an algorithm
he calls reinforce which is actually an
acronym but i won't bother you with the
exact
uh
semantics of the acronym in this case
so
we start here
on the left hand side with the gradient
of the expectation of the reward given
that policy and what we're going to do
first is we're just going to write that
out so we're going to expand this
expected reward given the policy into
the summation over actions
and then we have the probability of
selecting each of the actions times the
expected reward given that we've taken
specifically that action and we know
that to be
our action value qa
now we know that this qa does not
actually depend on our policy parameters
because we've already pinned down the
action by now right so given action a
qa is just some number and it doesn't
depend on our policy anymore that means
that we can push the gradients inside
the summation now and it will only
affect the policy not the q values
now we're going to do a little trick
which is quite useful in general we're
going to multiply this whole quantity
with something that is basically one
it's the possible probability of
selecting the action divided by that
same probability
and the reason to do this is because
then we can write it as an expectation
again so we're going to pull this
probability of deselecting the action a
up in front again and let's push back
the division by that same probability to
the back just for notational simplicity
and then we know that we have something
that is of the form of an expectation
again we have a summation over actions
of the probability of selecting an
action and then some term that depends
on the action somehow
so let's write this back as an
expectation where this can be written as
the expectation of the random reward so
we replace this expectation of the
uh the reward with the random reward
inside the expectation again
and this gets multiplied by this kind of
weird looking ratio which is the
gradient of
the probability of selecting action a t
divided by that same probability
and now as our final step which is
actually somewhat optional but it's just
a different way to write down the same
thing and actually the much more common
way to write down the same thing
is that this whole term turns out to be
equivalent to the expectation of the
reward times the gradient of the
logarithm of your policy
this is true simply because of the chain
rule because you might recall that the
gradient of the logarithm of something
is one over that thing but then we have
to apply the chain rule so if we're
considering something that has the form
of the of the gradient of the logarithm
of some function of say x
then the gradient with respect to x of
that thing is one over the function of x
times the gradient of the function of x
so that that same thing happens here so
this this
quotient here turns out to be equal to
the gradient of the logarithm of the
probability of selecting action a
this is just equivalent right this is
just writing the same thing down
differently
so now the important thing is that we've
arrived here on the right hand side to
something that has an expectation on the
outside
and then some term on the inside which
has the gradient but that means that we
now have something that we can sample we
can just get rid of the expectation in
some sense in our updates by just
sampling the thing inside and then we
would be following the stochastic
gradient rather than the true gradients
but we know from practice that
stochastic gradient descent and
stochastic gradient ascent are quite
useful algorithms and they work quite
well
so that seems to be fine
so this is just to condense this
derivation from the previous slide into
the main uh output which is that we've
now been able to rewrite the gradient of
the expectation of the reward under the
policy parameters as the expectation of
the reward
times the gradient of the logarithm of
the policy
of taking that action 18.
so we can sample this and turn this into
a stochastic grainy descent algorithm
which means we're going to update our
policy parameters theta
again this can be thought of as a vector
containing your reference for action
preferences for instance and we're going
to add a learning rate times the reward
times the gradient of the logarithm of
the probability of selecting an action
note we're adding not subtracting
because we're doing gradient ascent
rather than gradient descent
instead of subtracting the gradient of a
loss we're adding the gradient of a
value in this case
and then we can just execute this over
and over again and then we would be
performing
stochastic gradient ascent
on our
values which means the policy should be
getting better and better over time
and we can use the sampled rewards we
don't need to make any value estimates
so this is a value-free algorithm
now we can extend this and this has been
extended to also include values and
we'll talk about that in much more
length in later lectures
now one more thing that i want to say
before we go to other algorithms first
we're going to
derive this concretely to give more of
an intuition and then i want to add one
component so
so let's consider the softmax policy
that we talked about before where h is
just some preferences for an
action and to build our intuition for
what this policy gradient algorithm is
doing let's just step through that
example where we specifically
parameterize our policy using these
action preferences
what does this update that mean well we
can consider the preference for what
just one of these actions so instead of
considering this whole vector theta
we're only going to consider one
component in that vector
for action a
how does this get updated
well we add as mentioned before some
step size alpha times the reward that
you've seen times the gradient of the
selected action a t
whether or not this is action a or not
and then turns out this gradient here
this logarithm of the probability of
selecting a t
by the preference of action a
turns out to be this quantity below
which is
one minus the probability of selecting
the action if the action is equal to the
one that you've selected and otherwise
it's just minus the probability so
writing that out in case by case basis
means that the preference for action a t
the one that you selected
gets updated by adding
learning rates times reward times one
minus probability of selecting the
action
whereas the probabil the preferences for
all the other actions get updated by
subtracting a learning rate times the
reward times the probability of
selecting that action
i encourage you to step through this
more
detailed uh basically just take the
softmax distribution plug that in into
pi here and see whether you can derive
these results on the slides as an
informative exercise to go through
okay so now that we know these updates
can we interpret them somehow well let's
plug in a specific value let's say we
saw a reward of plus one
what now happens to these preferences
well the action that we selected will
get updated
to be higher the preference for the
action that we get we selected gets
updated to be higher
because
the step size here is a positive
quantity the reward is plus one as i
just mentioned and this one minus the
probability of selecting the action is
also the positive quantity so the
preference for selecting action a t will
increase if the reward is positive if
the reward is say plus one
at the same time the preference for all
the other actions will go down a little
bit
and note also that how much they go down
depends on how likely they were to be
selected actions that were very unlikely
to be selected don't go down that much
actions that were very likely to be
selected go down more
because we saw a different action that
was quite good at this point
how much do they go up or down that
depends on your step size parameter
so
what then happens is that the preference
increased for the action that had a plus
one and if say you sometimes get a
reward of minus one exactly the opposite
happens if you get a reward of minus one
the preference for the action that
you've selected will go down and the
preference for all the other actions
will go up
but what now if there are no negative
rewards what if the only possible
possible rewards are plus one and plus
two let's say
then always whenever you select an
action its preference will go up
but turns out everything still works out
because whenever you select a different
action its preference will go up faster
because the reward will be two rather
than one which means the preference
increases faster
and indeed it turns out we're still
doing valid stochastic gradient ascent
in that case so the intuition with minus
one plus one reward that one's a little
bit more intuitive when you get a plus
one your preference increases when you
get a minus one it decreases
but if you have
only positive rewards let's say plus one
and plus two then always when you take
an action the preference will increase
but the actions with the higher average
reward will increase faster which means
that
over the long run we're still following
a valid gradient the policy still
converges to the right policy
okay
so the intuition is as as uh summarized
here the preferences for actions with
higher rewards increase more or decrease
less making them more likely to be
selected again and that makes it a valid
algorithm to learn policies
the exploration here is not very
explicit though the expiration is purely
due to the fact that the policy happens
to be stochastic
this does work often if your policy is
stochastic enough for a long enough
amount of time you'll learn enough about
your policy and it's a valid gradient
algorithm which means that it will
continue to go up the only problem is
that the gradient itself because it's a
gradient algorithm it can get stuck in a
local optimum
so there's no guarantee here that this
algorithm will converge to the right
policy
and there's no guarantee that it won't
also suffer from
linear linearly increasing regret over
time
because it could get stuck at a
sub-optimal policy similar to the greedy
policy
but it turns out to be a lot better in
practice often and it's quite a common
approach also because of this property
that i mentioned before where we can
extend this algorithm quite easily to
for instance deep neural networks
but it's good to keep in mind that this
doesn't solve the full exploration
problem just by itself
we will get back to this later because
in later lectures we will talk more
about policy gradient algorithms and how
to apply these to the full reinforcement
printing setting
one more thing to say about policy
gradient algorithms already and i will
repeat this in a later lecture as well
is that we can modify them slightly by
basically changing the mean of the
reward in some sense
and the way that works is we first know
that the probabilities sum to ones this
is a valid probability so so they must
sum to one
and what this means is that for any
number b
which is just some number
we can multiply the gradient of the
policy with this b
pull the b and the gradient outside of
the summation but then what we see is
that we simply have b times the gradient
of one
but the gradient of one is simply zero
because one is a constant and we can't
change the constant by changing theta
the total summation won't change if we
change theta it will always sum to one
that means that this this whole quantity
here on the left hand side is always
zero and it doesn't matter what b is
as long as b does not depend on your
actions because otherwise we couldn't do
this first step where we pull b out of
the summation
now why is this important well turns out
we can use that any b and subtract it
from the reward
and this can change the weight of the
algorithm works in practice for instance
instead of having a reward of plus one
and plus two we can now effectively get
rewards of plus a half minus a half
so
this is not important turns out for the
expected value because of this there
little derivation above it does not
actually change the expected direction
of the gradients but it can change the
variance of the update and this is a
specifically useful or maybe especially
useful in the full case because later on
we will use this algorithm and we will
have different states and then this
value b is not allowed to depend on your
actions but it is allowed to depend on
state that means it can co-vary a little
bit with the rewards that you might get
and might therefore reduce variance
quite quite a lot
now don't worry about recalling this
whole trick with the baselines in one go
it's good to be aware of it but i'll
repeat it later in a different lecture
as well when you go to the full
sequential case
so now we've discussed three different
algorithms
and we can see here the effect of the
baseline and it can be quite important
uh this is an example just from susan
embardo where we look at some sort of a
step size alpha which is either point
one or point four and we look at the
difference uh between the policy
gradient algorithms with and without a
baseline
and
we see that with a baseline in this
example the algorithm performed better
but we can't actually guarantee
in general that policy gradients will
will find the optimal action
but there are algorithms for which we
can guarantee that
[Music]
so i already mentioned these gradient
methods can be extended to include full
context full rl case to partial
observability
and we will discuss them again but now
we're going to move to the question what
is actually possible what is the best we
can hope for
and we're going to talk about some
algorithms that we can apply
that will get the best thing in the
bandit case
so
um
we're going to discuss this theorem
first which is a very general theorem
by lyon robbins it's somewhat all the
results
and the interesting thing about this
theorem there's a couple of things that
are interesting about this theorem but
the first one is to note that this is a
statement about any algorithm so this is
more of a statement about the problem
than it is about an algorithm
and what this problem what is sorry what
this theorem states is that in the limit
the regret will grow at least
logarithmically in the number of time
steps
so note that
log t here's the leading term that
depends on time and there's this other
quantity behind it but that quantity
does not depend on time it only depends
on the differences
in value between
the optimal action and the action that
we've taken delta a
and
a kl which is a callback library
divergence which is a measure of the
difference between the distribution of
the reward under action a and the
distribution of the reward under the
optimal action a star
if you don't know what kl divergence is
that's okay
um it's something that's measures the
difference between these distributions
and it turns out it's roughly
proportional to the
uh delta squared so a different way to
think about this is that it just bounds
essentially
the regret in a term that is logarithmic
in t and then some term that depends on
the differences in means between these
actions
now the important thing the most
important thing here is that the regret
grows logarithmically and note that this
is a lower bound so what this says is
that for any algorithm you could
possibly think of the regret will go
grow at least logarithmically in time
but that's still a whole lot better than
linear growth which we had so far greedy
epsilon greedy and ballsy gradients they
can all have linear regret so can we get
logarithmic in practice are there
algorithms that have logarithmic regret
so what we'll talk about is whether we
can prove that we can find an algorithm
for which not just this lower bound is
logarithmic but the upper bound the
worse the algorithm will do is also
logarithmic in expectation
and of course i wouldn't be mentioning
this if this wasn't the case there are
algorithms which this is true
so in order to think about this we are
going to write under regret a little bit
differently we're first recalling the
definition of this delta is simply the
maximum action value of v star
minus the action value for action a
and then we can write down the regret in
two different ways so the first one we
already sh we already saw before
where the total regret at time t so lt
can be written as a summation over time
where we count in n
um
for n is zero uh sorry for n is one all
the way up to the current time sub t and
we just count the regret that we get at
every time step this is just for
analysis we don't actually know these
regrets this is not something an
algorithm uses it's just for our
analysis that we know that our regret is
going to be equal to the summation of
all the instantaneous regress that you
get at every time step
but we can also write this down
differently and some instead of over
time we're going to sum over all of the
actions and then simply look at the
count for each of these actions up to
time t so obviously this does still
depend on t right all of these
quantities depend on t
times the regret for that action which
does not depend on t
so here we're replacing the summation
over time with the summation of actions
and quite intuitively these things are
equal
so we can also just for each action
consider how often did we take that
action and how bad was it
and we want that to be low our goal is
to have the total regret to be small
so a good algorithm will ensure small
counts
whenever the regret is large and if the
regret is small the count can be a bit
bigger
in particular
the regret for the optimal action is
zero so we don't care how big the count
will be the multiplication of the count
and the regret for the optimal action is
always zero no matter what the count is
but for all the other actions we do care
and
the algorithms that will be able to do
well in this regime
can in some sense all be considered to
implement this strategy which is called
optimism in the face of uncertainty
and this is somewhat of a container
term because you can implement that in
many different ways but the idea is
quite intuitive where whenever we're
uncertain about the value of an action
we want to be optimistic about how well
how good it could be
and then we want to basically
pick it more often if we're more
uncertain about its value so we want to
be optimistic that it could be good
and let's make that concrete with an
example so let's consider we have some
belief probabilities here these are not
reward probabilities these are the
probabilities that we
subjectively assigned to the mean of
each of these actions
so for the red action well we're pretty
certain that the red action is positive
that its expected value is positive
we're also pretty certain it's not
larger than two it might be below zero
it might be negative but it's definitely
not going to be larger than say four
it's definitely not going to be smaller
than say minus two
for the blue action we're a bit more
uncertain we think it's closer to zero
probably but it might actually be as
large as two and it might be as small as
minus two although with small
probability
the green action a3 we've barely ever
selected so we think it's not that great
in on average we expect maybe its value
somewhere close to -1 or zero
but it might be as large as four because
we've maybe only selected it once or
twice and we really don't have a good
idea
now the question is now which action
should we pick and let's step through
this as an example
if we have more uncertainty about its
value i'm arguing that maybe it's more
important to explore that action because
the mean could actually be high we're
not sure where the mean is and it could
be that it's higher than we think it
currently is
so let's now consider what we should do
well in this case the red action like
action a
a1 sorry
does look quite good so maybe not
knowing anything else we do pick action
a1
and now we're going to observe some
reward it won't be that far off because
we're quite certain and maybe the reward
will be somewhere around zero now
obviously the reward could actually be
pretty far off and the mean could be
quite certain because over time this
becomes more and more certain
irrespective of the variance of the
rewards but let's say the reward was
like a little bit less than zero and
then we update our belief of where the
actual expected value for this action is
so maybe we move it a little bit to the
left and we also make it a little bit
little bit more narrow
because we haven't just seen a slightly
lower reward we've also seen an
additional reward so we have more data
so maybe the distribution over time
becomes more and more narrow
so this is our updated belief of where
these probabilities lie what should we
do next
well maybe by now it's time to select
the action a3 which was the green action
and maybe see what we can get for that
one because we're quite uncertain maybe
we've only selected it once or twice
before and let's say that then we find
this not quite unexpected reward of
larger than one like maybe it's one and
a half or something like that and we
update its distribution a little bit
towards it
it also becomes slightly more narrow
but it's
it's still quite a bit broader than all
the other distributions so maybe next
time around you select this action again
and maybe you find out that the mean
value for it is actually larger than
that than it is for the red
more peaked distribution
so that's the whole intuition of
optimism in the phase of uncertainty
and this underpins several different
algorithms the first of which that we're
going to discuss is the ucb algorithm
so the idea behind ucb is to use
something called an upper confidence
bound
so we're going to have some notion of
uncertainty and we're going to
basically estimate the upper confidence
for each action value such that we're
pretty sure that the actual action value
qa
will be smaller or equal
to our current estimate of the action
value plus this upper confidence bound
so obviously if we pick u to be very
very large this will be true
but we don't want to pick it too large
because then it becomes meaningless so
we want to pick this large enough that
we
if we're very uncertain that we
that we're still certain that the mean
is lower than this number
but we want to also pick it small enough
that this eventually converges to the
true mean
and then
we're going to select our actions
greedily
but not really greedy with respect to
just the action value estimates q
but with respect to q plus this bound u
so essentially we're going to to
define some sort of a time dependent and
action dependent bound
that depends on the uncertainty of the
action and we're going to define it in
such a way that if we're very uncertain
about an action that we'll pick it
if we're not very uncertain about an
action so if we're quite certain about
the action the only other reason why we
might pick the action is because the
estimate itself is high
the uncertainty somewhat intuitively
should depend on the number of times the
action have been has been selected
and in particular we kind of want a
large small stroke a small count
to imply a large bound because if this
count is very small for an action then
its uncertainty must be large and
therefore we should pick it occasionally
in order to check how good it actually
is but if the count then grows and in
particular if the count becomes very
large compared to other actions then the
bounds should become small compared to
the other actions
because this means that the estimated
value is accurate
so this is the idea behind the algorithm
and what happens then if you apply this
algorithm is that each action a only
gets selected if either
its value is going to be good so if you
have a high estimate you're going to
select the action right
or
if it's bound is very large which means
that we're very uncertain about the
action
and if neither of those is true if both
the action value is quite small compared
to other actions and the uncertainty is
quite small then we're not going to
select the action and that seems indeed
to be the reasonable thing to do because
if you're quite certain that an action
has a comparatively low value then at
some point we should stop exploring it
this is what epsilon greedy does not do
this is how epsilon really fails it
keeps on selecting these actions
indiscriminately of our uncertainty
about it and and
also not looking at the actual values
whereas ucb
stops selecting certain actions
or selects them much less often when the
values become low
and the uncertainty becomes low as well
now this is the intuition but can we
somehow maybe derive this bound can we
somehow figure out what the right way is
to approach this
and can we somehow
maybe come up with a bound that is
optimal in some way or the other
so
we're going to discuss that and then i'm
going to actually derive it on the
blackboard but first we're going to do
this on the slides
and what we're going to use is something
called the concentration inequality
and this specific inequality that we're
going to use is called huffling's
inequality
and what this gives us it gives us a
bound on how
wrong our estimate will be
so in general let's consider a bunch of
random variables x1 up to xn
and these are going to be identical and
independently distributed random
variables
so we're going to use these later on for
the reward so think of x1 to xn as being
the rewards for a specific action these
are all just random variables drawn from
the same distribution and in particular
for simplicity of these of the theorem
statement let's assume that these are
all between 0 and 1. the theorem is
easily extended to more general cases
but for simplicity we're just going to
assume that all of the rewards
essentially are going to be between 0
and 1.
there will be some true mean the
expected value of x for any x
in this set and this is going to be mu
now we're going to define x
bar so x t with a bar on top as being
the average
or the sample mean for all of these
random variables that we have so far
and what hufting's inequality then
allows us to do
is to consider
how far off could we could this mean be
so consider some
quantity u
if we add that quantity u
to the mean that we've estimated so far
how likely is it that this thing even
after adding this u is still smaller
than the actual mean
and turns out we can bound this quantity
to be a small number and in particular
it will be the exponentiation of minus 2
times n where n is the number of
elements going into this average
and u squared where u is this bound that
we add
so what do we see here this is a
probability it's a number that's going
to be between 0 and 1
in particular
this is a bound so the bound is not
actually capped at 1 but of course if
this is larger than 1 then we don't care
then the probability will be bounded at
1 instead
and what we see this is typically going
to be a small number and it's going to
be smaller and smaller the larger either
n is or u is and this makes intuitive
sense the more
numbers we have in our average
the less likely we are that if we then
add an added amount that this is still
going to be smaller than the actual mean
we're quite likely to be within u of the
actual mean if we have enough numbers in
here
similarly if we pick this u to be larger
the probability also decreases which
means that even for a given number of uh
elements in our mean
if we if you consider to be far away
enough
then it becomes exceedingly unlikely
that our sample mean is that far off
that is the intuition but the statement
is as given and we're going to use this
in a second to derive ucb
and the idea is to apply this to bandit
with balance with bounded rewards so as
i mentioned so let's assume for
simplicity that the reward is between
zero and one this is not necessary we
can extend the theory to more general
cases but for simplicity we're just
going to make this assumption
and then we're just going to plug in the
banded case to into this hufftings
inequality so specifically we're going
to consider our current estimate for the
value a qa
plus some uncertainty bound ua which
we're going to want to determine what we
want that to be
and we're going to consider the
probability that this whole quantity
even though we added u is still smaller
than the actual expected value and now
we know that we can bound that
probability in this fashion
and by symmetry of the argument you can
actually flip this around
uh and you could also consider like this
is considering how far off is it in one
direction this is considering how far
off are you in the opposite direction so
instead of adding the bond you could
also subtract it and then check whether
you're still even after subtracting
whether you're still large in an amine
i'm just mentioning that for
completeness it's a useful thing to be
aware of
now
the whole idea behind ucb is to
basically pick a probability so we have
here a bound on the probability and
we're going to say well
let's pick this thing let's let's pick u
so that's so that this doesn't exceed
some sort of a number so let's pick a
probability p
and we're going to say let's pick our
bound to be equal to that probability p
now we can solve this for
the bound u
if we want this to be true this first
statement that means we have to pick u
in this following way
but now we can we so what is what does
this mean well if we pick u specifically
in this way then we know that the
probability that we're going to have
a mean that is further away from our
sample mean than this bound
is smaller than this quantity p
and now we can pick p to be small and
decrease over time so that's the whole
idea
behind the algorithm we're going to
reduce the probability that we're going
to be more than this bound
off
and specifically we can pick it to be
one over t just plug that into the bound
there and we get this bound that looks
like this
which is
the bounds going to be the square root
of the logarithm of t divided by twice
the count for that action
now i didn't tell you how to pick p
i didn't tell you why we're picking it
to be one over t and in fact there are
different choices you could make there
and i'll get back to that in a moment
but this is just an example of oh you
could pick now the probability that
you're going to be more than
this bound you wrong with your sample
estimate so what does this capture well
if we don't have a lot of samples if n
is pretty small
then this bound will be pretty large
if n grows larger this bound will go
down
in addition to that the bound grows
over time what does this mean this means
that indefinitely we're going to
continue to select each action
but maybe less and less so if it's
really not a great action
so this ensures that we keep exploring
the fact that this grows with the square
root of log of t does ensure that we all
keep exploring every action but not that
much because if the action is really not
that great at some point the uncertainty
starts growing growing growing because
of this log t term so whenever we don't
select an action slowly the bound is
creeping up but quite slowly because log
t uses slow growing function
and the square root of log t is even
slower but then when we do select the
action
the bound goes down so then the only
reason to keep on selecting that action
is if the estimate the mean is large
enough
so the ucb algorithm
then looks like this where we've now
instantiated that upper bound by
plugging in this square root of log t
divided by the count n and we've
introduced this new hyper parameter c on
the previous slide
we basically within the square root we
were dividing by two
so this is this this is the same as
picking c here to be one over square
root of two but turns out c can also be
just considered kind of like a
hyperparameter for larger c you'll
explore more for smaller c you'll
explore less
in particular if you pick c to be zero
we get the greedy algorithm back that's
not great that doesn't explore enough
and the intuition now behind this
algorithm is that if your gap is large
that means that the count will be small
because the sample average is likely to
be small so it will take
more
uh if your if your gap is really large
it will essentially take larger gaps to
span
so the the bound must be quite large for
you to be considered
so that means in practice that either
the gap will be small or your account
will be small
that's not a formal statement this is
just your intuition
but in fact turns out we can prove that
in total the problem the products of
this gap times the count
will be bounded by the logarithm of t
for all actions and that means that we
have now established an algorithm which
has logarithmic regret as an upper bound
because we know that the log that the
regret is also logarithmic as a lower
bound this means that we now know that
the algorithm will have logarithmic
regret
and this was proven by peter hour
and this was uh one of the references
that i said at the beginning of this
lecture
where he proved famously that ucb
and he had a specific c there which is
uh not that important for the proof
actually but this is the specific
statement that he proved he plugged in a
specific c there or he derived it with a
specific c
and he showed that the regret for this
was logarithmic in the way that is shown
on the slide for more details please do
refer to the paper or you could also see
the proof for that specific algorithm
but i'm also going to prove it here
which will hopefully help you establish
[Music]
maybe some techniques on how you could
come up with this so instead of just
taking ucb and proving it
i'm going to try to derive it in some
sense from first principles there will
be a couple of
steps there that are maybe in some sense
cheating because i already know the
algorithm
but
it will hopefully still be useful
if you
want you can skip this section from the
lecture as well and just go on to the
other algorithms
but it's useful to at least go through
the proof either the one that i'm going
to give you now or maybe the one in the
paper by peter hour just to get an
intuition for how these things work
okay so we'll go to the
blackboard and i will try to in some
sense derive the um ucb algorithm
which means that i'll start at what we
want and then see if we can somehow come
up with the algorithm
from first principles
there will be a couple of steps here
that are going to be a little bit
creative in the sense of i'm jumping to
maybe something that won't be
immediately obvious and indeed when of
course you come up with such an
algorithm this often involves a little
bit of trial and error let me try this
oh that doesn't quite work let me try a
different thing
and of course now i already know where
to end up and i also don't want to make
this too lengthy this segment so
sometimes i will make a step that is
maybe a little bit of a shortcut but
hopefully it will still
give you some clarity of how you could
come up with such an algorithm
so to start off let's write down the
objective we have the total regret which
as noted in the
previous slides you can write down as
the number of times you selected an
action
times the
expected regret for that action
um
sorry this is the expected
regret that we're interested in
right
and then note that the only random
quantity in the right hand side are
these counts
also note that the instantaneous regret
for the optimal action which we'll call
a star
is by definition zero so that means that
we don't care about the counts for the
optimal action
we simply don't care how often we select
it in fact selecting a more often is a
good thing because that means that our
total regret is not growing
for all the other actions
so for all actions that are not the
optimal action
we want this count to be small
note that the regret itself does not
depend on time right this is just some
number that depends on the action so
that means in order for us to have a
suitable
count we want
sorry for a suitable request we want the
count to be low and in particular we
want a multiplication of
discounts times the regret
maybe to match the
lower bound that we know from the lyre
robbins theorem
so let's assume that we don't know
exactly what we are able to get but
let's hope that we can get something
that is smaller or equal to some
constant that will fill in later because
we don't know yet what that should be
times the logarithm of t
that would be great if we can get that
so
we'll start here and then we'll see
whether we can find an x
and also to find an algorithm such that
this is true by the way the algorithm i
should note that at the beginning the
algorithm that we're going to consider
is going to be the ucb style algorithm
so we're going to pick an action
such that
it
maximizes the expected
estimated
value plus some bound
and what we're going to do here is to
see if we can come up
with what u should be can we come up
with an upper confidence bound u
such that we have logarithmic regret
now in the science i already told you
one way to
instantiate that but let's assume we
didn't know that in advance and that we
can somehow see if we can derive that
so in order to get the lie robin's lower
bound we need the regret to grow
logarithmically
or something that grows equally fast
which means we need this property
now it could be that this is already
true on some timestamps right we know
that nt so the count for action a at
time t is a random quantity and at some
time steps we could already be in
declare
so let's assume that we have some time
step in the past before time step t or
it could be time sub t itself
let's call it time step
m
which is smaller or equal to t
on which we know that the count is small
enough in particular let's say
that our count
is going to be
uh well let's just say it this equation
above already holds
well that means that we're good because
this is the
total regret up to that time step
so if
this equation above already holds we're
good to go for time set m
which means
that we
only need to consider
um
let's just write it down equivalent
sorry explicitly
if this is true
then clearly it's also smaller than
log t which is the time set that we care
about
so
in in in what follows we only need to
consider the time steps after the very
last time this was stirred so let's say
there was some time step m
let's make it a little clearer this is
an
m
and this was true that means that up to
that point in time time step m the
regret in total was logarithmic even
though at some earlier time subject
before time step m
it could have been
it could have popped above this number
we know that by the time we're at time
step m
this is true now this time set m we
don't know what it is right it might be
some number that is close to t might be
some number that is close to one but we
don't care for now what we're just going
to say is oh there's going to be some
additional time steps
and they're going to be after this time
step m
and because we're already good to go at
time set m all that we have to look at
is these time steps that are after and
we know that for these time steps it
must be the case
that the count
is larger
by
assumption that m is the last possible
time step on which
this statement above hold
held
that means that in the subsequent time
set this statement must hold
okay so we can just put that to the side
for now we're basically considering a
couple of time steps
beyond this time step m in the past
and now what we're going to do is we're
going to look at how quickly does the
regret grow
after the first times that we've
violated this nice
property that we had low enough regret
and if we can show that regret grows
slowly enough when this is the case
then we can bound the total regret
so effectively what we will be doing is
we're looking at the total expected
count for a given action for a given
time step
and we're basically noting that you can
you can write this down explicitly as
the expectation of a sum of these
indicators that action
a was picked
that's just by definition
and what we did is we basically split
this up into two parts where we say well
we have
the number until until this time step m
it doesn't matter that we don't know
what it is there is maybe sometimes that
m could be m is one or could be m is t
somewhere between 1 and t in which this
was true the fact that
your gut was low enough
plus
the remainder of the time steps where
we're just going to concern ourselves
with these intermediate time steps now
and we know that
this by definition of what m is
is going to be
smaller or equal to the
x a times the logarithm of t divided by
delta a i'm just using that
property that we had up here
for
m so there are some types of m for which
we know that this is true
and i'm just plugging that in here
and then we have the remainder as we had
up there
just going to repeat that
for completeness
and then what we're going to do we're
going to notice that this first quantity
is now no longer a random variable we've
just bounded the random variable that we
had into something that is no longer a
random variable so we can take the
expectation away
so we can push the expectation all the
way into the summation that we have
so this is going to be equal to the exp
sorry
not the expectation
this is just going to be x a which we
haven't determined yet
log t divided by
a
plus and we can push the expectation all
the way into the summation
the indicator that a n is a
and we know that this is actually
conditional so let's be explicit about
that now
on n being
the time step beyond
m
which means that we know that at these
time steps
the count
will be sufficiently large
that this is true
and then we take it from there and we
can see
what will happen but note that here we
actually have the expectation of an
indicator so we have the expectation of
an indicator of an event
the expectation of an indicator is the
same as the probability of that event
so we can write this thing inside the
summation as a probability
so now i'm going to put a this condition
like i put it there explicitly but let's
put that to the side for now this is
going to be we're going to consider a
time step
n
and an action a when this is true but
i'm not going to include it in the
notation all the time because it becomes
a bit
cumbersome so now we're going to look at
can we bound
the
probability that we're picking the
action
given that
this is true
we're just logging that to the side we
don't know yet whether it'll be useful
but we know that that's true all these
time steps we might as well keep it
around
so now we can talk about the probability
of selecting an action
note that a is a sub-optimal action
right we already took care all the way
at the beginning we took care of the
optimal action and we said well we don't
care often so we select that so we don't
need to worry about bounding the count
for the optimal action we're only
worried about bounding the counts for
the sub-optimal actions so what is the
probability of selecting an action
well in order to select an action it
must have the highest value plus
bound and in particular
we can bound the probability of
selecting the action by saying well at
the very least
the
value for that action
plus the bound for that action the
estimated value plus the boundary for
the action
must exceed
those quantities for the optimal action
i put um greater or equal here i could
have also put uh greater because then
uh
like we were strictly
certain we're picking action a but let's
just assume that ties don't really
matter that much so these values are
never exactly the same
so it doesn't really matter whether i
could greater than or greater or equal
than
though this is not intended to be very
rigorous uh
proofs is more of a proof sketch or
derivation of how you could come up with
the algorithm
now let's simplify notation a little bit
let's get rid of all these um
brackets and subscripts and stuff so
let's actually write this down
just notationally
for now as q plus u
is greater or equal to q star
plus u star
this is just a notational shortcut this
i didn't change anything here
so can we somehow bound this probability
because we know that the probability of
selecting action a is going to be
smaller or equal to this probability
because in order to select action a you
must at least have a larger value plus
bound than the optimal action otherwise
you won't select it and indeed there
might be other actions that you might
select as well but we don't even need to
worry about those turns out
now this we can write out by considering
different cases
so it might make sense to look at
these estimates so we have an estimate
for
the action a and an estimate free action
a star
and let's just pick one of those let's
pick a star and consider whether our
estimate is in some sense wrong
so it could be that our estimate for a
star is quite low it's in fact
substantially lower than maybe it should
be the case and we could consider
some quantity
so let's write down this whole thing
again
and let's condition it
on
that q
star plus u star
is um
low so we're going to say this is lower
than expected it's it's low
lower than some number y that'll fit in
in a moment
and then of course we also have to
consider the opposite case so in one
case we say well
maybe
q
star is quite low and maybe then we can
bound this probability and in the other
case we're going to say well now q star
is not particularly low
but then maybe then again we can bound
that probability because maybe then
q needs to be quite high
so the alternate case is just
using the law of total probability
right um
i didn't change anything here i'm just
writing things out i'm considering case
case by case and of course i need to
multiply both of these with the
probability of this new condition that i
put in
and this is starting to look interesting
now
because here
we have something while i'm putting down
the rest
we have something that looks
similar to huffling's inequality
we have a random quantity
q star
we have some bound u star
and we have some quantity on the right
hand side y
so maybe we can plug into huffling's
inequality here and in order to do that
y should be the expectation of the
random quantity so maybe we can pick y
to be
the actual value
of the optimal action is that a good
choice
well let's just try let's just plug it
in and see what happens
so
if we plug in
y being the actual value of q star
we have this probability
p
q star
plus u star
is smaller or equal to q
star right this is just shorthand for
the actual value
of a star
and this we know from huffington's
inequality is going to be quite small
it's going to be specifically on some
time step sorry i was at time step i was
calling it n so let's use consistently
let's use
time step n
this is going to be
just using half things inequality as
given in the
lecture slides
we know that this is true
and now we can
see that maybe we can pick this bound u
to make this probability substantially
small enough
how small do we want it to be well we
were looking eventually going back all
the way to the beginning we're looking
at the probability of selecting an
action how small should we make it well
we're looking at the probability of
selecting the action on a specific time
step
and we want the summation of that so the
summation of these probabilities over
time
which is what we have up here
we want that to be small
now if we sum a function of time over
time and we want it to be smaller than
the logarithm of t
then maybe we can bound this probability
selecting the action
with one over n is this possible if it
were possible to bound the probability
of selecting the action with one over n
where n is the time step
then the total summation of that would
be smaller than the logarithm
of n
or up to whatever action you uh put
because the um
and i'll just put it to the side and
i'll remove it in a moment we know that
it's true that if n is one to t and we
have one over n
this summation is smaller or equal to
the logarithm of
t plus one
it's actually smaller
there's a
small quantity there it's slightly
larger than logarithm of t but smaller
than logarithmic t plus one
so this will be small enough if we can
somehow make sure that this is equal to
one over n
can we make that happen
well we have full
choice over what u is so let's just
solve this
so if we have e
minus n u squared and we want that to be
equal to 1 over n
then this implies
that we want u to be equal to the square
root
of the logarithm small n divided by
big n
so this is interesting because here we
see the ucb
bound showing up
but we've derived it from this need for
the probability on every step to be
1 over n in order to get this logarithm
of t in total
so now we can go back we've kind of come
to this conclusion that maybe this is a
good bound and now we can go back to the
rest of the proof and see if we can then
prove
everything holes that we want to hold
so let's go back
slightly so we're looking at this
probability let me use a different color
now
so we're looking at this probability
of selecting the action
and now what we've established is that
this quantity
can be bounded
by
one over n
this quantity over here
can obviously be bounded by one
so the first term
in this
is small or equal to one over n
so next
we move
and i'll use a different color to the
second term
now
for this last bit here
we can't say that much because we
already saw that the inverse of it above
is the probability that's quite small so
all we can say about this one is small
or equal to one perhaps perhaps we can't
say that much more
because we know that if the opposite of
this is is not very likely then this
event might be quite likely so now what
we're hoping to do is maybe we can bound
the other side
so let's see if we can do that
so what we're trying to bounce we're
going to see whether we can bound q plus
u
being greater or equal to q star
plus u star
given that we know that q star plus u
star is not particularly small it's
going to be greater than the average
for q star
can we make sure that this is small
well
we know
that
the probability of q plus u being bigger
than q star plus u star
that's going to be smaller than the
probability of them being larger than q
star because we know we're comparing to
something
the estimated q star plus u star which
itself is bigger than q star so we can
bound this probability
by saying q plus u
is greater or equal to q star
okay that looks less complicated that's
good
it's not quite in the form where we can
apply huffington equality or anything
like it
so we need to do a little bit more work
first of all we can notice that the
inequality is in the wrong direction for
huffling's inequality
so one thing that we can do is first we
can flip the signs
the negative of a random variable is
obviously still a random variable so
this
doesn't hurt
applying huffling's inequality at all
and we also we're also missing on the
right hand side
the expected value for the quantity that
we want to uh
for the random variable we have this
random variable on the left hand side
but we don't have its expectation on the
right hand side
so let's just add and subtract that
expectation so that we have the
expectation on the right hand side
and now
notice that this
minus q star
plus q
is by definition just
minus the regret for action a
for the action a that we're considering
so this means that we can simply write
this down we can move that to the other
side of the inequality so we can say the
probability of all this
again i'm simply rewriting things here
i'm not actually changing anything
is smaller or equal to q
so now we have kind of the desired form
to be able to apply huffling's
inequality we have a random variable on
the left hand side
we have this expectation on the right
hand side
and we have something else
now there's something else looks a
little bit complicated and potentially
messy because it has this
action gap this delta a in there
so it would be nice if we were able to
instead of this have something that it
more looks like
you
so can we
maybe somehow
have the property that this is larger or
equal to 2u
if we could do that
then we could just replace this
probability we could bound the
probability on the left-hand side
with something that has this nice clean
simple form where we just have this
random variable minus q
and we add some bound u and then we
compare that uh to see whether it's
lower it's still lower or equal to oh
sorry there should be a minus q here on
the
right hand side
whether it's it's uh small or equal to
its expectation
so can we do this well now we're going
to scroll all the way up and we are
noticed that we do have we do know
something
this is something i said all the way at
the beginning we were considering only
those time steps
on which it's true that in some sense we
were saying that the count is rather
large but instead we can also interpret
this as
the gap being fairly large so let me
just repeat this equation let's call it
equation one or something
and let me
repeat that equation all the way down
and see if we can use that
so we know from equation one i'm just
going to repeat it here that n times the
delta
is going to be greater than
x a which we hadn't determined yet
times the logarithm
of the
time step n
so that means that we do know that delta
a is larger than
x a
logarithm of n divided by n
well that's interesting so that looks
very similar to our upper bound
but in fact it's the
square of it because if we go up we saw
that u was defined
as the square root of log n divided by n
for any action
um
this is the count for action a at time
step t just to be
clear so if i write it down more
explicitly we suppress that's from the
notation but that's what it means
so maybe
so let's first actually write that down
so this term here
is equal to x a
u
t a
squared
but what we wanted is this other thing
right we wanted the delta to be larger
than 2u
instead we have the delta is larger than
x times u squared
so can we somehow make these the same
well we could
for instance pick x a to be 1 over delta
a
if we pick it in that way and notice
that we can pick xa in whichever way we
want as long as we don't make it
dependent on the time set t
then we have that delta
squared is larger than
u squared
which means that delta is larger than
mu
so this is not enough oh we notice we
have a we have a problem here with a
constants but we could have picked x in
a different way so let's pick it to be 4
divided by
delta
and then we'd have
i have to make a little bit of room here
in order to be able to then change the
equation
this would be 4 and then taking the
square root on both sides we have 2u
which is the required thing the thing
that we wanted
so by picking x to be a certain value 4
divided by the action gap for that
action
we can make sure that in the time step
under consideration delta has this
property
and that means that we can go back now
to the equation up here
and we can say that this thing must be
smaller or equal to the probability of
this
random variable minus q
plus two u
minus u
being smaller or equal to the
expectation of that random variable
and then of course we can simply cancel
the 2u
and the minus u
to just have a u
now we have something that is exactly in
the form of halfting's inequality so we
can say this is smaller or equal to the
exponentiation of
the count
times u squared
and by definition of u which we've
defined above to be equal to the square
root
of the logarithm of n divided by n
we know that this is going to be equal
to 1 over
n
as desired so then we can go back up
here again
let's
pick a nice green color again we know
that we can now bound this to be one
over
n
so that means if we go up here
that we were actually able
to bound
this to be
um
not quite one over n because we had
these two different cases but we were
actually able to bound it as 2 over n
which means that the summation over time
is still logarithmic in t
so now we can plug this in and we can
say
that this total summation here
is going to be smaller or equal to two
logarithm
of t
plus one
and therefore the whole term
is logarithmic in t we had this case
first when the count was relatively
small that was covered in the first part
so maybe let me do that with a color but
that was covered in the first part and
then in the second part we did a lot
more work but we were able to bound that
as well
in something that's logarithmic in t
and we were able to do that by just
first picking some generic numbers x and
later seeing how we should fill that in
and actually let us now fill that in
here we figured out below
that we were able to do that by picking
x to be equal to four oh sorry
four
divided by delta a
so you can you can do this in different
ways you can bound this slightly
differently as well you can pick other
constants you can go through it slightly
differently and you could come up with
slightly different bounds
notice for instance that we came up with
a certain definition of u that is
slightly different from the one from
peter hour because peter r had a
factor of two there which we didn't use
you could put that there as well and you
get a similar bounds which is also
logarithmic so we don't particularly
care about these constants that much we
more care about the rate
which we were able here to establish to
be logarithmic in t for the total count
for any action
so to repeat what we've done we've
basically done a case by case analysis
here first we went to this time step and
we said well there are maybe some time
steps
on which we already have low regret
and then we said well so now we're going
to only concern ourselves with the time
steps in which the regret is not that
low and it turns out we could use that
later with this equation one
to make sure that for those time steps
we know that the gap must be relatively
large
in the sense as as stated in equation
one and we were able to use that later
on
then for those time steps in which uh
this is the case we just bounded the
probability of selecting the action and
this is useful because bounding the
probability allowed us to eventually
bound the expected count
and by bounding the expected count to be
logarithmic
because the action gap itself is not a
quantity that depends on time if we're
able to bound the counts to be
logarithmic then the total regret is
also bounded to be
logarithmic again i'm i'm going to
stress here this is not a very rigorous
proof there were some steps that could
have been made a little bit more precise
the point here was not to be very
rigorous or precise to point here was
more to show you how you could come up
with these type of algorithms perhaps
and
how in some sense the ucb bound could be
derived by thinking about huffington's
inequality and thinking about what are
the properties that we want from
this regret so you want the count to be
low we want it maybe to only be
logarithmic and then turns out if you
want to use huffing's inequality
the upper bound that is used in usb
almost pops out in some sense
okay
i'm going to
go back to the slides
now
and continue
so next
we'll go into some bayesian approaches
to bandwidths
so what is the bayesian approach in the
bayesian approach we keep track of a
distribution
and in particular we're going to keep
track of a distribution over the
expected value for each action
this is different it's good to keep in
mind many of you will be familiar with
beijing approaches but just to be clear
we're not trying to model the
distribution of the rewards
instead we're going to quantify our
insurgency about where the expected
reward is and then update that over time
as we go
so know the probability here is not on
the
reward given action a no it's over the
expected
reward or the value of action a which
under probation approach you consider to
be a random quantity
not because it's random every time you
query it but because you have
uncertainty about what its true value is
and then theta slightly overloading
notation here but here using theta now
as the
um
parameters of the distribution so for
instance
so first like to be clear this
probability should be interpreted as a
belief so for instance for some number x
the probability of q
q of a being equal to x is our belief
that that's true it doesn't actually
impact the actual probability because
this is this is just a number
but it it makes our uncertainty about
this and as an example theta could for
instance contain the means and variances
of gaussians if we want to model each of
these uncertainties with the gaussian
distribution
that's choice right we can just pick a
distribution and then
uh hopefully we pick it in such a way
that at least the true value is
supported with the gaussian for instance
this would be the case because the
guardian has full support over the real
number line
and then
this allows us to do a couple of things
that we couldn't quite do before as
easily perhaps for instance we could
already
a priori we could pick certain actions
to have a higher expected value than
others because we know something about
them we might know that oh this value
we're quite certain is between 10 and 11
let's say
or this other value we don't know where
it is it's somewhere between 1 and 100
and
if we want to model these things
explicitly we can actually inject this
as prior information before we even
start running the algorithm
but you can also use these approaches if
you don't have a lot of prior
information for instance i'll show you
an example where we assume
very little and we basically say well we
know the true value somewhere between 0
and 1 but it could be anywhere between
there it's uniformly likely to be
anywhere
so if we do this and we update these
probabilities then we can use these
beliefs to guide our explanation if we
have these complete distributions over
where we expect the values to be we can
for instance very easily pick the upper
confidence intervals
now let's go through an example to make
this a bit more concrete so for instance
consider a bandit with bernoulli reward
distributions and i mean with that that
the distribution of the reward only has
support on zero and one
so the only thing that you basically
need to figure out is how likely is the
reward to be won and how likely is to be
zero
and then for instance the prior
information could be very little so for
each action we basically say well we
don't know
it's uniformly likely to be one zero a
half a third we don't really know we're
going to assign equal belief to each of
these possibilities on
the interval between zero and one that's
just one choice of prior
and that means that either each of these
values is equally likely
even though the rewards can only be plus
one and minus one the mean could be
anywhere between there
now one way to model this is to pick
beta distribution with parameters x a
and y a
and for instance if i pick a beta
distribution with initial parameters x a
is 1 and y a is 1 then we exactly get a
uniform distribution over the interval 0
to 1.
a beta distribution is quite a common
one to be used in the combination of
bernoulli data
and then we can update the posterior for
the beta distribution this is quite
simple
whenever the reward is 0 we update the x
parameter by incrementing it with one so
for the action a
t that we selected at this time step we
just increment x
so this will be the number of times the
reward was zero
plus one
for action 80
and if the reward happened to be 1 we
increment the other number y
note that x and and y together actually
give us gives us give us the count the
number of times the action was selected
plus two because they start at one each
of them before even selecting any of the
actions
this is how that looks so if you start
here at the left bottom plot you see a
uniform distribution between zero and
one
and again this is the belief that we
have for each of these values so we
believe each of the values between zero
and one to be equally likely so you have
this
block essentially uniform probability
mass
now if we then see a reward of plus one
we can do the bayesian updates to the
beta distribution which for the beta
distribution is quite simple to
implement
and then what we see is that the
probabilities immediately change quite
dramatically
and in particular
the mass on a mean of zero becomes
basically zero and this makes sense
because we've seen a reward of plus one
now so we know that a true expected
value cannot literally be zero anymore
if the only possible rewards are zero n
plus one and we've at least seen one
plus one then we know that the true
value must be at least infinitesimally
above zero so the probability of
literally zero goes to zero immediately
after just one sample
and the probability of one essentially
doubled so now we have this triangular
shape to the distribution which assigns
more probabilities to larger numbers
given the information that we have right
now but still some substantial
probability mass as well to lower
numbers
now let's say we see a reward of plus
one again then the distribution gets
this nonlinear shape it's actually cut
off here slightly on the slide but it
goes up even farther on the uh right
hand side it becomes more and more
likely that the true mean is one so far
we've only ever seen plus one so the
most likely mean value is actually one
but other high values are also quite
likely very low values become quite
unlikely
if now after this we see a reward of
zero
and another reward of zero
then the distribution has become
symmetric again and now the most
probability mass isn't exactly a half we
know it's extremely unlikely that it's
very close to zero it's also extremely
unlikely to be very close to one the
average value and instead it's much more
likely to be somewhere here in the
middle
this is just an example to step through
how you would update these distributions
so what you'd actually do if you if you
did this for ballooning banner with a
beta distribution is you just increment
these counts right but implicitly these
counts would then imply this
distribution
you're basically updating this
distribution this is what it looks like
now if you have these distributions they
could be gaussians they could be beta
distributions but then we can
use this to explore and in particular we
could again go back to this principle of
optimism in the face of uncertainty and
one simple thing we could do is we could
look at the standard deviation of each
of these beliefs
and
pick kind of like an upper confidence
bound so we take the mean value for
instance for action
a3 here the mean is the highest of the
bunch and we add say one standard
deviation and this brings us over here
now note that the red action has a lower
mean but adding one standard deviation
might actually bring it or two standard
deviations i don't know how much it is
exactly but some number of standard
deviations would actually bring us
higher
so the upper confidence bound for the
red action here action a2 is actually
larger than that for the green action a3
but the copper upper confidence bound
for a1 which has this hugely wide
distribution turns out to be even larger
so we see here that by adding this
uncertainty
we get the same principle of optimism
under the phase of uncertainty but now
instead of using an upper confidence
bound
which stems from huffling's inequality
instead we can also just try to
approximate the full distributions and
then you use those to do the optimism in
the phase of uncertainty
so one way to think about this is that
we have yet again we have a bonus in
some sense and we could use that to pick
our action but the bonus could be
derived from these
posterior distributions which we've
updated according to bayes
and then we could still do the same
principle as in ucb so basically we
could do a ucb like algorithm but using
the
bayesian approach rather than the
bounds that we had before
this is however not the only algorithm
you could do and now i'm going to tell
you a little bit more about a different
algorithm
called thompson sampling
so thompson sampling is also an
algorithm to
solve bandits and it's a bayesian
approach
and in particular um it's going to be
related to something that we're going to
describe first which is called
probability matching
so what is probability matching
this is an algorithm that is quite
different from ucb because it's a ran
it's a random algorithm so our
probabilities of picking an action will
be
a
random quantity like our action will be
random quantity our policy will be
stochastic
and the way it works is it picks an
action according to the likelihood
according to our beliefs
that this action is the optimal action
because we have these belief
distributions now we can reason about it
so we can pick the probability of each
action to be exactly the same as the
probability that action is optimal
now this is a somewhat unintuitive thing
but it is optimistic in the face of
uncertainty because if you have a large
uncertainty about where your action will
be
then the probability of it being maximal
also goes up maybe it's not the most
likely action to be maximal but it might
be a fairly
likely action to be maximal if your
uncertainty is high
so actions with larger probabilities
are either high-valued actions or you
have a lot of uncertainty about them
similar to the ucb algorithm and other
optimized optimistic approaches
however it's a little bit of an
intuitive thing as you mentioned because
it's not immediately obvious that this
is the right probability that you should
assign to an action right it's not
immediately obvious that picking
according to the probability that you're
the optimal action is also the right
probability to use for expiration
in addition to that it can be a little
bit difficult to compute this
probability analytically
for from the posterior uh well this can
be done numerically but even keeping
track of the posteriors of course can be
a tricky thing uh in a full basin update
but if you have the posteriors then you
can compute these probabilities
potentially numerically but turns out
there's also an easier approach which is
called pumps and sampling and this is in
fact
perhaps the oldest of banded algorithms
it's already from the 1930s
and it's named after the inventor of the
algorithm and the idea is quite simple
so we are still going to keep track of
these posterior distributions so we're
going to update these via bayes rule for
instance
and then the idea is to sample first
from each of these belief distributions
an actual action value
so please
carefully think about what this means
you have your belief distribution at
time step t about where you believe the
mean value for that action to be
and then we're going to sample the
distribution which gives us an action
value and we're going to do that for
each of these actions
then we're simply going to pick the
greedy action according to the sample
action values
turns out if you do that
fonts on saturn will select the actions
according to exactly the same
probability as probability matching
would would do
and the proof here is essentially
contained on the slide
because the instance of sorry we have an
indicator function there over an event
and the event is that action a was
picked
so
the event that action a is picked means
that action a had the highest sampled
action value
but the expectation of this instance is
equal to the probability
of that happening
so it turns out if you do
sorry thompson sampling if you first
sample the action values and then you
pick greedily this is exactly the same
as trying to just just computing these
whole probabilities and then selecting
your action according to that
so it's kind of an interesting simple
shortcut that if you have these
posterior distributions you can simply
sample values from these and then pick
greedy
so in some sense thomson sampling is
simply a technique that allows you to go
from these bayesian basin
probability distributions these
posterior distributions
to a policy
interestingly
thompson sampling actually achieves the
lie in robin's lower bounds for
bernoulli bandits and in some other
cases as well and therefore is optimal
in the same way that ucb is
and it has logarithmic regret
so this was actually proven not too long
ago well a couple of years ago now but
this wasn't immediately obvious to
people but thompson sampling is now
considered
similar to ucb one of the optimal
algorithms why did i tell you about
multiple algorithms and i'll tell you
about one more approach in a second so
it's good to mention this
that
not all of these approaches are quite as
easy to scale to the full reinforcement
training case so this is why we're going
to go through a couple of these
different algorithms
and then like we'll have
a lot of tools in our toolbox which we
can then see which ones we can apply
best at scale and for instance
thompson's happening has been applied to
the full reinforcement running setting
and also to the function approximation
case in which the observations coming at
you are just messy observations uh for
instance from a camera and you're
somewhere going to have to deal with it
you can't do this exactly then perhaps
but some of these approaches are easier
to estimate than others and the jury's
still a little bit out of which approach
is the most beneficial or the most
fruitful for future
work okay now we're going to continue to
our last
set of algorithms in a sense
and we're going to consider
planning to explore
so what do i mean when i say planning to
explore well so far we've viewed bandits
as one-step decision-making problems i
mentioned all the way at the beginning
that the stage doesn't really matter
but we can actually also view them as
sequential decision-making problems but
instead of reasoning about the
environment status being the sequential
part the thing that makes it sequential
we're going to talk about internal v
agent so at each time step the agent
updates some internal states to
summarize its past this state now does
not need to contain anything about the
environment because the environment
doesn't have any in additional
information but it should contain
something about
the rewards and the action that the
agent take has taken
so each action a
can be thought of as transitioning and
to a new information state as t plus one
by adding information
with some probability
and this probability depends on
given the state and the action it is a
random quantity like the next states
because it depends on the reward that
you're receiving
so given that we've taken action a t in
state st we're going to transition to
some state st plus one
and that means that we have some sort of
a markov decision process
where the states are fully internal to
the agent
this is just whatever is internal to the
agent as i mentioned these state
transitions there are probabilities
because the rewards can be random and
also the actions can be random so if you
condition on
one action then that randomness goes
away but you could also consider the
state the state probability distribution
which will also then have the randomness
of the action
depending on your algorithm
so thompson sampling is a random
has a random policy whereas ucb has a
deterministic policy
but what does this all mean well it
means that even in bandits we can think
of actions as affecting the future after
all
but now not because they change the
internal state of the environment in
whatever way because there is no state
through the environment there's nothing
to be changed
instead
they change uh they affect the future
because of how they affect the internal
state of the agent
so to make that more concrete
considerably banded again where there's
some probability mu that you'll find get
a reward of one and there's therefore a
probability of one minus mu that you get
a reward of zero for some action a
so for instance you can think of winning
or losing a game with that probability
and we want to find the arm with the
highest mean so we want to find the
strategy say that is most likely to win
the game
then we can consider the information
state to be a tuple alpha and beta where
alpha counts the number of times the
reward was zero and b is a constant
number of times where the reward was won
recall that this is very similar to the
beta distribution that we talked about
before
where the parameters x and y of the beta
distribution were essentially alpha and
beta plus 1 each
so
this information state is fully internal
to the agents and we know exactly how it
will update whenever you see a reward
right
but which reward to receive is a random
quantity so the state transition from
this information states from one time
set to the next is a random random
occurrence
so what we've done now is we basically
formulated the bandit as an infinite
markov decision process over information
states it's infinite because alpha and
beta in this example for instance can
continue to grow indefinitely right so
we're never actually returning to
exactly the same state that we were
before
you can of course change the algorithm
the agent and some agents might actually
return to the same internal state as
they were before but in this case it
would it wouldn't wouldn't ever loop it
would continue indefinitely but it is a
well-defined markov decision process
it's just an infinitely large one and we
can still think about how to solve this
using reinforcement learning techniques
for instance we could learn a bayesian
reward distribution and then use
planning
we only need to learn the reward
distribution because everything else is
given we know how we will update the
information state
given a certain reward so if we know the
likelihood of each reward happening then
we can use that to plan into the future
and this allows us then to
um
reason about which action to take and
therefore to explore
this is known as bayes adaptive
reinforcement turning and turns out if
you do that so if you learn a bayesian
reward distribution so you're trying to
learn what the reward could be the
expected reward could be and then you
take that into accounts
and you use it in your planning
algorithm
and then you plan infinitely far into
the future
all the way until the end of time and
then you use that plan to select the
action
this turns out to optimally trade off
exploration and exploitation with
respect to your distribution
now obviously this this this can
actually be extended to full rl by also
learning a transition model but it
becomes very unwieldy very quickly
i uh already mentioned planning
indefinitely into the future
and it's a little bit unclear how to
scale this effectively that doesn't mean
that it can't be done it just means that
it's not immediately obvious how to do
that
i do want to i did want to mention it
because it might be an interesting
approach for future research
okay now we're going to go back
to one final example
okay so for our final example i'll go
back to the blackboard and we see here a
variation of the simple problem that we
saw before as well
and essentially what we're going to ask
is this question of which action are we
going to select next if we've seen this
information so far
now as i mentioned at the beginning
someone intuitively maybe you would say
well
action b seems to be the right action
but i'm going to ask you to be a bit
more specific and i want you to think
about this so what i'm going to ask you
is what is the probability the actual
probability of on time step 3
picking action a
and
for a different algorithm so let's
consider the greedy algorithm
let's consider epsilon greedy as well
ucb
and sometimes happen
you can pause the video now and think
about this for a second and then you can
compare your answers to the answers that
i'm going to give
and if the answers don't line up if you
have a different answer but you think
yours is definitely correct and mine is
wrong please let me know
okay then now i'm going to give the
answers that i would give so the greedy
one is quite easy um
this is the example i gave all the way
at the beginning of the lecture as well
if we've only ever seen a reward of zero
for action a and a reward of plus one
for action b
assuming that we're going to estimate
the action values by simply averaging
the reward so far then clearly action b
will be degree 1. so the probability of
selecting action a is simply going to be
0.
for epsilon gradients quite similar
except that we're going to pick the
greedy actually only with probability
one minus epsilon and all of the other
actions with probability
epsilon
now the only subtlety here in some sense
is that the
probability of selecting action is then
not epsilon but it's epsilon divided by
two because picking uniformly at random
means we are also going to pick
b with probability epsilon divided by 2
whenever we pick a random action
now for ucb
what can we say for ucb ucb is not
actually a randomized algorithm it's a
deterministic algorithm so we're just
going to add
a bound to each of these actions
and ucb in its general form has some
hyperparameters c that gets multiplied
with the bound but in this specific case
that doesn't actually matter because
both actions got selected exactly once
which means that the bound for both
actions is exactly the same
so adding the same number to the
action as estimates doesn't actually
change which action will be grading so
that means that ucb in this case will
also not select action a
the difference is and this is something
i encourage you to think about now
on the next time step say that we get a
reward of zero for picking action b
what then happens to ucb
is that the bound for action b will go
down whereas the bound for action a will
go up
depending on your hyper parameter c
at some point action a will get
preferred whereas greedy just keeps on
picking action b indefinitely
and then for thomson sampling i didn't
actually specify exactly what the
distributions are but let's assume the
same as we did in the slides let's
assume we have a beta distribution which
is uniform between zero and one
and then we've that we've only seen this
data so far
so then it turns out for thompson
sampling the
posterior distributions will essentially
look something like this for action a
it'll look
something like this and for action b
it'll look
the opposite
sorry
not from zero to zero but from zero to
one
and if you then consider both these uh
uh these distributions these belief
distributions
and then you can consider you should
consider the likelihood
that action a is optimal compared to
action b turns out that's
if i did my math correctly
one quarter
you can feel free to check this
of course always keep open the
possibility that i made some small
mistakes somewhere and please do let me
know if you think i did
um but it's interesting to know the
differences also between ucb and
thomson's happening basically because i
discussed both these algorithms as being
optimal in some sense
in the sense that they uh over time get
logarithmic regrets that doesn't mean
that they always select exactly the same
actions so on this third time step ucb
will never select action a again
on the fourth time step it might
but in the third time it will definitely
select action b whereas thompson
sampling would actually select action a
25 percent of the time
that means that the total sequence of
actions and rewards that thompson
sampling gets could differ and in fact
are random whereas for ucb they're only
random because of the rewards but not
because of the policy
however it turns out that over time ucb
and thomson sampling select actions
under a similar ratio and this is why
they both have the same
regret so even though on every time step
they might be quite different in terms
of their choices
on the whole they select according to
similar ratios now of course you can
change this by tweaking the exploration
parameter c for ucb or by changing the
prior distributions that are used for
thomson sampling and these might change
things quite a bit
okay so that brings us to the end of
this lecture
now i'm going to go
back to the slides and i have nothing
more to say there except that this is
the end of the lecture
in future lectures just to recap this
lecture we assumed that there was only
one state in the environment
and we were going to optimize our policy
in that setting in future lectures we
will go back to the full reinforcement
learning case and first we're going to
consider how one can reason about the
values and how come on one planning
those values and then after this we're
going to consider how to do sample based
algorithms that can really learn from
scratch
and for now i thank you for your
attention