# Reinforcement Learning with Neural Network

This is the second iPython notebook from my series on Reinforcement Learning (RL).
In this notebook I will deal with Approximate Solution Methods for RL. In order to do so I will closely follow the second edition of an amazing book by Sutton and Barto on Reinforcement Learning; a draft of the book can be downloaded [Here](http://incompleteideas.net/book/the-book-2nd.html). The content of this notebook roughly corresponds to the main ideas presented in much more detail in the second part, Approximate Solution Methods, of the above book. Some of the presented algorithms are implemented in the examples, usually following the theoretical explanation. For the examples to run you need to have installed NumPy, AIGym, and of course iPython libraries.

I will assume that the reader have read either my first notebook on RL or the first part, Tabular Solution Methods, of the above book; and also that the reader is familiar with artificial neural networks.

## Setting up the scene

The only difference between the tabular methods presented in the first notebook, and the approximate methods I will present here is that the value function is not a table but rather a parametrised functional with **weight vector**, $\textbf{w} \in \mathbb{R}^{n}$. A functional is a map from a vector space into a field of scalars of this vector space, for more details see e.g. Wikipedia.
Since this part is concerned only with approximate solutions, to denote an approximate value of a state-value function, $v_{\pi}$, of a state, $s$, I will use the following notation $\widehat{v}(s,\textbf{w})$ ($\approx v_{\pi}(s)$).

### Weight vector

As a simple example, one can consider a linear function from the feature space to real numbers. The vector $\textbf{w}$, in this case, would precisely be the weight vector of features.
Or as mathematician would put it: Let both, the feature vector of states, denoted simply by $s$, and the weight vector, $\textbf{w}$, be elements of $\mathbb{R}^{n}$; and let the map,
\begin{equation}
(\cdot, \cdot): \mathbb{R}^{n} \times \mathbb{R}^{n} \rightarrow \mathbb{R} \,,
\end{equation}
be a scalar product on $\mathbb{R}^{n}$. Then the weight vector together with a scalar product define a functional,
\begin{equation}
(\textbf{w}, \cdot): \mathbb{R}^{n} \rightarrow \mathbb{R} \,,
\end{equation}
which is our state-value function, $\widehat{v}(\cdot) = (\textbf{w}, \cdot)$; and since every scalar product is a bi-linear map this function is linear.

Another, more relevant for us, example of $\widehat{v}$ might be the function computed by multi-layer artificial neural network (ANN); where the weight vector $\textbf{w}$ denotes the set of all connection weights of the ANN. The next issue popping-up is how one could update the weight vector.

### Weight vector update

Recall that in the tabular case, all updates to the estimates of value functions were moving the value towards an update target, i.e. in case of TD(0) the update took the form of
\begin{equation}
V(S_{t}) \leftarrow V(S_{t}) + \alpha \big[ R_{t+1} + \gamma V(S_{t+1}) - V(S_{t}) \big] \,.
\end{equation}
Thus, in case of tabular methods, the value function for a state $S_{t}$ was simply moved towards the update target, $u = R_{t+1} + \gamma V(S_{t+1})$. In the case of approximation, one must update the weight vector such that the new estimate for $s$ is more like $u$. If the update target, $u$, is a number this procedure is called function approximation. I will follow the notation of Sutton and Barto and denote this process as $s \mapsto u$.

Having said this, one can think of the updates as desired input-output behaviour, i.e. the estimated value for $s$ should be more like the update target $u$. Looking at the problem from this angle, one immediately realises that any of a wide range of function approximation methods for value prediction could, in theory, be used; e.g. ANN.

### Cost function

In comparison to tabular methods where no cost function was required because the learned value function could, in principle, become equal to the true value function; and changing the value function at one state did not affect any other state. In case of approximation, on the other hand, the state space is often continuous and hence infinite (or very large) and the weight vector is, by assumption, finite dimensional (or is of smaller dimensionality than the state space). This in general implies that making the estimate of one state more accurate makes others less accurate.

One can define a distribution, $\mu(s) \geq 0$ such that $\sum_{s} \mu(s) = 1$, representing how much one cares about the error in a state, $s$. The error is defined to be the square of the difference between the approximate value and the true value. Putting this together yields the **cost function**
\begin{equation}
\overline{VE}(\textbf{w}) = \sum_{s} \mu(s) \big[ v_{\pi}(s) - \widehat{v}(s, \textbf{w}) \big]^{2} \,.
\end{equation}

In general, the true value $v_{\pi}(s)$ is unknown, and therefore the above update can't be performed exactly. It can however be approximated, e.g. by update targets described in the previous paragraph.

### Semi-gradient Method

I will now assume that the sampled examples follow the same distribution, $\mu$, which was mentioned in the definition of the cost function, $\overline{VE}$; in other words I will assume on-policy methods. This allows me to re-write the cost function as an expected value
\begin{equation}
\begin{split}
\overline{VE}(\textbf{w}) =& \sum_{s} \mu(s) \big[ v_{\pi}(s) - \widehat{v}(s, \textbf{w}) \big]^{2} \\
=& \mathbb{E}_{\pi} \big[ v_{\pi}(S_{t}) - \widehat{v}(S_{t}, \textbf{w}) \big]^{2} \,,
\end{split}
\end{equation}
where $S_{t}$ denotes a sampled state obtained by following the policy $\pi$. Having said this, I can continue by useing **stochastic gradient descent** (SGD) methods to update weight vectors, namely
\begin{equation}
\begin{split}
\textbf{w}_{t+1} &= \textbf{w}_{t} - \frac{1}{2} \alpha \nabla \big[ v_{\pi}(S_{t}) - \widehat{v}(S_{t}, \textbf{w}) \big]^{2} \\
&= \textbf{w}_{t} + \alpha \big[ v_{\pi}(S_{t}) - \widehat{v}(S_{t}, \textbf{w}) \big] \nabla\widehat{v}(S_{t}, \textbf{w}) \,.
\end{split}
\end{equation}
Where $\nabla$ is a derivative, i.e. $\nabla f(\textbf{w})$ is the gradient$^{\dagger}$ of the function, $f$. Note that the second line of the above equation assumes that the update target, $v_{\pi}$, is independent of $\textbf{w}$. When the true value of $v_{\pi}$ is unknown, an estimate, $U_{t}$ can be used instead
\begin{equation}
\textbf{w}_{t+1} = \textbf{w}_{t} + \alpha \big[ U_{t} - \widehat{v}(S_{t}, \textbf{w}) \big] \nabla\widehat{v}(S_{t}, \textbf{w}) \,.
\end{equation}
If for all time-steps, $t$, $U_{t}$ is an unbiased estimate, i.e. $\mathbb{E}[U_{t}|S_{t}=s] = v_{\pi}(S_{t})$, then $\overline{VE}(\textbf{w}_{t})$ is guaranteed to converge to a local optimum under certain conditions on the learning parameter, $\alpha$. (See equation 2.7 in Sutton and Barto book.)

Unfortunately, ignoring all what was said about the independence of update target, and blindly substituting a bootstrapping estimate $U_{t}(\textbf{w}_{t})$ for $v_{\pi}$ doesn't provide the same robust convergence guaranties. There are two problems with it; the first is due to the fact that $U_{t}(\textbf{w}_{t})$ is biased; and the other problem exists because the second equality in the weight update equation above does not hold. ($\nabla$ would also need to act on $U_{t}(\textbf{w}_{t})$, which results in a disaster!) Therefore bootstrapping methods are not instances of SGD but rather of **semi-gradient methods**.

$^{\dagger}$ Strictly speaking it is a one-form not a vector, but since here I care only about Euclidean spaces it's okay. :D

## Digression:
Before delving into the details of implementation first I need to introduce two concepts, ValueFunction class and the openAI gym. Let me start with the former.

### ValueFunction class

In order to implement the algorithms presented here I will use ValueFunction class implemented in the ValueFunctino.py file. This class will take care of computations needed to choose epsilon greedy action, compute value function of a state-action pair or to perform the gradient descent update. For example, to compute the actin-value function, the value function takes only state as an argument, and returns an array of action values for each posible action from the given state. The constructor of this class takes three arguments:
* **in_len**: this parameter sets the length of the state vector the class should expect,
* **out_len**: this parameter sets how many outputs should the value function have, i.e. number of actions in a state,
* **degree**: this determines if higher order features are created from state vector, e.g. degree=2 tells the class to create second order features from the state vector passed to member functions.

There is a handfull of member functions I will use, namely
* **computeVF**: as already mentioned, this function takes a state as an argument, and returns an array of values for each action
* **epsGreedyPolicy**: this function takes two arguments, state and epsilon, and returns an action chosen $\epsilon$-greedily
* **softmaxPolicy**: this function takes two arguments, state and temperature, and returns an action, more on this later.
* **trainSemiGrad**: takes tour arguments, namely state, action, TD error, and learning rate. The arguments correspond exactly to the elements $S_{t}, A_{t}, \delta,$ and $\alpha$ in the equation $\textbf{w}_{t+1} = \textbf{w}_{t} + \alpha \delta \nabla \widehat{q}(S_{t}, A_{t}, \textbf{w}_{t})$.
* **trainEligibTraceSemiGrad**: takes six arguments, namely state, action, TD error, discount, decay factor, and learning rate. The arguments correspond exactly to the elements of the eligibility trace update equation  discussed after the Example 2.
* **trainDutchTraceSemiGrad**: same as trainEligibTraceSemiGrad, but uses dutch traces for training.
* **resetTraces**: this member function resets the all traces used for training to zero. It needs to be called at the beginning of each episode.

### Open AI gym

There is a number of environments in the open AI gym to train your algorithms on. One of them, CartPole, will be used in the following examples. The very first step is to import the gym; next to be done is to create the environment as follows
```python
env = gym.make('CartPole-v0')
```

The task for CartPole is to balance a pole on a cart for as long as possible by accelerating the cart right or left. At each time-step, the agent receives a reward of +1; and an episode ends when the state of the environment reaches a point from which fall of the pole is inevitable, e.g. the angle is too big, or when the pole is balanced for 200 steps without falling. The environments have attributes **action_space** and **observation_space**, returning the space of all actions and the type of state vector, respectively. Let me now introduce three environment's member functions which will be used
* **reset**: this function resets the environment, and returns an initial state vector,
* **step**: takes action as an argument and returns state vector, reward, done, and info
* **render**: this renders the environment on the screen.

The third element returned by the step member function, done, is a boolean variable indicating the end of episode. The fourth one, info, is a dictionary which in some environments may provide extra debug information, unless you want to cheat you should not use it for training. For CartPole specifically, the state vector is an array of length 4 with the elements denoting:
* displacement of the cart from the middle,
* velocity of the cart,
* angle by which the pole deviates from the vertical position, and
* angular velocity of the pole.

These numbers are taken with respect to the standard coordinate system; i.e. positive displacement is to the right, positive angle denotes clockwise deviation, and similarly for the other two.
The action space had only two allowed actions, i.e. accelerate left (action $0$) and accelerate right (action $1$).

### Example 1: CartPole - Benchmarking

If you have ever tried to balance a stick on your hand, you might have come up with many different strategies, I'll consider three
* Strategy 1: place the stick as vertical as possible and then hope for the best.
* Strategy 2: try to hold still, i.e. keep  the velocities low, and correct for only angle deviation.
* Strategy 3: move your hand from side to side at a certain speed depending on the length of the stick in a pendulum like way - if you are a physicist you know exactly what I'm talking about.

You will probably agree with me that the first strategy is quite dumb but let's give it a shot anyway. Since the CartPole environment doesn't have an action hope for the best, i.e. do nothing, I will pick actions randomly. The second one might seem reasonable. In order to translate it into the CartPole problem one might want to accelerate the cart to the side of the angle deviation of the pole. The last strategy takes into account also the angular velocity of the pole. Let's see what experiments will have to say about these.

```python
# In case you restart the kernel and wish to continue with later examples run this cell again.
import gym
import numpy as np
from ValueFunction import ValueFunction
from HelperFunctions_ASM import printTrainingReturns, printReturns, printHighestLowest
from HelperFunctions_ASM import getPerformanceVF

# For rendering in a separate window use
from HelperFunctions_ASM import visualizeVF

# For inline rendering (e.g. on a server) use the following
# %matplotlib inline
# from HelperFunctions_ASM import visualizeVF_inline as visualizeVF

# Create the CartPole environment.
env = gym.make('CartPole-v0')
print("The number of possible action is {0}, and the shape of a feature vector is {1}."\
      .format(env.action_space.n, env.observation_space.shape))
```

    WARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.
    The number of possible action is 2, and the shape of a feature vector is (4,).

**Strategy 1**: pick randomly, i.e. set $\epsilon$ to 1.0.

```python
actionVF = ValueFunction(in_len=4, out_len=2, degree=1)

getPerformanceVF(env, actionVF, epsilon=1, episodes=500)
```

    Performance measure: the return per episode averaged over 500 episodes was 23.1 with the standard deviation 12.0.
    The highest return was 79.0, and the lowest return was 9.0.

```python
visualizeVF(env, actionVF, epsilon=0)
```

    Return of this visualization was 9.0.

Well, this kinda sucks.

**Strategy 2**: if the deviation angle is positive go right, if negative go left.

```python
actionVF = ValueFunction(in_len=4, out_len=2, degree=1)
# Recall that the angle is the third feature in the feature vector.
actionVF.weights = np.array([[ 0, 0], [ 0, 0], [ -1, 1], [ 0, 0]])

getPerformanceVF(env, actionVF, epsilon=0, episodes=500)
```

    Performance measure: the return per episode averaged over 500 episodes was 42.0 with the standard deviation 8.8.
    The highest return was 72.0, and the lowest return was 24.0.

```python
visualizeVF(env, actionVF, epsilon=0)
```

    Return of this visualization was 58.0.

Slightly better but still not quite good.

**Strategy 3**: if the sum of the deviation angle and angular velocity is positive go right, if negative go left.

```python
actionVF = ValueFunction(in_len=4, out_len=2, degree=1)
actionVF.weights = np.array([[ 0, 0], [ 0, 0], [ -1, 1], [ -1, 1]])

# guess why I used only 100 episodes here
getPerformanceVF(env, actionVF, epsilon=0, episodes=100)
```

    Performance measure: the return per episode averaged over 100 episodes was 200.0 with the standard deviation 0.0.
    The highest return was 200.0, and the lowest return was 200.0.

```python
visualizeVF(env, actionVF, epsilon=0)
```

    Return of this visualization was 200.0.

Ho-ho-hoooooo, problem solved! Let's go home. :D

But joking aside, let's get to work.

## Semi-gradient control

So far I was talking only about state-value functions. Now, I would like to move the discussion to action-value functions. The only change is that now I will have "training examples" of the form $S_{t}, A_{t} \mapsto U_{t}$ instead of $S_{t} \mapsto U_{t}$. The gradient descent update (or semi-gradient descent update, in case $U_{t}(\textbf{w}_{t})$) takes the form
\begin{equation}
\begin{split}
\textbf{w}_{t+1} =& \textbf{w}_{t} + \alpha \big[ U_{t} - \widehat{q}(S_{t}, A_{t}, \textbf{w}_{t}) \big] \nabla \widehat{q}(S_{t}, A_{t}, \textbf{w}_{t}) \\
=& \textbf{w}_{t} + \alpha \delta_{t} \nabla \widehat{q}(S_{t}, A_{t}, \textbf{w}_{t})\,,
\end{split}
\end{equation}
where the quantity in the square brackets is the old friend **TD error** denoted $\delta_{t}$. Now, let's move right to the examples.

### one-step SARSA

The semi-gradient update for one-step SARSA is
\begin{equation}
\begin{split}
\textbf{w}_{t+1} =& \textbf{w}_{t} + \alpha \big[ R_{t+1} + \gamma \widehat{q}(S_{t+1}, A_{t+1}, \textbf{w}_{t}) - \widehat{q}(S_{t}, A_{t}, \textbf{w}_{t}) \big] \nabla \widehat{q}(S_{t}, A_{t}, \textbf{w}_{t}) \\
=& \textbf{w}_{t} + \alpha \delta_{t} \nabla \widehat{q}(S_{t}, A_{t}, \textbf{w}_{t})\,.
\end{split}
\end{equation}
The TD error takes the form $\delta_{t} = \big[ R_{t+1} + \gamma \widehat{q}(S_{t+1}, A_{t+1}, \textbf{w}_{t}) - \widehat{q}(S_{t}, A_{t}, \textbf{w}_{t}) \big]$.
I will implement this algorithm in the Example 2, below.

### Example 2: CartPole with one-step SARSA

Finally I got the the point where I will implement one-step SARSA algorithm to decide on which action should be taken in the CartPole environment.

I encourage the reader to play around with the hyper-parameters, e.g. increase the degree in ValueFunction which for a state vector (e.g. $x_{1}$, $x_{2}$) creates additional features ($x_{1}^{2}$, $x_{2}^{2}$, $x_{1} x_{2}$).

```python
# Next line resets the value function, comment it when you want to continue training.
actionVF = ValueFunction(in_len=4, out_len=2, degree=1)

episodes = 100
discount = 0.8
epsilon = 0.1
learning_rate = 0.0005

returns = np.zeros(episodes)

for i in range(episodes):
    done = False
    cum_reward = 0

    state1 = env.reset()
    action1 = actionVF.epsGreedyPolicy(state1, eps=epsilon)

    while not done:
        state2, reward, done, _ = env.step(action1)
        cum_reward = cum_reward + reward
        action2 = actionVF.epsGreedyPolicy(state2, eps=epsilon)
        # if the pole falls give a reward of -5 rather than 1
        # recall that an episode ends after 200 steps even if the pole didn't fall
        if (done and cum_reward != 200): reward = -5
        td_error = reward +discount*actionVF.computeVF(state2)[action2]\
                                   -actionVF.computeVF(state1)[action1]
        actionVF.trainSemiGrad(state1, action1, td_error, learning_rate)
        state1, action1 = state2, action2

    returns[i] = cum_reward

printTrainingReturns(returns)
```

    Performance measure: the return per episode averaged over:
     - the first 10% of the training was 87.7 with the standard deviation 55.6;
     - 10% to 50% of the training was 78.3 with the standard deviation 30.9;
     - 50% to 90% of the training was 76.3 with the standard deviation 26.8;
     - the last 10% of the training was 75.4 with the standard deviation 26.6.

```python
getPerformanceVF(env, actionVF, epsilon=0, episodes=500)
```

    Performance measure: the return per episode averaged over 500 episodes was 79.7 with the standard deviation 30.2.
    The highest return was 197.0, and the lowest return was 40.0.

```python
visualizeVF(env, actionVF, epsilon=0)
```

    Return of this visualization was 67.0.

```python
# Printing weight wector:
print(actionVF.weights)
```

    [[-0.00770507 -0.02782745]
     [ 0.17972142 -0.20323063]
     [ 0.00691165 -0.01387499]
     [-0.23032698  0.21739531]]

## Eligibility Traces for Approximate Solution Methods

For approximate solutions, the discussion of eligibility traces is analogous to the one in my first notebook. The performance as well as efficiency of a learning method is improved by considering **$n$-step updates**. The update, this time to the weight vector, $\textbf{w}$, rather than to the state value function. Using the $n$-step method looks as follows
\begin{equation}
\begin{split}
\textbf{w}_{t+n} &= \textbf{w}_{t+n-1} + \alpha \big[ G_{t:t+n} - \widehat{v}(S_{t}, \textbf{w}_{t+n-1}) \big] \nabla \widehat{v}(S_{t}, \textbf{w}_{t+n-1}) \,, \\
G_{t:t+n} &= \sum_{i=1}^{n} \gamma^{i-1} R_{t+i} + \gamma^{n} \widehat{v}(S_{t}, \textbf{w}_{t+n-1}) \,,
\end{split}
\end{equation}
where $G_{t:t+n}$ is the **$n$-step return**. For eligibility traces, the value function is being updated using a weighted sum of all $n$-step returns with weights summing up to 1 rather than just using one $n$-step return. For the choice of weights $\frac{\lambda}{1-\lambda}$, where $\lambda$ is called **decay factor**; the corresponding return,
\begin{equation}
G_{t}^{\lambda} = (1-\lambda) \sum_{n=1}^{\infty} \lambda^{n-1} G_{t:t+n} \,,
\end{equation}
is called **$\lambda$-return**. For episodes terminating at a terminal state, $S_{t}$, all subsequent $n$-step returns are equal to $G_{t}$. In such a case the above equation is modified to
\begin{equation}
G_{t}^{\lambda} = (1-\lambda) \sum_{n=1}^{T-t-1} \lambda^{n-1} G_{t:t+n} + \lambda^{T-t-1}G_{t} \,.
\end{equation}

### SARSA($\lambda$)

I will now modify the tabular **SARSA($\lambda$)** algorithm so that it is suitable for approximate solution methods. Recall that the eligibility trace for tabular methods is a vector, $z$, of the same shape as the value function. For the approximate solution methods, the eligibility trace is again a vector, $z$, but this time of the same shape as the weight vector $\textbf{w}$.

At the beginning of the episode, the eligibility trace vector (ETV) is initialised to zero. Then, on each time-step, $t$, ETV is incremented by the value gradient, and then fades away by $\gamma\lambda$ (a product of the discount and decay factors). More precisely
\begin{equation}
\begin{split}
\textbf{z}_{-1} &= 0 \,,\\
\textbf{z}_{t} &= \gamma\lambda \textbf{z}_{t-1} + \nabla \widehat{q}(S_{t}, A_{t}, \textbf{w}_{t}) \,.
\end{split}
\end{equation}
At each time-step $t$, the action-value function is updated as follows
\begin{equation}
\begin{split}
\delta_{t} &= R_{t+1} + \gamma \widehat{q}(S_{t+1}, A_{t+1}, \textbf{w}_{t}) - \widehat{q}(S_{t}, A_{t}, \textbf{w}_{t}) \,,\\
\textbf{w}_{t+1} &= \textbf{w}_{t} + \alpha\delta_{t} \textbf{z}_{t} \,,
\end{split}
\end{equation}
where, as usual, $\delta_{t}$ denotes the TD error at the time-step, $t$. To get the corresponding algorithm for state-value functions, called **TD($\lambda$)**, just replace $\widehat{q}(\cdot, \cdot, \textbf{w})$ by $\widehat{v}(\cdot, \textbf{w})$, and eliminate the actions, $A_{t}$ and $A_{t+1}$.

The discussion about different values of the decay parameter $\lambda$ takes over to the approximate solution methods:
- if $\lambda = 0$, then the eligibility trace is precisely the value gradient corresponding to the state-action pair $(S_{t}, A_{t})$; and the whole algorithm reduces to the one-step semi-gradient TD method.
- if $0 < \lambda < 1$, then more of the preceding states are updated; but due to the decay factor, the states further back in time are updated less and less.
- if $\lambda = 1$, then the resulting algorithm precisely corresponds to discounted Monte Carlo.

### Example 3: CartPole with SARSA($\lambda$)

Now, I will extend the above Example 2 by using SARSA($\lambda$) algorithm. Implementation of the weight vector update using eligibility trace is hidden in the ValueFunction class function **trainEligibTraceSemiGrad**. I encourage the reader to experiment with the hyper-parameters, to see for example that setting decay_factor to 0 is equivalent to one-step SARSA.

The training 300 episodes; the learning rate starts at 0.001, and then is divided by 2 every 50 episodes. In the current setting, adding more episodes does not results in any performance gains.

```python
# Next line resets the value function, comment it when you want to continue training.
actionVF = ValueFunction(in_len=4, out_len=2, degree=2)

episodes = 300
discount = 0.8
decay_factor = 0.9
epsilon = 0.1

returns = np.zeros(episodes)

for i in range(episodes):
    learning_rate = 0.001/2**(i//50)
    done = False
    cum_reward = 0

    actionVF.resetTraces()
    state1 = env.reset()
    action1 = actionVF.epsGreedyPolicy(state1, eps=epsilon)

    while not done:
        state2, reward, done, _ = env.step(action1)
        cum_reward = cum_reward + reward
        action2 = actionVF.epsGreedyPolicy(state2, eps=epsilon)
        # if the pole falls give a reward of -5 rather than 1
        # recall that an episode ends after 200 steps even if the pole didn't fall
        if (done and cum_reward != 200): reward = -5
        td_error = reward +discount*actionVF.computeVF(state2)[action2]\
                                   -actionVF.computeVF(state1)[action1]
        actionVF.trainEligibTraceSemiGrad(state1, action1, td_error, discount, decay_factor, learning_rate)
        state1, action1 = state2, action2

    returns[i] = cum_reward

printTrainingReturns(returns)
```

    Performance measure: the return per episode averaged over:
     - the first 10% of the training was 88.8 with the standard deviation 42.1;
     - 10% to 50% of the training was 85.8 with the standard deviation 35.5;
     - 50% to 90% of the training was 78.9 with the standard deviation 25.7;
     - the last 10% of the training was 86.5 with the standard deviation 27.9.

```python
getPerformanceVF(env, actionVF, epsilon=0, episodes=500)
```

    Performance measure: the return per episode averaged over 500 episodes was 87.3 with the standard deviation 31.2.
    The highest return was 200.0, and the lowest return was 47.0.

```python
visualizeVF(env, actionVF, epsilon=0)
```

    Return of this visualization was 59.0.

### Expected SARSA($\lambda$)

A slight modification to SARSA($\lambda$) algorithm gives the one-step Expected SARSA($\lambda$). Its semi-gradient update takes the following form
\begin{equation}
\textbf{w}_{t+1} = \textbf{w}_{t} + \alpha \big[ R_{t+1} + \gamma \sum_{a} \pi(a \,|\, S_{t+1}) \widehat{q}(S_{t+1}, a, \textbf{w}_{t}) - \widehat{q}(S_{t}, A_{t}, \textbf{w}_{t}) \big] \nabla \widehat{q}(S_{t}, A_{t}, \textbf{w}_{t}) \,.
\end{equation}

Since in the CartPole environment there are only two actions to choose from, hence this algorithm wouldn't add much more value than plain SARSA($\lambda$). (The room and necessity for exploration is much smaller than in an environment with many actions.)

### True online TD($\lambda$) (dutch trace)

As already written in my first notebook, the **true online TD($\lambda$)** is currently the best performing TD algorithm (this holds only for tabular methods and for linear function approximation). I'm not going to derive the algorithm here. (Alright, you guessed it. I'm too lazy. :D ) Hence, I will only state the update rules below. However I encourage the reader to consult Sutton and Barto book or the actual paper by Seijen et al., where this algorithm was introduced. Link to arXiv [here](https://arxiv.org/abs/1512.04087).

As usual, ETV is initialised to zero at the beginning of each episode; then at each time-step, $t$, it is updated as follows
\begin{equation}
\begin{split}
\textbf{z}_{-1} =& 0 \,, \\
\textbf{z}_{t} =& \gamma \lambda \textbf{z}_{t-1} + (1 - \alpha \gamma \lambda \, \textbf{z}_{t-1}^{\top} \!\cdot\! \textbf{x}_{t}) \textbf{x}_{t} \\
=& \gamma \lambda \textbf{z}_{t-1} + \textbf{x}_{t} (1 - \alpha \gamma \lambda \, \textbf{z}_{t-1}^{\top} \!\cdot\! \textbf{x}_{t}) \,,
\end{split}
\end{equation}
where $^{\top}$ denotes transpose of a vector, and $\cdot$ denotes matrix multiplication. $\textbf{x}_{t}$ is the feature vector of $S_{t}$ which is actually, in the linear case, the gradient of state-value function, $\nabla \widehat{v}(S_{t}, \textbf{w}_{t})$. The reason why I swapped the multiplication in the last term of the last line will become clear soon.

The update rule for the weight vector reads
\begin{equation}
\begin{split}
\textbf{w}_{t+1} =& \textbf{w}_{t} + \alpha \delta_{t} \textbf{z}_{t} + \alpha \big( (\textbf{w}_{t}^{\top} - \textbf{w}_{t-1}^{\top}) \!\cdot\! \textbf{x}_{t} \big) (\textbf{z}_{t} - \textbf{x}_{t}) \\
=& \textbf{w}_{t} + \alpha \delta_{t} \textbf{z}_{t} + \alpha (\textbf{z}_{t} - \textbf{x}_{t}) \big( (\textbf{w}_{t}^{\top} - \textbf{w}_{t-1}^{\top}) \!\cdot\! \textbf{x}_{t} \big) \,,
\end{split}
\end{equation}
where $\delta_{t}$ is as usual; and again the swap in the last term.

### True online SARSA($\lambda$) (dutch trace)

So far so good, though the above algorithm is not particularly helpful, because I would need the version with action-value function, not state-value function. Luckily, to modify it is not a big deal; one just needs to realise two things
* The first one that the feature vector, $\textbf{x}_{t}$, is just the gradient of $\nabla \widehat{v}(S_{t}, \textbf{w}_{t})$ (because of linearity of the algorithm);
* The other thing is obvious, use the gradient of action-value function, $\nabla \widehat{q}(S_{t}, A_{t}, \textbf{w}_{t})$, instead of gradient of state-value function, $\nabla \widehat{v}(S_{t}, \textbf{w}_{t})$.

They yield the following update equations
\begin{equation}
\begin{split}
\textbf{z}_{t} =& \gamma \lambda \textbf{z}_{t-1} + \nabla \widehat{q}(S_{t}, A_{t}, \textbf{w}_{t}) \!\cdot\! \big( I - \alpha \gamma \lambda \, \textbf{z}_{t-1}^{\top} \!\cdot\! \nabla \widehat{q}(S_{t}, A_{t}, \textbf{w}_{t}) \big) \,, \\
\textbf{w}_{t+1} =& \textbf{w}_{t} + \alpha \delta_{t} \textbf{z}_{t} + \alpha \big( \textbf{z}_{t} - \nabla \widehat{q}(S_{t}, A_{t}, \textbf{w}_{t}) \big) \!\cdot\! ( \textbf{w}_{t}^{\top} - \textbf{w}_{t-1}^{\top} ) \!\cdot\! \nabla \widehat{q}(S_{t}, A_{t}, \textbf{w}_{t}) \,,
\end{split}
\end{equation}
where $I$ is the identity matrix. The TD error is
\begin{equation}
\delta_{t} = R_{t+1} + \gamma \widehat{q}(S_{t+1}, A_{t+1}, \textbf{w}_{t}) - \widehat{q}(S_{t}, A_{t}, \textbf{w}_{t}) \,.
\end{equation}

Now comes the reason why I had to swap the two factors in the last terms of $\textbf{w}_{t}$ and $\textbf{z}_{t}$ updates. In case of action-value function, the weight vector becomes an $m \times n$ matrix with $m$ equal to the number of features in a feature vector, and $n$ equal to the number of possible actions. This is also true for the eligibility trace as well as for the gradient of action-value function. Hence, the terms $\textbf{z}^{\top} \!\cdot\! \nabla \widehat{q}$ and $\textbf{w}^{\top} \!\cdot\! \nabla \widehat{q}$ are no longer numbers but rather $n \times n$ matrices.

To get a full understanding of these adaptations, the reader is welcomed to have a look on the exact derivation of this algorithm in the appendix of the paper by Seijen et al.

### Comments on Action Selection

So far I have always used $\epsilon$-greedy policy to select actions. Such selection has one major drawback - the action selection is in general a highly discontinuous function of action-value estimates; meaning that a small change in action values may result in a different action being maximal, and hence to be prefered by the $\epsilon$-greedy policy. This in general results in a dramatic slow-down of the learning process. To smoothen out the action selection process, I will use a **soft-max** policy with temperature parameter instead. This policy is defined by the following equation
\begin{equation}
\pi( A \,|\, S, \textbf{w}) = \frac{\exp\big( \frac{1}{t} \widehat{q}(S, A, \textbf{w}) \big)}{\sum_{a} \exp \big( \frac{1}{t} \widehat{q}(S, a, \textbf{w}) \big)} \,.
\end{equation}
If the temperature parameter, $t$, is set to 1 then one gets the plain soft-max policy. It is not a good idea to use the plain soft-max poilicy in training because it would result in action-value estimates converging to their true values. However, by decreasing the temperature parameter one can get much faster convergence. The downside of this all is one more hyperparameter to tweak; this one, though, must be a decreasing function of episodes.

In the next example I will implement the true online SARSA($\lambda$) algorithm, but this time I will use soft-max policy with temparature parameter. Doing so will dramatically increase the performance.

### Example 3: CartPole with True online SARSA($\lambda$) (Dutch trace)

In this example I will implement the true online SARSA($\lambda$) (or simply SARSA($\lambda$) with Dutch traces) to solve the CartPole problem. The implementation of the weight vector update using Dutch trace is hidden in the ValueFunction member function **trainDutchTraceSemiGrad**. I will also use the soft-max policy with decreasing temperature during the training. If one uses the $\epsilon$-greedy policy instead of soft-max, the algorithm gives pretty much the same result as SARSA($\lambda$) with ordinary eligibility trace.

The structure of this training is best understood if one thinks of it as nested loops of 4 epochs each consisting of 400 'episodes'. The epoch number is denoted by j, and the episode number is denoted by k. The helper variable, $lr$, begins at 0.005 and is decremented by 0.001 on each passing epoch. The learning rate is equal to the helper variable, $lr$, divided by 2 each 50 episodes.
Note that $\frac{k+1}{400}$ is just the percentage fraction of the current 'episode', $k$. Multiplying this fraction by $2^{j+3}$, and taking the reciprocal value gives the temperature parameter. The idea behind this is that the last 'episode' in the loop strengthens the action with highest action-value function exactly $2^{j+3}$-times.

```python
# Next line resets the weights to zeros, comment when you want to continue training.
actionVF = ValueFunction(in_len=4, out_len=2, degree=2)

episodes = 1600
discount = 0.8
decay_factor = 0.9

returns = np.zeros(episodes)

for i in range(episodes):
    j, k = i//400, i%400
    lr = 0.005 - 0.001*j
    learning_rate = lr / 2**(k//50)
    temperature = 400/((k+1)*2**(j+3))

    done = False
    cum_reward = 0

    actionVF.resetTraces()
    state1 = env.reset()
    action1 = actionVF.softmaxPolicy(state1, temperature)

    while not done:
        state2, reward, done, _ = env.step(action1)
        cum_reward = cum_reward + reward
        action2 = actionVF.softmaxPolicy(state1, temperature)

        # if the pole falls give a reward of -5 rather than 1
        # recall that an episode ends after 200 steps even if the pole didn't fall
        if (done and cum_reward != 200): reward = -5
        td_error = reward +discount*actionVF.computeVF(state2)[action2]\
                                   -actionVF.computeVF(state1)[action1]
        actionVF.trainDutchTraceSemiGrad(state1, action1, td_error, discount, decay_factor, learning_rate)
        state1, action1 = state2, action2

    returns[i] = cum_reward

printTrainingReturns(returns)
```

    Performance measure: the return per episode averaged over:
     - the first 10% of the training was 34.1 with the standard deviation 20.7;
     - 10% to 50% of the training was 94.6 with the standard deviation 41.4;
     - 50% to 90% of the training was 183.9 with the standard deviation 43.6;
     - the last 10% of the training was 200.0 with the standard deviation 0.0.

```python
getPerformanceVF(env, actionVF, epsilon=0, episodes=100)
```

    Performance measure: the return per episode averaged over 100 episodes was 200.0 with the standard deviation 0.0.
    The highest return was 200.0, and the lowest return was 200.0.

This algorithm clearly solves the CartPole problem, but for what price - 1600 episodes! After watching the visualisation (next cell) one can see that the algorithm learned a policy lying in between the second and third strategy from the Example 1. Here I use a for loop with 600 loops so that you can see the next challenges - the agent balances the pole well, but sometimes drifts-off the screen.

```python
state = env.reset()
done = False
cum_reward = 0
for _ in range(600):
    action = actionVF.epsGreedyPolicy(state, 0)
    state, reward, done, _ = env.step(action)
    cum_reward = cum_reward + reward
    env.render()
print('Return of this visualization was {0}.'.format(cum_reward))
#visualizeVF(env, actionVF, epsilon=0)
```

    Return of this visualization was 600.0.

## Policy Gradient Methods

In this part I will consider **policy gradient methods**. This approach is a lot different from all what I have talked about so far. Up until now, the policy was always defined in terms of value function, e.g. choose an action with highest action-value estimate - greedy policy. Now however, the policy will be learned so that no value function is needed for action selection. Although value function won't be used to select actions, one might still want to use it to learn the policy parameters, this methods are called **actor-critic methods**. Don't worry if it sounds confusing for now, it will become clear soon.

I will stick with the notation introduced in the Sutton and Barto book, and denote the weight vector for policy gradient method, also called **policy parameter vector**, $\textbf{$\theta$}$. If value function is learned too, it's weight vector stays $\textbf{w}$. The learned policy will be denoted by $\pi(a \,|\, s, \textbf{$\theta$})$, which has to be differentiable with respect to its parameters, i.e. $\nabla \pi(a \,|\, s, \textbf{$\theta$})$ exists and is finite.

To ensure some degree of exploration, the policy should never become deterministic. A common choice of policy for a discrete action space is
\begin{equation}
\pi(a \,|\, s, \textbf{$\theta$}) = \frac{\exp \big( h(s, a, \textbf{$\theta$}) \big) }{\sum_{b} \exp \big( h(s, b, \textbf{$\theta$}) \big) } \,,
\end{equation}
where $h(s, a, \textbf{$\theta$})$ is some numerical preference. In other words, if one uses neural nets, then the activation function of the output layer is, in this case, soft-max function.

A careful reader might have spotted that this is pretty much what I have done in the exercise 3. There, the numerical preference, $h(s, a, \textbf{$\theta$})$, is precisely the action-value function, $\widehat{q}(s, a, \textbf{w})$, itself to which a soft-max function was applied in order to choose action. However, the main difference between the approach from the exercise 3 and the one I will present here lies in the update rule for policy parameter vector, $\textbf{$\theta$}$.

The parameter vector for the policy gradient methods is updated based on the gradient of some suitable **performance measure**, $J(\textbf{$\theta$})$, more precisely
\begin{equation}
\textbf{$\theta$}_{t+1} = \textbf{$\theta$}_{t} + \alpha \widehat{\nabla J(\textbf{$\theta$}_{t})} \,.
\end{equation}
Note that this is gradient ascent, not descent, because one tries to maximise $J(\textbf{$\theta$})$ as opposed to minimise; this will become clear in the next couple of lines.
The performance measure, $J$, depends on the task at hand. For episodic case it is defined as the true state-value function, $v_{\pi_{\textbf{$\theta$}}}(s_{0})$, of some non-random start state, $s_{0}$; and for the continuing case, one never-ending episode, it is the average rate of reward per time step. I will only be concerned about the episodic case. First, let me assume that all episodes start in some definite start state, $s_{0}$. One might think that it is necessary to learn the true state-value function of $s_{0}$ first to be able to perform the gradient step. This turns out to be false; a theoretical result known as **policy gradient theorem** establishes the following equation
\begin{equation}
\nabla J(\textbf{$\theta$}) \propto \sum_{s} \mu(s) \sum_{a} q_{\pi}(s, a) \nabla \pi(a \,|\, s, \textbf{$\theta$}) \,.
\end{equation}
I encourage the reader to see the detailed derivation of this result either in the above-mentioned book or in the [paper](https://papers.nips.cc/paper/1713-policy-gradient-methods-for-reinforcement-learning-with-function-approximation.pdf) by Sutton et al. called Policy Gradient Methods for Reinforcement Learning with Function Approximation. When you go over the derivation, try to argue why it is okay to interchange derivations and sums over action and state spaces in case when these spaces are infinite.

### REINFORCE

So far so good, now I would like to put all the things I was talking about together and derive the **REINFORCE** algorithm first proposed by Williams in 1992. (If you bother to read the paper check out how he came up with the name. ;)
Let me start with an adaptation of the performance measure gradient
\begin{equation}
\begin{split}
\nabla J(\textbf{$\theta$}) \propto& \sum_{s} \mu(s) \sum_{a} q_{\pi}(s, a) \nabla \pi(a \,|\, s, \textbf{$\theta$}) \\
=& \mathbb{E}_{\pi} \big[ \sum_{a} q_{\pi}(S_{t}, a) \nabla \pi(a \,|\, S_{t}, \textbf{$\theta$}) \big] \\
=& \mathbb{E}_{\pi} \big[ \sum_{a} \pi(a \,|\, S_{t}, \textbf{$\theta$}) q_{\pi}(S_{t}, a) \frac{\nabla \pi(a \,|\, S_{t}, \textbf{$\theta$})}{\pi(a \,|\, S_{t}, \textbf{$\theta$})} \big] \\
=& \mathbb{E}_{\pi} \big[ q_{\pi}(S_{t}, A_{t}) \frac{\nabla \pi(A_{t} \,|\, S_{t}, \textbf{$\theta$})}{\pi(A_{t} \,|\, S_{t}, \textbf{$\theta$})} \big] \\
=& \mathbb{E}_{\pi} \big[ G_{t} \frac{\nabla \pi(A_{t} \,|\, S_{t}, \textbf{$\theta$})}{\pi(A_{t} \,|\, S_{t}, \textbf{$\theta$})} \big] \,.
\end{split}
\end{equation}
If the policy, $\pi$, is being followed, then states are encountered precisely in the proportions determined by $\mu(s)$, and the second equality is justified. Weighting each term of the sum over the action space by the probability of taking the actions allows for using sampled action, $A_{t}$, inside the expected value; this justifies the fourth equality. The last equality follows from the fact that the expected value of a return, $G_{t}$, in a state, $S_{t}$, when taking an action, $A_{t}$, is precisely $q_{\pi}(S_{t}, A_{t})$. This last form is exactly what one needs to update the policy gradient vector
\begin{equation}
\begin{split}
\textbf{$\theta$}_{t+1} =& \textbf{$\theta$}_{t} + \alpha \gamma^{t} G_{t} \frac{\nabla \pi(A_{t} \,|\, S_{t}, \textbf{$\theta$})}{\pi(A_{t} \,|\, S_{t}, \textbf{$\theta$})} \\
=&\textbf{$\theta$}_{t} + \alpha \gamma^{t} G_{t} \nabla \ln \big(\pi(A_{t} \,|\, S_{t}, \textbf{$\theta$}) \big) \,,
\end{split}
\end{equation}
where I included the discount factor $\gamma^{t}$ which originates in the policy gradient theorem. To perform the full derivation including the discount factor is slightly cumbersome but doable. Note that the learning rate, $\alpha$, absorbs the proportionality coming from the policy gradient theorem. Note also that the complete return, $G_{t}$, is used in the update; which is not known until the episode ends. Hence, REINFORCE belongs to the group of Monte Carlo algorithms. The vector $\nabla \ln \big(\pi(A_{t} \,|\, S_{t}, \textbf{$\theta$}) \big)$ is often called **eligibility vector**. Let me now implement this algorithm in the Example 4.

### Example 4: CartPole with Policy Gradient

The very first example of policy gradient method I will implement is also the easiest one. It is the REINFORCE algorithm explained in the above paragraph. Again, I will only go for a linear model, since as seen in the Example 1 it's more than sufficient. Moreover, I will use **TensorFlow** to implement this algorithm.

#### REINFORCE class
As usual, the implementation is in a separate file. In order to use this implementation, one needs to create an instance of the **REINFORCE class**. The constructor of this class takes two arguments, number of features in a state vector and number of actions; then it builds up a one layer ANN. The class has two main functions:
* the first one is **chooseAction**, which takes a state vector as an argument;
* the second function, **train**, takes a list of states, actions, and rewards from the entire episode; discount factor; and learning rate. A state must have shape $(1,n)$, where $n$ is the number of features in the state vector.

Next, there are two member functions taking care of resource management:
* **close**, which releases all acquired resources, i.e. it closes the TensorFlow session of the agent; and
* **reset**, which re-acquires new resources, i.e. closes the current session and opens a new one.

For convenience, I also provided **save**, **restore**, and **printParameters**. The first two are used for saving and restoring learned parameters, both take path as an argument. The last one prints values of all trained parameters on the screen, in this case the policy parameter vector.

The last three functions I provide have to do with TensorBoard:
* **createFileWriter**, which takes a path as an argument, and creates TensorFlow FileWriter to save summaries;
* **makeSummary**, which creates summaries, and adds them to the FileWriter; and
* **closeFileWriter**, which flushes everything to disc and closes the file.

For convenience, I import all relevant libraries once again, and create a new CartPole environment.

```python
# In case you restart the kernel and wish to continue with later examples run this cell again.
import gym
import numpy as np
from HelperFunctions_ASM import printTrainingReturns, printReturns, printHighestLowest
from HelperFunctions_ASM import getPerformanceAgent

# For rendering in a separate window use
from HelperFunctions_ASM import visualizeAgent

# For inline rendering (e.g. on a server) use the following
# %matplotlib inline
# from HelperFunctions_ASM import visualizeAgent_inline as visualizeAgent

# Create the CartPole environment.
env = gym.make('CartPole-v0')
```

    WARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.

```python
from REINFORCE import REINFORCE

n_features, n_actions = 4, 2

agent = REINFORCE(n_features, n_actions)
```

```python
agent.reset()                               # Reset the policy vector in the agent.
agent.createFileWriter('./tb/REINFORCE')    # FileWriter is used visualise training in TensorBoard

discount = 0.995
episodes = 250
returns = np.zeros(episodes)

for i in range(episodes):
    learning_rate = 0.01 / 2**(i//50)

    done = False
    state = env.reset()

    states = []
    actions = []
    rewards = []

    while not done:
        action = agent.chooseAction(state[np.newaxis, :])
        states.append(state[np.newaxis, :])
        actions.append(action)
        state, reward, done, _ = env.step(action)
        rewards.append(reward)

    returns[i] = np.sum(rewards)
    if len(rewards) != 200:
        rewards[-1] = -5

    agent.train(states, actions, rewards, discount, learning_rate)

    if i%10 == 0:
        agent.makeSummary(i)                # Save policy vector to disc for later visualisation in TensorBoard

agent.closeFileWriter()                     # Flush everything to disc, and close FileWriter.

printTrainingReturns(returns)
```

    Performance measure: the return per episode averaged over:
     - the first 10% of the training was 67.1 with the standard deviation 29.3;
     - 10% to 50% of the training was 188.3 with the standard deviation 27.5;
     - 50% to 90% of the training was 190.2 with the standard deviation 17.5;
     - the last 10% of the training was 188.1 with the standard deviation 18.7.

```python
getPerformanceAgent(env, agent, 100)
```

    Performance measure: the return per episode averaged over 100 episodes was 190.2 with the standard deviation 18.8.
    The highest return was 200.0, and the lowest return was 92.0.

```python
visualizeAgent(env, agent)
```

    Return of this visualization was 178.0.

```python
agent.printParameters()
```

    policy/weights:0
    [[-1.8685938  1.8684399]
     [-2.122431   2.122391 ]
     [-3.7279317  3.7280452]
     [-8.071732   8.071443 ]]

Now, one can visualise the graph and saved summaries in TensorBoard, which can be opened from terminal by typing the following command.
```terminal
tensorboard --logdir=dir_to_this_notebook/tb/
```
After successfull lounched of TensorBoard, type *http://localhost:6006* in your browser.

### REINFORCE with Baseline

Because one can write the following equation
\begin{equation}
\begin{split}
0 =& b(s) \nabla 1 \\
=& b(s) \nabla \sum_{a} \pi(a \,|\, s, \textbf{$\theta$}) \\
=& b(s) \sum_{a} \nabla \pi(a \,|\, s, \textbf{$\theta$}) \,,
\end{split}
\end{equation}
it is possible to add one more term, called **baseline**, into the gradient of the performance measure, i.e.
\begin{equation}
\nabla J(\textbf{$\theta$}) \propto \sum_{s} \mu(s) \sum_{a} \big( q_{\pi}(s, a) - b(s) \big) \nabla \pi(a \,|\, s, \textbf{$\theta$}) \,.
\end{equation}
The corresponding policy gradient vector update,
\begin{equation}
\textbf{$\theta$}_{t+1} =\textbf{$\theta$}_{t} + \alpha \gamma^{t} \big( G_{t} - b(s) \big) \nabla \ln \big(\pi(A_{t} \,|\, S_{t}, \textbf{$\theta$}) \big) \,,
\end{equation}
yields a strict generalisation of the REINFORCE algorithm. To see this take $b(s) \equiv 0$.

The aim of doing all this is to reduce variance, and thus making the learning process more efficient. Of course, one could pick a static variance but a much better choice is a function of the states. The rationale for doing so is the fact that some states are better than others, and these states have high action values for many, if not all, actions. On the other hand, there are also bad states with many bad actions with low action values. Hence for the former states, it would be appropriate to have higher baseline; the later one would do better with a lower baseline.

What could one use as a baseline? Well, one good choice fulfilling all what was mentioned so far would be an estimate of the state-value function, $\widehat{v}(S_{t}, \textbf{w})$. Since REINFORCE belongs to the Monte Carlo group of algorithms, to learn state-value function estimate, it seems natural to choose one of the many available Monte Carlo methods. (Sure, I didn't bother to present any of the Monte Carlo methods in detail; however, it is not difficult to come up with the update rule. One just needs to save the whole episode and then, at the end, perform all the updates, e.g. $n$-step update with $n$ being the length of the episode.)

Although the baseline reduces variance and accelerates learning, the algorithm is still far from learning quickly. In the next part, I will try to present some more ideas how bootstrapping might come to rescue.

Nevertheless, I encourage the reader to add a baseline to the above REINFORCE algorithm.

### Actor-Critic Methods

**Actor-critic** methods are bootstrapping methods with a separate structures for representing policy and value functions, i.e. policy is independent of the value function. The part of the algorithm representing policy is called actor, because it selects actions; and the value function part is called critic, because it criticises the actor's actions. The critic critiques the actor only by outputting the TD error, which is in turn used to perform learning steps in both, actor and critic.

Although REINFORCE with baseline is very close to be an instance of actor-critic methods, it is not consider to be one mainly because it does not bootstrap. Moreover, since it doesn't bootstrap, it is unbiased; and it will converge asymptotically to a local minimum. The problem is that this convergence guarantee is slow due to high variance. Bootstrapping, in general, reduces variance and thus accelerates learning - the main reason for it's implementation.

A slight modification to REINFORCE with baseline is needed in order for it to classify as an actor-critic method. One only needs to change critic so that it will bootstrap. There are several ways how to do this, i.e. one-step bootstrapping, n-step bootstrapping, etc. I will start right away with the $\lambda$-return algorithm.

To this end one needs eligibility traces for both, weight vector, $\textbf{w}$, as well as policy gradient vector, $\textbf{$\theta$}$. I will denote these vectors by $\textbf{z}^{\textbf{w}}$ and $\textbf{z}^{\textbf{$\theta$}}$, respectively. Both are initialised to zero vectors at the beginning of each episode, i.e.
\begin{equation}
\begin{split}
\textbf{z}^{\textbf{$\theta$}}_{-1} =& 0 \,, \\
\textbf{z}^{\textbf{w}}_{-1} =& 0 \,.
\end{split}
\end{equation}

Then, on each time-step, $t$, both are incremented by the respective gradients; and then fade away by the product of the discount and decay factors. In terms of equation this yields
\begin{equation}
\begin{split}
\textbf{z}^{\textbf{$\theta$}}_{t} =& \gamma^{\textbf{$\theta$}}\lambda^{\textbf{$\theta$}} \textbf{z}^{\textbf{$\theta$}}_{t-1} + (\gamma^{\textbf{$\theta$}})^{t} \nabla_{\textbf{$\theta$}} \ln \big(\pi(A_{t} \,|\, S_{t}, \textbf{$\theta$}_{t}) \big) \,, \\
\textbf{z}^{\textbf{w}}_{t} =& \gamma^{\textbf{w}}\lambda^{\textbf{w}} \textbf{z}^{\textbf{w}}_{t-1} + \nabla_{\textbf{w}} \widehat{v}(S_{t}, \textbf{w}_{t}) \,,
\end{split}
\end{equation}
where each vector has its own decay factor with obvious notation. What reminds to do is the update both, the weight vector as well as the policy gradient vector. This is done as expected
\begin{equation}
\begin{split}
\textbf{$\theta$}_{t+1} =& \textbf{$\theta$}_{t} + \alpha^{\textbf{$\theta$}} \delta_{t} \textbf{z}^{\textbf{$\theta$}}_{t} \,, \\
\textbf{w}_{t+1} =& \textbf{w}_{t} + \alpha^{\textbf{w}} \delta_{t} \textbf{z}^{\textbf{w}}_{t} \,.
\end{split}
\end{equation}
For linear approximation methods, one could also use the true online algorithm (Dutch traces) to update $\textbf{w}$. I won't do it here since, as already mentioned, from now on I will only be concerned by neural networks, which almost always come with a non-linear activation function; and thus are non-linear function approximation methods. The TD error is computed as usual, though this time using state-value function, $\delta_{t} = \big[ R_{t+1} + \gamma^{\textbf{w}} \widehat{v}(S_{t+1}, \textbf{w}_{t}) - \widehat{v}(S_{t} \textbf{w}_{t}) \big]$.

### Example 5: CartPole with Actor-Critic

In the last exercise of this notebook I will demonstrate a huge advantage of Actor-Critic methods over all the others I have developed so far. In a very similar manner to the previous example, also here I will use TensorFlow. The models for both, actor and critic, are again linear; and, as usual, the implementation is hidden in a separate file.

#### Actor and Critic classes

The **Actor class** functions similarly to the REINFORCE class from the previous example. It has all the same member functions, the only difference is that the **train** member function now takes state, action, discount and decay factors, TD error, and learning rate as arguments. The other member functions stay the same.

In case of the **Critic class**, the **train** member function takes one less argument compared to the actor, namely state, discount and decay factors, TD error, and learning rate. It doesn't take actions, since to predict the TD error it learns state-value function. Also, there is no chooseAction member function, but instead the class has **computeTDerror** function. It takes two states, namely $S_{t}$ and $S_{t+1}$; reward, $R_{t+1}$; and discount factor.

In addition to these functions, both classes also have **resetTraces** member function, needed to reset traces at the beginning of each new episode.

```python
from ActorCritic import Actor, Critic

n_features, n_actions = 4, 2

actor = Actor(n_features, n_actions)
critic = Critic(n_features)
```

```python
actor.reset()
critic.reset()

actor.createFileWriter('./tb/actor')
critic.createFileWriter('./tb/critic')

discount_A = 0.995
discount_C = 0.8
decay_A = 1.0
decay_C = 0.8

episodes = 250
returns = np.zeros(episodes)

for i in range(episodes):
    learning_rate_A = 0.01 / 2**(i//50)
    learning_rate_C = learning_rate_A*7

    done = False
    cum_reward = 0

    actor.resetTraces()
    critic.resetTraces()
    old_state = env.reset()

    while not done:
        action = actor.chooseAction(old_state[np.newaxis, :])
        new_state, reward, done, _ = env.step(action)
        cum_reward = cum_reward + reward
        if (done and cum_reward != 200): reward = -5
        td_error = critic.computeTDerror(old_state[np.newaxis, :], new_state[np.newaxis, :], reward, discount_C)
        actor.train(old_state[np.newaxis, :], action, discount_A, decay_A, td_error, learning_rate_A)
        critic.train(old_state[np.newaxis, :], discount_C, decay_C, td_error, learning_rate_C)
        old_state = new_state

    returns[i] = cum_reward
    if i%10 == 0:
        actor.makeSummary(i)
        critic.makeSummary(i)

actor.closeFileWriter()
critic.closeFileWriter()

printTrainingReturns(returns)
```

    Performance measure: the return per episode averaged over:
     - the first 10% of the training was 112.9 with the standard deviation 66.5;
     - 10% to 50% of the training was 170.0 with the standard deviation 42.6;
     - 50% to 90% of the training was 198.1 with the standard deviation 9.5;
     - the last 10% of the training was 200.0 with the standard deviation 0.0.

```python
getPerformanceAgent(env, actor, 100)
```

    Performance measure: the return per episode averaged over 100 episodes was 196.6 with the standard deviation 12.5.
    The highest return was 200.0, and the lowest return was 143.0.

```python
visualizeAgent(env, actor)
```

    Return of this visualization was 200.0.

```python
actor.printParameters()
```

    policy/weights:0
    [[-0.85150903  0.8515178 ]
     [-3.8869119   3.8870802 ]
     [-4.2056746   4.2056165 ]
     [-6.1395283   6.1396074 ]]
    training/trace:0
    [[ 0.60862345 -0.6086233 ]
     [-0.47551307  0.4755132 ]
     [-0.08369061  0.0836906 ]
     [ 0.13692683 -0.1369268 ]]

Since summaries for actor and critic are saved in the same directory as those for REINFORCE agent, one does not need to relounche TensorBoard. The new summaries will appear automatically, if you have, of course, lounched TensorBoard in the previous example.

This and the previous exercises deserve a short comment. Sometimes, the agent learns an unstable policy, meaning that the cart would go off the screen if there were enough episodes, not just 200. In fact, in the unstable case, the more the cart is displaced from its initial possition, the faster it goes off the screen. This can be seen from the sign of the first weight, i.e. in case it is positive, the policy is unstable. This can be fixed by not capping the training episodes at 200 steps.

### Example 6: CartPole with non-linear REINFORCE

In this example I explore REINFORCE algorithm with non-linear layer with activation function $\mathrm{atan}(x)$. Initially, the algorithm performs worse than the linear counterpart — this is due to the need for train one more layer. However, eventually it outperforms the linear version.

```python
from non_linear_REINFORCE import nl_REINFORCE

n_features, n_actions = 4, 2

agent = nl_REINFORCE(n_features, n_actions)
```

```python
agent.reset()                               # Reset the policy vector in the agent.
agent.createFileWriter('./tb/nl_REINFORCE') # FileWriter is used visualise training in TensorBoard

discount = 0.995
episodes = 200
returns = np.zeros(episodes)

for i in range(episodes):
    learning_rate = 0.01 / 2**(i//80)

    done = False
    state = env.reset()

    states = []
    actions = []
    rewards = []

    while not done:
        action = agent.chooseAction(state[np.newaxis, :])
        states.append(state[np.newaxis, :])
        actions.append(action)
        state, reward, done, _ = env.step(action)
        rewards.append(reward)

    returns[i] = np.sum(rewards)
    if len(rewards) != 200:
        rewards[-1] = -5

    agent.train(states, actions, rewards, discount, learning_rate)

    if i%10 == 0:
        agent.makeSummary(i)                # Save policy vector to disc for later visualisation in TensorBoard

agent.closeFileWriter()                     # Flush everything to disc, and close FileWriter.

printTrainingReturns(returns)
```

    Performance measure: the return per episode averaged over:
     - the first 10% of the training was 17.3 with the standard deviation 6.1;
     - 10% to 50% of the training was 38.1 with the standard deviation 36.2;
     - 50% to 90% of the training was 192.0 with the standard deviation 22.3;
     - the last 10% of the training was 200.0 with the standard deviation 0.0.

```python
getPerformanceAgent(env, agent, 100)
```

    Performance measure: the return per episode averaged over 100 episodes was 200.0 with the standard deviation 0.0.
    The highest return was 200.0, and the lowest return was 200.0.

```python
visualizeAgent(env, agent)
```

    Return of this visualization was 200.0.

```python
agent.printParameters()
```

    policy/weights1:0
    [[-0.0282541   0.01876783  0.0428801   0.04617718]
     [ 1.3056211   0.16739683  0.40418524  1.3789247 ]
     [ 3.0054305   0.3741164   0.9068356   3.2327397 ]
     [ 6.09383     0.7131179   1.7369671   6.538426  ]]
    policy/weights2:0
    [[-1.8773029   1.8771524 ]
     [-0.22768097  0.22751907]
     [-0.5511264   0.5512068 ]
     [-1.999021    1.9992278 ]]

Again, visualisation can be done with the help of TensorBoard as before.
```terminal
tensorboard --logdir=dir_to_this_notebook/tb/
```
After successfull lounched of TensorBoard, type *http://localhost:6006* in your browser.

### Example 7: CartPole with non-linear Actor-Critic

The same story continues also for Actor-Critic; linear algorithm improve usually better initially however non-linear ones quickly catch up, and then outperform the linear ones. I encourage the reader to experiment with hyper-parameters and number of episodes. As with everything in life, one faces a trade-off between the level of complexity of networks, and the time spent on training (or number of episodes one needs) in order to obtain decent results.

```python
from non_linear_ActorCritic import nl_Actor, nl_Critic

n_features, n_actions = 4, 2

actor = nl_Actor(n_features, n_actions)
critic = nl_Critic(n_features)
```

```python
actor.reset()
critic.reset()

actor.createFileWriter('./tb/nl_actor')
critic.createFileWriter('./tb/nl_critic')

discount_A = 0.995
discount_C = 0.8
decay_A = 1.0
decay_C = 0.8

episodes = 400
returns = np.zeros(episodes)

for i in range(episodes):
    learning_rate_A = 0.01 / 2**(i//80)
    learning_rate_C = learning_rate_A*7

    done = False
    cum_reward = 0

    actor.resetTraces()
    critic.resetTraces()
    old_state = env.reset()

    while not done:
        action = actor.chooseAction(old_state[np.newaxis, :])
        new_state, reward, done, _ = env.step(action)
        cum_reward = cum_reward + reward
        if (done and cum_reward != 200): reward = -5
        td_error = critic.computeTDerror(old_state[np.newaxis, :], new_state[np.newaxis, :], reward, discount_C)
        actor.train(old_state[np.newaxis, :], action, discount_A, decay_A, td_error, learning_rate_A)
        critic.train(old_state[np.newaxis, :], discount_C, decay_C, td_error, learning_rate_C)
        old_state = new_state

    returns[i] = cum_reward
    if i%10 == 0:
        actor.makeSummary(i)
        critic.makeSummary(i)

actor.closeFileWriter()
critic.closeFileWriter()

printTrainingReturns(returns)
```

    Performance measure: the return per episode averaged over:
     - the first 10% of the training was 20.4 with the standard deviation 12.5;
     - 10% to 50% of the training was 128.6 with the standard deviation 61.2;
     - 50% to 90% of the training was 199.3 with the standard deviation 8.4;
     - the last 10% of the training was 200.0 with the standard deviation 0.0.

```python
getPerformanceAgent(env, actor, 100)
```

    Performance measure: the return per episode averaged over 100 episodes was 200.0 with the standard deviation 0.0.
    The highest return was 200.0, and the lowest return was 200.0.

```python
visualizeAgent(env, actor)
```

    Return of this visualization was 200.0.

```python
actor.printParameters()
```

    policy/weights1:0
    [[ 0.05949468 -0.05441877  0.3844086  -0.9858479 ]
     [-0.16988026  0.15610315 -1.1925418   2.026118  ]
     [-0.2952739   0.27236223 -2.125239    3.8163395 ]
     [-0.5490327   0.50491565 -4.380072    8.90089   ]]
    policy/weights2:0
    [[ 0.1630345  -0.16269252]
     [-0.15012963  0.14948061]
     [ 1.4013253  -1.4025971 ]
     [-3.2229307   3.2220747 ]]
    training/trace:0
    [[-0.04036621  0.0371312  -0.33479118  0.67640644]
     [ 0.01469889 -0.01352144  0.11658498 -0.18996628]
     [ 0.0072532  -0.00667187  0.06157876 -0.13730145]
     [-0.03348186  0.03080075 -0.27350533  0.54647416]]
    training/trace_1:0
    [[ 0.0345098  -0.03450977]
     [-0.03176262  0.03176259]
     [ 0.2928286  -0.29282844]
     [-0.56767815  0.567678  ]]

```python
critic.printParameters()
```

    value_function/weights1:0
    [[-2.686344    1.3936129  -3.0145488   1.3940256 ]
     [ 2.539597    2.9783134  -1.2554175  -2.854464  ]
     [ 1.4830793  -4.938145   -0.21833652  2.4927754 ]
     [-0.23441082  0.8347611   0.8397501   2.9257116 ]]
    value_function/weights2:0
    [[-0.64603674]
     [ 0.7381989 ]
     [-1.1949729 ]
     [ 0.3258322 ]]
    training/trace:0
    [[ 0.07145461 -0.07907555  0.1293024  -0.02486444]
     [ 0.11704247 -0.11170448  0.18376043 -0.03270137]
     [-0.03545566  0.03947157 -0.06455316  0.01243541]
     [-0.06312116  0.04250066 -0.07092051  0.00970631]]
    training/trace_1:0
    [[-0.1034631 ]
     [-0.8965101 ]
     [ 0.63110954]
     [ 0.63143545]]

***The end of the notebook.***


# Introduction

In this last notebook, we will create a full state feedback controller that will allow the human model to balance. There isn't time to give a good overview of controls, but we will use a very common optimal control method to quickly create a controller for our system.

# Setup

Bring in the results from the previous notebooks.

```python
from __future__ import print_function
from solution.visualization import *
```

We will be working with numerical matrices so we will need several functions from NumPy.

```python
from numpy import array, zeros, eye, asarray, dot, rad2deg
from numpy.linalg import inv
```

We will be simulating the model again with the controller in place and will make some plots of the state trajectories as before so bring in some useful functions from matplotlib. First enable inline plotting:

```python
%matplotlib inline
```

```python
from matplotlib.pyplot import plot, xlabel, ylabel, legend, rcParams
```

```python
rcParams['figure.figsize'] = (14, 8)
```

Bring in some SymPy functions for simpilfication and printing:

```python
from sympy import simplify, Matrix, matrix2numpy
from sympy.physics.vector import init_vprinting, vlatex
```

```python
init_vprinting(use_latex='mathjax', pretty_print=False)
```

We will be displaying numerical matrices and the IPython magic `%precision` can be used to adjust how many decimal places are shown for nice compact printing.

```python
%precision 3
```

# Linearization

We current have symbolic equations of motion in this form:

$$ \mathbf{M}(\mathbf{x}) \dot{\mathbf{x}} = \mathbf{f}(\mathbf{x}, \mathbf{u}, t) $$

where $\mathbf{M}$ is the mass matrix, $\mathbf{x}$ is the state vector made up of the generalized coordinates and speeds, and $\mathbf{f}(\mathbf{x}, \mathbf{u})$ is a vector of non-linear functions of the states, $\mathbf{x}$, and the specified inputs, $\mathbf{u}$.

This model is relatively easy to linearize and we will use a quick and dirty method to do so. In general, the non-linear equations of motion (for an unconstrained system) can be transformed into a linear form by expanding the Taylor series about small perturbations and truncating any non-linear terms. For this problem, this amounts to computing the [Jacobian](https://en.wikipedia.org/wiki/Jacobian) of the forcing vector, $\mathbf{f}$. The linear equations take this form:

$$ \mathbf{M} \Delta \dot{\mathbf{x}} = \mathbf{f}_A \Delta \mathbf{x} + \mathbf{f}_B \Delta \mathbf{u} $$

where $\mathbf{f}_A$ and $\mathbf{f}_B$ are the Jacobians of $\mathbf{f}$ with respect to the states and specified inputs, respectively:

$$ \mathbf{f}_A = \left[ \frac{\delta \mathbf{f} }{ \delta x_1}, \ldots, \frac{\delta \mathbf{f} }{ \delta x_n} \right] $$

$$ \mathbf{f}_B = \left[ \frac{\delta \mathbf{f} }{ \delta u_1}, \ldots, \frac{\delta \mathbf{f} }{ \delta u_n} \right] $$

and $\Delta \mathbf{x}$ and $\Delta \mathbf{u}$ are _differential_ changes in $\mathbf{x}$ and $\mathbf{u}$ respectively.

Once $\mathbf{f}_A$ and $\mathbf{f}_B$ are found, then the standard state space form of the linear equations can be formed by transforming the linear equations.

$$ \Delta \dot{x} = \mathbf{M}^{-1} \mathbf{f}_A \Delta \mathbf{x} + \mathbf{M}^{-1} \mathbf{f}_B \Delta \mathbf{u} $$

which can be rewritten as

$$ \Delta \dot{x} = \mathbf{A} \Delta \mathbf{x} + \mathbf{B} \Delta \mathbf{u} $$

where $\mathbf{A}$ is the state matrix and $\mathbf{B}$ is the input matrix.

The first step is to specify the equilbrium point about which to linearize. Our system has two unique equilibrium points, one when the person is upright, $\mathbf{x} = [0, \ldots, 0]$, and one when the person is upside down, $\mathbf{x} = [\pi, \ldots, \pi]$ (180 degrees). We are only concerned with the former because we want the model to "stand up".

```python
equilibrium_point = zeros(len(coordinates + speeds))
equilibrium_point
```

We will use SymPy's `sub()` method to convert the linear equations of motion to numerical arrays, so we create two dictionaries containing the numerical values of the equilibrium point and the constants.

```python
equilibrium_dict = dict(zip(coordinates + speeds, equilibrium_point))
equilibrium_dict
```

## Exercise

As an exercise, create a `parameter_dict` that maps the symbolic constants to the numerical constants. We have variable names for each already available from the previous notebooks (`constants` and `numerical_constants`).

```python
%load exercise_solutions/n09_control_parameter-dict.py
```

The `KanesMethod` object from earlier contains methods for linearizing the equations of motion. First, we'll create a `Linearizer` object:

```python
linearizer = kane.to_linearizer()
```

This creates an object that holds a *generalized form* of the equations of motion, which can be used to linearize the system in a robust way. When it does this, it finds the inputs `r`, and orders them alphabetically. We have our own order we want to use from before, so we need to change this:

```python
linearizer.r == Matrix(specified)
```

```python
linearizer.r = Matrix(specified)
```

To perform the actual linearization, we'll use the `linearize` method:

```python
help(linearizer.linearize)
```

Because we want to get the equations in the form

$$ \Delta \dot{x} = \mathbf{A} \Delta \mathbf{x} + \mathbf{B} \Delta \mathbf{u} $$

we'll set `A_and_B=True`. We can also perform the variable to numeric substitution in this step by passing in both the `equilibrium_dict` and the `parameter_dict` to the operating point.

```python
A, B = linearizer.linearize(op_point=[equilibrium_dict, parameter_dict], A_and_B=True)
```

Now we can convert from symbolic to numeric numpy matrices, and display our results:

```python
A = matrix2numpy(A, dtype=float)
B = matrix2numpy(B, dtype=float)
```

```python
A
```

```python
B
```

# Controller Design

Before we try to create a controller, we need to check to see if it is actually [controllable](https://en.wikipedia.org/wiki/Controlability). You can import a function from the `utils` module included with these notebooks to check.

```python
from utils import controllable
```

```python
controllable(A, B)
```

So yes, the system is controllable. Now we will use [optimal control theory](https://en.wikipedia.org/wiki/Optimal_control) to create a controller that compute the necessary joint torques at each time step to balance the person.

It turns out that if you setup a cost function in which you'd like to minimize the deviation from the desired state, 0, and minimize the effort in the joint torques, there is an elegant solution to find a matrix $\mathbf{K}$ which can be used to compute the joint torques given the current state value.

Given the cost function:

$$J = \frac{1}{2} \int_0^\infty\left[\mathbf{x}^T \mathbf{Q} \mathbf{x} + \mathbf{u}^T \mathbf{R} \mathbf{u}\right]dt$$

There is a matrix $\mathbf{K}$ that computes the inputs $\mathbf{u}(t)$ given the states $\mathbf{x}$:

$$\mathbf{u}(t) = -\mathbf{K} \mathbf{x}(t)$$

where $\mathbf{K}$ can be found by the solution to the algebraic [Riccati Equation](https://en.wikipedia.org/wiki/Riccati_equation) such that:

$$\mathbf{K} = \mathbf{R}^{-1} \mathbf{B}^T \mathbf{S}$$

and the Riccati equation is:

$$\textbf{0} = -\textbf{S}\textbf{A}-\textbf{A}^{\text{T}}\textbf{S}+\textbf{S}\textbf{B}\textbf{R}^{-1}\textbf{B}^{\text{T}}\textbf{S}-\textbf{Q}$$

SciPy provides a function that can compute the solution, $\mathbf{S}$, of the continous algebraic Riccati Equation.

```python
from scipy.linalg import solve_continuous_are
```

The weighting matrices, $\mathbf{Q}$ and $\mathbf{R}$, allow you to control how much weight is applied to minimizing the error in the states and the effort in the inputs. A typical starting point for these matrices is to set them equal to the identity matrix, which can be created with the NumPy `eye()` function.

```python
Q = eye(6)
Q
```

```python
R = eye(3)
R
```

$\mathbf{K}$ can now be computed.

```python
S = solve_continuous_are(A, B, Q, R)
K = dot(dot(inv(R), B.T),  S)
K
```

Remember that the `right_hand_side` function we generated earlier can take a function for the specified values.

```python
help(right_hand_side)
```

## Exercise

Create a function named `controller` that computes the specified inputs given the current state and time.

```python
def controller():
    """Returns the output of the controller, i.e. the joint torques, given the current state."""
    return
```

```python
%load exercise_solutions/n09_control_controller-function.py
```

Now we can integrate the equations of motion again, but this time with the controller in place.

## Exercise

Integrate the equations of motion and plot the coordinates and speeds as we did in the simulation notebook and see if the system behaves as expected.

```python

```

```python
%load exercise_solutions/n09_control_integrate-eom.py
```

# Plot Trajectories

## Exercise

Use the same plot functions as were used in the simulation notebook to inspect the state trajectories.

```python
# Plot the angles

```

```python
%load exercise_solutions/n09_control_plot-angles.py

```

```python
# Plot the angular velocities

```

```python
%load exercise_solutions/n09_control_plot-velocities.py
```

What do these plots tell you about the controller? Does it work?

# Visualization

Finally, regenerate the visualization to see the 3D animation of the system under the influence of the controller.

```python
scene.states_trajectories = y
```

```python
scene.display_ipython()
```