**Submission deadline:**
* **Regular problems: last lab session before or on Monday, 13.01.2020**
* **Bonus problems: Last lab during semester**

**Points: 5 + 9 bonus points**

Please note: some of the assignments are tedious or boring if you are already a NumPy ninja. The bonus problems were designed to give you a more satisfying alternative.

## Heads Up!

This assignment comes with starter code, but you are not forced to use it, as long as you execute all analysis demanded in the problems.

## A note about plots!

Plots are a way of communication. Just lke text, they can be paraphrased. You do not have to exactly reproducy my plots, but you must try to make suer yourp plots tell a similar story:
- label axis
- add titles
- choose plot type properly
- choose a color scale, limits, ticks

so that you can describe what is happening!

## Bugs?!

Please submit Github PRs or email us about any problems with the notebook - we will try to correct them quickly.

```
# Standard IPython notebook imports
%matplotlib inline

import os

from io import StringIO
import itertools
import httpimport
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from tqdm import tqdm_notebook

import scipy.stats as sstats
import scipy.optimize as sopt

import seaborn as sns

import sklearn.datasets
import sklearn.ensemble
import sklearn.svm
import sklearn.tree

import cvxopt

# In this way we can import functions straight from github
with httpimport.github_repo('janchorowski', 'nn_assignments',
                            module='common', branch='nn18'):
     from common.plotting import plot_mat

sns.set_style('whitegrid')
```

# SVM Theory
A linear SVM assigns points $x^{(i)}\in\mathbb{R}^n$ to one of two
classes,  $y^{(i)}\in\{-1,1\}$ using the decision rule:

\begin{equation}
y = \text{signum}(w^T x + b).
\end{equation}

SVM training consists of finding weights $w\in\mathbb{R}^n$
and bias $b\in\mathbb{R}$ that maximize the separation margin. This
corresponds to solving the following quadratic optimization problem:

\begin{equation}
\begin{split}
  \min_{w,b,\xi}  &\frac{1}{2}w^Tw  + C\sum_{i=1}^m \xi_i  \\
  \text{s.t. } & y^{(i)}(w^T x^{(i)} + b) \geq 1- \xi_i\;\; \forall_i \\
  & \xi_i \geq 0 \;\; \forall_i.
\end{split}
\end{equation}

# Problem 1 [2p]

Load the iris dataset.

1. [1p] Using the `sklearn.svm.SVC` library train a linear SVM that
separates the Virginica from the Versicolor class using the
petal length and petal width features. Plot the obtained decision boundary and
the support vectors (their locations and weights - coefficients $\alpha$).

2. [.5p] Now train a nonlinear SVM using the Gaussian kernel. Tune the parameetrs `C` and `gamma` (for the kernel) to reach maximum training accurracy. Plot the decision boundary and supprt vectors.

3. [.5p] Answer the following questions:
    - When the SVM is forced to maximally accurate on the train set, roughly how many support vectors do we get ?
    - what is the relationship between the regularization constant `C` and the support vector weights `alpha`?

```
# load iris, extract petal_length and petal_width of versicolors and virginicas

iris = sklearn.datasets.load_iris()
print('Features: ', iris.feature_names)
print('Targets: ', iris.target_names)
petal_length = iris.data[:,iris.feature_names.index('petal length (cm)')]
petal_width = iris.data[:, iris.feature_names.index('petal width (cm)')]

IrisX = np.array(iris.data.T)
IrisX = IrisX[:, iris.target!=0]

IrisX2F = np.vstack([petal_length, petal_width])
IrisX2F = IrisX2F[:, iris.target!=0]

# Set versicolor=0 and virginia=1
IrisY = (iris.target[iris.target!=0]-1).reshape(1,-1).astype(np.float64)

plt.scatter(IrisX2F[0,:], IrisX2F[1,:], c=IrisY.ravel(),
            cmap='spring', edgecolors='k')
plt.xlabel('petal_length')
plt.ylabel('petal_width')
```

```
#
# Fit a linear SVM using libsvm
#

svm_model = TODO
svm_model.fit(TODO)

print("libsvm error rate: %f" % ((svm_model.predict(IrisX2F.T)!=IrisY).mean(),))
```

    libsvm error rate: 0.050000

```
#
# Plot the decision boundary
#

petal_lengths, petal_widths = np.meshgrid(np.linspace(IrisX2F[0,:].min(), IrisX2F[0,:].max(), 100),
                                          np.linspace(IrisX2F[1,:].min(), IrisX2F[1,:].max(), 100))

IrisXGrid = np.vstack([petal_lengths.ravel(), petal_widths.ravel()])
predictions_Grid = svm_model.predict(IrisXGrid.T)

plt.contourf(petal_lengths, petal_widths, predictions_Grid.reshape(petal_lengths.shape), cmap='spring')

plt.scatter(IrisX2F[0,:], IrisX2F[1,:], c=IrisY.ravel(),
            cmap='spring', edgecolors='k')

plt.xlabel('petal_length')
plt.ylabel('petal_width')
plt.title('Decision boundary found by libsvm')
```

```
#
# Plot the decision boundary and the support vectors.
#
# You can extract the indices of support vectors and their weights from fielfs of the
# svm object. Display the loaction of support vectors and their weights (by changing the
# size in the scatterplot)
#
# TODO
#

support_vector_indices = svm_model.support_
support_vector_coefficients = svm_model.dual_coef_

plt.contourf(petal_lengths, petal_widths, predictions_Grid.reshape(petal_lengths.shape), cmap='spring')
plt.scatter(TODO,
        cmap='spring',
        edgecolors='k')

plt.xlabel('petal_length')
plt.ylabel('petal_width')
plt.title('Decision boundary found by libsvm')
```

```
#
# Fit a nonlinear SVM with a Gaussian kernel using libsvm.
# Optimize the SVM to make
#

svm_gauss_model = TODO
svm_gauss_model.fit(TODO)

print("libsvm error rate: %f" % ((svm_gauss_model.predict(IrisX2F.T)!=IrisY).mean(),))

TODO: plot the decision boundary with support vectors

plt.xlabel('petal_length')
plt.ylabel('petal_width')
plt.title('Decision boundary found by libsvm')
```

# Problem 2 [1p]

Reimplement the linear SVM using the use `cvxopt.solvers.qp`
Quadratic Programming (QP) solver. You will need to define the matrices
that define the problem. Compare the obtained solutions. Extract the
support vectors from the LIBSVM solution and plot the support vectors.

The `cvxopt.solvers.qp` solves the following optimization problem:

\begin{align}
\text{minimize over } x \text{: }& \frac{1}{2} x^T P x + q^T x \\
\text{subject to: } & Gx \leq h \\
& Ax = b
\end{align}

To solve the SVM problem you need to encode the weights $W$, biases $b$, and slack variables $\xi$ as elements of the vector $x$, then properly fill the matrices and vectors $P$, $q$, $G$, $h$. We can ignore setting the $A$ and $b$ parametrs, since there are no linear constraints.

```
#
# Now solve the SVM using the QP solver
#

n, m = IrisX2F.shape
C=10.0

P = np.zeros((n+1+m, n+1+m)) #w, bias, xi
q = np.zeros((n+1+m,1))
G = np.zeros((2*m, n+1+m)) # we have two constrains for each data point:
                           # that the margin is equal to 1-xi
                           # and that xi is nonnegative
h = np.zeros((2*m,1))
#
# TODO: fill in P, q, G, h
#

TODO

#
# Now run the solver
#
ret  = cvxopt.solvers.qp(cvxopt.matrix(P), cvxopt.matrix(q),
                         cvxopt.matrix(G), cvxopt.matrix(h), )

ret = np.array(ret['x'])

#
# extract the weights and biases
#
W = ret[:n].reshape(-1,1)
b = ret[n]

#
# Extract the weight and bias from libsvm for comparison
#
Wlibsvm = TODO
blibsvm = TODO

print('W', W.T, 'Wlibsvm', Wlibsvm)
print('b', b, 'blibsvm', blibsvm)
```

         pcost       dcost       gap    pres   dres
     0: -8.2045e+03  4.6654e+03  2e+04  2e+01  3e+00
     1:  5.3426e+01 -5.8951e+02  3e+03  2e+00  4e-01
     2:  1.7058e+02 -1.9137e+01  4e+02  3e-01  5e-02
     3:  1.4063e+02  7.5166e+01  1e+02  7e-02  1e-02
     4:  1.3820e+02  9.8722e+01  6e+01  3e-02  5e-03
     5:  1.4016e+02  1.1207e+02  3e+01  4e-03  8e-04
     6:  1.3507e+02  1.1797e+02  2e+01  1e-03  2e-04
     7:  1.2551e+02  1.2356e+02  2e+00  2e-04  3e-05
     8:  1.2445e+02  1.2440e+02  5e-02  4e-06  6e-07
     9:  1.2442e+02  1.2442e+02  1e-03  9e-08  2e-08
    10:  1.2442e+02  1.2442e+02  2e-05  1e-09  1e-10
    Optimal solution found.
    W [[2.75844069 4.827271  ]] Wlibsvm [[2.75873306 4.82861713]]
    b [-21.2054472] blibsvm [-21.20914737]

# Problem 3 [2p]

Repeat 100 bootstrap experiments to establish the effect of constant $C$ on SVM.

For each experiment do the following:

1. Sample (with replacement) a bootstrap dataset equal in size to the
    training dataset. This will be this experiment's training dataset.
2. Prepare the experiment's testing dataset by using samples not
    inluded in the bootstrap dataset.
3. For all $C$ from the set $\{10^{-4}, 10^{-3.5}, 10^{-3.}, \ldots, 10^{6}\}$
    fit a nonlinear SVM (Gaussian kernel, called \texttt{rbf} in
    LIBSVM using the default $\gamma$) and record the training and
    testing errors.

Analyze a box plot of errors as a function of $C$. Can you see its
influence on the training and testing error, as well as on the
testing error variability?

**Indicate regions of overfitting and underfitting.**

```
res = []
for rep in range(100):
    bootstrap_sel = TODO
    test_sel = TODO

    bootstrap_IrisX = IrisX[:,bootstrap_sel]
    bootstrap_IrisY = IrisY[:,bootstrap_sel]

    test_IrisX = IrisX[:,test_sel]
    test_IrisY = IrisY[:,test_sel]

    #
    # TODO: Loop over a list of exponents.
    #
    for Cexponent in TODO:
        C = 10.0**Cexponent
        svm_model = TODO
        svm_model.fit(TODO)
        train_acc = TODO
        test_acc = TODO

        res.append(dict(Cexponent=Cexponent, err=1-test_acc, subset='test'))
        res.append(dict(Cexponent=Cexponent, err=1-train_acc, subset='train'))

res = pd.DataFrame(res)
chart = sns.catplot(kind='box', x='Cexponent', y='err', col='subset',
            color='blue', data=res)
chart.set_xticklabels(rotation=45)
None
```

# Problem 4 [3p bonus]

Implement a nonlinear SVM by solving the dual problem using the Quadratic Programming solver. Compare results with LIBSVM.

Please see [page 20 if CS229 lecture notes](http://cs229.stanford.edu/notes/cs229-notes3.pdf) for problem formulation.

# Problem 5 [2p bonus]
Compare two ways to implement a multi-class
SVM: by training ``1-vs-1'' classifier for each class combination,
and by training a ``1-vs-rest'' classifier for each clas. See
http://www.csie.ntu.edu.tw/\~cjlin/papers/multisvm.pdf for
details.

# Problem 6 [4p bonus]

Implement a Kernelized linear regression. Train it on Iris using a Gaussian kernel. Compare to the non-linear SVM.

```

```


# Create our environment (connect-4 game)

Connect-4 is a game in which if one were to connect 4 of his moves either horizontally, vertically or diagonally, one records a win. In reinforcement learning setting, we grant reward = 1 for a win, -1 for a lose and 0.5 for a draw. The following game engine object provides five methods.
* Render :
       Showing the board state with X and Os.
* Reset :
       For playing over and over.
* Get available moves :
       Scan the board state and give available moves.
* Check game done :
       Based on which player is making the move, check if one has won the game or a draw has resulted.
* Make move :
       Record the move by players and return observation and reward.

```python
import numpy as np
import pandas as pd
from IPython.display import display

class connect_x:

    def __init__(self):
        self.board_height = 6
        self.board_width = 7
        self.board_state = np.zeros([self.board_height, self.board_width], dtype=np.int8)
        self.players = {'p1': 1, 'p2': 2}
        self.isDone = False
        self.reward = {'win': 1, 'draw': 0.5, 'lose': -1}

    def render(self):
        rendered_board_state = self.board_state.copy().astype(np.str)
        rendered_board_state[self.board_state == 0] = ' '
        rendered_board_state[self.board_state == 1] = 'O'
        rendered_board_state[self.board_state == 2] = 'X'
        display(pd.DataFrame(rendered_board_state))

    def reset(self):
        self.__init__()

    def get_available_actions(self):
        available_cols = []
        for j in range(self.board_width):
            if np.sum([self.board_state[:, j] == 0]) != 0:
                available_cols.append(j)
        return available_cols

    def check_game_done(self, player):
        if player == 'p1':
            check = '1 1 1 1'
        else:
            check = '2 2 2 2'

        # check vertically then horizontally
        for j in range(self.board_width):
            if check in np.array_str(self.board_state[:, j]):
                self.isDone = True
        for i in range(self.board_height):
            if check in np.array_str(self.board_state[i, :]):
                self.isDone = True

        # check left diagonal and right diagonal
        for k in range(0, self.board_height - 4 + 1):
            left_diagonal = np.array([self.board_state[k + d, d] for d in \
                            range(min(self.board_height - k, min(self.board_height, self.board_width)))])
            right_diagonal = np.array([self.board_state[d + k, self.board_width - d - 1] for d in \
                            range(min(self.board_height - k, min(self.board_height, self.board_width)))])
            if check in np.array_str(left_diagonal) or check in np.array_str(right_diagonal):
                self.isDone = True
        for k in range(1, self.board_width - 4 + 1):
            left_diagonal = np.array([self.board_state[d, d + k] for d in \
                            range(min(self.board_width - k, min(self.board_height, self.board_width)))])
            right_diagonal = np.array([self.board_state[d, self.board_width - 1 - k - d] for d in \
                            range(min(self.board_width - k, min(self.board_height, self.board_width)))])
            if check in np.array_str(left_diagonal) or check in np.array_str(right_diagonal):
                self.isDone = True

        if self.isDone:
            return self.reward['win']
        # check for draw
        elif np.sum([self.board_state == 0]) == 0:
            self.isDone = True
            return self.reward['draw']
        else:
            return 0.

    def make_move(self, a, player):
        # check if move is valid
        if a in self.get_available_actions():
            i = np.sum([self.board_state[:, a] == 0]) - 1
            self.board_state[i, a] = self.players[player]
        else:
            print('Move is invalid')
            self.render()

        reward = self.check_game_done(player)

        # give feedback as new state and reward
        return self.board_state.copy(), reward

env = connect_x()
```

# Experience replay

We’ll be using experience replay for training our DQN. It stores the transitions that the agent observes, allowing us to reuse this data later. By sampling from it randomly, the transitions that build up a batch are decorrelated. It has been shown that this greatly stabilizes and improves the DQN training procedure.

For a brief review of more advanced sampling algorithm mentioned in AlphaGo zero, its sampling procedure can be divided into 3 phases.

* Optimization:
      It checkpoints the deep learning agent by recording its weights every 1000 training iterations.
* Evaluation:
      Evaluate the checkpoints model by letting the two playing against each other for 400 games, the one wins > 55% will become the current best model (with epsilon = 0 = no exploration).
* Self-play:
      They will only use the current best model or player to generate training data for the optimization step.

For a more deep dive of the AlphaGo zero's impressive algorithm, readers can refer to the original paper or this post - AlphaGo Zero — a game changer, the author did a great job of summarizing the details from MCTS, ResNet to self-playing.

```python
import random

# memory block for deep q learning
class replayMemory:
    def __init__(self):
        self.memory = []

    def dump(self, transition_tuple):
        self.memory.append(transition_tuple)

    def sample(self, batch_size):
        return random.sample(self.memory, batch_size)

    def __len__(self):
        return len(self.memory)

memory = replayMemory()
```

# Deep Q-network

Our model will be using a convolutional neural network that takes into the state as an image of board_state and output the state action function value for all states. In effect, the network is trying to predict the expected return of taking each action given the current input. But be aware that, the game's action space is changing so agent can only take legal actions.

More advanced way to deal with training a better function approximator for fully observation games like Go and Connect-4 is to preprocess input images. AlphaGo zero separated the training input image into two binary matrix of 0 and 1s made by white and black stones. Also, to handle strategic moves better, it includes the history (last 8 moves of game) as input, plus 1 extra dimension as 0 or 1 indicating whether it is the players turn to make a move, in their case, the input is therfore 19 \* 19 \* 17 (2 * 8 + 1) image vector. The output is a policy vector of 19 \* 19 (the probabilities of making a moves) and a state value scalar.

For fast experimentation, our DQN will just use a single image map as 6 \* 7 \* 1 with 1 as the moves made by player 1 and 2 as the moves made by player 2.

```python
import torch
import torch.nn as nn
import torch.nn.functional as F

class DQN(nn.Module):

    def __init__(self, outputs):
        super(DQN, self).__init__()
        # 6 by 7, 10 by 11
        self.conv1 = nn.Conv2d(1, 32, kernel_size=5, padding=2)
        self.conv2 = nn.Conv2d(32, 32, kernel_size=5, padding=2)
        self.conv3 = nn.Conv2d(32, 32, kernel_size=5, padding=2)
        self.conv4 = nn.Conv2d(32, 32, kernel_size=5, padding=2)
        self.conv5 = nn.Conv2d(32, 32, kernel_size=5, padding=2)
        self.conv6 = nn.Conv2d(32, 32, kernel_size=5, padding=2)
        self.conv7 = nn.Conv2d(32, 32, kernel_size=5, padding=2)

        linear_input_size = 6 * 7 * 32
        self.MLP1 = nn.Linear(linear_input_size, 50)
        self.MLP2 = nn.Linear(50, 50)
        self.MLP3 = nn.Linear(50, 50)
        self.MLP4 = nn.Linear(50, outputs)

    def forward(self, x):
        x = F.leaky_relu(self.conv1(x))
        x = F.leaky_relu(self.conv2(x))
        x = F.leaky_relu(self.conv3(x))
        x = F.leaky_relu(self.conv4(x))
        x = F.leaky_relu(self.conv5(x))
        x = F.leaky_relu(self.conv6(x))
        x = F.leaky_relu(self.conv7(x))
        # flatten the feature vector except batch dimension
        x = x.view(x.size(0), -1)
        x = F.leaky_relu(self.MLP1(x))
        x = F.leaky_relu(self.MLP2(x))
        x = F.leaky_relu(self.MLP3(x))
        return self.MLP4(x)
```

# Agent
Below created 2 models of the same DQN object, termed as target_net and policy_net. The creation of 2 nets instead of 1 is to stablilize the training by a lagged goal provided by target_net. The reasonale behind this will be explained more in the "Optimizing policy" section.

To accelerate training with at least 100x, below uses the free CPU provided by Google Colab. To use GPU in Pytorch, first is setting the CUDA device, then send the model, input and output tensors to GPU memory.

```python
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
# Assuming that we are on a CUDA machine, this should print a CUDA device:
print(device)
```

    cuda:0

```python
import matplotlib.pyplot as plt
# epilson decay graph
EPS_START = 0.9
EPS_END = 0.05
EPS_DECAY = 2000

steps_done = np.arange(20000)
eps = EPS_END + (EPS_START - EPS_END) * np.exp(-1 * steps_done / EPS_DECAY)
plt.plot(steps_done, eps)
plt.title('Epsilon decay graph')
plt.xlabel('Episode no.')
plt.ylabel('Epsilon')
plt.show()
```

```python
import torch.optim as optim
import math

BATCH_SIZE = 256
GAMMA = 0.999

# get max no. of actions from action space
n_actions = env.board_width

height = env.board_height
width = env.board_width

policy_net = DQN(n_actions).to(device)
# target_net will be updated every n episodes to tell policy_net a better estimate of how far off from convergence
target_net = DQN(n_actions).to(device)
target_net.load_state_dict(policy_net.state_dict())
# set target_net in testing mode
target_net.eval()

optimizer = optim.Adam(policy_net.parameters())

def select_action(state, available_actions, steps_done=None, training=True):
    # batch and color channel
    state = torch.tensor(state, dtype=torch.float, device=device).unsqueeze(dim=0).unsqueeze(dim=0)
    epsilon = random.random()
    if training:
        eps_threshold = EPS_END + (EPS_START - EPS_END) * math.exp(-1 * steps_done / EPS_DECAY)
    else:
        eps_threshold = 0

    # follow epsilon-greedy policy
    if epsilon > eps_threshold:
        with torch.no_grad():
            # action recommendations from policy net
            r_actions = policy_net(state)[0, :]
            state_action_values = [r_actions[action] for action in available_actions]
            argmax_action = np.argmax(state_action_values)
            greedy_action = available_actions[argmax_action]
            return greedy_action
    else:
        return random.choice(available_actions)
```

# Optmizing policy

Now, let's define our model. But first, let quickly recap what a DQN is.

Model based environment is deterministic, but model-free is not.
Our aim will be to train a policy that tries to maximize the discounted,
cumulative reward
$R_{t_0} = \sum_{t=t_0}^{\infty} \gamma^{t - t_0} r_t$, where
$R_{t_0}$ is also known as the *return*. The discount,
$\gamma$, should be a constant between $0$ and $1$
that ensures the sum converges. It makes rewards from the uncertain far
future less important for our agent than the ones in the near future
that it can be fairly confident about.

The main idea behind Q-learning is that if we had a function
$Q^*: State \times Action \rightarrow \mathbb{R}$, that could tell
us what our return would be, if we were to take an action in a given
state, then we could easily construct a policy that maximizes our
rewards:

\begin{align}\pi^*(s) = \arg\!\max_a \ Q^*(s, a)\end{align}

However, we don't know everything about the world, so we don't have
access to $Q^*$. But, since neural networks are universal function
approximators, we can simply create one and train it to resemble
$Q^*$.

For our training update rule, we'll use a fact that every $Q$
function for a optimal policy in a Markov Decision Process obeys the Bellman optimality equation:

\begin{align}Q^{\pi}(s, a) = \mathbb{E}_{(s, a, s', r)}  r + \gamma Q^{\pi}(s', \pi(s'))\end{align}

In the training of a Deep Q network, the RHS is the 'Prediction' from policy_net, the LHS is the 'Truth' from target_net.
The difference between the two sides of the equality is known as the
temporal difference error, $\delta$:

\begin{align}\delta = Q(s, a) - (r + \gamma \max_a Q(s', a))\end{align}

To minimise this error, we will use the [Huber
loss](https://en.wikipedia.org/wiki/Huber_loss). The Huber loss acts
like the mean squared error when the error is small, but like the mean
absolute error when the error is large - this makes it more robust to
outliers when the estimates of $Q$ are very noisy. We calculate
this over a batch of transitions, $B$, sampled from the replay
memory:

\begin{align}\mathcal{L} = \frac{1}{|B|}\sum_{(s, a, s', r) \ \in \ B} \mathcal{L}(\delta)\end{align}

\begin{align}\text{where} \quad \mathcal{L}(\delta) = \begin{cases}
     \frac{1}{2}{\delta^2}  & \text{for } |\delta| \le 1, \\
     |\delta| - \frac{1}{2} & \text{otherwise.}
   \end{cases}\end{align}

```python
def optimize_model():
    if len(memory) < BATCH_SIZE:
        return
    transitions = memory.sample(BATCH_SIZE)
    state_batch, action_batch, reward_batch, next_state_batch = zip(*[(np.expand_dims(m[0], axis=0), \
                                        [m[1]], m[2], np.expand_dims(m[3], axis=0)) for m in transitions])
    # tensor wrapper
    state_batch = torch.tensor(state_batch, dtype=torch.float, device=device)
    action_batch = torch.tensor(action_batch, dtype=torch.long, device=device)
    reward_batch = torch.tensor(reward_batch, dtype=torch.float, device=device)

    # for assigning terminal state value = 0 later
    non_final_mask = torch.tensor(tuple(map(lambda s_: s_[0] is not None, next_state_batch)), device=device)
    non_final_next_state = torch.cat([torch.tensor(s_, dtype=torch.float, device=device).unsqueeze(0) for s_ in next_state_batch if s_[0] is not None])

    # prediction from policy_net
    state_action_values = policy_net(state_batch).gather(1, action_batch)

    # truth from target_net, initialize with zeros since terminal state value = 0
    next_state_values = torch.zeros(BATCH_SIZE, device=device)
    # tensor.detach() creates a tensor that shares storage with tensor that does not require grad
    next_state_values[non_final_mask] = target_net(non_final_next_state).max(1)[0].detach()
    # compute the expected Q values
    expected_state_action_values = (next_state_values * GAMMA) + reward_batch

    # Compute Huber loss
    loss = F.smooth_l1_loss(state_action_values, expected_state_action_values.unsqueeze(1)) # torch.tensor.unsqueeze returns a copy

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
```

# Training loop
Our agent (policy_net) will play against random agent, as a baby step, it will learn how to play and win the game at a beginner level. The agent is expected to perform better than ignorance player at the end of episodes. In short, the student is expected to outperform the teacher. It is the sole goal of training.

```python
# random agent
def random_agent(actions):
    return random.choice(actions)

# win rate test
def win_rate_test():
    win_moves_taken_list = []
    win = []
    for i in range(100):
        env.reset()
        win_moves_taken = 0

        while not env.isDone:
            state = env.board_state.copy()
            available_actions = env.get_available_actions()
            action = select_action(state, available_actions, training=False)
            state, reward = env.make_move(action, 'p1')
            win_moves_taken += 1

            if reward == 1:
                win_moves_taken_list.append(win_moves_taken)
                win.append(1)
                break

            available_actions = env.get_available_actions()
            action = random_agent(available_actions)
            state, reward = env.make_move(action, 'p2')

    return sum(win)/100, sum(win_moves_taken_list)/len(win_moves_taken_list)
```

```python
# avoid resetting
steps_done = 0
training_history = []
```

```python
from itertools import count

num_episodes = 20000
# control how lagged is target network by updating every n episodes
TARGET_UPDATE = 10

for i in range(num_episodes):
    env.reset()
    state_p1 = env.board_state.copy()

    # record every 20 epochs
    if i % 20 == 19:
        win_rate, moves_taken = win_rate_test()
        training_history.append([i + 1, win_rate, moves_taken])
        th = np.array(training_history)
        # print training message every 200 epochs
        if i % 200 == 199:
            print('Episode {}: | win_rate: {} | moves_taken: {}'.format(i, th[-1, 1], th[-1, 2]))

    for t in count():
        available_actions = env.get_available_actions()
        action_p1 = select_action(state_p1, available_actions, steps_done)
        steps_done += 1
        state_p1_, reward_p1 = env.make_move(action_p1, 'p1')

        if env.isDone:
            if reward_p1 == 1:
                # reward p1 for p1's win
                memory.dump([state_p1, action_p1, 1, None])
            else:
                # state action value tuple for a draw
                memory.dump([state_p1, action_p1, 0.5, None])
            break

        available_actions = env.get_available_actions()
        action_p2 = random_agent(available_actions)
        state_p2_, reward_p2 = env.make_move(action_p2, 'p2')

        if env.isDone:
            if reward_p2 == 1:
                # punish p1 for (random agent) p2's win
                memory.dump([state_p1, action_p1, -1, None])
            else:
                # state action value tuple for a draw
                memory.dump([state_p1, action_p1, 0.5, None])
            break

        # punish for taking too long to win
        memory.dump([state_p1, action_p1, -0.05, state_p2_])
        state_p1 = state_p2_

        # Perform one step of the optimization (on the policy network)
        optimize_model()

    # update the target network, copying all weights and biases in DQN
    if i % TARGET_UPDATE == TARGET_UPDATE - 1:
        target_net.load_state_dict(policy_net.state_dict())

print('Complete')
```

    Episode 199: | win_rate: 0.74 | moves_taken: 7.45945945945946
    Episode 399: | win_rate: 0.78 | moves_taken: 7.923076923076923
    Episode 599: | win_rate: 0.77 | moves_taken: 9.155844155844155
    Episode 799: | win_rate: 0.67 | moves_taken: 6.253731343283582
    Episode 999: | win_rate: 0.69 | moves_taken: 9.014492753623188
    Episode 1199: | win_rate: 0.67 | moves_taken: 9.17910447761194
    Episode 1399: | win_rate: 0.9 | moves_taken: 7.444444444444445
    Episode 1599: | win_rate: 0.72 | moves_taken: 8.98611111111111
    Episode 1799: | win_rate: 0.8 | moves_taken: 8.7375
    Episode 1999: | win_rate: 0.83 | moves_taken: 6.963855421686747
    Episode 2199: | win_rate: 0.94 | moves_taken: 5.968085106382978
    Episode 2399: | win_rate: 0.87 | moves_taken: 6.482758620689655
    Episode 2599: | win_rate: 0.89 | moves_taken: 5.786516853932584
    Episode 2799: | win_rate: 0.92 | moves_taken: 6.130434782608695
    Episode 2999: | win_rate: 0.88 | moves_taken: 6.715909090909091
    Episode 3199: | win_rate: 0.88 | moves_taken: 6.2727272727272725
    Episode 3399: | win_rate: 0.93 | moves_taken: 6.236559139784946
    Episode 3599: | win_rate: 0.95 | moves_taken: 5.894736842105263
    Episode 3799: | win_rate: 0.93 | moves_taken: 5.709677419354839
    Episode 3999: | win_rate: 0.94 | moves_taken: 6.0212765957446805
    Episode 4199: | win_rate: 0.89 | moves_taken: 6.022471910112359
    Episode 4399: | win_rate: 0.89 | moves_taken: 5.191011235955056
    Episode 4599: | win_rate: 0.85 | moves_taken: 6.188235294117647
    Episode 4799: | win_rate: 0.96 | moves_taken: 5.28125
    Episode 4999: | win_rate: 0.89 | moves_taken: 5.606741573033708
    Episode 5199: | win_rate: 0.87 | moves_taken: 6.0344827586206895
    Episode 5399: | win_rate: 0.97 | moves_taken: 5.948453608247423
    Episode 5599: | win_rate: 0.84 | moves_taken: 7.261904761904762
    Episode 5799: | win_rate: 0.91 | moves_taken: 6.021978021978022
    Episode 5999: | win_rate: 0.91 | moves_taken: 5.8791208791208796
    Episode 6199: | win_rate: 0.91 | moves_taken: 5.8901098901098905
    Episode 6399: | win_rate: 0.89 | moves_taken: 5.865168539325842
    Episode 6599: | win_rate: 0.93 | moves_taken: 5.494623655913978
    Episode 6799: | win_rate: 0.9 | moves_taken: 6.277777777777778
    Episode 6999: | win_rate: 0.94 | moves_taken: 6.4361702127659575
    Episode 7199: | win_rate: 0.88 | moves_taken: 5.988636363636363
    Episode 7399: | win_rate: 0.92 | moves_taken: 6.086956521739131
    Episode 7599: | win_rate: 0.91 | moves_taken: 6.087912087912088
    Episode 7799: | win_rate: 0.9 | moves_taken: 5.633333333333334
    Episode 7999: | win_rate: 0.85 | moves_taken: 6.105882352941176
    Episode 8199: | win_rate: 0.92 | moves_taken: 5.706521739130435
    Episode 8399: | win_rate: 0.87 | moves_taken: 5.873563218390805
    Episode 8599: | win_rate: 0.95 | moves_taken: 6.11578947368421
    Episode 8799: | win_rate: 0.96 | moves_taken: 5.552083333333333
    Episode 8999: | win_rate: 0.92 | moves_taken: 5.445652173913044
    Episode 9199: | win_rate: 0.91 | moves_taken: 5.571428571428571
    Episode 9399: | win_rate: 0.92 | moves_taken: 5.5
    Episode 9599: | win_rate: 0.91 | moves_taken: 6.010989010989011
    Episode 9799: | win_rate: 0.9 | moves_taken: 5.477777777777778
    Episode 9999: | win_rate: 0.87 | moves_taken: 5.574712643678161
    Episode 10199: | win_rate: 0.91 | moves_taken: 5.846153846153846
    Episode 10399: | win_rate: 0.95 | moves_taken: 5.021052631578947
    Episode 10599: | win_rate: 0.89 | moves_taken: 5.786516853932584
    Episode 10799: | win_rate: 0.88 | moves_taken: 5.75
    Episode 10999: | win_rate: 0.95 | moves_taken: 5.168421052631579
    Episode 11199: | win_rate: 0.81 | moves_taken: 5.469135802469136
    Episode 11399: | win_rate: 0.88 | moves_taken: 5.7272727272727275
    Episode 11599: | win_rate: 0.87 | moves_taken: 5.551724137931035
    Episode 11799: | win_rate: 0.91 | moves_taken: 5.604395604395604
    Episode 11999: | win_rate: 0.86 | moves_taken: 5.616279069767442
    Episode 12199: | win_rate: 0.93 | moves_taken: 6.301075268817204
    Episode 12399: | win_rate: 0.88 | moves_taken: 5.761363636363637
    Episode 12599: | win_rate: 0.91 | moves_taken: 5.2967032967032965
    Episode 12799: | win_rate: 0.88 | moves_taken: 5.363636363636363
    Episode 12999: | win_rate: 0.93 | moves_taken: 5.795698924731183
    Episode 13199: | win_rate: 0.94 | moves_taken: 5.0
    Episode 13399: | win_rate: 0.96 | moves_taken: 5.604166666666667
    Episode 13599: | win_rate: 0.9 | moves_taken: 5.455555555555556
    Episode 13799: | win_rate: 0.9 | moves_taken: 5.622222222222222
    Episode 13999: | win_rate: 0.9 | moves_taken: 5.444444444444445
    Episode 14199: | win_rate: 0.9 | moves_taken: 5.911111111111111
    Episode 14399: | win_rate: 0.92 | moves_taken: 5.673913043478261
    Episode 14599: | win_rate: 0.89 | moves_taken: 5.606741573033708
    Episode 14799: | win_rate: 0.93 | moves_taken: 5.483870967741935
    Episode 14999: | win_rate: 0.91 | moves_taken: 5.604395604395604
    Episode 15199: | win_rate: 0.89 | moves_taken: 5.786516853932584
    Episode 15399: | win_rate: 0.94 | moves_taken: 5.3936170212765955
    Episode 15599: | win_rate: 0.91 | moves_taken: 4.967032967032967
    Episode 15799: | win_rate: 0.89 | moves_taken: 5.51685393258427
    Episode 15999: | win_rate: 0.94 | moves_taken: 5.9361702127659575
    Episode 16199: | win_rate: 0.95 | moves_taken: 5.4
    Episode 16399: | win_rate: 0.91 | moves_taken: 5.56043956043956
    Episode 16599: | win_rate: 0.96 | moves_taken: 5.625
    Episode 16799: | win_rate: 0.94 | moves_taken: 5.372340425531915
    Episode 16999: | win_rate: 0.96 | moves_taken: 5.625
    Episode 17199: | win_rate: 0.87 | moves_taken: 5.172413793103448
    Episode 17399: | win_rate: 0.99 | moves_taken: 5.08080808080808
    Episode 17599: | win_rate: 0.93 | moves_taken: 5.096774193548387
    Episode 17799: | win_rate: 0.94 | moves_taken: 5.468085106382978
    Episode 17999: | win_rate: 0.94 | moves_taken: 5.680851063829787
    Episode 18199: | win_rate: 0.96 | moves_taken: 5.53125
    Episode 18399: | win_rate: 0.93 | moves_taken: 5.494623655913978
    Episode 18599: | win_rate: 0.89 | moves_taken: 5.910112359550562
    Episode 18799: | win_rate: 0.92 | moves_taken: 5.663043478260869
    Episode 18999: | win_rate: 0.97 | moves_taken: 5.43298969072165
    Episode 19199: | win_rate: 0.95 | moves_taken: 5.905263157894737
    Episode 19399: | win_rate: 0.97 | moves_taken: 5.350515463917525
    Episode 19599: | win_rate: 0.89 | moves_taken: 5.932584269662922
    Episode 19799: | win_rate: 0.91 | moves_taken: 5.329670329670329
    Episode 19999: | win_rate: 0.96 | moves_taken: 5.020833333333333
    Complete

```python
plt.plot(th[:, 0], th[:, 1], c='c')
win_rate_moving_average = np.array([[(i + 19) * 20, np.mean(th[i: i + 20, 1])] for i in range(len(th) - 19)])
plt.plot(win_rate_moving_average[:, 0], win_rate_moving_average[:, 1], c='b', label='moving average of win rate')
plt.legend()
plt.title('Playing against random agent')
plt.xlabel('Episode no.')
plt.ylabel('Win rate')
plt.show()

plt.plot(th[:, 0], th[:, 2], c='c')
win_steps_taken_moving_average = np.array([[(i + 19) * 20, np.mean(th[i: i + 20, 2])] for i in range(len(th) - 19)])
plt.plot(win_steps_taken_moving_average[:, 0], win_steps_taken_moving_average[:, 1], c='b', label='moving average of win steps taken')
plt.legend()
plt.xlabel('Episode no.')
plt.ylabel('Average steps taken for a win')
plt.show()
```

As one can see from the graph above,
* The variance of win rate is getting smaller and the moving average for win rate is also getting higher.
* The average win steps taken has gone down, which means the agent has learnt to win as fast as possible.

Both are the indication of the fact that our agent has learnt how to win Connect 4.

```python
path = 'DQN_plainCNN.pth'
torch.save(policy_net.state_dict(), path)
```

# Demonstration of a trained agent playing a game

```python
def demo():
    env.reset()
    env.render()

    while not env.isDone:
        state = env.board_state.copy()
        available_actions = env.get_available_actions()
        action = select_action(state, available_actions, training=False)
        # trained agent's move is denoted by O
        state, reward = env.make_move(action, 'p1')
        env.render()

        if reward == 1:
            break

        available_actions = env.get_available_actions()
        action = random_agent(available_actions)
        state, reward = env.make_move(action, 'p2')
        env.render()

demo()
```

      0
      1
      2
      3
      4
      5
      6

      0

      1

      2

      3

      4

      5

      0
      1
      2
      3
      4
      5
      6

      0

      1

      2

      3

      4

      5

      O

      0
      1
      2
      3
      4
      5
      6

      0

      1

      2

      3

      4

      5

      O

      X

      0
      1
      2
      3
      4
      5
      6

      0

      1

      2

      3

      4

      O

      5

      O

      X

      0
      1
      2
      3
      4
      5
      6

      0

      1

      2

      3

      X

      4

      O

      5

      O

      X

      0
      1
      2
      3
      4
      5
      6

      0

      1

      2

      3

      X

      4

      O

      5

      O
      O

      X

      0
      1
      2
      3
      4
      5
      6

      0

      1

      2

      3

      X

      4

      O
      X

      5

      O
      O

      X

      0
      1
      2
      3
      4
      5
      6

      0

      1

      2

      3

      X
      O

      4

      O
      X

      5

      O
      O

      X

      0
      1
      2
      3
      4
      5
      6

      0

      1

      2

      X

      3

      X
      O

      4

      O
      X

      5

      O
      O

      X

      0
      1
      2
      3
      4
      5
      6

      0

      1

      2

      X

      3

      X
      O

      4

      O
      X

      5

      O
      O
      O

      X

      0
      1
      2
      3
      4
      5
      6

      0

      1

      2

      X

      3

      X
      O

      4

      O
      X

      X

      5

      O
      O
      O

      X

      0
      1
      2
      3
      4
      5
      6

      0

      1

      2

      X

      3

      X
      O

      4

      O
      O
      X

      X

      5

      O
      O
      O

      X

      0
      1
      2
      3
      4
      5
      6

      0

      1

      2

      X

      3

      X
      O

      4

      O
      O
      X

      X

      5

      O
      O
      O
      X
      X

      0
      1
      2
      3
      4
      5
      6

      0

      1

      2

      X

      3

      X
      O

      4

      O
      O
      X

      X

      5

      O
      O
      O
      O
      X
      X

From the above gameplay, the agent (denoted by O) knows putting in (5, 1) is a winning move even after its original strategy is blocked by p2 in step 5 by putting X in (3, 2).

```python

```


This notebook is part of the `nbsphinx` documentation: http://nbsphinx.readthedocs.io/.

# Markdown Cells

We can use *emphasis*, **boldface**, `preformatted text`.

> It looks like strike-out text is not supported: ~~strikethrough~~.

* Red
* Green
* Blue

***

1. One
1. Two
1. Three

## Equations

Equations can be formatted really nicely, either inline, like $\text{e}^{i\pi} = -1$, or on a separate line, like

\begin{equation}
\int_{-\infty}^\infty f(x) \delta(x - x_0) dx = f(x_0)
\end{equation}

*Note:* Avoid leading and trailing spaces around math expressions, otherwise errors like the following will occur when Sphinx is running:

    ERROR: Unknown interpreted text role "raw-latex".

See also the [pandoc docs](http://pandoc.org/MANUAL.html#math):

> Anything between two `$` characters will be treated as TeX math. The opening `$` must have a non-space character immediately to its right, while the closing `$` must have a non-space character immediately to its left, and must not be followed immediately by a digit.

## Code

We can also write code with nice syntax highlighting:

```python3
print("Hello, world!")
```

## Tables

A     | B     | A and B
------|-------|--------
False | False | False
True  | False | False
False | True  | False
True  | True  | True

## Images

PNG file (local):

SVG file (local):

PNG file (remote):

SVG file (remote):

## HTML Elements (HTML only)

It is allowed to use plain HTML elements within Markdown cells.
Those elements are passed through to the HTML output and are ignored for the LaTeX output.
Below are a few examples.

HTML5 [audio](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/audio) elements can be created like this:

```html
<audio src="https://example.org/audio.ogg" controls>alternative text</audio>
```

Example:

<audio src="https://upload.wikimedia.org/wikipedia/commons/6/61/DescenteInfinie.ogg" controls>The HTML audio element is not supported!</audio>

HTML5 [video](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/video) elements can be created like this:

```html

```

Example:

The alternative text is shown in browsers that don't support those elements. The same text is also shown in Sphinx's LaTeX output.

**Note:** You can also use local files for the `<audio>` and `<video>` elements, but you have to create a link to the source file somewhere, because only then are the local files copied to the HTML output directory!
You should do that anyway to make the audio/video file accessible to browsers that don't support the `<audio>` and `<video>` elements.

## Info/Warning Boxes

**Warning:**

This is an *experimental feature*!
Its usage will probably change in the future or it might be removed completely!

Until there is an info/warning extension for Markdown/CommonMark (see [this issue](https://github.com/jupyter/notebook/issues/1292)), such boxes can be created by using HTML `<div>` elements like this:

```html
<div class="alert alert-info">

**Note:** This is a note!

</div>
```

For this to work reliably, you should obey the following guidelines:

* The `class` attribute has to be either `"alert alert-info"` or `"alert alert-warning"`, other values will not be converted correctly.
* No further attributes are allowed.
* For compatibility with CommonMark, you should add an empty line between the `<div>` start tag and the beginning of the content.

**Note:**

The text can contain further Markdown formatting.
It is even possible to have nested boxes:

... but please don't *overuse* this!

## Links to Other Notebooks

Relative links to local notebooks can be used:
[a link to a notebook in a subdirectory](subdir/a-notebook-in-a-subdir.ipynb),
[a link to an orphan notebook](orphan.ipynb)
(latter won't work in LaTeX output, because orphan pages are not included there).

This is how a link is created in Markdown:

```
[a link to a notebook in a subdirectory](subdir/a-notebook-in-a-subdir.ipynb)
```

Markdown also supports *reference-style* links:
[a reference-style link][mylink],
[another version of the same link][mylink].

[mylink]: subdir/a-notebook-in-a-subdir.ipynb

These can be created with this syntax:

```
[a reference-style link][mylink]

[mylink]: subdir/a-notebook-in-a-subdir.ipynb
```

Links to sub-sections are also possible, e.g.
[this subsection](subdir/a-notebook-in-a-subdir.ipynb#A-Sub-Section).

This link was created with:

```
[this subsection](subdir/a-notebook-in-a-subdir.ipynb#A-Sub-Section)
```

You just have to remember to replace spaces with hyphens!

BTW, links to sections of the current notebook work, too, e.g.
[beginning of this section](#Links-to-Other-Notebooks).

This can be done, as expected, like this:

```
[beginning of this section](#Links-to-Other-Notebooks)
```

## Links to `*.rst` Files (and Other Sphinx Source Files)

Links to files whose extension is in the configuration value [source_suffix](http://www.sphinx-doc.org/config.html#confval-source_suffix), will be converted to links to the generated HTML/LaTeX pages. Example: [A reStructuredText file](a-normal-rst-file.rst).

This was created with:

```
[A reStructuredText file](a-normal-rst-file.rst)
```

Links to sub-sections are also possible. Example: [Sphinx Directives](a-normal-rst-file.rst#Sphinx-Directives-for-Info/Warning-Boxes).

This was created with:

```
[Sphinx Directives](a-normal-rst-file.rst#Sphinx-Directives-for-Info/Warning-Boxes)
```

Again, you'll have to replace spaces with hyphens in sub-section titles.

## Links to Local Files (HTML only)

Links to local files (other than Jupyter notebooks and other Sphinx source files) are also possible, e.g. [requirements.txt](requirements.txt).

This was simply created with:

```
[requirements.txt](requirements.txt)
```

The linked files are automatically copied to the HTML output directory.
For LaTeX output, no link is created.

## Links to Domain Objects

Links to [Sphinx domain objects](http://www.sphinx-doc.org/en/stable/domains.html) (such as a Python class or JavaScript function) are also possible. For example: [example_python_function](a-normal-rst-file.rst#example_python_function).

This was created with:

```
[example_python_function](a-normal-rst-file.rst#example_python_function)
```

This is especially useful for use with the Sphinx [autodoc](http://www.sphinx-doc.org/ext/autodoc.html) extension!


# Week 6 - Sequential Models
## 1. Recurrent Neural Networks (RNNs)

### 1.1 RNN as a sequential model

The neural networks in previous tutorials are **feedforward neural networks**, which take one input (e.g. information about one credit card transaction) and transform it onto one output (e.g. fraudulent or not). Here, the model processes each instance independently.

In sequential data however, instances of data are not independent. Sequential data is data where order matters, including text streams, audio clips, video clips, time series like stock prices. Predicting the next value in the sequence often requires information about many past values in the sequence, not just the latest value.

**Recurrent neural networks** are a type of neural networks suitable for sequential data. Unike feedforward neural networks, RNN keeps a memory of past inputs. The newest input (latest time step in the sequence) is combined with this memory to generate an output that continues the sequence.

(E.g. to predict the closing price of a stock tomorrow, the RNN takes today's closing price, as well as memory of closing prices in the past n number of days.)

```python
!pip install --upgrade pandas_datareader

import pandas_datareader.data as web
import pandas as pd
import matplotlib.pyplot as plt
import datetime as dt
```

```python
# Use pandas datareader to obtain stock data from Yahoo Finance
data = web.DataReader('AAPL', 'yahoo', start='2021-01-01', end='2021-11-23')

data.head()
```

      High
      Low
      Open
      Close
      Volume
      Adj Close

      Date

      2021-01-04
      133.610001
      126.760002
      133.520004
      129.410004
      143301900.0
      128.617096

      2021-01-05
      131.740005
      128.429993
      128.889999
      131.009995
      97664900.0
      130.207306

      2021-01-06
      131.050003
      126.379997
      127.720001
      126.599998
      155088000.0
      125.824318

      2021-01-07
      131.630005
      127.860001
      128.360001
      130.919998
      109578200.0
      130.117844

      2021-01-08
      132.630005
      130.229996
      132.429993
      132.050003
      105158200.0
      131.240921

```python
# visualizing the daily closing price
close=data['Close']

fig, ax = plt.subplots(figsize=(8,5))
ax.plot(close.index,close,label="AAPL")
ax.set_xlabel("Date")
ax.set_ylabel("Closing Price ($)")
fig.suptitle('Daily closing price', fontsize=15)
ax.legend()
print(close)
```

### 1.2 RNN structure and layers

#### How is a recurrent neuron different from the usual neuron?

1. **The recurrent neuron as a hidden state / cell state** ($h_t$): the internal memory of the recurrent neuron. It is initialized as a zero matrix, and updated after each time step to include the information inputted at that time step.

2. **The recurrent neuron's output is a function of input and memory ($y_t = f(X_t,  h_{t-1})$), instead of just a function of input($y_t = f(X_t)$)**

#### Two layers of an RNN:
1. **Recurrent layer**: updating the hidden state

$h_t = f(h_{t-1} \times W_{hh} + X_t \times W_xh + B_1)$

$h_t$: hidden state at time step $t$. $h_0$ is initialized as a matrix of zeros.
$W_{hh}$: weight matrix determining how the previous hidden state affects the current hidden state
$W_{xh}$: weight matrix determining how the current input ($X$) affects the current hidden state
$B_1$: bias of the recurrent layer
$f()$: activation function of the recurrent layer

2. **Feedforward layer**: using the updated hidden state to produce an output

$\hat{y}_t = g(h_t \times W_{hy} + B_2)$

$\hat{y}_t$: output at time step $t$
$W_{hy}$: weight matrix determining how current hidden state affects output
$B_1$: bias of the feedforward layer
$g()$: activation function of the feedforward layer

Note: once trained, the same weight matrices will be used across all time steps.

#### Activation functions
Just like neural networks in previous tutorials, RNNs need activation functions to introduce non-linearity and enable learning of more complex patterns.

Common activation functions for RNN:
- sigmoid
- ReLU
- tanh

### 1.3 Back propagation through time
- Step 1: Values of an input sequence are passed through the model in order. At each time step, the model generates an output and calculates its loss.

    $$L_t = f_l(\hat{y}, y)$$

    $L_t$: loss of the output at time $t$

    $f_l$: loss function

    $\hat{y}$: prediction output

    $y$: actual label

- Step 2: The losses of all the time steps are summed into a total loss for the input sequence.

    $$L = \sum_{t} L_t $$

    $L$: total loss

- Step 3: Error is propagated back through each time step by taking the derivative of total loss with respect to each trainable parameter

    $$\frac{\partial L}{\partial \theta} = \sum_{t} \frac{\partial L_t}{\partial \theta}
    $$

    Where $\theta$ is any trainable parameter.

    The equation above represents the derivative of the total loss as a sum of the derivatives of losses at each time step. These time step losses are each calculated with:
    \begin{equation}
    \frac{\partial L_t}{\partial \theta} = \sum_{k=1}^{t} \frac{\partial L_t}{\partial y_t} \frac{\partial y_t}{\partial h_t}  \frac{\partial h_t}{\partial h_k} \frac{\partial h_k}{\partial \theta}
    \end{equation}

    Thus, the derivative of the total loss is:

    \begin{equation}
    \frac{\partial L}{\partial \theta} = \sum_{t} \sum_{k=1}^{t} \frac{\partial L_t}{\partial y_t} \frac{\partial y_t}{\partial h_t}  \frac{\partial h_t}{\partial h_k} \frac{\partial h_k}{\partial \theta}
    \end{equation}

    This derivative w.r.t each weight is multiplied by a learning rate to adjust the weights, and learning is achieved.

### 1.4 RNN implementation from scratch

So far, we have defined the structure of simple recurrent neural network; however, how does this translate into code?

In the following section, we will have a look into implementation from scratch for the prediction of simple sine function.

For starters, let's create a sample dataset by dividing the sine function into sequences of 50 units. For the model itself, we will try to predict the next value for the sequence.

```python
import math
import numpy as np
import matplotlib.pyplot as plt

#Preparing our dataset
sine =  np.array([math.sin(x) for x in np.arange(200)])

#Creating our train dataset
X = []
Y = []

#Defining the lenght of our sequence and number of samples
seq_len = 50
num_records = len(sine) - seq_len

for i in range(num_records - 50):
    X.append(sine[i:i+seq_len])
    Y.append(sine[i+seq_len])

X = np.array(X)
X = np.expand_dims(X, axis=2)

Y = np.array(Y)
Y = np.expand_dims(Y, axis=1)
```

To later test our model's performance, let's also create a validation dataset (we will use to after the training).

```python
#Creating our validation dataset
X_val = []
Y_val = []

for i in range(num_records - 50, num_records):
    X_val.append(sine[i:i+seq_len])
    Y_val.append(sine[i+seq_len])

X_val = np.array(X_val)
X_val = np.expand_dims(X_val, axis=2)

Y_val = np.array(Y_val)
Y_val = np.expand_dims(Y_val, axis=1)
```

Now, let's start defining our simple RNN model. As it has been mentioned earlier, our model will produce a single output after processing input through a hidden state of 100 units.

In the following cell, we also need to initialize the starting weights of our network:
- U - weights between the input and hidden layers
- V - weights between the hidden and output layers
- W - weights for shared weights in the RNN hidden layer

```python
#Creating RNN
learning_rate = 0.0001
num_epoch = 25
T = 50                   # length of sequence
hidden_dim = 100
output_dim = 1

bptt_truncate = 5
min_clip_value = -10
max_clip_value = 10

#Initializing weights
U = np.random.uniform(0, 1, (hidden_dim, T))
W = np.random.uniform(0, 1, (hidden_dim, hidden_dim))
V = np.random.uniform(0, 1, (output_dim, hidden_dim))
```

Finally, let's also define our activation function used in the hidden layer (sigmoid):

```python
def sigmoid(x):
    return 1 / (1 + np.exp(-x))
```

#### Training our model

As it has been mentioned in the previous sections, the training of RNN model can be divided into the following steps:
- Calculating loss on train dataset
- Forward pass
- Backward pass
- Updating weights

```python
for epoch in range(num_epoch):
    # defining loss
    loss = 0.0

    # generating initial prediction
    for i in range(Y.shape[0]):

        #Retrieving input and output for sequence
        x, y = X[i], Y[i]

        #Initializing value of previous hidden layer
        prev_s = np.zeros((hidden_dim, 1))
        for t in range(T):

            #Forward pass for every sequence
            new_input = np.zeros(x.shape)
            new_input[t] = x[t]

            mulu = np.dot(U, new_input)
            mulw = np.dot(W, prev_s)
            add = mulw + mulu

            #Passing through activation function
            s = sigmoid(add)
            mulv = np.dot(V, s)

            #Defining previous state
            prev_s = s

        #Calculating error
        loss_per_seq = (y - mulv)**2 / 2
        loss += loss_per_seq

    loss = loss / float(y.shape[0])

    print('Epoch: ', epoch + 1, ', Loss: ', loss)

    for i in range(Y.shape[0]):
        x, y = X[i], Y[i]

        layers = []
        prev_s = np.zeros((hidden_dim, 1))
        dU = np.zeros(U.shape)
        dV = np.zeros(V.shape)
        dW = np.zeros(W.shape)

        dU_t = np.zeros(U.shape)
        dV_t = np.zeros(V.shape)
        dW_t = np.zeros(W.shape)

        dU_i = np.zeros(U.shape)
        dW_i = np.zeros(W.shape)

        # Forward pass
        for t in range(T):
            new_input = np.zeros(x.shape)
            new_input[t] = x[t]

            #Multiplying the input witht the weights
            mulu = np.dot(U, new_input)
            mulw = np.dot(W, prev_s)

            #Adding to capture information from the previous timestep
            add = mulw + mulu

            #Passing through activation function
            s = sigmoid(add)
            mulv = np.dot(V, s)

            #Saving current and previous states
            layers.append({'s':s, 'prev_s':prev_s})
            prev_s = s

        dmulv = (mulv - y)

        # Backpropagation
        for t in range(T):
            dV_t = np.dot(dmulv, np.transpose(layers[t]['s']))
            dsv = np.dot(np.transpose(V), dmulv)

            ds = dsv
            dadd = add * (1 - add) * ds

            dmulw = dadd * np.ones_like(mulw)

            dprev_s = np.dot(np.transpose(W), dmulw)

            for i in range(t-1, max(-1, t-bptt_truncate-1), -1):
                ds = dsv + dprev_s
                dadd = add * (1 - add) * ds

                dmulw = dadd * np.ones_like(mulw)
                dmulu = dadd * np.ones_like(mulu)

                dW_i = np.dot(W, layers[t]['prev_s'])
                dprev_s = np.dot(np.transpose(W), dmulw)

                new_input = np.zeros(x.shape)
                new_input[t] = x[t]
                dU_i = np.dot(U, new_input)
                dx = np.dot(np.transpose(U), dmulu)

                dU_t += dU_i
                dW_t += dW_i

            dV += dV_t
            dU += dU_t
            dW += dW_t

            # Ensuring that the gradients' values fall within our predefined range
            # between min_clip_value and max_clip_value

            if dU.max() > max_clip_value:
                dU[dU > max_clip_value] = max_clip_value
            if dV.max() > max_clip_value:
                dV[dV > max_clip_value] = max_clip_value
            if dW.max() > max_clip_value:
                dW[dW > max_clip_value] = max_clip_value

            if dU.min() < min_clip_value:
                dU[dU < min_clip_value] = min_clip_value
            if dV.min() < min_clip_value:
                dV[dV < min_clip_value] = min_clip_value
            if dW.min() < min_clip_value:
                dW[dW < min_clip_value] = min_clip_value

        # Updating our weights
        U -= learning_rate * dU
        V -= learning_rate * dV
        W -= learning_rate * dW
```

Finally, after training, we can test our model with previously defined validation dataset.

```python
preds = []
for i in range(Y_val.shape[0]):
    x, y = X_val[i], Y_val[i]
    prev_s = np.zeros((hidden_dim, 1))

    for t in range(T):
        mulu = np.dot(U, x)
        mulw = np.dot(W, prev_s)
        add = mulw + mulu
        s = sigmoid(add)
        mulv = np.dot(V, s)
        prev_s = s

    preds.append(mulv)

preds = np.array(preds)

plt.plot(preds[:, 0, 0], 'b')
plt.plot(Y_val[:, 0], 'r')
plt.show()
```

### 1.5 RNN implementation in Tensorflow

```python
import tensorflow as tf
from keras.models import Sequential
from keras.layers import Dense, SimpleRNN
from tensorflow.keras.optimizers import SGD, Adam

from sklearn.preprocessing import MinMaxScaler
from sklearn.metrics import mean_squared_error
import numpy as np
```

```python
#check missing value
data.isna().sum()
```

```python
#scale data
scaler = MinMaxScaler()
scaled_data = scaler.fit_transform(close.values.reshape(-1,1))
scaled_data.shape
```

    (225, 1)

```python
#define functions to split data
def input_output_split(dataset, y_start, y_end, history):
  X = []
  y = []
  y_start += history
  for i in range(y_start, y_end):
    indices = range(i-history, i)
    X.append(np.reshape(dataset[indices], (history, 1)))
    y.append(dataset[i])
  return np.array(X), np.array(y)

def test_train_split(dataset, train_ratio, history):
  training_size = int(dataset.size * train_ratio)
  X_train, y_train = input_output_split(dataset, 0, training_size, history)
  X_test, y_test = input_output_split(dataset, training_size, len(dataset), history)
  return X_train, y_train, X_test, y_test

```

RNN expects input dataset to be 3D-shaped (instances, timesteps, features)

```python
#split data
X_train, y_train, X_test, y_test = test_train_split(scaled_data, 0.8, 10)

X_train.shape, y_train.shape
```

    ((170, 10, 1), (170, 1))

```python
#create model
def create_RNN(hidden_units, dense_units, input_shape, activation):
    rmodel = Sequential()

    # recurrent layer:
    rmodel.add(SimpleRNN(hidden_units, input_shape=input_shape,
                        activation=activation[0]))
    # feedfoward layer:
    rmodel.add(Dense(units=dense_units, activation=activation[1]))

    rmodel.compile(optimizer=Adam(learning_rate=1e-3), loss=tf.keras.losses.MeanSquaredError(), metrics=[tf.keras.metrics.MeanSquaredError()])

    return rmodel

RNN_model = create_RNN(8, 1, (10,1), activation=['sigmoid', 'relu'])

RNN_model.summary()
```

```python
#train model
RNN_model.fit(X_train, y_train, epochs=20, batch_size=1, verbose=2)
```

```python
#predict using test set
y_pred = RNN_model.predict(X_test)
mse = mean_squared_error(y_test, y_pred)
print('MSE: %.3f' % (mse))

y_pred = scaler.inverse_transform(y_pred).flatten()
print(y_pred)
```

    MSE: 0.003
    [139.70413 140.73264 141.66429 142.77484 142.67307 142.56572 141.56274
     140.91917 142.99507 144.19817 145.64993 147.53839 148.21982 148.4688
     147.93082 147.79124 148.27608 148.01979 150.64854 149.14284 148.19508
     148.84677 150.03354 149.84937 150.02527 149.4628  149.62015 147.55537
     147.13347 148.65814 148.9387  149.67935 151.59358 154.94313 157.27788]

```python
#visualize the predictions
y_pred = np.concatenate(([None]*(len(data)-len(y_pred)),y_pred))
result = pd.DataFrame({'predicted':y_pred, 'target':close}, index=data.index)
result.plot().set_ylabel("AAPL Closing Price ($)")
```

Reference:

https://www.ibm.com/cloud/learn/recurrent-neural-networks

http://introtodeeplearning.com/slides/6S191_MIT_DeepLearning_L2.pdf

https://www.youtube.com/watch?v=qjrad0V0uJE

https://towardsdatascience.com/sequence-models-and-recurrent-neural-networks-rnns-62cadeb4f1e1

https://towardsdatascience.com/under-the-hood-of-neural-networks-part-2-recurrent-af091247ba78

https://medium.com/deep-math-machine-learning-ai/chapter-10-deepnlp-recurrent-neural-networks-with-math-c4a6846a50a2

https://machinelearningmastery.com/understanding-simple-recurrent-neural-networks-in-keras/

https://www.youtube.com/watch?v=p5pDBGoJmR0