# Fundamentals of Data Science
Winter Semester 2021

## Prof. Fabio Galasso, Guido D'Amely, Alessandro Flaborea, Luca Franco, Muhammad Rameez Ur Rahman and Alessio Sampieri
<galasso@di.uniroma1.it>, <damely@di.uniroma1.it>, <flaborea@di.uniroma1.it>, <franco@diag.uniroma1.it>, <rahman@di.uniroma1.it>, <alessiosampieri27@gmail.com>

## Exercise 2: Classification

In Exercise 2, you will re-derive and implement logistic regression and optimize the parameters with Gradient Descent and with the Newton's method. Also, in this exercise you will re-derive and implement Gassian Discriminant Analysis.
We will use datasets generated from the make_classification function from the SkLearn library. Its first output contains the feature values $x^{(i)}_1$ and $x^{(i)}_2$ for the $i$-th data sample $x^{(i)}$. The second contains the ground truth label $y^{(i)}$ for each corresponding data sample.

The completed exercise should be handed in as a single notebook file. Use Markdown to provide equations. Use the code sections to provide your scripts and the corresponding plots.
Submit it by sending an email to galasso@di.uniroma1.it, flaborea@di.uniroma1.it, franco@diag.uniroma1.it and alessiosampieri27@gmail.com  by Wednesday November 17th 2021, 23:59.

## Notation

- $x^i$ is the $i^{th}$ feature vector
- $y^i$ is the expected outcome for the $i^{th}$ training example
- $m$ is the number of training examples
- $n$ is the number of features

Let's start by setting up our Python environment and importing the required libraries:

```python
%matplotlib inline
import numpy as np # imports a fast numerical programming library
import scipy as sp # imports stats functions, amongst other things
import matplotlib as mpl # this actually imports matplotlib
import matplotlib.cm as cm # allows us easy access to colormaps
import matplotlib.pyplot as plt # sets up plotting under plt
import pandas as pd # lets us handle data as dataframes
from sklearn.datasets import make_classification
import seaborn as sns
import math

# sets up pandas table display
pd.set_option('display.width', 500)
pd.set_option('display.max_columns', 100)
pd.set_option('display.notebook_repr_html', True)

import seaborn as sns # sets up styles and gives us more plotting options
```

## Question 1: Logistic Regression with Gradient Ascent **(10 Points)**

### Code and Theory

#### Exercise 1.a **(3 Points)** Equations for the log likelihood, its gradient, and the gradient ascent update rule.

Write and simplify the likelihood $L(\theta)$ and log-likelihood $l(\theta)$ of the parameters $\theta$.

Recall the probabilistic interpretation of the hypothesis $h_\theta(x)= P(y=1|x;\theta)$ and that $h_\theta(x)=\frac{1}{1+\exp(-\theta^T x)}$.

Also derive the gradient $\frac{\delta l(\theta)}{\delta \theta_j}$ of $l(\theta)$ and write the gradient update equation.

Question: Are we looking for a local minimum or a local maximum using the gradient ascent rule?

################# Do not write above this line #################

Your equations and answers here.

Note that $\sigma = h_\theta(x)$

As we know from the lecture $L(\theta)$ is equal to

 $L(\theta)$ = $\prod^{n}_{i=0} P(x^{(i)}|\theta)$

 we know that

 $P(y^{(i)}| x^{(i)},\theta) = \frac{1}{\sqrt{2\pi}\sigma}e^{(-\frac{(y^{(i)} - \theta x^{(i)})}{2\sigma^2})}$

 as we know

  $P(x^{(i)}|\theta) = P(y^{(i)}| x^{(i)},\theta) * y^{(i)}$

  $\Rightarrow L(\theta) = \prod^{n}_{i=0} \frac{1}{\sqrt{2\pi}\sigma}e^{(-\frac{(y^{(i)} - \theta x^{(i)})}{2\sigma^2})} * y^{(i)}$

  As we know from the lectures

  $l(\theta) = log(L(\theta))$ = $log(\prod^{n}_{i=0} P(x^{(i)}|\theta))$

  $l(\theta) = \sum^{n}_{i=0} log( \frac{1}{\sqrt{2\pi}\sigma}e^{(-\frac{(y^{(i)} - \theta x^{(i)})^2}{2\sigma^2})} * y^{(i)})$

  $l(\theta) = \sum^{n}_{i=0} log(\frac{1}{\sqrt{2\pi}\sigma})+ log(e^{(-\frac{(y^{(i)} - \theta x^{(i)})^2}{2\sigma^2})}) + y^{(i)})$

  $l(\theta) = \sum^{n}_{i=0} log(\frac{1}{\sqrt{2\pi}\sigma}) -\frac{(y^{(i)} - \theta x^{(i)})^2}{2\sigma^2}  + log(y^{(i)})$

################# Do not write below this line #################

#### Exercise 1.b **(7 Points)** Implementation of logistic regression with Gradient Ascent

Code up the equations above to learn the logistic regression parameters. The dataset used here is created using the make_classification function present in the SkLearn library. $x^{(i)}_1$ and $x^{(i)}_2$ represent the two features for the $i$-th data sample $x^{(i)}$ and $y^{(i)}$ is its ground truth label.

```python
X, y = make_classification(n_samples=500, n_features=2, n_informative=2, n_redundant=0, n_classes=2, random_state=5)
X.shape, y.shape
```

    ((500, 2), (500,))

```python
sns.scatterplot(x=X[:,0], y=X[:,1], hue=y);
```

Adding a column of 1's to $X$ to take into account the zero intercept

```python
x = np.hstack([np.ones((X.shape[0], 1)), X])
```

```python
[x[:5,:],x[-5:,:]] # Plot the first and last 5 lines of x, now containing features x0 (constant=1), x1 and x2
```

    [array([[ 1.        ,  2.25698215, -1.34710915],
            [ 1.        ,  1.43699308,  1.28420453],
            [ 1.        ,  0.57927295,  0.23690172],
            [ 1.        ,  0.42538132, -0.24611145],
            [ 1.        ,  1.13485101, -0.61162683]]),
     array([[ 1.        ,  1.56638944,  0.81749944],
            [ 1.        , -1.94913831, -1.90601147],
            [ 1.        ,  1.53440506, -0.11687238],
            [ 1.        , -0.39243599,  1.39209018],
            [ 1.        , -0.11881249,  0.96973739]])]

```python
[y[:5],y[-5:]] # Plot the first and last 5 lines of y
```

    [array([1, 1, 1, 0, 1]), array([1, 0, 0, 0, 1])]

Define the sigmoid function "sigmoid", the function to compute the gradient of the log likelihood  "grad_l" and the gradient ascent algorithm.

################# Do not write above this line #################

```python
def sigmoid(x):
    '''
    Function to compute the sigmoid of a given input x.

    Input:
    x: it's the input data matrix. The shape is (N, H)

    Output:
    g: The sigmoid of the input x
    '''

    #####################################################
    ##                 YOUR CODE HERE                  ##
    #####################################################

    x = np.array(x)

    g = 1 / ( 1 + np.exp(-x))

    return g

def log_likelihood(theta,features,target):
    '''
    Function to compute the log likehood of theta according to data x and label y

    Input:
    theta: it's the model parameter matrix.
    features: it's the input data matrix. The shape is (N, H)
    target: the label array

    Output:
    log_g: the log likehood of theta according to data x and label y
    '''

    #####################################################
    ##                 YOUR CODE HERE                  ##
    #####################################################

    x = np.array(features)
    y = np.array(target)
    n = len(target)
    aprox = 0.000000000000000000001

    y_pred = np.array(predictions(x, theta))
    y_pred = np.where(y_pred==0, aprox, y_pred)
    y_pred = np.where(y_pred>=1,1 - aprox, y_pred)
    log_l = np.sum((y*np.log(y_pred)+(1-y)*np.log(1-y_pred))*1/n)

    return log_l

def predictions(features, theta):
    '''
    Function to compute the predictions for the input features

    Input:
    theta: it's the model parameter matrix.
    features: it's the input data matrix. The shape is (N, H)

    Output:
    preds: the predictions of the input features
    '''

    #####################################################
    ##                 YOUR CODE HERE                  ##
    #####################################################

    preds = sigmoid(np.dot(features,theta))

    return preds

def update_theta(theta, target, preds, features, lr):
    '''
    Function to compute the gradient of the log likelihood
    and then return the updated weights

    Input:
    theta: the model parameter matrix.
    target: the label array
    preds: the predictions of the input features
    features: it's the input data matrix. The shape is (N, H)
    lr: the learning rate

    Output:
    theta: the updated model parameter matrix.
    '''
    n = len(target)
    gr = 0
    tr = target-preds
    for i in range(n):
      gr += tr[i]*features[i] * 1/n
    theta += gr*lr
    #####################################################
    ##                 YOUR CODE HERE                  ##
    #####################################################
    return theta

def gradient_ascent(theta, features, target, lr, num_steps):
    '''
    Function to execute the gradient ascent algorithm

    Input:
    theta: the model parameter matrix.
    target: the label array
    num_steps: the number of iterations
    features: the input data matrix. The shape is (N, H)
    lr: the learning rate

    Output:
    theta: the final model parameter matrix.
    log_likelihood_history: the values of the log likelihood during the process
    '''

    log_likelihood_history = np.zeros(num_steps)

    #####################################################
    ##                 YOUR CODE HERE                  ##
    #####################################################
    """
    for i in range(50):
      gr = 0
      for j in range(features.shape[0]):
        gr += 2*(target[j] - np.dot(theta.T,features[j])*features[j])
      theta += lr*gr
      print(theta)
    """
    for i in range(num_steps):
      theta = update_theta(theta, target, predictions(features, theta), features, lr)
      log_likelihood_history[i] = log_likelihood(theta,features,target)
    return theta, log_likelihood_history

```

################# Do not write below this line #################

Check your grad_l implementation:
grad_l applied to the theta_test (defined below) should provide a value for log_l_test close to the target_value (defined below); in other words the error_test should be 0, up to machine error precision.

```python
target_value = -1.630501731599431

output_test  = log_likelihood(np.array([-7,4,1]),x,y)
error_test=np.abs(output_test-target_value)

print("{:f}".format(error_test))
```

    0.000000

Let's now apply the function gradient_ascent and print the final theta as well as theta_history

```python
# Initialize theta0
theta0 = np.zeros(x.shape[1])

# Run Gradient Ascent method
n_iter=1000
theta_final, log_l_history = gradient_ascent(theta0,x,y,lr=0.5,num_steps=n_iter)
print(theta_final)

```

    [-0.46097042  2.90036399  0.23146846]

Let's plot the log likelihood over iterations

```python
fig,ax = plt.subplots(num=2)

ax.set_ylabel('l(Theta)')
ax.set_xlabel('Iterations')
_=ax.plot(range(len(log_l_history)),log_l_history,'b.')
```

Plot the data and the decision boundary:

```python
# Generate vector to plot decision boundary
x1_vec = np.linspace(X[:,0].min(),X[:,1].max(),2)

# Plot raw data
sns.scatterplot(x=X[:,0], y=X[:,1], hue=y, data=X)

# Plot decision boundary
plt.plot(x1_vec,(-x1_vec*theta_final[1]-theta_final[0])/theta_final[2], color="red")
plt.ylim(X[:,1].min()-1,X[:,1].max()+1)
# Save the theta_final value for later comparisons
theta_GA = theta_final.copy()
```

################# Do not write above this line #################

Discuss these two points:
1. You have implemented the gradient ascent rule. Could we have also used gradient descent instead for the proposed problem? Why/Why not?
2. Let's deeply analyze how the learning rate $\alpha$ and the number of iterations affect the final results. Run the algorithm you have written for different values of $\alpha$ and the number of iterations and look at the outputs you get. Is the decision boundary influenced by these parameters change? Why do you think these parameters are affecting/not affecting the results?

################# Do not write below this line #################

## Question 2: Logistic Regression with non linear boundaries (7 points)

#### Exercise 2.a **(4 Points)** Polynomial features for logistic regression

Define new features, e.g. of 2nd and 3rd degrees, and learn a logistic regression classifier by using the new features, by using the gradient ascent optimization algorithm you defined in Question 1.

In particular, we would consider a polynomial boundary with equation:

$f(x_1, x_2) = c_0 + c_1 x_1 + c_2 x_2 + c_3 x_1^2 + c_4 x_2^2 + c_5 x_1 x_2 + c_6 x_1^3 + c_7 x_2^3 + c_8 x_1^2 x_2 + c_9 x_1 x_2^2$

We would therefore compute 7 new features: 3 new ones for the quadratic terms and 4 new ones for the cubic terms.

Create new arrays by stacking x and the new 7 features (in the order x1x1, x2x2, x1x2, x1x1x1, x2x2x2, x1x1x2, x1x2x2). In particular create x_new_quad by additionally stacking with x the quadratic features, and x_new_cubic by additionally stacking with x the quadratic and the cubic features.

```python
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=500, n_features=2, n_informative=2, n_redundant=0, n_classes=2, random_state=5)
X.shape, y.shape
```

    ((500, 2), (500,))

```python
x = np.hstack([np.ones((X.shape[0], 1)), X])
```

```python
import seaborn as sns
import matplotlib.pyplot as plt
sns.scatterplot(x=X[:,0], y=X[:,1], hue=y);
```

```python
# First extract features x1 and x2 from x and reshape them to x1 vector arrays
x1 = x[:,1]
x2 = x[:,2]
x1 = x1.reshape(x1.shape[0], 1)
x2 = x2.reshape(x2.shape[0], 1)
print(x[:5,:]) # For visualization of the first 5 values
print(x1[:5,:]) # For visualization of the first 5 values
print(x2[:5,:]) # For visualization of the first 5 values
```

    [[ 1.          2.25698215 -1.34710915]
     [ 1.          1.43699308  1.28420453]
     [ 1.          0.57927295  0.23690172]
     [ 1.          0.42538132 -0.24611145]
     [ 1.          1.13485101 -0.61162683]]
    [[2.25698215]
     [1.43699308]
     [0.57927295]
     [0.42538132]
     [1.13485101]]
    [[-1.34710915]
     [ 1.28420453]
     [ 0.23690172]
     [-0.24611145]
     [-0.61162683]]

################# Do not write above this line #################

Your code here

```python
def new_features(x, degree=2):
    '''
    Function to create n-degree features from the input

    Input:
    x: the initial features
    degree: the maximum degree you wantthe features

    Output:
    features: the final features.
              2nd degree features must have the order [x, x1x1, x1x2, x2x2]
              2nd degree features must have the order [x, x1x1, x1x2, x2x2, x1x1x1, x1x1x2, x1x2x2, x2x2x2]
    '''

    features = np.ones((x[:,1].shape[0], 2+2**degree))
    # features = np.ones((x[:,1].shape[0]))
    #####################################################
    ##                 YOUR CODE HERE                  ##
    #####################################################

    x0 = x[:,0]
    x1 = x[:,1]
    x2 = x[:,2]

    features[:,0] = x0
    features[:,1] = x1
    features[:,2] = x2
    indexFeatures = 3

    for i in range(2, degree+1):
      for j in range(0,i+1):
        features[:,indexFeatures] = (x[:,1]**(i-j))*(x[:,2]**j)
        # features.append((x1**(i-j))*(x2**j))
        indexFeatures += 1

    return features

```

################# Do not write below this line #################

```python
x_new_quad = new_features(x, degree=2)
x_new_cubic = new_features(x, degree=3)

#reordering output features
temp = np.copy(x_new_quad[:, -1])
x_new_quad[:, -1] = x_new_quad[:, -2]
x_new_quad[:, -2] = temp

temp = np.copy(x_new_cubic[:, -1])
x_new_cubic[:, -1] = x_new_cubic[:, -2]
x_new_cubic[:, -2] = x_new_cubic[:, -3]
x_new_cubic[:, -3] = temp

```

Now use the gradient ascent optimization algorithm to learn theta by maximizing the log-likelihood, both for the case of x_new_quad and x_new_cubic.

```python
# Initialize theta0, in case of quadratic features
theta0_quad = np.zeros(x_new_quad.shape[1])

theta_final_quad, log_l_history_quad = gradient_ascent(theta0_quad,x_new_quad,y,lr=0.5,num_steps=n_iter)

# Initialize theta0, in case of quadratic and cubic features
theta0_cubic = np.zeros(x_new_cubic.shape[1])

# Run Newton's method, in case of quadratic and cubic features
theta_final_cubic, log_l_history_cubic = gradient_ascent(theta0_cubic,x_new_cubic,y,lr=0.5,num_steps=n_iter)

# check and compare with previous results
print(theta_final_quad)
print(theta_final_cubic)
```

    [ 0.07548038  3.32398303  0.27106753 -0.51873505 -0.34088582 -0.04846172]
    [ 0.82395453  2.41957969  1.66049077 -1.13927962 -0.20657699 -0.96074035
      0.34384121 -0.64078458  0.95132383  1.40927121]

```python
# Plot the log likelihood values in the optimization iterations, in one of the two cases.
fig,ax = plt.subplots(num=2)

ax.set_ylabel('l(Theta)')
ax.set_xlabel('Iterations')
_=ax.plot(range(len(log_l_history_quad)),log_l_history_quad,'b.')
```

#### Exercise 2.b **(3 Points)** Plot the computed non-linear boundary and discuss the questions

First, define a boundary_function to compute the boundary equation for the input feature vectors $x_1$ and $x_2$, according to estimated parameters theta, both in the case of quadratic (theta_final_quad) and of quadratic and cubic features (theta_final_cubic). Refer for the equation to the introductory part of Question 2.

################# Do not write above this line #################

Your code here

```python
def boundary_function(x1_vec, x2_vec, theta_final):

    x1_vec, x2_vec = np.meshgrid(x1_vec,x2_vec)

    if len(theta_final) == 6:
        # boundary function value for features up to quadratic
        c_0, c_1, c_2, c_3, c_4, c_5 = theta_final
        f = c_0 + c_1*x1_vec + c_2*x2_vec + c_3*(x1_vec**2) + c_4*(x2_vec**2) + c_5*(x1_vec*x2_vec)
    elif len(theta_final) == 10:
        # boundary function value for features up to cubic
        c_0, c_1, c_2, c_3, c_4, c_5, c_6, c_7, c_8, c_9 = theta_final
        f = c_0 + c_1*x1_vec + c_2*x2_vec + c_3*(x1_vec**2) + c_4*(x2_vec**2) + c_5*(x1_vec*x2_vec) + c_6*(x1_vec**3) + c_7*(x2_vec**3) + c_8*((x1_vec**2)*x2_vec) + c_9*(x1_vec*(x2_vec**2))
    else:
        raise("Number of Parameters is not correct")
    return x1_vec, x2_vec, f
```

################# Do not write below this line #################

Now plot the decision boundaries corresponding to the theta_final_quad and theta_final_cubic solutions.

```python
x1_vec = np.linspace(X[:,0].min()-1,X[:,0].max()+1,200);
x2_vec = np.linspace(X[:,1].min()-1,X[:,1].max()+1,200);

x1_vec, x2_vec, f = boundary_function(x1_vec, x2_vec, theta_final_quad)

sns.scatterplot(x=X[:,0], y=X[:,1], hue=y, data=X);

plt.contour(x1_vec, x2_vec, f, colors="red", levels=[0])
plt.show()
```

```python
x1_vec = np.linspace(X[:,0].min()-1,X[:,0].max()+1,200);
x2_vec = np.linspace(X[:,1].min()-1,X[:,1].max()+1,200);

x1_vec, x2_vec, f = boundary_function(x1_vec, x2_vec, theta_final_cubic)

sns.scatterplot(x=X[:,0], y=X[:,1], hue=y, data=X);

plt.contour(x1_vec, x2_vec, f, colors="red", levels=[0])
plt.show()
```

#### Confusion Matrix

Here you can see the confusion matrices related to the three models you've implemented.

```python
from sklearn.metrics import confusion_matrix
```

```python
## logistic regression with linear buondary

z = np.dot(x,theta_final)
probabilities = sigmoid(z)
y_hat = np.array(list(map(lambda x: 1 if x>0.5 else 0, probabilities)))
confusion_matrix(y, y_hat)
```

    array([[218,  35],
           [ 22, 225]])

```python
## logistic regression with non linear buondary - quadratic

z = np.dot(x_new_quad,theta_final_quad)
probabilities = sigmoid(z)
y_hat = np.array(list(map(lambda x: 1 if x>0.5 else 0, probabilities)))
confusion_matrix(y, y_hat)
```

    array([[220,  33],
           [ 15, 232]])

```python
## logistic regression with non linear buondary - cubic

z = np.dot(x_new_cubic,theta_final_cubic)
probabilities = sigmoid(z)
y_hat = np.array(list(map(lambda x: 1 if x>0.5 else 0, probabilities)))
confusion_matrix(y, y_hat)
```

    array([[225,  28],
           [ 11, 236]])

################# Do not write above this line #################

Write now your considerations. Discuss in particular:
- Look back at the plots you have generated. What can you say about the differences between the linear, quadratic, and cubic decision boundaries? Can you say if the model is improving in performances, increasing the degree of the polynomial? Do you think you can incur in underfitting increasing more and more the degree?
- Let's now delve into some quantitative analysis. The three tables you have generated represent the confusion matrix for the model you have implemented in the first two questions. What can you say about actual performances? Does the increase of the degree have a high effect on the results?

################# Do not write below this line #################

## Question 3: Multinomial Classification (Softmax Regression) **(13 Points)**

### Code and Theory **(10 Points)**
### Report **(3 Points)**

#### Exercise 3.a **(4 Points)**

In the multinomial classification we generally have $K>2$ classes. So the label for the $i$-th sample $X_i$ is $y_i\in\{1,...,K\}$, where $i=1,...,N$. The output class for each sample is estimated by returning a score $s_i$ for each of the K classes. This results in a vector of scores of dimension K.
In this exercise we'll use the *Softmax Regression* model, which is the natural extension of *Logistic Regression* for the case of more than 2 classes. The score array is given by the linear model:

\begin{equation}
s_i =  X_i \theta
\end{equation}

Scores may be interpreted probabilistically, upon application of the function *softmax*. The position in the vector with the highest probability will be predicted as the output class. The probability of the class k for the $i$-th data sample is:

\begin{equation}
p_{ik} = \frac{\exp(X_i \theta_k)}{\sum_{j=1}^K(X_i \theta_j))}
\end{equation}

We will adopt the *Cross Entropy* loss and optimize the model via *Gradient Descent*.
In the first of this exercise we have to:
-    Write the equations of the Cross Entropy loss for the Softmax regression model;
-    Compute the equation for the gradient of the Cross Entropy loss for the model, in order to use it in the gradient descent algorithm.

#### A bit of notation

*  N: is the number of samples
*  K: is the number of classes
*  X: is the input dataset and it has shape (N, H) where H is the number of features
*  y: is the output array with the labels; it has shape (N, 1)
*  $\theta$: is the parameter matrix of the model; it has shape (H, K)

################# Do not write above this line #################

Your equations here.

\begin{equation}
L(\theta) = ...
\end{equation}

\begin{equation}
\nabla_{\theta_k} L(\theta) = ...
\end{equation}

################# Do not write below this line #################

#### Exercise 3.b **(4 Points)**

Now we will implement the code for the equations. Let's implement the functions:
-  softmax
-  CELoss
-  CELoss gradient
-  gradient descent

We generate a toy dataset with *sklearn* library. Do not change anything outside the parts provided of your own code (else the provided checkpoint will not work).

```python
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=300, n_features=7, n_informative=7, n_redundant=0, n_classes=3, random_state=1)
X.shape, y.shape
```

    ((300, 7), (300,))

As a hint for the implementations of your functions: consider the labels $y$ as one-hot vector. This will allow matrix operations (element-wise multiplication and summation).

```python
import scipy
import numpy as np

def class2OneHot(vec):
    out_sparse = scipy.sparse.csr_matrix((np.ones(vec.shape[0]), (vec, np.array(range(vec.shape[0])))))
    out_onehot = np.array(out_sparse.todense()).T
    return out_onehot

y_onehot = class2OneHot(y)
```

Let's visualize the generated dataset. We use as visualizzation method the *Principal Component Analysis* (PCA). PCA summarize the high-dimensional feature vectors of each sample into 2  features, which we can illustrate with a 2D plot. Look at the following plot, the 3 generated classes do not seem separable.

```python
from sklearn.decomposition import PCA
import pandas as pd

pca = PCA(n_components=2)
principalComponents = pca.fit_transform(X)
principalDf = pd.DataFrame(data = principalComponents, columns = ['pc1', 'pc2'])
finalDf = pd.concat([principalDf, pd.DataFrame(y, columns = ['target'])], axis = 1)
```

```python
import seaborn as sns
import matplotlib.pyplot as plt
sns.scatterplot(x='pc1', y='pc2', hue='target', data=finalDf);
```

################# Do not write above this line #################

```python
def softmax(theta, X):
    '''
    Function to compute associated probability for each sample and each class.

    Input:
    theta: it's the model parameter matrix. The shape is (H, K)
    X: it's the input data matrix. The shape is (N, H)

    Output:
    softmax: it's the matrix containing probability for each sample and each class. The shape is (N, K)
    '''

    #####################################################
    ##                 YOUR CODE HERE                  ##
    #####################################################

    z = np.dot(X, theta)
    softmax = np.exp(z)/sum(np.exp(z))

    return softmax

def CELoss(theta, X, y_onehot):
    '''
    Function to compute softmax regression model and Cross Entropy loss.

    Input:
    theta: it's the model parameter matrix. The shape is (H, K)
    X: it's the input data matrix. The shape is (N, H)
    y_onehot: it's the label array in encoded as one hot vector. The shape is (N, K)

    Output:
    loss: The scalar that is the mean error for each sample.
    '''

    #####################################################
    ##                 YOUR CODE HERE                  ##
    #####################################################
    loss = -sum(y_onehot*np.log(softmax(theta, X)))

    return loss

def CELoss_jacobian(theta, X, y_onehot):
    '''
    Function to compute gradient of the cross entropy loss with respect the parameters.

    Input:
    theta: it's the model parameter matrix. The shape is (H, K)
    X: it's the input data matrix. The shape is (N, H)
    y_onehot: it's the label array in encoded as one hot vector. The shape is (N, K)

    Output:
    jacobian: A matrix with the partial derivatives of the loss. The shape is (H, K)
    '''

    #####################################################
    ##                 YOUR CODE HERE                  ##
    #####################################################

    return jacobian

def gradient_descent(theta, X, y_onehot, alpha=0.01, iterations=100):
    '''
    Function to compute gradient of the cross entropy loss with respect the parameters.

    Input:
    theta: it's the model parameter matrix. The shape is (H, K)
    X: it's the input data matrix. The shape is (N, H)
    y_onehot: it's the label array in encoded as one hot vector. The shape is (N, K)
    alpha: it's the learning rate, so it determines the speed of each step of the GD algorithm
    iterations: it's the total number of step the algorithm performs

    Output:
    theta: it's the updated matrix of the parameters after all the iterations of the optimization algorithm. The shape is (H, K)
    loss_history: it's an array with the computed loss after each iteration
    '''

    # We initialize an empty array to be filled with loss value after each iteration
    loss_history = np.zeros(iterations)

    # With a for loop we compute the steps of GD algo
    for it in range(iterations):
      m = X.shape[0] #First we get the number of training examples
      prob = softmax(theta, X)
      grad = (-1 / m) * np.dot(X.T,(y_onehot - prob)) + theta
      theta -= alpha*grad
      loss_history[it] = sum(CELoss(theta, X, y_onehot))
      #print(theta)
        #####################################################
        ##                 YOUR CODE HERE                  ##
        #####################################################

    return theta, loss_history

```

################# Do not write below this line #################

```python
# Initialize a theta matrix with random parameters
theta0 = np.random.rand(X.shape[1], len(np.unique(y)))

print("Initial Loss with initialized theta is:", CELoss(theta0, X, y_onehot))

# Run Gradient Descent method
n_iter = 1000
theta_final, log_l_history = gradient_descent(theta0, X, y_onehot, alpha=0.01, iterations=10000)
```

    Initial Loss with initialized theta is: [1162.97358142 1246.9958419   808.30349786]

```python
theta_final
```

    array([[-0.0645275 ,  0.08389829,  0.057148  ],
           [-0.02875407,  0.1159036 ,  0.02730884],
           [-0.29747352, -0.38801056,  0.29286725],
           [-0.29385156, -0.25532116,  0.33810673],
           [ 0.31222304, -0.1468008 , -0.22322808],
           [ 0.33922877,  0.31375042,  0.37038279],
           [ 0.28510912, -0.41488842,  0.36814386]])

```python
loss = CELoss(theta_final, X, y_onehot)
loss
```

    array([570.29195871, 554.60303773, 563.96386959])

```python
fig,ax = plt.subplots(num=2)

ax.set_ylabel('loss')
ax.set_xlabel('Iterations')
_=ax.plot(range(len(log_l_history)), log_l_history,'b.')
```

#### Exercise 3.c **(2 Points)**

Let's now evaluate the goodness of the learnt based on accuracy:

\begin{equation}
Accuracy = \frac{Number\ of\ correct\ predictions}{Total\ number\ of\ predictions}
\end{equation}

Implement the compute_accuracy function. You may compare the accuracy achieved with learnt model Vs. a random model (random $\Theta$) or one based on $\Theta$'s filled with zeros.

################# Do not write above this line #################

```python
def compute_accuracy(theta, X, y):
    '''
    Function to compute accuracy metrics of the softmax regression model.

    Input:
    theta: it's the final parameter matrix. The one we learned after all the iterations of the GD algorithm. The shape is (H, K)
    X: it's the input data matrix. The shape is (N, H)
    y: it's the label array. The shape is (N, 1)

    Output:
    accuracy: Score of the accuracy.
    '''
    result = list()

    for i in range(3):
      sum = 0
      for j in range(3):
        sum += np.exp(np.dot(X, theta[:,j].T))
      result.append(np.round(np.exp(np.dot(X, theta[:,i].T))/sum))

    result_final = np.zeros(y.shape)

    for i in range(3):
      for j in range(len(result[i])):
        if result[i][j] == 1:
          result_final[j] = i

    accuracy = np.sum([ 1 if result_final[i] == y[i] else 0 for i in range(len(result_final)) ])/len(y)
    return accuracy
```

################# Do not write below this line #################

```python
compute_accuracy(theta_final, X, y)
```

    0.7466666666666667

```python
theta0 = np.random.rand(X.shape[1], len(np.unique(y)))
compute_accuracy(theta0, X, y)
```

    0.14333333333333334

```python
compute_accuracy(np.zeros((X.shape[1], len(np.unique(y)))), X, y)
```

    0.3333333333333333

### Report **(3 Points)**

Experiment with different values for the learning rate $\alpha$ and the number of iterations. Look how the loss plot changes the convergence rate and the resulting accuracy metric. Report also execution time of each run. For this last step you could you %%time at the beginning of the cell to display time needed for the algorithm.

```python
%%time

# Initialize a theta matrix with random parameters
theta0 = np.random.rand(X.shape[1], len(np.unique(y)))

print("Initial Loss with initialized theta is:", CELoss(theta0, X, y_onehot))

# Run Gradient Descent method
n_iter = 100
theta_final, log_l_history = gradient_descent(theta0, X, y_onehot, alpha=0.001, iterations=n_iter)
```

    Initial Loss with initialized theta is: [ 849.61276667 1046.63156307  898.20123202]
    CPU times: user 93.4 ms, sys: 1.69 ms, total: 95.1 ms
    Wall time: 97.4 ms

**Write your Report here**

| LR | Iter | Accuracy | Time |
|---|---|---|---|
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
| ... |  |  |  |

## Question 4: Multinomial Naive Bayes **(6 Points)**

### Code and Theory

The Naive Bayes classifier is a probabilistic machine learning model often used for classification tasks, e.g. document classification problems.

In the multinomial Naive Bayes classification you generally have $K>2$ classes, and the features are assumed to be generated from a multinomial distribution.

##### __*Example Data*__
General models consider input data as values. In the case of MultinomialNB, being used mainly in the field of document classification, these data consider how many features $X_i$ are present in the sample. Basically, it is a count of features within each document.

Taking into account $D=3$ documents and a vocabulary consisting of $N=4$ words, the data are considered as follows.

|  | $w_1$ | $w_2$ | $w_3$ | $w_4$ |
|---|---|---|---|---|
| $d_1$  | 3 | 0 | 1 | 1 |
| $d_2$ | 2 | 1 | 3 |0|
| $d_3$ | 2 | 2 | 0 |2|

By randomly generating the class to which each document belongs we have $y=[1,0,1]$

##### __*A bit of notation*__
- $Y =\{y_1, y_2, ... , y_{|Y|}\}$: set of classes

- $V =\{w_1, w_2, ... , w_{|V|}\}$: set of vocabulary

-  $D =\{d_1, d_2, ... , d_{|D|}\}$: set of documents

-  $N_{yi}$:  count of a specific word $w_i$ in each unique class, e.g. for $y=1$ you select $D_1$ and $D_3$, then for third column  you have $N_{y,3}=1$
-  $N_y$: total count of features for a specific class, e.g. for $y=1$ you sum all rows values which the correspondent label is 1, so $N_y=11$

-  $n$: total number of features (words in vocabulary)
-  $\alpha$: smoothing parameters

##### __*Task*__
Find the class $y$ to which the document is most likely to belong given the words $w$.
Use the Bayes formula and the posterior probability for it.

Bayes Formula:
\begin{equation}
P(A|B) = \frac{P(A)*P(B|A)}{P(B)}
\end{equation}

Where:
- P(A): Prior probability of A
- P(B): Prior probability of B
- P(B|A): Likelihood, multiplying posterior probability, that is multinomial Naive Bayes is:
\begin{equation}
P(B|A) = \left(\frac{N_{yi}+\alpha}{N_{y}+\alpha*n\_features}\right)^{X_{doc,i}}
\end{equation}

**Reminder: do not change any part of this notebook outside the assigned work spaces**

#### Generate random dataset

```python
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

X, y = make_classification(n_samples=300, n_features=7, n_informative=7, n_redundant=0, n_classes=3, random_state=1)
X = np.floor(X)-np.min(np.floor(X))
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
X_train.shape, y_train.shape
```

    ((240, 7), (240,))

#### Step0: $N_y$ and $N_{yi}$

```python
def feature_count(X, y, classes, n_classes, n_features):
        '''
        Function to compute the count of a specific word in each unique class and the total count.

        Input:
        X: it's the input data matrix.
        y: label array
        classes: unique values of y
        n_classes: number of classes
        n_features: it's the number of word in Vocabulary.

        Output:
        N_yi: count of a specific word $w_i$ in each unique class
        N_y: total count of features for a specific class
        '''
        N_yi = np.zeros((n_classes, n_features)) # feature count
        N_y = np.zeros((n_classes)) # total count

        #####################################################
        ##                 YOUR CODE HERE                  ##
        #####################################################

        test = []

        for i in range(len(X)):
          for j in range(len(X[i])):
            N_yi[y[i], j] = N_yi[y[i], j] + X[i, j]

        for i in range(n_classes):
          N_y[i] = sum([1 if classe == i else 0 for classe in y ])

        return N_yi, N_y
```

```python
n_samples_train, n_features = X_train.shape
classes = np.unique(y_train)
n_classes = 3
alpha = 0.1

N_yi, N_y = feature_count(X_train, y_train, classes, n_classes, n_features)
```

#### Step1: Prior Probability
The probability of a document being in a specific category from the given set of documents.

######################################

Your equations here:
\begin{equation}
P(y_j) = \frac{total \; of \; y_j \; observe}{n_{Sample}}
\end{equation}

######################################

```python
def prior_(X, y, n_classes, n_samples):
        """
        Calculates prior for each unique class in y.

        Input:
        X: it's the input data matrix.
        y: label array
        n_classes: number of classes
        n_samples: number of documents

        Output:
        P: prior probability for each class. Shape: (, n_classes)
        """
        classes = np.unique(y)
        P = np.zeros(n_classes)

        # Implement Prior Probability P(A)
        #####################################################
        ##                 YOUR CODE HERE                  ##
        #####################################################
        wh = list()
        for i in range(len(y)):
          if not y[i] in wh:
            wh.append(y[i])
          P[wh.index(y[i])] += 1
        return P/n_samples
```

```python
prior_prob = prior_(X_train, y_train, n_classes, n_samples_train)
print(prior_prob)
```

    [0.3        0.34583333 0.35416667]

#### Step2
Posterior Probability: The conditional probability of a word occurring in a document given that the document belongs to a particular category.

\begin{equation}
P(w_i|y_j) = \left(\frac{N_{yi}+\alpha}{N_{y}+\alpha*n\_features}\right)^{X_{doc,i}}
\end{equation}

Likelihood for a single document:
######################################

Your equations here:
\begin{equation}
P(w|y_j) = \sum log(P(w_i|y_j)
\end{equation}

######################################

```python
def posterior_(x_i, i, h, N_y, N_yi, n_features, alpha):
        """
        Calculates posterior probability. aka P(w_i|y_j) using equation in the notebook.

        Input:
        x_i: feature x_i
        i: feature index.
        h: a class in y
        N_yi:   count of a specific word in each unique class
        N_y: total count of features for a specific class
        n_features: it's the number of word in Vocabulary.
        alpha: smoothing parameter

        Output:
        posterior: P(xi | y). Float.
        """

        # Implement Posterior Probability
        #####################################################
        ##                 YOUR CODE HERE                  ##
        #####################################################
        n_features = np.array(n_features)
        posterior = np.dot((N_yi[h][i].T+alpha),1/(N_y[h]+(n_features*alpha)))**x_i
        #posterior = np.dot((N_y[h]+alpha),1/(N_yi[h][i]+(n_features*alpha)))**x_i
        print(posterior)
        return posterior

def likelihood_(x, h, N_y, N_yi, n_features, alpha):
        """
        Calculates Likelihood P(w|j_i).

        Input:
        x: a row of test data. Shape(n_features,)
        h: a class in y
        N_yi:   count of a specific word in each unique class
        N_y: total count of features for a specific class
        n_features: it's the number of word in Vocabulary.
        alpha: smoothing parameter

        Output:
        likelihood: Float.
        """
        """
        muy = N_yi[h]
        d = n_features
        su = 0
        for i in range(n_features):
          su += np.dot(x[i] - muy, (x[i] - muy).T)
        su /= n_features
        tmp = []
        for i in range(x.shape[0]):
            tmp.append(posterior_(x[i], i, h, N_y, N_yi, n_features, alpha))
        """
        # Implement Likelihood
        #####################################################
        ##                 YOUR CODE HERE                  ##
        #####################################################
        likelihood = 0
        print(N_y)
        for i in range(n_features):
          likelihood += math.log(posterior_(x[i],i , h, N_y, N_yi, n_features, alpha))
        return likelihood
```

```python
# Example of likelihood for first document
likelihood_(X_test[0], 0, N_y, N_yi, n_features, alpha)
```

    [72. 83. 85.]
    222178405.41003954
    175260553.3600799
    571975803.1055853
    9417702.877034616
    55951560.11672744
    721456260.3038414
    527309537.39743763

    132.7435620321748

#### Step3
Joint Likelihood that, given the words, the documents belongs to specific class
######################################

Your equations here:
\begin{equation}
P(y_i|w) = ...
\end{equation}

######################################

Finally, from the probability that the document is in that class given the words, take the argument correspond to max value.

\begin{equation}
y(D) = argmax_{y \in Y} \frac{P(y|w)}{\sum_{j}P(y_j|w)}
\end{equation}

```python
def joint_likelihood(X, prior_prob, classes, n_classes, N_y, N_yi, n_features, alpha):
        """
        Calculates the joint probability P(y_i|w) for each class and makes it probability.
        Then take the argmax.

        Input:
        X: test data
        prior_prob:
        classes:
        n_classes:
        N_yi:   count of a specific word in each unique class
        N_y: total count of features for a specific class
        n_features: it's the number of word in Vocabulary.
        alpha: smoothing parameter

        Output:
        predicted_class: Predicted class of the documents. Int. Shape: (,#documents)
        """
        samples, features = X.shape
        predict_proba = np.zeros((samples,n_classes))

        # Calculate Joint Likelihood of each row for each class, then normalize in order to make them probabilities
        # Finally take the argmax to have the predicted class for each document
        #####################################################
        ##                 YOUR CODE HERE                  ##
        #####################################################
        print(alpha)

        return predicted_class
```

```python
yhat = joint_likelihood(X_test, prior_prob, classes, n_classes, N_y, N_yi, n_features, alpha)
```

#### Step4: Calculate the Accuracy Score

```python
print('Accuracy: ', np.round(accuracy_score(yhat, y_test),3))
```

**Sanity Check**

Here we use a function from the sklearn library, one of the most widely used in machine learning. MultinomialNB() implements the required algorithm, so the result of your implementation should be equal to the output of the following function.

```python
from sklearn import naive_bayes
clf = naive_bayes.MultinomialNB(alpha=0.1)
clf.fit(X_train,y_train)
sk_y = clf.predict(X_test)
print('Accuracy: ', np.round(accuracy_score(sk_y, y_test),3))
```


# Policy-Gradients with the REINFORCE algorithm

**Background**:
In this practical we will train an agent using the REINFORCE algorithm to learn to balance a pole in the OpenAI gym [Cartpole environment](https://gym.openai.com/envs/CartPole-v1).

**Learning objectives**:
* Understand the policy-gradient approach to directly training a parameterised policy to maximise expected future rewards.
* Understand how the policy-gradient theorem allows us to improve the policy using on-policy state = env.reset().

**What is expected of you**:
 * Go through the explanation, keeping the above learning objectives in mind.
 * Fill in the missing code ("#IMPLEMENT-ME") and train a model to solve the Cartpole-v1 environment in OpenAI gym (you solve it when reward=500).

# A Simple Policy-Gradient Cartpole Example

## Introduction

We have seen in your course that there are many different approaches to training RL agents. In this practical we will take a look at REINFORCE - a simple policy-based method. REINFORCE (and policy-based methods in general) directly optimise a parametrised policy in order to maximise future rewards.

We will try to learn a policy $\pi_\theta(a | s)$ which outputs a distribution over the possible actions $a$, given the current state $s$ of the environment. The goal is find a set of parameters $\theta$ to maximise the expected discounted return:
\begin{align}
J(\theta) = \mathbb{E}_{\tau \sim p_\theta} \left[\sum_{t=0}^T \gamma^t r(s_t, a_t)\right],
\end{align}
where $\tau$ is a trajectory sampled from $p_\theta$. The **policy-gradient** theorem gives us the derivative of this objective function:
\begin{align}
    \nabla_\theta J(\theta) = \mathbb{E}_{\tau \sim p_\theta} \left[\left(\sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(a_t|s_t)\right) \left(\sum_{t=0}^T \gamma^t r(s_t, a_t) \right) \right].
\end{align}

**NOTE**:
* We have a policy $\pi_\theta(a|s)$ which tells the agent which action $a$ to take, given the state $s$, and it is parameterised in terms of parameters $\theta$.
* Our goal is to maximise $J(\theta)$ by **choosing actions from this policy** that lead to high future rewards.
* We'll use gradient-based optimisation to update the policy parameters $\theta$. We therefore want the gradient of our objective wrt the policy parameters.
* We use the policy-gradient theorem to find a expression for the gradient. This is an expectation over trajectories from our policy and the environment.
* Since we can now sample trajectories $(s_0, a_0, r_1, s_1, a_1, r_2, \ldots)$ using our policy $\pi_\theta$, we can approximate this gradient using **[Monte-Carlo](https://en.wikipedia.org/wiki/Monte_Carlo_integration)** methods.

This algorithm is called **Monte-Carlo REINFORCE**, and is one type of policy-gradient algorithm. Let's use this to solve the Cartpole environment!

**Monte-Carlo REINFORCE**:

for each episode:
1. sample a trajectory $\tau$ using the policy $\pi_\theta$.
2. compute $\nabla_\theta J(\theta) \approx \left(\sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(a_t|s_t)\right) \left(\sum_{t=0}^T \gamma^t r(s_t, a_t) \right)$.
3. update policy parameters $\theta \leftarrow \theta + \alpha \nabla_\theta J(\theta)$

```python
# import various packages
from collections import deque

import numpy as np
import matplotlib.pyplot as plt
import gym

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.distributions import Categorical
```

```python
# use gpu if available
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# configure matplotlib
%matplotlib inline
plt.rcParams['figure.figsize'] = (15.0, 10.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'
```

## The Environment

Cartpole is a standard benchmark in reinforcement learning and is a good sandbox for trying things out. The goal is to balance a pendulum on top of a moving cart. We have 2 actions - either push the cart to the left or push to the right. The state space consists of the cart's position and velocity and the pendulum's angle and angular velocity. Let's create the environment and take a look at the state and action spaces:

```python
env = gym.make('CartPole-v1')
print('action space:', env.action_space)
print('observation space:', env.observation_space)
```

Here we can see that there are 2 discrete actions and a continuous state space.

### Taking a few steps

To get a better feel for the environment, we will use a random policy to genrate a short trajectory.

```python
SUB = str.maketrans("t0123456789+", "ₜ₀₁₂₃₄₅₆₇₈₉₊")
print('-'*115)

state = env.reset()
for i in range (5):
    # sample random action
    action = env.action_space.sample()
    # take action in the environment
    new_state, reward, done, _ = env.step(action)

    print('Step t=', i+1, ': (', 'st, at , rt, st+1'.translate(SUB),')')
    print('(', state, ',', action, ',', reward, ',', new_state, ')')
    print('-'*115)

    state = new_state
```

### Watching a random policy agent play

Let's also see how a random policy performs in this enviroment:

```python
env_1 = gym.make('CartPole-v1')
state = env_1.reset()
for t in range(200):
    # sample a random action
    action = env_1.action_space.sample()
    env_1.render()
    state, reward, done, _ = env_1.step(action)
env_1.close()
```

Not very good! The pole only stayed up for a few time steps... Now let's improve things using REINFORCE.

## The Policy

We begin by parameterising the policy $\pi_\theta(a | s)$ as a simple neural network which takes the state (a vector of 4 elements provided by `gym`) as input, and produces a Categorical distribution over the possible actions as output. Simple enough. Refer to [torch.nn](https://pytorch.org/docs/stable/nn.html)

```python
class Policy(nn.Module):
    def __init__(self):
        super(Policy, self).__init__()
        # IMPLEMENT-ME
        # Define neural network layers. Refer to nn.Linear (https://pytorch.org/docs/stable/nn.html#torch.nn.Linear)
        # We are going to use a neural network with one hidden layer of size 16.
        # The first layer should have an input size of env.observation_space.shape and an output size of 16
        self.fc1 = ...
        # The second layer should have an input size of 16 and an output size of env.action_space.n
        self.fc2 = ...

    def forward(self, x):
        # IMPLEMENT-ME
        # Implement the forward pass
        # apply a ReLU activation after the first linear layer
        x = ...
        # apply the second linear layer (without an activation).
        # the outputs of the second layer will act as the log probabilities for the Categorial distribution.
        x = ...
        return Categorical(logits=x)
```

### Selecting actions with our policy

For a given state our policy returns a pytorch `Categorial` object. We can sample from this distribution by calling it's `sample` method and we can find the log probability of an action using `log_prob`:

```python
policy = Policy().to(device)
state = env.reset()
# convert state (a numpy array) to a torch tensor
state = torch.from_numpy(state).float().to(device)
dist = policy(state)
action = dist.sample()

print("Sampled action: ", action.item())
print("Log probability of action: ", dist.log_prob(action).item())
```

### Computing the return

Given a sequence of rewards $(r(s_0, a_0), \ldots, r(s_T, a_T))$ we want to calculate the return $\sum_{t=0}^T \gamma^t r(s_t, a_t)$.

```python
def compute_returns(rewards, gamma):
    # IMPLEMENT-ME
    # compute the return using the above equation
    return returns
```

## REINFORCE

Now its time to implement the algorithm

**Monte-Carlo REINFORCE**:

for each episode:
1. sample a trajectory $\tau$ using the policy $\pi_\theta$.
2. compute $\nabla_\theta J(\theta) \approx \left(\sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(a_t|s_t)\right) \left(\sum_{t=0}^T \gamma^t r(s_t, a_t) \right)$.
3. update policy parameters $\theta \leftarrow \theta + \alpha \nabla_\theta J(\theta)$

### Hyperameters

```python
learning_rate = 1e-2
number_episodes = 1500
max_episode_length = 1000
gamma = 1.0
```

```python
def reinforce(seed, verbose=True):
    # set random seeds (for reproducibility)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    env.seed(seed)

    # instantiate the policy and optimizer
    policy = Policy().to(device)
    optimizer = optim.Adam(policy.parameters(), lr=learning_rate)

    scores = []
    scores_deque = deque(maxlen=100)
    for episode in range(1, number_episodes+1):
        #################################################################
        # 1. Collect trajectories using our policy and save the rewards #
        # and the log probability of each action taken.                 #
        #################################################################
        log_probs = []
        rewards = []
        state = env.reset()
        for t in range(max_episode_length):
            # IMPLEMENT-ME: get the distribution over actions for state
            dist = ...

            # IMPLEMENT-ME: sample an action from the distribution
            action = ...

            # IMPLEMENT-ME: compute the log probability
            log_prob = ...

            # IMPLEMENT-ME: take a step in the environment
            state, reward, done, _ = ...

            # save the reward and log probability
            rewards.append(reward)
            log_probs.append(log_prob.unsqueeze(0))

            if done:
                break

        # for reporting save the score
        scores.append(sum(rewards))
        scores_deque.append(sum(rewards))

        #################################################################
        # 2. evaluate the policy gradient                               #
        #################################################################

        # IMPLEMENT-ME: calculate the discounted return of the trajectory
        returns = ...

        log_probs = torch.cat(log_probs)
        # IMPLEMENT-ME: multiply the log probabilities by the returns and sum (see the policy-gradient theorem)
        # Remember to multiply the result by -1 because we want to maximise the returns
        policy_loss = ...

        #################################################################
        # 3. update the policy parameters (gradient descent)            #
        #################################################################
        optimizer.zero_grad()
        policy_loss.backward()
        optimizer.step()

        # report the score to check that we're making progress
        if episode % 50 == 0 and verbose:
            print('Episode {}\tAverage Score: {:.2f}'.format(episode, np.mean(scores_deque)))

        if np.mean(scores_deque) >= 495.0 and verbose:
            print('Environment solved in {:d} episodes!\tAverage Score: {:.2f}'.format(episode, np.mean(scores_deque)))
            break

    return policy, scores
```

```python
policy, scores = reinforce(42)
```

## Seeing our learned policy in action

Let's watch our learned policy balance the pole!

```python
env = gym.make('CartPole-v1')
state = env.reset()
for t in range(2000):
    dist = policy(torch.from_numpy(state).float().to(device))
    action = dist.sample()
    env.render()
    state, reward, done, _ = env.step(action.item())
    if done:
        break
env.close()
```

### Plotting the results

Finally, let's plot the learning curve.

```python
def moving_average(a, n) :
    ret = np.cumsum(a, dtype=float)
    ret[n:] = ret[n:] - ret[:-n]
    return ret / n
```

```python
fig = plt.figure()
ax = fig.add_subplot(111)
x = np.arange(1, len(scores)+1)
ax.plot(x, scores, label='Score')
m_average = moving_average(scores, 50)
ax.plot(x, m_average, label='Moving Average (w=100)', linestyle='--')
plt.legend()
plt.ylabel('Score')
plt.xlabel('Episode #')
plt.title('REINFORCE learning curve - CartPole-v1')
plt.show()
```

Here is what your graph should look like.

We can see that at the end of training our policy consistantly (more or less) recieves returns of 500.

## Investigating the variance of REINFORCE

We noted in class that REINFORCE is a high variance algorithm. We can investigate the variance by running multiple trials and averaging the results.

```python
np.random.seed(53)
seeds = np.random.randint(1000, size=5)
all_scores = []
for seed in seeds:
    print("started training with seed: ", seed)
    _, scores = reinforce(int(seed), verbose=False)
    print("completed training with seed: ", seed)
    all_scores.append(scores)
```

```python
smoothed_scores = [moving_average(s, 50) for s in all_scores]
smoothed_scores = np.array(smoothed_scores)
mean = smoothed_scores.mean(axis=0)
std = smoothed_scores.std(axis=0)
```

```python
fig = plt.figure()
ax = fig.add_subplot(111)
x = np.arange(1, len(mean)+1)
ax.plot(x, mean, '-', color='blue')
ax.fill_between(x, mean - std, mean + std, color='blue', alpha=0.2)
plt.ylabel('Score')
plt.xlabel('Episode #')
plt.title('REINFORCE averaged over 5 seeds')
plt.show()
```

Here is what your graph should look like.

## Reducing the variance of REINFORCE

In class we saw a couple of tricks to reduce the variance of REINFORCE and improve its performance. Firstly, future actions should not change past decision. Present actions only impact the future. Therefore, we can change our objective function to reflect this:
\begin{align}
    \nabla_\theta J(\theta) = \mathbb{E}_{\tau \sim p_\theta} \left[\sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(a_t|s_t) \sum_{t'= t}^T \gamma^{t'- t} r(s_{t'}, a_{t'})\right].
\end{align}

We can also reduce variance by subtracing a state dependent baseline to get:
\begin{align}
    \nabla_\theta J(\theta) = \mathbb{E}_{\tau \sim p_\theta} \left[\sum_{t=0}^{T} \nabla_\theta \log \pi_\theta(a_t|s_t) \sum_{t'= t}^T \left( \gamma^{t'- t} r(s_{t'}, a_{t'}) - b(s_{t'}) \right)\right].
\end{align}

For our baseline we'll use the average of the returns over the trajectory. As a final trick we normalise the returns by dividing by the standard deviation.

```python
def compute_returns_baseline(rewards, gamma):
    r = 0
    returns = []
    for step in reversed(range(len(rewards))):
        r = rewards[step] + gamma * r
        returns.insert(0, r)
    returns = np.array(returns)
    # IMPLEMENT-ME: normalize the returns by subtracting the mean and dividing by the standard deviation
    returns = ...
    return returns
```

```python
def reinforce_baseline(seed, verbose=True):
    # set random seeds (for reproducibility)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    env.seed(seed)

    # instantiate the policy and optimizer
    policy = Policy().to(device)
    optimizer = optim.Adam(policy.parameters(), lr=learning_rate)

    scores = []
    scores_deque = deque(maxlen=100)
    for episode in range(1, number_episodes+1):
        #################################################################
        # 1. Collect trajectories using our policy and save the rewards #
        # and the log probability of each action taken.                 #
        #################################################################
        log_probs = []
        rewards = []
        state = env.reset()
        for t in range(max_episode_length):
            # IMPLEMENT-ME: get the distribution over actions for state
            dist = ...

            # IMPLEMENT-ME: sample an action from the distribution
            action = ...

            # IMPLEMENT-ME: compute the log probability
            log_prob = ...

            # IMPLEMENT-ME: take a step in the environment
            state, reward, done, _ = ...

            # save the reward and log probability
            rewards.append(reward)
            log_probs.append(log_prob.unsqueeze(0))

            if done:
                break

        # for reporting save the score
        scores.append(sum(rewards))
        scores_deque.append(sum(rewards))

        #################################################################
        # 2. evaluate the policy gradient (with variance reduction)     #
        #################################################################

        # calculate the discounted return of the trajectory
        returns = compute_returns_baseline(rewards, gamma)
        returns = torch.from_numpy(returns).float().to(device)

        log_probs = torch.cat(log_probs)
        # IMPLEMENT-ME: multiply the log probabilities by the returns and sum (see the policy-gradient theorem)
        # Remember to multiply the result by -1 because we want to maximise the returns
        policy_loss = ...

        #################################################################
        # 3. update the policy parameters (gradient descent)            #
        #################################################################
        optimizer.zero_grad()
        policy_loss.backward()
        optimizer.step()

        # report the score to check that we're making progress
        if episode % 50 == 0 and verbose:
            print('Episode {}\tAverage Score: {:.2f}'.format(episode, np.mean(scores_deque)))

        if np.mean(scores_deque) >= 495.0 and verbose:
            print('Environment solved in {:d} episodes!\tAverage Score: {:.2f}'.format(episode, np.mean(scores_deque)))
            break

    return policy, scores
```

Let's see if these changes give us any improvement.

```python
np.random.seed(53)
seeds = np.random.randint(1000, size=5)
all_scores_baseline = []
for seed in seeds:
    print("started training with seed: ", seed)
    _, scores = reinforce_baseline(int(seed), verbose=False)
    print("completed training with seed: ", seed)
    all_scores_baseline.append(scores)
```

## Comparing the methods

Finally we'll compare the performance of the two methods.

```python
smoothed_scores_baseline = [moving_average(s, 50) for s in all_scores_baseline]
smoothed_scores_baseline = np.array(smoothed_scores_baseline)
mean_baseline = smoothed_scores_baseline.mean(axis=0)
std_baseline = smoothed_scores_baseline.std(axis=0)
```

```python
fig = plt.figure()
ax = fig.add_subplot(111)
x = np.arange(1, len(mean_baseline)+1)
ax.plot(x, mean_baseline, '-', color='green', label='Variance reduced REINFORCE')
ax.plot(x, mean, '-', color='blue', label='REINFORCE')
ax.fill_between(x, mean_baseline - std_baseline, mean_baseline + std_baseline, color='green', alpha=0.2)
ax.fill_between(x, mean - std, mean + std, color='blue', alpha=0.2)
plt.ylabel('Score')
plt.xlabel('Episode #')
plt.legend()
plt.title('Comparison of REINFORCE and Variance reduced REINFORCE (averaged over 5 seeds)')
plt.show()
```

Here is what your graph should look like.

```python

```


```python
Godfrey Beddard 'Applying Maths in the Chemical & Biomolecular Sciences an example-based approach' Chapter 9
```

```python
# import all python add-ons etc that will be needed later on
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from sympy import *
from scipy.integrate import quad
init_printing()                      # allows printing of SymPy results in typeset maths format
plt.rcParams.update({'font.size': 14})  # set font size for plots
```

# 11 The Hadamard Transform: Encoding and Decoding

## 11.1 Concept and motivation

To obtain the average value of a quantity, such as weight, individual measurements are usually made, then added together and divided by the number of measurements made. However, this is not the only way of obtaining the average. You may be familiar with the method of weighing several objects at a time and perhaps know also that doing this will reduce the error in the average value. This multiple weighing method is an example of using a _Hadamard transform_ and anything that can be measured in groups can be treated in the same way, for example, a spectrum or an image, thus the method is quite general. The reason that this multiple measuring method works is that the error is introduced by the balance not by the objects being weighed. A large weight therefore has the same error associated with its measurement as a smaller one does.

The reason for doing any transform experiment is always the same and either this is to achieve an improvement in signal to noise, or, a reduction in the time taken to do an experiment at a fixed signal to noise, which is effectively the same thing. Normally for $n$ measurements, the signal to noise increases only as $\sqrt{n}$ but in the Hadamard approach, the signal to noise achievable increases at least as $n/2$, which is a huge improvement if, for example, $n= 100$.

## 11.2 The Hadamard Transform

The Hadamard transform is a purely discrete transform and instead of forward and back transforming, as in the Fourier transform, the equivalent steps are encoding and decoding. The encoding is done by adding several measurements together according to a set of rules or algorithm. The rule is always written down as a matrix, two forms of which $\boldsymbol H$ and $\boldsymbol S$ can be used; in the first $\boldsymbol H$ is a matrix of ones and minus ones, the other $\boldsymbol S$, is a matrix of zeros and ones. We shall concentrate of the $\boldsymbol S$ matrix form; it is the most useful one to use experimentally because it involves making only one measurement at a time; the $\boldsymbol H$ matrix method involves making two measurements. Harwit & Sloane (1979) describe the Hadamard transform method in detail, but see Marshall (1978) for a brief description.

## 11.3 Encoding and decoding with $\boldsymbol S$ matrices

Suppose that there are three samples to be weighed of masses $x_{1,2,3}$ they could be grouped as $x_1 + x_2,\;   x_1 + x_3$ and $x_2 + x_3$ and weighed two at a time on a single pan balance. Written as equations where the $z$'s are the measured values then

$$\displaystyle  z_1 = x_1 +x_2 +0 \\
z_2 = x_1 +0+x_3 \tag{47}\\
z_3 = 0+x_2 +x_3$$

These equations can be solved simultaneously to find the weights ( $x$'s ), for example, $2x_1 = z_1 + z_2 - z_3$ and so forth. Instead of doing this, which would be hard if there were $100$ equations, the coefficients in equation (47) can be put into a matrix called an $\boldsymbol S$ matrix, (also called a Simplex matrix) and for this example this is

$$\displaystyle  \boldsymbol{S } =
\begin{bmatrix}
1 & 1 & 0\\
1 & 0 & 1\\
0 & 1 & 1\\
\end{bmatrix}$$

Note how the pattern is the same as that of the coefficients in the equations and that it cycles around so that each column is related to the next by one position of cyclic rotation. As a matrix equation, equation (47) is

$$\displaystyle  \boldsymbol{z}=\boldsymbol{Sx}$$

and to solve for (column) vector $\boldsymbol x$, (column) vector $\boldsymbol z$ is multiplied by the inverse of
matrix $\boldsymbol S$ and the result is

$$\displaystyle  \boldsymbol x = \boldsymbol S ^{-1} \boldsymbol z \tag{48}$$

To show that this works, suppose that the $\boldsymbol x$ values are $12, 5$, and $2$, then the individual $\boldsymbol z$ values are $x_1 +x_2 =17$, $z_2 =14$ ,and $z_3 =7$ and the calculation $\boldsymbol S^{-1}\boldsymbol z$ is

$$\displaystyle \begin{bmatrix}
1/2 & 1/2 & -1/2\\
1/2 & -1/2 & 1/2\\
-1/2 & 1/2 & 1/2\\
\end{bmatrix}
\begin{bmatrix}
17\\ 14\\ 7 \end{bmatrix} =  \begin{bmatrix} 12 \\ 5 \\2 \end{bmatrix} $$

Of course if there are more than three values then a different $\boldsymbol S$ matrix will have to be be used. The rules are that each column must be orthogonal to every other one and that each column must be related to the next by cyclic rotation, and thus they contain the same number of zeros and ones. The rules for producing the $\boldsymbol S$ matrix are described in Section 11.6. As a check the matrix inverse $\boldsymbol S^{-1}$ is calculated using SymPy.

```python
# check on matrix inverse
S=symbols('M')
S=Matrix([[1, 1, 0], [1, 0, 1], [0, 1,1]])
S**(-1)
```

$\displaystyle \left[\begin{matrix}\frac{1}{2} & \frac{1}{2} & - \frac{1}{2}\\\frac{1}{2} & - \frac{1}{2} & \frac{1}{2}\\- \frac{1}{2} & \frac{1}{2} & \frac{1}{2}\end{matrix}\right]$

## 11.4 Signal to Noise improvement

To see why this method works to improve signal to noise some error has to be added to each measurement. If each measurement has a standard deviation determined by the instrument (scales) used then a measurement of $x$ if done individually also has this error. When weighed in pairs, each pair of $x$'s has the same standard deviation $\sigma$ because this is a property of the scales not the weights. The mean square error between the true and estimated values $\psi$ is

$$\displaystyle  mse \equiv \left< m^2\right> = \sum_i(\psi_i - m_i)^2$$

which is used to estimate the overall error. If $\boldsymbol z$ is the vector of the estimated values, the calculation for weighing in groups is $\psi =\boldsymbol S^{-1}\boldsymbol z$

```python
z,m1,m2,m3,sigma = symbols('z m1 m2 m3 sigma')
z = Matrix([m1 + m2 + sigma, m1 + m3 + sigma,m2 + m3 + sigma]) # define matrix
psi = S**(-1)*z
psi
```

$\displaystyle \left[\begin{matrix}m_{1} + \frac{\sigma}{2}\\m_{2} + \frac{\sigma}{2}\\m_{3} + \frac{\sigma}{2}\end{matrix}\right]$

subtracting each $m_i$ from each $\psi_i$ and summing produces $mse = 3\sigma^2/4$. The mean square error of any single measurement is $\sigma^2$ so that an improvement is obtained by measuring in groups, although it is small in this case. If $n$ measurements are made then mean square error is reduced by $\displaystyle (n + 1)^2/4n$ and the signal to noise improved by $\displaystyle (n+1)/\sqrt{4n} \approx\sqrt{n}/2$.

## 11.5 Implementation

Instead of weights, suppose that a spectrum is to be measured. To do the experiment, the detector is placed at the focusing plane of a spectrometer, the exit slits of which are removed and then a mask consisting of strips of opaque (0) and transparent (1) regions is placed there instead, see Fig. 49. At the first position the total amount of light falling on the detector is measured, this is $z_1$ and corresponds to measuring at positions $1 + 2 + 3 + 5$. Next, the mask is moved by one position, $z_2$ is measured which corresponds to light transmitted by $2 + 3 + 4 + 6$, and so on until all measurements are taken, $7$ in this example. Each measurement corresponds to moving from one column to the next in the $\boldsymbol S$ matrix and the total light measured forms the $\boldsymbol z$ matrix. Once this encoded $\boldsymbol z$ matrix is established it is multiplied by $\boldsymbol S^{-1}$ and the signal $\boldsymbol x$  recovered. Experiments have been performed by physically moving a mask etched in glass but a programmable liquid crystal mask would be easier to use. If there are $n$ elements in the mask, then $n$ different wavelengths are measured at the end of the $n$ experiments. The resolution is determined by the width of the mask compared to the wavelength spread it covers.

Figure 49. The pattern of the mask replaces the slits of the spectrometer. The detector measures all the light transmitted by the mask at each position. Each mask is rotated by one element from the previous one. All the mask elements when placed together form the $\boldsymbol S$ matrix. A possible experimental set up is shown on the right.
____

In Fig. 50 is shown a simulated comparison of data taken in the normal way and with that taken using the Hadamard encoding method. The noise on the detector is normally distributed with a mean of $0$ and $\sigma = 1$. The improvement in signal to noise is clear. In the calculation normally distributed noise with a standard deviation of $0.5$ was added to a damped exponential. The data shown is the result of an  $n = 199$ Hadamard experiment. The expected improvement in signal to noise is the ratio of the mean square error of the two sets of data. The expected value for the increase in signal to noise ratio is $(n + 1)^2/\sqrt{4n} \approx 7$ for the $199$ data points used and the measured value for this particular calculation is 7.3. When there is additionally noise imposed on the data before it reaches the detector the Hadamard method only reduces the noise due to the detector.

The Hadamard technique has recently been applied by the author to time resolved x-ray crystallography (Nat. Methods. 2014 Nov; 11(11): 1131-1134. )and has also been applied to time-resolved spectroscopic measurements (Appl. Spectros. 2016,70,1292-1299). To perform a time-resolved absorption measurement, a train of pulses whose intensity is in the $0$ and $1$ pattern of one column of the $\boldsymbol S$ matrix is passed through the sample each time the reaction is started. Starting the reaction can be achieved by mixing reactants in a stopped flow reactor or by a flash of light in a flash photolysis or pump probe experiment. The total light transmitted by the sample is recorded after each pulse train has finished. The first train of pulses produces point $z_1$ as described above. The experiment is repeated by exciting the sample again, rotating the sequence of pulses by one element and the total light transmitted is recorded again, producing point $z_2$, and this is repeated for n experiments. The $\boldsymbol z$ data, a column matrix of $n$ elements, is the Hadamard encoded data and is decoded as in equation (48) to produce the $\boldsymbol x$ column matrix, which is the intensity transmitted by the sample at each time point.

The timescale of the measurement is set by the spacing between any two pulses in the sequence and the total measurement time is $n$ times this. The train of pulses can conveniently be produced by an electro or acousto - optic modulator, either by extracting pulses from a train of mode-locked laser pulses or from a continuous laser or other light source. If pulses could be produced close enough together in time, this method would remove the need for delay lines in picosecond and femtosecond pump-probe experiments.

Figure 50. Comparison of Hadamard and normal (conventional) on the same set of data. The initial ideal data is shown in both figures as the blue line, the 'measured' data is in red.

_______

## 11.6 Constructing the $\boldsymbol S$ matrix

Harwit & Sloane (1979) give several methods by which to construct the $\boldsymbol S$ matrix. The simplest is the Quadratic Residue method, which produces a sequence of ones and zeros of length $n$ but only if $n$ is a prime number satisfying the condition also that $4m + 3$ where $m$ is also an integer. Once $n$ is chosen, the numbers $i = 1, 4, 9, \cdots n^2 \cdots $ are divided by $n$ and the remainders are the indices in a sequence of numbers and these numbers have a value of one and the rest are zero. The $\boldsymbol S$ matrix is then made from this list by rotating each new column by one element compared to its neighbour in a cyclical manner.

The Quadratic residue $R$ is defined as $n^2 \equiv R \mod(b)$ meaning that integer division of $n^2$ by $b$ produces remainder R. As an example $7^2 = 9\mod(10)$, meaning that $10$ divides $49, 4$ times exactly with remainder $9$.

A flow diagram to make a row of the $\boldsymbol S$ matrix is shown in the sketch and some Python code to do this below it.

______

```python
# check for valid sequence length. First few values are 3, 7, 11, 19, 23, 31, 43, 47,
#------------
def valid_seq_length(n):

    maxi = 200
    Hseq = np.zeros(maxi,dtype=int)
    for i in range(maxi):                   # produce Hadamard sequence numbers
        for m in range(0,maxi):
            if isprime(i) and i == 4*m + 3:
                Hseq[i]=i
            pass
    if n in Hseq[0:maxi]:
        is_ok=True
    else:
        is_ok=False
    return is_ok
#------------
def isprime(n):     # check if integer n is a prime, range starts with 2 and only needs to go up the squareroot of n
    for x in range(2, int(n**0.5)+1):
        if n % x == 0:
            return False
    return True
#------------
def make_S_mat(Srow):
    n = len(Srow)
    S = np.zeros((n,n),dtype = int)
    for i in range(n):
        for j in range(n):
            S[i,j] = Srow[n-1-j]
        Srow = np.roll(Srow, -1)             # rotate by 1 element at a time
        pass
    return S
#------------
def quadratic_hadamard(n):    # quadratic residue method, this generates an S matrix.

    alist = np.zeros(n,dtype=int)
    for i in range(0,(n-1)//2):            # integer division
        alist[(i+1)*(i+1) % n] = 1.0       # alist = hadamard Srow need only go to half range of n as indices are symmetric
    alist[0] = 1.0
    Srow = list(alist)
    S = make_S_mat(Srow[::-1])             # *****   reverse to get 1's at start; no other reason ****

    return S                               # returns S matrix
#------------

print('Hadamard S matrices by Quadratic residue method')
for i in range(1,12):                      # print out S matrices up to size of 12 x 12
    if valid_seq_length(i):
        S = quadratic_hadamard(i)
        print(i)
        if i  <= 32 :                      # for larger matrices print just first line
            print('\n'.join( [''.join(['{:2}'.format(item) for item in row] ) for row in S] ) )
        else:
            print(''.join(['{:2}'.format(item) for item in S[0]]))
            xs=''.join( str(S[0][i]) for i in range(len(S[0])) )
            print(hex(int(xs,2)))
    pass

```

    Hadamard S matrices by Quadratic residue method
    3
     1 1 0
     0 1 1
     1 0 1
    7
     1 1 1 0 1 0 0
     0 1 1 1 0 1 0
     0 0 1 1 1 0 1
     1 0 0 1 1 1 0
     0 1 0 0 1 1 1
     1 0 1 0 0 1 1
     1 1 0 1 0 0 1
    11
     1 1 0 1 1 1 0 0 0 1 0
     0 1 1 0 1 1 1 0 0 0 1
     1 0 1 1 0 1 1 1 0 0 0
     0 1 0 1 1 0 1 1 1 0 0
     0 0 1 0 1 1 0 1 1 1 0
     0 0 0 1 0 1 1 0 1 1 1
     1 0 0 0 1 0 1 1 0 1 1
     1 1 0 0 0 1 0 1 1 0 1
     1 1 1 0 0 0 1 0 1 1 0
     0 1 1 1 0 0 0 1 0 1 1
     1 0 1 1 1 0 0 0 1 0 1

The figure below shows two $\boldsymbol S$ matrices of sizes $7$ and $11$. The circulant nature of these matrices is clear.

_____

In the calculation the inverse of the $\boldsymbol S$ matrix is needed. This can be inverted in SymPy as $\boldsymbol S^{-1}$ but for the particular form of this matrix, inversion can be obtained more quickly using the formula

$$ \boldsymbol S^{-1}= \frac{2}{n+1} ( 2 \boldsymbol S^T -\boldsymbol J_n  ) $$

where $T$ is the transpose and matrix $\boldsymbol J$ is an $n\times n$ matrix where every value is one (Harwit & Sloane 1979).