# Assignment 2 - Semi-gradient TD with a Neural Network

Welcome to Course 3 Programming Assignment 2. In the previous assignment, you implemented semi-gradient TD with State Aggregation for solving a **policy evaluation task**. In this assignment, you will implement **semi-gradient TD with a simple Neural Network** and use it for the same policy evaluation problem.

You will implement an agent to evaluate a fixed policy on the 500-State Randomwalk. As you may remember from the previous assignment, the 500-state Randomwalk includes 500 states. Each episode begins with the agent at the center and terminates when the agent goes far left beyond state 1 or far right beyond state 500. At each time step, the agent selects to move either left or right with equal probability. The environment determines how much the agent moves in the selected direction.

**In this assignment, you will:**
- Implement stochastic gradient descent method for state-value prediction.
- Implement semi-gradient TD with a neural network as the function approximator and Adam algorithm.
- Compare performance of semi-gradient TD with a neural network and semi-gradient TD with tile-coding.

## Packages
We import the following libraries that are required for this assignment:

- [numpy](www.numpy.org) : Fundamental package for scientific computing with Python.
- [matplotlib](http://matplotlib.org) : Library for plotting graphs in Python.
- [RL-Glue](http://www.jmlr.org/papers/v10/tanner09a.html) : Library for reinforcement learning experiments.
- [tqdm](https://tqdm.github.io/) : A package to display progress bar when running experiments.
- BaseOptimizer : An abstract class that specifies the optimizer API for Agent.
- plot_script : Custom script to plot results.
- RandomWalkEnvironment : The Randomwalk environment script from Course 3 Assignment 1.

```python
# Do not modify this cell!

# Import necessary libraries
# DO NOT IMPORT OTHER LIBRARIES - This will break the autograder.
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

import os, shutil
from tqdm import tqdm

from rl_glue import RLGlue
from environment import BaseEnvironment
from agent import BaseAgent
from optimizer import BaseOptimizer
import plot_script
from randomwalk_environment import RandomWalkEnvironment
```

## Section 1: Create semi-gradient TD with a Neural Network
In this section, you will implement an Agent that learns with semi-gradient TD with a neural network. You will use a neural network with one hidden layer. The input of the neural network is the one-hot encoding of the state number. We use the one-hot encoding of the state number instead of the state number itself because we do not want to build the prior knowledge that integer number inputs close to each other have similar values. The hidden layer contains 100 rectifier linear units (ReLUs) which pass their input if it is bigger than one and return 0 otherwise. ReLU gates are commonly used in neural networks due to their nice properties such as the sparsity of the activation and having non-vanishing gradients. The output of the neural network is the estimated state value. It is a linear function of the hidden units as is commonly the case when estimating the value of a continuous target using neural networks.

The neural network looks like this:

For a given input, $s$, value of $s$ is computed by:
$$
\begin{align}
\psi &= sW^{[0]} + b^{[0]} \\
x &= \textit{max}(0, \psi) \\
v &= xW^{[1]} + b^{[1]}
\end{align}
$$

where $W^{[0]}$, $b^{[0]}$, $W^{[1]}$, $b^{[1]}$  are the parameters of the network and will be learned when training the agent.

## 1-1: Implement helper methods

Before implementing the agent, you first implement some helper functions which you will later use in agent's main methods.

### Implement `get_value()`
First, you will implement get_value() method which feeds an input $s$ into the neural network and returns the output of the network $v$ according to the equations above. To implement get_value(), take into account the following notes:

- `get_value()` gets the one-hot encoded state number denoted by s as an input.
- `get_value()` receives the weights of the neural network as input, denoted by weights and structured as an array of dictionaries. Each dictionary corresponds to weights from one layer of the neural network to the next. Each dictionary includes $W$ and $b$. The shape of the elements in weights are as follows:
    - weights[0]["W"]: num_states $\times$ num_hidden_units
    - weights[0]["b"]: 1 $\times$ num_hidden_units
    - weights[1]["W"]: num_hidden_units $\times$ 1
    - weights[1]["b"]: 1 $\times$ 1

- The input of the neural network is a sparse vector. To make computation faster, we take advantage of input sparsity. To do so, we provided a helper method `my_matmul()`. **Make sure that you use `my_matmul()` for all matrix multiplications except for element-wise multiplications in this notebook.**
- The max operator used for computing $x$ is element-wise.

```python
def my_matmul(x1, x2):
    """
    Given matrices x1 and x2, return the multiplication of them
    """

    result = np.zeros((x1.shape[0], x2.shape[1]))
    x1_non_zero_indices = x1.nonzero()
    if x1.shape[0] == 1 and len(x1_non_zero_indices[1]) == 1:
        result = x2[x1_non_zero_indices[1], :]
    elif x1.shape[1] == 1 and len(x1_non_zero_indices[0]) == 1:
        result[x1_non_zero_indices[0], :] = x2 * x1[x1_non_zero_indices[0], 0]
    else:
        result = np.matmul(x1, x2)
    return result
```

```python
#GRADED FUNCTION: [get_value]

def get_value(s, weights):
    """
    Compute value of input s given the weights of a neural network
    """
    ### Compute the ouput of the neural network, v, for input s (3 lines)

    ### START CODE HERE ###
    h1 = my_matmul(s, weights[0]["W"]) + weights[0]["b"]
    relu_mask = h1 > 0
    x1 = h1 * relu_mask
    v = my_matmul(x1, weights[1]["W"]) + weights[1]["b"]
    ### END CODE HERE ###
    return v
```

Run the following code to test your implementation of the `get_value()` function:

```python
## Test Code for get_value() ##

# Suppose num_states = 5, num_hidden_layer = 1, and num_hidden_units = 10
num_hidden_layer = 1
s = np.array([[0, 0, 0, 1, 0]])

weights_data = np.load("asserts/get_value_weights.npz")
weights = [dict() for i in range(num_hidden_layer+1)]
weights[0]["W"] = weights_data["W0"]
weights[0]["b"] = weights_data["b0"]
weights[1]["W"] = weights_data["W1"]
weights[1]["b"] = weights_data["b1"]

estimated_value = get_value(s, weights)
print ("Estimated value: {}".format(estimated_value))
assert(np.allclose(estimated_value, np.array([[-0.21915705]])))

print ("Passed the assert!")

```

    Estimated value: [[-0.21915705]]
    Passed the assert!

**Expected output**:

    Estimated value: [[-0.21915705]]

### Implement `get_gradient()`
You will also implement `get_gradient()` method which computes the gradient of the value function for a given input, using backpropagation. You will later use this function to update the value function.

As you know, we compute the value of a state $s$ according to:
$$
\begin{align}
\psi &= sW^{[0]} + b^{[0]} \\
x &= \textit{max}(0, \psi) \\
v &= xW^{[1]} + b^{[1]}
\end{align}
$$

To update the weights of the neural network ($W^{[0]}$, $b^{[0]}$, $W^{[1]}$, $b^{[1]}$), we compute the gradient of $v$ with respect to the weights according to:

$$
\begin{align}
\frac{\partial v}{\partial W^{[0]}} &= s^T(W^{[1]T} \odot I_{x>0}) \\
\frac{\partial v}{\partial b^{[0]}} &= W^{[1]T} \odot I_{x>0} \\
\frac{\partial v}{\partial W^{[1]}} &= x^T \\
\frac{\partial v}{\partial b^{[1]}} &= 1
\end{align}
$$
where $\odot$ denotes element-wise matrix multiplication and $I_{x>0}$ is the gradient of the ReLU activation function which is an indicator whose $i$th element is 1 if $x[i]>0$ and 0 otherwise.

```python
#GRADED FUNCTION: [get_gradient]

def get_gradient(s, weights):
    """
    Given inputs s and weights, return the gradient of v with respect to the weights
    """

    ### Compute the gradient of the value function with respect to W0, b0, W1, b1 for input s (6~8 lines)
    # grads[0]["W"] = ?
    # grads[0]["b"] = ?
    # grads[1]["W"] = ?
    # grads[1]["b"] = ?
    # Note that grads[0]["W"], grads[0]["b"], grads[1]["W"], and grads[1]["b"] should have the same shape as
    # weights[0]["W"], weights[0]["b"], weights[1]["W"], and weights[1]["b"] respectively
    # Note that to compute the gradients, you need to compute the activation of the hidden layer (x)

    grads = [dict() for i in range(len(weights))]

    ### START CODE HERE ###
    h1 = my_matmul(s, weights[0]["W"]) + weights[0]["b"]
    relu_mask = h1 > 0
    x1 = h1 * relu_mask

    dW1 = x1.T
    db1 = 1

    dx1 = weights[1]["W"].T
    dh1 = dx1 * relu_mask
    db0 = dh1
    dW0 = my_matmul(s.T, dh1)

    grads[1]["W"] = dW1
    grads[1]["b"] = db1
    grads[0]["W"] = dW0
    grads[0]["b"] = db0

    ### END CODE HERE ###

    return grads

```

Run the following code to test your implementation of the `get_gradient()` function:

```python
## Test Code for get_gradient() ##

# Suppose num_states = 5, num_hidden_layer = 1, and num_hidden_units = 2
num_hidden_layer = 1
s = np.array([[0, 0, 0, 1, 0]])

weights_data = np.load("asserts/get_gradient_weights.npz")
weights = [dict() for i in range(num_hidden_layer+1)]
weights[0]["W"] = weights_data["W0"]
weights[0]["b"] = weights_data["b0"]
weights[1]["W"] = weights_data["W1"]
weights[1]["b"] = weights_data["b1"]

grads = get_gradient(s, weights)

grads_answer = np.load("asserts/get_gradient_grads.npz")

print("grads[0][\"W\"]\n", grads[0]["W"], "\n")
print("grads[0][\"b\"]\n", grads[0]["b"], "\n")
print("grads[1][\"W\"]\n", grads[1]["W"], "\n")
print("grads[1][\"b\"]\n", grads[1]["b"], "\n")

assert(np.allclose(grads[0]["W"], grads_answer["W0"]))
assert(np.allclose(grads[0]["b"], grads_answer["b0"]))
assert(np.allclose(grads[1]["W"], grads_answer["W1"]))
assert(np.allclose(grads[1]["b"], grads_answer["b1"]))

print("Passed the asserts!")

```

    grads[0]["W"]
     [[0.         0.        ]
     [0.         0.        ]
     [0.         0.        ]
     [0.76103773 0.12167502]
     [0.         0.        ]]

    grads[0]["b"]
     [[0.76103773 0.12167502]]

    grads[1]["W"]
     [[0.69198983]
     [0.82403662]]

    grads[1]["b"]
     1

    Passed the asserts!

**Expected output**:

    grads[0]["W"]
     [[0.         0.        ]
     [0.         0.        ]
     [0.         0.        ]
     [0.76103773 0.12167502]
     [0.         0.        ]]

    grads[0]["b"]
     [[0.76103773 0.12167502]]

    grads[1]["W"]
     [[0.69198983]
     [0.82403662]]

    grads[1]["b"]
     [[1.]]

### Implement stochastic gradient descent method for state-value prediction
In this section, you will implement stochastic gradient descent (SGD) method for state_value prediction. Here is the basic SGD update for state-value prediction with TD:

$$\mathbf{w_{t+1}} = \mathbf{w_{t}} + \alpha \delta_t \nabla \hat{v}(S_t,\mathbf{w_{t}})$$

At each time step, we update the weights in the direction  $g_t = \delta_t \nabla \hat{v}(S_t,\mathbf{w_t})$ using a fixed step-size $\alpha$. $\delta_t = R_{t+1} + \gamma \hat{v}(S_{t+1},\mathbf{w_{t}}) - \hat{v}(S_t,\mathbf{w_t})$ is the TD-error. $\nabla \hat{v}(S_t,\mathbf{w_{t}})$ is the gradient of the value function with respect to the weights.

The following cell includes the SGD class. You will complete the `update_weight()` method of SGD assuming that the weights and update g are provided.

**As you know, in this assignment, we structured the weights as an array of dictionaries. Note that the updates $g_t$, in the case of TD, is $\delta_t \nabla \hat{v}(S_t,\mathbf{w_t})$. As a result, $g_t$ has the same structure as $\nabla \hat{v}(S_t,\mathbf{w_t})$ which is also an array of dictionaries.**

```python
#GRADED FUNCTION: [SGD]

class SGD(BaseOptimizer):
    def __init__(self):
        pass

    def optimizer_init(self, optimizer_info):
        """Setup for the optimizer.

        Set parameters needed to setup the stochastic gradient descent method.

        Assume optimizer_info dict contains:
        {
            step_size: float
        }
        """
        self.step_size = optimizer_info.get("step_size")

    def update_weights(self, weights, g):
        """
        Given weights and update g, return updated weights
        """
        for i in range(len(weights)):
            for param in weights[i].keys():

                ### update weights (1 line)
                # weights[i][param] = None

                ### START CODE HERE ###
                weights[i][param] += self.step_size * g[i][param]
                ### END CODE HERE ###

        return weights
```

Run the following code to test your implementation of the `update_weights()` function:

```python
# Do not modify this cell!

## Test Code for update_weights() ##

# Suppose num_states = 5, num_hidden_layer = 1, and num_hidden_units = 2
num_hidden_layer = 1

weights_data = np.load("asserts/update_weights_weights.npz")
weights = [dict() for i in range(num_hidden_layer+1)]
weights[0]["W"] = weights_data["W0"]
weights[0]["b"] = weights_data["b0"]
weights[1]["W"] = weights_data["W1"]
weights[1]["b"] = weights_data["b1"]

g_data = np.load("asserts/update_weights_g.npz")
g = [dict() for i in range(num_hidden_layer+1)]
g[0]["W"] = g_data["W0"]
g[0]["b"] = g_data["b0"]
g[1]["W"] = g_data["W1"]
g[1]["b"] = g_data["b1"]

test_sgd = SGD()
optimizer_info = {"step_size": 0.3}
test_sgd.optimizer_init(optimizer_info)
updated_weights = test_sgd.update_weights(weights, g)

# updated weights asserts
updated_weights_answer = np.load("asserts/update_weights_updated_weights.npz")

print("updated_weights[0][\"W\"]\n", updated_weights[0]["W"], "\n")
print("updated_weights[0][\"b\"]\n", updated_weights[0]["b"], "\n")
print("updated_weights[1][\"W\"]\n", updated_weights[1]["W"], "\n")
print("updated_weights[1][\"b\"]\n", updated_weights[1]["b"], "\n")

assert(np.allclose(updated_weights[0]["W"], updated_weights_answer["W0"]))
assert(np.allclose(updated_weights[0]["b"], updated_weights_answer["b0"]))
assert(np.allclose(updated_weights[1]["W"], updated_weights_answer["W1"]))
assert(np.allclose(updated_weights[1]["b"], updated_weights_answer["b1"]))

print("Passed the asserts!")

```

    updated_weights[0]["W"]
     [[ 1.17899492  0.53656321]
     [ 0.58008221  1.47666572]
     [ 1.01909411 -1.10248056]
     [ 0.72490408  0.06828853]
     [-0.20609725  0.69034095]]

    updated_weights[0]["b"]
     [[-0.18484533  0.92844539]]

    updated_weights[1]["W"]
     [[0.70488257]
     [0.58150878]]

    updated_weights[1]["b"]
     [[0.88467086]]

    Passed the asserts!

**Expected output**:

    updated_weights[0]["W"]
     [[ 1.17899492  0.53656321]
     [ 0.58008221  1.47666572]
     [ 1.01909411 -1.10248056]
     [ 0.72490408  0.06828853]
     [-0.20609725  0.69034095]]

    updated_weights[0]["b"]
     [[-0.18484533  0.92844539]]

    updated_weights[1]["W"]
     [[0.70488257]
     [0.58150878]]

    updated_weights[1]["b"]
     [[0.88467086]]

### Adam Algorithm
In this assignment, instead of using SGD for updating the weights, we use a more advanced algorithm called Adam. The Adam algorithm improves the SGD update with two concepts: adaptive vector step-sizes and momentum. It keeps estimates of the mean and second moment of the updates, denoted by $\mathbf{m}$ and $\mathbf{v}$ respectively:
$$\mathbf{m_t} = \beta_m \mathbf{m_{t-1}} + (1 - \beta_m)g_t \\
\mathbf{v_t} = \beta_v \mathbf{v_{t-1}} + (1 - \beta_v)g^2_t
$$

Given that $\mathbf{m}$ and $\mathbf{v}$ are initialized to zero, they are biased toward zero. To get unbiased estimates of the mean and second moment, Adam defines $\mathbf{\hat{m}}$ and $\mathbf{\hat{v}}$ as:
$$ \mathbf{\hat{m_t}} = \frac{\mathbf{m_t}}{1 - \beta_m^t} \\
\mathbf{\hat{v_t}} = \frac{\mathbf{v_t}}{1 - \beta_v^t}
$$

The weights are then updated as follows:
$$ \mathbf{w_t} = \mathbf{w_{t-1}} + \frac{\alpha}{\sqrt{\mathbf{\hat{v_t}}}+\epsilon} \mathbf{\hat{m_t}}
$$

When implementing the agent you will use the Adam algorithm instead of SGD because it is more efficient. We have already provided you the implementation of the Adam algorithm in the cell below. You will use it when implementing your agent.

```python
class Adam(BaseOptimizer):
    def __init__(self):
        pass

    def optimizer_init(self, optimizer_info):
        """Setup for the optimizer.

        Set parameters needed to setup the Adam algorithm.

        Assume optimizer_info dict contains:
        {
            num_states: integer,
            num_hidden_layer: integer,
            num_hidden_units: integer,
            step_size: float,
            self.beta_m: float
            self.beta_v: float
            self.epsilon: float
        }
        """

        self.num_states = optimizer_info.get("num_states")
        self.num_hidden_layer = optimizer_info.get("num_hidden_layer")
        self.num_hidden_units = optimizer_info.get("num_hidden_units")

        # Specify Adam algorithm's hyper parameters
        self.step_size = optimizer_info.get("step_size")
        self.beta_m = optimizer_info.get("beta_m")
        self.beta_v = optimizer_info.get("beta_v")
        self.epsilon = optimizer_info.get("epsilon")

        self.layer_size = np.array([self.num_states, self.num_hidden_units, 1])

        # Initialize Adam algorithm's m and v
        self.m = [dict() for i in range(self.num_hidden_layer+1)]
        self.v = [dict() for i in range(self.num_hidden_layer+1)]

        for i in range(self.num_hidden_layer+1):

            # Initialize self.m[i]["W"], self.m[i]["b"], self.v[i]["W"], self.v[i]["b"] to zero
            self.m[i]["W"] = np.zeros((self.layer_size[i], self.layer_size[i+1]))
            self.m[i]["b"] = np.zeros((1, self.layer_size[i+1]))
            self.v[i]["W"] = np.zeros((self.layer_size[i], self.layer_size[i+1]))
            self.v[i]["b"] = np.zeros((1, self.layer_size[i+1]))

        # Initialize beta_m_product and beta_v_product to be later used for computing m_hat and v_hat
        self.beta_m_product = self.beta_m
        self.beta_v_product = self.beta_v

    def update_weights(self, weights, g):
        """
        Given weights and update g, return updated weights
        """

        for i in range(len(weights)):
            for param in weights[i].keys():

                ### update self.m and self.v
                self.m[i][param] = self.beta_m * self.m[i][param] + (1 - self.beta_m) * g[i][param]
                self.v[i][param] = self.beta_v * self.v[i][param] + (1 - self.beta_v) * (g[i][param] * g[i][param])

                ### compute m_hat and v_hat
                m_hat = self.m[i][param] / (1 - self.beta_m_product)
                v_hat = self.v[i][param] / (1 - self.beta_v_product)

                ### update weights
                weights[i][param] += self.step_size * m_hat / (np.sqrt(v_hat) + self.epsilon)

        ### update self.beta_m_product and self.beta_v_product
        self.beta_m_product *= self.beta_m
        self.beta_v_product *= self.beta_v

        return weights

```

## 1-2: Implement Agent Methods
In this section, you will implement `agent_init()`, `agent_start()`, `agent_step()`, and `agent_end()`.

In `agent_init()`, you will:

   - specify the neural network structure by filling self.layer_size with the size of the input layer, hidden layer, and output layer.
   - initialize the network's parameters. We show the parameters as an array of dictionaries, self.weights, where each dictionary corresponds to weights from one layer to the next. Each dictionary includes $W$ and $b$. To initialize the parameters, you will use a normal distribution with mean 0 and standard deviation $\sqrt{\frac{2}{\text{# input of each node}}}$. This initialization heuristic is commonly used when using ReLU gates and helps keep the output of a neuron from getting too big or too small. To initialize the network's parameters, use **self.rand_generator.normal()** which draws random samples from a normal distribution. The parameters of self.rand_generator.normal are mean of the distribution, standard deviation of the distribution, and output shape in the form of tuple of integers.

In `agent_start()`, you will:
   - specify self.last_state and self.last_action.

In `agent_step()` and `agent_end()`, you will:
   - compute the TD error using $v(S_t)$ and $v(S_{t+1})$. To compute the value function for $S_t$ and $S_{t+1}$, you will get their one-hot encoding using `one_hot()` method that we provided below. You feed the one-hot encoded state number to the neural networks using `get_value()` method that you implemented above. Note that `one_hot()` method returns the one-hot encoding of a state as a numpy array of shape (1, num_states).
   - retrieve the gradients using `get_gradient()` function that you implemented.
   - use Adam_algorithm that we provided to update the neural network's parameters, self.weights.
   - use `agent_policy()` method to select actions with. (only in `agent_step()`)

```python
def one_hot(state, num_states):
    """
    Given num_state and a state, return the one-hot encoding of the state
    """
    # Create the one-hot encoding of state
    # one_hot_vector is a numpy array of shape (1, num_states)

    one_hot_vector = np.zeros((1, num_states))
    one_hot_vector[0, int((state - 1))] = 1

    return one_hot_vector
```

```python
#GRADED FUNCTION: [Agent]

class TDAgent(BaseAgent):
    def __init__(self):
        self.name = "td_agent"
        pass

    def agent_init(self, agent_info={}):
        """Setup for the agent called when the experiment first starts.

        Set parameters needed to setup the semi-gradient TD with a Neural Network.

        Assume agent_info dict contains:
        {
            num_states: integer,
            num_hidden_layer: integer,
            num_hidden_units: integer,
            step_size: float,
            discount_factor: float,
            self.beta_m: float
            self.beta_v: float
            self.epsilon: float
            seed: int
        }
        """

        # Set random seed for weights initialization for each run
        self.rand_generator = np.random.RandomState(agent_info.get("seed"))

        # Set random seed for policy for each run
        self.policy_rand_generator = np.random.RandomState(agent_info.get("seed"))

        # Set attributes according to agent_info
        self.num_states = agent_info.get("num_states")
        self.num_hidden_layer = agent_info.get("num_hidden_layer")
        self.num_hidden_units = agent_info.get("num_hidden_units")
        self.discount_factor = agent_info.get("discount_factor")

        ### Define the neural network's structure (1 line)
        # Specify self.layer_size which shows the number of nodes in each layer
        # self.layer_size = np.array([None, None, None])
        # Hint: Checkout the NN diagram at the beginning of the notebook

        ### START CODE HERE ###
        self.layer_size = [self.num_states, self.num_hidden_units, 1]
        ### END CODE HERE ###

        # Initialize the neural network's parameter (2 lines)
        self.weights = [dict() for i in range(self.num_hidden_layer+1)]
        for i in range(self.num_hidden_layer+1):

            ### Initialize self.weights[i]["W"] and self.weights[i]["b"] using self.rand_generator.normal()
            # Note that The parameters of self.rand_generator.normal are mean of the distribution,
            # standard deviation of the distribution, and output shape in the form of tuple of integers.
            # To specify output shape, use self.layer_size.

            ### START CODE HERE ###
            self.weights[i]["W"] = self.rand_generator.normal(0, np.sqrt(2/self.layer_size[i]), (self.layer_size[i], self.layer_size[i+1]))
            self.weights[i]["b"] = self.rand_generator.normal(0, np.sqrt(2/self.layer_size[i]), (1, self.layer_size[i+1]))

            ### END CODE HERE ###

        # Specify the optimizer
        self.optimizer = Adam()
        optimizer_info = {"num_states": agent_info["num_states"],
                          "num_hidden_layer": agent_info["num_hidden_layer"],
                          "num_hidden_units": agent_info["num_hidden_units"],
                          "step_size": agent_info["step_size"],
                          "beta_m": agent_info["beta_m"],
                          "beta_v": agent_info["beta_v"],
                          "epsilon": agent_info["epsilon"]}
        self.optimizer.optimizer_init(optimizer_info)

        self.last_state = None
        self.last_action = None

    def agent_policy(self, state):

        ### Set chosen_action as 0 or 1 with equal probability.
        chosen_action = self.policy_rand_generator.choice([0,1])
        return chosen_action

    def agent_start(self, state):
        """The first method called when the experiment starts, called after
        the environment starts.
        Args:
            state (Numpy array): the state from the
                environment's evn_start function.
        Returns:
            The first action the agent takes.
        """
        ### select action given state (using self.agent_policy()), and save current state and action (2 lines)
        # self.last_state = ?
        # self.last_action = ?

        ### START CODE HERE ###
        action = self.agent_policy(state)
        self.last_state = state
        self.last_action = action
        ### END CODE HERE ###

        return self.last_action

    def agent_step(self, reward, state):
        """A step taken by the agent.
        Args:
            reward (float): the reward received for taking the last action taken
            state (Numpy array): the state from the
                environment's step based, where the agent ended up after the
                last step
        Returns:
            The action the agent is taking.
        """

        ### Compute TD error (5 lines)
        # delta = None

        ### START CODE HERE ###
        last_state_onehot = one_hot(self.last_state, self.num_states)
        state_onehot = one_hot(state, self.num_states)

        v_last_state = get_value(last_state_onehot, self.weights)
        v_state = get_value(state_onehot, self.weights)

        td_error = reward + self.discount_factor * v_state - v_last_state

        ### END CODE HERE ###

        ### Retrieve gradients (1 line)
        # grads = None

        ### START CODE HERE ###
        grads = get_gradient(last_state_onehot, self.weights)
        ### END CODE HERE ###

        ### Compute g (1 line)
        g = [dict() for i in range(self.num_hidden_layer+1)]
        for i in range(self.num_hidden_layer+1):
            for param in self.weights[i].keys():

                # g[i][param] = None
                ### START CODE HERE ###
                g[i][param] = grads[i][param] * td_error
                ### END CODE HERE ###

        ### update the weights using self.optimizer (1 line)
        # self.weights = None

        ### START CODE HERE ###
        self.weights = self.optimizer.update_weights(self.weights, g)
        ### END CODE HERE ###

        ### update self.last_state and self.last_action (2 lines)

        ### START CODE HERE ###
        self.last_state = state
        self.last_action = self.agent_policy(self.last_state)

        ### END CODE HERE ###

        return self.last_action

    def agent_end(self, reward):
        """Run when the agent terminates.
        Args:
            reward (float): the reward the agent received for entering the
                terminal state.
        """

        ### compute TD error (3 lines)
        # delta = None

        ### START CODE HERE ###
        last_state_onehot = one_hot(self.last_state, self.num_states)
        v_last_state = get_value(last_state_onehot, self.weights)
        td_error = reward - v_last_state
        ### END CODE HERE ###

        ### Retrieve gradients (1 line)
        # grads = None

        ### START CODE HERE ###
        grads = get_gradient(last_state_onehot, self.weights)

        ### END CODE HERE ###

        ### Compute g (1 line)
        g = [dict() for i in range(self.num_hidden_layer+1)]
        for i in range(self.num_hidden_layer+1):
            for param in self.weights[i].keys():

                # g[i][param] = None
                ### START CODE HERE ###
                g[i][param] = grads[i][param] * td_error
                ### END CODE HERE ###

        ### update the weights using self.optimizer (1 line)
        # self.weights = None

        ### START CODE HERE ###
        self.weights = self.optimizer.update_weights(self.weights, g)
        ### END CODE HERE ###

    def agent_message(self, message):
        if message == 'get state value':
            state_value = np.zeros(self.num_states)
            for state in range(1, self.num_states + 1):
                s = one_hot(state, self.num_states)
                state_value[state - 1] = get_value(s, self.weights)
            return state_value
```

Run the following code to test your implementation of the `agent_init()` function:

```python
## Test Code for agent_init() ##

agent_info = {"num_states": 5,
              "num_hidden_layer": 1,
              "num_hidden_units": 2,
              "step_size": 0.25,
              "discount_factor": 0.9,
              "beta_m": 0.9,
              "beta_v": 0.99,
              "epsilon": 0.0001,
              "seed": 0
             }

test_agent = TDAgent()
test_agent.agent_init(agent_info)

print("layer_size: {}".format(test_agent.layer_size))
assert(np.allclose(test_agent.layer_size, np.array([agent_info["num_states"],
                                                    agent_info["num_hidden_units"],
                                                    1])))

print("weights[0][\"W\"] shape: {}".format(test_agent.weights[0]["W"].shape))
print("weights[0][\"b\"] shape: {}".format(test_agent.weights[0]["b"].shape))
print("weights[1][\"W\"] shape: {}".format(test_agent.weights[1]["W"].shape))
print("weights[1][\"b\"] shape: {}".format(test_agent.weights[1]["b"].shape), "\n")

assert(test_agent.weights[0]["W"].shape == (agent_info["num_states"], agent_info["num_hidden_units"]))
assert(test_agent.weights[0]["b"].shape == (1, agent_info["num_hidden_units"]))
assert(test_agent.weights[1]["W"].shape == (agent_info["num_hidden_units"], 1))
assert(test_agent.weights[1]["b"].shape == (1, 1))

print("weights[0][\"W\"]\n", (test_agent.weights[0]["W"]), "\n")
print("weights[0][\"b\"]\n", (test_agent.weights[0]["b"]), "\n")
print("weights[1][\"W\"]\n", (test_agent.weights[1]["W"]), "\n")
print("weights[1][\"b\"]\n", (test_agent.weights[1]["b"]), "\n")

agent_weight_answer = np.load("asserts/agent_init_weights_1.npz")
assert(np.allclose(test_agent.weights[0]["W"], agent_weight_answer["W0"]))
assert(np.allclose(test_agent.weights[0]["b"], agent_weight_answer["b0"]))
assert(np.allclose(test_agent.weights[1]["W"], agent_weight_answer["W1"]))
assert(np.allclose(test_agent.weights[1]["b"], agent_weight_answer["b1"]))

print("Passed the asserts!")

```

    layer_size: [5, 2, 1]
    weights[0]["W"] shape: (5, 2)
    weights[0]["b"] shape: (1, 2)
    weights[1]["W"] shape: (2, 1)
    weights[1]["b"] shape: (1, 1)

    weights[0]["W"]
     [[ 1.11568467  0.25308164]
     [ 0.61900825  1.4172653 ]
     [ 1.18114738 -0.6180848 ]
     [ 0.60088868 -0.0957267 ]
     [-0.06528133  0.25968529]]

    weights[0]["b"]
     [[0.09110115 0.91976332]]

    weights[1]["W"]
     [[0.76103773]
     [0.12167502]]

    weights[1]["b"]
     [[0.44386323]]

    Passed the asserts!

**Expected output**:

    layer_size: [5 2 1]
    weights[0]["W"] shape: (5, 2)
    weights[0]["b"] shape: (1, 2)
    weights[1]["W"] shape: (2, 1)
    weights[1]["b"] shape: (1, 1)

    weights[0]["W"]
     [[ 1.11568467  0.25308164]
     [ 0.61900825  1.4172653 ]
     [ 1.18114738 -0.6180848 ]
     [ 0.60088868 -0.0957267 ]
     [-0.06528133  0.25968529]]

    weights[0]["b"]
     [[0.09110115 0.91976332]]

    weights[1]["W"]
     [[0.76103773]
     [0.12167502]]

    weights[1]["b"]
     [[0.44386323]]

Run the following code to test your implementation of the `agent_start()` function:

```python
# Do not modify this cell!

## Test Code for agent_start() ##

agent_info = {"num_states": 500,
              "num_hidden_layer": 1,
              "num_hidden_units": 100,
              "step_size": 0.1,
              "discount_factor": 1.0,
              "beta_m": 0.9,
              "beta_v": 0.99,
              "epsilon": 0.0001,
              "seed": 10
             }

# Suppose state = 250
state = 250

test_agent = TDAgent()
test_agent.agent_init(agent_info)
test_agent.agent_start(state)

print("Agent state: {}".format(test_agent.last_state))
print("Agent selected action: {}".format(test_agent.last_action))

assert(test_agent.last_state == 250)
assert(test_agent.last_action == 1)

print("Passed the asserts!")

```

    Agent state: 250
    Agent selected action: 1
    Passed the asserts!

**Expected output**:

    Agent state: 250
    Agent selected action: 1

Run the following code to test your implementation of the `agent_step()` function:

```python
# Do not modify this cell!

## Test Code for agent_step() ##

agent_info = {"num_states": 5,
              "num_hidden_layer": 1,
              "num_hidden_units": 2,
              "step_size": 0.1,
              "discount_factor": 1.0,
              "beta_m": 0.9,
              "beta_v": 0.99,
              "epsilon": 0.0001,
              "seed": 0
             }

test_agent = TDAgent()
test_agent.agent_init(agent_info)

# load initial weights
agent_initial_weight = np.load("asserts/agent_step_initial_weights.npz")
test_agent.weights[0]["W"] = agent_initial_weight["W0"]
test_agent.weights[0]["b"] = agent_initial_weight["b0"]
test_agent.weights[1]["W"] = agent_initial_weight["W1"]
test_agent.weights[1]["b"] = agent_initial_weight["b1"]

# load m and v for the optimizer
m_data = np.load("asserts/agent_step_initial_m.npz")
test_agent.optimizer.m[0]["W"] = m_data["W0"]
test_agent.optimizer.m[0]["b"] = m_data["b0"]
test_agent.optimizer.m[1]["W"] = m_data["W1"]
test_agent.optimizer.m[1]["b"] = m_data["b1"]

v_data = np.load("asserts/agent_step_initial_v.npz")
test_agent.optimizer.v[0]["W"] = v_data["W0"]
test_agent.optimizer.v[0]["b"] = v_data["b0"]
test_agent.optimizer.v[1]["W"] = v_data["W1"]
test_agent.optimizer.v[1]["b"] = v_data["b1"]

# Assume the agent started at State 3
start_state = 3
test_agent.agent_start(start_state)

# Assume the reward was 10.0 and the next state observed was State 1
reward = 10.0
next_state = 1
test_agent.agent_step(reward, next_state)

# updated weights asserts
print("updated_weights[0][\"W\"]\n", test_agent.weights[0]["W"], "\n")
print("updated_weights[0][\"b\"]\n", test_agent.weights[0]["b"], "\n")
print("updated_weights[1][\"W\"]\n", test_agent.weights[1]["W"], "\n")
print("updated_weights[1][\"b\"]\n", test_agent.weights[1]["b"], "\n")

agent_updated_weight_answer = np.load("asserts/agent_step_updated_weights.npz")
assert(np.allclose(test_agent.weights[0]["W"], agent_updated_weight_answer["W0"]))
assert(np.allclose(test_agent.weights[0]["b"], agent_updated_weight_answer["b0"]))
assert(np.allclose(test_agent.weights[1]["W"], agent_updated_weight_answer["W1"]))
assert(np.allclose(test_agent.weights[1]["b"], agent_updated_weight_answer["b1"]))

# last_state and last_action assert
print("Agent last state:", test_agent.last_state)
print("Agent last action:", test_agent.last_action, "\n")

assert(test_agent.last_state == 1)
assert(test_agent.last_action == 1)

print ("Passed the asserts!")

```

    updated_weights[0]["W"]
     [[ 1.10893459  0.30763738]
     [ 0.63690565  1.14778865]
     [ 1.23397791 -0.48152743]
     [ 0.72792093 -0.15829832]
     [ 0.15021996  0.39822163]]

    updated_weights[0]["b"]
     [[0.29798822 0.96254535]]

    updated_weights[1]["W"]
     [[0.76628754]
     [0.11486511]]

    updated_weights[1]["b"]
     [[0.58530057]]

    Agent last state: 1
    Agent last action: 1

    Passed the asserts!

**Expected output**:

    updated_weights[0]["W"]
     [[ 1.10893459  0.30763738]
     [ 0.63690565  1.14778865]
     [ 1.23397791 -0.48152743]
     [ 0.72792093 -0.15829832]
     [ 0.15021996  0.39822163]]

    updated_weights[0]["b"]
     [[0.29798822 0.96254535]]

    updated_weights[1]["W"]
     [[0.76628754]
     [0.11486511]]

    updated_weights[1]["b"]
     [[0.58530057]]

    Agent last state: 1
    Agent last action: 1

Run the following code to test your implementation of the `agent_end()` function:

```python
# Do not modify this cell!

## Test Code for agent_end() ##

agent_info = {"num_states": 5,
              "num_hidden_layer": 1,
              "num_hidden_units": 2,
              "step_size": 0.1,
              "discount_factor": 1.0,
              "beta_m": 0.9,
              "beta_v": 0.99,
              "epsilon": 0.0001,
              "seed": 0
             }

test_agent = TDAgent()
test_agent.agent_init(agent_info)

# load initial weights
agent_initial_weight = np.load("asserts/agent_end_initial_weights.npz")
test_agent.weights[0]["W"] = agent_initial_weight["W0"]
test_agent.weights[0]["b"] = agent_initial_weight["b0"]
test_agent.weights[1]["W"] = agent_initial_weight["W1"]
test_agent.weights[1]["b"] = agent_initial_weight["b1"]

# load m and v for the optimizer
m_data = np.load("asserts/agent_step_initial_m.npz")
test_agent.optimizer.m[0]["W"] = m_data["W0"]
test_agent.optimizer.m[0]["b"] = m_data["b0"]
test_agent.optimizer.m[1]["W"] = m_data["W1"]
test_agent.optimizer.m[1]["b"] = m_data["b1"]

v_data = np.load("asserts/agent_step_initial_v.npz")
test_agent.optimizer.v[0]["W"] = v_data["W0"]
test_agent.optimizer.v[0]["b"] = v_data["b0"]
test_agent.optimizer.v[1]["W"] = v_data["W1"]
test_agent.optimizer.v[1]["b"] = v_data["b1"]

# Assume the agent started at State 4
start_state = 4
test_agent.agent_start(start_state)

# Assume the reward was 10.0 and reached the terminal state
reward = 10.0
test_agent.agent_end(reward)

# updated weights asserts
print("updated_weights[0][\"W\"]\n", test_agent.weights[0]["W"], "\n")
print("updated_weights[0][\"b\"]\n", test_agent.weights[0]["b"], "\n")
print("updated_weights[1][\"W\"]\n", test_agent.weights[1]["W"], "\n")
print("updated_weights[1][\"b\"]\n", test_agent.weights[1]["b"], "\n")

agent_updated_weight_answer = np.load("asserts/agent_end_updated_weights.npz")
assert(np.allclose(test_agent.weights[0]["W"], agent_updated_weight_answer["W0"]))
assert(np.allclose(test_agent.weights[0]["b"], agent_updated_weight_answer["b0"]))
assert(np.allclose(test_agent.weights[1]["W"], agent_updated_weight_answer["W1"]))
assert(np.allclose(test_agent.weights[1]["b"], agent_updated_weight_answer["b1"]))

print ("Passed the asserts!")

```

    updated_weights[0]["W"]
     [[ 1.10893459  0.30763738]
     [ 0.63690565  1.14778865]
     [ 1.17531054 -0.51043162]
     [ 0.75062903 -0.13736817]
     [ 0.15021996  0.39822163]]

    updated_weights[0]["b"]
     [[0.30846523 0.95937346]]

    updated_weights[1]["W"]
     [[0.68861703]
     [0.15986364]]

    updated_weights[1]["b"]
     [[0.586074]]

    Passed the asserts!

**Expected output:**

    updated_weights[0]["W"]
     [[ 1.10893459  0.30763738]
     [ 0.63690565  1.14778865]
     [ 1.17531054 -0.51043162]
     [ 0.75062903 -0.13736817]
     [ 0.15021996  0.39822163]]

    updated_weights[0]["b"]
     [[0.30846523 0.95937346]]

    updated_weights[1]["W"]
     [[0.68861703]
     [0.15986364]]

    updated_weights[1]["b"]
     [[0.586074]]

## Section 2 - Run Experiment

Now that you implemented the agent, we can run the experiment. Similar to Course 3 Programming Assignment 1, we will plot the learned state value function and the learning curve of the TD agent. To plot the learning curve, we use Root Mean Squared Value Error (RMSVE).

## 2-1: Run Experiment for Semi-gradient TD with a Neural Network

We have already provided you the experiment/plot code, so you can go ahead and run the two cells below.

Note that running the cell below will take **approximately 12 minutes**.

```python
# Do not modify this cell!

true_state_val = np.load('data/true_V.npy')
state_distribution = np.load('data/state_distribution.npy')

def calc_RMSVE(learned_state_val):
    assert(len(true_state_val) == len(learned_state_val) == len(state_distribution))
    MSVE = np.sum(np.multiply(state_distribution, np.square(true_state_val - learned_state_val)))
    RMSVE = np.sqrt(MSVE)
    return RMSVE

# Define function to run experiment
def run_experiment(environment, agent, environment_parameters, agent_parameters, experiment_parameters):

    rl_glue = RLGlue(environment, agent)

    # save rmsve at the end of each episode
    agent_rmsve = np.zeros((experiment_parameters["num_runs"],
                            int(experiment_parameters["num_episodes"]/experiment_parameters["episode_eval_frequency"]) + 1))

    # save learned state value at the end of each run
    agent_state_val = np.zeros((experiment_parameters["num_runs"],
                                environment_parameters["num_states"]))

    env_info = {"num_states": environment_parameters["num_states"],
                "start_state": environment_parameters["start_state"],
                "left_terminal_state": environment_parameters["left_terminal_state"],
                "right_terminal_state": environment_parameters["right_terminal_state"]}

    agent_info = {"num_states": environment_parameters["num_states"],
                  "num_hidden_layer": agent_parameters["num_hidden_layer"],
                  "num_hidden_units": agent_parameters["num_hidden_units"],
                  "step_size": agent_parameters["step_size"],
                  "discount_factor": environment_parameters["discount_factor"],
                  "beta_m": agent_parameters["beta_m"],
                  "beta_v": agent_parameters["beta_v"],
                  "epsilon": agent_parameters["epsilon"]
                 }

    print('Setting - Neural Network with 100 hidden units')
    os.system('sleep 1')

    # one agent setting
    for run in tqdm(range(1, experiment_parameters["num_runs"]+1)):
        env_info["seed"] = run
        agent_info["seed"] = run
        rl_glue.rl_init(agent_info, env_info)

        # Compute initial RMSVE before training
        current_V = rl_glue.rl_agent_message("get state value")
        agent_rmsve[run-1, 0] = calc_RMSVE(current_V)

        for episode in range(1, experiment_parameters["num_episodes"]+1):
            # run episode
            rl_glue.rl_episode(0) # no step limit

            if episode % experiment_parameters["episode_eval_frequency"] == 0:
                current_V = rl_glue.rl_agent_message("get state value")
                agent_rmsve[run-1, int(episode/experiment_parameters["episode_eval_frequency"])] = calc_RMSVE(current_V)
            elif episode == experiment_parameters["num_episodes"]: # if last episode
                current_V = rl_glue.rl_agent_message("get state value")

        agent_state_val[run-1, :] = current_V

    save_name = "{}".format(rl_glue.agent.name).replace('.','')

    if not os.path.exists('results'):
                os.makedirs('results')

    # save avg. state value
    np.save("results/V_{}".format(save_name), agent_state_val)

    # save avg. rmsve
    np.savez("results/RMSVE_{}".format(save_name), rmsve = agent_rmsve,
                                                   eval_freq = experiment_parameters["episode_eval_frequency"],
                                                   num_episodes = experiment_parameters["num_episodes"])

# Run Experiment

# Experiment parameters
experiment_parameters = {
    "num_runs" : 20,
    "num_episodes" : 1000,
    "episode_eval_frequency" : 10 # evaluate every 10 episode
}

# Environment parameters
environment_parameters = {
    "num_states" : 500,
    "start_state" : 250,
    "left_terminal_state" : 0,
    "right_terminal_state" : 501,
    "discount_factor" : 1.0
}

# Agent parameters
agent_parameters = {
    "num_hidden_layer": 1,
    "num_hidden_units": 100,
    "step_size": 0.001,
    "beta_m": 0.9,
    "beta_v": 0.999,
    "epsilon": 0.0001,
}

current_env = RandomWalkEnvironment
current_agent = TDAgent

# run experiment
run_experiment(current_env, current_agent, environment_parameters, agent_parameters, experiment_parameters)

# plot result
plot_script.plot_result(["td_agent"])

shutil.make_archive('results', 'zip', 'results')
```

You plotted the learning curve for 1000 episodes. As you can see the RMSVE is still decreasing. Here we provide the pre-computed result for 5000 episodes and 20 runs so that you can see the performance of semi-gradient TD with a neural network after being trained for a long time.

Does semi-gradient TD with a neural network find a good approximation within 5000 episodes?

As you may remember from the previous assignment, semi-gradient TD with 10-state aggregation converged within 100 episodes. Why is TD with a neural network slower?

Would it be faster if we decrease the number of hidden units? Or what about if we increase the number of hidden units?

## 2-2: Compare Performance of Semi-gradient TD with a Neural Network and Semi-gradient TD with Tile-coding

In this section, we compare the performance of semi-gradient TD with a Neural Network and semi-gradient TD with tile-coding. Tile-coding is a kind of coarse coding that uses multiple overlapping partitions of the state space to produce features. For tile-coding, we used 50 tilings each with 6 tiles. We set the step-size for semi-gradient TD with tile-coding to $\frac{0.1}{\text{# tilings}}$. See the figure below for the comparison between semi-gradient TD with tile-coding and semi-gradient TD with a neural network and Adam algorithm. This result is for 5000 episodes and 20 runs:

How are the results?

Semi-gradient TD with tile-coding is much faster than semi-gradient TD with a neural network. Why?

Which method has a lower RMSVE at the end of 5000 episodes?

### Wrapping up!

You have successfully implemented Course 3 Programming Assignment 2.

You have implemented **semi-gradient TD with a Neural Network and Adam algorithm** in 500-state Random Walk.

You also compared semi-gradient TD with a neural network and semi-gradient TD with tile-coding.

From the experiments and lectures, you should be more familiar with some of the strengths and weaknesses of using neural networks as the function approximator for an RL agent. On one hand, neural networks are powerful function approximators capable of representing a wide class of functions. They are also capable of producing features without exclusively relying on hand-crafted mechanisms. On the other hand, compared to a linear function approximator with tile-coding, neural networks can be less sample efficient. When implementing your own Reinforcement Learning agents, you may consider these strengths and weaknesses to choose the proper function approximator for your problems.

---

**Note**: Apart from using the 'Submit' button in the notebook, you have to submit an additional `zip` file containing the 'npy' files that were generated from running the experiment cells. To do so:

1. Generate the zip file by running the experiment cells in the notebook. On the top of the notebook, navigate to 'File->Open' to open the directory view of this assignment. Select "results.zip" and click on "Download". Alternatively, you can download just the results folder and run "zip -jr results.zip results/" (The flag 'j' is required by the grader!).
2. Go to the "My submission" tab and click on "+ Create submission".
3. Click on "C3M2 Data-file Grader" and upload your `results.zip`.

**These account for 40% of the marks, so don't forget to do so!**

```python

```

Prediction and Control with Function Approximation/Week 2/Notebook_ Semi-gradient TD with a Neural Network/C3M2_Assignment2-v7.ipynb


# Taylor problem 2.20 Template

last revised: 08-Jan-2019 by Dick Furnstahl [furnstahl.1@osu.edu]

**This is a template for solving problem 2.20.  Go through and fill in the blanks where ### appears.**

The goal of this problem is to plot and comment on the trajectory of a projectile subject to linear air resistance, considering four different values of the drag coefficient.

The problem statement fixes the initial angle above the horizontal and suggests using convenient values for the initial speed (magnitude of the velocity) and gravitational strength $g$.  We'll set up the problem more generally and look at special cases.

The equations are derived in the book:

$$\begin{align}
  x(t) &= v_{x0}\tau (1 - e^{-t/\tau}) \\
  y(t) &= (v_{y0} + v_{\textrm{ter}}) \tau (1 - e^{-t/\tau}) - v_{\textrm{ter}} t
\end{align}$$

where $v_{\textrm{ter}} = g\tau$.

Plan:
1. Define functions for $x$ and $y$, which will depend on $t$, $\tau$, $g$, and the initial velocity.  Make the functions look like the equations from Taylor to reduce the possibility of error.
2. Set up an array of the time $t$.
3. Determine $x$ and $y$ arrays for different values of $\tau$.
4. Make a plot of $y$ versus $x$ for each value of $\tau$, all on the same plot.
5. Save the plot for printing.

```python
### What modules do we need to import? (Can always add more later!)
import numpy as np
```

### 1. Define functions for $x$ and $y$

```python
def x_traj(t, tau, v_x0=1., g=1.):
    """Horizontal position x(t) from equation (2.36) in Taylor.
       The initial position at t=0 is x=y=0.
    """
    return v_x0 * tau * (1. - np.exp(-t/tau))

def y_traj(t, tau, v_y0=1., g=1.):
    """Vertical position y(t) from equation (2.36) in Taylor.
       The initial position at t=0 is x=y=0.
    """
    v_ter = g * tau
    return (v_y0 + v_ter) * tau * (1. - np.exp(-t/tau)) - v_ter*t
```

### 2. Set up an array of the time $t$

```python
t_min = 0.
t_max = 3.
delta_t = 0.00001      ### pick a reasonable delta_t
t_pts = np.arange(-1, 101)  ### fill in the blanks
t_pts  # check that we did what we thought!
```

    array([ -1,   0,   1,   2,   3,   4,   5,   6,   7,   8,   9,  10,  11,
            12,  13,  14,  15,  16,  17,  18,  19,  20,  21,  22,  23,  24,
            25,  26,  27,  28,  29,  30,  31,  32,  33,  34,  35,  36,  37,
            38,  39,  40,  41,  42,  43,  44,  45,  46,  47,  48,  49,  50,
            51,  52,  53,  54,  55,  56,  57,  58,  59,  60,  61,  62,  63,
            64,  65,  66,  67,  68,  69,  70,  71,  72,  73,  74,  75,  76,
            77,  78,  79,  80,  81,  82,  83,  84,  85,  86,  87,  88,  89,
            90,  91,  92,  93,  94,  95,  96,  97,  98,  99, 100])

### 3., 4. Make $x$ and $y$ arrays for different $\tau$ and plot

```python
%matplotlib inline
### What module(s) should you import?
import matplotlib.pyplot as plt
```

```python
# generate random integer values
from random import seed
from random import randint
# seed random number generator
seed(1)
# generate some integers
def x_pts_fnc(x):
    for _ in range(x):
        x_pts = randint(0, 10)
```

```python
plt.rcParams.update({'font.size': 16})  # This is to boost the font size

fig = plt.figure(figsize=(10,10))
ax = fig.add_subplot(1, 1, 1)   ### How do you create a single subplot?

tau_1 = 0.3
ax.plot(x_traj(t_pts, tau_1), y_traj(t_pts, tau_1), 'b-',
        label=r'$\tau = 0.3$')

tau_2 = 1.0
ax.plot(x_traj(t_pts, tau_2), y_traj(t_pts, tau_2), 'r:',
        label=r'$\tau = 1.0$')

tau_3 = 3.0
ax.plot(x_traj(t_pts, tau_3), y_traj(t_pts, tau_3),  'g--',
        label=r'$\tau = 3.0$')
### plot a line with tau_3 and line type 'g--' with a label

tau_4 = 100.
ax.plot(x_traj(t_pts, tau_4), y_traj(t_pts, tau_4),  'k-',
        label=r'$\tau = 100.0$')
### plot a line with tau_4 and line type 'k- with a label

ax.set_ylim(-0.1, 1.5)
ax.set_xlim(0., 3)
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_aspect(1)  # so that the x and y spacing is the same

ax.legend();

```

### 5. Save the plot for printing

```python
# save the figure
fig.savefig('Taylor_prob_2.20.png', bbox_inches='tight')
### Find the figure file and display it in your browser, then save or print.
### Check you graph against the one from the next section.
```

## More advanced python: plot again with a loop

Now do it as a loop, cycling through properties, and add a vertical line at the asymptotic distance.

```python
import matplotlib.pyplot as plt

plt.rcParams.update({'font.size': 18})

from cycler import cycler
my_cycler = (cycler(color=['k', 'g', 'b', 'r']) +
             cycler(linestyle=['-', '--', ':', '-.']))

fig = plt.figure(figsize=(10,10))
ax = fig.add_subplot(1,1,1)
ax.set_prop_cycle(my_cycler)

v_x0 = 1.
tau_list = [10000, 3.0, 1.0, 0.3, 0.1]
for tau in tau_list:
    ax.plot(x_traj(t_pts, tau), y_traj(t_pts, tau),
            label=rf'$\tau = {tau:.1f}$')
    ax.axvline(v_x0 * tau, color='black', linestyle='dotted')

ax.set_ylim(-0.1, 1.5)
ax.set_xlim(0., 3)
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_aspect(1)  # so that the x and y spacing is the same

ax.legend();

```

**If it is new to you, look up how a for loop in Python works and try to figure out what is happening here.  Ask if you are confused!**

```python

```


 # 5 2-D Anisotropic Diffusion, non coercive case

 In this tutorial, we consider the study and analysis of the eigenvalue problem associated to the following anisotropic diffusion operator
$$
    - \nabla \cdot \mathbf{b} \otimes \mathbf{b} \nabla \phi = \omega^2 \phi
$$
where $\mathbf{b} = (\iota, 1)$ is the normalized vector of the magnetic field.

In fact, the original problem is 3D and the iota profile is given as

$$
    \iota(r) = 0.47262 + 0.32392 r + 0.49604 r^2 + 5.3991 . 10^{-6} r^3 - 3.165 . 10^{-5} r^4 + 4.7963 . 10^{-5} r^5 - 2.2824 . 10^{-5} r^6
$$

which is the LHD-$\iota$-profile.

## Constants coefficients case

Now taking the given radius $r = 0.9$ we get

$$
\mathbf{b} = (b_1,b_2)^\top = (1.1659397624413860850012270020670, 1. )^\top
$$

```python
from numpy import linspace, meshgrid, zeros, pi

import matplotlib.pyplot as plt
%matplotlib inline

from IPython.display import Math

from sympy.core.containers import Tuple
from sympy import symbols
from sympy import Symbol
from sympy import Lambda
from sympy import Function
from sympy import IndexedBase

from gelato.glt import glt_symbol
from gelato.glt import glt_lambdify
from gelato.glt import glt_approximate_eigenvalues
from gelato.printing.latex import glt_latex
from gelato.calculus   import (Dot, Cross, Grad, Curl, Rot, Div, dx, dy)
from gelato.calculus   import Constant
from gelato.fem.assembly import assemble_matrix
from gelato.fem.utils    import compile_kernel
from gelato.fem.utils    import compile_symbol

from spl.fem.splines import SplineSpace
from spl.fem.tensor  import TensorFemSpace
```

```python
x,y = symbols('x y')

u = Symbol('u')
v = Symbol('v')

b1 = Constant('b1')
b2 = Constant('b2')
b = Tuple(b1, b2)

a = Lambda((x,y,v,u), Dot(b, Grad(v)) * Dot(b, Grad(u)))

```

```python
# ...  create a finite element space
p1  = 3 ; p2  = 3
ne1 = 16 ; ne2 = 16

grid_1 = linspace(0., 1., ne1+1)
grid_2 = linspace(0., 1., ne2+1)

V1 = SplineSpace(p1, grid=grid_1)
V2 = SplineSpace(p2, grid=grid_2)

V = TensorFemSpace(V1, V2)
# ...
```

```python
a_m = Lambda((x,y,v,u), v * u)
kernel_m_f90 = compile_kernel('kernel_m', a_m, V,
                            backend='fortran')
```

```python
Mass = assemble_matrix(V, kernel_m_f90).tocsr()
```

```python
symbol_m = compile_symbol('symbol_m', a_m, V,
                          backend='fortran')
```

```python
d_constants = {'b2': 1.}
d_types = {'b1': 'double'}
```

```python
kernel_a_f90 = compile_kernel('kernel_1', a, V,
                            d_constants=d_constants,
                            d_args=d_types,
                            backend='fortran')
```

```python
d_args = {'b1': 1.165939762441386085001227002067}
```

```python
M = assemble_matrix(V, kernel_a_f90, args=d_args).tocsr()
```

```python
# import eigenvalue solver
from scipy.linalg import eig, inv

M = M.todense()
Mass = Mass.todense()
M = M * inv(Mass)
w, v = eig(M)
wr = w.real
wr.sort()
```

```python
symbol = glt_symbol(a, space=V, evaluate=False)

```

```python
symbol_a = compile_symbol('symbol_a', a, V,
                          d_constants=d_constants,
                          d_args=d_types,
                          backend='fortran')
```

```python
# ... example of symbol evaluation
[n1,n2] = [W.nbasis for W in V.spaces]

t1 = linspace(-pi,pi, n1)
t2 = linspace(-pi,pi, n2)
x1 = linspace(0.,1., n1)
x2 = linspace(0.,1., n2)
e_a = zeros((n1, n2), order='F')
e_m = zeros((n1, n2), order='F')
symbol_a(x1,x2,t1,t2, e_a, d_args['b1'])
symbol_m(x1,x2,t1,t2, e_m)
e = e_a/e_m
t = e.ravel()
t.sort()
# ...
```

```python
plt.plot(t, "+b", label="glt symbol")
plt.plot(wr, "xr", label="eigenvalues")
plt.legend(loc=2);
```

```python
# ... compute the symbol f_a / f_m
def plot_symbol(nx, ny):

    t1 = linspace(-pi,pi, nx)
    t2 = linspace(-pi,pi, ny)
    x1 = linspace(0.,1., nx)
    x2 = linspace(0.,1., ny)
    e_a = zeros((nx, ny), order='F')
    e_m = zeros((nx, ny), order='F')
    symbol_a(x1,x2,t1,t2, e_a, d_args['b1'])
    symbol_m(x1,x2,t1,t2, e_m)
    e = e_a/e_m
    e = e / (4 * pi**2)

    T1,T2 = meshgrid(t1,t2)
    plt.contourf(T1, T2, e)
    plt.axis("equal")
# ...
```

```python
# ... plot
plot_symbol(100, 100)
plt.colorbar();
# ...
```

```python
_b1 = d_args['b1']
_b2 = d_constants['b2']

n = range(-ne1//2, ne1//2+1)
m = range(-ne2//2, ne2//2+1)
n,m = meshgrid(n,m)
n = n.transpose()
m = m.transpose()
f = (_b1*n + _b2*m)**2
```

```python
nx = ne1+1 ; ny = ne2+1
t1 = linspace(-pi,pi, nx)
t2 = linspace(-pi,pi, ny)

x1 = linspace(0.,1., nx)
x2 = linspace(0.,1., ny)
e_a = zeros((nx, ny), order='F')
e_m = zeros((nx, ny), order='F')
symbol_a(x1,x2,t1,t2, e_a, d_args['b1'])
symbol_m(x1,x2,t1,t2, e_m)
e = e_a/e_m
e = e / (4 * pi**2)

plt.contourf(t1, t2, abs(e-f))
plt.axis("equal")
plt.colorbar();
```

```python
def mode(i,j):
    t1 = i * 2 * pi / ne1
    t2 = j * 2 * pi / ne2

    x = 0.5 + i * 2. / ne1
    y = 0.5 + j * 2. / ne2

    nx = 1 ; ny = 1
    x1 = [x] ; x2 = [y]
    t1 = [t1] ; t2 = [t2]
    e_a = zeros((nx, ny), order='F')
    e_m = zeros((nx, ny), order='F')
    symbol_a(x1,x2,t1,t2, e_a, d_args['b1'])
    symbol_m(x1,x2,t1,t2, e_m)
    e = e_a/e_m
    e = e / (4 * pi**2)

    return e
```

```python
table = []
for i in range(-10,10+1):
    for j in range(-10,10+1):
        v = mode(i,j)
        w = (_b1*i + _b2*j)**2
        if v < 0.2:
            line = []
            line.append("(" + str(i) + "," + str(j) + ")")
            # approximated eigenvalue
            line.append(str(v))
            # exact eigenvalue
            line.append(str(w))
            # absolute error
            line.append(str(abs(v-w)))
            # relative error
            if abs(w) > 1.e-10:
                line.append(str(abs(v-w)/w))
            else:
                line.append("")
            table.append(line)
```

```python
from tabulate import tabulate

headers = ["mode", "approximated", "exact", "abs-err", "rel-err"]
print(tabulate(table, headers, tablefmt="pipe"))
```

    | mode   | approximated      |    exact | abs-err            | rel-err            |
    |:-------|:------------------|---------:|:-------------------|:-------------------|
    | (-2,2) | [[0.11026398]]    | 0.110144 | [[0.00011996]]     | [[0.00108913]]     |
    | (-1,1) | [[0.02753633]]    | 0.027536 | [[3.27085629e-07]] | [[1.18784708e-05]] |
    | (0,0)  | [[3.1052801e-14]] | 0        | [[3.1052801e-14]]  |                    |
    | (1,-1) | [[0.02753633]]    | 0.027536 | [[3.27085629e-07]] | [[1.18784708e-05]] |
    | (2,-2) | [[0.11026398]]    | 0.110144 | [[0.00011996]]     | [[0.00108913]]     |

## Variable coefficients case

We consider the following variable coefficients case for the magnetic field

$$
 \mathbf{b}(\theta, \varphi) =
 \begin{pmatrix}
 \iota (1 + 0.1 \sin(2 \pi \varphi))
 \\
 (1 + 0.1 \sin(2 \pi \varphi))
 \end{pmatrix}
$$

```python
# ... define the weak formulation
x,y = symbols('x y')

u = Symbol('u')
v = Symbol('v')

b0 = Function('b0')
b1 = Function('b1')

a = Lambda((x,y,v,u),
           (b0(x,y)*dx(v) + b1(x,y)*dy(v)) * (b0(x,y)*dx(u) + b1(x,y)*dy(u)))
# ...

# ... user defined function
def b0(x,y):
    from numpy import sin
    from scipy import pi

    r = 1.1659397624413860850012270020670 * (1.0 + 0.1 * sin(2*pi*y))
    return r

def b1(x,y):
    from numpy import sin
    from scipy import pi

    r = 1.0 * (1.0 + 0.1 * sin(2*pi*y))
    return r
# ...

# ... create an interactive pyccel context
from pyccel.epyccel import ContextPyccel

context = ContextPyccel(name='context_2')
context.insert_function(b0, ['double', 'double'], kind='function', results=['double'])
context.insert_function(b1, ['double', 'double'], kind='function', results=['double'])

context.compile()
# ...

# ...
kernel_f90 = compile_kernel('kernel_2', a, V,
                            context=context,
                            verbose=True,
                            backend='fortran')
# ...

# ...
M_f90 = assemble_matrix(V, kernel_f90)
# ...

```

    pyccel: [semantic]
     |warning| Duplicated import  (sin)
     |warning| Duplicated import  (pi)

    pyccel: [codegen]
     |warning| Duplicated import  (sin)
     |warning| Duplicated import  (pi)

    > input     := Lambda((x, y, v, u), (b0(x, y)*dx(u) + b1(x, y)*dy(u))*(b0(x, y)*dx(v) + b1(x, y)*dy(v)))
    > weak form := (Ni_x*b0(x, y) + Ni_y*b1(x, y))*(Nj_x*b0(x, y) + Nj_y*b1(x, y))
    pyccel: [semantic]
     |warning| Duplicated import  (sin)
     |warning| Duplicated import  (pi)

    pyccel: [semantic]
     |warning| Duplicated import  (sin)
     |warning| Duplicated import  (pi)

    pyccel: [codegen]
     |warning| Duplicated import  (sin)
     |warning| Duplicated import  (pi)

```python
symbol = glt_symbol(a, space=V, evaluate=False)
```

```python
Math(glt_latex(symbol))
```

$$- 2 \mathfrak{a}_p(\theta_1) \mathfrak{a}_p(\theta_2) \operatorname{b_{0}}{\left (x,y \right )} \operatorname{b_{1}}{\left (x,y \right )} + \mathfrak{m}_p(\theta_1) \mathfrak{s}_p(\theta_2) \operatorname{b_{1}}^{2}{\left (x,y \right )} + \mathfrak{m}_p(\theta_2) \mathfrak{s}_p(\theta_1) \operatorname{b_{0}}^{2}{\left (x,y \right )}$$

```python
symbol_a2 = compile_symbol('symbol_a2', a, V,
                            context=context,
                            backend='fortran')
```

    pyccel: [semantic]
     |warning| Duplicated import  (sin)
     |warning| Duplicated import  (pi)

    pyccel: [semantic]
     |warning| Duplicated import  (sin)
     |warning| Duplicated import  (pi)

    pyccel: [codegen]
     |warning| Duplicated import  (sin)
     |warning| Duplicated import  (pi)

```python
nx = 100 ; ny = 100
t1 = linspace(-pi,pi, nx)
t2 = linspace(-pi,pi, ny)

x1 = linspace(0.,1., nx)
x2 = linspace(0.,1., ny)
e_a = zeros((nx, ny), order='F')
e_m = zeros((nx, ny), order='F')
symbol_a2(x1,x2,t1,t2, e_a)
symbol_m(x1,x2,t1,t2, e_m)
e = e_a/e_m
e = e / (4 * pi**2)

T1,T2 = meshgrid(t1,t2)
plt.contourf(T1, T2, e)
plt.axis("equal")
plt.colorbar();
```

```python
from IPython.core.display import HTML
def css_styling():
    styles = open("../../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>
<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
    }
    div.cell{
        width:600px;
        margin-left:16% !important;
        margin-right:auto;
    }
    h1 {
        font-family: 'Alegreya Sans', sans-serif;
    }
    h2 {
        font-family: 'Fenix', serif;
    }
    h3{
		font-family: 'Fenix', serif;
        margin-top:12px;
        margin-bottom: 3px;
       }
	h4{
		font-family: 'Fenix', serif;
       }
    h5 {
        font-family: 'Alegreya Sans', sans-serif;
    }
    div.text_cell_render{
        font-family: 'Alegreya Sans',Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 135%;
        font-size: 120%;
        width:600px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro";
			font-size: 90%;
    }
/*    .prompt{
        display: None;
    }*/
    .text_cell_render h1 {
        font-weight: 200;
        font-size: 50pt;
		line-height: 100%;
        color:#054BCD;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
        display: block;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 16pt;
        color: #054BCD;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

```python

```


# UCL AI Society Machine Learning Tutorials
### Session 02. Introduction to Machine Learning

## Contents
1. Train/test set split
2. Various Cross Validation teqniques
3. Regressions
4. Classifications
5. All in one challenge: Boston house

## Aim
At the end of this session, you will be able to:
- Understand how to prepare your dataset.
- Use the most popular library in Machine Learning
- Implement simple regression and classification models.
- Practice a whole machine learning project individually

## 0. Introduction to Classification
From the last section, you've learned what Regression is. Classification is another important branch in Machine Learning.
So what is classification?
It is an area of supervised learning that tries to predict which class or category some entity belongs to, based on its features.

Classification can have four types of result:
1. True Negative
2. Treu Positive
3. False Negative
4. False Positive
We will dive more into this, later in this tutorial series.

In this notebook we will discover when we should use classification rather than regression, and types of classification algorithms.

Type of Classifiers:
- Logistic Regression (Its name is regression, but it's much more close to classification than regression)
- Softmax
- Naive Bayes
- SVM
- KNN
- Decision Tree

This notebook will cover logistic regression and Decision tree. Others can possibly dealt in the next session.

# 1. Logistic Regression

Useful videos:
1. [Andrew Ng's explanation 1](https://www.youtube.com/watch?v=-la3q9d7AKQ)
2. [Andrew Ng's explanation 2](https://www.youtube.com/watch?v=t1IT5hZfS48)
3. [Andrew Ng's explanation 3](https://www.youtube.com/watch?v=F_VG4LNjZZw)
4. [Andrew Ng's explanation 4](https://www.youtube.com/watch?v=HIQlmHxI6-0)

Logistic regression is a well-motivated approach to discriminative classification which leads to a smooth, convex, optimisation problem.

Logistic regression is also a basis of Neural Network. Logistic Regression is sometimes called, a single node of Artificial Neuron. We will get back to what this means afterwards when we are doing Deep Learning.

```python
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from sklearn import linear_model
from scipy.special import expit
```

## 1-1. In which case do we use classification?

**Let's generate a toy dataset that is suitable for classification**

```python
np.random.seed(0)
n_samples= 100

X = np.random.normal(size=n_samples)
y = (X > 0).astype(np.float)

X[X > 0] *= 5
X += .7 * np.random.normal(size=n_samples)
X = X[:, np.newaxis]
```

```python
X.shape, y.shape
```

```python
plt.scatter(X, y)
```

What if our data looks like the above? Would you still use your linear regression model?
Probably not. When your data has classes and your task is to classify the data, you normally use classification method, and Logistic Regression is a good start in learning classification.
Please do watch the Andrew Ng's video on Logistic Regression to fully understand mathematically.

I've written a function called `compare_logistic_linear`, which fits the data into the logistic regression model and a simple ordinary least squared linear regression model. Then, it plots the two in one plot for better visual representation on why you should consider using classification rather than regression.

Plus, note the the term 'logistic regression' has a word 'regression' inside.
It is because the logistic regression is a generalised linear model using the same basic formula of linear regression but it is regressing for the probability of a categorical outcome by using `sigmoid` function.

```python
def compare_logistic_linear(model, X_data, y_data):
    """
    This function plots the given data - X_data and y_data
    then fit the data both into given `model` and LinearRegression model.
    Then shows the difference by plotting both of them.
    """
    plt.clf()
    plt.scatter(X_data.ravel(), y_data, color='black', zorder=20)
    X_test = np.linspace(-5, 10, 300)

    loss = expit(X_test * model.coef_ + model.intercept_).ravel()
    plt.plot(X_test, loss, color='red', linewidth=3)

    # Ordinary Least Squared Linear Regression
    ols = linear_model.LinearRegression()
    ols.fit(X_data, y_data)
    plt.plot(X_test, ols.coef_ * X_test + ols.intercept_, linewidth=1)
    plt.axhline(.5, color='.5')

    plt.ylabel('y')
    plt.xlabel('X')
    plt.xticks(range(-5, 10))
    plt.yticks([0, 0.5, 1])
    plt.ylim(-.25, 1.25)
    plt.xlim(-4, 10)
    plt.legend(('Logistic Regression Model', 'Linear Regression Model'),
               loc="lower right", fontsize='small')
    plt.tight_layout()
    plt.show()
```

```python
logistic_clf = linear_model.LogisticRegression(C=1e5, solver='lbfgs')
logistic_clf.fit(X, y)
```

```python
compare_logistic_linear(logistic_clf, X, y)
```

## 1-2. Implementing a logistic classifier from scratch

```python
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
import math
from sklearn.metrics import classification_report,accuracy_score

"""
If you are not familiar with this iris dataset, feel free to explore this dataset by EDA.
However, for now, I recommend you to just assume that you are given this data.
We will explore this dataset in the later section of this notebook (2-3. Decision Tree)
"""
iris = load_iris()
X = iris.data[:, :2]  # We will only use two features. Do not change X.
y = (iris.target != 0) * 1
```

```python
X[:3], y[:3]
```

```python
x_train, x_test, y_train, y_test = train_test_split(X, y,random_state=0)
```

```python
x_train.shape, x_test.shape, y_train.shape, y_test.shape
```

If you have watched Andrew Ng's lecture video, you will know that the cost function for logistic regression is below:
\begin{align}
J\left(\theta \right) & =  -{\frac{1}{n}}[\sum_{i=1}^n \left(y_i \log_2(P_r(\hat{y}=1|x_i;\theta))+(1-y_i)\log_2(1-P_r(\hat{y}=1|x_i;\theta)) \right)]\\
\end{align}

You are asked to complete the LogisticRegression Class below.
y_hat should be:
\begin{align}
\hat{y} & = \frac{1}{1+e^{-\theta^{*t}x}}
\end{align}

Since you are already familiar with the Iris dataset which is an in-built dataset provided by scikit learn, we will use that one.

```python
def sigmoid(z):
    # TODO
    # return y_hat. Look at the cell above. e to the -theta time x is just a z, our input of this function.
    return None

def loss(h, y):
    # TODO:
    # return the loss function J
    return None

def logRegParamEstimates(x_train, y_train):
    ## TODO: Fill in the 'None's commented section ##

    intercept = np.ones(("""What should be the dimension?"""))
    x_train = np.concatenate((intercept, x_train), axis=1)
    y_train[y_train > 1] = 0
    # Initialisation of theta.
    theta = np.zeros("""What should be the dimension of theta?""")
    lr = 0.01  # learning rate
    for i in range(100):
        z = np.dot("""dot product of which two?""")
        h = sigmoid("""sigmoid of what?""")
        # gradient part below was going to be TODO'ed, but since it can be hard to some of you guys, I'll just leave it.
        # But don't overlook this gradient part.
        gradient= ( 1 / (math.log(2) * x_train.shape[0]) ) * np.dot( (h-y_train), x_train)

        # TODO: update theta
        # you reassign theta as theta minus gradient*learning_rate
        # Imagine you are walking down the gradient convex with a step size of `learning rate`*gradient.
        # Think why you should subtract the `gradient*learning_rate' rather than adding it to the theta.
        theta = None

    return theta

def logistic_regression_predict(x_train, y_train, x_test):
    ## TODO: Fill in the 'None's  ##

    # Get theta matrix by calling the above function
    theta = None
    # This can be an hint to the solution somewhere above where you are asked to fill in a dimension.
    intercept = np.ones((x_test.shape[0], 1))
    x_test = np.concatenate((intercept, x_test), axis=1)
    sig = sigmoid(np.dot(x_test, theta))
    y_pred1=[]
    for i in range(len(sig)):
        if sig[i]>=0.5:
            y_pred1.append(1)
        else:
            y_pred1.append(3)
    return y_pred1

```

```python
y_pred1 = logistic_regression_predict(x_train, y_train,x_test)
print('Accuracy on test set: '+str(accuracy_score(y_test,y_pred1)))
print(classification_report(y_test,y_pred1))#text report showing the main classification metrics
```

```python

```

```python

```

# 2. Decision Tree
We will now look into different classification algorithm, which is called a Decision Tree Classifer.
Before we dive into it, let's first get to know what tree is in computer science.

## 2-1. Getting used to Tree Structure (optional)
The most basic tree is a Binary Tree, and it is crucial to know what Binary Tree is to undertand the Decision Tree Classifiers.
There are two search algorithms in tree structure. One is **Breadth First Search (BFS)** and the other is **Depth First Search (DFS)**. BFS and DFS are not only used in tree but also used in general graph structure. However, in this notebook we will only look into the cases in trees.

Note:
Don't worry if you are not familiar with recursion and data structure. This section is just to show you what Tree is. Just get the gist of the way it works.

1. **Binary Tree Breath First Search (BFS)**

```python
# BFS uses Queue as its Data Structure
import queue
```

```python
class Node():
    def __init__(self, val):
        self.val = val
        self.left = None
        self.right = None
```

```python
def completeBinaryTree(lst):
    def helper(index):
        if index >= len(lst):
            return None
        node = Node(lst[index])
        node.left = helper(index * 2 + 1)     # Think why the index should be 2n+1
        node.right = helper(index * 2 + 2)    # Think why the index should be 2n+2
        return node

    return helper(0)
```

```python
# This function is implemented as BFS algorithm.
def printTree(node):
    all_lines = []
    line = []

    q = queue.Queue()
    q.put(node)
    q.put(Node(-1))

    while q.qsize() > 0:
        node = q.get()

        if not node:
            continue
        else:
            if node.val == -1:
                if q.qsize() > 0:
                    all_lines.append(line)
                    line = []
                    q.put(Node(-1))
            else:
                line.append(node.val)
                q.put(node.left)
                q.put(node.right)

    return all_lines
```

```python
# Try different lists and visualise how the tree looks like
node_bfs = completeBinaryTree([1, 2, 3, 4, 5, 6, 7])
printTree(node_bfs)
```

```python

```

2. **Binary Tree Depth First Search (DFS)**

Here, we will reuse `Node` data structure, `completeBinaryTree` and `printTree` functions.
This time, to learn what DFS is, let me show you a simple DFS example.

Consider every route/path from the root (top Node of the tree) to the leaf (bottom Node of the tree).
Every step, you should swim down the tree and sum the value of each node.
Among the paths, if at least one sum of the path is equal to the number that you were given, you return True and False otherwise.

For example, say you were given a number 10,
and your tree looks like:
```
         1
     2       3
   4   5   6   7
```
By DFS, you should search every possible route:
1-2-4, 1-2-5, 1-3-6, 1-3-7.  Among these, sum of 1-3-6 make 10, so you return True.
If the given number was 18, you return False, because no single route can make 18.

```python
# Depth First Search in Binary Tree
def path_sum(node, targetSum):
    def dfsHelper(node, curSum):

        if node is None:
            if curSum == targetSum:
                return True
            else:
                return False
        else:
            curSum += node.val
            is_left = dfsHelper(node.left, curSum)
            is_right = dfsHelper(node.right, curSum)

        return is_left or is_right

    dfsHelper(node, 0)

    return dfsHelper(node, 0)
```

```python
node_dfs = completeBinaryTree([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16])
print(printTree(node_dfs))

print(path_sum(node_dfs, 16))
print(path_sum(node_dfs, 22))
print(path_sum(node_dfs, 35))
```

## 2-2. Manually emulating a simple Binary Tree Classifier (optional)
Try chaining configurations and code to see what's exactly happpening in this section

```python
import pandas as pd
```

```python
# We'll make a Classifier that classifies names based on major, height and gender
name = ['Nick', 'Danny', 'Claire', 'Rachel', 'Kamen', 'Bianca', 'Alice']
major = ['CS', 'CS', 'Neuroscience', 'Neuroscience', 'CS', 'Neuroscience', 'History']
height = [180, 177, 163, 168, 182, 170, 165]
gender = ['M', 'M', 'F', 'F', 'M', 'F', 'F']
```

```python
# Node number --> should be a global var
num = 0

# print as a Dataframe
data = pd.DataFrame({'Name': name, 'Major': major, 'Height': height, 'Gender': gender})
data
```

```python
# Height classifier Node
def Height_Node(df, idx, depth):
    global num
    num += 1
    print('Node_num : {} | Node Depth : {} | Height_Node'.format(num, depth))

    for i in idx:
        num += 1
        # Gender probably can have a strong correlation with heights
        if df['Gender'][i] == 'M':
            # 180 is classifying standard for men
            if df['Height'][i] < 180:
                print('Node_num : {} | Node Depth : {} | Name : {}'.format(num, depth, df['Name'][i]))
            else:
                print('Node_num : {} | Node Depth : {} | Name : {}'.format(num, depth, df['Name'][i]))

        else:
            # 165 is classifying standard for women
            if df['Height'][i] < 165:
                print('Node_num : {} | Node Depth : {} | Name : {}'.format(num, depth, df['Name'][i]))
            else:
                print('Node_num : {} | Node Depth : {} | Name : {}'.format(num, depth, df['Name'][i]))
```

```python
# Major classifier Node
def Major_Node(df, idx, depth):
    global num
    num += 1
    print('Node_num : {} | Node Depth : {} | Gender_Node'.format(num, depth))

    # List for storing index of CS guys
    CS = []

    for i in idx:
        # Store index if one's major is CS
        if df['Major'][i] == 'CS':
            CS.append(i)

        # print Node number, depth and name if Neuroscience
        else:
            num += 1
            print('Node_num : {} | Node Depth : {} | Name : {}'.format(num, depth, df['Name'][i]))

    print('CS Index : ', CS)

    # Classify CS guys by height, and depth should be increased by 1
    Height_Node(df, CS, depth + 1)
```

```python
# Gender classifier Node
def Gender_Node(df, depth):
    global num
    num += 1
    print('Node_num : {} | Node Depth : {} | Gender_Node'.format(num, depth))

    male = []
    female = []

    # classify by gender
    for idx, gen in enumerate(df['Gender']):
        if gen == 'M':
            male.append(idx)
        elif gen == 'F':
            female.append(idx)

    print('Male Index : ', male)
    print('Female Index : ', female)

    # Call Major classifier after classifying peeps by gender
    # Always increase depth by 1 when calling next node.
    # Major Node will then call Height Node
    Major_Node(df, male, depth + 1)
    Major_Node(df, female, depth + 1)
```

```python
Gender_Node(data, 1); num=0  # num=0 is just to reinitialise `num` if you run this cell again
```

```python

```

## 2-3. Decision Tree
Now you're ready to dive into the real Decision Tree Classifier!
Please do take time to watch videos below. They are worth watching.

### Must watch: Good video about Decision tree
1. Decision Tree Basics: https://www.youtube.com/watch?v=7VeUPuFGJHk&list=PLblh5JKOoLUICTaGLRoHQDuF_7q2GfuJF&index=38
2. Decision Tree Advanced: https://www.youtube.com/watch?v=wpNl-JwwplA&list=PLblh5JKOoLUICTaGLRoHQDuF_7q2GfuJF&index=39

```python
import numpy as np
import pandas as pd
from sklearn.datasets import load_iris

# Always check documentation
# https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.model_selection import GridSearchCV
```

```python
# Load iris data set
iris = load_iris()
```

```python
# iris is a dictionary that contains data, target data, and other information
# so you can access values by specifying key name
# for more detailed info,
# visit: https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_iris.html
iris.keys()
```

```python
# X data has its shape of 150x4
# y data has its shape of (150,)  --> which means that it is just an array of numbers --> can be reshaped to (150, 1)
iris.data.shape, iris.target.shape
```

```python
# each row has 4 training features, which are ouputted below
iris.data[0], iris.feature_names
```

```python
# each row the answer is one of three flowers- setosa, versicolor, or virginica
iris.target, iris.target_names
```

```python
# Split the iris dataset. tes
X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target, test_size=0.2)
X_train.shape, X_test.shape, y_train.shape, y_test.shape
```

```python
# get decision tree classifier from scikit learn
dtree = DecisionTreeClassifier()
```

Introducing `GridSearchCV`

```python
# Hyperparameters candidates
# max_depth: The maximum depth of the tree.
#            If None, then nodes are expanded until all leaves are pure or until all leaves contain less than min_samples_split samples.

# min_samples_split: The minimum number of samples required to split an internal node:
#                    If int, then consider min_samples_split as the minimum number.
#                    If float, then min_samples_split is a fraction and ceil(min_samples_split * n_samples) are the minimum number of samples for each split.
param_grid = {'max_depth': [1, 2, 3], 'min_samples_split': [2, 3]}

### Grid Search Cross Validation ###
# Since i've set 3 params for max_depth and 2 for min_samples_split,
# they will try out 6 combinations of those and pick one combination that let our model perform the best
## Question: what does cv parameter do, and why did I set cv=3 ?  ##
grid_dtree = GridSearchCV(dtree, param_grid=param_grid, cv=3, refit=True, iid=True)
# iid is set to True to avoid DeprecationWarning. You don't need to consider what iid is.
```

```python
# Train
grid_dtree.fit(X_train, y_train)
```

```python
# cv_results_ is an attribute of GridSearchCV
# and it is a dict with keys as column headers and values as columns, that can be imported into a pandas DataFrame.
scores_df = pd.DataFrame(grid_dtree.cv_results_)
print(scores_df)
scores_df[
    ['params', 'mean_test_score', 'rank_test_score', 'split0_test_score', 'split1_test_score', 'split2_test_score']]
```

```python
# Try printing out different attributes that GridSearchCV have
print('GridSearchCV-> best parameter combination : ', grid_dtree.best_params_)
print('GridSearchCV-> best score : {0:.4f}'.format(grid_dtree.best_score_))

# Get the best model/estimator
estimator = grid_dtree.best_estimator_
estimator
```

```python
pred = estimator.predict(X_test)
print(f'Test Dataset accuracy : {accuracy_score(y_test, pred)}')
```

```python

```

## Now it's time to exercise by yourself

**Let's classify digits**

```python
import numpy as np
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score

# Below are new imports
from sklearn.model_selection import KFold, train_test_split
from sklearn.datasets import load_digits
from sklearn.model_selection import GridSearchCV
import matplotlib.pyplot as plt
```

```python
digits = load_digits()
```

```python
# TODO: Explore digits Dataset by yourself.
# TODO: Try visualising each digit, by using matplotlib or any visualisation tool you may prefer
```

```python
digits.keys()
```

```python
def visualise_digit(index):
    # Can be challenging but you should try!
    pass
```

```python
feature = digits.data
label = digits.target

dt_clf = DecisionTreeClassifier()

param_grid = {'max_depth': np.arange(1, 30), 'min_samples_split': np.arange(2, 7)}
```

```python
# We will going to use KFold CV, just to revise
n_splits = 5    # This is 5 for reason, but feel free to try out different numbers
kfold = KFold(n_splits=n_splits)
n_iter = 0
cv_accuracy = []

# TODO: Split with ratio 7:3
X_train, X_test, y_train, y_test = """train test split should be familiar to you now"""

# TODO: Do `n_splits` KFold cross validation
for train_idx, vali_idx in kfold.split("""with which splitted dataset should you be doing kfold?"""):
    X_fold_train, X_fold_vali = None, None
    y_fold_train, y_fold_vali = None, None

    # TODO: Train your model with GridSearchCV
    grid_dtree = GridSearchCV("""Fill in parmas""", iid=True)
    grid_dtree.fit(None)

    # TODO: predict the output for each fold with validation set
    fold_pred = grid_dtree.predict(None)

    n_iter += 1
    acc_score = accuracy_score(y_fold_vali, fold_pred)
    print('Iter : {0}, CV accuary : {1:.4f}'.format(n_iter, acc_score))

    cv_accuracy.append(acc_score)
```

```python
# TODO: calculate the mean of cv accuracy
cv_accuracy_mean = None
print("Mean acc : {0:.4f}".format(cv_accuracy_mean))

# TODO: What's the best param combi and best score?
print('GridSearchCV -> best param : ', None)
print('GridSearchCV -> best accuracy : {0:.4f}'.format(None))

# TODO: get the best performing model
estimator = None

# TODO: predict with test data
pred = estimator.predict(None)

# How's the accuarcy of unseen data?
test_accuracy = accuracy_score(pred, y_test)
print("test accuarcy : {0:.4f}".format(test_accuracy))
```

```python

```

For those who want to learn further, learn:
- KNeighbors Classifier
- Ensemble
- Voting Classifier
- Gradient Boosting
- Random Forest

Some of those above can be dealt in the next session :) so stay tuned!

```python

```


# SD211 TP1 Systèmes de recommandation
*ZHAO Fubang*

```python
import numpy as np
from scipy import sparse
import matplotlib.pyplot as  plt
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
from scipy.optimize import check_grad
from scipy import linalg
%matplotlib inline
```

# 1 Présentation du modèle
## Question 1.1

```python
def load_movielens(filename, minidata=False):
    """
    Cette fonction lit le fichier filename de la base de donnees
    Movielens, par exemple
    filename = '~/datasets/ml-100k/u.data'
    Elle retourne
    R : une matrice utilisateur-item contenant les scores
    mask : une matrice valant 1 si il y a un score et 0 sinon
    """

    data = np.loadtxt(filename, dtype=int)

    R = sparse.coo_matrix((data[:, 2], (data[:, 0]-1, data[:, 1]-1)),
                          dtype=float)
    R = R.toarray()  # not optimized for big data

    # code la fonction 1_K
    mask = sparse.coo_matrix((np.ones(data[:, 2].shape),
                              (data[:, 0]-1, data[:, 1]-1)), dtype=bool )
    mask = mask.toarray()  # not optimized for big data

    if minidata is True:
        R = R[0:100, 0:200].copy()
        mask = mask[0:100, 0:200].copy()

    return R, mask
```

```python
R, mask = load_movielens('ml-100k/u.data', minidata=False)
print R.shape, mask.shape
```

    (943, 1682) (943, 1682)

## Reponse:
L'option minidata est pour juste choisir un dataset de taille 100*200 donc on peut réduire le temps de calculation.

## Question 1.2

```python
print np.nonzero(R)[0].shape
```

    (100000,)

## Reponse:

Selon le taille de R et le nombre de non_zero valeurs, on peut voir que:
1. Le nombre d'utilisateurs: 943;
2. Le nombre de films: 1682;
3. Le nombre total de notes: 100000

## Question 1.3

```python
def objSimple(P, Q, R, rho):
    tmp = R - Q*P
    val = (tmp ** 2)/2. + rho/2. * (Q ** 2 + P ** 2)
    return val
```

```python
fig = plt.figure(figsize=(12, 8))
ax = fig.gca(projection='3d')

# Make data.
P = np.arange(-5, 5, 0.05)
Q = np.arange(-5, 5, 0.05)
P, Q = np.meshgrid(P, Q)
Z = objSimple(P, Q, 10, 0.2)

# Plot the surface.
surf = ax.plot_surface(P, Q, Z, cmap=cm.coolwarm,
                       linewidth=0, antialiased=True)

# Add a color bar which maps values to colors.
fig.colorbar(surf, shrink=0.5, aspect=5)

plt.show()
```

## Reponse:
On prend une situation extrême, c'est à dire qu'on utilise les nombres pour remplacer les matrices.
Selon le graphe, on peut voir que le minimum n'est pas au milieu du surface. Donc, c'est pas convexe.

# 2 Trouver $P$ quand $Q_0$ est fixé
## Question 2.1

$$g(P) = \frac{1}{2}\|1_K\circ(R - Q^0P)\|^2_F+\frac{\rho}{2}\|Q^0\|^2_F+\frac{\rho}{2}\|P\|^2_F$$

$$\triangledown g(P) = - (Q^0)^T(1_K \circ (R - Q^0P)) + \rho P$$

$${\triangledown} ^2 g(P) = (Q^0)^TQ^0 + \rho I$$

Il est évident que la matrice est définie positive, donc $g(P)$ est convexe.

Son gradient est $\triangledown g(P) = (Q^0)^T((Q^0P - R)\circ 1_K) + \rho P$

## Question 2.2

```python
def func(P_ravel, Q0, R, mask, rho, c):
    P = P_ravel.reshape((c, P_ravel.shape[0]/c))
    tmp = (R - Q0.dot(P)) * mask
    val = np.sum(tmp ** 2)/2. + rho/2. * (np.sum(Q0 ** 2) + np.sum(P ** 2))
    return val

def grad(P_ravel, Q0, R, mask, rho, c):
    P = P_ravel.reshape((6, P_ravel.shape[0]/c))
    tmp = (R - Q0.dot(P)) * mask
    grad_P = -Q0.T.dot(tmp) + rho * P
    return grad_P.ravel()
```

```python
U, s, Vh = linalg.svd(R)
c = 6
rho = 0.2
Q0 = U[:, :c]
P0 = Vh[:c, :]
check_grad(func, grad, P0.ravel(), Q0, R, mask, rho, c)
```

## Reponse:
On verifie le calcul par *check_grad*. D'abord, il faut séparer la fonction *objective* à deux fonctions: *func* et *grad*. Après, il faut changer le paramètre *P* à un array(C'est à dire qu'il faut le reformer à un matrice. Selon le résultat de *check_grad*, on peut voir que les deux gradients sont presque égales.

## Question 2.3

```python
def objective(P, Q0, R, mask, rho):
    """
    La fonction objectif du probleme simplifie.
    Prend en entree
    P : la variable matricielle de taille C x I
    Q0 : une matrice de taille U x C
    R : une matrice de taille U x I
    mask : une matrice 0-1 de taille U x I
    rho : un reel positif ou nul

    Sorties :
    val : la valeur de la fonction
    grad_P : le gradient par rapport a P
    """
    tmp = (R - Q0.dot(P)) * mask
    val = np.sum(tmp ** 2)/2. + rho/2. * (np.sum(Q0 ** 2) + np.sum(P ** 2))
    grad_P = -Q0.T.dot(tmp) + rho * P
    return val, grad_P

def gradient(g, P0, gamma, epsilon):
    values = []
    new_value, grad = g(P0, Q0, R, mask, rho)
    values.append(new_value)
    while np.sum((grad) ** 2) ** 0.5 > epsilon:
        step = -gamma * grad / (np.sum(grad ** 2) ** 0.5)
        P0 = P0 + step
        new_value, grad = g(P0, Q0, R, mask, rho)
        values.append(new_value)
    return new_value, values
```

## Question 2.4

```python
v, vs = gradient(objective, P0, 1, 1)
print 'The minimal value is: %.2f' %v
print 'The iteration times are: %r' %len(vs)
plt.figure()
plt.plot(vs)
plt.show()
```

# 3 Raffinements algorithmiques pour le problème à $Q_0$ fixé
## Question 3.1

```python
def gradientRecLin(g, P0, epsilon):
    values = []
    new_value, grad = g(P0, Q0, R, mask, rho)
    values.append(new_value)
    while np.sum((grad) ** 2) ** 0.5 > epsilon:
        temp1 = rho*np.sum(grad*P0) - np.sum((Q0.dot(grad)*(R - Q0.dot(P0))*mask))
        temp2 = rho*np.sum((grad)**2) + np.sum((Q0.dot(grad)*mask)**2)
        y = temp1/temp2
        P0 = P0 - y*grad
        new_value, grad = g(P0, Q0, R, mask, rho)
        values.append(new_value)
    return new_value, values
```

```python
v_r, vs_r = gradientRecLin(objective, P0, 1)
print 'The minimal value is: %.2f' %v_r
print 'The iteration times are: %r' %len(vs_r)
plt.figure()
plt.plot(vs_r)
plt.show()
```

## Question 3.2
#### Théorème:
Soit $q(x) = \frac{1}{2}x^t A x + b^t x + c$ une fonction quadratique, où A est une matrice symétrique définie positive. On peut utiliser le méthode des gradients conjugués.

\begin{align}
g(P) &= \frac{1}{2}\|1_K\circ(R - Q^0P)\|^2_F+\frac{\rho}{2}\|Q^0\|^2_F+\frac{\rho}{2}\|P\|^2_F\\
&= Tr[\frac{1}{2}(1_K \circ (R - Q_0P))^T(1_K \circ (R - Q_0P)) + \frac{\rho}{2}P^T P ] + \frac{\rho}{2}\|Q_0\|^2_F
\end{align}

Pour transformer le calcul de matrice au celui de vecteur, on définit quelques variables:

$X = P.reshape(CI,1) =
\begin{Bmatrix}
        P_{1} \\
        P_{2} \\
        \vdots \\
        P_{I-1} \\
        P_{I} \\
\end{Bmatrix}$

$R' = R.reshape(UI,1) =
\begin{Bmatrix}
        R_{1} \\
        R_{2} \\
        \vdots \\
        R_{I-1} \\
        R_{I} \\
\end{Bmatrix}$

$Q' =
\begin{Bmatrix}
    Q_0 & 0 & \cdots & 0 \\
    0 & Q_0 & \cdots & 0 \\
    \vdots & \vdots & \ddots & \vdots \\
    0 & 0 & \cdots & Q_0 \\
\end{Bmatrix} $

(Q'.shape = $UI*CI$)

$M' =
\begin{Bmatrix}
    mask_{1, 1} & 0 & \cdots & 0 \\
    0 & mask_{2, 1} & \cdots & 0 \\
    \vdots & \vdots & \ddots & \vdots \\
    0 & 0 & \cdots & mask_{U, I} \\
\end{Bmatrix} $

(M'.shape = $UI*UI$)

On définit que $f(X)=g(P)$, et:
\begin{align}
f(x) & = g(P) \\
& = Tr[\frac{1}{2}(1_K \circ (R - Q_0P))^T(1_K \circ (R - Q_0P)) + \frac{\rho}{2}P^T P ] + \frac{\rho}{2}\|Q_0\|^2_F \\
& = \frac{1}{2}(M'*(R'-Q'X))^T(M'*(R'-Q'X)) + \frac{\rho}{2}X^TX + \frac{\rho}{2}\|Q_0\|^2_F\\
& = \frac{1}{2}(X^T((M'Q')^TM'Q' + \rho I_d)X - 2R'^TM'Q'X + R'^TM'R') + \frac{\rho}{2}\|Q_0\|^2_F \\
\end{align}
On a que:

$A = Q'^TM'Q' + \rho I_{CI}$

$b^t = -R'^TM'Q'$

$c = \frac{1}{2}R'^TM''R' + \frac{\rho}{2}\|Q_0\|^2_F$

et:

$f(x) = \frac{1}{2}x^t A x + b^t x + c$

Il est évident que A est définie positive, donc on peut utiliser la méthode des gradients conjugués.
Car pour ce problème, c'est pas pratique de transformer tous les matrices aux vecteurs, on utilise un algorithme pour la fonction quelconque lors de calculer le résultat numérique.

```python
def gradientGc(g, P0, epsilon):
    values = []
    new_value, grad = g(P0, Q0, R, mask, rho)
    d0 = grad
    values.append(new_value)
    while np.sum((grad) ** 2) ** 0.5 > epsilon:
        temp1 = rho*np.sum(d0*P0) - np.sum((Q0.dot(d0)*(R - Q0.dot(P0))*mask))
        temp2 = rho*np.sum((d0)**2) + np.sum((Q0.dot(d0)*mask)**2)
        y = temp1/temp2
        P0 = P0 - y*d0
        grad_old = grad
        new_value, grad = g(P0, Q0, R, mask, rho)
        values.append(new_value)
        b = np.sum(grad ** 2)/np.sum(grad_old ** 2)
        d0 =  grad + b * d0
    return new_value, values
```

```python
v_gc, vs_gc = gradientGc(objective, P0, 1)
print 'The minimal value is: %.2f' %v_gc
print 'The iteration times are: %r' %len(vs_gc)
plt.figure()
plt.plot(vs_gc)
plt.show()
```

## Question 3.3
Pour minimiser la fonction jusqu'à $\epsilon<=1$:
1. La méthode de gradient a besoins de 1128 fois d'itération avec $\gamma=1$, le valeur mnimal est 303938.38.
2. La méthode de recherche linéaire a besoins de 12 fois d'itération , le valeur mnimal est 303937.25 qui est plus petit que la méthode de gradient.
3. La méthode de gradient conjugué a besoins de juste 8 fois d'itération , le valeur mnimal est 303936.98 qui est plus petit que la méthode de gradient.

#Résolution du problème complet
## Question 4.1

```python
def total_objective(P, Q, R, mask, rho):
    """
    La fonction objectif du probleme complet.
    Prend en entree
    P : la variable matricielle de taille C x I
    Q : la variable matricielle de taille U x C
    R : une matrice de taille U x I
    mask : une matrice 0-1 de taille U x I
    rho : un reel positif ou nul

    Sorties :
    val : la valeur de la fonction
    grad_P : le gradient par rapport a P
    grad_Q : le gradient par rapport a Q
    """

    tmp = (R - Q.dot(P)) * mask

    val = np.sum(tmp ** 2)/2. + rho/2. * (np.sum(Q ** 2) + np.sum(P ** 2))

    grad_P = -Q0.T.dot(tmp) + rho * P

    grad_Q = -tmp.dot(P0.T) + rho * Q

    return val, grad_P, grad_Q

def gradientRecLinP(g, P00, Q00, epsilon):
    new_value, grad, _ = g(P00, Q00, R, mask, rho)
    k = 0
    while k < 30:
        temp1 = rho*np.sum(grad*P00) - np.sum(Q00.dot(grad)*(R - Q00.dot(P00))*mask)
        temp2 = rho*np.sum((grad)**2) + np.sum((Q00.dot(grad)*mask)**2)
        y = temp1/temp2
        P00 = P00 - y*grad
        new_value, grad, _ = g(P00, Q00, R, mask, rho)
        k = k + 1
    return new_value, P00

def gradientRecLinQ(g, P00, Q00, epsilon):
    new_value, _, grad = g(P00, Q00, R, mask, rho)
    k = 0
    while k < 30:
        temp1 = rho*np.sum(grad*Q00) - np.sum(grad.dot(P00)*(R - Q00.dot(P00))*mask)
        temp2 = rho*np.sum((grad)**2) + np.sum((grad.dot(P00)*mask)**2)
        y = temp1/temp2
        Q00 = Q00 - y*grad
        new_value, _, grad = g(P00, Q00, R, mask, rho)
        k = k + 1
    return new_value, Q00

def gradientTotal(P00, Q00, epsilon, times):
    k = 0
    values = []
    P11, Q11 = P00, Q00
    while k < times:
        v, Q11 = gradientRecLinQ(total_objective, P11, Q11, epsilon)
        values.append(v)
        v, P11 = gradientRecLinP(total_objective, P11, Q11, epsilon)
        values.append(v)
        k = k + 1
    return v, values, P11, Q11
```

```python
P1 = np.ones(P0.shape)
Q1 = np.ones(Q0.shape)
print P1.shape, Q1.shape
```

    (6, 1682) (943, 6)

```python
v_t, vs_t, P_rl, Q_rl = gradientTotal(P1, Q1, 100, 5)
print 'The minimal value is: %.2f' %v_t
print 'The iteration times are: %r' %len(vs_t)
plt.figure()
plt.plot(vs_t)
plt.show()
```

## Question 4.2
*for  k >= 1 do*

$$P_k = argmin_P(\frac{1}{2}(1_K \circ ||R-Q_{k-1}P||_F^2) + \frac{\rho}{2}(||P||_F^2 + ||Q_{k-1}||_F^2)$$

$$Q_k = argmin_Q(\frac{1}{2}(1_K \circ ||R-QP_k||_F^2) + \frac{\rho}{2}(||P_k||_F^2 + ||Q||_F^2)$$

*end for*

Pour ce problème, la fonction est toujours convexe par rapport à Q et P. Donc à chaque étape, l'objectif décroît. Cependant, on peut seulement avoir le minimum local car la fonction n'est pas convexe lorque tous Q et P ne sont pas fixé;

## Question 4.3

## Question 4.4

## Question 4.5

```python
resLS = np.argmax((Q_rl[449,:].dot(P_rl))*(1 - mask[449,:]))

print(resLS)
```

    951

Le meillieur film pour utilisateur 449 est: 951