## Burgers Equation
#### Parameter estimation for Burgers' Equation using Gaussian processes (Backward Euler scheme)

#### Problem Setup

$u_t + u u_{x} - (\nu - e) u_{xx} = 0$

$u(x,t) = \frac{x}{1+t}$

$u_0(x) := u(x,0) = x$

$x \in [0, 1], t \in \{0, \tau \}$

Using the backward Euler scheme, the equation can be re-written as:

$\frac{u_n - u_{n-1}}{\tau} + u_n \frac{d}{dx}u_n - (\nu - e) \frac{d^2}{dx^2}u_n = 0$

and setting $u_n = \mu_n$ to deal with the non-linearity:

$u_n + \tau \mu_n \frac{d}{dx}u_n - \tau (\nu - e) \frac{d^2}{dx^2}u_n = u_{n-1}$

Consider $u_n$ to be a Gaussian processes.

$u_n \sim \mathcal{GP}(0, k_{uu}(x_i, x_j, \theta))$

And the linear operator:

$\mathcal{L}_x^\nu = \cdot + \tau \mu_n \frac{d}{dx}\cdot - \tau (\nu - e) \frac{d^2}{dx^2} \cdot$

so that

$\mathcal{L}_x^\nu u_n = u_{n-1}$

Problem at hand: estimate $\nu$ (should be $\nu = e$ in the end).

For the sake of simplicity, take $u := u_n$ and $f := u_{n-1}$.

#### step 1: Simulate data

Take data points at $t = 0$ for $(u_{n-1})$ and $t = \tau$ for $(u_n)$, where $\tau$ is the time step.

```python
import numpy as np
import sympy as sp
from scipy.optimize import minimize
import matplotlib.pyplot as plt
```

```python
tau = 0.001
def get_simulated_data(tau):
    x = np.random.rand(50)
    y_u = x/(1+tau)
    y_f = x
    return (x, y_u,y_f)

(x, y_u, y_f) = get_simulated_data(tau)
```

```python
plt.plot(x, y_u, 'ro')
plt.show()
```

```python
plt.plot(x, y_f, 'bo')
plt.show()
```

#### Step 2:Evaluate kernels

$k_{nn}(x_i, x_j; \theta) = \theta exp(-\frac{1}{2l}(x_i-x_j)^2)$

```python
x_i, x_j, theta, l, nu = sp.symbols('x_i x_j theta l nu')
mu_n = np.mean(y_f)
kuu_sym = theta*sp.exp(-1/(2*l)*((x_i - x_j)**2))
kuu_fn = sp.lambdify((x_i, x_j, theta, l), kuu_sym, "numpy")
def kuu(x, theta, l):
    k = np.zeros((x.size, x.size))
    for i in range(x.size):
        for j in range(x.size):
            k[i,j] = kuu_fn(x[i], x[j], theta, l)
    return k
```

$k_{ff}(x_i,x_j;\theta,\phi) \\
= \mathcal{L}_{x_i}^\nu \mathcal{L}_{x_j}^\nu k_{uu}(x_i, x_j; \theta) \\
= \mathcal{L}_{x_i}^\nu \left( k_{uu} + \tau \mu_n \frac{d}{dx_j}k_{uu} - \tau(\nu - e)\frac{d^2}{dx_j^2}k_{uu} \right) \\
 = k_{uu} + \tau \mu_n \frac{d}{dx_i}k_{uu} - \tau(\nu-e)\frac{d^2}{dx_i^2}k_{uu} + \tau\mu_n\frac{d}{dx_j}k_{uu} + \tau\mu_n\frac{d}{dx_i}\tau\mu_n\frac{d}{dx_j}k_{uu} - \tau(\nu - e)\frac{d^2}{dx_i^2}\tau\mu_n\frac{d}{dx_j}k_{uu} - \tau(\nu-e)\frac{d^2}{dx_j^2}k_{uu} - \tau\mu_n\frac{d}{dx_i}\tau(\nu-e)\frac{d^2}{dx_j^2}k_{uu} + \tau(\nu - e)\frac{d^2}{dx_i^2}\tau(\nu-e)\frac{d^2}{dx_j^2}k_{uu}$

```python
kff_sym = kuu_sym \
        + tau*mu_n*(sp.diff(kuu_sym, x_i)) \
        - tau*(nu-np.e)*sp.diff(kuu_sym, x_i, x_i) \
        + tau*mu_n*sp.diff(kuu_sym, x_j) \
        + tau**2*mu_n**2*sp.diff(kuu_sym, x_i, x_j) \
        - tau**2*(nu-np.e)*mu_n*sp.diff(kuu_sym, x_j, x_i, x_i) \
        - tau*(nu-np.e)*sp.diff(kuu_sym, x_j, x_j) \
        - tau**2*mu_n*(nu-np.e)*sp.diff(kuu_sym, x_j, x_j, x_i) \
        + tau**2*(nu-np.e)**2*sp.diff(kuu_sym, x_j, x_j, x_i, x_i)
kff_fn = sp.lambdify((x_i, x_j, theta, l, nu), kff_sym, "numpy")
def kff(x, theta, l, nu):
    k = np.zeros((x.size, x.size))
    for i in range(x.size):
        for j in range(x.size):
            k[i,j] = kff_fn(x[i], x[j], theta, l, nu)
    return k
```

$k_{fu}(x_i,x_j;\theta,\phi) \\
= \mathcal{L}_{x_i}^\nu k_{uu}(x_i, x_j; \theta) \\
= k_{uu} + \tau \mu_n \frac{d}{dx_i}k_{uu} - \tau(\nu - e) \frac{d^2}{dx_i^2}k_{uu}$

```python
kfu_sym = kuu_sym + tau*mu_n*sp.diff(kuu_sym, x_i) - tau*(nu-np.e)*sp.diff(kuu_sym, x_i, x_i)
kfu_fn = sp.lambdify((x_i, x_j, theta, l, nu), kfu_sym, "numpy")
def kfu(x, theta, l, nu):
    k = np.zeros((x.size, x.size))
    for i in range(x.size):
        for j in range(x.size):
            k[i,j] = kfu_fn(x[i], x[j], theta, l, nu)
    return k
```

```python
def kuf(x, theta, l, nu):
    return kfu(x,theta, l, nu).T
```

#### Step 3: Compute NLML

```python
def nlml(params, x, y1, y2, s):
    theta_exp = np.exp(params[0]) # don't see why mu or nu must be positive
    l_exp = np.exp(params[1])
    K = np.block([
        [kuu(x, theta_exp, l_exp) + s*np.identity(x.size), kuf(x, theta_exp, l_exp, params[2])],
        [kfu(x, theta_exp, l_exp, params[2]), kff(x, theta_exp, l_exp, params[2]) + s*np.identity(x.size)]
    ])
    y = np.concatenate((y1, y2))
    print(np.linalg.det(K))
    val = 0.5*(np.log(abs(np.linalg.det(K))) + np.mat(y) * np.linalg.inv(K) * np.mat(y).T)
    return val.item(0)

```

```python
# nlml((5,2,1), mu_n, x, y_u, y_f, 1e-2)

```

Mapping w.r.t. theta:

```python
#z = np.linspace(-10, 10)
#z_out = [nlml((q, 0.53, 6.22), x, y_u, y_f, 1e-3) for q in z]
#plt.plot(z, z_out)

#nlml((-0.47, 0.5, np.e), x, y_u, y_f, 1e-3)
```

Mapping w.r.t. nu:

```python
#nlml((-2.4, 0.5, np.e) ,x ,y_u,y_f,1e-3) # mu should be roughly 0.5

#z = np.linspace(-10, 10)
#z_out = [nlml((0.53, 0.53, q), x, y_u, y_f, 1e-6) for q in z]
#plt.plot(z, z_out)

# Relatively consistently we get the info, that vu = np.e is the maximizer, not the minimizer of the function:
#print(nlml((0.53, 0.525, np.e - 0.2) ,x ,y_u,y_f,1e-6))
#print(nlml((0.53, 0.525, np.e) ,x ,y_u,y_f,1e-6))
#print(nlml((0.53, 0.525, np.e + 0.2) ,x ,y_u,y_f,1e-6))

# This is possibly hinting at a sign error somewhere
```

#### Step 4: Optimise hyperparameters

```python
m = minimize(nlml, (np.random.rand(1), 0.5, np.e), args=(x, y_u, y_f, 1e-3), method="Nelder-Mead", options = {'maxiter' : 1000})
```

    3.042479607885426e-289
    3.211504591379622e-289
    2.705343498901258e-289
    3.045866688178866e-289
    2.667469683555439e-289
    2.5750665933306594e-289
    2.3696734601830804e-289
    2.185272801279879e-289
    1.8575882622698195e-289
    1.9563998173887035e-289
    1.5604475454613908e-289
    1.195360032029416e-289
    1.1321244075621684e-289
    7.961810210777395e-290
    7.526304817032698e-290
    4.7950841144188857e-290
    3.369233974253082e-290
    1.5749050625598486e-290
    1.373243922729275e-290
    5.3615650427673284e-291
    3.6316181008283006e-291
    1.203896017568132e-291
    2.7629759532199307e-291
    1.0214558835641094e-291
    7.549794235087319e-291
    1.0354504530100237e-290
    3.79936132106044e-291
    7.932536460860595e-291
    3.9392615226491996e-291
    8.022568995184481e-291
    3.872901697701663e-291
    2.3984044809562556e-291
    5.8803043325259495e-291
    6.443529281594926e-291
    5.663592130832428e-291
    8.128831064273123e-291
    4.70437232819429e-291
    5.417029382744925e-291
    5.4450885729180534e-291
    4.998241575355497e-291
    6.269064228695591e-291
    7.247426497263659e-291
    5.755946304204629e-291
    5.696626624004703e-291
    7.398294730531573e-291
    8.328966224180752e-291
    1.0320476050053674e-290
    7.9040435479855e-291
    8.25806362805192e-291
    8.331869847745779e-291
    8.84833212844404e-291
    1.4548025677788654e-290
    1.0128046093944295e-290
    1.010444873704958e-290
    1.4353675823118626e-290
    8.238612766634865e-291
    6.333723727007314e-291
    1.0725772327100948e-290
    1.2033346272451032e-290
    6.034947965741531e-291
    6.2168521586712064e-291
    9.491384217217396e-291
    1.2215531006702021e-290
    1.3072472545868666e-290
    1.9456141506562074e-290
    2.0746489158104317e-290
    1.6361768324630202e-290
    1.2746996118356228e-290
    6.569952781015258e-291
    1.5504582415392838e-290
    1.999905109682775e-290
    1.138382226657001e-290
    1.385127934395591e-290
    1.2954316024752876e-290
    1.332693055136552e-290
    1.0098821718123158e-290
    1.3928311331942932e-290
    1.2172004347984677e-290
    1.300484306592837e-290
    1.5584444649914195e-290
    1.22916543815668e-290
    1.3081352366958155e-290
    1.3130522887969534e-290
    1.1710162115610029e-290
    1.3335721710982394e-290
    1.284518806094573e-290
    1.2923126390620203e-290
    1.3988313610040983e-290
    1.269476781741983e-290
    1.2886614747285333e-290
    1.293351326115241e-290
    1.3047227808244396e-290
    1.310993948843718e-290
    1.2500751977040135e-290
    1.2998070807602015e-290
    1.3153524571885094e-290
    1.2801620149978336e-290
    1.2738533875046376e-290
    1.3454596075437664e-290
    1.3053236030837564e-290
    1.3025149017782212e-290
    1.3185236702148285e-290
    1.2566703192473965e-290
    1.3045010867151362e-290
    1.3168919090248406e-290
    1.371180451948931e-290
    1.341699460051791e-290
    1.2711413370527375e-290
    1.2535869892939049e-290
    1.3189573358794402e-290
    1.3017857045209372e-290
    1.301495278161119e-290
    1.3549733863243344e-290
    1.3295847662300316e-290
    1.3350221764836223e-290
    1.344033483881101e-290
    1.2991076533347112e-290
    1.2722468299576116e-290
    1.3322571752853282e-290
    1.3483810269775947e-290
    1.292870227307488e-290
    1.2749209644970125e-290
    1.2476159897560091e-290
    1.339405577748121e-290
    1.3758026750954533e-290
    1.3490457935087472e-290
    1.379041366079343e-290
    1.2947395166724097e-290
    1.307467351842352e-290
    1.3916465099141506e-290
    1.3029973230139468e-290
    1.3004741120254598e-290
    1.3294498785750784e-290
    1.278402094228633e-290
    1.331005694981482e-290
    1.3350167286093053e-290
    1.328004621910627e-290
    1.309324365581318e-290
    1.325525613845143e-290
    1.3082674107741344e-290
    1.3241025280817044e-290
    1.3491553257629533e-290
    1.3143836115964853e-290
    1.3147073886200703e-290
    1.3180091491605076e-290
    1.3121590351473222e-290
    1.305532595443574e-290
    1.3056564953780745e-290
    1.3194657281050077e-290
    1.3187072607052716e-290
    1.3154599209760836e-290
    1.3133888819474485e-290
    1.3168481724015914e-290
    1.322377019302309e-290
    1.3147049355736595e-290
    1.3118861156235442e-290
    1.317566616616631e-290
    1.317286862000096e-290
    1.316829408377248e-290
    1.319463037256115e-290
    1.3158923886314727e-290
    1.31667796211023e-290
    1.3168052522978547e-290
    1.3154521897203146e-290
    1.3152706312978763e-290
    1.3144920439970796e-290
    1.316526820525933e-290
    1.3162579595793682e-290
    1.3163300605137524e-290
    1.3160017162209353e-290
    1.3148823327910448e-290
    1.3163241977180237e-290
    1.3171192474122106e-290
    1.31573253455097e-290
    1.3157810487288286e-290
    1.315900239448257e-290
    1.315969588927048e-290
    1.3159936685262285e-290
    1.3164129354162765e-290
    1.3159025985600324e-290
    1.3155402608608054e-290
    1.3161281662973015e-290
    1.3159603330989056e-290
    1.3159686635123664e-290
    1.316099383281274e-290
    1.3160495928678136e-290
    1.3161950315818192e-290
    1.3159756996876206e-290
    1.315867815386356e-290
    1.3160630736936854e-290
    1.3160902518513062e-290
    1.3159990583821882e-290
    1.315975628940763e-290
    1.3159941192408428e-290
    1.3160618025712554e-290
    1.3159972245228635e-290
    1.3159305310250826e-290
    1.31602993678775e-290
    1.3160151286621835e-290
    1.3160340743801878e-290
    1.3160240854177034e-290
    1.3159943558963245e-290
    1.316021041421481e-290
    1.3159981777607144e-290
    1.3160176084377993e-290
    1.3160087876604254e-290
    1.3160135433801913e-290
    1.31603757122559e-290
    1.3160073111682713e-290
    1.316015575943632e-290
    1.3160193249422104e-290
    1.3160074164712364e-290
    1.3160139906079882e-290
    1.316013197969384e-290
    1.316026532925396e-290
    1.3160194296870287e-290
    1.3160194821007346e-290
    1.3160074863463636e-290
    1.3160217711693276e-290

```python
m.x[2]
```

    4.355178275479192

```python
m
```

     final_simplex: (array([[1.79784263, 2.81784441, 4.35517828],
           [1.79783886, 2.81784109, 4.35520677],
           [1.79783707, 2.8178407 , 4.35510619],
           [1.79784109, 2.81784399, 4.35526421]]), array([-332.29109637, -332.29109637, -332.29109637, -332.29109637]))
               fun: -332.29109636678197
           message: 'Optimization terminated successfully.'
              nfev: 219
               nit: 121
            status: 0
           success: True
                 x: array([1.79784263, 2.81784441, 4.35517828])

```python

```


# Goal
CartPole-v0 of OpenAi Gym
" A pole is attached by an un-actuated joint to a cart, which moves along a frictionless track. The system is controlled by applying a force of +1 or -1 to the cart. The pendulum starts upright, and the goal is to prevent it from falling over. A reward of +1 is provided for every timestep that the pole remains upright. The episode ends when the pole is more than 15 degrees from vertical, or the cart moves more than 2.4 units from the center."

# Representation
- At each iteration the agent observe a state of the environement.
- After each action it has to choose an action developing by that a new transition.
- After each iteration he receives a reward of **+1** for having maintained.
- The state will be represented ( for the model ) as the difference between the actual state and the previous one ( to observe the motion ).

# Libraries
- Neural networks (torch.nn)
- Environement (GYM)
- Optimization (torch.optim)
- Automatic differentiation (torch.autograd)
- utilities for vision tasks (torchvision - a separate package).
- Matplotlib (ploting graphs)

" Automatic differentiation (autodiff) refers to a general way of takinga program which computes a value, and automatically constructing aprocedure for computing derivatives of that value. "

```python
import gym
import math
import random
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
from collections import namedtuple
from itertools import count
from PIL import Image

import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
import torchvision.transforms as T

env = gym.make('CartPole-v0').unwrapped

# set up matplotlib
is_ipython = 'inline' in matplotlib.get_backend()
if is_ipython:
    from IPython import display

plt.ion()

# if gpu is to be used
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
```

# Replay Memory
Now the agent is going to play some games with random moves and store each game's data.
And we'll sample in that data to suppress the correlation between them and stabilizing by that the DQN algorithm.

```python
Transition = namedtuple('Transition',
                        ('state', 'action', 'next_state', 'reward'))

class ReplayMemory(object):

    def __init__(self, capacity):
        self.capacity = capacity
        self.memory = []
        self.position = 0

    def push(self, *args):
        """Saves a transition."""
        if len(self.memory) < self.capacity:
            self.memory.append(None)
        self.memory[self.position] = Transition(*args)
        self.position = (self.position + 1) % self.capacity

    def sample(self, batch_size):
        return random.sample(self.memory, batch_size)

    def __len__(self):
        return len(self.memory)

```

- Transition - a named tuple representing a single transition in our environment. It essentially maps (state, action) pairs to their (next_state, reward) result, with the state being the screen difference image as described later on.
- ReplayMemory - a cyclic buffer of bounded size that holds the transitions observed recently. It also implements a .sample() method for selecting a random batch of transitions for training.

# DQN Algorithm
The aim of the algorithm is to maximize the discounted cumulative Reward :
\begin{align}
R_t_0 = \sum_{t=t_0}^{\infty}  \gamma^(t−t_0)  r_{t}
\end{align}

$R_t0$ : The Return.

$gamma$ : The discount constant [0 - 1] (ensuring the sum converges) making the actions in near future more important than the ones in a far future.

The goal is :

$ Q^*(state, action) = Reward $
to find a policy that maximize the rewards :
$ \pi_^*(state) = argmax(Q^*(state, action_i)) $
But we don't know the exact representation of the Q function, so we're going to train a neural network to estimate it.

### Update Rule
Knowing that evry Q function for some policy respect the Bellman equation
$Q_\pi (s,a)=r + \gamma Q^\pi (s′,\pi(s′))$

the difference between the two sides is known as **temporal difference error** $\delta$ :
$Q_\pi (s,a) = r+\gamma Q_\pi (s′,\pi (s′))$

And the neural network will minimize this error using the **Huber Loss** .
Huber Loss features :
- When the error is small : acts like mean squarred error.
- When the error is big : acts like mean absolute squared error.

- B : Batch of transitions

# The Model
Our model will be a convolutional neural network that takes in the difference between the current and previous screen patches. It has two outputs, representing Q(s,left) and Q(s,right) (where s is the input to the network). In effect, the network is trying to predict the expected return of taking each action given the current input.

```python
class DQN(nn.Module):

    def __init__(self, h, w, outputs):
        super(DQN, self).__init__()
        self.conv1 = nn.Conv2d(3, 16, kernel_size=5, stride=2)
        self.bn1 = nn.BatchNorm2d(16)
        self.conv2 = nn.Conv2d(16, 32, kernel_size=5, stride=2)
        self.bn2 = nn.BatchNorm2d(32)
        self.conv3 = nn.Conv2d(32, 32, kernel_size=5, stride=2)
        self.bn3 = nn.BatchNorm2d(32)

        # Number of Linear input connections depends on output of conv2d layers
        # and therefore the input image size, so compute it.
        def conv2d_size_out(size, kernel_size = 5, stride = 2):
            return (size - (kernel_size - 1) - 1) // stride  + 1
        convw = conv2d_size_out(conv2d_size_out(conv2d_size_out(w)))
        convh = conv2d_size_out(conv2d_size_out(conv2d_size_out(h)))
        linear_input_size = convw * convh * 32
        self.head = nn.Linear(linear_input_size, outputs)

    # Called with either one element to determine next action, or a batch
    # during optimization. Returns tensor([[left0exp,right0exp]...]).
    def forward(self, x):
        x = F.relu(self.bn1(self.conv1(x)))
        x = F.relu(self.bn2(self.conv2(x)))
        x = F.relu(self.bn3(self.conv3(x)))
        return self.head(x.view(x.size(0), -1))
```

# Input extraction and Training loop
The code below are utilities for extracting and processing rendered images from the environment. It uses the torchvision package, which makes it easy to compose image transforms. Once you run the cell it will display an example patch that it extracted.

```python
resize = T.Compose([T.ToPILImage(),
                    T.Resize(40, interpolation=Image.CUBIC),
                    T.ToTensor()])

def get_cart_location(screen_width):
    world_width = env.x_threshold * 2
    scale = screen_width / world_width
    return int(env.state[0] * scale + screen_width / 2.0)  # MIDDLE OF CART

def get_screen():
    # Returned screen requested by gym is 400x600x3, but is sometimes larger
    # such as 800x1200x3. Transpose it into torch order (CHW).
    screen = env.render(mode='rgb_array').transpose((2, 0, 1))
    # Cart is in the lower half, so strip off the top and bottom of the screen
    _, screen_height, screen_width = screen.shape
    screen = screen[:, int(screen_height*0.4):int(screen_height * 0.8)]
    view_width = int(screen_width * 0.6)
    cart_location = get_cart_location(screen_width)
    if cart_location < view_width // 2:
        slice_range = slice(view_width)
    elif cart_location > (screen_width - view_width // 2):
        slice_range = slice(-view_width, None)
    else:
        slice_range = slice(cart_location - view_width // 2,
                            cart_location + view_width // 2)
    # Strip off the edges, so that we have a square image centered on a cart
    screen = screen[:, :, slice_range]
    # Convert to float, rescale, convert to torch tensor
    # (this doesn't require a copy)
    screen = np.ascontiguousarray(screen, dtype=np.float32) / 255
    screen = torch.from_numpy(screen)
    # Resize, and add a batch dimension (BCHW)
    return resize(screen).unsqueeze(0).to(device)
BATCH_SIZE = 128
GAMMA = 0.999
EPS_START = 0.9
EPS_END = 0.05
EPS_DECAY = 200
TARGET_UPDATE = 10

# Get screen size so that we can initialize layers correctly based on shape
# returned from AI gym. Typical dimensions at this point are close to 3x40x90
# which is the result of a clamped and down-scaled render buffer in get_screen()
init_screen = get_screen()
_, _, screen_height, screen_width = init_screen.shape

# Get number of actions from gym action space
n_actions = env.action_space.n

policy_net = DQN(screen_height, screen_width, n_actions).to(device)
target_net = DQN(screen_height, screen_width, n_actions).to(device)
target_net.load_state_dict(policy_net.state_dict())
target_net.eval()

optimizer = optim.RMSprop(policy_net.parameters())
memory = ReplayMemory(10000)

steps_done = 0

def select_action(state):
    global steps_done
    sample = random.random()
    eps_threshold = EPS_END + (EPS_START - EPS_END) * \
        math.exp(-1. * steps_done / EPS_DECAY)
    steps_done += 1
    if sample > eps_threshold:
        with torch.no_grad():
            # t.max(1) will return largest column value of each row.
            # second column on max result is index of where max element was
            # found, so we pick action with the larger expected reward.
            return policy_net(state).max(1)[1].view(1, 1)
    else:
        return torch.tensor([[random.randrange(n_actions)]], device=device, dtype=torch.long)

def plot_durations():
    plt.figure(2)
    plt.clf()
    durations_t = torch.tensor(episode_durations, dtype=torch.float)
    plt.title('Training...')
    plt.xlabel('Episode')
    plt.ylabel('Duration')
    plt.plot(durations_t.numpy())
    # Take 100 episode averages and plot them too
    if len(durations_t) >= 100:
        means = durations_t.unfold(0, 100, 1).mean(1).view(-1)
        means = torch.cat((torch.zeros(99), means))
        plt.plot(means.numpy())

    plt.pause(0.001)  # pause a bit so that plots are updated
    if is_ipython:
        display.clear_output(wait=True)
        display.display(plt.gcf())

episode_durations = []

def optimize_model():
    if len(memory) < BATCH_SIZE:
        return
    transitions = memory.sample(BATCH_SIZE)
    # Transpose the batch (see https://stackoverflow.com/a/19343/3343043 for
    # detailed explanation). This converts batch-array of Transitions
    # to Transition of batch-arrays.
    batch = Transition(*zip(*transitions))

    # Compute a mask of non-final states and concatenate the batch elements
    # (a final state would've been the one after which simulation ended)
    non_final_mask = torch.tensor(tuple(map(lambda s: s is not None,
                                          batch.next_state)), device=device, dtype=torch.uint8)
    non_final_next_states = torch.cat([s for s in batch.next_state
                                                if s is not None])
    state_batch = torch.cat(batch.state)
    action_batch = torch.cat(batch.action)
    reward_batch = torch.cat(batch.reward)

    # Compute Q(s_t, a) - the model computes Q(s_t), then we select the
    # columns of actions taken. These are the actions which would've been taken
    # for each batch state according to policy_net
    state_action_values = policy_net(state_batch).gather(1, action_batch)

    # Compute V(s_{t+1}) for all next states.
    # Expected values of actions for non_final_next_states are computed based
    # on the "older" target_net; selecting their best reward with max(1)[0].
    # This is merged based on the mask, such that we'll have either the expected
    # state value or 0 in case the state was final.
    next_state_values = torch.zeros(BATCH_SIZE, device=device)
    next_state_values[non_final_mask] = target_net(non_final_next_states).max(1)[0].detach()
    # Compute the expected Q values
    expected_state_action_values = (next_state_values * GAMMA) + reward_batch

    # Compute Huber loss
    loss = F.smooth_l1_loss(state_action_values, expected_state_action_values.unsqueeze(1))

    # Optimize the model
    optimizer.zero_grad()
    loss.backward()
    for param in policy_net.parameters():
        param.grad.data.clamp_(-1, 1)
    optimizer.step()

num_episodes = 50
for i_episode in range(num_episodes):
    # Initialize the environment and state
    env.reset()
    last_screen = get_screen()
    current_screen = get_screen()
    state = current_screen - last_screen
    for t in count():
        # Select and perform an action
        action = select_action(state)
        _, reward, done, _ = env.step(action.item())
        reward = torch.tensor([reward], device=device)

        # Observe new state
        last_screen = current_screen
        current_screen = get_screen()
        if not done:
            next_state = current_screen - last_screen
        else:
            next_state = None

        # Store the transition in memory
        memory.push(state, action, next_state, reward)

        # Move to the next state
        state = next_state

        # Perform one step of the optimization (on the target network)
        optimize_model()
        if done:
            episode_durations.append(t + 1)
            plot_durations()
            break
    # Update the target network, copying all weights and biases in DQN
    if i_episode % TARGET_UPDATE == 0:
        target_net.load_state_dict(policy_net.state_dict())

print('Complete')
env.render()
env.close()
plt.ioff()
plt.show()
```

    Complete

```python

```


# Tutorial 1: Gradient Descent and AutoGrad
**Week 1, Day 2: Linear Deep Learning**

**By Neuromatch Academy**

__Content creators:__ Saeed Salehi, Vladimir Haltakov, Andrew Saxe

__Content reviewers:__ Polina Turishcheva, Antoine De Comite, Kelson Shilling-Scrivo

__Content editors:__ Anoop Kulkarni, Spiros Chavlis

__Production editors:__ Khalid Almubarak, Spiros Chavlis

**Our 2021 Sponsors, including Presenting Sponsor Facebook Reality Labs**

---
# Tutorial Objectives

Day 2 Tutorial 1 will continue on buiding PyTorch skillset and motivate its core functionality, Autograd. In this notebook, we will cover the key concepts and ideas of:

* Gradient descent
* PyTorch Autograd
* PyTorch nn module

```python
# @title Tutorial slides

from IPython.display import IFrame
IFrame(src=f"https://mfr.ca-1.osf.io/render?url=https://osf.io/3qevp/?direct%26mode=render%26action=download%26mode=render", width=854, height=480)
```

These are the slides for all videos in this tutorial. If you want to locally dowload the slides, click [here](https://osf.io/3qevp/download).

---
# Setup

This a GPU-Free tutorial!

```python
# @title Install dependencies
!pip install git+https://github.com/NeuromatchAcademy/evaltools --quiet

from evaltools.airtable import AirtableForm
# init airtable form
atform = AirtableForm('appn7VdPRseSoMXEG','W1D2_T1','https://portal.neuromatchacademy.org/api/redirect/to/9c55f6cb-cdf9-4429-ac1c-ec44fe64c303')
```

```python
# Imports
import torch
import numpy as np
from torch import nn
from math import pi
import matplotlib.pyplot as plt
```

```python
# @title Figure settings
import ipywidgets as widgets       # interactive display
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/content-creation/main/nma.mplstyle")
```

```python
# @title Plotting functions

from mpl_toolkits.axes_grid1 import make_axes_locatable

def ex3_plot(model, x, y, ep, lss):
  f, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))
  ax1.set_title("Regression")
  ax1.plot(x, model(x).detach().numpy(), color='r', label='prediction')
  ax1.scatter(x, y, c='c', label='targets')
  ax1.set_xlabel('x')
  ax1.set_ylabel('y')
  ax1.legend()

  ax2.set_title("Training loss")
  ax2.plot(np.linspace(1, epochs, epochs), losses, color='y')
  ax2.set_xlabel("Epoch")
  ax2.set_ylabel("MSE")

  plt.show()

def ex1_plot(fun_z, fun_dz):
  """Plots the function and gradient vectors

  """
  x, y = np.arange(-3, 3.01, 0.02), np.arange(-3, 3.01, 0.02)
  xx, yy = np.meshgrid(x, y, sparse=True)
  zz = fun_z(xx, yy)
  xg, yg = np.arange(-2.5, 2.6, 0.5), np.arange(-2.5, 2.6, 0.5)
  xxg, yyg = np.meshgrid(xg, yg, sparse=True)
  zxg, zyg = fun_dz(xxg, yyg)

  plt.figure(figsize=(8, 7))
  plt.title("Gradient vectors point towards steepest ascent")
  contplt = plt.contourf(x, y, zz, levels=20)
  plt.quiver(xxg, yyg, zxg, zyg, scale=50, color='r', )
  plt.xlabel('$x$')
  plt.ylabel('$y$')
  ax = plt.gca()
  divider = make_axes_locatable(ax)
  cax = divider.append_axes("right", size="5%", pad=0.05)
  cbar = plt.colorbar(contplt, cax=cax)
  cbar.set_label('$z = h(x, y)$')
  plt.show()
```

```python
# @title Set random seed

# @markdown Executing `set_seed(seed=seed)` you are setting the seed

# for DL its critical to set the random seed so that students can have a
# baseline to compare their results to expected results.
# Read more here: https://pytorch.org/docs/stable/notes/randomness.html

# Call `set_seed` function in the exercises to ensure reproducibility.
import random
import torch

def set_seed(seed=None, seed_torch=True):
  if seed is None:
    seed = np.random.choice(2 ** 32)
  random.seed(seed)
  np.random.seed(seed)
  if seed_torch:
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True

  print(f'Random seed {seed} has been set.')

# In case that `DataLoader` is used
def seed_worker(worker_id):
  worker_seed = torch.initial_seed() % 2**32
  np.random.seed(worker_seed)
  random.seed(worker_seed)
```

```python
# @title Set device (GPU or CPU). Execute `set_device()`
# especially if torch modules used.

# inform the user if the notebook uses GPU or CPU.

def set_device():
  device = "cuda" if torch.cuda.is_available() else "cpu"
  if device != "cuda":
    print("GPU is not enabled in this notebook. \n"
          "If you want to enable it, in the menu under `Runtime` -> \n"
          "`Hardware accelerator.` and select `GPU` from the dropdown menu")
  else:
    print("GPU is enabled in this notebook. \n"
          "If you want to disable it, in the menu under `Runtime` -> \n"
          "`Hardware accelerator.` and select `None` from the dropdown menu")

  return device
```

```python
SEED = 2021
set_seed(seed=SEED)
DEVICE = set_device()
```

---
# Section 0: Introduction

Today, we will go through 3 tutorials. Starting with Gradient Descent, the workhorse of deep learning algorithms, in this tutorial. The second tutorial will help us build a better intuition about neural networks and basic hyper-parameters. Finally, in tutorial 3, we learn about the learning dynamics, what the (a good) deep network is learning, and why sometimes they may perform poorly.

```python
# @title Video 0: Introduction
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1Qf4y1578t", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"i7djAv2jnzY", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

#add event to airtable
atform.add_event('Video 0:Introduction')

display(out)
```

---
# Section 1: Gradient Descent Algorithm

*Time estimate: ~30-45 mins*

Since the goal of most learning algorithms is **minimizing the risk (also known as the cost or loss) function**, optimization is often the core of most machine learning techniques! The gradient descent algorithm, along with its variations such as stochastic gradient descent, is one of the most powerful and popular optimization methods used for deep learning. Today we will introduce the basics, but you will learn much more about Optimization in the coming days (Week 1 Day 4).

## Section 1.1: Gradients & Steepest Ascent

```python
# @title Video 1: Gradient Descent
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1Pq4y1p7em", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"UwgA_SgG0TM", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

#add event to airtable
atform.add_event('Video 1: Gradient Descent')

display(out)
```

Before introducing the gradient descent algorithm, let's review a very important property of gradients. The gradient of a function always points in the direction of the steepest ascent. The following exercise will help clarify this.

### Analytical Exercise 1.1: Gradient vector (Optional)

Given the following function:

\begin{equation}
z = h(x, y) = \sin(x^2 + y^2)
\end{equation}

find the gradient vector:

\begin{equation}
  \begin{bmatrix}
  \dfrac{\partial z}{\partial x} \\ \\ \dfrac{\partial z}{\partial y}
  \end{bmatrix}
\end{equation}

*hint: use the chain rule!*

**Chain rule**: For a composite function $F(x) = g(h(x)) \equiv (g \circ h)(x)$:

\begin{equation}
F'(x) = g'(h(x)) \cdot h'(x)
\end{equation}

or differently denoted:

\begin{equation}
\frac{dF}{dx} = \frac{dg}{dh} ~ \frac{dh}{dx}
\end{equation}

---
#### Solution:
We can rewrite the function as a composite function:

\begin{equation}
z = f\left( g(x,y) \right), ~~ f(u) = \sin(u), ~~ g(x, y) = x^2 + y^2
\end{equation}

Using the [chain rule](https://en.wikipedia.org/wiki/Chain_rule):

\begin{align}
\dfrac{\partial z}{\partial x} &= \dfrac{\partial f}{\partial g} \dfrac{\partial g}{\partial x} = \cos(g(x,y)) ~ (2x) = \cos(x^2 + y^2) \cdot 2x \\ \\
\dfrac{\partial z}{\partial y} &= \dfrac{\partial f}{\partial g} \dfrac{\partial g}{\partial y} = \cos(g(x,y)) ~ (2y) = \cos(x^2 + y^2) \cdot 2y
\end{align}

### Coding Exercise 1.1: Gradient Vector

Implement (complete) the function which returns the gradient vector for $z=\sin(x^2 + y^2)$.

```python
def fun_z(x, y):
  """Function sin(x^2 + y^2)

  Args:
    x (float, np.ndarray): variable x
    y (float, np.ndarray): variable y

  Return:
    z (float, np.ndarray): sin(x^2 + y^2)
  """
  z = np.sin(x**2 + y**2)
  return z

def fun_dz(x, y):
  """Function sin(x^2 + y^2)

  Args:
    x (float, np.ndarray): variable x
    y (float, np.ndarray): variable y

  Return:
    (tuple): gradient vector for sin(x^2 + y^2)
  """
  #################################################
  ## Implement the function which returns gradient vector
  ## Complete the partial derivatives dz_dx and dz_dy
  # Complete the function and remove or comment the line below
  raise NotImplementedError("Gradient function `fun_dz`")
  #################################################
  dz_dx = ...
  dz_dy = ...
  return (dz_dx, dz_dy)

#add event to airtable
atform.add_event('Coding Exercise 1.1: Gradient Vector')

## Uncomment to run
# ex1_plot(fun_z, fun_dz)
```

```python
# to_remove solution
def fun_z(x, y):
  """Function sin(x^2 + y^2)

  Args:
    x (float, np.ndarray): variable x
    y (float, np.ndarray): variable y

  Return:
    z (float, np.ndarray): sin(x^2 + y^2)
  """
  z = np.sin(x**2 + y**2)
  return z

def fun_dz(x, y):
  """Function sin(x^2 + y^2)

  Args:
    x (float, np.ndarray): variable x
    y (float, np.ndarray): variable y

  Return:
    (tuple): gradient vector for sin(x^2 + y^2)
  """
  dz_dx = 2 * x * np.cos(x**2 + y**2)
  dz_dy = 2 * y * np.cos(x**2 + y**2)
  return (dz_dx, dz_dy)

#add event to airtable
atform.add_event('Coding Exercise 1.1: Gradient Vector')

## Uncomment to run
with plt.xkcd():
  ex1_plot(fun_z, fun_dz)
```

We can see from the plot that for any given $x_0$ and $y_0$, the gradient vector $\left[ \dfrac{\partial z}{\partial x}, \dfrac{\partial z}{\partial y}\right]^{\top}_{(x_0, y_0)}$ points in the direction of $x$ and $y$ for which $z$ increases the most. It is important to note that gradient vectors only see their local values, not the whole landscape! Also, length (size) of each vector, which indicates the steepness of the function, can be very small near local plateaus (i.e. minima or maxima).

Thus, we can simply use the aforementioned formula to find the local minima.

In 1847, Augustin-Louis Cauchy used **negative of gradients**  to develop the Gradient Descent algorithm as an **iterative** method to **minimize** a **continuous** and (ideally) **differentiable function** of **many variables**.

```python
# @title Video 2: Gradient Descent - Discussion
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1Rf4y157bw", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"8s22ffAfGwI", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

#add event to airtable
atform.add_event('Video 2: Gradient Descent ')

display(out)
```

## Section 1.2: Gradient Descent Algorithm

Let $f(\mathbf{w}): \mathbb{R}^d \rightarrow \mathbb{R}$ be a differentiable function. Gradient Descent is an iterative algorithm for minimizing the function $f$, starting with an initial value for variables $\mathbf{w}$, taking steps of size $\eta$ (learning rate) in the direction of the negative gradient at the current point to update the variables $\mathbf{w}$.

\begin{equation}
\mathbf{w}^{(t+1)} = \mathbf{w}^{(t)} - \eta \nabla f \left( \mathbf{w}^{(t)} \right)
\end{equation}

where $\eta > 0$ and $\nabla f (\mathbf{w})= \left( \frac{\partial f(\mathbf{w})}{\partial w_1}, ..., \frac{\partial f(\mathbf{w})}{\partial w_d} \right)$. Since negative gradients always point locally in the direction of steepest descent, the algorithm makes small steps at each point **towards** the minimum.

**Vanilla Algorithm**

---
> **Inputs:** initial guess $\mathbf{w}^{(0)}$, step size $\eta > 0$, number of steps $T$.

> **For** $t = 0, 1, 2, \dots , T-1$ **do** \
$\qquad$ $\mathbf{w}^{(t+1)} = \mathbf{w}^{(t)} - \eta \nabla f \left( \mathbf{w}^{(t)} \right)$\
**end**

> **Return:** $\mathbf{w}^{(t+1)}$

---

Hence, all we need is to calculate the gradient of the loss function with respect to the learnable parameters (i.e., weights):

\begin{equation}
\dfrac{\partial Loss}{\partial \mathbf{w}} = \left[ \dfrac{\partial Loss}{\partial w_1}, \dfrac{\partial Loss}{\partial w_2} , \dots, \dfrac{\partial Loss}{\partial w_d} \right]^{\top}
\end{equation}

### Analytical Exercise 1.2: Gradients

Given $f(x, y, z) = \tanh \left( \ln \left[1 + z \frac{2x}{sin(y)} \right] \right)$, how easy is it to derive $\dfrac{\partial f}{\partial x}$, $\dfrac{\partial f}{\partial y}$ and $\dfrac{\partial f}{\partial z}$?

**Hint:** You don't have to actually calculate them!

## Section 1.3: Computational Graphs and Backprop

```python
# @title Video 3: Computational Graph
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1c64y1B7ZG", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"2z1YX5PonV4", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

#add event to airtable
atform.add_event('Video 3: Computational Graph ')

display(out)
```

 *Exercise 1.2* is an example of how overwhelming the derivation of gradients can get, as the number of variables and nested functions increases. This function is still extraordinarily simple compared to the loss functions of modern neural networks. So how can we (as well as PyTorch and similar frameworks) approach such beasts?

Let’s look at the function again:

\begin{equation}
f(x, y, z) = \tanh \left(\ln \left[1 + z \frac{2x}{sin(y)} \right] \right)
\end{equation}

We can build a so-called computational graph (shown below) to break the original function into smaller and more approachable expressions.

Starting from $x$, $y$, and $z$ and following the arrows and expressions, you would see that our graph returns the same function as $f$. It does so by calculating intermediate variables $a,b,c,d,$ and $e$. This is called the **forward pass**.

Now, let’s start from $f$, and work our way against the arrows while calculating the gradient of each expression as we go. This is called the **backward pass**, from which the **backpropagation of errors** algorithm gets its name.

By breaking the computation into simple operations on intermediate variables, we can use the chain rule to calculate any gradient:

\begin{equation}
\dfrac{\partial f}{\partial x} = \dfrac{\partial f}{\partial e}~\dfrac{\partial e}{\partial d}~\dfrac{\partial d}{\partial c}~\dfrac{\partial c}{\partial a}~\dfrac{\partial a}{\partial x} = \left( 1-\tanh^2(e) \right) \cdot \frac{1}{d+1}\cdot z \cdot \frac{1}{b} \cdot 2
\end{equation}

Conveniently, the values for $e$, $b$, and $d$ are available to us from when we did the forward pass through the graph. That is, the partial derivatives have simple expressions in terms of the intermediate variables $a,b,c,d,e$ that we calculated and stored during the forward pass.

### Analytical Exercise 1.3: Chain Rule (Optional)
For the function above, calculate the $\dfrac{\partial f}{\partial y}$ using the computational graph and chain rule.

---
#### Solution:
\begin{equation}
\dfrac{\partial f}{\partial y} = \dfrac{\partial f}{\partial e}~\dfrac{\partial e}{\partial d}~\dfrac{\partial d}{\partial c}~\dfrac{\partial c}{\partial b}~\dfrac{\partial b}{\partial y} = \left( 1-\tanh^2(e) \right) \cdot \frac{1}{d+1}\cdot z \cdot \frac{-a}{b^2} \cdot \cos(y)
\end{equation}

For more: [Calculus on Computational Graphs: Backpropagation](https://colah.github.io/posts/2015-08-Backprop/)

---
# Section 2: PyTorch AutoGrad

*Time estimate: ~30-45 mins*

```python
# @title Video 4: Auto-Differentiation
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1UP4y1s7gv", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"IBYFCNyBcF8", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

#add event to airtable
atform.add_event('Video 4: Auto-Differentiation ')

display(out)
```

Deep learning frameworks such as PyTorch, JAX, and TensorFlow come with a very efficient and sophisticated set of algorithms, commonly known as Automatic Differentiation. AutoGrad is PyTorch's automatic differentiation engine. Here we start by covering the essentials of AutoGrad, and you will learn more in the coming days.

## Section 2.1: Forward Propagation

Everything starts with the forward propagation (pass). PyTorch tracks all the instructions, as we declare the variables and operations, and it builds the graph when we call the `.backward()` pass. PyTorch rebuilds the graph every time we iterate or change it (or simply put, PyTorch uses a dynamic graph).

For gradient descent, it is only required to have the gradients of cost function with respect to the variables we wish to learn. These variables are often called "learnable / trainable parameters" or simply "parameters" in PyTorch. In neural nets, weights and biases are often the learnable parameters.

### Coding Exercise 2.1: Buiding a Computational Graph

In PyTorch, to indicate that a certain tensor contains learnable parameters, we can set the optional argument `requires_grad` to `True`. PyTorch will then track every operation using this tensor while configuring the computational graph. For this exercise, use the provided tensors to build the following graph, which implements a single neuron with scalar input and output.

```python
class SimpleGraph:
  def __init__(self, w, b):
    """Initializing the SimpleGraph

    Args:
      w (float): initial value for weight
      b (float): initial value for bias
    """
    assert isinstance(w, float)
    assert isinstance(b, float)
    self.w = torch.tensor([w], requires_grad=True)
    self.b = torch.tensor([b], requires_grad=True)

  def forward(self, x):
    """Forward pass

    Args:
      x (torch.Tensor): 1D tensor of features

    Returns:
      torch.Tensor: model predictions
    """
    assert isinstance(x, torch.Tensor)
    #################################################
    ## Implement the the forward pass to calculate prediction
    ## Note that prediction is not the loss, but the value after `tanh`
    # Complete the function and remove or comment the line below
    raise NotImplementedError("Forward Pass `forward`")
    #################################################
    prediction = ...
    return prediction

def sq_loss(y_true, y_prediction):
  """L2 loss function

  Args:
    y_true (torch.Tensor): 1D tensor of target labels
    y_prediction (torch.Tensor): 1D tensor of predictions

  Returns:
    torch.Tensor: L2-loss (squared error)
  """
  assert isinstance(y_true, torch.Tensor)
  assert isinstance(y_prediction, torch.Tensor)
  #################################################
  ## Implement the L2-loss (squred error) given true label and prediction
  # Complete the function and remove or comment the line below
  raise NotImplementedError("Loss function `sq_loss`")
  #################################################
  loss = ...
  return loss

# add event to airtable
atform.add_event('Coding Exercise 2.1: Computational Graph ')

feature = torch.tensor([1])  # input tensor
target = torch.tensor([7])  # target tensor
## Uncomment to run
# simple_graph = SimpleGraph(-0.5, 0.5)
# print(f"initial weight = {simple_graph.w.item()}, "
#       f"\ninitial bias = {simple_graph.b.item()}")
# prediction = simple_graph.forward(feature)
# square_loss = sq_loss(target, prediction)
# print(f"for x={feature.item()} and y={target.item()}, "
#       f"prediction={prediction.item()}, and L2 Loss = {square_loss.item()}")
```

```python
# to_remove solution
class SimpleGraph:
  def __init__(self, w, b):
    """Initializing the SimpleGraph

    Args:
      w (float): initial value for weight
      b (float): initial value for bias
    """
    assert isinstance(w, float)
    assert isinstance(b, float)
    self.w = torch.tensor([w], requires_grad=True)
    self.b = torch.tensor([b], requires_grad=True)

  def forward(self, x):
    """Forward pass

    Args:
      x (torch.Tensor): 1D tensor of features

    Returns:
      torch.Tensor: model predictions
    """
    assert isinstance(x, torch.Tensor)
    prediction = torch.tanh(x * self.w + self.b)
    return prediction

def sq_loss(y_true, y_prediction):
  """L2 loss function

  Args:
    y_true (torch.Tensor): 1D tensor of target labels
    y_prediction (torch.Tensor): 1D tensor of predictions

  Returns:
    torch.Tensor: L2-loss (squared error)
  """
  assert isinstance(y_true, torch.Tensor)
  assert isinstance(y_prediction, torch.Tensor)
  loss = (y_true - y_prediction)**2
  return loss

# add event to airtable
atform.add_event('Coding Exercise 2.1: Computational Graph ')

feature = torch.tensor([1])  # input tensor
target = torch.tensor([7])  # target tensor
## Uncomment to run
simple_graph = SimpleGraph(-0.5, 0.5)
print(f"initial weight = {simple_graph.w.item()}, "
      f"\ninitial bias = {simple_graph.b.item()}")
prediction = simple_graph.forward(feature)
square_loss = sq_loss(target, prediction)
print(f"for x={feature.item()} and y={target.item()}, "
      f"prediction={prediction.item()}, and L2 Loss = {square_loss.item()}")
```

It is important to appreciate the fact that PyTorch can follow our operations as we arbitrarily go through classes and functions.

## Section 2.2: Backward Propagation

Here is where all the magic lies. In PyTorch, `Tensor` and `Function` are interconnected and build up an acyclic graph, that encodes a complete history of computation. Each variable has a `grad_fn` attribute that references a function that has created the Tensor (except for Tensors created by the user - these have `None` as `grad_fn`).  The example below shows that the tensor `c = a + b` is created by the `Add` operation and the gradient function is the object `<AddBackward...>`. Replace `+` with other single operations (e.g., `c = a * b` or `c = torch.sin(a)`) and examine the results.

```python
a = torch.tensor([1.0], requires_grad=True)
b = torch.tensor([-1.0], requires_grad=True)
c = a + b
print(f'Gradient function = {c.grad_fn}')
```

For more complex functions, printing the `grad_fn` would only show the last operation, even though the object tracks all the operations up to that point:

```python
print(f'Gradient function for prediction = {prediction.grad_fn}')
print(f'Gradient function for loss = {square_loss.grad_fn}')
```

Now let's kick off the backward pass to calculate the gradients by calling `.backward()` on the tensor we wish to initiate the backpropagation from. Often, `.backward()` is called on the loss, which is the last node on the graph. Before doing that, let's calculate the loss gradients by hand:

$$\frac{\partial{loss}}{\partial{w}} = - 2 x (y_t - y_p)(1 - y_p^2)$$

$$\frac{\partial{loss}}{\partial{b}} = - 2 (y_t - y_p)(1 - y_p^2)$$

Where $y_t$ is the target (true label), and $y_p$ is the prediction (model output). We can then compare it to PyTorch gradients, which can be obtained by calling `.grad` on the relevant tensors.

**Important Notes:**
* Learnable parameters (i.e. `requires_grad` tensors) are "contagious". Let's look at a simple example: `Y = W @ X`, where `X` is the feature tensors and `W` is the weight tensor (learnable parameters, `requires_grad`), the newly generated output tensor `Y` will be also `requires_grad`. So any operation that is applied to `Y` will be part of the computational graph. Therefore, if we need to plot or store a tensor that is `requires_grad`, we must first `.detach()` it from the graph by calling the `.detach()` method on that tensor.

* `.backward()` accumulates gradients in the leaf nodes (i.e., the input nodes to the node of interest). We can call `.zero_grad()` on the loss or optimizer to zero out all `.grad` attributes (see [autograd.backward](https://pytorch.org/docs/stable/autograd.html#torch.autograd.backward) for more information).

* Recall that in python we can access variables and associated methods with `.method_name`. You can use the command `dir(my_object)` to observe all variables and associated methods to your object, e.g., `dir(simple_graph.w)`.

```python
# analytical gradients (remember detaching)
ana_dloss_dw = - 2 * feature * (target - prediction.detach())*(1 - prediction.detach()**2)
ana_dloss_db = - 2 * (target - prediction.detach())*(1 - prediction.detach()**2)

square_loss.backward()  # first we should call the backward to build the graph
autograd_dloss_dw = simple_graph.w.grad  # we calculate the derivative w.r.t weights
autograd_dloss_db = simple_graph.b.grad  # we calculate the derivative w.r.t bias

print(ana_dloss_dw == autograd_dloss_dw)
print(ana_dloss_db == autograd_dloss_db)
```

## References and more:

* [A gentle introduction to torch.autograd](https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html)

* [Automatic Differentiation package - torch.autograd](https://pytorch.org/docs/stable/autograd.html)

* [Autograd mechanics](https://pytorch.org/docs/stable/notes/autograd.html)

* [Automatic Differentiation with torch.autograd](https://pytorch.org/tutorials/beginner/basics/autogradqs_tutorial.html)

---
# Section 3: PyTorch's Neural Net module (`nn.Module`)

*Time estimate: ~30 mins*

```python
# @title Video 5: PyTorch `nn` module
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1MU4y1H7WH", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"jzTbQACq7KE", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

#add event to airtable
atform.add_event('Video 5: PyTorch `nn` module')

display(out)
```

PyTorch provides us with ready-to-use neural network building blocks, such as layers (e.g., linear, recurrent, etc.), different activation and loss functions, and much more, packed in the [`torch.nn`](https://pytorch.org/docs/stable/nn.html) module. If we build a neural network using `torch.nn` layers, the weights and biases are already in `requires_grad` mode and will be registered as model parameters.

For training, we need three things:

* **Model parameters:** Model parameters refer to all the learnable parameters of the model, which are accessible by calling `.parameters()` on the model. Please note that NOT all the `requires_grad` tensors are seen as model parameters. To create a custom model parameter, we can use [`nn.Parameter`](https://pytorch.org/docs/stable/generated/torch.nn.parameter.Parameter.html) (*A kind of Tensor that is to be considered a module parameter*).

* **Loss function:** The loss that we are going to be optimizing, which is often combined with regularization terms (coming up in few days).

* **Optimizer:** PyTorch provides us with many optimization methods (different versions of gradient descent). Optimizer holds the current state of the model and by calling the `step()` method, will update the parameters based on the computed gradients.

You will learn more details about choosing the right model architecture, loss function, and optimizer later in the course.

## Section 3.1: Training loop in PyTorch

We use a regression problem to study the training loop in PyTorch.

The task is to train a wide nonlinear (using $\tanh$ activation function) neural net for a simple $\sin$ regression task. Wide neural networks are thought to be really good at generalization.

```python
# @markdown #### Generate the sample dataset
set_seed(seed=SEED)
n_samples = 32
inputs = torch.linspace(-1.0, 1.0, n_samples).reshape(n_samples, 1)
noise = torch.randn(n_samples, 1) / 4
targets = torch.sin(pi * inputs) + noise
plt.figure(figsize=(8, 5))
plt.scatter(inputs, targets, c='c')
plt.xlabel('x (inputs)')
plt.ylabel('y (targets)')
plt.show()
```

Let's define a very wide (512 neurons) neural net with one hidden layer and `nn.Tanh()` activation function.

```python
## A Wide neural network with a single hidden layer
class WideNet(nn.Module):

  def __init__(self):
    """Initializing the WideNet
    """
    n_cells = 512
    super().__init__()
    self.layers = nn.Sequential(
        nn.Linear(1, n_cells),
        nn.Tanh(),
        nn.Linear(n_cells, 1),
    )

  def forward(self, x):
    """Forward pass

    Args:
      x (torch.Tensor): 2D tensor of features

    Returns:
      torch.Tensor: model predictions
    """
    return self.layers(x)
```

We can now create an instance of our neural net and print its parameters.

```python
# creating an instance
set_seed(seed=SEED)
wide_net = WideNet()
print(wide_net)
```

```python
# Create a mse loss function
loss_function = nn.MSELoss()

# Stochstic Gradient Descent optimizer (you will learn about momentum soon)
lr = 0.003  # learning rate
sgd_optimizer = torch.optim.SGD(wide_net.parameters(), lr=lr, momentum=0.9)
```

The training process in PyTorch is interactive - you can perform training iterations as you wish and inspect the results after each iteration.

Let's perform one training iteration. You can run the cell multiple times and see how the parameters are being updated and the loss is reducing. This code block is the core of everything to come: please make sure you go line-by-line through all the commands and discuss their purpose with your pod.

```python
# Reset all gradients to zero
sgd_optimizer.zero_grad()

# Forward pass (Compute the output of the model on the features (inputs))
prediction = wide_net(inputs)

# Compute the loss
loss = loss_function(prediction, targets)
print(f'Loss: {loss.item()}')

# Perform backpropagation to build the graph and compute the gradients
loss.backward()

# Optimizer takes a tiny step in the steepest direction (negative of gradient)
# and "updates" the weights and biases of the network
sgd_optimizer.step()
```

### Coding Exercise 3.1: Training Loop
Using everything we've learned so far, we ask you to complete the `train` function below.

```python
def train(features, labels, model, loss_fun, optimizer, n_epochs):

  """Training function

  Args:
    features (torch.Tensor): features (input) with shape torch.Size([n_samples, 1])
    labels (torch.Tensor): labels (targets) with shape torch.Size([n_samples, 1])
    model (torch nn.Module): the neural network
    loss_fun (function): loss function
    optimizer(function): optimizer
    n_epochs (int): number of training iterations

  Returns:
    list: record (evolution) of training losses
  """
  loss_record = []  # keeping recods of loss

  for i in range(n_epochs):
    #################################################
    ## Implement the missing parts of the training loop
    # Complete the function and remove or comment the line below
    raise NotImplementedError("Training loop `train`")
    #################################################
    ...  # set gradients to 0
    predictions = ...  # Compute model prediction (output)
    loss = ...  # Compute the loss
    ...  # Compute gradients (backward pass)
    ...  # update parameters (optimizer takes a step)

    loss_record.append(loss.item())
  return loss_record

#add event to airtable
atform.add_event('Coding Exercise 3.1: Training Loop')

set_seed(seed=2021)
epochs = 1847 # Cauchy, Exercices d'analyse et de physique mathematique (1847)
## Uncomment to run
# losses = train(inputs, targets, wide_net, loss_function, sgd_optimizer, epochs)
# ex3_plot(wide_net, inputs, targets, epochs, losses)
```

```python
# to_remove solution
def train(features, labels, model, loss_fun, optimizer, n_epochs):

  """Training function

  Args:
    features (torch.Tensor): features (input) with shape torch.Size([n_samples, 1])
    labels (torch.Tensor): labels (targets) with shape torch.Size([n_samples, 1])
    model (torch nn.Module): the neural network
    loss_fun (function): loss function
    optimizer(function): optimizer
    n_epochs (int): number of training iterations

  Returns:
    list: record (evolution) of training losses
  """
  loss_record = []  # keeping recods of loss

  for i in range(n_epochs):
    optimizer.zero_grad()  # set gradients to 0
    predictions = model(features)  # Compute model prediction (output)
    loss = loss_fun(predictions, labels)  # Compute the loss
    loss.backward()  # Compute gradients (backward pass)
    optimizer.step()  # update parameters (optimizer takes a step)

    loss_record.append(loss.item())
  return loss_record

#add event to airtable
atform.add_event('Coding Exercise 3.1: Training Loop')

set_seed(seed=2021)
epochs = 1847 # Cauchy, Exercices d'analyse et de physique mathematique (1847)
## Uncomment to run
losses = train(inputs, targets, wide_net, loss_function, sgd_optimizer, epochs)
with plt.xkcd():
  ex3_plot(wide_net, inputs, targets, epochs, losses)
```

---
# Summary

In this tutorial we covered one of the most basic concepts of deep learning; the computational graph and how a network learns via gradient descent and the backpropagation algorithm. We have seen all of these using PyTorch modules and we compared the analytical solutions with the ones provided directly by the PyTorch module.

```python
# @title Video 6: Tutorial 1 Wrap-up
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1Pg41177VU", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"TvZURbcnXc4", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

#add event to airtable
atform.add_event('Video 6: Tutorial 1 Wrap-up')

display(out)
```

```python
# @title Airtable Submission Link
from IPython import display as IPyDisplay
IPyDisplay.HTML(
    f"""
  <div>
    <a href= "{atform.url()}" target="_blank">
    </a>
    </div>""" )
```


# Model selection assessment

The objective of this notebook is to compare the different classical model selection approaches, and to learn one more adapted to clonesig.

A few notations
- $N$ is the number of observations
- $J$ is the number of clones, $D_J$ is the number of parameters for a model with $J$ clones
- $\ell$ is the log likelihood of the model, namely $\ell(\mathcal{D};\boldsymbol{\theta}) = \sum_{n=1}^N \log \left[ \sum_{u=1}^U \sum_{s=1}^L \sum_{m=1}^{M_{max_n}} \xi_u \mu_{st} \pi_{us} \nu_{mn} BB(B_n|D_n, C_n, p, U_n=u, M_n=m;\boldsymbol{\theta})\right]$

We have $D_J = J * (L - 1 + 2)$, with $L$ the number of signatures. Indeed, for each clone, we have $L-1$ parameters for the signature proportions ($\pi$), the frequency of the clone ($\phi_u$), and the proportion of the clone $\xi_u$. We have to remove 1 because $\sum_{u=1}^J \xi_u=1$, and add 1 for the overdispersion parameter $\tau$.

The definitions of the different indices have been adapted so that they are all in the form of $\ell(\mathcal{D};\boldsymbol{\theta}_J) - pen(J)$ for easier comparison, following notations from [Assessing a Mixture Model for Clustering with the
Integrated Classification Likelihood -
Christophe Biernacki, Gilles Celeux, Gérard Govaert, 2006](https://hal.inria.fr/inria-00073163/document)

**BIC**
$$BIC = \ell(\mathcal{D};\boldsymbol{\theta}_J) - \frac{D_J}{2}\log N$$

**AIC**
$$AIC = \ell(\mathcal{D};\boldsymbol{\theta}_J) - D_J $$

**AICC**
some adapted version of AIC for small N (might be relevant for TCGA)
$$AICc = AIC + \frac{D_J(D_J+1)}{n-D_J -1}$$

**ICL**
$$ICL=BIC + H = BIC - \sum_{n=1}^N \sum_{u=1}^J \sum_{s=1}^L \sum_{m=1}^{M_{max_n}} q_{un}r_{nus} v_{mnu}  \log (q_{un}r_{nus}v_{mnu})$$

In fact I don't understand anything, there are a lot of different definitions.... This is the definition in [EM for mixtures - Initialization requires special care](https://hal.inria.fr/hal-01113242/document) but there are others:

in [Slope heuristics for variable selection and clustering via
Gaussian mixture](https://hal.inria.fr/inria-00284620v2/document) it is $$ICL_{qn}=BIC + H = BIC - \frac{\sum_{n=1}^N \sum_{u=1}^J \sum_{s=1}^L \sum_{m=1}^{M_{max_n}} U_{un} S_{uns} M_{num} \log (q_{un}r_{nus}v_{mnu})}{N}$$ with $U_{un}=1$ if $u=\mbox{argmax}_v q_{vn}$ and $S_{uns}=1$ if $u=\mbox{argmax}_v q_{vn}$ and $s=\mbox{argmax}_t r_{nut}$, and $M_{num}=1$ if $u=\mbox{argmax}_v q_{vn}$ and $m=\mbox{argmax}_{m'} v_{m'nu}$.

in [Assessing a Mixture Model for Clustering with the
Integrated Classification Likelihood](https://hal.inria.fr/inria-00073163/document) it is $$ICL_{q}=BIC + H = BIC - \sum_{n=1}^N \sum_{u=1}^J \sum_{s=1}^L \sum_{m=1}^{M_{max_n}} U_{un} S_{uns} M_{num} \log (q_{un}r_{nus} v_{mnu})$$

Clearly the computation in this last paper is correct, so let's keep it, and the normalized version. I named them ICL_{q} and ICL_{qn} to avoid confusion. Obviously, the MOBSTER paper (new ITH method using ICL) made a mistake about that in the manuscript...

**SH (slope heuristics)**
$$SH=\ell(\mathcal{D};\boldsymbol{\theta}_J) -\rho D_J $$
where $\rho_{opt}= 2\rho_{min}$ and $\rho_{min}$ is estimated as is the slope of $\ell(\mathcal{D};\boldsymbol{\theta}_J)$ with respect to $D_J$

**"elbow" detection**
the idea is to compute the discrete second derivative as a proxy for the cuvature
$$ x[i+1] + x[i-1] - 2 * x[i]$$

```python
%matplotlib inline

import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np

from clonesig.data_loader import MAFLoader, SimLoader, beta_binomial
from clonesig.estimator import _get_projected
from clonesig.estimator import *
import time
from sklearn.cluster import KMeans
from sklearn import linear_model
from scipy.stats import pearsonr, linregress
from sklearn.model_selection import GroupKFold
from sklearn.svm import LinearSVC, SVC
from scipy.spatial.distance import cosine
pd.options.display.max_columns = 200

phd_folder_path = '/Users/JudithAbecassis/Documents/PhD'
os.chdir('{}/TCGA_signatures'.format(phd_folder_path))
import warnings
warnings.filterwarnings("ignore")
```

    running on server, otherwise please investigate
    running on server, otherwise please investigate

```python
from IPython.display import HTML

HTML('''
<form action="javascript:code_toggle()"><input type="submit" value="Click here to toggle on/off the raw code."></form>''')
```

<form action="javascript:code_toggle()"><input type="submit" value="Click here to toggle on/off the raw code."></form>

## Penalty learning

We have generated a set of 6300 simulated dataset with the following characteristics
- each example has a cancertype, and a subset of active signatures drawn at random from this cancer type active signatures, as reported in the table in SigProfiler package.
- each example has a number of mutations among 100, 300, 600, 1000 or 5000
- each example has a theoretic proportion of genome that is diploid of 0, 20, 40, 60, 80, 100. For each simulation, the actual average major copy number was computed
- each example has a different number of clones, between 1 and 6
- for each clone, a different exposure among the active signatures was computed. Those exposure vectors were simulated such that the cosine distance of the mixture was greater than 0.05 between 2 successive clones. The min, max and average distance between the profiles of all pairs of clones was computed.

For each cancer type (35 distinct), there were 180 simulations (1 for each combination of number of clones, number of mutations, and percentage of the genome that is diploid). For each simulation, clonesig was applied twice, one with only the subset of signatures active in the cancer type, and one with the all 65 signatures, with between 1 and 10 clones (included).

All simulations are otherwise independant.

### Learn several models: see if we should have CN or no CN, and number of signatures or degree of freedom
In most models, the penalty depends on $D_J$, the number of parameters in the model. This can be revisited in terms of degree of freedom. The parameters $\phi$, $\xi$ and $\rho$ determining the CCFs and proportions of the different clones in the mixture are unquestionable. Regarding the signatures however, one can notice that the signatures are neither orthogonal (some signatures are very similar), nor independant (some signatures are associated with the same underlying biological process). We have approximated the degree of freedom by the number of non-zero eigenvalues of the cosine similarity matrix of signatures (in practice, we considered the number of eigenvalues greater than $0.5$ in absolute value). Another source of degree of freedom is the copy number. Indeed, for each observed mutation, several values of the number of mutated copies are considered, so if the maximal average multiplicity for mutations in the sample is $M_{max_{avg}}$, a unique clone CCF corresponds to $M_{max_{avg}}$ possible VAFs, adding some freedom to the model.

We have hence considered 4 possible shapes for $D_J$, indexed with letters A to D.
\begin{align}
D_J^A &= J \times (L + 1) \times M_{max_{avg}} \times \log(N) \\
D_J^B &= J \times (L + 1) \times \log(N) \\
D_J^C &= J \times (\mbox{dof}_L + 1) \times M_{max_{avg}} \times \log(N) \\
D_J^D &= J \times (\mbox{dof}_L + 1) \times \log(N) \\
\end{align}

Moreover, if we consider the variations of the degree of freedom associated with $L$ signatures, $\mbox{dof}_L$, as a function of $L$ for the 35 available cancer types, and for the all 65 signatures, we note that there is a gap, as the maximal number of signatures in one cancer type is 19, and that the slope seems different for a subset or for all the signatures. The dependency is quite different, so we have attempted to lean a single model or 2 different models for each shape of $D_J$: one for the case where Clonesig is run with only cancer type-specific signatures, and one for the case where Clonesig is run with all the 65 signatures.

To learn the penalty, the data was divided in train and test sets (

```python
sig_file_path = 'external_data/sigProfiler_SBS_signatures_2018_03_28.csv'
cancer_type_sig_filename = 'external_data/curated_match_signature_cancertype_tcgawes_literature.csv'

# open the matrix describing the signatures
SIG = pd.read_csv(sig_file_path)
SIG_MATRIX = SIG.values[:, 2:].astype(float).T
L, K = SIG_MATRIX.shape
NEW_SIG_MATRIX = SIG_MATRIX + 10**-20 * (SIG_MATRIX == 0)
MU = NEW_SIG_MATRIX / NEW_SIG_MATRIX.sum(axis=1)[:, np.newaxis]

# get the signatures specific of the cancer type
cancer_type_df = pd.read_csv(cancer_type_sig_filename, sep='\t', index_col=0)
cancer_type_sig = cancer_type_df.values
#select = cancer_type_sig[cancer_type, :]
#subMU = MU[select.astype(bool), :]

nb_sig_list = list()
dof_list = list()
median_list = list()
mean_list = list()
for i, cancer_type in enumerate(cancer_type_df.columns.tolist()):
    select = cancer_type_sig[:, i]
    subMU = MU[select.astype(bool), :]
    ev, _ = np.linalg.eig(1-sp.spatial.distance.squareform(sp.spatial.distance.pdist(subMU, 'cosine')))
    print(cancer_type, len(ev), sum(ev>0.5))
    subL = subMU.shape[0]
    nb_sig_list.append(subL)
    dof_list.append(sum(ev>0.5))
    dist_list = list()
    for j in range(10000):
        #nb_active_sig = np.min((np.random.poisson(int(subL/2)) + 1, subL))
        nb_active_sig = 4
        #nb_active_sig = subL
        active_signatures = np.random.choice(subL, nb_active_sig, replace=False)
        pi = np.zeros((2, subL))
        pi[0, active_signatures] = np.random.dirichlet(alpha=np.ones(nb_active_sig))
        pi[1, active_signatures] = np.random.dirichlet(alpha=np.ones(nb_active_sig))
        dist_list.append(cosine(pi[0, :].dot(subMU), pi[1, :].dot(subMU)))
    median_list.append(np.median(dist_list))
    mean_list.append(np.mean(dist_list))
df_cancer_type = pd.DataFrame(data=np.array([nb_sig_list, dof_list, median_list, mean_list]).T, columns=['nb_sig', 'dof', 'median_dist', 'mean_list'], index=cancer_type_df.columns)
```

    ACC 7 6
    BLCA 8 5
    BRCA 20 12
    CESC 6 5
    CHOL 14 10
    COADREAD 18 11
    DLBC 17 10
    ESCA 11 8
    GBM 5 3
    HNSC 15 10
    KICH 12 8
    KIRC 13 8
    KIRP 6 4
    LGG 10 7
    LIHC 23 10
    LUAD 12 8
    LUSC 7 5
    MESO 10 7
    OV 13 7
    PAAD 19 13
    PCPG 13 9
    PRAD 15 8
    SARC 16 10
    SKCM 12 10
    STAD 16 10
    TGCT 20 11
    THCA 11 7
    THYM 18 10
    UCEC 13 8
    UCS 14 9
    UVM 14 8

```python
MU_act = MU[cancer_type_sig.sum(axis=1).astype(bool), :]

sns.set_context("talk")
fig, ax = plt.subplots(nrows=1, ncols=3, figsize=(16, 4))
sns.regplot(x='nb_sig', y='dof', data=df_cancer_type, ax=ax[0])
slope, _, _,_ ,_ = linregress(df_cancer_type['nb_sig'],df_cancer_type['dof'])
ax[0].set_title('type-specific signatures,\nslope is {}'.format(np.round(slope, 3)))
ax[0].set_xlabel("number of signatures")
ax[0].set_ylabel("degree of freedom")

df_cancer_type_full = df_cancer_type.copy()
ev, _ = np.linalg.eig(1-sp.spatial.distance.squareform(sp.spatial.distance.pdist(MU_act, 'cosine')))
df_cancer_type_full.loc['all'] = [MU_act.shape[0], sum(ev>0.5), np.nan, np.nan]
sns.regplot(x='nb_sig', y='dof', data=df_cancer_type_full, ax=ax[1])
slope, _, _,_ ,_ = linregress(df_cancer_type_full['nb_sig'],df_cancer_type_full['dof'])
ax[1].set_title('all active signature matrices,\nslope is {}'.format(np.round(slope, 3)))
plt.subplots_adjust(wspace=0.3)
ax[1].set_xlabel("number of signatures")
ax[1].set_ylabel("degree of freedom")

ev, _ = np.linalg.eig(1-sp.spatial.distance.squareform(sp.spatial.distance.pdist(MU, 'cosine')))
df_cancer_type_full.loc['all_artefact'] = [MU.shape[0], sum(ev>0.5), np.nan, np.nan]
sns.regplot(x='nb_sig', y='dof', data=df_cancer_type_full, ax=ax[2])
slope, _, _,_ ,_ = linregress(df_cancer_type_full['nb_sig'],df_cancer_type_full['dof'])
ax[2].set_title('all signature matrices,\nslope is {}'.format(np.round(slope, 3)))
plt.subplots_adjust(wspace=0.3)
ax[2].set_xlabel("number of signatures")
ax[2].set_ylabel("degree of freedom")
plt.savefig('20200222_paper_figures/nb_sig_dof.pdf', bbox_inches='tight')

```

```python
df_cancer_type = df_cancer_type.assign(nb_sig=df_cancer_type.nb_sig.astype(int))
df_cancer_type = df_cancer_type.assign(cancer_type=list(range(len(df_cancer_type))))
df_cancer_type_bis = df_cancer_type.copy()
df_cancer_type_bis = df_cancer_type_bis.assign(nb_sig=MU.shape[0])
big_df_cancer_type = pd.concat((df_cancer_type, df_cancer_type_bis), axis=0)
#get dof of the 65 signatures
ev, _ = np.linalg.eig(1-sp.spatial.distance.squareform(sp.spatial.distance.pdist(MU, 'cosine')))
dof65 = sum(ev>0.5)
big_df_cancer_type.loc[big_df_cancer_type.nb_sig==65, 'dof'] = dof65

df_cancer_type_ter = df_cancer_type.copy()
df_cancer_type_ter = df_cancer_type_ter.assign(nb_sig=MU_act.shape[0])
big_df_cancer_type = pd.concat((big_df_cancer_type, df_cancer_type_ter), axis=0)
ev, _ = np.linalg.eig(1-sp.spatial.distance.squareform(sp.spatial.distance.pdist(MU_act, 'cosine')))
dof47 = sum(ev>0.5)
big_df_cancer_type.loc[big_df_cancer_type.nb_sig==47, 'dof'] = dof47

big_df_cancer_type.sample(20)
```

      nb_sig
      dof
      median_dist
      mean_list
      cancer_type

      UVM
      14
      8.0
      0.172666
      0.220271
      30

      SKCM
      65
      26.0
      0.216352
      0.260411
      23

      COADREAD
      47
      21.0
      0.191984
      0.239050
      5

      KIRP
      47
      21.0
      0.164307
      0.220022
      12

      ESCA
      65
      26.0
      0.197923
      0.249900
      7

      LUAD
      12
      8.0
      0.179708
      0.232747
      15

      LUAD
      47
      21.0
      0.179708
      0.232747
      15

      THCA
      65
      26.0
      0.176920
      0.222253
      26

      UCEC
      13
      8.0
      0.176217
      0.226683
      28

      THCA
      47
      21.0
      0.176920
      0.222253
      26

      CHOL
      14
      10.0
      0.190910
      0.239220
      4

      PRAD
      65
      26.0
      0.161416
      0.211261
      21

      STAD
      65
      26.0
      0.181524
      0.232345
      24

      BRCA
      65
      26.0
      0.172637
      0.223101
      2

      CESC
      6
      5.0
      0.183533
      0.232871
      3

      UCEC
      65
      26.0
      0.176217
      0.226683
      28

      OV
      47
      21.0
      0.139589
      0.195655
      18

      COADREAD
      65
      26.0
      0.191984
      0.239050
      5

      GBM
      5
      3.0
      0.108601
      0.164607
      8

      LGG
      47
      21.0
      0.186504
      0.231429
      13

```python
simu = pd.read_csv('20200222_clonesig_model_selection_criterion.csv', sep='\t')
len(simu[simu.nb_sig==65]), len(simu[simu.nb_sig==47]), len(simu[simu.nb_sig<47])
```

    (6695, 7797, 7797)

```python
np.random.seed(0)
coef_df_list = list()
for repet in range(10):
    print(repet)
    simu = pd.read_csv('20200222_clonesig_model_selection_criterion.csv', sep='\t')
    simu = simu.assign(true_loglikelihood=simu.apply(lambda x: x['ll_{}'.format(int(x['nb_clones']))], axis=1))
    combis = simu[['cancer_type', 'nb_mut', 'nb_clones', 'perc_diploid']].drop_duplicates()
    combis = combis.assign(split=np.random.choice(['test', 'train'], p=[0.2, 0.8], size=len(combis)))
    simu_split = pd.merge(simu, combis, on=['cancer_type', 'nb_mut', 'nb_clones', 'perc_diploid'])
    big_simu = pd.merge(simu_split, big_df_cancer_type[['cancer_type', 'nb_sig', 'dof']], on=['cancer_type', 'nb_sig'])
    usable_simus = pd.melt(big_simu, id_vars=['cancer_type', 'perc_diploid', 'nb_sig',
                                          'nb_clones', 'nb_mut', 'bic',
                                          'min_dist', 'max_dist', 'avg_dist',
                                          'avg_major_cn', 'actual_perc_diploid',
                                          'avg_tot_cn', 'dof', 'split',
                                          'aic', 'aicc', 'icl_q', 'icl_qn',
                                          'bic_alt', 'sh_4', 'sh_5', 'sh_6',
                                          'sh_7', 'sh_8', 'sh_9', 'sh_10',
                                          'max_curvature', 'timestamp',
                                          'true_loglikelihood'],
                          value_vars=['ll_{}'.format(i) for i in range(1, 10)],
                          value_name='loglikelihood')
    feature_list = ('loglikelihood', 'D_cn_nb_sig_lnN', 'D_nocn_nb_sig_lnN', 'D_cn_dof_lnN', 'D_nocn_dof_lnN')
    usable_simus = usable_simus.assign(fitted_nb_clones=usable_simus.variable.str.split('_').str.get(1).astype(int))
    usable_simus = usable_simus.assign(D_cn_nb_sig=usable_simus.avg_major_cn * usable_simus.fitted_nb_clones * (usable_simus.nb_sig + 1))
    usable_simus = usable_simus.assign(D_nocn_nb_sig=usable_simus.fitted_nb_clones * (usable_simus.nb_sig + 1))
    usable_simus = usable_simus.assign(D_cn_dof=usable_simus.avg_major_cn * usable_simus.fitted_nb_clones * (usable_simus.dof + 1))
    usable_simus = usable_simus.assign(D_nocn_dof=usable_simus.fitted_nb_clones * (usable_simus.dof + 1))
    usable_simus = usable_simus.assign(nD_cn_nb_sig_lnN=usable_simus.D_cn_nb_sig*np.log(usable_simus.nb_mut)/usable_simus.nb_mut)
    usable_simus = usable_simus.assign(nD_nocn_nb_sig_lnN=usable_simus.D_nocn_nb_sig*np.log(usable_simus.nb_mut)/usable_simus.nb_mut)
    usable_simus = usable_simus.assign(nD_cn_dof_lnN=usable_simus.D_cn_dof*np.log(usable_simus.nb_mut)/usable_simus.nb_mut)
    usable_simus = usable_simus.assign(nD_nocn_dof_lnN=usable_simus.D_nocn_dof*np.log(usable_simus.nb_mut)/usable_simus.nb_mut)
    usable_simus = usable_simus.assign(nloglikelihood=usable_simus.loglikelihood/usable_simus.nb_mut)

    usable_simus = usable_simus.assign(true_D_cn_nb_sig=usable_simus.avg_major_cn * usable_simus.nb_clones * (usable_simus.nb_sig + 1))
    usable_simus = usable_simus.assign(true_D_nocn_nb_sig=usable_simus.nb_clones * (usable_simus.nb_sig + 1))
    usable_simus = usable_simus.assign(true_D_cn_dof=usable_simus.avg_major_cn * usable_simus.nb_clones * (usable_simus.dof + 1))
    usable_simus = usable_simus.assign(true_D_nocn_dof=usable_simus.nb_clones * (usable_simus.dof + 1))
    usable_simus = usable_simus.assign(ntrue_D_cn_nb_sig_lnN=usable_simus.true_D_cn_nb_sig*np.log(usable_simus.nb_mut)/usable_simus.nb_mut)
    usable_simus = usable_simus.assign(ntrue_D_nocn_nb_sig_lnN=usable_simus.true_D_nocn_nb_sig*np.log(usable_simus.nb_mut)/usable_simus.nb_mut)
    usable_simus = usable_simus.assign(ntrue_D_cn_dof_lnN=usable_simus.true_D_cn_dof*np.log(usable_simus.nb_mut)/usable_simus.nb_mut)
    usable_simus = usable_simus.assign(ntrue_D_nocn_dof_lnN=usable_simus.true_D_nocn_dof*np.log(usable_simus.nb_mut)/usable_simus.nb_mut)
    usable_simus = usable_simus.assign(ntrue_loglikelihood=usable_simus.true_loglikelihood/usable_simus.nb_mut)

    usable_simus = usable_simus.assign(posneg=np.random.choice([1, -1], p=[0.5, 0.5], size=len(usable_simus)))

    simu_dict = {'full': usable_simus.copy(),
                 'full47': usable_simus[usable_simus.nb_sig<=47],
                 '65sigs': usable_simus[usable_simus.nb_sig==65],
                 '47sigs': usable_simus[usable_simus.nb_sig==47],
                 'submu': usable_simus[usable_simus.nb_sig<47]}
    pen_shapes = ['D_cn_nb_sig_lnN', 'D_nocn_nb_sig_lnN', 'D_cn_dof_lnN',
                  'D_nocn_dof_lnN']
    coef_df = pd.DataFrame(index=simu_dict.keys(), columns=pen_shapes)
    for learn_set in simu_dict.keys():
        print(learn_set)
        for pen_shape in pen_shapes:
            sim = simu_dict[learn_set]
            wrong_simus = sim[sim.nb_clones!=sim.fitted_nb_clones]
            feature_list = ['loglikelihood', pen_shape]
            nX_diff = - wrong_simus[['n'+c for c in feature_list]].values + wrong_simus[['ntrue_' + c for c in feature_list]].values
            ny_diff = np.array([1]*len(nX_diff))
            nX_diff = nX_diff * wrong_simus.posneg.values[:, np.newaxis]
            ny_diff = ny_diff * wrong_simus.posneg.values

            nX_diff_train, nX_diff_test = nX_diff[wrong_simus.split=='train'], nX_diff[wrong_simus.split=='test']
            ny_diff_train, ny_diff_test = ny_diff[wrong_simus.split=='train'], ny_diff[wrong_simus.split=='test']

            clf = LinearSVC(fit_intercept=False, max_iter=5000)
            clf.fit(nX_diff_train, ny_diff_train)
            #print(learn_set, pen_shape, clf.score(nX_diff_train, ny_diff_train), clf.score(nX_diff_test, ny_diff_test), clf.coef_/clf.coef_[0][0])
            coef_df.loc[learn_set, pen_shape] = clf.coef_[0][1]/clf.coef_[0][0]
    coef_df_list.append(coef_df)

```

    0
    65sigs
    submu
    47sigs
    full47
    full
    1
    65sigs
    submu
    47sigs
    full47
    full
    2
    65sigs
    submu
    47sigs
    full47
    full
    3
    65sigs
    submu
    47sigs
    full47
    full
    4
    65sigs
    submu
    47sigs
    full47
    full
    5
    65sigs
    submu
    47sigs
    full47
    full
    6
    65sigs
    submu
    47sigs
    full47
    full
    7
    65sigs
    submu
    47sigs
    full47
    full
    8
    65sigs
    submu
    47sigs
    full47
    full
    9
    65sigs
    submu
    47sigs
    full47
    full

```python

```

```python
p = pd.Panel({n: df for n, df in enumerate(coef_df_list)})
```

Mean and confidence interval amplitude for the coefficients

```python
p.mean(axis=0)
```

      D_cn_nb_sig_lnN
      D_nocn_nb_sig_lnN
      D_cn_dof_lnN
      D_nocn_dof_lnN

      65sigs
      -0.008638
      -0.013532
      -0.020119
      -0.032859

      submu
      -0.025857
      -0.042594
      -0.039675
      -0.065452

      47sigs
      -0.009849
      -0.015603
      -0.020847
      -0.034018

      full47
      -0.013053
      -0.020768
      -0.025659
      -0.042146

      full
      -0.010805
      -0.016772
      -0.022896
      -0.037475

```python
1.96 * p.std(axis=0) / np.sqrt(10)
```

      D_cn_nb_sig_lnN
      D_nocn_nb_sig_lnN
      D_cn_dof_lnN
      D_nocn_dof_lnN

      65sigs
      0.000326
      0.000094
      0.000177
      0.000225

      submu
      0.000171
      0.000200
      0.000294
      0.000357

      47sigs
      0.000048
      0.000068
      0.000109
      0.000149

      full47
      0.000228
      0.000101
      0.000160
      0.000199

      full
      0.000080
      0.000086
      0.000146
      0.000195

### Evaluation of the learnt criterion on the test dataset
(the last of the 10 models is kept for evaluation, as the confidence interval for coefficient estimates are small)

```python
def get_result(x, pen_shape, strategy):
    if pen_shape=='D_cn_nb_sig_lnN':
        D = x['avg_major_cn'] * (x['nb_sig']+1) * np.log(x['nb_mut'])
    elif pen_shape=='D_nocn_nb_sig_lnN':
        D = (x['nb_sig']+1) * np.log(x['nb_mut'])
    elif pen_shape=='D_cn_dof_lnN':
        D = x['avg_major_cn'] * (x['dof']+1) * np.log(x['nb_mut'])
    elif pen_shape=='D_nocn_dof_lnN':
        D = (x['dof']+1) * np.log(x['nb_mut'])

    if strategy=='full':
        coef_dict = {int(x['nb_sig']): coef_df.loc['full47', pen_shape]}
    else:
        coef_dict = {int(x['nb_sig']): coef_df.loc['submu', pen_shape]}
        coef_dict[65] = coef_df.loc['65sigs', pen_shape]
        coef_dict[47] = coef_df.loc['47sigs', pen_shape]
    return np.argmax([x['ll_{}'.format(i)] + coef_dict[int(x['nb_sig'])] * i * D  for i in range(1, 11)])+1

for pen_shape in pen_shapes:
    for strategy in ('full', 'separate'):
        def spe_func(x):
            return get_result(x, pen_shape, strategy)
        big_simu = big_simu.assign(**{'{}+{}'.format(pen_shape, strategy): big_simu.apply(spe_func, axis=1)})

```

```python
big_table = pd.melt(big_simu[big_simu.nb_sig!=65], id_vars=['cancer_type', 'perc_diploid', 'nb_sig',
                                      'nb_clones', 'nb_mut',
                                      'min_dist', 'max_dist', 'avg_dist',
                                      'avg_major_cn', 'actual_perc_diploid',
                                      'avg_tot_cn', 'dof', 'split'],
                    value_vars=['bic', 'aic', 'aicc', 'icl_q', 'icl_qn', 'bic_alt', 'sh_4', 'sh_5',
       'sh_6', 'sh_7', 'sh_8', 'sh_9', 'sh_10', 'max_curvature'] +
                    ['{}+{}'.format(pen_shape, strategy) for pen_shape in pen_shapes for strategy in ('full', 'separate')],
                   var_name='method', value_name='selected nb clones')
```

```python
colors = ["terracota", "taupe", "turquoise", "salmon", "light mustard", "wisteria", "dark purple"]
pal = sns.xkcd_palette(colors)
sns.set_context("poster")

big_table=big_table.assign(method=big_table.method.str.replace('D_cn_nb_sig_lnN', '$D_J^A$').str.replace('D_nocn_nb_sig_lnN', '$D_J^B$').str.replace('D_cn_dof_lnN', '$D_J^C$').str.replace('D_nocn_dof_lnN', '$D_J^D$'))
g = sns.catplot(data=big_table[(big_table.split=='test')&((big_table.method.isin(['bic', 'aic', 'icl_qn'])|(big_table.method.str.contains('\+'))))], x="selected nb clones", col="nb_clones",
                row="method", hue='nb_mut', margin_titles=True, kind="count", palette=pal, height=3, aspect=1.3)
for row in g.axes:
    row[-1].texts = []
g = g.set_titles(col_template='{col_name} clones', row_template='{row_name}')
plt.savefig('20200222_paper_figures/repartition_nb_clones_criterion_test.pdf', bbox_inches='tight')

```

```python
big_table = big_table.assign(ok=big_table.nb_clones==big_table['selected nb clones'])
fig, ax=plt.subplots(figsize=(10, 7))
test_df_acc=(big_table[(big_table.split=='test')&((big_table.method.isin(['bic', 'aic', 'icl_qn'])|(big_table.method.str.contains('\+'))))].groupby('method').ok.sum().sort_values()/len(big_simu[big_simu.split=='test']) * 100).to_frame()
test_df_acc.head()
#plt.bar(x=range(len(test_df_acc)), height=test_df_acc.ok)
test_df_acc.plot.barh(ax=ax)
ax.set_xlabel('accuracy (%)')
ax.get_legend().remove()
plt.savefig('20200222_paper_figures/model_selection_criterion_test_accuracy.pdf', bbox_inches='tight')

```

## Sanity check: variation of the accuracy with other parameters
### Accuracy and cancer type

```python
big_table_restrict = big_table[(big_table.method.isin(['bic', 'aic', 'icl_qn'])|(big_table.method.str.contains('\+')))]
plt.figure(figsize=(10, 7))
#TODO replace by cancer types names
sns.heatmap(big_table_restrict.pivot_table(index='cancer_type', columns='method', values=['ok'], aggfunc=sum) / big_table_restrict.pivot_table(index='cancer_type', columns='method', values=['ok'], aggfunc='count'), cmap='Greens')

```

### Accuracy and number of clones

```python
plt.figure(figsize=(13, 7))
sns.heatmap(big_table_restrict.pivot_table(index='nb_clones', columns='method', values=['ok'], aggfunc=sum) / big_table_restrict.pivot_table(index='nb_clones', columns='method', values=['ok'], aggfunc='count'), cmap='Greens', annot=True)
```

### Accuracy and copy number abundance

```python
plt.figure(figsize=(13, 7))
sns.heatmap(big_table_restrict.pivot_table(index='perc_diploid', columns='method', values=['ok'], aggfunc=sum) / big_table_restrict.pivot_table(index='perc_diploid', columns='method', values=['ok'], aggfunc='count'), cmap='Greens', annot=True)

```

### Accuracy and minimal signature cosine distance between 2 clones

```python
#big_table = big_table.assign(round_min_dist=np.round(big_table.min_dist, 2))
#big_table.loc[big_table.round_min_dist>0.5, 'round_min_dist'] = 0.5
val, bins = pd.qcut(big_table_restrict.min_dist, 10, retbins=True, precision=3, duplicates='drop')
big_table_restrict = big_table_restrict.assign(round_min_dist=val)
fig = plt.figure(figsize=(25,15))
sns.heatmap(big_table_restrict.pivot_table(index='round_min_dist', columns='method', values=['ok'], aggfunc=sum) / big_table_restrict.pivot_table(index='round_min_dist', columns='method', values=['ok'], aggfunc='count'), cmap='Greens', annot=True)
#https://stackoverflow.com/questions/43290853/how-to-create-heat-map-from-irregular-xyz-data-in-pyplot

```

### Accuracy and number of observed mutations

```python
plt.figure(figsize=(13, 7))
sns.heatmap(big_table_restrict.pivot_table(index='nb_mut', columns='method', values=['ok'], aggfunc=sum) / big_table_restrict.pivot_table(index='nb_mut', columns='method', values=['ok'], aggfunc='count'), cmap='Greens', annot=True)

```

```python

```

```python

```