TL:DR Devito allows users to build finite difference schemes in python and "compile-down" to optimized C++

Finite difference methods have a long history in quantitative finance.  They are the preferred technique to price and risk manage instruments based on low dimensional probabilistic models that lack analytic solutions.  To implement the finite difference method, one chooses a discretization scheme (or stencil) that approximates the derivatives in the model.  This scheme converts a system of differential equations into a system of algebraic equations that can be solved numerically.

Naturally, a scheme introduces error into the calculation that is roughly proportional to the order of the discretization.  Higher order schemes decrease the error in the system but are more difficult to implement by hand in compiled numerical code. Most option pricing libraries provide utilities for building common, low-order schemes. For instance, QuantLib provides forward, backward and central difference operators that can be combined to build a finite difference pricing engine for a given model.

A new python library called [Devito](http://www.opesci.org/devito-public) takes a different approach to finite difference model building.  Instead of building a discretization scheme directly in C++, it instead allows users to build an arbitrary scheme in python and "compile" that scheme down to optimized C++.  This code can either be JIT compiled and executed immediately or retained for later execution.

In order to make this approach possible, Devito leverages Sympy, the python-based computer algebra system.  Devito uses Sympy to algebraically manipulate the system of differential equations and generates the equivalent stencil equations used in the numerical implementation.  It then generates optimized C++ code after passing the stencil equations through various optimization algorithms.

This approach is equivalent to what happens in a compiler.  To wit, "high-level" code is parsed, passed through optimization algorithms and printed down to "low-level" code.  In this example, python is the high level code and C++ is the low level code.

This approach has several advantages; one, the python implementation is close to the mathematical language; two, the often tedious algebraic manipulations needed to build the stencil is off-loaded to the computer algebra system (Sympy); three, the C++ implementation can be highly optimized of efficient computation.

### Black Scholes Equation

As a first example, let's implement a finite difference solution of the Black Scholes model, the Hello World of quant finance. The Black Scholes PDE for a European option with strike, K, spot, S, volatility $\sigma$, risk free rate, r, expiring at T, is given by

$$
\begin{equation}
\frac{\partial V}{\partial t} + \frac{\sigma^2S^2}{2}\frac{\partial ^2 V}{\partial S^2} + rS\frac{\partial V}{\partial S} - rV = 0
\end{equation}
$$

with the terminal condition for a Call option given by

$$V(S,T) = \max(S-K,0)$$

This PDE can be solved analytically:

$$V(S,K,t,r, \sigma)=SN(d_1)−Ke^{−rt}N(d_2)$$

$$d_1=\frac{\ln\left(\frac{S}{K}\right)+\left(r+0.5\sigma^2\right)T}{\sigma\sqrt{T}}\,\,\,d_2=d_1−\sigma\sqrt{T}$$

In Numpy, this pricing function is

```python
import numpy as np
from scipy.stats import norm
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
#Black Scholes Formula
def d1(S0, K, r, sigma, t):
    return (np.log(S0/K) + (r + sigma**2 / 2) * t)/(sigma * np.sqrt(t))

def d2(S0, K, r, sigma, t):
    return (np.log(S0/K) + (r - sigma**2 / 2) * t)/(sigma * np.sqrt(t))

def BlackScholes(S0, K, r, sigma, T):
    return S0 * norm.cdf(d1(S0, K, r, sigma, T)) - K * np.exp(-r*T) * norm.cdf(d2(S0, K, r, sigma, T))

```

### BS Finite Difference Scheme in Numpy

In order to solve the BS PDE using finite difference techniques, we need to choose a discretization scheme. Keeping things simple, we choose central differences in space and forward differencing in time.  Indexing the spatial dimension with $i$ and the temporal dimension with $n$, Black Scholes PDE  then becomes

$$
\begin{align}
V_{i}^{n+1} &= V_{i}^n -  \frac{\sigma^2 (i\Delta S)^2 \Delta t}{2\Delta S^2}\left(V_{i+1}^n - 2 V_{i}^n + V_{i-1}^n\right) - r(i\Delta S)\frac{\sigma \Delta t}{2\Delta S}\left(V_{i+1}^n- V_{i-1}^n \right)
+ r \Delta t V_{i}^n \\
& = V_{i}^n - \frac{\sigma^2 (i)^2 \Delta t}{2}\left(V_{i+1}^n - 2 V_{i}^n + V_{i-1}^n\right) - r(i)\frac{\sigma \Delta t}{2}\left(V_{i+1}^n- V_{i-1}^n \right)
+ r \Delta t V_{i}^n
\end{align}
$$

Let's first implement a Numpy version of the scheme for comparison's sake. The numerical algorithm essentially works as follows

1. Evaluate the pricing function at the boundary (payoff)
2. Iterate backwards in time, solving the difference equation at each iteration
3. Terminate at t=0 (the evaluation date) to obtain the present value (PV)

In terms of data structures, all we need in order to implement this algorithm is two Numpy array buffers.

#### A word on spatial boundary conditions

We need to choose spatial boundary conditions on pricing function.  In practice, this means enforcing some conditions on either the pricing function itself or derivatives of the pricing function.  Following [Wilmott Vol 3](https://www.amazon.com/Paul-Wilmott-Quantitative-Finance-Set/dp/0470018704), we choose the following boundary conditions

1. For $S = 0 \;\; \frac{\partial V}{\partial t}(0,t) - rV(0,t) = 0$

In discrete form this becomes:
$$V_n^0 = (1 - r \delta t) V_{k-1}^0$$

2. For $S = S_{max}\;\; \frac{\partial^2 V}{\partial S^2}(S,t) = 0$

In discrete form this becomes:
$$V_n^i = 2V_n^{i-1} - V_n^{i-2}$$

In principle, one can choose different boundary conditions without greatly effecting the computed pricing functions.  For the remaining variables needed to solve the problem we make the following choices

```python
# Some variable declarations needed for numerical implementation
nx = 20    # number of space steps
s = 0.2    # vol
r = 0.05   # interest rate
T = 1      # time to expiry
K = 100    # Strike
dt = (0.9/(s*nx)**2) # time step size
nt = int(T/dt) + 1  # number of time steps
dt = T/nt
dx = 2*K/nx
```

Defining a Call payoff function and initializing it at the t=T boundary

```python
def payoff(S,K):
    return np.maximum(S-K,0)

v_payoff = np.vectorize(payoff)
v_bs = np.vectorize(BlackScholes)

# Init C at the payoff
S = np.arange(0,(nx+1)*dx,dx)
C = v_payoff(S,K)
```

Now we can define a diffuse function that carries out the finite difference algorithm. Here we name our pricing function `C` for Call in order to distinguish it from `V` used below in the Devito implementation.

```python
def diffuse(C, nt):
    for n in range(nt):
        Cn = C.copy()
        delta = (0.5/dx)*(Cn[2:] - Cn[0:-2])
        gamma = (1/(dx**2))*(Cn[2:] - 2*Cn[1:-1] + Cn[0:-2])
        theta = -(0.5*s**2)*np.multiply(np.square(S[1:-1]),gamma) - r*np.multiply(S[1:-1],delta) + r*Cn[1:-1]
        C[1:-1] = Cn[1:-1] - dt*theta
        #spatial bc's
        C[0] = Cn[0]*(1 - r*dt)
        C[nx-1] = 2*C[nx-2] - C[nx-3]

```

With our implementation in place, let's diffuse the model back 10 time steps.

```python
diffuse(C, nt=10)
```

Comparing this to analytic solution to BS gives

```python
C
```

    array([  0.00000000e+00,   6.77083365e-15,   2.47797465e-11,
             1.38986768e-08,   2.31379196e-06,   1.50919507e-04,
             4.48551091e-03,   6.67669285e-02,   5.33585468e-01,
             2.44970519e+00,   7.05423863e+00,   1.43368053e+01,
             2.32467881e+01,   3.28816476e+01,   4.27765661e+01,
             5.27502671e+01,   6.27445527e+01,   7.27434913e+01,
             8.27433082e+01,   9.27431251e+01,   1.00000000e+02])

```python
v_bs(S0=S, K=100, r=r, sigma=s, T=10*dt)
```

    /home/givenskm/anaconda3/envs/devito/lib/python3.6/site-packages/ipykernel/__main__.py:3: RuntimeWarning: divide by zero encountered in log
      app.launch_new_instance()
    /home/givenskm/anaconda3/envs/devito/lib/python3.6/site-packages/ipykernel/__main__.py:6: RuntimeWarning: divide by zero encountered in log

    array([  0.00000000e+00,   2.13184792e-53,   8.12368101e-27,
             1.49012572e-15,   1.86102962e-09,   8.60357129e-06,
             1.81033147e-03,   5.89084888e-02,   5.84232552e-01,
             2.66791893e+00,   7.33121846e+00,   1.45076224e+01,
             2.33168378e+01,   3.29033374e+01,   4.27809370e+01,
             5.27490737e+01,   6.27415850e+01,   7.27399614e+01,
             8.27396309e+01,   9.27395669e+01,   1.02739555e+02])

Ploting the PV gives reasonable looking results

```python
fig, ax = plt.subplots()
ax.plot(S, C)
```

## Devito Implementation

Now we want to implement the same model in Devito. First, a brief word about the Devito API. (For more info, consult the Devito documentation)

A key idea in Devito is to present users with a `Function` object that has dual identity, one symbolic and the other numeric.  That is, `Function` objects can be manipulated by Sympy in order to generate the stencil equations. Later, when the finite difference scheme is being numerically solved, `Function` objects hold numerical data in two Numpy array buffers in similar way to how data was stored during the Numpy implementation given above.  This dual approach allows users to implement and solve the finite difference problems in a natural way.  They don't have to implement and reason about separate data structures for the differential equations as well as the numerical implementation.

For our implementation, we first need to declare instances of `Grid`, `Function` and `TimeFunction` (time varying function).  We then define the stencil equation using the `TimeFunction` `V`'s Sympy identity.

```python
from devito import Grid, TimeFunction, Function, Operator
from sympy import Eq, solve
```

```python
# Initialize `u` for space order 2
grid = Grid(shape=(nx+1,), extent=(200.,))
V = TimeFunction(name='V', grid=grid, space_order=2)
X = Function(name='X', grid=grid)

# Create an equation with second-order derivatives
eq = Eq(V.dt, 0.5*s**2*X*X*(V.dx2) + r*X*(V.dx) - r*V)
stencil = solve(eq, V.forward)[0]
eq_stencil = Eq(V.forward, stencil)
```

Looking at the stencil, one can see that it is equivalent to the Numpy version of the stencil defined earlier.

```python
print(stencil)
```

    (0.025*dt*h_x*(-V(t, x - h_x) + V(t, x + h_x))*X(x) + 0.02*dt*(-2.0*V(t, x) + V(t, x - h_x) + V(t, x + h_x))*X(x)**2 + 0.05*h_x**2*(-dt + 20.0)*V(t, x))/h_x**2

Next we implement boundary conditions by setting the `Data` attribute of V equal to the payoff value.  We also enforce spatial boundary conditions as before.  Finally, we construct an `Operator` object from the stencil and boundary conditions.  This is the object that carries out the diffusion in order to solve the finite difference problem.

```python
# Init C at the payoff
S = np.arange(0,(nx+1)*dx,dx)
V.data[0] = v_payoff(S,K)
V.data[1] = v_payoff(S,K)
X.data[:] = S

# Create boundary condition expressions
x = grid.dimensions
t = grid.stepping_dim
bc = [Eq(V.indexed[t+1, 0], V.indexed[t, 0]*(1 - r*dt))]  # bottom
bc += [Eq(V.indexed[t+1,-1], 2*V.indexed[t+1,-2] - V.indexed[t+1,-3])]  # top

# Define the operator
op = Operator([eq_stencil] + bc)
```

The data arrays before simulation are initialized to the payoff boundary

```python
V.data[1]
```

    array([   0.,    0.,    0.,    0.,    0.,    0.,    0.,    0.,    0.,
              0.,    0.,   10.,   20.,   30.,   40.,   50.,   60.,   70.,
             80.,   90.,  100.], dtype=float32)

Diffusing the operator 10 time steps as before we get

```python
op(time=10, dt=dt)
```

    CustomCompiler: compiled /tmp/devito-oebkv76j/41acee72a2d7a564a563c14bcd4b5358a3ed3147.c [1.23 s]
    =========================================================================================
    Section section_0<10,20> with OI=1.44 computed in 0.000 s [4.00 GFlops/s]
    Section main<10> with OI=1.67 computed in 0.000 s [0.11 GFlops/s, 0.01 GPts/s]
    =========================================================================================

Now lets inspect the data array

```python
V.data[0]
```

    array([  0.00000000e+00,   6.77083621e-15,   2.47797494e-11,
             1.38986778e-08,   2.31379227e-06,   1.50919528e-04,
             4.48551122e-03,   6.67669326e-02,   5.33585429e-01,
             2.44970512e+00,   7.05423832e+00,   1.43368015e+01,
             2.32467213e+01,   3.28807259e+01,   4.27697105e+01,
             5.27169189e+01,   6.26308594e+01,   7.24206314e+01,
             8.20446320e+01,   9.12274399e+01,   1.00405121e+02], dtype=float32)

Comparing that with the Numpy implemenation gives similar results

```python
C
```

    array([  0.00000000e+00,   6.77083365e-15,   2.47797465e-11,
             1.38986768e-08,   2.31379196e-06,   1.50919507e-04,
             4.48551091e-03,   6.67669285e-02,   5.33585468e-01,
             2.44970519e+00,   7.05423863e+00,   1.43368053e+01,
             2.32467881e+01,   3.28816476e+01,   4.27765661e+01,
             5.27502671e+01,   6.27445527e+01,   7.27434913e+01,
             8.27433082e+01,   9.27431251e+01,   1.00000000e+02])

As one can see from the results, the closest agreement between all three solutions is away from the boundaries and the cusp S = K.  This makes sense as the pricing function isn't smooth at those points.  I will explore the numerical errors in a later post.

For fun, let's you can take a look at the C++ code that was generated and JIT compiled during evaluation

```python
print(op.ccode)
```

    #define _POSIX_C_SOURCE 200809L
    #include "stdlib.h"
    #include "math.h"
    #include "sys/time.h"
    #include "xmmintrin.h"
    #include "pmmintrin.h"

    struct profile
    {
      double section_0;
      double section_1;
    } ;

    int Kernel(float *restrict V_vec, float *restrict X_vec, const float dt, const float h_x, const int t_size, const int t_s, const int t_e, const int time_size, const int time_s, const int time_e, const int x_size, const int x_s, const int x_e, void *_timings)
    {
      float (*restrict V)[x_size] __attribute__((aligned(64))) = (float (*)[x_size]) V_vec;
      float (*restrict X) __attribute__((aligned(64))) = (float (*)) X_vec;
      struct profile *timings = (struct profile*) _timings;
      /* Flush denormal numbers to zero in hardware */
      _MM_SET_DENORMALS_ZERO_MODE(_MM_DENORMALS_ZERO_ON);
      _MM_SET_FLUSH_ZERO_MODE(_MM_FLUSH_ZERO_ON);
      struct timeval start_section_1, end_section_1;
      gettimeofday(&start_section_1, NULL);
      for (int time = t_s, t0 = (time)%(2), t1 = (time + 1)%(2); time < t_e - 1; time += 1, t0 = (time)%(2), t1 = (time + 1)%(2))
      {
        for (int x = x_s + 1; x < x_e - 1; x += 1)
        {
          V[t1][x] = (2.5e-2F*dt*h_x*(-V[t0][x - 1] + V[t0][x + 1])*X[x] + 2.0e-2F*dt*(X[x]*X[x])*(-2.0F*V[t0][x] + V[t0][x - 1] + V[t0][x + 1]) + 5.0e-2F*(h_x*h_x)*(-dt + 2.0e+1F)*V[t0][x])/pow(h_x, 2);
        }
        V[t1][0] = 9.97222222222222e-1F*V[t0][0];
        V[t1][-1] = -V[t1][-3] + 2*V[t1][-2];
      }
      gettimeofday(&end_section_1, NULL);
      timings->section_1 += (double)(end_section_1.tv_sec-start_section_1.tv_sec)+(double)(end_section_1.tv_usec-start_section_1.tv_usec)/1000000;
      return 0;
    }

## Conclusion

In this post we solved Black Scholes equation using finite difference methods in both Numpy and Devito.  As you can see, their output is similar but their implementation is different.  The great thing about Devito is that it allows users to build complex stencils and solve pde's without having to worry about hand writing optimized C++ code.  This achieves a nice seperation of concerns between model building in Python and numerical implemenation in C++.

In future posts, I plan on implementing more finance models in Devito and comparing Devito's performance with conventional pricing libraries like QuantLib.  Until then, thanks for visiting my blog.  See you next time.


## Mathematical Typesetting With Latex

This will only serve as a brief introduction to mathematical typesetting with latex as a full guide could take a few notebooks on its own. However, most of these commands are easy to find if you need to do something special. The basics are as follows:

1. To put a variable in-line, surround it in dollar signs like so: `$variable$`.
1. To typeset an equation, surround it (on a new line) in double dollar signs: `$$equation$$`
1. you could also use this syntax
   ```latex
    \begin{equation}
       equation
   \end{equation}
    ```
1. Special symbols start with a backslash (`\`). For example, the greek letter $\alpha$ would be called like this `$\alpha$`.

1. Fractions can be typeset with `\frac{numerator}{denominator}`
1. Square roots are `\sqrt{stuff inside square root}`

Change the cell below to markdown format and run it to see some of these in action.

If you want to type inline variables it's as easy as $x$. If you want special symbols in line, simply call it by its name, such as $\alpha$, $\gamma$, $\epsilon$, $\Delta$, $\nabla$ and so on. [here](http://tug.ctan.org/info/symbols/comprehensive/symbols-a4.pdf) is a link to a comprehensive symbol list, and the command you have to call to render it. Of course, sometimes a Google search is faster.

You can start equations easily
$$ c  = \sqrt{a^2 + b^2}$$
notice how in the above, the variables appearing under the square root are surrounded by curly brackets, and powers are done with carrots.

You can also type equations like this
\begin{equation}
a = \frac{x^3 +\Gamma}{\int e^{-x^2} dx}
\end{equation}

Notice here how the numerator of the fraction appears in curly braces, and the denominator appears in separate curly braces. Notice how to find a capital gamma, we simply wrote the first letter as a capital. Also notice how in the exponent, this time we included curly brackets. This is because without them only the first character will become a superscript.

\begin{equation}
a = x^-10
\end{equation}

The above doesn't render how you want it to, but
\begin{equation}
a = x^{-10}
\end{equation}
does once you include the curly braces.

You can also include vectors on variables
\begin{equation}
\vec{F} = m \vec{a}
\end{equation}
### Latex Common Errors

You'll need to convert the cells below to markdown in order to see how these common mistakes render, and their solutions.

##### Exponents need to be wrapped in curly braces if they're more than one character long.
\begin{equation}
a = x^-10
\end{equation}

Doesn't render how you want it to, but
\begin{equation}
a = x^{-10}
\end{equation}
does once you include the curly braces.

##### The same is true for subscripts
\begin{equation}
a = x_10
\end{equation}

Doesn't render how you want it to, but
\begin{equation}
a = x_{10}
\end{equation}
does once you include the curly braces.

##### Spaces in equations are ignored
$$
If you use words as a varable, they become hard to read = x_a^2
$$

If you need to include words in your equations, you can do so with `\text{}`
$$
\text{Now we can read this easily once we've rendered the math} = x_a^2
$$
##### You cannot include percent signs without a little extra work
$$
100 % That's because a percent sign is a comment in latex!
$$
To escape the comment, simply add a backslash
$$ 100 \% $$
Latex is very extensive for mathematical typesetting, and we can't cover it all here. However, it's fairly intuitive and with a little practice significantly faster than using the equation editors in other tools. Feel free to try creating your own equations now.

---
### Exercises
1. Fix the following to correctly represent the formula for calculating the surface area of a sphere
$$
Area = 4 pi r_2
$$

2. Fix the following to correctly represent the formula for calculating the volume of a sphere
\begin{equation}
Volume = \frac{4} / {3} pi r3
\end{equation}
3. Make sure pi is showing up as a symbol in questions 1 & 2.
4. In this sentence, change the variables x, Y, and y to ensure they render as LateX variables.
5. In a new cell below, write out the following formula in LateX.

---

[](https://github.com/callysto/curriculum-notebooks/blob/master/LICENSE.md)


```python
import numpy as np
import sympy as sp
import pandas as pd
import math
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split

data=pd.read_csv('C:\\Users\\Utsav\\Desktop\\pulsar_prediction\\pulsar_stars.csv', sep=',',header=0)

pulsarData=data.values
pulsarData=np.array(pulsarData)
```

```python
#split the data

train, test = train_test_split(data, test_size=0.2)
pulsar=np.array(train)
train=np.array(train)
test=np.array(test)

print(len(test))
print(len(train))
#print(train)
```

    3580
    14318

```python
#copy labels of test & train data elsewhere
#replace all test and train labels by 1 (for intercept term)

y_train=np.zeros(len(train))
y_test=np.zeros(len(test))
for i in range(len(test)):
    y_test[i]=test[i][8]
    test[i][8]=1

for i in range(len(train)):
    y_train[i]=train[i][8]
    train[i][8]=1

```

```python
%%time
#logistic regression using sklearn
from sklearn.linear_model import LogisticRegression
from sklearn import metrics

logreg = LogisticRegression()
logreg.fit(train, y_train)

y_pred = logreg.predict(test)

```

    Wall time: 186 ms

```python
# calculate score

score=0
for i in range(len(test)):
    if y_pred[i]==y_test[i]:
        score=score+1
score=score*100/len(test)

print("Accuracy = " + "%.2f" % score + "%")

#calculate no of flase negatives
score=0

for i in range(len(test)):
    if y_pred[i]==0 and y_test[i]==1:
        score=score+1

#score=score*100/len(test)
percent = score/len(test)

print("No of false negatives = %d" % score)
print("No of positives in test data = %d" % sum(y_test))
print("No of positives predicted : %d" % sum(y_pred))
```

    Accuracy = 97.99%
    No of false negatives = 57
    No of positives in test data = 332
    No of positives predicted : 290

```python
#using maxabs scaler

from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()
train_scaled=scaler.fit_transform(train)
test_scaled=scaler.fit_transform(test)

logreg = LogisticRegression()
logreg.fit(train_scaled, y_train)

y_pred = logreg.predict(test_scaled)

#calculate score
score=0
for i in range(len(test)):
    if y_pred[i]==y_test[i]:
        score=score+1
score=score*100/len(test)

print("Accuracy = " + "%.2f" % score + "%")

#calculate no of flase negatives
score=0

for i in range(len(test)):
    if y_pred[i]==0 and y_test[i]==1:
        score=score+1

#score=score*100/len(test)
percent = score/len(test)

print("No of false negatives = %d" % score)
print("No of positives in test data = %d" % sum(y_test))
print("No of positives predicted : %d" % sum(y_pred))
```

    Accuracy = 97.35%
    No of false negatives = 88
    No of positives in test data = 332
    No of positives predicted : 251

Own Implementation

```python
from scipy.optimize import fmin_tnc

theta = np.zeros((X.shape[1], 1))

def sigmoid(z):
    sigma = 1 / (1 + np.exp(-z))
    return sigma

def probability(theta, x):
    return sigmoid(np.dot(x, theta))

def cost_function(theta, x, y):
    cost = -np.sum(y*np.log(probability(theta, x)) + (1-y)*np.log(1 - probability(theta, x)))
    return cost

def gradient (theta, x, y):
    return np.dot(x.T, probability(theta , x) - y)

def fit (x, y, theta):
    param = fmin_tnc(func=cost_function, x0=theta, fprime=gradient,args=(x, y.flatten()))
    return param[0]

parameters = fit(X, y, theta)
print (parameters)

def predict(test, y_test, theta):
    for i in range(len(test)):
        p = probability(parameters, test[i,:])
        if p>0.5:
            y_test[i] = 1
        else:
            y_test[i] = 0

```

    [ 3.65133667e-02 -3.80217159e-02  6.78890765e+00 -6.33306875e-01
     -2.47733463e-02  4.29645538e-02 -5.28755145e-03 -3.77353568e-03
     -9.14482752e+00]

```python
#predict
predict(test, y_pred, parameters)
```

```python
# calculate score:
score=0
for i in range(len(test)):
    if y_pred[i]==y_test[i]:
        score=score+1
score=score*100/len(test)

print("Accuracy = " + "%.2f" % score + "%")

#calculate no of flase negatives
score=0

for i in range(len(test)):
    if y_pred[i]==0 and y_test[i]==1:
        score=score+1

#score=score*100/len(test)
percent = score/len(test)

print("No of false negatives = %d" % score)
print("No of positives in test data = %d" % sum(y_test))
print("No of positives predicted : %d" % sum(y_pred))

```

    Accuracy = 98.02%
    No of false negatives = 57
    No of positives in test data = 332
    No of positives predicted : 289

However we can decrease the number of false negatives if we predict 1 at p>0.4 instead of 0.5

```python
from scipy.optimize import fmin_tnc

theta = np.zeros((X.shape[1], 1))

def sigmoid(z):
    sigma = 1 / (1 + np.exp(-z))
    return sigma

def probability(theta, x):
    return sigmoid(np.dot(x, theta))

def cost_function(theta, x, y):
    cost = -np.sum(y*np.log(probability(theta, x)) + (1-y)*np.log(1 - probability(theta, x)))
    return cost

def gradient (theta, x, y):
    return np.dot(x.T, probability(theta , x) - y)

def fit (x, y, theta):
    param = fmin_tnc(func=cost_function, x0=theta, fprime=gradient,args=(x, y.flatten()))
    return param[0]

parameters = fit(X, y, theta)
print (parameters)

def predict(test, y_test, theta):
    for i in range(len(test)):
        p = probability(parameters, test[i,:])
        if p>0.4:
            y_test[i] = 1
        else:
            y_test[i] = 0

```

    [ 3.65133667e-02 -3.80217159e-02  6.78890765e+00 -6.33306875e-01
     -2.47733463e-02  4.29645538e-02 -5.28755145e-03 -3.77353568e-03
     -9.14482752e+00]

```python
#predict
predict(test, y_pred, parameters)
```

```python
# calculate score:
score=0
for i in range(len(test)):
    if y_pred[i]==y_test[i]:
        score=score+1
score=score*100/len(test)

print("Accuracy = " + "%.2f" % score + "%")

#calculate no of flase negatives
score=0

for i in range(len(test)):
    if y_pred[i]==0 and y_test[i]==1:
        score=score+1

#score=score*100/len(test)
percent = score/len(test)

print("No of false negatives = %d" % score)
print("No of positives in test data = %d" % sum(y_test))
print("No of positives predicted : %d" % sum(y_pred))

```

    Accuracy = 98.10%
    No of false negatives = 49
    No of positives in test data = 332
    No of positives predicted : 302

```python

```

```python

```


# Robust nonlinear regression in scipy
TAGS: Optimization and fitting

AUTHORS: Nikolay Mayorov

One of the main applications of nonlinear least squares is nonlinear regression or curve fitting. That is by given pairs $\left\{ (t_i, y_i) \: i = 1, \ldots, n \right\}$ estimate parameters $\mathbf{x}$ defining a nonlinear function $\varphi(t; \mathbf{x})$, assuming the model:
\begin{equation}
y_i = \varphi(t_i; \mathbf{x}) + \epsilon_i
\end{equation}

Where $\epsilon_i$ is the measurement (observation) errors. In the least-squares estimation we search $\mathbf{x}$ as the solution of the following optimization problem:
\begin{equation}
\frac{1}{2} \sum_{i=1}^n (\varphi(t_i; \mathbf{x}) - y_i)^2 \rightarrow \min_\mathbf{x}
\end{equation}

Such formulation is intuitive and convinient from mathematical point of view. From the probabilistic point of view the least-squares solution is known to be the [maximum likelihood](https://en.wikipedia.org/wiki/Maximum_likelihood) estimate, provided that all $\epsilon_i$ are independent and normally distributed random variables.

So theoretically it is not optimal when $\epsilon_i$ have distribution other than normal. Although in engineering practice it is usally not important, i.e. if errors behave as some reasonable random variables with zero mean a result of least-squares estimation will be satisfactory.

The real problems start when data is contaminated by outliers (completly wrong measurements). In this case the least-squares solution can become significantly biased to avoid very high residuals on outliers. To qualitatively explain why it is happening, let's consider the simplest least-squares problem:
\begin{align}
\frac{1}{2} \sum_{i=1}^n (a - x_i)^2 \rightarrow \min_a
\end{align}
And the solution is the mean value:
\begin{align}
a^* = \frac{1}{n} \sum_{i=1}^n x_i
\end{align}

Now imagine: $a = 1, n=4, x_1 = 0.9, x_2 = 1.05, x_3=0.95, x_4=10 \text{ (outlier) }$. The solution $a^* = 3.225$, i.e. completely ruined by a single outlier.

One of the well known robust estimators is l1-estimator, in which the sum of absolute values of the residuals is minimized. For demonstration, again consider the simplest problem:
\begin{equation}
\sum_{i=1}^n |a - x_i| \rightarrow \min_a
\end{equation}

And the well-known solution is the [median](https://en.wikipedia.org/wiki/Median) of $\{x_i\}$, such that the small number of outliers won't affect the solution. In our toy problem we have $a^* = 1$.

The only disadvantage of l1-estimator is that arising optimization problem is hard, as the function is nondifferentiable everywhere, which is particularly troublesome for efficient nonlinear optimization. It means that we are better to stay with differentiable problems, but somehow incorporate robustness in estimation. To accomplish this we introduce a sublinear function $\rho(z)$ (i.e. its growth should be slower than linear) and formulate a new least-squares-like optimization problem:
\begin{equation}
\frac{1}{2} \sum_{i=1}^n \rho \left((\varphi(t_i; x) - y_i)^2 \right) \rightarrow \min_x
\end{equation}

Turns out that this problem can be reduced to standard nonlinear least squares by modifying a vector of residuals and Jacobian matrix on each iteration, such that computed gradient and Hessian approximation match the ones of the objective function. Refer to the [paper](http://link.springer.com/chapter/10.1007%2F3-540-44480-7_21) for details.

The implemented choices of $\rho$ are the following:

1. Linear functior which gives a standard least squares: $\rho(z) = z$.
2. [Huber loss](https://en.wikipedia.org/wiki/Huber_loss): $\rho(z) = \begin{cases}
z & z \leq 1 \\
\sqrt{z} - 1 & z > 1
\end{cases}$
3. Smooth approximation to absolute value loss, "soft l1 loss": $\rho(z) = 2 (\sqrt{1 + z} - 1)$
4. Cauchy loss: $\rho(z) = \ln(1 + z)$.
5. Loss by arctan: $\rho(z) = \arctan z$.

The functions 2 and 3 are relatively mild and give approximately absolute value loss for large residuals. The last two functions are strongly sublinear and give significant attenuation for outliers.

The loss functions above are written with the assumption that the soft threshold between inliners and outliers is equal to 1.0. To generalize, we introduce the scaling parameter $C$ and evaluate the loss as
\begin{equation}
\hat{\rho}(r^2) = C^2 \rho \left( \left(\frac{r}{C} \right)^2 \right)
\end{equation}

Note that if residuals are small, we have $\hat{\rho}(r^2) \approx \rho(r^2) \approx r^2$ for any $\rho(z)$ defined above.

To illustrate the behaviour of the introduced functions we build a plot:

```python
import numpy as np

%matplotlib inline
import matplotlib.pyplot as plt

from matplotlib import rcParams
rcParams['figure.figsize'] = (10, 6)
rcParams['legend.fontsize'] = 16
rcParams['axes.labelsize'] = 16
```

```python
r = np.linspace(0, 5, 100)

linear = r**2

huber = r**2
huber[huber > 1] = 2 * r[huber > 1] - 1

soft_l1 = 2 * (np.sqrt(1 + r**2) - 1)

cauchy = np.log1p(r**2)

arctan = np.arctan(r**2)
```

```python
plt.plot(r, linear, label='linear')
plt.plot(r, huber, label='huber')
plt.plot(r, soft_l1, label='soft_l1')
plt.plot(r, cauchy, label='cauchy')
plt.plot(r, arctan, label='arctan')
plt.xlabel("$r$")
plt.ylabel(r"$\rho(r^2)$")
plt.legend(loc='upper left')
```

-----------------

Now we will show how robust loss functions work on a model example. We define the model function as
\begin{equation}
f(t; A, \sigma, \omega) = A e^{-\sigma t} \sin (\omega t)
\end{equation}

Which can model a observed displacement of a linear damped oscillator.

Define data generator:

```python
def generate_data(t, A, sigma, omega, noise=0, n_outliers=0, random_state=0):
    y = A * np.exp(-sigma * t) * np.sin(omega * t)
    rnd = np.random.RandomState(random_state)
    error = noise * rnd.randn(t.size)
    outliers = rnd.randint(0, t.size, n_outliers)
    error[outliers] *= 35
    return y + error
```

Define model parameters:

```python
A = 2
sigma = 0.1
omega = 0.1 * 2 * np.pi
x_true = np.array([A, sigma, omega])

noise = 0.1

t_min = 0
t_max = 30
```

Data for fitting the parameters will contain 3 outliers:

```python
t_train = np.linspace(t_min, t_max, 30)
y_train = generate_data(t_train, A, sigma, omega, noise=noise, n_outliers=4)
```

Define the function computing residuals for least-squares minimization:

```python
def fun(x, t, y):
    return x[0] * np.exp(-x[1] * t) * np.sin(x[2] * t) - y
```

Use all ones as the initial estimate.

```python
x0 = np.ones(3)
```

```python
from scipy.optimize import least_squares
```

Run standard least squares:

```python
res_lsq = least_squares(fun, x0, args=(t_train, y_train))
```

Run robust least squares with `loss='soft_l1'`, set `f_scale` to 0.1 which means that inlier residuals are approximately lower than 0.1.

```python
res_robust = least_squares(fun, x0, loss='soft_l1', f_scale=0.1, args=(t_train, y_train))
```

Define data to plot full curves.

```python
t_test = np.linspace(t_min, t_max, 300)
y_test = generate_data(t_test, A, sigma, omega)
```

Compute predictions with found parameters:

```python
y_lsq = generate_data(t_test, *res_lsq.x)
y_robust = generate_data(t_test, *res_robust.x)
```

```python
plt.plot(t_train, y_train, 'o', label='data')
plt.plot(t_test, y_test, label='true')
plt.plot(t_test, y_lsq, label='lsq')
plt.plot(t_test, y_robust, label='robust lsq')
plt.xlabel('$t$')
plt.ylabel('$y$')
plt.legend()
```

We clearly see that standard least squares react significantly on outliers and give very biased solution, whereas robust least squares (with soft l1-loss) give the solution very close to the true model.


# Seminar: SVM, Kernel trick
Machine Learning by prof. Evgeny Burnaev

Author: Andrey Lange

Given training data $x_i \in \mathbb{R}^p$, $y_i\in\{-1, 1\}$, $i=1,\ldots, n$, we have the following problem:

Primal problem:

\begin{equation}
    \frac{1}{2} ||w||^2 + C \sum_{i=1}^{n} \xi_i \longrightarrow \min_{w, b, \xi},\\
      \begin{split}
      \textrm {subject to }\ & y_i (w^T \phi(x_i) + b) \geq 1 - \xi_i,\\
                             & \xi_i \geq 0.
      \end{split}
\end{equation}

Dual problem:

\begin{equation}
    \frac{1}{2} \sum_{i=1}^n\sum_{j=1}^n \alpha_i\alpha_j y_i y_j \underbrace{\phi(x_i)^{T} \phi(x_j)}_{K(x_i, x_j)} - \sum_{i=1}^n \alpha_i \longrightarrow  \min_{\alpha}, \\
    \begin{split}
    \textrm {subject to }\ & \sum_{i=1}^n y_i \alpha_i = 0,\\
                           & 0 \leq \alpha_i \leq C.
    \end{split}
\end{equation}

$\alpha_i = 0$ for non-support vectors,

$\alpha_i < C$ if $x_i$ lies on the margin,

$\alpha_i = C$ if $x_i$ lies inside a margin or misclassified.

The decision function is a sum over only SUPPORT data points $S$:

\begin{equation}
y(x)= w^T\phi(x) + b = \sum_{x_i \in S} \alpha_i y_i K(x_i, x) + b \qquad \qquad (1)
\end{equation}

The sum is over only the support vectors!

Less support vectors (sparsity) - faster classifications!

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
```

## Topic 1: Separation by a hyperplane

```python
from sklearn import svm, datasets, metrics

X, y = datasets.make_blobs(n_samples=100, centers=2, cluster_std=1.0, center_box = (-15, 15), random_state=0)
X = np.concatenate((X, [[4, 8], [0, 0]]))
y = np.concatenate((y, [1, 0]))

plt.scatter(X[:, 0], X[:, 1], c=y, s=30, cmap=plt.cm.Paired)
plt.xlim(-1, 6)
plt.ylim(-1, 8)

clf = svm.SVC(kernel='linear')
clf.fit(X, y)

# create grid to evaluate model
xx = np.linspace(X[:, 0].min(), X[:, 0].max(), 30)
yy = np.linspace(X[:, 1].min(), X[:, 1].max(), 30)
YY, XX = np.meshgrid(yy, xx)
xy = np.vstack([XX.ravel(), YY.ravel()]).T
Z = clf.decision_function(xy).reshape(XX.shape)

# plot decision boundary and margins
plt.contour(XX, YY, Z, colors='k', levels=[-1, 0, 1], alpha=0.5, linestyles=['--', '-', '--'])
plt.scatter(clf.support_vectors_[:, 0], clf.support_vectors_[:, 1], s=100,
           linewidth=1, facecolors='none', edgecolors='k')

print('accuracy on the training set:', metrics.accuracy_score(y, clf.predict(X)))
```

### Question 1.1: Change the code above to make accuracy of 1. Do not tune any hyperparameters of SVM. What prevents ideal classification in this case?

### Solution 1.1:
Simply remove strings

```python
X = np.concatenate((X, [[4, 8], [0, 0]]))
y = np.concatenate((y, [1, 0]))
```

that add outliers to the dataset and classes will become separable

### Question 1.2: How to calculate the value of accuracy without any function call, just looking on the code and on the picture?

### Solution 1.2:

The accuracy is 100/102, because only 2 outliers among 102 points were missclassified

## Topic 2: Primal SVM and explicit transforms

```python
df = pd.read_csv('https://raw.githubusercontent.com/adasegroup/ML2022_seminars/master/seminar4/data/dataset0.csv')
X = df[['X1', 'X2']]
y = df['y']
plt.scatter(X['X1'], X['X2'], c=['brown' if c == 1 else 'b' for c in y])
plt.title('labeled dataset')
plt.show()

clf = svm.SVC(kernel='linear')
clf.fit(X, y)

plt.scatter(X['X1'], X['X2'], c=['brown' if c == 1 else 'b' for c in clf.predict(X)])
plt.title('predicted class labels: all blue class was missclassified!')
plt.show()

print('Linear SVM is not good if neither basic nor kernel is used:')
print('accuracy on the training set:', metrics.accuracy_score(y, clf.predict(X)))
```

### Question 2.1: One of the ideas of SVM is that to transform the initial features to some other space where the classes are better separated by a hyperplane. What about using the polar coordinates in the example above? Apply it and show how SVM works with such a new basis using the following template.

```python
df = pd.read_csv('https://raw.githubusercontent.com/adasegroup/ML2022_seminars/master/seminar4/data/dataset0.csv')

X = df[['X1', 'X2']]
y = df['y']

def to_polar(x, y):
    r = np.sqrt(x**2 + y**2)
    phi = np.arctan2(y, x)
    return phi, r

# --- HERE IS YOUR CODE!!!
# change this block so that it gives new pandas DataFrame called X_transform
# with columns 'phi' and 'r' and X and X_transform have the corresponding rows of data.
X_transform = pd.DataFrame()
X_transform['phi'] = X['X1'] # its wrong
X_transform['r'] = X['X2']   # its wrong

# --- HERE IS YOUR CODE!!!
clf = svm.SVC(kernel='linear')
clf.fit(X_transform, y)
print('accuracy on the training set:', metrics.accuracy_score(y, clf.predict(X_transform)))

plt.scatter(X_transform['phi'], X_transform['r'], c=['brown' if c == 1 else 'b' for c in y])
plt.xlabel('phi')
plt.ylabel('r')

pp = np.linspace(X_transform['phi'].min(), X_transform['phi'].max(), 30)
rr = np.linspace(X_transform['r'].min(), X_transform['r'].max(), 30)
RR, PP = np.meshgrid(rr, pp)
pr = np.vstack([PP.ravel(), RR.ravel()]).T
Z = clf.decision_function(pr).reshape(PP.shape)
plt.contour(PP, RR, Z, colors='k', levels=[-1, 0, 1], linestyles=['--', '-', '--'])
plt.scatter(clf.support_vectors_[:, 0], clf.support_vectors_[:, 1], s=100,
            linewidth=1, facecolors='none', edgecolors='k')
plt.show()
```

### Solution 2.1:
One can use this code
```python
phi, r = to_polar(X['X1'], X['X2'])
X_transform = pd.DataFrame({'phi': phi, 'r': r})
```

### Question 2.2: The separating hyperplane (line in 2D) in polar basis above looks strange: why shouldn't it be a little higher because many blue points are missclassified - they are on the other side of the line! We need to tune the hyperparameters of the model - constant $C$ in SVM. Try different values of $C$, using the code below and then find the optimal value of $C$, which provides the maximum accuracy.

```python
df = pd.read_csv('https://raw.githubusercontent.com/adasegroup/ML2022_seminars/master/seminar4/data/dataset0.csv')

X = df[['X1', 'X2']]
y = df['y']

phi, r = to_polar(X['X1'], X['X2'])
X_transform = pd.DataFrame({'phi': phi, 'r': r})
# # --- YOUR CODE HERE!!!
# select the optimal C among some set of values (say from 10**-5 to 10**5)
# providing the best model 'clf' and print its accuracy metric on the training dataset
clf = svm.SVC(kernel='linear', C=1).fit(X_transform, y)
print('accuracy on the training set:', metrics.accuracy_score(y, clf.predict(X_transform)))
# # --- YOUR CODE HERE!!!

plt.scatter(X_transform['phi'], X_transform['r'], c=['brown' if c == 1 else 'b' for c in y])
plt.xlabel('phi')
plt.ylabel('r')

pp = np.linspace(X_transform['phi'].min(), X_transform['phi'].max(), 30)
rr = np.linspace(X_transform['r'].min(), X_transform['r'].max(), 30)
RR, PP = np.meshgrid(rr, pp)
pr = np.vstack([PP.ravel(), RR.ravel()]).T
Z = clf.decision_function(pr).reshape(PP.shape)
plt.contour(PP, RR, Z, colors='k', levels=[-1, 0, 1], linestyles=['--', '-', '--'])
plt.show()
```

### Solution 2.2:
You can use the code
```python
# tuning of C
best_C = None
best_acc = 0
for c in [0.1, 1, 10, 100, 1000, 10**4, 10**5]:
    clf = svm.SVC(kernel='linear', C=c).fit(X_transform, y)
    acc = metrics.accuracy_score(y, clf.predict(X_transform))
    if (acc > best_acc):
        best_acc = acc
        best_C = c
clf = svm.SVC(kernel='linear', C=best_C)
clf.fit(X_transform, y)
print('best C:', best_C)
print('accuracy on the training set:', metrics.accuracy_score(y, clf.predict(X_transform)))
```
or using `GridsearchCV()`:
```python
# tuning of C
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import accuracy_score, make_scorer
clf = svm.SVC(kernel='linear')
clf = GridSearchCV(clf,
                   {'C': [0.1, 1, 10, 100, 1000, 10**4, 10**5]},
                   scoring=make_scorer(accuracy_score),  # we can skip this line because accuracy_score is by default
                   cv = [(range(len(y)), range(len(y)))] # only 1 split and training and testing sets are the same
                   )
clf.fit(X_transform, y)
print('best C:', clf.best_params_['C'])
print('accuracy on the training set:', clf.best_score_)
```
We obtain:
```
best C: 10000
accuracy on the training set: 1.0
```

### Question 2.3: Usually, SVM transforms into a space with the dimension higher than number of original features because it makes easier to separate classes by a hyperplane there. Unlike to polar coordinates, one can try the transform
### $$y_1=x_1, \quad y_2=x_2, \quad y_3=|x_1|+|x_2|.$$
### Why classes are better separable in that 3D space? Imagine the picture in 3D. Since we have previously tuned the model even reaching the accuracy of 1, improve this transform and the range for hyperparameter $C$ for better fitting.

```python
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import accuracy_score, make_scorer

df = pd.read_csv('https://raw.githubusercontent.com/adasegroup/ML2022_seminars/master/seminar4/data/dataset0.csv')

X = df[['X1', 'X2']]
y = df['y']

# --- YOUR CODE HERE !!! improve the transform
def to_y1y2y3(x1, x2):
    y1 = x1
    y2 = x2
    y3 = abs(x1) + abs(x2)
    return y1, y2, y3
# --- YOUR CODE HERE !!! improve the transform

# improve the range for hyperparameter C
y1, y2, y3 = to_y1y2y3(X['X1'], X['X2'])
X_transform = pd.DataFrame({'Y1': y1, 'Y2': y2, 'Y3': y3})
clf = GridSearchCV(svm.SVC(kernel='linear'),
                   {'C': [1]},         # YOUR CODE HERE, IMPROVE THE RANGE  !!!
                   scoring=make_scorer(accuracy_score),  # we can skip this line because accuracy_score is by default
                   cv = [(np.array(range(len(y))), np.array(range(len(y))))] # only 1 split and training and testing sets are the same
                   )
clf.fit(X_transform, y)
print('best C:', clf.best_params_['C'])
print('accuracy on the training set:', clf.best_score_)

xx1 = np.linspace(X['X1'].min(), X['X1'].max(), 50)
xx2 = np.linspace(X['X2'].min(), X['X2'].max(), 50)
XX1, XX2 = np.meshgrid(xx1, xx2)
YY1, YY2, YY3 = to_y1y2y3(XX1, XX2)
xy = np.vstack([YY1.ravel(), YY2.ravel(), YY3.ravel()]).T
Z = clf.decision_function(xy).reshape(XX1.shape)
plt.scatter(X['X1'], X['X2'], c=['brown' if c == 1 else 'b' for c in y])
plt.contour(XX1, XX2, Z, colors='k', levels=[-1, 0, 1], linestyles=['--', '-', '--'])
plt.show()
```

```python
X
```

      X1
      X2

      0
      -0.086167
      0.104660

      1
      -0.183062
      -0.085844

      2
      -0.141990
      0.295572

      3
      0.016888
      0.429102

      4
      0.040901
      0.397820

      ...
      ...
      ...

      390
      0.092060
      0.425511

      391
      -0.142642
      -0.141563

      392
      0.457797
      0.142104

      393
      -0.029519
      0.053592

      394
      -0.290575
      0.216828

395 rows × 2 columns

### Solution 2.3:
You can use the transform
```
def to_y1y2y3(x1, x2):
    y1 = x1
    y2 = x2
    y3 = x1**2 + x2**2
    return y1, y2, y3
```
and select the best $C$ from the set of values (for example)
```
{'C': [1, 10, 100, 1000, 10**5, 10**6]}
```
It provides
```
best C: 1000000
accuracy on the training set: 1.0
```

### Question 2.4: Execute the code below. There is a solution for SVM classification by some decision curve that looks like a circle and the ROC-curve that have 1 point between $(0, 0)$ and $(1, 1)$. Why this point is only one? Obtain its coordinates by another way - using the confusion matrix. What part of blue points are missclassified?

### Question 2.5: Unlike to explicit transform (like polar coordinates $(x_1, x_2) \to (\phi, r)$ or $(x_1, x_2) \to (y_1, y_2, y_3)$) in primal SVM problem it can be implied implicitly by the corresponding kernel in the equivalent dual SVM problem. Apply RBF (Radial Basis Function) kernel to this example with initial features $x_1$ and $x_2$ and tune the hyperparameters $C$ and $\gamma$.

```python
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import accuracy_score, make_scorer

df = pd.read_csv('https://raw.githubusercontent.com/adasegroup/ML2022_seminars/master/seminar4/data/dataset0.csv')

X = df[['X1', 'X2']]
y = df['y']

# improve the ranges for hyperparameters C and gamma
clf = GridSearchCV(
                   svm.SVC(kernel='rbf'),                 # kernel is changed to non-linear 'rbf'
                   {'C': [1], 'gamma': [1]},              # YOUR IMPROVEMENT HERE !!!
                   scoring=make_scorer(accuracy_score),   # we can skip this line because accuracy_score is by default
                   cv = [(np.array(range(len(y))), np.array(range(len(y))))]  # only 1 split and training and testing sets are the same
                  )
clf.fit(X, y) # note that we use the initial X, not X_transform !
print('best params:', clf.best_params_)
print('accuracy on the training set:', clf.best_score_)

# ROC - curve
plt.figure()
from sklearn.metrics import accuracy_score, make_scorer, roc_curve

fpr, tpr, thresh = roc_curve(y, clf.decision_function(X))
plt.title('ROC curve')
plt.xlabel('False positive rate')
plt.ylabel('True positive rate')
plt.plot(fpr, tpr, '--o')

fpr, tpr, _ = roc_curve(y, clf.predict(X))
plt.title('ROC curve')
plt.xlabel('False positive rate')
plt.ylabel('True positive rate')
plt.plot(fpr, tpr, '--o')

plt.show()
print('points: ', list(zip(fpr, tpr, thresh)))

# Plot classification
xx1 = np.linspace(X['X1'].min(), X['X1'].max(), 30)
xx2 = np.linspace(X['X2'].min(), X['X2'].max(), 30)
XX1, XX2 = np.meshgrid(xx1, xx2)
xy = np.vstack([XX1.ravel(), XX2.ravel()]).T
Z = clf.decision_function(xy).reshape(XX1.shape)
plt.scatter(X['X1'], X['X2'], c=['brown' if c == 1 else 'b' for c in y])
plt.contour(XX1, XX2, Z, colors='k', levels=[0], linestyles=['-'])
plt.title('many blue points were missclassified!')       # YOUR IMPROVEMENT HERE :)
plt.show()
```

### Solution 2.4:
It gets the values of the point:
```
from sklearn.metrics import confusion_matrix
C = confusion_matrix(y, clf.predict(X))
TN = C[0, 0]
FN = C[1, 0]
FP = C[0, 1]
TP = C[1, 1]
tpr = TP/(TP + FN)
fpr = FP/(FP + TN)

print('False positive rate:', fpr)
print('True positive rate:', tpr)
```
Blue points are "negative" or 0-class, browns are "positive" or 1-class. Missclassified blue points are "negative" points that were classified as "positive" but it is False (outside the curve), so their number is FP. The percentage of them among all blue (negative) is exactly fpr.

### Solution 2.5:
We can use for example this grid of hyperparameters:
```
{'C': [1, 100], 'gamma': [1, 100]}
```
It provides
```
best params: {'C': 100, 'gamma': 100}
accuracy on the training set: 1.0
```

### Question 2.6: Here is the example with polar coordinates from above shown in initial features:

```python
df = pd.read_csv('https://raw.githubusercontent.com/adasegroup/ML2022_seminars/master/seminar4/data/dataset0.csv')

X = df[['X1', 'X2']]
y = df['y']

def to_polar(x, y):
    r = np.sqrt(x**2 + y**2)
    phi = np.arctan2(y, x)
    return phi, r

phi, r = to_polar(X['X1'], X['X2'])
X_transform = pd.DataFrame({'phi': phi, 'r': r})

clf = svm.SVC(kernel='linear', C=1)
clf.fit(X_transform, y)
print('accuracy on the training set:', metrics.accuracy_score(y, clf.predict(X_transform)))

plt.scatter(X['X1'], X['X2'], c=['brown' if c == 1 else 'b' for c in y])
plt.xlabel('X1')
plt.ylabel('X2')

xx1 = np.linspace(X['X1'].min(), X['X1'].max(), 30)
xx2 = np.linspace(X['X2'].min(), X['X2'].max(), 30)
XX2, XX1 = np.meshgrid(xx2, xx1)
X1X2 = np.vstack([XX1.ravel(), XX2.ravel()]).T

phi_rho = np.array([to_polar(x1, x2) for x1, x2 in X1X2])
Z = clf.decision_function(phi_rho).reshape(PP.shape)
plt.contour(XX1, XX2, Z, colors='k', levels=[-1, 0, 1], linestyles=['--', '-', '--'])

plt.scatter(X['X1'][clf.support_], X['X2'][clf.support_], s=100, linewidth=1, facecolors='none', edgecolors='k')
plt.show()
```

### Find the expression for the equivalent kernel analytically. Train a model and show the same picture of classification but using the kernel instead of explicit transform (modify the code below):

```python
df = pd.read_csv('https://raw.githubusercontent.com/adasegroup/ML2022_seminars/master/seminar4/data/dataset0.csv')

X = df[['X1', 'X2']]
y = df['y']

def to_polar(x1, x2):
    r = np.sqrt(x1**2 + x2**2)
    varphi = np.arctan2(x2, x1)
    return varphi, r

# modify this function of kernel calculation
def my_kernel(X, Y):
    # here phi(x1, x2) is the basis function of the transformation (varphi, r) = phi(x1, x2)
    # now it is just a linear kernel, make it to be 'kernel of polar coordinates':
    phi_x = np.array([(x1, x2) for x1, x2 in X])
    phi_y = np.array([(x1, x2) for x1, x2 in Y])
    return np.matmul(phi_x, phi_y.T) # all values of inner product phi(x) * phi^T(y)

clf = svm.SVC(kernel=my_kernel, C=1) # kernel is used instead of explicit transform (varphi, r) = phi(x1, x2), dual problem
clf.fit(np.asarray(X), np.asarray(y))                        # initial features X are used, but no explicit polar coordinates!
print('accuracy on the training set:', metrics.accuracy_score(y, clf.predict(np.asarray(X))))

plt.scatter(X['X1'], X['X2'], c=['brown' if c == 1 else 'b' for c in y])
plt.xlabel('X1')
plt.ylabel('X2')

xx1 = np.linspace(X['X1'].min(), X['X1'].max(), 30)
xx2 = np.linspace(X['X2'].min(), X['X2'].max(), 30)
XX2, XX1 = np.meshgrid(xx2, xx1)
X1X2 = np.vstack([XX1.ravel(), XX2.ravel()]).T
Z = clf.decision_function(X1X2).reshape(PP.shape)
plt.contour(XX1, XX2, Z, colors='k', levels=[-1, 0, 1], linestyles=['--', '-', '--'])

plt.scatter(X['X1'][clf.support_], X['X2'][clf.support_], s=100, linewidth=1, facecolors='none', edgecolors='k')
plt.show()
```

### Solution 2.6

For $\phi(x_1, x_2) = \left(\sqrt{x_1^2 + x_2^2}, \mathrm{arctg} \frac{x_2}{x_1}\right)$ we have

$$K(x, y) = \phi(x)^T \phi(y) = \sqrt{(x_1^2 + x_2^2)(y_1^2 + y_2^2)} + \textrm{arctg} \frac{x_2}{x_1} \mathrm{arctg} \frac{y_2}{y_1}.$$

Construct your kernel $K(x, y)=\phi(x)^T\phi(y)$ using `to_polar(x1, x2)` as $\phi(x_1, x_2)$:
```
phi_x = np.array([to_polar(x1, x2) for x1, x2 in X])
phi_y = np.array([to_polar(x1, x2) for x1, x2 in Y])
```

# Topic 3: Training and testing sets

```python
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import accuracy_score, make_scorer
from sklearn.model_selection import train_test_split

df = pd.read_csv('https://raw.githubusercontent.com/adasegroup/ML2022_seminars/master/seminar4/data/dataset1.csv')

X = df[['X1', 'X2']]
y = df['y']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=1/3, random_state=0)

clf = GridSearchCV(svm.SVC(kernel='rbf'),                                     # kernel is 'rbf'!
                            {'C': [1], 'gamma': [1.5]},                       # Underfitting
#                           {'C': [1000], 'gamma': [100]},                    # Overfitting
#                      ... YOUR IMPROVEMENT HERE !!!
                   scoring=make_scorer(accuracy_score),
                   cv=3)    # we can skip this line because accuracy is by default
clf.fit(X_train, y_train) # note that we use X_train, no explicit X_transform !
print('best params:', clf.best_params_)
print('accuracy on the training set:', clf.score(X_train, y_train))
print('accuracy on the testing set:', clf.score(X_test, y_test))

# plots results
xx1 = np.linspace(X['X1'].min(), X['X1'].max(), 50)
xx2 = np.linspace(X['X2'].min(), X['X2'].max(), 50)
XX1, XX2 = np.meshgrid(xx1, xx2)
xy = np.vstack([XX1.ravel(), XX2.ravel()]).T
Z = clf.decision_function(xy).reshape(XX1.shape)

def flter(vv, mask):
    return [v for v, m in zip(vv, mask) if m]

plt.scatter(flter(X_train['X1'], y_train), flter(X_train['X2'], y_train), c='brown', alpha = 0.5, label='y_train == 1')
plt.scatter(flter(X_train['X1'], 1-y_train), flter(X_train['X2'], 1-y_train), c='b', alpha = 0.5, label='y_train == 0')
plt.legend(bbox_to_anchor=(0.999, 1))
plt.contour(XX1, XX2, Z, colors='k', levels=[0], linestyles=['-'])
plt.title('$\mathbf{Training}$ set and $\mathbf{trained}$ decision curve')
plt.show()

plt.scatter(flter(X_test['X1'], y_test), flter(X_test['X2'], y_test), c='brown', label='y_test == 1')
plt.scatter(flter(X_test['X1'], 1-y_test), flter(X_test['X2'], 1-y_test), c='b', label='y_test == 0')
plt.legend(bbox_to_anchor=(0.999, 1))
plt.contour(XX1, XX2, Z, colors='k', levels=[0], linestyles=['-'])
plt.title('$\mathbf{Testing}$ set and $\mathbf{trained}$ decision curve')
plt.show()
```

### Question 3.1: Try different sets of hyperparameters including:
#### Underfitting:
```
accuracy on the training set: 0.707317073171 - poor :(
accuracy on the testing set: 0.780487804878  - poor :(
```
#### Overfitting:
```
accuracy on the training set: 1.0            - even 1.0! but it is useless :|
accuracy on the testing set: 0.841463414634  - poor :(
```
### See the decision curve in both cases (it is the hyperplane in the space of RBF kernel!). Explain them in terms of bias-variance tradeoff.  Tune hyperparameters $C$ and $\gamma$ aiming to increase the score on the testing set.

### Solution 3.1:
We can use for example this grid of hyperparameters:
```
{'C': [1, 10, 100, 10**3, 10**4, 10**5, 10**6],
 'gamma': [1, 1.5, 2, 2.5, 3, 5, 10, 100]},

```
or even shorter ranges. The scores are
```
best params: {'C': 10, 'gamma': 2}
accuracy on the training set: 0.841463414634
accuracy on the testing set: 0.878048780488
```

# Topic 4: Dual SVM

### Question 4.1: In the following example with RBF kernel 4 points are given by their coordinates. Explain why $y([0, 0])=b$. Use formula (1) in the problem formulation in the beginning.

```python
from sklearn import svm, datasets, metrics

# data of 4 points:
X = np.array([[-1, 0], [0, 1], [0, -1], [0, -1.5]])
y = np.array([1, 1, -1, -1])

plt.scatter(X[:, 0], X[:, 1], c=y, s=30, cmap=plt.cm.Paired)

clf = svm.SVC(kernel='rbf', C=2, gamma='auto')
clf.fit(X, y)

# create grid to evaluate model
xx = np.linspace(X[:, 0].min(), X[:, 0].max(), 30)
yy = np.linspace(X[:, 1].min(), X[:, 1].max(), 30)
YY, XX = np.meshgrid(yy, xx)
xy = np.vstack([XX.ravel(), YY.ravel()]).T
Z = clf.decision_function(xy).reshape(XX.shape)

# plot decision boundary and margins
plt.contour(XX, YY, Z, colors='k', levels=[-1, 0, 1], alpha=0.5, linestyles=['--', '-', '--'])
plt.scatter(clf.support_vectors_[:, 0], clf.support_vectors_[:, 1], s=100,
           linewidth=1, facecolors='none', edgecolors='k')
plt.plot([0], [0], 'dr');

# Why y(x_0) == b, x_0=[0, 0]?
print('why are they the same:', clf.decision_function(np.array([[0, 0]])), '==', clf.intercept_, '?')
```

### Solution 4.1:
RBF kernel depends on between-point distance, $K(x, y) = \exp\left(-\frac{||x-y||^2}{2\sigma^2}\right)$, and, since $x_0=(0, 0)$ is equidistant from 3 support vectors, then $K(x_1, x_0)=K(x_2, x_0)=K(x_3, x_0)$. Using the constraint $\sum_{i=1}^n y_i \alpha_i = 0$ and the fact that $x_4=[0, -1.5]$ is not a support vector ($\alpha_4=0$), we obtain

$$
y(x_0)= \sum_{x_i \in S} \alpha_i y_i K(x_i, x_0) + b = K(x_1, x_0) \sum_{x_i \in S} \alpha_i y_i +b = b.
$$

### Question 4.2: Let in the example above  $x_1$ and $x_2$ - brown points, $x_3$  - cyan support vector, $x_4$ - cyan non-support vector. Check what's true:
\begin{align*}
1. \qquad & \alpha_2 = С, \\
2. \qquad & \alpha_1 > \alpha_3,\\
3. \qquad & \alpha_3 - \alpha_1 = \alpha_2,\\
4. \qquad & \alpha_2 - \alpha_3 < 2C?
\end{align*}

### Solution 4.2:

\begin{align*}
1. \qquad & \text{No}, x_2 \ \text{lies on the margin and thus}\ \alpha_2 < С, \\
2. \qquad & \text{No}, \text{because}\ \alpha_1 + \alpha_2 - \alpha_3 - \alpha_4 = 0,\ \alpha_2 > 0\ (\text{because}\ x_2\ \text{is support vector}), \alpha_4 = 0\ (\text{because}\ x_4\ \text{is not a support vector}),\\
3. \qquad & \text{Yes}, \alpha_1 + \alpha_2 - \alpha_3 = 0,\\
4. \qquad & \text{Yes}, \alpha_2 - \alpha_3 <= |\alpha_2 - \alpha_3| <= |\alpha_2| + |\alpha_3| = \alpha_2 + \alpha_3 < 2C,\ \text{since}\ x_2\ \text{and}\ x_3\ \text{are on the margin}.
\end{align*}

```python

```


# SciPy / Havana

In case I get to go to [SciPy / Havana](http://conf.scipyla.org/), I'm boning up on SymPy, an important component within the SciPy Ecosystem.

Here's the kind of thing one might do with SymPy, a computer algebra system:

```python
import sympy as sym
```

```python
x = sym.symbols('x')
sym.init_printing()
```

```python
sym.Integral(sym.sin(x), (x, 0, sym.pi))
```

```python
sym.Integral(sym.sin(x), (x, 0, sym.pi)).doit()
```

In addition, we have numpy, with its multi-dimensional array object.  Numpy comes with a host of numeric recipes, already built in...

```python
import numpy as np
```

[What are the ethics](http://ucsmp.uchicago.edu/resources/conferences/2012-03-01/) of using CAS in a poorly developing country, such as the United States, known for high infant mortality and poverty rates?

Not everyone can easily afford a TI N-spire, and schools tend to not provide adequate computer infrastructure, even for accessing free and open source tools.

Fortunately, even without a TI, we have CAS in the form of the SciPy ecosystem.  Adapting a solution from a CAS blog post, [Quadratics and CAS](https://casmusings.wordpress.com/2017/06/30/quadratics-and-cas/), I was able to derive the same solution for the coefficients a, b and c.

```python
np.polyfit(x=[10,5,-2],y=[210, 40, -30], deg=2)
```

    array([  2.,   4., -30.])

But don't we want algebra students to be able to derive the solution manually?  Yes, we do. However, once this step becomes a means to an end, rather than an end in itself, a CAS will save time and facilitate deeper explorations.

For further reading:
* [Learning Math with Python: Youtube Lectures by Juan Klopper](https://youtu.be/P8DIM41B-mM?list=PLsu0TcgLDUiIqN76ZuRkhDUCd-c_EjIkN)
* [Tutorial de SymPy: introducción y comandos básicos ... ](https://youtu.be/OGQRcYVys1Q)


```python
# initial setup
try:
    # settings colab:
    import google.colab
except ModuleNotFoundError:
    # settings local:
    %run "../../../common/0_notebooks_base_setup.py"
```

---

#### Distribución Binomial

La distribución Binomial es una **distribución de probabilidad discreta**.

Mide el número de éxitos en una secuencia de n ensayos independientes entre sí, con una probabilidad fija p de ocurrencia del éxito entre los ensayos.

Sea n los intentos, x la cantidad de éxitos que buscamos y p la probabilidad de éxito en cada prueba (independientes entre sí).

\begin{equation}
  P(X = x) = {n \choose x} p^x (1-p)^{n-x}
\end{equation}

**Ejemplos**:

* X: cantidad de pacientes en un grupo de 10 para los que fue efectivo un tratamiento médico

* X: cantidad de caras al lanzar una moneda 5 veces

* X: cantidad de 3 al lanzar un dado 10 veces

Una variable aleatoria con distribución Bernoulli es una Binomial con n= 1

---

Vamos a ver ahora cómo generar datos con esta distibución de probabilidad.

Necesitamos un generador de números aleatorios, que expone métodos para generar números aleatorios con alguna distribución de probabilidad especificada. Construimos este generador de este modo `np.random.default_rng()`

https://docs.scipy.org/doc/numpy/reference/random/generator.html

Estas son las distribuciones de probabilidad disponibles:
https://docs.scipy.org/doc/numpy/reference/random/generator.html#distributions

Vamos a generar datos con distribución empleando el método `binomial` con `n` la cantidad de veces que hacemos el test, `p` la probabilidad de éxito
https://docs.scipy.org/doc/numpy/reference/random/generated/numpy.random.Generator.binomial.html#numpy.random.Generator.binomial

```python
import seaborn as sns
def distribution_plotter(data, label, bins=None):
    sns.set(rc={"figure.figsize": (10, 7)})
    sns.set_style("white")
    dist = sns.distplot(data, bins= bins, hist_kws={'alpha':0.2}, kde_kws={'linewidth':5})
    dist.set_title('Distribucion de ' + label + '\n', fontsize=16)
```

Simulemos el resultado de tirar una moneda 10 veces, repitiendo el experimento 10000 veces, y grafiquemos el resultado.

```python
import numpy as np
random_generator = np.random.default_rng()
n=10
p=0.5
sample_size = 100
random_binomial_data = random_generator.binomial(n=n, p=p, size = sample_size)

distribution_plotter(random_binomial_data, "Binomial")
```

Prueben ahora generando 1000, 10000 y 100000 datos y comparen los gráficos. ¿Qué pueden concluir?


# タイソンによるオレゴネーターの簡略化モデル
$$
\begin{equation}
\begin{array}{rcl}
\displaystyle
\epsilon_1 \frac{du}{d\tau} & =&  u(1-u)-\frac{bv(u-a)}{u+a}  \\
\displaystyle
\frac{dv}{d\tau} & =&  u-v
\end{array}
\end{equation}
$$

```python
import numpy as np
from scipy.integrate import solve_ivp
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
sns.set('poster', 'whitegrid', 'dark', rc={"lines.linewidth": 2, 'grid.linestyle': '--'})
```

```python
def tyson(t, x, a, b, e):
    return [(x[0]*(1-x[0]) - b*x[1]*(x[0]-a)/(x[0]+a))/e, x[0] - x[1]]
```

```python
a=0.01
b=1
e=0.1
t0 = 0.0
```

```python
t1 = 100.0
x0 = [0.1, 0.4]
s0 = solve_ivp(tyson, [t0, t1], x0, args=([a,b,e]), method='DOP853', dense_output=True)
```

```python
def f(x, y):
    return (x*(1-x) - b*y*(x-a)/(x+a))/e
def g(x,y):
    return x - y
N = 20000
T = np.linspace(t0, t1, N)
sol = s0.sol(T)
fig = plt.figure(figsize=(8,5))

ax = fig.add_subplot(111)
ax.set_xlabel("$u$")
ax.set_ylabel("$v$")
ax.set_xlim(0.,0.8)
ax.set_ylim(0.,0.5)
Nx, Ny = 17, 17
X, Y = np.meshgrid(np.linspace(0.0, 0.8, Nx), np.linspace(0.0, 0.5, Ny))
U, V = f(X, Y), g(X, Y)
U, V = U/np.hypot(U, V), V/np.hypot(U, V)
ax.quiver(X, Y, U, V, color='darkgray', angles='xy')
ax.plot(np.linspace(0,0.8,2), np.linspace(0,0.8,2), '-', color='gray')
X2 = np.linspace(0,0.8,1024)
ax.plot(X2, (X2+a)*X2*(1-X2)/(b*(X2-a)), '-', color='gray')
ax.plot(sol.T[-N//10:,0], sol.T[-N//10:,1], '-k')
# plt.savefig("tyson.pdf", bbox_inches='tight')
```


# Backpropagation

## Course outline:

1. Backpropagation and chaine rule

3. Lab: with numpy and pytorch

```python
%matplotlib inline
```

## Backpropagation and chaine rule

We will set up a two layer network [source pytorch tuto](https://pytorch.org/tutorials/beginner/pytorch_with_examples.html)
:

$$ \mathbf{Y} = \text{max}(\mathbf{X} \mathbf{W}^{(1)}, 0) \mathbf{W}^{(2)}$$

A fully-connected ReLU network with one hidden layer and no biases, trained to
predict y from x using Euclidean error.

### Chaine rule

Forward pass with **local** partial derivatives of ouput given inputs:

\begin{align*}
x       \rightarrow &  \boxed{z^{(1)} = x^Tw^{(1)}} \rightarrow &  \boxed{h^{(1)} = \max(z^{(1)}, 0)} \rightarrow &  \boxed{z^{(2)}=h^{(1)T}w^{(2)}} \rightarrow &  \boxed{L(z^{(2)}, y) = (z^{(2)} - y)^2}\\
w^{(1)} \nearrow    &                                  &                    w^{(2)} \nearrow & &\\
                    & \frac{\partial z^{(1)}}{\partial w^{(1)}}=x & \frac{\partial h^{(1)}}{\partial z^{(1)}} = \{^{1~\text{if}~z^{(1)}>0}_{\text{else}~0}& \frac{\partial z^{(2)} }{\partial w^{(2)}}=h^{(1)} & \frac{\partial L}{\partial z^{(2)}}=2(z^{(2)}-y)\\
                    & \frac{\partial z^{(1)}}{\partial x}=w^{(1)} & & \frac{\partial z^{(2)} }{\partial h^{(1)}}= w^{(2)} &
\end{align*}

Backward: compute gradient of the loss given each parameters vectors applying chaine rule from the loss downstream to the parameters:

For $w^{(2)}$:

\begin{align}
\frac{\partial L}{\partial w^{(2)}} =& \frac{\partial L}{\partial z^{(2)}} \frac{\partial z^{(2)}}{\partial w^{(2)}}\\
                                    =& 2(z^{(2)}-y) h^{(1)}
\end{align}

For $w^{(1)}$:

\begin{align}
\frac{\partial L}{\partial w^{(1)}} =& \frac{\partial L}{\partial z^{(2)}} \frac{\partial z^{(2)}}{\partial h^{(1)}} \frac{\partial h^{(1)}}{\partial z^{(1)}} \frac{\partial z^{(1)}}{\partial w^{(1)}}\\
                                    =& 2(z^{(2)}-y) w^{(2)} \{^{1~\text{if}~z^{(1)}>0}_{\text{else}~0} x
\end{align}

### Recap: Vector derivatives

Given a function $z = x`w$ with $z$ the output, $x$ the input and $w$ the coeficients.

- Scalar to Scalar: $x \in \mathbb{R}, z \in \mathbb{R}$, $w \in \mathbb{R}$

Regular derivative:
$$
\frac{\partial z}{\partial w} = x \in \mathbb{R}
$$

If $w$ changes by a small amount, how much will $z$ change?

- Vector to Scalar: $x \in \mathbb{R}^N, z \in \mathbb{R}$, $w \in \mathbb{R}^N$

Derivative is **Gradient** of partial derivative:
$\frac{\partial z}{\partial w} \in \mathbb{R}^N$

\begin{align}
\frac{\partial z}{\partial w} = \nabla_w z &= \begin{bmatrix}
       \frac{\partial z}{\partial w_1} \\
       \vdots \\
       \frac{\partial z}{\partial w_i} \\
       \vdots \\
       \frac{\partial z}{\partial w_N}
     \end{bmatrix}
\end{align}

For each element $w_i$ of $w$, if it changes by a small amount then how much will y change?

- Vector to Vector: $w \in \mathbb{R}^N, z \in \mathbb{R}^M$

Derivative is **Jacobian** of partial derivative:

TO COMPLETE

$\frac{\partial z}{\partial w} \in \mathbb{R}^{N \times M}$

### Backpropagation summary

Backpropagation algorithm in a graph:
1. Forward pass, for each node compute local partial derivatives of ouput given inputs
2. Backward pass: apply chain rule from the end to each parameters
    - Update parameter with gradient descent using the current upstream gradient and the current local gradient
    - Compute upstream gradient for the backward nodes

Think locally and remember that at each node:
- For the loss the gradient is the error
- At each step, the upstream gradient is obtained by multiplying the upstream gradient (an error) with the current parameters (vector of matrix).
- At each step, the current local gradient equal the input, therfore the current update is the current upstream gradient time the input.

```python
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import sklearn.model_selection
```

## Lab: with numpy and pytorch

### Load iris data set
Goal: Predict Y = [petal_length, petal_width] = f(X = [sepal_length, sepal_width])

- Plot data with seaborn
- Remove setosa samples
- Recode 'versicolor':1, 'virginica':2
- Scale X and Y
- Split data in train/test 50%/50%

```python
iris = sns.load_dataset("iris")
#g = sns.pairplot(iris, hue="species")
df = iris[iris.species != "setosa"]
g = sns.pairplot(df, hue="species")
df['species_n'] = iris.species.map({'versicolor':1, 'virginica':2})

# Y = 'petal_length', 'petal_width'; X = 'sepal_length', 'sepal_width')
X_iris = np.asarray(df.loc[:, ['sepal_length', 'sepal_width']], dtype=np.float32)
Y_iris = np.asarray(df.loc[:, ['petal_length', 'petal_width']], dtype=np.float32)
label_iris = np.asarray(df.species_n, dtype=int)

# Scale
from sklearn.preprocessing import StandardScaler
scalerx, scalery = StandardScaler(), StandardScaler()
X_iris = scalerx.fit_transform(X_iris)
Y_iris = StandardScaler().fit_transform(Y_iris)

# Split train test
X_iris_tr, X_iris_val, Y_iris_tr, Y_iris_val, label_iris_tr, label_iris_val = \
    sklearn.model_selection.train_test_split(X_iris, Y_iris, label_iris, train_size=0.5, stratify=label_iris)
```

### Backpropagation with numpy

This implementation uses numpy to manually compute the forward pass, loss, and
backward pass.

```python
# X=X_iris_tr; Y=Y_iris_tr; X_val=X_iris_val; Y_val=Y_iris_val

def two_layer_regression_numpy_train(X, Y, X_val, Y_val, lr, nite):
    # N is batch size; D_in is input dimension;
    # H is hidden dimension; D_out is output dimension.
    # N, D_in, H, D_out = 64, 1000, 100, 10
    N, D_in, H, D_out = X.shape[0], X.shape[1], 100, Y.shape[1]

    W1 = np.random.randn(D_in, H)
    W2 = np.random.randn(H, D_out)

    losses_tr, losses_val = list(), list()

    learning_rate = lr
    for t in range(nite):
        # Forward pass: compute predicted y
        z1 = X.dot(W1)
        h1 = np.maximum(z1, 0)
        Y_pred = h1.dot(W2)

        # Compute and print loss
        loss = np.square(Y_pred - Y).sum()

        # Backprop to compute gradients of w1 and w2 with respect to loss
        grad_y_pred = 2.0 * (Y_pred - Y)
        grad_w2 = h1.T.dot(grad_y_pred)
        grad_h1 = grad_y_pred.dot(W2.T)
        grad_z1 = grad_h1.copy()
        grad_z1[z1 < 0] = 0
        grad_w1 = X.T.dot(grad_z1)

        # Update weights
        W1 -= learning_rate * grad_w1
        W2 -= learning_rate * grad_w2

        # Forward pass for validation set: compute predicted y
        z1 = X_val.dot(W1)
        h1 = np.maximum(z1, 0)
        y_pred_val = h1.dot(W2)
        loss_val = np.square(y_pred_val - Y_val).sum()

        losses_tr.append(loss)
        losses_val.append(loss_val)

        if t % 10 == 0:
            print(t, loss, loss_val)

    return W1, W2, losses_tr, losses_val

W1, W2, losses_tr, losses_val = two_layer_regression_numpy_train(X=X_iris_tr, Y=Y_iris_tr, X_val=X_iris_val, Y_val=Y_iris_val,
                                                                 lr=1e-4, nite=50)
plt.plot(np.arange(len(losses_tr)), losses_tr, "-b", np.arange(len(losses_val)), losses_val, "-r")
```

### Backpropagation with PyTorch Tensors

[source](https://pytorch.org/tutorials/beginner/pytorch_with_examples.html)

Numpy is a great framework, but it cannot utilize GPUs to accelerate its numerical computations.
For modern deep neural networks, GPUs often provide speedups of 50x or greater, so unfortunately numpy won’t
be enough for modern deep learning.
Here we introduce the most fundamental PyTorch concept: the Tensor. A PyTorch Tensor is conceptually identical
to a numpy array: a Tensor is an n-dimensional array, and PyTorch provides many functions for operating on these Tensors.
Behind the scenes, Tensors can keep track of a computational graph and gradients, but they’re also useful as a generic tool for scientific computing.
Also unlike numpy, PyTorch Tensors can utilize GPUs to accelerate their numeric computations. To run a PyTorch Tensor
on GPU, you simply need to cast it to a new datatype.
Here we use PyTorch Tensors to fit a two-layer network to random data. Like the numpy example above we need to manually
implement the forward and backward passes through the network:

```python
import torch

# X=X_iris_tr; Y=Y_iris_tr; X_val=X_iris_val; Y_val=Y_iris_val

def two_layer_regression_tensor_train(X, Y, X_val, Y_val, lr, nite):

    dtype = torch.float
    device = torch.device("cpu")
    # device = torch.device("cuda:0") # Uncomment this to run on GPU

    # N is batch size; D_in is input dimension;
    # H is hidden dimension; D_out is output dimension.
    N, D_in, H, D_out = X.shape[0], X.shape[1], 100, Y.shape[1]

    # Create random input and output data
    X = torch.from_numpy(X)
    Y = torch.from_numpy(Y)
    X_val = torch.from_numpy(X_val)
    Y_val = torch.from_numpy(Y_val)

    # Randomly initialize weights
    W1 = torch.randn(D_in, H, device=device, dtype=dtype)
    W2 = torch.randn(H, D_out, device=device, dtype=dtype)

    losses_tr, losses_val = list(), list()

    learning_rate = lr
    for t in range(nite):
        # Forward pass: compute predicted y
        z1 = X.mm(W1)
        h1 = z1.clamp(min=0)
        y_pred = h1.mm(W2)

        # Compute and print loss
        loss = (y_pred - Y).pow(2).sum().item()

        # Backprop to compute gradients of w1 and w2 with respect to loss
        grad_y_pred = 2.0 * (y_pred - Y)
        grad_w2 = h1.t().mm(grad_y_pred)
        grad_h1 = grad_y_pred.mm(W2.t())
        grad_z1 = grad_h1.clone()
        grad_z1[z1 < 0] = 0
        grad_w1 = X.t().mm(grad_z1)

        # Update weights using gradient descent
        W1 -= learning_rate * grad_w1
        W2 -= learning_rate * grad_w2

        # Forward pass for validation set: compute predicted y
        z1 = X_val.mm(W1)
        h1 = z1.clamp(min=0)
        y_pred_val = h1.mm(W2)
        loss_val = (y_pred_val - Y_val).pow(2).sum().item()

        losses_tr.append(loss)
        losses_val.append(loss_val)

        if t % 10 == 0:
            print(t, loss, loss_val)

    return W1, W2, losses_tr, losses_val

W1, W2, losses_tr, losses_val = two_layer_regression_tensor_train(X=X_iris_tr, Y=Y_iris_tr, X_val=X_iris_val, Y_val=Y_iris_val,
                                                                 lr=1e-4, nite=50)

plt.plot(np.arange(len(losses_tr)), losses_tr, "-b", np.arange(len(losses_val)), losses_val, "-r")
```

### Backpropagation with PyTorch: Tensors and autograd

[source](https://pytorch.org/tutorials/beginner/pytorch_with_examples.html)

A fully-connected ReLU network with one hidden layer and no biases, trained to
predict y from x by minimizing squared Euclidean distance.
This implementation computes the forward pass using operations on PyTorch
Tensors, and uses PyTorch autograd to compute gradients.
A PyTorch Tensor represents a node in a computational graph. If ``x`` is a
Tensor that has ``x.requires_grad=True`` then ``x.grad`` is another Tensor
holding the gradient of ``x`` with respect to some scalar value.

```python
import torch

# X=X_iris_tr; Y=Y_iris_tr; X_val=X_iris_val; Y_val=Y_iris_val
# del X, Y, X_val, Y_val

def two_layer_regression_autograd_train(X, Y, X_val, Y_val, lr, nite):

    dtype = torch.float
    device = torch.device("cpu")
    # device = torch.device("cuda:0") # Uncomment this to run on GPU

    # N is batch size; D_in is input dimension;
    # H is hidden dimension; D_out is output dimension.
    N, D_in, H, D_out = X.shape[0], X.shape[1], 100, Y.shape[1]

    # Setting requires_grad=False indicates that we do not need to compute gradients
    # with respect to these Tensors during the backward pass.
    X = torch.from_numpy(X)
    Y = torch.from_numpy(Y)
    X_val = torch.from_numpy(X_val)
    Y_val = torch.from_numpy(Y_val)

    # Create random Tensors for weights.
    # Setting requires_grad=True indicates that we want to compute gradients with
    # respect to these Tensors during the backward pass.
    W1 = torch.randn(D_in, H, device=device, dtype=dtype, requires_grad=True)
    W2 = torch.randn(H, D_out, device=device, dtype=dtype, requires_grad=True)

    losses_tr, losses_val = list(), list()

    learning_rate = lr
    for t in range(nite):
        # Forward pass: compute predicted y using operations on Tensors; these
        # are exactly the same operations we used to compute the forward pass using
        # Tensors, but we do not need to keep references to intermediate values since
        # we are not implementing the backward pass by hand.
        y_pred = X.mm(W1).clamp(min=0).mm(W2)

        # Compute and print loss using operations on Tensors.
        # Now loss is a Tensor of shape (1,)
        # loss.item() gets the scalar value held in the loss.
        loss = (y_pred - Y).pow(2).sum()

        # Use autograd to compute the backward pass. This call will compute the
        # gradient of loss with respect to all Tensors with requires_grad=True.
        # After this call w1.grad and w2.grad will be Tensors holding the gradient
        # of the loss with respect to w1 and w2 respectively.
        loss.backward()

        # Manually update weights using gradient descent. Wrap in torch.no_grad()
        # because weights have requires_grad=True, but we don't need to track this
        # in autograd.
        # An alternative way is to operate on weight.data and weight.grad.data.
        # Recall that tensor.data gives a tensor that shares the storage with
        # tensor, but doesn't track history.
        # You can also use torch.optim.SGD to achieve this.
        with torch.no_grad():
            W1 -= learning_rate * W1.grad
            W2 -= learning_rate * W2.grad

            # Manually zero the gradients after updating weights
            W1.grad.zero_()
            W2.grad.zero_()

            y_pred = X_val.mm(W1).clamp(min=0).mm(W2)

            # Compute and print loss using operations on Tensors.
            # Now loss is a Tensor of shape (1,)
            # loss.item() gets the scalar value held in the loss.
            loss_val = (y_pred - Y).pow(2).sum()

        if t % 10 == 0:
            print(t, loss.item(), loss_val.item())

        losses_tr.append(loss.item())
        losses_val.append(loss_val.item())

    return W1, W2, losses_tr, losses_val

W1, W2, losses_tr, losses_val = two_layer_regression_autograd_train(X=X_iris_tr, Y=Y_iris_tr, X_val=X_iris_val, Y_val=Y_iris_val,
                                                                 lr=1e-4, nite=50)
plt.plot(np.arange(len(losses_tr)), losses_tr, "-b", np.arange(len(losses_val)), losses_val, "-r")
```

### Backpropagation with PyTorch: nn

[source](https://pytorch.org/tutorials/beginner/pytorch_with_examples.html)

This implementation uses the nn package from PyTorch to build the network.
PyTorch autograd makes it easy to define computational graphs and take gradients,
but raw autograd can be a bit too low-level for defining complex neural networks;
this is where the nn package can help. The nn package defines a set of Modules,
which you can think of as a neural network layer that has produces output from
input and may have some trainable weights.

```python
import torch

# X=X_iris_tr; Y=Y_iris_tr; X_val=X_iris_val; Y_val=Y_iris_val
# del X, Y, X_val, Y_val

def two_layer_regression_nn_train(X, Y, X_val, Y_val, lr, nite):

    # N is batch size; D_in is input dimension;
    # H is hidden dimension; D_out is output dimension.
    N, D_in, H, D_out = X.shape[0], X.shape[1], 100, Y.shape[1]

    X = torch.from_numpy(X)
    Y = torch.from_numpy(Y)
    X_val = torch.from_numpy(X_val)
    Y_val = torch.from_numpy(Y_val)

    # Use the nn package to define our model as a sequence of layers. nn.Sequential
    # is a Module which contains other Modules, and applies them in sequence to
    # produce its output. Each Linear Module computes output from input using a
    # linear function, and holds internal Tensors for its weight and bias.
    model = torch.nn.Sequential(
        torch.nn.Linear(D_in, H),
        torch.nn.ReLU(),
        torch.nn.Linear(H, D_out),
    )

    # The nn package also contains definitions of popular loss functions; in this
    # case we will use Mean Squared Error (MSE) as our loss function.
    loss_fn = torch.nn.MSELoss(reduction='sum')

    losses_tr, losses_val = list(), list()

    learning_rate = lr
    for t in range(nite):
        # Forward pass: compute predicted y by passing x to the model. Module objects
        # override the __call__ operator so you can call them like functions. When
        # doing so you pass a Tensor of input data to the Module and it produces
        # a Tensor of output data.
        y_pred = model(X)

        # Compute and print loss. We pass Tensors containing the predicted and true
        # values of y, and the loss function returns a Tensor containing the
        # loss.
        loss = loss_fn(y_pred, Y)

        # Zero the gradients before running the backward pass.
        model.zero_grad()

        # Backward pass: compute gradient of the loss with respect to all the learnable
        # parameters of the model. Internally, the parameters of each Module are stored
        # in Tensors with requires_grad=True, so this call will compute gradients for
        # all learnable parameters in the model.
        loss.backward()

        # Update the weights using gradient descent. Each parameter is a Tensor, so
        # we can access its gradients like we did before.
        with torch.no_grad():
            for param in model.parameters():
                param -= learning_rate * param.grad
            y_pred = model(X_val)
            loss_val = (y_pred - Y_val).pow(2).sum()

        if t % 10 == 0:
            print(t, loss.item(), loss_val.item())

        losses_tr.append(loss.item())
        losses_val.append(loss_val.item())

    return model, losses_tr, losses_val

model, losses_tr, losses_val = two_layer_regression_nn_train(X=X_iris_tr, Y=Y_iris_tr, X_val=X_iris_val, Y_val=Y_iris_val,
                                                                 lr=1e-4, nite=50)

plt.plot(np.arange(len(losses_tr)), losses_tr, "-b", np.arange(len(losses_val)), losses_val, "-r")
```

### Backpropagation with PyTorch optim

This implementation uses the nn package from PyTorch to build the network.
Rather than manually updating the weights of the model as we have been doing,
we use the optim package to define an Optimizer that will update the weights
for us. The optim package defines many optimization algorithms that are commonly
used for deep learning, including SGD+momentum, RMSProp, Adam, etc.

```python
import torch

# X=X_iris_tr; Y=Y_iris_tr; X_val=X_iris_val; Y_val=Y_iris_val

def two_layer_regression_nn_optim_train(X, Y, X_val, Y_val, lr, nite):

    # N is batch size; D_in is input dimension;
    # H is hidden dimension; D_out is output dimension.
    N, D_in, H, D_out = X.shape[0], X.shape[1], 100, Y.shape[1]

    X = torch.from_numpy(X)
    Y = torch.from_numpy(Y)
    X_val = torch.from_numpy(X_val)
    Y_val = torch.from_numpy(Y_val)

    # Use the nn package to define our model and loss function.
    model = torch.nn.Sequential(
        torch.nn.Linear(D_in, H),
        torch.nn.ReLU(),
        torch.nn.Linear(H, D_out),
    )
    loss_fn = torch.nn.MSELoss(reduction='sum')

    losses_tr, losses_val = list(), list()

    # Use the optim package to define an Optimizer that will update the weights of
    # the model for us. Here we will use Adam; the optim package contains many other
    # optimization algoriths. The first argument to the Adam constructor tells the
    # optimizer which Tensors it should update.
    learning_rate = lr
    optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
    for t in range(nite):
        # Forward pass: compute predicted y by passing x to the model.
        y_pred = model(X)

        # Compute and print loss.
        loss = loss_fn(y_pred, Y)

        # Before the backward pass, use the optimizer object to zero all of the
        # gradients for the variables it will update (which are the learnable
        # weights of the model). This is because by default, gradients are
        # accumulated in buffers( i.e, not overwritten) whenever .backward()
        # is called. Checkout docs of torch.autograd.backward for more details.
        optimizer.zero_grad()

        # Backward pass: compute gradient of the loss with respect to model
        # parameters
        loss.backward()

        # Calling the step function on an Optimizer makes an update to its
        # parameters
        optimizer.step()

        with torch.no_grad():
            y_pred = model(X_val)
            loss_val = loss_fn(y_pred, Y_val)

        if t % 10 == 0:
            print(t, loss.item(), loss_val.item())

        losses_tr.append(loss.item())
        losses_val.append(loss_val.item())

    return model, losses_tr, losses_val

model, losses_tr, losses_val = two_layer_regression_nn_optim_train(X=X_iris_tr, Y=Y_iris_tr, X_val=X_iris_val, Y_val=Y_iris_val,
                                                                 lr=1e-3, nite=50)
plt.plot(np.arange(len(losses_tr)), losses_tr, "-b", np.arange(len(losses_val)), losses_val, "-r")
```

```python

```