$\newcommand{\xv}{\mathbf{x}}
 \newcommand{\wv}{\mathbf{w}}
 \newcommand{\yv}{\mathbf{y}}
 \newcommand{\zv}{\mathbf{z}}
 \newcommand{\Chi}{\mathcal{X}}
 \newcommand{\R}{\rm I\!R}
 \newcommand{\sign}{\text{sign}}
 \newcommand{\Tm}{\mathbf{T}}
 \newcommand{\Xm}{\mathbf{X}}
 \newcommand{\Xlm}{\mathbf{X1}}
 \newcommand{\Wm}{\mathbf{W}}
 \newcommand{\Vm}{\mathbf{V}}
 \newcommand{\Ym}{\mathbf{Y}}
 \newcommand{\Zm}{\mathbf{Z}}
 \newcommand{\Zlm}{\mathbf{Z1}}
 \newcommand{\I}{\mathbf{I}}
 \newcommand{\muv}{\boldsymbol\mu}
 \newcommand{\Sigmav}{\boldsymbol\Sigma}
 \newcommand{\Phiv}{\boldsymbol\Phi}
$

# Neural Networks

Neural networks, or artificial neural networks, are the computational models inspired by the brain. Mimicing the neurons' synaptic connecions (Figure 1), we build or stack multiple neuron-like hidden units to map data into nonlinear space for rich representation.

<center>Figure 1. Anatomy of a neuron (wikipedia) </center>

Now, let us review the perceptron model.

In perceptron, passing the output of linear model to the step function, we get discrete outputs.
Now, you can think a perceptron as a neuron. With a threshold zero, when the linear model outputs are over it, it passes the signal to next neuron.

By connecting the perceptrons, we can actually build synaptic connections.
We call this model as *multi-layer perceptron* (MLP).

**Q:** For inputs $x \in \{-1, +1 \}$, think about what the following picture represents and answer for it.

1)

2)

3)

Answer:

1) +1

2) -1

3) -1

## Feed Forward Neural Networks

Fitting the data with MLP is a combinatorial optimization problem with non-smooth step function.
So, we can consider smooth step function, a s-shaped sigmoid function.
We call this smooth function as **activation function**.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

fig, ax = plt.subplots()

# x - y axis
ax.axhline(y=0, color='k', linewidth=1)
ax.axvline(x=0, color='k', linewidth=1)

# step function in blue
plt.plot([0, 6], [1, 1], 'b-', linewidth=3)
plt.plot([-6, 0], [-1, -1], 'b-', linewidth=3)
plt.plot([0, 0], [-1, 1], 'b-', linewidth=3)

# tanh in red
x = np.linspace(-6, 6, 100)
plt.plot(x, np.tanh(x), 'r-', linewidth=3)

```

## Non-linear Extension of Linear Model

As we discussed, feed forward neural networks have a rich representation. Thus, it can represent the linear model with single layer.

Considering the multiple outputs, we formulated this in matrix:

$$
\begin{align}
E &= \frac{1}{N} \frac{1}{K} \sum_{n=1}^{N} \sum_{k=1}^{K} (t_{nk} - y_{nk})^2 \\
\\
\Ym &= \Xlm \cdot \Wm
\end{align}
$$

Here, we assume the first column of $\Xlm$ is the bias column with 1's.
Thus, the weight matrix $\Wm$ is $(D+1) \times K$ with the bias row in the first row.

From this model, we can convert the raw data $\Xm$ to $\Phiv$, which is a nonlinear mapping.

$$
\phi: \Xm \rightarrow \Phiv
$$

Then, we can rewrite the linear model with as follows:

$$
\begin{align}
E &= \frac{1}{N} \frac{1}{K} \sum_{n=1}^{N} \sum_{k=1}^{K} (t_{nk} - y_{nk})^2
\\
\Ym &= \Phiv \Wm \\
\\
\Ym_{nk} &= \Phiv_n^\top \Wm_k
\end{align}
$$

Now, let $\phi(\xv) = h(\xv)$ where $h$ is the *activation function*.

$$
\begin{align}
\Zm &= h(\Xlm \cdot \Vm) \\
\\
\Ym & = \Zlm \cdot \Wm
\end{align}
$$

Figure below depics this model.

The size of each matrix is listed:
- $\Xm: N \times D$
- $\Xlm: N \times (D+1)$
- $\Vm: (D+1) \times G$
- $\Zm: N \times G$
- $\Zlm: N \times (G+1)$
- $\Wm: (G+1) \times K$
- $\Ym: N \times K$

For this two-layer network, we call the blue circle layer with the activation functions as **hidden layer** and the organge layer with summation as **output layer**.

# Why Sigmoid?

The resemblance to the step function can be good reason. But is there any other reason for choosing a sigmoid function as activation?

Let us take a look at a polinomial function and the sigmoid.

$$
y = x^4 + 3 x^2 + 7 x + 3   \quad\quad\text{vs.}\quad\quad y = tanh(x)
$$

```python
# polinomial function
def h_poly(x):
    return x**4 + 3 * x**2 + 7 * x + 3

# sigmoid function
def h_sigmoid(x):
    return np.tanh(x)

##### Gradient functions

# polinomial function
def dh_poly(x):
    return 4 * x**3 + 6 * x + 7

# polinomial function
def dh_sigmoid(x):
    h = h_sigmoid(x)
    return 1 - h ** 2

x = np.linspace(-6, 6, 100)

plt.figure(figsize=(16,8))
plt.subplot(121)
plt.plot(x, h_poly(x), label="$y = x^4 + 3 x^2 + 7 x + 3$")
plt.plot(x, dh_poly(x), label="$dy$")
plt.legend()

plt.subplot(122)
plt.plot(x, h_sigmoid(x), label="$y = tanh(x)$")
plt.plot(x, dh_sigmoid(x), label="$dy$")
plt.legend()

```

Here, we can see the polinomial gradients are very huge when $x$ is moving away from 0. A gradient descent procedure takes this huge step for the large positive or negative $x$ values, which can make learning divergent and unstable.

In the right figure, we can see the gradient is nearly turned off for large $x$ values. Only on the nonlinear region of sigmoid function, small gradient is applied for stable learning.

# Gradient Descent

From the error function $E$,

$$
E = \frac{1}{N} \frac{1}{K}\sum_{n=1}^{N} \sum_{k=1}^{K} (t_{nk} - y_{nk})^2,
$$

we can derive the gradient to update the weights for each layer.

Since we can change the output and eventually the error by changing the weights $\Vm$ and $\Wm$,

$$
\begin{align}
v_{dg} &\leftarrow v_{dg} - \alpha_h \frac{\partial{E}} {\partial{v_{dg}}} \\
\\
w_{gk} &\leftarrow w_{gk} - \alpha_o \frac{\partial{E}} {\partial{w_{gk}}},
\end{align}
$$

where $\alpha_h$ and $\alpha_o$ are the learning rate for hidden and output layer respectively.

$$
\begin{align}
\frac{\partial{E}}{\partial{w_{gk}}} &= \frac{\partial{\Big( \frac{1}{N} \frac{1}{K} \sum_{n=1}^{N} \sum_{l=1}^{K} (t_{nl} - y_{nl})^2} \Big)}{\partial{w_{gk}}} \\
   &= -2  \frac{1}{N} \frac{1}{K} \sum_{n=1}^{N}(t_{nk} - y_{nk}) \frac{\partial{y_{nl}}}{\partial{w_{gk}}}
\end{align}
$$

where

$$
y_{nl} = z1_{n}^\top w_{*l} = \sum_{g=0}^{G} z1_{ng} w_{gl} .
$$

The gradient for the output layer can be computed as follows:
$$
\begin{align}
\frac{\partial{E}}{\partial{w_{gk}}} &= -2 \frac{1}{N} \frac{1}{K} \sum_{n=1}^{N} (t_{nk} - y_{nk})  z1_{nk} \\
    &= -2 \frac{1}{N} \frac{1}{K} \Zlm^\top (\Tm - \Ym).
\end{align}
$$

For the hidden layer,

$$
\begin{align}
\frac{\partial{E}}{\partial{v_{dg}}} &= \frac{\partial{\Big( \frac{1}{N} \frac{1}{K} \sum_{n=1}^{N} \sum_{l=1}^{K} (t_{nl} - y_{nl})^2} \Big)}{\partial{v_{dg}}} \\
   &= -2  \frac{1}{N} \frac{1}{K} \sum_{n=1}^{N} \sum_{l=1}^{K} (t_{nl} - y_{nl}) \frac{\partial{y_{nl}}}{\partial{v_{dg}}}
\end{align}
$$

where

$$
y_{nl} = \sum_{g=0}^{G} z1_{ng} w_{gl} = \sum_{g=0}^G w_{gl} h (\sum_{d=0}^D v_{dg} x1_{nd}) .
$$

Let $a_{ng} = \sum_{d=0}^D x1_{nd} v_{dg}$. Then, we can use a chain rule for the derivation.

$$
\begin{align}
\frac{\partial{E}}{\partial{v_{dg}}} &= -2  \frac{1}{N} \frac{1}{K} \sum_{n=1}^{N} \sum_{l=1}^{K} (t_{nl} - y_{nl}) \frac{\partial{\Big(  \sum_{q=0}^G w_{ql} h (\sum_{p=0}^D v_{pq} x1_{np}) \Big)}}{\partial{v_{dg}}} \\
    &= -2  \frac{1}{N} \frac{1}{K} \sum_{n=1}^{N} \sum_{l=1}^{K} (t_{nl} - y_{nl}) \sum_{q=0}^G w_{ql} \frac{\partial{\Big( h (\sum_{p=0}^D v_{pq} x1_{np}) \Big)}}{\partial{v_{dg}}} \\
    &= -2  \frac{1}{N} \frac{1}{K} \sum_{n=1}^{N} \sum_{l=1}^{K} (t_{nl} - y_{nl}) \sum_{q=0}^G w_{ql} \frac{\partial{h(a_{ng})}}{\partial{a_{ng}}} \frac{\partial{a_{ng}}}{\partial{v_{dg}}} \\
    &= -2  \frac{1}{N} \frac{1}{K} \sum_{n=1}^{N} \sum_{l=1}^{K} (t_{nl} - y_{nl}) \sum_{q=0}^G w_{ql} \frac{\partial{h(a_{ng})}}{\partial{a_{ng}}} x1_{nd}.
\end{align}
$$

When $h = tanh$,

$$
\frac{\partial{h(a_{ng})}}{\partial{a_{ng}}} = \frac{z_{ng}}{\partial{a_{ng}}} = (1 - z_{ng}^2).
$$

Thus,

$$
\frac{\partial{E}}{\partial{v_{dg}}} = -2  \frac{1}{N} \frac{1}{K} \sum_{n=1}^{N} \sum_{l=1}^{K} (t_{nk} - y_{nl}) \sum_{g=0}^G w_{gl} (1 - z_{ng}^2) x1_{nd}.
$$

Rewriting this in matrix form,

$$
\frac{\partial{E}}{\partial{v_{dg}}} = -2 \frac{1}{N} \frac{1}{K} \Xlm^\top \Big( (\Tm - \Ym) \Wm^\top \odot (1 - \Zm^2) \Big).
$$

Here, $\odot$ denotes the element-wise multiplication.

To summarize, the backpropagation performs the this weight updates iteratively:
$$
\begin{align}
\Vm &\leftarrow \Vm + \rho_h \frac{1}{N} \frac{1}{K} \Xlm^\top \Big( (\Tm - \Ym) \Wm^\top \odot (1 - \Zm^2) \Big), \\
\Wm &\leftarrow \Wm + \rho_o \frac{1}{N} \frac{1}{K} \Zlm^\top \Big( \Tm - \Ym \Big)
\end{align}
$$
where $\rho_h$ and $\rho_o$ are the learning rate for hidden and output layer weights.

Implemented iteration follows.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

import IPython.display as ipd  # for display and clear_output
import time  # for sleep
```

```python
# Make some training data
n = 20
X = np.linspace(0.,20.0,n).reshape((n,1)) - 10
T = 0.2 + 0.05 * (X+10) + 0.4 * np.sin(X+10) + 0.2 * np.random.normal(size=(n,1))

# Make some testing data
Xtest = X + 0.1*np.random.normal(size=(n,1))
Ttest = 0.2 + 0.05 * (Xtest+10) + 0.4 * np.sin(Xtest+10) + 0.2 * np.random.normal(size=(n,1))

nSamples = X.shape[0]
nOutputs = T.shape[1]
```

```python
# Set parameters of neural network
nHiddens = 10

rhoh = 0.5
rhoo = 0.1

rh = rhoh / (nSamples*nOutputs)
ro = rhoo / (nSamples*nOutputs)

# Initialize weights to uniformly distributed values between small normally-distributed between -0.1 and 0.1
V = 0.1*2*(np.random.uniform(size=(1+1,nHiddens))-0.5)
W = 0.1*2*(np.random.uniform(size=(1+nHiddens,nOutputs))-0.5)

# Add constant column of 1's
def addOnes(A):
    return np.insert(A, 0, 1, axis=1)
X1 = addOnes(X)
Xtest1 = addOnes(Xtest)

# Take nReps steepest descent steps in gradient descent search in mean-squared-error function
nReps = 30000
# collect training and testing errors for plotting
errorTrace = np.zeros((nReps,2))

N_ = X1.shape[0]
K_ = W.shape[1]

fig = plt.figure(figsize=(10,8))
for reps in range(nReps):

    # Forward pass on training data
    Z = np.tanh(X1 @ V)
    Z1 = addOnes(Z)
    Y = Z1 @ W

    # Error in output
    error = T - Y

    print("V:", V.shape)
    print("X1:", X1.T.shape)
    print("error:", error.shape)
    print("W.T:", W.T.shape)
    print("Z:", Z.shape)
    print(np.square(Z).shape)

    # TODO: Backward pass - the backpropagation and weight update steps
    V = V + ((rh / (N_ * K_)) * X1.T * ((error * W.T) @ (1 - np.square(Z))))
    W = W + (ro / (N_ * K_)) * Z1.T * error

    # error traces for plotting
    errorTrace[reps,0] = np.sqrt(np.mean((error**2)))
    Ytest = addOnes(np.tanh(Xtest1 @ V)) @ W  #!! Forward pass in one line
    errorTrace[reps,1] = np.sqrt(np.mean((Ytest-Ttest)**2))

    if reps % 1000 == 0 or reps == nReps-1:
        plt.clf()
        plt.subplot(3,1,1)
        plt.plot(errorTrace[:reps,:])
        plt.ylim(0,0.7)
        plt.xlabel('Epochs')
        plt.ylabel('RMSE')
        plt.legend(('Train','Test'),loc='upper left')

        plt.subplot(3,1,2)
        plt.plot(X,T,'o-',Xtest,Ttest,'o-',Xtest,Ytest,'o-')
        plt.xlim(-10,10)
        plt.legend(('Training','Testing','Model'),loc='upper left')
        plt.xlabel('$x$')
        plt.ylabel('Actual and Predicted $f(x)$')

        plt.subplot(3,1,3)
        plt.plot(X,Z)
        plt.ylim(-1.1,1.1)
        plt.xlabel('$x$')
        plt.ylabel('Hidden Unit Outputs ($z$)');

        ipd.clear_output(wait=True)
        ipd.display(fig)
ipd.clear_output(wait=True)
```

```python

```

$\newcommand{\xv}{\mathbf{x}}
 \newcommand{\wv}{\mathbf{w}}
 \newcommand{\yv}{\mathbf{y}}
 \newcommand{\zv}{\mathbf{z}}
 \newcommand{\av}{\mathbf{a}}
 \newcommand{\Chi}{\mathcal{X}}
 \newcommand{\R}{\rm I\!R}
 \newcommand{\sign}{\text{sign}}
 \newcommand{\Tm}{\mathbf{T}}
 \newcommand{\Xm}{\mathbf{X}}
 \newcommand{\Xlm}{\mathbf{X1}}
 \newcommand{\Wm}{\mathbf{W}}
 \newcommand{\Vm}{\mathbf{V}}
 \newcommand{\Ym}{\mathbf{Y}}
 \newcommand{\Zm}{\mathbf{Z}}
 \newcommand{\Zlm}{\mathbf{Z1}}
 \newcommand{\I}{\mathbf{I}}
 \newcommand{\muv}{\boldsymbol\mu}
 \newcommand{\Sigmav}{\boldsymbol\Sigma}
 \newcommand{\Phiv}{\boldsymbol\Phi}
$

# Optimization

So far, we have been using gradient descent to find minimum or maximum values in our error function.
In general, we call this maximization or minimization problem as an **optimization problem**.

In optimization problems, we look for the largest or the smallest value that a function can take. By systematically choosing input vales within the constraint set, optimization problem seeks for the best available values of an objective function.

So, for a given function $f(x)$ that maps $f: \Xm \rightarrow \Ym $ where $\Ym \subset \R$,
we are looking for a $x^* \in \Xm$ that satisfies

$$
\begin{cases}
    f(x^*) \le f(x) &\forall x       & \quad \text{if } \text{ minimization}\\
    f(x^*) \ge f(x) &\forall x  & \quad \text{if } \text{ maximization}.
  \end{cases}
$$

The optimization problems are often expressed in following notation.

$$
\begin{equation*}
\begin{aligned}
& \underset{x}{\text{minimize}}
& & f(x) \\
& \text{subject to}
& & x \le b_i, \; i = 1, \ldots, m,\\
 &&& x \ge 0.
\end{aligned}
\end{equation*}
$$

## Least Squares

$$
\begin{equation*}
\begin{aligned}
& \underset{\wv}{\text{minimize}}
& & \Vert \Xm \wv - t\Vert^2
\end{aligned}
\end{equation*}
$$

As we discussed, least-squares problems can be solved analytically, $\wv = (\Xm^\top \Xm)^{-1} \Xm^\top t$.
We easily formulate least-sqaures and solve very efficiently.

## Linear Programming

$$
\begin{equation*}
\begin{aligned}
& \underset{\xv}{\text{minimize}}
& & \wv^\top \xv \\
& \text{subject to}
& & \av_i^\top \xv \le b_i, \; i = 1, \ldots, m.
\end{aligned}
\end{equation*}
$$

Linear programming or linear optimization finds a maximum or minimum from a mathematical model that is represented by linear relationships.
There is no analytical formular for a solution, but there are reliable algorithms that solve LP efficiently.

## Convex Optimization

$$
\begin{equation*}
\begin{aligned}
& \underset{x}{\text{minimize}}
& & f_0(x) \\
& \text{subject to}
& & f_i(x) \leq b_i, \; i = 1, \ldots, m.
\end{aligned}
\end{equation*}
$$

Convex condition:
$$
f_i(\alpha x_1 + (1-\alpha) x_2) \le \alpha f_i(x_1) + (1-\alpha) f_i(x_2)
$$

Convex optimization generalizes the linear programming problems. A convex optimization problem has the constraint set that forms convex functions. As a general model of LP, convex optimization problems do not have analytical solution but they also have reliable and efficient algorithms for it. Thus, it can be solved very quickly and reliably up to very large problems.
However, it is difficulty to recognize if it is convex or not.

## Nonlinear Optimization

For non-convex problems, we can apply local optimization methods, which is called nonlinear programming.
Starting from initial guess, it searchs for a minimal point near neighborhood. It can be fast and can be applicable large problems. However, there is no guarantee for discovery of global optimum.

## Newton's method

Newton's method approximates the curve with quadratic function repeatedly to find a temporary point or stationary point of $f$.

If we assume that for each measurement point $x^{(k)}$, we can compute $f(x^{(k)})$, $f^{\prime}(x^{(k)})$, and $f^{\prime\prime}(x^{(k)})$.
Using second order Taylor expansion, we can approximate $q(x)$ for $f(x + \Delta x)$:

$$
q(x) = f(x^{(k)}) + f^{\prime}(x^{(k)}) \Delta x + \frac{1}{2} f^{\prime\prime}(x^{(k)}) \Delta x^2
$$

where $\Delta x = (x - x^{(k)})$.

Minimizing this quadratic function,

$$
0 = q^\prime(x) = f^{\prime}(x^{(k)}) + f^{\prime\prime}(x^{(k)}) \Delta x.
$$

Setting $x = x^{(k+1)}$, we can get

$$
 x^{(k+1)} =  x^{(k)} - \frac{f^{\prime}(x^{(k)})}{f^{\prime\prime}(x^{(k)})}.
$$

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import scipy.optimize as opt
from scipy.optimize import rosen, minimize

```

```python
# examples are from http://people.duke.edu/~ccc14/sta-663-2017/14C_Optimization_In_Python.html

x = np.linspace(-5, 5, 1000)
y = np.linspace(-5, 5, 1000)

xs, ys = np.meshgrid(x, y)
zs = rosen(np.vstack([xs.ravel(), ys.ravel()])).reshape(xs.shape)

plt.figure(figsize=(8,8))
plt.contour(xs, ys, zs, np.arange(10)**5, cmap='jet')
plt.text(1, 1, 'x', va='center', ha='center', color='k', fontsize=30);
```

```python
from scipy.optimize import rosen_der, rosen_hess

def reporter(p):
    """record the points visited"""
    global ps
    ps.append(p)

# starting position
x0 = np.array([4,-4.1])

ps = [x0]
minimize(rosen, x0, method="Newton-CG", jac=rosen_der, hess=rosen_hess, callback=reporter)

```

         fun: 1.3642782750354208e-13
         jac: array([ 1.21204353e-04, -6.08502470e-05])
     message: 'Optimization terminated successfully.'
        nfev: 38
        nhev: 26
         nit: 26
        njev: 38
      status: 0
     success: True
           x: array([0.99999963, 0.99999926])

```python
ps = np.array(ps)
plt.figure(figsize=(16, 8))
plt.subplot(121)
plt.contour(xs, ys, zs, np.arange(10)**5, cmap='jet')
plt.plot(ps[:, 0], ps[:, 1], '-ro')
plt.subplot(122)
plt.semilogy(range(len(ps)), rosen(ps.T));
```

## Vs. others?

Now, let us take a look at other optimization tools including naive steepest descent and scaled conjugate gradient ([Moller, 1997](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.50.8063&rep=rep1&type=pdf)).
To run this properly, you need to download [grad.py](https://webpages.uncc.edu/mlee173/teach/itcs4156online/notes/grad.py) under your current work folder.

```python
from grad import steepest

res = steepest(np.array(x0), rosen_der, rosen, stepsize=0.0001, wtracep=True, ftracep=True)

ps = np.array(res['wtrace'])
plt.figure(figsize=(16, 8))
plt.subplot(121)
plt.contour(xs, ys, zs, np.arange(10)**5, cmap='jet')
plt.plot(ps[:, 0], ps[:, 1], '-ro')
plt.subplot(122)
plt.semilogy(range(len(ps)), res['ftrace']);
```

```python
from grad import scg

res = scg(np.array(x0), rosen_der, rosen, wtracep=True, ftracep=True)
res1 = scg(np.array([-4, 4]), rosen_der, rosen, wtracep=True, ftracep=True)

ps = np.array(res['wtrace'])
ps1 = np.array(res1['wtrace'])
plt.figure(figsize=(16, 8))
plt.subplot(121)
plt.contour(xs, ys, zs, np.arange(10)**5, cmap='jet')
plt.plot(ps[:, 0], ps[:, 1], '-ro')
plt.plot(ps1[:, 0], ps1[:, 1], '-bo')
plt.subplot(122)
plt.semilogy(range(len(ps)), res['ftrace']);
```

```python
x0 = [-4, 4]
ps = [x0]
minimize(rosen, x0, method="Newton-CG", jac=rosen_der, hess=rosen_hess, callback=reporter)

ps = np.array(ps)
plt.figure(figsize=(16, 8))
plt.subplot(121)
plt.contour(xs, ys, zs, np.arange(10)**5, cmap='jet')
plt.plot(ps[:, 0], ps[:, 1], '-ro')
plt.subplot(122)
plt.semilogy(range(len(ps)), rosen(ps.T));
```

# Neural Network!

Now, let us use this optimization trick for our neural networks.

```python
# standardization class
class Standardizer:
    """ class version of standardization """
    def __init__(self, X, explore=False):
        self._mu = np.mean(X,0)
        self._sigma = np.std(X,0)
        if explore:
            print ("mean: ", self._mu)
            print ("sigma: ", self._sigma)
            print ("min: ", np.min(X,0))
            print ("max: ", np.max(X,0))

    def set_sigma(self, s):
        self._sigma[:] = s

    def standardize(self,X):
        return (X - self._mu) / self._sigma

    def unstandardize(self,X):
        return (X * self._sigma) + self._mu

```

```python

""" Neural Network
        referenced NN code by Chuck Anderson in R and C++

                                by Jake Lee (lemin)

    example usage:
        X = numpy.array([0,0,1,0,0,1,1,1]).reshape(4,2)
        T = numpy.array([0,1,1,0,1,0,0,1]).reshape(4,2)

        nn = nnet.NeuralNet([2,3,2])
        nn.train(X,T, wprecision=1e-20, fprecision=1e-2)
        Y = nn.use(X)

"""
from grad import scg, steepest
from copy import copy

class NeuralNet:
    """ neural network class for regression

        Parameters
        ----------
        nunits: list
            the number of inputs, hidden units, and outputs

        Methods
        -------
        set_hunit
            update/initiate weights

        pack
            pack multiple weights of each layer into one vector

        forward
            forward processing of neural network

        backward
            back-propagation of neural network

        train
            train the neural network

        use
            appply the trained network for prediction

        Attributes
        ----------
        _nLayers
            the number of hidden unit layers

        rho
            learning rate

        _W
            weights
        _weights
            weights in one dimension (_W is referencing _weight)

        stdX
            standardization class for data
        stdT
            standardization class for target

        Notes
        -----

    """

    # TODO: Try to implement Neural Network class with the member variables and methods described above

```

```python
X = np.array([0,0,1,0,0,1,1,1]).reshape(4,2)
T = np.array([0,1,1,0,1,0,0,1]).reshape(4,2)

nn = NeuralNet([2,3,2])
nn.train(X, T)
Y = nn.use(X)
```

```python
Y
```

    array([[  6.57854210e-07,   1.00000047e+00],
           [  1.00000100e+00,  -1.17983002e-07],
           [  1.00000194e+00,  -2.09443329e-07],
           [  2.12270757e-06,   1.00000155e+00]])

```python
T
```

    array([[0, 1],
           [1, 0],
           [1, 0],
           [0, 1]])

```python
X
```

    array([[0, 0],
           [1, 0],
           [0, 1],
           [1, 1]])

```python
# repeating the previous example

# Make some training data
n = 20
X = np.linspace(0.,20.0,n).reshape((n,1)) - 10
T = 0.2 + 0.05 * (X+10) + 0.4 * np.sin(X+10) + 0.2 * np.random.normal(size=(n,1))

# Make some testing data
Xtest = X + 0.1*np.random.normal(size=(n,1))
Ttest = 0.2 + 0.05 * (Xtest+10) + 0.4 * np.sin(Xtest+10) + 0.2 * np.random.normal(size=(n,1))

nSamples = X.shape[0]
nOutputs = T.shape[1]
```

```python
nn = NeuralNet([1,3,1])
nn.train(X, T, ftracep=True)
Ytest, Z = nn.use(Xtest, retZ=True)

plt.figure(figsize=(10,8))
plt.subplot(3,1,1)
plt.plot(nn.ftrace)
plt.ylim(0,0.7)
plt.xlabel('Epochs')
plt.ylabel('RMSE')
plt.legend(('Train','Test'),loc='upper left')

plt.subplot(3,1,2)
plt.plot(X,T,'o-',Xtest,Ttest,'o-',Xtest,Ytest,'o-')
plt.xlim(-10,10)
plt.legend(('Training','Testing','Model'),loc='upper left')
plt.xlabel('$x$')
plt.ylabel('Actual and Predicted $f(x)$')

plt.subplot(3,1,3)
plt.plot(X, Z[1])
plt.ylim(-1.1,1.1)
plt.xlabel('$x$')
plt.ylabel('Hidden Unit Outputs ($z$)');
```


| |Pierre Proulx, ing, professeur|
|:---|:---|
|Département de génie chimique et de génie biotechnologique |** GCH200-Phénomènes d'échanges I **|

# Chapitre 17

```python
from CoolProp.CoolProp import PropsSI
import thermo as th
# fonction Dab équation 17.2-1, utile aussi pour extrapoler
def Dab(A,B,p,T,MA,MB,polaire): #  cm2/sec
    a=[2.745e-4,3.64e-4]
    b=[1.823,2.334]
    TcA=PropsSI('Tcrit',A)
    PcA=PropsSI('Pcrit',A)/101325  # en atmosphères
    TcB=PropsSI('Tcrit',B)
    PcB=PropsSI('Pcrit',B)/101325
    f1=(PcA*PcB)**(1/3)
    f2=(TcA*TcB)**(5/12)
    f3=(1/MA+1/MB)**(1/2)
    f=f1*f2*f3
    if polaire:
        return f*a[1]*(T/(TcA*TcB)**0.5)**b[1]/p   # polaire
    else:
        return f*a[0]*(T/(TcA*TcB)**0.5)**b[0]/p   # non-polaire
# fin de la fonction Dab
Dab('CO','CO2',1,296.1,28.01,44.01,polaire=False) # exemple 17.2-1, en cm2/sec
```

    0.1517419935491598

### Voyons si on peut utiliser la même équation pour résoudre l'exemple 17.2-2

```python
P=171.7
T=373
R=82.0578
c=P/(R*T)
cdab=c*Dab('CO2','CO2',P,T,44.01,46.01,polaire=False) # exemple 17.2-2 isotope C14 pour le CO2
print(cdab,' Le résultat est près de la valeur expérimentale de 5.89e-6')
```

    5.609988881945637e-06  Le résultat est près de la valeur expérimentale de 5.89e-6

### Voyons si on peut utiliser la même équation pour résoudre l'exemple 17.2-3

```python
R=82.0578
P=136
T=313
c=P/(R*T)
cdab=c*Dab('CH4','ethane',P,T,16.04,30.07,polaire=False) # exemple 17.2-3
print(cdab,' résultat pas mal encore, environ 10% en erreur')
```

    6.663373783520741e-06  résultat pas mal encore, environ 10% en erreur

### En conclusion on voit que l'équation 17.2-1 semble convenir à l'évaluation du coefficient de diffusion binaire à moins de 10%, basé sur ces trois exemples de la section 17.2.

### L'exemple 17.3-1 montre bien que le résultat calculé avec 17-2.1 est tout-à-fait satisfaisant, on n'aura donc pas à utiliser les équations de la section 17.3

# ***

## Section 17.4 diffusion en phase liquide

> ### L'équation empirique de Wilke et Chang est encore la méthode proposée pour les petites et moyennes molécules.
>> #### ####  $D_{AB}=7.4 \times 10^{-8} T \frac {\sqrt{\phi_B M_B}} {\mu \tilde {V_A}^{0.6}}$ (équation 17.4-8)

le terme $V_A$ est au point d'ebullition de la substance diffusant dans le solvant

```python
import thermo as th
T=273.15+15
P=101325
phib=1
benzene=th.Chemical('benzene',T,P)
TNT=th.Chemical('118-96-7',T,P)
TNT.calculate(TNT.Tb) # on veut calculer la densité au point d'ébullition
VA=TNT.MW/(TNT.rhol/1000) # en g-moles!!!   ceci devrait être Ok
#VA=140                      # cette valeur de 140 semble etre la densité à 25 degrés C
mu=benzene.mul*1e3          # en centipoises
MB=benzene.MW
Da= lambda T: 7.4e-8*T*(phib*MB)**0.5/(mu*VA**0.6)  # attention Va est au point d'ebullition
VA,Da(T)
```

    (219.25241425268013, 1.0644668694655854e-05)

```python
#
# Diffusion de l'oxygène dans l'eau, la valeur mesurée est 2.1e-5 cm2/sec
#
T=298
P=101325
eau=th.Chemical('water',T,P)
oxygene=th.Chemical('O2',T,P)
phib=2.26
#
oxygene.calculate(oxygene.Tb)
VA=oxygene.MW/(oxygene.rhol/1000)     # ramener aux unités g-moles !!!
#VA=25.6            # trouvée dans la littérature
mu=eau.mul*1e3
MB=eau.MW
print('Coefficient de diffusion par Wilke et Chang pour oxygene-eau ',Da(T), 'cm2/sec')
```

    Coefficient de diffusion par Wilke et Chang pour oxygene-eau  2.132707592624732e-05 cm2/sec

```python
#
# exercice 17A10 a
#
T=273+12.5
P=101325
eau=th.Chemical('water',T,P)
ac=th.Chemical('acetic acid',T,P)
phib=2.26
ac.calculate(ac.Tb) # recalcule les propriétés à Tb pour trouver la densité voulue
VA=ac.MW/(ac.rhol/1000)
mu=eau.mul*1e3
MB=eau.MW
print('Wilke et Chang pour acide acetique-eau ',Da(T), 'cm2/sec')
print(ac.rhol/1000)
#
#b
#
T=273+15
P=101325
eau.calculate(T)
me=th.Chemical('methanol',T,P)
me.calculate(me.Tb)
VA-me.MW/(me.rhol/1000)
mu=eau.mul*1e3
print('Wilke et Chang pour acide methanol-eau a 15 ',Da(T), 'cm2/sec')
T=273+100
eau.calculate(T)
mu=eau.mul*1e3
print('Wilke et Chang pour acide methanol-eau a 100 ',Da(T), 'cm2/sec')
```

    Wilke et Chang pour acide acetique-eau  9.102665934383858e-06 cm2/sec
    0.9393154185433927
    Wilke et Chang pour acide methanol-eau a 15  9.8257478185632e-06 cm2/sec
    Wilke et Chang pour acide methanol-eau a 100  5.153435554932639e-05 cm2/sec

## Section 17.5: Diffusion de particules colloidales dans un liquide:
> #### Équation de Stokes-Einstein: $D_{A}=\frac {kT}{f}$ avec $f={6 \pi \mu R_A}$(équation 17.5-4)
> #### si la particule n'est pas une sphère, ce qui arrive souvent... on utilise le concept de sphère équivalente en calculant le rayon de la sphère ayant le même volume que la particule colloidale.
>> $
\begin{array}{cc}
\mathrm{Forme\mathbf{}} & \mathrm{Coefficient\,\, de\,\, friction\mathbf{{\displaystyle }}}\\
\mathrm{Sphere} & f=6\pi\mu R\\
\mathrm{Ellipsoide\, prolate\, p=a/b>1} & f=\frac{6\pi\mu Lb(p^{2}-1)^{1/2}}{p^{1/3}ln(p+(p^{2}-1)^{1/2})}\\
\mathrm{Ellipsoide\, oblate\, p=a/b<1} & f=\frac{6\pi\mu Lb(1-p^{2})^{1/2}}{p^{1/3}tan^{-1}(p+(1-p^{2})^{1/2}p^{-1})}\\
\mathrm{Disque\, plat\, mince\, de\, rayon\, a} & f=16\mu a\\
\mathrm{Cylindre\, de\, rayon\, a\, et\, longueur\, L} &  f=\frac{4\pi L}{ln(L/a)+0.193}
\end{array}
$

```python
#
# Équation de Stokes-Einstein
#
import math
Db= lambda T: (k*T)/(6*math.pi*muB*RA)   # utilisons des valeurs en SI
k=1.38e-23
RA=1.59e-9             # rayon équivalent pour un lysozyme ()
T=273.15+20
eau.calculate(T,101325)
muB=eau.mul
print('Lysozyme ',Db(T),' m2/sec, valeur mesurée 1.13 e-10')
RA=4.7e-9              # hemoglobine
print('Hémoglobine ',Db(T),' m2/sec, valeur mesurée 5.2 e-11')
# voir Adams and Fatt, Respiration  Physiology (1967) 2, 293-301;
```

    Lysozyme  1.3476530359594026e-10  m2/sec, valeur mesurée 1.13 e-10
    Hémoglobine  4.559081547181809e-11  m2/sec, valeur mesurée 5.2 e-11

## Sections 17.7 et 17.8

```python
from IPython.display import *
import sympy as sp
sp.init_printing(use_latex=True)
x_a,x_b,omega_a,omega_b,M_a,M_b=sp.symbols('x_a x_b omega_a omega_b M_a M_b')
x_a=omega_a/M_a/(omega_a/M_a+omega_b/M_b)
display(x_a.simplify())
eq=sp.Eq(sp.diff(x_a,omega_a),sp.diff(x_a,omega_a).simplify())
display(eq)
```


```python
#hide
%matplotlib inline
import datetime

import numpy as np
from IPython.display import display, Markdown
import matplotlib.pyplot as plt
import matplotlib
import pandas as pd
import seaborn as sns
import arviz as az
import pymc3 as pm
import theano.tensor as tt
from sunode.wrappers.as_theano import solve_ivp
import sunode
import sympy as sym

import requests
import io

sns.set_context('talk')
plt.style.use('seaborn-whitegrid')
```

```python
import load_covid_data
```

## Data

```python
df_sign = load_covid_data.load_data(filter_n_days_100=10)
df_sign = df_sign[df_sign.state.isna()]

countries = pd.Series(df_sign.country.unique())
countries = countries.loc[~countries.isin(['Cruise Ship (total)', 'Cruise Ship'])]
n_countries = len(countries)
countries = countries.rename('country').to_frame().set_index('country')
```

```python
df_sign = df_sign.reset_index()
df_sign.index.name = 'observation'

previous = (
    df_sign
    .reset_index()
    .groupby(['country'])
    .apply(
        lambda x:
        x.set_index('date')
        .assign(date=x.date.values)
        .sort_index()
        .shift()
    )
)

df_sign = (
    df_sign
    .reset_index()
    .set_index(['country', 'date'])
    .assign(
        previous_date=previous.date,
        previous_confirmed=previous.confirmed,
    )
    .reset_index()
    .set_index('observation')
)

df_sign['new_confirmed'] = df_sign.eval('confirmed - previous_confirmed')
df_sign['days_since_last_data'] = (df_sign['date'] - df_sign['previous_date']).dt.days
df_sign = df_sign.dropna(subset=['new_confirmed'])
```

## Growth Rate Predictions

```python
pop_file = 'https://github.com/datasets/population/raw/master/data/population.csv'
populations = pd.read_csv(pop_file)
populations = populations.sort_values(['Country Name', 'Year']).groupby(['Country Name']).last()

idxs = populations.index.get_indexer([c.replace('(total)', '').strip() for c in countries.index])
countries['population'] = [populations.iloc[i].Value if idx != -1 else np.nan for i, idx in enumerate(idxs)]
```

```python
countries.loc['Egypt', 'population'] = populations.loc['Egypt, Arab Rep.'].Value
countries.loc['Iran', 'population'] = populations.loc['Iran, Islamic Rep.'].Value
countries.loc['Korea, South', 'population'] = populations.loc['Korea, Rep.'].Value
countries.loc['Hong Kong', 'population'] = populations.loc['Hong Kong SAR, China'].Value
countries.loc['Czechia', 'population'] = populations.loc['Czech Republic'].Value
countries.loc['US (total)', 'population'] = populations.loc['United States'].Value
```

```python
countries['population'] = countries.population.values.astype('d')
```

```python
multi_country = True

if multi_country:
    max_countries = 50
    countries = countries.iloc[:n_countries]
    n_countries = len(countries)
    df_sign = df_sign[df_sign.country.isin(countries.index)]
else:
    n_countries = 1
    country = 'Italy'

    countries = countries.loc[country: country]
    df_sign = df_sign[df_sign.country == country]
```

```python
start_day = df_sign.date.min() - datetime.timedelta(days=1)
end_data_day = df_sign.date.max()
end_day = end_data_day + datetime.timedelta(days=30)

data_time = pd.date_range(start_day, end_data_day)
time = pd.date_range(start_day, end_day)
country = countries.index
```

```python
log_scale = False

if not log_scale:
    def sir_rhs(t, y, p):
        exp = np.vectorize(sym.exp)
        infections = exp(y.log_infection_rate) * y.S * y.I
        remissions = p.remission_rate * y.I

        change = [
            sunode.symode.lambdify.interpolate_spline(t, p.infection_change_polys[i], 0, len(data_time), degree=4)
            for i in range(len(infections))
        ]

        return {
            'S': -infections,
            'I': infections - remissions,
            'R': remissions,
            'log_infection_rate': change,
        }
else:
    def sir_rhs(t, y, p):
        exp = np.vectorize(sym.exp)
        infections = exp(y.log_infection_rate + y.I + y.S)
        remissions = p.remission_rate * exp(y.I)

        change = [
            sunode.symode.lambdify.interpolate_spline(t, p.infection_change_polys[i], 0, len(data_time), degree=4)
            for i in range(len(infections))
        ]

        return {
            'S': -infections / exp(y.S),
            'I': (infections - remissions) / exp(y.I),
            'R': remissions / exp(y.R),
            'log_infection_rate': change,
        }

with pm.Model() as model:
    log_pop = np.log(countries['population'].values)
    if multi_country:
        sd = pm.HalfNormal('init_logI_sd')
        mu = pm.Normal('init_logI_mu')
        raw = pm.Normal('init_logI_raw', shape=n_countries)
        init_logI = mu + sd * raw - log_pop
        pm.Deterministic('init_logI', init_logI)
    else:
        init_logI = pm.Normal('init_logI_raw', shape=n_countries)
        init_logI = init_logI - log_pop
        pm.Deterministic('init_logI', init_logI)

    if multi_country:
        sd = pm.HalfNormal('log_infect_sd')
        mu = pm.Normal('log_infect_mu')
        raw = pm.Normal('log_infect_raw', shape=n_countries)
        log_infect = mu + sd * raw
        pm.Deterministic('log_infect', log_infect)
    else:
        log_infect = pm.Normal('log_infect', shape=n_countries)

    if multi_country:
        sd = pm.HalfNormal('log_remiss_sd')  # In 1/day/infected
        mu = pm.Normal('log_remiss_mu')
        raw = pm.Normal('log_remiss_raw', shape=n_countries)
        log_remiss = mu + sd * raw
        pm.Deterministic('log_remiss', log_remiss)
    else:
        log_remiss = pm.Normal('log_remiss', shape=n_countries)

    sd = pm.HalfNormal('infection_change_polys_sd')
    raw = pm.StudentT('infection_change_polys_raw', shape=(n_countries, 8), nu=5)
    infection_change_polys = sd * raw / 100

    if not log_scale:
        init = {
            'S': (1 - tt.exp(init_logI), (n_countries,)),
            'I': (tt.exp(init_logI), (n_countries,)),
            'R': (np.zeros(n_countries), (n_countries,)),
            'log_infection_rate': (log_infect, (n_countries,)),
        }
    else:
        init = {
            'S': (tt.log1p(-tt.exp(init_logI)), (n_countries,)),
            'I': (init_logI, (n_countries,)),
            'R': np.full(n_countries, -20, dtype='d'),
            'log_infection_rate': (log_infect, (n_countries,)),
        }

    params = {
        #'infection_rate': (tt.exp(log_infect), (n_countries,)),
        'remission_rate': (tt.exp(log_remiss), (n_countries,)),
        'infection_change_polys': (infection_change_polys, (n_countries, 8)),
        '_placeholder': (np.zeros(n_countries), (n_countries,)),
    }

    solutions = []
    problems = []
    solvers = []
    for i in range(n_countries):
        init_i = {
            name: (var[i:i+1], (1, *shape_others))
            for i, (name, (var, (_, *shape_others))) in enumerate(init.items())
        }
        params_i = {
            name: (var[i:i+1], (1, *shape_others))
            for i, (name, (var, (_, *shape_others))) in enumerate(params.items())
        }

        solution_i, flat_solution, problem, solver, *_ = solve_ivp(
            t0=0,
            y0=init_i,
            params=params_i,
            tvals=np.arange(len(data_time)),
            rhs=sir_rhs,
            solver_kwargs={
                'solver': 'ADAMS',
                #'constraints': np.r_[np.ones(n_countries * 3), np.zeros(n_countries)],
                'constraints': np.array([1., 1, 1, 0]),
            }
        )
        problems.append(problem)
        solvers.append(solver)
        solutions.append(solution_i)

    solution = {}
    for name in init:
        solution[name] = tt.join(1, *[vals[name] for vals in solutions])

    for name in init:
        if log_scale:
            pm.Deterministic(f'solution_{name}', tt.exp(solution[name]))
        else:
            pm.Deterministic(f'solution_{name}', solution[name])

    # The number of people who have at some time been infected
    if log_scale:
        log_cases = tt.log1p(-tt.exp(solution['S'])) + np.log(countries.population.values[None, :])
    else:
        cases = (1 - solution['S']) * countries.population.values[None, :]
    #cases = solution['I'] * countries.population.values[None, :]

    idx_country = countries.index.get_indexer(df_sign.country)
    idx_data_time = data_time.get_indexer(df_sign.date)
    idx_previous_time = data_time.get_indexer(df_sign.previous_date)

    assert -1 not in idx_country
    assert -1 not in idx_data_time
    assert -1 not in idx_previous_time

    if log_scale:
        log_current_infected = log_cases[idx_data_time, idx_country]
        log_previous_infected = log_cases[idx_previous_time, idx_country]
        #mu = tt.exp(log_current_infected) - tt.exp(log_previous_infected) + 1
        mu_confirmed = tt.exp(log_current_infected)
    else:
        current_infected = cases[idx_data_time, idx_country]
        previous_infected = cases[idx_previous_time, idx_country]
        #mu = current_infected - previous_infected + 1
        mu_confirmed = current_infected

    alpha = pm.HalfNormal('confirmed_alpha', shape=n_countries, sd=10)
    pm.Deterministic('mu_confirmed', mu_confirmed)
    pm.NegativeBinomial(
        'confirmed_count',
        mu_confirmed,
        alpha[idx_country],
        observed=np.maximum(0, df_sign['confirmed'].values)
    )

    if log_scale:
        log_cases = solution['R'] + np.log(countries.population.values[None, :])
    else:
        cases = solution['R'] * countries.population.values[None, :]

    if log_scale:
        log_current_recovered = log_cases[idx_data_time, idx_country]
        log_previous_recovered = log_cases[idx_previous_time, idx_country]
        #mu = tt.exp(log_current_infected) - tt.exp(log_previous_infected) + 1
        mu_recovered = tt.exp(log_current_recovered)
    else:
        current_recovered = cases[idx_data_time, idx_country]
        previous_recovered = cases[idx_previous_time, idx_country]
        #mu = current_infected - previous_infected + 1
        mu_recovered = current_recovered

    alpha = pm.HalfNormal('recovered_alpha', shape=n_countries, sd=10)
    pm.Deterministic('mu_recovered', mu_recovered)
    pm.NegativeBinomial(
        'recovered_count',
        mu_recovered,
        alpha[idx_country],
        observed=np.maximum(0, df_sign['recovered'].values + df_sign['deaths'].values)
    )
```

```python
solutions[1]['R'].tag.test_value.shape
```

```python
cases.tag.test_value.shape
```

```python
lib = sunode._cvodes.lib
```

```python
    jac_prod = problem.make_sundials_jac_prod()
    lin_solver = lib.SUNLinSol_SPGMR(solver._state_buffer.c_ptr, lib.PREC_NONE, 5)
    lib.SUNLinSolInitialize_SPGMR(lin_solver)

    lib.CVodeSetLinearSolver(solver._ode, lin_solver, sunode.basic.ffi.NULL)

    lib.CVodeSetJacTimes(solver._ode, sunode.basic.ffi.NULL, jac_prod.cffi)

    jac_prod = problem.make_sundials_adjoint_jac_prod()
    lin_solverB = lib.SUNLinSol_SPGMR(solver._state_buffer.c_ptr, lib.PREC_NONE, 5)
    lib.SUNLinSolInitialize_SPGMR(lin_solverB)

    lib.CVodeSetLinearSolverB(solver._ode, solver._odeB, lin_solverB, sunode.basic.ffi.NULL)

    lib.CVodeSetJacTimesB(solver._ode, solver._odeB, sunode.basic.ffi.NULL, jac_prod.cffi)

```

```python
for problem, solver in zip(problems, solvers):
    lib.CVodeSStolerances(solver._ode, 1e-7, 1e-7)
    lib.CVodeSStolerancesB(solver._ode, solver._odeB, 1e-6, 1e-6)
    lib.CVodeQuadSStolerancesB(solver._ode, solver._odeB, 1e-6, 1e-6)
    lib.CVodeSetMaxNumSteps(solver._ode, 500)
    lib.CVodeSetMaxNumStepsB(solver._ode, solver._odeB, 500)
    #lib.CVodeSetMinStep(solver._ode, 1e-12)
    #lib.CVodeSetMinStepB(solver._ode, solver._odeB, 1e-12)
    lib.CVodeSetMaxOrd(solver._ode, 4)
    lib.CVodeSetMaxOrdB(solver._ode, solver._odeB, 4)
    #lib.CVodeSetInitStep(solver._ode, 1e-7)
    #lib.CVodeSetInitStepB(solver._ode, solver._odeB, -1e-6)
    #lib.CVodeSetMaxStep(solver._ode, 50)
    #lib.CVodeSetMaxStepB(solver._ode, solver._odeB, 50)
```

```python
func = model.logp_dlogp_function()
func.set_extra_values({})
```

```python
np.random.seed(42)
```

```python
x0 = np.random.randn(func.size)
func(x0)
```

    /home/adr/anaconda3/envs/pymc/lib/python3.7/site-packages/theano/tensor/subtensor.py:2197: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result.
      rval = inputs[0].__getitem__(inputs[1:])

    (array(-47547.31119204),
     array([-4.97753393e+03, -1.98826271e+03, -1.52161765e+00, -3.26821370e+03,
             2.34153375e-01,  2.34136957e-01, -1.57921282e+00, -7.67434729e-01,
             4.69474386e-01, -5.42560044e-01,  4.63417693e-01,  4.65729754e-01,
            -2.41962272e-01,  1.91328024e+00,  1.72491783e+00,  5.62287529e-01,
             1.01283112e+00, -3.14247333e-01,  9.08024076e-01,  1.41230370e+00,
            -1.92681031e+04, -4.00767998e+04, -6.75282047e-02,  1.42474819e+00,
             5.44382725e-01, -1.73547728e+05,  1.15099358e+00, -3.75698018e-01,
             6.00638690e-01,  2.91693750e-01,  6.01706612e-01, -1.85227818e+00,
             1.34972247e-02,  1.05771093e+00, -8.22544912e-01,  1.22084365e+00,
            -2.08863595e-01,  1.95967012e+00,  1.32818605e+00, -1.96861236e-01,
            -6.66215336e+03,  2.75131562e+04,  5.75779173e+04,  3.01103696e-01,
             1.47852199e+00,  7.19844208e-01,  4.60638771e-01, -1.05712223e+00,
            -3.43618290e-01,  1.76304016e+00, -3.24083969e-01,  3.85082280e-01,
             6.76922000e-01, -6.11676289e-01, -1.03099952e+00, -9.31280119e-01,
             8.39217523e-01,  3.09212376e-01, -3.31263431e-01, -9.75545127e-01,
            -2.32288255e+03,  2.21265401e-01,  1.06652222e+00,  1.11605347e+00,
            -8.61304631e-01, -1.18979086e+00,  8.63226214e-02, -1.00235041e+00,
            -4.22901760e-01, -2.45155171e+01, -5.64908086e+02, -1.31813745e+03,
            -6.42093462e+02, -9.14923053e+01, -1.66338689e+00, -8.80596998e-01,
            -1.04303339e-01,  3.52505639e-01, -1.09927813e-01,  1.33238439e+00,
             2.61086481e-01, -4.17876772e-01, -1.23429196e+00,  5.90217313e-01,
             8.58021172e-01,  5.73244434e-01, -9.40810100e-01, -3.86154427e-01,
             6.01926601e-01, -5.85093128e-01, -1.16273905e-01, -9.78713816e-01,
             7.66869052e-01,  3.84929185e-01,  4.56492741e-01,  1.22952144e+00,
            -3.49219908e-01, -3.09053930e-01, -6.13611588e-03,  2.78439987e-01,
             1.21260771e+00,  4.87521726e-01,  4.01818444e-01,  8.52934648e-01,
             1.92541136e-01, -4.69530201e-01, -1.32244868e+00, -2.08224149e-01,
            -3.05014018e-01,  8.92361861e-02,  1.32608243e+00,  3.18121778e-02,
            -7.22238510e-02, -1.33538471e+00,  2.29137415e-01, -3.55393567e-01,
             4.16440882e-02,  1.10152047e+00, -1.08735951e+00, -8.10650723e-01,
            -8.43657691e-01,  9.36388925e-01, -1.20794617e+00,  1.20759243e+00,
            -6.58846969e-01, -1.34135591e+00,  9.93656039e-01,  6.38598450e-01,
            -1.19344610e-01,  5.75018745e-01,  1.25652088e+00, -8.21982889e-02,
             1.04003158e+00, -5.43912129e-01,  9.43750840e-01, -1.25631366e+00,
             8.37183880e-01,  3.78619463e-01, -8.62110010e-01,  1.13356183e+00,
            -2.70156449e-01, -1.16907113e+00,  1.27174267e+00, -2.20060274e-01,
            -3.07702959e-01, -8.35988369e-01,  1.13654683e+00,  1.17485216e+00,
            -5.93967782e-01, -3.50204016e-01, -2.96865951e-01, -4.05991905e-01,
             7.46946986e-01, -2.75729766e-01, -3.45747613e-01,  7.77836071e-01,
            -1.31994999e+00, -5.44164595e-01,  1.11350581e+00, -7.25331655e-01,
             9.82871553e-01, -8.40378289e-01, -1.09605751e+00,  8.67908172e-01,
            -9.75061664e-01, -4.79013474e-01, -8.69018608e-01, -1.32367892e+00,
             2.90961669e-01,  8.12198452e-01,  9.21579893e-01,  8.63969924e-01,
             9.24121792e-02, -4.00069968e-01, -3.27021746e-01, -8.73134556e-01,
            -1.56017428e-02, -1.22613484e+00,  3.13200682e-01, -1.31628304e+00,
            -6.96287155e-01,  8.96808592e-01,  1.04531525e+00, -5.53211619e-01,
             2.65503724e-01, -7.77524688e-01, -5.43539593e-01,  8.73020843e-02,
             8.88702001e-01,  1.24597392e+00,  5.15271432e-01, -8.96217922e-01,
            -2.54578712e-01,  1.14080991e+00, -2.06577989e-01, -4.49046903e-01,
             9.17307994e-01, -1.83602371e-01, -6.98031600e-02,  1.08744170e+00,
            -4.18626994e-01, -6.33120759e-01, -1.05269844e+00, -1.03474570e+00,
             1.19832541e+00, -1.59908004e+02, -6.11705130e+02, -8.62561141e+02,
            -2.36422868e+02, -8.87968820e+03, -9.70604735e+02, -8.78299176e+02,
            -1.89798482e+03, -1.21455159e+03, -3.61493042e+02, -1.20464743e+03,
            -1.81803587e+02, -3.18191975e+02, -5.28797736e+01, -2.76740937e+02,
            -3.86149413e+03, -5.84999233e+01, -5.87752386e+02, -8.17712775e+01,
            -2.59847776e+02, -1.82653088e+03, -2.32731723e+02, -1.41356571e+02,
            -3.00064440e+02, -7.00352254e+02, -4.08492074e+02, -8.90550476e+02,
            -6.07873945e+02, -3.12459010e+02, -4.73296992e+03, -9.58080827e+02,
             2.27362579e+01, -4.40776317e+02, -2.80599280e+02, -1.72701319e+03,
            -1.57198789e+02]))

```python
out = sunode._cvodes.ffi.new('long[1]')
lib.CVodeGetNumNonlinSolvIters(solver._ode, out)
out[0]
```

    150

```python
%timeit func(x0)
```

    61.5 ms ± 187 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

```python
import theano
func_dets = theano.function(model.free_RVs, model.deterministics)
dets = {var.name: vals for var, vals in zip(model.deterministics, func_dets(**func.array_to_dict(x0)))}
```

```python
import theano
func_logps = theano.function(model.free_RVs, [var.logp_elemwiset for var in model.observed_RVs])
logps = {var.name: vals for var, vals in zip(model.observed_RVs, func_logps(**func.array_to_dict(x0)))}
```

```python
(~np.isfinite(logps['recovered_count'])).nonzero()
```

```python
dets['solution_R'].min()
```

```python
with model:
    trace = pm.sample(tune=1000, chains=6, cores=6, discard_tuned_samples=True, compute_convergence_checks=False)
```

```python
import arviz
tr = arviz.from_pymc3(
    trace=trace,
    coords={
        'data_time': data_time,
        'country': country,
        'time': time,
        'observation': df_sign.index,
        'infection_poly': np.arange(8),
    },
    dims={
        'init_logI_raw': ['country'],
        'init_logI': ['country'],
        'log_infect_raw': ['country'],
        'log_infect': ['country'],
        'log_remiss_raw': ['country'],
        'log_remiss': ['country'],
        'solution_I': ['data_time', 'country'],
        'solution_S': ['data_time', 'country'],
        'solution_R': ['data_time', 'country'],
        'solution_log_infection_rate': ['data_time', 'country'],
        'infection_change_polys_raw': ['country', 'infection_poly'],
        'infection_change_polys': ['country', 'infection_poly'],
        'confirmed_alpha': ['country'],
        'recovered_alpha': ['country'],
        'mu_confirmed': ['observation'],
        'mu_recovered': ['observation'],
    },
    log_likelihood=False,
).posterior
```

```python
import xarray as xr
```

```python
xr.DataArray(countries['population'])
```

```python
arviz.plot_trace(tr, var_names=['init_logI'], compact=True);
```

```python
(tr.mu_confirmed - xr.DataArray(df_sign.confirmed, dims=('observation',))).isel(draw=-1).plot.line(x='observation')
```

```python
(tr.mu_recovered - xr.DataArray(df_sign.recovered + df_sign.deaths, dims=('observation',))).isel(draw=-1).plot.line(x='observation')
```

```python
((1 - tr.solution_S) * xr.DataArray(countries['population'])).isel(country=1, draw=-1).plot.line(x='data_time', add_legend=False);
```

```python
((tr.solution_I) * xr.DataArray(countries['population'])).isel(country=1, draw=-1).plot.line(x='data_time', add_legend=False);
```

```python
#tr.solution_log_infection_rate
```

```python
(
    (tr.infection_change_polys_sd * tr.infection_change_polys_raw)
    .cumsum('infection_poly')
    .median(['draw', 'chain'])
    .plot
    .line(x='infection_poly')
);
```

```python
from scipy import interpolate
```

```python
vals = (tr.infection_change_polys_raw * tr.infection_change_polys_sd / 100).transpose('infection_poly', ...).values
degree = 4
lower = 0
upper = len(tr.data_time)

n_knots = degree + len(vals) + 1
knots = np.arange(n_knots)
lower_knot = degree #+ lower * (n_knots - 2 * degree - 1)
upper_knot = degree + (n_knots - 2 * degree - 1)

knots = (knots - lower_knot) / (upper_knot - lower_knot)
knots = knots * (upper - lower) + lower

x = np.arange(upper)
spline = interpolate.BSpline(knots, vals.reshape((vals.shape[0], -1)), degree)
spline_int = spline.antiderivative()
infection_rate_change = (spline_int(x) - spline_int(lower)).reshape((upper, *vals.shape[1:]))
```

```python
tr['infection_rate_change'] = (('data_time', *tr.infection_change_polys_raw.transpose('infection_poly', ...).dims[1:]), infection_rate_change)
```

```python
tr.infection_rate_change.sel(country='Germany', chain=2).isel(draw=slice(-100, None)).plot.line(x='data_time', add_legend=False);
```

```python
sns.lineplot(
    x='data_time',
    y='infection_rate',
    hue='country',
    units='draw',
    estimator=None,
    data = (
        np.exp(tr.solution_log_infection_rate)
        .rename('infection_rate')
        #.sel(country=['Germany', 'Japan'], chain=2)
        .sel(chain=2)
        .isel(draw=slice(-100, None))
        .to_dataframe()
        .reset_index()
    )
)
```

```python
np.log(df_sign[df_sign.country == 'Germany'].set_index('date').confirmed).plot()
```

```python
np.log(df_sign[df_sign.country == 'Italy'].set_index('date').confirmed).plot()
```

```python
tr.infection_change_polys_sd.plot.line(x='draw', add_legend=False);
```

```python
df_sign.query("country == 'Japan'").set_index('date').confirmed.plot()
```

```python
((1 - tr.solution_S) * xr.DataArray(countries['population'])).isel(country=0, draw=-1).plot.line(x='data_time', add_legend=False);
```

```python
df_sign.query("country == 'Japan'").set_index('date').eval('recovered + deaths').plot()
```

```python
(tr.solution_R * xr.DataArray(countries['population'])).isel(country=0, draw=-1).plot.line(x='data_time', add_legend=False);
```

```python
tr.solution_S.diff('data_time').max()
```

```python
df_sign.plot.line(x='date', y='cases')
```


# Recording of class
Background: people join from Americas/East Asia

Solution: recording of class
- Will be put on secret youtube links - not searchable
- Delete mid June

Consent: via poll

# Session 3:
## Non-linear ML

- A tour of the most essential supervised models

*Andreas Bjerre-Nielsen*

# Review

# Universal Approximation

Definition: Given enough data the algorithm comes as close to sampling distribution as possible
- Note: does not care about causation or selection! Purely prediction

Examples?

- We can also make input non-linear using `PolynomialFeatures` of any order.
  - Follows from iterative Taylor expansion
  - Problem: spurious coefficients and large coefficients
- Others approaches? non-linear?

## Agenda
More supervised tools

1. [Measuring classification performance](#Measuring-classification-performance)
1. [Nested cross validation](#Nested-cross-validation)

Non-linear ML models

1. [Kernel-based models](#Kernel-based-models)
1. [Tree-based modelling](#Tree-based-models)
1. [Ensemble learning](#Ensemble-learning)
1. [Neural networks and deep learning](#Neural-networks-and-deep-learning)

## Loading up

```python
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
```

## Data for this lecture

We begin by loading the titanic dataset, *survived* is target.

```python
from sklearn.model_selection import train_test_split

df = sns.load_dataset('titanic')\
        .assign(male=lambda df: df.sex=='male',
                age_null = lambda df: df.age.isnull())
drop_cols = ['survived','adult_male','class','who','alive','embark_town','sex', 'deck']
y,X = df['survived'], pd.get_dummies(df.drop(drop_cols, axis=1), dummy_na=True, drop_first=True).fillna(-99)
X_train, X_test, y_train, y_test = train_test_split(X,y)

auc_scores = {}
```

# Measuring classification performance

## Breakdown by error type (1)

We measure the accaracy as the rate of true predictions, i.e. \begin{align}ACC&=\frac{True}{True+False}\end{align}

Can we decompose?

## Breakdown by error type (2)
Yes, we can decompose into
$$ACC=\frac{TP+TN}{TP+TN+FP+FN}$$

## Breakdown by error type (3)

Some powerful measures:

- Precision: share of *predicted positive* that are true
    - PRE = $\frac{TP}{TP+FP}$
    - = true positive rate
- Recall: share of *actual positive* that are true
   - REC = $\frac{TP}{TP+FN}=\frac{TP}{AP}$
   - = 1- false negative rate
- F1: mix recall and precision: $\frac{2\cdot PRE\cdot REC}{PRE+ REC}$

```python
from sklearn.metrics import precision_score, recall_score, f1_score
```

## Breakdown by error type (4)

Classification models provide a predicted likelihood of being in the class or not:
- Receiver Operating Characteristic (ROC) curve by varying thresholds for predicted true.
    - ROC is a *theoretical* measure of model performance based on probabilities.
    - AUC: Area Under the (ROC) Curve.

## Breakdown by error type (5)

Example of Area Under the (ROC) Curve.

# Nested cross validation

## Nested cross validation (1)

- Model test does not consider uncertainty from fact that we are also tuning hyperparameters:
  - Leads too overfitting (Varma & Simon 2006; Cawley, Talbot 2010).
- Solution is **nested cross validation**.
  - Validation step should not be modelled as 1) train; 2) test.
  - Better way is 1) model selection: train, validate; 2) test.
  - Implement as pp 204-205 in Python for Machine Learning:
      - first inner loop: `GridSearchCV`
      - second outer loop: `cross_val_score`

## Nested cross validation (2)
A depiction of the process:

## Nested cross validation (3)
Example of application: [Bjerre-Nielsen et al. (2021, PNAS)](https://doi.org/10.1073/pnas.2020258118)
  - Show that student surveillance using phones does not need to better models of academic performance
    - Important for algorithmic policy making!

# Kernel-based models

### *Learn from others like you*

## Kernels

What is a kernel? A mapping $k$ that computes *similarity* between two vectors.

- E.g. how similar are the two observations $x_i$ and $x_j$ from space $\mathcal{X}$?
- Formally a kernel is a mapping $k: \mathcal{X}\times\mathcal{X}\rightarrow\mathbb{R}$

How can we use it as a supervised model?

Let be a binary target, $y\in\{1,-1\}$.

We can use a kernel to weight observations from training data-set:

$${\hat {y}_i}=\operatorname {sgn} \sum _{j=1}^{n}y_{j}k(x _{i},x_j )$$

## Kernels

One example of a kernel is the **k-nearest neighbor**.

1. Compute $||x_i-x_j||$, i.e. distance norm between $x_i$ and all $x_j$, e.g. Euclidian distance ($L_2$ norm)
2. The kernel $k(x_i,x_j)$ equals $1/w$ if it's one of the $k$ smallest distances; otherwise $k(x_i,x_j)=0$.

Many alternative kernels:
- Radius neighbors, which weights all observations within a threshold $r$ evenly, i.e. if $k(x_i,x_j)<r$.
- E.g. polynomial, radial basis function and many more

## Application of kernels

We can use KNN to aggregate over the $k$ most similar observations.
- The KNN classifier computes the mode
- The KNN regressor computes the mean
- Note neighbor kernel are common econometrics too

Generally, we may use the Nadaraya-Watson estimator using for kernel $k$ (aka. kernel regression)

$$\widehat{y}_{k}(x)={\frac {\sum _{i=1}^{n}k(x-x_{i})y_{i}}{\sum _{i=1}^{n}k(x-x_{i})}}$$

## Application of kernels

```python
from sklearn.metrics import roc_auc_score
from sklearn.neighbors import KNeighborsClassifier
clf_knn = KNeighborsClassifier(n_neighbors=5)
clf_knn.fit(X_train,y_train)
auc_scores['5_nearest_neighbors'] = round(roc_auc_score(y_test, clf_knn.predict_proba(X_test)[:, 1]), 2)
auc_scores
```

    {'5_nearest_neighbors': 0.74}

## Application of kernels

Example of the k-nearest neighbor classifier where the binary values are illustrated as red (= -1) / blue (= 1).

*(artist: [Antti Ajanki](https://commons.wikimedia.org/wiki/File:KnnClassification.svg), CC-SA 1.0, no changes)*

## Application of kernels

Another  application is: **Support Vector Machine** (SVM) is like logistic regression (separates green/blue):

*(artist: [Larhmam](https://commons.wikimedia.org/wiki/File:SVM_margin.png), CC-SA 4.0, no changes)*

## Application of kernels

SVMs have two major advantages over logistic regression
- Computational efficiency (optimization under constraints)
- Has non-linear implementation kernels that are decomposable by inner product space, i.e. $k(x_i,x_j)=\left\langle\varphi(x_i),\varphi(x_j) \right\rangle$

*(artist: [Shehzadex](https://commons.wikimedia.org/wiki/File:Kernel_yontemi_ile_veriyi_daha_fazla_dimensiyonlu_uzaya_tasima_islemi.png), CC-SA 4.0, no changes)*

# Tree-based models

### *Divide and conquer*

## Decision Tree

Situation - we want to predict who will become a criminal

## Decision Tree

How can we make a tree to determine who becomes criminal?

## Decision Tree

Situation - want to predict who will become a criminal

## Decision Tree

Split 1: by place of birth

## Decision Tree

Split 2: by alcoholic mother

## Decision Tree

Alternative split 1: by alcoholic mother

## Decision Tree

How can we automate the splitting process of data?

Yes, we can do as follows:

- Use measure/criterion to evaluate the value of all potential splits
- Apply the criterion iteratively (greedy approach)

## Decision Tree

Splitting criterion: entropy

## Decision Tree

Applying the entropy criterion - split by alcoholic mother

## Decision Tree

Applying the entropy criterion - split by place of birth

## Decision Tree - a generalization

Generally, we estimate trees using the Classification And Regression Tree (**CART**) approach:

- At each step evaluate parent (before split) against children (after split)

  - Can use other measures for deciding to split, e.g. gini impurity, variation (sum square pairwise over observations)
  - When used for regression requires different criterion, e.g. variation

## Decision Tree - a generalization

Some terminology of a CART:

## Decision Tree - a generalization

What are some of the properties of the CART?
- Main **advantage** has no *underfitting*, can universally approximate
- Main **disadvantage** is high *overfitting*
- We can trade-off the two by tuning hyperparameters, e.g. maximal depth
  - Another solution is to grow forests rather trees (next topic)

## Decision Tree - python demonstration

We now load the Decision Tree classification model and train it

```python
from sklearn.metrics import roc_auc_score
from sklearn.tree import DecisionTreeClassifier
clf_dt = DecisionTreeClassifier()
clf_dt.fit(X_train,y_train)
auc_scores['decision_tree'] = round(roc_auc_score(y_test, clf_dt.predict_proba(X_test)[:, 1]),2)
auc_scores
```

    {'5_nearest_neighbors': 0.74, 'decision_tree': 0.8}

# Ensemble learning

### *Leveraging wisdom of the crowd*

## Ensemble learning
Main idea
- Create and train many supervised models

How is this implemented?
- Aggregation of independent models (e.g. bagging)
  - E.g. `mode` in classification and `mean` in regression
  - Each model is weighted equally in aggregation
- Sequential improvement of models (e.g. boosting)
  - Next model learn by current model's errors

## Bagging
One way to implement model aggregation:

- What is it?
  - Acronym for **B**ootstrap **Agg**regate **'ing**
  - I.e. repeatedly estimate model on bootstrap samples

- Why should we use it?
  - Advantage: Decreases overfitting, implies better generalization performance
  - Disadvantage: Takes a longer to train/estimate model!

## Bagging
A quick reminder of **bootstrap**:

- Given a table of length $n$, randomly select $n$ rows with replacement $B$ times

Example with five draws from dimensional array, $X=[0, 1, 2, 3, 4,5]$:

```python
import numpy as np

n,B = 6, 5
X = np.arange(n)
bootstrap = np.random.RandomState(0).choice(X,size=[B,n])
bootstrap
```

    array([[4, 5, 0, 3, 3, 3],
           [1, 3, 5, 2, 4, 0],
           [0, 4, 2, 1, 0, 1],
           [5, 1, 5, 0, 1, 4],
           [3, 0, 3, 5, 0, 2]])

## Bagging

Example of bagging algorithm. Assume we have training data set with features $X$ and target $Y$:
  - For $b = 1, ..., B$:
    1. Draw bootstrap sample from training data $X, Y$; denote $X_b, Y_b$.
    2. Train a classification or regression model $f_b$ on $X_b, Y_b$.

- After training, predictions for unseen samples $x'$ can be made by averaging the predictions from all the individual classification on $x'$:

$$\hat{f}=\frac{1}{B}\sum_{b=1}^{B}f_b(x')$$

## Random forest

*Random forest* is a  popular machine learning method that uses bagging by design:

- Train/estimate the forest:
    1. Draw $B$ bootstrap samples on training data, one for each CART
    1. Grow a decision tree from the bootstrap sample. At each node:
      - Randomly select $d$ features without replacement.
- Apply the forest:
   - For each observation we aggregate predictions from CARTs

NOTE: bootstrapping from rows and subsampling from columns

## Random forest

How does a random forest look work? Example of $N=B$ trees:

*(by [Machado et al. 2015](http://dx.doi.org/10.1186/s13567-015-0219-7) licensed under [CC-BY](https://www.researchgate.net/figure/Random-forest-model-Example-of-training-and-classification-processes-using-random_fig5_280533599))*

## Random forest

Why do we subsample feature?
  - Reduces overfitting by making trees more independent - key part of the procedure.

Random forest is nice to use - off the shelf and generally good performance, however, not transparent

- When fitting CARTs, we estimate the information gain for every feature at each split
- We can summarize the *importance* of a feature as its relative amount of information gain delivered during classification

## Random forest

We now load the Random Forest classification model and apply it

```python
from sklearn.ensemble import RandomForestClassifier
clf_rf = RandomForestClassifier()
clf_rf.fit(X_train,y_train)
auc_scores['random_forest'] = round(roc_auc_score(y_test, clf_rf.predict_proba(X_test)[:, 1]), 2)
auc_scores
```

    {'5_nearest_neighbors': 0.74, 'decision_tree': 0.8, 'random_forest': 0.88}

## Boosting

*Social learning*

- Background
  - [Kearns (1988)](http://www.cis.upenn.edu/~mkearns/papers/boostnote.pdf) posed the fundamental question: how can we turn weak learners (not universal approximators) into strong learners?
- Answer to Kearns' problem came by the following procedure:
  - Train models repeatedly
  - For each new model set weights to focus on previous models' errors.

- Intuition: Many weak models that learn from each other’s mistakes, combine into one strong model

## Boosting

How can we implement boosting?
- ***Ada***<text><text>*ptive* ***Boost***: Sequentially update training weights - upweigh with previous errors of models
- ***Gradient***<text><text> ***Boost***: Sequentially train on previous errors of models - gradient approach

Why should we use these methods - often works even better than random forest

## Boosting

Implementation of adaptive boosting:

1. Create a weight vector $w$ that encodes the importance of each training sample
2. For $j$ out of $m$ boosting iterations:
  1. Train a weighted weak classifier train (e.g. decision tree with max depth 10)
  1. Predict class labels
  1. Update $w$ based on the errors that makes (steps *c* to *f* in Raschka page 248)
3. To make predictions apply weighted voting, i.e. giving more prediction weight to less error prone
classifiers

## Boosting

We now load a gradient boosting classification model and apply it

```python
from sklearn.ensemble import GradientBoostingClassifier
clf_gbf = GradientBoostingClassifier()
clf_gbf.fit(X_train,y_train)
auc_scores['xgboost'] = round(roc_auc_score(y_test, clf_rf.predict_proba(X_test)[:, 1]), 2)
auc_scores
```

    {'5_nearest_neighbors': 0.74,
     'decision_tree': 0.8,
     'random_forest': 0.88,
     'xgboost': 0.88}

# Neural networks and deep learning

### Models inspired by biology

## Neural background

What happens when we combine multiple neurons? Network of neurons?

(artist: [Gerd Altmann](https://www.publicdomainpictures.net/en/view-image.php?image=372933&picture=artificial-intelligence-201103), CC-0)

## Neural background

All the linear and logistic models we saw can be expressed in the following form:

(Image from [Raschka, 2017](https://github.com/rasbt/python-machine-learning-book-2nd-edition))

## Neural background

Recall: logistic regression was one of the simple linear models / neurons

How does the perform on the titanic data?

```python
from sklearn.linear_model import LogisticRegression
clf_lr = LogisticRegression(max_iter=100000)
clf_lr.fit(X_train,y_train)
auc_scores['random_log_reg'] = round(roc_auc_score(y_test, clf_lr.predict_proba(X_test)[:, 1]), 3)
auc_scores
```

    {'5_nearest_neighbors': 0.74,
     'decision_tree': 0.8,
     'random_forest': 0.88,
     'xgboost': 0.88,
     'random_log_reg': 0.867}

## Neural example

What about slightly more advanced data?
- Suppose we have circular data:
  - $y$ is blue if $x$ near origin;
  - otherwise, $y$ is red.
- Let $P(y=red)=0.7$.

## Neural example

How does the model data look like before seeing $x$?

## Neural example

What if we depict the predictions of logistic regression?

## Neural example

Could we somehow create intermediate features that help us model?

We create four logistic regressions, two horizontal (left) and two vertical (right)
- none of them are accurate, what about together?

## Neural example

When we combine the four models, taking the maximal value of $P(y=red)$:

## Neural networks

What happened in the previous example? Manually created a neural network.
- General structure of an (artificial) neural network, 1 hidden layer:

(Image from [Raschka, 2017](https://github.com/rasbt/python-machine-learning-book-2nd-edition))

## Neural networks

Can we compare the neural network with a single neuron?
- Yes, the neuron has no hidden layers.
- But what are the intermediate/hidden layer?
  - They consist of $h$ extra auxiliary models that are used to predict the output.

Why might neural architecture be useful?

## Neural networks

Fundamental results in 80's:

- Methods for estimating neural networks were developed
- [Cybenko (1989)](https://doi.org/10.1007%2FBF02551274) and [Hornik (1991)](https://doi.org/10.1016%2F0893-6080%2891%2990009-T) demonstrate that neural networks with a **single hidden layer** can universally approximate for $h\rightarrow\infty$

Is this important?

- Not at the time - models e.g. non-linear SVM, random forest could be trained with less power for similar accuracy.
- Consequence: popularity of neural networks fell in 1990's and early 2000's.

## Neural networks

Today neural networks have changed computing, e.g. to infer content of images:

(author: [MTheiler](https://commons.wikimedia.org/wiki/User:MTheiler), license: CC-BY-SA 4.0)

## Neural networks

What happend since the early 2000's?

Break-through in 2008-12 driven by three factors
- Data not abundant and accessible as today
- Computers not powerful enough, in particular number of cores
  - .. solved by modern Graphical Processing Units (GPU) for gaming
- Changes in architecture:
  - Multiple layers rather than 1 hidden layer, known as ***deep learning***
  - Many more advances, e.g. ***convolution*** of layers that divides input into independent parts

## Mechanics of neural networks

Model works with **feed-forward**: start with input, then proceed through layers

(Image from [Raschka, 2017](https://github.com/rasbt/python-machine-learning-book-2nd-edition))

## Mechanics of neural networks

The models in the both layers consist of linear models with activation, $\phi$:

\begin{align}
a_k^{(hid)} &= \phi\Big(\sum_{i=0}^m a_i^{(in)}\cdot w_{i,k}^{(hid)}\Big), \quad k\in\{1,..,h\} \\
a_l^{(out)} &= \phi\Big(\sum_{i=0}^h a_i^{(hid)}\cdot w_{i,l}^{(out)}\Big), \quad l\in\{1,..,t\} \\
            &= \phi\left(\sum_{i=0}^h \phi\Big(\sum_{j=0}^m a_j^{(in)}\cdot w_{j,k}^{(hid)}\Big)\cdot w_{i,l}^{(out)}\right), \quad k\in\{1,..,h\}, l\in\{1,..,t\} \\
\end{align}

What happens if $\phi$ is linear?

## Mechanics of neural networks

Estimation of neural network is possible using ***backpropagation***

- Idea, use chain rule: $\frac{\partial y}{\partial x}=\frac{\partial y}{\partial u} \frac{\partial u}{\partial x}$ where $y=f(u)$, $u=g(x)$ (Leibniz notation)
- Fix initial parameters drawn randomly
- Apply gradient descent repeatedly for a given number of epochs as follows
  1. Compute the errors in output stage using feed-forward
  1. Optimize model parameters of output layer given output errors
  1. Optimize model parameters of hidden layer activations given using chain rule
    - (keeping parameters of output layer fixed)

## Mechanics of neural networks

Drawbacks of neural networks

- Minimal transparency of how input affects output, discrimination?
- Optimization problem is non-linear

(By *KSmrq*, licensed under CC-BY-SA 3.0, [source](https://commons.wikimedia.org/w/index.php?title=User:KSmrq))

## Appliction in economics

Methodological
- [Hartford et al. (2017)](http://proceedings.mlr.press/v70/hartford17a/hartford17a.pdf) develop framework for using neural networks in instrumental variables to parse non-linear, high dimensional treatment
  - show application to treatment of URL shown in search advertisement
- [Farrell, Liang, Misra (2021)](https://doi.org/10.3982/ECTA16901) develop a generalized framework for using neural networks in estimation and inference
- [Chernozhukov et al. (2017).](https://doi.org/10.1257/aer.p20171038), [Chernozhukov et al. (2018).](https://doi.org/10.1111/ectj.12097) develops a two step framework called Double Machine Learning to apply machine learning in estimation.
  - Allows for applications of machine learning

## Appliction in economics

Data imputation
- Input: Google Street View (360° images)
  - Infer neighborhood safety ([Naik, Raskar & Hidalgo, 2016](https://doi.org/10.1257/aer.p20161030))
  - To measure and track development in sociodemographic compositon ([Gebru et al. 2017](https://doi.org/10.1073/pnas.1700035114); [Nikhil et al. 2017](https://doi.org/10.1073/pnas.1619003114))
- [Layout Parser](https://layout-parser.github.io/),  which can automatically parse the structure of historical records [Zeiyang et al. (2021)](https://arxiv.org/abs/2103.15348)
  - M. Wust and C.M. Dahl use techniques to extract information about health visitors in 60's

## Appliction in economics

Reinforcement learning
- Dynamic structural models that are solved approximately.
- Work in progress by colleagues in Copenhagen: Estimate choice models using neural networks. Can increase state space drastically!
-  [Calvano et al. 2020](https://doi.org/10.1257/aer.20190623) demonstrate that pricing algorithms implicitly collude (i.e. coordinate on price setting) even when optimized own pricing
  - How should we regulate price setting online? (and offline)

## Training a neural network

Let's to fit the network with 2 epochs

```python
import pickle
with open(f"../base/data/mnist/mnist.pkl",'rb') as f:
    mnist = pickle.load(f)
X_train,y_train,X_test,y_test = \
    mnist["training_images"], mnist["training_labels"], mnist["test_images"], mnist["test_labels"]
```

## Training a neural network

We load the code from Raschka chapter 12. Available in this [auxiliary notebook](neural_network_auxiliary.ipynb).

```python
from neuralnet import NeuralNetMLP

clf_nn = {}
for epoch in [2, 20, 200]:
    clf_nn[epoch] = \
        NeuralNetMLP(n_hidden=100,
                     l2=0.01,
                     epochs=epoch,
                     eta=0.0005,
                     minibatch_size=100,
                     shuffle=True,
                     seed=1)
```

## Training a neural network

Let's to fit the network with 2 epochs

```python
clf_nn[2].fit(X_train=X_train[:55000],
              y_train=y_train[:55000],
              X_valid=X_train[55000:],
              y_valid=y_train[55000:])
```

    2/2 | Cost: 44472.59 | Train/Valid Acc.: 88.74%/91.50%

    <neuralnet.NeuralNetMLP at 0x7fbc98d6f3d0>

## Training a neural network

Let's to fit the network with 20 epochs

```python
clf_nn[20].fit(X_train=X_train[:55000],
               y_train=y_train[:55000],
               X_valid=X_train[55000:],
               y_valid=y_train[55000:])
```

    20/20 | Cost: 26911.76 | Train/Valid Acc.: 92.89%/94.60%

    <neuralnet.NeuralNetMLP at 0x7fbc98d6f1c0>

## Training a neural network

Let's to fit the network with 200 epochs

```python
clf_nn[200].fit(X_train=X_train[:55000],
                y_train=y_train[:55000],
                X_valid=X_train[55000:],
                y_valid=y_train[55000:])
```

    200/200 | Cost: 14340.23 | Train/Valid Acc.: 96.47%/96.80%

    <neuralnet.NeuralNetMLP at 0x7fbcbaed02b0>

## Training a neural network

What happened to errors?
- Almost halved for ten fold increase in epochs.

# Outro supervised learning

## Outro supervised ML

We have seen that sacrificing 'unbiased' property, we get much stronger models
- General tradeoff: overfitting/underfitting
- Issue: non-linear models are less ***transparent***

## Outro - non-linear ML

Tree based models
- Simple, in particular decision tree
- Ensemble version performed best
- Later in course we extend to allow for estimating causal effects (causal forest and related models)

Kernel/support vector machines
- Less common nowadays, replaced by neural networks

## Outro - non-linear ML

Very big potential for research of neural networks
- Understanding causal effects of non-linear data, e.g.
  - Visual and audible input
  - Human life trajectory
- Understanding consequences of algorithms
- Potentially for estimation

## Outro - non-linear ML

We have only scratched the surface of neural networks
- Architectures can include:
  - Depth, activation, regularization, temporal structure, convolution
  - Deep learning is half engineering/half science

The world is your oyster
- Deep learning courses are available online and in most universities
- Two big frameworks: pyTorch or TensorFlow
  - May want to invest in GPU!
- Melissa Dell from Harvard has a course called, *Unleashing Novel Data at Scale*, see here for the [reading list](https://dell-research-harvard.github.io/teaching/economics-2355) that may be of interest


```python
import sys
sys.path.append('..')
import torch
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sympy import simplify_logic

from lens.utils.base import validate_network
from lens.utils.relu_nn import get_reduced_model, prune_features
from lens import logic
import lens

torch.manual_seed(0)
np.random.seed(0)
```

```python
x = pd.read_csv('dsprites_c_train.csv', index_col=0)
y = pd.read_csv('dsprites_y_train.csv', index_col=0)
```

```python
base_concepts = ['color', 'shape', 'scale', 'rotation', 'x_pos', 'y_pos']
base_concepts
```

    ['color', 'shape', 'scale', 'rotation', 'x_pos', 'y_pos']

```python
colors = ['white']
shapes = ['square', 'ellipse', 'heart']
scale = ['very small', 'small', 's-medium', 'b-medium', 'big', 'very big']
rotation = ['0°', '5°', '10°', '15°', '20°', '25°', '30°', '35°']
x_pos = ['x0', 'x2', 'x4', 'x6', 'x8', 'x10', 'x12', 'x14', 'x16', 'x18', 'x20', 'x22', 'x24', 'x26', 'x28', 'x30']
y_pos = ['y0', 'y2', 'y4', 'y6', 'y8', 'y10', 'y12', 'y14', 'y16', 'y18', 'y20', 'y22', 'y24', 'y26', 'y28', 'y30']
concepts = colors + shapes + scale + rotation + x_pos + y_pos
```

```python
x_train = torch.tensor(x.values, dtype=torch.float)
print(x_train.shape)
print(y_train.shape)
print(n_classes)
x
```

    torch.Size([5530, 50])
    torch.Size([5530, 18])
    18

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      ...
      40
      41
      42
      43
      44
      45
      46
      47
      48
      49

      0
      1.0
      1.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      1.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      1
      1.0
      0.0
      1.0
      0.0
      1.0
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.0

      2
      1.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.0
      0.0
      1.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      3
      1.0
      0.0
      0.0
      1.0
      1.0
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      1.0
      0.0

      4
      1.0
      0.0
      0.0
      1.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      5525
      1.0
      0.0
      0.0
      1.0
      0.0
      0.0
      0.0
      1.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      5526
      1.0
      0.0
      0.0
      1.0
      1.0
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      5527
      1.0
      0.0
      1.0
      0.0
      1.0
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      5528
      1.0
      0.0
      1.0
      0.0
      0.0
      0.0
      0.0
      0.0
      1.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      1.0
      0.0

      5529
      1.0
      0.0
      1.0
      0.0
      0.0
      0.0
      0.0
      1.0
      0.0
      0.0
      ...
      0.0
      0.0
      1.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

5530 rows × 50 columns

```python
y_train = torch.zeros((y.shape[0], y.shape[1]), dtype=torch.float)
y_train = torch.tensor(y.values, dtype=torch.float)
x_test = x_train
n_classes = y_train.size(1)
print(n_classes)
y_train
```

    18

    tensor([[1., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 1., 0.],
            ...,
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.]])

```python
y_train.sum(dim=0)
```

    tensor([321., 311., 295., 324., 286., 295., 325., 302., 302., 319., 293., 297.,
            302., 301., 329., 326., 321., 281.])

```python
torch.manual_seed(0)
np.random.seed(0)

layers = [
    torch.nn.Linear(x_train.size(1), 20 * n_classes),
    torch.nn.LeakyReLU(),
    lens.nn.XLinear(20, 10, n_classes),
    torch.nn.LeakyReLU(),
    lens.nn.XLinear(10, 5, n_classes),
    torch.nn.LeakyReLU(),
    lens.nn.XLinear(5, 1, n_classes),
    torch.nn.Softmax(),
]
model = torch.nn.Sequential(*layers)

optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
loss_form = torch.nn.BCELoss()
model.train()
need_pruning = True
for epoch in range(6000):
    # forward pass
    optimizer.zero_grad()
    y_pred = model(x_train)
    # Compute Loss
    loss = loss_form(y_pred, y_train)

    for module in model.children():
        if isinstance(module, torch.nn.Linear):
            loss += 0.001 * torch.norm(module.weight, 1)
            break

    # backward pass
    loss.backward()
    optimizer.step()

    if epoch > 3000 and need_pruning:
        prune_features(model, n_classes)
        #need_pruning = False

    # compute accuracy
    if epoch % 500 == 0:
        y_pred_d = torch.argmax(y_pred, dim=1)
        y_train_d = torch.argmax(y_train, dim=1)
        accuracy = y_pred_d.eq(y_train_d).sum().item() / y_train.size(0)
        print(f'Epoch {epoch}: train accuracy: {accuracy:.4f}')
```

    /home/pietro/anaconda3/envs/dev/lib/python3.7/site-packages/torch/nn/modules/container.py:117: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
      input = module(input)

    Epoch 0: train accuracy: 0.0588
    Epoch 500: train accuracy: 0.8325
    Epoch 1000: train accuracy: 0.9467
    Epoch 1500: train accuracy: 0.9467
    Epoch 2000: train accuracy: 0.9467
    Epoch 2500: train accuracy: 1.0000
    Epoch 3000: train accuracy: 1.0000
    Epoch 3500: train accuracy: 1.0000
    Epoch 4000: train accuracy: 1.0000
    Epoch 4500: train accuracy: 1.0000
    Epoch 5000: train accuracy: 1.0000
    Epoch 5500: train accuracy: 1.0000

# Local explanations

```python
np.set_printoptions(precision=2, suppress=True)
outputs = []
for i, (xin, yin) in enumerate(zip(x_train, y_train)):
    model_reduced = get_reduced_model(model, xin)
    for module in model_reduced.children():
        if isinstance(module, torch.nn.Linear):
            wa = module.weight.detach().numpy()
            break
    output = model_reduced(xin)

    pred_class = torch.argmax(output)
    true_class = torch.argmax(y_train[i])

    # generate local explanation only if the prediction is correct
    if pred_class.eq(true_class):
        local_explanation = logic.relu_nn.explain_local(model, x_train, y_train, xin, concepts)
        print(f'Input {(i+1)}')
        print(f'\tx={xin.detach().numpy()}')
        print(f'\ty={y_train[i].detach().numpy()}')
        print(f'\ty={output.detach().numpy()}')
        #print(f'\tw={wa}')
        print(f'\tExplanation: {local_explanation}')
        print()
    outputs.append(output)
    if i > 1:
        break
```

    Input 1
    	x=[1. 1. 0. 0. 1. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0.
     0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0.
     0. 0.]
    	y=[1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
    	y=[1.   0.02 0.04 0.02 0.03 0.46 0.56 0.   0.   0.   0.   0.   0.48 0.
     0.62 0.   0.04 0.  ]
    	Explanation: square & very small

    Input 2
    	x=[1. 0. 1. 0. 1. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0.
     0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0.
     0. 0.]
    	y=[0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
    	y=[0.72 0.   0.   0.   0.   0.   1.   0.47 0.36 0.55 0.48 0.33 0.47 0.
     0.62 0.   0.04 0.  ]
    	Explanation: ellipse & very small

    Input 3
    	x=[1. 0. 0. 1. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 1. 0. 0. 0.
     0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
     0. 0.]
    	y=[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0.]
    	y=[0.   0.   0.   0.   0.55 0.   0.   0.   0.   0.   0.53 0.   0.   0.
     0.62 0.   1.   0.  ]
    	Explanation: heart & big

# Combine local explanations

```python
y_train_d = torch.argmax(y_train, dim=1)
for target_class in range(n_classes):
    global_explanation, predictions, counter = logic.combine_local_explanations(model, x_train, y_train,
                                                                              topk_explanations=10,
                                                                              target_class=target_class,
                                                                              concept_names=concepts)

    y2 = torch.argmax(y_train, dim=1) == target_class
    accuracy = sum(predictions == y2.detach().numpy().squeeze()) / len(predictions)
    print(f'Class {target_class} - Global explanation: "{global_explanation}" - Accuracy: {accuracy:.4f}')
```

    Class 0 - Global explanation: "square & very small" - Accuracy: 1.0000
    Class 1 - Global explanation: "square & small" - Accuracy: 1.0000
    Class 2 - Global explanation: "square & s-medium" - Accuracy: 1.0000
    Class 3 - Global explanation: "square & b-medium" - Accuracy: 1.0000
    Class 4 - Global explanation: "square & big" - Accuracy: 1.0000
    Class 5 - Global explanation: "square & very big" - Accuracy: 1.0000
    Class 6 - Global explanation: "ellipse & very small" - Accuracy: 1.0000
    Class 7 - Global explanation: "ellipse & small" - Accuracy: 1.0000
    Class 8 - Global explanation: "ellipse & s-medium" - Accuracy: 1.0000
    Class 9 - Global explanation: "ellipse & b-medium" - Accuracy: 1.0000
    Class 10 - Global explanation: "ellipse & big" - Accuracy: 1.0000
    Class 11 - Global explanation: "ellipse & very big" - Accuracy: 1.0000
    Class 12 - Global explanation: "heart & very small" - Accuracy: 1.0000
    Class 13 - Global explanation: "heart & small" - Accuracy: 1.0000
    Class 14 - Global explanation: "heart" - Accuracy: 0.7231
    Class 15 - Global explanation: "heart & b-medium" - Accuracy: 1.0000
    Class 16 - Global explanation: "heart & big" - Accuracy: 1.0000
    Class 17 - Global explanation: "heart & very big" - Accuracy: 1.0000


```python
from sympy import exp
from mpmath import legenp
# from sympy import legendre
```

```python
import sympy
```

```python
x = sympy.Symbol("x")
str_expr = "legenp(l, m, x)"
expr = sympy.sympify(str_expr)
f = sympy.lambdify([l, m, x], expr, "mpmath")
```

```python
sympy.Symbol?
```

```python
sympy.diff(f, x)
```

```python
sympy.sympify([x], , "mpmath")
```

    [x]

```python
lambdify([l, m, ra, de], expr, "scipy")
```

    mpf('5.0')

```python
sympy.diff(legenp(1, 0, x))
```