## Universal Approximation Theorem - Gradient Descent Optimisation

Here we study the possibility to approximate arbitrary functions $f: [0,1]\rightarrow \mathbb{R}$ by using MLPs with a single hidden layer ($n_1$ hidden units). Since this is a regression problem we use the MSE cost:

The MSE cost for a neural net with a 1d input $x$, a single hidden layer with $n$ units and a linear output layer is given by

$J_{\rm MSE}(\mathbf{\theta}) = \frac{1}{2m}\sum_{i=1}^m \left(y^{(i)} - (\sum_{k=1}^{n_1} w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+b_2)\right)^2$

where we have used the sigmoid ('logit') function $\sigma$ as activation function.

Finally, we will use mini-batch-gradient descent to minimize MSE cost.

The dataset is given by suitable $x$-values (in the interval $[0,1]$) and associated function values $f(x)$, i.e. $\{(x^{(i)},y^{(i)}\,=\,f(x^{(i)}))\, |\, i=1,\dots,m\}$. The data for the training will be generated on the fly.

Goals:
* Learn how a given function can be represented with a single layer MLP.
* Understand that, in principle, it can be learned from sample data.
* Understand that the optimization based on plain gradient is not always straightforward.
* Experience that the choice of the hyper-parameters number of hidden units, batchsize, learning rate is tricky.

## PART I: Calculation of the Formulas

Compute the formulas for gradient descent for this problem, i.e. compute the derivatives w.r.t. parameters $w_{1,k},w_{2,k},b_{1,k},b_2$ and formulate the according update rules.

__YOUR SOLUTION:__

MSE Cost:
$$J_{\rm MSE}(\mathbf{\theta}) = \frac{1}{2m}\sum_{i=1}^m \left(y^{(i)} - \left(\sum_{k=1}^{n} w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+b_2\right)\right)^2$$

Derivative in $\mathbf{\theta}_k$:
$$
\begin{align}
\frac {\partial{J_{\rm MSE}(\theta)}} {\partial{\theta_k}} &=
\frac {\partial} {\partial{\theta_k}}
\frac{1}{2m}\sum_{i=1}^m \left(y^{(i)} - \left(\sum_{k=1}^{n} w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+b_2\right)\right)^2 \\
&=
\frac{1}{2m}\sum_{i=1}^m
\frac {\partial} {\partial{\theta_k}}
\left(y^{(i)} - \left(\sum_{k=1}^{n} w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+b_2\right)\right)^2 \\
&=
\frac{1}{m}\sum_{i=1}^m
\left(y^{(i)} - \left(\sum_{k=1}^{n} w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+b_2\right)\right)
\frac {\partial} {\partial{\theta_k}}
\left(y^{(i)} - \left(\sum_{k=1}^{n} w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+b_2\right)\right)
\\
&=
\frac{1}{m}\sum_{i=1}^m
\left(y^{(i)} - \left(\sum_{k=1}^{n} w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+b_2\right)\right)
\left(0 - \frac {\partial} {\partial{\theta_k}} \left(\sum_{k=1}^{n} w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+b_2\right)\right)
\\
&=
\frac{1}{m}\sum_{i=1}^m
\left(y^{(i)} - \left(\sum_{k=1}^{n} w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+b_2\right)\right)
\left(-1* \frac {\partial} {\partial{\theta_k}} \sum_{k=1}^{n}
w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+
\frac {\partial} {\partial{\theta_k}} b_2\right)
\\
&=
-\frac{1}{m}\sum_{i=1}^m
\left(y^{(i)} - \left(\sum_{k=1}^{n} w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+b_2\right)\right)
\left(\sum_{k=1}^{n}\frac {\partial} {\partial{\theta_k}}
w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+
\frac {\partial} {\partial{\theta_k}} b_2\right)
\\
\end{align}
$$

__Partial derivatives__ w.r.t. parameters $w_{1,k},w_{2,k},b_{1,k},b_2$:

- Derivative in $w_{1,k}$:
$$
\begin{align}
\frac {\partial{J_{\rm MSE}(\theta)}} {\partial{w_{1,k}}}
&=
-\frac{1}{m}\sum_{i=1}^m
\left(y^{(i)} - \left(\sum_{k=1}^{n} w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+b_2\right)\right)
\left(\sum_{k=1}^{n}\frac {\partial} {\partial{w_{1,k}}}
w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+
\frac {\partial} {\partial{w_{1,k}}} b_2\right)
\\
&=
-\frac{1}{m}\sum_{i=1}^m
\left(y^{(i)} - \left(\sum_{k=1}^{n} w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+b_2\right)\right)
\left(\sum_{k=1}^{n} w_{2,k}
\frac {\partial} {\partial{w_{1,k}}}
\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+
0\right)
\\
&=
-\frac{1}{m}\sum_{i=1}^m
\left(y^{(i)} - \left(\sum_{k=1}^{n} w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+b_2\right)\right)
\left(\sum_{k=1}^{n} w_{2,k}
\sigma'(w_{1,k}\cdot x^{(i)} +b_{1,k})(x^{(i)})\right)
\\
\end{align}
$$

- Derivative in $w_{2,k}$:
$$
\begin{align}
\frac {\partial{J_{\rm MSE}(\theta)}} {\partial{w_{2,k}}}
&=
-\frac{1}{m}\sum_{i=1}^m
\left(y^{(i)} - \left(\sum_{k=1}^{n} w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+b_2\right)\right)
\left(\sum_{k=1}^{n}\frac {\partial} {\partial{w_{2,k}}}
w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+
\frac {\partial} {\partial{w_{2,k}}} b_2\right)
\\
&=
-\frac{1}{m}\sum_{i=1}^m
\left(y^{(i)} - \left(\sum_{k=1}^{n} w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+b_2\right)\right)
\left(\sum_{k=1}^{n}
\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+
0\right)
\\
&=
-\frac{1}{m}\sum_{i=1}^m
\left(y^{(i)} - \left(\sum_{k=1}^{n} w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+b_2\right)\right)
\left(\sum_{k=1}^{n}
\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})\right)
\\
\end{align}
$$

- Derivative in $b_{1,k}$:
$$
\begin{align}
\frac {\partial{J_{\rm MSE}(\theta)}} {\partial{b_{1,k}}}
&=
-\frac{1}{m}\sum_{i=1}^m
\left(y^{(i)} - \left(\sum_{k=1}^{n} w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+b_2\right)\right)
\left(\sum_{k=1}^{n}\frac {\partial} {\partial{b_{1,k}}}
w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+
\frac {\partial} {\partial{b_{1,k}}} b_2\right)
\\
&=
-\frac{1}{m}\sum_{i=1}^m
\left(y^{(i)} - \left(\sum_{k=1}^{n} w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+b_2\right)\right)
\left(\sum_{k=1}^{n}w_{2,k}
\frac {\partial} {\partial{b_{1,k}}}
\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+
0\right)
\\
&=
-\frac{1}{m}\sum_{i=1}^m
\left(y^{(i)} - \left(\sum_{k=1}^{n} w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+b_2\right)\right)
\left(\sum_{k=1}^{n}w_{2,k}
\sigma'(w_{1,k}\cdot x^{(i)} +b_{1,k})(1)\right)
\\
&=
-\frac{1}{m}\sum_{i=1}^m
\left(y^{(i)} - \left(\sum_{k=1}^{n} w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+b_2\right)\right)
\left(\sum_{k=1}^{n}w_{2,k}
\sigma'(w_{1,k}\cdot x^{(i)} +b_{1,k})\right)
\\
\end{align}
$$

- Derivative in $b_2$:
$$
\begin{align}
\frac {\partial{J_{\rm MSE}(\theta)}} {\partial{b_2}}
&=
-\frac{1}{m}\sum_{i=1}^m
\left(y^{(i)} - \left(\sum_{k=1}^{n} w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+b_2\right)\right)
\left(\sum_{k=1}^{n}\frac {\partial} {\partial{b_2}}
w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+
\frac {\partial} {\partial{b_2}} b_2\right)
\\
&=
-\frac{1}{m}\sum_{i=1}^m
\left(y^{(i)} - \left(\sum_{k=1}^{n} w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+b_2\right)\right)
\left(0+1\right)
\\
&=
-\frac{1}{m}\sum_{i=1}^m
\left(y^{(i)} - \left(\sum_{k=1}^{n} w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+b_2\right)\right)
\\
\end{align}
$$

__The Update Rules__:

Some definitions and additional notations to clarify the calculation:

$$
\begin{align}
\sigma'(z) &= \sigma(z)(1-\sigma(z))
\\
\hat{y}^{(i)} &=h_{\theta}(x^{(i)})=\sum_{k=1}^{n} w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+b_2
\\
\alpha&: \,learning\,rate
\\
\sigma(w_{1,k}\cdot x +b_{1,k})&: \,result \,from \,hidden \,layer
\end{align}
$$

- Update rule for $w_{1,k}$:
$$
\begin{align}
\frac {\partial{J_{\rm MSE}(\theta)}} {\partial{w_{1,k}}}
&=
-\frac{1}{m}\sum_{i=1}^m
\left(y^{(i)} - \left(\sum_{k=1}^{n} w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+b_2\right)\right)
\left(\sum_{k=1}^{n} w_{2,k}
\sigma'(w_{1,k}\cdot x^{(i)} +b_{1,k})(x^{(i)})\right)
\\
\\
\nabla_{w_{1,k}}J_{\rm MSE}(\theta)
&=
-\frac{1}{m}\sum_{i=1}^m
\left(y^{(i)} -\hat{y}^{(i)}\right)
\left(\sum_{k=1}^{n}
\left(w_{2,k} (\sigma'(w_{1,k}\cdot x^{(i)} +b_{1,k}))(x^{(i)})\right)\right)
\\
&=
-\frac{1}{m}\sum_{i=1}^m
\left(y^{(i)} -\hat{y}^{(i)}\right)
\left(\sum_{k=1}^{n}
\left(w_{2,k} (\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k}))(1-\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k}))(x^{(i)})\right)\right)
\\
\\
w_{1,k} \leftarrow w_{1,k}-\alpha\nabla_{w_{1,k}}J_{\rm MSE}(\theta)
\end{align}
$$

- Update rule for $w_{2,k}$:
$$
\begin{align}
\frac {\partial{J_{\rm MSE}(\theta)}} {\partial{w_{2,k}}}
&=
-\frac{1}{m}\sum_{i=1}^m
\left(y^{(i)} - \left(\sum_{k=1}^{n} w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+b_2\right)\right)
\left(\sum_{k=1}^{n}
\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})\right)
\\
\\
\nabla_{w_{2,k}}J_{\rm MSE}(\theta)
&=
-\frac{1}{m}\sum_{i=1}^m
\left(y^{(i)} - \hat{y}^{(i)}\right)
\left(\sum_{k=1}^{n}
\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})\right)
\\
\\
w_{2,k} \leftarrow w_{2,k}-\alpha\nabla_{w_{2,k}}J_{\rm MSE}(\theta)
\end{align}
$$

- Update rule for $b_{1,k}$:
$$
\begin{align}
\frac {\partial{J_{\rm MSE}(\theta)}} {\partial{b_{1,k}}}
&=
-\frac{1}{m}\sum_{i=1}^m
\left(y^{(i)} - \left(\sum_{k=1}^{n} w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+b_2\right)\right)
\left(\sum_{k=1}^{n}w_{2,k}
\sigma'(w_{1,k}\cdot x^{(i)} +b_{1,k})\right)
\\
\\
\nabla_{b_{1,k}}J_{\rm MSE}(\theta)
&=
-\frac{1}{m}\sum_{i=1}^m
\left(y^{(i)} - \hat{y}^{(i)}\right)
\left(\sum_{k=1}^{n}
\left(w_{2,k} \sigma'(w_{1,k}\cdot x^{(i)} +b_{1,k})\right)\right)
\\
&=
-\frac{1}{m}\sum_{i=1}^m
\left(y^{(i)} - \hat{y}^{(i)}\right)
\left(\sum_{k=1}^{n}
\left(w_{2,k} (\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k}))(1-\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k}))\right)\right)
\\
\\
b_{1,k} \leftarrow b_{1,k}-\alpha\nabla_{b_{1,k}}J_{\rm MSE}(\theta)
\end{align}
$$

- Update rule for $b_2$:
$$
\begin{align}
\frac {\partial{J_{\rm MSE}(\theta)}} {\partial{b_2}}
&=
-\frac{1}{m}\sum_{i=1}^m
\left(y^{(i)} - \left(\sum_{k=1}^{n} w_{2,k}\sigma(w_{1,k}\cdot x^{(i)} +b_{1,k})+b_2\right)\right)
\\
\\
\nabla_{b_2}J_{\rm MSE}(\theta)
&=
-\frac{1}{m}\sum_{i=1}^m
\left(y^{(i)} - \hat{y}^{(i)}\right)
\\
\\
b_2 \leftarrow b_2-\alpha\nabla_{b_2}J_{\rm MSE}(\theta)
\end{align}
$$

## PART II: Implementation

In this part, you need to implement and train the model:

1. Implement the cells below for the functions `predict`, `cost`, `gradient`, `train`.
2. Generate the training data - by assuming a function on the unit interval $[0,1]$. Here, we provide two families of functions:
    * Beta distribution function: $b_{\alpha,\beta}(x)=x^\alpha\cdot(1-x)^\beta$
    * Sine function: $sin_\omega(x)=\sin(2\pi\omega\cdot x)$
You can also add some moderate amount of noise to the data in the form $y = f(x)+\sigma_Y * \epsilon$ where $\epsilon$ are standard normally distributed random numbers.
3. Normalise the data (both $x$- and $y$ values.
4. Train the model and convince yourself that the functional relationship could be learned. Can it be reliably learned? - Try several trainings in a row (starting from different random initial weights).

#### Plot Utility

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

def plot_function(x,y):
    plt.plot(x, y)
    plt.xlabel('x')
    plt.show()

def plot_compare_function(x,y1,y2, label1='', label2=''):
    plt.plot(x, y1, label=label1)
    plt.xlabel('x')
    plt.plot(x, y2, label=label2)
    if label1 and label2:
        plt.legend()
    plt.show()

```

### Model

```python
def sigmoid(z):
    return 1. / (1. + np.exp(-z))
```

```python
def predict(X,W1,b1,W2,b2):
    """
    Computes the output for the single hidden layer network (n1 units) with 1d input and 1d output.

    Arguments:
    W1 -- weights of the hidden layer with shape (n1,1)
    b1 -- biases of the hidden units with shape (n1,1)
    W2 -- weights of the output layer with shape (1,n1)
    b2 -- bias of the output
    X  -- input data with m samples and shape (1,m)

    Returns:
    A2 -- Output from the network of shape (1,m)

    """

    ### START YOUR CODE ###

    # Computing process in hidden layer
    h = np.dot(W1, X) + b1
    A1 = sigmoid(h)

    # Computing process in output layer
    A2 = (np.dot(W2, A1) + b2)
    #print(A2.shape)

    ### END YOUR CODE ###

    return A2
```

#### TEST - Prediction

```python
W1 = np.array([0.4,0.2,-0.4]).reshape(3,1) # n1 = 3
b1 = np.array([0.1,0.1,0.1]).reshape(3,1)
W2 = np.array([1,2,1]).reshape(1,3)
b2 = -1
X = np.linspace(-1,1,5).reshape((1,5))
Ypred = predict(X,W1,b1,W2,b2)
Yexp = np.array([0.99805844, 1.04946333, 1.09991675, 1.14913132, 1.19690185]).reshape(1,5)
np.testing.assert_array_almost_equal(Ypred,Yexp,decimal=8)
```

#### Cost

```python
def cost(Y,Ypred):
    """
    Computes the MSE cost for given ground truth Y and predicted Ypred
    Uses the predict function defined above.

    Arguments:
    Y -- ground truth output with shape (1,m)
    Ypred -- predicted output with shape (1,m)

    Returns:
    cost -- the MSE cost divided by 2.
    """
    ### START YOUR CODE ###

    J = (Y-Ypred)**2
    J /= 2*J.shape[1]
    cost = J.sum()

    ### END YOUR CODE ###
    return cost
```

#### TEST - Cost

```python
W1 = np.array([4,5,6]).reshape(3,1)
W2 = np.array([1,2,3]).reshape(1,3)
b1 = np.array([1,1,1]).reshape(3,1)
b2 = 2
X = np.linspace(-1,1,5).reshape(1,5)
Ypred = predict(X,W1,b1,W2,b2)
Y = 2.0*np.ones(5).reshape(1,5)
c = cost(Y,Ypred)
cexp = 9.01669099
np.testing.assert_almost_equal(c,cexp,decimal=8)
```

#### Gradient

```python
def gradient(W1,b1,W2,b2,X,Y):
    """
    Computes the gradient of the MSE cost for a single hidden layer network with 1d input and 1d output.
    The parts of the gradient associated with the weights array and bias array for the hidden layer,
    the weights array and the bias for the output layer are provided as separate numpy arrays of according
    dimension.

    Arguments:
    W1 -- weights of hidden layer with shape (n1,1)
    b1  -- biases of hidden layer with shape (n1,1)
    W2 -- weights of output layer with shape (1,n1)
    b2  -- biases of output layer
    X  -- input data with shape (1,m)
    Y  -- labels with shape (1,m)

    Returns:
    gradient -- dictionary with the gradients w.r.t. W1, W2, b1, b2 and according keys
                'dW1' with shape (n1,1)
                'db1' with shape (n1,1)
                'dW2' with shape (1,n1)
                'db2' a scalar
    """
    ### START YOUR CODE ###

    m = X.shape[1]
    n = W1.size

    # Hidden layer -> n1 x m
    h = np.dot(W1, X) + b1
    A1 = sigmoid(h)

    # Output layer -> 1 x m
    A2 = (np.dot(W2, A1) + b2)

    #dd = (Y-Y^) -> 1 x m
    dd = Y - A2

    w1 = A1*(1-A1)*W2.T*X
    dW1 = -(1/m) * np.dot(w1, dd.T)

    w2 = A1
    dW2 = -(1/m) * np.dot(dd, w2.T)

    b1 = A1*(1-A1)*W2.T
    db1 = -(1/m) * np.dot(b1, dd.T)

    db2 = -(1/m) * np.sum(dd)
    #print(dd)

    ### END YOUR CODE ###
    return {'dW1':dW1, 'dW2':dW2, 'db1':db1, 'db2':db2}
```

#### TEST - Gradient

```python
W1 = np.array([4,5,6]).reshape(3,1)
W2 = np.array([1,2,3]).reshape(1,3)
b1 = np.array([1,1,1]).reshape(3,1)
b2 = 2
X = np.array([1,2,3,4,5,6,7]).reshape((1,7))
Y = np.array([2,2,2,2,2,2,2]).reshape((1,7))
gradJ = gradient(W1,b1,W2,b2,X,Y)
dW1exp = np.array([0.00590214,0.00427602,0.00234663]).reshape(W1.shape)
db1exp = np.array([0.00579241,0.004247,0.00234079]).reshape(b1.shape)
dW2exp = np.array([5.99209251,5.99579451,5.99714226]).reshape(W2.shape)
db2exp = 5.99792323
np.testing.assert_array_almost_equal(gradJ['dW1'],dW1exp,decimal=8)
np.testing.assert_array_almost_equal(gradJ['db1'],db1exp,decimal=8)
np.testing.assert_array_almost_equal(gradJ['dW2'],dW2exp,decimal=8)
np.testing.assert_almost_equal(gradJ['db2'],db2exp,decimal=8)
```

#### Training Loop

```python
def train(X,Y,n1,nepochs,batchsize=32,learning_rate=0.1):
    """
    Performs the training by using MBGD for a MLP with a single hidden layer (n1 units) and 1d input and output layer.

    It starts with initializing the parameters:
    * the weights and the biases for the hidden units : W1,b1 of shape (n1,1)
    * the weights and the bias for the output layer: W2 of shape (1,n1) and scalar b2

    Then, it loops over the epochs and per epoch over the mini-batches. The number of batches is determined from the
    batchsize.
    """
    # initialize weights
    W1 = np.random.uniform(-1,1,n1).reshape(n1,1) / (2*np.sqrt(n1))
    b1 = np.zeros((n1,1), dtype=float)
    W2 = np.random.uniform(-1,1,n1).reshape(1,n1) / (2*np.sqrt(n1))
    b2 = 0.0

    m = X.shape[1]
    mb = int(m/batchsize)
    indices = np.arange(m)

    # remember the epoch id and cost after each epoch for constructing the learning curve at the end
    costs = []
    epochs = []

    # Initial cost value:
    epochs.append(0)
    Ypred = predict(X,W1,b1,W2,b2)
    costs.append(cost(Y,Ypred))

    # training loop
    for epoch in range(nepochs):
        ### START YOUR CODE ###

        np.random.shuffle(indices)

        for i in range(mb):
            # get batch indexes
            idx = indices[i*batchsize:(i+1)*batchsize]

            # prepare data
            xbatch = X[:,idx].reshape(1,batchsize)
            ybatch = Y[:,idx].reshape(1,batchsize)

            gr = gradient(W1, b1, W2, b2, xbatch, ybatch)
            W1 = W1 - learning_rate * gr['dW1']
            W2 = W2 - learning_rate * gr['dW2']
            b1 = b1 - learning_rate * gr['db1']
            b2 = b2 - learning_rate * gr['db2']

        ### END YOUR CODE ###

        epochs.append(epoch+1)
        Ypred = predict(X,W1,b1,W2,b2)
        costs.append(cost(Y,Ypred))

    print(costs[-1])
    params = {'W1':W1, 'W2':W2,'b1':b1,'b2':b2}
    return params, np.array(epochs), np.array(costs)
```

### Generate the (Training) Data

```python
def beta_fct(x,alpha,beta):
    """
    Parameters:
    x - input array
    alpha, beta -- larger values lead to more pronounced peaks
    """
    c = alpha/(alpha+beta)
    norm = c**alpha*(1-c)**beta
    return x**alpha*(1-x)**beta/norm
```

```python
def sin_fct(x,omega):
    """
    Parameters:
    x -- input array
    omega -- frequency that defines the integer number of cycles within the unit interval
    """
    return np.sin(x*2*np.pi*omega)
```

```python
def generate_inputs(m, func, random=True, vargs=None, sigmaY=0.0):
    """
    Generates m (x,y=f(x))-samples by either generating random x-values in the unit interval (random=True) or by
    generating a grid of such values. Then the y values (used as labels below) are created from the function object
    `func`.
    Parameter needed to define the function `func` can be passed as vargs-dict.
    """
    if random:
        x = np.random.rand(1,m)
    else:
        x = np.linspace(0,1,m).reshape(1,m)
    y = func(x,**vargs) + sigmaY*np.random.randn(*(1,m))
    return x,y
```

### Normalize the Input and Output

It turns out that it is important to normalize the input and the output data here.
Remember the mean and stdev computed for the training data so that you can also apply it to the test data!

```python
def normalize(X, mu=None, stdev=None):
    """
    Normalizes the data by using z-normalization. If mu and stdev are NOT specified, mean and stedev are
    computed from the given samples.

    Returns:
    X1 -- normalized data (array of the same shape as input)
    mu -- mean
    stdev -- standard deviation
    """
    ### START YOUR CODE ###
    if not mu:
        mu = np.mean(X)
    if not stdev:
        stdev = np.std(X)
    X1 = (X-mu)/stdev
    ### END YOUR CODE ###

    return X1,mu,stdev
```

```python
def inv_normalize(X1, mu, stdev):
    """
    Invert the normalization. This is needed to bring the predicted values back to the original scale.

    Returns:
    X -- unnormalized data (array of the same shape as input X1)
    """
    ### START YOUR CODE ###
    X = X1 * stdev + mu
    ### END YOUR CODE ###

    return X
```

### Perform the Training

Includes generating and normalizing the data and training. Test data can be generated as non-random.

Make sure that you do the test performance on the right scales (of both x-values and y-values)!

```python
mtrain = 1000
func = beta_fct
vargs={'alpha':2.0,'beta':2.0}
#func = sin_fct
#vargs={'omega':1.0}
X,Y = generate_inputs(mtrain,func,vargs=vargs, random=False, sigmaY=0.0)
X1, muX, stdevX = normalize(X)
Y1, muY, stdevY = normalize(Y)
```

```python
plt.plot(X1[0,:],Y1[0,:],'+')
```

```python
n_hidden = 10
nepochs = 1000 # number of epochs
batchsize = 64
learning_rate = 0.1

params, epochs, costs = train(X1, Y1, n_hidden, nepochs, batchsize=batchsize,learning_rate=learning_rate)
plt.semilogy(epochs,costs)
```

### Test

Compute the predicted values on the test set and compute the MSE cost.
Prepare a (x,y)-plot with the ground truth test values and the predicted values.

```python
xx = np.arange(0,1,0.05).reshape(1,int(1/0.05))
yy = func(xx, **vargs)
plt.plot(xx[0,:],yy[0,:],'r--')

xx0,_,_ = normalize(xx, muX, stdevX)
yypred0 = predict(xx0, params['W1'],params['b1'],params['W2'],params['b2'])
yypred = inv_normalize(yypred0, muY, stdevY)
plt.plot(xx[0,:],yypred[0,:],'g-')

#plt.plot(X[0,:],Y[0,:],'b.')
```

```python

```


# Some brief concepts on numerical python - NUMPY

MAIN QUEST: Exercise about training, test and validation. (1 week)

SUB QUEST: NONE

Python is not designed specifically for mathematical and scientific computing.
In particular, Python lists are very flexible containers, but they are poorly suited to efficiently represent common mathematical constructs like vectors and matrices.

Fortunately, the **numpy** package (module) exists. It is a package that provide high-performance vector, matrix and higher-dimensional data structures for Python. It is implemented in C and Fortran so when calculations are vectorized (formulated with vectors and matrices), performance is very good. It is used in almost all numerical computation using Python.

Why not simply use Python lists for computations instead of creating a new array type?

There are several reasons:

* Python lists are very general. They can contain any kind of object. They are dynamically typed. They do not support mathematical functions such as matrix and dot multiplications, etc. Implementating such functions for Python lists would not be very efficient because of the dynamic typing.
* Numpy arrays are statically typed and homogeneous. The type of the elements is determined when array is created.
* Numpy arrays are memory efficient.
* Because of the static typing, fast implementation of mathematical functions such as multiplication and addition of numpy arrays can be implemented in a compiled language (C and Fortran is used).

## Basics of Numpy

To use **numpy** it is needed to import the module:

```python
import numpy as np
```

## Creating numpy arrays
There are a number of ways to initialize new numpy arrays, for example from

1. A Python list or tuples
2. Using array-generating functions, such as `arange`, `linspace`, etc.
3. Reading data from files

numpy.array is a function that returns a numpy.ndarray. There is no object type numpy.array.

### 1. From a list
For example, to create new vector and matrix arrays from Python lists we can use the `numpy.array` function.

```python
# a vector: the argument to the array function is a Python list
v = np.array([1,2,3,4])
v
```

    array([1, 2, 3, 4])

```python
type(v)
```

    numpy.ndarray

```python
v.dtype
```

    dtype('int32')

```python
# a matrix: the argument to the array function is a nested Python list
M = np.array([[1, 2], [3, 4]])
M
```

    array([[1, 2],
           [3, 4]])

If we want, we can explicitly define the type of the array data when we create it, using the `dtype` keyword argument:

```python
M = np.array([[1, 2], [3, 4]], dtype=int)
M
```

    array([[1, 2],
           [3, 4]])

Common type that can be used with dtype are: int, float, complex, bool, object, etc.

We can also explicitly define the bit size of the data types, for example: int64, int16, float128, complex128.

### 2. Using array-generating functions
For larger arrays it is inpractical to initialize the data manually, using explicit python lists. Instead we can use one of the many functions in `numpy` that generates arrays of different forms. Some of the more common are:

**Zeros and Ones**

```python
np.zeros(5, dtype=float)
```

    array([0., 0., 0., 0., 0.])

```python
np.ones(5,dtype=float)
```

    array([1., 1., 1., 1., 1.])

```python
a = np.zeros((2,3),dtype=np.int64)
a.shape
```

    (2, 3)

**arange**

```python
x = np.arange(0, 20, 1) # arguments: start, stop, step
x
```

    array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16,
           17, 18, 19])

**linspace and logspace**

```python
print("A linear grid of 5 elements between 0 and 1:")
print(np.linspace(0, 1,5))
```

    A linear grid of 5 elements between 0 and 1:
    [0.   0.25 0.5  0.75 1.  ]

**Creating random arrays**

```python
# uniform random numbers in [0,1]
np.random.rand(5,5)
```

    array([[0.00594451, 0.07510901, 0.44177116, 0.9870086 , 0.48399101],
           [0.03449061, 0.56418171, 0.74848344, 0.82323553, 0.51349367],
           [0.18969478, 0.85156371, 0.5108862 , 0.6572645 , 0.56298342],
           [0.93090994, 0.04054921, 0.3798979 , 0.11006237, 0.86609357],
           [0.8468136 , 0.43201715, 0.08609731, 0.69660495, 0.989217  ]])

```python
# 5 samples from a normal distribution with a mean of 10 and a variance of 3:
np.random.normal(10, 3, 5)
```

    array([13.24030147,  6.97283327, 10.02331165, 11.99978282,  7.13639879])

Although out of the scope, we will need permutations at some point. Thus, here comes how to create a permutation of the elements of a given list.

```python
np.random.permutation([1,2,3,4,5,6,7])
```

    array([2, 6, 4, 1, 7, 3, 5])

Observe that each time you execute the former line, the result changes. If you want to get the same permutation everytime, we can set the seed of the Pseudo-Random Number Generator.

```python
np.random.seed(1)
np.random.permutation([1,2,3,4,5,6,7])
```

    array([7, 3, 2, 1, 5, 4, 6])

*Few remarks on NANs:*

By definition, NaN is a float point number which is not equal to any other number

```python
np.nan != np.nan
```

    True

Thus, the equal operator can not be used for detecting NaN

Instead, isnan function is used:

```python
np.isnan(x)
```

    array([False, False, False, False, False, False, False, False, False,
           False, False, False, False, False, False, False, False, False,
           False, False])

## Manipulating arrays

```python
lst = [10, 20, 30, 40] #python list
arr = np.array([10, 20, 30, 40],dtype='int64') #numpy array
M = np.array([[10, 20, 30, 40],[50, 60, 70, 80]]) #numpy matrix
```

** Element indexing **

```python
#get the first element of list
lst[0]
```

    10

```python
#get the first element of array
arr[0]
```

    10

```python
# M is a matrix, or a 2 dimensional array, taking two indices
print(M)
print(M[0,0]) # element from first row first column
print(M[1,1]) # element from second row second column
print(M[1,2])
```

    [[10 20 30 40]
     [50 60 70 80]]
    10
    60
    70

If we omit an index of a multidimensional array it returns the whole row
(or, in general, a N-1 dimensional array)

```python
M[1] # second row
```

    array([50, 60, 70, 80])

The same thing can be achieved with using `:` instead of an index:

```python
M[1,:] # second row, all columns
```

    array([50, 60, 70, 80])

```python
M[:,3] # all rows, fourth column
```

    array([40, 80])

We can assign new values to elements in an array using indexing:

```python
M[0,0] = 1
M
```

    array([[ 1, 20, 30, 40],
           [50, 60, 70, 80]])

```python
# also works for rows and columns
M[1,:] = 0
M
```

    array([[ 1, 20, 30, 40],
           [ 0,  0,  0,  0]])

Arrays are homogeneous; i.e. all elements of an array must be of the same type

```python
#Lists are heterogeneous
lst[1] = 'a string inside a list'
lst

```

    [10, 'a string inside a list', 30, 40]

```python
#Arrays are homogeneous
arr[1] = 'a string inside an array'
```

Once an array has been created, its dtype is fixed and it can only store elements of the same type. For this example where the dtype is integer, if we store a floating point number it will be automatically converted into an integer:

```python
arr[1] = 1.234
arr
```

    array([10,  1, 30, 40], dtype=int64)

** Index slicing **

Index slicing is the technical name for the syntax `M[lower:upper:step]` to extract part of an array:

```python
A = np.array([1,2,3,4,5])
#slice from second to fourth element, step is one
A[1:3:1]
```

    array([1, 3, 5])

```python
A[1:3:1] = [-2,-3]
A
```

    array([ 1, -2, -3,  4,  5])

We can omit any of the three parameters in `M[lower:upper:step]`, by default `lower` is the beginning , `upper` is the end of the array, and `step` is one

```python
A[::2] # step is 2, lower and upper defaults to the beginning and end of the array
```

    array([ 1, -3,  5])

```python
A[:3] # first three elements
```

    array([ 1, -2, -3])

```python
A[3:] # elements from index 3
```

    array([4, 5])

Negative indices counts from the end of the array:

```python
A[-1] # the last element in the array
```

    5

```python
A[-3:] # the last three elements
```

    array([-3,  4,  5])

```python
A[::-1] #Step backwards, it returns an array with elements in reverse order
```

    array([ 5,  4, -3, -2,  1])

Index slicing works exactly the same way for multidimensional arrays, but every dimension separated by comma:

```python
M
```

    array([[ 1, 20, 30, 40],
           [ 0,  0,  0,  0]])

```python
#a block from the original array
#all rows, two central columns
M[:, 1:3]

```

    array([[20, 30],
           [ 0,  0]])

```python
# all row, skiping even columns
M[:, ::2]
```

    array([[ 1, 30],
           [ 0,  0]])

** Indexing with other arrays  (*Fancy indexing*)**

Arrays allow for a more sophisticated kind of indexing: you can index an array with another array, and in particular with an array of boolean values.  This is particluarly useful to **filter**
information from an array that matches a certain condition.

```python
import numpy as np
arr = np.array([10,1,30,40])
print(arr)
mask = arr < 9 # construct a boolean array
               #where i-th eleement is True if the i-th element of arr is less than 9
mask
```

    [10  1 30 40]

    array([False,  True, False, False])

```python
print('Values below 9:', arr[mask])
```

    Values below 9: [1]

The index mask can be converted to position index using the `where` function

```python
indices = np.where(mask,99999.,0.)
indices
```

    array([    0., 99999.,     0.,     0.])

```python
print('Resetting all values below 9 to 10...')
arr[arr < 9] = 10
print(arr)
arr < 9
```

    Resetting all values below 9 to 10...
    [10 10 30 40]

    array([False, False, False, False])

It is also possible to select using integer arrays that represent indexes.

```python
print(arr)
row_indices = [1, 2 ,3]
arr[row_indices]
```

    [10 10 30 40]

    array([10, 30, 40])

```python
a = np.array([2, 4, 6, 8], float)
b = np.array([0, 0, 1, 3, 2, 1], int)  # the 0th, 0th, 1st, 3rd, 2nd, and 1st elements of a
a[b]
```

    array([2., 2., 4., 8., 6., 4.])

For multidimensional arrays, we have to send multiple one-dimensional integer arrays to the
selection bracket, one for each axis.

```python
a = np.array([[1, 4], [9, 16]], float)
print(a)
b = np.array([0, 0, 1, 1, 1], int)
c = np.array([0, 1, 1, 1, 0], int)
a[b,c]
```

    [[ 1.  4.]
     [ 9. 16.]]

    array([ 1.,  4., 16., 16.,  9.])

`np.where` can also be used as a conditional statement.

```python
print (a)
print (np.where(a==1.0,1,0))

```

    [[ 1.  4.]
     [ 9. 16.]]
    [[1 0]
     [0 0]]

# The machine learning pipeline

Modeling churn means to understand what keeps the customer engaged to our product. Its analysis goal is to predict or describe the **churn rate** i.e. the rate at which customer leave or cease the subscription to a service. Its value lies in the fact that engaging new customers is often more costly than retaining existing ones. For that reason subscription business-based companies usually have proactive policies towards customer retention.

In this case study, we aim at building a machine learning based model for customer churn prediction on data from a Telecom company. Each row on the dataset represents a subscribing telephone customer. Each column contains customer attributes such as phone number, call minutes used during different times of day, charges incurred for services, lifetime account duration, and whether or not the customer is still a customer.

This case is partially inspired in Eric Chiang's analysis of churn rate. Data is available from the University of California Irvine machine learning repositories data set.

## Goal
 + Implement a full machine learning pipeline.
 + Understand the concepts of training, validation, and test.

```python
%load datalist.py
```

```python
%reset -f
from datalist import *

dl = DataList()
dl.read_csv('./files/churn_curated_numerical.csv',hasHeader = False)
```

```python
#IF YOU DON'T HAVE DATALIST READY YOU CAN USE:

import pandas as pd

dl=pd.read_csv('./files/churn_curated_numerical.csv',header=-1)
```

```python
data = dl.get_values()
```

```python
data.shape[0] * data.shape[1]
```

    63327

```python
X = data[:,:-1]
y = data[:,-1]
```

```python
np.unique(y)
```

    array([0., 1.])

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
plt.pie(np.c_[len(y)-np.sum(y),np.sum(y)][0],labels=['No Churn','Churn'],colors=['r','g'],shadow=True,autopct ='%.2f' )
fig = plt.gcf()
fig.set_size_inches(6,6)
np.array([len(y)-np.sum(y),np.sum(y)])
```

## Data

Observe data

```python
import numpy as np

np.mean(X,axis=0)
np.var(X,axis=0)
```

    array([1.58532433e+03, 1.79478761e+03, 8.75182028e-02, 2.00104799e-01,
           1.87315130e+02, 2.96580639e+03, 4.02647298e+02, 8.57114045e+01,
           2.57112237e+03, 3.96791913e+02, 1.85762804e+01, 2.55694661e+03,
           3.82815581e+02, 5.17804314e+00, 7.79202952e+00, 6.05575823e+00,
           5.68002683e-01, 1.72999748e+00])

A problem in Scikit-Learn is modeled as follows:

+ Input data is structured in Numpy arrays. The size of the array is expected to be [n_samples, n_features]:

    + *n_samples*: The number of samples ($N$): each sample is an item to process (e.g. classify). A sample can be a document, a picture, a sound, a video, an astronomical object, a row in database or CSV file, or whatever you can describe with a fixed set of quantitative traits.

    + *n_features*: The number of features ($d$) or distinct traits that can be used to describe each item in a quantitative manner. Features are generally real-valued, but may be boolean, discrete-valued or even cathegorical.

$${\rm feature~matrix:} {\bf X}~=~\left[
\begin{matrix}
x_{11} & x_{12} & \cdots & x_{1d}\\
x_{21} & x_{22} & \cdots & x_{2d}\\
x_{31} & x_{32} & \cdots & x_{3d}\\
\vdots & \vdots & \ddots & \vdots\\
\vdots & \vdots & \ddots & \vdots\\
x_{N1} & x_{N2} & \cdots & x_{Nd}\\
\end{matrix}
\right]$$

$${\rm label~vector:} {\bf y}~=~ [y_1, y_2, y_3, \cdots y_N]$$

The number of features must be fixed in advance. However it can be very high dimensional (e.g. millions of features) with most of them being zeros for a given sample.

Create and fit a decision tree (you can find it in the module sklearn.tree and the name is DecisionTreeClassifier)

```python
from sklearn import tree

clf = tree.DecisionTreeClassifier()
clf.fit(X,y)
```

    DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,
                           max_features=None, max_leaf_nodes=None,
                           min_impurity_decrease=0.0, min_impurity_split=None,
                           min_samples_leaf=1, min_samples_split=2,
                           min_weight_fraction_leaf=0.0, presort=False,
                           random_state=None, splitter='best')

Predict the data you used for training/fiting the classifer

```python
yhat = clf.predict(X)
```

EXERCISE: We need a measure of how well the classifier is performing. `yhat` is a list and our target outcome `y`is also a list. Create a measure of accuracy.

```python
#Your code here
```

    accuracy: 100.0%

One sensible way of measuring the goodness of a classifier is measuring the error rate, i.e. the number of times the classifier fails divided by the total number of elements.

$$err = \frac{1}{N}\sum \mathbb{1}_{\tilde{y}!=y}$$

where $\mathbb{1}_{\text{cond}}$ is the indicator function given a condition, $\text{cond}$, defined as

$$\mathbb{1}_{\text{cond}}=\left \{\begin{align} 1 & \quad\text{if cond = True}\\ 0 & \quad\text{otherwise} \end{align}\right.$$
Alternatively, we can report the accuracy, defined as the rate of success

$$acc = \frac{1}{N}\sum \mathbb{1}_{\tilde{y}==y}.$$

Observe that $acc = 1-err$.

`sklearn` reports this result using the method from module `metrics`, `.accuracy_score`.

```python
from sklearn import metrics

metrics.accuracy_score(y,yhat)
```

    1.0

 QUESTION: Is this a good result?

 QUESTION: Is this the value we expect to have when we apply this method in production?

In real applications we will train a classifier on a given data set but then apply the classifier to unseen data. Let us simulate this process by spliting the data set in two sets. We will call data we use for fiting the classifier training and data used for assessing the performance, test data.

EXERCISE: Split the data set 70% for training purposes and the rest for test purposes. You should end up with four variables `X_train`, `y_train`, `X_test`, `y_test`.

OTHER REQUIREMENTS: Reshuffle data using a permutation of the indexes (`np.random.permutation(...)`) and set the seed of the random number generator using `np.random.seed(42)`

```python
#Your code here
```

Split data in training and set, use the module cross_validation, train_test_split , use random_state=42 as an argument for reproductibility.

```python
from sklearn import model_selection

X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, train_size=0.7, random_state=42)
```

Let us try a new algorithm, nearest neighbor, with parameter n_neighbors = 1

```python
from sklearn import neighbors

clf = neighbors.KNeighborsClassifier(n_neighbors=1)
clf.fit(X_train,y_train)
yhat = clf.predict(X_train)
from sklearn import metrics

metrics.accuracy_score(y_train,yhat)

```

    1.0

```python
yhat = clf.predict(X_test)
from sklearn import metrics

metrics.accuracy_score(y_test,yhat)
```

    0.811

 QUESTION: Is this a good result?

```python
#INSERT SNOOPING CODE

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()

X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, train_size=0.7, random_state=42)

X_train_scaled = scaler.fit_transform(X_train)

clf = neighbors.KNeighborsClassifier(n_neighbors=1)
clf.fit(X_train_scaled,y_train)
yhat = clf.predict(X_train_scaled)
from sklearn import metrics

metrics.accuracy_score(y_train,yhat)

X_test_scaled = scaler.transform(X_test)

yhat = clf.predict(X_test_scaled)
from sklearn import metrics

metrics.accuracy_score(y_test,yhat)
```

    0.866

```python
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)

clf = neighbors.KNeighborsClassifier(n_neighbors=1)
clf.fit(X_train_scaled,y_train)
yhat = clf.predict(X_train_scaled)
from sklearn import metrics

metrics.accuracy_score(y_train,yhat)

```

    1.0

```python
from sklearn.preprocessing import StandardScaler

X_test_scaled = scaler.transform(X_test)

yhat = clf.predict(X_test_scaled)
from sklearn import metrics

metrics.accuracy_score(y_test,yhat)
```

    0.866

This result is ok, but how consistent is it? Maybe we have been lucky with the train-test partion. We can repeat this process for different values of the random_state (or just use random permutations) and report the average result.

EXERCISE: We want a vector of accuracies, `acc`, of shape (10,1) with the values of testing a 1-NN classifier on a `train_size=0.7` for the different random_states stored in the array `r_state`. Use `sklearn.cross_validation.train_test_split` function.

```python
r_state = [0,1,2,3,4,5,42,43,44,45]
```

```python
acc = []
for r_s in r_state:
    X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.7, random_state=r_s)
    scaler = StandardScaler()
    X_train_scaled = scaler.fit_transform(X_train)

    clf = neighbors.KNeighborsClassifier(n_neighbors=1)
    clf.fit(X_train_scaled,y_train)
    yhat = clf.predict(X_train_scaled)
    from sklearn import metrics

    metrics.accuracy_score(y_train,yhat)

    X_test_scaled = scaler.transform(X_test)

    yhat = clf.predict(X_test_scaled)
    from sklearn import metrics

    acc.append(metrics.accuracy_score(y_test,yhat))
acc = np.array([acc]).T
```

Check your code with the following visualization code:

```python
import matplotlib.pyplot as plt
fig = plt.figure()
plt.boxplot(acc)
plt.scatter(np.ones(acc.shape)+0.01*np.random.normal(size=(10,1)),acc,alpha = 0.5,color='r')
```

 QUIZ: Report the average accuracy.

```python
#Your code
value = (sum(acc)/len(acc))[0]
```

```python
assert np.abs(value-0.8691)<0.0001
```

# Model selection (I)

We have tried a 1-Nearest Neighbors classifiers but we could try also different values for the Nearest Neighbors. The selection of a model between different alternatives is called model selection. We can use the same strategy as before and report accuracies for the three models. Let us do it comparintg 1-NN, 3-NN and a DecisionTree.

EXERCISE: We want a matrix of accuracies, `acc`, of shape (10,3) with the values of testing a decision tree, a 1-NN, and a 3-NN classifier on a `train_size=0.7` for the different random_states stored in the array `r_state`.

```python
#Your code

from sklearn import metrics
from sklearn import tree
from sklearn import neighbors
from sklearn.preprocessing import StandardScaler

r_state = [0,1,2,3,4,5,42,43,44,45]

acc = np.zeros((len(r_state),3))

for i in range(len(r_state)):
    X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, train_size=0.7, random_state=r_state[i])

    #Your code here
    scaler = StandardScaler()
    X_train_scaled = scaler.fit_transform(X_train)

    clf = tree.DecisionTreeClassifier()
    clf.fit(X_train_scaled,y_train)
    X_test_scaled = scaler.transform(X_test)
    yhat_tr = clf.predict(X_test_scaled)

    clf = neighbors.KNeighborsClassifier(n_neighbors=1)
    clf.fit(X_train_scaled,y_train)
    X_test_scaled = scaler.transform(X_test)
    yhat_nn1 = clf.predict(X_test_scaled)

    clf = neighbors.KNeighborsClassifier(n_neighbors=3)
    clf.fit(X_train_scaled,y_train)
    X_test_scaled = scaler.transform(X_test)
    yhat_nn3 = clf.predict(X_test_scaled)

    acc[i,0] = metrics.accuracy_score(y_test,yhat_tr)
    acc[i,1] = metrics.accuracy_score(y_test,yhat_nn1)
    acc[i,2] = metrics.accuracy_score(y_test,yhat_nn3)
```

```python
import matplotlib.pyplot as plt
fig = plt.figure()
plt.boxplot(acc)
plt.scatter(np.tile(np.array([1,2,3]),(10,1))+0.01*np.random.normal(size=(10,3)),acc,alpha = 0.5,color='r')
```

QUIZ: What is the best of the three methods?

QUIZ: What is the expected accuracy of the selected method in exploitation over unseen data?

```python
acc[:,2].sum()/len(acc[:,2])
```

    0.8952

MAIN QUEST EXERCISE: The  `breast_cancer` dataset from `datasets` (check `load_breast_cancer`) reports a set of clinical trials with the outcome of breast cancer detection. We want to build a method to predict whether a patient has a potential cancer or not according to her clinical trials.

For that purpose we will use two different models, a support vector machine and a gradient boosting machine. We will train different settings of a support vector machine (`svm.SVC`) and a single gradient boosting machine (`ensemble.GradientBoostingMachine`) with the following parameters:

`svm.SVC(C=10.0,gamma = 1e-5,random_state=42)`

`svm.SVC(C=100.0,gamma = 1e-5,random_state=42)`

`svm.SVC(C=1000.0,gamma = 1e-6,random_state=42)`

`ensemble.GradientBoostingClassifier(random_state=42)`

For selection purposes and accuracy evaluation we will use `model_selection.train_test_split`.
The data set will be divided using parameters `test_size = 100` and `random_state=42`. As a result of this first division we will have a big training set and a 100 samples test set. Following that and using the same settings we will divide the remaining training set into the final training set and the validation set with 100 samples again.

Prepare the following answers:

Check the sizes of the training, validation and test sets.

Report the training accuracy of all four methods.

Report the validation accuracy for all methods.

Report the performance of all methods using the test set.

In the light of the answers obtained from the exercise:

Question 1: What is the size of the training set?

Question 2: What method do you select?

Question 3: What is the expected performance of the method selected?

```python
from sklearn.datasets import load_breast_cancer

data = load_breast_cancer()
X = data.data
y = data.target

X.shape
```

    (569, 30)

```python
#Your code
from sklearn import svm
from sklearn import ensemble

X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, test_size = 100, random_state=42)
X_train, X_val, y_train, y_val = model_selection.train_test_split(X_train, y_train, test_size = 100, random_state=42)

```

```python
print(X_train.shape[0])
print(X_val.shape[0])
print(X_test.shape[0])
```

    369
    100
    100

```python
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
clf = svm.SVC(C=10.0,gamma = 1e-5,random_state=42)
clf.fit(X_train_scaled,y_train)
X_val_scaled = scaler.transform(X_val)
yhat_val = clf.predict(X_val_scaled)
print(f"svm.SVC(C=10.0,gamma = 1e-5,random_state=42): acc={metrics.accuracy_score(y_val,yhat_val)}")

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
clf = svm.SVC(C=100.0,gamma = 1e-5,random_state=42)
clf.fit(X_train_scaled,y_train)
X_val_scaled = scaler.transform(X_val)
yhat_val = clf.predict(X_val_scaled)
print(f"svm.SVC(C=100.0,gamma = 1e-5,random_state=42): acc={metrics.accuracy_score(y_val,yhat_val)}")

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
clf = svm.SVC(C=1000.0,gamma = 1e-6,random_state=42)
clf.fit(X_train_scaled,y_train)
X_val_scaled = scaler.transform(X_val)
yhat_val = clf.predict(X_val_scaled)
print(f"svm.SVC(C=1000.0,gamma = 1e-6,random_state=42): acc={metrics.accuracy_score(y_val,yhat_val)}")

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
clf = ensemble.GradientBoostingClassifier(random_state=42)
clf.fit(X_train_scaled,y_train)
X_val_scaled = scaler.transform(X_val)
yhat_val = clf.predict(X_val_scaled)
print(f"ensemble.GradientBoostingClassifier(random_state=42): acc={metrics.accuracy_score(y_val,yhat_val)}")
```

    svm.SVC(C=10.0,gamma = 1e-5,random_state=42): acc=0.76
    svm.SVC(C=100.0,gamma = 1e-5,random_state=42): acc=0.97
    svm.SVC(C=1000.0,gamma = 1e-6,random_state=42): acc=0.97
    ensemble.GradientBoostingClassifier(random_state=42): acc=0.99

```python
#Choosen is GBC with acc 99%
#lets check performance
X_test_scaled = scaler.transform(X_test)
yhat_test = clf.predict(X_test_scaled)
print(f"Acc={metrics.accuracy_score(y_test,yhat_test)}")
```

    Acc=0.96


# Linear and Quadratic Discriminant Analysis

## Linear Discriminant Analysis

### Classifying with Bayes' Theorem

In the previous chapter we discussed logistic regression for the case of two response classes (e.g. 0 and 1). It models the conditional probability $\Pr(Y=k|X=x)$ directly through the use of the Sigmoid function. In this chapter we discuss an alternative approach that models the distribution of the predictors $X$ separately for each response class (i.e. given $Y$), and then uses Bayes' theorem to transform these into conditional probabilities for $\Pr(Y=k|X=x)$. Its main advantage compared to logistic regressions is that if classes are well-separated, parameter estimates for logistic regression tend to be unstable whereas linear discriminant analysis (LDA) does not suffer from this problem. Beyond that, LDA is a popular algorithm for multiple-class classification (i.e. the response has more than two classes, for example buy/hold/sell etc.) where logistic regression is not used that often (James et al. (2013)).

LDA assigns an object to class $k$ for which the computed probability is highest. These probabilities are calculated using Bayes' rule which states that

\begin{equation}
\begin{aligned}
\underbrace{\Pr(Y=k|X)}_{\text{posterior probability}} &= \frac{\overbrace{\Pr(X|Y=k)}^{\text{conditional probability}}  \quad \overbrace{\Pr(k)}^{\text{prior probability}}}{\underbrace{\Pr(X)}_{\text{evidence}}} \\[3ex]
 &= \frac{\Pr(X|Y=k) \Pr(k)}{\sum_{\ell=1}^K \Pr(X|Y=\ell) \Pr(\ell)}
\end{aligned}
\end{equation}

Above, $\Pr(k)$ is simply the prior probability of class $k$ (with $\sum_{k=1}^K \Pr(k) = 1$) that a randomly chosen observation comes from the $k$th class. $\Pr(X|Y=k)$ on the other hand is the class conditional density of $X$ in class $Y=k$. Following the notation in (Friedman et al. (2001)), we denote $\Pr(X|Y=k) \equiv f_k(x)$ to indicate that it is a density function. LDA's decision rule thus classifies an observation into class $k$ if

\begin{align}
\Pr(Y=k|X) &> \Pr(Y=j|X) \qquad \forall j \neq k \nonumber \\
\end{align}

or, if we substitute both sides of the inequality with Bayes' rule:

\begin{align}
\frac{f_k(x) \Pr(k)}{\sum_{\ell=1}^K f_\ell(x) \Pr(\ell)} &> \frac{f_j(x) \Pr(j)}{\sum_{\ell=1}^K f_\ell(x) \Pr(\ell)} \qquad \forall j \neq k \nonumber
\end{align}

The evidence term (denominator in above equation) can be omitted from the decision rule because it is merely a scaling factor (Raschka (2014)). This then yields the following simple decision boundary:

\begin{align}
f_k(x) \Pr(k) &> f_j(x) \Pr(j)  \qquad \forall j \neq k
\end{align}

This expression can also be written as
\begin{equation}
\delta_k(x) = \arg \max_k \; f_k(x) \Pr(k)
\end{equation}

### Bayes Decision Rule in LDA with One Feature

Suppose that $f_k(x)$ follows a Gaussian distribution. For the one-dimensional setting, that is if we have just one feature $p=1$, the normal density takes the well known form

\begin{equation}
f_k(x) = \frac{1}{\sqrt{2\pi \sigma_k^2}} \, \exp\left( - \frac{(x - \mu_k)^2}{2 \sigma_k^2} \right)
\end{equation}

where $\mu_k$ and $\sigma_k^2$ are the mean and variance for the $k$th class, respectively. For the moment, let us also assume that there is a shared variance term across all $K$ classes, i.e. $\sigma_1^2 = \sigma_2^2 = \ldots = \sigma_k^2$. Then plugging the normal distribution into our maximization problem, taking the log and doing some algebra - see the appendix of the script for detailed steps - we find that an observation is assigned to class $k$ for which $\delta_k(x)$ is greatest:

\begin{equation}
\delta_k(x) = \arg \max_k \left[\frac{x \mu_k}{\sigma^2} - \frac{\mu_k}{2\sigma^2} + \ln(\Pr(k)) \right]
\end{equation}

Below figure shows how LDA classifies data based on the above result. In the left subplot we see two separate normal densities representing a situation with two classes ($K \in \{\text{blue, green\}}$). $\Pr(k=\text{blue}) = \Pr(k=\text{green}) = 0.5$; equal for both classes. Both densities have the same variance $\sigma_1^2 = \sigma_2^2 = 1$ but different location parameter, $\mu_1 = -1.25, \mu_2 = 1.25$.

If we were to know these parameter, then LDA's decision boundary would be drawn exactly at zero (dashed line). If $\Pr(k=\text{blue}) > \Pr(k=\text{green})$, Bayes' decision boundary would move to the right, if $\Pr(k=\text{blue}) < \Pr(k=\text{green})$, to the left. There is some overlapping area leading to some uncertainty, but overall the error rate is minimized to a minimum. In reality however, we do not know the true location and scale parameter and hence we have to estimate them - what we will discuss momentarily. The right plot displays histograms of 50 randomly drawn observations from the aforementioned normal distribution. Given this data, LDA calculates $\mu_k, \sigma^2$ and uses $\delta_k(x)$ to draw the decision boundary (solid vertical line). Data points to the left belong to the blue class, all others to the green class. The dashed vertical line again displays the optimal decision boundary. Because we don't know the true location and scale parameter LDA relies on estimates. This introduces inaccuracy that is reduced the larger the data sample is (assuming our normal assumption is correct).

### Assumptions and Parameter Estimation

So far we have discussed how LDA draws its decision boundary with the help of Bayes rule and given the assumptions that the features follow a normal distribution. But in order to follow through with our classification task, estimates for $\Pr(k), \mu_k$, and $\sigma^2$ are required. Estimating the prior probability $\Pr(k)$ is no difficult job: we simply compute the fraction of training observations that belong to the $k$th class: $\hat{\Pr}(k) = n_k / n$, where $n_k$ represents the count of samples from class $k$ and $n$ the count of all samples. Location parameter $\mu_k$ is estimated using the average of training observation of the $k$th class and $\sigma^2$, the scale parameter, is the weighted average of the sample variance for each of the $K$ classes (Note that Friedman et al. (2001) and James et al (2013) both use a biased corrected version of $\hat{\sigma}^2$ (and $\hat{\Sigma}$ for the case of $p>0$) by dividing the summed terms by $n-K$ instead of $n$. The formula given here uses an uncorrected estimate of $\sigma$ and in that follows `Sklearn`'s implementation.)

\begin{align}
\hat{\mu}_k &= \frac{1}{n_k} \sum_{i:y_i=k} x_i \\
\hat{\sigma}^2 &= \frac{1}{n} \sum_{k=1}^K \sum_{i:y_i=k} (x_i - \hat{\mu}_k)^2
\end{align}

Given the assumption of normality and given these estimates for location and scale we are able to establish a decision rule that assigns each new data point to the class for which $\delta_k(x)$ is highest.

### Bayes Decision Rule in LDA with More Than One Feature

Above we have used the one-dimensional case with one predictor to introduce how LDA classifies an observation. Now we extend the classifier to work with multiple features ($p>1$). Again we assume that $X = (X_1, X_2, \ldots, X_p)$ is drawn from a (multivariate) normal distribution with a class specific mean vector $\mu_k$ of length $p$ and common covariance matrix $\Sigma$ of dimension $p \times p$. This is expressed as $X \sim N(\mu, \Sigma)$. The multivariate Gaussian density is defined as

\begin{equation}
f_k(x) = \frac{1}{(2\pi)^{p/2}|\Sigma|^{1/2}} \, \exp \left( -\frac{1}{2} (x-\mu_k)^T \Sigma^{-1} (x-\mu_k) \right)
\end{equation}

As before we plug this expression into our maximization problem, take the logarithm and perform a little bit of algebra (For the interested the different steps are shown in the appendix of the script). This yields to the following LDA's Bayes classifier rule, based on which an observation $X=x$ is assigned to the class for which $\delta_k(x)$ is largest.

\begin{equation}
\delta_k(x) = \arg \max_k \left[x^T \Sigma^{-1} \mu_k - \frac{1}{2} \mu_k^T\Sigma^{-1}\mu_k + \ln(\Pr(k))\right]
\end{equation}

The estimates for $\Pr(k), \mu_k$, and $\Sigma$ follow again the same approach as in the case of only one predictor.

The next figure plots LDA's Bayes decision boundary for a random training set with two features $X_1, X_2$. The colors indicate the binary response with blue circles indicating customers who accepted a product offer and green circles representing those who declined it. The bivariate normal contours (ellipses) represent iso-lines with the same probabilities. LDA uses Bayes' decision rule discussed above to classify any new data point into class $k$.

### LDA in Python
#### Setup

We will apply LDA in Python with the functions that are provided through the `sklearn` package and the 'Default' data set we used to introduce logistic regression in a previous chapter. `Sklearn`, short for Scikit-learn, is the go-to source for clustering, classification or regression algorithms in machine learning. It offers an abundant variety of functions and functionalities and is actively developed by a large community.

`Sklearn` is one of the most extensive package in Python with hundreds of functions. It is good practice to not load the full library as we did for example with `numpy` but to only load those functions that are needed to run your task at hand. This saves computer memory and with that improves the efficiency of your algorithm, especially if you are using your household PC to run it on larger data sets.

```python
%matplotlib inline
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn import metrics
plt.rcParams['font.size'] = 14
plt.style.use('seaborn-whitegrid')
```

```python
# Default data set is not available online. Data was extracted from R package "ISLR"
df = pd.read_csv('Data/Default.csv', sep=',')

# Factorize 'No' and 'Yes' in columns 'default' and 'student'
df['defaultFac'] = df.default.factorize()[0]
df['studentFac'] = df.student.factorize()[0]
df.head(5)
```

      default
      student
      balance
      income
      defaultFac
      studentFac

      0
      No
      No
      729.526495
      44361.625074
      0
      0

      1
      No
      Yes
      817.180407
      12106.134700
      0
      1

      2
      No
      No
      1073.549164
      31767.138947
      0
      0

      3
      No
      No
      529.250605
      35704.493935
      0
      0

      4
      No
      No
      785.655883
      38463.495879
      0
      0

```python
# Assign data to feature matrix X_train and response vector y_train
X_train = df[['balance', 'income', 'studentFac']].as_matrix()
y_train = df.defaultFac.as_matrix()
```

#### LDA Classifier Object & Fit
Now we are in a position to run the LDA classifier. This, as you can see from the three lines below, is as easy as it gets.

```python
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA

# Create LDA object and run classifier
lda = LDA(solver='lsqr')
lda = lda.fit(X_train, y_train)
lda
```

    LinearDiscriminantAnalysis(n_components=None, priors=None, shrinkage=None,
                  solver='lsqr', store_covariance=False, tol=0.0001)

The parameter `solver='lsqr'` specifies the method by which the covariance matrix is estimated. `lsqr` follows the approach introduced in the preceding subsection. Others such as `svd` or `eigen` are available. See [Scikit-learn's guide](http://scikit-learn.org/stable/modules/lda_qda.html#estimation-algorithms) or the [function description](http://scikit-learn.org/stable/modules/generated/sklearn.discriminant_analysis.LinearDiscriminantAnalysis.html).

Every function in `sklearn` has different attributes and methods. `Sklearn`s convention is to store anything that is derived from the data in attributes that end with a trailing underscore. That is to separate them from parameters that are set by the user (Mueller and Guido (2017)). For example the estimated covariance matrix can be printed with this command.

```python
lda.covariance_
```

    array([[  2.05277550e+05,  -9.37165654e+05,   4.21453998e+01],
           [ -9.37165654e+05,   1.77777941e+08,  -4.57857337e+03],
           [  4.21453998e+01,  -4.57857337e+03,   2.07468022e-01]])

In a Jupyter notebook, to see all options you can simply type `lda.` and hit tab.

#### LDA Performance

Here are some basic metrics on how the LDA classifier performed on the training data.

```python
print('default-rate: {0: .4f}'.format(np.sum(y_train)/len(y_train)))
print('score:        {0: .4f}'.format(lda.score(X_train, y_train)))
print('error-rate:   {0: .4f}'.format(1-lda.score(X_train, y_train)))
```

    default-rate:  0.0333
    score:         0.9725
    error-rate:    0.0275

Overall, 3.33% of all observations defaulted. If we would simply label each entry as 'non-default' we would have an error rate of this magnitude. So, in comparison to this *naive* classifier, LDA seems to have some skill in predicting the default.

> **IMPORTANT NOTE: In order to be in line with James et al. (2015), the textbook for this course, we have not performed any train/test split of the data. Therefore we will use the same full matrix `X_train` and response vector `y_train` as test data. Performance metrics might be applied to both test and training data but in the end the test results are those that we are ultimately interested in. To drive this point home, I have relabeled the  `X_train` and `y_train` to `X_test`, `y_test`.  Nevertheless, be aware that in this unique case it is the same data!**

Let us print the confusion matrix introduced in the previous chapter to see the class-wise performance. For reference, the confusion matrix is also printed as `DataFrame` but moving forward, be sure to know that row represent the true values and columns predicted labels.

```python
# Relabel variables as discussed
X_test = X_train
y_test = y_train

# Predict labels
y_pred = lda.predict(X_test)

# Sklearn's confusion matrix
print(metrics.confusion_matrix(y_test, y_pred))

# Manual confusion matrix as pandas DataFrame
confm = pd.DataFrame({'Predicted default status': y_pred,
                      'True default status': y_test})
confm.replace(to_replace={0:'No', 1:'Yes'}, inplace=True)
print(confm.groupby(['True default status','Predicted default status']).size().unstack('Predicted default status'))
```

    [[9645   22]
     [ 253   80]]
    Predicted default status    No  Yes
    True default status
    No                        9645   22
    Yes                        253   80

The confusion matrix tells us that for the non-defaulters, LDA only missclassified 22 of them. This is an excellent rate. However, out of the 333 (=253 + 80) people who actually defaulted, LDA classified only 80 correctly. This means our classifier missed out on 76.0% of those who actually defaulted! For a credit card applicant with a bad credit score this is good news. For a credit card company, not so much.

#### Varying the Threshold Levels
Why does LDA miss all these 'defaulters'? Implicitly, Bayes classifier minimizes the **overall** error rate, meaning that it yields the smallest possible total number of misclassified observations - irrespective of the class-specific error rate. Bayes classifier works by assigning an observation to class 'default' for which the posterior probability $Pr(\text{default = Yes}|X=x) > 0.5$. A credit card company who seeks to have as few defaults as possible, this threshold might be too large. Instead, such a company might decide to label any customer with a posterior probability of default above 20% to the 'default' class ($Pr(\text{default = Yes}|X=x) > 0.2$). Let us investigate how the results in such a case would look like.

```python
# Calculated posterior probabilities
posteriors = lda.predict_proba(X_test)
posteriors[:5, :]
```

    array([[ 0.996778  ,  0.003222  ],
           [ 0.99731184,  0.00268816],
           [ 0.98529382,  0.01470618],
           [ 0.99881647,  0.00118353],
           [ 0.99597848,  0.00402152]])

The function `lda.predict_proba()` provides the posterior probabilities of $\Pr(\text{default = 0}|X=x)$ in the first column and $\Pr(\text{default = 1}|X=x)$ in the second. The latter column is what we are interested in. Out of convenience we use `sklearn`'s `binarize` function to classify all probabilities above the threshold of 0.2 as 1 (=default) and generate the confusion matrix.

```python
from sklearn.preprocessing import binarize

# Set threshold and get classes
thresh = 0.2
y_pred020 = binarize([posteriors[:, 1]], thresh)[0]

# new confusion matrix (threshold of 0.2)
print(metrics.confusion_matrix(y_test, y_pred020))
```

    [[9435  232]
     [ 140  193]]

Now LDA misclassifies only 140 out of 333 defaults, or 42.0%. Thats a sharp improvement over the 76.0% from before. But this comes at a price: Before, of those who did not default LDA mislabeled only 22 (or 0.2%) incorrectly. This number increased now to 232 (or 2.4%). Combined, the total error rate increased from 2.75% to 3.72%. For a credit card company, this might be a price they are willing to pay to have a more accurate identification of individuals who default.

Below code snippet calculates and plots the overall error rate, the proportion of missed defaulting customers and the fraction of error among the non-defaulting customers as a function of the threshold value for the posterior probability that is used to assign classes.

```python
# Array of thresholds
thresh = np.linspace(0, 0.5, num=100)

er   = []  # Total error rate
der  = []  # Defaults error rate
nder = []  # Non-Defaults error rate

for t in thresh:
    # Sort/arrange data
    y_pred_class = binarize([posteriors[:, 1]], t)[0]
    confm = metrics.confusion_matrix(y_test, y_pred_class)

    # Calculate error rates
    er   = np.append(er, (confm[0, 1] + confm[1, 0]) / len(posteriors))
    der  = np.append(der, confm[1, 0] / np.sum(confm[1, :]))
    nder = np.append(nder, confm[0, 1] / np.sum(confm[0, :]))
```

```python
# Plot
plt.figure(figsize=(12, 6))
plt.plot(thresh, er, label='Total error rate')
plt.plot(thresh, der, label='Missed defaults')
plt.plot(thresh, nder, label='Missed non-defaults')
plt.xlim(0, 0.5)
plt.xlabel('Threshold')
plt.ylabel('Error Rate')
plt.legend();
```

How do we know what threshold value is best? Unfortunately there's no formula for it. "Such a decision must be based on *domaine knowledge*, such as detailed information about costs associated with defaults" (James et al. (2013, p.147)) and it will always be a **trade-off: if we increase the threshold we reduce the missed non-defaults but at the same time increase the missed defaults**.

## Performance Metrics

This is now the perfect opportunity to refresh our memory on a few classification performance measures introduced in the previous chapters and add a few more to have a full bag of performance metrics. The following table will help in doing this.

We will use the following abbreviations ([Markham (2016)](https://github.com/justmarkham/scikit-learn-videos/blob/master/09_classification_metrics.ipynb)):

* True Positives (TP): correctly predicted defaults
* True Negatives (TN): correctly predicted non-defaults
* False Positives (FP): incorrectly predicted defaults (a "Type I error")
* False Negatives (FN): incorrectly predicted non-defaults (a "Type II error")

```python
# Assign confusion matrix values to variables
confm = metrics.confusion_matrix(y_test, y_pred)
print(confm)
TP = confm[1, 1]  # True positives
TN = confm[0, 0]  # True negatives
FP = confm[0, 1]  # False positives
FN = confm[1, 0]  # False negatives
```

    [[9645   22]
     [ 253   80]]

So far we've encountered the following performance metrics:

* Score
* Error rate
* Sensitivity and
* Specificity.

We briefly recapture their meaning, how they are calculated and how to call them in Scikit-learn. We will make use of the functions in the `metrics` sublibrary of `sklearn`

#### Score
* Score = (TN + TP) / (TN + TP + FP + FN)
* Fraction of (overall) correctly predicted classes

```python
print((TP + TN) / (TP + TN + FP + FN))
print(metrics.accuracy_score(y_test, y_pred))
print(lda.score(X_test, y_test))
```

    0.9725
    0.9725
    0.9725

#### Error rate
* Error rate = 1 - Score or Error rate = (FP + FN) / (TN + TP + FP + FN)
* Fraction of (overall) incorrectly predicted classes
* Also known as "Misclassification Rate"

```python
print((FP + FN) / (TP + TN + FP + FN))
print(1 - metrics.accuracy_score(y_test, y_pred))
print(1 - lda.score(X_test, y_test))
```

    0.0275
    0.0275
    0.0275

#### Specificity

* Specificity = TN / (TN + FP)
* Fraction of correctly predicted negatives (e.g. 'non-defaults')

```python
print(TN / (TN + FP))
```

    0.997724216406

#### Sensitivity or Recall
* Sensitivity = TP / (TP + FN)
* Fraction of correctly predicted 'positives' (e.g. 'defaults'). Basically asks the question: "When the actual value is positive, how often is the prediction correct?"
* Also known as **True positive rate**
* Counterpart to *Precision*

```python
print(TP / (TP + FN))
print(metrics.recall_score(y_test, y_pred))
```

    0.24024024024
    0.24024024024

The above four classification performance metrics we already encountered. There are two more metrics we want to cover: **Precision** and the **F-Score**.

#### Precision
* Precision = TP / (TP + FP)
* Refers to the accuracy of a positive ('default') prediction. Basically asks the question: "When a positive value is predicted, how often is the prediction correct?"
* Counterpart to *Recall*

```python
print(TP / (TP + FP))
print(metrics.precision_score(y_test, y_pred))
```

    0.78431372549
    0.78431372549

#### F-Score

Van Rijsbergen (1979) introduced a measure that is still widely used to evaluate the accuracy of predictions in two-class (binary) classification problems: the F-Score. It combines Precision and Recall (aka Sensitivity) in one metric and tells us something about the relations between data's positive labels and those given by a classifier. It is a single measure of a classification procedure's usefullness and in general the rule is that the higher the F-Score, the better the predictive power of the classification procedure. It is defined as:

\begin{align}
F_{\beta} &= \frac{(1 + \beta^2) \cdot \text{precision} \cdot \text{recall}}{\beta^2 \cdot \text{precision} + \text{recall}} \\[2ex]
          &= \frac{(1+\beta^2) \cdot TP}{(1+\beta^2) \cdot TP + \beta^2 \cdot FN + FP}
\end{align}

This measure employs a parameter $\beta$ that captures a user's preference (Guggenbuehler (2015)). The most common value for $\beta$ is 1. This $F_1$-score weights both precision and recall evenly (simple harmonic mean). In rare cases the $F_2$-score is used, which puts twice as much weights on recall as on precision (Hripcsak and Rotschild (2005)).

```python
print(metrics.confusion_matrix(y_test, y_pred))
print(metrics.f1_score(y_test, y_pred))
print(((1+1**2) * TP)/((1+1**2) * TP + FN + FP))
print(metrics.classification_report(y_test, y_pred))
```

    [[9645   22]
     [ 253   80]]
    0.367816091954
    0.367816091954
                 precision    recall  f1-score   support

              0       0.97      1.00      0.99      9667
              1       0.78      0.24      0.37       333

    avg / total       0.97      0.97      0.97     10000

Let us compare this to the situation where we set the posterior probability threshold for 'default' at 20%.

```python
# Confusion matrix & clf-report for cut-off
# value Pr(default=yes | X = x) > 0.20
print(metrics.confusion_matrix(y_test, y_pred020))
print(metrics.classification_report(y_test, y_pred020))
```

    [[9435  232]
     [ 140  193]]
                 precision    recall  f1-score   support

              0       0.99      0.98      0.98      9667
              1       0.45      0.58      0.51       333

    avg / total       0.97      0.96      0.96     10000

We see that by reducing the cut-off level from $\Pr(\text{default} = 1| X=x) > 0.5$ to $\Pr(\text{default} = 1| X=x) > 0.2$ precision decreases but recall improves. This changes the $F_1$-score.

Does this mean that a threshold of 20% is better? In general, one could argue for a 'yes'. Yet, as mentioned before, this boils down to domain knowledge. Where the $F_1$-score is of help, together with the other metrics introduced, is when we compare models against each other and want to determine which one performed best. For example if we compare results from logistic regression and LDA (and both used a cut-off level of 50%) the F-score would suggest that the one with the higher value performed better.

For the interested reader an excellent and easily accessible summary on performance metrics is the article by [Sokolova and Lapalme (2009)](http://rali.iro.umontreal.ca/rali/sites/default/files/publis/SokolovaLapalme-JIPM09.pdf).

## ROC Curve

Having introduced the major performance measures, let us now discuss the so called ROC curve (short for "receiver operating characteristics"). This is a very popular way of visualizing the performance of binary classifiers. Its origin are in signal detection theory durign WWII (Flaach (2017)) but it has since found application in medical decision making and machine learning ([Fawcett (2006)](http://people.inf.elte.hu/kiss/13dwhdm/roc.pdf)). ROC investigates the relationship between sensitivity and specificity of a binary classifier. Sensitivity (or true positive rate) measures the proportion of positives (defaults) correctly classified. Specificity (or true negative rate) measures the proportion of negatives (non-defaults) correctly classified.

Above we calculated that if we use $\Pr(\text{default = Yes}|X=x) > 0.5$ to classify posterior probabilities as defaults, LDA has its best overall error rate but misses on 76.0% of the customers who acutally defaulted. By decreasing the threshold to 0.2 we improved the accuracy of detecting defaults but this came at the cost of a higher overall error rate. This was the trade-off we faced. The ROC curve serves to visualize a variation of this trade-off. It varies the cut-off threshold from 0 to 1 and calculates for each threshold the true positive rate (aka sensitivity) and false positive rate (equals 1 - specificity). These values are then plotted with the former on the vertical and the later on the horizontal axis.

Though this might feel a bit abstract if one is not familiar with all these technical terms, the interpretation is fortunately fairly simple. The ideal ROC curve will hug the top left corner. In that case, the area under the curve (AUC) is biggest. The bigger the AUC, the better the classifier. A perfect classifier has an AUC of 1.

Here's how we calculate the ROC numbers, the corresponding area under the curve and how we plot it.

```python
# Compute ROC curve and ROC area (AUC) for each class
fpr, tpr, thresholds = metrics.roc_curve(y_test, posteriors[:, 1])
roc_auc = metrics.auc(fpr, tpr)
```

```python
plt.figure(figsize=(6, 6))
plt.plot(fpr, tpr, lw=2, label='ROC curve (area = {0: 0.2f})'.format(roc_auc))
plt.plot([0, 1], [0, 1], lw=2, c = 'k', linestyle='--')
plt.xlim([-0.01, 1.0])
plt.ylim([-0.01, 1.01])
plt.xlabel('False Positive Rate (1 - Specificity)')
plt.ylabel('True Positive Rate (Sensitivity)')
plt.title('Receiver operating characteristic (ROC)', fontweight='bold', fontsize=18)
plt.legend(loc="lower right");
```

An AUC value of 0.95 is close to the maximum of 1 and should be deemed very good. The dashed black line puts this in perspective: it represents the "no information" classifier; this is what we would expect if the probability of default is not associated with 'student' status and 'balance'. Such a classifier, that performs no better than chance, is expected to have an AUC of 0.5.

## Quadratic Discriminant Analysis

### Underlying Assumptions

For LDA we assume that observations within each class are drawn from a multivariate normal distribution with a class-specific mean vector and a common covariance metrix: $X \sim N(\mu_k, \Sigma)$. Quadratic discriminant analysis (QDA) relaxes these assumptions somewhat. The basic assumption is still that the observations follow a multivariate distribution, however, QDA allows for class specific means and covariance matrices: $X \sim N(\mu_k, \Sigma_k)$, where $\Sigma_k$ is a covariance matrix for the $k$th class. With that, the Bayes classifier assigns an observation to the class for which

\begin{equation}
\delta_k(x) = \arg \max_k \; - \frac{1}{2} \ln(|\Sigma_k|) - \frac{1}{2} (x - \mu_k)^T \Sigma_k^{-1} (x - \mu_k) + \ln(\Pr(k))
\end{equation}

is highest. For a derivation of this result see again the appendix of the script. As was the case for LDA, the parameter $\mu_k, \Sigma_k$ and $\Pr(k)$ are again estimated from the training data with the same formulas introduced in this notebook.

Below figure depict both LDA and QDA. Both classifiers were trained on the same data. Due to the different variability of the two classes the QDA algorithm seems to perform slightly better in this case.

Under what circumstances should we prefer QDA over LDA? As always, there's no straight answer to this question. Obviously, performance should be king. However, it is said that LDA tends to be a better bet than QDA if the training set is small. In contrast, if the hold-out set is large or the assumption of a common covariance matrix is clearly incorrect, then QDA is recommended. Beyond that, we have to keep in mind that QDA estimates $K p(p+1)/2$ parameters. So if the number of parameters $p$ is large, QDA might take some time to process (James et al. (2013)).

> **Naive Bayes**

> Naive Bayes is the name for a family of popular ML algorithms that are often used in text mining, which is a field of ML that deals with extracting quantitative information from text. A simple example of it is the analysis of Twitter feeds in order to predict stock market reactions. There exist different variations of Naive Bayes applications. Gaussian Naive Bayes works similar to QDA - with the exception that contrary to QDA the covariance matrices $\Sigma$ are assumed to be diagonal. This means $\Sigma_k$ only contains the variances of the different features for class $k$. Its covariance terms (the off-diagonal elements) are assumed to be zero. Because of its popularity, Naive Bayes is well documented in text books and on the web. A good starting point is Scikit-learn's tutorial on [Naive Bayes](http://scikit-learn.org/stable/modules/naive_bayes.html), Collins (2013) or Russell and Norvig (2009, p.499). To apply the algorithm in Python you want to use `sklearn.naive_bayes.GaussianNB()` or (for text mining preferably) `sklearn.naive_bayes.MultinomialNB()`.

### QDA in Python

The application of QDA follows the one detailed for LDA. Therefore we let the code speak for itself.

```python
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis as QDA

# Run qda on training data
qda = QDA().fit(X_train, y_train)
qda
```

    QuadraticDiscriminantAnalysis(priors=None, reg_param=0.0,
                   store_covariance=False, store_covariances=None, tol=0.0001)

```python
# Predict classes for qda
y_pred_qda = qda.predict(X_test)
posteriors_qda = qda.predict_proba(X_test)[:, 1]

# Print performance metrics
print(metrics.confusion_matrix(y_test, y_pred_qda))
print(qda.score(X_test, y_test))
print(metrics.classification_report(y_test, y_pred_qda))
```

    [[9636   31]
     [ 239   94]]
    0.973
                 precision    recall  f1-score   support

              0       0.98      1.00      0.99      9667
              1       0.75      0.28      0.41       333

    avg / total       0.97      0.97      0.97     10000

The performance seems to be slightly better than with LDA. Let's plot the ROC curve for both LDA and QDA.

```python
# Compute ROC curve and ROC area (AUC) for each class
fpr_qda, tpr_qda, _ = metrics.roc_curve(y_test, posteriors_qda)
roc_auc_qda = metrics.auc(fpr_qda, tpr_qda)
```

```python
plt.figure(figsize=(6, 6))
plt.plot(fpr, tpr, lw=2, label='LDA ROC (AUC = {0: 0.2f})'.format(roc_auc))
plt.plot(fpr_qda, tpr_qda, lw=2, label='QDA ROC (AUC = {0: 0.2f})'.format(roc_auc_qda))
plt.plot([0, 1], [0, 1], lw=2, c = 'k', linestyle='--')
plt.xlim([-0.01, 1.0])
plt.ylim([-0.01, 1.01])
plt.xlabel('False Positive Rate (1 - Specificity)')
plt.ylabel('True Positive Rate (Sensitivity)')
plt.title('ROC Curve', fontweight='bold', fontsize=18)
plt.legend(loc="lower right");
```

With respect to Sensitivity (Recall) and Specificity LDA and QDA perform virtually identical. Therefore, one might give the edge here to QDA because of its slighly better Recall and $F_1$-Score.

## Reality and the Gaussian Assumption for LDA & QDA

Despite the rather strict assumptions regarding normal distribution, LDA and QDA perform well on an amazingly large and diverse set of classification tasks. Friedman et al. (2001, p. 111) put it this way:

> *"Both techniques are widely used, and entire books are devoted to LDA. It seems that whatever exotic tools are the rage of the day, we should always have available these two simple tools. The question arises why LDA and QDA have such a good track record. The reason is not likely to be that the data are approximately Gaussian, and in addition for LDA that the covariances are approximately equal. More likely a reason is that the data can only support simple decision boundaries such as linear or quadratic, and the estimates provided via the Gaussian models are stable. This is a bias variance tradeoff - we can put up with the bias of a linear decision boundary because it can be estimated with much lower variance than more exotic alternatives. This argument is less believable for QDA, since it can have many parameters itself - although perhaps fewer than the non-parametric alternatives."*

Whether LDA or QDA should be applied to categorical/binary features warrants a separate note. It is true that discriminant analysis was designed for continuous features (Ronald A. Fisher (1936)) where the underlying assumption is that the values are normally distributed. However, as above quote shows, studies have proven the robustness of the model even in light of violations of the rather rigid normality assumption. This is not only true for continuous features but also for categorical/binary features. For more details see Huberty et al. (1986). It follows that applying LDA and QDA is possible, though the user should cautiously control the output. We will discuss appropriate cross validation methods to do so in the next chapter.

# Further Ressources

In writing this notebook, many ressources were consulted. For internet ressources the links are provided within the textflow above and will therefore not be listed again. Beyond these links, the following ressources were consulted and are recommended as further reading on the discussed topics:

* Collins, Michael, 2013, The Naive Bayes Model, Maximum-Likelihood Estimation, and the EM Algorithm, Technical report, Columbia University, New York.
* Fawcett, Tom, 2006, An introduction to ROC analysis, *Pattern Recognition Letters* 27, 861–874.
* Fisher, Roland A., 1936, The Use of Multiple Measurements in Taxonomic Problems, *Annals of Human Genetics* 7, 179-188.
* Flach, Peter A., 2017, Roc analysis, in Claude Sammut, and Geoffrey I. Webb, eds., *Encyclopedia of Machine Learning and Data Mining*, 1109–1116 (Springer Science & Business Media, New York, NY).
* Friedman, Jerome, Trevor Hastie, and Robert Tibshirani, 2001, *The Elements of Statistical Learning* (Springer, New York, NY).
* Guggenbuehler, Jan P., 2015, Predicting Net New Money Using Machine Learning Algorithms and Newspaper Articles, Technical report, University of Zurich, Zurich.
* James, Gareth, Daniela Witten, Trevor Hastie, and Robert Tibshirani, 2013, *An Introduction to Statistical Learning: With Applications in R* (Springer Science & Business Media, New York, NY).
* Jobson, J. David, and Bob Korkie, 1980, Estimation for Markowitz Efficient Portfolios, *Journal of the American Statistical Association* 75, 544–554.
* Hripcsak, George, and Adam S Rothschild, 2005, Agreement, the F-measure, and Reliability in Information Retrieval, *Journal of the American Medical Informatics Association* 12, 296–298.
* Huberty, Carl J., Joseph M. Wisenbaker, Jerry D. Smith, and Janet C. Smith, 1986, Using Categorical Variables in Discriminant Analysis, *Multivariate Behavioral Research* 21, 479-496.
* Ledoit, Olivier, and Michael Wolf, 2004, Honey, i shrunk the sample covariance matrix, *The Journal of Portfolio Management* 30, 110–119.
* Müller, Andreas C., and Sarah Guido, 2017, *Introduction to Machine Learning with Python* (O’Reilly Media, Sebastopol, CA).
* Raschka, Sebastian, 2014, Naive Bayes and Text Classification I - Introduction and Theory from website, http://sebastianraschka.com/Articles/2014_naive_bayes_1.html, 08/31/2017
* Russell, Stuart, and Peter Norvig, 2009, *Artificial Intelligence: A Modern Approach* (Prentice Hall Press, Upper Saddle River, NJ).
* Sokolova, Marina, and Guy Lapalme, 2009, A systematic analysis of performance measures for classification tasks, *Information Processing & Management* 45, 427–437.
* Van Rijsbergen, Cornelis Joost, 1979, *Information Retrieval* (Butterworths, London).


```julia
using ForwardDiff
using PyPlot
using LinearAlgebra
```

# Automatic Differentiation
Automatic Differentation is a set of techniques to compute derivatives efficiently dating back to the 1960s. Different from numerical differentation using finite differences, or symbolic differentiation using a library like SymPy or MATLAB, or manually by hand. Automatic differentiation calculates the derivative of a function $f(x)$ at a particular value of $x$, forgoing the analytical closed form solution we might be used to from manual differentiation.

The idea is to break up a function $f(x, y)$ into its simplest operations such as addition, subtraction, division, exponentiation, etc and then use the derivatives of these elementary operations to compute a certain derivative of the function. For example say we have the function,
$$
f(x_1, x_2) = x_2 \sin(x_1) + x_2^2
$$
and we want to know the derivative of this function with respect to $x_1, \frac{\partial f}{\partial x_1}$. We can think of $f(x_1, x_2)$ as a set of nested functions or _primals_, like so:
$$
\begin{align}
f(x_1, x_2) &= f_2(f_1(f_0(x_1))) = f_2(f_1(w_0)) = f_2(w_1) = r\\
f_0(x_1) &= \sin(x_1) = w_0\\
f_1(w_0) &= x_2 w_0 = w_1 \\
f_2(w_1) &= w_1 + x_2^2 = r
\end{align}
$$

Then we can describe the derivative of this function with respect to either of the input variables as:
$$
\begin{align}
% derivative with respect to x_1
\frac{\color{red}{\partial  f}}{\color{blue} {\partial  x_1}} &=
\frac{\color{red}{\partial  f}}{\partial w_1}
\frac{\partial w_1}{\color{blue} {\partial  x_1}} =
\frac{\color{red}{\partial  f}}{\partial w_1}
\frac{\partial w_1}{\partial w_0}
\frac{\partial w_0}{\color{blue} {\partial  x_1}}\\
\end{align}
$$
We can perform this computation in the forward direction starting with $\frac{\partial f}{ {\partial w_1}}$, or in the reverse direction starting with $\frac{\partial w_0}{ {\partial  x_1}} $.

We can represent these primal functions in a _computation graph_ which gives us a view of the operations occuring when computing the function.

## Forward Mode
To start with the forward mode, we again list out the _primal_ functions of $f(x_1, x_2)$:
$$
\begin{align}
w_0 &= sin(x_1) \\
w_1 &= x_2 w_0 \\
w_2 &= x_2^2 \\
r &= w_1 + w_2
\end{align}
$$
While calculating these primal functions we keep a record of the derivatives, or _tangent_ functions of $f(x_1, x_2)$ where $\dot{w_0} = \frac{\partial w_0}{\partial x_1}$. Keep in mind that we can keep track of our "seeds" $\dot{x_1} = \frac{\partial x_1}{\partial x_1} = 1, \dot{x_2} = \frac{\partial x_2}{\partial x_1} = 0$:
$$
\begin{align}
\dot{w_0} &= cos(x_1) \cdot \dot{x_1} = \cos(x_1) \\
\dot{w_1} &= x_2 \dot{w_0} + \dot{x_2} w_0 = x_2 \cdot \cos(x_1)\\
\dot{w_2} &= 2 x_2 \cdot \dot{x_2} = 0\\
r &= \dot{w_1} + \dot{w_2} = x_2 \cdot \cos(x_1)
\end{align}
$$
using the chain rule and product rule where appropriate. This now gives us the value of $\frac{\partial f}{\partial x_1}$ at analytical accuracy.

In order to solve for $\frac{\partial f}{\partial x_2}$, we can use the same primal functions, but this time our seeds are $\dot{x_1} = \frac{\partial x_1}{\partial x_1} = 0, \dot{x_2} = \frac{\partial x_2}{\partial x_1} = 1$, which gives us:
$$
\begin{align}
\dot{w_0} &= cos(x_1) \cdot \dot{x_1} = 0 \\
\dot{w_1} &= x_2 \dot{w_0} + \dot{x_2} w_0 = \sin(x_1)\\
\dot{w_2} &= 2 x_2 \cdot \dot{x_2} = 2 x_2\\
r &= \dot{w_1} + \dot{w_2} = \sin(x_1) + 2x_2
\end{align}
$$

## Reverse Mode
To perform the reverse auto diff, or _adjoint_ derivative, we start with the adjoint derivative $\bar{f}$:
$$
\begin{align}
\bar{f} &= \frac{\partial f}{\partial f} = 1 \\
\end{align}
$$
Our goal is to find the adjoints of our input variables which we found using forward autodiff above:
$$
\begin{align}
\bar{x_1} &= \frac{\partial f}{\partial x_1} \\
\bar{x_2} &= \frac{\partial f}{\partial x_2} \\
\end{align}
$$

This uses the computation graph as a reference to compute the adjoint derivatives.

The reverse mode requires us to calculate the primal functions in a forward pass. Afterwards we perform a reverse pass to calculate the adjoint derivatives of the primals. In other words instead of computing the derivatives simultaneous to the forward pass, we store values and dependencies of intermediate variables:
$$
\begin{align}
w_0 &= sin(x_1) \\
w_1 &= x_2 w_0 \\
w_2 &= x_2^2 \\
r &= w_1 + w_2
\end{align}
$$
In order to calculate $\bar{w_i}$ for a particular primal function, or node in the computation graph, we look at the nodes children:
$$
\bar{w_i} = \frac{\partial f}{\partial w_i} = \sum_{j \textrm{ child of } i} \bar{w_j} \frac{\partial w_j}{\partial w_i}
$$

Starting from the last expression and applying the formula for adjoint derivatives of nodes:
$$
\begin{align}
% w1
\bar{w_1} &= \bar{r}\frac{\partial r}{\partial w_1} &\bar{r}\frac{\partial r}{\partial w_1} &= 1 \cdot \frac{\partial}{\partial w_1}[w_2 + w_1] =  1\\
% w2
\bar{w_2} &= \bar{r}\frac{\partial r}{\partial w_2} &\bar{r}\frac{\partial r}{\partial w_2} &= 1 \cdot \frac{\partial}{\partial w_1}[w_2 + w_1] =  1\\
% w0
\bar{w_0} &= \bar{w_1} \frac{\partial w_1}{\partial w_0} & \bar{w_1}\frac{\partial w_1}{\partial w_0} &= 1 \cdot \frac{\partial}{\partial w_0}[x_2 w_0] = x_2\\
% x1
\bar{x_1} &=  \bar{w_0}\frac{w_0}{\partial x_1} & \bar{w_0}\frac{w_0}{\partial x_1}&= x_2 \cdot\frac{\partial}{\partial x_1}[ \sin{x_1}] = x_2 \cdot \cos(x_1) \\
% x2
\bar{x_2} &= \bar{w_1} \frac{\partial w_1}{\partial x_2} + \bar{w_2} \frac{\partial w_2}{\partial x_2} & &= 1 \cdot\frac{\partial}{\partial x_2}[x_2 w_0] + 1 \cdot\frac{\partial}{\partial x_2}[x_2^2]= \sin(x_1) + 2x_2\\
\end{align}
$$

We can see that by making use of the reverse automatic differentiation and the adjoint derivatives, we actually obtain the same derivatives we computed in the forward pass above. However, using the reverse pass we obtain both derivatives $\frac{\partial f}{\partial x_1}, \frac{\partial f}{\partial x_2}$ in one pass, whereas we need to perform two passes just using the forward pass.

This fact makes reverse mode autodiff better to use when the number of inputs far exceeds the number of outputs, as is typical in machine learning. For general vector valued functions, reverse mode produces one row of the Jacobian at a time, whereas forward mode produces a single column.

There are other types of mixtures between forward and reverse mode autodiff such as forward-on-reverse autodiff. Additionally, some of the algebra can be computed using _dual_ numbers, which is how Julia's `ForwardDiff.jl` is implemented.

## ForwardDiff.jl
There are a couple ways to implement this from scratch, including operator overloading, and source code conversion. However, it's often easiest to use a pre-written library. In Julia there is `ForwardDiff`, Python has `autograd`, and various machine learning engines such as TensorFlow and PyTorch have their own automatic differentiation implementations.

Julia is particularly well suited for autodifferentiation because of multiple dispatch, which overloads math functions and can store their partial derivatives. ForwardDiff is much faster than `autograd` when the input size is smaller than around 10000.

### Derivatives
Say we want to graph the function $g(x)$ and its derivative:

```julia
g(x) = 0.5x^4 + 1.3x^3 -4.0x^2 - 7.5x + 0
∇g(x) = ForwardDiff.derivative(g, x)

xs = LinRange(-4, 3, 100)
# vectorize along the arrays
ys = g.(xs)
ys_prime = ∇g.(xs)
plot(xs, ys, "r", label="g(x)")
plot(xs, ys_prime, "b", label="g'(x)")
legend()
```

### Gradients
Now to get fancier, let's graph a contour map and vector field of the function we worked with earlier with the `ForwardDiff.gradient` function.

```julia
# calculate function at particular x
f(x) = x[2] * sin(x[1]) + x[2]^2
# use autodiff to calculate the derivative
∂f(x) = ForwardDiff.jacobian(f, x)
∇f(x) = ForwardDiff.gradient(f, x)
```

    ∇f (generic function with 1 method)

With this function, we can calculate the gradient at every point in the graph so that we can obtain a vector field. The gradient is a directional vector that describes the change of the function in all directions:
$$
\nabla f(x_1, x_2) = \begin{bmatrix}
\frac{\partial f}{\partial x_1} \\
\frac{\partial f}{\partial x_2}
\end{bmatrix}
$$
As discussed before, in order to solve for this with forward mode autodiff we would need to perform two forward passes.

```julia
# create contour map of the function f(x)
function plot_source_landscape(fun)
    n, up, low = 100, -4, 4
    # calculate outer product to make a meshgrid
    x = ones(n) * LinRange(low, up, n)'
    y = LinRange(low, up, n) * ones(n)'
    # use array comprehension to populate z array
    z = [fun([x[i, j]; y[i, j]]) for i = 1:n, j=1:n]
    contour(x, y, z)
end
plot_source_landscape(f)
```

```julia
# now add on the gradient field
function plot_gradient_landscape(fun, fun_grad)
    n, up, low = 20, -4, 4
    # calculate outer product to make a meshgrid
    x = ones(n) * LinRange(low, up, n)'
    y = LinRange(low, up, n) * ones(n)'
    # use array comprehension to populate z array
    z = [fun([x[i, j]; y[i, j]]) for i = 1:n, j=1:n]
    # get the normals
    u = [fun_grad([x[i, j]; y[i, j]])[1] for i = 1:n, j=1:n]
    v = [fun_grad([x[i, j]; y[i, j]])[2] for i = 1:n, j=1:n]
    # scale u, v
    u
    v

    contour(x, y, z)
    quiver(x, y, u, v)
end
# see the gradients along with the original function
plot_gradient_landscape(f, ∇f)
```

### Jacobians
We can also find jacobians for linearizing vector valued functions. Take the simple example:
$$
g(x) =
\begin{bmatrix}
2 x_1^2 \\
-x_2^3 + x_1
\end{bmatrix}
$$

The Jacobian of this function with respect to $x$ is:

$$
\frac{\partial g}{\partial x} =
\begin{bmatrix}
\frac{\partial g_1}{\partial x_1} & \frac{\partial g_1}{\partial x_2} \\
\frac{\partial g_2}{\partial x_1} & \frac{\partial g_2}{\partial x_2}
\end{bmatrix}
=
\begin{bmatrix}
4 x_1 & 0 \\
1 & -3 x_2^2
\end{bmatrix}
$$

```julia
g(x) = [2*x[1]^2, -x[2]^3 + x[1]]
∂g(x) = ForwardDiff.jacobian(g, x)
```

    ∂g (generic function with 1 method)

```julia
x0 = [0.4, 0.5]
println("g(x): ", g(x0))
println("Jacobian of g(x): ", ∂g(x0))
```

    g(x): [0.32000000000000006, 0.275]
    Jacobian of g(x): [1.6 0.0; 1.0 -0.75]

Which matches what we would predict from analytically deriving the Jacobian.

Now say we had a function representing the dynamics of a pendulum $\dot{x} = f(x, u)$:
$$
x = \begin{bmatrix}
\theta \\
\dot{\theta}
\end{bmatrix}, \quad
\dot{x} = \begin{bmatrix}
\dot{\theta} \\
\ddot{\theta}
\end{bmatrix}
$$
$$
\dot{x} = f(x, u) = \begin{bmatrix}
\dot{\theta} \\
-\frac{g}{l} \sin\theta + \frac{1}{m l^2} u
\end{bmatrix}
$$

where $l$ is the length of the pendulum, $g$ is the acceleration due to gravity, $\theta$ is the angle the swinging mass $m$ makes with the vertical, and $u$ is a control force.

Linearizing these dynamics can be done using the jacobian $A \approx \frac{\partial f}{\partial x}, B = \frac{\partial f}{\partial u}$. For the pendulum, this looks like:
$$ \dot{x} = \begin{bmatrix}
0 & 1 \\
-\frac{g}{l} \cos(\theta) & 0
\end{bmatrix} x +
\begin{bmatrix}
0 \\
\frac{1}{m l^2}\end{bmatrix} u.
$$
Effectively, computing the $B$ matrix is a Jacobian that ignores the other two variables in the dynamics function. In order to use ForwardDiff, we need to rewrite the dynamics to only take in one vector $x$. We can then take the derivative of the entire function with respect to all three variables, and slice it up to get $A, B$. So we can either rewrite the dynamics function as $f_{auto}$,

$$
f_{auto}(x) = \begin{bmatrix}
{x_2} \\
-\frac{g}{l} \sin x_1 + \frac{1}{m l^2} x_3
\end{bmatrix}
$$

```julia
# constants set
grav = 9.81
l = 1
m = 1
fₐ(x) = [x[2], -(grav/l)*sin(x[1]) + (1/m*l^2)x[3]]
```

    fₐ (generic function with 1 method)

```julia
x = [1.0, 1.0, 1.0]
J = ForwardDiff.jacobian(fₐ, x)
A = J[:, 1:2]
B = J[:, 3]
println("A: ", A)
println("B: ", B)
```

    A: [0.0 1.0; -5.300365620566452 0.0]
    B: [0.0, 1.0]

Or we can just leave it in the form $f(x, u)$, but with an anonymous function in the `ForwardDiff.jacobian(f, x)` call.

```julia
# alternatively, just do something like this:
fₚ(x, u) = [x[2], -(grav/l)*sin(x[1]) + (1/m*l^2)u]
# just as before, first 2x2 is A, second 2x1 is B
J = ForwardDiff.jacobian(i -> fₚ([i[1], i[2]], i[3]), x)
```

    2×3 Matrix{Float64}:
      0.0      1.0  0.0
     -5.30037  0.0  1.0

### Aside on Pendulum Dynamics
When producing the phase portrait of the pendulum's dynamics, we want to plot $\theta$ vs $\dot{\theta}$ and observe the behavior. We notice that there are clearly some sources and sinks in the dynamics. We can use the functions `quiver` and `streamplot` to plot the phase space. Since both methods of plotting ask for velocities $u, v$ of the input variables, we can calculate these as,

$$
u = \dot{\theta} = x_2 \qquad v = \ddot{\theta} = -\frac{g}{l} \sin x_1
$$

which is just the system dynamics. When viewing streamlines, it's important to note that the lines are representing the phase path a particle will take given any initial starting position. A stream line joins together the velocity field which makes it look exactly like the original path. This makes it simple to identify equilibrium points.

```julia
# now add on the gradient field
# pendulum dynamics
p(x) = [x[2], -(grav/l)*sin(x[1])]

# produce the x, y space to perform dynamics on
function plot_pendulum_landscape()
    n, up, low = 20, -10, 10
    # calculate outer product to make a meshgrid
    θ = ones(n) * LinRange(low, up, n)'
    ω = LinRange(low, up, n) * ones(n)'
    # use array comprehension to populate z array
    z = [p([θ[i, j]; ω[i, j]]) for i = 1:n, j=1:n]
    # get the normals
    u = [p([θ[i, j]; ω[i, j]])[1] for i = 1:n, j=1:n]
    v = [p([θ[i, j]; ω[i, j]])[2] for i = 1:n, j=1:n]
    return θ, ω, u, v
end
pθ, pω, pu, pv = plot_pendulum_landscape();
```

```julia
# plot the vector field of the phase portrait using the derivative of θ, ω
using LaTeXStrings
quiver(pθ/π, pω, pu, pv)
xlabel(L"\theta  \quad (\pi)");
ylabel(L"\dot{\theta}");
```

```julia
# we can view the phase portrait with stream lines, which looks like the original paths in the phase plot
streamplot(pθ/π, pω, pu, pv)
xlabel(L"\theta  \quad (\pi)");
ylabel(L"\dot{\theta} ");
```

## References
1. [Ari Seff. _What is Automatic Differentiation?_](https://www.youtube.com/watch?v=wG_nF1awSSY)
2. [ArchQuant. _Automatic Differentiation Explained with Example_](https://www.youtube.com/watch?v=jS-0aAamC64&t=579s)