# Hopfield Network

## Introduction

### Structure

### Updating
Here we update the network asynchronously by randomly choosing one node and update its ouput.
For a network with $N$ nodes:
$$s_{i}(t+1)=sgn(\sum_{j=1}^{N}w_{ij}s_{j}(t)-\theta_{i})$$
$$sgn(x)=1 (x\geq0)$$
$$sgn(x)=-1(otherwise)$$
$$W=W^{T}$$
$$W_{ii}>=0$$
Where $w_{ij}$ indicates the entry at $i_{th}$ row and $j_{th}$ column of $W$.
asynchronous
### Energy
#### Definition
$$E=-\frac{1}{2}\sum_{i=1}^{N}\sum_{j=1}^{N}w_{ij}s_{i}s_{j}+\sum_{i=1}^{N}\theta_{i}s_{i}$$
or in matrix form:
$$E=-\frac{1}{2}s^{T}Ws+\theta^{T}s$$
where:
$$s=\begin{bmatrix}s_{1}&s_{2}&\cdots&s_{N}\end{bmatrix}^{T}$$
$$\theta=\begin{bmatrix}\theta_{1}&\theta_{2}&\cdots&\theta_{N}\end{bmatrix}^{T}$$
#### Proof
* E is bounded
$$\lvert E\rvert\leq\frac{1}{2}\sum_{i=1}^{N}\sum_{j=1}^{N}\lvert w_{ij}\rvert \lvert s_{i}\rvert\lvert s_{j}\rvert+\sum_{i=1}^{N}\lvert\theta_{i}\rvert\lvert s_{i}\rvert$$
* E is monotonically decreasing

Assume that the $k_{th}$ node has been updated:
$$s(t+1)=s(t)+\Delta s(t)$$
$$\Delta s(t)=\epsilon_{k}\Delta s_{k}(t)$$
\begin{equation}
\begin{split}
\Delta E&=E(t+1)-E(t)\\
&=-\frac{1}{2}\left(\left(s(t)+\Delta s(t)\right)^{T}W\left(s(t)+\Delta s(t)\right)\right)+\theta^{T}\left(s(t)+\Delta s(t)\right)\\
&-\left(-\frac{1}{2}s(t)^{T}Ws(t)+\theta^{T}s(t)\right)\\
&=-\frac{1}{2}\left(\Delta s(t)^{T}Ws(t)+ s(t)^{T}W\Delta s(t)+\Delta s(t)^{T}W\Delta s(t)\right)+\theta^{T}\Delta s(t)\\
&=-\frac{1}{2}\left[\Delta s_{k}(t)\sum_{j=1}^{N}w_{kj}s_{j}(t)+\Delta s_{k}(t)\sum_{i=1}^{N}w_{ik}s_{i}(t)+w_{kk}\Delta s_{k}^{2}(t)\right]+\Delta s_{k}(t)\theta_{k}
\end{split}
\end{equation}
where $\epsilon_{k}$ denotes the vector with $k_{th}$ entry equal to 1.
Given that $w_{ij}=w_{ji}$,
$$\Delta E=-\Delta s_{k}(t)\left(\sum_{j=1}^{N}w_{kj}s_{j}(t)-\theta_{k}\right)-\frac{1}{2}w_{kk}\Delta s_{k}^{2}(t)
$$
Note that$\Delta s_{k}(t)\left(\sum_{j=1}^{N}w_{kj}s_{j}(t)-\theta_{k}\right)\geq0$ and that $w_{kk}\geq0$, we obtain:
$$\Delta E\leq-\frac{1}{2}w_{kk}\Delta s_{k}^{2}(t)\leq 0$$
Hence, E is monotonically decreasing.

Thus, we have the conclusion that the energy of the network will converge to a local minimum after several iterations.

### A numerical example for convergence

```python
import numpy as np
import random
```

```python
def updateState(state,W,theta,verbose=False,reshape=None):
    networkSize=np.shape(state)[0]
    if np.shape(state)!=np.shape(theta):
        raise(ValueError("The shapes of states and thresholds didn't match."))
    if networkSize!=np.shape(W)[0]:
        raise(ValueError("The shapes of states and W didn't match."))
    if np.shape(W)[0]!=np.shape(W)[1]:
        raise(ValueError("W must be a square matrix."))
    lastState=None
    iteration=0
    n=0
    k=random.randint(0,networkSize-1)
    if verbose:
        if reshape==None:
            print("Original state:\n{}".format(state))
        else:
            print("Original state:\n{}".format(np.reshape(state,reshape)))
    while n<networkSize:
        iteration+=1
        #lastState=np.array(state,copy=True)
        lastState=np.copy(state)
        if np.sum(W[k,:]*state)-theta[k]>0:
            state[k]=1
        elif np.sum(W[k,:]*state)-theta[k]<0:
            state[k]=-1
        else:
            state[k]=0
        if type(lastState)==type(None) or np.any(lastState!=state):
            n=0
        else:
            n+=1
        if verbose:
            print("Node number {} has been updated at epoch {}.".format(k,iteration))
            if reshape==None:
                print("New state:\n{}".format(state))
            else:
                print("New state:\n{}".format(np.reshape(state,reshape)))
        if k<networkSize-1:
            k+=1
        else:
            k=0
    return state
```

```python
initState=np.array([1,-1,-1,-1])
W=np.array([[0,2,2,2],[2,0,2,2],[2,2,0,2],[2,2,2,0]])
theta=np.array([1,2,3,4])
updateState(initState,W,theta,verbose=True)
```

    Original state:
    [ 1 -1 -1 -1]
    Node number 0 has been updated at epoch 1.
    New state:
    [-1 -1 -1 -1]
    Node number 1 has been updated at epoch 2.
    New state:
    [-1 -1 -1 -1]
    Node number 2 has been updated at epoch 3.
    New state:
    [-1 -1 -1 -1]
    Node number 3 has been updated at epoch 4.
    New state:
    [-1 -1 -1 -1]
    Node number 0 has been updated at epoch 5.
    New state:
    [-1 -1 -1 -1]

    array([-1, -1, -1, -1])

```python
initState=np.array([1,1,1,-1])
updateState(initState,W,theta,verbose=True,reshape=None)
```

    Original state:
    [ 1  1  1 -1]
    Node number 2 has been updated at epoch 1.
    New state:
    [ 1  1 -1 -1]
    Node number 3 has been updated at epoch 2.
    New state:
    [ 1  1 -1 -1]
    Node number 0 has been updated at epoch 3.
    New state:
    [-1  1 -1 -1]
    Node number 1 has been updated at epoch 4.
    New state:
    [-1 -1 -1 -1]
    Node number 2 has been updated at epoch 5.
    New state:
    [-1 -1 -1 -1]
    Node number 3 has been updated at epoch 6.
    New state:
    [-1 -1 -1 -1]
    Node number 0 has been updated at epoch 7.
    New state:
    [-1 -1 -1 -1]
    Node number 1 has been updated at epoch 8.
    New state:
    [-1 -1 -1 -1]

    array([-1, -1, -1, -1])

### Design $W$
Design $W$ to store $M$ samples in $X=\begin{bmatrix}X_{1}&X_{2}&\cdots&X_{M}\end{bmatrix}$
#### Hebb's learning rule
$$w_{ij}=\alpha\sum_{k=1}^{M}x_{i}^{k}x_{j}^{k},(i\neq j)$$
$$w_{ii}=0$$
or in matrix form
$$W=\alpha\sum_{k=1}^{M}\left(X_{k}X_{k}^{T}-I\right)$$
Assume we need to store two samples: H and V.

```python
cross=[
-1,-1,+1,-1,-1,
-1,-1,+1,-1,-1,
+1,+1,+1,+1,+1,
-1,-1,+1,-1,-1,
-1,-1,+1,-1,-1]#Cross
diamond=[
-1,-1,+1,-1,-1,
-1,+1,-1,+1,-1,
+1,-1,-1,-1,+1,
-1,+1,-1,+1,-1,
-1,-1,+1,-1,-1]#Diamond
X=np.array([cross,diamond]).T
alpha=0.5
dim,size=np.shape(X)
W=np.zeros([dim,dim],dtype=np.float)

for i in range(size):
    x=np.reshape(X[:,i],[dim,1])
    W=W+x*x.T
W=W-size*np.eye(dim)
W=alpha*W
initState=-np.ones(dim)
theta=np.ones(dim)
updateState(initState,W,theta,verbose=True,reshape=[5,5])
```

    Original state:
    [[-1. -1. -1. -1. -1.]
     [-1. -1. -1. -1. -1.]
     [-1. -1. -1. -1. -1.]
     [-1. -1. -1. -1. -1.]
     [-1. -1. -1. -1. -1.]]
    Node number 6 has been updated at epoch 1.
    New state:
    [[-1. -1. -1. -1. -1.]
     [-1.  1. -1. -1. -1.]
     [-1. -1. -1. -1. -1.]
     [-1. -1. -1. -1. -1.]
     [-1. -1. -1. -1. -1.]]
    Node number 7 has been updated at epoch 2.
    New state:
    [[-1. -1. -1. -1. -1.]
     [-1.  1. -1. -1. -1.]
     [-1. -1. -1. -1. -1.]
     [-1. -1. -1. -1. -1.]
     [-1. -1. -1. -1. -1.]]
    Node number 8 has been updated at epoch 3.
    New state:
    [[-1. -1. -1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1. -1. -1. -1.]
     [-1. -1. -1. -1. -1.]
     [-1. -1. -1. -1. -1.]]
    Node number 9 has been updated at epoch 4.
    New state:
    [[-1. -1. -1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1. -1. -1. -1.]
     [-1. -1. -1. -1. -1.]
     [-1. -1. -1. -1. -1.]]
    Node number 10 has been updated at epoch 5.
    New state:
    [[-1. -1. -1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1. -1.]
     [-1. -1. -1. -1. -1.]
     [-1. -1. -1. -1. -1.]]
    Node number 11 has been updated at epoch 6.
    New state:
    [[-1. -1. -1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1. -1.]
     [-1. -1. -1. -1. -1.]
     [-1. -1. -1. -1. -1.]]
    Node number 12 has been updated at epoch 7.
    New state:
    [[-1. -1. -1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1. -1.]
     [-1. -1. -1. -1. -1.]
     [-1. -1. -1. -1. -1.]]
    Node number 13 has been updated at epoch 8.
    New state:
    [[-1. -1. -1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1. -1.]
     [-1. -1. -1. -1. -1.]
     [-1. -1. -1. -1. -1.]]
    Node number 14 has been updated at epoch 9.
    New state:
    [[-1. -1. -1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1. -1. -1. -1. -1.]
     [-1. -1. -1. -1. -1.]]
    Node number 15 has been updated at epoch 10.
    New state:
    [[-1. -1. -1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1. -1. -1. -1. -1.]
     [-1. -1. -1. -1. -1.]]
    Node number 16 has been updated at epoch 11.
    New state:
    [[-1. -1. -1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1. -1. -1.]
     [-1. -1. -1. -1. -1.]]
    Node number 17 has been updated at epoch 12.
    New state:
    [[-1. -1. -1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1. -1. -1.]
     [-1. -1. -1. -1. -1.]]
    Node number 18 has been updated at epoch 13.
    New state:
    [[-1. -1. -1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1. -1. -1. -1.]]
    Node number 19 has been updated at epoch 14.
    New state:
    [[-1. -1. -1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1. -1. -1. -1.]]
    Node number 20 has been updated at epoch 15.
    New state:
    [[-1. -1. -1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1. -1. -1. -1.]]
    Node number 21 has been updated at epoch 16.
    New state:
    [[-1. -1. -1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1. -1. -1. -1.]]
    Node number 22 has been updated at epoch 17.
    New state:
    [[-1. -1. -1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1.  1. -1. -1.]]
    Node number 23 has been updated at epoch 18.
    New state:
    [[-1. -1. -1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1.  1. -1. -1.]]
    Node number 24 has been updated at epoch 19.
    New state:
    [[-1. -1. -1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1.  1. -1. -1.]]
    Node number 0 has been updated at epoch 20.
    New state:
    [[-1. -1. -1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1.  1. -1. -1.]]
    Node number 1 has been updated at epoch 21.
    New state:
    [[-1. -1. -1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1.  1. -1. -1.]]
    Node number 2 has been updated at epoch 22.
    New state:
    [[-1. -1.  1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1.  1. -1. -1.]]
    Node number 3 has been updated at epoch 23.
    New state:
    [[-1. -1.  1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1.  1. -1. -1.]]
    Node number 4 has been updated at epoch 24.
    New state:
    [[-1. -1.  1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1.  1. -1. -1.]]
    Node number 5 has been updated at epoch 25.
    New state:
    [[-1. -1.  1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1.  1. -1. -1.]]
    Node number 6 has been updated at epoch 26.
    New state:
    [[-1. -1.  1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1.  1. -1. -1.]]
    Node number 7 has been updated at epoch 27.
    New state:
    [[-1. -1.  1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1.  1. -1. -1.]]
    Node number 8 has been updated at epoch 28.
    New state:
    [[-1. -1.  1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1.  1. -1. -1.]]
    Node number 9 has been updated at epoch 29.
    New state:
    [[-1. -1.  1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1.  1. -1. -1.]]
    Node number 10 has been updated at epoch 30.
    New state:
    [[-1. -1.  1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1.  1. -1. -1.]]
    Node number 11 has been updated at epoch 31.
    New state:
    [[-1. -1.  1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1.  1. -1. -1.]]
    Node number 12 has been updated at epoch 32.
    New state:
    [[-1. -1.  1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1.  1. -1. -1.]]
    Node number 13 has been updated at epoch 33.
    New state:
    [[-1. -1.  1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1.  1. -1. -1.]]
    Node number 14 has been updated at epoch 34.
    New state:
    [[-1. -1.  1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1.  1. -1. -1.]]
    Node number 15 has been updated at epoch 35.
    New state:
    [[-1. -1.  1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1.  1. -1. -1.]]
    Node number 16 has been updated at epoch 36.
    New state:
    [[-1. -1.  1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1.  1. -1. -1.]]
    Node number 17 has been updated at epoch 37.
    New state:
    [[-1. -1.  1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1.  1. -1. -1.]]
    Node number 18 has been updated at epoch 38.
    New state:
    [[-1. -1.  1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1.  1. -1. -1.]]
    Node number 19 has been updated at epoch 39.
    New state:
    [[-1. -1.  1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1.  1. -1. -1.]]
    Node number 20 has been updated at epoch 40.
    New state:
    [[-1. -1.  1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1.  1. -1. -1.]]
    Node number 21 has been updated at epoch 41.
    New state:
    [[-1. -1.  1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1.  1. -1. -1.]]
    Node number 22 has been updated at epoch 42.
    New state:
    [[-1. -1.  1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1.  1. -1. -1.]]
    Node number 23 has been updated at epoch 43.
    New state:
    [[-1. -1.  1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1.  1. -1. -1.]]
    Node number 24 has been updated at epoch 44.
    New state:
    [[-1. -1.  1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1.  1. -1. -1.]]
    Node number 0 has been updated at epoch 45.
    New state:
    [[-1. -1.  1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1.  1. -1. -1.]]
    Node number 1 has been updated at epoch 46.
    New state:
    [[-1. -1.  1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1.  1. -1. -1.]]
    Node number 2 has been updated at epoch 47.
    New state:
    [[-1. -1.  1. -1. -1.]
     [-1.  1. -1.  1. -1.]
     [ 1. -1. -1. -1.  1.]
     [-1.  1. -1.  1. -1.]
     [-1. -1.  1. -1. -1.]]

    array([-1., -1.,  1., -1., -1., -1.,  1., -1.,  1., -1.,  1., -1., -1.,
           -1.,  1., -1.,  1., -1.,  1., -1., -1., -1.,  1., -1., -1.])

It's interesting that if we use a new initial state we will get a the inversion of the diamond in the training sample.

```python
initState=np.ones(dim)
theta=np.ones(dim)
finalState=updateState(initState,W,theta,reshape=[5,5])
print("Final state:\n{}".format(np.reshape(finalState,[5,5])))
```

    Final state:
    [[ 1.  1. -1.  1.  1.]
     [ 1. -1.  1. -1.  1.]
     [-1.  1.  1.  1. -1.]
     [ 1. -1.  1. -1.  1.]
     [ 1.  1. -1.  1.  1.]]

#### Pseudo Inverse
To be continued...

### Hopfield network and TSP
The [travelling salesman problem (TSP)](https://en.wikipedia.org/wiki/Travelling_salesman_problem) asks the following question: "Given a list of cities and the distances between each pair of cities, what is the shortest possible route that visits each city exactly once and returns to the origin city?" It is an NP-hard problem in combinatorial optimization, important in operations research and theoretical computer science.

TSP is a special case of the travelling purchaser problem and the vehicle routing problem.
#### Design energy function
Suppose we have $N$ cities in total. We represent the route of the salesman with a matrix $V_{N\times N}$. $v_{ij}\in \{0,1\}$ where $v_{ij}$ denotes the element in the $i_{th}$ row and the $j_{th}$ column of $V$. $v_{ij}=1$ indicates that the $i_{th}$ city is reached at step $j$. We use matrix $D$ to denote the distances among cities where the element in the $i_{th}$ row and $j_{th}$ column $d_{ij}$ denotes the cost to go from city $i$ to city $j$ (e.g. distance between two cities).

Now we define an energy function as follow:
$$E=\frac{A}{2}E_{1}+\frac{B}{2}E_{2}+\frac{C}{2}E_{3}+\frac{D}{2}E_{4}$$
where,
$$E_{1}=\sum_{k=1}^{N}\sum_{i,j,i\neq j}v_{ki}v_{kj}$$
$$E_{2}=\sum_{k=1}^{N}\sum_{i,j,i\neq j}v_{ik}v_{jk}$$
$$E_{3}=\left(N-\sum_{i=1}^{N}\sum_{j=1}^{N}v_{ij}\right)^{2}$$
$$E_{4}=\sum_{k=1}^{N}\sum_{i=1}^{N}\sum_{j=1}^{N}d_{ij}v_{ik}(v_{j,(k-1)}+v_{j,(k+1)})$$

$E_{1}$ and $E_{2}$ penalize the possible violations over the constraints that each row/column of the matrix $V$ contains at most one 1, since each city should be vistied once and the salesman can visit one city at a time.

$E_{3}$ indicates the constrain that a total of $N$ cities must be visited.

$E_{4}$ represents the cost we want to minimize.

```python

```


# Mathematics for Machine Learning: Linear Algebra
## Week2
## Module 2:

### Modulus & inner product (vectors)

#### usefull pakages

```python
from sympy import solve, Poly, Eq, Function, exp, sqrt, simplify,acos

from sympy.abc import x, y, z, a, b, c, d , e, f
from sympy.vector import CoordSys3D

import numpy as np
```

### lenght of the vector:

```python
C = CoordSys3D('C') # cartesian coordinate
r = x*C.i + y*C.j + z*C.k
sqrt(r.dot(r))
```

$\displaystyle \sqrt{x^{2} + y^{2} + z^{2}}$

### dot product :

```python
s = a*C.i + b*C.j + c*C.k
r.dot(s)
```

$\displaystyle a x + b y + c z$

#### commutative so s.r = r.s

```python
assert r.dot(s) == s.dot(r)
```

#### distributive: r.(s+t) == r.s + r.t

```python
t = d*C.i + e*C.j + f*C.k
```

```python
r.dot(s+t)
```

$\displaystyle x \left(a + d\right) + y \left(b + e\right) + z \left(c + f\right)$

```python
r.dot(s) + r.dot(t)
```

$\displaystyle a x + b y + c z + d x + e y + f z$

```python
assert r.dot(s+t).equals(r.dot(s) + r.dot(t))
```

```python
(r-s).dot(r-s)
```

$\displaystyle \left(- a + x\right)^{2} + \left(- b + y\right)^{2} + \left(- c + z\right)^{2}$

```python
from sympy.physics.vector import *
N = ReferenceFrame('N')

```

```python
(a*N.x+b*N.y).magnitude()
```

$\displaystyle \sqrt{a^{2} + b^{2}}$

```python
s=(a*N.x+b*N.y)
r=(x*N.x + y*N.y)
```

```python
(r.dot(s))
```

$\displaystyle a x + b y$

```python
r.magnitude()*s.magnitude()
```

$\displaystyle \sqrt{a^{2} + b^{2}} \sqrt{x^{2} + y^{2}}$

```python
Tetha = acos((r.dot(s))/(r.magnitude()*s.magnitude()))
```

```python
Tetha
```

$\displaystyle \operatorname{acos}{\left(\frac{a x + b y}{\sqrt{a^{2} + b^{2}} \sqrt{x^{2} + y^{2}}} \right)}$

```python
(r.dot(r))
```

$\displaystyle x^{2} + y^{2}$

```python
(r.dot(-r))
```

$\displaystyle - x^{2} - y^{2}$

```python
 rn = (y*N.x + -x*N.y)  #normal
```

```python
(r.dot(rn))
```

$\displaystyle 0$

### Projection

```python
i, j, k = C.base_vectors()
v1 = i + j + k
v2 = 3*i + 4*j
v1.projection(v2)
```

$\displaystyle (\frac{7}{3})\mathbf{\hat{i}_{C}} + (\frac{7}{3})\mathbf{\hat{j}_{C}} + (\frac{7}{3})\mathbf{\hat{k}_{C}}$

```python

```

```python
v2.projection(v1)
```

$\displaystyle (\frac{21}{25})\mathbf{\hat{i}_{C}} + (\frac{28}{25})\mathbf{\hat{j}_{C}}$

#### Scalar Projection

```python
v2.projection(v1,scalar=True) == v1.dot(v2)/(v2.magnitude()**2)
```

    True

```python
v2.projection(v1,scalar=True)
```

$\displaystyle \frac{7}{25}$

```python
v1.projection(v2,scalar=True) == v2.dot(v1)/(v1.magnitude()**2)
```

    True

```python
v1.projection(v2,scalar=True)
```

$\displaystyle \frac{7}{3}$

## Changing basis

1)

```python
v = np.array([1,1,2,3])
```

```python
b1 = np.array([1,0,0,0])
b2 = np.array([0,2,-1,0])
b3 = np.array([0,1,2,0])
b4 = np.array([0,0,0,0])
```

```python
v.dot(b1)/(b1.dot(b1))

```

    1.0

```python
v.dot(b2)/(b2.dot(b2))
```

    0.0

```python
v.dot(b3)/(b3.dot(b3))
```

    1.0

```python
v.dot(b3)/(b3.dot(b3))
```

    1.0

## Vector operations assessment

```python
v1 = 2*i + j
c = 3*i -4 *j
#c2 = i-j
#assert c.dot(c2) == 0
assert v1.dot(c)/(c.dot(c)) == c.projection(v1,scalar=True)
c.projection(v1,scalar=True)
```

$\displaystyle \frac{2}{25}$

```python
c2.projection(v1)
```

```python
v = -4*i-3*j+8*k
b1 = i+2*j+3*k
b2 = -2*i+1*j
b3 = -3*i-6*j+5*k
```

```python
b1.projection(v,scalar=True)
```

$\displaystyle 1$

```python
b2.projection(v,scalar=True)
```

$\displaystyle 1$

```python
b3.projection(v,scalar=True)
```

$\displaystyle 1$

```python
x = np.array([3, 2, 4])
v = np.array([-1, 2, -3])
t = 2
```

```python
x_n = x + t * v
```

```python
x_n
```

    array([ 1,  6, -2])

```python

```


## bayespropestimation basis

The bayespropestimation module uses simple Bayesian probability to derive the posterior distributions of two samples. The parameters of these samples can then be approximated by simulation, as can the difference in the parameters.

#### Sections

- Deriving the posterior distribution
- Comparing samples
- Choice of priors
- Estimating the Bayes Factor

#### Deriving the posterior distribution

Let $Y_k$ be the count of successes from $n_k$ trials from population $k$.  $Y_k$ can be defined as:

$$Y_k \sim \textrm{Bi}(n_k, \theta_k)$$

Where $\theta_k$ is the parameter we wish to estimate.  $\theta_k$ can be defined as:

$$\theta_k \sim \textrm{Beta}(\alpha, \beta)$$

Where $\alpha$ and $\beta$ denote fixed parameters of the beta distribution.  Using Bayes theorem $\Big( p(B|A) = \frac{p(A|B)p(B)}{p(A)}\Big)$ it can be seen that:

\begin{equation}
\begin{aligned}
p(\theta_k | Y_k) &= \frac{L(Y_k | \theta_k)p(\theta_k)}{p(Y_k)}
\\
p(\theta_k | Y_k) &\propto L(Y_k | \theta_k)p(\theta_k)
\\
&\propto \Big( {n_k \choose y_k} \theta_k^{y_k} (1 - \theta_k)^{(n_k - y_k)} \Big) \Big(\frac{\theta^{(\alpha -1)}(1 - \theta_k)^{(\beta - 1)}}{\mathbb{B}(\alpha, \beta)}\Big)
\\
&\propto \theta^{(y_k + \alpha -1)}_k (1 - \theta_k)^{(n_k - y_k + \beta - 1)}
\\
&\propto \frac{\theta^{(y_k + \alpha -1)}(1 - \theta_k)^{(n_k - y_k + \beta  - 1)}}{\mathbb{B}(y_k + \alpha, n_k - y_k + \beta)}
\end{aligned}
\end{equation}

Which is identical to the beta distribution with parameters $y_k + \alpha$ and $n_k - y_k + \beta$.  Therefore:

$$ \theta_k | Y_k \sim \textrm{Beta}(y_k + \alpha, n_k - y_k + \beta)$$

#### Comparing samples

For two samples from populations $A$ and $B$, the posterior distributions can be defined as:

$$ \theta_A | Y_A \sim \textrm{Beta}(y_A + \alpha, n_A - y_A + \beta)$$

$$ \theta_B | Y_B \sim \textrm{Beta}(y_B + \alpha, n_B - y_B + \beta)$$

Credible intervals, MAP and expectations of $\theta_A | Y_A$ and $\theta_B | Y_B$ can be derived directly from the posterior density, or they can be approximated by random draws from the posterior density.

Let $\Delta_{AB} = \theta_B$ - $\theta_A$, i.e. the difference between the parameter estimates for sample $A$ and sample $B$.  $\Delta_{AB}$ can be defined as:

$$\theta_B - \theta_A = \Delta_{AB}  \propto \textrm{Beta}(y_A + \alpha, n_A - y_A + \beta) - \textrm{Beta}(y_B + \alpha, n_B - y_B + \beta)$$

This distribution is proper if a normalising constant is added.  However $\Delta_{AB}$ can be easily approximated by taking random draws from $\theta_B | Y_B$ and $\theta_A | Y_A$ and computing the difference.

#### Choice of priors

The model assumes that $\theta_B | Y_B$ and $\theta_A | Y_A$ have the same prior distribution.  Unless specified, an uninformative Jeffrey's prior is used, i.e

$$\theta_k \sim \textrm{Beta}(0.5, 0.5)$$

If an informative prior is available the user can define their own Beta prior.

#### Estimating the Bayes Factor

The Bayes Factor for comparing directional hypotheses, e.g. $\theta_B > \theta_A$ vs $\theta_B < \theta_A$, is easy to estimate from the draws from the posterior distribution of $\Delta_{AB}$.

The bayes factor ($K$) can be defined as $\frac{P(D | H_1)}{P(D | H_2)}$ where $D$ denotes the data and $H_1$ and $H_2$ denote the hypotheses $\theta_B > \theta_A$ and $\theta_B < \theta_A$ respectively.  $K$ can therefore be defined as:

$$ K = \frac{P(y_A, y_B | \theta_B > \theta_A)}{P(y_A, y_B | \theta_B < \theta_A)} $$

Given that the marginal likelihood of $P(y_k | \theta_k, \alpha, \beta)$ is defined as $\int p(y_k | \theta_k)p(\theta_k) \, d\theta_k$. The $P(y_A, y_B | \theta_B > \theta_A)$ is:

$$ P(y_A, y_B | \theta_B > \theta_A) = \int_{\theta_B > \theta_A}  L(y_A | \theta_A) p(\theta_A) L(y_B | \theta_B) p(\theta_B) \, d\theta_A d\theta_B$$

The contents of the integral are proportional to $\Delta_{AB}$, therefore:

$$ P(y_A, y_B | \theta_B > \theta_A) \propto P(\theta_B - \theta_A > 0 | y_A, y_B)$$

Where $P(\theta_A, \theta_B | y_A, y_B) \propto \Delta_{AB}$.  Consequently:

$$ K = \frac{P(y_A, y_B | \theta_B > \theta_A)}{P(y_A, y_B | \theta_B < \theta_A)} = \frac{P(\theta_B - \theta_A > 0 | y_A, y_B)}{ P(\theta_B - \theta_A < 0 | y_A, y_B)}$$

These two numerator and denominator can be easily approximated using can therefore be approximated using random draws from $\Delta_{AB}$.  Interpretation of Bayes factors follows the [Jeffreys scale](https://en.wikipedia.org/w/index.php?title=Bayes_factor#Interpretation).

```python

```


# Slope Deflection Example - Frame 1
Illustrates the solution of simple, statically-indeterminate framed structures
using the method of **slope-deflection** and symbolic algebra.
This version uses an improved version of 'sdutil' that computes end shears as well as moments.

```python
from sympy import *                 # use symbolic algebra
init_printing(use_latex='mathjax')  # print nice math (use_latex arg is workaround for FormatterWarning messages)
from IPython import display
```

```python
from sdutil2 import SD, FEF
```

Analyze the following frame using the method of **slope-deflection**.

```python
display.SVG('frame1.svg')
```

This frame has 5 independant joint displacements - rotations at the four
joints *a*,
*b*,
*c*,
and
*d*,
and a lateral displacement of joints
*b*
and
*c*
together.  Actually, we know that joint *d* does not rotate, but we
will pretend it does for now, and will handle that as a displacement constraint
later.

So now we write the slope deflection equations.  These express the end moments
as functions of the five unknown displacements.  Note also that we do not treat the
pin-ended member *ab* specially -- it is not necessary to do so.

We use a very simple utility function *SD()* to compute the end moments and shears of a segment,
as a function of the end displacements:

```python
help(SD)
```

    Help on function SD in module sdutil2:

    SD(L, EI, t0, t1, d01=0)
        Return the end moments and shears of a segment as computed by the slope
        deflection equations.
           L = length,
           EI = stiffness,
           t0 = rotation at end 0 (left),
           t1 = rotation at end 1,
           d01 = relative lateral displacment end 0 to end 1.
        SD sign convention: moments and rotation are +ive CW,
        relative displacement d01 is +ive for CW rotation of chord.

```python
# Slope deflection equations:
# theta_a,b,c,d - rotations of joints
# Delta_ab - horizontal displacement of joint b (and c) relative to a (and d)
var('EI theta_a theta_b theta_c theta_d Delta_ab')
Mab,Mba,Vab,Vba = SD(3,EI,theta_a,theta_b,Delta_ab)
Mbc,Mcb,Vbc,Vcb = SD(6,2*EI,theta_b,theta_c,0) + FEF.udl(6,36)
Mcd,Mdc,Vcd,Vdc = SD(2,EI,theta_c,theta_d,Delta_ab)
```

Display just one of the end moments to see what the values look like.  They are indeed algebraic expressions.

```python
Mbc
```

$$\frac{EI}{3} \left(4 \theta_{b} + 2 \theta_{c}\right) - 108.0$$

Now we write five linear equations to solve for the five unknown displacements.
Three are equilibrium equations and involve moments at three of the joints: $Mab = 0$, $Mba+Mbc=0$, and $Mcb+Mcd=0$.
One equation involves the equilibrium of the column shears with the horizontal applied
load: $Vab+Vdc-60=0$.  The fifth equation is just the known rotation at joint *d*: $\theta_d=0$.

In sympy, we build a list of the five equations, all expressed as $f(..,..) = 0$ and giving only the left side of each (the right side always being 0 is not given).  We then call the solver and give it a list of variables to solve for:

```python
eqns = [Mab, Mba+Mbc, Mcb+Mcd, Vab+Vdc-60, theta_d]
soln = solve(eqns,[theta_a,theta_b,theta_c,theta_d,Delta_ab])
soln
```

$$\left \{ \Delta_{ab} : \frac{14.2834008097166}{EI}, \quad \theta_{a} : - \frac{22.4210526315789}{EI}, \quad \theta_{b} : \frac{59.1255060728745}{EI}, \quad \theta_{c} : - \frac{37.7975708502024}{EI}, \quad \theta_{d} : 0.0\right \}$$

The above solution is returned as a Python dictionary, making it easy to substitute for those
variables in other equations.

In fact, that is done now, substituting the displacement solutions into each
of the slope-deflection equations.  The result is a list of the member end moments (and shears).

The '.n(4)' method forces a numerical answer with 4 significant figures.

```python
{v:eval(v).subs(soln).n(4) for v in 'Mab,Mba,Mbc,Mcb,Mcd,Mdc,Vab,Vdc'.split(',')}
```

    {'Mab': -4.737e-15,
     'Mba': 54.36,
     'Mbc': -54.36,
     'Mcb': 97.02,
     'Mcd': -97.02,
     'Mdc': -59.22,
     'Vab': -18.12,
     'Vdc': 78.12}

And now - the reactions at supports *a* and *d*:

```python
Ra = Vbc
Rd = -Vcb
Ha = -Vab
Hd = -Vdc
Md = Mdc
[v.subs(soln).n(4) for v in [Ra,Ha,Rd,Hd,Md]]
```

$$\left [ 100.9, \quad 18.12, \quad 115.1, \quad -78.12, \quad -59.22\right ]$$

#### Equilibrium checks:

```python
# sum horizontal forces
(Ha + Hd + 60).subs(soln)
```

$$0$$

```python
# sum vertical forces
(Ra + Rd - 36*6).subs(soln)
```

$$0$$

```python
# sum moments about a
(60*3 + 36*6*3 + Hd*1 + Md - Rd*6).subs(soln)
```

$$0$$

### Plots
Plot bending moment in *bc* as a function of distance *x* from left end, *b*:

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
```

```python
var('x')
Mbcx = (Ra*x + Vab*3 - 36*x*x/2).subs(soln)  # M as a function of x
fn = lambdify((x),Mbcx)   # convert moment expression to a function of one argument
z = np.linspace(0,6)
plt.plot(z,fn(z),[0,6],[0,0])
plt.fill_between(z,fn(z),color='lightgrey')
##plt.show()
```

Find the locations along span *bc* of zero moment:

```python
[r.n(4) for r in solve(Mbcx)]
```

$$\left [ 0.6039, \quad 5.001\right ]$$

Find point of maximum moment and the maximum moment in the span:

```python
xs = solve([Mbcx.diff(x)])
xs
```

$$\left \{ x : 2.80251911830858\right \}$$

```python
xs[x].n(4), Mbcx.subs(xs).n(4)
```

$$\left ( 2.803, \quad 87.01\right )$$

```python

```


```python
import numpy as np
from scipy.misc import derivative
def f(x): return x**5
derivative(f, 1.0, dx=1e-6, order=15)
```

    4.9999999998068354

```python
derivative(f, 1.0, dx=1e-6, order=15, n=2)
```

    19.999127035691139

```python
p = np.poly1d([1,0,0,0,0,0]);
print (p)

np.polyder(p,1)(1.0)
p.deriv()(1.0)

np.polyder(p,2)(1.0)
p.deriv(2)(1.0)
```

       5
    1 x

    20.0

```python
from sympy import diff, symbols
x = symbols('x', real=True)
diff(x**5, x)
```

    5*x**4

```python
diff(x**5, x, x)
```

    20*x**3

```python
diff(x**5, x).subs(x, 1.0)
```

    5.00000000000000

```python
diff(x**5, x, x).subs(x, 1.0)
```

    20.0000000000000

```python
def g(x): return np.exp(-x) * np.sin(x)
derivative(g, 1.0, dx=1e-6, order=101)

from sympy import sin as Sin, exp as Exp
diff(Exp(-x) * Sin(x), x).subs(x, 1.0)
```

    -0.110793765306699

```python
y, z = symbols('y z', real=True)
diff(Exp(x * y * z), z, z, y, x).subs({x:1.0, y:1.0, z:2.0})
```

    133.003009780752

```python

```


```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline
```

# Class 11: Introduction to Dynamic Optimization: A Two-Period Cake-Eating Problem

Dynamic optimization, the optimal choice over elements in a time series, is at the heart of macroeconomic theory. People and firms routinely make decisions that affect their future opportunities. For example, a household chooses how much of its income to save and therefore allocates some of its current consumption to the future. In this Notebook, we consider a simple two-period dynamic optimization model. A person has some initial quantity of cake and chooses the optimal way to eat it over the next two periods. We solve for the optimal consumption path.

## Preferences

Suppose that a person has only two periods to eat cake. Let $C_0$ denote the quantity of cake consumed in period 0 and let $C_1$ denote the quantity consumed in period 1. The person's utility in period 0 from consuming $C_0$ and $C_1$ is given by:

\begin{align}
U(C_0,C_1) & = \log C_0 + \beta \log C_1,
\end{align}

where $0\leqslant\beta\leqslant 1$ is the person's *subjective discount factor*. $U$ is an *intertemporal utility function* and $\log C$ is the *flow* of utility generated each period. The presence of $\beta$ indicates the the person places lower weight on consumption in the future. The utility function implies indifference curves of the form:

\begin{align}
C_1 & =  e^{ \frac{\bar{U} - \log C_0}{\beta}},
\end{align}

where $\bar{U}$ is some given level of utility.

### Example: Plot Indifference Curves for Different Utility Levels

For $C_0$ between 0.01 and 10 and $\beta=0.95$, plot indifference curves for $\bar{U}=1, 2, 3, 4$. Set x- and y-axis limits to [0,10].

```python
# Define a function for computing c1 given a beta value and an array of c0 values

# Create a variable called 'c0' that stores c0 values from 0.01 to 10

# Create a variable called 'beta' that stores the value of beta

# Plot indifference curves

# Place legend to right of figure. PROVIDED
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))

```

### Example: Plot Indifference Curves for Different $\beta$ Values

For $C_0$ between 0.01 and 30 and $\bar{U} = 3$, plot indifference curves for $\beta=1, 0.95,0.5,0.25$. Set x-axis limits to [0,30] and y-axis limits to [0,10].

```python
# Create a variable called u that stores the value of utility for the indifference curves

# Create variable called 'c0' that stores c0 values from 0.01 to 30

# Plot indifference curves

# Place legend to right of figure. PROVIDED
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))

```

## Budget Constraints and the Boundary Condition

Let $K_t$ denote the quantity of cake available in period $t$ and let $K_0$ denote the initial quantity of cake. The cake neither grows nor depreciates over time so the person faces the following budget constraints each period:

\begin{align}
C_0 & =  K_0 - K_1\\
C_1 & =  K_1 - K_2.
\end{align}

Finally, the person's problem is subject to a *boundary condition* that imposes a limit on the endpoint of the problem. Since the person only consumes cake in periods 0 and 1, there is no value in leaving leftover cake at the end of period 1. This reasoning implies the boundary condition:

\begin{align}
K_2 & = 0.
\end{align}

The boundary condition allows us to eliminate $K_2$ from the period 1 budget constraint. Combining the two budget constraints to eliminate $K_1$ implies an *intertemporal budget constraint* that links consumption in period 0 with consumption in period 1:

\begin{align}
C_0 & =  K_0 - C_1.
\end{align}

Notice that the implied price of $C_0$ in terms of $C_1$ is 1.

### Example: Plot the Intertemporal Budget Constraint

For $C_0$ between 0.01 and 20 and plot indifference curves for $K_0=5, 10, 15, 20$. Set x-axis limits to [0,10] and y-axis limits to [0,30].

```python
# Define a function for computing c1 given a k0 value and an array of c0 values

# Create variable called 'c0' that stores c0 values from 0.01 to 10

# Plot budget constraints

# Place legend to right of figure. PROVIDED
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))

```

## Optimization

The person's problem is to choose $C_0$, $C_1$, $K_1$, and $K_2$ to maximize his utility subject to the two budget constraints and the boundary condition. However, the problem can be simplified considerably. Use the boundary condition to eliminate $K_2$ from the budget constraints. Then use the budget constraints to substitute $C_0$ and $C_1$ out of the utility function. The result is an unconstrained optimization problem in $K_1$.

\begin{align}
\max_{K_1} \; \log (K_0 - K_1) + \beta \log K_1,
\end{align}

Take the derivative with respect to $K_1$ and set the derivative equal to zero to obtain the *first-order* or *optimality* condition:

\begin{align}
-\frac{1}{K_0 - K_1} + \frac{\beta}{K_1} & = 0.
\end{align}

Solve for $K_1$ to find the optimal cake holding for period 1:

\begin{align}
\boxed{K_1  = \frac{\beta}{1+\beta}K_0}
\end{align}

Then use the budget constraints to infer optimal cake consumption in period 0:
\begin{align}
\boxed{C_0  = \frac{1}{1+\beta}K_0}
\end{align}

and period 1:

\begin{align}
\boxed{C_1  = \frac{\beta}{1+\beta}K_0}.
\end{align}

Notice that $C_0 + C_1 = K_0$ so all of the cake is eaten by the end.

### Example: Solve for the Optimal Consumption Path

Compute $K_1, C_0, C_1$ for $\beta = 0.9$ and $K_0 = 20$.

```python
# Define a function that returns the solution to the two-period cake-eating problem

    # Initialize arrays for cake and consumption

    # Assign values to cake array elements

    # Assign values to consumption array elements as implied by the budget constraints

# Create a cariable called 'cake0' that stores initial cake value

# Create a cariable called 'beta' that stores beta value

# Compute the optimal path

```

$\beta <1$ means that the person places greater weight on consumption in period 0 and therefore the person chooses to consume a greater share of the cake in period 0.


# Fourier Analysis, Wavelet Analysis, and Signal Processing

This note reviews basic operations for Fourier analysis and wavelet analysis, and demonstrate the Python implementation of the operations. The main reference is the wavelet paper by Torrence and Compo (1998), we will refer to it as TC98 later in the note.

The codes below are just some preparations at the beginning of the python code.

```python
import numpy as np
import pandas as pd
import matplotlib as mpl
import matplotlib.ticker as ticker
import matplotlib.pyplot as plt

# Plot settings.
mpl.rcParams['lines.linewidth'] = 0.5
mpl.rcParams['xtick.top'] = True
mpl.rcParams['xtick.bottom'] = True
mpl.rcParams['xtick.direction'] = 'out'
mpl.rcParams['xtick.major.size'] = 4
mpl.rcParams['xtick.minor.visible'] = True
mpl.rcParams['xtick.minor.top'] = True
mpl.rcParams['xtick.minor.bottom'] = True
mpl.rcParams['xtick.minor.size'] = 2
mpl.rcParams['ytick.left'] = True
mpl.rcParams['ytick.right'] = True
mpl.rcParams['ytick.direction'] = 'out'
mpl.rcParams['ytick.major.size'] = 4
mpl.rcParams['ytick.minor.visible'] = True
mpl.rcParams['ytick.minor.left'] = True
mpl.rcParams['ytick.minor.right'] = True
mpl.rcParams['ytick.minor.size'] = 2
mpl.rcParams['axes.spines.right'] = True
mpl.rcParams['axes.spines.top'] = True
mpl.rcParams['axes.grid'] = 'on'
mpl.rcParams['grid.alpha'] = 0.5
mpl.rcParams['font.size'] = 8
```

## Input signal
In this note, the input signal refers to a **discrete-time signal** $x_n$, where $n=0,1,\ldots,N-1$ and $N$ is the total number of record. $x_n$ is the real-world measurement of the **continuous-time signal** $x(t)$ at certain *time resolution* $\delta t$.

Both $x_n$ and $x(t)$ are in the unit of $\mathtt{x}$. The main difference is that $x(t)$ is explicitly a function of time $t$, whereas $x_n$ maps to the discrete time $t_n=n\delta t$ through the index $n$. For $x_n$, the *duration* of the signal is $T=N\delta t$.

```python
# Load Nino3 data.
nino3 = "http://paos.colorado.edu/research/wavelets/wave_idl/sst_nino3.dat"
nino3 = pd.read_table(nino3)
x0 = nino3.values.squeeze()     # original data.
dt = 0.25                       # time rate, 0.25 year.
n0 = x0.size                    # original length.
t0 = 1871+0.25*np.arange(n0)    # times in year.

fig = plt.figure(figsize=(6,2.5))
ax = fig.add_axes([0.2,0.2,0.75,0.75])
ax.plot(t0,x0, 'k')
ax.set(xlim=[1870,2000], ylabel='($^{\circ}C$)', xlabel='Time (year)');
```

The above codes load and plot the NINO3 SST data, which is the input signal we will use. To facillitate later calculations, the input signal is padded to an integer power of 2 with zeros on both sides. For the NINO3 SST data, $N=503$, $\delta t=0.25$ yr, $T=125.75$ yr.

```python
# remove DC offset, calculate the power.
x0 -= x0.mean()         # remove DC offset, make 0-freqency signal be 0.
sigma2 = np.var(x0)     # the variance, but also the power if <x[n]> = 0.
```

The variance $\sigma^2$ of the input signal is also the power (defined below) if the input signal is of zero-mean. As we will see shortly, the DC-offset corresponds to the Fourier coefficient at zero-frequency, and it is advantageous to let this coefficient be 0.

For certain discrete-time signal $x_n$, one can define the **energy** $E$ and the **time-averaged power** $P$:
\begin{align}
E &= \delta t\sum_{n=0}^{N-1}\left|x_n\right|^2\\
P &= \frac{\delta t}{T}\sum_{n=0}^{N-1}\left|x_n\right|^2 = \frac{1}{N}\sum_{n=0}^{N-1}\left|x_n\right|^2
\end{align}

The corresponding definitions for a continuous-time signal $x(t)$ are
\begin{align}
E &= \int_0^T dt\;\left|x(t)\right|^2\\
P &= \frac{1}{T}\int_0^T dt\;\left|x(t)\right|^2.
\end{align}

Note that the energy and power defined here is irrelevant to the energy and power in physics. It is easy to see that the power $P$ is the same as $\sigma^2$ when $\left<x_n\right>=0$.

## Fourier Analysis
The discrete Fourier transform (DFT) of the input signal $x_n$ and the inverse transform are defined as
\begin{align}
\hat{x}_k &= \frac{1}{N}\sum_{n=0}^{N-1}x_n e^{-2\pi ikn/N},\\
x_n &= \sum_{k=0}^{N-1} \hat{x}_k e^{2\pi ikn/N}.
\end{align}

Here we use "hat" version $\hat{x}$ to refer to the Fourier transform of certain quantity $x$. It should be clear that the frequency index $k=0,1,\ldots,N-1$ and the time index $n=0,1,\ldots,N-1$. As the time index $n$ maps the signal $x_n$ to time $t_n$, the frequency index $k$ maps the Fourier transform $\hat{x}_k$ to frequency $f_n$. We will re-visit the frequency index and its mapping shortly.

An important property of the Fourier transform is the **Parseval's relation**
\begin{equation}
P = \frac{1}{N}\sum_{n=0}^{N-1}\left|x_n\right|^2 = \sum_{k=0}^{N-1}\left|\hat{x}_k\right|^2.
\end{equation}

The equation says that the power is independent on whether it is counted in the time domain or frequency domain. Notice that the power in the frequency domain at each $k$ adds directly. This is the result of the normalization we used. There are other valid normalizations for DFT, as listed in Table 1.1 in *Analysis methods for multi-spacecraft data* by Paschmann and Daly (1998).

```python
# Reproduce Fig 3, FFT.

# padding 0, ensure N is an integer power of 2.
N = np.int(np.power(2,np.ceil(np.log2(n0))))
T = N*dt
dn = N-n0
xn = np.pad(x0,(0,dn), 'constant', constant_values=(0,0))
tn = np.pad(t0,(0,dn), 'linear_ramp', end_values=(0,t0[-1]+dn*dt))

# Plot settings.
mpl.rcParams['xtick.minor.visible'] = False
mpl.rcParams['xtick.minor.top'] = False
mpl.rcParams['xtick.minor.bottom'] = False
mpl.rcParams['xtick.minor.size'] = 0
mpl.rcParams['axes.grid'] = 'off'

fig = plt.figure(figsize=(4,4))
ax = fig.add_axes([0.2,0.2,0.75,0.75])

ns = np.arange(N)/N     # time index, 0,1,...,N-1.
ks = np.arange(N)       # freq index, 0,1,...,N-1.
xk = np.array([sum(xn*np.exp(-2*np.pi*1j*k*ns)) for k in ks])/N    # Eq.3

fft_ps = abs(xk)**2 * 2     # power spectrum, in X^2.
fft_fk = ks/T               # frequency.

# Throw the symmetric half and the 0-frequency coefficient.
xs = 1/fft_fk[1:N//2]       # period.
ys = fft_ps[1:N//2]/(2*sigma2/N)  # normalize to power, c.f. Fig.3
ax.plot(xs, ys, 'k')
ax.set(xscale='log', xlim=[64,0.5], xlabel='Period (years)',
    ylim=[0,15], ylabel='Variance ($\sigma^2$)')
ax.xaxis.set_major_locator(plt.LogLocator(base=2,numticks=8))
ax.xaxis.set_major_formatter(ticker.FormatStrFormatter('%.1f'))

# calculate mean red-noise spectrum.
series = pd.Series(xn)
# auto-correlation of lag-1 and lag-2.
alpha1 = series.autocorr(1)
alpha2 = series.autocorr(2)
alpha = (alpha1+np.sqrt(alpha2))*0.5    # see text below Eq.16
# the red-noise spectrum background.
pk = (1-alpha**2)/(1+alpha**2-2*alpha*np.cos(2*np.pi*ks[0:N//2]/N))   # Eq.16

xs = 1/fft_fk[1:N//2]
ys = pk[1:]
ax.plot(xs, ys, 'k--')

# calculate the 95% confidence level.
from scipy.stats import chi2
dof = 2     # degree-of-freedom is 2 for FFT.
confidence_level = 0.95
chiv2 = chi2.ppf(confidence_level, dof) / dof
qk = pk[1:] * chiv2  # see text left to Fig.5
ax.plot(xs, qk, 'k--');
```

### Fourier power spectrum
The above codes reproduce Fig. 3 in TC98. Right now, let's only focus on the solid line, which is the Fourier power spectrum normalized to $2\sigma^2/N$.

The Fourier **power spectrum** (PS) is defined as
\begin{equation}
ps_k = 2\left|x_k\right|^2.
\end{equation}

The tricky part here is that the power spectrum is only define for $k=1,2,\ldots,N/2$, which corresponds to the period $\tau_k = T/k$ from $2\delta t$ to $T$. To see why, let's look at the Fourier transform closer.

```python
xk[[0,1,255,256,257,511]]
```

### Frequency index $k$
The above line prints the Fourier transform $\hat{x}_k$ at several interesting places. $k=0$ corresponds to zero-frequency. It is effectively 0 because we have removed the DC-offset.

It's also easy to notice that $\hat{x}_{1} = \hat{x}^*_{511}$ and $\hat{x}_{255} = \hat{x}^*_{257}$. Indeed, for all non-zero frequency index $k=1,2,\ldots,N-1$, we have the **reality condition**
\begin{equation}
\hat{x}_k = \hat{x}^*_{N-k}.
\end{equation}

The frequency index $k$ maps the Fourier transform $\hat{x}_k$ to the **frequency** $f_k$ in the mapping rule:
\begin{equation}
f_k = \begin{cases}
   \frac{k}{T}, &k=0,1,\ldots,N/2\\
-\frac{N-k}{T}, &k=N/2,\ldots,N-1.
\end{cases}
\end{equation}
$\hat{x}_{N/2}$ ($\hat{x}_{256}$ in this case) corresponds to the Nyquist frequency $f_{Nyq} = 1/2\delta t$, which is the highest frequency can be resolved for a given time resolution $\delta t$. $\hat{x}_{N/2}$ can be mapped to either $-f_{Nyq}$ or $f_{Nyq}$.

Given the above discussion, it should be clear that why the power spectrum is defined only for $k=1,2,\ldots,N/2$. For $k>0$, energy contributions from $\pm f_k$ are equal and can be merged as $2\left|x_k\right|^2$, whereas for $k=0$, the energy contribution is 0.

The **period** $\tau_k$ that corresponds to the power spectrum is
\begin{equation}
\tau_k = T/k,\quad k=1,2,\ldots,N/2.
\end{equation}

Another common quantity in use is the **power spectral density** (PSD)
\begin{equation}
psd_k = 2T\left|x_k\right|^2.
\end{equation}
Note that the power spectrum is in the unit of $\mathtt{x}^2$ and the power spectrum density is in the unit of $\mathtt{x}^2-t$.

### Significant level and confidence level
The lower dashed line in the above figure is the mean red-noise spectrum, and the upper dashed line is the 95% confidence spectrum.

According to Section 4.a in TC98, the Fourier power spectrum for a simple red-noise model is
\begin{equation}
P_k = \frac{1-\alpha^2}{1+\alpha^2-2\alpha\cos(2\pi k/N)},
\end{equation}
where $k=1,2,\ldots,N/2$, $\alpha=(\alpha_1+\sqrt{\alpha_2})/2$, $\alpha_{1,2}$ are the lag-1 and lag-2 auto-correlations of the input signal $x_n$. For the NINO3 data, $\alpha=0.72$. The red-noise spectrum is normalized and serves as a background for significant peaks.

The 95% confidence spectrum is calculated based on the red-noise spectrum.
\begin{equation}
    Q_k = P_k \;\chi^2(\nu_0,c),
\end{equation}
where $c=0.95$ is the specified confidence level, $\nu_0$ is the **degree of freedom** ($\nu_0=2$ for Fourier transform). Both of them are inputs for the $\chi^2$ distribution.

# Wavelet function
In this section, we implement the several wavelet functions introduced in TC98. This is a preparation for next section, where we implement the continuous wavelet transform.

```python
# Reproduce Fig 2.
# Define wavelet functions.
def morlet(eta,w0=6):
    return np.power(np.pi,-0.25)*np.exp(eta*(-0.5*eta+(0+1j)*w0))

def paul(eta, m=4):
    return np.power(2j,m)*np.math.factorial(m)/np.sqrt(np.pi*np.math.factorial(2*m))*np.power(1-eta*(0+1j),-m-1)

from scipy.special.orthogonal import hermitenorm
from numpy.lib.polynomial import polyval
from scipy.special import gamma

def dog(eta, m=2):
    return np.power(-1,np.mod(m+1,2))*polyval(hermitenorm(m),eta)*np.exp(-0.5*eta**2)/np.sqrt(gamma(m+0.5))

def morletf(sw, w0=6):
    return np.power(np.pi,-0.25)*np.exp(-0.5*(sw-w0)**2)*(sw>0)

def paulf(sw, m=4):
    return np.power(2,m)/np.sqrt(m*np.math.factorial(2*m-1))*np.power(sw,m)*np.exp(-sw)*(sw>0)

def dogf(sw, m=2):
    return -np.power(1j,m)/np.sqrt(gamma(m+0.5))*np.power(sw,m)*np.exp(-0.5*sw**2)
```

For a generic wavelet function $\psi_0(z)$, it's easy to calculate its Fourier transform $\hat{\psi}_0(z)$. For both of them the subscript 0 indicates they are the mother wavelet function.

Based on Table 1 in TC98, three commonly used mother wavelet functions and their Fourier tranforms are
* Morlet. $\omega_0$ is the free parameter. $\omega_0=6$ in the paper.
\begin{align}
\psi_0(z) &= \pi^{-1/4}e^{i\omega z}e^{-z^2/2}\\
\hat{\psi}_0(z) &= \pi^{-1/4} H(z) e^{-(z-\omega_0)^2/2}.
\end{align}
* Paul. $m$ is the free parameter. $m=4$ in the paper.
\begin{align}
\psi_0(z) &= \frac{2^m i^m m!}{\sqrt{\pi(2m)!}}(1-iz)^{-(m+1)}\\
\hat{\psi}_0(z) &= \frac{2^m}{\sqrt{m(2m-1)!}}H(z) z^m e^{-z}.
\end{align}
* DOG (derivative of a Gaussian). $m$ is the free parameter. $m=2,6$ in the paper.
\begin{align}
\psi_0(z) &= \frac{(-1)^{m+1}}{\sqrt{\Gamma(m+1/2)}}\frac{d^m e^{-z^2/2}}{d\tau^m}\\
\hat{\psi}_0(z) &= \frac{-i^{m}}{\sqrt{\Gamma(m+1/2)}}\eta^m e^{-z^2/2}.
\end{align}
In above equations
\begin{equation}
H(z) = \begin{cases}
1 & z>0\\
0 & z\leq 0.
\end{cases}
\end{equation}

Run the codes below and check the waveforms of these wavelet functions.

```python
# Plot settings.
mpl.rcParams['xtick.minor.visible'] = True
mpl.rcParams['xtick.minor.top'] = True
mpl.rcParams['xtick.minor.bottom'] = True
mpl.rcParams['xtick.minor.size'] = 2
mpl.rcParams['xtick.top'] = False
mpl.rcParams['xtick.minor.top'] = False
mpl.rcParams['ytick.right'] = False
mpl.rcParams['ytick.minor.right'] = False
mpl.rcParams['axes.spines.right'] = False
mpl.rcParams['axes.spines.top'] = False

fig = plt.figure(figsize=(4,7))
fig.subplots_adjust(wspace=0.5, hspace=0.8, top=0.9)

t = np.linspace(-4,4,500,endpoint=True)
dt = t[1]-t[0]
s = 10*dt
eta = np.zeros(len(t), dtype=complex)
eta.real = t
sw = np.linspace(-2,2,500,endpoint=True)

# a. Morlet
ax = fig.add_subplot(421)
psi0 = morlet(eta)
psi = psi0*np.sqrt(dt/s)
ax.set(xlim=[-4,4], ylim=[-0.3,0.35])
ax.xaxis.set_major_locator(plt.MultipleLocator(2))
ax.xaxis.set_minor_locator(plt.MultipleLocator(1))
ax.yaxis.set_major_locator(plt.MultipleLocator(0.3))
ax.yaxis.set_minor_locator(plt.MultipleLocator(0.15))
ax.axhline(color='black',alpha=0.3)
ax.plot(t,psi.real, 'k-')
ax.plot(t,psi.imag, 'k--')
ax.text(-0.2,1.1, 'a. Morlet', ha='left',va='bottom', transform=ax.transAxes)
ax.text(0.5,1.5, '$\psi(t/s)$', ha='center',va='bottom', transform=ax.transAxes)

# b. Paul (m=4)
ax = fig.add_subplot(423)
psi0 = paul(eta)
psi = psi0*np.sqrt(dt/s)
ax.set(xlim=[-4,4], ylim=[-0.3,0.35])
ax.xaxis.set_major_locator(plt.MultipleLocator(2))
ax.xaxis.set_minor_locator(plt.MultipleLocator(1))
ax.yaxis.set_major_locator(plt.MultipleLocator(0.3))
ax.yaxis.set_minor_locator(plt.MultipleLocator(0.15))
ax.axhline(color='black',alpha=0.3)
ax.plot(t,psi.real, 'k-')
ax.plot(t,psi.imag, 'k--')
ax.text(-0.2,1.1, 'b. Paul (m=4)', ha='left',va='bottom', transform=ax.transAxes)

# c. DOG (m=2)
ax = fig.add_subplot(425)
psi0 = dog(eta)
psi = psi0*np.sqrt(dt/s)
ax.set(xlim=[-4,4], ylim=[-0.3,0.35])
ax.xaxis.set_major_locator(plt.MultipleLocator(2))
ax.xaxis.set_minor_locator(plt.MultipleLocator(1))
ax.yaxis.set_major_locator(plt.MultipleLocator(0.3))
ax.yaxis.set_minor_locator(plt.MultipleLocator(0.15))
ax.axhline(color='black',alpha=0.3)
ax.plot(t,psi.real, 'k-')
ax.text(-0.2,1.1, 'c. DOG (m=2)', ha='left',va='bottom', transform=ax.transAxes)

# d. DOG (m=6)
ax = fig.add_subplot(427)
psi0 = dog(eta, m=6)
psi = psi0*np.sqrt(dt/s)
ax.set(xlim=[-4,4], ylim=[-0.3,0.35])
ax.xaxis.set_major_locator(plt.MultipleLocator(2))
ax.xaxis.set_minor_locator(plt.MultipleLocator(1))
ax.yaxis.set_major_locator(plt.MultipleLocator(0.3))
ax.yaxis.set_minor_locator(plt.MultipleLocator(0.15))
ax.axhline(color='black',alpha=0.3)
ax.plot(t,psi.real, 'k-')
ax.set(xlabel='$t/s$')
ax.text(-0.2,1.1, 'd. DOG (m=6)', ha='left',va='bottom', transform=ax.transAxes)

# a'. Morlet
ax = fig.add_subplot(422)
ax.text(0.5,1.5, '$\hat{\psi}(s\omega)$', ha='center',va='bottom', transform=ax.transAxes)
psi0 = morletf(sw * 2 * np.pi)
psi = psi0*np.sqrt(2*np.pi*s/dt)
ax.set(xlim=[-2,2], ylim=[0,6])
ax.xaxis.set_major_locator(plt.MultipleLocator(1))
ax.xaxis.set_minor_locator(plt.MultipleLocator(0.5))
ax.yaxis.set_major_locator(plt.MultipleLocator(2))
ax.yaxis.set_minor_locator(plt.MultipleLocator(1))
ax.axvline(color='black',alpha=0.3)
ax.plot(sw,psi.real, 'k-')

# b'. Paul
ax = fig.add_subplot(424)
psi0 = paulf(sw * 2 * np.pi)
psi = psi0*np.sqrt(2*np.pi*s/dt)
ax.set(xlim=[-2,2], ylim=[0,6])
ax.xaxis.set_major_locator(plt.MultipleLocator(1))
ax.xaxis.set_minor_locator(plt.MultipleLocator(0.5))
ax.yaxis.set_major_locator(plt.MultipleLocator(2))
ax.yaxis.set_minor_locator(plt.MultipleLocator(1))
ax.axvline(color='black',alpha=0.3)
ax.plot(sw,psi.real, 'k-')

# c'. DOG
ax = fig.add_subplot(426)
psi0 = dogf(sw * 2 * np.pi)
psi = psi0*np.sqrt(2*np.pi*s/dt)
ax.set(xlim=[-2,2], ylim=[0,6])
ax.xaxis.set_major_locator(plt.MultipleLocator(1))
ax.xaxis.set_minor_locator(plt.MultipleLocator(0.5))
ax.yaxis.set_major_locator(plt.MultipleLocator(2))
ax.yaxis.set_minor_locator(plt.MultipleLocator(1))
ax.axvline(color='black',alpha=0.3)
ax.plot(sw,psi.real, 'k-')

# d'. DOG
ax = fig.add_subplot(428)
psi0 = dogf(sw * 2 * np.pi, m=6)
psi = psi0*np.sqrt(2*np.pi*s/dt)
ax.set(xlim=[-2,2], ylim=[0,6])
ax.xaxis.set_major_locator(plt.MultipleLocator(1))
ax.xaxis.set_minor_locator(plt.MultipleLocator(0.5))
ax.yaxis.set_major_locator(plt.MultipleLocator(2))
ax.yaxis.set_minor_locator(plt.MultipleLocator(1))
ax.axvline(color='black',alpha=0.3)
ax.plot(sw,psi.real, 'k-')
ax.plot(sw,psi.imag, 'k-')
ax.set(xlabel='$s\omega/(2\pi)$');
```

The above plot reproduces Fig. 2 in TC98. What are plotted are the daughter wavelet functions $\psi_s$ and $\hat{\psi}_s$ for $s/\delta t = 10$.

For a certain scale $s$, the mother and daughter wavelet functions differ in normalization, as given by Equation 6 and 8 in TC98
\begin{align}
\psi_s(z) &= \sqrt{\delta t/s}\; \psi_0(z)\\
\hat{\psi}_s(z) &= \sqrt{2\pi s/\delta t}\; \hat{\psi}_0(z).
\end{align}

To be clear, later in the note, we will stick to the mother wavelet function $\psi_0$ and $\hat{\psi}_0$, and write the normalization explicitly.

The Morlet and Paul wavelets are complex, thus their $\psi$ have both the real (solid line) and imaginary (dashed line) parts. The DOG wavelet is real, thus its $\psi$ only has the real (solid line) part, and its $\hat{\psi}$ is symmetric around 0. (Remember the reality condition leads that for a real-valued signal, the real part of its Fourier transform is symmetric around the zero-frequency.)

## Wavelet Analysis
Wavelet analysis is one step forward than Fourier analysis. Similar to Fourier analysis, wavelet analysis can reveal the power distribution in the frequency domain, characterized by the scale index $j$, which is the counterpart of frequency index $k$ in Fourier analysis. In addition, wavelet analysis also reveals the power distribution in the time domain, characterized by the time index $n$.

The **continuous wavelet transform** (CWT) of the input signal $x_n$ is a complex-valued quantity, depends on the scale index $j$ and the time index $n$. It can be calculated in the time domain
\begin{equation}
W_{n,j} = \sum_{l=0}^{N-1} x_l \sqrt{\frac{\delta t}{s_j}}\psi^*_0 \left[\frac{(l-n)\delta t}{s_j}\right],
\end{equation}
or equivalently in the frequency domain
\begin{equation}
W_{n,j} = \sum_{k=0}^{N-1} \hat{x}_k \sqrt{\frac{2\pi s_j}{\delta t}}\hat{\psi}^*_0(2\pi k s_j/T) e^{2\pi ikn/N}.
\end{equation}
The codes below implement the CWT in the time domain and reproduces Fig. 1 in TC98.

```python
#### Reproduce Fig 1.

# Plot settings.
mpl.rcParams['xtick.top'] = True
mpl.rcParams['xtick.minor.top'] = True
mpl.rcParams['ytick.right'] = True
mpl.rcParams['ytick.minor.right'] = True
mpl.rcParams['axes.spines.right'] = True
mpl.rcParams['axes.spines.top'] = True
mpl.rcParams['axes.grid'] = 'on'

unit ='($^{\circ}C$)'
label ='NINO3 SST'

fig = plt.figure(figsize=(8,8))
gs = plt.GridSpec(3,1,height_ratios=[2,3,3])
gs.update(left=0.2, right=0.8, hspace=0.6, top=0.9, bottom=0.1)

# a. Nino3 time series.
ax = plt.subplot(gs[0])
ax.plot(t0, x0, color='black')
ax.set(xlim=[1870,2000], ylabel=unit)
ax.xaxis.set_major_locator(plt.MultipleLocator(20))
ax.xaxis.set_minor_locator(plt.MultipleLocator(5))
ax.yaxis.set_major_locator(plt.MultipleLocator(1))
ax.yaxis.set_minor_locator(plt.MultipleLocator(0.5))
ax.text(-0.2,1.1, 'a. NINO3 SST', ha='left',va='bottom', transform=ax.transAxes)

dt = 0.25
T = n0*dt
xn = (x0-x0.mean())/x0.std()
xn = np.pad(xn,(0,dn), 'constant', constant_values=(0,0))

# b. Morlet CWT
# make scale.
s0 = 2*dt
s1 = 64
dj = 0.125
J = np.int(np.log2(s1/s0)/dj)+1
sj = s0*np.power(2,dj*np.arange(J))

# In time domain.
wns = np.empty((N,J),dtype=complex)
for n in range(N):
    for j in range(J):
        z = (np.arange(N)-n)*dt/sj[j]
        wns[n,j] = sum(xn*np.conjugate(morlet(z)))*np.sqrt(dt/sj[j])

## In freq domain.
#wns = np.empty((N,J),dtype=complex)
#xk = np.fft.fft(xn, N)
#wk = np.fft.fftfreq(N,dt)*2*np.pi
#for j in range(J):
#    daughter = np.sqrt(2*np.pi*sj[j]/dt)*morletf(sj[j]*wk)
#    wns[:,j] = np.fft.ifft(xk*daughter)

wns = wns[0:N-dn,:]
cwtpower = np.abs(wns)**2
vs = [1,2,5,10]

# Fourier wavelength.
w0 = 6
s2t = 4*np.pi/(w0+np.sqrt(2+w0**2))
tauj = sj*s2t
ylim = np.array([64,0.5])

# main contour.
xi, yi = np.meshgrid(t0, tauj)
zi = np.transpose(np.copy(cwtpower))
ax = plt.subplot(gs[1])
ax.yaxis.tick_left()
ax.contourf(xi, yi, zi, vs, cmap='Greys', extend='max')
ax.contour(xi, yi, zi, vs, colors='black', linewidths=0.3)
ax.text(-0.2,1.1, 'b. Morlet, $\omega_0$=6', ha='left',va='bottom', transform=ax.transAxes)

# Wavelet power spectrum and periods for later code.
wps_nj = cwtpower
lambdas = tauj

# Add 95% confidence level.
gamma0 = 2.32   # Table 2.
pj = (1-alpha**2)/(1+alpha**2-2*alpha*np.cos(2*np.pi*dt/tauj))
dof = 2
chiv2 = chi2.ppf(confidence_level, dof) / dof
qj = pj * chiv2
for j in range(J): zi[j,:] /= qj[j]
ax.contour(xi,yi, zi, [-99,1], colors='red', linewidths=0.5)

# Add 2nd y-axis for the scale.
ax2 = fig.add_subplot(gs[1], sharex=ax, frameon=False)
ax2.yaxis.tick_right()
ax2.yaxis.grid(False)
ax2.get_xaxis().set_visible(False)
ax2.yaxis.set_label_position("right")
ax2.set(ylim=ylim, yscale='log', ylabel='Scale (years)')
ax2.yaxis.set_major_locator(plt.LogLocator(base=2,numticks=8))
ax2.yaxis.set_major_formatter(ticker.FormatStrFormatter('%.1f'))
ax2.yaxis.set_minor_locator(plt.NullLocator())

# Add COI. Table 1, e-folding time.
tmid = 0.5*(t0[0]+t0[-1])
coi = (T*0.5-abs(t0-tmid))/np.sqrt(2)
ax2.plot(t0, coi, lw=0.5, color='black')
ax2.fill_between(t0, coi, y2=1e5, lw=0, facecolor='none', hatch='X', edgecolor='black', alpha=0.3)
ax.set(xlim=[1870,2000], ylim=ylim*s2t, yscale='log', ylabel='Period (years)')
ax.yaxis.set_major_locator(plt.LogLocator(base=2,numticks=8))
ax.yaxis.set_major_formatter(ticker.FormatStrFormatter('%d'))
ax.yaxis.set_minor_locator(plt.NullLocator())

# b. DOG CWT
# make scale.
s0 = 0.125
s1 = 16
dj = 0.125
J = np.int(np.log2(s1/s0)/dj)+1
sj = s0*np.power(2,dj*np.arange(J))

# In time domain.
wns = np.empty((N,J),dtype=complex)
for n in range(N):
    for j in range(J):
        wns[n,j] = sum(xn*np.conjugate(dog((np.arange(N)-n)*dt/sj[j])))*np.sqrt(dt/sj[j])

## In freq domain.
#wns = np.empty((N,J),dtype=complex)
#xk = np.fft.fft(xn, N)
#wk = np.fft.fftfreq(N,dt)*2*np.pi
#for j in range(J):
#    daughter = np.sqrt(2*np.pi*sj[j]/dt)*dogf(sj[j]*wk)
#    wns[:,j] = np.fft.ifft(xk*daughter)

wns = wns[0:N-dn,:]
cwtpower = np.abs(wns)**2
vs = [2,100]
colors = ['0.9','0.9']

# Fourier wavelength.
m = 2
s2t = 2*np.pi/np.sqrt(m+0.5)
tauj = sj*s2t
ylim = np.array([16,0.125])

# main contour.
xi, yi = np.meshgrid(t0, tauj)
zi = np.transpose(np.copy(cwtpower))
ax = plt.subplot(gs[2])
ax.yaxis.tick_left()
ax.contourf(xi, yi, zi, vs, colors=colors, extend='max')
ax.contour(xi, yi, zi, vs, colors='black', linewidths=0.3)
ax.text(-0.2,1.1, 'c. DOG, m=2', ha='left',va='bottom', transform=ax.transAxes)

# Add 95% confidence level.
gamma0 = 1.43   # Table 2.
pj = (1-alpha**2)/(1+alpha**2-2*alpha*np.cos(2*np.pi*dt/tauj))
dof = 1
chiv2 = chi2.ppf(confidence_level, dof) / dof
qj = pj * chiv2
for j in range(J): zi[j,:] /= qj[j]
ax.contour(xi,yi, zi, [-99,1], colors='red', linewidths=0.5)

# Add 2nd y-axis for the scale.
ax2 = fig.add_subplot(gs[2], sharex=ax, frameon=False)
ax2.yaxis.tick_right()
ax2.yaxis.grid(False)
ax2.get_xaxis().set_visible(False)
ax2.yaxis.set_label_position("right")
ax2.set(ylim=ylim, yscale='log', ylabel='Scale (years)')
ax2.yaxis.set_major_locator(plt.LogLocator(base=2,numticks=8))
ax2.yaxis.set_major_formatter(ticker.FormatStrFormatter('%.1f'))
ax2.yaxis.set_minor_locator(plt.NullLocator())

# Add COI. Table 1, e-folding time.
tmid = 0.5*(t0[0]+t0[-1])
coi = (T*0.5-abs(t0-tmid))/np.sqrt(2)*s2t
ax.plot(t0, coi, lw=0.5, color='black')
ax.fill_between(t0, coi, y2=1e5, lw=0, facecolor='none', hatch='X', edgecolor='black', alpha=0.3)
ax.set(xlim=[1870,2000], ylim=ylim*s2t, yscale='log', ylabel='Period (years)')
ax.yaxis.set_major_locator(plt.LogLocator(base=2,numticks=8))
ax.yaxis.set_major_formatter(ticker.FormatStrFormatter('%d'))
ax.yaxis.set_minor_locator(plt.NullLocator())
```

### Scale index $j$
Similar to the frequency index $k$ in the Fourier transform, the wavelet transform has a scale index $j$, for the dimension of frequency domain. The scales $s_j$ are in the unit of $t$, and convert to periods $\tau_j$ by $c_{\lambda} = \tau_j/s_j$. From Table 1 in TC98
* Morlet. $c_\lambda = \cfrac{4\pi}{\omega_0 + \sqrt{2+\omega_0}}$. For example, $\tau_j \simeq 1.033 s_j$ for $\omega_0 = 6$.
* Paul. $c_\lambda = \cfrac{4\pi}{2m+1}$. For example, $\tau_j \simeq 1.396 s_j$ for $m=4$.
* DOG. $c_\lambda = \cfrac{2\pi}{\sqrt{m+1/2}}$. For example, $\tau_j \simeq 3.974 s_j$ for $m=2$.

In the above plot, $\tau_j$ are the y-axis on the left and $s_j$ are the y-axis on the right. For example in panel c, the conversion coefficient is 3.974 for DOG $m=2$, hence the period is roughly 4 times of the corresponding scale.

Different from the Fourier transform, there is freedom to choose scales $s_j$. One can generate $s_j$ by setting the smallest scale $s_0$, the step  of power $\delta j$, and the number of scales $J$
\begin{equation}
s_j = s_0 2^{j\delta j},\quad j=0,1,\ldots,J-1.
\end{equation}
As seen from panel b and c in the above plot, the scales are chosen to make the period ranges from 0.5 to 64 yr. Recall that $\delta t=0.25$ yr and $T = 128$ yr, this period range corresponds to $2\delta t$ and $T/2$ respectively. In terms of $\delta j$, it is usually set to be $0.125$ or $0.25$.

### Cone of influence (COI)
The cross hatched regions in panel b and c in the above plot correspond to the **cone of influence**, which marks the region affected by the boundary effect. For the time $t_n$, one can defind the time difference from the center $\Delta t_n = 0.5T - |t_n-t_0-0.5T|$. Every $\Delta t_n$ corresponds to a scale $s_n$. At the scale $s_n$, times within $\Delta t_n$ on both boundaries are affected by the boundary effect. From Table 1 in TC98, the conversion constant $c_{COI} = s_n/\Delta t_n$ depends on type of wavelet.
* Morlet. $1/\sqrt{2}$.
* Paul. $\sqrt{2}$.
* DOG. $1/\sqrt{2}$.

To be specific, the COI is a function of time $t_n$, and its value can be expressed in terms of either scales $s_n$ or periods $\tau_n$.
* For scale, COI's values are $s_n = \Delta t_n c_{COI}$.
* For period, COI's values are $\tau_n = c_\lambda \Delta t_n c_{COI}$.

### Wavelet power spectrum
Similar to the Fourier power spectrum, the **wavelet power spectrum** is
\begin{equation}
wps_{n,j} = \left|W_{n,j}\right|^2.
\end{equation}

The wavelet power spectrum is in the unit of $\mathtt{x}^2$, defined for $n=0,1,\ldots,N-1$ and $j=0,1,\ldots,J-1$. Consequently, averaging over time defines the **global wavelet spectrum** (GWS), which gives the power distribution in the frequency domain. And averaging over scale defines the **scale-averaged wavelet power** (SWP), which gives the power distribution in the time domain. We will compare the former to the Fourier power spectrum, but the latter is what missed by the Fourier analysis.
\begin{align}
gws_j &= \frac{1}{N} \sum_{n=0}^{N-1} {\left|W_{n,j}\right|^2},\\
swp_n &= \frac{\delta j\delta t}{C_\delta} \sum_{j=0}^{J-1} \frac{\left|W_{n,j}\right|^2}{s_j}.
\end{align}

### Confidence level for WPS
The last point about the above plot is the red contours in panel b and c. They are the 95% confidence level $Q_j$ defined on each scale $s_j$. It's definition is the same as the Fourier transform. The only difference is that now $Q_j$ is defined on the scale index $j$.

According to Section 4.a in TC98, the red-noise background is
\begin{equation}
P_j = \frac{1-\alpha^2}{1+\alpha^2-2\alpha\cos(2\pi\delta t/\tau_j)},
\end{equation}
where $j=0,2,\ldots,J-1$, $\alpha=(\alpha_1+\sqrt{\alpha_2})/2$, $\alpha_{1,2}$ are the lag-1 and lag-2 auto-correlations of the input signal $x_n$.

The 95% confidence spectrum is calculated based on the red-noise spectrum.
\begin{equation}
    Q_j = P_j \;\chi^2(\nu_0,c),
\end{equation}
where $c=0.95$ is the specified confidence level, $\nu_0$ is the **degree of freedom**. $\nu_0 = 2$ for complex wavelets (Morlet and Paul), and $\nu_0 = 1$ for real wavelts (DOG). Both of them are inputs for the $\chi^2$ distribution.

The $Q_j$ applies to all times for the wavelet power spectrum, therefore the contour of $WPS_nj = Q_j$ is equivalent to the contour of $wps_{nj}/Q_j = 1$.

### Global wavelet spectrum (GWS)
Now we compare the global wavelet spectrum
\begin{equation}
gws_j = \frac{1}{N} \sum_{n=0}^{N-1} {\left|W_{n,j}\right|^2}
\end{equation}
to its counterpart in Fourier analysis: the Fourier power spectrum.

Here are the codes:

```python
# Reproduce Fig 6.
# GWS vs FFT.
mpl.rcParams['xtick.minor.visible'] = False
mpl.rcParams['xtick.minor.top'] = False
mpl.rcParams['xtick.minor.bottom'] = False
mpl.rcParams['xtick.minor.size'] = 0
mpl.rcParams['axes.grid'] = 'off'

def smooth(x0, N):
    n1 = N // 2
    n2 = N - n1

    x1 = np.pad(x0, (n1, n2), 'constant', constant_values=(0, 0))
    x2 = np.empty(len(x0), dtype=float)
    x2[0] = sum(x1[0:N])
    for i in range(1, len(x0)):
        x2[i] = x2[i - 1] - x1[i - 1] + x1[N + i - 1]
    return x2 / N

xs = 1/fft_fk[1:n0//2]   # period.
ys = smooth(fft_ps, 5)[1:n0//2]*(N/(2*sigma2))  # Fourier PS, 5-point smoothed.

fig = plt.figure(figsize=(4, 4))
ax = fig.add_axes([0.2, 0.2, 0.75, 0.75])
ax.plot(xs, ys, color='black')
ax.plot(lambdas, gws, lw=2, color='black')

gws = np.sum(wps_nj,0)/N  # Eq.22

# significance level.
from scipy.stats import chi2

gamma0 = 2.32   # Table 2. Morlet w0 = 6.
s2t = 4*np.pi/(w0+np.sqrt(2+w0**2))
sj = lambdas/s2t
pj = (1-alpha**2)/(1+alpha**2-2*alpha*np.cos(2*np.pi*dt/lambdas))
ax.plot(lambdas, pj, 'k--')
dof = 2*np.sqrt(1 + (N*dt/(gamma0*sj))**2)   # Eq (23).
chiv2 = chi2.ppf(confidence_level, dof) / dof
ys = pj * chiv2
ax.plot(lambdas, ys, 'k--')

ax.set(xscale='log', xlim=[64, 0.5], xlabel='Period (years)',
       ylim=[0, 15], ylabel='Variance ($\sigma^2$)')
ax.xaxis.set_major_locator(plt.LogLocator(base=2, numticks=8))
ax.xaxis.set_major_formatter(ticker.FormatStrFormatter('%.1f'))
ax.xaxis.set_minor_locator(plt.NullLocator());
```

The above plot reproduces Fig. 6 in TC98. The thick solid line is the global wavelet spectrum, whereas the thin solid line is the Fourier power spectrum after a 5-point running average. The lower and upper dashed lines are the mean red-noise spectrum and the 95% confidence level. Note that the 2 dashed lines here are based on the wavelet analysis, which are conceptually different from those in the section on Fourier analysis.

#### Confidence level for GWS

As discussed in Section 5.a in TC98, the red-noise spectrum for Fourier analysis applies to wavelet analysis. This is the same red-noise spectrum when we discuss the red contours for the wavelet power spectrum.
\begin{equation}
P_j = \frac{1-\alpha^2}{1+\alpha^2-2\alpha\cos(2\pi\delta t/\tau_j)},
\end{equation}
where the subscript $j$ indicates that $P_j$ maps to the scale index $j=0,1,\ldots,J-1$.

And the 95% confidence spectrum for the global wavelet spectrum is
\begin{equation}
    Q_j = P_j \;\chi^2(\nu_j,c),
\end{equation}
However, for GWS, the degree of freedom is a function of the scale index $j$:
\begin{equation}
\nu_j = \nu_0\sqrt{1+\left(\frac{N\delta t}{\gamma s_j}\right)^2},
\end{equation}
where $\gamma$ is a constant called **decorrelation factor** which depends on the choice of wavelet. According to Table 2 in TC98,
* Morlet. $\gamma=2.32$ for $\omega_0=6$.
* Paull. $\gamma=1.17$ for $m=4$.
* DOG. $\gamma=1.43$ for $m=2$ and $\gamma=1.37$ for $m=6$.

### Scale-averaged wavelet power (SWP)
Next, we examine the scale-averaged wavelet power
\begin{equation}
swp_n = \frac{\delta j\delta t}{C_\delta} \sum_{j=0}^{J-1} \frac{\left|W_{n,j}\right|^2}{s_j},
\end{equation}
for averaging over all scales; or
\begin{equation}
swp_n = \frac{\delta j\delta t}{C_\delta} \sum_{j=j_1}^{j_2} \frac{\left|W_{n,j}\right|^2}{s_j},
\end{equation}
for averaging from the scale index $j_1$ to $j_2$. The total number of scales get averaged is $n_a = j2-j1+1$.

```python
# Reproduce Fig. 8
mpl.rcParams['xtick.minor.visible'] = True
mpl.rcParams['xtick.minor.top'] = True
mpl.rcParams['xtick.minor.bottom'] = True
mpl.rcParams['xtick.minor.size'] = 2
mpl.rcParams['xtick.top'] = False
mpl.rcParams['xtick.minor.top'] = False
mpl.rcParams['ytick.right'] = False
mpl.rcParams['ytick.minor.right'] = False
mpl.rcParams['axes.spines.right'] = False
mpl.rcParams['axes.spines.top'] = False

Cdelta = 0.776  # Table 2.
dj0 = 0.60      # Table 2.
index = [i for i, s in enumerate(sj) if 2<=s<=8]
na = len(index)

# The scale-averaged wavelet power.
swp_n = dj*dt/Cdelta*np.sum(wps_nj[:,index]/sj[index],1)*sigma2   # Eq.24

# confidence level.
s_avg = 1/np.sum(1/sj[index])                   # Eq. 25
s_mid = sj[0]*np.power(2,0.5*(index[0]+index[-1])*dj)
dof = 2*na*s_avg/s_mid*np.sqrt(1+(na*dj/dj0)**2)# Eq. 28
pbar = s_avg * sum(pj[index]/sj[index])         # Eq. 27
chiv2 = chi2.ppf(confidence_level, dof) / dof
Q = (dj*dt/Cdelta/s_avg)*pbar*chiv2*sigma2      # Eq. 26

fig = plt.figure(figsize=(6,1.5))
ax = fig.add_axes([0.2, 0.2, 0.75, 0.75])
ax.plot(t0, swp_n, color='black')
ax.axhline(y=Q, color='black')
ax.set(xlim=(1870,2000), ylim=(0,0.8), xlabel='Year', ylabel='NINO3 ($^\circ C^2$)');
```

The above plot shows the scale-averaged wavelet power $swp_n$, which is a function of time $t_n$, for the subset of scales $s_j \in [2,8]$ yr.

#### Confidence level for SWP
The 95% confidence level for SWP is different from GWS and WPS. For SWP, the confidence level is a number $Q$, based on Equation (26) in TC98,
\begin{equation}
Q = \frac{\delta j\delta t\sigma^2}{C_\delta S_{avg}}\overline{P}\frac{\chi_\nu^2}{\nu},
\end{equation}
where from Equation (25) and (27) in TC98,
\begin{align}
S_{avg} &= 1/\sum_{j=j_1}^{j_2}1/s_j,\\
\overline{P} &= S_{avg}\sum_{j=j_1}^{j_2}\frac{P_j}{s_j},
\end{align}
$P_j$ is the same red-noise spectrum defined for GWS and WPS. $C_\delta$ is the **recontruction factor**, a constant depends on wavelet (Table 2 in TC98):
* Morlet. $C_\delta = 0.776$ for $\omega_0 = 6$.
* Paul. $C_\delta = 1.132$ for $m=4$.
* DOG. $C_\delta = 3.541$ for $m=2$, and $C_\delta = 1.966$ for $m=6$.

The $\nu$ again is the degree of freedom, but defined differently for SWP, from Equation (28) in TC98,
\begin{equation}
\nu = \frac{\nu_0 n_a S_{avg}}{S_{mid}} \sqrt{1+\left(\frac{n_a\delta j}{\delta j_0}\right)^2},
\end{equation}
where $n_a = j2-j1+1$, $S_{mid} = s_0\,2^{\delta j(j_1+j_2)/2}$, and $\delta j_0$ is the **factor for scale averaging**, a constant depends on wavelet (Table 2 in TC98):
* Morlet. $\delta j_0 = 0.60$ for $\omega_0 = 6$.
* Paul. $\delta j_0 = 1.5$ for $m=4$.
* DOG. $\delta j_0 = 1.4$ for $m=2$, and $\delta j_0 = 0.97$ for $m=6$.

# Summary
In this note, I reviewed
1. The Fourier transform, its red-noise spectrum and 95% confidence level;
2. The mother wavelet functions, including Morlet ($\omega_0=6$), Paul ($m=4$), DOG ($m=2,6$);
3. The continuous wavelet transform (CWT) in the time domain and frequency domain;
    1. The wavelet power spectrum (WPS) and its 95% confidence level;
    2. The global wavelet spectrum (GWS) and its 95% confidence level;
    3. The scale-averaged wavelet power (SWP) and its 95% confidence level.
Figures in TC98 are reproduced with Python code. The purpose of the code is to show every computation detail for the equations in the paper. Developing more abstract functions is not pursued here.

Materials for Section 6 in TC98 covers extensions to wavelet analysis, which I don't have the energy to pursue further.


# 15.4. Computing exact probabilities and manipulating random variables

```
from sympy import *
from sympy.stats import *
init_printing()
```

```
X, Y = Die('X', 6), Die('Y', 6)
```

```
P(Eq(X, 3))
```

```
P(X > 3)
```

```
P(X > Y)
```

```
P(X + Y > 6, X < 5)
```

```
Z = Normal('Z', 0, 1)  # Gaussian variable
```

```
P(Z > pi)
```

```
E(Z**2), variance(Z**2)
```

```
f = density(Z)
```

```
var('x')
f(x)
```

```
%matplotlib inline
plot(f(x), (x, -6, 6))
```

```
Eq(Integral(f(x), (x, pi, oo)),
   simplify(integrate(f(x), (x, pi, oo))))
```


# A Python Tour of Data Science: Data Exploitation

[Michaël Defferrard](http://deff.ch), *PhD student*, [EPFL](http://epfl.ch) [LTS2](http://lts2.epfl.ch)

The data `X.npy` and `y.npy` can be obtained by running the [data acquisition and exploration demo](01_demo_acquisition_exploration.ipynb).

```python
# Cross-platform (Windows / Mac / Linux) paths.
import os.path
folder = os.path.join('..', 'data', 'credit_card_defaults')

import numpy as np
X = np.load(os.path.join(folder, 'X.npy'))
y = np.load(os.path.join(folder, 'y.npy'))
n, d = X.shape
print('The data is a {} with {} samples of dimensionality {}.'.format(type(X), n, d))
```

## 1 Pre-Processing

Back to [NumPy](http://www.numpy.org/), the fundamental package for scientific computing with Python. It provides multi-dimensional arrays, data types and linear algebra routines. Note that [scikit-learn](http://scikit-learn.org) provides many helpers for those tasks.

Pre-processing usually consists of:
1. Data types transformation. The data has not necessarilly the format the chosen learning algorithm expects. This was done in the previous notebook before doing statistics with `statsmodels`.
1. Data normalization. Some algorithms expect data to be centered and scaled. Some will train faster.
1. Data randomization. If the samples are presented in sequence, it'll train faster if they are not correlated.
1. Train / test splitting. You may have to be careful here, e.g. not including future events in the training set.

```python
# Center and scale.
# Note: on a serious project, should be done after train / test split.
X = X.astype(np.float)
X -= X.mean(axis=0)
X /= X.std(axis=0)
```

```python
# Training and testing sets.
test_size = 10000
print('Split: {} testing and {} training samples'.format(test_size, y.size - test_size))
perm = np.random.permutation(y.size)
X_test  = X[perm[:test_size]]
X_train = X[perm[test_size:]]
y_test  = y[perm[:test_size]]
y_train = y[perm[test_size:]]
```

## 2 A first Predictive Model

The ingredients of a Machine Learning (ML) model are:
1. A predictive function, e.g. the linear transformation $f(x) = x^Tw + b$.
1. An error function, e.g. the least squares $E = \sum_{i=1}^n \left( f(x_i) - y_i \right)^2 = \| f(X) - y \|_2^2$.
1. An optional regularization, e.g. the Thikonov regularization $R = \|w\|_2^2$.
1. Which makes up the loss / objective function $L = E + \alpha R$.

Our model has a sole hyper-parameter, $\alpha \geq 0$, which controls the shrinkage.

A Machine Learning (ML) problem can often be cast as a (convex or smooth) optimization problem which objective is to find the parameters (here $w$ and $b$) who minimize the loss, e.g.
$$\hat{w}, \hat{b} = \operatorname*{arg min}_{w,b} L = \operatorname*{arg min}_{w,b} \| Xw + b - y \|_2^2 + \alpha \|w\|_2^2.$$

If the problem is convex and smooth, one can compute the gradients
$$\frac{\partial L}{\partial{w}} = 2 X^T (Xw+b-y) + 2\alpha w,$$
$$\frac{\partial L}{\partial{b}} = 2 \sum_{i=1}^n (x_i^Tw+b-y_i) = 2 \sum_{i=1}^n (x_i^Tw-y_i) + 2n \cdot b,$$

which can be used in a [gradient descent](https://en.wikipedia.org/wiki/Gradient_descent) scheme or to form closed-form solutions:
$$\frac{\partial L}{\partial{w}} = 0 \ \rightarrow \ 2 X^T X\hat{w} + 2\alpha \hat{w} = 2 X^T y - 2 X^T b \ \rightarrow \ \hat{w} = (X^T X + \alpha I)^{-1} X^T (y-b),$$
$$\frac{\partial L}{\partial{b}} = 0 \ \rightarrow \ 2n\hat{b} = 2\sum_{i=1}^n (y_i) - \underbrace{2\sum_{i=1}^n (x_i^Tw)}_{=0 \text{ if centered}} \ \rightarrow \ \hat{b} = \frac1n I^T y = \operatorname{mean}(y).$$

What if the resulting problem is non-smooth ? See the [PyUNLocBoX](http://pyunlocbox.readthedocs.io), a convex optimization toolbox which implements [proximal splitting methods](https://en.wikipedia.org/wiki/Proximal_gradient_method).

### 2.1 Take a *symbolic* Derivative

Let's verify our manually derived gradients ! [SymPy](http://www.sympy.org/) is our computer algebra system (CAS) (like [Mathematica](https://www.wolfram.com/mathematica), [Maple](https://www.maplesoft.com/products/Maple)) of choice.

```python
import sympy as sp
sp.init_printing()

X, y, w, b, a = sp.symbols('x y w b a')
L = (X*w + b - y)**2 + a*w**2

dLdw = sp.diff(L, w)
dLdb = sp.diff(L, b)

from IPython.display import display
display(L)
display(dLdw)
display(dLdb)
```

### 2.2 Build the Classifier

Relying on the derived equations, we can implement our model relying only on the [NumPy](http://www.numpy.org/) linear algebra capabilities (really wrappers to [BLAS](http://www.netlib.org/blas) / [LAPACK](http://www.netlib.org/lapack) implementations such as [ATLAS](http://math-atlas.sourceforge.net), [OpenBLAS](http://www.openblas.net) or [MKL](https://software.intel.com/intel-mkl)).

A ML model is best represented as a class, with hyper-parameters and parameters stored as attributes, and is composed of two essential methods:
1. `y_pred = model.predict(X_test)`: return the predictions $y$ given the features $X$.
1. `model.fit(X_train, y_train)`: learn the model parameters such as to predict $y$ given $X$.

```python
class RidgeRegression(object):
    """Our ML model."""

    def __init__(self, alpha=0):
        "The class' constructor. Initialize the hyper-parameters."
        self.a = alpha

    def predict(self, X):
        """Return the predicted class given the features."""
        return np.sign(X.dot(self.w) + self.b)

    def fit(self, X, y):
        """Learn the model's parameters given the training data, the closed-form way."""
        n, d = X.shape
        self.b = np.mean(y)
        Ainv = np.linalg.inv(X.T.dot(X) + self.a * np.identity(d))
        self.w = Ainv.dot(X.T).dot(y - self.b)

    def loss(self, X, y, w=None, b=None):
        """Return the current loss.
        This method is not strictly necessary, but it provides
        information on the convergence of the learning process."""
        w = self.w if w is None else w  # The ternary conditional operator
        b = self.b if b is None else b  # makes those tests concise.
        import autograd.numpy as np  # See below for autograd.
        return np.linalg.norm(np.dot(X, w) + b - y)**2 + self.a * np.linalg.norm(w, 2)**2
```

Now that our model can learn its parameters and predict targets, it's time to evaluate it. Our metric for binary classification is the accuracy, which gives the percentage of correcly classified test samples. Depending on the application, the time spent for inference or training might also be important.

```python
def accuracy(y_pred, y_true):
    """Our evaluation metric, the classification accuracy."""
    return np.sum(y_pred == y_true) / y_true.size

def evaluate(model):
    """Helper function to instantiate, train and evaluate the model.
    It returns the classification accuracy, the loss and the execution time."""
    import time
    t = time.process_time()
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    acc = accuracy(y_pred, y_test)
    loss = model.loss(X_test, y_test)
    t = time.process_time() - t
    print('accuracy: {:.2f}%, loss: {:.2f}, time: {:.2f}ms'.format(acc*100, loss, t*1000))
    return model

alpha = 1e-2*n
model = RidgeRegression(alpha)
evaluate(model)

models = []
models.append(model)
```

Okay we got around 80% accuracy with such a simple model ! Inference and training time looks good.

For those of you who don't now about numerical mathematics, solving a linear system of equations by inverting a matrix can be numerically instable. Let's do it the proper way and use a proper solver.

```python
def fit_lapack(self, X, y):
    """Better way (numerical stability): solve the linear system with LAPACK."""
    n, d = X.shape
    self.b = np.mean(y)
    A = X.T.dot(X) + self.a * np.identity(d)
    b = X.T.dot(y - self.b)
    self.w = np.linalg.solve(A, b)

# Let's monkey patch our object (Python is a dynamic language).
RidgeRegression.fit = fit_lapack

# Yeah just to be sure.
models.append(evaluate(RidgeRegression(alpha)))
assert np.allclose(models[-1].w, models[0].w)
```

### 2.3 Learning as Gradient Descent

Descending the gradient of our objective will lead us to a local minimum. If the objective is convex, that minimum will be global. Let's implement the gradient computed above and a simple gradient descent algorithm
$$w^{(t+1)} = w^{(t)} - \gamma \frac{\partial L}{\partial w}$$
where $\gamma$ is the learning rate, another hyper-parameter.

```python
class RidgeRegressionGradient(RidgeRegression):
    """This model inherits from `ridge_regression`. We overload the constructor, add a gradient
    function and replace the learning algorithm, but don't touch the prediction and loss functions."""

    def __init__(self, alpha=0, rate=0.1, niter=1000):
        """Here are new hyper-parameters: the learning rate and the number of iterations."""
        super().__init__(alpha)
        self.rate = rate
        self.niter = niter

    def grad(self, X, y, w):
        A = X.dot(w) + self.b - y
        return 2 * X.T.dot(A) + 2 * self.a * w

    def fit(self, X, y):
        n, d = X.shape
        self.b = np.mean(y)

        self.w = np.random.normal(size=d)
        for i in range(self.niter):
            self.w -= self.rate * self.grad(X, y, self.w)

            # Show convergence.
            if i % (self.niter//10) == 0:
                print('loss at iteration {}: {:.2f}'.format(i, self.loss(X, y)))

models.append(evaluate(RidgeRegressionGradient(alpha, 1e-6)))
```

Tyred of derivating gradients by hand ? Welcome [autograd](https://github.com/HIPS/autograd/), our tool of choice for [automatic differentation](https://en.wikipedia.org/wiki/Automatic_differentiation). Alternatives are [Theano](http://deeplearning.net/software/theano/) and [TensorFlow](https://www.tensorflow.org/).

```python
class RidgeRegressionAutograd(RidgeRegressionGradient):
    """Here we derive the gradient during construction and update the gradient function."""
    def __init__(self, *args):
        super().__init__(*args)
        from autograd import grad
        self.grad = grad(self.loss, argnum=2)

models.append(evaluate(RidgeRegressionAutograd(alpha, 1e-6)))
```

### 2.4 Learning as generic Optimization

Sometimes we don't want to implement the optimization by hand and would prefer a generic optimization algorithm. Let's make use of [SciPy](https://www.scipy.org/), which provides high-level algorithms for, e.g. [optimization](http://docs.scipy.org/doc/scipy/reference/optimize.html), [statistics](http://docs.scipy.org/doc/scipy/reference/stats.html), [interpolation](http://docs.scipy.org/doc/scipy/reference/tutorial/interpolate.html), [signal processing](http://docs.scipy.org/doc/scipy/reference/tutorial/signal.html), [sparse matrices](http://docs.scipy.org/doc/scipy/reference/sparse.html), [advanced linear algebra](http://docs.scipy.org/doc/scipy/reference/tutorial/linalg.html).

```python
class RidgeRegressionOptimize(RidgeRegressionGradient):

    def __init__(self, alpha=0, method=None):
        """Here's a new hyper-parameter: the optimization algorithm."""
        super().__init__(alpha)
        self.method = method

    def fit(self, X, y):
        """Fitted with a general purpose optimization algorithm."""
        n, d = X.shape
        self.b = np.mean(y)

        # Objective and gradient w.r.t. the variable to be optimized.
        f = lambda w: self.loss(X, y, w)
        jac = lambda w: self.grad(X, y, w)

        # Solve the problem !
        from scipy.optimize import minimize
        w0 = np.random.normal(size=d)
        res = minimize(f, w0, method=self.method, jac=jac)
        self.w = res.x

models.append(evaluate(RidgeRegressionOptimize(alpha, method='Nelder-Mead')))
models.append(evaluate(RidgeRegressionOptimize(alpha, method='BFGS')))
```

Accuracy may be lower (depending on the random initialization) as the optimization may not have converged to the global minima. Training time is however much longer ! Especially for gradient-less optimizers such as Nelder-Mead.

## 3 More interactivity

Interlude: the interactivity of Jupyter notebooks can be pushed forward with [IPython widgets](https://ipywidgets.readthedocs.io). Below, we construct a slider for the model hyper-parameter $\alpha$, which will train the model and print its performance at each change of the value. Handy when exploring the effects of hyper-parameters ! Although it's less usefull if the required computations are long.

```python
import ipywidgets
from IPython.display import clear_output

slider = ipywidgets.widgets.FloatSlider(
    value=-2,
    min=-4,
    max=2,
    step=1,
    description='log(alpha) / n',
)

def handle(change):
    """Handler for value change: fit model and print performance."""
    value = change['new']
    alpha = np.power(10, value) * n
    clear_output()
    print('alpha = {:.2e}'.format(alpha))
    evaluate(RidgeRegression(alpha))

slider.observe(handle, names='value')
display(slider)

slider.value = 1  # As if someone moved the slider.
```

## 4 Machine Learning made easier

Tired of writing algorithms ? Try [scikit-learn](http://scikit-learn.org), which provides many ML algorithms and related tools, e.g. metrics, cross-validation, model selection, feature extraction, pre-processing, for [predictive modeling](https://en.wikipedia.org/wiki/Predictive_modelling).

```python
from sklearn import linear_model, metrics

# The previously developed model: Ridge Regression.
model = linear_model.RidgeClassifier(alpha)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
models.append(model)

# Evaluate the predictions with a metric: the classification accuracy.
acc = metrics.accuracy_score(y_test, y_pred)
print('accuracy: {:.2f}%'.format(acc*100))

# It does indeed learn the same parameters.
assert np.allclose(models[-1].coef_, models[0].w, rtol=1e-1)
```

```python
# Let's try another model !
models.append(linear_model.LogisticRegression())
models[-1].fit(X_train, y_train)
acc = models[-1].score(X_test, y_test)
print('accuracy: {:.2f}%'.format(acc*100))
```

## 5 Deep Learning (DL)

Of course ! We got two low-level Python libraries: (1) [TensorFlow](https://www.tensorflow.org/) and (2) [Theano](http://deeplearning.net/software/theano/). Both of them treat data as tensors and construct a computational graph ([dataflow paradigm](https://en.wikipedia.org/wiki/Dataflow_programming)), composed of any mathematical expressions, that get evaluated on CPUs or GPUs. Theano is the pioneer and features an optimizing compiler which will turn the computational graph into efficient code. TensorFlow has a cleaner API (not need to define expressions as strings) and does not require a compilation step (which is painful when developing models).

While you'll only use Theano / TensorFlow to develop DL models, these are the higher-level libraries you'll use to define and test DL architectures on your problem:
* [Keras](https://keras.io/): TensorFlow & Theano backends
* [Lasagne](http://lasagne.readthedocs.io): Theano backend
* [nolearn](https://github.com/dnouri/nolearn): sklearn-like abstraction of Lasagne
* [Blocks](http://blocks.readthedocs.io): Theano backend
* [TFLearn](http://tflearn.org): TensorFlow backend

```python
import os
os.environ['KERAS_BACKEND'] = 'theano'
import keras

class NeuralNet(object):

    def __init__(self):
        """Define Neural Network architecture."""
        self.model = keras.models.Sequential()
        self.model.add(keras.layers.Dense(output_dim=46, input_dim=23, activation='relu'))
        self.model.add(keras.layers.Dense(output_dim=1, activation='sigmoid'))
        self.model.compile(loss='binary_crossentropy', optimizer='sgd', metrics=['accuracy'])

    def fit(self, X, y):
        y = y / 2 + 0.5  # [-1,1] -> [0,1]
        self.model.fit(X, y, nb_epoch=5, batch_size=32)

    def predict(self, X):
        classes = self.model.predict_classes(X, batch_size=32)
        return classes[:,0] * 2 - 1

models.append(NeuralNet())
models[-1].fit(X_train, y_train)

loss_acc = models[-1].model.evaluate(X_test, y_test/2+0.5, batch_size=32)
print('\n\nTesting set: {}'.format(loss_acc))
```

## 6 Evaluation

Now that we tried several predictive models, it is time to evaluate them with our chosen metrics and choose the one best suited to our particular problem. Let's plot the *classification accuracy* and the *prediction time* for each classifier with [matplotlib](http://matplotlib.org), the goto 2D plotting library for scientific Python. Its API is similar to matlab.

Result: The NeuralNet gives the best accuracy, by a small margin over the much simple logistic regression, but is the slowest method. Which to choose ? Again, it depends on your priorities.

```python
from matplotlib import pyplot as plt
plt.style.use('ggplot')
%matplotlib inline
# Or notebook for interaction.

names, acc, times = [], [], []
for model in models:
    import time
    t = time.process_time()
    y_pred = model.predict(X_test)
    times.append((time.process_time()-t) * 1000)
    acc.append(accuracy(y_pred, y_test) * 100)
    names.append(type(model).__name__)

plt.figure(figsize=(15,5))
plt.subplot(121)
plt.plot(acc, '.', markersize=20)
plt.title('Accuracy [%]')
plt.xticks(range(len(names)), names, rotation=90)

plt.subplot(122)
plt.plot(times, '.', markersize=20)
plt.title('Prediction time [ms]')
plt.xticks(range(len(names)), names, rotation=90)
plt.show()
```