# About this Notebook

This notebook mainly implements a Low-Rank Tensor Completion (LRTC) model with Truncated Nuclear Norm (TNN) regularization.

## Quick Run

This notebook is publicly available for any usage at our data imputation project. Please click [**transdim - GitHub**](https://github.com/xinychen/transdim).

## Low-Rank Tensor Completion

We start by importing the necessary dependencies.

```python
import numpy as np
from numpy.linalg import inv as inv
```

### Tensor Unfolding (`ten2mat`) and Matrix Folding (`mat2ten`)

Using numpy reshape to perform 3rd rank tensor unfold operation. [[**link**](https://stackoverflow.com/questions/49970141/using-numpy-reshape-to-perform-3rd-rank-tensor-unfold-operation)]

```python
def ten2mat(tensor, mode):
    return np.reshape(np.moveaxis(tensor, mode, 0), (tensor.shape[mode], -1), order = 'F')
```

```python
X = np.array([[[1, 2, 3, 4], [3, 4, 5, 6]],
              [[5, 6, 7, 8], [7, 8, 9, 10]],
              [[9, 10, 11, 12], [11, 12, 13, 14]]])
print('tensor size:')
print(X.shape)
print('original tensor:')
print(X)
print()
print('(1) mode-1 tensor unfolding:')
print(ten2mat(X, 0))
print()
print('(2) mode-2 tensor unfolding:')
print(ten2mat(X, 1))
print()
print('(3) mode-3 tensor unfolding:')
print(ten2mat(X, 2))
```

    tensor size:
    (3, 2, 4)
    original tensor:
    [[[ 1  2  3  4]
      [ 3  4  5  6]]

     [[ 5  6  7  8]
      [ 7  8  9 10]]

     [[ 9 10 11 12]
      [11 12 13 14]]]

    (1) mode-1 tensor unfolding:
    [[ 1  3  2  4  3  5  4  6]
     [ 5  7  6  8  7  9  8 10]
     [ 9 11 10 12 11 13 12 14]]

    (2) mode-2 tensor unfolding:
    [[ 1  5  9  2  6 10  3  7 11  4  8 12]
     [ 3  7 11  4  8 12  5  9 13  6 10 14]]

    (3) mode-3 tensor unfolding:
    [[ 1  5  9  3  7 11]
     [ 2  6 10  4  8 12]
     [ 3  7 11  5  9 13]
     [ 4  8 12  6 10 14]]

```python
def mat2ten(mat, tensor_size, mode):
    index = list()
    index.append(mode)
    for i in range(tensor_size.shape[0]):
        if i != mode:
            index.append(i)
    return np.moveaxis(np.reshape(mat, list(tensor_size[index]), order = 'F'), 0, mode)
```

### Singular Value Thresholding (SVT) for TNN

```python
def svt_tnn(mat, alpha, rho, theta):
    """This is a Numpy dependent singular value thresholding (SVT) process."""
    u, s, v = np.linalg.svd(mat, full_matrices = 0)
    vec = s.copy()
    vec[theta :] = s[theta :] - alpha / rho
    vec[vec < 0] = 0
    return np.matmul(np.matmul(u, np.diag(vec)), v)
```

**Potential alternative for this**:

If you prefer, there is a competitively efficient singular value thresholding (SVT) process which uses the randomized SVD (`randomized_svd`) of `sklearn`.

```python
from sklearn.utils.extmath import randomized_svd
```

```python
def svt_tnn(mat, alpha, rho, theta):
    """This is a sklearn dependent singular value thresholding (SVT) process."""
    u, s, v = randomized_svd(mat, n_components = min(mat.shape[0], mat.shape[1]), n_iter = 1)
    vec = s.copy()
    vec[theta :] = s[theta :] - alpha / rho
    vec[vec < 0] = 0
    return np.matmul(np.matmul(u, np.diag(vec)), v)
```

**Understanding these codes**:

- **`line 1`**: Necessary inputs including any input matrix $\boldsymbol{X}$, weight of Truncated Nuclear Norm (TNN) regularization $\alpha$, learning rate $\rho$, and positive integer number $\theta$ for nuclear norm truncation.

- **`line 2`**: Compute the Singular Value Decomposition (SVD) for any matrix $\boldsymbol{X}$ with `numpy.linalg.svd` (i.e., SVD function in `Numpy`'s linear algebra package).

- **`line 3-5`**: Truncate singular values $\sigma_{\theta+1},...$ with the following rule:

\begin{equation}
\sigma_{i}=\left[\sigma_{i}(\boldsymbol{X})-\frac{\alpha}{\rho}\right]_{+}.
\end{equation}

- **`line 6`**: Return the resulted matrix.

### Define Performance Metrics

- **RMSE**
- **MAPE**

```python
def Compute_RMSE(var, var_hat):
    return np.sqrt(np.sum((var - var_hat) ** 2) / var.shape[0])
```

```python
def Compute_MAPE(var, var_hat):
    return np.sum(np.abs(var - var_hat) / var) / var.shape[0]
```

### Define LRTC-TNN Function with `Numpy`

```python
def LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, maxiter):
    """Low-Rank Tenor Completion with Truncated Nuclear Norm, LRTC-TNN."""

    dim = np.array(sparse_tensor.shape)
    pos_train = np.where(sparse_tensor != 0)
    pos_missing = np.where(sparse_tensor == 0)
    pos_test = np.where((dense_tensor != 0) & (sparse_tensor == 0))

    X = np.zeros(np.insert(dim, 0, len(dim))) # \boldsymbol{\mathcal{X}}
    T = np.zeros(np.insert(dim, 0, len(dim))) # \boldsymbol{\mathcal{T}}
    Z = sparse_tensor.copy()
    for it in range(maxiter):
        for k in range(len(dim)):
            X[k] = mat2ten(svt_tnn(ten2mat(Z - T[k] / rho, k), alpha[k], rho, np.int(np.ceil(theta * dim[k]))), dim, k)
        Z[pos_missing] = np.mean(X + T / rho, axis = 0)[pos_missing]
        T = T + rho * (X - np.broadcast_to(Z, np.insert(dim, 0, len(dim))))
        tensor_hat = np.einsum('k, kmnt -> mnt', alpha, X)
        if (it + 1) % 50 == 0:
            print('Iter: {}'.format(it + 1))
            print('RMSE: {:.6}'.format(Compute_RMSE(dense_tensor[pos_test], tensor_hat[pos_test])))
            print()

    print('Imputation MAPE: {:.6}'.format(Compute_MAPE(dense_tensor[pos_test], tensor_hat[pos_test])))
    print('Imputation RMSE: {:.6}'.format(Compute_RMSE(dense_tensor[pos_test], tensor_hat[pos_test])))
    print()

    return tensor_hat
```

**Understanding these codes**:

- **`line 18-19`**: Update $\boldsymbol{\mathcal{Z}}_{k}^{l+1},k=1,2,3$.

- **`line 20-22`**: Update $\boldsymbol{\mathcal{X}}_{k}^{l+1}$ by

\begin{equation}
\boldsymbol{\mathcal{X}}_{k}^{l+1}=\mathcal{P}_{\Omega}(\boldsymbol{\mathcal{Y}})+\mathcal{P}_{\Omega}^{\perp}\left(\boldsymbol{\mathcal{Z}}_{k}^{l+1}-\frac{1}{\rho}\boldsymbol{\mathcal{T}}_{k}^{l}\right),k=1,2,3.
\end{equation}

- **`line 23`**: Update $\boldsymbol{\mathcal{T}}_{k}^{l+1}$ by

\begin{equation}
\boldsymbol{\mathcal{T}}_{k}^{l+1}=\boldsymbol{\mathcal{T}}_{k}^{l}+\rho_k\left(\boldsymbol{\mathcal{X}}_{k}^{l+1}-\boldsymbol{\mathcal{Z}}_{k}^{l+1}\right).
\end{equation}

## Data Organization

### 1) Matrix Structure

We consider a dataset of $m$ discrete time series $\boldsymbol{y}_{i}\in\mathbb{R}^{f},i\in\left\{1,2,...,m\right\}$. The time series may have missing elements. We express spatio-temporal dataset as a matrix $Y\in\mathbb{R}^{m\times f}$ with $m$ rows (e.g., locations) and $f$ columns (e.g., discrete time intervals),

$$Y=\left[ \begin{array}{cccc} y_{11} & y_{12} & \cdots & y_{1f} \\ y_{21} & y_{22} & \cdots & y_{2f} \\ \vdots & \vdots & \ddots & \vdots \\ y_{m1} & y_{m2} & \cdots & y_{mf} \\ \end{array} \right]\in\mathbb{R}^{m\times f}.$$

### 2) Tensor Structure

We consider a dataset of $m$ discrete time series $\boldsymbol{y}_{i}\in\mathbb{R}^{nf},i\in\left\{1,2,...,m\right\}$. The time series may have missing elements. We partition each time series into intervals of predifined length $f$. We express each partitioned time series as a matrix $Y_{i}$ with $n$ rows (e.g., days) and $f$ columns (e.g., discrete time intervals per day),

$$Y_{i}=\left[ \begin{array}{cccc} y_{11} & y_{12} & \cdots & y_{1f} \\ y_{21} & y_{22} & \cdots & y_{2f} \\ \vdots & \vdots & \ddots & \vdots \\ y_{n1} & y_{n2} & \cdots & y_{nf} \\ \end{array} \right]\in\mathbb{R}^{n\times f},i=1,2,...,m,$$

therefore, the resulting structure is a tensor $\mathcal{Y}\in\mathbb{R}^{m\times n\times f}$.

## Missing Data Imputation

In the following, we apply the above defined TRMF function to the task of missing data imputation task on the following spatiotemporal multivariate time series datasets/matrices:

- **Guangzhou data set**: [Guangzhou urban traffic speed data set](https://doi.org/10.5281/zenodo.1205228).
- **Birmingham data set**: [Birmingham parking data set](https://archive.ics.uci.edu/ml/datasets/Parking+Birmingham).
- **Hangzhou data set**: [Hangzhou metro passenger flow data set](https://doi.org/10.5281/zenodo.3145403).
- **Settle data set**: [Seattle freeway traffic speed data set](https://github.com/zhiyongc/Seattle-Loop-Data).

The original data sets have been adapted into our experiments, and it is now available at the fold of `datasets`.

### Experiments on Guangzhou Data Set

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_matrix = scipy.io.loadmat('../datasets/Guangzhou-data-set/random_matrix.mat')
random_matrix = random_matrix['random_matrix']
random_tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/random_tensor.mat')
random_tensor = random_tensor['random_tensor']

missing_rate = 0.2

# =============================================================================
### Random missing (RM) scenario:
binary_tensor = np.round(random_tensor + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 0.002
theta = 0.30
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 2.84958

    Iter: 100
    RMSE: 2.84361

    Iter: 150
    RMSE: 2.84324

    Iter: 200
    RMSE: 2.84309

    Imputation MAPE: 0.0662603
    Imputation RMSE: 2.84309

    Running time: 127 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_matrix = scipy.io.loadmat('../datasets/Guangzhou-data-set/random_matrix.mat')
random_matrix = random_matrix['random_matrix']
random_tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/random_tensor.mat')
random_tensor = random_tensor['random_tensor']

missing_rate = 0.4

# =============================================================================
### Random missing (RM) scenario:
binary_tensor = np.round(random_tensor + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 0.002
theta = 0.30
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 7.74311

    Iter: 100
    RMSE: 3.10197

    Iter: 150
    RMSE: 3.11387

    Iter: 200
    RMSE: 3.11496

    Imputation MAPE: 0.0722136
    Imputation RMSE: 3.11496

    Running time: 142 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_matrix = scipy.io.loadmat('../datasets/Guangzhou-data-set/random_matrix.mat')
random_matrix = random_matrix['random_matrix']
random_tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/random_tensor.mat')
random_tensor = random_tensor['random_tensor']

missing_rate = 0.2

# =============================================================================
### Non-random missing (NM) scenario:
binary_tensor = np.zeros(dense_tensor.shape)
for i1 in range(dense_tensor.shape[0]):
    for i2 in range(dense_tensor.shape[1]):
        binary_tensor[i1, i2, :] = np.round(random_matrix[i1, i2] + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 0.002
theta = 0.05
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 6.57525

    Iter: 100
    RMSE: 4.21873

    Iter: 150
    RMSE: 3.97082

    Iter: 200
    RMSE: 3.97132

    Imputation MAPE: 0.0938764
    Imputation RMSE: 3.97132

    Running time: 134 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_matrix = scipy.io.loadmat('../datasets/Guangzhou-data-set/random_matrix.mat')
random_matrix = random_matrix['random_matrix']
random_tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/random_tensor.mat')
random_tensor = random_tensor['random_tensor']

missing_rate = 0.4

# =============================================================================
### Non-random missing (NM) scenario:
binary_tensor = np.zeros(dense_tensor.shape)
for i1 in range(dense_tensor.shape[0]):
    for i2 in range(dense_tensor.shape[1]):
        binary_tensor[i1, i2, :] = np.round(random_matrix[i1, i2] + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 0.0005
theta = 0.05
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 4.07891

    Iter: 100
    RMSE: 4.06736

    Iter: 150
    RMSE: 4.06617

    Iter: 200
    RMSE: 4.06606

    Imputation MAPE: 0.0956982
    Imputation RMSE: 4.06606

    Running time: 123 seconds

**Experiment results** of missing data imputation using LRTC-TNN:

|  scenario |`alpha` (vector input)|`rho`|`theta`|`maxiter`|       mape |      rmse |
|:----------|-----:|---------:|---------:|-------- --:|-------- --:|----------:|
|**0.2, RM**| $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$ | 0.002 | 0.30 | 200 | **0.0663** | **2.84**|
|**0.4, RM**| $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$ | 0.002 | 0.30 | 200 | **0.0722** | **3.11**|
|**0.2, NM**| $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$ | 0.002 | 0.05 | 200 | **0.0939** | **3.97**|
|**0.4, NM**| $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$ |0.0005 | 0.05 | 200 | **0.0957** | **4.07**|

### Experiments on Birmingham Data Set

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Birmingham-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_matrix = scipy.io.loadmat('../datasets/Birmingham-data-set/random_matrix.mat')
random_matrix = random_matrix['random_matrix']
random_tensor = scipy.io.loadmat('../datasets/Birmingham-data-set/random_tensor.mat')
random_tensor = random_tensor['random_tensor']

missing_rate = 0.1

# =============================================================================
### Random missing (RM) scenario
### Set the RM scenario by:
binary_tensor = np.round(random_tensor + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 0.0005
theta = 0.20
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 135.489

    Iter: 100
    RMSE: 12.3854

    Iter: 150
    RMSE: 12.3673

    Iter: 200
    RMSE: 12.3742

    Imputation MAPE: 0.0411626
    Imputation RMSE: 12.3742

    Running time: 2 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Birmingham-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_matrix = scipy.io.loadmat('../datasets/Birmingham-data-set/random_matrix.mat')
random_matrix = random_matrix['random_matrix']
random_tensor = scipy.io.loadmat('../datasets/Birmingham-data-set/random_tensor.mat')
random_tensor = random_tensor['random_tensor']

missing_rate = 0.3

# =============================================================================
### Random missing (RM) scenario
### Set the RM scenario by:
binary_tensor = np.round(random_tensor + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 0.0005
theta = 0.15
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 281.746

    Iter: 100
    RMSE: 109.418

    Iter: 150
    RMSE: 21.0593

    Iter: 200
    RMSE: 17.507

    Imputation MAPE: 0.0521031
    Imputation RMSE: 17.507

    Running time: 2 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Birmingham-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_matrix = scipy.io.loadmat('../datasets/Birmingham-data-set/random_matrix.mat')
random_matrix = random_matrix['random_matrix']
random_tensor = scipy.io.loadmat('../datasets/Birmingham-data-set/random_tensor.mat')
random_tensor = random_tensor['random_tensor']

missing_rate = 0.1

# =============================================================================
### Non-random missing (NM) scenario
### Set the RM scenario by:
binary_tensor = np.zeros(dense_tensor.shape)
for i1 in range(dense_tensor.shape[0]):
    for i2 in range(dense_tensor.shape[1]):
        binary_tensor[i1, i2, :] = np.round(random_matrix[i1, i2] + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 0.0002
theta = 0.10
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 225.961

    Iter: 100
    RMSE: 20.8381

    Iter: 150
    RMSE: 20.8942

    Iter: 200
    RMSE: 20.9185

    Imputation MAPE: 0.0877233
    Imputation RMSE: 20.9185

    Running time: 2 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Birmingham-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_matrix = scipy.io.loadmat('../datasets/Birmingham-data-set/random_matrix.mat')
random_matrix = random_matrix['random_matrix']
random_tensor = scipy.io.loadmat('../datasets/Birmingham-data-set/random_tensor.mat')
random_tensor = random_tensor['random_tensor']

missing_rate = 0.3

# =============================================================================
### Non-random missing (NM) scenario
### Set the RM scenario by:
binary_tensor = np.zeros(dense_tensor.shape)
for i1 in range(dense_tensor.shape[0]):
    for i2 in range(dense_tensor.shape[1]):
        binary_tensor[i1, i2, :] = np.round(random_matrix[i1, i2] + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 0.0002
theta = 0.05
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 416.54

    Iter: 100
    RMSE: 186.726

    Iter: 150
    RMSE: 57.7627

    Iter: 200
    RMSE: 47.65

    Imputation MAPE: 0.13257
    Imputation RMSE: 47.65

    Running time: 2 seconds

**Experiment results** of missing data imputation using LRTC-TNN:

|  scenario |`alpha` (vector input)|`rho` | `theta` |`maxiter`|       mape |      rmse |
|:----------|-----:|---------:|---------:|-------- --:|-------- --:|----------:|
|**0.1, RM**| $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$ | 0.0005 | 0.20 | 200 | **0.0412** | **12.37**|
|**0.3, RM**| $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$ | 0.0005 | 0.15 | 200 | **0.0521** | **17.51**|
|**0.1, NM**| $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$ | 0.0002 | 0.10 | 200 | **0.0877** | **20.92**|
|**0.3, NM**| $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$ | 0.0002 | 0.05 | 200 | **0.1326** | **47.65**|

### Experiments on Hangzhou Data Set

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Hangzhou-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_matrix = scipy.io.loadmat('../datasets/Hangzhou-data-set/random_matrix.mat')
random_matrix = random_matrix['random_matrix']
random_tensor = scipy.io.loadmat('../datasets/Hangzhou-data-set/random_tensor.mat')
random_tensor = random_tensor['random_tensor']

missing_rate = 0.2

# =============================================================================
### Random missing (RM) scenario
### Set the RM scenario by:
binary_tensor = np.round(random_tensor + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 0.0002
theta = 0.10
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 39.063

    Iter: 100
    RMSE: 24.9086

    Iter: 150
    RMSE: 24.6781

    Iter: 200
    RMSE: 24.6533

    Imputation MAPE: 0.181488
    Imputation RMSE: 24.6533

    Running time: 13 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Hangzhou-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_matrix = scipy.io.loadmat('../datasets/Hangzhou-data-set/random_matrix.mat')
random_matrix = random_matrix['random_matrix']
random_tensor = scipy.io.loadmat('../datasets/Hangzhou-data-set/random_tensor.mat')
random_tensor = random_tensor['random_tensor']

missing_rate = 0.4

# =============================================================================
### Random missing (RM) scenario
### Set the RM scenario by:
binary_tensor = np.round(random_tensor + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 0.0002
theta = 0.10
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 48.7718

    Iter: 100
    RMSE: 35.2227

    Iter: 150
    RMSE: 26.5963

    Iter: 200
    RMSE: 25.7474

    Imputation MAPE: 0.187988
    Imputation RMSE: 25.7474

    Running time: 13 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Hangzhou-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_matrix = scipy.io.loadmat('../datasets/Hangzhou-data-set/random_matrix.mat')
random_matrix = random_matrix['random_matrix']
random_tensor = scipy.io.loadmat('../datasets/Hangzhou-data-set/random_tensor.mat')
random_tensor = random_tensor['random_tensor']

missing_rate = 0.2

# =============================================================================
### Non-random missing (NM) scenario
### Set the NM scenario by:
binary_tensor = np.zeros(dense_tensor.shape)
for i1 in range(dense_tensor.shape[0]):
    for i2 in range(dense_tensor.shape[1]):
        binary_tensor[i1, i2, :] = np.round(random_matrix[i1, i2] + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 0.0002
theta = 0.05
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 66.9782

    Iter: 100
    RMSE: 32.2313

    Iter: 150
    RMSE: 31.6912

    Iter: 200
    RMSE: 31.7906

    Imputation MAPE: 0.20379
    Imputation RMSE: 31.7906

    Running time: 11 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Hangzhou-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_matrix = scipy.io.loadmat('../datasets/Hangzhou-data-set/random_matrix.mat')
random_matrix = random_matrix['random_matrix']
random_tensor = scipy.io.loadmat('../datasets/Hangzhou-data-set/random_tensor.mat')
random_tensor = random_tensor['random_tensor']

missing_rate = 0.4

# =============================================================================
### Non-random missing (NM) scenario
### Set the NM scenario by:
binary_tensor = np.zeros(dense_tensor.shape)
for i1 in range(dense_tensor.shape[0]):
    for i2 in range(dense_tensor.shape[1]):
        binary_tensor[i1, i2, :] = np.round(random_matrix[i1, i2] + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 0.0002
theta = 0.05
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 83.7915

    Iter: 100
    RMSE: 54.7268

    Iter: 150
    RMSE: 33.813

    Iter: 200
    RMSE: 32.3401

    Imputation MAPE: 0.210977
    Imputation RMSE: 32.3401

    Running time: 13 seconds

**Experiment results** of missing data imputation using LRTC-TNN:

|  scenario |`alpha` (vector input)|`rho`|`theta`|`maxiter`|       mape |      rmse |
|:----------|-----:|---------:|---------:|--------:|----------:|----------:|
|**0.2, RM**| $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$ | 0.0005 | 0.05 | 200 | **0.1792** | **25.00**|
|**0.4, RM**| $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$ | 0.0005 | 0.05 | 200 | **0.1857** | **26.15**|
|**0.2, NM**| $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$ | 0.0005 | 0.04 | 200 | **0.2033** | **27.85**|
|**0.4, NM**| $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$ | 0.0005 | 0.04 | 200 | **0.2093** | **29.89**|

### Experiments on Seattle Data Set

```python
import pandas as pd

dense_mat = pd.read_csv('../datasets/Seattle-data-set/mat.csv', index_col = 0)
RM_mat = pd.read_csv('../datasets/Seattle-data-set/RM_mat.csv', index_col = 0)
dense_mat = dense_mat.values
RM_mat = RM_mat.values
dense_tensor = dense_mat.reshape([dense_mat.shape[0], 28, 288])
RM_tensor = RM_mat.reshape([RM_mat.shape[0], 28, 288])

missing_rate = 0.2

# =============================================================================
### Random missing (RM) scenario
### Set the RM scenario by:
binary_tensor = np.round(RM_tensor + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 0.001
theta = 0.30
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 3.01931

    Iter: 100
    RMSE: 3.01648

    Iter: 150
    RMSE: 3.01518

    Iter: 200
    RMSE: 3.01493

    Imputation MAPE: 0.0460414
    Imputation RMSE: 3.01493

    Running time: 228 seconds

```python
import pandas as pd

dense_mat = pd.read_csv('../datasets/Seattle-data-set/mat.csv', index_col = 0)
RM_mat = pd.read_csv('../datasets/Seattle-data-set/RM_mat.csv', index_col = 0)
dense_mat = dense_mat.values
RM_mat = RM_mat.values
dense_tensor = dense_mat.reshape([dense_mat.shape[0], 28, 288])
RM_tensor = RM_mat.reshape([RM_mat.shape[0], 28, 288])

missing_rate = 0.4

# =============================================================================
### Random missing (RM) scenario
### Set the RM scenario by:
binary_tensor = np.round(RM_tensor + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 0.001
theta = 0.30
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 3.22771

    Iter: 100
    RMSE: 3.22321

    Iter: 150
    RMSE: 3.22076

    Iter: 200
    RMSE: 3.21982

    Imputation MAPE: 0.0501131
    Imputation RMSE: 3.21982

    Running time: 214 seconds

```python
import pandas as pd

dense_mat = pd.read_csv('../datasets/Seattle-data-set/mat.csv', index_col = 0)
NM_mat = pd.read_csv('../datasets/Seattle-data-set/NM_mat.csv', index_col = 0)
dense_mat = dense_mat.values
NM_mat = NM_mat.values
dense_tensor = dense_mat.reshape([dense_mat.shape[0], 28, 288])

missing_rate = 0.2

# =============================================================================
### Non-random missing (NM) scenario
### Set the NM scenario by:
binary_tensor = np.zeros((dense_mat.shape[0], 28, 288))
for i1 in range(binary_tensor.shape[0]):
    for i2 in range(binary_tensor.shape[1]):
        binary_tensor[i1, i2, :] = np.round(NM_mat[i1, i2] + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 0.0005
theta = 0.05
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 4.20882

    Iter: 100
    RMSE: 4.18975

    Iter: 150
    RMSE: 4.18834

    Iter: 200
    RMSE: 4.18746

    Imputation MAPE: 0.069471
    Imputation RMSE: 4.18746

    Running time: 217 seconds

```python
import pandas as pd

dense_mat = pd.read_csv('../datasets/Seattle-data-set/mat.csv', index_col = 0)
NM_mat = pd.read_csv('../datasets/Seattle-data-set/NM_mat.csv', index_col = 0)
dense_mat = dense_mat.values
NM_mat = NM_mat.values
dense_tensor = dense_mat.reshape([dense_mat.shape[0], 28, 288])

missing_rate = 0.4

# =============================================================================
### Non-random missing (NM) scenario
### Set the NM scenario by:
binary_tensor = np.zeros((dense_mat.shape[0], 28, 288))
for i1 in range(binary_tensor.shape[0]):
    for i2 in range(binary_tensor.shape[1]):
        binary_tensor[i1, i2, :] = np.round(NM_mat[i1, i2] + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 0.0005
theta = 0.05
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 15.0981

    Iter: 100
    RMSE: 4.55854

    Iter: 150
    RMSE: 4.51553

    Iter: 200
    RMSE: 4.51677

    Imputation MAPE: 0.0762777
    Imputation RMSE: 4.51677

    Running time: 229 seconds

**Experiment results** of missing data imputation using LRTC-TNN:

|  scenario |`alpha` (vector input)|`rho`|`theta`|`maxiter`|       mape |      rmse |
|:----------|-----:|---------:|---------:|---------:|-------- --:|----------:|
|**0.2, RM**| $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$ | 0.005 | 0.30 | 200 | **0.0467** | **3.07**|
|**0.4, RM**| $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$ | 0.001 | 0.30 | 200 | **0.0507** | **3.24**|
|**0.2, NM**| $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$ |0.0005 | 0.05 | 200 | **0.0720** | **4.30**|
|**0.4, NM**| $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$ |0.0005 | 0.05 | 200 | **0.0770** | **4.55**|


#**Continuous Optimisation Exam Autumn 2019**
#*Exercise* 1.

```python
from sympy import *
from sympy.calculus.util import is_convex
from sympy.abc import x, y, z

init_printing()
```

# Minimization under inequality constraints

We are dealing with the following problem on $\mathbb{R}^3$ :

\begin{align}
min f(x, y,z) = x^2+2y^2+3z^2-4x-4y
\end{align}

\begin{align}
 g(x,y,z) = x+y+z-3
\end{align}

```python
#f = x**2 + y**2 + x*y + x
#g = x + y - 1

f = x**2+2*y**2+3*z**2-4*x-4*y
g = x+y+z-3

X = f.free_symbols
```

```python
f
```

```python
g
```

```python
if len(X) == 2:
  plotting.plot3d(f, (x, -10, 10), (y, -10, 10),size=(10,7));
```

## 1. Is the set $K = {(x, y, z)/x + y + z ≤ 3}$ closed in $\mathbb{R}^3$ ? open ? convex ? (explain in two words for each question), how would you call this set ?

If we have a subset $K$ such that $K=\{\forall$ $X \in \mathbb{R}^n | g(X)\}\leq 0$

$K$ is **closed** for any continuous $g$

$K = {(x, y, z)/x + y + z ≤ 3}$

the boundary of $K$ is $x + y + z ≤ 3$

For any point of $K$ there should exist a ball $\mathcal B(X,ϵ)$ included in $K$

$\forall X\in K,\quad \exists \epsilon$ such that $\mathcal B(X,ϵ) \subset K$

**intuition**: the $\color{blue}{points}$ at the border does not satisfy the **opennes** condition

=>$K$ is not **open** (large inequality)

Convexity definition:

$\forall X_1,X_2 \in K \quad \forall θ \in [0\,;1] \qquad θ\,X_1 + (1-\theta)\,X_2 \in K$

## 2. Is the function $f(x, y, z)$ convex ? strictly convex ? α-convex ? (if yes give the value of $\alpha$)

```python
for index in X:
  if is_convex(f,index):
    print("the function if convex for the variable", index)
```

    the function if convex for the variable y
    the function if convex for the variable x
    the function if convex for the variable z

```python
GradOf_f = [f.diff(x) for x in X]
Grad_f = Matrix(GradOf_f)
Grad_f
```

$\displaystyle \left[\begin{matrix}4 y - 4\\2 x - 4\\6 z\end{matrix}\right]$

```python
GradOf_g = [g.diff(x) for x in X]
Grad_g = Matrix(GradOf_g)
Grad_g
```

$\displaystyle \left[\begin{matrix}1\\1\\1\end{matrix}\right]$

```python
H = hessian(f, list(X))
H
```

$\displaystyle \left[\begin{matrix}4 & 0 & 0\\0 & 2 & 0\\0 & 0 & 6\end{matrix}\right]$

```python
eigenvalues = H.eigenvals()
eigenvalues
```

```python
alpha = min(eigenvalues)
alpha
```

```python
if alpha > 0:
  print("the function is α-convex with α =", alpha)
  print(alpha,"-convexity implies strict convexity")
else:
  print("the function is not α-convex with α =", alpha)

```

    the function is α-convex with α = 2
    2 -convexity implies strict convexity

#3. Justify the existence and uniqueness of the solution of $(P)$.

**Existence of a minimum**
1. ✔$K$ is non empty $\quad \Rightarrow$ trivial $(0,0,0)$ belongs to $K$
1. ✔$K$ closed $\quad \Rightarrow$ already proven ($K$ is **closed** for any continuous $g$)
1. ✔$f$ is continous
1. ❌$K$ is bounded
> if $K$ not bounded, is $f$ $∞$ at infinity? (i.e when $\|X\| → \infty$ does $\|f(X)\| → \infty$ ?)
>>easy to demonstrate that at some point $\|f(X)\|^2>\|X\|^2$

**Uniqueness of a minimum**
1. ✔$f$ is strictly convex
1. ✔$K$ is convex

## 4. Apply the Kuhn-Tucker theorem : are the hypotheses verified ? write the KuhnTucker relations.

**hypotheses**
1. ✔$f$ and $g$ convex $\Rightarrow$ $f$ is 2-convex and $g$ is linear
1. ✔differentiable on $K$ $\Rightarrow$ polynoms are differentiable

$f' + λ.g' = 0$

$λ.g = 0$

$λ\geq0$

$g \leq 0$

```python
lam = symbols('λ', real = True)
L = f + lam* g
L
```

```python
XLamb = L.free_symbols
XLamb
```

```python
gradL = [diff(L,x) for x in XLamb] # gradient of Lagrangian w.r.t. (x,y,z, etc.)
Matrix(gradL)
```

$\displaystyle \left[\begin{matrix}x + y + z - 3\\2 x + λ - 4\\4 y + λ - 4\\6 z + λ\end{matrix}\right]$

```python
stationary_points=solve(gradL)
stationary_points
```

```python
Min_f = f.subs(stationary_points)
print("the minimal value of the function f on K is ", Min_f)
```

    the minimal value of the function f on K is  -6

We have proved before that there exists only a minimum,

*Alternate solution with the matrix*

```python
C = Matrix([[0,4,0,1], [0,0,6,1],[1,1,1,0],[2,0,0,1]])
D = Matrix([4,0,3,4])
C, D
```

$\displaystyle \left( \left[\begin{matrix}0 & 4 & 0 & 1\\0 & 0 & 6 & 1\\1 & 1 & 1 & 0\\2 & 0 & 0 & 1\end{matrix}\right], \  \left[\begin{matrix}4\\0\\3\\4\end{matrix}\right]\right)$

```python
C.inv()*D
```

$\displaystyle \left[\begin{matrix}2\\1\\0\\0\end{matrix}\right]$


# Convex approximate dynamic programming

We consider a stochastic control problem of the form
\begin{equation}
\begin{array}{ll}
\mbox{minimize} & \underset{T \to \infty}\lim {\mathbb E} \left[\frac{1}{T} \sum_{t=0}^{T-1} \|{x_t}\|_2^2 + \|{\phi(x_t)}\|_2^2\right]\\[.2cm]
\mbox{subject to} & x_{t+1} = Ax_t + B\phi(x_t) + \omega_t,
\end{array}
\label{eq:adp}
\end{equation}
where $x_t\in\mathbf{R}^n$ is the state, $\phi:\mathbf{R}^n \to \mathcal U \subseteq \mathbf{R}^m$ is
the policy, $\mathcal U$ is a convex set representing the allowed set of controls,
and $\omega_t\in\Omega$ is a (random, i.i.d.) disturbance.
Here the variable is the policy $\phi$, and the expectation is taken over
disturbances and the initial state $x_0$. If $\mathcal U$ is not an affine
set, then this problem is in general very difficult to solve.

A common heuristic for solving stochastic control problems is
approximate dynamic programming (ADP), which parametrizes $\phi$
and replaces the minimization over functions $\phi$ with a minimization over parameters.
In this example, we take $\mathcal U$ to be the unit ball and we represent $\phi$
as a particular quadratic *control-Lyapunov* policy.
Evaluating $\phi$ corresponds to solving the SOCP
\begin{equation}
\begin{array}{ll}
\mbox{minimize} & u^T P u + x_t^T Q u + q^T u \\
\mbox{subject to} & \|{u}\|_2 \leq 1,
\end{array}
\label{eq:policy}
\end{equation}
with variable $u$ and parameters $P$, $Q$, $q$, and $x_t$. We can run
stochastic gradient descent (SGD) on $P$, $Q$, and $q$ to
approximately solve the original problem, which requires requires differentiating
through the quadratic policy. Note that if $u$ were unconstrained, the original problem
could be solved exactly, via linear quadratic regulator (LQR) theory.

```python
import cvxpy as cp
import numpy as np
import tensorflow as tf
from scipy.linalg import solve_discrete_are
from scipy.linalg import sqrtm

from cvxpylayers.tensorflow.cvxpylayer import CvxpyLayer

# Generate data
tf.random.set_seed(1)
np.random.seed(1)

n = 2
m = 3

A = np.eye(n) + 1e-2 * np.random.randn(n, n)
B = 1e-2 / 3 * np.random.randn(n, m)
Q = np.eye(n)
R = np.eye(m)

# Compute LQR control policy
P_lqr = solve_discrete_are(A, B, Q, R)
P = R + B.T@P_lqr@B
P_sqrt_lqr = sqrtm(P)

# Construct CVXPY problem and layer
x_cvxpy = cp.Parameter((n, 1))
P_sqrt_cvxpy = cp.Parameter((m, m))
P_21_cvxpy = cp.Parameter((n, m))
q_cvxpy = cp.Parameter((m, 1))

u_cvxpy = cp.Variable((m, 1))
y_cvxpy = cp.Variable((n, 1))

objective = .5 * cp.sum_squares(P_sqrt_cvxpy @ u_cvxpy) + x_cvxpy.T @ y_cvxpy + q_cvxpy.T @ u_cvxpy
problem = cp.Problem(cp.Minimize(objective), [cp.norm(u_cvxpy) <= 1, y_cvxpy == P_21_cvxpy @ u_cvxpy])
assert problem.is_dpp()
policy = CvxpyLayer(
    problem, [x_cvxpy, P_sqrt_cvxpy, P_21_cvxpy, q_cvxpy], [u_cvxpy])
```

Below, we train the policy and plot the estimated average cost for each iteration of
SGD for a numerical example, with $x \in
\mathbf{R}^2$ and $u \in \mathbf{R}^3$, a time horizon of $T=25$, and a batch
size of $8$. We initialize our policy's parameters with the LQR solution,
ignoring the constraint on $u$.

```python
import matplotlib.pyplot as plt

def train(iters):
    # Initialize with LQR control lyapunov function
    P_sqrt = tf.Variable(P_sqrt_lqr)
    P_21 = tf.Variable(A.T @ P_lqr @ B)
    q = tf.Variable(tf.zeros((m, 1), dtype=tf.float64))
    variables = [P_sqrt, P_21, q]
    A_tf, B_tf, Q_tf, R_tf = map(tf.constant, [A, B, Q, R])

    def g(x, u):
        return tf.squeeze(tf.transpose(x) @ Q_tf @ x + tf.transpose(u) @ R_tf @ u)

    def evaluate(x0, P_sqrt, P_21, q, T):
        x = x0
        cost = 0.
        for _ in range(T):
            u, = policy(x, P_sqrt, P_21, q)
            cost += g(x, u) / T
            x = A_tf @ x + B_tf @ u + .2 * tf.random.normal((n, 1), dtype=tf.float64)
        return cost

    def eval_loss(N=8, T=25):
        return sum([evaluate(tf.zeros((n, 1), dtype=tf.float64), P_sqrt, P_21, q, T=T)
                    for _ in range(N)]) / N

    results = []
    optimizer = tf.keras.optimizers.SGD(learning_rate=0.02, momentum=0.9)
    for i in range(iters):
        tf.random.set_seed(1)
        np.random.seed(1)
        with tf.GradientTape() as tape:
            loss = eval_loss()
        gradients = tape.gradient(loss, variables)
        optimizer.apply_gradients(zip(gradients, variables))
        results.append(loss.numpy())
        print("(iter %d) loss: %g " % (i, results[-1]))
    return results

results = train(iters=100)
```

    (iter 0) loss: 1.18634
    (iter 1) loss: 1.17733
    (iter 2) loss: 1.16115
    (iter 3) loss: 1.13998
    (iter 4) loss: 1.11706
    (iter 5) loss: 1.09234
    (iter 6) loss: 1.06502
    (iter 7) loss: 1.03633
    (iter 8) loss: 1.00895
    (iter 9) loss: 0.982198
    (iter 10) loss: 0.954998
    (iter 11) loss: 0.926645
    (iter 12) loss: 0.900704
    (iter 13) loss: 0.874988
    (iter 14) loss: 0.851813
    (iter 15) loss: 0.829279
    (iter 16) loss: 0.809518
    (iter 17) loss: 0.79314
    (iter 18) loss: 0.77915
    (iter 19) loss: 0.767274
    (iter 20) loss: 0.756987
    (iter 21) loss: 0.74633
    (iter 22) loss: 0.735943
    (iter 23) loss: 0.725892
    (iter 24) loss: 0.716452
    (iter 25) loss: 0.70797
    (iter 26) loss: 0.700616
    (iter 27) loss: 0.694235
    (iter 28) loss: 0.688617
    (iter 29) loss: 0.683476
    (iter 30) loss: 0.678953
    (iter 31) loss: 0.67472
    (iter 32) loss: 0.670976
    (iter 33) loss: 0.667654
    (iter 34) loss: 0.664699
    (iter 35) loss: 0.662062
    (iter 36) loss: 0.659702
    (iter 37) loss: 0.657582
    (iter 38) loss: 0.655674
    (iter 39) loss: 0.653949
    (iter 40) loss: 0.652387
    (iter 41) loss: 0.650967
    (iter 42) loss: 0.649673
    (iter 43) loss: 0.648491
    (iter 44) loss: 0.647407
    (iter 45) loss: 0.646412
    (iter 46) loss: 0.645496
    (iter 47) loss: 0.64465
    (iter 48) loss: 0.643867
    (iter 49) loss: 0.64314
    (iter 50) loss: 0.642464
    (iter 51) loss: 0.641835
    (iter 52) loss: 0.641246
    (iter 53) loss: 0.640696
    (iter 54) loss: 0.640179
    (iter 55) loss: 0.639693
    (iter 56) loss: 0.639236
    (iter 57) loss: 0.638805
    (iter 58) loss: 0.638397
    (iter 59) loss: 0.63801
    (iter 60) loss: 0.637644
    (iter 61) loss: 0.637296
    (iter 62) loss: 0.636964
    (iter 63) loss: 0.636648
    (iter 64) loss: 0.636346
    (iter 65) loss: 0.636058
    (iter 66) loss: 0.635782
    (iter 67) loss: 0.635517
    (iter 68) loss: 0.635262
    (iter 69) loss: 0.635018
    (iter 70) loss: 0.634782
    (iter 71) loss: 0.634555
    (iter 72) loss: 0.634336
    (iter 73) loss: 0.634125
    (iter 74) loss: 0.63392
    (iter 75) loss: 0.633722
    (iter 76) loss: 0.633531
    (iter 77) loss: 0.633345
    (iter 78) loss: 0.633164
    (iter 79) loss: 0.632989
    (iter 80) loss: 0.632819
    (iter 81) loss: 0.632653
    (iter 82) loss: 0.632492
    (iter 83) loss: 0.632335
    (iter 84) loss: 0.632182
    (iter 85) loss: 0.632033
    (iter 86) loss: 0.631887
    (iter 87) loss: 0.631745
    (iter 88) loss: 0.631606
    (iter 89) loss: 0.631471
    (iter 90) loss: 0.631339
    (iter 91) loss: 0.631209
    (iter 92) loss: 0.631082
    (iter 93) loss: 0.630958
    (iter 94) loss: 0.630837
    (iter 95) loss: 0.630718
    (iter 96) loss: 0.630601
    (iter 97) loss: 0.630487
    (iter 98) loss: 0.630375
    (iter 99) loss: 0.630265

```python
plt.figure()
plt.plot(results)
plt.xlabel('iteration')
plt.ylabel('average cost')
plt.show()
```


# Software profesional en Acústica 2020-21 (M2i)

*This notebook contains a modification of the notebook [FEM_Helmholtz_equation_Robin](https://github.com/spatialaudio/computational_acoustics/blob/master/FEM_Helmholtz_equation_Robin.ipynb), created by Sascha Spors, Frank Schultz, Computational Acoustics Examples, 2018. The text/images are licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/). The code is released under the [MIT license](https://opensource.org/licenses/MIT).*

# Numerical Solution of the Helmholtz Equation with Robin Boundary Conditions using the Finite Element Method

This notebook illustrates the numerical solution of the wave equation for an harmonic excitation and Robin boundary conditions using the [Finite Element Method](https://en.wikipedia.org/wiki/Finite_element_method) (FEM). The method aims at an approximate solution by subdividing the area of interest into smaller parts with simpler geometry, linking these parts together and applying methods from the calculus of variations to solve the problem numerically. The FEM is a well established method for the numerical approximation of the solution of partial differential equations (PDEs). The solutions of PDEs are often known analytically only for rather simple geometries. FEM based simulations allow to gain insights into other more complex cases.

## Problem Statement

The inhomogeneous [Helmholtz equation](https://en.wikipedia.org/wiki/Helmholtz_equation) is given as

\begin{equation}
\Delta P(\mathbf{x}, \omega) + \frac{\omega^2}{c^2} P(\mathbf{x}, \omega) = - Q(\mathbf{x}, \omega) .
\end{equation}

We aim for a numerical solution of the Helmholtz equation on the domain $V$ with respect to the homogeneous Robin boundary condition

\begin{equation}
V_n(\mathbf{x}, \omega) + \frac{1}{Z(\mathbf{x}, \omega)} P(\mathbf{x}, \omega) = 0 \qquad \text{for } x \in \partial V ,
\end{equation}

where $V_n(\mathbf{x}, \omega)$ denotes the particle velocity in inward normal direction to the boundary $\partial V$ of $V$ and $Z(\mathbf{x}, \omega)$ the acoustic impedance of the boundary.
The particle velocity can be linked to the pressure using the Euler equation

\begin{equation}
-\mathrm{j} \omega \rho_0 V_n(\mathbf{x}, \omega) = \frac{\partial}{\partial n} P(\mathbf{x}, \omega) ,
\end{equation}

where $\rho_0$ denotes the static density of air.
Introducing this into the Robin boundary equation above yields

\begin{equation}
\frac{\partial}{\partial n} P(\mathbf{x}, \omega) - \mathrm{j} \underbrace{\frac{\omega \rho_0}{Z}}_{\sigma} P(\mathbf{x}, \omega) = 0 \qquad \text{for } x \in \partial V .
\end{equation}

The medium impedance of air for free-field propagation is $Z_0 = \rho_0 c$, hence $\sigma_0 = \frac{\omega}{c}$ in this case. Free-field conditions can be simulated by matching the impedance of the boundary to $Z_0$.

## Variational Formulation

Starting from the [variational formulation of the Helmholtz equation](FEM_Helmholtz_equation_2D.ipynb#Variational-Formulation) (before application of Green's first theorem)

\begin{equation}
{-} \int_V \nabla P(\mathbf{x}, \omega) \cdot \nabla V(\mathbf{x}, \omega) \mathrm{d}x  + \int_{\partial V} V(\mathbf{x}, \omega) \frac{\partial}{\partial n}  P(\mathbf{x}, \omega) \mathrm{d}s + \frac{\omega^2}{c^2} \int_V P(\mathbf{x}, \omega) V(\mathbf{x}, \omega) \mathrm{d}x = -\int_V Q(\mathbf{x}, \omega) V(\mathbf{x}, \omega) \mathrm{d}x
\end{equation}

and introducing the Robin boundary condition into the second integral yields

\begin{equation}
{-} \int_V \nabla P(\mathbf{x}, \omega) \cdot \nabla V(\mathbf{x}, \omega) \mathrm{d}x  + \mathrm{j} \sigma \int_{\partial V} V(\mathbf{x}, \omega) P(\mathbf{x}, \omega) \mathrm{d}s + \frac{\omega^2}{c^2} \int_V P(\mathbf{x}, \omega) V(\mathbf{x}, \omega) \mathrm{d}x = -\int_V Q(\mathbf{x}, \omega) V(\mathbf{x}, \omega) \mathrm{d}x .
\end{equation}

It is common to express this integral equation in terms of the bilinear $a(P, V)$ and linear $L(V)$ forms

\begin{equation}
a(P, V) = \frac{\omega^2}{c^2} \int_V P(\mathbf{x}, \omega) V(\mathbf{x}, \omega) \mathrm{d}x - \int_V \nabla P(\mathbf{x}, \omega) \cdot \nabla V(\mathbf{x}, \omega) \mathrm{d}x + \mathrm{j} \sigma \int_{\partial V} V(\mathbf{x}, \omega) P(\mathbf{x}, \omega) \mathrm{d}s , \end{equation}

\begin{equation}
L(V) = -\int_V Q(\mathbf{x}, \omega) V(\mathbf{x}, \omega) \mathrm{d}x ,
\end{equation}

where

\begin{equation}
a(P, V) = L(V) .
\end{equation}

Computational implementations of the FEM (like FEniCS) may not be able to handle complex numbers.
In this case the problem can be split into two coupled problems with respect to the real and imaginary part. By introducing $P(\mathbf{x}, \omega) = P_r(\mathbf{x}, \omega) + \mathrm{j} P_i(\mathbf{x}, \omega)$ and $V(\mathbf{x}, \omega) = V_r(\mathbf{x}, \omega) + \mathrm{j} V_i(\mathbf{x}, \omega)$ and identifying the real and imaginary parts of the bilinear and linear forms, we get

\begin{equation}
a_r = \int_V \left( \frac{\omega^2}{c^2} V_r(\mathbf{x}, \omega) P_r(\mathbf{x}, \omega) -  \frac{\omega^2}{c^2} V_i(\mathbf{x}, \omega) P_i(\mathbf{x}, \omega) - \nabla P_r(\mathbf{x}, \omega) \cdot \nabla V_r(\mathbf{x}, \omega) + \nabla P_i(\mathbf{x}, \omega) \cdot \nabla V_i(\mathbf{x}, \omega) \right) \mathrm{d}x -
\sigma \cdot \int_{\partial V} \left( V_i(\mathbf{x}, \omega) P_r(\mathbf{x}, \omega) + V_r(\mathbf{x}, \omega) P_i(\mathbf{x}, \omega) \right) \mathrm{d}s,
\end{equation}

\begin{equation}
a_i = \int_V \left( \frac{\omega^2}{c^2} V_r(\mathbf{x}, \omega) P_i(\mathbf{x}, \omega) +  \frac{\omega^2}{c^2} V_i(\mathbf{x}, \omega) P_r(\mathbf{x}, \omega) - \nabla P_i(\mathbf{x}, \omega) \cdot \nabla V_r(\mathbf{x}, \omega) + \nabla P_r(\mathbf{x}, \omega) \cdot \nabla V_i(\mathbf{x}, \omega) \right) \mathrm{d}x +
\sigma \cdot \int_{\partial V} \left( V_r(\mathbf{x}, \omega) P_r(\mathbf{x}, \omega) - V_i(\mathbf{x}, \omega) P_i(\mathbf{x}, \omega) \right) \mathrm{d}s
\end{equation}

for the bilinear form.

## Numerical Solution

The numerical solution of the variational problem is based on [FEniCS](https://fenicsproject.org/), an open-source framework for numerical solution of PDEs.
Its high-level Python interface `dolfin` is used in the following to define the problem and compute the solution.
The implementation is based on the variational formulation derived above.
It is common in the FEM to denote the solution of the problem by $u$ and the test function by $v$.
The definition of the problem in FEniCS is very close to the mathematical formulation of the problem.

For the subsequent examples the solution of inhomogeneous wave equation for a point source $Q(\mathbf{x}) = \delta(\mathbf{x}-\mathbf{x_s})$ at position $\mathbf{x_s}$ is computed using the FEM.
A function is defined for this purpose, as well as for the plotting of the resulting sound field.

```python
import dolfin
import mshr
import matplotlib.pyplot as plt
%matplotlib inline

def Helmholtz_Robin(mesh, frequency, xs, sigma=dolfin.Constant(0), c=343):

    # squared wavenumber
    k2 = dolfin.Constant(2*dolfin.pi*frequency/c)**2

    # define function space
    V = dolfin.VectorFunctionSpace(mesh, "CG", 1, dim=2)

    # define variational problem
    (u_r, u_i) = dolfin.TrialFunction(V)
    (v_r, v_i) = dolfin.TestFunction(V)

    a_r = ( k2 * dolfin.inner(u_r,v_r) - k2 * dolfin.inner(u_i,v_i) - dolfin.inner(dolfin.grad(u_r), dolfin.grad(v_r)) + dolfin.inner(dolfin.grad(u_i), dolfin.grad(v_i)) ) * dolfin.dx  - sigma*dolfin.inner(u_r, v_i) * dolfin.ds - sigma*dolfin.inner(u_i, v_r) * dolfin.ds
    a_i = ( k2 * dolfin.inner(u_r,v_i) + k2 * dolfin.inner(u_i,v_r) - dolfin.inner(dolfin.grad(u_r), dolfin.grad(v_i)) - dolfin.inner(dolfin.grad(u_i), dolfin.grad(v_r)) ) * dolfin.dx  + sigma*dolfin.inner(u_r, v_r) * dolfin.ds - sigma*dolfin.inner(u_i, v_i) * dolfin.ds
    L_r = dolfin.Constant(0) * v_r * dolfin.dx
    L_i = dolfin.Constant(0) * v_i * dolfin.dx

    a = a_r + a_i
    L = L_r + L_i
    A, b = dolfin.assemble_system(a, L)

    # define inhomogenity
    delta = dolfin.PointSource(V.sub(0), xs, -1)  # negative amplitude accounts for -Q(x,w) in inhomogeneous wave equation
    delta.apply(b)

    # compute solution
    u = dolfin.Function(V)
    dolfin.solve(A, u.vector(), b)
    (u_r, u_i) = dolfin.split(u)

    return u_r

def plot_soundfield(u):
    '''plots solution of FEM-based simulation'''
    fig = plt.figure(figsize=(10,10))
    fig = dolfin.plot(u)
    plt.title(r'$P(\mathbf{x}, \omega)$')
    plt.xlabel(r'$x$ in m')
    plt.ylabel(r'$y$ in m')
    plt.colorbar(fig, fraction=0.038, pad=0.04);
```

### Sound Field in a Rectangular Room

The two-dimensional sound field in a rectangular room (rectangular plate) with homogeneous Robin boundary conditions is computed for the frequency $f=120$ Hz and source position $x_s = (1.2,3.2)$ m.

```python
f = 120  # frequency
xs = dolfin.Point(1.2, 3.2)  # source position

# define geometry and mesh
mesh = dolfin.RectangleMesh(dolfin.Point(0, 0), dolfin.Point(5, 4), 200, 200, "right/left")
```

First, the case of sound-hard (Neumann) boundary conditions with $\sigma = 0$ ($Z \to \infty$) is considered.

```python
# compute solution for sigma=0
u = Helmholtz_Robin(mesh, f, xs, sigma=dolfin.Constant(0))

# plot sound field
plot_soundfield(u)
plot_soundfield(abs(u))
plt.title(r'$|P(\mathbf{x}, \omega)|$');
```

Now the case of a matched boundary (free-field propagation) with $\sigma = \frac{\omega}{c}$ ($Z = Z_0$) is considered.

```python
# compute solution for free-field propogation
u = Helmholtz_Robin(mesh, f, xs, sigma=dolfin.Constant(2*dolfin.pi*f/343))

# plot sound field
plot_soundfield(u)
plot_soundfield(abs(u))
plt.title(r'$|P(\mathbf{x}, \omega)|$');
```

The last example shows the simulation result for sound-soft (Dirichlet) boundary conditions with $\sigma \to \infty$ ($Z=0$).

```python
# compute solution for very large sigma
u = Helmholtz_Robin(mesh, f, xs, sigma=dolfin.Constant(1e15))

# plot sound field
plot_soundfield(u)
plot_soundfield(abs(u))
plt.title(r'$|P(\mathbf{x}, \omega)|$');
```

The effect of the homogeneous Dirichlet boundary condition (zero pressure at the boundary) can be observed conveniently by inspecting the magnitude of the sound field close to the boundary.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT).


# State space modeling: Local Linear Trends

This notebook describes how to extend the statsmodels statespace classes to create and estimate a custom model. Here we develop a local linear trend model.

The Local Linear Trend model has the form (see Durbin and Koopman 2012, Chapter 3.2 for all notation and details):

$$
\begin{align}
y_t & = \mu_t + \varepsilon_t \qquad & \varepsilon_t \sim
    N(0, \sigma_\varepsilon^2) \\
\mu_{t+1} & = \mu_t + \nu_t + \xi_t & \xi_t \sim N(0, \sigma_\xi^2) \\
\nu_{t+1} & = \nu_t + \zeta_t & \zeta_t \sim N(0, \sigma_\zeta^2)
\end{align}
$$

It is easy to see that this can be cast into state space form as:

$$
\begin{align}
y_t & = \begin{pmatrix} 1 & 0 \end{pmatrix} \begin{pmatrix} \mu_t \\ \nu_t \end{pmatrix} + \varepsilon_t \\
\begin{pmatrix} \mu_{t+1} \\ \nu_{t+1} \end{pmatrix} & = \begin{bmatrix} 1 & 1 \\ 0 & 1 \end{bmatrix} \begin{pmatrix} \mu_t \\ \nu_t \end{pmatrix} + \begin{pmatrix} \xi_t \\ \zeta_t \end{pmatrix}
\end{align}
$$

Notice that much of the state space representation is composed of known values; in fact the only parts in which parameters to be estimated appear are in the variance / covariance matrices:

$$
\begin{align}
H_t & = \begin{bmatrix} \sigma_\varepsilon^2 \end{bmatrix} \\
Q_t & = \begin{bmatrix} \sigma_\xi^2 & 0 \\ 0 & \sigma_\zeta^2 \end{bmatrix}
\end{align}
$$

```python
%matplotlib inline

import numpy as np
import pandas as pd
from scipy.stats import norm
import statsmodels.api as sm
import matplotlib.pyplot as plt
```

To take advantage of the existing infrastructure, including Kalman filtering and maximum likelihood estimation, we create a new class which extends from `statsmodels.tsa.statespace.MLEModel`. There are a number of things that must be specified:

1. **k_states**, **k_posdef**: These two parameters must be provided to the base classes in initialization. The inform the statespace model about the size of, respectively, the state vector, above $\begin{pmatrix} \mu_t & \nu_t \end{pmatrix}'$, and   the state error vector, above $\begin{pmatrix} \xi_t & \zeta_t \end{pmatrix}'$. Note that the dimension of the endogenous vector does not have to be specified, since it can be inferred from the `endog` array.
2. **update**: The method `update`, with argument `params`, must be specified (it is used when `fit()` is called to calculate the MLE). It takes the parameters and fills them into the appropriate state space matrices. For example, below, the `params` vector contains variance parameters $\begin{pmatrix} \sigma_\varepsilon^2 & \sigma_\xi^2 & \sigma_\zeta^2\end{pmatrix}$, and the `update` method must place them in the observation and state covariance matrices. More generally, the parameter vector might be mapped into many different places in all of the statespace matrices.
3. **statespace matrices**: by default, all state space matrices (`obs_intercept, design, obs_cov, state_intercept, transition, selection, state_cov`) are set to zeros. Values that are fixed (like the ones in the design and transition matrices here) can be set in initialization, whereas values that vary with the parameters should be set in the `update` method. Note that it is easy to forget to set the selection matrix, which is often just the identity matrix (as it is here), but not setting it will lead to a very different model (one where there is not a stochastic component to the transition equation).
4. **start params**: start parameters must be set, even if it is just a vector of zeros, although often good start parameters can be found from the data. Maximum likelihood estimation by gradient methods (as employed here) can be sensitive to the starting parameters, so it is important to select good ones if possible. Here it does not matter too much (although as variances, they should't be set zero).
5. **initialization**: in addition to defined state space matrices, all state space models must be initialized with the mean and variance for the initial distribution of the state vector. If the distribution is known, `initialize_known(initial_state, initial_state_cov)` can be called, or if the model is stationary (e.g. an ARMA model), `initialize_stationary` can be used. Otherwise, `initialize_approximate_diffuse` is a reasonable generic initialization (exact diffuse initialization is not yet available). Since the local linear trend model is not stationary (it is composed of random walks) and since the distribution is not generally known, we use `initialize_approximate_diffuse` below.

The above are the minimum necessary for a successful model. There are also a number of things that do not have to be set, but which may be helpful or important for some applications:

1. **transform / untransform**: when `fit` is called, the optimizer in the background will use gradient methods to select the parameters that maximize the likelihood function. By default it uses unbounded optimization, which means that it may select any parameter value. In many cases, that is not the desired behavior; variances, for example, cannot be negative. To get around this, the `transform` method takes the unconstrained vector of parameters provided by the optimizer and returns a constrained vector of parameters used in likelihood evaluation. `untransform` provides the reverse operation.
2. **param_names**: this internal method can be used to set names for the estimated parameters so that e.g. the summary provides meaningful names. If not present, parameters are named `param0`, `param1`, etc.

```python
"""
Univariate Local Linear Trend Model
"""
class LocalLinearTrend(sm.tsa.statespace.MLEModel):
    def __init__(self, endog):
        # Model order
        k_states = k_posdef = 2

        # Initialize the statespace
        super(LocalLinearTrend, self).__init__(
            endog, k_states=k_states, k_posdef=k_posdef,
            initialization='approximate_diffuse',
            loglikelihood_burn=k_states
        )

        # Initialize the matrices
        self.ssm['design'] = np.array([1, 0])
        self.ssm['transition'] = np.array([[1, 1],
                                       [0, 1]])
        self.ssm['selection'] = np.eye(k_states)

        # Cache some indices
        self._state_cov_idx = ('state_cov',) + np.diag_indices(k_posdef)

    @property
    def param_names(self):
        return ['sigma2.measurement', 'sigma2.level', 'sigma2.trend']

    @property
    def start_params(self):
        return [np.std(self.endog)]*3

    def transform_params(self, unconstrained):
        return unconstrained**2

    def untransform_params(self, constrained):
        return constrained**0.5

    def update(self, params, *args, **kwargs):
        params = super(LocalLinearTrend, self).update(params, *args, **kwargs)

        # Observation covariance
        self.ssm['obs_cov',0,0] = params[0]

        # State covariance
        self.ssm[self._state_cov_idx] = params[1:]
```

Using this simple model, we can estimate the parameters from a local linear trend model. The following example is from Commandeur and Koopman (2007), section 3.4., modeling motor vehicle fatalities in Finland.

```python
import requests
from io import BytesIO
from zipfile import ZipFile

# Download the dataset
ck = requests.get('http://staff.feweb.vu.nl/koopman/projects/ckbook/OxCodeAll.zip').content
zipped = ZipFile(BytesIO(ck))
df = pd.read_table(
    BytesIO(zipped.read('OxCodeIntroStateSpaceBook/Chapter_2/NorwayFinland.txt')),
    skiprows=1, header=None, sep='\s+', engine='python',
    names=['date','nf', 'ff']
)
```

Since we defined the local linear trend model as extending from `MLEModel`, the `fit()` method is immediately available, just as in other statsmodels maximum likelihood classes. Similarly, the returned results class supports many of the same post-estimation results, like the `summary` method.

```python
# Load Dataset
df.index = pd.date_range(start='%d-01-01' % df.date[0], end='%d-01-01' % df.iloc[-1, 0], freq='AS')

# Log transform
df['lff'] = np.log(df['ff'])

# Setup the model
mod = LocalLinearTrend(df['lff'])

# Fit it using MLE (recall that we are fitting the three variance parameters)
res = mod.fit(disp=False)
print(res.summary())
```

                               Statespace Model Results
    ==============================================================================
    Dep. Variable:                    lff   No. Observations:                   34
    Model:               LocalLinearTrend   Log Likelihood                  27.510
    Date:                Tue, 02 Feb 2021   AIC                            -49.020
    Time:                        07:05:52   BIC                            -44.623
    Sample:                    01-01-1970   HQIC                           -47.563
                             - 01-01-2003
    Covariance Type:                  opg
    ======================================================================================
                             coef    std err          z      P>|z|      [0.025      0.975]
    --------------------------------------------------------------------------------------
    sigma2.measurement     0.0010      0.003      0.346      0.730      -0.005       0.007
    sigma2.level           0.0074      0.005      1.564      0.118      -0.002       0.017
    sigma2.trend        2.498e-11      0.000   1.66e-07      1.000      -0.000       0.000
    ===================================================================================
    Ljung-Box (L1) (Q):                   0.00   Jarque-Bera (JB):                 0.68
    Prob(Q):                              0.95   Prob(JB):                         0.71
    Heteroskedasticity (H):               0.75   Skew:                            -0.02
    Prob(H) (two-sided):                  0.64   Kurtosis:                         2.29
    ===================================================================================

    Warnings:
    [1] Covariance matrix calculated using the outer product of gradients (complex-step).

Finally, we can do post-estimation prediction and forecasting. Notice that the end period can be specified as a date.

```python
# Perform prediction and forecasting
predict = res.get_prediction()
forecast = res.get_forecast('2014')
```

```python
fig, ax = plt.subplots(figsize=(10,4))

# Plot the results
df['lff'].plot(ax=ax, style='k.', label='Observations')
predict.predicted_mean.plot(ax=ax, label='One-step-ahead Prediction')
predict_ci = predict.conf_int(alpha=0.05)
predict_index = np.arange(len(predict_ci))
ax.fill_between(predict_index[2:], predict_ci.iloc[2:, 0], predict_ci.iloc[2:, 1], alpha=0.1)

forecast.predicted_mean.plot(ax=ax, style='r', label='Forecast')
forecast_ci = forecast.conf_int()
forecast_index = np.arange(len(predict_ci), len(predict_ci) + len(forecast_ci))
ax.fill_between(forecast_index, forecast_ci.iloc[:, 0], forecast_ci.iloc[:, 1], alpha=0.1)

# Cleanup the image
ax.set_ylim((4, 8));
legend = ax.legend(loc='lower left');
```

### References

    Commandeur, Jacques J. F., and Siem Jan Koopman. 2007.
    An Introduction to State Space Time Series Analysis.
    Oxford ; New York: Oxford University Press.

    Durbin, James, and Siem Jan Koopman. 2012.
    Time Series Analysis by State Space Methods: Second Edition.
    Oxford University Press.


# 机器学习练习 2 - 逻辑回归

这个笔记包含了以Python为编程语言的Coursera上机器学习的第二次编程练习。请参考 [作业文件](ex2.pdf) 详细描述和方程。
在这一次练习中，我们将要实现逻辑回归并且应用到一个分类任务。我们还将通过将正则化加入训练算法，来提高算法的鲁棒性，并用更复杂的情形来测试它。

代码修改并注释：黄海广，haiguang2000@qq.com

## 逻辑回归

在训练的初始阶段，我们将要构建一个逻辑回归模型来预测，某个学生是否被大学录取。设想你是大学相关部分的管理者，想通过申请学生两次测试的评分，来决定他们是否被录取。现在你拥有之前申请学生的可以用于训练逻辑回归的训练样本集。对于每一个训练样本，你有他们两次测试的评分和最后是被录取的结果。为了完成这个预测任务，我们准备构建一个可以基于两次测试评分来评估录取可能性的分类模型。

让我们从检查数据开始。

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
```

```python
path = 'ex2data1.txt'
data = pd.read_csv(path, header=None, names=['Exam 1', 'Exam 2', 'Admitted'])
data.head()
```

      Exam 1
      Exam 2
      Admitted

      0
      34.623660
      78.024693
      0

      1
      30.286711
      43.894998
      0

      2
      35.847409
      72.902198
      0

      3
      60.182599
      86.308552
      1

      4
      79.032736
      75.344376
      1

让我们创建两个分数的散点图，并使用颜色编码来可视化，如果样本是正的（被接纳）或负的（未被接纳）。

```python
positive = data[data['Admitted'].isin([1])]
negative = data[data['Admitted'].isin([0])]

fig, ax = plt.subplots(figsize=(12,8))
ax.scatter(positive['Exam 1'], positive['Exam 2'], s=50, c='b', marker='o', label='Admitted')
ax.scatter(negative['Exam 1'], negative['Exam 2'], s=50, c='r', marker='x', label='Not Admitted')
ax.legend()
ax.set_xlabel('Exam 1 Score')
ax.set_ylabel('Exam 2 Score')
plt.show()
```

看起来在两类间，有一个清晰的决策边界。现在我们需要实现逻辑回归，那样就可以训练一个模型来预测结果。方程实现在下面的代码示例在"exercises" 文件夹的 "ex2.pdf" 中。

# sigmoid 函数
g 代表一个常用的逻辑函数（logistic function）为S形函数（Sigmoid function），公式为： \\[g\left( z \right)=\frac{1}{1+{{e}^{-z}}}\\]
合起来，我们得到逻辑回归模型的假设函数：
	\\[{{h}_{\theta }}\left( x \right)=\frac{1}{1+{{e}^{-{{\theta }^{T}}X}}}\\]

```python
def sigmoid(z):
    return 1 / (1 + np.exp(-z))
```

让我们做一个快速的检查，来确保它可以工作。

```python
nums = np.arange(-10, 10, step=1)

fig, ax = plt.subplots(figsize=(12,8))
ax.plot(nums, sigmoid(nums), 'r')
plt.show()
```

棒极了！现在，我们需要编写代价函数来评估结果。
代价函数：
$J\left( \theta  \right)=\frac{1}{m}\sum\limits_{i=1}^{m}{[-{{y}^{(i)}}\log \left( {{h}_{\theta }}\left( {{x}^{(i)}} \right) \right)-\left( 1-{{y}^{(i)}} \right)\log \left( 1-{{h}_{\theta }}\left( {{x}^{(i)}} \right) \right)]}$

```python
def cost(theta, X, y):
    theta = np.matrix(theta)
    X = np.matrix(X)
    y = np.matrix(y)
    first = np.multiply(-y, np.log(sigmoid(X * theta.T)))
    second = np.multiply((1 - y), np.log(1 - sigmoid(X * theta.T)))
    return np.sum(first - second) / (len(X))
```

现在，我们要做一些设置，和我们在练习1在线性回归的练习很相似。

```python
# add a ones column - this makes the matrix multiplication work out easier
data.insert(0, 'Ones', 1)

# set X (training data) and y (target variable)
cols = data.shape[1]
X = data.iloc[:,0:cols-1]
y = data.iloc[:,cols-1:cols]

# convert to numpy arrays and initalize the parameter array theta
X = np.array(X.values)
y = np.array(y.values)
theta = np.zeros(3)
```

让我们来检查矩阵的维度来确保一切良好。

```python
theta
```

    array([0., 0., 0.])

```python
X.shape, theta.shape, y.shape
```

    ((100, 3), (3,), (100, 1))

让我们计算初始化参数的代价函数(theta为0)。

```python
cost(theta, X, y)
```

    0.6931471805599453

看起来不错，接下来，我们需要一个函数来计算我们的训练数据、标签和一些参数thata的梯度。

# gradient descent(梯度下降)
* 这是批量梯度下降（batch gradient descent）
* 转化为向量化计算： $\frac{1}{m} X^T( Sigmoid(X\theta) - y )$
$$\frac{\partial J\left( \theta  \right)}{\partial {{\theta }_{j}}}=\frac{1}{m}\sum\limits_{i=1}^{m}{({{h}_{\theta }}\left( {{x}^{(i)}} \right)-{{y}^{(i)}})x_{_{j}}^{(i)}}$$

```python
def gradient(theta, X, y):
    theta = np.matrix(theta)
    X = np.matrix(X)
    y = np.matrix(y)

    parameters = int(theta.ravel().shape[1])
    grad = np.zeros(parameters)

    error = sigmoid(X * theta.T) - y

    for i in range(parameters):
        term = np.multiply(error, X[:,i])
        grad[i] = np.sum(term) / len(X)

    return grad
```

注意，我们实际上没有在这个函数中执行梯度下降，我们仅仅在计算一个梯度步长。在练习中，一个称为“fminunc”的Octave函数是用来优化函数来计算成本和梯度参数。由于我们使用Python，我们可以用SciPy的“optimize”命名空间来做同样的事情。

我们看看用我们的数据和初始参数为0的梯度下降法的结果。

```python
gradient(theta, X, y)
```

    array([ -0.1       , -12.00921659, -11.26284221])

现在可以用SciPy's truncated newton（TNC）实现寻找最优参数。

```python
import scipy.optimize as opt
result = opt.fmin_tnc(func=cost, x0=theta, fprime=gradient, args=(X, y))
result
```

    (array([-25.16131863,   0.20623159,   0.20147149]), 36, 0)

让我们看看在这个结论下代价函数计算结果是什么个样子~

```python
cost(result[0], X, y)
```

    0.20349770158947458

接下来，我们需要编写一个函数，用我们所学的参数theta来为数据集X输出预测。然后，我们可以使用这个函数来给我们的分类器的训练精度打分。
逻辑回归模型的假设函数：
	\\[{{h}_{\theta }}\left( x \right)=\frac{1}{1+{{e}^{-{{\theta }^{T}}X}}}\\]
当${{h}_{\theta }}$大于等于0.5时，预测 y=1

当${{h}_{\theta }}$小于0.5时，预测 y=0 。

```python
def predict(theta, X):
    probability = sigmoid(X * theta.T)
    return [1 if x >= 0.5 else 0 for x in probability]
```

```python
theta_min = np.matrix(result[0])
predictions = predict(theta_min, X)
correct = [1 if ((a == 1 and b == 1) or (a == 0 and b == 0)) else 0 for (a, b) in zip(predictions, y)]
accuracy = (sum(map(int, correct)) % len(correct))
print ('accuracy = {0}%'.format(accuracy))
```

    accuracy = 89%

我们的逻辑回归分类器预测正确，如果一个学生被录取或没有录取，达到89%的精确度。不坏！记住，这是训练集的准确性。我们没有保持住了设置或使用交叉验证得到的真实逼近，所以这个数字有可能高于其真实值（这个话题将在以后说明）。

## 正则化逻辑回归

在训练的第二部分，我们将要通过加入正则项提升逻辑回归算法。如果你对正则化有点眼生，或者喜欢这一节的方程的背景，请参考在"exercises"文件夹中的"ex2.pdf"。简而言之，正则化是成本函数中的一个术语，它使算法更倾向于“更简单”的模型（在这种情况下，模型将更小的系数）。这个理论助于减少过拟合，提高模型的泛化能力。这样，我们开始吧。

设想你是工厂的生产主管，你有一些芯片在两次测试中的测试结果。对于这两次测试，你想决定是否芯片要被接受或抛弃。为了帮助你做出艰难的决定，你拥有过去芯片的测试数据集，从其中你可以构建一个逻辑回归模型。

和第一部分很像，从数据可视化开始吧！

```python
path =  'ex2data2.txt'
data2 = pd.read_csv(path, header=None, names=['Test 1', 'Test 2', 'Accepted'])
data2.head()
```

      Test 1
      Test 2
      Accepted

      0
      0.051267
      0.69956
      1

      1
      -0.092742
      0.68494
      1

      2
      -0.213710
      0.69225
      1

      3
      -0.375000
      0.50219
      1

      4
      -0.513250
      0.46564
      1

```python
positive = data2[data2['Accepted'].isin([1])]
negative = data2[data2['Accepted'].isin([0])]

fig, ax = plt.subplots(figsize=(12,8))
ax.scatter(positive['Test 1'], positive['Test 2'], s=50, c='b', marker='o', label='Accepted')
ax.scatter(negative['Test 1'], negative['Test 2'], s=50, c='r', marker='x', label='Rejected')
ax.legend()
ax.set_xlabel('Test 1 Score')
ax.set_ylabel('Test 2 Score')
plt.show()
```

哇，这个数据看起来可比前一次的复杂得多。特别地，你会注意到其中没有线性决策界限，来良好的分开两类数据。一个方法是用像逻辑回归这样的线性技术来构造从原始特征的多项式中得到的特征。让我们通过创建一组多项式特征入手吧。

```python
degree = 5
x1 = data2['Test 1']
x2 = data2['Test 2']

data2.insert(3, 'Ones', 1)

for i in range(1, degree):
    for j in range(0, i):
        data2['F' + str(i) + str(j)] = np.power(x1, i-j) * np.power(x2, j)

data2.drop('Test 1', axis=1, inplace=True)
data2.drop('Test 2', axis=1, inplace=True)

data2.head()
```

      Accepted
      Ones
      F10
      F20
      F21
      F30
      F31
      F32
      F40
      F41
      F42
      F43

      0
      1
      1
      0.051267
      0.002628
      0.035864
      0.000135
      0.001839
      0.025089
      0.000007
      0.000094
      0.001286
      0.017551

      1
      1
      1
      -0.092742
      0.008601
      -0.063523
      -0.000798
      0.005891
      -0.043509
      0.000074
      -0.000546
      0.004035
      -0.029801

      2
      1
      1
      -0.213710
      0.045672
      -0.147941
      -0.009761
      0.031616
      -0.102412
      0.002086
      -0.006757
      0.021886
      -0.070895

      3
      1
      1
      -0.375000
      0.140625
      -0.188321
      -0.052734
      0.070620
      -0.094573
      0.019775
      -0.026483
      0.035465
      -0.047494

      4
      1
      1
      -0.513250
      0.263426
      -0.238990
      -0.135203
      0.122661
      -0.111283
      0.069393
      -0.062956
      0.057116
      -0.051818

现在，我们需要修改第1部分的成本和梯度函数，包括正则化项。首先是成本函数：

# regularized cost（正则化代价函数）
$$J\left( \theta  \right)=\frac{1}{m}\sum\limits_{i=1}^{m}{[-{{y}^{(i)}}\log \left( {{h}_{\theta }}\left( {{x}^{(i)}} \right) \right)-\left( 1-{{y}^{(i)}} \right)\log \left( 1-{{h}_{\theta }}\left( {{x}^{(i)}} \right) \right)]}+\frac{\lambda }{2m}\sum\limits_{j=1}^{n}{\theta _{j}^{2}}$$

```python
def costReg(theta, X, y, learningRate):
    theta = np.matrix(theta)
    X = np.matrix(X)
    y = np.matrix(y)
    first = np.multiply(-y, np.log(sigmoid(X * theta.T)))
    second = np.multiply((1 - y), np.log(1 - sigmoid(X * theta.T)))
    reg = (learningRate / (2 * len(X))) * np.sum(np.power(theta[:,1:theta.shape[1]], 2))
    return np.sum(first - second) / len(X) + reg
```

请注意等式中的"reg" 项。还注意到另外的一个“学习率”参数。这是一种超参数，用来控制正则化项。现在我们需要添加正则化梯度函数：

如果我们要使用梯度下降法令这个代价函数最小化，因为我们未对${{\theta }_{0}}$ 进行正则化，所以梯度下降算法将分两种情形：
\begin{align}
  & Repeat\text{ }until\text{ }convergence\text{ }\!\!\{\!\!\text{ } \\
 & \text{     }{{\theta }_{0}}:={{\theta }_{0}}-a\frac{1}{m}\sum\limits_{i=1}^{m}{[{{h}_{\theta }}\left( {{x}^{(i)}} \right)-{{y}^{(i)}}]x_{_{0}}^{(i)}} \\
 & \text{     }{{\theta }_{j}}:={{\theta }_{j}}-a\frac{1}{m}\sum\limits_{i=1}^{m}{[{{h}_{\theta }}\left( {{x}^{(i)}} \right)-{{y}^{(i)}}]x_{j}^{(i)}}+\frac{\lambda }{m}{{\theta }_{j}} \\
 & \text{          }\!\!\}\!\!\text{ } \\
 & Repeat \\
\end{align}

对上面的算法中 j=1,2,...,n 时的更新式子进行调整可得：
${{\theta }_{j}}:={{\theta }_{j}}(1-a\frac{\lambda }{m})-a\frac{1}{m}\sum\limits_{i=1}^{m}{({{h}_{\theta }}\left( {{x}^{(i)}} \right)-{{y}^{(i)}})x_{j}^{(i)}}$

```python
def gradientReg(theta, X, y, learningRate):
    theta = np.matrix(theta)
    X = np.matrix(X)
    y = np.matrix(y)

    parameters = int(theta.ravel().shape[1])
    grad = np.zeros(parameters)

    error = sigmoid(X * theta.T) - y

    for i in range(parameters):
        term = np.multiply(error, X[:,i])

        if (i == 0):
            grad[i] = np.sum(term) / len(X)
        else:
            grad[i] = (np.sum(term) / len(X)) + ((learningRate / len(X)) * theta[:,i])

    return grad
```

就像在第一部分中做的一样，初始化变量。

```python
# set X and y (remember from above that we moved the label to column 0)
cols = data2.shape[1]
X2 = data2.iloc[:,1:cols]
y2 = data2.iloc[:,0:1]

# convert to numpy arrays and initalize the parameter array theta
X2 = np.array(X2.values)
y2 = np.array(y2.values)
theta2 = np.zeros(11)
```

让我们初始学习率到一个合理值。，果有必要的话（即如果惩罚太强或不够强）,我们可以之后再折腾这个。

```python
learningRate = 1
```

现在，让我们尝试调用新的默认为0的theta的正则化函数，以确保计算工作正常。

```python
costReg(theta2, X2, y2, learningRate)
```

    0.6931471805599454

```python
gradientReg(theta2, X2, y2, learningRate)
```

    array([0.00847458, 0.01878809, 0.05034464, 0.01150133, 0.01835599,
           0.00732393, 0.00819244, 0.03934862, 0.00223924, 0.01286005,
           0.00309594])

现在我们可以使用和第一部分相同的优化函数来计算优化后的结果。

```python
result2 = opt.fmin_tnc(func=costReg, x0=theta2, fprime=gradientReg, args=(X2, y2, learningRate))
result2
```

    (array([ 0.53010249,  0.29075567, -1.60725763, -0.5821382 ,  0.01781027,
            -0.21329509, -0.40024142, -1.37144139,  0.02264303, -0.9503358 ,
             0.0344085 ]),
     22,
     1)

最后，我们可以使用第1部分中的预测函数来查看我们的方案在训练数据上的准确度。

```python
theta_min = np.matrix(result2[0])
predictions = predict(theta_min, X2)
correct = [1 if ((a == 1 and b == 1) or (a == 0 and b == 0)) else 0 for (a, b) in zip(predictions, y2)]
accuracy = (sum(map(int, correct)) % len(correct))
print ('accuracy = {0}%'.format(accuracy))
```

    accuracy = 78%

虽然我们实现了这些算法，值得注意的是，我们还可以使用高级Python库像scikit-learn来解决这个问题。

```python
from sklearn import linear_model#调用sklearn的线性回归包
model = linear_model.LogisticRegression(penalty='l2', C=1.0)
model.fit(X2, y2.ravel())
```

    LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
                       intercept_scaling=1, l1_ratio=None, max_iter=100,
                       multi_class='auto', n_jobs=None, penalty='l2',
                       random_state=None, solver='lbfgs', tol=0.0001, verbose=0,
                       warm_start=False)

```python
model.score(X2, y2)
```

    0.6610169491525424

这个准确度和我们刚刚实现的差了好多，不过请记住这个结果可以使用默认参数下计算的结果。我们可能需要做一些参数的调整来获得和我们之前结果相同的精确度。

这就是练习2的全部！ 敬请期待下一个练习：多类图像分类。

```python

```


# Session 13: Mixtures of Bernoulli distributions

------------------------------------------------------
*Introduction to Data Science & Machine Learning*

*Pablo M. Olmos olmos@tsc.uc3m.es*

------------------------------------------------------

In this notebook we will implement the EM algorithm for mixtures of Bernoulli distributions. This model is also known as [*latent class analysis*](https://en.wikipedia.org/wiki/Latent_class_model). As well as being of practical importance on its own right, understading this model and its learning also lay the fundation for **hidden Markov models (HMMs)** over discrete variables. HMMs will be dicussed in future course sessions.

### Mixtures of Bernoulli distributions

Consider a set of i.i.d. $D$-dimensional binary (0-1) vectors. Examples of this kind of data are binary images, binary detection results, or genetic markers. Consider also a mixture of multinomials (or multivariate Bernoullis) model for each of the vectors, $\mathbf{x}$,

$$\displaystyle p(\mathbf{x} | \boldsymbol{\Theta},\mathbf{\pi} ) = \sum_{k=1}^K \pi_k p_k(\mathbf{x}^{(i)}|\boldsymbol{\theta}_k) = \sum_{k=1}^K \pi_k \prod_{j=1}^D \theta_{jk}^{x_{j}} (1-\theta_{jk})^{1-x_{j}},$$
where $\boldsymbol{\Theta}=[\boldsymbol{\theta}_1,\ldots,\boldsymbol{\theta}_K]$. If we are given a data set $\mathbf{X}=\{\mathbf{x}^{(1)},\ldots,\mathbf{x}^{(N)}\}$ then the log likelihood function for this model is given by

\begin{align}
\log p(\mathbf{X}|\boldsymbol{\Theta},\mathbf{\pi}) = \sum_{i=1}^N \log\left(\sum_{k=1}^D \pi_k p(\mathbf{x}^{(i)}|\boldsymbol{\theta}_k)\right)
\end{align}

To prevent overfitting, we will also use a **prior distribution** for the model parameters. For $\boldsymbol{\Theta}$ we have

\begin{align}
\theta_{jk}&\sim\text{Beta}(\alpha,\beta)\\
p(\boldsymbol{\Theta}) &= \prod_{k=1}^{K} \prod_{j=1}^D p(\theta_{jk})\\
p(\theta_{jk}) &= \frac{\theta_{jk}^{\alpha-1}(1-\theta_{jk})^{\beta-1}}{\text{B}(\alpha,\beta)},
\end{align}
where $\text{B}(\alpha,\beta)$ is the [Beta function](https://en.wikipedia.org/wiki/Beta_function). Check [here](https://en.wikipedia.org/wiki/Beta_distribution) for more details about the Beta distribution.

For $\mathbf{\pi}$ we use a uniform [Dirichlet distribution](https://en.wikipedia.org/wiki/Dirichlet_distribution):
\begin{align}
\mathbf{\pi}&\sim \text{Dir}(\frac{1}{K}, \ldots, \frac{1}{K}) \Rightarrow p(\mathbf{\pi}) =\frac{1}{\text{B}(\frac{1}{K}, \ldots, \frac{1}{K})}\prod_{k=1}^{K}\pi_k^{\frac{1}{K}-1},
\end{align}
where $\text{B}(\frac{1}{K}, \ldots, \frac{1}{K})$ is the multivariate Beta function.

### EM learning

We now derive the EM algorithm for maximizing the posterior distribution $p(\mathbf{\Theta},\mathbf{\pi}|\mathbf{X})$. To do this, we introduce an explict discrete latent variable $z\in\{1,\ldots,K\}$ assoacted to each data point $\mathbf{x}$:
\begin{align}
p(\mathbf{x},z) = \prod_{k=1}^{K} \left(\pi_k ~p_{k}(\mathbf{x})\right)^{\mathbb{1} [z==k]}, ~~ p(z)=\prod_{k=1}^K \pi_k^{\mathbb{1} [z==k]}
\end{align}

#### Complete log-likelihood

We write the complete data log-likelihood as follows:
\begin{align}
\log p(\mathbf{X},\mathbf{z}|\boldsymbol{\Theta},\mathbf{\pi})=\sum_{i=1}^{N}\sum_{k=1}^{K}\mathbb{1} [z^{(i)}==k]\left(\log \pi_k + \sum_{j=1}^D \left[x_{j}^{(i)}\log\theta_k+(1-x_{j}^{(i)})\log(1-\theta_k)\right]\right)
\end{align}

#### Posterior distribution of $\mathbf{z}$ given $\boldsymbol{\Theta},\mathbf{\pi}$

In the $E$, we compute the expected complete data log-likelihood w.r.t. posterior distribution of $\mathbf{z}$ given the current values of $\boldsymbol{\Theta},\mathbf{\pi}$:
\begin{align}
p(z^{(i)}=k|\mathbf{x}^{(i)},\boldsymbol{\Theta}_{(t-1)},\mathbf{\pi}_{(t-1)}) \triangleq r_{ik} = \frac{\pi_{(k,t-1)} p_k(\mathbf{x}|\boldsymbol{\theta}_k) }{\sum_{q=1}^K \pi_{(q,t-1)} p_k(\mathbf{x}^{(i)}|\boldsymbol{\theta}_q)}, ~~~ k=1,\ldots, K
\end{align}

#### E-step
It is easy to show that
\begin{align}
\mathcal{Q}(\boldsymbol{\Theta},\mathbf{\pi},\boldsymbol{\Theta}_{(t-1)},\mathbf{\pi}_{(t-1)})&=\mathbb{E}_{p(\mathbf{z}|\mathbf{X}^{(i)},\boldsymbol{\Theta}_{(t-1)},\mathbf{\pi}_{(t-1)})}[\log p(\mathbf{X},\mathbf{z}|\boldsymbol{\Theta},\mathbf{\pi})]\\
&=\sum_{i=1}^{N}\sum_{k=1}^K r_{ik} \left(\log \pi_k + \sum_{j=1}^D \left[x_{j}^{(i)}\log\theta_k+(1-x_{j}^{(i)})\log(1-\theta_k)\right]\right)
\end{align}

#### M-step

We have to find
\begin{align}
\boldsymbol{\Theta}_t,\mathbf{\pi}_t = \arg \max_{\boldsymbol{\Theta},\mathbf{\pi}} ~~\mathcal{Q}(\boldsymbol{\Theta},\mathbf{\pi},\boldsymbol{\Theta}_{(t-1)},\mathbf{\pi}_{(t-1)})+ \log p(\boldsymbol{\Theta})+\log p(\mathbf{\pi})
\end{align}

As a result, one can prove that the maximum is attained at

\begin{align}
r_k &\triangleq \sum_{i=1}^N r_{ik} \\\\
\pi^t_k &= \frac{r_k+\frac{1}{K}-1}{N+1-K}\\\\
\boldsymbol{\theta}_k &= \frac{\sum_{i=1}^N r_{ik}\mathbf{x}^{(i)}+\alpha-1}{r_k+\alpha+\beta-2}
\end{align}

To see details about the derivation of this result, check out chapter 9 of Bishop's book. Also Chapter 11 of Murphy's book.

## Implementation of the E-step

We first write a Python function that evaluates the responsibilities $r_{ik}$, $i=1,\ldots,N$, $k=1,\ldots,K$ for each data point. It should take as input the current values of $\boldsymbol{\pi}$, $\boldsymbol{\Theta}$, and the matrix $\mathbf{X}$ of observations.

```python
import matplotlib.pyplot as plt
import numpy as np
from IPython.display import Image
import matplotlib.mlab as mlab
from mpl_toolkits.mplot3d import Axes3D

%matplotlib inline
```

```python

# THE FOLLOWING FUNCTION takes the whole matrix of data points and the vector
# of probabilites of a single cluster. The function returns the probability of each data
# point given cluster parameters p(x|theta_k)

def eval_bern_pdf(X,Thetak):
    #Your code here
    M = np.exp(X*np.log(Thetak)+(1-X)*np.log(1-Thetak))
    pmf = np.prod(M,1)
    return pmf

# THE FOLLOWING FUNCTION takes the whole matrix of data points, the vector
# of probabilites of a single cluster and the cluster probabilites. The function  returns
# the matrix of responsibilites

def responsibilities(X,P,Theta,K):
    N,D = X.shape
    R = np.zeros([N,K])

    for k in range(K):
        #Your code here

        R[:,k] = eval_bern_pdf(X,Theta[k,:])

    R *= P.T
    R /= np.sum(R,1).reshape([-1,1])

    return R

```

```python
#Test your code with the following example

N = 2
D = 5
K = 3

np.random.seed(10)

X = np.random.randint(0,2,[N,D])
Theta = np.random.rand(K,D)

print(Theta)

P = np.random.rand(K,1)
P /= np.sum(P)

print(P)

R = responsibilities(X,P,Theta,K)

# R should be a [N,K] matrix with the following values
# [[  2.20811374e-02   2.19873976e-01   7.58044887e-01]
#  [  7.49839314e-04   3.34167429e-01   6.65082732e-01]]

print(X)

print(R)
```

    [[0.22479665 0.19806286 0.76053071 0.16911084 0.08833981]
     [0.68535982 0.95339335 0.00394827 0.51219226 0.81262096]
     [0.61252607 0.72175532 0.29187607 0.91777412 0.71457578]]
    [[0.51277512]
     [0.13436922]
     [0.35285566]]
    [[1 1 0 1 0]
     [1 1 0 1 1]]
    [[2.20811374e-02 2.19873976e-01 7.58044887e-01]
     [7.49839314e-04 3.34167429e-01 6.65082732e-01]]

## Implementation of the M step

```python
# COMPLETE THE FOLLOWING FUNCTION, which computes the the soft number of points
# associated to each cluster given the matrix R

def points_cluster(R):
    #Your code here
    return np.sum(R,0)

# COMPLETE THE FOLLOWING FUNCTION, which updates the value of the cluster probabilites

def new_P(R,N):
    #Your code here
    Rk = points_cluster(R)
    return Rk/N

def new_Theta(R,X,K,alpha,beta):

    Theta = np.zeros([K,D])
    Rk = points_cluster(R)

    for k in range(K):
        #Your code here
        Theta[k,:] = (np.sum(X*R[:,k].reshape([-1,1]),0)+alpha-1) / (Rk[k]+alpha+beta-2)

    return Theta

```

```python
alpha = 2.
beta = 2.

Theta = new_Theta(R,X,K,alpha,beta)

print(Theta)

# Theta should be a [K,D] matrix with the following values
# [[ 0.50564332  0.50564332  0.49435668  0.50564332  0.49472737]
# [ 0.60846367  0.60846367  0.39153633  0.60846367  0.52237502]
#  [ 0.70786949  0.70786949  0.29213051  0.70786949  0.48642146]]

```

    [[ 0.50564332  0.50564332  0.49435668  0.50564332  0.49472737]
     [ 0.60846367  0.60846367  0.39153633  0.60846367  0.52237502]
     [ 0.70786949  0.70786949  0.29213051  0.70786949  0.48642146]]

## Computing the log-likelihood

```python
# COMPLETE THE FOLLOWING FUNCTION, which computes the model log-likelihood
# given the model parameters

def log_lik(X,P,Theta,K):

    LL = 0.

    for k in range(K):
        #Your code here
        LL += eval_bern_pdf(X,Theta[k,:])*P[k]

    return np.sum(np.log(LL)) #Your code here

```

```python
#For the toy example the LL should be -5.31111037373

print(log_lik(X,P,Theta,K))
```

    -5.31111037373

## EM for a synthetic dataset

Let's generate samples from a mixtures of Bernoulli distributions and run the EM algorithm to recover the true model parameters.

```python
## True model parameters
K = 5
D = 100
N = 5000
X = []

Theta_true = np.random.rand(K,D)
P_true = np.random.rand(K,1)
P_true /= np.sum(P_true)

N_Z_true = np.random.multinomial(N,P_true.reshape([-1,]))

for k in range(K):
    X.append(np.random.rand(N_Z_true[k],D)<=Theta_true[k,:])
    print(X[k].shape)

X = np.concatenate(X)
```

    (1137, 100)
    (352, 100)
    (2184, 100)
    (1259, 100)
    (68, 100)

```python
## EM initialization
K = 5

P = np.random.rand(K,1)
P /= np.sum(P_true)

Theta = np.ones([K,D])*1.0/K

Num_Iter = 100

alpha = 0.2
beta = 0.2

LL = np.zeros([Num_Iter,1])

for i in range(Num_Iter):

    LL[i] = log_lik(X,P,Theta,K)

    R = responsibilities(X,P,Theta,K)

    P = new_P(R,N)

    Theta = new_Theta(R,X,K,alpha,beta)

```

```python
# Plot the evolution of the negative LL. It should always decrease!

plt.semilogy(-LL,'-*')
```

```python
plt.plot(P_true,label='true probs', color = [0, 0.5, 0])
plt.plot(P,label='EM probs', color = [1, 0, 0])
plt.legend()
plt.title('Cluster Probabilites')
```

## EM for a data set of handwritten digits

Load and run the EM over the [sklearn dataset of digits](http://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html#sklearn.datasets.load_digits). Each datapoint is a 8x8 image of a digit.

```python
from sklearn.datasets import load_digits
digits = load_digits()    #Gray scale, we have to binarize
print(digits.data.shape)

plt.gray()
plt.matshow(digits.images[15])
plt.show()

#Binarization

Bin_Images = np.copy(digits.images)

val_min=np.min(Bin_Images)
val_max=np.max(Bin_Images)

Bin_Images = (Bin_Images - val_min) / (val_max - val_min)

Bin_Images = np.round(Bin_Images)

plt.gray()
plt.matshow(Bin_Images[15])
plt.show()

```

```python
# EM

K = 10
D = 64
X = Bin_Images.reshape([-1,D])

P = np.random.rand(K,1)
P /= np.sum(P_true)

Theta = np.ones([K,D])*1.0/K

Num_Iter = 1000

alpha = 2.
beta = 2.

LL = np.zeros([Num_Iter,1])

for i in range(Num_Iter):

    LL[i] = log_lik(X,P,Theta,K)

    R = responsibilities(X,P,Theta,K)

    P = new_P(R,N)

    Theta = new_Theta(R,X,K,alpha,beta)

# Plot the evolution of the negative LL. It should always decrease!
plt.semilogy(-LL,'-*')

```

```python
# Plot as a 8x8 gray-scale image each theta_k vector
# (it represents the probability of each pixel to take value 1)

for k in range(K):
    plt.gray()
    plt.matshow(Theta[k,:].reshape([8,8]))
    plt.show()
```

## EM for the MNIST Database (optional)

Repeat the experiment for the MNIST Database. Take $10^4$ images **at random**, and binarize the database.

```python
from sklearn.datasets import fetch_mldata
mnist = fetch_mldata('MNIST original')
```

```python
images = mnist.data  #70000 images

plt.gray()
plt.matshow(images[0,:].reshape([28,28]))
plt.show()

val_min=np.min(images)
val_max=np.max(images)

Bin_Images = (images - val_min) / (val_max - val_min)

Bin_Images = np.round(Bin_Images)

plt.gray()
plt.matshow(Bin_Images[0,:].reshape([28,28]))
plt.show()

mask = np.random.permutation(Bin_Images.shape[0])

Bin_Images = Bin_Images[mask,:]
```

```python
# EM

K = 10
D = 784
N = 1000

X = Bin_Images[:N,:]

P = np.random.rand(K,1)
P /= np.sum(P_true)

Theta = 0.5/K*np.ones([K,D])+0.5*np.random.rand(K,D)

Num_Iter = 1000

alpha = 2.1
beta = 2.1

LL = np.zeros([Num_Iter,1])

for i in range(Num_Iter):

    LL[i] = log_lik(X,P,Theta,K)

    R = responsibilities(X,P,Theta,K)

    P = new_P(R,N)

    Theta = new_Theta(R,X,K,alpha,beta)

    if(i % 20 ==0):
        print('Iteration ',i)

```

    Iteration  0
    Iteration  20
    Iteration  40
    Iteration  60
    Iteration  80
    Iteration  100
    Iteration  120
    Iteration  140
    Iteration  160
    Iteration  180
    Iteration  200
    Iteration  220
    Iteration  240
    Iteration  260
    Iteration  280
    Iteration  300
    Iteration  320
    Iteration  340
    Iteration  360
    Iteration  380
    Iteration  400
    Iteration  420
    Iteration  440
    Iteration  460
    Iteration  480
    Iteration  500
    Iteration  520
    Iteration  540
    Iteration  560
    Iteration  580
    Iteration  600
    Iteration  620
    Iteration  640
    Iteration  660
    Iteration  680
    Iteration  700
    Iteration  720
    Iteration  740
    Iteration  760
    Iteration  780
    Iteration  800
    Iteration  820
    Iteration  840
    Iteration  860
    Iteration  880
    Iteration  900
    Iteration  920
    Iteration  940
    Iteration  960
    Iteration  980

```python
# Plot the evolution of the negative LL. It should always decrease!
plt.plot(-LL,'-*')
```

```python
# Plot as a 8x8 gray-scale image each theta_k vector
# (it represents the probability of each pixel to take value 1)

for k in range(K):
    plt.gray()
    plt.matshow(Theta[k,:].reshape([28,28]))
    plt.show()
```

```python

```

Notebooks/Session 13 Mixture of Bernoullis/.ipynb_checkpoints/Session_13_EM_mix_Bern-checkpoint.ipynb


# Solving a diffusion PDE on a GAMer mesh using FEniCS
This tutorial will walk you through the process of setting up and solving a diffusion partial differential equation (PDE) using a mesh imported from GAMer into FEniCS

```python
import pygamer
import math
```

## Generating the mesh using GAMer

```python
# generate the surface mesh for a cube
smesh = pygamer.surfacemesh.cube(5)

# Mark sides of the face to use later for boundary conditions
for fid in smesh.faceIDs:
    indices = fid.indices()
    z_coords = [smesh.getVertex([idx]).data().position[2] for idx in list(indices)]
    if all([z==1 for z in z_coords]):
        fid.data().marker = 2
    elif all([z==-1 for z in z_coords]):
        fid.data().marker = 3
    else:
        fid.data().marker = 4

# construct the volumetric mesh
vmesh = pygamer.makeTetMesh([smesh], "q1.1/15O10/2ACVY")

# write volumetric mesh into dolfin readable format
pygamer.writeDolfin('outputTetMesh.xml', vmesh)
pygamer.writeVTK('outputTetMesh.vtk', vmesh)
```

    Number of vertices: 1538
    Number of Faces: 3072
    Number of Regions: 1
    Number of Holes: 0
    Region point: Tensor({0}:-0.875; {1}:0.333333; {2}:-0.333333)

## Setup the model problem in FEniCS

### Model problem: strong formulation

We will model diffusion in a $2 \times 2 \times 2$ cube with an exponentially decaying source term centered at the origin, an influx boundary condition at the $z=1$ surface, an outflux boundary condition at the $z=-1$ surface, and a $u=0$ Dirichlet boundary condition on the four other sides of the cube.

Find $u(\mathbf{x},t)$ such that

\begin{align}
\frac{\partial u}{\partial t} &= D\nabla^2 u + f ~~\text{in}~~ \Omega \\
D(n \cdot \nabla u) &= j_\text{in} ~~\text{on}~~ \partial\Omega_\text{in} \\
D(n \cdot \nabla u) &= j_\text{out} ~~\text{on}~~ \partial\Omega_\text{out} \\
u(\mathbf{x},t=0) &= u_0 \\
u(\mathbf{x},t) &= 0 ~~\text{on}~~ \partial\Omega_\text{dirichlet} \\
\partial\Omega &= \partial\Omega_\text{in} \cup \partial\Omega_\text{out} \cup \partial\Omega_\text{dirichlet}
\end{align}

where $f$ is the source term, $f(\mathbf{x}) = 32 - e^{||\mathbf{x}||}$, $D$ is the diffusion coefficient, $j_\text{in}$ and $j_\text{out}$ are the influx/outflux rates for the Neumann boundary conditions, and $u_0$ is the initial condition.

```python
import dolfin as d

# Model parameters
D    = 1 # diffusion coefficient
u0   = d.Constant(0.001) # initial condition
dt   = 0.1 # time step size
T    = 1.0 # final time
jin  = 10.0 # influx rate
jout = -50.0 # outflux rate
dist = d.Expression("sqrt(pow(x[0],2) + pow(x[1],2) + pow(x[2],2))", degree=1)
f    = d.Expression("32 - exp(dist)", dist=dist, degree=1) # source term
t    = 0.0 # current time

numsteps = round(T/dt)
```

### Model problem: weak/variational formulation

The finite element method uses a variational formulation of the PDE which allows us to search for solutions from a larger function space. Equations input into FEniCS are done so through their variational formulation; we will now show how to convert our problem into a valid FEniCS input.

First we multiply the governing PDE by a test function, $v$, coming from a function space $V$, integrate over the domain and apply the divergence theorem to obtain the following variational formulation:

\begin{equation}
\int_\Omega \frac{\partial u}{\partial t}v \,dx = -\int_\Omega D\nabla u \cdot \nabla v \,dx + \int_{\partial\Omega} D(n\cdot\nabla u)v \,ds + \int_\Omega fv \,dx ~~\text{in}~~ \Omega
\end{equation}

We discretize the time-derivative with a backward Euler scheme due to its simplicity and unconditional stability properties

\begin{equation}
\frac{\partial u}{\partial t} \approx \frac{u - u_n}{\Delta t}
\end{equation}

where $u_n$ represents the (known) solution computed at the previous timestep. We further simplify the variational formulation by inserting the Neumann boundary conditions and using the shorthand notation where $\langle \cdot , \cdot \rangle$ represents the inner-product over $\Omega$ and $\langle \cdot , \cdot \rangle_{\partial\Omega_\text{in}}$ represents the inner-product on the boundary $\partial\Omega_\text{in}$. Terms are separated onto the left or right hand sides by their dependence on $u$:

\begin{equation}
\langle u,v \rangle  + D\Delta t \langle \nabla u,\nabla v \rangle = \Delta t\langle f,v \rangle +  \Delta t \langle j_\text{in},v \rangle_{\partial\Omega_\text{in}} + \Delta t \langle j_\text{out},v \rangle_{\partial\Omega_\text{out}} + \langle u_n,v \rangle
\end{equation}

Notice that since $u_n$ is a known value it is not dependent on the unknown, $u$, and therefore is placed on the right hand side. Also, since $D(\mathbf{n}\cdot \nabla u)=0$ on the no-flux boundary those terms drop out of the variational formulation.

In the abstract form this is written as,

\begin{align}
a(u,v) &= \langle u,v \rangle  + D\Delta t \langle \nabla u,\nabla v \rangle \\
L(v) &= \Delta t\langle f,v \rangle + \Delta t \langle j_\text{in},v \rangle_{\partial\Omega_\text{in}} + \Delta t \langle j_\text{out},v \rangle_{\partial\Omega_\text{out}} + \langle u_n,v \rangle
\end{align}

where $a(u,v)$ is a bilinear form and $L(v)$ is a linear functional. Note that the Dirichlet boundary condition does not appear in the variational form and therefore must be enforced separately.

```python
# Import the mesh and construct a linear Lagrange function space over the mesh
dolfin_mesh = d.Mesh('outputTetMesh.xml')
V = d.FunctionSpace(dolfin_mesh,'P',1)
```

```python
# Define trial and test functions
u = d.TrialFunction(V)
v = d.TestFunction(V)
# Known solution corresponding to previous timestep. Initialize with initial condition
un = d.interpolate(u0,V)
```

```python
# Earlier we marked faces on the influx boundary with 2 and faces on the outflux boundaries as 3

# define integration domains
face_dimension = 2 # we marked faces which have a topological dimension of 2
meshBoundary = d.MeshFunction("size_t",dolfin_mesh,face_dimension,dolfin_mesh.domains())
dx = d.Measure('dx',domain=dolfin_mesh)
ds = d.Measure('ds',domain=dolfin_mesh,subdomain_data=meshBoundary)
```

```python
# define bilinear and linear forms
a = u*v*dx + D*dt*d.inner(d.grad(u),d.grad(v))*dx
L = dt*f*v*dx + dt*jin*v*ds(2) + dt*jout*v*ds(3) + un*v*dx
```

### Solve and store solution

Solutions are saved in the $\texttt{.vtk}$ format and can be visualized using software such as Paraview

```python
filename = 'solution/dolfinOut.pvd'
vtkfile = d.File(filename)

# store the initial condition
u = d.Function(V)
un.rename("u", "solution")
u.rename("u", "solution")
vtkfile << (un,t)

bc = d.DirichletBC(V,d.Constant(0),meshBoundary,4)

# Main loop
for idx in range(numsteps):
    # step forward in time
    t += dt

    d.solve(a==L,u,bcs=bc)

    # save solution to file
    vtkfile << (u,t)

    # assign just computed solution to un
    un.assign(u)

    print('Time step %d complete ...' % int(idx+1))
```

    Time step 1 complete ...
    Time step 2 complete ...
    Time step 3 complete ...
    Time step 4 complete ...
    Time step 5 complete ...
    Time step 6 complete ...
    Time step 7 complete ...
    Time step 8 complete ...
    Time step 9 complete ...
    Time step 10 complete ...

```python

```


```python
from sympy import *
init_printing()
x, y, t = symbols('x y t')
gamma_s, gamma_g, pi_s, pi_g = symbols('gamma_s gamma_g pi_s pi_g')
```

```python
# Conserved Variables

q_1, q_2, q_3, q_4, q_5, q_6, q_7 = symbols('q_1 q_2 q_3 q_4 q_5 q_6 q_7')

# Physical variables

phi_s = q_7
phi_g = 1 - q_7

rho_s = q_1/phi_s;
rho_g = q_4/phi_g;

u_s = q_2/q_1;
u_g = q_5/q_4;

E_s = q_3/phi_s;
E_g = q_6/phi_g;

p_s = (gamma_s - 1)*(E_s - Rational(1,2)*rho_s*u_s*u_s) - gamma_s*pi_s;
p_g = (gamma_g - 1)*(E_g - Rational(1,2)*rho_g*u_g*u_g) - gamma_g*pi_g;

# Conserved fluxes

f_1 = phi_s*rho_s*u_s
f_2 = phi_s*(rho_s*u_s*u_s + p_s)
f_3 = phi_s*u_s*(E_s + p_s)

f_4 = phi_g*rho_g*u_g
f_5 = phi_g*(rho_g*u_g*u_g + p_g)
f_6 = phi_g*u_g*(E_g + p_g)

f_7 = 0

# Primitive variables

v_1 = rho_s

v_2 = u_s
v_3 = p_s
v_4 = rho_g

v_5 = u_g
v_6 = p_g
v_7 = phi_s

Q = Matrix([q_1, q_2, q_3, q_4, q_5, q_6, q_7])
V = Matrix([v_1, v_2, v_3, v_4, v_5, v_6, v_7])
F = Matrix([f_1, f_2, f_3, f_4, f_5, f_6, f_7])

# Matrix B

B = zeros(7, 7)

u_I = u_s
p_I = p_g

B[1,6] = -p_I;
B[2,6] = -p_I*u_I;
B[4,6] =  p_I;
B[5,6] =  p_I*u_I;
B[6,6] =  u_I;

A = F.jacobian(Q) + B
```

```python
DV_DQ  = simplify(V.jacobian(Q))
```