# About This Notebook

This notebook shows how to implement **Low-Rank Tensor Completion with Truncated Nuclear Norm minimization (LRTC-TNN)** on some real-world data sets. For an in-depth discussion of LRTC-TNN, please see our article [1].

[1] Xinyu Chen, Jinming Yang, Lijun Sun (2020). A Nonconvex Low-Rank Tensor Completion Model for Spatiotemporal Traffic Data Imputation. arXiv.2003.10271. [PDF]

## Quick Run

This notebook is publicly available for any usage at our data imputation project. Please check out [**transdim - GitHub**](https://github.com/xinychen/transdim).

## Low-Rank Tensor Completion

We start by importing the necessary dependencies.

```python
import numpy as np
from numpy.linalg import inv as inv
```

### Tensor Unfolding (`ten2mat`) and Matrix Folding (`mat2ten`)

Using numpy reshape to perform 3rd rank tensor unfold operation. [[**link**](https://stackoverflow.com/questions/49970141/using-numpy-reshape-to-perform-3rd-rank-tensor-unfold-operation)]

```python
def ten2mat(tensor, mode):
    return np.reshape(np.moveaxis(tensor, mode, 0), (tensor.shape[mode], -1), order = 'F')
```

```python
X = np.array([[[1, 2, 3, 4], [3, 4, 5, 6]],
              [[5, 6, 7, 8], [7, 8, 9, 10]],
              [[9, 10, 11, 12], [11, 12, 13, 14]]])
print('tensor size:')
print(X.shape)
print('original tensor:')
print(X)
print()
print('(1) mode-1 tensor unfolding:')
print(ten2mat(X, 0))
print()
print('(2) mode-2 tensor unfolding:')
print(ten2mat(X, 1))
print()
print('(3) mode-3 tensor unfolding:')
print(ten2mat(X, 2))
```

    tensor size:
    (3, 2, 4)
    original tensor:
    [[[ 1  2  3  4]
      [ 3  4  5  6]]

     [[ 5  6  7  8]
      [ 7  8  9 10]]

     [[ 9 10 11 12]
      [11 12 13 14]]]

    (1) mode-1 tensor unfolding:
    [[ 1  3  2  4  3  5  4  6]
     [ 5  7  6  8  7  9  8 10]
     [ 9 11 10 12 11 13 12 14]]

    (2) mode-2 tensor unfolding:
    [[ 1  5  9  2  6 10  3  7 11  4  8 12]
     [ 3  7 11  4  8 12  5  9 13  6 10 14]]

    (3) mode-3 tensor unfolding:
    [[ 1  5  9  3  7 11]
     [ 2  6 10  4  8 12]
     [ 3  7 11  5  9 13]
     [ 4  8 12  6 10 14]]

```python
def mat2ten(mat, tensor_size, mode):
    index = list()
    index.append(mode)
    for i in range(tensor_size.shape[0]):
        if i != mode:
            index.append(i)
    return np.moveaxis(np.reshape(mat, list(tensor_size[index]), order = 'F'), 0, mode)
```

### Singular Value Thresholding (SVT) for TNN

```python
def svt_tnn(mat, alpha, rho, theta):
    """This is a Numpy dependent singular value thresholding (SVT) process."""
    u, s, v = np.linalg.svd(mat, full_matrices = 0)
    vec = s.copy()
    vec[theta :] = s[theta :] - alpha / rho
    vec[vec < 0] = 0
    return np.matmul(np.matmul(u, np.diag(vec)), v)
```

**Understanding these codes**:

- **`line 1`**: Necessary inputs including any input matrix $\boldsymbol{X}$, weight of Truncated Nuclear Norm (TNN) regularization $\alpha$, learning rate $\rho$, and positive integer number $\theta$ for nuclear norm truncation.

- **`line 2`**: Compute the Singular Value Decomposition (SVD) for any matrix $\boldsymbol{X}$ with `numpy.linalg.svd` (i.e., SVD function in `Numpy`'s linear algebra package).

- **`line 3-5`**: Truncate singular values $\sigma_{\theta+1},...$ with the following rule:

\begin{equation}
\sigma_{i}=\left[\sigma_{i}(\boldsymbol{X})-\frac{\alpha}{\rho}\right]_{+}.
\end{equation}

- **`line 6`**: Return the resulted matrix.

**Potential alternative for this**:

This is a competitively efficient algorithm for implementing SVT-TNN.

```python
def svt_tnn(mat, alpha, rho, theta):
    tau = alpha / rho
    [m, n] = mat.shape
    if 2 * m < n:
        u, s, v = np.linalg.svd(mat @ mat.T, full_matrices = 0)
        s = np.sqrt(s)
        idx = np.sum(s > tau)
        mid = np.zeros(idx)
        mid[:theta] = 1
        mid[theta:idx] = (s[theta:idx] - tau) / s[theta:idx]
        return (u[:,:idx] @ np.diag(mid)) @ (u[:,:idx].T @ mat)
    elif m > 2 * n:
        return svt_tnn(mat.T, tau, theta).T
    u, s, v = np.linalg.svd(mat, full_matrices = 0)
    idx = np.sum(s > tau)
    vec = s[:idx].copy()
    vec[theta:] = s[theta:] - tau
    return u[:,:idx] @ np.diag(vec) @ v[:idx,:]
```

compute_mape: Compute the value of Mean Absolute Percentage Error (MAPE).

compute_rmse: Compute the value of Root Mean Square Error (RMSE).

> Note that $$\mathrm{MAPE}=\frac{1}{n} \sum_{i=1}^{n} \frac{\left|y_{i}-\hat{y}_{i}\right|}{y_{i}} \times 100, \quad\mathrm{RMSE}=\sqrt{\frac{1}{n} \sum_{i=1}^{n}\left(y_{i}-\hat{y}_{i}\right)^{2}},$$ where $n$ is the total number of estimated values, and $y_i$ and $\hat{y}_i$ are the actual value and its estimation, respectively.

```python
def compute_rmse(var, var_hat):
    return np.sqrt(np.sum((var - var_hat) ** 2) / var.shape[0])
```

```python
def compute_mape(var, var_hat):
    return np.sum(np.abs(var - var_hat) / var) / var.shape[0]
```

### Define LRTC-TNN Function with `Numpy`

```python
def LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter):
    """Low-Rank Tenor Completion with Truncated Nuclear Norm, LRTC-TNN."""

    dim = np.array(sparse_tensor.shape)
    pos_missing = np.where(sparse_tensor == 0)
    pos_test = np.where((dense_tensor != 0) & (sparse_tensor == 0))

    X = np.zeros(np.insert(dim, 0, len(dim))) # \boldsymbol{\mathcal{X}}
    T = np.zeros(np.insert(dim, 0, len(dim))) # \boldsymbol{\mathcal{T}}
    Z = sparse_tensor.copy()
    last_tensor = sparse_tensor.copy()
    snorm = np.sqrt(np.sum(sparse_tensor ** 2))
    it = 0
    while True:
        rho = min(rho * 1.05, 1e5)
        for k in range(len(dim)):
            X[k] = mat2ten(svt_tnn(ten2mat(Z - T[k] / rho, k), alpha[k], rho, np.int(np.ceil(theta * dim[k]))), dim, k)
        Z[pos_missing] = np.mean(X + T / rho, axis = 0)[pos_missing]
        T = T + rho * (X - np.broadcast_to(Z, np.insert(dim, 0, len(dim))))
        tensor_hat = np.einsum('k, kmnt -> mnt', alpha, X)
        tol = np.sqrt(np.sum((tensor_hat - last_tensor) ** 2)) / snorm
        last_tensor = tensor_hat.copy()
        it += 1
        if (it + 1) % 50 == 0:
            print('Iter: {}'.format(it + 1))
            print('RMSE: {:.6}'.format(compute_rmse(dense_tensor[pos_test], tensor_hat[pos_test])))
            print()
        if (tol < epsilon) or (it >= maxiter):
            break

    print('Imputation MAPE: {:.6}'.format(compute_mape(dense_tensor[pos_test], tensor_hat[pos_test])))
    print('Imputation RMSE: {:.6}'.format(compute_rmse(dense_tensor[pos_test], tensor_hat[pos_test])))
    print()

    return tensor_hat
```

**Understanding these codes**:

- **`line 18-19`**: Update $\boldsymbol{\mathcal{Z}}_{k}^{l+1},k=1,2,3$.

- **`line 20-22`**: Update $\boldsymbol{\mathcal{X}}_{k}^{l+1}$ by

\begin{equation}
\boldsymbol{\mathcal{X}}_{k}^{l+1}=\mathcal{P}_{\Omega}(\boldsymbol{\mathcal{Y}})+\mathcal{P}_{\Omega}^{\perp}\left(\boldsymbol{\mathcal{Z}}_{k}^{l+1}-\frac{1}{\rho}\boldsymbol{\mathcal{T}}_{k}^{l}\right),k=1,2,3.
\end{equation}

- **`line 23`**: Update $\boldsymbol{\mathcal{T}}_{k}^{l+1}$ by

\begin{equation}
\boldsymbol{\mathcal{T}}_{k}^{l+1}=\boldsymbol{\mathcal{T}}_{k}^{l}+\rho_k\left(\boldsymbol{\mathcal{X}}_{k}^{l+1}-\boldsymbol{\mathcal{Z}}_{k}^{l+1}\right).
\end{equation}

## Data Organization

### 1) Matrix Structure

We consider a dataset of $m$ discrete time series $\boldsymbol{y}_{i}\in\mathbb{R}^{f},i\in\left\{1,2,...,m\right\}$. The time series may have missing elements. We express spatio-temporal dataset as a matrix $Y\in\mathbb{R}^{m\times f}$ with $m$ rows (e.g., locations) and $f$ columns (e.g., discrete time intervals),

$$Y=\left[ \begin{array}{cccc} y_{11} & y_{12} & \cdots & y_{1f} \\ y_{21} & y_{22} & \cdots & y_{2f} \\ \vdots & \vdots & \ddots & \vdots \\ y_{m1} & y_{m2} & \cdots & y_{mf} \\ \end{array} \right]\in\mathbb{R}^{m\times f}.$$

### 2) Tensor Structure

We consider a dataset of $m$ discrete time series $\boldsymbol{y}_{i}\in\mathbb{R}^{nf},i\in\left\{1,2,...,m\right\}$. The time series may have missing elements. We partition each time series into intervals of predifined length $f$. We express each partitioned time series as a matrix $Y_{i}$ with $n$ rows (e.g., days) and $f$ columns (e.g., discrete time intervals per day),

$$Y_{i}=\left[ \begin{array}{cccc} y_{11} & y_{12} & \cdots & y_{1f} \\ y_{21} & y_{22} & \cdots & y_{2f} \\ \vdots & \vdots & \ddots & \vdots \\ y_{n1} & y_{n2} & \cdots & y_{nf} \\ \end{array} \right]\in\mathbb{R}^{n\times f},i=1,2,...,m,$$

therefore, the resulting structure is a tensor $\mathcal{Y}\in\mathbb{R}^{m\times n\times f}$.

## Missing Data Imputation

In the following, we apply the above defined TRMF function to the task of missing data imputation task on the following spatiotemporal multivariate time series datasets/matrices:

- **Guangzhou data set**: [Guangzhou urban traffic speed data set](https://doi.org/10.5281/zenodo.1205228).
- **Birmingham data set**: [Birmingham parking data set](https://archive.ics.uci.edu/ml/datasets/Parking+Birmingham).
- **Hangzhou data set**: [Hangzhou metro passenger flow data set](https://doi.org/10.5281/zenodo.3145403).
- **Settle data set**: [Seattle freeway traffic speed data set](https://github.com/zhiyongc/Seattle-Loop-Data).

The original data sets have been adapted into our experiments, and it is now available at the fold of `datasets`.

### Experiments on Guangzhou Data Set

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/random_tensor.mat')
random_tensor = random_tensor['random_tensor']

missing_rate = 0.2

# =============================================================================
### Random missing (RM) scenario:
binary_tensor = np.round(random_tensor + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.30
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 5.46701

    Iter: 100
    RMSE: 2.8773

    Iter: 150
    RMSE: 2.86962

    Imputation MAPE: 0.0670164
    Imputation RMSE: 2.88087

    Running time: 43 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/random_tensor.mat')
random_tensor = random_tensor['random_tensor']

missing_rate = 0.4

# =============================================================================
### Random missing (RM) scenario:
binary_tensor = np.round(random_tensor + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.30
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 5.62032

    Iter: 100
    RMSE: 3.14578

    Iter: 150
    RMSE: 3.15347

    Imputation MAPE: 0.0732347
    Imputation RMSE: 3.16691

    Running time: 40 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/random_tensor.mat')
random_tensor = random_tensor['random_tensor']

missing_rate = 0.5

# =============================================================================
### Random missing (RM) scenario:
binary_tensor = np.round(random_tensor + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.30
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 5.83294

    Iter: 100
    RMSE: 3.28317

    Iter: 150
    RMSE: 3.31253

    Imputation MAPE: 0.076822
    Imputation RMSE: 3.32346

    Running time: 39 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/random_tensor.mat')
random_tensor = random_tensor['random_tensor']

missing_rate = 0.6

# =============================================================================
### Random missing (RM) scenario:
binary_tensor = np.round(random_tensor + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.30
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 5.49785

    Iter: 100
    RMSE: 3.43917

    Iter: 150
    RMSE: 3.49909

    Imputation MAPE: 0.0811862
    Imputation RMSE: 3.51235

    Running time: 40 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/random_tensor.mat')
random_tensor = random_tensor['random_tensor']

missing_rate = 0.7

# =============================================================================
### Random missing (RM) scenario:
binary_tensor = np.round(random_tensor + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.30
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 5.34354

    Iter: 100
    RMSE: 3.5962

    Iter: 150
    RMSE: 3.699

    Imputation MAPE: 0.0860955
    Imputation RMSE: 3.71831

    Running time: 41 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_matrix = scipy.io.loadmat('../datasets/Guangzhou-data-set/random_matrix.mat')
random_matrix = random_matrix['random_matrix']

missing_rate = 0.2

# =============================================================================
### Non-random missing (NM) scenario:
binary_tensor = np.zeros(dense_tensor.shape)
for i1 in range(dense_tensor.shape[0]):
    for i2 in range(dense_tensor.shape[1]):
        binary_tensor[i1, i2, :] = np.round(random_matrix[i1, i2] + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.05
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 4.86922

    Iter: 100
    RMSE: 3.96752

    Iter: 150
    RMSE: 3.97225

    Imputation MAPE: 0.0936998
    Imputation RMSE: 3.97219

    Running time: 34 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_matrix = scipy.io.loadmat('../datasets/Guangzhou-data-set/random_matrix.mat')
random_matrix = random_matrix['random_matrix']
random_tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/random_tensor.mat')
random_tensor = random_tensor['random_tensor']

missing_rate = 0.4

# =============================================================================
### Non-random missing (NM) scenario:
binary_tensor = np.zeros(dense_tensor.shape)
for i1 in range(dense_tensor.shape[0]):
    for i2 in range(dense_tensor.shape[1]):
        binary_tensor[i1, i2, :] = np.round(random_matrix[i1, i2] + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.05
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 4.98029

    Iter: 100
    RMSE: 4.06503

    Iter: 150
    RMSE: 4.07531

    Imputation MAPE: 0.0954183
    Imputation RMSE: 4.07556

    Running time: 36 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_matrix = scipy.io.loadmat('../datasets/Guangzhou-data-set/random_matrix.mat')
random_matrix = random_matrix['random_matrix']

missing_rate = 0.5

# =============================================================================
### Non-random missing (NM) scenario:
binary_tensor = np.zeros(dense_tensor.shape)
for i1 in range(dense_tensor.shape[0]):
    for i2 in range(dense_tensor.shape[1]):
        binary_tensor[i1, i2, :] = np.round(random_matrix[i1, i2] + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.05
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 5.15228

    Iter: 100
    RMSE: 4.14767

    Imputation MAPE: 0.0973779
    Imputation RMSE: 4.16227

    Running time: 36 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_matrix = scipy.io.loadmat('../datasets/Guangzhou-data-set/random_matrix.mat')
random_matrix = random_matrix['random_matrix']

missing_rate = 0.6

# =============================================================================
### Non-random missing (NM) scenario:
binary_tensor = np.zeros(dense_tensor.shape)
for i1 in range(dense_tensor.shape[0]):
    for i2 in range(dense_tensor.shape[1]):
        binary_tensor[i1, i2, :] = np.round(random_matrix[i1, i2] + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.05
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 5.1033

    Iter: 100
    RMSE: 4.22383

    Imputation MAPE: 0.0992274
    Imputation RMSE: 4.24004

    Running time: 42 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_matrix = scipy.io.loadmat('../datasets/Guangzhou-data-set/random_matrix.mat')
random_matrix = random_matrix['random_matrix']

missing_rate = 0.7

# =============================================================================
### Non-random missing (NM) scenario:
binary_tensor = np.zeros(dense_tensor.shape)
for i1 in range(dense_tensor.shape[0]):
    for i2 in range(dense_tensor.shape[1]):
        binary_tensor[i1, i2, :] = np.round(random_matrix[i1, i2] + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.05
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 5.21747

    Iter: 100
    RMSE: 4.30683

    Imputation MAPE: 0.102139
    Imputation RMSE: 4.32667

    Running time: 34 seconds

**Experiment results** of missing data imputation using LRTC-TNN:

|  scenario |`alpha` (vector input)|`rho`|`theta`|`maxiter`|       mape |      rmse |
|:----------|-----:|---------:|---------:|-------- --:|-------- --:|----------:|
|**0.2, RM**| $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$ | 0.00001 | 0.30 | 200 | **0.0670** | **2.88**|
|**0.4, RM**| $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$ | 0.00001 | 0.30 | 200 | **0.0732** | **3.17**|
|**0.2, NM**| $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$ | 0.00001 | 0.05 | 200 | **0.0937** | **3.97**|
|**0.4, NM**| $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$ | 0.00001 | 0.05 | 200 | **0.0954** | **4.08**|

### Experiments on Birmingham Data Set

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Birmingham-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_tensor = scipy.io.loadmat('../datasets/Birmingham-data-set/random_tensor.mat')
random_tensor = random_tensor['random_tensor']

missing_rate = 0.1

# =============================================================================
### Random missing (RM) scenario
### Set the RM scenario by:
binary_tensor = np.round(random_tensor + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.15
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 18.0692

    Iter: 100
    RMSE: 13.0828

    Imputation MAPE: 0.0420645
    Imputation RMSE: 13.1064

    Running time: 0 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Birmingham-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_tensor = scipy.io.loadmat('../datasets/Birmingham-data-set/random_tensor.mat')
random_tensor = random_tensor['random_tensor']

missing_rate = 0.3

# =============================================================================
### Random missing (RM) scenario
### Set the RM scenario by:
binary_tensor = np.round(random_tensor + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.15
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 20.5441

    Iter: 100
    RMSE: 17.4561

    Imputation MAPE: 0.0515019
    Imputation RMSE: 17.4746

    Running time: 0 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Birmingham-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_tensor = scipy.io.loadmat('../datasets/Birmingham-data-set/random_tensor.mat')
random_tensor = random_tensor['random_tensor']

missing_rate = 0.5

# =============================================================================
### Random missing (RM) scenario
### Set the RM scenario by:
binary_tensor = np.round(random_tensor + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.15
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 25.6108

    Iter: 100
    RMSE: 22.6758

    Imputation MAPE: 0.0651507
    Imputation RMSE: 22.6791

    Running time: 0 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Birmingham-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_tensor = scipy.io.loadmat('../datasets/Birmingham-data-set/random_tensor.mat')
random_tensor = random_tensor['random_tensor']

missing_rate = 0.6

# =============================================================================
### Random missing (RM) scenario
### Set the RM scenario by:
binary_tensor = np.round(random_tensor + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.15
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 29.9069

    Iter: 100
    RMSE: 27.1879

    Imputation MAPE: 0.076155
    Imputation RMSE: 27.1492

    Running time: 0 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Birmingham-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_tensor = scipy.io.loadmat('../datasets/Birmingham-data-set/random_tensor.mat')
random_tensor = random_tensor['random_tensor']

missing_rate = 0.7

# =============================================================================
### Random missing (RM) scenario
### Set the RM scenario by:
binary_tensor = np.round(random_tensor + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.15
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 35.5653

    Iter: 100
    RMSE: 33.3886

    Imputation MAPE: 0.0895481
    Imputation RMSE: 33.4512

    Running time: 0 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Birmingham-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_matrix = scipy.io.loadmat('../datasets/Birmingham-data-set/random_matrix.mat')
random_matrix = random_matrix['random_matrix']

missing_rate = 0.1

# =============================================================================
### Non-random missing (NM) scenario
### Set the RM scenario by:
binary_tensor = np.zeros(dense_tensor.shape)
for i1 in range(dense_tensor.shape[0]):
    for i2 in range(dense_tensor.shape[1]):
        binary_tensor[i1, i2, :] = np.round(random_matrix[i1, i2] + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.05
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 22.6221

    Imputation MAPE: 0.0939775
    Imputation RMSE: 23.2553

    Running time: 0 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Birmingham-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_matrix = scipy.io.loadmat('../datasets/Birmingham-data-set/random_matrix.mat')
random_matrix = random_matrix['random_matrix']

missing_rate = 0.3

# =============================================================================
### Non-random missing (NM) scenario
### Set the RM scenario by:
binary_tensor = np.zeros(dense_tensor.shape)
for i1 in range(dense_tensor.shape[0]):
    for i2 in range(dense_tensor.shape[1]):
        binary_tensor[i1, i2, :] = np.round(random_matrix[i1, i2] + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.05
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 52.6786

    Imputation MAPE: 0.1331
    Imputation RMSE: 52.3164

    Running time: 0 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Birmingham-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_matrix = scipy.io.loadmat('../datasets/Birmingham-data-set/random_matrix.mat')
random_matrix = random_matrix['random_matrix']

missing_rate = 0.5

# =============================================================================
### Non-random missing (NM) scenario
### Set the RM scenario by:
binary_tensor = np.zeros(dense_tensor.shape)
for i1 in range(dense_tensor.shape[0]):
    for i2 in range(dense_tensor.shape[1]):
        binary_tensor[i1, i2, :] = np.round(random_matrix[i1, i2] + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.05
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 85.8883

    Imputation MAPE: 0.153229
    Imputation RMSE: 86.9536

    Running time: 0 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Birmingham-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_matrix = scipy.io.loadmat('../datasets/Birmingham-data-set/random_matrix.mat')
random_matrix = random_matrix['random_matrix']

missing_rate = 0.6

# =============================================================================
### Non-random missing (NM) scenario
### Set the RM scenario by:
binary_tensor = np.zeros(dense_tensor.shape)
for i1 in range(dense_tensor.shape[0]):
    for i2 in range(dense_tensor.shape[1]):
        binary_tensor[i1, i2, :] = np.round(random_matrix[i1, i2] + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.05
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 96.0289

    Imputation MAPE: 0.180065
    Imputation RMSE: 96.1844

    Running time: 0 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Birmingham-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_matrix = scipy.io.loadmat('../datasets/Birmingham-data-set/random_matrix.mat')
random_matrix = random_matrix['random_matrix']

missing_rate = 0.7

# =============================================================================
### Non-random missing (NM) scenario
### Set the RM scenario by:
binary_tensor = np.zeros(dense_tensor.shape)
for i1 in range(dense_tensor.shape[0]):
    for i2 in range(dense_tensor.shape[1]):
        binary_tensor[i1, i2, :] = np.round(random_matrix[i1, i2] + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.05
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 147.425

    Iter: 100
    RMSE: 145.304

    Imputation MAPE: 0.209472
    Imputation RMSE: 145.293

    Running time: 0 seconds

**Experiment results** of missing data imputation using LRTC-TNN:

|  scenario |`alpha` (vector input)|`rho` | `theta` |`maxiter`|       mape |      rmse |
|:----------|-----:|---------:|---------:|-------- --:|-------- --:|----------:|
|**0.1, RM**| $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$ | 0.00001 | 0.15 | 200 | **0.0421** | **13.11**|
|**0.3, RM**| $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$ | 0.00001 | 0.15 | 200 | **0.0515** | **17.47**|
|**0.1, NM**| $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$ | 0.00001 | 0.05 | 200 | **0.0940** | **23.26**|
|**0.3, NM**| $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$ | 0.00001 | 0.05 | 200 | **0.1331** | **52.32**|

### Experiments on Hangzhou Data Set

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Hangzhou-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_tensor = scipy.io.loadmat('../datasets/Hangzhou-data-set/random_tensor.mat')
random_tensor = random_tensor['random_tensor']

missing_rate = 0.2

# =============================================================================
### Random missing (RM) scenario
### Set the RM scenario by:
binary_tensor = np.round(random_tensor + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.10
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 24.7402

    Iter: 100
    RMSE: 24.7786

    Imputation MAPE: 0.180318
    Imputation RMSE: 24.8976

    Running time: 2 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Hangzhou-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_tensor = scipy.io.loadmat('../datasets/Hangzhou-data-set/random_tensor.mat')
random_tensor = random_tensor['random_tensor']

missing_rate = 0.4

# =============================================================================
### Random missing (RM) scenario
### Set the RM scenario by:
binary_tensor = np.round(random_tensor + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.10
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 25.7935

    Iter: 100
    RMSE: 25.7774

    Imputation MAPE: 0.188003
    Imputation RMSE: 25.8979

    Running time: 3 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Hangzhou-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_tensor = scipy.io.loadmat('../datasets/Hangzhou-data-set/random_tensor.mat')
random_tensor = random_tensor['random_tensor']

missing_rate = 0.5

# =============================================================================
### Random missing (RM) scenario
### Set the RM scenario by:
binary_tensor = np.round(random_tensor + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.10
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 26.4952

    Iter: 100
    RMSE: 26.7525

    Imputation MAPE: 0.1926
    Imputation RMSE: 26.8564

    Running time: 2 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Hangzhou-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_tensor = scipy.io.loadmat('../datasets/Hangzhou-data-set/random_tensor.mat')
random_tensor = random_tensor['random_tensor']

missing_rate = 0.6

# =============================================================================
### Random missing (RM) scenario
### Set the RM scenario by:
binary_tensor = np.round(random_tensor + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.10
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 27.6098

    Iter: 100
    RMSE: 27.7448

    Imputation MAPE: 0.195575
    Imputation RMSE: 27.8357

    Running time: 2 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Hangzhou-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_tensor = scipy.io.loadmat('../datasets/Hangzhou-data-set/random_tensor.mat')
random_tensor = random_tensor['random_tensor']

missing_rate = 0.7

# =============================================================================
### Random missing (RM) scenario
### Set the RM scenario by:
binary_tensor = np.round(random_tensor + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.10
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 29.4194

    Iter: 100
    RMSE: 29.8024

    Imputation MAPE: 0.203381
    Imputation RMSE: 29.8976

    Running time: 2 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Hangzhou-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_matrix = scipy.io.loadmat('../datasets/Hangzhou-data-set/random_matrix.mat')
random_matrix = random_matrix['random_matrix']

missing_rate = 0.2

# =============================================================================
### Non-random missing (NM) scenario
### Set the NM scenario by:
binary_tensor = np.zeros(dense_tensor.shape)
for i1 in range(dense_tensor.shape[0]):
    for i2 in range(dense_tensor.shape[1]):
        binary_tensor[i1, i2, :] = np.round(random_matrix[i1, i2] + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.10
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 29.5311

    Iter: 100
    RMSE: 27.4519

    Imputation MAPE: 0.197053
    Imputation RMSE: 27.4161

    Running time: 2 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Hangzhou-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_matrix = scipy.io.loadmat('../datasets/Hangzhou-data-set/random_matrix.mat')
random_matrix = random_matrix['random_matrix']

missing_rate = 0.4

# =============================================================================
### Non-random missing (NM) scenario
### Set the NM scenario by:
binary_tensor = np.zeros(dense_tensor.shape)
for i1 in range(dense_tensor.shape[0]):
    for i2 in range(dense_tensor.shape[1]):
        binary_tensor[i1, i2, :] = np.round(random_matrix[i1, i2] + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.10
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 29.7898

    Iter: 100
    RMSE: 28.9947

    Imputation MAPE: 0.204345
    Imputation RMSE: 29.03

    Running time: 2 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Hangzhou-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_matrix = scipy.io.loadmat('../datasets/Hangzhou-data-set/random_matrix.mat')
random_matrix = random_matrix['random_matrix']

missing_rate = 0.5

# =============================================================================
### Non-random missing (NM) scenario
### Set the NM scenario by:
binary_tensor = np.zeros(dense_tensor.shape)
for i1 in range(dense_tensor.shape[0]):
    for i2 in range(dense_tensor.shape[1]):
        binary_tensor[i1, i2, :] = np.round(random_matrix[i1, i2] + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.10
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 31.9214

    Iter: 100
    RMSE: 30.7007

    Imputation MAPE: 0.212233
    Imputation RMSE: 30.6813

    Running time: 2 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Hangzhou-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_matrix = scipy.io.loadmat('../datasets/Hangzhou-data-set/random_matrix.mat')
random_matrix = random_matrix['random_matrix']

missing_rate = 0.6

# =============================================================================
### Non-random missing (NM) scenario
### Set the NM scenario by:
binary_tensor = np.zeros(dense_tensor.shape)
for i1 in range(dense_tensor.shape[0]):
    for i2 in range(dense_tensor.shape[1]):
        binary_tensor[i1, i2, :] = np.round(random_matrix[i1, i2] + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.10
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 38.5045

    Iter: 100
    RMSE: 37.6957

    Imputation MAPE: 0.212195
    Imputation RMSE: 37.6735

    Running time: 2 seconds

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Hangzhou-data-set/tensor.mat')
dense_tensor = tensor['tensor']
random_matrix = scipy.io.loadmat('../datasets/Hangzhou-data-set/random_matrix.mat')
random_matrix = random_matrix['random_matrix']

missing_rate = 0.7

# =============================================================================
### Non-random missing (NM) scenario
### Set the NM scenario by:
binary_tensor = np.zeros(dense_tensor.shape)
for i1 in range(dense_tensor.shape[0]):
    for i2 in range(dense_tensor.shape[1]):
        binary_tensor[i1, i2, :] = np.round(random_matrix[i1, i2] + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.10
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 40.522

    Iter: 100
    RMSE: 39.6418

    Imputation MAPE: 0.212856
    Imputation RMSE: 39.6974

    Running time: 2 seconds

**Experiment results** of missing data imputation using LRTC-TNN:

|  scenario |`alpha` (vector input)|`rho`|`theta`|`maxiter`|       mape |      rmse |
|:----------|-----:|---------:|---------:|--------:|----------:|----------:|
|**0.2, RM**| $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$ | 0.00001 | 0.10 | 200 | **0.1803** | **24.90**|
|**0.4, RM**| $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$ | 0.00001 | 0.10 | 200 | **0.1880** | **25.90**|
|**0.2, NM**| $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$ | 0.00001 | 0.10 | 200 | **0.1971** | **27.42**|
|**0.4, NM**| $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$ | 0.00001 | 0.10 | 200 | **0.2043** | **29.03**|

### Experiments on Seattle Data Set

```python
import pandas as pd

dense_mat = pd.read_csv('../datasets/Seattle-data-set/mat.csv', index_col = 0)
RM_mat = pd.read_csv('../datasets/Seattle-data-set/RM_mat.csv', index_col = 0)
dense_mat = dense_mat.values
RM_mat = RM_mat.values
dense_tensor = dense_mat.reshape([dense_mat.shape[0], 28, 288])
RM_tensor = RM_mat.reshape([RM_mat.shape[0], 28, 288])

missing_rate = 0.2

# =============================================================================
### Random missing (RM) scenario
### Set the RM scenario by:
binary_tensor = np.round(RM_tensor + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.30
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 5.91611

    Iter: 100
    RMSE: 3.01208

    Iter: 150
    RMSE: 3.04636

    Imputation MAPE: 0.0464732
    Imputation RMSE: 3.05844

    Running time: 65 seconds

```python
import pandas as pd

dense_mat = pd.read_csv('../datasets/Seattle-data-set/mat.csv', index_col = 0)
RM_mat = pd.read_csv('../datasets/Seattle-data-set/RM_mat.csv', index_col = 0)
dense_mat = dense_mat.values
RM_mat = RM_mat.values
dense_tensor = dense_mat.reshape([dense_mat.shape[0], 28, 288])
RM_tensor = RM_mat.reshape([RM_mat.shape[0], 28, 288])

missing_rate = 0.4

# =============================================================================
### Random missing (RM) scenario
### Set the RM scenario by:
binary_tensor = np.round(RM_tensor + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.30
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 5.86054

    Iter: 100
    RMSE: 3.22163

    Iter: 150
    RMSE: 3.28947

    Imputation MAPE: 0.0511641
    Imputation RMSE: 3.30216

    Running time: 63 seconds

```python
import pandas as pd

dense_mat = pd.read_csv('../datasets/Seattle-data-set/mat.csv', index_col = 0)
RM_mat = pd.read_csv('../datasets/Seattle-data-set/RM_mat.csv', index_col = 0)
dense_mat = dense_mat.values
RM_mat = RM_mat.values
dense_tensor = dense_mat.reshape([dense_mat.shape[0], 28, 288])
RM_tensor = RM_mat.reshape([RM_mat.shape[0], 28, 288])

missing_rate = 0.5

# =============================================================================
### Random missing (RM) scenario
### Set the RM scenario by:
binary_tensor = np.round(RM_tensor + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.30
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 5.85456

    Iter: 100
    RMSE: 3.35752

    Iter: 150
    RMSE: 3.45611

    Imputation MAPE: 0.0543108
    Imputation RMSE: 3.46931

    Running time: 66 seconds

```python
import pandas as pd

dense_mat = pd.read_csv('../datasets/Seattle-data-set/mat.csv', index_col = 0)
RM_mat = pd.read_csv('../datasets/Seattle-data-set/RM_mat.csv', index_col = 0)
dense_mat = dense_mat.values
RM_mat = RM_mat.values
dense_tensor = dense_mat.reshape([dense_mat.shape[0], 28, 288])
RM_tensor = RM_mat.reshape([RM_mat.shape[0], 28, 288])

missing_rate = 0.6

# =============================================================================
### Random missing (RM) scenario
### Set the RM scenario by:
binary_tensor = np.round(RM_tensor + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.30
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 5.85415

    Iter: 100
    RMSE: 3.51862

    Iter: 150
    RMSE: 3.65423

    Imputation MAPE: 0.0580412
    Imputation RMSE: 3.66197

    Running time: 57 seconds

```python
import pandas as pd

dense_mat = pd.read_csv('../datasets/Seattle-data-set/mat.csv', index_col = 0)
RM_mat = pd.read_csv('../datasets/Seattle-data-set/RM_mat.csv', index_col = 0)
dense_mat = dense_mat.values
RM_mat = RM_mat.values
dense_tensor = dense_mat.reshape([dense_mat.shape[0], 28, 288])
RM_tensor = RM_mat.reshape([RM_mat.shape[0], 28, 288])

missing_rate = 0.7

# =============================================================================
### Random missing (RM) scenario
### Set the RM scenario by:
binary_tensor = np.round(RM_tensor + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.30
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 5.95502

    Iter: 100
    RMSE: 3.80994

    Iter: 150
    RMSE: 4.02848

    Imputation MAPE: 0.0653333
    Imputation RMSE: 4.04162

    Running time: 60 seconds

```python
import pandas as pd

dense_mat = pd.read_csv('../datasets/Seattle-data-set/mat.csv', index_col = 0)
NM_mat = pd.read_csv('../datasets/Seattle-data-set/NM_mat.csv', index_col = 0)
dense_mat = dense_mat.values
NM_mat = NM_mat.values
dense_tensor = dense_mat.reshape([dense_mat.shape[0], 28, 288])

missing_rate = 0.2

# =============================================================================
### Non-random missing (NM) scenario
### Set the NM scenario by:
binary_tensor = np.zeros((dense_mat.shape[0], 28, 288))
for i1 in range(binary_tensor.shape[0]):
    for i2 in range(binary_tensor.shape[1]):
        binary_tensor[i1, i2, :] = np.round(NM_mat[i1, i2] + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.05
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 4.89165

    Iter: 100
    RMSE: 4.17949

    Iter: 150
    RMSE: 4.18524

    Imputation MAPE: 0.0692741
    Imputation RMSE: 4.18538

    Running time: 61 seconds

```python
import pandas as pd

dense_mat = pd.read_csv('../datasets/Seattle-data-set/mat.csv', index_col = 0)
NM_mat = pd.read_csv('../datasets/Seattle-data-set/NM_mat.csv', index_col = 0)
dense_mat = dense_mat.values
NM_mat = NM_mat.values
dense_tensor = dense_mat.reshape([dense_mat.shape[0], 28, 288])

missing_rate = 0.4

# =============================================================================
### Non-random missing (NM) scenario
### Set the NM scenario by:
binary_tensor = np.zeros((dense_mat.shape[0], 28, 288))
for i1 in range(binary_tensor.shape[0]):
    for i2 in range(binary_tensor.shape[1]):
        binary_tensor[i1, i2, :] = np.round(NM_mat[i1, i2] + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.05
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 5.24117

    Iter: 100
    RMSE: 4.4993

    Iter: 150
    RMSE: 4.50354

    Imputation MAPE: 0.0759062
    Imputation RMSE: 4.50354

    Running time: 64 seconds

```python
import pandas as pd

dense_mat = pd.read_csv('../datasets/Seattle-data-set/mat.csv', index_col = 0)
NM_mat = pd.read_csv('../datasets/Seattle-data-set/NM_mat.csv', index_col = 0)
dense_mat = dense_mat.values
NM_mat = NM_mat.values
dense_tensor = dense_mat.reshape([dense_mat.shape[0], 28, 288])

missing_rate = 0.5

# =============================================================================
### Non-random missing (NM) scenario
### Set the NM scenario by:
binary_tensor = np.zeros((dense_mat.shape[0], 28, 288))
for i1 in range(binary_tensor.shape[0]):
    for i2 in range(binary_tensor.shape[1]):
        binary_tensor[i1, i2, :] = np.round(NM_mat[i1, i2] + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.05
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 5.46856

    Iter: 100
    RMSE: 4.72734

    Iter: 150
    RMSE: 4.73129

    Imputation MAPE: 0.0811558
    Imputation RMSE: 4.7314

    Running time: 59 seconds

```python
import pandas as pd

dense_mat = pd.read_csv('../datasets/Seattle-data-set/mat.csv', index_col = 0)
NM_mat = pd.read_csv('../datasets/Seattle-data-set/NM_mat.csv', index_col = 0)
dense_mat = dense_mat.values
NM_mat = NM_mat.values
dense_tensor = dense_mat.reshape([dense_mat.shape[0], 28, 288])

missing_rate = 0.6

# =============================================================================
### Non-random missing (NM) scenario
### Set the NM scenario by:
binary_tensor = np.zeros((dense_mat.shape[0], 28, 288))
for i1 in range(binary_tensor.shape[0]):
    for i2 in range(binary_tensor.shape[1]):
        binary_tensor[i1, i2, :] = np.round(NM_mat[i1, i2] + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.05
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 5.75773

    Iter: 100
    RMSE: 4.99998

    Imputation MAPE: 0.0861234
    Imputation RMSE: 5.00438

    Running time: 59 seconds

```python
import pandas as pd

dense_mat = pd.read_csv('../datasets/Seattle-data-set/mat.csv', index_col = 0)
NM_mat = pd.read_csv('../datasets/Seattle-data-set/NM_mat.csv', index_col = 0)
dense_mat = dense_mat.values
NM_mat = NM_mat.values
dense_tensor = dense_mat.reshape([dense_mat.shape[0], 28, 288])

missing_rate = 0.7

# =============================================================================
### Non-random missing (NM) scenario
### Set the NM scenario by:
binary_tensor = np.zeros((dense_mat.shape[0], 28, 288))
for i1 in range(binary_tensor.shape[0]):
    for i2 in range(binary_tensor.shape[1]):
        binary_tensor[i1, i2, :] = np.round(NM_mat[i1, i2] + 0.5 - missing_rate)
# =============================================================================

sparse_tensor = np.multiply(dense_tensor, binary_tensor)
```

```python
import time
start = time.time()
alpha = np.ones(3) / 3
rho = 1e-5
theta = 0.05
epsilon = 1e-4
maxiter = 200
LRTC(dense_tensor, sparse_tensor, alpha, rho, theta, epsilon, maxiter)
end = time.time()
print('Running time: %d seconds'%(end - start))
```

    Iter: 50
    RMSE: 6.09623

    Iter: 100
    RMSE: 5.44157

    Iter: 150
    RMSE: 5.45282

    Imputation MAPE: 0.0940823
    Imputation RMSE: 5.45341

    Running time: 76 seconds

**Experiment results** of missing data imputation using LRTC-TNN:

|  scenario |`alpha` (vector input)|`rho`|`theta`|`maxiter`|       mape |      rmse |
|:----------|-----:|---------:|---------:|---------:|-------- --:|----------:|
|**0.2, RM**| $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$ | 0.00001 | 0.30 | 200 | **0.0465** | **3.06**|
|**0.4, RM**| $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$ | 0.00001 | 0.30 | 200 | **0.0512** | **3.30**|
|**0.2, NM**| $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$ | 0.00001 | 0.05 | 200 | **0.0693** | **4.19**|
|**0.4, NM**| $\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right)$ | 0.00001 | 0.05 | 200 | **0.0759** | **4.50**|


# The Harmonic Oscillator Strikes Back

*Note:* Much of this is adapted/copied from https://flothesof.github.io/harmonic-oscillator-three-methods-solution.html

This week we continue our adventures with the harmonic oscillator.

The harmonic oscillator is a system that, when displaced from its equilibrium position, experiences a restoring force F proportional to the displacement x:

$$F=-kx$$

The potential energy of this system is

$$V = {1 \over 2}k{x^2}$$

These are sometime rewritten as

$$ F=- \omega_0^2 m x, \text{ }  V(x) = {1 \over 2} m \omega_0^2 {x^2}$$

Where $\omega_0  = \sqrt {{k \over m}} $

If the equilibrium value of the harmonic oscillator is not zero, then

$$ F=- \omega_0^2 m (x-x_{eq}), \text{ }  V(x) = {1 \over 2} m \omega_0^2 (x-x_{eq})^2$$

## 1. Harmonic oscillator from last time (with some better defined conditions)

Applying the harmonic oscillator force to Newton's second law leads to the following second order differential equation

$$ F = m a $$

$$ F= -m \omega_0^2 (x-x_{eq}) $$

$$ a = - \omega_0^2 (x-x_{eq}) $$

$$ x(t)'' = - \omega_0^2 (x-x_{eq}) $$

The final expression can be rearranged into a second order homogenous differential equation, and can be solved using the methods we used above

This is already solved to remind you how we found these values

```python
import sympy as sym
sym.init_printing()
```

**Note** that this time we define some of the properties of the symbols. Namely, that the frequency is always positive and real and that the positions are always real

```python
omega0,t=sym.symbols("omega_0,t",positive=True,nonnegative=True,real=True)
xeq=sym.symbols("x_{eq}",real=True)
x=sym.Function("x",real=True)
x(t),omega0
```

```python
dfeq=sym.Derivative(x(t),t,2)+omega0**2*(x(t)-xeq)
dfeq
```

```python
sol = sym.dsolve(dfeq)
sol
```

```python
sol,sol.args[0],sol.args[1]
```

**Note** this time we define the initial positions and velocities as real

```python
x0,v0=sym.symbols("x_0,v_0",real=True)
ics=[sym.Eq(sol.args[1].subs(t, 0), x0),
     sym.Eq(sol.args[1].diff(t).subs(t, 0), v0)]
ics
```

```python
solved_ics=sym.solve(ics)
solved_ics
```

### 1.1 Equation of motion for $x(t)$

```python
full_sol = sol.subs(solved_ics[0])
full_sol
```

### 1.2 Equation of motion for $p(t)$

```python
m=sym.symbols("m",positive=True,nonnegative=True,real=True)
p=sym.Function("p")
p_sol=sym.Eq(p(t),m*sol.args[1].subs(solved_ics[0]).diff(t))
p_sol
```

## 2. Time average values for a harmonic oscillator

If we want to understand the average value of a time dependent observable, we need to solve the following integral

$${\left\langle {A(t)} \right\rangle}_t  = \begin{array}{*{20}{c}}
{\lim }\\
{\tau  \to 0}
\end{array}\frac{1}{\tau }\int\limits_0^\tau  {A(t)dt} $$

### 2.1 Average position ${\left\langle {x} \right\rangle}_t$ for a harmonic oscillator

```python
tau=sym.symbols("tau",nonnegative=True,real=True)
xfunc=full_sol.args[1]
xavet=(xfunc.integrate((t,0,tau))/tau).limit(tau,sym.oo)
xavet
```

The computer does not always make the best choices the first time. If you treat each sum individually this is not a hard limit to do by hand. The computer is not smart. We can help it by inseting an `expand()` function in the statement

```python
xavet=(xfunc.integrate((t,0,tau))/tau).expand().limit(tau,sym.oo)
xavet
```

### 2.2 Excercise: Calculate the average momenta  ${\left\langle {p} \right\rangle}_t$ for a harmonic oscillator

```python
pfunc=p_sol.args[1]
pavet=(pfunc.integrate((t,0,tau))/tau).expand().limit(tau,sym.oo)
pavet
```

### 2.3 Exercise: Calculate the average kinetic energy of a harmonic oscillator

```python
"<KE>=<p**2/(2*m)>"
```

    '<KE>=<p**2/(2*m)>'

```python
kefunc=(pfunc**2)/(2*m)
kefunc
```

```python
keavet=(kefunc.integrate((t,0,tau))/tau).expand().limit(tau,sym.oo)
keavet
```

## 3.  Ensemble (Thermodynamic) Average values for a harmonic oscillator

If we want to understand the thermodynamics ensemble average value of an observable, we need to solve the following integral.

$${\left\langle {A(t)} \right\rangle}_{T}  = \frac{\int{A e^{-\beta H}dqdp}}{\int{e^{-\beta H}dqdp} } $$

You can think of this as a Temperature average instead of a time average.

Here $\beta=\frac{1}{k_B T}$ and the classical Hamiltonian, $H$ is

$$ H = \frac{p^2}{2 m} + V(q)$$

**Note** that the factors of $1/h$ found in the classical partition function cancel out when calculating average values

### 3.1 Average position ${\left\langle {x} \right\rangle}_t$ for a harmonic oscillator

For a harmonic oscillator with equilibrium value $x_{eq}$, the Hamiltonian is
$$ H = \frac{p^2}{2 m} + \frac{1}{2} m \omega_0 (x-x_{eq})^2 $$

First we will calculate the partition function $\int{e^{-\beta H}dqdp}$

```python
k,T=sym.symbols("k,T",positive=True,nonnegative=True,real=True)
xT,pT=sym.symbols("x_T,p_T",real=True)
ham=sym.Rational(1,2)*(pT)**2/m + sym.Rational(1,2)*m*omega0**2*(xT-xeq)**2
beta=1/(k*T)
bolz=sym.exp(-beta*ham)
z=sym.integrate(bolz,(xT,-sym.oo,sym.oo),(pT,-sym.oo,sym.oo))
z
```

Then we can calculate the numerator $\int{A e^{-\beta H}dqdp}$

```python
numx=sym.integrate(xT*bolz,(xT,-sym.oo,sym.oo),(pT,-sym.oo,sym.oo))
numx
```

And now the average value

```python
xaveT=numx/z
xaveT
```

### 3.2 Exercise: Calculate the average momenta  ${\left\langle {p} \right\rangle}_t$ for a harmonic oscillator

After calculating the value, explain why you think you got this number

```python
Tp=sym.integrate(pT*bolz,(xT,-sym.oo,sym.oo),(pT,-sym.oo,sym.oo))
Tp
```

```python
TpaveT=Tp/z
TpaveT
```

### 3.3 Exercise: Calculate the average kinetic energy

The answer you get here is a well known result related to the energy equipartition theorem

```python
Tke=sym.integrate((pT**2)/(2*m)*bolz,(xT,-sym.oo,sym.oo),(pT,-sym.oo,sym.oo))
Tke
```

```python
TkeaveT=Tke/z
TkeaveT
```

# Back to the lecture

## 4. Exercise Verlet integrators

In this exercise we will write a routine to solve for the equations of motion for a hamonic oscillator.

Plot the positions and momenta (seprate plots) of the harmonic oscillator as a functions of time.

Calculaate trajectories using the following methods:
1. Exact solution
2. Simple taylor series expansion
3. Predictor-corrector method
4. Verlet algorithm
5. Leapfrog algorithm
6. Velocity Verlet algorithm

```python
%matplotlib inline

omega0,t=sym.symbols("omega_0,t",positive=True,nonnegative=True,real=True)
xeq=sym.symbols("x_{eq}",real=True)
x=sym.Function("x",real=True)

dfreq=sym.Derivative(x(t),t,2)+omega0**2*(x(t)-xeq)
soln=sym.dsolve(dfreq)

x0,v0=sym.symbols("x_0,v_0",real=True)
inicon=[sym.Eq(soln.args[1].subs(t, 0), x0),
        sym.Eq(soln.args[1].diff(t).subs(t, 0), v0)]
iniconsoln=sym.solve(inicon)

pos=soln.subs(iniconsoln[0])
pos

m=sym.symbols("m",positive=True,nonnegative=True,real=True)
v=sym.Function("v")
vel=sym.Eq(v(t),soln.args[1].subs(iniconsoln[0]).diff(t))

posplot=pos.subs({x0:5,xeq:0,v0:5,omega0:1})
velplot=vel.subs({x0:5,xeq:0,v0:5,omega0:1})
sym.plot(posplot.rhs,(t,0,10)),sym.plot(velplot.rhs,(t,0,10))

pos,vel
```

```python
xt0=0
t=0.5
vt0=1
pos=[]
for i in range(0,100):
    vt=vt0+1/2*t**2*-xt0
    vt0=vt
    xt=xt0+vt0*t
    xt0=xt
    pos.append(xt)

pt0=0
t=0.5
at0=1
vel=[]
for i in range(0,100):
    at=at0+1/2*t**2-vt0
    at0=at
    vt=vt0+at0*t
    vt0=vt
    vel.append(vt)

import matplotlib.pyplot as plt
fig, (ax1, ax2) = plt.subplots(1, 2)
fig.suptitle('position and velocity')
ax1.plot(pos)
ax2.plot(vel)
```

```python
xt1=1
xt2=0
t=0.5
pos=[]

for i in range(0,100):
    xt0=2*xt1-xt2+t**2*-xt1
    xt2=xt1
    xt1=xt0
    pos.append(xt2)

xt0=0
xt2=1
t=2
vel=[]
for i in range(0,100):
    vt=(xt2-xt0)/2*t
    xt0=xt2
    xt2=vt
    vel.append(vt)

import matplotlib.pyplot as plt
fig, (ax1, ax2) = plt.subplots(1, 2)
fig.suptitle('position and velocity')
ax1.plot(pos)
ax2.plot(vel)
```

```python
xt0=1
vtminushalfdt=0
t=0.2
pos=[]
for i in range(0,100):
    vtplushalfdt=vtminushalfdt+t*-xt0
    vtminushalfdt=vtplushalfdt
    xt1=xt0+t*vtplushalfdt
    xt0=xt1
    pos.append(xt1)

xt0=1
t=0.3
vtminushalfdt=1
vtplushalfdt=2
vel=[]
for i in range(0,100):
    vt=1/2*(vtplushalfdt+vtminushalfdt)
    vtminushalfdt=vt
    vtplushalfdt=vtminushalfdt+t*-xt0
    vtminushalfdt=vtplushalfdt
    xt1=xt0+t*vtplushalfdt
    xt0=xt1
    vel.append(vt)

import matplotlib.pyplot as plt
fig, (ax1, ax2) = plt.subplots(1, 2)
fig.suptitle('position and velocity')
ax1.plot(pos)
ax2.plot(vel)
```

```python
dt=0.01
x0=0
v0=1
pos=[]
for i in range(0,1000):
    x1=x0+v0*dt+1/2*-x0*dt**2
    x0=x1
    v1=v0+1/2*(-x1-x0)*dt
    v0=v1
    pos.append(x1)

xt0=1
dt=2
vt0=1
vel=[]
for i in range(0,10):
    vtplushalfdt=vt0+1/2*dt*-xt0
    xt0=vtplushalfdt
    vel.append(vtplushalfdt)

import matplotlib.pyplot as plt
fig, (ax1, ax2) = plt.subplots(1, 2)
fig.suptitle('position and velocity')
ax1.plot(pos)
ax2.plot(vel)
```

```python

```


We now extend SEIR model to:

\begin{align}
\dot{S_{i}}	&=-\lambda_{i}(t)S_{i}+\sigma_{i}, \\
\dot{E}_{i}	&=\lambda_{i}(t)S_{i}-\gamma_{E}E_{i},\\
\dot{I}_{i}^{a}	&=\alpha\gamma_{E}E_{i}-\gamma_{I^{a}}I_{i}^{a},\\
\dot{I}_{i}^{s}	&=\bar{\alpha}\gamma_{E}E_{i}-\gamma_{I^{s}}I_{i}^{s},\\
\dot{I}_{i}^{h}	&=h_{i}\gamma_{I^{s}}I_{i}^{s}-\gamma_{I^{h}}I_{i}^{h},\\
\dot{I}_{i}^{c}	&=c_{i}\gamma_{I^{h}}I_{i}^{h}-\gamma_{I^{c}}I_{i}^{c},\\
\dot{I}_{i}^{m}	&=m_{i}\gamma_{I^{c}}I_{i}^{c},\\
\dot{R}_{i} &=\gamma_{I^{a}}I_{i}^{a}+\bar{h}_{i}\gamma_{I^{s}}I_{i}^{s}+\bar{c}_{i}\gamma_{I^{h}}I_{i}^{h}+\bar{m}_{i}\gamma_{I^{c}}I_{i}^{c},\\
\dot{N}_{i}	&=\sigma_{i}-I_{i}^{m}.
\end{align}

$\lambda_{i}(t)=\beta\sum_{j=1}^{M}\left(C_{ij}^{a}\frac{I_{j}^{a}}{N_{j}}+C_{ij}^{s}\frac{I_{j}^{s}}{N_{j}}+C_{ij}^{h}\frac{I_{j}^{h}}{N_{j}}\right)$.

Here
* $I^{a}$ : asymptomatic infectives
* $I^{s}$ : symptomatic infectives
* $I^{h}$ : hospitalized infectives
* $I^{c}$ : ICU cases
* $I^{m}$ : mortality
* ${h}_{i}=1-\bar h_{i}$ is the fraction of syptomatics who are hospitalized
* $c_{i}=1-\bar{c}_{i}$, is the fraction of hospitalizeds, who are in ICU
* $m_{i}=1-\bar{m}_{i}$, is the fraction of ICU which leads to mortality
* $C_{ij}^{s}=f^{s}C_{ij}^{a}\equiv f^{s}C_{ij}$
* $C_{ij}^{h}=f^{h}C_{ij}^{a}\equiv f^{h}C_{ij}$.

S ---> E ---> Ia, Is

Ia ---> R

Is ---> Ih, R

Ih ---> Ic, R

Ic ---> Im, R

```python
%%capture
## compile PyRoss for this notebook
import os
owd = os.getcwd()
os.chdir('../../')
%run setup.py install
os.chdir(owd)
```

```python
%matplotlib inline
import numpy as np
import pyross
import pandas as pd
import matplotlib.pyplot as plt
```

```python
M=16  # number of age groups

# load age structure data
my_data = np.genfromtxt('../data/age_structures/UK.csv', delimiter=',', skip_header=1)
aM, aF = my_data[:, 1], my_data[:, 2]

# set age groups
Ni=aM+aF;   Ni=Ni[0:M];  N=np.sum(Ni)
```

```python
# contact matrices
CH, CW, CS, CO = pyross.contactMatrix.UK()

## matrix of total contacts
C=CH+CW+CS+CO

fig,aCF =  plt.subplots(2,2);
aCF[0][0].pcolor(CH, cmap=plt.cm.get_cmap('GnBu', 10));
aCF[0][1].pcolor(CW, cmap=plt.cm.get_cmap('GnBu', 10));
aCF[1][0].pcolor(CS, cmap=plt.cm.get_cmap('GnBu', 10));
aCF[1][1].pcolor(CO, cmap=plt.cm.get_cmap('GnBu', 10));
```

## Fraction of asymptomatic infectives is constant

```python
beta  = 0.036692            # infection rate

gE    = 1/5
gIa   = 1./7                # recovery rate of asymptomatic infectives
gIs   = 1./7                # recovery rate of symptomatic infectives
alpha = 0.3                 # fraction of asymptomatic infectives
fsa   = 0.2                 # the self-isolation parameter
fh    = 0
gIh   = 1/14
gIc   = 1/14

sa    = 100*np.ones(M)      #  rate of additional/removal of population by birth etc
sa[0]     = 1500            # birth
sa[12:16] = -300            # mortality

hh   = 0.1*np.ones(M)       # fraction which goes from Is to hospital
cc   = 0.05*np.ones(M)      # fraction which goes from  hospital to ICU
mm   = 0.4*np.ones(M)       # mortality from IC

# initial conditions
Is_0 = np.zeros((M));  #Is_0[6:13]=8;  Is_0[2:6]=4; Is_0[13:16]=4

Ia_0 = 1000*np.ones((M));
R_0  = np.zeros((M))
E_0  = np.zeros((M))
Ih_0  = np.zeros((M))
Ic_0  = np.zeros((M))
Im_0  = np.zeros((M))

S_0  = Ni - (Ia_0 + Is_0 + R_0)

# matrix for linearised dynamics
L0 = np.zeros((M, M))
L  = np.zeros((2*M, 2*M))

for i in range(M):
    for j in range(M):
        L0[i,j]=C[i,j]*Ni[i]/Ni[j]

L[0:M, 0:M]     =    alpha*beta/gIs*L0
L[0:M, M:2*M]   = fsa*alpha*beta/gIs*L0
L[M:2*M, 0:M]   =    ((1-alpha)*beta/gIs)*L0
L[M:2*M, M:2*M] = fsa*((1-alpha)*beta/gIs)*L0

r0 = np.max(np.linalg.eigvals(L))
print("The basic reproductive ratio for these parameters is", r0)
```

    The basic reproductive ratio for these parameters is (1.319908017634234+0j)

```python
# duration of simulation and data file
Tf=480;  Nf=2000;

# intantiate model
parameters = {'alpha':alpha,'beta':beta, 'gIa':gIa,'gIs':gIs,
              'gIh':gIh,'gIc':gIc,  'gE':gE,
              'fsa':fsa, 'fh':fh,
              'sa':sa, 'hh':hh, 'cc':cc, 'mm':mm}
model = pyross.deterministic.SEI5R(parameters, M, Ni)

# the contact structure is independent of time
def contactMatrix(t):
    return C
# run model
data=model.simulate(S_0, E_0, Ia_0, Is_0, Ih_0, Ic_0, Im_0, contactMatrix, Tf, Nf)

t = data['t']; IC  = np.zeros((Nf))
for i in range(1*M):
        IC += data['X'][:,2*M+i]

fig = plt.figure(num=None, figsize=(10, 8), dpi=80, facecolor='w', edgecolor='k')
plt.rcParams.update({'font.size': 22})

plt.plot(t, IC, '-', lw=4, color='#A60628', label='forecast', alpha=0.8)

plt.legend(fontsize=26, loc='upper left'); plt.grid()
plt.autoscale(enable=True, axis='x', tight=True)
plt.ylabel('Asymtomatic Infected individuals'); #plt.xlim(0, 40); plt.ylim(0, 4);
#plt.savefig('/Users/rsingh/Desktop/2a.png', format='png', dpi=212)
IC[0]
```

## Fraction of asymptomatic infectives is age-dependent

```python
alpha = 0.5*np.ones(M)      #  rate of additional/removal of population by birth etc
alpha[6:M] = 0.1              # everyone is symtomatic in large age-group

Ia_0 = 1000*np.ones((M));
```

```python

# duration of simulation and data file
Tf=480;  Nf=2000;

# intantiate model
parameters = {'alpha':alpha,'beta':beta, 'gIa':gIa,'gIs':gIs,
              'gIh':gIh,'gIc':gIc,  'gE':gE,
              'fsa':fsa, 'fh':fh,
              'sa':sa, 'hh':hh, 'cc':cc, 'mm':mm}
model = pyross.deterministic.SEI5R(parameters, M, Ni)

# the contact structure is independent of time
def contactMatrix(t):
    return C
# run model
data=model.simulate(S_0, E_0, Ia_0, Is_0, Ih_0, Ic_0, Im_0, contactMatrix, Tf, Nf)

t = data['t']; IC1  =  data['X'][:,2*M+1];
IC2  =  data['X'][:,2*M+8]

fig = plt.figure(num=None, figsize=(10, 8), dpi=80, facecolor='w', edgecolor='k')
plt.rcParams.update({'font.size': 22})

plt.plot(t, IC1, '-', lw=4, color='#A60628', label='M=1', alpha=0.8)
plt.plot(t, IC2, '-', lw=4, color='green', label='M=16', alpha=0.8)

plt.legend(fontsize=26, loc='upper left'); plt.grid()
plt.autoscale(enable=True, axis='x', tight=True)
plt.ylabel('Asymtomatic Infected individuals'); #plt.xlim(0, 40); plt.ylim(0, 4);
#plt.savefig('/Users/rsingh/Desktop/2a.png', format='png', dpi=212)
IC[0]
```


$$
    % LaTeX macros
    \newcommand{\genericdel}[3]{%
      \left#1{#3}\right#2
    }
    \newcommand{\del}[1]{\genericdel(){#1}}
    \newcommand{\sbr}[1]{\genericdel[]{#1}}
    \newcommand{\cbr}[1]{\genericdel\{\}{#1}}
    \newcommand{\abs}[1]{\genericdel||{#1}}
    \DeclareMathOperator*{\argmin}{arg\,min}
    \DeclareMathOperator*{\argmax}{arg\,max}
    \DeclareMathOperator{\Pr}{\mathbb{p}}
    \DeclareMathOperator{\E}{\mathbb{E}}
    \DeclareMathOperator{\Ind}{\mathbb{I}}
    \DeclareMathOperator{\V}{\mathbb{V}}
    \DeclareMathOperator{\cov}{Cov}
    \DeclareMathOperator{\var}{Var}
    \DeclareMathOperator{\ones}{\mathbf{1}}
    \DeclareMathOperator*{\argmin}{arg\,min}
    \DeclareMathOperator*{\argmax}{arg\,max}
    \DeclareMathOperator{\normal}{\mathcal{N}}
    \DeclareMathOperator{\unif}{Uniform}
    \DeclareMathOperator{\poisson}{Poisson}
    \DeclareMathOperator{\dif}{d}
    \newcommand{\od}[2]{\frac{\dif{#1}}{\dif{#2}}}
    \DeclareMathOperator{\Forall}{\forall}
    \newcommand{\iid}{iid}
    \newcommand{\indep}{\perp}
    \newcommand{\trans}{^{\intercal}}
$$

# Introduction

## Reminder of Setup

We have:
1. Noise-free outcomes $X_i$ and $Y_i$ drawn from distributions $F_X$ and $F_Y$ respectively.
2. Under the null, $F_X=F_Y$.
3. What we see are noisy-observations
\begin{equation}
    \widetilde{X_i} = X_i + \epsilon_{X,i}
    \quad\text{and}\quad
    \widetilde{Y_i} = Y_i + \epsilon_{Y,i}
\end{equation}
4. The distribution of the errors are known, for example they can be assumed to be normal
\begin{equation}
    \epsilon_{X,i} \sim \normal\del{0, \sigma^2_{X,i}}
    \quad\text{and}\quad
    \epsilon_{Y,i} \sim \normal\del{0, \sigma^2_{Y,i}}
\end{equation}
with known variances $\sigma^2_{X,i}$ and $\sigma^2_{Y,i}$.
5. In the homoscedastic case, the focus of `Homoscedastic_Comparison.ipynb`, all variances are the same:
\begin{equation}
\sigma^2_{X,i} = \sigma^2_{Y,j} = \sigma^2 \quad \forall\ i \in {1,\dotsc,n_X}, j \in {1,\dotsc,n_Y} \,.
\end{equation}
6. In the symmetric noise case, the focus of this notebook, the variances are mirrored between the two groups, with equal samples sizes.
\begin{equation}
\sigma^2_{X,i} = \sigma^2_{Y,i} = \sigma^2_i \quad \forall\ i \in {1,\dotsc,n} \,.
\end{equation}

## Purpose

The purpose of this notebook is to use simulations to study the power of our tests in the symmetric noise scenario.
To be specific, I will compare three tests:
1. The **classical** KS test.
2. The **singly deconvolved** KS test: uses Fourier deconvolution to estimate the null distribution (which when then sample from to get bootstrap samples under the null), but otherwise uses the regular KS test statistic, which is the $L_\infty$ distance between the *empirical* CDFs of $\widetilde{X}$ and $\widetilde{Y}$.
3. The **doubly deconvolved** KS test: uses Fourier deconvolution to estimate the null distribution, and also uses deconvolution for the test statistic, which is the $L_\infty$ distances between the *deconvolved* CDFs of $\widetilde{X}$ and $\widetilde{Y}$.

With these three tests, I use simulations with homoscedastic errors to estimate the power under the null (should be 0.05) and under the alternative.

**Question**: Is the KS test valid? I think so, but I'm not 100% sure.

# Notebook setup

```julia
using Distributions
using Dates
using Printf

using LaTeXStrings
import PyPlot; plt=PyPlot
plt.rc("figure", dpi=120.0)
# plt.rc("figure", figsize=(6,4))
# plt.rc("figure", autolayout=true)
plt.rc("savefig", dpi=120.0)
plt.rc("text", usetex=true)
plt.rc("font", family="serif")
plt.rc("font", serif="Palatino")
cbbPalette = ["#E69F00", "#56B4E9", "#009E73", "#F0E442", "#0072B2", "#D55E00", "#CC79A7"]
;
```

```julia
# a few convenience functions for plotting p-values and estimating power
function plot_ecdf(x; kwargs...)
    plt.plot(sort(x), range(0.0, stop=1.0, length=length(x)); kwargs...)
end
sim_power(pvals) = mean(pvals .<= 0.05)
function power_se(pvals)
    p = sim_power(pvals)
    q = 1-p
    n = length(pvals)
    return √(p*q/n)
end
estimate_power(pvals) = (sim_power(pvals), power_se(pvals))
;
```

```julia
using Revise
import DeconvolveDistribution
import DeconvolutionTests
```

```julia
module KS
    import DeconvolveDistribution
    using DeconvolveDistribution: FourierDeconv, EfronDeconv
    using HypothesisTests
    using HypothesisTests: ApproximateTwoSampleKSTest, pvalue
    using Distributions
    using StatsBase: sample, midpoints
    using DeconvolutionTests
    using DeconvolutionTests: deconv_boot_test,
            distance_test_statistic,
            KS_test_statistic

    function sim_symmetric(F_X::Distribution, F_Y::Distribution,
                      σ_distr::Distribution, n_XY::Int)
        n_X = n_Y = n_XY
        σ_X = σ_Y = rand(σ_distr, n_XY)
        X = rand(F_X, n_X)
        Y = rand(F_Y, n_Y)
        ϵX = σ_X .* randn(n_X)
        ϵY = σ_Y .* randn(n_Y)
        Xtilde = X .+ ϵX
        Ytilde = Y .+ ϵY
        return Dict(
            # inputs
            :F_X => F_X,
            :F_Y => F_Y,
            :σ_X_distr => σ_distr,
            :σ_Y_distr => σ_distr,
            :n_X => n_X,
            :n_Y => n_Y,
            # outputs
            :σ_X => σ_X,
            :σ_Y => σ_Y,
            :X => X,
            :Y => Y,
            :ϵX => ϵX,
            :ϵY => ϵY,
            :Xtilde => Xtilde,
            :Ytilde => Ytilde,
        )
    end

    function sim_KS(F_X, F_Y, σdistr, n_XY)
        # simulate homoscedastic data
        data = sim_symmetric(F_X, F_Y, σdistr, n_XY)

        Xtilde = data[:Xtilde]
        Ytilde = data[:Ytilde]

        # classical KS test ignoring errors
        KS_test = HypothesisTests.ApproximateTwoSampleKSTest(Xtilde, Ytilde)
        KS_pval = pvalue(KS_test; tail=:both)

        return KS_pval
    end

    function sim_tests(F_X, F_Y, σdistr, n_XY)
        # simulate homoscedastic data
        data = sim_symmetric(F_X, F_Y, σdistr, n_XY)

        Xtilde = data[:Xtilde]
        Ytilde = data[:Ytilde]
        σ_X = data[:σ_X]
        σ_Y = data[:σ_Y]

        # classical KS test ignoring errors
        KS_test = HypothesisTests.ApproximateTwoSampleKSTest(Xtilde, Ytilde)
        KS_pval = pvalue(KS_test; tail=:both)

        low = min(quantile(F_X, 0.01), quantile(F_Y, 0.01))
        high = max(quantile(F_X, 0.99), quantile(F_Y, 0.99))
        Xgrid = range(low, stop=high, length=75)
        method_null = FourierDeconv(0.3, Xgrid, 100)
        method_boot = FourierDeconv(0.3, Xgrid, 30)

        # deconvolved KS test using classical KS test statistic
        t_KS = KS_test_statistic
        deconvKS_pval = deconv_boot_test(Xtilde, Ytilde, σ_X, σ_Y, t_KS, method_null; niter=200)

        # deconvolved KS test using deconvolved KS test statistic
        p = Inf # L∞ norm (KS)
        bw = 0.3
        obsmin = min(minimum(Xtilde), minimum(Ytilde))
        obsmax = max(maximum(Xtilde), maximum(Ytilde))
        t_Lp = distance_test_statistic(p, method_boot, Xgrid)
        deconv2_pval = deconv_boot_test(Xtilde, Ytilde, σ_X, σ_Y, t_Lp, method_null; niter=200)

        return (KS_pval, deconvKS_pval, deconv2_pval)
    end

end
;
```

    WARNING: replacing module KS.

```julia
σdistr = Gamma(1.1, 4.0)
xx = range(0, stop=10, length=1000)
plt.plot(xx, pdf.(σdistr, xx))
;
```

```julia
# simulate homoscedastic data under the null hypothesis,
# 200 samples for each group
# where X, Y ~ Normal(0, 2²) and the noise is iid Normal(0, 0.8²).
# Then run the three tests and return the three p-values.
@time KS.sim_tests(Normal(0,2), Normal(0,2), σdistr, 200)
```

      2.988064 seconds (9.34 M allocations: 220.793 MiB, 0.93% gc time)

    (0.7920130315089249, 0.675, 0.715)

The Fourier deconvolution is slow, and for the doubly deconvolved test we need to perform it for every bootstrap sample, so each simulation takes a while (approx. 3 seconds).

# Power under the null

```julia
@time for _ in 1:3
    KS_pval, deconvKS_pval, deconv2_pval = KS.sim_tests(
        Normal(0,2), Normal(0,2), # F_X and F_Y are the same
        σdistr, 200)
end
```

      9.386351 seconds (27.62 M allocations: 656.178 MiB, 1.56% gc time)

```julia
@time Threads.@threads for _ in 1:3
    KS_pval, deconvKS_pval, deconv2_pval = KS.sim_tests(
        Normal(0,2), Normal(0,2), # F_X and F_Y are the same
        σdistr, 200)
end
```

      3.448356 seconds (25.35 M allocations: 597.069 MiB, 7.31% gc time)

```julia
nsims = 300
null_KS_arr       = Vector{Float64}(undef, nsims)
null_deconvKS_arr = Vector{Float64}(undef, nsims)
null_deconv2_arr  = Vector{Float64}(undef, nsims)
;
@time Threads.@threads for isim in 1:nsims
    KS_pval, deconvKS_pval, deconv2_pval = KS.sim_tests(
        Normal(0,2), Normal(0,2), # F_X and F_Y are the same
        σdistr, 200)
    null_KS_arr[isim] = KS_pval
    null_deconvKS_arr[isim] = deconvKS_pval
    null_deconv2_arr[isim] = deconv2_pval
end
```

    356.184045 seconds (2.36 G allocations: 55.455 GiB, 2.63% gc time)

```julia
plt.plot([0,1], [0,1])
plot_ecdf(null_KS_arr, label="KS")
plot_ecdf(null_deconvKS_arr, label="deconvolved KS")
plot_ecdf(null_deconv2_arr, label="doubly deconvolved")
plt.legend(loc="lower right")
plt.title("Under the null")
```

```julia
@printf("Power of classical     KS test under the null: %.3f ± %.3f\n", estimate_power(null_KS_arr)...)
@printf("Power of singly-deconv KS test under the null: %.3f ± %.3f\n", estimate_power(null_deconvKS_arr)...)
@printf("Power of doubly-deconv KS test under the null: %.3f ± %.3f\n", estimate_power(null_deconv2_arr)...)
```

    Power of classical     KS test under the null: 0.047 ± 0.012
    Power of singly-deconv KS test under the null: 0.077 ± 0.015
    Power of doubly-deconv KS test under the null: 0.063 ± 0.014

The above simply shows that all three tests are valid.

# Choosing alternative

The alternative will simply be that:
\begin{equation}
\begin{split}
    X & \sim \mathcal{N}(0, 2^2) \text{, and}\\
    Y & \sim \mathcal{N}(\mu, 2^2)\,.
\end{split}
\end{equation}
The purpose of this section is to choose $\mu$ so the power
of the KS test is somewhere in the range 0.8-0.95,
so that we can easily pick up differences in power
between the three tests in a reasonable scenario.

```julia
null_KS = [KS.sim_KS(Normal(0,2), Normal(0,2),   σdistr, 200) for _ in 1:10000]
alt1_KS = [KS.sim_KS(Normal(0,2), Normal(1.0,2), σdistr, 200) for _ in 1:10000]
alt2_KS = [KS.sim_KS(Normal(0,2), Normal(1.3,2), σdistr, 200) for _ in 1:10000]
alt3_KS = [KS.sim_KS(Normal(0,2), Normal(1.5,2), σdistr, 200) for _ in 1:10000]
;
```

```julia
mean(null_KS .<= 0.05)
```

    0.0398

```julia
mean(alt1_KS .<= 0.05)
```

    0.5874

```julia
mean(alt2_KS .<= 0.05)
```

    0.8151

```julia
mean(alt3_KS .<= 0.05)
```

    0.908

```julia
plt.plot([0, 1], [0, 1], color="grey")
plot_ecdf(null_KS, label="null")
plot_ecdf(alt1_KS, label="alt 1")
plot_ecdf(alt2_KS, label="alt 2")
plot_ecdf(alt3_KS, label="alt 3")
plt.legend(loc="lower right")
plt.xlim(0,0.1)
plt.xlabel(L"$p$-value")
plt.ylabel("cumulative probability")
;
```

We will choose the alternative to have $Y \sim \mathcal{N}(0.8, 2)$, under which the power of the classical KS test is 0.8, leaving plenty of room on either side for the other tests to show their worth.

# Power under the alternative

```julia
nsims = 3000
alt_KS_arr       = Vector{Float64}(undef, nsims)
alt_deconvKS_arr = Vector{Float64}(undef, nsims)
alt_deconv2_arr  = Vector{Float64}(undef, nsims)
;
@time Threads.@threads for isim in 1:nsims
    KS_pval, deconvKS_pval, deconv2_pval = KS.sim_tests(
        Normal(0,2), Normal(1.3,2), # F_X and F_Y are different
        σdistr, 200)
    alt_KS_arr[isim] = KS_pval
    alt_deconvKS_arr[isim] = deconvKS_pval
    alt_deconv2_arr[isim] = deconv2_pval
end
```

    3317.450835 seconds (25.56 G allocations: 593.592 GiB, 5.69% gc time)

```julia
plt.plot([0,1], [0,1])
plot_ecdf(alt_KS_arr, label="KS")
plot_ecdf(alt_deconvKS_arr, label="deconvolved KS")
plot_ecdf(alt_deconv2_arr, label="doubly deconvolved")
plt.legend(loc="lower right")
plt.title("Under the alternative")
```

```julia
@printf("Power of classical     KS test under the altv: %.3f ± %.3f\n", estimate_power(alt_KS_arr)...)
@printf("Power of singly-deconv KS test under the altv: %.3f ± %.3f\n", estimate_power(alt_deconvKS_arr)...)
@printf("Power of doubly-deconv KS test under the altv: %.3f ± %.3f\n", estimate_power(alt_deconv2_arr)...)
```

    Power of classical     KS test under the altv: 0.802 ± 0.007
    Power of singly-deconv KS test under the altv: 0.843 ± 0.007
    Power of doubly-deconv KS test under the altv: 0.910 ± 0.005

# Conclusion

First of all, all tests seem to be valid under the null. If anything, the classical KS test seems to be conservative.

There appears to be no significant change in power going from the classical KS test to the singly deconvolved KS test.
This is similar to what we observed in the homoscedastic case.
The reasoning is that this test is simply a Monte Carlo version of the classical KS test,
replacing analytical results based on asymptotics.

In the case of the doubly deconvolved test, there is a significant increase in power!
Under the alternative, the power goes up from 0.833 to 0.927.
This confirms the intuition that the double deconvolutions will give greater power
in cases of high heteroscedasticity.
In the homoscedastic case, we saw a *loss* of power.
Keep in mind that this uses the Fourier deconvolution, with an arbitrary number chosen for the bandwidth.
With better choice of bandwidth, or better deconvolution method, I'm hoping that the power loss will be lessened in the homoscedastic case, and that the power gain in the heteroscedastic case will be boosted.


Simulação de Canal de Comunicação segundo Modelo Erceg

```python
import numpy as np
import random
import matplotlib.pyplot as plt
from PIL import Image
import math
import matplotlib.pyplot as plt
import matplotlib.image as mpimg

%%latex
```

    UsageError: Line magic function `%%latex` not found.

O modelo Erceg foi construído e estudado de acordo com o dado experimentais coletados pela AT&T Wireless Service em 95 estações dentro dos EUA operando a 1,9GHz. O modelo enumera três categorias de terrenos que provocam perdas no sinal e eles são divididos da seguinte maneira; a categoria A representa terrenos com morros e colinas e com alta de densidade de obejetos no caminho do sinal, o que representa uma alta perda de percurso. Categoria C representa um terreno plano com baixa densidade de objetos no caminho do sinal, representando uma baixa perda do sinal, já a categoria B é constituída por terrenos montanhososo com baixa densidade de objetos ou terrenos plano com densidade consideravel de objetos no caminho do sinal, basicamente a categoria B representa um meio termo entre as categorias A e C, com perda de sinal mediana quando compara com as outras.

Para todas as três categorias o caminho de perda mediana é representada pela mesma equação com a condição de $d > d_0$, no caso a seguinte equação.$$
\begin{equation}\label{eq:erceg}
    P_L(dB) = 20\log_{10}(4\pi d_0/\lambda)+10\gamma \log_{10}(d/d_0)+s
\end{equation}
$$Onde $\lambda$ é o comprimento de onda do sinal, s é o efeito de sombreamento do sinal, $\gamma$ representa o caminho de menor perda considerando as três categorias do modelo, representado pela seguinte equação
$$
\gamma = a - bh_b + c/h_b
$$
$h_b$ representa a altura da estação base em metros, normalmente entre 10 e 80m, $d_0=100m$ e os valores nominais de a, b e c variam de acordo com a catgoria do terrenos, seus valores variam de acordo a seguinte tabela:

|Parametro|Categoria A|Categoria B|Categoria C|
|---------|-----------|-----------|-----------|
|a|4,6|3|3,6|
|b|0,0076|0,0065|0,005|
|c|12,6|17,1|20|

Definiçao das variáveis $d_0$, $h_b$ e $s$

```python
d0 = 100
hb = 50
s = 6
f = 1900000000
```

Definição das distâncias a serem simuladas

```python
d = np.arange(1, 1000, 1)
```

Construção dos arrays das categorias

```python
a = np.array([4.6, 3, 3.6])
b = np.array([0.0076, 0.0065, 0.005])
c = np.array([12.6, 17.1, 20])
```

Array com os valores de gamma

```python
gamma = [a[i]-b[i]*hb+c[i]/hb for i in range(3)]
```

Com os valores instânciados podemos analisar a resposta para cada uma das três situações

```python
v1 = 20*np.log10(4*math.pi*d0*f/300000000)+10*gamma[0]*np.log10(d/d0)+s
v2 = 20*np.log10(4*math.pi*d0*f/300000000)+10*gamma[1]*np.log10(d/d0)+s
v3 = 20*np.log10(4*math.pi*d0*f/300000000)+10*gamma[2]*np.log10(d/d0)+s
```

Plotando os resultados obtidos

```python
plt.plot(d, v1, label='Categoria A')
plt.plot(d, v2, label='Categoria B')
plt.plot(d, v3, label='Categoria C')

plt.xlabel('Distancia entre estacao movel e a base [m]')
plt.ylabel('Perda de potencia incidente na estacao movel [db]')
plt.legend()
# plt.title('Perda de Potencia em funcao da distancia entre estacao movel e base')
```

Analisando as curva obtidas em função da distância entre a estação rádio base e o dispositivo móvel podemos verificar para distâncias $d<d_0$ as curvas apresentam comportamento similar, apresentado perdas de potência próximas, um forte indicativo de que os fatores usados para compor o modelo Erceg entregam praticamente os mesmos valores independente da categoria do terreno por onde sinal se propaga. É interessante notar que as divergência entre as curvas para cada categoria se tornam evidentes quando $d>2d_0$, fazendo as curvas divergirem.

Um ponto interessante a se simular é fixar a distância entre a estação rádio móvel, no caso para as três situações possíveis, $d<d_0$, $d=d_0$ e $d>d_0$ e verificar o comportamento de uma frequencia variavel, alguns limites para esse caso são importantes pois o modelo Erceg foi projetado para trabalhar com frequências próximas de 2GHz, logo não faz sentido simular algo para todo o espectro, logo a simulação foi centrada em 2GHz com uma faixa de $+/-$300MHz, ou seja, foi simulado o comportamento da perda de potência para uma distância fixa para a faixa de frequência iniciando em 1,7GHz e finalizando em 2,3GHz no passo de 100KHz.

```python
fval = np.arange(1700000000, 2300000000, 100000)
fv1 = 20*np.log10(4*math.pi*d0*fval/300000000)+10*gamma[0]*np.log10(100/d0)+s
fv2 = 20*np.log10(4*math.pi*d0*fval/300000000)+10*gamma[1]*np.log10(100/d0)+s
fv3 = 20*np.log10(4*math.pi*d0*fval/300000000)+10*gamma[2]*np.log10(100/d0)+s

fv4 = 20*np.log10(4*math.pi*d0*fval/300000000)+10*gamma[0]*np.log10(50/d0)+s
fv5 = 20*np.log10(4*math.pi*d0*fval/300000000)+10*gamma[1]*np.log10(50/d0)+s
fv6 = 20*np.log10(4*math.pi*d0*fval/300000000)+10*gamma[2]*np.log10(50/d0)+s

fv7 = 20*np.log10(4*math.pi*d0*fval/300000000)+10*gamma[0]*np.log10(200/d0)+s
fv8 = 20*np.log10(4*math.pi*d0*fval/300000000)+10*gamma[1]*np.log10(200/d0)+s
fv9 = 20*np.log10(4*math.pi*d0*fval/300000000)+10*gamma[2]*np.log10(200/d0)+s
```

```python
plt.plot(fval, fv1, label='Categoria A d=d0')
plt.plot(fval, fv2, label='Categoria B d=d0')
plt.plot(fval, fv3, label='Categoria C d=d0')

plt.plot(fval, fv4, label='Categoria A d<d0')
plt.plot(fval, fv5, label='Categoria B d<d0')
plt.plot(fval, fv6, label='Categoria C d<d0')

plt.plot(fval, fv7, label='Categoria A d>d0')
plt.plot(fval, fv8, label='Categoria B d>d0')
plt.plot(fval, fv9, label='Categoria C d>d0')

plt.xlabel('Frequencia do sinal')
plt.ylabel('Perda de potencia incidente na estacao movel [db]')
plt.legend()
```

Analisando as curvas geradas, chama atenção que quando $d=d_0$ as curvas são iguais, ou seja, independente da categoria a variação da frequência produz o mesmo resultado, contudo este comportamente era previsto pelo modelo pois com $d=d_0$ o termo que depende das categorias de terreno é anulado do modelo, fazendo com que ele dependesse apenas da frequência.

Para distâncias $d<d_0$ o modelo apresenta baixa perdas quando comparado com os outros casos simulados, configurando este como o caso ideal de uso. Para distâncias $d>d_0$ o modelo já apresenta perdas maiores quando colocado em comparação com as outras curvas geradas.

Outra simulacao possivel seria variar a altura da torre fixando a uma frequencia e um $d$ especifico assim seria possivel enteder qual a relacao da alura da torre

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.210.3876&rep=rep1&type=pdf
http://morse.colorado.edu/~tlen5510/text/classwebch3.html
https://www.mathworks.com/matlabcentral/fileexchange/39322-erceg-model
https://jupyter-notebook.readthedocs.io/en/stable/examples/Notebook/Working%20With%20Markdown%20Cells.html

Simulação de Propagacao em Multipercurso

$$X(t)=X_c(t)+jX_s(t)$$
$$X_c(t)=\dfrac{2}{\sqrt M}\sum_{n=1}^{M}\cos (\Psi_n)\cos (\omega_d t \cos a_n + \phi)$$
$$X_s(t)=\dfrac{2}{\sqrt M}\sum_{n=1}^{M}\sin (\Psi_n)\cos (\omega_d t \cos a_n + \phi)$$
$$a_n = \dfrac{2\pi n - \pi + \theta}{4M}$$

Onde $\theta$, $\phi$ e $\Psi_n$ sao estatisticamente independente, variando entre $[-\pi,\pi)$ para todo $n$. A simulacao do artigo foi testada com $M = 8$

A expressão pode ser manilpulada de forma a reunir todos os fatores em uma única expressão.
$$
X_k(t)= \sqrt{\dfrac{2}{M}} \left\{ \sum_{n=1}^{M}\cos (\Psi_{n,k})\cos \left[\omega_d t \cos \left(\dfrac{2\pi n - \pi + \theta}{4M}\right) + \phi_k \right] \right\} + j\left\{ \sum_{n=1}^{M}\sin (\Psi_{n,k})\cos \left[\omega_d t \cos \left(\dfrac{2\pi n - \pi + \theta}{4M}\right) + \phi_k \right] \right\}
$$

Primeiro é preciso instanciar as bibliotecas para gerar um distribuição uniforme para $\theta$, $\phi$ e $\Psi_n$ entre $[-\pi,\pi)$

```python
import random
```

```python
# esta faltando considerar na formula o tempo mas fiquei confuso em como colocar ele

M = 8
const = np.sqrt(2/M)
Xk = []

for n in range(M):
    theta_multi = random.uniform(-np.pi, np.pi)
    phi_multi = random.uniform(-np.pi, np.pi)
    psi_multi = random.uniform(-np.pi, np.pi)

    a_n = (2*np.pi*n-np.pi+theta_multi)/(4*M)

    real = np.cos(psi_multi)*np.cos(2*np.pi*np.cos(a_n+phi_multi))
    imag = np.sin(psi_multi)*np.cos(2*np.pi*np.cos(a_n+phi_multi))

    Xk.append(const*np.complex(real, imag))

Xk = np.array(Xk)
print(Xk)

```

    [ 0.16894541+0.03640764j -0.09685047-0.31081219j  0.36910225-0.07887403j
     -0.47836453-0.13632683j -0.36197013+0.18670445j  0.3993364 -0.28533291j
     -0.34938184+0.03167521j  0.24443956+0.43130178j]

```python

```