# Orthogonal Random Forest: Use Cases and Examples

Orthogonal Random Forest (ORF) combines orthogonalization,
a technique that effectively removes the confounding effect in two-stage estimation,
with generalized random forests, a flexible method for estimating treatment effect heterogeneity. Due to the orthogonalization aspect of this method, the ORF performs especially well in the presence of high-dimensional confounders. For more details, see [this paper](https://arxiv.org/abs/1806.03467).

The EconML SDK implements the following OrthoForest variants:

* ContinuousTreatmentOrthoForest: suitable for continuous treatments

* DiscreteTreatmentOrthoForest: suitable for discrete treatments

In this notebook, we show the performance of the ORF on synthetic data.

**Notebook contents:**

1. Example usage with continuous treatment synthetic data

2. Example usage with binary treatment synthetic data

3. Example usage with multiple discrete treatment synthetic data

4. Example usage with real continuous treatment observational data

```python
import econml
```

```python
# Main imports
from econml.ortho_forest import ContinuousTreatmentOrthoForest, WeightedModelWrapper, DiscreteTreatmentOrthoForest

# Helper imports
import numpy as np
from itertools import product
from sklearn.linear_model import Lasso, LassoCV, LogisticRegression, LogisticRegressionCV
#from sklearn.multioutput import MultiOutputRegressor
import matplotlib.pyplot as plt

%matplotlib inline
```

## 1. Example Usage with Continuous Treatment Synthetic Data

### 1.1. DGP
We use the data generating process (DGP) from [here](https://arxiv.org/abs/1806.03467). The DGP is described by the following equations:

\begin{align}
T =& \langle W, \beta\rangle + \eta, & \;\eta \sim \text{Uniform}(-1, 1)\\
Y =& T\cdot \theta(X) + \langle W, \gamma\rangle + \epsilon, &\; \epsilon \sim \text{Uniform}(-1, 1)\\
W \sim& \text{Normal}(0,\, I_{n_w})\\
X \sim& \text{Uniform}(0,1)^{n_x}
\end{align}

where $W$ is a matrix of high-dimensional confounders and $\beta, \gamma$ have high sparsity.

For this DGP,
\begin{align}
\theta(x) = \exp(2\cdot x_1).
\end{align}

```python
# Treatment effect function
def exp_te(x):
    return np.exp(2*x[0])
```

```python
# DGP constants
np.random.seed(123)
n = 1000
n_w = 30
support_size = 5
n_x = 1
# Outcome support
support_Y = np.random.choice(range(n_w), size=support_size, replace=False)
coefs_Y = np.random.uniform(0, 1, size=support_size)
epsilon_sample = lambda n: np.random.uniform(-1, 1, size=n)
# Treatment support
support_T = support_Y
coefs_T = np.random.uniform(0, 1, size=support_size)
eta_sample = lambda n: np.random.uniform(-1, 1, size=n)

# Generate controls, covariates, treatments and outcomes
W = np.random.normal(0, 1, size=(n, n_w))
X = np.random.uniform(0, 1, size=(n, n_x))
# Heterogeneous treatment effects
TE = np.array([exp_te(x_i) for x_i in X])
T = np.dot(W[:, support_T], coefs_T) + eta_sample(n)
Y = TE * T + np.dot(W[:, support_Y], coefs_Y) + epsilon_sample(n)

# ORF parameters and test data
# The following parameters are set according to theory
subsample_power = 0.88
subsample_ratio = ((n/np.log(n_w))**(subsample_power)) / n
lambda_reg = np.sqrt(np.log(n_w) / (10 * subsample_ratio * n))
X_test = np.array(list(product(np.arange(0, 1, 0.01), repeat=n_x)))
```

### 1.2. Train Estimator

**Note:** The models in the final stage of the estimation (``model_T_final``, ``model_Y_final``) need to support sample weighting.

If the models of choice do not support sample weights (e.g. ``sklearn.linear_model.LassoCV``), the ``econml`` packages provides a convenient wrapper for these models ``WeightedModelWrapper`` in order to allow sample weights.

If the model of choice is a linear (regression) model such as Lasso, you should set ``sample_type="weighted"``. Otherwise, set ``sample_type="sampled"``.

```python
est = ContinuousTreatmentOrthoForest(
    n_trees=200, min_leaf_size=5,
    max_splits=50, subsample_ratio=2*subsample_ratio, bootstrap=False,
    model_T=Lasso(alpha=lambda_reg),
    model_Y=Lasso(alpha=lambda_reg),
    model_T_final=WeightedModelWrapper(Lasso(alpha=lambda_reg), sample_type="weighted"),
    model_Y_final=WeightedModelWrapper(Lasso(alpha=lambda_reg), sample_type="weighted"),
    random_state=123)
```

```python
est.fit(Y, T, X, W)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    9.2s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:   48.0s
    [Parallel(n_jobs=-1)]: Done 200 out of 200 | elapsed:  1.4min finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    6.5s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:   45.1s
    [Parallel(n_jobs=-1)]: Done 200 out of 200 | elapsed:  1.4min finished

    <econml.ortho_forest.ContinuousTreatmentOrthoForest at 0x125a40550>

```python
treatment_effects = est.const_marginal_effect(X_test)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:   35.7s
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:  3.6min finished

### 1.3. Performance Visualization

```python
y = treatment_effects[:, 0]
plt.plot(X_test, y, label='ORF estimate')
expected_te = np.array([exp_te(x_i) for x_i in X_test])
plt.plot(X_test[:, 0], expected_te, 'b--', label='True effect')
plt.ylabel("Treatment Effect")
plt.xlabel("x")
plt.legend()
plt.show()
```

## 2. Example Usage with Binary Treatment Synthetic Data

### 2.1. DGP
We use the following DGP:

\begin{align}
T \sim & \text{Bernoulli}\left(f(W)\right), &\; f(W)=\sigma(\langle W, \beta\rangle + \eta), \;\eta \sim \text{Uniform}(-1, 1)\\
Y = & T\cdot \theta(X) + \langle W, \gamma\rangle + \epsilon, & \; \epsilon \sim \text{Uniform}(-1, 1)\\
W \sim & \text{Normal}(0,\, I_{n_w}) & \\
X \sim & \text{Uniform}(0,\, 1)^{n_x}
\end{align}

where $W$ is a matrix of high-dimensional confounders, $\beta, \gamma$ have high sparsity and $\sigma$ is the sigmoid function.

For this DGP,
\begin{align}
\theta(x) = \exp( 2\cdot x_1 ).
\end{align}

```python
# DGP constants
np.random.seed(1234)
n = 1000
n_w = 30
support_size = 5
n_x = 1
# Outcome support
support_Y = np.random.choice(range(n_w), size=support_size, replace=False)
coefs_Y = np.random.uniform(0, 1, size=support_size)
epsilon_sample = lambda n: np.random.uniform(-1, 1, size=n)
# Treatment support
support_T = support_Y
coefs_T = np.random.uniform(0, 1, size=support_size)
eta_sample = lambda n: np.random.uniform(-1, 1, size=n)

# Generate controls, covariates, treatments and outcomes
W = np.random.normal(0, 1, size=(n, n_w))
X = np.random.uniform(0, 1, size=(n, n_x))
# Heterogeneous treatment effects
TE = np.array([exp_te(x_i) for x_i in X])
# Define treatment
log_odds = np.dot(W[:, support_T], coefs_T) + eta_sample(n)
T_sigmoid = 1/(1 + np.exp(-log_odds))
T = np.array([np.random.binomial(1, p) for p in T_sigmoid])
# Define the outcome
Y = TE * T + np.dot(W[:, support_Y], coefs_Y) + epsilon_sample(n)

# ORF parameters and test data
# The following parameters are set according to theory
subsample_power = 0.88
subsample_ratio = ((n/np.log(n_w))**(subsample_power)) / n
lambda_reg = np.sqrt(np.log(n_w) / (10 * subsample_ratio * n))
X_test = np.array(list(product(np.arange(0, 1, 0.01), repeat=n_x)))
```

### 2.2. Train Estimator

```python
est = DiscreteTreatmentOrthoForest(
    n_trees=200, min_leaf_size=10,
    max_splits=30, subsample_ratio=2*subsample_ratio, bootstrap=False,
    propensity_model = LogisticRegression(C=1/(X.shape[0]*lambda_reg), penalty='l1'),
    model_Y = Lasso(alpha=lambda_reg),
    propensity_model_final=LogisticRegression(C=1/(X.shape[0]*lambda_reg), penalty='l1'),
    model_Y_final=WeightedModelWrapper(Lasso(alpha=lambda_reg), sample_type="weighted")
)
```

```python
est.fit(Y, T, X, W)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    5.6s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:   27.4s
    [Parallel(n_jobs=-1)]: Done 200 out of 200 | elapsed:   44.1s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    3.2s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:   22.9s
    [Parallel(n_jobs=-1)]: Done 200 out of 200 | elapsed:   37.6s finished

    <econml.ortho_forest.DiscreteTreatmentOrthoForest at 0x12bbc44a8>

```python
treatment_effects = est.const_marginal_effect(X_test)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:   33.4s
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed:  3.4min finished

### 2.3. Performance Visualization

```python
y = treatment_effects[:, 0]
plt.plot(X_test, y, label='ORF estimate')
expected_te = np.array([exp_te(x_i) for x_i in X_test])
plt.plot(X_test[:, 0], expected_te, 'b--', label='True effect')
plt.ylabel("Treatment Effect")
plt.xlabel("x")
plt.legend()
plt.show()
```

## 3. Example Usage with Multiple Treatment Synthetic Data

### 3.1 DGP
We use the following DGP:

\begin{align}
Y = & \sum_{t=1}^{n_{\text{treatments}}} 1\{T=t\}\cdot \theta_{T}(X) + \langle W, \gamma\rangle + \epsilon, \; \epsilon \sim \text{Unif}(-1, 1), \\
\text{Pr}[T=t \mid W] \propto & \exp\{\langle W, \beta_t \rangle\}, \;\;\;\; \forall t\in \{0, 1, \ldots, n_{\text{treatments}}\}
\end{align}

where $W$ is a matrix of high-dimensional confounders, $\beta_t, \gamma$ are sparse.

For this particular example DGP we used $n_{\text{treatments}}=3$ and
\begin{align}
\theta_1(x) = & \exp( 2 x_1 ),\\
\theta_2(x) = &  3 \cdot \sigma(100\cdot (x_1 - .5)),\\
\theta_3(x) = & -2 \cdot \sigma(100\cdot (x_1 - .25)),
\end{align}
where $\sigma$ is the sigmoid function.

```python
def get_test_train_data(n, n_w, support_size, n_x, te_func, n_treatments):
    # Outcome support
    support_Y = np.random.choice(range(n_w), size=support_size, replace=False)
    coefs_Y = np.random.uniform(0, 1, size=support_size)
    epsilon_sample = lambda n: np.random.uniform(-1, 1, size=n)
    # Treatment support
    support_T = support_Y
    coefs_T = np.random.uniform(0, 1, size=(support_size, n_treatments))
    eta_sample = lambda n: np.random.uniform(-1, 1, size=n)
    # Generate controls, covariates, treatments and outcomes
    W = np.random.normal(0, 1, size=(n, n_w))
    X = np.random.uniform(0, 1, size=(n, n_x))
    # Heterogeneous treatment effects
    TE = np.array([te_func(x_i, n_treatments) for x_i in X])
    log_odds = np.dot(W[:, support_T], coefs_T)
    T_sigmoid = np.exp(log_odds)
    T_sigmoid = T_sigmoid/np.sum(T_sigmoid, axis=1, keepdims=True)
    T = np.array([np.random.choice(n_treatments, p=p) for p in T_sigmoid])
    TE = np.concatenate((np.zeros((n,1)), TE), axis=1)
    Y = TE[np.arange(n), T] + np.dot(W[:, support_Y], coefs_Y) + epsilon_sample(n)
    X_test = np.array(list(product(np.arange(0, 1, 0.01), repeat=n_x)))

    return (Y, T, X, W), (X_test, np.array([te_func(x, n_treatments) for x in X_test]))
```

```python
import scipy.special
def te_func(x, n_treatments):
    return [np.exp(2*x[0]), 3*scipy.special.expit(100*(x[0] - .5)) - 1, -2*scipy.special.expit(100*(x[0] - .25))]

np.random.seed(123)
(Y, T, X, W), (X_test, te_test) = get_test_train_data(1000, 3, 3, 1, te_func, 4)
```

### 3.2 Train Estimator

```python
est = DiscreteTreatmentOrthoForest(n_trees=500,
                                   propensity_model = LogisticRegression(C=1/(X.shape[0]*lambda_reg), penalty='l1'),
                                   model_Y = WeightedModelWrapper(Lasso(alpha=lambda_reg)))
```

```python
est.fit(Y, T, X, W)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:   14.4s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:  1.4min
    [Parallel(n_jobs=-1)]: Done 272 tasks      | elapsed:  3.2min
    [Parallel(n_jobs=-1)]: Done 500 out of 500 | elapsed:  5.8min finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:   11.3s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:  1.3min
    [Parallel(n_jobs=-1)]: Done 272 tasks      | elapsed:  3.2min
    [Parallel(n_jobs=-1)]: Done 500 out of 500 | elapsed:  5.8min finished

    <econml.ortho_forest.DiscreteTreatmentOrthoForest at 0x1267494a8>

```python
treatment_effects = est.const_marginal_effect(X_test)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:  2.5min
    [Parallel(n_jobs=-1)]: Done 100 out of 100 | elapsed: 15.1min finished

### 3.3 Performance Visualization

```python
y = treatment_effects
for it in range(y.shape[1]):
    plt.plot(X_test, y[:, it], label='ORF estimate T={}'.format(it))
    plt.plot(X_test[:, 0], te_test[:, it], '--', label='True effect T={}'.format(it))
plt.ylabel("Treatment Effect")
plt.xlabel("x")
plt.legend()
plt.show()
```

## 4. Example usage with real continuous treatment observational data

We applied our technique to Dominick’s dataset, a popular historical dataset of store-level orange juice prices and sales provided by University of Chicago Booth School of Business.

The dataset is comprised of a large number of covariates $W$, but researchers might only be interested in learning the elasticity of demand as a function of a few variables $x$ such
as income or education.

We applied the `ContinuousTreatmentOrthoForest` to estimate orange juice price elasticity
as a function of income, and our results, unveil the natural phenomenon that lower income consumers are more price-sensitive.

### 4.1. Data

```python
# A few more imports
import os
import pandas as pd
import urllib.request
from sklearn.preprocessing import StandardScaler
```

```python
# Import the data
file_name = "oj_large.csv"

if not os.path.isfile(file_name):
    print("Downloading file (this might take a few seconds)...")
    urllib.request.urlretrieve("https://msalicedatapublic.blob.core.windows.net/datasets/OrangeJuice/oj_large.csv", file_name)
oj_data = pd.read_csv(file_name)
oj_data.head()
```

      store
      brand
      week
      logmove
      feat
      price
      AGE60
      EDUC
      ETHNIC
      INCOME
      HHLARGE
      WORKWOM
      HVAL150
      SSTRDIST
      SSTRVOL
      CPDIST5
      CPWVOL5

      0
      2
      tropicana
      40
      9.018695
      0
      3.87
      0.232865
      0.248935
      0.11428
      10.553205
      0.103953
      0.303585
      0.463887
      2.110122
      1.142857
      1.92728
      0.376927

      1
      2
      tropicana
      46
      8.723231
      0
      3.87
      0.232865
      0.248935
      0.11428
      10.553205
      0.103953
      0.303585
      0.463887
      2.110122
      1.142857
      1.92728
      0.376927

      2
      2
      tropicana
      47
      8.253228
      0
      3.87
      0.232865
      0.248935
      0.11428
      10.553205
      0.103953
      0.303585
      0.463887
      2.110122
      1.142857
      1.92728
      0.376927

      3
      2
      tropicana
      48
      8.987197
      0
      3.87
      0.232865
      0.248935
      0.11428
      10.553205
      0.103953
      0.303585
      0.463887
      2.110122
      1.142857
      1.92728
      0.376927

      4
      2
      tropicana
      50
      9.093357
      0
      3.87
      0.232865
      0.248935
      0.11428
      10.553205
      0.103953
      0.303585
      0.463887
      2.110122
      1.142857
      1.92728
      0.376927

```python
# Prepare data
Y = oj_data['logmove'].values
T = np.log(oj_data["price"]).values
scaler = StandardScaler()
W1 = scaler.fit_transform(oj_data[[c for c in oj_data.columns if c not in ['price', 'logmove', 'brand', 'week', 'store']]].values)
W2 = pd.get_dummies(oj_data[['brand']]).values
W = np.concatenate([W1, W2], axis=1)
X = oj_data[['INCOME']].values
```

### 4.2. Train Estimator

```python
# Define some parameters
n_trees = 500
min_leaf_size = 50
max_splits = 20
subsample_ratio = 0.02
bootstrap = False
```

```python
est = ContinuousTreatmentOrthoForest(
        n_trees=n_trees, min_leaf_size=min_leaf_size, max_splits=max_splits,
        subsample_ratio=subsample_ratio, bootstrap=bootstrap,
        model_T=Lasso(alpha=0.5), model_Y=Lasso(alpha=0.5),
        model_T_final=WeightedModelWrapper(LassoCV(), sample_type="weighted"),
        model_Y_final=WeightedModelWrapper(LassoCV(), sample_type="weighted")
       )
```

```python
min_income = 10.4
max_income = 10.9
delta = (max_income - min_income) / 50
X_test = np.arange(min_income, max_income + delta - 0.001, delta).reshape(-1, 1)
```

```python
est.fit(Y, T, X, W)
te_pred = est.const_marginal_effect(X_test)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    2.7s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:    6.5s
    [Parallel(n_jobs=-1)]: Done 272 tasks      | elapsed:   12.8s
    [Parallel(n_jobs=-1)]: Done 500 out of 500 | elapsed:   21.8s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.7s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:    4.5s
    [Parallel(n_jobs=-1)]: Done 272 tasks      | elapsed:   10.8s
    [Parallel(n_jobs=-1)]: Done 500 out of 500 | elapsed:   19.7s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:  1.0min
    [Parallel(n_jobs=-1)]: Done  51 out of  51 | elapsed:  3.4min finished

### 4.3. Performance Visualization

```python
# Plot Oranje Juice elasticity as a function of income
plt.plot(np.ndarray.flatten(X_test), te_pred[:, 0], label="OJ Elasticity")
plt.xlabel(r'$\log$(Income)')
plt.ylabel('Orange Juice Elasticity')
plt.xlim(10.45, 10.9)
plt.ylim(-2.9, -2.3)
plt.legend()
plt.title("Orange Juice Elasticity vs Income")
plt.show()
```

### 4.4 Bootstrap Confidence Intervals

```python
from econml.bootstrap import BootstrapEstimator
boot_est = BootstrapEstimator(ContinuousTreatmentOrthoForest(
        n_trees=n_trees, min_leaf_size=min_leaf_size, max_splits=max_splits,
        subsample_ratio=subsample_ratio, bootstrap=bootstrap,
        model_T=Lasso(alpha=0.5), model_Y=Lasso(alpha=0.5),
        model_T_final=WeightedModelWrapper(LassoCV(), sample_type="weighted"),
        model_Y_final=WeightedModelWrapper(LassoCV(), sample_type="weighted")
       ), n_bootstrap_samples=10)
```

```python
boot_est.fit(Y, T, X, W)
te_pred_interval = boot_est.const_marginal_effect_interval(X_test, lower=1, upper=99)
```

    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    2.7s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:    6.6s
    [Parallel(n_jobs=-1)]: Done 272 tasks      | elapsed:   12.9s
    [Parallel(n_jobs=-1)]: Done 500 out of 500 | elapsed:   21.7s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.7s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:    4.5s
    [Parallel(n_jobs=-1)]: Done 272 tasks      | elapsed:   10.8s
    [Parallel(n_jobs=-1)]: Done 500 out of 500 | elapsed:   19.7s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.7s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:    4.5s
    [Parallel(n_jobs=-1)]: Done 272 tasks      | elapsed:   10.8s
    [Parallel(n_jobs=-1)]: Done 500 out of 500 | elapsed:   19.6s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.7s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:    4.5s
    [Parallel(n_jobs=-1)]: Done 272 tasks      | elapsed:   10.8s
    [Parallel(n_jobs=-1)]: Done 500 out of 500 | elapsed:   20.0s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.7s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:    4.8s
    [Parallel(n_jobs=-1)]: Done 272 tasks      | elapsed:   11.1s
    [Parallel(n_jobs=-1)]: Done 500 out of 500 | elapsed:   19.9s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.7s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:    4.5s
    [Parallel(n_jobs=-1)]: Done 272 tasks      | elapsed:   11.0s
    [Parallel(n_jobs=-1)]: Done 500 out of 500 | elapsed:   20.2s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.7s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:    4.4s
    [Parallel(n_jobs=-1)]: Done 272 tasks      | elapsed:   10.8s
    [Parallel(n_jobs=-1)]: Done 500 out of 500 | elapsed:   19.7s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.7s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:    4.5s
    [Parallel(n_jobs=-1)]: Done 272 tasks      | elapsed:   11.0s
    [Parallel(n_jobs=-1)]: Done 500 out of 500 | elapsed:   19.8s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.7s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:    4.5s
    [Parallel(n_jobs=-1)]: Done 272 tasks      | elapsed:   11.3s
    [Parallel(n_jobs=-1)]: Done 500 out of 500 | elapsed:   20.3s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.7s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:    4.5s
    [Parallel(n_jobs=-1)]: Done 272 tasks      | elapsed:   11.1s
    [Parallel(n_jobs=-1)]: Done 500 out of 500 | elapsed:   20.8s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.7s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:    4.5s
    [Parallel(n_jobs=-1)]: Done 272 tasks      | elapsed:   10.8s
    [Parallel(n_jobs=-1)]: Done 500 out of 500 | elapsed:   19.7s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.7s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:    4.6s
    [Parallel(n_jobs=-1)]: Done 272 tasks      | elapsed:   11.1s
    [Parallel(n_jobs=-1)]: Done 500 out of 500 | elapsed:   20.5s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.7s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:    4.4s
    [Parallel(n_jobs=-1)]: Done 272 tasks      | elapsed:   10.7s
    [Parallel(n_jobs=-1)]: Done 500 out of 500 | elapsed:   19.7s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.7s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:    4.4s
    [Parallel(n_jobs=-1)]: Done 272 tasks      | elapsed:   10.9s
    [Parallel(n_jobs=-1)]: Done 500 out of 500 | elapsed:   19.6s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.7s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:    4.6s
    [Parallel(n_jobs=-1)]: Done 272 tasks      | elapsed:   10.9s
    [Parallel(n_jobs=-1)]: Done 500 out of 500 | elapsed:   19.7s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.7s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:    4.5s
    [Parallel(n_jobs=-1)]: Done 272 tasks      | elapsed:   10.9s
    [Parallel(n_jobs=-1)]: Done 500 out of 500 | elapsed:   20.4s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.7s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:    4.5s
    [Parallel(n_jobs=-1)]: Done 272 tasks      | elapsed:   10.8s
    [Parallel(n_jobs=-1)]: Done 500 out of 500 | elapsed:   19.7s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.7s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:    4.6s
    [Parallel(n_jobs=-1)]: Done 272 tasks      | elapsed:   10.9s
    [Parallel(n_jobs=-1)]: Done 500 out of 500 | elapsed:   19.7s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.8s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:    4.5s
    [Parallel(n_jobs=-1)]: Done 272 tasks      | elapsed:   10.9s
    [Parallel(n_jobs=-1)]: Done 500 out of 500 | elapsed:   19.8s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:    0.8s
    [Parallel(n_jobs=-1)]: Done 112 tasks      | elapsed:    4.8s
    [Parallel(n_jobs=-1)]: Done 272 tasks      | elapsed:   11.7s
    [Parallel(n_jobs=-1)]: Done 500 out of 500 | elapsed:   20.7s finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:  1.1min
    [Parallel(n_jobs=-1)]: Done  51 out of  51 | elapsed:  3.4min finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:  1.0min
    [Parallel(n_jobs=-1)]: Done  51 out of  51 | elapsed:  3.3min finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:  1.0min
    [Parallel(n_jobs=-1)]: Done  51 out of  51 | elapsed:  3.3min finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:  1.0min
    [Parallel(n_jobs=-1)]: Done  51 out of  51 | elapsed:  3.3min finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:  1.0min
    [Parallel(n_jobs=-1)]: Done  51 out of  51 | elapsed:  3.3min finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:  1.0min
    /anaconda/lib/python3.6/site-packages/joblib/externals/loky/process_executor.py:706: UserWarning: A worker stopped while some jobs were given to the executor. This can be caused by a too short worker timeout or by a memory leak.
      "timeout or by a memory leak.", UserWarning
    [Parallel(n_jobs=-1)]: Done  51 out of  51 | elapsed: 34.8min finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:  1.1min
    [Parallel(n_jobs=-1)]: Done  51 out of  51 | elapsed:  3.6min finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:  1.2min
    [Parallel(n_jobs=-1)]: Done  51 out of  51 | elapsed:  3.7min finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:  1.1min
    [Parallel(n_jobs=-1)]: Done  51 out of  51 | elapsed:  3.6min finished
    [Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.
    [Parallel(n_jobs=-1)]: Done  16 tasks      | elapsed:  1.1min
    [Parallel(n_jobs=-1)]: Done  51 out of  51 | elapsed:  3.5min finished

```python
plt.plot(np.ndarray.flatten(X_test), te_pred[:, 0], label="OJ Elasticity")
plt.fill_between(np.ndarray.flatten(X_test),
                 te_pred_interval[0][:, 0],
                 te_pred_interval[1][:, 0], alpha=.5, label="1-99% CI")
plt.xlabel(r'$\log$(Income)')
plt.ylabel('Orange Juice Elasticity')
plt.title("Orange Juice Elasticity vs Income")
plt.legend()
plt.show()
```

notebooks/Orthogonal Random Forest Examples.ipynb


## Time Series Classification using Gaussian processes with Signature Covariances
#### _Using SVGP with SignatureRBF kernel over sequences and inducing tensors as inducing point locations_
_Dataset: LIngua BRAsileira de Sinais (LIBRAS)_
***

```python
# imports
import os
os.environ['CUDA_VISIBLE_DEVICES'] = '3'

import sys
sys.path.append('..') # add to path parent dir of gpsig

# computations
import numpy as np
import tensorflow as tf
tf.logging.set_verbosity(tf.logging.ERROR)
import gpflow as gp
import gpsig

# preprocessing, model evaluation
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score, classification_report

# plotting
import matplotlib.pyplot as plt
plt.style.use('seaborn-darkgrid')

# data loading
import requests, zipfile, io
from sktime.utils.load_data import load_from_arff_to_dataframe
```

***
### Load dataset
Load a multivariate time series dataset (LIBRAS) from timeseriesclassification.com (https://arxiv.org/abs/1811.00075).

__LIBRAS__ is an acronym for the official Brazilian sign language. The dataset consists of 15 classes with 24 instances each. Each example corresponds to a 2-dimensional curve followed by the hand while performing the given sign with 45 time-observations.

```python
dataset_name = 'Libras'
dataset_url = 'https://timeseriesclassification.com/Downloads/{}.zip'.format(dataset_name)

# download, extract and load dataset
r = requests.get(dataset_url, stream=True)
if r.ok:
    z = zipfile.ZipFile(io.BytesIO(r.content))
    z.extractall('./{}/'.format(dataset_name))
else:
    print('Error: the dataset could not be downloaded.')

X_train, y_train = load_from_arff_to_dataframe('./{0}/{0}_TRAIN.arff'.format(dataset_name))
X_test, y_test = load_from_arff_to_dataframe('./{0}/{0}_TEST.arff'.format(dataset_name))
```

***
### Format and preprocess data

We first take care of formatting the outputs $\mathbf{Y} = (\mathbf{y}_i)_{i=1,\dots,n_{\mathbf{x}}}$. We are going to use the GPflow MultiClass likelihood as observation model as the problem class is multi-class classification. It expects that class labels are assigned from $\{0, 1, \dots, n_{c}-1\}$ with $n_c$ the number of classes, hence we simply relabel all examples.

```python
labels_dict = {c : i for i, c in enumerate(np.unique(y_train))}
y_train = np.asarray([labels_dict[c] for c in y_train])
y_test = np.asarray([labels_dict[c] for c in y_test])
```

To format the inputs $\mathbf{X} = (\mathbf{x}_i)_{i=1,\dots,n_{\mathbf{X}}} \subset \mathbb{R}^d_{seq}$, we convert the data to a list of sequences, with each sequence $\mathbf{x} \in \mathbf{X}$ given as an $(l_{\mathbf{x}}, d)$ numpy array.

```python
X_train = [np.stack(x, axis=1) for x in X_train.values]
X_test = [np.stack(x, axis=1) for x in X_test.values]
```

Now as preprocessing, we first add a monotonically increasing coordinate with unit steps between time-observations, that we call "time", and its role is to allow signatures to separate sequences, that are not equal, but can be warped into one another. Hence, denoting time-augmentation as $\hat{\cdot}: \mathbb{R}^d_{seq} \rightarrow \mathbb{R}^{d+1}_{seq}$ defined for $\mathbf{x} = (x_i)_{i=1,\dots,l_\mathbf{x}}$ as $\hat{\mathbf{x}} := (t_i, x_i)_{i=1,\dots,l_{\mathbf{x}}}$, where $t_i := i / l_{\mathbf{x}}$.

```python
# implemented in gpsig as a convenience function
X_train = gpsig.preprocessing.add_time_to_list(X_train)
X_test = gpsig.preprocessing.add_time_to_list(X_test)
```

As we would like to have a validation set that we monitor while optimizing the hyperparameters of the model, we split the training data 80-20 in a stratified manner.

```python
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2, shuffle=True, stratify=y_train)
```

Continuing with the preprocessing, we jointly normalize using the training set all state-space dimensions (in this case, the time coordinate and the two coordinates corresponding to hand-movements) to zero mean and unit variance.

```python
scaler = StandardScaler()
scaler.fit(np.concatenate(X_train, axis=0))

X_train = [scaler.transform(x) for x in X_train]
X_val = [scaler.transform(x) for x in X_val]
X_test = [scaler.transform(x) for x in X_test]
```

Finally, as the dataformat of the inputs should be a numpy array of size $(n_{\mathbf{X}}, \max_{\mathbf{x} \in \mathbf{X}} l_{\mathbf{x}}, d)$, we tabulate all sequences to equal length (in case they weren't already), by repeating for each $\mathbf{x} \in \mathbf{X}$ the last observation $\max_{\tilde{\mathbf{x}} \in \mathbf{X}} l_{\tilde{\mathbf{x}}} - l_{\mathbf{x}}$ times.

```python
# using another convenience function from gpsig
X_train = gpsig.preprocessing.tabulate_list_of_sequences(X_train)
X_val = gpsig.preprocessing.tabulate_list_of_sequences(X_val)
X_test = gpsig.preprocessing.tabulate_list_of_sequences(X_test)
```

    100%|██████████| 144/144 [00:00<00:00, 20299.79it/s]
    100%|██████████| 36/36 [00:00<00:00, 11684.20it/s]
    100%|██████████| 180/180 [00:00<00:00, 25359.40it/s]

To check that everything we did makes sense, we investigate the sizes of the data arrays and plot some examples from the inputs and a histogram from the outputs.

```python
print('X_train shape: {}'.format(X_train.shape))
print('y_train shape: {}'.format(y_train.shape))
print('X_val shape: {}'.format(X_val.shape))
print('y_val shape: {}'.format(y_val.shape))
print('X_test shape: {}'.format(X_test.shape))
print('y_test shape: {}'.format(y_test.shape))

fig, axes = plt.subplots(1, 4, figsize=(20, 5))
axes[0].plot(X_train[0])
axes[1].plot(X_train[20])
axes[2].plot(X_train[50])
axes[3].hist(y_train)

plt.show()
```

***
### Setup SVGP model

```python
num_train, len_examples, num_features = X_train.shape
num_val = X_val.shape[0]
num_test = X_test.shape[0]
num_classes = np.unique(y_train).size
```

We are going to use $M=5$ levels in the signature kernel, i.e. the truncated signature kernel of depth $M=5$. The data will be summarized using $n_{\mathbf Z} = 200$ inducing tensors in the variational posterior. For simpliciy, we use $p=0$ lags in the signature kernel.

```python
num_levels = 4
num_inducing = 200
num_lags = 0
```

To initialize the inducing tensors, we use a function from gpsig that samples random sparse tensors from the input sequences, and initialize the lengthscales by $l_i^{(0)} := \sqrt{\mathbb{E}\vert x_i - \tilde{x}_i\vert^2\cdot d}$ for $i \in \{1, 2, \dots, d\}$ using $n = 1000$ randomly selected observation samples from the data.

```python
Z_init = gpsig.utils.suggest_initial_inducing_tensors(X_train, num_levels, num_inducing, labels=y_train, increments=True, num_lags=num_lags)
l_init = gpsig.utils.suggest_initial_lengthscales(X_train, num_samples=1000)
```

Since GPflow kernels usually expect as inputs numpy arrays in a 2d axis format, we flatten out the last two dimensions of the input arrays (which are then expanded again inside the kernel calculations).

```python
input_dim = len_examples * num_features
X_train = X_train.reshape([-1, input_dim])
X_val = X_val.reshape([-1, input_dim]) if X_val is not None else None
X_test = X_test.reshape([-1, input_dim])
```

Next instantiate the different parts of the model.
***
#### _InducingTensors_
Arguments:
- Z - a numpy array of size $(M(M+1)/2, n_{\mathbf Z}, d)$ if increments is False, $(M(M+1)/2, n_{\mathbf Z}, 2, d)$ if increments is True (see kwargs)
- num_levels - the same number of levels as in the signature kernel

Kwargs:
- increments - if this is set to true, then a different size Z is expected and the inducing tensors are represented for $\mathbf{z} \in \mathbf{Z}$ as $\mathbf{z} = (z_m)_{m=1,\dots,M}$ with
\begin{align}
    z_m = \left(\kappa(x_{m, 1, 2}, \cdot) - \kappa(x_{m, 1, 1}, \cdot)\right) \otimes \left(\kappa(x_{m, 2, 2}, \cdot) - \kappa(x_{m, 2, 1}, \cdot)\right) \otimes \dots \otimes \left(\kappa(x_{m, m, 2}, \cdot) - \kappa(x_{m, m, 1}, \cdot)\right),
\end{align}
while for increments==False they are simply
\begin{align}
    z_m = \kappa(x_{m, 1}, \cdot) \otimes \kappa(x_{m, 2}, \cdot) \otimes \dots \otimes \kappa(x_{m, m}, \cdot),
\end{align}
where $\kappa(\cdot, \cdot)$ is a positive definite kernel over $\mathbb{R}^d$. We find that using the increments version greatly improves their flexibility.
- learn_weights - experimental, if False, then everything as before, if True, then an additional linear combination layer is inserted with the inducing tensors $\mathbf{z}^\prime \in \mathbf{Z}^\prime$ defined as $\mathbf{z}^\prime = (z_m^\prime)_{m=0,\dots,M}$ and
\begin{align}
    \mathbf{z}_m^\prime = \sum_{i=1}^{n_{\mathbf Z}} \alpha_i \mathbf{z}_m \hspace{10pt}\text{for}\hspace{10pt} \mathbf{z}_m \in \mathbf{Z}_m,
\end{align}
where $\mathbf{Z}_m$ is as before.

Suggested setting is to use increments=True and learn_weights=False.

```python
feat = gpsig.inducing_variables.InducingTensors(Z_init, num_levels=num_levels, increments=True)
```

***
#### _SignatureRBF_
The signature kernel for sequences using as state-space embedding the RBF kernel  $\kappa: \mathbb{R}^d \times \mathbb{R}^d \rightarrow \mathbb{R}^d$, i.e. $\kappa(x, x^\prime) = \exp\left(-\frac{1}{2} (x - x^\prime)^\top \Sigma^{-1} (x - x^\prime) \right)$ for $x, x^\prime \in \mathbb{R}^d$

Concretely, for two sequences $\mathbf{x}, \mathbf{x}^\prime \in \mathbb{R}^d$, the sequence kernel derived from signatures can be written as

\begin{align}
    k(\mathbf{x}, \mathbf{x}^\prime) = \sum_{m=0}^M \sum_{\substack{1 \leq i_1 \leq \dots \leq i_m < l_{\mathbf x}} \\ 1 \leq j_1 \leq \dots \leq j_m < l_{\mathbf{x}^\prime}} \sigma_m^2 \cdot c(i_1, \dots, i_m) \cdot c(j_1, \dots j_m) \cdot \prod_{l=1}^m \left(\kappa(x_{i_l+1}, x^\prime_{j_l+1}) + \kappa(x_{i_l}, x^\prime_{j_l}) - \kappa(x_{i_l}, x^\prime_{j_l+1}) - \kappa(x_{i_l+1}, x^\prime_{j_l}) \right),
\end{align}
for some explicitly known constants $c: \mathbb{N}^m \rightarrow \mathbb{R}$ for each $m \in \{1, \dots, M\}$.

Arguments:
- input_dim - the total size of an input sample to the kernel, i.e. $l_{\mathbf X} \cdot d$
- num_features - the state-space dimension of the input sequences, i.e $d$ for sequences in $\mathbb{R}^d_{seq}$
- num_levels - the degree of cut-off for the truncated signatures

Kwargs:
- num_lags - (default 0) the number of lags ($p$) added to each sequence, i.e. $\hat{\mathbf{x}} = (t_i, x_{t_i}, x_{t_i - s_1}, \dots, x_{t_i - s_{p}})$ for $\mathbf{x} = (t_i, x_i)$
- order - (default 1) specifies the approximation order ($D$) to the signature kernel (see http://jmlr.org/papers/v20/16-314.html), denoted by $1 \leq D \leq M$. Relates to the constants $c(i_1, i_2, \dots, i_m)$ in the above formula. An illustrative example is to consider a sequence with two elements $\mathbf{x} = (x_1, x_2)$. Then, for $D=1$
\begin{align}
    \mathbf{S}^{(1)}(\mathbf{x}) = \left(1, (x_2 - x_1), 0, 0, \dots\right),
\end{align}
and for $D=M$
\begin{align}
    \mathbf{S}^{(M)}(\mathbf{x}) = \left(1, (x_2 - x_1), (x_2 - x_1)^{\otimes 2}/2!, (x_2 - x_1)^{\otimes 3}/3!, \dots, (x_2 - x_1)^{\otimes M}/M!, 0, 0, \dots \right),
\end{align}
while for $D \in \{1, 2, \dots, M\}$ in general
\begin{align}
    \mathbf{S}^{(D)}(\mathbf{x}) = \left(1, (x_2 - x_1), (x_2 - x_1)^{\otimes 2}/2!, (x_2 - x_1)^{\otimes 3}/3!, \dots (x_2 - x_1)^{\otimes D}/D!, 0, 0, \dots \right).
\end{align}

- normalization - if False, no normalization, if True each level of the signature kernel is normalized as
\begin{align}
\tilde{k}(\mathbf x, \mathbf{x}^\prime) = \sum_{m=0}^M \sigma_m^2 \frac{k_m(\mathbf{x}, \mathbf{x}^\prime)}{\sqrt{k_m(\mathbf{x}, \mathbf{x})k_m(\mathbf{x}^\prime, \mathbf{x}^\prime)}},
\end{align}
that we find in practice to help with training more efficiently

- variances - initialization of $(\sigma_0, \sigma_1, \dots, \sigma_M) \in \mathbb{R}_+^{M+1}$
- lengthscales: - initialization of the lengthscales $(l_1, l_2, \dots, l_{d})$ of the base kernel

- difference - if True, everything as above, if False, the cumulative sequence is used (i.e. $\tilde{\mathbf{x}} = \left( \sum_{j=1}^i x_i\right)_{i=1,\dots,l_{\mathbf x}})$

- active_dims - if specified, should contain a list of the dimensions in the input that should be sliced out and fed to the kernel. if not specified, defaults to range(input_dim)

Low-rank options:
 - low_rank - boolean indicating whether to use low-rank kernel
 - num_components - number of components used in Nystrom approximation
 - rank_bound - max rank of low-rank factor in signature algs, if None, defaults to num_components.
 - sparsity - controls the sparsity of the random projection matrix used in the low-rank algorithm. Possible values are:
     - 'sqrt' - approximately $O(n \cdot \sqrt{n})$ non-zero entries per row;
     - 'log' - approximately $O(n \cdot \log(n))$ non-zero entries per row;
     - 'lin' - approximately $O(n)$ non-zero entries per row;

```python
k = gpsig.kernels.SignatureRBF(input_dim, num_features, num_levels, lengthscales=l_init)
```

***
The likelihood and the SVGP model use the GPflow syntax, see https://gpflow.readthedocs.io/en/master/. The SVGP from gpsig uses the same syntax as the GPflow SVGP, except that it expects that _feat_ is either an _InducingTensors_ or _InducingSequences_ object derived from the _SignatureInducing_ base class.

```python
if num_classes == 2:
    lik = gp.likelihoods.Bernoulli()
    num_latent = 1
else:
    lik = gp.likelihoods.MultiClass(num_classes)
    num_latent = num_classes

m = gpsig.models.SVGP(X_train, y_train[:, None], kern=k, feat=feat, likelihood=lik, num_latent=num_latent, minibatch_size=50)
```

***
### Train model using stochastic variational inference

Setup the metrics for training, for monitoring we look at the validation nlpp and the validation accuracy (for early stopping the nlpp is taken into account).

```python
acc = lambda m, X, y: accuracy_score(y, np.argmax(m.predict_y(X)[0], axis=1))
nlpp = lambda m, X, y: -np.mean(m.predict_density(X, y[:, None]))

val_acc = lambda m: acc(m, X_val, y_val)
val_nlpp = lambda m: nlpp(m, X_val, y_val)

test_acc = lambda m: acc(m, X_test, y_test)
test_nlpp = lambda m: nlpp(m, X_test, y_test)

opt = gpsig.training.NadamOptimizer # use Nadam optimizer i.e. Adam with Nesterov momentum
```

***
First, as usual, fix the kernel hyperparameters and train for a fixed number of iterations to tighten the ELBO bound.

```python
# train model with kernel hyperparameters fixed
m.kern.set_trainable(False)
hist = gpsig.training.optimize(m, opt(1e-3), max_iter=5000, print_freq=10, save_freq=100, val_scorer=[val_acc, val_nlpp])
```

    -------------------------
      Starting optimization
    -------------------------
    Iteration 100	|	Time: 3.40	|	ELBO: -1121.63	|	Val. 0: 0.5556	|	Val. 1: 1.7456
    Iteration 200	|	Time: 8.86	|	ELBO: -1000.52	|	Val. 0: 0.5833	|	Val. 1: 1.4835
    Iteration 300	|	Time: 10.97	|	ELBO: -781.96	|	Val. 0: 0.5556	|	Val. 1: 1.3053
    Iteration 400	|	Time: 13.03	|	Time: 12.43	|	ELBO: -647.71	|	Val. 0: 0.6944	|	Val. 1: 1.2388
    Iteration 500	|	Time: 15.18	|	ELBO: -786.55	|	Val. 0: 0.6944	|	Val. 1: 1.2604
    Iteration 600	|	Time: 17.25	|	ELBO: -663.91	|	Val. 0: 0.6389	|	Val. 1: 1.2695
    Iteration 700	|	Time: 19.35	|	ELBO: -649.09	|	Val. 0: 0.6667	|	Val. 1: 1.2587
    Iteration 800	|	Time: 21.44	|	ELBO: -623.64	|	Val. 0: 0.6667	|	Val. 1: 1.2524
    Iteration 900	|	Time: 23.52	|	ELBO: -474.87	|	Val. 0: 0.6944	|	Val. 1: 1.2520
    Iteration 1000	|	Time: 25.63	|	ELBO: -427.00	|	Val. 0: 0.6944	|	Val. 1: 1.2701
    Iteration 1100	|	Time: 27.71	|	ELBO: -530.32	|	Val. 0: 0.6944	|	Val. 1: 1.2753
    Iteration 1200	|	Time: 29.87	|	ELBO: -532.95	|	Val. 0: 0.6944	|	Val. 1: 1.2887
    Iteration 1300	|	Time: 32.00	|	ELBO: -490.00	|	Val. 0: 0.7222	|	Val. 1: 1.2933
    Iteration 1400	|	Time: 34.13	|	ELBO: -512.04	|	Val. 0: 0.7222	|	Val. 1: 1.3025
    Iteration 1500	|	Time: 36.18	|	ELBO: -434.69	|	Val. 0: 0.7222	|	Val. 1: 1.3058
    Iteration 1600	|	Time: 38.23	|	ELBO: -405.42	|	Val. 0: 0.7222	|	Val. 1: 1.3071
    Iteration 1700	|	Time: 40.29	|	ELBO: -355.31	|	Val. 0: 0.7222	|	Val. 1: 1.3178
    Iteration 1800	|	Time: 42.35	|	ELBO: -412.34	|	Val. 0: 0.7222	|	Val. 1: 1.3183
    Iteration 1900	|	Time: 44.41	|	ELBO: -415.05	|	Val. 0: 0.7222	|	Val. 1: 1.3239
    Iteration 2000	|	Time: 46.45	|	ELBO: -491.81	|	Val. 0: 0.7222	|	Val. 1: 1.3219
    Iteration 2100	|	Time: 48.52	|	ELBO: -445.51	|	Val. 0: 0.7222	|	Val. 1: 1.3233
    Iteration 2200	|	Time: 50.59	|	ELBO: -597.32	|	Val. 0: 0.7222	|	Val. 1: 1.3265
    Iteration 2300	|	Time: 52.69	|	ELBO: -397.63	|	Val. 0: 0.7222	|	Val. 1: 1.3295
    Iteration 2400	|	Time: 54.79	|	ELBO: -435.97	|	Val. 0: 0.7222	|	Val. 1: 1.3377
    Iteration 2500	|	Time: 56.87	|	ELBO: -528.33	|	Val. 0: 0.7222	|	Val. 1: 1.3361
    Iteration 2600	|	Time: 58.95	|	ELBO: -620.61	|	Val. 0: 0.7222	|	Val. 1: 1.3356
    Iteration 2700	|	Time: 61.05	|	ELBO: -441.77	|	Val. 0: 0.7222	|	Val. 1: 1.3295
    Iteration 2800	|	Time: 63.11	|	ELBO: -422.39	|	Val. 0: 0.7222	|	Val. 1: 1.3213
    Iteration 2900	|	Time: 65.18	|	ELBO: -463.82	|	Val. 0: 0.7222	|	Val. 1: 1.3122
    Iteration 3000	|	Time: 67.28	|	ELBO: -446.47	|	Val. 0: 0.7222	|	Val. 1: 1.3105
    Iteration 3100	|	Time: 69.40	|	ELBO: -412.17	|	Val. 0: 0.7222	|	Val. 1: 1.3153
    Iteration 3200	|	Time: 71.50	|	ELBO: -380.50	|	Val. 0: 0.7222	|	Val. 1: 1.3158
    Iteration 3300	|	Time: 73.61	|	ELBO: -472.70	|	Val. 0: 0.7222	|	Val. 1: 1.3056
    Iteration 3400	|	Time: 75.73	|	ELBO: -437.77	|	Val. 0: 0.7222	|	Val. 1: 1.2948
    Iteration 3500	|	Time: 77.79	|	ELBO: -434.26	|	Val. 0: 0.7222	|	Val. 1: 1.2833
    Iteration 3600	|	Time: 79.87	|	ELBO: -395.73	|	Val. 0: 0.7222	|	Val. 1: 1.2809
    Iteration 3700	|	Time: 81.93	|	ELBO: -404.99	|	Val. 0: 0.7222	|	Val. 1: 1.2853
    Iteration 3800	|	Time: 84.01	|	ELBO: -480.19	|	Val. 0: 0.7222	|	Val. 1: 1.2771
    Iteration 3900	|	Time: 86.07	|	ELBO: -458.17	|	Val. 0: 0.7222	|	Val. 1: 1.2670
    Iteration 4000	|	Time: 88.13	|	ELBO: -426.39	|	Val. 0: 0.7222	|	Val. 1: 1.2699
    Iteration 4100	|	Time: 90.28	|	ELBO: -445.48	|	Val. 0: 0.7222	|	Val. 1: 1.2657
    Iteration 4200	|	Time: 92.31	|	ELBO: -462.27	|	Val. 0: 0.7222	|	Val. 1: 1.2514
    Iteration 4300	|	Time: 94.37	|	ELBO: -408.11	|	Val. 0: 0.7222	|	Val. 1: 1.2586
    Iteration 4400	|	Time: 96.48	|	ELBO: -426.68	|	Val. 0: 0.7222	|	Val. 1: 1.2502
    Iteration 4500	|	Time: 98.54	|	ELBO: -397.91	|	Val. 0: 0.7222	|	Val. 1: 1.2470
    Iteration 4600	|	Time: 100.61	|	ELBO: -428.43	|	Val. 0: 0.7222	|	Val. 1: 1.2443
    Iteration 4700	|	Time: 102.75	|	ELBO: -428.42	|	Val. 0: 0.7222	|	Val. 1: 1.2249
    Iteration 4800	|	Time: 104.88	|	ELBO: -490.34	|	Val. 0: 0.7222	|	Val. 1: 1.2377
    Iteration 4900	|	Time: 106.96	|	ELBO: -430.13	|	Val. 0: 0.7222	|	Val. 1: 1.2169
    Iteration 5000	|	Time: 109.03	|	ELBO: -396.78	|	Val. 0: 0.7222	|	Val. 1: 1.2198

    Optimization session finished...

***
Second, unfix the kernel hyperparameters and train with early stopping with a patience of 5000 iterations.

```python
# unfix kernel hyperparameters and train with early stopping with 5000 iterations patience
m.kern.set_trainable(True)
hist = gpsig.training.optimize(m, opt(1e-2), max_iter=50000, print_freq=10, save_freq=100, history=hist,
                               val_scorer=[val_acc, val_nlpp], save_best_params=True, lower_is_better=True, patience=5000)
m.assign(hist['best']['params'])
```

    ---------------------------
      Continuing optimization
    ---------------------------
    Iteration 5100	|	Time: 112.37	|	ELBO: -478.86	|	Val. 0: 0.7222	|	Val. 1: 1.2471
    Iteration 5200	|	Time: 114.87	|	ELBO: -365.37	|	Val. 0: 0.7500	|	Val. 1: 0.9598
    Iteration 5300	|	Time: 117.11	|	ELBO: -333.20	|	Val. 0: 0.8333	|	Val. 1: 0.7189
    Iteration 5400	|	Time: 119.34	|	ELBO: -294.74	|	Val. 0: 0.8333	|	Val. 1: 0.6909
    Iteration 5500	|	Time: 121.55	|	ELBO: -336.91	|	Val. 0: 0.8333	|	Val. 1: 0.6614
    Iteration 5600	|	Time: 123.78	|	ELBO: -295.32	|	Val. 0: 0.8333	|	Val. 1: 0.6539
    Iteration 5700	|	Time: 126.00	|	ELBO: -293.04	|	Val. 0: 0.8333	|	Val. 1: 0.6189
    Iteration 5800	|	Time: 128.25	|	ELBO: -299.08	|	Val. 0: 0.8333	|	Val. 1: 0.6271
    Iteration 5900	|	Time: 130.52	|	ELBO: -293.35	|	Val. 0: 0.8333	|	Val. 1: 0.6007
    Iteration 6000	|	Time: 132.81	|	ELBO: -284.44	|	Val. 0: 0.8333	|	Val. 1: 0.6013
    Iteration 6100	|	Time: 135.10	|	ELBO: -287.71	|	Val. 0: 0.8333	|	Val. 1: 0.5941
    Iteration 6200	|	Time: 137.37	|	ELBO: -284.68	|	Val. 0: 0.8333	|	Val. 1: 0.5925
    Iteration 6300	|	Time: 139.59	|	ELBO: -278.59	|	Val. 0: 0.8333	|	Val. 1: 0.5917
    Iteration 6400	|	Time: 141.85	|	ELBO: -283.39	|	Val. 0: 0.8333	|	Val. 1: 0.5892
    Iteration 6500	|	Time: 144.10	|	ELBO: -276.77	|	Val. 0: 0.8333	|	Val. 1: 0.5884
    Iteration 6600	|	Time: 146.35	|	ELBO: -285.58	|	Val. 0: 0.8333	|	Val. 1: 0.5821
    Iteration 6700	|	Time: 148.58	|	ELBO: -293.30	|	Val. 0: 0.8333	|	Val. 1: 0.5870
    Iteration 6800	|	Time: 150.75	|	ELBO: -274.88	|	Val. 0: 0.8333	|	Val. 1: 0.5805
    Iteration 6900	|	Time: 152.96	|	ELBO: -278.94	|	Val. 0: 0.8333	|	Val. 1: 0.5768
    Iteration 7000	|	Time: 155.25	|	ELBO: -282.24	|	Val. 0: 0.8333	|	Val. 1: 0.5853
    Iteration 7100	|	Time: 157.52	|	ELBO: -277.44	|	Val. 0: 0.8333	|	Val. 1: 0.5803
    Iteration 7200	|	Time: 159.78	|	ELBO: -270.71	|	Val. 0: 0.8333	|	Val. 1: 0.5768
    Iteration 7300	|	Time: 162.02	|	ELBO: -288.22	|	Val. 0: 0.8333	|	Val. 1: 0.5762
    Iteration 7400	|	Time: 164.27	|	ELBO: -294.57	|	Val. 0: 0.8333	|	Val. 1: 0.5827
    Iteration 7500	|	Time: 166.51	|	ELBO: -277.12	|	Val. 0: 0.8333	|	Val. 1: 0.5838
    Iteration 7600	|	Time: 168.74	|	ELBO: -277.81	|	Val. 0: 0.8333	|	Val. 1: 0.5816
    Iteration 7700	|	Time: 171.01	|	ELBO: -278.01	|	Val. 0: 0.8333	|	Val. 1: 0.5838
    Iteration 7800	|	Time: 173.25	|	ELBO: -278.70	|	Val. 0: 0.8333	|	Val. 1: 0.5743
    Iteration 7900	|	Time: 175.48	|	ELBO: -281.60	|	Val. 0: 0.8333	|	Val. 1: 0.5707
    Iteration 8000	|	Time: 177.73	|	ELBO: -277.16	|	Val. 0: 0.8333	|	Val. 1: 0.5755
    Iteration 8100	|	Time: 179.98	|	ELBO: -276.42	|	Val. 0: 0.8333	|	Val. 1: 0.5703
    Iteration 8200	|	Time: 182.24	|	ELBO: -278.54	|	Val. 0: 0.8333	|	Val. 1: 0.5757
    Iteration 8300	|	Time: 184.46	|	ELBO: -277.54	|	Val. 0: 0.8333	|	Val. 1: 0.5783
    Iteration 8400	|	Time: 186.67	|	ELBO: -269.46	|	Val. 0: 0.8333	|	Val. 1: 0.5710
    Iteration 8500	|	Time: 188.89	|	ELBO: -280.54	|	Val. 0: 0.8333	|	Val. 1: 0.5674
    Iteration 8600	|	Time: 191.12	|	ELBO: -281.97	|	Val. 0: 0.8333	|	Val. 1: 0.5692
    Iteration 8700	|	Time: 193.28	|	ELBO: -276.61	|	Val. 0: 0.8333	|	Val. 1: 0.5720
    Iteration 8800	|	Time: 195.51	|	ELBO: -275.33	|	Val. 0: 0.8333	|	Val. 1: 0.5719
    Iteration 8900	|	Time: 197.75	|	ELBO: -268.73	|	Val. 0: 0.8333	|	Val. 1: 0.5713
    Iteration 9000	|	Time: 200.00	|	ELBO: -271.03	|	Val. 0: 0.8333	|	Val. 1: 0.5762
    Iteration 9100	|	Time: 202.20	|	ELBO: -288.88	|	Val. 0: 0.8333	|	Val. 1: 0.5591
    Iteration 9200	|	Time: 204.37	|	ELBO: -273.12	|	Val. 0: 0.8333	|	Val. 1: 0.5749
    Iteration 9300	|	Time: 206.61	|	ELBO: -280.12	|	Val. 0: 0.8333	|	Val. 1: 0.5601
    Iteration 9400	|	Time: 208.89	|	ELBO: -271.76	|	Val. 0: 0.8333	|	Val. 1: 0.5694
    Iteration 9500	|	Time: 211.18	|	ELBO: -270.29	|	Val. 0: 0.8333	|	Val. 1: 0.5719
    Iteration 9600	|	Time: 213.39	|	ELBO: -262.86	|	Val. 0: 0.8333	|	Val. 1: 0.5723
    Iteration 9700	|	Time: 215.63	|	ELBO: -271.01	|	Val. 0: 0.8333	|	Val. 1: 0.5811
    Iteration 9800	|	Time: 217.87	|	ELBO: -273.79	|	Val. 0: 0.8333	|	Val. 1: 0.5731
    Iteration 9900	|	Time: 220.12	|	ELBO: -273.97	|	Val. 0: 0.8333	|	Val. 1: 0.5719
    Iteration 10000	|	Time: 222.35	|	ELBO: -273.58	|	Val. 0: 0.8333	|	Val. 1: 0.5719
    Iteration 10100	|	Time: 224.57	|	ELBO: -275.36	|	Val. 0: 0.8333	|	Val. 1: 0.5810
    Iteration 10200	|	Time: 226.82	|	ELBO: -276.69	|	Val. 0: 0.8333	|	Val. 1: 0.5614
    Iteration 10300	|	Time: 229.04	|	ELBO: -276.71	|	Val. 0: 0.8333	|	Val. 1: 0.5799
    Iteration 10400	|	Time: 231.27	|	ELBO: -280.37	|	Val. 0: 0.8333	|	Val. 1: 0.5718
    Iteration 10500	|	Time: 233.46	|	ELBO: -268.30	|	Val. 0: 0.8333	|	Val. 1: 0.5777
    Iteration 10600	|	Time: 235.73	|	ELBO: -265.90	|	Val. 0: 0.8333	|	Val. 1: 0.5696
    Iteration 10700	|	Time: 238.03	|	ELBO: -265.95	|	Val. 0: 0.8333	|	Val. 1: 0.5756
    Iteration 10800	|	Time: 240.29	|	ELBO: -273.70	|	Val. 0: 0.8333	|	Val. 1: 0.5744
    Iteration 10900	|	Time: 242.57	|	ELBO: -273.79	|	Val. 0: 0.8333	|	Val. 1: 0.5696
    Iteration 11000	|	Time: 244.81	|	ELBO: -270.69	|	Val. 0: 0.8333	|	Val. 1: 0.5722
    Iteration 11100	|	Time: 247.07	|	ELBO: -278.47	|	Val. 0: 0.8333	|	Val. 1: 0.5787
    Iteration 11200	|	Time: 249.27	|	ELBO: -281.86	|	Val. 0: 0.8333	|	Val. 1: 0.5697
    Iteration 11300	|	Time: 251.49	|	ELBO: -266.02	|	Val. 0: 0.8333	|	Val. 1: 0.5781
    Iteration 11400	|	Time: 253.76	|	ELBO: -286.40	|	Val. 0: 0.8333	|	Val. 1: 0.5662
    Iteration 11500	|	Time: 256.01	|	ELBO: -280.06	|	Val. 0: 0.8333	|	Val. 1: 0.5668
    Iteration 11600	|	Time: 258.24	|	ELBO: -273.66	|	Val. 0: 0.8333	|	Val. 1: 0.5755
    Iteration 11700	|	Time: 260.46	|	ELBO: -286.40	|	Val. 0: 0.8333	|	Val. 1: 0.5760
    Iteration 11800	|	Time: 262.72	|	ELBO: -274.43	|	Val. 0: 0.8333	|	Val. 1: 0.5617
    Iteration 11900	|	Time: 264.97	|	ELBO: -276.54	|	Val. 0: 0.8333	|	Val. 1: 0.5644
    Iteration 12000	|	Time: 267.21	|	ELBO: -274.56	|	Val. 0: 0.8333	|	Val. 1: 0.5679
    Iteration 12100	|	Time: 269.45	|	ELBO: -271.60	|	Val. 0: 0.8333	|	Val. 1: 0.5765
    Iteration 12200	|	Time: 271.67	|	ELBO: -272.05	|	Val. 0: 0.8333	|	Val. 1: 0.5732
    Iteration 12300	|	Time: 273.91	|	ELBO: -272.93	|	Val. 0: 0.8333	|	Val. 1: 0.5777
    Iteration 12400	|	Time: 276.24	|	ELBO: -272.24	|	Val. 0: 0.8333	|	Val. 1: 0.5759
    Iteration 12500	|	Time: 278.47	|	ELBO: -282.31	|	Val. 0: 0.8333	|	Val. 1: 0.5663
    Iteration 12600	|	Time: 280.76	|	ELBO: -277.25	|	Val. 0: 0.8333	|	Val. 1: 0.5691
    Iteration 12700	|	Time: 283.03	|	ELBO: -280.07	|	Val. 0: 0.8333	|	Val. 1: 0.5615
    Iteration 12800	|	Time: 285.28	|	ELBO: -288.13	|	Val. 0: 0.8333	|	Val. 1: 0.5733
    Iteration 12900	|	Time: 287.53	|	ELBO: -277.51	|	Val. 0: 0.8333	|	Val. 1: 0.5714
    Iteration 13000	|	Time: 289.74	|	ELBO: -271.82	|	Val. 0: 0.8333	|	Val. 1: 0.5748
    Iteration 13100	|	Time: 291.99	|	ELBO: -272.73	|	Val. 0: 0.8333	|	Val. 1: 0.5683
    Iteration 13200	|	Time: 294.20	|	ELBO: -273.97	|	Val. 0: 0.8333	|	Val. 1: 0.5709
    Iteration 13300	|	Time: 296.44	|	ELBO: -271.48	|	Val. 0: 0.8333	|	Val. 1: 0.5738
    Iteration 13400	|	Time: 298.70	|	ELBO: -265.79	|	Val. 0: 0.8333	|	Val. 1: 0.5752
    Iteration 13500	|	Time: 300.94	|	ELBO: -279.60	|	Val. 0: 0.8333	|	Val. 1: 0.5719
    Iteration 13600	|	Time: 303.15	|	ELBO: -280.40	|	Val. 0: 0.8333	|	Val. 1: 0.5706
    Iteration 13700	|	Time: 305.36	|	ELBO: -278.65	|	Val. 0: 0.8333	|	Val. 1: 0.5714
    Iteration 13800	|	Time: 307.60	|	ELBO: -265.49	|	Val. 0: 0.8333	|	Val. 1: 0.5687
    Iteration 13900	|	Time: 309.85	|	ELBO: -272.96	|	Val. 0: 0.8333	|	Val. 1: 0.5697
    Iteration 14000	|	Time: 312.07	|	ELBO: -264.32	|	Val. 0: 0.8333	|	Val. 1: 0.5799
    Iteration 14100	|	Time: 314.30	|	ELBO: -284.11	|	Val. 0: 0.8333	|	Val. 1: 0.5655
    Iteration 14200	|	Time: 316.54	|	ELBO: -293.33	|	Val. 0: 0.8333	|	Val. 1: 0.5705
    No improvement over validation loss has occured for 5000 iterations: stopping early...

    Optimization session finished...

***
Third, fix the kernel again and re-merge the validation set into the training data. Train for a fixed number of iterations to assimilate the remaining information into the variational approximation.

```python
X_train, y_train = np.concatenate((X_train, X_val), axis=0), np.concatenate((y_train, y_val), axis=0)
num_train = X_train.shape[0]
m.X, m.Y = X_train, y_train
m.num_data = num_train

m.kern.set_trainable(False)
hist = gpsig.training.optimize(m, opt(1e-3), max_iter=5000, print_freq=10, save_freq=100, history=hist)
```

    ---------------------------
      Continuing optimization
    ---------------------------
    Iteration 14300	|	Time: 319.47	|	ELBO: -317.29
    Iteration 14400	|	Time: 321.53	|	ELBO: -276.40
    Iteration 14500	|	Time: 323.54	|	ELBO: -285.63
    Iteration 14600	|	Time: 325.55	|	ELBO: -312.04
    Iteration 14700	|	Time: 327.58	|	ELBO: -332.78
    Iteration 14800	|	Time: 329.61	|	ELBO: -305.83
    Iteration 14900	|	Time: 331.60	|	ELBO: -293.91
    Iteration 15000	|	Time: 333.58	|	ELBO: -331.48
    Iteration 15100	|	Time: 335.61	|	ELBO: -298.33
    Iteration 15200	|	Time: 337.64	|	ELBO: -279.17
    Iteration 15300	|	Time: 339.63	|	ELBO: -283.91
    Iteration 15400	|	Time: 341.63	|	ELBO: -272.08
    Iteration 15500	|	Time: 343.66	|	ELBO: -277.04
    Iteration 15600	|	Time: 345.73	|	ELBO: -314.53
    Iteration 15700	|	Time: 347.78	|	ELBO: -305.81
    Iteration 15800	|	Time: 349.81	|	ELBO: -307.69
    Iteration 15900	|	Time: 351.87	|	ELBO: -275.54
    Iteration 16000	|	Time: 353.91	|	ELBO: -291.97
    Iteration 16100	|	Time: 355.93	|	ELBO: -320.32
    Iteration 16200	|	Time: 357.93	|	ELBO: -298.66
    Iteration 16300	|	Time: 359.94	|	ELBO: -316.82
    Iteration 16400	|	Time: 361.97	|	ELBO: -277.22
    Iteration 16500	|	Time: 364.03	|	ELBO: -288.52
    Iteration 16600	|	Time: 366.03	|	ELBO: -313.09
    Iteration 16700	|	Time: 368.04	|	ELBO: -321.67
    Iteration 16800	|	Time: 370.06	|	ELBO: -291.88
    Iteration 16900	|	Time: 372.14	|	ELBO: -270.59
    Iteration 17000	|	Time: 374.19	|	ELBO: -304.62
    Iteration 17100	|	Time: 376.26	|	ELBO: -284.59
    Iteration 17200	|	Time: 378.35	|	ELBO: -277.04
    Iteration 17300	|	Time: 380.38	|	ELBO: -282.94
    Iteration 17400	|	Time: 382.46	|	ELBO: -298.18
    Iteration 17500	|	Time: 384.55	|	ELBO: -276.15
    Iteration 17600	|	Time: 386.58	|	ELBO: -284.30
    Iteration 17700	|	Time: 388.63	|	ELBO: -321.97
    Iteration 17800	|	Time: 390.65	|	ELBO: -277.34
    Iteration 17900	|	Time: 392.70	|	ELBO: -274.85
    Iteration 18000	|	Time: 394.74	|	ELBO: -289.39
    Iteration 18100	|	Time: 396.79	|	ELBO: -319.76
    Iteration 18200	|	Time: 398.88	|	ELBO: -287.17
    Iteration 18300	|	Time: 401.00	|	ELBO: -319.40
    Iteration 18400	|	Time: 403.07	|	ELBO: -302.44
    Iteration 18500	|	Time: 405.15	|	ELBO: -286.23
    Iteration 18600	|	Time: 407.20	|	ELBO: -319.64
    Iteration 18700	|	Time: 409.21	|	ELBO: -295.01
    Iteration 18800	|	Time: 411.20	|	ELBO: -294.33
    Iteration 18900	|	Time: 413.21	|	ELBO: -287.34
    Iteration 19000	|	Time: 415.19	|	ELBO: -295.15
    Iteration 19100	|	Time: 417.18	|	ELBO: -285.86
    Iteration 19200	|	Time: 419.21	|	ELBO: -278.66

    Optimization session finished...

***
### Evaluate model performance

```python
print('Test nlpp.: {:.03f}'.format(test_nlpp(m)))
print('Test acc.: {:.03f}'.format(test_acc(m)))
print('Test classification report:')
print(classification_report(y_test, np.argmax(m.predict_y(X_test)[0], axis=1)))
```

    Test nlpp.: 0.303
    Test acc.: 0.917
    Test classification report:
                  precision    recall  f1-score   support

               0       1.00      0.92      0.96        12
               1       0.90      0.75      0.82        12
               2       0.71      0.83      0.77        12
               3       0.80      1.00      0.89        12
               4       0.78      0.58      0.67        12
               5       1.00      1.00      1.00        12
               6       0.92      1.00      0.96        12
               7       0.92      1.00      0.96        12
               8       1.00      1.00      1.00        12
               9       0.92      0.92      0.92        12
              10       1.00      1.00      1.00        12
              11       1.00      0.92      0.96        12
              12       0.92      0.92      0.92        12
              13       0.92      0.92      0.92        12
              14       1.00      1.00      1.00        12

        accuracy                           0.92       180
       macro avg       0.92      0.92      0.92       180
    weighted avg       0.92      0.92      0.92       180

```python
m.as_pandas_table()
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>class</th>
      <th>prior</th>
      <th>transform</th>
      <th>trainable</th>
      <th>shape</th>
      <th>fixed_shape</th>
      <th>value</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>SVGP/feature/Z</th>
      <td>Parameter</td>
      <td>None</td>
      <td>(none)</td>
      <td>True</td>
      <td>(10, 200, 2, 3)</td>
      <td>True</td>
      <td>[[[[-1.36666351 -0.09387724  0.05260296], [ 0....</td>
    </tr>
    <tr>
      <th>SVGP/kern/lengthscales</th>
      <td>Parameter</td>
      <td>None</td>
      <td>+ve</td>
      <td>False</td>
      <td>(3,)</td>
      <td>True</td>
      <td>[5.826849902048951, 4.966218487190286, 4.97390...</td>
    </tr>
    <tr>
      <th>SVGP/kern/sigma</th>
      <td>Parameter</td>
      <td>None</td>
      <td>+ve</td>
      <td>False</td>
      <td>()</td>
      <td>True</td>
      <td>17.563948650680643</td>
    </tr>
    <tr>
      <th>SVGP/kern/variances</th>
      <td>Parameter</td>
      <td>None</td>
      <td>+ve</td>
      <td>False</td>
      <td>(5,)</td>
      <td>True</td>
      <td>[0.005267731259876558, 0.038240026377996665, 0...</td>
    </tr>
    <tr>
      <th>SVGP/likelihood/invlink/epsilon</th>
      <td>Parameter</td>
      <td>Beta(0.2,5.0)</td>
      <td>[0.0, 1.0]</td>
      <td>False</td>
      <td>()</td>
      <td>True</td>
      <td>0.0009999999999999996</td>
    </tr>
    <tr>
      <th>SVGP/q_mu</th>
      <td>Parameter</td>
      <td>None</td>
      <td>(none)</td>
      <td>True</td>
      <td>(200, 15)</td>
      <td>True</td>
      <td>[[-0.19498280339215954, 0.3107899346033142, 0....</td>
    </tr>
    <tr>
      <th>SVGP/q_sqrt</th>
      <td>Parameter</td>
      <td>None</td>
      <td>LoTri->vec</td>
      <td>True</td>
      <td>(15, 200, 200)</td>
      <td>True</td>
      <td>[[[0.9616282232164973, 0.0, 0.0, 0.0, 0.0, 0.0...</td>
    </tr>
  </tbody>
</table>
</div>

```python
print('The variances of each level:')
plt.bar(range(num_levels+1), m.kern.sigma.value * m.kern.variances.value)
```

```python
print('The lengthsacles of each coordinate:')
plt.bar(range(num_features), m.kern.lengthscales.value)
```

### Inspect training curve

```python
time = [y['time'] for x, y in hist.items() if str(x).isnumeric()]
elbo = [y['elbo'] for x, y in hist.items() if str(x).isnumeric()]
val_acc = [y['val'][0] for x, y in hist.items() if str(x).isnumeric() and 'val' in y]
val_nlpp = [y['val'][1] for x, y in hist.items() if str(x).isnumeric() and 'val' in y]
```

```python
fig, axes = plt.subplots(1, 3, figsize=(20, 5))

axes[0].plot(time, elbo)
axes[0].set_title('ELBO over time')
axes[1].plot(time[:len(val_nlpp)], val_acc, 'tab:orange')
axes[1].set_title('Validation acc. over time')
axes[2].plot(time[:len(val_nlpp)], val_nlpp, 'tab:red')
axes[2].set_title('Validation nlpp. over time')
```