```python
# ============================================================
# Notebook setup
# ============================================================

%load_ext autoreload
%autoreload 2

# Control figure size
interactive_figures = True
if interactive_figures:
    # Normal behavior
    %matplotlib widget
    figsize=(9, 3)
else:
    # PDF export behavior
    figsize=(14, 3)

from util import nn
import numpy as np
from matplotlib import pyplot as plt
import pandas as pd

# Load HPC data
data_folder = '/app/data'
hpc = pd.read_csv(data_folder+ '/hpc.csv', parse_dates=['timestamp'])

# Identify input columns
hpc_in = hpc.columns[1:-1]

# Standardization
tr_end, val_end = 3000, 4500
hpcs = hpc.copy()
tmp = hpcs.iloc[:tr_end]
hpcs[hpc_in] = (hpcs[hpc_in] - tmp[hpc_in].mean()) / tmp[hpc_in].std()

# Training, validation, and test set
trdata = hpcs.iloc[:tr_end]
valdata = hpcs.iloc[tr_end:val_end]
tsdata = hpcs.iloc[val_end:]

# Anomaly labels
hpc_labels = pd.Series(index=hpc.index, data=(hpc['anomaly'] != 0), dtype=int)

# Cost model
c_alarm, c_missed, tolerance = 1, 5, 12
cmodel =  nn.HPCMetrics(c_alarm, c_missed, tolerance)
```

    The autoreload extension is already loaded. To reload it, use:
      %reload_ext autoreload

# Density Estimation with Neural Models

## Density Estimation vs Autoencoders

**Anomaly detection can be formulated as density estimation**

* This is probably _the cleanest formulation_ for the problem
* ...And usually leads to good results

**KDE as an estimation technique**

* ...Works reasonably well for low-dimensional data
* ...Becomes _slower and more data hungry_ for higher-dimensional data

**Autoencoders overcome some of these limitations**

* They are _faster and less data hungry_ for high-dimensional data
* They can provide _additional insight_ in the anomalies
* ...But they tend to be _worse_ than D.E. in terms of _pure detection power_

**Let's try to understand why this may be the case...**

## Density Estimation vs Autoencoders

**Anomaly Detection based on D.E. checks whether:**

$$
- \log f({\bf x}, \lambda) \geq \theta
$$

* Where $\bf x$ is the input vector, $f$ the density estimator, and $\lambda$ its parameter vector
* $\theta$ is the anomaly detection threshold

**Anomaly Detection based on autoencoders usually relies on:**

$$
\|g({\bf x}, \lambda) - {\bf x}\|_2^2 \geq \theta^\prime
$$

* Where $g$ is the autoencoder, with parameter vector $\lambda$
* $\theta^\prime$ is again a suitably-chosen detection threshold

## Density Estimation vs Autoencoders

**The detection condition for autoencoders admits a probabilistic interpretation**

Like we did for linear regression, we can rewrite:

$$
\|g({\bf x}, \lambda) - {\bf x}\|_2^2 \longrightarrow
\sum_{j=1}^m (g_j({\bf x}, \lambda) - x_j)^2 \longrightarrow
\log \prod_{j=1}^m \exp\left((g_j({\bf x}, \lambda) - x_j)^2\right)
$$

From which, with an _affine transformation_, for some fixed $\sigma$ we get:

$$
\log \frac{1}{\sigma\sqrt{2\pi}} + \frac{1}{\sigma^2} \log \prod_{j=1}^m \exp \left((g_j({\bf x}, \lambda) - x_j)^2\right) \quad \longrightarrow\\
\longrightarrow\quad \log \prod_{j=1}^m \frac{1}{\sigma\sqrt{2\pi}} \exp \left(\left(\frac{g_j({\bf x}, \lambda) - x_j}{\sigma}\right)^2\right)
$$

* The transformation preserves all the optimal points

## Density Estimation vs Autoencoders

**Therefore, optimizing the MSE is equivalent to optimizing**

$$
-\log \prod_{j=1}^m \varphi (x_j \mid g_j({\bf x}, \lambda), \sigma)
$$

* I.e. the log likelihood (estimated conditional probability of the data)...
* ...Assuming that the prediction for each $x_i$ is _independent and normally distributed_
* ...with _mean_ equal to the predictions $g_j({\bf x}, \lambda)$ and fixed _standard deviation_ $\sigma$

**This is similar to what we observed for Linear Regression**

* In LR, we assume normality, independence and fixed variance _on the samples_
* Here, we do it _also on the features_

## Density Estimation vs Autoencoders

**The bottomline**

* Even with autoencoders, at training time we _solve a density estimation problem_
* ...But we do it _with some limiting assumptions_

> **This is why D.E.-based anomaly detection _tends to work better_**

**So we have**

* Either a density estimator with issues on high-dimensional data (KDE)
* ...Or a worse D.E. with good support for high-dimensional data  (autoencoders)

> **Can we get the best of both worlds?**

## Flow Models

**Ideally, we wish _a neural approach for density estimation_**

There are only a handful of approaches, often referred to as _flow models_:

* [Normalizing Flows](https://arxiv.org/abs/1505.05770)
* [Real Non-Volume Preserving transformations (Real NVP)](https://arxiv.org/abs/1605.08803)
* [Generative Flow with 1x1 convolutions (Glow)](https://arxiv.org/abs/1807.03039)

**These are all (fairly) advanced and recent approaches**

* Main idea: transforming _a simple (and known) probability distribution_...
* ..._Into a complex (and unknown) distribution_ that matches that of the available data

As many ML models, they are trained for maximum likelihood

* I.e. to maximize the estimated probability of the available data

## Flow Models

**All flow models rely on the _change of variable formula_**

* Let $x$ be a random variable representing the source of our data
* Let $p_x(x)$ be its (unknown) density function
* Let $z$ be a random _latent variable_ with known distribution $p_z$
* Let $f$ be a _bijective_ (i.e. invertible) transformation

Then, the change of variable formula states that:

$$
p_x(x) = p_z(f(x)) \left| \det \left(\frac{\partial f(x)}{\partial x^T} \right)\right|
$$

* Where $\det$ is the determinant and $\partial f / \partial x^T$ is the Jacobian of $f$

**The formula links the two distributions via _the flow model $f$_**

## Flow  Models

**Let's consider how we can use the formula**

$$
p_x(x) = p_z(f(x)) \left| \det \left(\frac{\partial f(x)}{\partial x^T} \right)\right|
$$

* Given _an example $x$_ (e.g. from our dataset)
* We _compute the mapping $f(x)$_, i.e. the corresponding value for the latent variable $z$
* ...Plus the _determinant of the Jacobian_ $\partial f / \partial x^T$ in $x$
* Then we can use the formula to compute the _probability of the example_

**The challenge is defining the transformation $f$ (i.e. the mapping)**

* It must be _invertible_ (for the formula to hold)
* It must be _non-linear_ (to handle any distribution)
* It should allow for an _easy computation of the determinant_

## Real NVP

**We will use [Real Non-Volume Preserving transformations](https://arxiv.org/abs/1605.08803) as an example**

Real NVPs are _a type of neural network_

* _Input:_ a vector $x$ representing an example
* _Output:_ a vector $z$ of values for the latent variable
* _Key property:_ $z$ should have a chosen probability distribution
* ...Typically: standard Normal distribution for each $z_i$:

$$
z \sim \mathcal{N}({\bf 0}, I)
$$

In other words

* $z$ follows a multivariate distribution
* ...But the covariance matrix is diagonal, i.e. each component is independent

## Real NVP

**A Real NVP architecture consists of a stack of _affine coupling layers_**

Each layer treats its input $x$ as split into two components, i.e. $x = (x^1, x^2)$

* One component is _passed forward_ as it is
* The second is processed via an _affine transformation_

$$\begin{align}
y^1 &= x^1 \\
y^2 &= e^{s(x^1)} \odot x^2 + t(x^1)
\end{align}$$

**The affine transformation is parameterized with two functions:**

* $x^2$ is _scaled_ using $e^{s(x^1)}$, $x^2$ is _translated_ using $t(x^1)$
* $\odot$ is the element-wise product (Hadamard product)

Since we have functions rather than fixed vectors, _the transformation is non-linear_

## Real NVP - Affine Coupling Layers

**Visually, each layer has the following _compute graph:_**

* We are using part of the input (i.e. $x^1$)...
* ...To transform the remaining part (i.e. $x^2$)

**Both $s$ and $t$ are usually implemented as Multilayer Perceptrons**

* I.e. pretend there are a few fully connected layers when you see $s$ and $t$

## Real NVP - Affine Coupling Layers

**Each affine coupling layer is _easy to invert_**

Since part of the input (i.e. $x^1$) has been passed forward unchanged, we have that:

$$\begin{align}
x^1 &= y^1 \\
x^2 &= (y^2 - t(y^1)) \oslash e^{s(y^1)}
\end{align}$$

* $\oslash$ is the element-wise division

## Real NVP - Affine Coupling Layers

**The _determinant_ of each layer is easy to compute**

The Jacobian of the transformation is:

$$
\frac{\partial y}{\partial x^T} = \left(\begin{array}{cc}
I & 0 \\
\frac{\partial t(x^1)}{\partial x^T} & \text{diag}(e^{s(x^1)})
\end{array}\right)
$$

The most (only, actually) important thing is that _the matrix is triangular:_

* ...Hence, its determinant is the product of the terms on the main diagonal:

$$
\det\left(\frac{\partial y}{\partial x^T}\right) = \prod_{j \in I_{x_1}} e^{s(x^1_i)} = \exp \left( \sum_{j \in I_{x_1}} s(x^1_i) \right)
$$

## Real NVP - Considerations

**Overall, we have a transformation that:**

* ...Is _non-linear_, and can be made arbitrarily _deep_
* ...Is _Invertible_ (so as to allow application of the change of variable formula)
* ...Is well suited for _determinant computation_

**Depth and non-linearity are very important:**

* The whole approach works _only if_ we can construct a mapping between $x$ and $z$...
* ...I.e. if we can transform one probability distribution into the other

A poor mapping will lead to poor estimates

## Real NVP - Considerations

**At training time we maximize the log likelihood...**

...Hence we care about _log probabilities_:

$$
\log p_x(x) = \log p_z(f(x)) +\log\, \left| \det \left(\frac{\partial f(x)}{\partial x^T} \right)\right|
$$

* If we choose a Normal distribution for $z$, the log _cancels all exponentials in the formula_
* I.e. the one in the Normal PDF and the one in the determinant computation

**In general, we want to make sure that all variables are transformed**

* We need to be careful to define the $x^1, x^2$ components on different layers...
* ...So that no variable is passed forward unchanged along the whole network

A simple approach: _alternate the roles_ (i.e. swap the role of $x^1, x^2$ at every layer)

## Real NVP as Generative Models

**Since Real NVPs are invertible, they can be used as _generative models_**

Formally, they can _sample_ from the distribution they have learned

* We just need to sample from $p_z$, i.e. on the latent space
  - ...And this is easy since the distribution is simple an known
* Then we go through the whole architecture _backwards_
  - ...Using the inverted version of the affine coupling layers

**In fact, generating data is often their _primary purpose_**

They can (or could) be used for:

* Super resolution
* Procedural content generation
* Data augmentation (relevant in an industrial context)

Recent versions allow for [data generation with controlled attributes](https://openai.com/blog/glow/)

# Implementing Real NVPs

## Implementing Real NVPs

**We will now see how to implement Real NVPs**

The basis from our code comes from the [official keras documentation](https://keras.io/examples/generative/real_nvp/)

* It will rely partially on low-level APIs of keras

We start by importing several packages:

```python
import tensorflow as tf
import tensorflow_probability as tfp
from tensorflow.keras.layers import Dense
from tensorflow.keras.callbacks import EarlyStopping
from tensorflow.keras.regularizers import l2
from sklearn.datasets import make_moons
```

* `tensorflow_probability` is a tensorflow extension for probabilistic computations
* ...And allows for easy manipulation of probability distributions

## Affine Coupling Layer

**Then we define a function to build each _affine coupling layer_:**

```python
def coupling(input_shape, nunits=64, nhidden=2, reg=0.01):
    assert(nhidden >= 0)
    x = keras.layers.Input(shape=input_shape)
    # Build the layers for the t transformation (translation)
    t = x
    for i in range(nhidden):
        t = Dense(nunits, activation="relu", kernel_regularizer=l2(reg))(t)
    t = Dense(input_shape, activation="linear", kernel_regularizer=l2(reg))(t)
    # Build the layers for the s transformation (scale)
    s = x
    for i in range(nhidden):
        s = Dense(nunits, activation="relu", kernel_regularizer=l2(reg))(s)
    s = Dense(input_shape, activation="tanh", kernel_regularizer=l2(reg))(s)
    # Return the layers, wrapped in a keras Model object
    return keras.Model(inputs=x, outputs=[s, t])
```

## Affine Coupling Layer

**This part of the code builds _the translation (i.e. $t$) function:_**

```python
def coupling(input_shape, nunits=64, nhidden=2, reg=0.01):
    ...
    x = keras.layers.Input(shape=input_shape)
    t = x
    for i in range(nhidden):
        t = Dense(nunits, activation="relu", kernel_regularizer=l2(reg))(t)
    t = Dense(input_shape, activation="linear", kernel_regularizer=l2(reg))(t)
    ...
```

* It's _just a Multi-Layer Perceptron_ built using the functional API
* The output represents an offset, hence the "linear" activation function in the last layer

## Affine Coupling Layer

**This part of the code builds _the translation (i.e. $t$) function:_**

```python
def coupling(input_shape, nunits=64, nhidden=2, reg=0.01):
    ...
    x = keras.layers.Input(shape=input_shape)
    t = x
    for i in range(nhidden):
        t = Dense(nunits, activation="relu", kernel_regularizer=l2(reg))(t)
    t = Dense(input_shape, activation="linear", kernel_regularizer=l2(reg))(t)
    ...
```

* The output and input have the same shape, but $x^1$ and $x^2$ may have _different size_
* This will be resolved by _masking_ some of the output of the affine layer
* ...The masked portions _will have no effect_, with effectively the same result
* The main drawback is higher memory consumption (and computational cost)

## Affine Coupling Layer

**This part of the code builds _the scaling (i.e. $s$) function:_**

```python
def coupling(input_shape, nunits=64, nhidden=2, reg=0.01):
    ...
    x = keras.layers.Input(shape=input_shape)
    ...
    s = x
    for i in range(nhidden):
        s = Dense(nunits, activation="relu", kernel_regularizer=l2(reg))(s)
    s = Dense(input_shape, activation="tanh", kernel_regularizer=l2(reg))(s)
    ...
```

* Another MLP, with a bipolar sigmoid ("tanh") activation function in the output layer
* Using "tanh" limits the amount of scaling per affine coupling layer
* ...Which in turn makes training more numerically stable
* For the same reason, we use L2 regularizers on the MPL weights

## RNVP Model

**Then, we define a Real NVP architecture by subclassing keras.model**

```python
class RealNVP(keras.Model):
    def __init__(self, input_shape, num_coupling, units_coupling=32, depth_coupling=0,
            reg_coupling=0.01): ...
    @property
    def metrics(self): ...
    def call(self, x, training=True): ...
    def log_loss(self, x): ...
    def score_samples(self, x): ...
    def train_step(self, data): ...
    def test_step(self, data): ...
```

* We will now discuss _the most important methods_
* Sometimes with a few  simplifications (for sake of clarity)

## RNVP Model

**The `__init__` method (constructor) initializes the internal fields**

```python
    def __init__(self, input_shape, num_coupling, units_coupling=32, depth_coupling=0,
            reg_coupling=0.01):
        super(RealNVP, self).__init__()
        self.distribution = tfp.distributions.MultivariateNormalDiag(
            loc=np.zeros(input_shape, dtype=np.float32),
            scale_diag=np.ones(input_shape, dtype=np.float32)
        )
        half_n = int(np.ceil(input_shape/2))
        m1 = ([0, 1] * half_n)[:input_shape]
        m2 = ([1, 0] * half_n)[:input_shape]
        self.masks = np.array([m1, m2] * (num_coupling // 2), dtype=np.float32)
        self.loss_tracker = keras.metrics.Mean(name="loss")
        self.layers_list = [coupling(input_shape, units_coupling, depth_coupling, reg_coupling)
                            for i in range(num_coupling)]
```

## RNVP Model

**The `__init__` method (constructor) initializes the internal fields**

```python
    def __init__(self, input_shape, num_coupling, units_coupling=32, depth_coupling=0,
            reg_coupling=0.01):
        ...
        self.distribution = tfp.distributions.MultivariateNormalDiag(
            loc=np.zeros(input_shape, dtype=np.float32),
            scale_diag=np.ones(input_shape, dtype=np.float32)
        )
        ...
```

Here we build a `tfp` object to handle the known distribution

* As it is customary, we chosen a Multivariate Normal distribution
* ...With independent components, zero mean, and unary standard deviation

## RNVP Model

**The `__init__` method (constructor) initializes the internal fields**

```python
    def __init__(self, input_shape, num_coupling, units_coupling=32, depth_coupling=0,
            reg_coupling=0.01):
        ...
        half_n = int(np.ceil(input_shape/2))
        m1 = ([0, 1] * half_n)[:input_shape]
        m2 = ([1, 0] * half_n)[:input_shape]
        self.masks = np.array([m1, m2] * (num_coupling // 2), dtype=np.float32)
        ...
```

Here we build the masks to discriminate the $x_1$ and $x_2$ components at each layer

* As in the original RNVP paper, we use an _alternating checkboard pattern_
  - I.e. we take even indexes at one layer, and odd indexes at the next layer
* ...So that all variables are transformed, if we have at least 2 affine coupling layers

## RNVP Model

**The `__init__` method (constructor) initializes the internal fields**

```python
    def __init__(self, input_shape, num_coupling, units_coupling=32, depth_coupling=0,
            reg_coupling=0.01):
        ...
        self.layers_list = [coupling(input_shape, units_coupling, depth_coupling, reg_coupling)
                            for i in range(num_coupling)]
```

Finally, here we build the model layers

* Each one consists in an affine coupling
* ...And contains in turn two Multi Layer Perceptrons
* Recall that we need at least 2 affine couplings to transform all variables

## RNVP Model

**The `call` method handles the transformation, in both directions**

```python
def call(self, x, training=True):
    log_det_inv, direction = 0, 1
    if training: direction = -1
    for i in range(self.num_coupling)[::direction]:
        x_masked = x * self.masks[i]
        reversed_mask = 1 - self.masks[i]
        s, t = self.layers_list[i](x_masked)
        s, t = s*reversed_mask, t*reversed_mask
        gate = (direction - 1) / 2
        x = reversed_mask * (x * tf.exp(direction * s) + direction * t * tf.exp(gate * s)) \
            + x_masked
        log_det_inv += gate * tf.reduce_sum(s, axis=1)
    return x, log_det_inv
```

## RNVP Model

**The `call` method handles the transformation, in both directions**

```python
def call(self, x, training=True):
    log_det_inv, direction = 0, 1
    if training: direction = -1
    for i in range(self.num_coupling)[::direction]:
    ...
```

The `direction` variable controls the direction of the transformation

* By default, this implementation transforms $z$ into $x$
  - I.e. it works _backwards_, compared to our theoretical discussion
* This is the case since RNVP are often mainly used as _generative models_
* At training time, we always want to transform $x$ into $z$
* ...And this is why `direction = -1` when `training` is `True`

## RNVP Model

**The `call` method handles the transformation, in both directions**

```python
def call(self, x, training=True):
    for i in range(self.num_coupling)[::direction]:
        x_masked = x * self.masks[i]
        reversed_mask = 1 - self.masks[i]
        s, t = self.layers_list[i](x_masked)
        s, t = s*reversed_mask, t*reversed_mask
        ...
```

* Here we mask $x$, i.e. filter the $x_1$ subset of variables
* ...We compute the value of the $s$ and $t$ function
* Then we filter such values using a the reversed (i.e. negated) mask
* I.e. prepare $s$ and $t$ for their application to the $x_2$ subset

## RNVP Model

**The `call` method handles the transformation, in both directions**

```python
def call(self, x, training=True):
        ...
        gate = (direction - 1) / 2
        x = reversed_mask * (x * tf.exp(direction * s) + direction * t * tf.exp(gate * s)) \
            + x_masked
        ...
```

Here we compute the main transformation (backwards, as mentioned):

* If `training = True`, we have `direction = -1` and we compute:
$$\begin{align}
x^1 &= y^1 \\
x^2 &= (y^2 - t(y^1)) \oslash e^{s(y^1)}
\end{align}$$

## RNVP Model

**The `call` method handles the transformation, in both directions**

```python
def call(self, x, training=True):
        ...
        gate = (direction - 1) / 2
        x = reversed_mask * (x * tf.exp(direction * s) + direction * t * tf.exp(gate * s)) \
            + x_masked
        ...
```

Here we compute the main transformation (backwards, as mentioned):

* If `training = False`, we have `direction = 1` and we compute:
$$\begin{align}
y^1 &= x^1 \\
y^2 &= e^{s(x^1)} \odot x^2 + t(x^1)
\end{align}$$

## RNVP Model

**The `call` method handles the transformation, in both directions**

```python
def call(self, x, training=True):
    ...
    for i in range(self.num_coupling)[::direction]:
        ...
        log_det_inv += gate * tf.reduce_sum(s, axis=1)
    return x, log_det_inv
```

At each layer, we also compute the $\log \det$ of the Jacobian

* ...Which is simply the sum of the $s$ function values
* Determinants of different layers should be multiplied (due to the chain rule)...
* ...Which means that their $\log$ is simply summed

At then end of the process, the determinant has been computed

## RNVP Model

**The `score_samples` method performs _density estimation_**

```python
def score_samples(self, x):
    y, logdet = self(x)
    log_probs = self.distribution.log_prob(y) + logdet
    return log_probs
```

The process relies on the change of variable formula:

* First, it triggers the `call` method with `training=True`
  - I.e. transforms data points $x$ into their latent representation $z$
* Then, it computes the (log) density of $z$
  - Using `tensorfllow_probability` comes in handy at this point
* ...And then sums the log determinant

## RNVP Model

**The `log_loss` method computes the _loss function_**

```python
def log_loss(self, x):
    log_densities = self.score_samples(x)
    return -tf.reduce_mean(log_densities)
```

This is done by:

* Obtaining the estimated densities via `score_samples`
* ...Summing up (in log scale, i.e. a product in the original scale)
* ...And finally swapping the sign of the resut
  - ...Since we want to _maximize_ the likelihood

## RNVP Model

**The `train_step` method is called by the keras `fit` method**

```python
def train_step(self, data):
    with tf.GradientTape() as tape:
        loss = self.log_loss(data)
    g = tape.gradient(loss, self.trainable_variables)
    self.optimizer.apply_gradients(zip(g, self.trainable_variables))
    self.loss_tracker.update_state(loss)
    return {"loss": self.loss_tracker.result()}
```

The `GradientTape` is _how tensorflow handles differentiation_

* All tensor operations made in the scope of a `GradientTape` are tracked
* ...So that a gradient can then be extracted
* Then we apply the gradient to the model weights (using the optimizer)
* ...And finally we track the loss

# Using Real NVPs

## Using Real NVP

**We are ready to test our model**

We will use a classical benchmark for density estimation (shaped like two half moons)

```python
from sklearn.datasets import make_moons
data = make_moons(3000, noise=0.05)[0].astype(np.float32)
nn.plot_distribution_2D(samples=data, figsize=figsize)
```

* We use `float32` numbers for easier interplay with tensorflow

## Training

**Now, we need to train a Real NVP model**

* We will use the whole dataset (this is just a simple test)
* ...But first, we need to _standardize_ it

```python
data_s = (data - data.mean(axis=0)) / data.std(axis=0)
```

**Standardization is very important when using Real NVPs**

* This is true for Neural Networks in general, for the usual reasons
* But even more in this case, since _the distribution for $z$ is standardized_
  - Standardizing the data makes it easier to learn a mapping

## Training

**Next we can perform training, as usual in keras**

```python
from tensorflow.keras.callbacks import EarlyStopping
model = nn.RealNVP(input_shape=2, num_coupling=10, units_coupling=32, depth_coupling=2, reg_coupling=0.01)
model.compile(optimizer='Adam')
cb = [EarlyStopping(monitor='loss', patience=40, min_delta=0.0001, restore_best_weights=True)]
history = model.fit(data_s, batch_size=256, epochs=200, verbose=1, callbacks=cb)
```

    Epoch 1/200
    12/12 [==============================] - 4s 4ms/step - loss: 2.9582
    Epoch 2/200
    12/12 [==============================] - 0s 4ms/step - loss: 2.7342
    Epoch 3/200
    12/12 [==============================] - 0s 4ms/step - loss: 2.5690
    Epoch 4/200
    12/12 [==============================] - 0s 4ms/step - loss: 2.5072
    Epoch 5/200
    12/12 [==============================] - 0s 4ms/step - loss: 2.4623
    Epoch 6/200
    12/12 [==============================] - 0s 4ms/step - loss: 2.4218
    Epoch 7/200
    12/12 [==============================] - 0s 4ms/step - loss: 2.3766
    Epoch 8/200
    12/12 [==============================] - 0s 4ms/step - loss: 2.3357
    Epoch 9/200
    12/12 [==============================] - 0s 4ms/step - loss: 2.2893
    Epoch 10/200
    12/12 [==============================] - 0s 4ms/step - loss: 2.2384
    Epoch 11/200
    12/12 [==============================] - 0s 4ms/step - loss: 2.1972
    Epoch 12/200
    12/12 [==============================] - 0s 4ms/step - loss: 2.1474
    Epoch 13/200
    12/12 [==============================] - 0s 4ms/step - loss: 2.0999
    Epoch 14/200
    12/12 [==============================] - 0s 4ms/step - loss: 2.0578
    Epoch 15/200
    12/12 [==============================] - 0s 4ms/step - loss: 2.0285
    Epoch 16/200
    12/12 [==============================] - 0s 4ms/step - loss: 2.0015
    Epoch 17/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.9644
    Epoch 18/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.9246
    Epoch 19/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.8893
    Epoch 20/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.8999
    Epoch 21/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.8473
    Epoch 22/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.8471
    Epoch 23/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.8188
    Epoch 24/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.8037
    Epoch 25/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.7702
    Epoch 26/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.7690
    Epoch 27/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.7703
    Epoch 28/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.7137
    Epoch 29/200
    12/12 [==============================] - 0s 6ms/step - loss: 1.7134
    Epoch 30/200
    12/12 [==============================] - 0s 6ms/step - loss: 1.6914
    Epoch 31/200
    12/12 [==============================] - 0s 6ms/step - loss: 1.6766
    Epoch 32/200
    12/12 [==============================] - 0s 6ms/step - loss: 1.6859
    Epoch 33/200
    12/12 [==============================] - 0s 6ms/step - loss: 1.6902
    Epoch 34/200
    12/12 [==============================] - 0s 6ms/step - loss: 1.6717
    Epoch 35/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.6347
    Epoch 36/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.6327
    Epoch 37/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.6462
    Epoch 38/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.6361
    Epoch 39/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.6378
    Epoch 40/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.6284
    Epoch 41/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.6124
    Epoch 42/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.6401
    Epoch 43/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.5696
    Epoch 44/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.5517
    Epoch 45/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.5557
    Epoch 46/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.5577
    Epoch 47/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.5653
    Epoch 48/200
    12/12 [==============================] - 0s 6ms/step - loss: 1.5537
    Epoch 49/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.5318
    Epoch 50/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.5194
    Epoch 51/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.5062
    Epoch 52/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.5209
    Epoch 53/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.4900
    Epoch 54/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.5006
    Epoch 55/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.5075
    Epoch 56/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.4839
    Epoch 57/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.5291
    Epoch 58/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.4826
    Epoch 59/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.4759
    Epoch 60/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.4576
    Epoch 61/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.4511
    Epoch 62/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.4506
    Epoch 63/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.6070
    Epoch 64/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.5913
    Epoch 65/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.5237
    Epoch 66/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.4656
    Epoch 67/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.4327
    Epoch 68/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.4296
    Epoch 69/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.4188
    Epoch 70/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.4284
    Epoch 71/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.4163
    Epoch 72/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.4116
    Epoch 73/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.4076
    Epoch 74/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.3908
    Epoch 75/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.3748
    Epoch 76/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.3666
    Epoch 77/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.3662
    Epoch 78/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.3726
    Epoch 79/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.3892
    Epoch 80/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.3593
    Epoch 81/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.3530
    Epoch 82/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.3609
    Epoch 83/200
    12/12 [==============================] - 0s 6ms/step - loss: 1.3697
    Epoch 84/200
    12/12 [==============================] - 0s 6ms/step - loss: 1.3839
    Epoch 85/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.3756
    Epoch 86/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.3377
    Epoch 87/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.3218
    Epoch 88/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.3029
    Epoch 89/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.3069
    Epoch 90/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.3089
    Epoch 91/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.3012
    Epoch 92/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.3344
    Epoch 93/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.3451
    Epoch 94/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.3361
    Epoch 95/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.3095
    Epoch 96/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.3090
    Epoch 97/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2886
    Epoch 98/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2938
    Epoch 99/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.3136
    Epoch 100/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.3002
    Epoch 101/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2870
    Epoch 102/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2937
    Epoch 103/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.3117
    Epoch 104/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.2813
    Epoch 105/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.2606
    Epoch 106/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.3260
    Epoch 107/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.2983
    Epoch 108/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.2689
    Epoch 109/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.2598
    Epoch 110/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.2481
    Epoch 111/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.2718
    Epoch 112/200
    12/12 [==============================] - 0s 6ms/step - loss: 1.3031
    Epoch 113/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.3096
    Epoch 114/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.3070
    Epoch 115/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2780
    Epoch 116/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2654
    Epoch 117/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2539
    Epoch 118/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2532
    Epoch 119/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2457
    Epoch 120/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.2600
    Epoch 121/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.2409
    Epoch 122/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.2498
    Epoch 123/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.2358
    Epoch 124/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2445
    Epoch 125/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2543
    Epoch 126/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2642
    Epoch 127/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2511
    Epoch 128/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.2336
    Epoch 129/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2400
    Epoch 130/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2448
    Epoch 131/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2364
    Epoch 132/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2631
    Epoch 133/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2741
    Epoch 134/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2498
    Epoch 135/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2297
    Epoch 136/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2220
    Epoch 137/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2168
    Epoch 138/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2350
    Epoch 139/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2172
    Epoch 140/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.2114
    Epoch 141/200
    12/12 [==============================] - 0s 6ms/step - loss: 1.2108
    Epoch 142/200
    12/12 [==============================] - 0s 6ms/step - loss: 1.2151
    Epoch 143/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.2098
    Epoch 144/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2037
    Epoch 145/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2017
    Epoch 146/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.2081
    Epoch 147/200
    12/12 [==============================] - 0s 6ms/step - loss: 1.2162
    Epoch 148/200
    12/12 [==============================] - 0s 6ms/step - loss: 1.2217
    Epoch 149/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.2057
    Epoch 150/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2211
    Epoch 151/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2013
    Epoch 152/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.1969
    Epoch 153/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2202
    Epoch 154/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2212
    Epoch 155/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2261
    Epoch 156/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.2378
    Epoch 157/200
    12/12 [==============================] - 0s 6ms/step - loss: 1.2072
    Epoch 158/200
    12/12 [==============================] - 0s 6ms/step - loss: 1.2131
    Epoch 159/200
    12/12 [==============================] - 0s 6ms/step - loss: 1.2159
    Epoch 160/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.2079
    Epoch 161/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2056
    Epoch 162/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.1981
    Epoch 163/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2145
    Epoch 164/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2155
    Epoch 165/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.1989
    Epoch 166/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.1963
    Epoch 167/200
    12/12 [==============================] - 0s 6ms/step - loss: 1.2261
    Epoch 168/200
    12/12 [==============================] - 0s 6ms/step - loss: 1.2210
    Epoch 169/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2309
    Epoch 170/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2015
    Epoch 171/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.1927
    Epoch 172/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2029
    Epoch 173/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2130
    Epoch 174/200
    12/12 [==============================] - 0s 6ms/step - loss: 1.2294
    Epoch 175/200
    12/12 [==============================] - 0s 6ms/step - loss: 1.1953
    Epoch 176/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.1965
    Epoch 177/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.1975
    Epoch 178/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.1932
    Epoch 179/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.1830
    Epoch 180/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.1871
    Epoch 181/200
    12/12 [==============================] - 0s 6ms/step - loss: 1.1859
    Epoch 182/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.1701
    Epoch 183/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.1729
    Epoch 184/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.1856
    Epoch 185/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.1811
    Epoch 186/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.1907
    Epoch 187/200
    12/12 [==============================] - 0s 6ms/step - loss: 1.2068
    Epoch 188/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.2103
    Epoch 189/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.1961
    Epoch 190/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.2080
    Epoch 191/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.1792
    Epoch 192/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.1700
    Epoch 193/200
    12/12 [==============================] - 0s 6ms/step - loss: 1.1747
    Epoch 194/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.1817
    Epoch 195/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.1716
    Epoch 196/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.1947
    Epoch 197/200
    12/12 [==============================] - 0s 4ms/step - loss: 1.1874
    Epoch 198/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.1886
    Epoch 199/200
    12/12 [==============================] - 0s 5ms/step - loss: 1.2214
    Epoch 200/200
    12/12 [==============================] - 0s 6ms/step - loss: 1.2064

## Training

**As usual with NNs, choosing the right architecture can be complicated**

```python
model = RealNVP(input_shape=2, num_coupling=16, units_coupling=32, depth_coupling=2, reg_coupling=0.01)
```

* We went for a relatively deep model (10 affine coupling)
* Each coupling has also a good degree of non-linearity (2 hidden layers)
* We used a small degree of L2 regularization to stabilize the training process

**We also use relatively _large batch size_**

```python
history = model.fit(data_s, batch_size=256, epochs=200, verbose=2, callbacks=cb)
```

* Large batch sizes are usually a good choice with density estimation approaches
* Batches should be ideally be representative of the distribution

## Training

**Let's see the evolution of the training loss over time**

```python
nn.plot_training_history(history, figsize=figsize)
```

## Latent Space Representation

**We can obtain the latent space representation by calling the trained model**

This will trigger the `call` method with default parameters (i.e. `training=True`)

```python
z, _ = model(data_s)
nn.plot_distribution_2D(samples=z, figsize=figsize)
```

## Density Estimation

**We can estimate the density of any data point**

```python
nn.plot_distribution_2D(estimator=model, xr=np.linspace(-2, 2, 100, dtype=np.float32),
                                         yr=np.linspace(-2, 2, 100, dtype=np.float32), figsize=figsize)
```

* A good approximation! With a strange low-density connection between the moons

## Data Generation

**We can also generate data, by sampling from $p_z$ and then calling `predict`**

This will trigger the `call` method with `training=False`

```python
samples = model.distribution.sample(3000)
x, _ = model.predict(samples)
nn.plot_distribution_2D(samples=x, figsize=figsize)
```

## Data Generation

**We can also plot the mapping for selected data points...**

...Which gives and intuition of how the transformation works

```python
nn.plot_rnvp_transformation(model, figsize=figsize)
```

# RNVP  for Anomaly Detection

## RNVP for Anomaly Detection

**RNVPs can be used for anomaly detection like any other density estimator**

First, we build and compile the model (for the HPC data)

```python
input_shape = len(hpc_in)
hpc_rnvp = nn.RealNVP(input_shape=input_shape,
                num_coupling=6, units_coupling=32, depth_coupling=1, reg_coupling=0.01)
hpc_rnvp.compile(optimizer='Adam')
```

We chose a _simpler_ architecture this time

* With RNVP, dealing with higher dimensional data has actually some advantage
* In particular, we have richer input for the $s$ and $t$ functions
  - In the "moons" dataset, $s$ and $t$ had 2/2 = 1 input feature
  - Now we have 159/2 = 79--80 features

## RNVP for Anomaly Detection

**Then we perform training as usual**

```python
X = trdata[hpc_in].astype(np.float32).values
cb = [EarlyStopping(monitor='loss', patience=10, min_delta=0.001, restore_best_weights=True)]
history = hpc_rnvp.fit(X, batch_size=256, epochs=100, verbose=1, callbacks=cb)
```

    Epoch 1/100
    12/12 [==============================] - 2s 7ms/step - loss: 719.0934
    Epoch 2/100
    12/12 [==============================] - 0s 7ms/step - loss: 905.2411
    Epoch 3/100
    12/12 [==============================] - 0s 7ms/step - loss: 412.6078
    Epoch 4/100
    12/12 [==============================] - 0s 7ms/step - loss: 1169.5496
    Epoch 5/100
    12/12 [==============================] - 0s 6ms/step - loss: 754.5610
    Epoch 6/100
    12/12 [==============================] - 0s 7ms/step - loss: 678.0566
    Epoch 7/100
    12/12 [==============================] - 0s 7ms/step - loss: 434.5121
    Epoch 8/100
    12/12 [==============================] - 0s 7ms/step - loss: 365.8380
    Epoch 9/100
    12/12 [==============================] - 0s 7ms/step - loss: 264.4407
    Epoch 10/100
    12/12 [==============================] - 0s 7ms/step - loss: 195.1885
    Epoch 11/100
    12/12 [==============================] - 0s 6ms/step - loss: 151.8016
    Epoch 12/100
    12/12 [==============================] - 0s 7ms/step - loss: 123.2256
    Epoch 13/100
    12/12 [==============================] - 0s 7ms/step - loss: 85.4567
    Epoch 14/100
    12/12 [==============================] - 0s 6ms/step - loss: 126.3301
    Epoch 15/100
    12/12 [==============================] - 0s 6ms/step - loss: 859.1494
    Epoch 16/100
    12/12 [==============================] - 0s 6ms/step - loss: 63.1145
    Epoch 17/100
    12/12 [==============================] - 0s 6ms/step - loss: 49.0259
    Epoch 18/100
    12/12 [==============================] - 0s 6ms/step - loss: 41.8723
    Epoch 19/100
    12/12 [==============================] - 0s 6ms/step - loss: 23.2193
    Epoch 20/100
    12/12 [==============================] - 0s 6ms/step - loss: 1.9452
    Epoch 21/100
    12/12 [==============================] - 0s 6ms/step - loss: -11.8411
    Epoch 22/100
    12/12 [==============================] - 0s 7ms/step - loss: -21.3956
    Epoch 23/100
    12/12 [==============================] - 0s 6ms/step - loss: -29.9412
    Epoch 24/100
    12/12 [==============================] - 0s 6ms/step - loss: -36.1167
    Epoch 25/100
    12/12 [==============================] - 0s 6ms/step - loss: -42.5414
    Epoch 26/100
    12/12 [==============================] - 0s 8ms/step - loss: -47.1233
    Epoch 27/100
    12/12 [==============================] - 0s 8ms/step - loss: -52.5653
    Epoch 28/100
    12/12 [==============================] - 0s 6ms/step - loss: -50.0787
    Epoch 29/100
    12/12 [==============================] - 0s 6ms/step - loss: -61.3525
    Epoch 30/100
    12/12 [==============================] - 0s 7ms/step - loss: -65.5290
    Epoch 31/100
    12/12 [==============================] - 0s 6ms/step - loss: -69.6306
    Epoch 32/100
    12/12 [==============================] - 0s 7ms/step - loss: -73.5066
    Epoch 33/100
    12/12 [==============================] - 0s 7ms/step - loss: -76.8681
    Epoch 34/100
    12/12 [==============================] - 0s 6ms/step - loss: -79.3996
    Epoch 35/100
    12/12 [==============================] - 0s 6ms/step - loss: -83.0936
    Epoch 36/100
    12/12 [==============================] - 0s 6ms/step - loss: -86.4273
    Epoch 37/100
    12/12 [==============================] - 0s 7ms/step - loss: -57.7764
    Epoch 38/100
    12/12 [==============================] - 0s 6ms/step - loss: 4.6803
    Epoch 39/100
    12/12 [==============================] - 0s 7ms/step - loss: 78.7103
    Epoch 40/100
    12/12 [==============================] - 0s 7ms/step - loss: 77.7866
    Epoch 41/100
    12/12 [==============================] - 0s 7ms/step - loss: 518.9639
    Epoch 42/100
    12/12 [==============================] - 0s 7ms/step - loss: -18.1266
    Epoch 43/100
    12/12 [==============================] - 0s 6ms/step - loss: -51.2412
    Epoch 44/100
    12/12 [==============================] - 0s 6ms/step - loss: -63.3980
    Epoch 45/100
    12/12 [==============================] - 0s 7ms/step - loss: -79.5090
    Epoch 46/100
    12/12 [==============================] - 0s 7ms/step - loss: -86.7409
    Epoch 47/100
    12/12 [==============================] - 0s 6ms/step - loss: -91.8204
    Epoch 48/100
    12/12 [==============================] - 0s 6ms/step - loss: -95.7429
    Epoch 49/100
    12/12 [==============================] - 0s 7ms/step - loss: -94.0352
    Epoch 50/100
    12/12 [==============================] - 0s 7ms/step - loss: -93.8613
    Epoch 51/100
    12/12 [==============================] - 0s 8ms/step - loss: -95.4797
    Epoch 52/100
    12/12 [==============================] - 0s 7ms/step - loss: -81.3049
    Epoch 53/100
    12/12 [==============================] - 0s 7ms/step - loss: -96.0608
    Epoch 54/100
    12/12 [==============================] - 0s 7ms/step - loss: -106.8405
    Epoch 55/100
    12/12 [==============================] - 0s 7ms/step - loss: -109.5008
    Epoch 56/100
    12/12 [==============================] - 0s 8ms/step - loss: -114.8781
    Epoch 57/100
    12/12 [==============================] - 0s 6ms/step - loss: -116.7885
    Epoch 58/100
    12/12 [==============================] - 0s 6ms/step - loss: -118.7055
    Epoch 59/100
    12/12 [==============================] - 0s 6ms/step - loss: -120.1718
    Epoch 60/100
    12/12 [==============================] - 0s 6ms/step - loss: -122.3495
    Epoch 61/100
    12/12 [==============================] - 0s 7ms/step - loss: -117.5011
    Epoch 62/100
    12/12 [==============================] - 0s 7ms/step - loss: -126.3227
    Epoch 63/100
    12/12 [==============================] - 0s 6ms/step - loss: -128.0181
    Epoch 64/100
    12/12 [==============================] - 0s 6ms/step - loss: -129.3858
    Epoch 65/100
    12/12 [==============================] - 0s 6ms/step - loss: -130.7806
    Epoch 66/100
    12/12 [==============================] - 0s 6ms/step - loss: -131.5900
    Epoch 67/100
    12/12 [==============================] - 0s 6ms/step - loss: -133.4695
    Epoch 68/100
    12/12 [==============================] - 0s 6ms/step - loss: -134.7427
    Epoch 69/100
    12/12 [==============================] - 0s 6ms/step - loss: -136.1255
    Epoch 70/100
    12/12 [==============================] - 0s 6ms/step - loss: -132.4985
    Epoch 71/100
    12/12 [==============================] - 0s 6ms/step - loss: -138.4926
    Epoch 72/100
    12/12 [==============================] - 0s 6ms/step - loss: -139.9982
    Epoch 73/100
    12/12 [==============================] - 0s 6ms/step - loss: -139.2789
    Epoch 74/100
    12/12 [==============================] - 0s 6ms/step - loss: -141.9384
    Epoch 75/100
    12/12 [==============================] - 0s 6ms/step - loss: -142.7077
    Epoch 76/100
    12/12 [==============================] - 0s 6ms/step - loss: -122.0286
    Epoch 77/100
    12/12 [==============================] - 0s 6ms/step - loss: -138.0880
    Epoch 78/100
    12/12 [==============================] - 0s 6ms/step - loss: -141.1452
    Epoch 79/100
    12/12 [==============================] - 0s 6ms/step - loss: -146.9416
    Epoch 80/100
    12/12 [==============================] - 0s 6ms/step - loss: -149.3540
    Epoch 81/100
    12/12 [==============================] - 0s 6ms/step - loss: -151.3710
    Epoch 82/100
    12/12 [==============================] - 0s 6ms/step - loss: -152.5702
    Epoch 83/100
    12/12 [==============================] - 0s 6ms/step - loss: -153.2488
    Epoch 84/100
    12/12 [==============================] - 0s 6ms/step - loss: -154.0018
    Epoch 85/100
    12/12 [==============================] - 0s 6ms/step - loss: -156.0649
    Epoch 86/100
    12/12 [==============================] - 0s 6ms/step - loss: -156.7068
    Epoch 87/100
    12/12 [==============================] - 0s 6ms/step - loss: -157.5935
    Epoch 88/100
    12/12 [==============================] - 0s 7ms/step - loss: -158.7546
    Epoch 89/100
    12/12 [==============================] - 0s 7ms/step - loss: -159.1322
    Epoch 90/100
    12/12 [==============================] - 0s 6ms/step - loss: -160.5154
    Epoch 91/100
    12/12 [==============================] - 0s 6ms/step - loss: -151.3180
    Epoch 92/100
    12/12 [==============================] - 0s 7ms/step - loss: -159.1134
    Epoch 93/100
    12/12 [==============================] - 0s 7ms/step - loss: -159.9041
    Epoch 94/100
    12/12 [==============================] - 0s 7ms/step - loss: -161.6100
    Epoch 95/100
    12/12 [==============================] - 0s 6ms/step - loss: -162.6764
    Epoch 96/100
    12/12 [==============================] - 0s 6ms/step - loss: -162.8461
    Epoch 97/100
    12/12 [==============================] - 0s 6ms/step - loss: -163.9352
    Epoch 98/100
    12/12 [==============================] - 0s 6ms/step - loss: -165.5593
    Epoch 99/100
    12/12 [==============================] - 0s 6ms/step - loss: -166.6348
    Epoch 100/100
    12/12 [==============================] - 0s 6ms/step - loss: -167.3380

## RNVP for Anomaly Detection

**Here is the loss evolution over time**

```python
nn.plot_training_history(history, figsize=figsize)
```

## RNVP for Anomaly Detection

**Then we can generate a signal as usual**

```python
X = hpcs[hpc_in].astype(np.float32).values
signal_hpc = pd.Series(index=hpcs.index, data=-hpc_rnvp.score_samples(X))
nn.plot_signal(signal_hpc, hpc_labels, figsize=figsize)
```

* The signal is very similar to that of KDE (not a surprise)

## RNVP for Anomaly Detection

**Finally, we can tune the threshold**

```python
th_range = np.linspace(1e5, 1.5e6, 100)
thr, val_cost = nn.opt_threshold(signal_hpc[tr_end:val_end],
                                        valdata['anomaly'],
                                        th_range, cmodel)
print(f'Best threshold: {thr:.3f}')
tr_cost = cmodel.cost(signal_hpc[:tr_end], hpcs['anomaly'][:tr_end], thr)
print(f'Cost on the training set: {tr_cost}')
print(f'Cost on the validation set: {val_cost}')
ts_cost = cmodel.cost(signal_hpc[val_end:], hpcs['anomaly'][val_end:], thr)
print(f'Cost on the test set: {ts_cost}')
```

    Best threshold: 1217171.717
    Cost on the training set: 0
    Cost on the validation set: 269
    Cost on the test set: 265

* Once again, the performance is on par with KDE
* ...But we have better support for high-dimensional data!

notebooks/2. Density Estimation with Neural Networks.ipynb


# Music Machine Learning - Support Vector Machines (SVM)

### Author: Philippe Esling (esling@ircam.fr)

In this course we will cover
1. A recap on the [motivation](#motivation) behind Support Vector Machines (SVM)
2. A simple use case for [understanding and fitting SVMs](#fitting) with the scikit library
3. Developing our own [implementation](#implem) of linear SVMs
4. An introuction to [kernels](#improve) in SVMs and how to implement them

## Support Vector Machines (SVM)

In this tutorial, we will cover a more advanced classification algorithm through the use of *Support Vector Machines* (SVMs). The goal is to gain an intuition of how SVMs works and how to use *Gaussian kernel* with SVMs to find out the decision boundary. The implementation proposed here follows a simplified version of the *Sequential Minimal Optimization* (SMO) algorithm for training support vector machines, which is detailed in the following [paper](http://cs229.stanford.edu/materials/smo.pdf). The complete SMO algorithm can then be quite easily implemented following the [full paper](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/tr-98-14.pdf).

Once again, to simplify your work, we provide the following set of functions that you should find in the `data` folder, and in the `helper_plot` file

  |**File**|*Explanation*|
  |-------:|:------------|
  |`svm_linear.mat`|Example data for a quasi-linear problem|
  |`svm_gaussian.mat`|Example data for a non-linear but well-defined problem|
  |`plot_data`|Allows to plot the set of data points|
  |`visualize_boundary`|Plots a non-linear decision boundary from a SVM model|
  |`plot_svc_decision_function`|Plots a linear decision boundary|

```python
import numpy as np
from scipy import io as sio
from matplotlib import pyplot as plt
from helper_plot import plot_data, visualize_boundary, visualize_boundary_linear, hdr_plot_style, plot_svc_decision_function
```

## Motivation

In the previous course on neural networks, we learned how to train a simple neuron. We saw that this simple model could be interpreted as defining a line, separating the space into two different regions. This allows to consider problems of *discriminative classification*: rather than modeling each class, we simply find a line that divides the classes from each other. As an example of these _linearly separable_ problems, consider the simple case of a classification task, in which the two classes of points are well separated:

```python
from sklearn.datasets.samples_generator import make_blobs
hdr_plot_style()
X, y = make_blobs(n_samples=50, centers=2, random_state=0, cluster_std=0.65)
plt.figure(figsize=(10, 8)); plt.grid(True)
plt.scatter(X[:, 0], X[:, 1], c=y, s=80, edgecolor='w', cmap='winter');
plt.title("Linear classification example");
```

As we have seen previously, a single neuron classifier would attempt to draw a straight line separating the two sets of data, and thereby create a model for classification, by optimizing the parameters $\theta=\{\mathbf{w}, b\}$ of

$$
\begin{equation}
y=\sigma\left(\sum_{i = 1}^{n}w_{i}.x_{i} + b\right)
\label{eq1}
\end{equation}
$$

For two dimensional data like that shown here, this is a task we could do by hand. But the major problem that we can directly see, is that there is an infinite number of lines that can perfectly discriminate between the two classes, as exemplified in the next example.

```python
xfit = np.linspace(-1, 3.5)
plt.figure(figsize=(10, 8)); plt.grid(True)
plt.scatter(X[:, 0], X[:, 1], c=y, s=80, edgecolor='w', cmap='winter')
for m, b in [(1, 0.65), (0.5, 1.6), (-0.2, 2.9)]:
    plt.plot(xfit, m * xfit + b, '-w')
plt.xlim(-1, 3.5);
```

Note that these solutions are all valid, and will mostly depend on the (random) initialization of our parameters. Although these all perfectly discriminate between our samples, we can clearly see that they are more or less "optimal" to the problem at hand. So our question today is how could we formalize this problem, and if we could find a formulation to adress and solve this issue.

### Notion of _margin_

The intuition behind comparing different solutions is that rather than estimating only the line between the classes, we can also look at the *margin* of some width defined around those line. In our simple case, it would look like this

```python
xfit = np.linspace(-1, 3.5)
plt.figure(figsize=(10, 8)); plt.grid(True)
plt.scatter(X[:, 0], X[:, 1], c=y, s=80, edgecolor='w', cmap='winter')
for m, b, d in [(1, 0.65, 0.25), (0.5, 1.6, 0.55), (-0.2, 2.9, 0.15)]:
    yfit = m * xfit + b
    plt.plot(xfit, yfit, '-w')
    plt.fill_between(xfit, yfit - d, yfit + d, edgecolor='w', color='#AAAAAA', alpha=0.6)
plt.xlim(-1, 3.5);
```

In *Support Vector Machines (SVMs)*, the goal is to find that optimal model, in which our separating line maximizes this margin. Hence, SVMs are part of a class of models called *maximum margin* estimator.

## Fitting a SVM

More formally the goal of SVMs is that given a set of input vectors $ \mathbf{x} $ separated in two classes (noted $ \mathbf{x}^{+} $ and $ \mathbf{x}^{-} $), we would like to find the *optimal boundary* (separating line) between the two classes. To do so, we want to find the separation that gives the *maximal margin* between the two sets. Hence, we can first define the following problem

$$
\begin{equation}
\begin{cases}
\mathbf{w}\cdot\mathbf{x}^{+}+\mathbf{b}\geq1\\
\mathbf{w}\cdot\mathbf{x}^{-}+\mathbf{b}\leq-1
\end{cases}
\label{eq1}
\end{equation}
$$

By setting the variable $ y_{i}={+}/{-}1 $, we can rewrite this problem as

$$
\begin{equation}
y_{i}\left(x_{i}\cdot\mathbf{w}+\mathbf{b}\right)-1\geq0
\label{eq2}
\end{equation}
$$

The idea is that for some of the input vectors (called *support vectors*), we will have an equality in equation (1), where $ \mathbf{w}\cdot\mathbf{x}^{+}+\mathbf{b}=1 $ and $ \mathbf{w}\cdot\mathbf{x}^{-}+\mathbf{b}=-1 $. By substracting those two equation, we see that we are trying to find a solution to

$$
\begin{equation}
\frac{\mathbf{w}}{\left\Vert \mathbf{w}\right\Vert }\cdot\left(x^{+}-x^{-}\right)=\frac{2}{\left\Vert \mathbf{w}\right\Vert }=Width
\label{eq3}
\end{equation}
$$

Therefore, solving the set of equations amounts to find the maximal margin between the support vectors (as they *support* classification).

### Fitting the SVM with scikit-learn

First, we can see the result of fitting this approach to our original data. To do so, we will use the SVM model found in `scikit-learn` to train an SVM model on this data. To do so, we can use the following code

```python
from sklearn.svm import SVC # "Support vector classifier"
model = SVC(kernel='linear')
model.fit(X, y)
```

    SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
      decision_function_shape='ovr', degree=3, gamma='auto', kernel='linear',
      max_iter=-1, probability=False, random_state=None, shrinking=True,
      tol=0.001, verbose=False)

To visualize the results, we will use our quick convenience function that plots the SVM decision boundaries

```python
plt.figure(figsize=(10, 8)); plt.grid(True)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor='w', cmap='winter')
plot_svc_decision_function(model);
```

This separating line is the one that maximizes the margin between our two sets of points.
Notice that some training points are right on the margin. These points are the pivotal elements of this fit, and are known as the *support vectors*, and give the algorithm its name. In `scikit-learn`, we can directly obtain the values of these points, stored in the ``support_vectors_`` attribute of the classifier:

```python
model.support_vectors_
```

    array([[0.39920934, 3.01626962],
           [2.45161058, 3.35844964],
           [2.06209253, 2.05847948],
           [3.02263145, 2.12999162]])

The major advantage of SVM is that for the fit, only the position of the support vectors matter; any points further from the margin which are on the correct side do not modify the fit. Technically, this is because these points do not contribute to the loss function used to fit the model, so their position and number do not matter so long as they do not cross the margin. We can see this, for example, if we plot the model learned from the first 50 or 150 points of this dataset

```python
# Plotting SVM trained on subsampled dataset
def plot_svm_subsample(N=10, ax=None):
    X, y = make_blobs(n_samples=200, centers=2, random_state=0, cluster_std=0.60)
    X = X[:N]
    y = y[:N]
    model = SVC(kernel='linear', C=1E10)
    model.fit(X, y)
    ax = ax or plt.gca()
    ax.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor = 'w', cmap='winter')
    ax.set_xlim(-1, 4)
    ax.set_ylim(-1, 6)
    plot_svc_decision_function(model, ax)

fig, ax = plt.subplots(1, 2, figsize=(16, 6))
fig.subplots_adjust(left=0.0625, right=0.95, wspace=0.1)
for axi, N in zip(ax, [60, 150]):
    plot_svm_subsample(N, axi)
    axi.set_title('N = {0}'.format(N))
```

In the left panel, we see the model and the support vectors for 60 training points.
In the right panel, we have doubled the number of training points, but the model has not changed: the three support vectors from the left panel are still the support vectors from the right panel.
This insensitivity to the exact behavior of distant points is one of the strengths of the SVM model.

## Implementing our own linear SVM

Here, we are going to try to implement our own linear SVM. We recall that, given a set of input vectors $ \mathbf{x} $ separated in two classes ($ \mathbf{x}^{+} $ and $ \mathbf{x}^{-} $, we would like to find the *optimal boundary* (separating line) between the two classes. This means that we are trying to find a solution to

$$
\begin{equation}
\frac{\mathbf{w}}{\left\Vert \mathbf{w}\right\Vert }\cdot\left(x^{+}-x^{-}\right)=\frac{2}{\left\Vert \mathbf{w}\right\Vert }=Width
\label{eq3}
\end{equation}
$$

Therefore, solving the set of equations amounts to find the maximal margin between the support vectors (as they *support* classification). As seen in the slides, we know that this problem can also be expressed as

$$
\begin{equation}
f\left(x\right)=\sum_{i=1}^{m}\alpha_{i}y_{i}\left\langle \phi\left(x_{i}\right),\phi\left(x\right)\right\rangle +b
\end{equation}
$$

Where the function $\phi(.)$ will be used later to introduce _kernels_, but for now, we will consider that $\phi(x)=x$, which reduces our problem to

$$
\begin{equation}
f\left(x\right)=\sum_{i=1}^{m}\alpha_{i}y_{i}\left\langle x_{i},x \right\rangle +b
\end{equation}
$$

we can introduce a kernel $K\left(x_{i},x\right)=\phi\left(x_{i}\right)\cdot\phi\left(x\right)$ in the equation in order to solve non-linear optimization problems. We can use this formulation to perform the *predictions* of the algorithm and in the first part of this tutorial, we will only consider the *linear* kernel $K\left(x_{i},x\right)=x_{i}\cdot x$.

Now, if we look at the way the SVM model that we trained is implemented in `scikit-learn`, we have the following variables

```python
print(vars(model))
```

    {'decision_function_shape': 'ovr', '_impl': 'c_svc', 'kernel': 'linear', 'degree': 3, 'gamma': 'auto', 'coef0': 0.0, 'tol': 0.001, 'C': 1.0, 'nu': 0.0, 'epsilon': 0.0, 'shrinking': True, 'probability': False, 'cache_size': 200, 'class_weight': None, 'verbose': False, 'max_iter': -1, 'random_state': None, '_sparse': False, 'class_weight_': array([1., 1.]), 'classes_': array([0, 1]), '_gamma': 0.5, 'support_': array([18, 25,  7, 46], dtype=int32), 'support_vectors_': array([[0.39920934, 3.01626962],
           [2.45161058, 3.35844964],
           [2.06209253, 2.05847948],
           [3.02263145, 2.12999162]]), 'n_support_': array([2, 2], dtype=int32), 'dual_coef_': array([[-0.24571733, -0.99437155,  1.        ,  0.24008889]]), 'intercept_': array([3.45657164]), 'probA_': array([], dtype=float64), 'probB_': array([], dtype=float64), 'fit_status_': 0, 'shape_fit_': (50, 2), '_intercept_': array([-3.45657164]), '_dual_coef_': array([[ 0.24571733,  0.99437155, -1.        , -0.24008889]])}

For the moment, our variables of interest are
- `support_vectors_` : _Support vectors_ (dataset points that support classification)
- `dual_coef_` : The $\alpha_{i} * y_{i}$ (coefficients from the dual formulation) for the _support_vectors_
- `n_support_` : Number of support vectors for each class (ordered)
- `_intercept_` : Value of $b$ in our formulation (bias)

In order to begin our own implementation of the SVM, we are going to start by computing the _decision function_ of the model. We are going to slightly change the model structure, to better reflect the mathematical formulation than `scikit-learn`. Hence, our model structure and all of its corresponding parameters will be defined as

``` Python
model = {};
model["x"];            # Values of _support vectors_ in input data
model["y"];            # Classes labels of _support vectors_
model["n"];            # Number of support for each class
model["b"];            # Value of bias
model["alphas"];       # Value of alphas
model["kernel"];       # Type of kernel
model["w"];            # Weights of vectors in primal formulation
```

In the following, we show how to shift from the `scikit-learn` version to ours

```python
#%% Creating a fake (random) model
our_model = {};
# Values of _support vectors_ in input data
our_model["x"] = model.support_vectors_
# Corresponding classes of the support vectors
our_model["y"] = np.concatenate([[0] * model.n_support_[0], [1] * model.n_support_[1]])
# Number of vectors for each class
our_model["n"] = model.n_support_
# Value of threshold
our_model["b"] = model._intercept_
# Values of alphas
our_model["alphas"] = model._dual_coef_[0]
print(our_model)
```

    {'x': array([[0.39920934, 3.01626962],
           [2.45161058, 3.35844964],
           [2.06209253, 2.05847948],
           [3.02263145, 2.12999162]]), 'y': array([0, 0, 1, 1]), 'n': array([2, 2], dtype=int32), 'b': array([-3.45657164]), 'alphas': array([ 0.24571733,  0.99437155, -1.        , -0.24008889])}

Computing the _predictions_ of the SVM is performed by calling `model.decision_function(X)` in `scikit_learn`. We are going to implement this behavior ourselves by coding our own `decision_function` function

``` Python
def decision_function(model, X)
    """
    % Returns a vector of predictions using a trained SVM model.
    % model   : svm model returned from svmTrain.
    % X       : m x n matrix where each example is a row.
    """
    return pred
```

For the time being we are using a _linear kernel_, but we might later use a different type of kernel, so we will directly use a `kernel` function which computes our kernel between the support vectors `x_s` and a dataset `X`.

***

**Exercice**
1. Implement the linear kernel function
1. Complete the decision function based on the dual formulation

***

```python
def kernel(x, X, type='linear'):

    ######################
    # YOUR CODE GOES HERE
    ######################

    if (type == 'linear'):
        k = ...
    return k

# This function should replicates the scikit-learn one
def decision_function(model, X):

    ######################
    # YOUR CODE GOES HERE
    ######################

    return ...

# Evaluate the decision function
ax = plt.figure(figsize=(10,8)).gca()
ax.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor = 'w', cmap='winter')
ax.set_xlim(-1, 4); ax.set_ylim(-1, 6)
xlim = ax.get_xlim(); ylim = ax.get_ylim()
# Create grid to evaluate model
x = np.linspace(xlim[0], xlim[1], 30)
yp = np.linspace(ylim[0], ylim[1], 30)
Y, Xp = np.meshgrid(yp, x)
xy = np.vstack([Xp.ravel(), Y.ravel()]).T
P = decision_function(our_model, xy.transpose())
P = P.reshape((30, 30))
# plot decision boundary and margins
ax.contour(Xp, Y, P, colors='w',levels=[-1, 0, 1], alpha=0.9,linestyles=['--', '-', '--']);
ax.scatter(our_model["x"][:, 0], our_model["x"][:, 1],s=300, linewidth=2, edgecolor='w', facecolors='none');
```

## Learning phase

In order to perform learning, we will try to maximize the *dual formulation* problem expressed as

$$
\begin{equation}
w\left(\alpha\right)=\sum_{i}\alpha_{i}-\frac{1}{2}\sum_{i}\sum_{j}y_{i}y_{j}\alpha_{i}\alpha_{j}\left\langle \phi\left(x_{i}\right),\phi\left(x_{j}\right)\right\rangle
\end{equation}
$$

with $0\leq \alpha_{i} \leq C$ and $\sum_{i=1}^{m}{\alpha_{i} y_{i}} = 0$. $C$ is a chosen *regularization hyper-parameter*. To ease your work, we will implement only a simplified version of the SMO algorithm which is mainly focused on the optimization aspects. The SMO algorithm selects two $\alpha$ parameters, $\alpha_{i}$ and $\alpha_{j}$ and optimizes the objective value jointly for both these $\alpha_{i,j}$. Finally it adjusts the $b$ parameter based on the new $\alpha_{i,j}$. This process is repeated until the $\alpha_{i,j}$ converge.

### Selecting $\alpha$

The full SMO algorithm use heuristics to choose which $\alpha_{i}$ and $\alpha_{j}$ to optimize (which greatly increase the convergence speed. However, for this simplified version, we will simply iterate on all $\alpha_{i},i=1,\cdots ,m$ and select a random $\alpha_{j}$, to perform the joint optimization of $\alpha_{i}$ and $\alpha_{j}$

### Optimizing $\alpha_{i}$ and $\alpha_{j}$

To jointly optimize the value, we first find bounds $L$ and $H$ such that $L\leq \alpha_{j} \leq H$ holds to satisfy the constraint of the original problem $0\leq \alpha_{i} \leq C$

$$
\begin{array}{ccc}
if\mbox{ }y_{i}\neq y_{j}, & L=max\left(0,\alpha_{j}-\alpha_{i}\right), & H=min\left(C,C+\alpha_{j}-\alpha_{i}\right)\\
if\mbox{ }y_{i}=y_{j}, & L=max\left(0,\alpha_{i}+\alpha_{j}-C\right), & H=min\left(C,\alpha_{i}+\alpha_{j}\right)
\end{array}
$$

Now we want to find $\alpha_{j}$ so as to maximize the objective function. It can be shown (as in the original paper) that the optimal $\alpha_{j}$ is given by

$$
\alpha_{j}=\alpha_{j}-\frac{y_{j}\left(E_{i}-E_{j}\right)}{\eta}
$$

where

$$
\begin{array}{ccc}
E_{k} & = & f\left(x_{k}\right)-y_{k}\\
\eta & = & 2\left\langle x_{i},x_{j}\right\rangle -\left\langle x_{i},x_{i}\right\rangle -\left\langle x_{j},x_{j}\right\rangle
\end{array}
$$

$E_{k}$ defines the error between the prediction of the SVM on the example $k$ and its desired label. When computing $\eta$, we can also replace the inner product by a kernel function $K$. Finally, we clip the value of $\alpha_{j}$ to lie within the selected bounds.

$$
\alpha_{j}=\begin{cases}
H & if\mbox{ }\alpha_{j}>H\\
\alpha_{j} & if\mbox{ }L\leq\alpha_{j}\leq H\\
L & if\mbox{ }\alpha_{j}<L
\end{cases}
$$

Now that we have solved $\alpha_{j}$, we can update the value for $\alpha_{i}$ through

$$
\alpha_{i}^{t}=\alpha_{i}^{t-1}+y_{i}y_{j}\left(\alpha_{j}^{(t-1)}-\alpha_{j}\right)
$$

### Computing threshold $b$

Finally, we can compute the threshold $b$ to satisfy the original constraints by selecting

$$
b=\begin{cases}
b_{1} & if\mbox{ }0<\alpha_{i}<C\\
b_{2} & if\mbox{ }0<\alpha_{j}<C\\
\left(b_{1}+b_{2}\right)/2 & otherwise
\end{cases}
$$

where

$$
b_{1}=b-E_{i}-y_{i}\left(\alpha_{i}^{(t-1)}-\alpha_{i}\right)\left\langle x_{i},x_{i}\right\rangle -y_{j}\left(\alpha_{j}^{(t-1)}-\alpha_{j}\right)\left\langle x_{i},x_{j}\right\rangle
$$
$$
b_{2}=b-E_{i}-y_{i}\left(\alpha_{i}^{(t-1)}-\alpha_{i}\right)\left\langle x_{i},x_{j}\right\rangle -y_{j}\left(\alpha_{j}^{(t-1)}-\alpha_{j}\right)\left\langle x_{j},x_{j}\right\rangle
$$

For the first part of this tutorial, we will compute the main iterations of the algorithm (minimization of the objective function), while relying on a *linear* kernel. This implies that we will only be able to perform linear discrimination. However, remember that the formulation of the SVMs provide an *optimal* and (gloriously) *convex* answer to this problem.

```Python
def svm_train(X, Y, C, kernelFunction, tol, maxIter):
    """
    % Trains an SVM classifier using a simplified SMO algorithm.
    % X       : m x n matrix of m training examples (with n-dimensional features).
    % Y       : column vector of class identifiers
    % C       : standard SVM regularization parameter
    % tol     : tolerance value used for determining equality of floating point numbers.
    % maxIter : number of iterations over the dataset before the algorithm stops.
    """
    return model
```

***

**Exercise**

  1. Update the `svm_train` code to perform the training of a SVM with linear kernel.
  2. Run your `svm_train` function on our linear dataset to obtain similar results as `scikit-learn`
  3. Update the framework to display the decision boundary at each iteration of the algorithm.

***

```python
def svm_train(X, Y_b, C, kernel_type, tol=1e-3, max_iter=5):
    """
    SVMTRAIN Trains an SVM classifier using a simplified version of the SMO
    # X       : m x n matrix of m training examples (with n-dimensional features).
    # Y       : column vector of class identifiers (0 or 1)
    # C       : standard SVM regularization parameter
    # tol     : tolerance value used for determining equality of floating point numbers.
    # maxIter : number of iterations over the dataset before the algorithm stops.
    """
    # Data parameters
    m, n = X.shape
    # Variables
    alphas = np.zeros((m, 1))
    b, passes, eta, L, H = 0, 0, 0, 0, 0
    E = np.zeros((m, 1))
    Y = Y_b.copy()
    Y[Y == 0] = -1

    # Pre-compute the kernel matrix (in practice, the optimized packages will _not_ do this)
    K = np.array(kernel(X, X.transpose(), kernel_type))

    # Train
    print('Training ...')
    while passes < max_iter:
        # Check that some alphas changed
        num_changed_alphas = 0;
        # Iterative over all alpha_i
        for i in range(m):

            ######################
            # YOUR CODE GOES HERE
            # => Perform one pass of the SMO algorithm for each alpha_i
            ######################

        if (num_changed_alphas == 0):
            passes = passes + 1
        else:
            passes = 0
    print(' Done !');
    # Save the model
    idx = (alphas > 0).transpose()[0];
    model = {}
    model["x"] = X[idx, :]
    model["y"] = Y[idx]
    model["b"] = b
    model["alphas"] = alphas[idx, 0] * Y[idx]
    model["kernel"] = kernel
    return model

# Training Linear SVM
print('Training Linear SVM ...')
# Parameters of the training
C = 1                  # C regularization value
tolerance = 1e-5       # Tolerance value for thresholds
nb_iterations = 50      # Number of iterations
# Perform the training (try to fiddle the previous values to see effect).
model = svm_train(X, y, C, 'linear', tolerance, nb_iterations);
# Evaluate the decision function
ax = plt.figure(figsize=(10, 8)).gca()
ax.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor = 'w', cmap='winter')
ax.set_xlim(-1, 4); ax.set_ylim(-1, 6)
xlim = ax.get_xlim(); ylim = ax.get_ylim()
# Create grid to evaluate model
x = np.linspace(xlim[0], xlim[1], 30)
yp = np.linspace(ylim[0], ylim[1], 30)
Y, Xp = np.meshgrid(yp, x)
xy = np.vstack([Xp.ravel(), Y.ravel()]).T
P = decision_function(model, xy.transpose())
P = P.reshape((30, 30))
# plot decision boundary and margins
ax.contour(Xp, Y, P, colors='w',levels=[-1, 0, 1], alpha=0.9,linestyles=['--', '-', '--'])
ax.scatter(model["x"][:, 0], model["x"][:, 1],s=300, linewidth=2, edgecolor='w', facecolors='none');
plt.title('Our trained SVM :)');
```

# Beyond linear problems with _kernels_

For the time being, our SVM is only able to solve _linear_ problems. However, the SVM approach becomes truly powerful is when it is combined with *kernels*. As seen in the slides, the major idea is to project our (non-linearly separable) data into a higher-dimensional space, where it becomes linearly separable (allowing to fit for nonlinear relationships with a linear classifier).

To better understand this idea, we can look at the same non linearly separable data that we discussed in the slides. Here, we try to fit a _linear_ SVM to this non-linear problem.

```python
from sklearn.datasets.samples_generator import make_circles
X, y = make_circles(100, factor=.1, noise=.1)

clf = SVC(kernel='linear').fit(X, y)
plt.figure(figsize=(10, 8)); plt.grid(True)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor='w', cmap='winter')
plot_svc_decision_function(clf, plot_support=False);
```

It is clear that no linear discrimination will *ever* be able to separate this data. As discussed, we need to think about how we might project the data into a higher dimension such that a linear separator *would* be sufficient. For example, one simple projection we could use would be to compute a *radial basis function* centered on the middle group of points.

```python
r = np.exp(-(X ** 2).sum(1))
```

We can visualize this extra data dimension using a three-dimensional plot

```python
from mpl_toolkits import mplot3d

def plot_3D(elev=30, azim=30, X=X, y=y):
    plt.figure(figsize=(10, 8));
    ax = plt.subplot(projection='3d')
    ax.scatter3D(X[:, 0], X[:, 1], r, c=y, s=50, edgecolor='w', cmap='winter')
    ax.view_init(elev=elev, azim=azim)
    ax.set_xlabel('x')
    ax.set_ylabel('y')
    ax.set_zlabel('z')
plot_3D(elev=30, azim=-50, X=X, y=y);
```

We can see that with this additional dimension, the data becomes trivially linearly separable, by drawing a separating plane. Here we had to choose and carefully tune our projection. In general, the need to make such a choice is a problem, as we would like to somehow automatically find the best basis functions to use. One strategy to this end is to compute a basis function centered at *every* point in the dataset, and let the SVM algorithm find the result. This type of basis function transformation is known as a *kernel transformation*, as it is based on a similarity relationship (or kernel) between **each pair of points**.

To see this idea in action first, we will (as previously) use `scikit-learn`, where we can apply _kernelized SVM_ simply by changing our _linear kernel_ to an _RBF (radial basis function)_ kernel, using the ``kernel`` model hyperparameter:

```python
model = SVC(kernel='rbf', C=1E6)
model.fit(X, y)
```

    SVC(C=1000000.0, cache_size=200, class_weight=None, coef0=0.0,
      decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf',
      max_iter=-1, probability=False, random_state=None, shrinking=True,
      tol=0.001, verbose=False)

Once fitted, we can see that our SVM is able to find the largest margin between these two sets of data

```python
plt.figure(figsize=(10, 8)); plt.grid(True)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor='w', cmap='winter')
plot_svc_decision_function(model)
plt.scatter(model.support_vectors_[:, 0], model.support_vectors_[:, 1], s=300, lw=1, facecolors='none');
```

Using this kernelized support vector machine, we learn a suitable nonlinear decision boundary.
This kernel transformation strategy is used often in machine learning to turn fast linear methods into fast nonlinear methods, especially for models in which the kernel trick can be used.

## Implementing the gaussian kernel

As we have seen previously, we know that the complete SVM optimization problem can also be expressed as

$$
\begin{equation}
f\left(x\right)=\sum_{i=1}^{m}\alpha_{i}y_{i}\left\langle \phi\left(x_{i}\right),\phi\left(x\right)\right\rangle +b
\end{equation}
$$

where we can introduce the kernel $K\left(x_{i},x\right)=\phi\left(x_{i}\right)\cdot\phi\left(x\right)$ in the equation in order to solve non-linear optimization problems. We can use this formulation to perform the *predictions* of the algorithm. Consequently, the complete *dual formulation* problem expressed as

$$
\begin{equation}
w\left(\alpha\right)=\sum_{i}\alpha_{i}-\frac{1}{2}\sum_{i}\sum_{j}y_{i}y_{j}\alpha_{i}\alpha_{j}\left\langle \phi\left(x_{i}\right),\phi\left(x_{j}\right)\right\rangle
\end{equation}
$$

with $0\leq \alpha_{i} \leq C$ and $\sum_{i=1}^{m}{\alpha_{i} y_{i}} = 0$. The Gaussian kernel (also called *Radial Basis Function (RBF)* kernel) allows to perform the optimization of non-linear classification problems. It is defined as

$$
K\left(x,y\right)=exp\left(\frac{-\left\Vert x-y\right\Vert ^{2}}{\left(2\sigma^{2}\right)}\right)
$$

As seen in the slides, the underlying idea is that instead of trying to solve a non-linear separation problem in a given space, we might first transform the space into a target space in which the problem is linearly separable. Therefore, we transform the space through a kernel (which is directly expressed in the minimization problem) and solve the linear separation problem in the target space. Given the starter code and the previous exercise, you should be able to directly plug this kernel inside the `svmTrain` and `svmPredict` functions.

***

**Exercise**

  1. Update the `kernel` function code to be able to rely on a Gaussian kernel.
  2. Check that you can run the `decision_function` code to rely on this Gaussian kernel.
  2. Run the `svm_train` function on our new (non-linear) dataset to obtain similar results as `scikit-learn`
  3. Once again, display the decision boundary at each iteration of the algorithm.

***

```python
def kernel(x, X, type='linear'):

    ######################
    # YOUR CODE GOES HERE
    ######################

    if (type == 'linear'):
        k = ...
    elif kernel == 'gaussian':
        k = ...
    return k

plt.title('Traning Dataset')
# Parameters of the training
C = 1                  # C regularization value
tolerance = 1e-3       # Tolerance value for thresholds
nb_iterations = 20     # Number of iterations
sigma = 0.1            # Variance of Gaussian kernel
# Train the SVM with a Gaussian (RBF) kernel
model = svm_train(X, y, C, 'gaussian', tolerance, nb_iterations)
# Evaluate the decision function
ax = plt.figure(figsize=(10, 8)).gca()
ax.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor = 'w', cmap='winter')
ax.set_xlim(-1, 4); ax.set_ylim(-1, 6)
xlim = ax.get_xlim(); ylim = ax.get_ylim()
# Create grid to evaluate model
x = np.linspace(xlim[0], xlim[1], 30)
yp = np.linspace(ylim[0], ylim[1], 30)
Y, Xp = np.meshgrid(yp, x)
xy = np.vstack([Xp.ravel(), Y.ravel()]).T
P = decision_function(model, xy.transpose())
P = P.reshape((30, 30))
# plot decision boundary and margins
ax.contour(Xp, Y, P, colors='w',levels=[-1, 0, 1], alpha=0.9,linestyles=['--', '-', '--'])
ax.scatter(model["x"][:, 0], model["x"][:, 1],s=300, linewidth=2, edgecolor='w', facecolors='none');
plt.title('Our trained SVM :)');
```

## Supplementary kernels

Now that you have mastered the concept of kernels and updated your code to include the Gaussian kernel, you can augment your classifier and optimizaton problem to include several different kernels as those proposed afterwards.

#### Polynomial kernel.

Intuitively, the polynomial kernel looks not only at the given features of input samples to determine their similarity, but also combinations of these.

$$
K\left(x,y\right)=\left(x^{T}y+1\right)^{d}
$$

#### Sigmoid (tanh) kernel.

This kernel takes two parameters: $a$ and $r$. For $a > 0$, we can view $a$ as a scaling parameter of the input data, and $r$ as a shifting parameter that controls the threshold of mapping. For $a < 0$, the dot-product of the input data is not only scaled but reversed.

$$
K\left(x,y\right)=tanh\left(kx^{T}y+\Theta\right)
$$

***

**Exercise**

  1. Update the `kernel` function code to include these supplementary kernels.
  2. Test both the `decision_function` and `svm_train` codes for these new kernels.

***

```python

```

## Audio application

As seen in the previous tutorial, we have considered only a *binary classification* problem (where elements can only belong to one of two classes). This simplifies the work as we simply need to separate the space between two types of points. However, in real-world problems, we usually want to solve *multi-class* problems with any given number of classes. For this tutorial, we will rely on a simple trick which is to consider that a $n$-class problem can simply be restated as $n$ different binary classification problems. The underlying idea is that each class defines a binary classifier which tells us if a given element is part of this class, or belongs to *any of the other* classes.

***

**Exercise**

  1. Change the training loop to define $n$separate binary problems
  2. For each problem, train an SVM to perform classification.
  3. Evaluate the global classification accuracy of your system.

***

```python

```

```python
from helper_data import import_dataset, compute_transforms, compute_features
class_path = 'data/classification'
# 0.1 - Import the classification dataset
data_struct = import_dataset(class_path, 'classification')
# 0.2 - Pre-process the audio to obtain spectral transforms
data_struct = compute_transforms(data_struct)
# 0.3 - Compute a set of temporal and spectral features
data_struct = compute_features(data_struct)
```