# ***Introduction to Radar Using Python and MATLAB***
## Andy Harrison - Copyright (C) 2019 Artech House
<br/>

# Delay Line
***

Referring to Figure 4.2, the frequency response of a delay line canceller is given by (Equation 11.5)

\begin{equation}
    \left| H(f)\right|^2 = 4\,\sin^2\left( \pi f \right),
\end{equation}
***

Begin by getting the library path

```python
import lib_path
```

Set up the normalized frequency space using the `linspace` routine from `scipy`

```python
from numpy import linspace

frequency = linspace(0, 4, 1000)
```

Calculate the frequency response for a single delay line canceller using the `delay_line` routine from `countermeasures`

```python
from Libs.ecm.countermeasures import delay_line

response = delay_line(frequency) / 4.0
```

Import the `matplotlib` routines to display the frequency response of the single delay line canceller

```python
from matplotlib import pyplot as plt
```

Import the `log10` and `finfo` routines for plotting the frequency response in deciBels

```python
from numpy import log10, finfo
```

```python
# Set the figure size

plt.rcParams["figure.figsize"] = (15, 10)

# Display the results

plt.plot(frequency, 10 * log10(response + finfo(float).eps), '')

# Set the plot title and labels

plt.title('Delay Line Response', size=14)

plt.xlabel('Normalized Frequency (f / PRF)', size=12)

plt.ylabel('Amplitude (dB)', size=12)

# Set the y-axis lim

plt.ylim([-30, 1])

# Turn on the grid

plt.grid(linestyle=':', linewidth=0.5)

# Set the tick label size

plt.tick_params(labelsize=12)
```

Set the pulse repetition stagger for a stagger type delay line canceller

```python
prf_stagger = 5.0/4.0
```

Calculate the reponse for each PRF separately and then combine to get the final result

```python
response_prf1 = delay_line(frequency) / 4.0

response_prf2 = delay_line(prf_stagger * frequency) / 4.0

response = 0.5 * (response_prf1 + response_prf2)
```

```python
# Set the figure size

plt.rcParams["figure.figsize"] = (15, 10)

# Display the results

plt.plot(frequency, 10 * log10(response_prf1 + finfo(float).eps), '', label='PRF 1')

plt.plot(frequency, 10 * log10(response_prf2 + finfo(float).eps), '--', label='PRF 2')

plt.plot(frequency, 10 * log10(response + finfo(float).eps), ':', label='PRF Staggered')

# Place the legend

plt.legend(loc='lower left', prop={'size': 10})

# Set the plot title and labels

plt.title('Delay Line Response', size=14)

plt.xlabel('Normalized Frequency (f / PRF)', size=12)

plt.ylabel('Amplitude (dB)', size=12)

# Set the y-axis lim

plt.ylim([-30, 1])

# Turn on the grid

plt.grid(linestyle=':', linewidth=0.5)

# Set the tick label size

plt.tick_params(labelsize=12)
```


**1**. (20 points)

Consider the linear transformation $f(x)$ on $\mathbb{R}^3$ that takes the standard basis $\left\{e_1,e_2,e_3\right\}$ to $\left\{v_1,v_2,v_3\right\}$ where

$$v_1=\left(\begin{matrix}10\\-10\\16\end{matrix}\right), v_2=\left(\begin{matrix}2\\-5\\20\end{matrix}\right) \textrm {and } v_3=\left(\begin{matrix}1\\-4\\13\end{matrix}\right)$$

1. Write a matrix $A$ that represents the same linear transformation. (4 points)

2. Compute the rank of $A$ using two different methods (do not use `matrix_rank`!). (4 points)

3. Find the eigenvalues and eigenvectors of $A$. (4 points)

4. What is the matrix representation of $f$ with respect to the eigenbasis? (8 points)

```python
import numpy as np
import scipy.linalg as la
#Question 1
A = np.array([[10,2,1],[-10,-5,-4],[16,20,13]])
A
```

    array([[ 10,   2,   1],
           [-10,  -5,  -4],
           [ 16,  20,  13]])

```python
#Question 2 method 1
l,v = la.eig(A)
len(l[l!=0])
```

    3

```python
#method 2
la.det(A)
#Since determinant not equals 0, A is a on-singular matrix and rank =3
```

    162.0

```python
#methon 3
I = np.eye(3)
x,res,rank,s = la.lstsq(I,A)
rank
```

    3

```python
#method 4:rref
from sympy import *
MA = Matrix([[10,2,1],[-10,-5,-4],[16,20,13]])
rref_a,pivot= MA.rref()
rref_a
#rank(A)=3
```

    Matrix([
    [1, 0, 0],
    [0, 1, 0],
    [0, 0, 1]])

```python
#Question 3
eigenvalues,eigenvectors=la.eig(A)
eigenvalues = np.real_if_close(eigenvalues)
print('eigenvalues:')
print(eigenvalues)
print('eigenvectors:')
print(eigenvectors)
```

    eigenvalues:
    [ 9.  3.  6.]
    eigenvectors:
    [[ 0.57735027  0.         -0.12038585]
     [-0.57735027 -0.4472136  -0.24077171]
     [ 0.57735027  0.89442719  0.96308682]]

```python
#Question 4
#eigenvalue的diagonal或eigenvectors的转换
ans=np.diag(la.eig(A)[0])
ans = np.real_if_close(ans)
ans
```

    array([[ 9.,  0.,  0.],
           [ 0.,  3.,  0.],
           [ 0.,  0.,  6.]])

**2**. (20 points)

You are given the following x-y coordinates (first column is x, second is y)

```
array([[  0.        ,   4.12306991],
       [  3.        , -15.47355729],
       [  4.        , -11.68725507],
       [  3.        , -20.33756693],
       [  5.        ,  -6.06401989],
       [  6.        ,  32.79353057],
       [  8.        ,  82.48658405],
       [  9.        ,  84.02971858],
       [  4.        ,  -1.30587276],
       [  8.        ,  68.59409878]])
```

- Find the coefficients $(a, b, c)$ of the least-squares fit of a quadratic function $y = a + bx + cx^2$ to the data.
- Plot the data and fitted curve using `matplotlib`.

Note: Use `numpy.linalg.leastsq` function to solve this.

```python
coordinates = np.array(
    [[  0.        ,   4.12306991],
       [  3.        , -15.47355729],
       [  4.        , -11.68725507],
       [  3.        , -20.33756693],
       [  5.        ,  -6.06401989],
       [  6.        ,  32.79353057],
       [  8.        ,  82.48658405],
       [  9.        ,  84.02971858],
       [  4.        ,  -1.30587276],
       [  8.        ,  68.59409878]])
y = coordinates[:,1].reshape(-1,1)
x = coordinates[:,0].reshape(-1,1)
x_square = x*x
X = np.column_stack((x,x_square))
final_x = np.column_stack((np.ones(x.shape[0]).reshape(-1,1),X))
#final_x
coefficient, res, rank, s, = np.linalg.lstsq(final_x,y,rcond=None)
#if the 'coefficient command above cannot work, please try the code below. Since these two codes works differently on my own laptop and vm.Thanks a lot.'
#coefficient, res, rank, s, = np.linalg.lstsq(final_x,y)
print('a=',coefficient[0][0])
print('b=',coefficient[1][0])
print('c=',coefficient[2][0])
```

    a= -0.357628957528
    b= -11.7853123187
    c= 2.53125198581

```python
import matplotlib.pyplot as plt
def func(x,coefficient):
    a = coefficient[0][0]
    b = coefficient[1][0]
    c = coefficient[2][0]
    return a+b*x+c*x*x

plot_x = np.linspace(0,10,50)
plot_y = func(plot_x,coefficient)
plt.plot(x, y, 'o', label='Original data', markersize=5)
plt.plot(plot_x, plot_y, 'r', label='Fitted line')
plt.legend()
plt.show()
```

**3**. (20 points)

Consider the following system of equations:

$$\begin{align*}
2x_1& - x_2&    +x_x    &=& 6\\
-x_1& +2x_2& -  x_3 &=& 2\\
 x_1   &  -x_2& + x_3 &=& 1
\end{align*}$$

1. Consider the system in matrix form $Ax=b$ and define $A$, $b$ in numpy.
2. Show that $A$ is positive-definite
3. Use the appropriate matrix decomposition function in numpy and back-substitution to solve the system. Remember to use the structure of the problem to determine the appropriate decomposition.

```python
#define A,b
A = np.array(
            [[2,-1,1],
             [-1,2,-1],
            [1,-1,1]])
b = np.array([6,2,1]).reshape(-1,1)
print('A:')
print(A)
print('b: ')
print(b)
```

    A:
    [[ 2 -1  1]
     [-1  2 -1]
     [ 1 -1  1]]
    b:
    [[6]
     [2]
     [1]]

```python
#eigen value are all positive, the matrix is positive-definite
value,vextor = la.eig(A)
value
```

    array([ 3.73205081+0.j,  1.00000000+0.j,  0.26794919+0.j])

```python
#Since A is a positive definite and symmetric matrix, u^TAu>0 and A=A^T ,it is appropriate to use cholesky decomposition
C = la.cholesky(A)
C
```

    array([[ 1.41421356, -0.70710678,  0.70710678],
           [ 0.        ,  1.22474487, -0.40824829],
           [ 0.        ,  0.        ,  0.57735027]])

```python
C1 = la.cho_factor(A)
la.cho_solve(C1, b)
```

    array([[ 5.],
           [ 3.],
           [-1.]])

**4**. (40 points)

You are given the following set of data to fit a quadratic polynomial to

```python
x = np.arange(10)
y = np.array([  1.58873597,   7.55101533,  10.71372171,   7.90123225,
                -2.05877605, -12.40257359, -28.64568712, -46.39822281,
                -68.15488905, -97.16032044])
```

- Find the least squares solution by using the normal equations $A^T A \hat{x} = A^T y$. (5 points)

- Write your own **gradient descent** optimization function to find the least squares solution for the coefficients $\beta$ of a quadratic polynomial. Do **not** use a gradient descent algorithm from a package such as `scipy-optimize` or `scikit-learn`.  You can use a simple for loop - start with the parameters `beta = np.zeros(3)` with a learning rate $\alpha = 0.0001$ and run for 100000 iterations. (15 points)

- Plot the data together with the fitted polynomial. (10 points)

```python
#normal equation
x = np.arange(10)
y = np.array([  1.58873597,   7.55101533,  10.71372171,   7.90123225,
                -2.05877605, -12.40257359, -28.64568712, -46.39822281,
                -68.15488905, -97.16032044])
y = y.reshape(-1,1)
x = x.reshape(-1,1)
x_square = x*x
X = np.column_stack((x,x_square))
X = np.column_stack((np.ones(x.shape[0]).reshape(-1,1),X))
la.solve(X.T@X, X.T@y)
```

    array([[ 2.55079998],
           [ 7.31478229],
           [-2.04118936]])

```python
#gradient descent
def gradientDescent(β,X,y,learning_rate,iters,m):
    for i in range(iters):
        y_model = X@β #fitted y
        error = y_model-y
        gradient = np.dot(X.T,error)/m
        β -= gradient*learning_rate
    return β

β = np.zeros(3).reshape(-1,1)
m = 10 #obs
learning_rate = 0.0001
iters = 100000
coeff = gradientDescent(β,X,y,learning_rate,iters,m)
coeff
```

    array([[ 2.73393241],
           [ 7.23151702],
           [-2.03359595]])

```python
plt.scatter(x,y,label='Original data')
plt.plot(x, coeff[1]*x + coeff[2]*x**2 + coeff[0], 'r', label='Fitted line')
plt.legend()
plt.show()
```


# "The Annotated TabNet"
> Creating TabNet from Scratch in Tensorflow 2.0

- toc: true
- branch: master
- badges: true
- comments: true
- author: Sachin Abeywardana
- categories: [Tensorflow]
- use_math: true
- image: images/tabnet.png

## Introduction
We are talking about [TabNet](https://arxiv.org/abs/1908.07442) today which is a network designed for Tabular data. One aspect that tree based models such as Random Forest (RF) and XgBoost can claim over Neural Nets is the explainability of the model.

Personally, one of the coolest features of this network is the ability for the network to point out which features were used in it's final decision. And these features change from input to input. We have a visual demo using MNIST below. RF does not have this kind of flexibility and is only one static graph for any kind of input.

## Acknowledgement
Kudos to [Paul Tune](https://www.linkedin.com/in/paul-tune-0ba18116/) for pointing out how to configure the loss function in keras. In terms of the actual implementation it was done very much thanks to this [YouTube talk](https://www.youtube.com/watch?v=ysBaZO8YmX8&ab_channel=AbhishekThakur) done by [Sebastien Fischman](https://www.linkedin.com/in/sebastienfischman/) and hosted by [Abhishek Thakur](https://www.linkedin.com/in/abhi1thakur/). Much of the TensorFlow code was very much a reimplimentation of the [PyTorch code](https://github.com/dreamquark-ai/tabnet/blob/develop/pytorch_tabnet/tab_network.py).

```python
# hide
!pip install tensorflow_addons
```

    Collecting tensorflow_addons
      Downloading https://files.pythonhosted.org/packages/74/e3/56d2fe76f0bb7c88ed9b2a6a557e25e83e252aec08f13de34369cd850a0b/tensorflow_addons-0.12.1-cp37-cp37m-manylinux2010_x86_64.whl (703kB)
         |████████████████████████████████| 706kB 5.5MB/s
    Requirement already satisfied: typeguard>=2.7 in /usr/local/lib/python3.7/dist-packages (from tensorflow_addons) (2.7.1)
    Installing collected packages: tensorflow-addons
    Successfully installed tensorflow-addons-0.12.1

```python
# hide
import multiprocessing as mp

import matplotlib.pyplot as plt
import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow_addons.activations import sparsemax

%matplotlib inline
```

## Model
This section will outline the blocks that we have used in creating the overall model. See the [diagram](tabnet_diagram) above to refer to the sections mentioned below. We will not be talking about part (b) of the diagram as that from my understanding is only used for completing missing values.

### Fully-Connected Block
Let's look at section (c) of the diagram and especially one set of the blue-green-red boxes and call that one Fully-Connected layer. Instead of ReLU the authors have used a GLU activation (the calculation of which is shown in the first two lines). The `FCBlock` will form as one of the building blocks of the entire architecture.

```python
def GLU(x):
    return x * tf.sigmoid(x)

class FCBlock(layers.Layer):
    def __init__(self, units):
        super().__init__()
        self.layer = layers.Dense(units)
        self.bn = layers.BatchNormalization()

    def call(self, x):
        return GLU(self.bn(self.layer(x)))
```

### Feature Transformer
Next bit of the feature transformer in section (c) is the Shared Block and the Decision Block which look extrememly similar. This is why I sub-classed `DecisionBlock` to have parent `SharedBlock`. Also worth mentioning that this is all possible thanks to the new features of TensorFlow 2.0+.

The shared block is simply stacking two `FCBlock`s with a residual connection which is multiplied by `sqrt(0.5)`. I'm not sure if this multiplication is necessary but I will leave it in.

The decision block is almost the same except that there are two residual connections.

Now in terms of functionality, refer to section (a) of the diagram. The user needs to define the number of steps as one of the hyper-parameters. The feature transformer is repeated number of steps + 1 many times. In this, the `SharedBlock` weights are shared across all of that, whereas the `DecisionBlock` weights are different for each step.

```python
class SharedBlock(layers.Layer):
    def __init__(self, units, mult=tf.sqrt(0.5)):
        super().__init__()
        self.layer1 = FCBlock(units)
        self.layer2 = FCBlock(units)
        self.mult = mult

    def call(self, x):
        out1 = self.layer1(x)
        out2 = self.layer2(out1)
        return out2 + self.mult * out1

class DecisionBlock(SharedBlock):
    def __init__(self, units, mult=tf.sqrt(0.5)):
        super().__init__(units, mult)

    def call(self, x):
        out1 = x * self.mult + self.layer1(x)
        out2 = out1 * self.mult + self.layer2(out1)
        return out2
```

### Attentive Transformer
This is the part where the magic happens. The attentive transformer decides which bits of the input features (x) it needs to pay attention (mask) at each step.

Before, talking about the Attentive transformer we need to talk about the `split` module and the `prior` layers. The split module simply splits the output of the feature transformer into two portions. One portion which feeds into the attentive transformer, and the other which goes into the output of our overall network. The portions (`n_d, n_a`) are hyper-parameters that the user needs to specify and it would sum to the number of output nodes of the decision layer.

The attentive layer takes the `n_a` output nodes from the decision block, runs it through a dense layer and batch norm layer before passing through a `sparsemax` layer. Sparsemax is similar to softmax in that the output sums to one. However, it drives some of the smaller outputs to exactly zero unlike softmax (which can only get close to zero). It is important to note that the dense layer used here projects `n_a` nodes to the same dimensionality as the input features, as we will use this as a mask to select features.

The prior layer is used as a tool to suppress some of the inputs that were used before in a previous step. In the first step none of the input had been used before, and therefore the output of the attentive transformer is unaffected by the prior. However, in the second step (and onwards), the prior is updated to be `prior = previous_prior * (constant - previous_mask)`. The constant mentioned here is a number close to one. Therefore, if the `previous_mask` output by the attentive mask had numbers close to one, the prior is pushed closer to zero, whereas the unused inputs having a mask close to zero, would have a prior closer to one. The priors closer to zero would effectively act as a suppressant before passing into the sparsemax activation in the attentive layer.

The reason that we attempt to use this kind of prior is so that each step attempts to find unique features which adds to the output of the network.

```python
class Prior(layers.Layer):
    def __init__(self, gamma=1.1):
        super().__init__()
        self.gamma = gamma

    def reset(self):
        self.P = 1.0

    def call(self, mask):
        self.P = self.P * (self.gamma - mask)
        return self.P
```

```python
class AttentiveTransformer(layers.Layer):
    def __init__(self, units):
        super().__init__()
        self.layer = layers.Dense(units)
        self.bn = layers.BatchNormalization()

    def call(self, x, prior):
        return sparsemax(prior * self.bn(self.layer(x)))
```

### Putting it All Together
Much of the architecture has already been discussed in the two previous sections. However, there is two more missing pieces.

Firstly, the loss function. In additional to the loss function that is used for the task (CrossEntropy in the case of MNIST classification), there is an additional loss on the mask values to drive the values towards either 0 or 1. The (entropy) loss is defined as follows:
$$
\begin{align}
L_{mask} = - M \log (M + \epsilon)
\end{align}
$$
The plot below shows entropy as a function of one mask value. We average out all available mask entropy values to get final loss. Note how they are minimised at either extreme.

```python
#hide_input
p = np.arange(0.01, 1, 0.01)
e = - p * np.log(p)
plt.plot(p, e)
plt.xlabel("mask")
plt.ylabel("entropy")
plt.show()
```

The second part that we have not discussed so far is the actual output of the model. The `n_d` number of inputs that do not go through the attentive layer gets passed through a ReLU activation at each step before being added up for the final output. This is compared against a target by using a task specific loss function (cross-entropy in our case).

One other thing I'd like to bring your attention to (see what I did there?) is the fact that I used a for loop inside the `call` function in the Module below. It might seem like a minor thing, but this kind of flexibility in building a model was not available before to my knowledge.

### Mask Importance
The whole point of building this model was to be able to select features and to be able to explain the model. The way that we calculate feature importance is by calculating the importance of each step, and by multiplying that by the mask.

Step importance is calculated as the sum of the (`n_d`) outputs of the feature transformer.

```python
# collapse
class TabNet(keras.Model):
    def __init__(self, input_dim, output_dim, steps, n_d, n_a, gamma=1.3):
        super().__init__()
        # hyper-parameters
        self.n_d, self.n_a, self.steps = n_d, n_a, steps
        # input-normalisation
        self.bn = layers.BatchNormalization()
        # Feature Transformer
        self.shared = SharedBlock(n_d+n_a)
        self.first_block = DecisionBlock(n_d+n_a)
        self.decision_blocks = [DecisionBlock(n_d+n_a)] * steps
        # Attentive Transformer
        self.attention = [AttentiveTransformer(input_dim)] * steps
        self.prior_scale = Prior(gamma)
        # final layer
        self.final = layers.Dense(output_dim)

        self.eps = 1e-8
        self.add_layer = layers.Add()

    @tf.function
    def call(self, x):
        self.prior_scale.reset()
        final_outs = []
        mask_losses = []

        x = self.bn(x)
        attention = self.first_block(self.shared(x))[:,:self.n_a]
        for i in range(self.steps):
            mask = self.attention[i](attention, self.prior_scale.P)
            entropy = mask * tf.math.log(mask + self.eps)
            mask_losses.append(
                -tf.reduce_sum(entropy, axis=-1) / self.steps
            )

            prior = self.prior_scale(mask)
            out = self.decision_blocks[i](self.shared(x * prior))
            attention, output = out[:,:self.n_a], out[:,self.n_a:]
            final_outs.append(tf.nn.relu(output))

        final_out = self.add_layer(final_outs)
        mask_loss = self.add_layer(mask_losses)

        return self.final(final_out), mask_loss

    def mask_importance(self, x):
        self.prior_scale.reset()
        feature_importance = 0

        x = self.bn(x)
        attention = self.first_block(self.shared(x))[:,:self.n_a]
        for i in range(self.steps):
            mask = self.attention[i](attention, self.prior_scale.P)

            prior = self.prior_scale(mask)
            out = self.decision_blocks[i](self.shared(x * prior))
            attention, output = out[:,:self.n_a], out[:,self.n_a:]
            step_importance = tf.reduce_sum(tf.nn.relu(output), axis=1, keepdims=True)
            feature_importance += mask * step_importance

        return feature_importance
```

```python
# hide
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
x_train = x_train.reshape((len(x_train), -1)).astype(np.float32)
x_test = x_test.reshape((len(x_test), -1)).astype(np.float32)
```

## Loss and Fitting
Now this is the part that I thought was strange in the keras API, unless I'm doing something wrong. Looking at the `TabNet` class above, it returns the actual prediction, and the `mask_loss`. From my understanding of loss functions in keras, it is always the "true y" and the "predictions" over which we need to define a loss.

As shown below my initial attempt was to unpack the predictions into logits and mask losses, and then in the next line to take the cross entropy and average out the mask loss. This however resulted in an error.

What did work however, was to define `mask_loss` function defined below which ignores the `y_true` input, and to add the `loss_weights` during `model.compile`. It seems like attempting to add a breakpoint and debug does not seem to work after you run `model.compile`.

```python
# collapse
from keras.losses import SparseCategoricalCrossentropy

sce = SparseCategoricalCrossentropy(from_logits=True)
reg_sparse = 0.01
def full_loss(y_true, y_pred):
    logits, mask_loss = y_pred
    return sce(y_true, logits) + reg_sparse * mask_loss.mean()
```

```python
# collapse
def mask_loss(y_true, mask_losses):
    return tf.reduce_mean(mask_losses)

model = TabNet(784, 10, 2, 10, 10, 1.3)
model.compile(
    'Adam',
    loss=[sce, mask_loss],
    loss_weights=[1, 0.01]
)
model.fit(
    x_train,
    y_train,
    epochs=5,
    batch_size=256,
    validation_split=0.15,
    workers=mp.cpu_count()
)
```

    Epoch 1/5
    200/200 [==============================] - 24s 96ms/step - loss: 1.7993 - output_1_loss: 1.7842 - output_2_loss: 1.5173 - val_loss: 0.6537 - val_output_1_loss: 0.6461 - val_output_2_loss: 0.7665
    Epoch 2/5
    200/200 [==============================] - 19s 94ms/step - loss: 0.5893 - output_1_loss: 0.5828 - output_2_loss: 0.6446 - val_loss: 0.2964 - val_output_1_loss: 0.2923 - val_output_2_loss: 0.4080
    Epoch 3/5
    200/200 [==============================] - 19s 94ms/step - loss: 0.2887 - output_1_loss: 0.2849 - output_2_loss: 0.3869 - val_loss: 0.2182 - val_output_1_loss: 0.2151 - val_output_2_loss: 0.3119
    Epoch 4/5
    200/200 [==============================] - 19s 95ms/step - loss: 0.2148 - output_1_loss: 0.2118 - output_2_loss: 0.3055 - val_loss: 0.1908 - val_output_1_loss: 0.1882 - val_output_2_loss: 0.2625
    Epoch 5/5
    200/200 [==============================] - 19s 94ms/step - loss: 0.1764 - output_1_loss: 0.1737 - output_2_loss: 0.2686 - val_loss: 0.1775 - val_output_1_loss: 0.1751 - val_output_2_loss: 0.2383

    <tensorflow.python.keras.callbacks.History at 0x7f6402c12510>

## Results
Model has 91.8% accuracy on test set. See below for calculation.

```python
# collapse
y_pred = model.predict(x_test)[0].argmax(axis=-1)
(y_pred == y_test).mean()
```

    0.918

Let's see what features got picked up for the first ten images:

```python
#hide_input
N = 5
query = x_test[:N]
importance = model.mask_importance(query).numpy()
y_pred = model.predict(query)[0].argmax(axis=-1)

for k in range(N):
    fig, ax = plt.subplots(1, 3)
    img = x_test[k].reshape(28, 28)
    feature_importance = importance[k].reshape(28, 28)
    num_nonzero = (importance[k] != 0).sum()

    ax[0].imshow(img, cmap="gray")
    ax[0].set_title(f"Prediction {y_pred[k]}")
    ax[1].imshow(feature_importance)
    ax[1].set_title(f"{num_nonzero} important features")

    ax[2].imshow(img / 500 + (feature_importance!=0).astype(np.float))
    fig.show()
```

As can be seen in most images the important features on the actual digit itself. However, with the first two images above there are no important features in the actual digit.

It is worth noting that the MNIST digit is not a binary image and even in the far corners the pixel values may not be exactly zero. Alternatively, it is also fathomable to think that the net is making sure that the pixels are zero in certain parts of the image to ensure that the digit is of a certain shape.

## Conclusion
Hopefully that demystified TabNets. I personally feel like the sparsemax activation is not explored enough in other areas of DL and probably has more to contribute in the coming years.

In terms of TabNet, it's great to see explainability being the pure focus of a paper. The experiments conducted in the paper claims that it has beaten XgBoost in some benchmarks and hopefully TabNet will be the gold standard in this space.

Finally, it was really good to play with Tensorflow 2 in a familiar environment to pytorch users.

## Shameless Self Promotion
See [here for my course](https://www.udemy.com/course/machine-learning-and-data-science-2021/?couponCode=DEEPSCHOOL-MARCH) on Machine Learning and Deep Learning (Use code DEEPSCHOOL-MARCH to 85% off).

```python

```


#### **Table of Content**
Static model
* [Free-energy](#free energy)
* [Mean-field assumption](#mean-field approximation)
* [Laplace approximation](#laplace approximation)
* [Optimisation](#optimisation)
* [Implementational consideration](#implementational consideration)
* [Example 1](#example1)
* [Example 2](#example2)
* [Example 3](#example3)

Appendix
* [Multivariate normal distribution](#appendix1)
* [Linearisation](#appendix3)

$\DeclareMathOperator{\tr}{tr}$

# Laplace approximation & free-energy

### § Free-energy

The Fristonian free-energy is

$$
F = U - S
$$

which comprises an energy term

$$
U = \left\langle \ln p(y, \vartheta|m) \right\rangle_q = \left\langle L \right\rangle_q
$$

and a (negative) Shannon entropy term

$$
S = \left\langle \ln q(\vartheta)\right\rangle
$$

### § Mean-field assumption

With mean-field approximation over the variational density, $q$

$$
q(\vartheta) = \prod_i q_i(\vartheta_i) = \prod_i q_i.
$$

Let us for now assume the following case

$$
q(\vartheta) = q_i(\vartheta_i) q_j(\vartheta_j) = q_i q_j,\;\;\; j = \backslash i
$$

and note that the following satisfies $\delta F / \delta q = 0$

$$
\begin{align}
    \ln q_i &= \ln Z_i + V(\vartheta_i) = \ln Z_i + \left\langle L(\vartheta)\right\rangle_{q_j}\\
    \end{align}.
$$

where $V_i=V(\vartheta_i)$ is known as the _variational energy_ and speask to the joint probability of data and parameters, $L$, expected under its Markov blanket, $q_j$.

### § Laplace approximation

Under Laplace assumption, let us suppose that

$$
\begin{eqnarray*}
    q_i &=& N(\vartheta_i| \mu_i, \Sigma_i)\\
    \end{eqnarray*}
$$

are Gaussian, where

$$
\begin{eqnarray*}
    \Sigma_i &=& \left\langle (\vartheta_i - \mu_i)(\vartheta_i - \mu_i)^T\right\rangle
    \end{eqnarray*}
$$

is the covariance matrix.

Following the factorisation, we write

$$
\begin{eqnarray*}
    F &=& U - H\\
    U &=& \left\langle L(\vartheta)\right\rangle_{q_i q_j}\\
    H &=& \left\langle\ln q_i\right\rangle_{q_i} + \left\langle\ln q_j\right\rangle_{q_j}
    \end{eqnarray*}
$$

The (neg-)entropies are (see note [A.1](#appendix1))

$$
\begin{eqnarray*}
    H &=& -\frac{D_i}{2}(1 + \ln 2\pi) -
        \frac 1 2\ln|\Sigma_i| - \frac{D_j}{2}(1 + \ln 2\pi) -\frac 1 2 \ln|\Sigma_j|\\
    &=& -\frac{D_i}{2}\ln 2\pi e -\frac 1 2\ln|\Sigma_i| - \frac{D_j}{2}\ln 2\pi e -\frac 1 2\ln|\Sigma_j|
    \end{eqnarray*}
$$

Further, with second-order truncation (dropping bilinear terms)

$$
\begin{eqnarray*}
    L(\vartheta) &\approx&
        L(\mu) +
        (\vartheta_i - \mu_i)^T L^{(\vartheta_i)} +
        (\vartheta_j - \mu_j)^T L^{(\vartheta_j)} \\
    && +
        \frac 1 2\left[
        (\vartheta_i - \mu_i)^T L^{(\vartheta_i\vartheta_i)} (\vartheta_i - \mu_i) +
        (\vartheta_j - \mu_j)^T L^{(\vartheta_j\vartheta_j)} (\vartheta_j - \mu_j)
        \right]\\
    &=& L(\mu) +
        (\vartheta_i - \mu_i)^T L^{(\vartheta_i)} +
        (\vartheta_j - \mu_j)^T L^{(\vartheta_j)} \\
    && +
        \frac 1 2\left\{
        \tr\left[(\vartheta_i - \mu_i) (\vartheta_i - \mu_i)^T L^{(\vartheta_i\vartheta_i)}\right] +
        \tr\left[(\vartheta_j - \mu_j) (\vartheta_j - \mu_j)^T L^{(\vartheta_j\vartheta_j)}\right]
        \right\}
        \end{eqnarray*}
$$

where $L^{(\vartheta_i\vartheta_i)}$ denotes $\partial^2 L/\partial\vartheta_i\partial \vartheta_i$ evaluated at its mode $\mu$,

and write

$$
\begin{eqnarray*}
    U &=& L(\mu) +
        \frac 1 2
        \left\{
        \tr\left[
        \langle (\vartheta_i - \mu_i) (\vartheta_i - \mu_i)^T\rangle L^{(\vartheta_i \vartheta_i)}
        \right] +
        \tr\left[
        \langle (\vartheta_j - \mu_j) (\vartheta_j - \mu_j)^T\rangle L^{(\vartheta_j \vartheta_j)}
        \right]
        \right\}\\
    &=& L(\mu) + \frac 1 2 \left\{
        \tr\left[\Sigma_i L^{(\vartheta_i \vartheta_i)}\right] +
        \tr\left[\Sigma_j L^{(\vartheta_j \vartheta_j)}\right]
        \right\}
    \end{eqnarray*}
$$

Here, one uses the property: $a^TCa = \tr[aa^TC]$.

Thus, the free-energy becomes

$$
\DeclareMathOperator{\tr}{tr}
F = L(\mu) + \frac 1 2\left[
    \tr\left(\Sigma_i L^{(\vartheta_i \vartheta_i)}\right) +
    \tr\left(\Sigma_j L^{(\vartheta_j \vartheta_j)}\right)
    \right] +
    \frac{D_i}{2}\ln 2\pi e +
    \frac 1 2\ln|\Sigma_i| +
    \frac{D_j}{2}\ln 2\pi e +\frac 1 2\ln|\Sigma_j|
$$

Solve for $\Sigma_i$ (or just by examination through completing the square)

$$
\begin{eqnarray*}
    \delta F/\delta\Sigma_i &=& 0 =
        \frac 1 2 {L^{(\vartheta_i \vartheta_i)}}^T + \frac 1 2 \Sigma_i^{-1}\\
        \Sigma_i^{-1} &=& - {L^{(\vartheta_i \vartheta_i)}}^T = - L^{(\vartheta_i \vartheta_i)}
    \end{eqnarray*}
$$

Note that the same procedure can be applied to individual variational density to give

$$
\begin{eqnarray*}
    \ln q(\vartheta_i) &\approx& V(\vartheta_i)\\
    &=& \left\langle L(\vartheta_i, \mu_j) +
        \frac 1 2
        (\vartheta_j - \mu_j)^T
        L^{(\vartheta_j \vartheta_j)}(\vartheta_i, \mu_j)
        (\vartheta_i - \mu_j)
        \right\rangle_{q_j}\\
    V(\vartheta_i, \mu_j) &=& L(\vartheta_i, \mu_j) +
        \frac 1 2 \tr\left(\Sigma_j L^{(\vartheta_j \vartheta_j)}\right)
    \end{eqnarray*}
$$

This is useful when optimse for variational mode $\mu_i$.

### § Optimsation

To optimise the sufficient statistics for the variational density, $q$, under Laplace approximation, we only need to find their mean and covariance, which are illustrated in the preceding sections

$$
\begin{eqnarray*}
    \mu_i &=& \underset{\vartheta_i}{\operatorname{argmax}} V(\vartheta_i)\\
    \Sigma_i &=& - L(\mu)^{(\vartheta_j \vartheta_j)^{-1}}
    \end{eqnarray*}
$$

It is worth noting that the covariance is a function of its variational mode and does not depend on the mean-field approximation.

* Optimise for $\mu_i$

The update scheme should read
$$
\DeclareMathOperator{\tr}{tr}
\begin{eqnarray*}
    V(\mu_i)^{(\vartheta_{i:k})} &=& L(\mu)^{(\vartheta_{i:k})} +
        \frac 1 2 \tr\left(\Sigma_j L^{(\vartheta_j \vartheta_j \vartheta_{i:k})}\right)\\
    V(\mu_i)^{(\vartheta_{i:k}\vartheta_{i:l})} &=&
        L(\mu)^{(\vartheta_{i:k}\vartheta_{i:l})} +
        \frac 1 2 \tr\left(\Sigma_j L^{(\vartheta_{j}\vartheta_{j}\vartheta_{i:k}\vartheta_{i:l})}\right)\\
    \Delta\mu_i &=& (\exp(tJ) - I)J^{-1}V(\mu_i)^{(\vartheta_i)}\\
    &\approx& -V(\mu_i)^{(\vartheta_{i}\vartheta_{i})^{-1}} V(\mu_i)^{(\vartheta_{i})}, \;\;\;\hbox{if }t \to \infty
    \end{eqnarray*}
$$

where superscript $(\vartheta_{i:k})$ denotes partial derivative to the $k$-th element of $\vartheta_i$. For detailed derivation see _[linearisation](#appendix3)_.

### § Implementational consideration

Suppose one has parameter that factorises into $\vartheta = \{\theta, \varphi\}$

and let $\theta=(\theta_1, \theta_2)^T$ and $\varphi = (\varphi_1, \varphi_2)^T$

$$
\DeclareMathOperator{\tr}{tr}
\begin{eqnarray*}
    V^{(\varphi\varphi)} &=& \left(
        \begin{array}{cc}
            V^{(\varphi_1\varphi_1)} &
            V^{(\varphi_1\varphi_2)} \\
            V^{(\varphi_2\varphi_1)} &
            V^{(\varphi_2\varphi_2)}
            \end{array}\right)\\
    &=& \left(
        \begin{array}{cc}
            L^{(\varphi_1\varphi_1)} &
            L^{(\varphi_1\varphi_2)} \\
            L^{(\varphi_2\varphi_1)} &
            L^{(\varphi_2\varphi_2)}
            \end{array}\right) +
        \frac{1}{2}\left(
        \begin{array}{cc}
            \tr\left[\Sigma_\theta L^{(\theta\theta\varphi_1\varphi_1)}\right] &
            \tr\left[\Sigma_\theta L^{(\theta\theta\varphi_1\varphi_2)}\right] \\
            \tr\left[\Sigma_\theta L^{(\theta\theta\varphi_2\varphi_1)}\right] &
            \tr\left[\Sigma_\theta L^{(\theta\theta\varphi_2\varphi_2)}\right]
            \end{array}\right)
    \end{eqnarray*}
$$

Unpack $L^{(\theta\theta\varphi_1\varphi_1)}$ in the second term

$$
\left(\begin{array}{cc}
    L^{(\theta_1\theta_1\varphi_1\varphi_1)} &
    L^{(\theta_1\theta_2\varphi_1\varphi_1)} \\
    L^{(\theta_2\theta_1\varphi_1\varphi_1)} &
    L^{(\theta_2\theta_2\varphi_1\varphi_1)}
    \end{array}\right)
$$

It is straightforward to show that, for instance,
$\tr\left[\Sigma_\theta L^{(\theta\theta\varphi_1\varphi_2)}\right]$
is equivalent to $\Sigma^T\!\!\!:_\theta^T L\!\!:^{(\theta\theta)(\varphi_1\varphi_2)}$,
where $\Sigma\!\!:$ denotes matrix vectorisation,
and $\Sigma^T\!\!\!:$ stands for vectorisation along columns.

Thus, $L\!\!:^{(\theta\theta)(\varphi\varphi)}$ becomes a matrix that reads

\begin{align}
    \left(\begin{array}{cc}
        L\!\!:^{(\theta\theta)(\varphi_1\varphi_1)} &
        L\!\!:^{(\theta\theta)(\varphi_1\varphi_2)} \\
        L\!\!:^{(\theta\theta)(\varphi_2\varphi_1)} &
        L\!\!:^{(\theta\theta)(\varphi_2\varphi_2)}
        \end{array}
        \right) =
    \left(\begin{array}{cc}
        L^{(\theta_1\theta_1\varphi_1\varphi_1)} &
        L^{(\theta_1\theta_1\varphi_1\varphi_2)} \\
        L^{(\theta_2\theta_1\varphi_1\varphi_1)} &
        L^{(\theta_2\theta_1\varphi_1\varphi_2)} \\
        L^{(\theta_1\theta_2\varphi_1\varphi_1)} &
        L^{(\theta_1\theta_2\varphi_1\varphi_2)} \\
        L^{(\theta_2\theta_2\varphi_1\varphi_1)} &
        L^{(\theta_2\theta_2\varphi_1\varphi_2)} \\
        L^{(\theta_1\theta_1\varphi_2\varphi_1)} &
        L^{(\theta_1\theta_1\varphi_2\varphi_2)} \\
        L^{(\theta_2\theta_1\varphi_2\varphi_1)} &
        L^{(\theta_2\theta_1\varphi_2\varphi_2)} \\
        L^{(\theta_1\theta_2\varphi_2\varphi_1)} &
        L^{(\theta_1\theta_2\varphi_2\varphi_2)} \\
        L^{(\theta_2\theta_2\varphi_2\varphi_1)} &
        L^{(\theta_2\theta_2\varphi_2\varphi_2)} \\
        \end{array}
        \right)
    \end{align}

and one accordingly has

$$
V^{(\varphi\varphi)} = L^{(\varphi\varphi)} + \frac 1 2
    \left( I\otimes \Sigma^T\!\!\!:_\theta\right)^T
    L\!\!:^{(\theta\theta)(\varphi\varphi)}
$$

The same procedure can be repeated for $V^{(\theta\theta)}, V^{(\varphi)}$ and $V^{(\theta)}$ to give

$$
\begin{eqnarray*}
    V^{(\theta\theta)} &= L^{(\theta\theta)}& +
        \frac 1 2
        \left( I\otimes \Sigma^T\!\!\!:_\varphi\right)^T
        L\!\!:^{(\varphi\varphi)(\theta\theta)}\\
    V^{(\varphi)} &= L^{(\varphi)}& +
        \frac 1 2
        \left( I\otimes \Sigma^T\!\!\!:_\theta\right)^T
        L\!\!:^{(\theta\theta)(\varphi)}\\
    V^{(\theta)} &= L^{(\theta)}& +
        \frac 1 2
        \left( I\otimes \Sigma^T\!\!\!:_\varphi\right)^T
        L\!\!:^{(\varphi\varphi)(\theta)}\\
    \end{eqnarray*}
$$

### § Example 1

In this example we will attempt to optimise the following objective function using the updating scheme above.

$
L(u, v) = -\frac{1}{16} (u - \frac{1}{8}v^2)^2 - \frac{1}{128} (v^2 - 2)^2
$

```python
import numpy as np
import scipy.stats as stat
import matplotlib.pyplot as plt
from IPython import display
import time
%matplotlib inline
```

```python
import theano
import theano.tensor as T

print("Theano version: ", theano.__version__)
```

    Using gpu device 0: Tesla K40c (CNMeM is enabled with initial size: 90.0% of memory, cuDNN Version is too old. Update to v5, was 2000.)

    Theano version:  0.9.0dev1.dev-ec0419a6cf1d3667ece1d5e0f95498c229471f19

```python
c0 = [1. / 16, 1. / 8, 1. / 128]
u, v = T.vectors(2)
Su, Sv = T.matrices(2)  # this is not necessary, only for the purpose of tutorial

L = (-c0[0] * (u - c0[1] * v ** 2) ** 2 - c0[2] * (v ** 2 - 2) ** 2)[0]
```

```python
fSu, fSv = Su.flatten(1), Sv.flatten(1)

Lu = T.jacobian(L, u)
Lv = T.jacobian(L, v)

Luu = T.hessian(L, u).flatten(1)
Lvv = T.hessian(L, v).flatten(1)

Luuv = T.join(0, [T.jacobian(Luu[i], v) for i in range(1)])
Lvvu = T.join(0, [T.jacobian(Lvv[i], u) for i in range(1)])

Luuvv = T.join(0, [T.hessian(Luu[i], v).flatten(1) for i in range(1)])
Lvvuu = T.join(0, [T.hessian(Lvv[i], u).flatten(1) for i in range(1)])

Uu = T.join(0, [T.sum(fSv * Lvvu[:, i]) for i in range(1)])
Uv = T.join(0, [T.sum(fSu * Luuv[:, i]) for i in range(1)])

Uuu = T.join(0, [T.sum(fSv * Lvvuu[:, i]) for i in range(1)])
Uvv = T.join(0, [T.sum(fSu * Luuvv[:, i]) for i in range(1)])

Vu = Lu + 0.5 * Uu
Vv = Lv + 0.5 * Uv

Vuu = (Luu + 0.5 * Uuu).reshape((1, 1))
Vvv = (Lvv + 0.5 * Uvv).reshape((1, 1))

F = L + 0.5 * (Su.dot(Luu) + Sv.dot(Lvv)) + T.log(T.nlinalg.det(Su)) + T.log(T.nlinalg.det(Sv))

du = -(T.nlinalg.pinv(Vuu).dot(Vu))
dv = -(T.nlinalg.pinv(Vvv).dot(Vv))

# alternatively
# du = (T.exp(Vuu) - 1).dot(T.nlinalg.pinv(Vuu).dot(Vu))
# dv = (T.exp(Vvv) - 1).dot(T.nlinalg.pinv(Vvv).dot(Vv))

Cu = - T.nlinalg.pinv(Luu.reshape((1, 1)))
Cv = - T.nlinalg.pinv(Lvv.reshape((1, 1)))
```

```python
update = theano.function([u, v, Su, Sv], [du, dv, Cu, Cv], allow_input_downcast=True)
elb = theano.function([u, v, Su, Sv], F, allow_input_downcast=True)
```

```python
tu, tv = 1., 1.  # delta_u, delta_v

u0, v0, Su0, Sv0 = np.array([1.]), np.array([1.]), np.array([[1.]]), np.array([[1.]])
tCu, tCv = Su0, Sv0
tol = 0.001
```

```python
plt.figure()
plt.hold(True)

step = 0
for maxM in range(256):
    plt.subplot(211)
    plt.plot(v0, u0, 'r*', alpha=1)
    for maxN in range(256):
        tu, tv, tCu, tCv = update(u0, v0, Su0, Sv0)
        u0 += tu
        v0 += tv

        plt.subplot(211)
        plt.plot(v0, u0, 'b.', alpha=.5);
        plt.subplot(212)
        plt.plot(step, elb(u0, v0, Su0, Sv0), 'r+');
        display.display(plt.gcf())
        display.clear_output(wait=True)
        time.sleep(0.)
        step += 1
        if (tu ** 2) < tol and (tv ** 2) < tol: break
    if np.sum((Su0 - tCu) ** 2) < tol and np.sum((Sv0 - tCv) ** 2) < tol: break
    Su0, Sv0 = tCu, tCv
print("Converged.")
print("u: ", u0, "v: ", v0, "Cu: ", Su0, "Cv: ", Sv0)
```

### § Example 2

Instead of writing multiple lines of code, in this example we create a function `vilaps()` which allows us to reuse the updating scheme given different model/objective function.

This function uses Theano to generate a compiled updating scheme, with optional observation `y` and hyperprior `hp` arguments.

```python
import scipy.stats as stat
import numpy as np
import theano
import functools
import theano.tensor as T

print("Theano version: ", theano.__version__)
```

    Theano version:  0.9.0dev1.dev-ec0419a6cf1d3667ece1d5e0f95498c229471f19

```python
def vilaps(L, b, bdims, y=None, hp=None):

    def _jac_(*args, **kwargs):
        return T.jacobian(*args, disconnected_inputs='warn', **kwargs)

    def _hes_(*args, **kwargs):
        return T.hessian(*args, disconnected_inputs='warn', **kwargs)

    assert len(b) == len(bdims)

    if y is None:
        y = []
    else:
        y = [y]

    if hp is None:
        hp = []

    mfp = len(bdims)

    Li  = [_jac_(L, b[i]) for i in range(mfp)]
    Lii = [_hes_(L, b[i]).flatten(1) for i in range(mfp)]

    Liij  = [[T.join(0, [_jac_(Lii[i][k], b[j]) for k in range(bdims[i] ** 2)])
              for j in range(mfp)]
             for i in range(mfp)]
    Liijj = [[T.join(0, [_hes_(Lii[i][k], b[j]).flatten(1) for k in range(bdims[i] ** 2)])
              for j in range(mfp)]
             for i in range(mfp)]

    Ci  = [- T.nlinalg.pinv(Lii[i].reshape((bdims[i], bdims[i]))) for i in range(mfp)]
    fCi = [Ci[i].flatten(1) for i in range(mfp)]

    Uj = [[T.join(0, [T.sum(fCi[j] * Liij[j][i][:, k])  for k in range(bdims[i])])
           for j in range(mfp)
           if j != i]
          for i in range(mfp)]

    Ujj = [[T.join(0, [T.sum(fCi[j] * Liijj[j][i][:, k]) for k in range(bdims[i] ** 2)])
            for j in range(mfp)
            if j != i]
           for i in range(mfp)]

    Vi  = [Li[i] + 0.5 * functools.reduce(lambda x, y: x + y, Uj[i]) for i in range(mfp)]
    Vii = [(Lii[i] + 0.5 * functools.reduce(lambda x, y: x + y, Ujj[i])).reshape((bdims[i], bdims[i]))
           for i in range(mfp)]

    bGN = [-T.nlinalg.pinv(Vii[i]).dot(Vi[i]) for i in range(mfp)]
    bLM = [(T.exp(Vii[i]) - 1).dot(T.nlinalg.pinv(Vii[i])).dot(Vi[i]) for i in range(mfp)]

    F = L
    for i in range(mfp):
        F += 0.5 * T.sum(fCi[i] * Lii[i] + T.log(T.nlinalg.det(Ci[i])))

    viupdate = theano.function(
        b + hp + y,
        bGN + bLM + Ci + [F],
        allow_input_downcast=True)

    return viupdate
```

This time we try to invert the following model that generates our observation $y$

$
\begin{align}
    p(y|\mu, \tau) &= N(y|\mu, \tau^{-1})\\
    p(\mu|\tau) &= N(\mu|\mu_0, (\lambda_0\tau)^{-1})\\
    p(\tau) &= \hbox{Ga}(\tau|a_0, b_0)
    \end{align}
$

where we assume mean-field approximation

$
q(\mu,\tau) = q(\mu)q(\tau)
$

and Laplace assumption.

```python
hv = [9., 3.]
hu = [5., 2.]
rngv = stat.gamma(a=hv[0], scale=hv[1])
N = 1000

Y = np.zeros((N, 1))
U = np.zeros(N)
V = np.zeros(N)

for s in range(N):
    v = rngv.rvs()
    rngu = stat.norm(loc=hu[0], scale=np.sqrt(1 / (hu[1] * v)))
    u = rngu.rvs()
    Y[s, :] = stat.norm(loc=u, scale=1 / v).rvs()
    U[s] = u
    V[s] = v

```

```python
y = T.matrix()
u, v, pu, pv = T.vectors(4)

L1 = T.sum(-0.5 * (y - u).dot(T.nlinalg.matrix_inverse(T.diag(v))) * (y - u)) -\
    0.5 * ((u - pu[0]).dot(T.nlinalg.matrix_inverse(T.diag(pu[1] * v)))).dot(u - pu[0]) -\
    T.gammaln(pv[0]) * pv[1] ** pv[0] + (pv[0] - 1) * T.log(v) - v / pv[1]
L1 = T.sum(L1)
```

```python
b = [u, v]
hp = [pu, pv]
bdims = [1, 1]

# The line below generates an warning (or raise an DisconnectedInput exception if the disconnected_input flag
# is otherwise not set for T.jacobian and T.hessian). This happens when AD takes a derivative of a constant.
viu = vilaps(L1, b, bdims, y, hp);
```

    /home/yen/env/p34/lib/python3.4/site-packages/theano/gradient.py:533: UserWarning: grad method was asked to compute the gradient with respect to a variable that is not part of the computational graph of the cost, or is used only by a non-differentiable operator: <TensorType(float32, vector)>
      handle_disconnected(elem)
    /home/yen/env/p34/lib/python3.4/site-packages/theano/gradient.py:559: UserWarning: grad method was asked to compute the gradient with respect to a variable that is not part of the computational graph of the cost, or is used only by a non-differentiable operator: <DisconnectedType>
      handle_disconnected(rval[i])

```python
u0, v0 = np.array([2.]), np.array([5.])
tol = 0.0001

plt.figure()
for step in range(256):
    gnu, gnv, lmu, lmv, Cu, Cv, FE = viu(u0, v0, hu, hv, Y)
    tu, tv = gnu, gnv
    u0 += tu
    v0 += tv
    plt.subplot(211)
    plt.plot(u0, v0, 'bo', alpha=.5)
    plt.subplot(212)
    plt.plot(step, FE, 'r+')
    display.display(plt.gcf())
    display.clear_output(wait=True)
    if np.square(tu) < tol and np.square(tv) < tol: break
```

```python
plt.figure(figsize=(16, 5))
plt.subplot(121)
plt.plot(U, V, 'b.', alpha=.1);
plt.plot(hu[0], np.prod(hv), 'r*', markersize=12, alpha=.7);
plt.plot(u0, v0, 'y*', markersize=12, alpha=.7);
plt.grid(True)
plt.subplot(122)
plt.plot(U, V, 'b.', alpha=.1);
plt.plot(hu[0], np.prod(hv), 'r*', markersize=12, alpha=.7);
plt.plot(u0, v0, 'y*', markersize=12, alpha=.7);
plt.grid(True)
plt.xlim([4.8, 5.2])
plt.ylim([20, 40])
plt.legend(["sample", "true", "approx"]);
```

### § Example 3

This example repeats [Example 1](#example1) using the function we just wrote.

```python
c0 = [1. / 16, 1. / 8, 1. / 128]
u, v = T.vectors(2)
L2 = (-c0[0] * (u - c0[1] * v ** 2) ** 2 - c0[2] * (v ** 2 - 2) ** 2)[0]

viu = vilaps(L2, [u, v], [1, 1], hp=None, y=None)
```

```python
plt.figure()
u0 = [1.]
v0 = [1.]
tol = 0.0001

for step in range(256):
    gnu, gnv, lmu, lmv, Cu, Cv, FE = viu(u0, v0)
    tu, tv = gnu, gnv
    u0 += tu
    v0 += tv
    plt.subplot(211)
    plt.plot(u0, v0, 'bo', alpha=.5)
    plt.subplot(212)
    plt.plot(step, FE, 'r+')
    display.display(plt.gcf())
    display.clear_output(wait=True)
    if np.all(np.square([tu, tv]) < tol): break
print(u0, v0)
```

### A.1 Multivariate normal distribution

Say $\pmb x \sim N(\pmb\mu, \pmb\Lambda^{-1})$

Then,

$
\ln p(\pmb x) = -\frac D 2\ln 2\pi - \frac 1 2 \ln|\pmb\Lambda^{-1}| - \frac 1 2 (\pmb x-\pmb\mu)^T\pmb\Lambda(\pmb x-\pmb\mu)
$

$
H(\pmb x) = \langle -\ln \pmb x\rangle = \frac D 2\ln 2\pi e + \frac 1 2 \ln |\pmb\Lambda^{-1}|
$

$
D = \dim(\pmb x)
$

### A.2 Linearisation

The following derives a local linearisation of a dynamical system, proposed by Ozaki (1992).

Suppose we have a stochastic dynamical system with a continuous time Gaussian white noize $n(t)$.

$$
\begin{equation}
\dot{y}(t) = f(y(t))) + n(t) \tag{1}
\end{equation}
$$

At first, we approximate the equation by a Gaussian process,

$$
\begin{equation}
\dot{y}(t) = K_ty(t) + n(t) \tag{2}
\end{equation}
$$

with some appropriate $K_t$ on a small interval $[t, t+\Delta t)$.

From the assumption that the Jacobian of the linear part of $(2)$ over each interval $[t, t+\Delta t)$ is given by the Jacobian $J_t = \frac{\partial f(y)}{\partial y}$ at $y = y_t$ of the original dynamical system, we have the following equation by using  chain rule,

$$
\begin{eqnarray*}
\ddot{y}(s) &=& \frac{d\dot{y}(s)}{ds} \\
&=& \frac{\partial f(y)}{\partial y}\frac{\partial y}{\partial s} \\
&=& J_t\dot{y}(s) \tag{3}
\end{eqnarray*}
$$

for $t \leq s \leq t + \Delta t$.

By integrating (3) on $[t, t + \tau)(0 \leq \tau < \Delta t)$ we have

$$
\begin{eqnarray*}
\dot{y}(t + \tau) &=& \exp(J_t\tau)\dot{y}(t) \\
&=& \exp(J_t\tau)f(y(t)) \tag{4}
\end{eqnarray*}
$$

Integrating the equation $(4)$ again with respect to $\tau$  on $[0, \Delta t)$  utilizing the method of integration by parts,
finally, we get the following equation which gives the value of $y(t+\Delta t)$ as a function of $y(t)$.

$$
\begin{equation}
y(t + \Delta t) = y(t) + J_t^{-1}\{\exp(J_t\Delta t) - 1\}f(y(t))\tag{5}
\end{equation}
$$

* Ozaki T (1992) "A bridge between nonlinear time series models and nonlinear stochastic dynamical systems: a local linearization approach", Statistics Sinica 2, 1, 113-135.

Laplace approximation for static model.ipynb


f=m*¨x

```python
%matplotlib inline

```

```python
import matplotlib.pyplot as plt
import numpy as np
from sympy import *
```

```python
import matplotlib as mpl
label_size = 14
mpl.rcParams['xtick.labelsize'] = label_size
mpl.rcParams['ytick.labelsize'] = label_size
```

```python
t = np.linspace(0,50,100)
plt.figure(figsize = (15,4))

plt.plot(t, 0.5*np.cos(0.5*t)+ 0.1*np.sin(0.5*t), '-', lw = 1, ms=4, label='$x(t)$')

plt.plot(t, 0.5*0.1*np.cos(0.5*t)- 0.5*0.5*np.sin(0.5*t), 'ro-', lw = 1, ms=4, label=r'$\dot{x}(t)$')

plt.xlabel('$t$', fontsize = 20)
plt.legend()
plt.show()

```

```python
x = np.arange(0,10,1)
y=x**2
plt.plot(x,y,'r')
plt.scatter(x,y)
```

```python
t = np.arange(-10,10,0.1)
#t = np.linspace(0,50,100)
#y=t**2
y = np.sin(t)
plt.plot(t,y)
```

```python
!pip install seaborn
```

    Requirement already satisfied: seaborn in d:\anaconda\lib\site-packages (0.9.0)
    Requirement already satisfied: numpy>=1.9.3 in d:\anaconda\lib\site-packages (from seaborn) (1.15.4)
    Requirement already satisfied: scipy>=0.14.0 in d:\anaconda\lib\site-packages (from seaborn) (1.1.0)
    Requirement already satisfied: pandas>=0.15.2 in d:\anaconda\lib\site-packages (from seaborn) (0.23.4)
    Requirement already satisfied: matplotlib>=1.4.3 in d:\anaconda\lib\site-packages (from seaborn) (3.0.2)
    Requirement already satisfied: python-dateutil>=2.5.0 in d:\anaconda\lib\site-packages (from pandas>=0.15.2->seaborn) (2.7.5)
    Requirement already satisfied: pytz>=2011k in d:\anaconda\lib\site-packages (from pandas>=0.15.2->seaborn) (2018.7)
    Requirement already satisfied: cycler>=0.10 in d:\anaconda\lib\site-packages (from matplotlib>=1.4.3->seaborn) (0.10.0)
    Requirement already satisfied: kiwisolver>=1.0.1 in d:\anaconda\lib\site-packages (from matplotlib>=1.4.3->seaborn) (1.0.1)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in d:\anaconda\lib\site-packages (from matplotlib>=1.4.3->seaborn) (2.3.0)
    Requirement already satisfied: six>=1.5 in d:\anaconda\lib\site-packages (from python-dateutil>=2.5.0->pandas>=0.15.2->seaborn) (1.12.0)
    Requirement already satisfied: setuptools in d:\anaconda\lib\site-packages (from kiwisolver>=1.0.1->matplotlib>=1.4.3->seaborn) (40.6.3)

```python
import seaborn as sns
sns.set(style='ticks', palette='Set2')
```

```python
t = np.linspace(0,50,100)
frecuencias = np.array([0.1, 0.2, 0.5, 0.6])
plt.figure(figsize=(7,4))
for f in frecuencias:
    plt.plot(t, 0.5*np.cos(f*t)+ 0.1*np.sin(f*t), 'o-', label='$\omega_0 = %s$'%f)

plt.xlabel('$t$', fontsize = 16)
plt.ylabel('$x(t)$', fontsize = 16)
plt.title('Oscilaciones', fontsize = 16)
plt.legend(loc='center left',bbox_to_anchor=(1.05, 0.5), prop={'size':14})
plt.show()
```

Plotly  para graficar en 3D es robusta y buena

```python
from ipywidgets import *
```

```python
def masa_resorte(t=0):
    fig = plt.figure()
    ax = fig.add_subplot(1,1,1)
    ax.plot(0.5*np.cos(0.5*t)+ 0.1*np.sin(0.5*t),[0], 'ko', ms=10)
    ax.set_xlim(left = -0.6, right=0.6)
    ax.axvline(x=0, color = 'r')
    ax.axhline(y=0, color = 'gray', lw=1)
    fig.canvas.draw()

interact(masa_resorte, t=(0,50,0.01))
```

    interactive(children=(FloatSlider(value=0.0, description='t', max=50.0, step=0.01), Output()), _dom_classes=('…

    <function __main__.masa_resorte(t=0)>

```python
def masa_resorte(t=0):
    fig = plt.figure()
    ax = fig.add_subplot(1,1,1)
    ax.plot(0.5*np.cos(0.5*t)+ 0.1*np.sin(0.5*t),[0], 'ko', ms=10)
    ax.set_xlim(left = -0.6, right=0.6)
    ax.axvline(x=0, color = 'r')
    ax.axhline(y=0, color = 'gray', lw=1)
    fig.canvas.draw()

interact_manual(masa_resorte, t=(0,50,0.01))
```

    interactive(children=(FloatSlider(value=0.0, description='t', max=50.0, step=0.01), Button(description='Run In…

    <function __main__.masa_resorte(t=0)>

$$\begin{eqnarray}
\dot{x} &=&  y \\
\dot{y} &=& -\frac{k}{m} x - \frac{B}{m} y
\end{eqnarray}$$

```python
import matplotlib.pyplot as plt
import numpy as np
from scipy.integrate import odeint
%matplotlib inline
```

```python
k = 3.0
m = 1.0
B = 0.5

def armonico(variables, t):
    x, y = variables
    return [y, -k* x / m - B / m * y]

inicial = [0.8, 0]

tiempo = np.arange(0.0, 20.0, 0.01)

resultado = odeint(armonico, inicial, tiempo)

xx, yy = resultado.T

plt.plot(tiempo, xx, c = 'r', label="Posicion")
plt.plot(tiempo, yy, c = 'k', label="Velocidad")
plt.legend(loc = 'best', prop ={'size':14})
plt.xlabel('tiempo', fontsize=14)
plt.show()

```

```python
from ipywidgets import *
```

```python
def amortiguador(t=0):
    fig = plt.figure()
    ax = fig.add_subplot(1,1,1)
    ax.plot(tiempo, xx, '-k')
    ax.plot(tiempo[t], xx[t], 'ro')
    ax.text(4, 0.6, 'tiempo =  %s'%tiempo[t])
    ax.text(4, 0.5, 'posicion =  %s'%xx[t])
    fig.canvas.draw()

t_f = len(tiempo)
interact_manual(amortiguador, t= (0, t_f, 1))
```

    interactive(children=(IntSlider(value=0, description='t', max=2000), Button(description='Run Interact', style=…

    <function __main__.amortiguador(t=0)>

```python
#k = 3.0
#m = 1.0
#B = 0.5

def constantes(k=0,m=0,B=0):
#    print(k,m,B)

    def armonico(variables, t):
        x, y = variables
        return [y, -k* x / m - B / m * y]

    inicial = [0.8, 0]

    tiempo = np.arange(0.0, 20.0, 0.01)

    resultado = odeint(armonico, inicial, tiempo)

    xx, yy = resultado.T

    plt.plot(tiempo, xx, c = 'r', label="Posicion")
    plt.plot(tiempo, yy, c = 'k', label="Velocidad")
    plt.legend(loc = 'best', prop ={'size':14})
    plt.xlabel('tiempo', fontsize=14)
    plt.show()

interact_manual(constantes,k=(0,10,0.1),m=(0,10,0.1),B=(0,10,0.1))

```

    interactive(children=(FloatSlider(value=0.0, description='k', max=10.0), FloatSlider(value=0.0, description='m…

    <function __main__.constantes(k=0, m=0, B=0)>

```python

```


# Two Degree-of-Freedom four well Potential

## Introduction and Development of the Problem

In this chapter we continue the study of Collins et al. {% cite collins2011 --file SNreac %} by considering the phase space structures that govern different reaction pathways and we  then consider the influence of symmetry breaking, bifurcation, and energy on these phase space reaction pathways. In order to reveal the phase space geometry we take advantage of lagrangian descriptors, which will allow us to reconstruct the phase space structures that govern the chemical reaction processes.

We focus our attention on the Hamiltonian dynamics that takes place in a PES with four wells as a basic model for isomerization dynamics. Each potential well corresponds to a equilibrium configuration of the molecule under study, and wells are separated by saddle critical points, which represent transition states. In this setup, we understand that reaction occurs when an initial condition that starts for instance in one of the wells of the PES is capable of evolving to a neighboring well by crossing the saddle critical point that connects both wells.

Consider the Hamiltonian:

\begin{equation}
H(x,y,p_x,p_y) = \frac{1}{2}\left(p_x^2 + p_y^2\right) + V(x,y)
\label{hamiltonian}
\end{equation}

where we have supposed for simplicity that the mass in each DoF is $m_x = m_y = 1$. The PES is defined as follows:

\begin{equation}\label{pes_model}
V(x,y) = x^4 - \alpha x^2 - \delta x + y^4 - y^2 + \beta x^2 y^2
\end{equation}

and $\delta$ represents the asymmetry parameter in the double well potential of the $x$ DoF. The dynamics of the Hamiltonian in Eq. \eqref{hamiltonian} is described by Hamilton's equations of motion:

\begin{equation}
\begin{cases}
\dot{x} = \dfrac{\partial H}{\partial p_x} = p_x \\[.4cm]
\dot{y} = \dfrac{\partial H}{\partial p_y} = p_y \\[.4cm]
\dot{p}_x = -\dfrac{\partial H}{\partial x} = -\dfrac{\partial V}{\partial x} = -4 x^3 + 2 \alpha x + \delta - 2 \beta x y^2 \\[.4cm]
\dot{p}_y = -\dfrac{\partial H}{\partial y} = -\dfrac{\partial V}{\partial y} = -4 y^3 + 2 y - 2 \beta x^2 y
\end{cases}
\label{ham_eqs}
\end{equation}

Now we describe the dynamics of the Hamiltonian in Eq. \eqref{hamiltonian} in the case where both DoF are uncoupled, so that the coupling strength is $\beta = 0$. We start our analysis by focusing on the symmetric Hamiltonian with $\delta = 0$, and later we will move on to discuss the asymmetric case.

## Revealing Phase Space Structure: Symmetric and Uncoupled System

fig:1  Potential energy surface in Eq. \eqref{pes_model} for the symmetric  uncoupled case.

Consider the symmetric and uncoupled system with energy $H_0$. Since the system is conservative, dynamics is constrained to the three-dimensional energy hypersurface:

\begin{equation}
\mathcal{S}(H_0) = \left\{ (x,y,p_x,p_y) \in \mathbb{R}^4 \; | \; H_0 = \frac{1}{2}\left(p_x^2+p_y^2\right) + x^4 - x^2 + y^4 - y^2 \right\}
\end{equation}

where the total energy can be split between both DoF to yield:

\begin{equation}\label{eqsymun}
H(x,y,p_x,p_y) = H_{x}(x,p_x) + H_{y}(y,p_y)
\end{equation}

so that the partitioned energy is:

\begin{equation}
H_{x}(x,p_x) = \frac{1}{2} \, p_x^2 + W(x) \quad,\quad H_{y}(y,p_y) = \frac{1}{2} \, p_y^2 + W(y)
\end{equation}

and the potential in each DoF is a double-well in the form:

\begin{equation}
W(z) = z^4 - z^2
\label{1D_potSymm}
\end{equation}

Therefore, the symmetric and uncoupled system is integrable, as the energy in each DOF is conserved, and thus we have two independent constants of the motion, implying that all motion is regular and no chaotic dynamics is allowed. The Hill's region, which has its origins in Celestial Mechanics, is defined as the intersection of the energy hypersurface with configuration space, that is:

\begin{equation}
\mathcal{C}(H_0) = \left\{ (x,y,p_x,p_y) \in \mathbb{R}^4 \; | \; x^4 - x^2 + y^4 - y^2 \leq H_0 \right\}
\end{equation}

This region determines the energetically allowed configurations for the DoFs of the system, and all the points outside the Hill's region are energetically forbidden. The boundary of the Hill's region, $\partial \mathcal{C}(H_0)$, is known in the literature as the zero velocity curve and corresponds to phase space points for which kinetic energy is zero.

In order to determine the phase space structures that characterize isomerization dynamics, we need to look first at the equilibrium points of Hamilton's equations. A phase space point  $\mathbf{x}_e = (x_e,y_e,p_{x,e},p_{y,e})$ is an equilibrium of Eq. \eqref{ham_eqs} when it satisfies $p_{x,e} = p_{y,e} = 0$ and $\nabla V(x_e,y_e) = \mathbf{0}$. The local stability of this stationary point is determined by the eigenvalues of the Jacobian matrix obtained by linearizing Hamilton's equations in its neighborhood. In the symmetric and uncoupled system, we have 9 equilibrium points with configuration coordinates and energies given below:

-  Four potential wells at the points $ \left(\pm\sqrt{2}/2,\pm\sqrt{2}/2\right) $ with energies $ V(\pm\sqrt{2}/2,\pm\sqrt{2}/2) = -1/2 $.
-  Four index-1 saddles located at $ (\pm\sqrt{2}/2,0)$, $(0,\pm\sqrt{2}/2) $ with energies $ V(\pm\sqrt{2}/2,0) = V(0,\pm\sqrt{2}/2) = -1/4 $.
-  One index-2 saddle at the origin with energy $ V(0,0) = 0 $.

Potential wells of a PES correspond to center-stability equilibrium points of the Hamiltonian, and are characterized by the fact that the Hessian matrix of the PES evaluated at these critical points, denoted by $\text{Hess}_V$, has a pair of real and positive eigenvalues. This results in the Jacobian of the linearization having two pairs of complex (and purely imaginary) eigenvalues, which yields quasiperiodic motion in their neighborhood. In the context of chemical reaction dynamics, potential wells correspond to stable isomer configurations of the molecule under study. On the other hand, index-1 saddles of the PES are identified with saddle points of the potential energy landscape, and the associated phase space equilibrium point is of saddle-center stability type. This means that $\text{Hess}_V$ has two real eigenvalues of different sign, which is equivalent to the Jacobian having a pair of real eigenvalues of opposite sign and a pair of purely imaginary eigenvalues. Geometrically, an index-1 saddle is a local maximum in one direction and a local minimum in a perpendicular direction, as seen in the normal coordinates associated to the eigenvectors. Moreover, the eigenvector pointing in the maximum direction can be taken as a local approximation to define the reaction coordinate. The role that the index-1 saddles have in the model PES for isomerization that we  consider is to connect pairs of wells, and therefore, they control access of phase space trajectories from well to well. From the perspective of chemical reactions, index-1 saddles are identified with transition state configurations of the given molecule, and, as we will see shortly, the phase space structures they give rise to are essential for the accurate computation of chemical reaction rates. Finally, index-2 saddles of a two-dimensional PES have the geometrical shape of a local maximum, i.e. a 'hilltop', which is characterized by the fact that the Hessian of the PES has two real and negative eigenvalues. This implies that when we linearize Hamilton's equations about this type of equilibrium point, the Jacobian yields two pairs of real eigenvalues, each pair having opposite signs. Thus, their stability is of saddle-saddle type, which has the dynamical effect of deflecting incoming trajectories in the neighborhood of an index-2 saddle at an exponential rate.

At this point, it is important to define what we mean for a trajectory to be trapped in one of the wells of the PES. From the energy landscape displayed in Fig. [fig:1](#fig:PES_isom_rout), this concept can be determined by studying the change in the sign of the configuration coordinates $x$ or $y$. For example, if we focus on the lower-left well, we can say that a trajectory is trapped in that well when the configuration coordinates of both $x$ and $y$ DoF remain negative during its evolution. Moreover, in this setup we can identify *reactive* trajectories as those that move from one well to another along their evolution. Since we are dealing in this case with a symmetric PES with respect to the origin, and there is one well in each quadrant, reaction would imply a change in the sign of one of the configuration coordinates of the trajectory. Notice that two types of reactive trajectories are possible when considering an initial condition starting on the lower-left well whose potential destination is the upper-right well. First, we could have sequential isomerization which implies transition from the lower-left well to an adjacent well through the phase space bottleneck region in the neighborhood of the index-1 saddle point that sits between both wells. This situation is only possible given that the system has enough energy to surmount the energy barrier (the energy of the index-1 saddle). A second alternative for reaching the upper-right well from the lower-left well is that trajectories cross directly through the index-2 saddle located at the origin (the hilltop of the PES), which is known as concerted isomerization. For an illustration of these two isomerization processes refer to Fig. [fig:1](#fig:PES_isom_rout).

We describe next the isomerization dynamics for the symmetric and uncoupled Hamiltonian system in terms of the phase space geometrical structures associated to the index-1 and index-2 saddles present in the model PES. We begin our discussion by fixing a value for the total energy of the system $H_0$. As we have mentioned earlier, this energy is distributed among both DoF, so that we can write $H_0 = H_{x,0} + H_{y,0}$ where $H_{x,0} $ and $H_{y,0}$ are the energies in the $x$ and $y$ DoFs respectively. We discuss some of the different cases in terms of the energy below:

-  **Energy Level** $(-1/2 \leq H_0 \leq -1/4)$: For this case the wells of the PES are isolated from each other since the energy of the system is below that of the index-1 saddles that interconnect the wells. Therefore, motion of an initial condition that starts in one of the wells will remain trapped in that well forever, displaying librational quasiperiodic motion.

-  **Energy Level** $(-1/4 < H_0 \leq 0)$: In this situation the energy is above that of any of the index-1 saddles in the PES, but below the energy of the index-2 saddle at the origin. Therefore, all potential wells are connected and isomerization can take place. This allows the transit of trajectories from well to well through the phase space bottlenecks that open in the neigborhood of the equilibrium points associated to index-1 saddles. However, the index-2 saddle region of the PES is energetically forbidden, and hence, only sequential isomerization is allowed. Given that the system is completely symmetric in this case, in order to describe the phase space structures that govern isomerization dynamics and characterize the bottleneck regions in the vicinity of the index-1 saddles of the PES, we focus our analysis on the equilibrium point $(0,\sqrt{2}/2,0,0)$ associated to the upper index-1 saddle, separating the upper-left and upper-right wells. This stationary point has saddle-center stability.

## Implications for Reaction Dynamics: Symmetric and Uncoupled System

We will introduce now the dynamical concepts. The phase space dividing surface separating the upper-left from the upper-right well regions of the PES is also defined in this situation by intersecting the energy surface with the slice $x = 0$, that is:

\begin{equation}\label{ds_sym_unc}
	\mathcal{D}\left(H_0\right) = \mathcal{S}\left(H_0\right) \cap \lbrace x = 0 \rbrace = \left\{\left(x,y,p_x,p_y\right) \in \mathbb{R}^4 \; \bigg| \; H_0 = \frac{1}{2}\left(p_x^2+p_y^2\right) + y^4 - y^2 \;,\; x = 0 \right\}
\end{equation}

which is a non-invariant surface with the local non-recrossing property. Moreover, it has the topology of a sphere $S^2$ with two hemispheres, known as the forward and backward dividing surface. They are given by:

\begin{equation}
	\begin{split}
	\mathcal{D}_{f}(H_0) &= \left\{\left(x,y,p_x,p_y\right) \in \mathbb{R}^4 \; \bigg| \; H_0 = \frac{1}{2}\left(p_x^2+p_y^2\right) + y^4 - y^2 \;,\; x = 0 \;,\; p_x > 0 \right\} \\
	\mathcal{D}_{b}(H_0) &= \left\{\left(x,y,p_x,p_y\right) \in \mathbb{R}^4 \; \bigg| \; H_0 = \frac{1}{2}\left(p_x^2+p_y^2\right) + y^4 - y^2 \;,\; x = 0 \;,\; p_x < 0 \right\}
	\end{split}
	\end{equation}

The two hemispheres meet at the equator, which is a NHIM (or an UPO), with the form:

\begin{equation}
	\mathcal{N}(H_0) = \left\{\left(x,y,p_y,p_y\right) \in \mathbb{R}^4 \; \bigg| \; H_0 = \frac{p_y^2}{2} + y^4 - y^2 \;,\; x = p_x = 0 \right\}
	\end{equation}

and has the topology of a circle $S^1$. The NHIM has stable and unstable manifolds:

\begin{equation}
	\mathcal{W}^{u}(H_0) = \mathcal{W}^{s}(H_0) = \left\{\left(x,y,p_y,p_y\right) \in \mathbb{R}^4 \; \bigg| \; H_y = \frac{p_y^2}{2} + y^4 - y^2 = H_0 \;,\; H_x = \frac{p_x^2}{2} + x^4 - x^2 = 0 \right\}
	\end{equation}

and topologically they have the structure of $S^1 \times \mathbb{R}$, representing *tube* or *cylindrical manifolds*. Observe that for the symmetric and uncoupled case that we are analyzing they have a homoclinic structure in phase space. All these relevant phase space structures that are responsible for the reaction mechanisms in phase space through the bottleneck that connects the upper-left ad upper-right wells of the PES are depicted in Fig. [fig:2](#fig:phasePort_1DoF_symm).

fig:2  A) Dividing surface described in Eq. \eqref{ds_sym_unc} for the symmetric uncoupled Hamiltonian system with energy $H_0 = -0.15$ in the neighborhood of the equilibrium point $\mathbf{x}_e = (0,\sqrt{2}/2,0,0)$. B) Symmetric double well potential given in Eq. \eqref{1D_potSymm}. C) and D) depict the phase portraits for the $x-p_x$ and $y-p_y$ planes respectively. We have marked with a magenta line the dividing surface $x = 0$ that separates the upper-left and upper-right wells of the PES.

fig:3  Phase space structures at energy $H = -0.2$. A) LDs calculated using the p-norm definition, with $p = 1/2$, integration time $\tau = 5$ on the phase space slice $y = -1/\sqrt{2}$ for the symmetric uncoupled Hamiltonian in Eq. (\ref{eqsymun}). B) Dynamical evolution of the initial conditions selected in panel A.

fig:4  Potential energy surface for the symmetric uncoupled Hamiltonian in Eq. (\ref{eqsymun}) at energy $H = -0.2$.

We will describe now the phase space dynamics. Since the Hamiltonian is separable, the energy of the system can naturally partitioned between each DoF, that is, $H_{0}=H_{x,0}+H_{y,0}$. By looking at the phase portraits in Fig [fig:2](#fig:phasePort_1DoF_symm) we will discuss the different cases:

1. $h_x \, , \, h_y < 0$: Since this energy levels correspond to trajectories inside the homoclinic curve in the phase portraits (B,D) depicted in Fig [fig:2](#fig:phasePort_1DoF_symm), then we know that neither $x$ nor $y$ will change sign, and this means that trajectories will be trapped in the lower-right well.

2.  $h_x < 0 \, , \, h_y > 0$: Since the energy in the $y$ DoF is positive, this means we are outside the separatrix in the $y-p_y$ plane, and thus the trajectory will be periodic in $y$ and the sign of $y$ changes along the trajectory. Moreover, since the energy in $x$ is negative, this gives that we are inside the separatrix in the $x-p_x$ plane, corresponding to $x$ not changing sign along the trajectory evolution. Consequently, we will have trajectories that move through the bottleneck corresponding to the right index-1 back and forth between the lower-right and upper-right wells.

3.  $h_x > 0 \, , \, h_y < 0$: Since the energy in the $x$ DoF is positive, this means we are outside the separatrix in the $x-p_x$ plane, and thus the trajectory will be periodic in $x$ and the sign of $x$ changes along the trajectory. Moreover, since the energy in $y$ is negative, this gives that we are inside the separatrix in the $y-p_y$ plane, corresponding to $y$ not changing sign along the trajectory evolution. Consequently, we will have trajectories that move through the bottleneck corresponding to the bottom index-1 back and forth between the lower-right and lower-left wells.

4.  $h_x = 0 \, , \, h_y < 0$: Since the energy in the $x$ DoF is zero, this means we are on the separatrix in the $x-p_x$ plane, and thus the trajectory will not change sign in $x$ and it will asymptotically approach $x = 0$. Moreover, since the energy in $y$ is negative, this gives that we are inside the separatrix in the $y-p_y$ plane, corresponding to $y$ not changing sign along the trajectory evolution. Consequently, we will have trajectories that evolve on the spherical cylinder of the bottom index-1 that will asymptotically approach the UPO.

5.  $h_x < 0 \, , \, h_y = 0$: Since the energy in the $y$ DoF is zero, this means we are on the separatrix in the $y-p_y$ plane, and thus the trajectory will not change sign in $y$ and it will asymptotically approach $y = 0$. Moreover, since the energy in $x$ is negative, this gives that we are inside the separatrix in the $x-p_x$ plane, corresponding to $x$ not changing sign along the trajectory evolution. Consequently, we will have trajectories that evolve on the spherical cylinder of the right index-1 that will asymptotically approach the UPO.

So far we have discussed the nature of the trajectories for all the different cases of positive and negative total energy of the system. At this point we will explore the dynamics revealed by the Lagrangian descriptors (LDs) for the same cases of the total energy and conclude that the LDs recovers the same phase space as in Fig. [fig:2](#fig:phasePort_1DoF_symm). In Fig. [fig:3](#fig:LD_sym_h_neg) we have calculated the LDs using a small integration time $\tau = 5$, for the phase space slice $y = -1/\sqrt{2}$ and for total energy $H = -0.2$. This energy is above the energy of all four index-1 saddles and below the energy of the index-2 saddle. Therefore all the wells of the potential are connected and reaction can take place by crossing the open bottleneck, which is the NHIM (UPO) and its stable and unstable manifolds (spherical cylinders) that coincide in this case, and passing from the $x>0$ to $x<0$ (or from $y>0$ to $y<0$ or vice versa) or vice versa. So these geometrical structures act as a 'reactive highway' allowing the system to transit from well to well which would correspond to a given molecule undergoing an isomerization reaction. Remember that since the chosen energy is below the energy of the index-2 saddle but above the energy of all four index-1 saddles  this means that we can have sequential isomerization but the region in the neighborhood of the index-2 saddle is forbidden and hence the system cannot exhibit concerted isomerization. In particular in Fig. [fig:3](#fig:LD_sym_h_neg) (A) we can notice that the spherical cylinders of the left and right index-1 saddles do not intersect with the cylinders of the bottom index-1 saddle and that the phase space consists of three regions: 1) the spherical cylinder of the index-1 in the right, where if we pick an initial condition inside this region the trajectory, in the configuration space, will be reactive and it will move through the bottleneck (UPO), corresponding to the right index-1, back and forth between the lower-right and upper-right wells (magenta). This is because the y-coordinate change sign while the x-coordinate doesn't. We are outside of the separatrix in the $y-p_{y}$ plane and the trajectory is periodic. On the other hand we are inside the separatrix in the $x-p_{x}$ plane and hence the trajectory doesn't change sign in this direction. We should mention here that the boundary of the cylinders is the boundary of the energy. 2) The spherical cylinder of the index-1 on the bottom, where if we pick any initial condition inside this cylinder the trajectory will be reactive and it move through the bottleneck (UPO), corresponding to the bottom index-1, back and forth between the lower-right and lower-left wells (red). This is because the x-coordinate change sign while the y-coordinate doesn't. We are outside of the separatrix in the $x-p_{x}$ plane and the trajectory is periodic. On the other hand we are inside the separatrix in the $y-p_{y}$ plane and hence the trajectory doesn't change sign in this direction. 3) The region inside the homoclinic curve, where if we pick an initial condition inside this region the trajectory will be trapped in the well (cyan) and they will never escape. In this case both the trajectories are inside the separatrix and hence non of them change sign. To summarize in this case, that the system is uncoupled, the stable and unstable manifolds of the UPOs of the index-1 saddles of the PES do not give rise to heteroclinic intersections, so that the dynamics of the system remains trapped in the lower-left well or it can exhibit sequential isomerization to the lower-right or upper-right well.

The potential energy surface for the symmetric case and for total energy $H = -0.2$, which is above the energy of all the index-1 saddles and below the energy of the index-2 saddle, is depicted in the Fig. [fig:4](#fig:LD_pot_neg), where we can see the reactive trajectories (red and magenta) in the spherical cylinders, which are impenetrable barriers on the constant energy surface and separating the reactive and non-reactive trajectories in the phase space, and the trapped trajectories (cyan) in one of the wells. Hence the cylinders determine the initial conditions that can pass through the bottleneck as they evolve in time.  The method of LDs also detects the unstable periodic orbits (UPOs) of all the index-1 saddles that are normally hyperbolic invariant manifolds (NHIMs). The method of LDs recovers all the phase space structures that we described above.

## Revealing Phase Space Structures: Asymmetric and Uncoupled Dynamics

We explore now the dynamics of the asymmetric and uncoupled system. Consider that the system has energy $H_0$, and since the DoFs are separable, we can decompose the total energy as $H_0 = H_{x,0} + H_{y,0}$. Moreover, the energy in each DoF can be written as:

\begin{equation}\label{eqasymun}
	H_{x}(x,p_x) = \frac{1}{2} \, p_x^2 + U(x) \quad,\quad H_{y}(y,p_y) = \frac{1}{2} \, p_y^2 + W(y)
\end{equation}
where the potential in the $x$ DoF is an asymmetric double-well in the form:
\begin{equation}\label{potasymun}
	U(x) = x^4 - x^2 - \delta x
\end{equation}

and $\delta$ represents the asymmetry parameter. The potential energy $W(y)$ of the $y$ DoF is a symmetric double well given by Eq. \eqref{1D_potSymm}. Energy conservation implies that phase space dynamics occurs in the three-dimensional energy hypersurface:

\begin{equation}
\mathcal{S}(H_0) = \left\{ (x,y,p_x,p_y) \in \mathbb{R}^4 \; | \; H_0 = \frac{1}{2}\left(p_x^2+p_y^2\right) + x^4 - x^2 - \delta x + y^4 - y^2 \right\}
\end{equation}

We concentrate on describing the phase space structures that determine transport between the upper-left and upper-right well regions of the PES for values of the asymmetry parameter below that corresponding to the saddle-node bifurcation. The dividing surface separating both wells is defined in this situation by intersecting the energy surface with the slice $x = x_s$, and the value of $x_s$ is that of $x_2$ in Eq. \eqref{roots_cubic_asymm}, that is:

\begin{equation}
\mathcal{D}\left(H_0\right) = \mathcal{S}\left(H_0\right) \cap \lbrace x = x_{s} \rbrace = \left\{(x,y,p_x,p_y) \in \mathbb{R}^4 \; \bigg| \; H_0 = \frac{1}{2}\left(p_x^2+p_y^2\right) + y^4 - y^2 - \frac{x_s}{2} \left(x_s + \frac{3\delta}{2}\right) \right\}
\label{ds_asym_unc}
\end{equation}

where we have used that the potential energy $H_{x,s} = U(x_s) = x_s^4 - x_s^2 - \delta x_s = -x_s^2/2 - (3\delta/4) \, x_s$, a property that follows from the fact that $x_s$ is a critical point of $U(x)$. The dividing surface is non-invariant and has the local non-recrossing property. Moreover, it has the topology of a sphere $S^2$ with two hemispheres, the forward and backward dividing surfaces, given by:

\begin{equation}
\begin{split}
\mathcal{D}_{f}(H_0) &= \left\{(x,y,p_x,p_y) \in \mathbb{R}^4 \; \bigg| \; H_0 = \frac{1}{2}\left(p_x^2+p_y^2\right) + y^4 - y^2 - \frac{x_s}{2} \left(x_s + \frac{3\delta}{2}\right) \;,\; p_x > 0 \right\} \\
\mathcal{D}_{b}(H_0) &= \left\{(x,y,p_x,p_y) \in \mathbb{R}^4 \; \bigg| \; H_0 = \frac{1}{2}\left(p_x^2+p_y^2\right) + y^4 - y^2 - \frac{x_s}{2} \left(x_s + \frac{3\delta}{2}\right) \;,\; p_x < 0 \right\}
\end{split}
\end{equation}

The two hemispheres meet at the equator, which is a NHIM (or an UPO) described by:

\begin{equation}
\mathcal{N}(H_0) = \left\{(x,y,p_x,p_y) \in \mathbb{R}^4 \; \bigg| \; H_0 = \frac{1}{2} \, p_y^2 + y^4 - y^2 - \frac{x_s}{2} \left(x_s + \frac{3\delta}{2}\right) \;,\; p_x = 0 \right\}
\end{equation}

with the topology of a circle $S^1$. The NHIM has stable and unstable manifolds:

\begin{equation}
\mathcal{W}^{u}(H_0) = \mathcal{W}^{s}(H_0) = \left\{\left(x,y,p_y,p_y\right) \in \mathbb{R}^4 \; \bigg| \; H_{y,0} = \frac{p_y^2}{2} + y^4 - y^2 \;,\; H_{x,s} = \frac{p_x^2}{2} + x^4 - x^2 - \delta x \right\}
\end{equation}

and topologically they have the structure of $S^1 \times \mathbb{R}$, representing *tube* or *cylindrical manifolds*. Observe that for the asymmetric and uncoupled case that we are analyzing they have a homoclinic structure in phase space. All these relevant phase space structures that are responsible for the reaction mechanisms in phase space through the bottleneck that connects the upper-left ad upper-right wells of the PES.

The equilibrium points $\mathbf{x}_e = (x_e,y_e,p_x^e,p_y^e)$ satisfy the equations:

\begin{equation}
	p_x^e = p_y^e = 0 \quad , \quad  -4 y_e^3 + 2 y_e = 0 \quad , \quad 4 x_e^3 - 2 \alpha x_e - \delta = 0 \;.
\end{equation}

Solving for the $y$ DoF we get $y_e \in \lbrace -\sqrt{2}/2,0,\sqrt{2}/2 \rbrace$, and the $x$ DoF gives a cubic polynomial $f(x) = 4x^3 - 2 \alpha x - \delta$ whose roots can be obtained analytically using the formulas in {% cite brizzard2015 --file SNreac %}. If we take $\gamma$ such that $f^{\prime\prime}(\gamma) = 0$ and define:

\begin{equation}
	\mu = \sqrt{-\frac{1}{3}f^{\prime}(\gamma)} \quad,\quad \phi = \arccos\left(-\dfrac{f(\gamma)}{\mu^3}\right)
	\label{cubic_rules}
\end{equation}

then for this cubic polynomial we have that $\gamma = 0$ and therefore:

\begin{equation}
	\mu = \sqrt{\dfrac{2\alpha}{3}} \quad,\quad \phi = \arccos \left( \left(\dfrac{3}{2\alpha} \right)^{3/2} \delta \right)
\end{equation}

The roots of the cubic polynomial have the form:

\begin{equation}
	\begin{cases}
	x_1 = \gamma + \mu \cos\left(\dfrac{\phi}{3}\right) = \mu \cos\left(\dfrac{\phi}{3}\right) \\[.3cm]
	x_{2,3} = \gamma - \mu \cos\left(\dfrac{\pi \pm \phi}{3}\right) = - \mu \cos\left(\dfrac{\pi \pm \phi}{3}\right) = -\dfrac{x_1}{2} \pm \dfrac{\mu\sqrt{3}}{2} \sin \left(\dfrac{\phi}{3}\right)
	\end{cases}
	\label{roots_cubic_asymm}
\end{equation}

Observe that the two roots $x_{2,3}$ merge into one, i.e. $x_{2,3} = -x_1/2$, when $\phi = 0$ and this occurs for:

\begin{equation}
	\left(\dfrac{3}{2\alpha} \right)^{3/2} \delta = 1 \quad \Leftrightarrow \quad \delta_c(\alpha) = \left(\dfrac{2\alpha}{3} \right)^{3/2}
	\label{crit_asym_uncoup}
\end{equation}

Consequently, there is a bifurcation (saddle-node) in the geometry of the potential energy corresponding to the $x$ DoF as the value of the asymmetry parameter $\delta$ is varied. If $\delta \in [0,\delta_c)$, then we have three real roots. At the critical values $\delta = \delta_c$, $x_2$ and $x_3$ merge and thus the cubic polynomial has two real roots $x_1 = \mu$ and $x_2 = x_3 = -\mu / 2 = -\sqrt{\alpha/6}$. Moreover, for values of the asymmetry parameter $\delta > \delta_c$ only the root $x_1$ remains. In summary, when $0 \leq \delta < \delta_c$, the PES has 9 critical points, for $\delta = \delta_c$ only 6 critical points remain, since in each line $y = -\sqrt{2}/2, \, 0, \, -\sqrt{2}/2$ two roots merge (we have three simultaneous saddle-node bifurcations). Finally, for $\delta > \delta_c$ we are left with 3 critical points of the PES. We give below the energies of the critical points of the PES:

\begin{eqnarray}
	(x_k,0) \quad ,& \quad  &V(x_k,0) = -\dfrac{x_k}{2} \left(\alpha x_k + \dfrac{3\delta}{2}\right) \;,\quad k \in \lbrace 1,2,3 \rbrace \\[.2cm]
	\left(x_k,\pm \sqrt{2}/2\right) \quad ,& \quad &V(x_k,\pm \sqrt{2}/2) = -\dfrac{x_k}{2} \left(\alpha x_k + \dfrac{3}{2}\delta \right) - \frac{1}{4} \;,\quad k \in \lbrace 1,2,3 \rbrace
\end{eqnarray}

In particular, the energies of the critical points where the bifurcations take place is:

\begin{equation}\label{energyeq}
	V(-\sqrt{\alpha/6},0) = \frac{\alpha^2}{12} \quad.\quad V(-\sqrt{\alpha/6},\pm \sqrt{2}/2) = \frac{\alpha^2-3}{12}
\end{equation}

In Fig. [fig:5](#fig:asym_pot_dif_delta) we show the potential energy surface for different values of $\delta$. We can see that for different $\delta$ the critical points vary. In the first row is depicted the surface for $\delta = 0.2 < \delta_{c}$. In this case the PES has four minima (wells), four index-1 saddles and one index-2 saddle point ('hilltop'), notice that the location of the critical points is different than the symmetric case and the index-2 saddle is not at the begging of the axis (0,0) anymore. Another difference between this case and the symmetric one, is that the energies have changed and the index-1 saddles don't have equal energies anymore. When $\delta = \delta_{c}$ (second row) the PES has 6 critical points, we can see the creation of a cusp and the three critical points in the middle of the panel B have collide simultaneously, saddle node bifurcations, with the three critical points on the left of the panel B. In the last row we can see that after the bifurcation occurs for $\delta = 0.8$ the PES has only three critical points left, one index-1 saddle and the two wells on the right.

## Implications for Reaction Dynamics: Asymmetric and Uncoupled Dynamics

In Fig. [fig:6](#fig:LD_delta_02) we calculate LDs for the fixed value of the asymmetric parameter $\delta = 0.2$ and for different energies. In particular in this figure we have calculated LDs for an integration time of $\tau = 5$ in the phase space slice $y = -1/\sqrt{2}$. We discuss the phase space structures for different energy values. For instance, in Fig. [fig:6](#fig:LD_delta_02) A) we use an energy value of $H = -0.2$ which is below the energy of the left index-1 saddle and below the energy of the index-2 saddle. Hence, the bottleneck that connects the lower-left and upper-left wells is still closed. The only option for trajectories that start on the lower-left well is to be trapped in that well or to evolve to the lower-right well by crossing the bottom index-1 saddle. In panel B) we use an energy of $H=-0.1$ which is above the energy of the left index-1 and still below the energy of index-2 saddle. In this case trajectories starting on the lower-left well have three different types of motion. They can be trapped in the well forever, or they can move to the lower-right or upper-left wells, since the phase space bottleneck connecting the wells is now open. Therefore, trajectories can move between the wells and the system would show  sequential isomerization.

In Fig. [fig:7](#fig:LD_delta_crit) we illustrate the systems' behavior using LDs for $p = 1/2$ integration time $\tau = 10$ for the asymmetric and uncoupled case $\delta = \delta_{c}$ and for three different energies based on the equations (\eqref{energyeq}) at the three bifurcations points, in particular we choose one energy $H = \alpha^{2}-3/12 =-1/6$ and one energy between $\alpha^{2}-3/12<H<\alpha^{2}/12$ that is $H = -1/12$. The magenta curve represents the energy boundary. For this value of $\delta$, that is the critical value, the three simultaneous saddle-node bifurcations have just occur and there are four wells and two index-1 saddles left. A) The energy of the system is $H = -1/6$ and corresponds to the phase space slice $y = -1/\sqrt{2}$. In this case we have two regions. One inside the spherical cylinder of the index-1 in the right and one outside of this cylinder, where is the trapping area. We choose an initial condition in each of these regions and in panel B we show their time evolution projected onto configuration space. The trajectory of any initial condition inside the spherical cylinder of index-1 (red) will remain inside the cylinder and move between the lower and upper wells in comparison to any trajectory starting outside of the cylinder will remain trapped in the well (blue), the coordinates don't change sign. C) The energy of the system is $H = -1/12$ and corresponds to the phase space slice $y = -1/\sqrt{2}$. In this case there are three regions. One inside the spherical cylinder of index-1, one area in the spherical cylinder of the parabolic point that is in the line of $x = x_{c}$ and one area outside of this. D) The phase space slide $p_y = 0$. Here we can see how the initial conditions from panel C evolve. The trajectory inside the parabolic spherical cylinder (cyan) will remain inside the cylinder, horizontal trapped motion. The initial condition (red) is trapping in the well. Finally the blue initial condition we move up and down between the two wells (sequential isomerization).

fig:5  Potential energy surface on the left column and configuration space on the right column for the asymmetric uncoupled potential in Eq. (\ref{potasymun}) for different values of the asymmetric parameter $\delta$. A) and B) $\delta = 0.2 < \delta_c$, C) and D) $\delta = \delta_c$, E) and F) $\delta = 0.8 > \delta_c$.

fig:6  LDs calculated using $p = 1/2$ and $\tau = 5$ for the asymmetric uncoupled Hamiltonian in Eq. (\ref{eqasymun}) for $\delta = 0.2$. Energies are: A) $H = -0.2$. B) $H = -0.1$.

fig:7  LDs calculated using $p = 1/2$ for $\tau = 10$ in the asymmetric and uncoupled Hamiltonian in Eq. (\ref{eqasymun}) for $\delta = \delta_c$. The left column corresponds to the phase space slice $y = -1/\sqrt{2}$ and the right column is for $p_x = 0$, except panel D) that is for $p_y = 0$. The energy of the system is: A) and B) $H = -1/6$; C) and D) $H = -1/12$ and $\tau = 10$.

# References
{% bibliography --file SNreac --cited %}


# Model Failure Explanations for Regression

**Example.** In this example, we create a model to forecast GDP growth based on macroeconomic variables, then ask why it fails to predict the 2008-2009 financial crisis. We find that a high volume of real estate loans as a percent of GDP in 2007-2008 was detrimental to model performance.

**General formulation.** More generally, we can ask why our model performed poorly on sample $X$, as measured by loss function $L$. Here, our sample is the years 2008-2009, and our loss function is $R^2$.

```python
import warnings
warnings.simplefilter('ignore')

import gshap
from gshap.datasets import load_gdp

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
from sklearn.decomposition import PCA
from sklearn.metrics import r2_score
from sklearn.neighbors import KNeighborsRegressor as KNeighborsRegressorMixin

import os
import random
from datetime import datetime

np.random.seed(0)
random.seed(0)

# directory for saving figures
# set to `None` to avoid saving
PLOT_DIR = 'plots'
```

**Model.** We train a model on a GDP dataset with annual data, in which we produce 1-year ahead GDP growth forecasts based on 6 macroeconomic variables: growth in government expenditure, investment, consumption, exports, and imports, and real estate loans from commercial banks as a percent of GDP. Our model first uses principal components analysis to reduce our data to 5 dimensions, then applies a K-nearest neighbors algorithm with 4 neighbors. The number of principal components and neighbors were tuned using Scikit-learn's grid search, training on data from 1975-1990 and validating on data from 1991-2007. Our model achieves a validation $R^2$ of .26. Retraining the model on the full training dataset (1975-2007), we achieve a training $R^2$ of .40. However, our forecasts for 2008-2009 perform worse than chance: $R^2=-3.13$.

For brevity, the hyperparameter tuning code is in a separate notebook.

```python
# define a version of KNN which begins by running PCA on input data
class KNeighborsRegressor(KNeighborsRegressorMixin):
    def __init__(self, n_neighbors=4, n_components=5):
        super().__init__(n_neighbors)
        self.n_components = n_components

    def fit(self, X, y):
        self.pca = PCA(n_components=self.n_components)
        X = self.pca.fit_transform(X)
        return super().fit(X, y)

    def predict(self, X):
        X = self.pca.transform(X)
        return super().predict(X)

# load the GDP dataset
gdp = load_gdp()
X, y = gdp.data, gdp.target.iloc[1:]

# lag the X variables by 1 time step (1 year)
X, date = X.drop('date', axis=1), pd.to_datetime(X['date']).iloc[1:]
X_lagged = X.shift(1).iloc[1:]

# train on years 1975-2007; predict on 2008-2009
train_idx = date < datetime(2008, 1, 1)
test_idx = ~train_idx & (date <= datetime(2009, 1, 1))
X_train, X_test, y_train, y_test = (
    X_lagged[train_idx], X_lagged[test_idx],
    y[train_idx], y[test_idx]
)

# training and evaluation
reg = KNeighborsRegressor().fit(X_train, y_train)
y_pred_train = reg.predict(X_train)
y_pred_test = reg.predict(X_test)
print('Train score: % .4f' % r2_score(y_train, y_pred_train))
print('Test score: % .4f' % r2_score(y_test, y_pred_test))
```

    Train score:  0.3958
    Test score: -3.1278

From this figure, it is clear that our model fails to forecast the financial crisis.

```python
date_plt = date[date <= datetime(2009, 1, 1)]
y_plt = 100*np.concatenate((y_train, y_test))
y_pred_plt = 100*np.concatenate((y_pred_train, y_pred_test))
plt.plot(date_plt, y_plt, color='mediumblue')
plt.plot(date_plt, y_pred_plt, color='chartreuse')
plt.vlines(date_plt[y_train.shape[0]], -3.5, 6.5, color='r', linestyle='dashed')
plt.legend(('Actual', 'Forecast'))
plt.ylabel('GDP growth rate (%)')
if PLOT_DIR is not None:
    plt.savefig(os.path.join(PLOT_DIR, 'gdp_forecast.png'))
plt.show()
```

**G-SHAP values.** We compute the G-SHAP values, where $g$ is a loss function. In our case, the loss function is $R^2$.

\begin{equation}
    g(f,X,\Omega) = L(f,X,y)
    = 1-\frac
    {\sum_t (\%\Delta\text{GDP}_t - f(x_{t-1}))^2}
    {\sum_t (\%\Delta\text{GDP}_t - \overline{\%\Delta\text{GDP}})^2}
\end{equation}

Where additional arguments to $g$ are the labels for $X$; $\Omega=\{y\}$.

```python
# performance on training data
g = lambda y_pred: r2_score(y_train, y_pred)
explainer_train = gshap.KernelExplainer(reg.predict, X_train, g)
gshap_values_train = explainer_train.gshap_values(X_train, nsamples=1000)

# performance on test data
g = lambda y_pred: r2_score(y_test, y_pred)
explainer_test = gshap.KernelExplainer(reg.predict, X_train, g)
gshap_values_test = explainer_test.gshap_values(X_test, nsamples=1000)
```

**Interpretation.** For interpretation, we make two comparisons. First, we compare the training data $X_\text{train}=\{x_{t-1} : t=1975...2007\}$ to a background dataset $Z$ (the shuffled training data) in terms of $g$ using training labels; $y_\text{train}=\{\%\Delta\text{GDP}_t : t=1975...2007\}$. Second, we compare the test data $X_\text{test}=\{x_{t-1} : t=2008,2009\}$ to $Z$ in terms of $g$ using test labels; $y_\text{test}=\{\%\Delta\text{GDP}_t : t=2008,2009\}$. By making these two comparisons, we can see the extent to which feature $j$ contributed to model performance for the training data, and the extent to which feature $j$ contributed to model performance for the test data. If feature $j$'s contribution diminished between the training and test data, we can conclude it was responsible for our model's poor test performance. A negative G-SHAP value indicates that feature $j$ degraded performance.

In the first comparison, our model's $R^2$ is .40 for the training data and -.28 for the background data. Each G-SHAP value $\phi_g^j$ is the amount of this difference (.68) explained by feature $j$. We interpret the G-SHAP values for the test data comparison comparison similarly.

For model failure explanations more broadly, our model has a loss $L(f,X,y)$ for our test sample and $L(f,Z,y)$ for the background data. Each G-SHAP value $\phi_g^j$ is the amount of this difference $L(f,X,y)-L(f,Z,y)$ explained by feature $j$.

```python
g_comparison, g_background = explainer_train.compare(X_train, bootstrap_samples=1000)
print('R-squared for the training data: %.4f' % g_comparison)
print('R-squared for the shuffled background data: %.4f' % g_background)
print('Sum of G-SHAP values: %.4f' % gshap_values_train.sum())
```

    R-squared for the training data: 0.3958
    R-squared for the shuffled background data: -0.2847
    Sum of G-SHAP values: 0.6736

```python
g_comparison, g_background = explainer_test.compare(X_test, bootstrap_samples=1000)
print('R-squared for the test data: %.4f' % g_comparison)
print('R-squared for the shuffled background data: %.4f' % g_background)
print('Sum of the G-SHAP values: %.4f' % gshap_values_test.sum())
```

    R-squared for the test data: -3.1278
    R-squared for the shuffled background data: -3.9406
    Sum of the G-SHAP values: 0.8859

**Results.** This plot tells us which features contribute most to model performance. Because the performance of the background dataset differs in terms of $g$ when using the training versus test labels, we normalize the G-SHAP values to sum to 1 in order to compare performance. We see that real estate loans as a percent of GDP in 2007-2008 decreased the $R^2$ of our model's 2008-2009 GDP growth forecast by 34\%.

```python
df = pd.DataFrame({
    'Features': X.columns,
    'Train': gshap_values_train/gshap_values_train.sum(),
    'Test': gshap_values_test/gshap_values_test.sum(),
})
df = df.sort_values('Test')
df = pd.melt(
    df,
    id_vars='Features',
    value_vars=['Test', 'Train'],
    var_name='Dataset',
    value_name='G-SHAP',
)
sns.catplot(
    x='G-SHAP',
    y='Features',
    hue='Dataset',
    data=df,
    kind='bar',
    orient='h'
)
if PLOT_DIR is not None:
    plt.savefig(os.path.join(PLOT_DIR, 'gdp_gshap.png'))
plt.show()
```