Copyright (c) 2015, 2016 [Sebastian Raschka](http://sebastianraschka.com/)

[Li-Yi Wei](http://liyiwei.org/)

https://github.com/1iyiwei/pyml

[MIT License](https://github.com/1iyiwei/pyml/blob/master/LICENSE.txt)

# Python Machine Learning - Code Examples

# Chapter 5 - Compressing Data via Dimensionality Reduction

Principle component analysis (PCA)
* unsupervised

Linear discriminant analysis (LDA)
* supervised

Kernel PCA
* non-linear mapping

Note that the optional watermark extension is a small IPython notebook plugin that I developed to make the code reproducible. You can just skip the following line(s).

```python
%load_ext watermark
%watermark -a '' -u -d -v -p numpy,scipy,matplotlib,sklearn
```

    last updated: 2016-10-12

    CPython 3.5.2
    IPython 4.2.0

    numpy 1.11.1
    scipy 0.17.1
    matplotlib 1.5.1
    sklearn 0.18

*The use of `watermark` is optional. You can install this IPython extension via "`pip install watermark`". For more information, please see: https://github.com/rasbt/watermark.*

### Overview

- [Unsupervised dimensionality reduction via principal component analysis](#Unsupervised-dimensionality-reduction-via-principal-component-analysis-128)
  - [Total and explained variance](#Total-and-explained-variance)
  - [Feature transformation](#Feature-transformation)
  - [Principal component analysis in scikit-learn](#Principal-component-analysis-in-scikit-learn)
- [Supervised data compression via linear discriminant analysis](#Supervised-data-compression-via-linear-discriminant-analysis)
  - [Computing the scatter matrices](#Computing-the-scatter-matrices)
  - [Selecting linear discriminants for the new feature subspace](#Selecting-linear-discriminants-for-the-new-feature-subspace)
  - [Projecting samples onto the new feature space](#Projecting-samples-onto-the-new-feature-space)
  - [LDA via scikit-learn](#LDA-via-scikit-learn)
- [Using kernel principal component analysis for nonlinear mappings](#Using-kernel-principal-component-analysis-for-nonlinear-mappings)
  - [Kernel functions and the kernel trick](#Kernel-functions-and-the-kernel-trick)
  - [Implementing a kernel principal component analysis in Python](#Implementing-a-kernel-principal-component-analysis-in-Python)
    - [Example 1 – separating half-moon shapes](#Example-1-–-separating-half-moon-shapes)
    - [Example 2 – separating concentric circles](#Example-2-–-separating-concentric-circles)
  - [Projecting new data points](#Projecting-new-data-points)
  - [Kernel principal component analysis in scikit-learn](#Kernel-principal-component-analysis-in-scikit-learn)
- [Summary](#Summary)

```python
from IPython.display import Image

%matplotlib inline
```

# Unsupervised dimensionality reduction via principal component analysis

PCA is a common way to reduce dimensionality of a given dataset.
It can also be considered as a unsupervised learning method.

Given the input data matrix $\mathbf{X}$

Goal: find a transformation matrix $\mathbf{W}$ that will project each row $\mathbf{x}$ of $\mathbf{X}$ into a lower dimensional vector $\mathbf{z}$ so that the variances of the projected components are maximized:
$$\mathbf{z} = \mathbf{x} \mathbf{W}$$

$\mathbf{X}$: size $n \times d$ where $n$ is the number of data samples and $d$ is the input data dimensionality

$\mathbf{W}$: size $d \times k$

$\mathbf{z}$ and $\mathbf{x}$ are both row vectors with dimensionality $k$ and $d$, usually $k << d$.

In this 2D example, we want to project the dataset into 1D.
We will pick the first principle component (PC1) as it maximizes variance among projected samples.

With $\mathbf{W}$, we can project the entire input $\mathbf{X}$ into a lower dimensional space data set as:
$$\mathbf{Z} = \mathbf{X} \mathbf{W}$$

$\mathbf{Z}$: size $n \times k$

We can also recover an approximated version $\mathbf{X'}$ of $\mathbf{X}$ from $\mathbf{Z}$ and $\mathbf{W}$ as:
$$\mathbf{X'} = \mathbf{Z} \mathbf{W}^T$$

It can be shown that $\mathbf{X'}$ is the best approximation of $\mathbf{X}$, i.e. minimizing
$$||\mathbf{X'} - \mathbf{X}||^2$$

## Algorithm

$$
\mathbf{Z} = \mathbf{X} \mathbf{W}
$$

$\mathbf{W}$ can be computed from $\mathbf{X}$ as follows.

First, compute the $d \times d$ covariance matrix $\Sigma$ from the columns (i.e. features) of $\mathbf{X}$:
$$
\begin{align}
\Sigma_{ij} = \frac{1}{n} \left(\mathbf{x_{(i)}} - \mu_i\right)^T \left(\mathbf{x_{(j)}} - \mu_j\right)
\end{align}
$$
, where
* $\Sigma_{ij}$ is the $(i, j)$th component of $\Sigma$.
* $\mathbf{x_{(i)}}$ is the $i^{th}$ column/feature of $\mathbf{X}$ and $\mu_i$ its mean (a scalar).

Alternatively we can compute $\Sigma$ by summing the covariance matrices of each individual sample $x^{(i)}$ (rows of $\mathbf{X}$):
$$
\begin{align}
\Sigma = \frac{1}{n} \sum_i \left(\mathbf{x^{(i)}} - \mu\right)^T \left( \mathbf{x^{(i)} - \mu}\right)
\end{align}
$$
, where $\mu$ is the (vector) mean of all rows of $\mathbf{X}$.

We then compute the eigen-values/vectors of $\Sigma$.

Recall $\mathbf{v}$ is an eigen-vector of a matrix $\Sigma$ with eigen-value $\lambda$ if
$$\lambda \mathbf{v} = \Sigma \mathbf{v}$$
That is, an eigen-vector remains itself after transforming by the matrix.

$\mathbf{W}$ can be constructed by horizontally stacking (as columns) the eigen-vectors of $\Sigma$ with the $k$ largest eigen-values (which we assume are all non-negative) as columns.

These columns are called the principle components, and thus the name principle component analysis (PCA).

## Math

Let's try to find the first principle component $\mathbf{w_1}$ so that when the input vector $\mathbf{x}$ is projected into $\mathbf{z}$ its variance is maximized:
$$\mathbf{z} = \mathbf{w_1}^T \mathbf{x}$$

$\mathbf{x}$ differnt rows of the matrix $\mathbf{X}$ verticalized as columns.

Consider $\mathbf{x}$ as a random vector that can take values from $\mathbf{X}$:

$$
\begin{align}
Var(\mathbf{z}) &= E\left( \left(\mathbf{w_1}^T (\mathbf{x} - \mu)\right)^2 \right)
\\
&= E\left( \mathbf{w_1}^T (\mathbf{x} - \mu) (\mathbf{x} - \mu)^T \mathbf{w_1} \right)
\\
&= \mathbf{w_1}^T \Sigma \mathbf{w_1}
\end{align}
$$

We want to find $\mathbf{w_1}$ to maximize $Var(\mathbf{z})$ subject to the unit vector constraint $|\mathbf{w_1}| = 1$.

Using Lagrangian multiplier we want to maximize:
$$\mathbf{w_1}^T \Sigma \mathbf{w_1} - \alpha(\mathbf{w_1}^T\mathbf{w_1}-1)$$

Take derivative of the above with $\mathbf{w_1}$ and set it to zero we have:
$$\Sigma \mathbf{w_1} = \alpha \mathbf{w_1}$$

And thus
$$\mathbf{w_1}^T \Sigma \mathbf{w_1} = \alpha \mathbf{w_1}^T \mathbf{w_1} = \alpha$$
Which means we want to maximize $\alpha$, and thus it should be the largest eigen-value of $\Sigma$ and $\mathbf{w_1}$ the corresponding eigen-vector.

We can continue the same trick to find the rest of the principle components by making sure each new one is orthogonal to all existing ones.

## Code example for the math above

Use the wine data set as it has 13 features for dimensionality reduction

```python
import pandas as pd

df_wine = pd.read_csv('https://archive.ics.uci.edu/ml/'
                      'machine-learning-databases/wine/wine.data',
                      header=None)

df_wine.columns = ['Class label', 'Alcohol', 'Malic acid', 'Ash',
                   'Alcalinity of ash', 'Magnesium', 'Total phenols',
                   'Flavanoids', 'Nonflavanoid phenols', 'Proanthocyanins',
                   'Color intensity', 'Hue',
                   'OD280/OD315 of diluted wines', 'Proline']

df_wine.head()
```

      Class label
      Alcohol
      Malic acid
      Ash
      Alcalinity of ash
      Magnesium
      Total phenols
      Flavanoids
      Nonflavanoid phenols
      Proanthocyanins
      Color intensity
      Hue
      OD280/OD315 of diluted wines
      Proline

      0
      1
      14.23
      1.71
      2.43
      15.6
      127
      2.80
      3.06
      0.28
      2.29
      5.64
      1.04
      3.92
      1065

      1
      1
      13.20
      1.78
      2.14
      11.2
      100
      2.65
      2.76
      0.26
      1.28
      4.38
      1.05
      3.40
      1050

      2
      1
      13.16
      2.36
      2.67
      18.6
      101
      2.80
      3.24
      0.30
      2.81
      5.68
      1.03
      3.17
      1185

      3
      1
      14.37
      1.95
      2.50
      16.8
      113
      3.85
      3.49
      0.24
      2.18
      7.80
      0.86
      3.45
      1480

      4
      1
      13.24
      2.59
      2.87
      21.0
      118
      2.80
      2.69
      0.39
      1.82
      4.32
      1.04
      2.93
      735

### Note:

If the link to the Wine dataset provided above does not work for you, you can find a local copy in this repository at [./../datasets/wine/wine.data](./../datasets/wine/wine.data).

Or you could fetch it via

```python
df_wine = pd.read_csv('https://raw.githubusercontent.com/1iyiwei/pyml/master/code/datasets/wine/wine.data', header=None)

df_wine.columns = ['Class label', 'Alcohol', 'Malic acid', 'Ash',
'Alcalinity of ash', 'Magnesium', 'Total phenols',
'Flavanoids', 'Nonflavanoid phenols', 'Proanthocyanins',
'Color intensity', 'Hue', 'OD280/OD315 of diluted wines', 'Proline']
df_wine.head()
```

      Class label
      Alcohol
      Malic acid
      Ash
      Alcalinity of ash
      Magnesium
      Total phenols
      Flavanoids
      Nonflavanoid phenols
      Proanthocyanins
      Color intensity
      Hue
      OD280/OD315 of diluted wines
      Proline

      0
      1
      14.23
      1.71
      2.43
      15.6
      127
      2.80
      3.06
      0.28
      2.29
      5.64
      1.04
      3.92
      1065

      1
      1
      13.20
      1.78
      2.14
      11.2
      100
      2.65
      2.76
      0.26
      1.28
      4.38
      1.05
      3.40
      1050

      2
      1
      13.16
      2.36
      2.67
      18.6
      101
      2.80
      3.24
      0.30
      2.81
      5.68
      1.03
      3.17
      1185

      3
      1
      14.37
      1.95
      2.50
      16.8
      113
      3.85
      3.49
      0.24
      2.18
      7.80
      0.86
      3.45
      1480

      4
      1
      13.24
      2.59
      2.87
      21.0
      118
      2.80
      2.69
      0.39
      1.82
      4.32
      1.04
      2.93
      735

Splitting the data into 70% training and 30% test subsets.

```python
from distutils.version import LooseVersion as Version
from sklearn import __version__ as sklearn_version

if Version(sklearn_version) < '0.18':
    from sklearn.cross_validation import train_test_split
else:
    from sklearn.model_selection import train_test_split

X, y = df_wine.iloc[:, 1:].values, df_wine.iloc[:, 0].values

X_train, X_test, y_train, y_test = \
    train_test_split(X, y, test_size=0.3, random_state=0)
```

Standardizing the data.

```python
from sklearn.preprocessing import StandardScaler

sc = StandardScaler()
X_train_std = sc.fit_transform(X_train)
X_test_std = sc.transform(X_test)
```

---

**Note**

Accidentally, I wrote `X_test_std = sc.fit_transform(X_test)` instead of `X_test_std = sc.transform(X_test)`. In this case, it wouldn't make a big difference since the mean and standard deviation of the test set should be (quite) similar to the training set. However, as remember from Chapter 3, the correct way is to re-use parameters from the training set if we are doing any kind of transformation -- the test set should basically stand for "new, unseen" data.

My initial typo reflects a common mistake is that some people are *not* re-using these parameters from the model training/building and standardize the new data "from scratch." Here's simple example to explain why this is a problem.

Let's assume we have a simple training set consisting of 3 samples with 1 feature (let's call this feature "length"):

- train_1: 10 cm -> class_2
- train_2: 20 cm -> class_2
- train_3: 30 cm -> class_1

mean: 20, std.: 8.2

After standardization, the transformed feature values are

- train_std_1: -1.21 -> class_2
- train_std_2: 0 -> class_2
- train_std_3: 1.21 -> class_1

Next, let's assume our model has learned to classify samples with a standardized length value < 0.6 as class_2 (class_1 otherwise). So far so good. Now, let's say we have 3 unlabeled data points that we want to classify:

- new_4: 5 cm -> class ?
- new_5: 6 cm -> class ?
- new_6: 7 cm -> class ?

If we look at the "unstandardized "length" values in our training datast, it is intuitive to say that all of these samples are likely belonging to class_2. However, if we standardize these by re-computing standard deviation and and mean you would get similar values as before in the training set and your classifier would (probably incorrectly) classify samples 4 and 5 as class 2.

- new_std_4: -1.21 -> class 2
- new_std_5: 0 -> class 2
- new_std_6: 1.21 -> class 1

However, if we use the parameters from your "training set standardization," we'd get the values:

- sample5: -18.37 -> class 2
- sample6: -17.15 -> class 2
- sample7: -15.92 -> class 2

The values 5 cm, 6 cm, and 7 cm are much lower than anything we have seen in the training set previously. Thus, it only makes sense that the standardized features of the "new samples" are much lower than every standardized feature in the training set.

---

Eigendecomposition of the covariance matrix:

```python
import numpy as np
cov_mat = np.cov(X_train_std.T)
eigen_vals, eigen_vecs = np.linalg.eig(cov_mat)

print('\nEigenvalues \n%s' % eigen_vals)
```

    Eigenvalues
    [ 4.8923083   2.46635032  1.42809973  1.01233462  0.84906459  0.60181514
      0.52251546  0.08414846  0.33051429  0.29595018  0.16831254  0.21432212
      0.2399553 ]

**Note**:

Above, I used the [`numpy.linalg.eig`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.eig.html) function to decompose the symmetric covariance matrix into its eigenvalues and eigenvectors.

>>> eigen_vals, eigen_vecs = np.linalg.eig(cov_mat)

    This is not really a "mistake," but probably suboptimal. It would be better to use [`numpy.linalg.eigh`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.eigh.html) in such cases, which has been designed for [Hermetian matrices](https://en.wikipedia.org/wiki/Hermitian_matrix). The latter always returns real  eigenvalues; whereas the numerically less stable `np.linalg.eig` can decompose nonsymmetric square matrices, you may find that it returns complex eigenvalues in certain cases. (S.R.)

### Total and explained variance

```python
tot = sum(eigen_vals)
var_exp = [(i / tot) for i in sorted(eigen_vals, reverse=True)]
cum_var_exp = np.cumsum(var_exp)
```

```python
import matplotlib.pyplot as plt

plt.bar(range(1, 14), var_exp, alpha=0.5, align='center',
        label='individual explained variance')
plt.step(range(1, 14), cum_var_exp, where='mid',
         label='cumulative explained variance')
plt.ylabel('Explained variance ratio')
plt.xlabel('Principal components')
plt.legend(loc='best')
plt.tight_layout()
# plt.savefig('./figures/pca1.png', dpi=300)
plt.show()
```

## Feature transformation

Now let's apply PCA ...

```python
# Make a list of (eigenvalue, eigenvector) tuples
eigen_pairs = [(np.abs(eigen_vals[i]), eigen_vecs[:, i])
               for i in range(len(eigen_vals))]

# Sort the (eigenvalue, eigenvector) tuples from high to low
eigen_pairs.sort(key=lambda k: k[0], reverse=True)

# Note: I added the `key=lambda k: k[0]` in the sort call above
# just like I used it further below in the LDA section.
# This is to avoid problems if there are ties in the eigenvalue
# arrays (i.e., the sorting algorithm will only regard the
# first element of the tuples, now).
```

```python
w = np.hstack((eigen_pairs[0][1][:, np.newaxis],
               eigen_pairs[1][1][:, np.newaxis]))
print('Matrix W:\n', w)
```

    Matrix W:
     [[ 0.14669811  0.50417079]
     [-0.24224554  0.24216889]
     [-0.02993442  0.28698484]
     [-0.25519002 -0.06468718]
     [ 0.12079772  0.22995385]
     [ 0.38934455  0.09363991]
     [ 0.42326486  0.01088622]
     [-0.30634956  0.01870216]
     [ 0.30572219  0.03040352]
     [-0.09869191  0.54527081]
     [ 0.30032535 -0.27924322]
     [ 0.36821154 -0.174365  ]
     [ 0.29259713  0.36315461]]

**Note**
Depending on which version of NumPy and LAPACK you are using, you may obtain the the Matrix W with its signs flipped. E.g., the matrix shown in the book was printed as:

```
[[ 0.14669811  0.50417079]
[-0.24224554  0.24216889]
[-0.02993442  0.28698484]
[-0.25519002 -0.06468718]
[ 0.12079772  0.22995385]
[ 0.38934455  0.09363991]
[ 0.42326486  0.01088622]
[-0.30634956  0.01870216]
[ 0.30572219  0.03040352]
[-0.09869191  0.54527081]
```

Please note that this is not an issue: If $v$ is an eigenvector of a matrix $\Sigma$, we have

$$\Sigma v = \lambda v,$$

where $\lambda$ is our eigenvalue,

then $-v$ is also an eigenvector that has the same eigenvalue, since

$$\Sigma(-v) = -\Sigma v = -\lambda v = \lambda(-v).$$

```python
X_train_pca = X_train_std.dot(w)
colors = ['r', 'b', 'g']
markers = ['s', 'x', 'o']

for l, c, m in zip(np.unique(y_train), colors, markers):
    plt.scatter(X_train_pca[y_train == l, 0],
                X_train_pca[y_train == l, 1],
                c=c, label=l, marker=m)

plt.xlabel('PC 1')
plt.ylabel('PC 2')
plt.legend(loc='lower left')
plt.tight_layout()
# plt.savefig('./figures/pca2.png', dpi=300)
plt.show()
```

Notice the nicely formed clusters, even though PCA does not consider class labels (unsupervised).

```python
X_train_std[0].dot(w)
```

    array([ 2.59891628,  0.00484089])

What happens if we use the last two eigen-vectors?

```python
w_tail = np.hstack((eigen_pairs[-1][1][:, np.newaxis],
                    eigen_pairs[-2][1][:, np.newaxis]))
print('Matrix W (tail end):\n', w_tail)
```

    Matrix W (tail end):
     [[-0.05546872  0.27566086]
     [ 0.03327316 -0.0813845 ]
     [-0.10061857 -0.01297513]
     [ 0.05616586  0.0989088 ]
     [ 0.09584239 -0.09592977]
     [-0.42126512  0.28389764]
     [ 0.8472247   0.11672921]
     [ 0.1662568  -0.03965663]
     [-0.16619747  0.08606027]
     [ 0.03961736 -0.57165189]
     [-0.10538369 -0.19884453]
     [-0.09950556 -0.65086971]
     [-0.01606618  0.07123771]]

```python
X_train_pca = X_train_std.dot(w_tail)
colors = ['r', 'b', 'g']
markers = ['s', 'x', 'o']

for l, c, m in zip(np.unique(y_train), colors, markers):
    plt.scatter(X_train_pca[y_train == l, 0],
                X_train_pca[y_train == l, 1],
                c=c, label=l, marker=m)

plt.xlabel('PC -1')
plt.ylabel('PC -2')
plt.legend(loc='lower left')
plt.tight_layout()
# plt.savefig('./figures/pca2.png', dpi=300)
plt.show()
```

Notice the badly formed clusters!

## Principal component analysis in scikit-learn

PCA is actually part of scikit-learn, so we can use it directly instead of going through the code above.

```python
from sklearn.decomposition import PCA

pca = PCA()
X_train_pca = pca.fit_transform(X_train_std)
pca.explained_variance_ratio_
```

    array([ 0.37329648,  0.18818926,  0.10896791,  0.07724389,  0.06478595,
            0.04592014,  0.03986936,  0.02521914,  0.02258181,  0.01830924,
            0.01635336,  0.01284271,  0.00642076])

```python
plt.bar(range(1, 14), pca.explained_variance_ratio_, alpha=0.5, align='center')
plt.step(range(1, 14), np.cumsum(pca.explained_variance_ratio_), where='mid')
plt.ylabel('Explained variance ratio')
plt.xlabel('Principal components')
plt.show()
```

```python
pca = PCA(n_components=2)
X_train_pca = pca.fit_transform(X_train_std)
X_test_pca = pca.transform(X_test_std)
```

```python
plt.scatter(X_train_pca[:, 0], X_train_pca[:, 1])
plt.xlabel('PC 1')
plt.ylabel('PC 2')
plt.show()
```

```python
from matplotlib.colors import ListedColormap

def plot_decision_regions(X, y, classifier, resolution=0.02):

    # setup marker generator and color map
    markers = ('s', 'x', 'o', '^', 'v')
    colors = ('red', 'blue', 'lightgreen', 'gray', 'cyan')
    cmap = ListedColormap(colors[:len(np.unique(y))])

    # plot the decision surface
    x1_min, x1_max = X[:, 0].min() - 1, X[:, 0].max() + 1
    x2_min, x2_max = X[:, 1].min() - 1, X[:, 1].max() + 1
    xx1, xx2 = np.meshgrid(np.arange(x1_min, x1_max, resolution),
                           np.arange(x2_min, x2_max, resolution))
    Z = classifier.predict(np.array([xx1.ravel(), xx2.ravel()]).T)
    Z = Z.reshape(xx1.shape)
    plt.contourf(xx1, xx2, Z, alpha=0.4, cmap=cmap)
    plt.xlim(xx1.min(), xx1.max())
    plt.ylim(xx2.min(), xx2.max())

    # plot class samples
    for idx, cl in enumerate(np.unique(y)):
        plt.scatter(x=X[y == cl, 0], y=X[y == cl, 1],
                    alpha=0.8, c=cmap(idx),
                    marker=markers[idx], label=cl)
```

Training logistic regression classifier using the first 2 principal components.

```python
from sklearn.linear_model import LogisticRegression

lr = LogisticRegression()
lr = lr.fit(X_train_pca, y_train)
```

```python
plot_decision_regions(X_train_pca, y_train, classifier=lr)
plt.xlabel('PC 1')
plt.ylabel('PC 2')
plt.legend(loc='lower left')
plt.tight_layout()
# plt.savefig('./figures/pca3.png', dpi=300)
plt.show()
```

```python
plot_decision_regions(X_test_pca, y_test, classifier=lr)
plt.xlabel('PC 1')
plt.ylabel('PC 2')
plt.legend(loc='lower left')
plt.tight_layout()
# plt.savefig('./figures/pca4.png', dpi=300)
plt.show()
```

```python
pca = PCA(n_components=None)
X_train_pca = pca.fit_transform(X_train_std)
pca.explained_variance_ratio_
```

    array([ 0.37329648,  0.18818926,  0.10896791,  0.07724389,  0.06478595,
            0.04592014,  0.03986936,  0.02521914,  0.02258181,  0.01830924,
            0.01635336,  0.01284271,  0.00642076])

# Supervised data compression via linear discriminant analysis

PCA
* unsupervised (no class information)
* project data into dimensions that maximize variance/spread

LDA
* supervised (with class information)
* project data into dimensions to (1) maximize inter-class spread, (2) minimize intra-class spread.

### Exercise
Under what circumstances would PCA and LDA produce very different results? Provide some intuitive examples.

## Algorithm

Similar to PCA, given a data matrix $\mathbf{X}$, we want to calculate a projection matrix $\mathbf{W}$ which separates the projected vectors as much as possible.
Unlike PCA which is unsupervised, for LDA we have the class information.
Thus, the goal is to spread out different classes while cluster each individual classes via projection $\mathbf{W}$.

Assume we have $K$ classes, each with center $\mu_{i}$ as computed from the mean of all $n_i$ samples within class $i$.
$\mu$ is the mean of all samples across all classes.

Below, each $\mu$ and $\mathbf{x}$ is a row vector, and the transpose $T$ is a column vector.

We first compute the between-class scatter matrix:
$$\mathbf{S_B} = \sum_{i=1}^K n_i (\mu_i - \mu)^T (\mu_i - \mu)$$
And the within-class scatter matrix:
$$
\begin{align}
\mathbf{S_i} & = \sum_{\mathbf{x} \in C_i} (\mathbf{x} - \mu_i)^T (\mathbf{x} - \mu_i) \\
\mathbf{S_W} & = \sum_{i=1}^K \mathbf{S_i}
\end{align}
$$

Note: these scatter matrices are very similar to the covariance matrices except for scaling constants.

We then perform eigen decomposition of
$$\mathbf{S_W}^{-1}\mathbf{S_B}$$
And construct $\mathbf{W}$ from the first $k$ eigen-vectors with the largest eigen-values.
This step is similar to PCA, except that we use the above matrix instead of $\Sigma$, the covariance matrix of all input data $\mathbf{X}$.

Intuitively, since we want to maximize the spread with $\mathbf{S_B}$ and minimize the spread with $\mathbf{S_W}$, we want to perform the eigen decomposition via $\mathbf{S_W}^{-1}\mathbf{S_B}$.

## Math
Below, we first discuss how to compute such inter and intra class spreads, followed by how to optimize $\mathbf{W}$.

The between/inter-class spread can be computed as the scatter/covariance of the projected class centers weighted by the class sizes:
$$\sum_{i=1}^K  n_i \left(\mathbf{W}^T (\mu_i - \mu) \right)^2 = \mathbf{W}^T \left( \sum_{i=1}^K n_i (\mu_i - \mu)^T (\mu_i - \mu) \right) \mathbf{W} = \mathbf{W}^T \mathbf{S_B}\mathbf{W}$$

The within/intra-class spread of each projected class $i$ can be computed analogously:
$$\sum_{\mathbf{x} \in C_i} \left(\mathbf{W}^T (\mathbf{x}-\mu_i)\right)^2 = \mathbf{W}^T \mathbf{S_i} \mathbf{W}$$
And thus the total within/intra-class spread is:
$$\sum_{i=1}^K \mathbf{W}^T \mathbf{S_i} \mathbf{W} = \mathbf{W}^T \mathbf{S_W} \mathbf{W}$$

The goal of maximize/minimize inter/intra-class spread can be formulated as maximizing the ratio of determinants:
$$J(\mathbf{W}) = \frac{\left|\mathbf{W}^T \mathbf{S_B} \mathbf{W}\right|}{\left|\mathbf{W}^T \mathbf{S_W} \mathbf{W}\right|}$$

Recall that the determinant of a matrix is the product of its eigen-values.

Linear algebra can show that constructing $\mathbf{W}$ from the largest eigen-vectors of $\mathbf{S_W}^{-1}\mathbf{S_B}$ can maximize $J(\mathbf{W})$ above.

## Code example

Computing the scatter matrices

Calculate the mean vectors for each class:

```python
np.set_printoptions(precision=4)

mean_vecs = []
for label in range(1, 4):
    mean_vecs.append(np.mean(X_train_std[y_train == label], axis=0))
    print('MV %s: %s\n' % (label, mean_vecs[label - 1]))
```

    MV 1: [ 0.9259 -0.3091  0.2592 -0.7989  0.3039  0.9608  1.0515 -0.6306  0.5354
      0.2209  0.4855  0.798   1.2017]

    MV 2: [-0.8727 -0.3854 -0.4437  0.2481 -0.2409 -0.1059  0.0187 -0.0164  0.1095
     -0.8796  0.4392  0.2776 -0.7016]

    MV 3: [ 0.1637  0.8929  0.3249  0.5658 -0.01   -0.9499 -1.228   0.7436 -0.7652
      0.979  -1.1698 -1.3007 -0.3912]

Compute the within-class scatter matrix:

```python
d = 13  # number of features
S_W = np.zeros((d, d))
for label, mv in zip(range(1, 4), mean_vecs):
    class_scatter = np.zeros((d, d))  # scatter matrix for each class
    for row in X_train_std[y_train == label]:
        row, mv = row.reshape(d, 1), mv.reshape(d, 1)  # make column vectors
        class_scatter += (row - mv).dot((row - mv).T)
    S_W += class_scatter                          # sum class scatter matrices

print('Within-class scatter matrix: %sx%s' % (S_W.shape[0], S_W.shape[1]))
```

    Within-class scatter matrix: 13x13

Better: covariance matrix since classes are not equally distributed:

```python
print('Class label distribution: %s'
      % np.bincount(y_train)[1:])
```

    Class label distribution: [40 49 35]

```python
d = 13  # number of features
S_W = np.zeros((d, d))
for label, mv in zip(range(1, 4), mean_vecs):
    class_scatter = np.cov(X_train_std[y_train == label].T)
    S_W += class_scatter
print('Scaled within-class scatter matrix: %sx%s' % (S_W.shape[0],
                                                     S_W.shape[1]))
```

    Scaled within-class scatter matrix: 13x13

Compute the between-class scatter matrix:

```python
mean_overall = np.mean(X_train_std, axis=0)
d = 13  # number of features
S_B = np.zeros((d, d))
for i, mean_vec in enumerate(mean_vecs):
    n = X_train[y_train == i + 1, :].shape[0]
    mean_vec = mean_vec.reshape(d, 1)  # make column vector
    mean_overall = mean_overall.reshape(d, 1)  # make column vector
    S_B += n * (mean_vec - mean_overall).dot((mean_vec - mean_overall).T)

print('Between-class scatter matrix: %sx%s' % (S_B.shape[0], S_B.shape[1]))
```

    Between-class scatter matrix: 13x13

## Selecting linear discriminants for the new feature subspace

Solve the generalized eigenvalue problem for the matrix $S_W^{-1}S_B$:

```python
eigen_vals, eigen_vecs = np.linalg.eig(np.linalg.inv(S_W).dot(S_B))
```

**Note**:

Above, I used the [`numpy.linalg.eig`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.eig.html) function to decompose the symmetric covariance matrix into its eigenvalues and eigenvectors.

>>> eigen_vals, eigen_vecs = np.linalg.eig(cov_mat)

    This is not really a "mistake," but probably suboptimal. It would be better to use [`numpy.linalg.eigh`](http://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.eigh.html) in such cases, which has been designed for [Hermetian matrices](https://en.wikipedia.org/wiki/Hermitian_matrix). The latter always returns real  eigenvalues; whereas the numerically less stable `np.linalg.eig` can decompose nonsymmetric square matrices, you may find that it returns complex eigenvalues in certain cases. (S.R.)

Sort eigenvectors in decreasing order of the eigenvalues:

```python
# Make a list of (eigenvalue, eigenvector) tuples
eigen_pairs = [(np.abs(eigen_vals[i]), eigen_vecs[:, i])
               for i in range(len(eigen_vals))]

# Sort the (eigenvalue, eigenvector) tuples from high to low
eigen_pairs = sorted(eigen_pairs, key=lambda k: k[0], reverse=True)

# Visually confirm that the list is correctly sorted by decreasing eigenvalues

print('Eigenvalues in decreasing order:\n')
for eigen_val in eigen_pairs:
    print(eigen_val[0])
```

    Eigenvalues in decreasing order:

    452.721581245
    156.43636122
    7.19589628751e-14
    5.68434188608e-14
    3.81852082594e-14
    2.7235398212e-14
    2.7235398212e-14
    2.67247705375e-14
    2.67247705375e-14
    1.19267702437e-14
    1.19267702437e-14
    1.10185855395e-14
    1.16091559497e-15

```python
tot = sum(eigen_vals.real)
discr = [(i / tot) for i in sorted(eigen_vals.real, reverse=True)]
cum_discr = np.cumsum(discr)

plt.bar(range(1, 14), discr, alpha=0.5, align='center',
        label='individual "discriminability"')
plt.step(range(1, 14), cum_discr, where='mid',
         label='cumulative "discriminability"')
plt.ylabel('"discriminability" ratio')
plt.xlabel('Linear Discriminants')
plt.ylim([-0.1, 1.1])
plt.legend(loc='best')
plt.tight_layout()
# plt.savefig('./figures/lda1.png', dpi=300)
plt.show()
```

```python
w = np.hstack((eigen_pairs[0][1][:, np.newaxis].real,
              eigen_pairs[1][1][:, np.newaxis].real))
print('Matrix W:\n', w)
```

    Matrix W:
     [[-0.0662 -0.3797]
     [ 0.0386 -0.2206]
     [-0.0217 -0.3816]
     [ 0.184   0.3018]
     [-0.0034  0.0141]
     [ 0.2326  0.0234]
     [-0.7747  0.1869]
     [-0.0811  0.0696]
     [ 0.0875  0.1796]
     [ 0.185  -0.284 ]
     [-0.066   0.2349]
     [-0.3805  0.073 ]
     [-0.3285 -0.5971]]

## Projecting samples onto the new feature space

```python
X_train_lda = X_train_std.dot(w)
colors = ['r', 'b', 'g']
markers = ['s', 'x', 'o']

for l, c, m in zip(np.unique(y_train), colors, markers):
    plt.scatter(X_train_lda[y_train == l, 0] * (-1),
                X_train_lda[y_train == l, 1] * (-1),
                c=c, label=l, marker=m)

plt.xlabel('LD 1')
plt.ylabel('LD 2')
plt.legend(loc='lower right')
plt.tight_layout()
# plt.savefig('./figures/lda2.png', dpi=300)
plt.show()
```

## LDA via scikit-learn

```python
#from sklearn.lda import LDA # deprecated
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA

lda = LDA(n_components=2)
X_train_lda = lda.fit_transform(X_train_std, y_train)
```

```python
from sklearn.linear_model import LogisticRegression
lr = LogisticRegression()
lr = lr.fit(X_train_lda, y_train)

plot_decision_regions(X_train_lda, y_train, classifier=lr)
plt.xlabel('LD 1')
plt.ylabel('LD 2')
plt.legend(loc='lower left')
plt.tight_layout()
# plt.savefig('./images/lda3.png', dpi=300)
plt.show()
```

```python
X_test_lda = lda.transform(X_test_std)

plot_decision_regions(X_test_lda, y_test, classifier=lr)
plt.xlabel('LD 1')
plt.ylabel('LD 2')
plt.legend(loc='lower left')
plt.tight_layout()
# plt.savefig('./images/lda4.png', dpi=300)
plt.show()
```

# Using kernel principal component analysis for nonlinear mappings

PCA/LDA problematic for non-linearly separable data

Idea:
1. elevate the dimension of the input data (similar to kernel SVM)
2. reduce the dimension (similar to PCA)

Projected data becomes linearly separable

Thus:
* the projected data can then be handled by linear classifiers
* why it make sense to increase dimension before reduce it

Note: PCA is unsupervised, but it matters whether the projected data is suitable for further classification.

## Algorithm

Let $\mathbf{X}$ be the usual matrix of the input data set, with size $n \times d$, where $n$ is the number of data vectors and $d$ is their dimensionality.

Similar to kernel SVM, we want to elevate each data vector into a higher $k$-dimensional space via a function $\phi$ (usually $k >> d$).

Specifically, we denote $\phi(\mathbf{X})$ as the matrix for which the ith row is $\phi(X^{(i)})$.

Ordinary PCA performs eigen analysis of the covariance matrix of $\mathbf{X}$:
$$\Sigma = \frac{1}{n} \mathbf{X}^T \mathbf{X} $$

Kernel PCA performs eigen analysis of the elevated covariance matrix:
$$\Sigma = \frac{1}{n} \phi(\mathbf{X})^T \phi(\mathbf{X})$$

Now $\Sigma$ is of size $k \times k$, which is very large and thus expensive to compute.

Fortunately, all we need to know is to
1. compute its eigen vectors
2. project all input vectors into the lower dimensional space formed by the selected eigen vectors (with largest eigen values similar to traditional PCA)

That is, we actually never need to know the eigen vectors explicitly, only their dot products with the input vectors.
This is where the kernel trick comes in, by replacing high dimensional dot products with fast kernel evaluations.

Specifically, we just need to compute
$$
\mathbf{K} = \phi(\mathbf{X}) \phi(\mathbf{X})^T
$$
, a $n \times n$ matrix, much smaller than $\Sigma$, via kernel trick.

The projection of $\phi(\mathbf{X})$ into $m$-dimension can be found from the $m$ largest eigen-vectors of $\mathbf{K}$.

## Math

$$\Sigma = \frac{1}{n} \phi(\mathbf{X})^T \phi(\mathbf{X})$$

Let $\mathbf{v}$ be an eigen vector of $\Sigma$ with eigen value $\lambda$:
$$\Sigma \mathbf{v} = \lambda \mathbf{v}$$

And for the elevated data matrix $\phi(\mathbf{X})$, we just need to know its projection with $\mathbf{v}$:
$$\mathbf{a} = \phi(\mathbf{X}) \mathbf{v}$$

Note that
$$
\begin{align}
\mathbf{a} &= \phi(\mathbf{X}) \mathbf{v}
\\
&= \frac{1}{\lambda} \phi(\mathbf{X}) \Sigma \mathbf{v}
\\
&= \frac{1}{\lambda n} \phi(\mathbf{X}) \phi(\mathbf{X})^T \phi(\mathbf{X}) \mathbf{v}
\\
&= \frac{1}{\lambda n} \phi(\mathbf{X}) \phi(\mathbf{X})^T \mathbf{a}
\end{align}
$$

If we denote
$$\mathbf{K} = \phi(\mathbf{X}) \phi(\mathbf{X})^T$$
we have
$$\lambda \mathbf{a} = \frac{\mathbf{K}}{n} \mathbf{a}$$

Note: $\Sigma$ and $\frac{\mathbf{K}}{n}$ have the same eigen values

Thus, $\mathbf{a}$ can be computed as an eigen vector of $\frac{\mathbf{K}}{n}$, where $\mathbf{K}$, the similarity (kernel) matrix, has size $n \times n$ is thus much smaller than $\Sigma$ with size $k \times k$.

Furthermore, each entry of $\mathbf{K}$ can be computed via fast kernel evaluation
$$\mathbf{K}_{ij} = \mathbf{k}\left(\mathbf{x}^{(i)}, \mathbf{x}^{(j)}\right)$$
instead of the original dot product between two $k$-dimensional vectors
$$\mathbf{K}_{ij} = \phi\left(\mathbf{x^{(i)}})^T \phi(\mathbf{x^{(j)}}\right)$$

This so called kernel trick, of approximating high dimensional dot products with fast kernel evaluatioin, shows up again, after what we have seen in the kernel SVM part.

### Mean shift

Recall that in (ordinary) PCA, each entry of $\Sigma$ is a covariance:
$$\Sigma_{ij} = \frac{1}{n} (\mathbf{x_{(i)}} - \mathbf{\mu_i})^T (\mathbf{x_{(j)}} - \mathbf{\mu_j})$$
, where $\mathbf{\mu}$ is the mean of all $\mathbf{x}$, i.e. the rows of $\mathbf{X}$.

For kernel PCA, we need to perform a similar mean shift for $\mathbf{K}$.
Specifically, since $\mathbf{K}$ is the covariance matrix of $\phi(\mathbf{x})$, we have
$$\mathbf{\mu} = \frac{1}{n} \sum_{k=1}^n \phi(x^{(k)})$$
And each entry of the mean-shifted $\mathbf{K'}$ is:
$$
\begin{align}
\mathbf{K'}_{ij} & = \left(\phi(\mathbf{x^{(i)}}) - \mathbf{\mu}\right) \left(\phi(\mathbf{x^{(j)}}) - \mathbf{\mu}\right)^T \\
& = \phi(\mathbf{x^{(i)}}) \phi(\mathbf{x^{(j)}})^T - \mathbf{\mu} \phi(\mathbf{x^{(j)}})^T - \phi(\mathbf{x^{(i)}}) \mathbf{\mu}^T + \mathbf{\mu} \mathbf{\mu}^T \\
& = \mathbf{k}(\mathbf{x^{(i)}}, \mathbf{x^{(j)}}) - \frac{1}{n} \sum_{i=1}^n \mathbf{k}(\mathbf{x^{(i)}}, \mathbf{x^{(j)}}) - \frac{1}{n} \sum_{j=1}^n \mathbf{k}(\mathbf{x^{(i)}}, \mathbf{x^{(j)}})
\\
&+ \frac{1}{n^2} \sum_{i=1}^n \sum_{j=1}^n \mathbf{k}(\mathbf{x^{(i)}}, \mathbf{x^{(j)}})
\end{align}
$$
Coalescing all entries $\mathbf{K'_{ij}}$ into $\mathbf{K'}$ we have
$$\mathbf{K'} = \mathbf{K} - \mathbf{1_n} \mathbf{K} - \mathbf{K} \mathbf{1_n} + \mathbf{1_n} \mathbf{K} \mathbf{1_n}$$
where $\mathbf{1_n}$ is a matrix of the same size as $\mathbf{K}$ with all entries equal to $\frac{1}{n}$.

### New data set

In the above we perform kernel PCA for a given dataset $\mathbf{X}$.
How about a new dataset, such as a test data $\mathbf{X'}$, which is not part of $\mathbf{X}$?

In ordinary PCA, we can simply project $\mathbf{X'}$ through $\mathbf{W}$, the matrix whose columns are the (selected) eigen-vectors of $\Sigma$:
$$\mathbf{X'} \mathbf{W}$$

However, for kernel PCA, we only compute the eigen-vectors of the (mean-shifted) kernel matrix $\mathbf{K}$, not the original covariance matrix $\Sigma$.
Fortunately, we can accomplish our goal via smart math tricks, as follows.

First, let's express each eigen-vector $\mathbf{v}$ of $\Sigma$ via the eigen-vectors $\mathbf{A}$ of $\mathbf{K}$.
Recall
$$
\begin{align}
\mathbf{v}
&=
\frac{1}{\lambda} \Sigma \mathbf{v}
\\
&=
\frac{1}{n \lambda} \phi(\mathbf{X})^T \phi(\mathbf{X}) \mathbf{v}
\\
&=
\frac{1}{n \lambda} \phi(\mathbf{X})^T \mathbf{a}
\\
&=
\frac{1}{n \lambda} \sum_{i=1}^n \mathbf{a^{(i)}} \phi(\mathbf{x}^{(i)})
\end{align}
$$

Thus, to project a new sample $\mathbf{x'}$ with an eigen vector $\mathbf{v}$, we can use the kernel trick again with the already computed $\mathbf{a}$ vectors:
$$
\begin{align}
\phi(\mathbf{x'})^T \mathbf{v} &=
\frac{1}{n \lambda} \sum_{i=1}^n \mathbf{a}^{(i)} \phi(\mathbf{x'})^T \phi(\mathbf{x}^{(i)})
\\
&=
\frac{1}{n \lambda} \sum_{i=1}^n \mathbf{a^{(i)}} \mathbf{k}(\mathbf{x'}, \mathbf{x^{(i)}})
\end{align}
$$

## Implementing a kernel principal component analysis in Python

Code the math above ...

```python
from scipy.spatial.distance import pdist, squareform
from scipy import exp
from scipy.linalg import eigh
import numpy as np

def rbf_kernel_pca(X, gamma, n_components):
    """
    RBF kernel PCA implementation.

    Parameters
    ------------
    X: {NumPy ndarray}, shape = [n_samples, n_features]

    gamma: float
      Tuning parameter of the RBF kernel

    n_components: int
      Number of principal components to return

    Returns
    ------------
     X_pc: {NumPy ndarray}, shape = [n_samples, k_features]
       Projected dataset

    """
    # Calculate pairwise squared Euclidean distances
    # in the MxN dimensional dataset.
    sq_dists = pdist(X, 'sqeuclidean')

    # Convert pairwise distances into a square matrix.
    mat_sq_dists = squareform(sq_dists)

    # Compute the symmetric kernel matrix.
    K = exp(-gamma * mat_sq_dists)

    # Center the kernel matrix.
    N = K.shape[0]
    one_n = np.ones((N, N)) / N
    K = K - one_n.dot(K) - K.dot(one_n) + one_n.dot(K).dot(one_n)

    # Obtaining eigenpairs from the centered kernel matrix
    # numpy.eigh returns them in sorted order
    eigvals, eigvecs = eigh(K)

    # Collect the top k eigenvectors (projected samples)
    X_pc = np.column_stack((eigvecs[:, -i]
                            for i in range(1, n_components + 1)))

    return X_pc
```

### Example 1: Separating half-moon shapes

```python
import matplotlib.pyplot as plt
from sklearn.datasets import make_moons

X, y = make_moons(n_samples=100, random_state=123)

plt.scatter(X[y == 0, 0], X[y == 0, 1], color='red', marker='^', alpha=0.5)
plt.scatter(X[y == 1, 0], X[y == 1, 1], color='blue', marker='o', alpha=0.5)

plt.tight_layout()
# plt.savefig('./figures/half_moon_1.png', dpi=300)
plt.show()
```

```python
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

scikit_pca = PCA(n_components=2)
X_spca = scikit_pca.fit_transform(X)

fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(7, 3))

ax[0].scatter(X_spca[y == 0, 0], X_spca[y == 0, 1],
              color='red', marker='^', alpha=0.5)
ax[0].scatter(X_spca[y == 1, 0], X_spca[y == 1, 1],
              color='blue', marker='o', alpha=0.5)

ax[1].scatter(X_spca[y == 0, 0], np.zeros((50, 1)) + 0.02,
              color='red', marker='^', alpha=0.5)
ax[1].scatter(X_spca[y == 1, 0], np.zeros((50, 1)) - 0.02,
              color='blue', marker='o', alpha=0.5)

ax[0].set_xlabel('PC1')
ax[0].set_ylabel('PC2')
ax[1].set_ylim([-1, 1])
ax[1].set_yticks([])
ax[1].set_xlabel('PC1')

plt.tight_layout()
# plt.savefig('./figures/half_moon_2.png', dpi=300)
plt.show()
```

```python
from matplotlib.ticker import FormatStrFormatter

X_kpca = rbf_kernel_pca(X, gamma=15, n_components=2)

fig, ax = plt.subplots(nrows=1,ncols=2, figsize=(7,3))
ax[0].scatter(X_kpca[y==0, 0], X_kpca[y==0, 1],
            color='red', marker='^', alpha=0.5)
ax[0].scatter(X_kpca[y==1, 0], X_kpca[y==1, 1],
            color='blue', marker='o', alpha=0.5)

ax[1].scatter(X_kpca[y==0, 0], np.zeros((50,1))+0.02,
            color='red', marker='^', alpha=0.5)
ax[1].scatter(X_kpca[y==1, 0], np.zeros((50,1))-0.02,
            color='blue', marker='o', alpha=0.5)

ax[0].set_xlabel('PC1')
ax[0].set_ylabel('PC2')
ax[1].set_ylim([-1, 1])
ax[1].set_yticks([])
ax[1].set_xlabel('PC1')
ax[0].xaxis.set_major_formatter(FormatStrFormatter('%0.1f'))
ax[1].xaxis.set_major_formatter(FormatStrFormatter('%0.1f'))

plt.tight_layout()
# plt.savefig('./figures/half_moon_3.png', dpi=300)
plt.show()
```

### Example 2: Separating concentric circles

```python
from sklearn.datasets import make_circles

X, y = make_circles(n_samples=1000, random_state=123, noise=0.1, factor=0.2)

plt.scatter(X[y == 0, 0], X[y == 0, 1], color='red', marker='^', alpha=0.5)
plt.scatter(X[y == 1, 0], X[y == 1, 1], color='blue', marker='o', alpha=0.5)

plt.tight_layout()
# plt.savefig('./figures/circles_1.png', dpi=300)
plt.show()
```

```python
scikit_pca = PCA(n_components=2)
X_spca = scikit_pca.fit_transform(X)

fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(7, 3))

ax[0].scatter(X_spca[y == 0, 0], X_spca[y == 0, 1],
              color='red', marker='^', alpha=0.5)
ax[0].scatter(X_spca[y == 1, 0], X_spca[y == 1, 1],
              color='blue', marker='o', alpha=0.5)

ax[1].scatter(X_spca[y == 0, 0], np.zeros((500, 1)) + 0.02,
              color='red', marker='^', alpha=0.5)
ax[1].scatter(X_spca[y == 1, 0], np.zeros((500, 1)) - 0.02,
              color='blue', marker='o', alpha=0.5)

ax[0].set_xlabel('PC1')
ax[0].set_ylabel('PC2')
ax[1].set_ylim([-1, 1])
ax[1].set_yticks([])
ax[1].set_xlabel('PC1')

plt.tight_layout()
# plt.savefig('./figures/circles_2.png', dpi=300)
plt.show()
```

```python
X_kpca = rbf_kernel_pca(X, gamma=15, n_components=2)

fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(7, 3))
ax[0].scatter(X_kpca[y == 0, 0], X_kpca[y == 0, 1],
              color='red', marker='^', alpha=0.5)
ax[0].scatter(X_kpca[y == 1, 0], X_kpca[y == 1, 1],
              color='blue', marker='o', alpha=0.5)

ax[1].scatter(X_kpca[y == 0, 0], np.zeros((500, 1)) + 0.02,
              color='red', marker='^', alpha=0.5)
ax[1].scatter(X_kpca[y == 1, 0], np.zeros((500, 1)) - 0.02,
              color='blue', marker='o', alpha=0.5)

ax[0].set_xlabel('PC1')
ax[0].set_ylabel('PC2')
ax[1].set_ylim([-1, 1])
ax[1].set_yticks([])
ax[1].set_xlabel('PC1')

plt.tight_layout()
# plt.savefig('./figures/circles_3.png', dpi=300)
plt.show()
```

## Projecting new data points

Note the code below computes eigen values of $\mathbf{K}$ instead of $\frac{\mathbf{K}}{n}$, and thus the eigen values will be $n \times$ scaled.

```python
from scipy.spatial.distance import pdist, squareform
from scipy import exp
from scipy.linalg import eigh
import numpy as np

def rbf_kernel_pca(X, gamma, n_components):
    """
    RBF kernel PCA implementation.

    Parameters
    ------------
    X: {NumPy ndarray}, shape = [n_samples, n_features]

    gamma: float
      Tuning parameter of the RBF kernel

    n_components: int
      Number of principal components to return

    Returns
    ------------
     X_pc: {NumPy ndarray}, shape = [n_samples, k_features]
       Projected dataset

     lambdas: list
       Eigenvalues

    """
    # Calculate pairwise squared Euclidean distances
    # in the MxN dimensional dataset.
    sq_dists = pdist(X, 'sqeuclidean')

    # Convert pairwise distances into a square matrix.
    mat_sq_dists = squareform(sq_dists)

    # Compute the symmetric kernel matrix.
    K = exp(-gamma * mat_sq_dists)

    # Center the kernel matrix.
    N = K.shape[0]
    one_n = np.ones((N, N)) / N
    K = K - one_n.dot(K) - K.dot(one_n) + one_n.dot(K).dot(one_n)

    # Obtaining eigenpairs from the centered kernel matrix
    # numpy.eigh returns them in sorted order
    eigvals, eigvecs = eigh(K)

    # Collect the top k eigenvectors (projected samples)
    alphas = np.column_stack((eigvecs[:, -i]
                              for i in range(1, n_components + 1)))

    # Collect the corresponding eigenvalues
    lambdas = [eigvals[-i] for i in range(1, n_components + 1)]

    return alphas, lambdas
```

```python
def project_x(x_new, X, gamma, alphas, lambdas):
    pair_dist = np.array([np.sum((x_new - row)**2) for row in X])
    k = np.exp(-gamma * pair_dist)
    return k.dot(alphas / lambdas)
```

```python
X, y = make_moons(n_samples=100, random_state=123)
alphas, lambdas = rbf_kernel_pca(X, gamma=15, n_components=1)
```

```python
select_new = -1
x_new = X[select_new]
x_new
```

    array([ 0.4816, -0.3551])

```python
x_proj = alphas[select_new] # original projection

# projection of the "new" datapoint
x_reproj = project_x(x_new, X, gamma=15, alphas=alphas, lambdas=lambdas)

# should be the same
print(x_proj)
print(x_reproj)
```

    [ 0.1192]
    [ 0.1192]

```python
plt.scatter(alphas[y == 0, 0], np.zeros((50)),
            color='red', marker='^', alpha=0.5)
plt.scatter(alphas[y == 1, 0], np.zeros((50)),
            color='blue', marker='o', alpha=0.5)
plt.scatter(x_proj, 0, color='black',
            label='original projection of point X[25]', marker='^', s=100)
plt.scatter(x_reproj, 0, color='green',
            label='remapped point X[25]', marker='x', s=500)
plt.legend(scatterpoints=1)

plt.tight_layout()
# plt.savefig('./figures/reproject.png', dpi=300)
plt.show()
```

```python
X, y = make_moons(n_samples=100, random_state=123)
X_1, y_1 = X[:-1, :], y[:-1]
alphas, lambdas = rbf_kernel_pca(X_1, gamma=15, n_components=1)

# projection of the "new" datapoint
x_new = X[-1, :]

x_reproj = project_x(x_new, X_1, gamma=15, alphas=alphas, lambdas=lambdas)

plt.scatter(alphas[y_1 == 0, 0], np.zeros((np.sum(y_1 == 0))),
            color='red', marker='^', alpha=0.5)
plt.scatter(alphas[y_1 == 1, 0], np.zeros((np.sum(y_1 == 1))),
            color='blue', marker='o', alpha=0.5)
plt.scatter(x_reproj, 0, color='green',
            label='new point [ 100.0,  100.0]', marker='x', s=500)
plt.legend(scatterpoints=1)

```

```python
plt.scatter(alphas[y_1 == 0, 0], np.zeros((np.sum(y_1 == 0))),
            color='red', marker='^', alpha=0.5)
plt.scatter(alphas[y_1 == 1, 0], np.zeros((np.sum(y_1 == 1))),
            color='blue', marker='o', alpha=0.5)
plt.scatter(x_proj, 0, color='black',
            label='some point [1.8713,  0.0093]', marker='^', s=100)
plt.scatter(x_reproj, 0, color='green',
            label='new point [ 100.0,  100.0]', marker='x', s=500)
plt.legend(scatterpoints=1)

plt.tight_layout()
# plt.savefig('./figures/reproject.png', dpi=300)
plt.show()
```

## Kernel principal component analysis in scikit-learn

Kernel PCA is part of the scikit-learn library and can be direclty used

```python
from sklearn.decomposition import KernelPCA

X, y = make_moons(n_samples=100, random_state=123)
scikit_kpca = KernelPCA(n_components=2, kernel='rbf', gamma=15)
X_skernpca = scikit_kpca.fit_transform(X)

plt.scatter(X_skernpca[y == 0, 0], X_skernpca[y == 0, 1],
            color='red', marker='^', alpha=0.5)
plt.scatter(X_skernpca[y == 1, 0], X_skernpca[y == 1, 1],
            color='blue', marker='o', alpha=0.5)

plt.xlabel('PC1')
plt.ylabel('PC2')
plt.tight_layout()
# plt.savefig('./figures/scikit_kpca.png', dpi=300)
plt.show()
```

## Question

How should we choose between the following two?
1. Linear PCA $\rightarrow$ kernel SVM
2. kernel PCA $\rightarrow$ linear SVM

# Summary

# Reading

* PML Chapter 5
* IML Chapter 6.3, 6.6, 13.12


# Tutorial: Setting up the energy-momentum tensor of a massless scalar field

## Authors: Leonardo Werneck & Zach Etienne

# This module documents the construction of the energy-momentum tensor of a massless scalar field.

**Notebook Status:**  Validated

**Validation Notes:** The expressions generated by the NRPy+ module corresponding to this tutorial notebook are used to demonstrate that the initial data for a massless scalar field satisfy Einstein's equations as expected [in this tutorial notebook](Tutorial-Start_to_Finish-BSSNCurvilinear-Setting_up_ScalarField_initial_data).

## Python module containing the final expressions constructed here: **[ScalarField/ScalarField_Tmunu.py](../edit/ScalarField/ScalarField_Tmunu.py)**

# Table of Contents
$$\label{toc}$$

The module is organized as follows

0. [Preliminaries](#preliminaries): The energy momentum tensor of a massless scalar field
1. [Step 1](#initializenrpy): Initialize core NRPy+ modules
1. [Step 2](#sf4d): The 4-derivatives of the scalar field: $\partial^{\mu}\varphi$
1. [Step 3](#energy_momentum_tensor): The energy momentum tensor: $T^{\mu\nu}$
1. [Step 4](#code_validation): Validation against the [ScalarField/ScalarField_Tmunu.py](../edit/ScalarField/ScalarField_Tmunu.py) module
1. [Step 5](#latex_pdf_output): Output this module to $\LaTeX$-formatted PDF

# Preliminaries: The energy momentum tensor of a massless scalar field \[Back to [top](#toc)\]
$$\label{preliminaries}$$

The energy-momentum tensor for a massless scalar field is given by eq. (5.232) of [B&S](https://books.google.com.br/books/about/Numerical_Relativity.html?id=dxU1OEinvRUC&redir_esc=y), which we right here in contravariant form

$$
T^{\mu\nu} = \partial^{\mu}\varphi\partial^{\nu}\varphi - \frac{1}{2}g^{\mu\nu}\left(\partial^{\lambda}\varphi\partial_{\lambda}\varphi\right)\ .
$$

This is a key tensor in the problem of gravitational collapse of a massless scalar field, since it will be responsible for how the geometry changes in the presence of the scalar field. In this tutorial module we will be implementing this tensor.

# Step 1: Initialize core NRPy+ modules \[Back to [top](#toc)\]
$$\label{initializenrpy}$$

Let's start by importing all the needed modules from NRPy+:

```python
# Step 1.a: import all needed modules from NRPy+:
import sympy as sp                         # SymPy: The Python computer algebra package upon which NRPy+ depends
import NRPy_param_funcs as par             # NRPy+: Parameter interface
import indexedexp as ixp                   # NRPy+: Symbolic indexed expression (e.g., tensors, vectors, etc.) support
import reference_metric as rfm             # NRPy+: Reference metric support
import BSSN.BSSN_quantities as Bq          # NRPy+: BSSN quantities
import BSSN.ADM_in_terms_of_BSSN as BtoA   # NRPy+: ADM quantities in terms of BSSN quantities
import BSSN.ADMBSSN_tofrom_4metric as ADMg # NRPy+: ADM 4-metric to/from ADM or BSSN quantities

# Step 1.b: Set the coordinate system for the numerical grid
coord_system = "Spherical"
par.set_parval_from_str("reference_metric::CoordSystem",coord_system)

# Step 1.c: Set spatial dimension (must be 3 for BSSN, as BSSN is
#           a 3+1-dimensional decomposition of the general
#           relativistic field equations)
DIM = 3
par.set_parval_from_str("grid::DIM",DIM)

# Step 1.d: Given the chosen coordinate system, set up
#           corresponding reference metric and needed
#           reference metric quantities
#    The following function call sets up the reference metric
#    and related quantities, including rescaling matrices ReDD,
#    ReU, and hatted quantities.
rfm.reference_metric()

# Step 1.e: Set the theta and phi axes to be the symmetry axes; i.e., axis "1" and "2",
#           corresponding to the i1 and i2 directions. This sets all spatial derivatives
#           in the theta and phi directions to zero (analytically).
par.set_parval_from_str("indexedexp::symmetry_axes","12")

# Step 1.e: Import all basic (unrescaled) BSSN scalars & tensors
Bq.BSSN_basic_tensors()
alpha = Bq.alpha
betaU = Bq.betaU

# Step 1.g: Define ADM quantities in terms of BSSN quantities
BtoA.ADM_in_terms_of_BSSN()
gammaDD = BtoA.gammaDD
gammaUU = BtoA.gammaUU

# Step 1.h: Define scalar field quantitites
sf_dD   = ixp.declarerank1("sf_dD")
Pi      = sp.Symbol("sfM",real=True)
```

# Step 2: The 4-derivatives of the scalar field: $\partial^{\mu}\varphi$ \[Back to [top](#toc)\]
$$\label{sf4d}$$

Consider the ADM 4-metric (eq. 2.119 of B&S)

$$
g^{\mu\nu}=\begin{pmatrix}
-\alpha^{-2} & \alpha^{-2}\beta^{i}\\
\alpha^{-2}\beta^{j} & \gamma^{ij} - \alpha^{-2}\beta^{i}\beta^{j}
\end{pmatrix}\ ,
$$

and the definition of the scalar field's conjugate momentum, $\Pi$, as given by eq. 2.522 of B&S

$$
\Pi\equiv-\frac{1}{\alpha}\left[\partial_{t}\varphi - \beta^{i}\partial_{i}\varphi\right]\ .
$$

Then we have

\begin{align}
\partial^{t}\varphi &= g^{tt}\partial_{t}\varphi + g^{ti}\partial_{i}\varphi\nonumber\\
&= -\alpha^{-2}\partial_{t}\varphi + \alpha^{-2}\beta^{i}\partial_{i}\varphi\nonumber\\
&= \alpha^{-1}\left[-\frac{1}{\alpha}\left(\partial_{t}\varphi - \beta^{i}\partial_{i}\varphi\right)\right]\nonumber\\
&= \frac{\Pi}{\alpha}\ .
\end{align}

```python
# Step 2a: Set up \partial^{t}\varphi = Pi/alpha
sf4dU = ixp.zerorank1(DIM=4)
sf4dU[0] = Pi / alpha
```

Next, we look at

\begin{align}
\partial^{i}\varphi &= g^{it}\partial_{t}\varphi + g^{ij}\partial_{j}\varphi\nonumber\\
&=\alpha^{-2}\beta^{i}\partial_{t}\varphi + \gamma^{ij}\partial_{j}\varphi - \alpha^{-2}\beta^{i}\beta^{j}\partial_{j}\varphi\nonumber\\
&=-\alpha^{-1}\beta^{i}\left[-\frac{1}{\alpha}\left(\partial_{t}\varphi-\beta^{j}\partial_{j}\varphi\right)\right] + \gamma^{ij}\partial_{j}\varphi\nonumber\\
&=-\frac{\Pi}{\alpha}\beta^{i} + \gamma^{ij}\partial_{j}\varphi\ .
\end{align}

```python
# Step 2b: Set up \partial^{i}\varphi = -Pi*beta^{i}/alpha + gamma^{ij}\partial_{j}\varphi
for i in range(DIM):
    sf4dU[i+1] = -Pi * betaU[i] / alpha
    for j in range(DIM):
        sf4dU[i+1] += gammaUU[i][j] * sf_dD[j]
```

The last step is to set up the contraction

\begin{align}
\partial^{\lambda}\varphi\partial_{\lambda}\varphi &= \partial^{t}\varphi\partial_{t}\varphi + \partial^{i}\varphi\partial_{i}\varphi\nonumber\\
&=\frac{\Pi}{\alpha}\partial_{t}\varphi - \frac{\Pi}{\alpha}\beta^{i}\partial_{i}\varphi + \gamma^{ij}\partial_{i}\varphi\partial_{j}\varphi\nonumber\\
&= -\Pi\left[-\frac{1}{\alpha}\left(\partial_{t}\varphi-\beta^{i}\partial_{i}\varphi\right)\right] + \gamma^{ij}\partial_{i}\varphi\partial_{j}\varphi\nonumber\\
&= -\Pi^2 + \gamma^{ij}\partial_{i}\varphi\partial_{j}\varphi\ .
\end{align}

```python
# Step 2c: Set up \partial^{i}\varphi\partial_{i}\varphi = -Pi**2 + gamma^{ij}\partial_{i}\varphi\partial_{j}\varphi
sf4d2 = -Pi**2
for i in range(DIM):
    for j in range(DIM):
        sf4d2 += gammaUU[i][j] * sf_dD[i] * sf_dD[j]
```

# Step 3: The energy momentum tensor: $T^{\mu\nu}$ \[Back to [top](#toc)\]
$$\label{energy_momentum_tensor}$$

We start by setting up the ADM 4-metric $g^{\mu\nu}$, given by eq. (2.119) in [B&S](https://books.google.com.br/books/about/Numerical_Relativity.html?id=dxU1OEinvRUC&redir_esc=y),

$$
g^{\mu\nu}=\begin{pmatrix}
-\alpha^{-2} & \alpha^{-2}\beta^{i}\\
\alpha^{-2}\beta^{j} & \gamma^{ij} - \alpha^{-2}\beta^{i}\beta^{j}
\end{pmatrix}\ .
$$

We do this be calling the [BSSN.adm_four_metric_conversions.py](../edit/BSSN/adm_four_metric_conversions.py) module.

```python
# Step 3a: Setting up g^{\mu\nu}
ADMg.g4UU_ito_BSSN_or_ADM("ADM",gammaDD=gammaDD,betaU=betaU,alpha=alpha, gammaUU=gammaUU)
g4UU = ADMg.g4UU
```

We then focus on the energy momentum tensor $T^{\mu\nu}$ for a massless scalar field, $\varphi$ (cf. eq. 5.232 of [B&S](https://books.google.com.br/books/about/Numerical_Relativity.html?id=dxU1OEinvRUC&redir_esc=y) with $V(\varphi)=0$)

$$
T^{\mu\nu} = \partial^{\mu}\varphi\partial^{\nu}\varphi - \frac{1}{2}g^{\mu\nu}\left(\underbrace{\partial_{\lambda}\varphi\partial^{\lambda}\varphi}_{\equiv \rm sf4d2}\right)\ .
$$

```python
# Step 3b: Setting up T^{\mu\nu} for a massless scalar field
T4UU = ixp.zerorank2(DIM=4)
for mu in range(4):
    for nu in range(4):
        T4UU[mu][nu] = sf4dU[mu] * sf4dU[nu] - sp.Rational(1,2) * g4UU[mu][nu] * sf4d2
```

# Step 4: Validation against the [ScalarField/ScalarField_Tmunu.py](../edit/ScalarField/ScalarField_Tmunu.py) module  \[Back to [top](#toc)\]
$$\label{code_validation}$$

Here we perform a code validation. We verify agreement in the SymPy expressions for the energy-momentum tensor of a scalar field between
1. this tutorial notebook and
2. the [ScalarField/ScalarField_Tmunu.py](../edit/ScalarField/ScalarField/ScalarField_Tmunu.py) NRPy+ module.

By default, we analyze the RHSs in Spherical coordinates, though other coordinate systems may be chosen.

```python
import ScalarField.ScalarField_Tmunu as sfTmunu # NRPyCritCol: Scalar field energy-momentum tensor
sfTmunu.ScalarField_Tmunu()

print("Consistency check between this tutorial and the ScalarField.ScalarField_Tmunu.py module: ALL SHOULD BE ZERO\n")

for mu in range(4):
    for nu in range(4):
        print("T4UU["+str(mu)+"]["+str(nu)+"] - sfTmunu.T4UU["+str(mu)+"]["+str(nu)+"] = "+str(sp.simplify(T4UU[mu][nu] - sfTmunu.T4UU[mu][nu])))
```

    Consistency check between this tutorial and the ScalarField.ScalarField_Tmunu.py module: ALL SHOULD BE ZERO

    T4UU[0][0] - sfTmunu.T4UU[0][0] = 0
    T4UU[0][1] - sfTmunu.T4UU[0][1] = 0
    T4UU[0][2] - sfTmunu.T4UU[0][2] = 0
    T4UU[0][3] - sfTmunu.T4UU[0][3] = 0
    T4UU[1][0] - sfTmunu.T4UU[1][0] = 0
    T4UU[1][1] - sfTmunu.T4UU[1][1] = 0
    T4UU[1][2] - sfTmunu.T4UU[1][2] = 0
    T4UU[1][3] - sfTmunu.T4UU[1][3] = 0
    T4UU[2][0] - sfTmunu.T4UU[2][0] = 0
    T4UU[2][1] - sfTmunu.T4UU[2][1] = 0
    T4UU[2][2] - sfTmunu.T4UU[2][2] = 0
    T4UU[2][3] - sfTmunu.T4UU[2][3] = 0
    T4UU[3][0] - sfTmunu.T4UU[3][0] = 0
    T4UU[3][1] - sfTmunu.T4UU[3][1] = 0
    T4UU[3][2] - sfTmunu.T4UU[3][2] = 0
    T4UU[3][3] - sfTmunu.T4UU[3][3] = 0

# Step 5: Output this module to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[Tutorial-ScalarField_Tmunu.pdf](Tutorial-ScalarField_Tmunu.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
import cmdline_helper as cmd    # NRPy+: Multi-platform Python command-line interface
cmd.output_Jupyter_notebook_to_LaTeXed_PDF("Tutorial-ScalarField_Tmunu")
```

    Created Tutorial-ScalarField_Tmunu.tex, and compiled LaTeX file to PDF file
        Tutorial-ScalarField_Tmunu.pdf


### Multi-head Attention with 256-dimensional embedding

Author: Jeanne Elizabeth Daniel

November 2019

We employ the multi-head attention encoder of the Transformer (Vaswani et al., 2017), to model the multilingual questions. The Transformer is a new type of encoder-decoder model that relies solely on attention to draw global dependencies between the input and output sequences.

Attention allows the model to focus on different parts of the input sequence at every step of the output sequence. This enables modelling dependencies without any regards for their distance in the sequences. This architecture is devoid of any recurrence or convolutions, and thus its training can be parallelizable.

The encoder component is a stack of encoders, identical in structure, but all with their own set of weights. Each encoder consists of a self-attention layer, followed by a fully-connected feedforward layer.

The attention used by the Transformer is the scaled dot-product attention with a set of queries in matrix $\boldsymbol{Q}$, a set of keys in matrix $\boldsymbol{K}$, and a set of values in matrix $\boldsymbol{V}$, and is computed as follows:

\begin{equation}
    \mathrm{Attention} (\boldsymbol{Q}, \boldsymbol{K}, \boldsymbol{V}) = \mathrm{softmax} \Bigg(\frac{\boldsymbol{Q} \boldsymbol{K}^{\top}}{ \sqrt{d_K}}\Bigg) \boldsymbol{V},
\end{equation}

where $d_K$ is the dimension of the keys and acts as a scaling factor. Multi-headed attention allows for attention to be aggregated across $h$ different, randomly-initialized representation subspaces. Thus,

\begin{equation}
    \mathrm{MultiHead} (\boldsymbol{Q}, \boldsymbol{K}, \boldsymbol{V}) = \mathrm{Concat} (\mathrm{head}_1, \dots, \mathrm{head}_h) \boldsymbol{W}^{O},
\end{equation}

where Concat refers to concatenating each head, defined as:

\begin{equation}
    \mathrm{head}_i = \mathrm{Attention} (\boldsymbol{Q}\boldsymbol{W}^{Q}_i, \boldsymbol{K}\boldsymbol{W}^{K}_i, \boldsymbol{V}\boldsymbol{W}^{V}_i ),
\end{equation}

with $\boldsymbol{W}^{Q}_i \in \mathbb{R}^{d_{\mathrm{model}} \times d_V }, \boldsymbol{W}^K_i \in \mathbb{R}^{d_{\mathrm{model}} \times d_V }$,
$\boldsymbol{W}_i^V \in \mathbb{R}^{d_{\mathrm{model}} \times d_V}$,
and $\boldsymbol{W}^O \in \mathbb{R}^{h d_v \times d_{\mathrm{model}}}$.

The scalar $d_V$ represents the dimension of the values and $d_{\mathrm{model}}$ denotes the dimension of the model's embedding space. This multi-headed attention function can also be parallelized and trained across multiple computers. The authors also inject information about the relative and absolute positions of the values in the sequence using positional encoding to allow for the modelling of time-dependencies.

This is done by summing the positional encodings with the input embeddings, which are defined as
\begin{eqnarray}
PE_{(pos, 2i)}  &  = & \sin (pos/10000^{2i/d_{\mathrm{model}}}),\\
PE_{(pos, 2i+1)} & = & \cos (pos/10000^{2i/d_{\mathrm{model}}}).
\end{eqnarray}
Combining all these elements results in state-of-the-art embeddings that, when compared to previous models, has reduced computational complexity per layer, parallelizable computation, and better long-term dependency modelling.

```python
import sys
import os
#sys.path.append(os.path.join(\"..\")) # path to source relative to current directory"
```

```python
import numpy as np
import gensim
```

```python
import preprocess_data
import pandas as pd
```

```python
from tensorflow.keras.constraints import Constraint
from tensorflow.keras import regularizers
```

```python
import tensorflow as tf
physical_devices = tf.config.experimental.list_physical_devices("GPU")
tf.config.experimental.set_memory_growth(physical_devices[0], True)
from tensorflow.keras.models import Model
from tensorflow.keras.preprocessing import sequence
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import GlobalAveragePooling1D, GlobalMaxPooling1D
from tensorflow.keras.layers import Dense, Dropout, Embedding, LSTM, Bidirectional, TimeDistributed, Input, Flatten, AdditiveAttention
```

```python
from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint
```

```python
data = pd.read_csv('dataset_7B', delimiter = ';', engine = 'python')
data_text = data.loc[data['set'] == 'Train'][['helpdesk_question']]
number_of_classes = data.loc[data['set'] == 'Train']['helpdesk_reply'].value_counts().shape[0]
data = data[['helpdesk_question', 'helpdesk_reply', 'set', 'low_resource']]
```

```python
responses = pd.DataFrame(data.loc[data['set'] == 'Train']['helpdesk_reply'].value_counts()).reset_index()
responses['reply'] = responses['index']
responses['index'] = responses.index
responses = dict(responses.set_index('reply')['index'])
```

```python
len(responses)
```

    89

```python
data_text['index'] = data_text.index
documents = data_text
```

```python
dictionary = preprocess_data.create_dictionary(data_text, 1, 0.25, 95000) #our entire vocabulary
```

```python
df_train = data.loc[data['set'] == 'Train']
df_train = df_train.reset_index()[['helpdesk_question', 'helpdesk_reply']]

df_valid = data.loc[data['set'] == 'Valid']
df_valid = df_valid.reset_index()[['helpdesk_question', 'helpdesk_reply']]

df_test = data.loc[data['set'] == 'Test']
df_test = df_test.reset_index()[['helpdesk_question', 'helpdesk_reply']]

df_LR = data.loc[(data['set'] == 'Test') & (data['low_resource'] == 'True') ]
df_LR = df_LR.reset_index()[['helpdesk_question', 'helpdesk_reply']]
```

```python
df_train.shape
```

    (96412, 2)

```python
unique_words = dictionary
```

```python
len(unique_words) + 1
```

    57545

```python
max_length = 30
min_token_length = 0
```

```python
word_to_id, id_to_word = preprocess_data.create_lookup_tables(unique_words)
```

#### Transforming the input sentence into a sequence of word IDs

```python
train_x_word_ids = []
for question in df_train['helpdesk_question'].apply(preprocess_data.preprocess_question,
                                                    args = [unique_words, min_token_length]):
    word_ids = preprocess_data.transform_sequence_to_word_ids(question, word_to_id)
    train_x_word_ids.append(np.array(word_ids, dtype = float))
train_x_word_ids = np.stack(train_x_word_ids)
print(train_x_word_ids.shape)

val_x_word_ids = []
for question in data['helpdesk_question'].loc[data['set'] == 'Valid'].apply(preprocess_data.preprocess_question,
                                                                          args = [unique_words, min_token_length]):
    word_ids = preprocess_data.transform_sequence_to_word_ids(question, word_to_id)
    val_x_word_ids.append(np.array(word_ids, dtype = float))
val_x_word_ids = np.stack(val_x_word_ids)

test_x_word_ids = []
for question in data['helpdesk_question'].loc[data['set'] == 'Test'].apply(preprocess_data.preprocess_question,
                                                                          args = [unique_words, min_token_length]):
    word_ids = preprocess_data.transform_sequence_to_word_ids(question, word_to_id)
    test_x_word_ids.append(np.array(word_ids, dtype = float))

test_x_word_ids = np.stack(test_x_word_ids)

LR_x_word_ids = []
for question in data['helpdesk_question'].loc[(data['set'] == 'Test') &
                                              (data['low_resource'] == 'True')].apply(preprocess_data.preprocess_question,
                                                                          args = [unique_words, min_token_length]):
    word_ids = preprocess_data.transform_sequence_to_word_ids(question, word_to_id)
    LR_x_word_ids.append(np.array(word_ids, dtype = float))
LR_x_word_ids = np.stack(LR_x_word_ids)
```

    (96412, 30, 1)

```python
def get_dummies(reply, all_responses):

    """ Constructs a one-hot vector for replies

    Args:
        reply: query item
        all_responses: dict containing all the template responses with their corresponding IDs

    Return:
        a one-hot vector where the corresponding ID of the reply is the one-hot index

    """

    Y = np.zeros(len(all_responses), dtype = int)
    Y[all_responses[reply]] += 1
    return Y
```

```python
train_y = np.array(list(df_train['helpdesk_reply'].apply(get_dummies, args = [responses])))
valid_y = np.array(list(df_valid['helpdesk_reply'].apply(get_dummies, args = [responses])))
test_y  = np.array(list(df_test['helpdesk_reply'].apply(get_dummies,  args = [responses])))
LR_y    = np.array(list(df_LR['helpdesk_reply'].apply(get_dummies,    args = [responses])))
```

```python
train_x_word_ids = train_x_word_ids.reshape(train_x_word_ids.shape[:-1])
val_x_word_ids   = val_x_word_ids.reshape(val_x_word_ids.shape[:-1])
test_x_word_ids  = test_x_word_ids.reshape(test_x_word_ids.shape[:-1])
LR_x_word_ids    = LR_x_word_ids.reshape(LR_x_word_ids.shape[:-1])
```

#### Transform vectors where the input sentence yields a sequence of length 0

```python
train_zero_vectors = np.where(train_x_word_ids.sum(axis = 1) == 0.0)[0]
for t in range(train_zero_vectors.shape[0]):
    train_x_word_ids[train_zero_vectors[t]][0] += 1
```

```python
val_zero_vectors = np.where(val_x_word_ids.sum(axis = 1) == 0.0)[0]
for t in range(val_zero_vectors.shape[0]):
    val_x_word_ids[val_zero_vectors[t]][0] += 1
```

### Building the encoder (from the Transformer)

Original code obtained from https://www.tensorflow.org/tutorials/text/transformer with minor adaptions

```python
def get_angles(pos, i, d_model):

    """ Multiplying angle rates and positions gives a map of the position encoding angles as a
    function of depth. The angle rates range from 1 [rads/step] to min_rate [rads/step] over the
    vector depth.

    Args:
        pos: vector of positions
        i: embedding vector
        d_model: dimension of embedding vector

    Returns:
        Vector of angle radians

    """

    angle_rate = 1/np.power(10000, ((2*i)/np.float32(d_model)))
    return pos * angle_rate

def positional_encoding(position, d_model):

    """ Calculate positional encodings to inject information about relative and absolute positions/
    The positional encodings are obtained by taking the sine and cosine of the angle radians.

    Args:
        position: maximum position encoding
        d_model: dimension of embedding vector

    Returns:
        A positional encoding vector

    """

    angle_rads = get_angles(np.arange(position)[:, np.newaxis],
                            np.arange(d_model)[np.newaxis, :],
                            d_model)

    angle_rads[:, 0::2] = np.sin(angle_rads[:, 0::2])
    angle_rads[:, 1::2] = np.cos(angle_rads[:, 1::2])
    pos_encoding = angle_rads[np.newaxis, ...]
    return tf.cast(pos_encoding, dtype=tf.float32)
```

```python
def scaled_dot_product_attention(q, k, v, mask):

    """ Calculate the attention weights. q, k, v must have matching leading dimensions.
    k, v must have matching penultimate dimension, i.e.: seq_len_k = seq_len_v.
    The mask has different shapes depending on its type(padding or look ahead)
    but it must be broadcastable for addition.

    Args:
        q: query shape == (..., seq_len_q, depth)
        k: key shape == (..., seq_len_k, depth)
        v: value shape == (..., seq_len_v, depth_v)
        mask: Float tensor with shape broadcastable
          to (..., seq_len_q, seq_len_k). Defaults to None.

    Returns:
        output, attention_weights
    """

    matmul_qk = tf.matmul(q, k, transpose_b=True)  # (..., seq_len_q, seq_len_k)

    # scale matmul_qk
    dk = tf.cast(tf.shape(k)[-1], tf.float32)
    scaled_attention_logits = matmul_qk / tf.math.sqrt(dk)

    attention_weights = tf.nn.softmax(scaled_attention_logits, axis=-1)  # (..., seq_len_q, seq_len_k)
    output = tf.matmul(attention_weights, v)  # (..., seq_len_q, depth_v)
    return output, attention_weights

class MultiHeadAttention(tf.keras.layers.Layer):

    """ Multi-head attention consists of four parts: linear layers that split into heads,
    scaled dot-product attention, the concatenation of heads, and a final linear layer.

    """

    def __init__(self, d_model, num_heads):

        super(MultiHeadAttention, self).__init__()
        self.num_heads = num_heads
        self.d_model = d_model

        assert d_model % self.num_heads == 0

        self.depth = d_model // self.num_heads
        self.wq = tf.keras.layers.Dense(d_model)
        self.wk = tf.keras.layers.Dense(d_model)
        self.wv = tf.keras.layers.Dense(d_model)

        self.dense = tf.keras.layers.Dense(d_model)

    def split_heads(self, x, batch_size):

        """ Split the last dimension into (num_heads, depth).
        Transpose the result such that the shape is (batch_size, num_heads, seq_len, depth)

        Args:
            x: feed forward layer
            batch_size: number of items in a batch

        Returns:
            tuple containing (batch size, number of heads, sequence length, depth)

        """

        x = tf.reshape(x, (batch_size, -1, self.num_heads, self.depth))
        return tf.transpose(x, perm=[0, 2, 1, 3])

    def call(self, v, k, q, mask):

        """ Call function to split the heads of the linear layers.
        Returns the scaled attention dense layer and attention weights

        Args:
            q: query shape == (..., seq_len_q, depth)
            k: key shape == (..., seq_len_k, depth)
            v: value shape == (..., seq_len_v, depth_v)
            mask: float tensor with shape broadcastable

        Returns:
            output, attention_weights

        """

        batch_size = tf.shape(q)[0]

        q = self.wq(q)  # (batch_size, seq_len, d_model)
        k = self.wk(k)  # (batch_size, seq_len, d_model)
        v = self.wv(v)  # (batch_size, seq_len, d_model)

        q = self.split_heads(q, batch_size)  # (batch_size, num_heads, seq_len_q, depth)
        k = self.split_heads(k, batch_size)  # (batch_size, num_heads, seq_len_k, depth)
        v = self.split_heads(v, batch_size)  # (batch_size, num_heads, seq_len_v, depth)

        # scaled_attention.shape == (batch_size, num_heads, seq_len_q, depth)
        # attention_weights.shape == (batch_size, num_heads, seq_len_q, seq_len_k)
        scaled_attention, attention_weights = scaled_dot_product_attention(
            q, k, v, mask)

        scaled_attention = tf.transpose(scaled_attention, perm=[0, 2, 1, 3])  # (batch_size,
                                                                              #seq_len_q, num_heads, depth)
        concat_attention = tf.reshape(scaled_attention,
                                      (batch_size, -1, self.d_model))  # (batch_size, seq_len_q, d_model)
        output = self.dense(concat_attention)  # (batch_size, seq_len_q, d_model)
        return output, attention_weights

def point_wise_feed_forward_network(d_model, dff):

    """ Construct a two-layer feedforward NN with layer dimensions d_model and dff respectively
    and ReLU activations between layers.

    Args:
        d_model: dimension of embedding layer
        dff: dimension of the second layer

    Returns:
        A two-layer feedforward NN

    """

    return tf.keras.Sequential([
        tf.keras.layers.Dense(dff, activation='relu'),  # (batch_size, seq_len, dff)
        tf.keras.layers.Dense(d_model)  # (batch_size, seq_len, d_model)
    ])

class EncoderLayer(tf.keras.layers.Layer):

    """ Each encoder layer consists of Multi-head attention (with padding mask) and pointwise
    feedforward networks.

    """

    def __init__(self, d_model, num_heads, dff, rate=0.1):
        super(EncoderLayer, self).__init__()

        self.mha = MultiHeadAttention(d_model, num_heads)
        self.ffn = point_wise_feed_forward_network(d_model, dff)

        self.layernorm1 = tf.keras.layers.LayerNormalization(epsilon=1e-6)
        self.layernorm2 = tf.keras.layers.LayerNormalization(epsilon=1e-6)

        self.dropout1 = tf.keras.layers.Dropout(0.1)
        self.dropout2 = tf.keras.layers.Dropout(0.1)

    def call(self, x, training=False, mask=None):

        """ Constructs the encoder layer.

        Args:
            x: sequential layer
            training: flag indicating training or testing
            mask: float tensor with shape broadcastable

        """

        attn_output, _ = self.mha(x, x, x, mask)  # (batch_size, input_seq_len, d_model)
        attn_output = self.dropout1(attn_output, training=training)
        out1 = self.layernorm1(x + attn_output)  # (batch_size, input_seq_len, d_model)

        ffn_output = self.ffn(out1)  # (batch_size, input_seq_len, d_model)
        ffn_output = self.dropout2(ffn_output, training=training)
        out2 = self.layernorm2(out1 + ffn_output)  # (batch_size, input_seq_len, d_model)

        return out2
```

```python
class Encoder(tf.keras.layers.Layer):

    """ The Encoder consists of an input embedding, summed with positional encoding, and N encoder layers.
    The summation is the input to the encoder layers. The output of the encoder is the input to the decoder.

    """

    def __init__(self, num_layers, d_model, num_heads, dff, input_vocab_size,
                 maximum_position_encoding, rate=0.1):
        super(Encoder, self).__init__()

        self.d_model = d_model
        self.num_layers = num_layers
        self.embedding = Embedding(input_vocab_size, d_model,)
        self.pos_encoding = positional_encoding(maximum_position_encoding, self.d_model)
        self.enc_layers = [EncoderLayer(d_model, num_heads, dff, rate) for _ in range(num_layers)]
        self.dropout = Dropout(rate)

    def call(self, x, training, mask=None):

        """ This function constructs the encoder.
        Note we move the dropout to right before the summation (of embedding and positional encodings).

        Args:
            x: sequential layer
            training: flag indicating training or testing
            mask: float tensor with shape broadcastable

        Returns:
            An encoder model
        """

        seq_len = tf.shape(x)[1]
        x = self.embedding(x)
        x = self.dropout(x, training = training)
        x *= tf.math.sqrt(tf.cast(self.d_model, tf.float32))
        x += self.pos_encoding[:, :seq_len, :]
        #x = self.dropout(x, training = training)
        for i in range(self.num_layers):
            x = self.enc_layers[i](x, training, mask)

        return x
```

```python
def multihead_attention_encoder(num_layers, max_features, input_length=30, model_dim=512, dff = 128,
                                num_heads=4):

    """ Constructs a multihead attention encoder model

    Args:
        num_layers: number of encoder layers
        max_features: size of vocabulary
        input_length: length of input sequence
        model_dim: dimension of embedding vector
        dff: dimension of second layer in pointwise FFNN
        num_heads: number of heads to split

    Returns:
        Model object

    """

    inputs = Input(shape=(input_length, ))
    x = Encoder(num_layers, model_dim, num_heads, dff, max_features, maximum_position_encoding = 10000,
                rate=0.5)(inputs)
    x = GlobalAveragePooling1D()(x)
    outputs = Dense(89, activation='softmax')(x)
    return Model(inputs=inputs, outputs=outputs)
```

#### Multi-head Attention Encoder with Average Pooling

We use average pooling to construct a single feature vector from the variable-length sequence of encodings produced by the MHA Encoder. This is then connected to a classification layer. Our MHA has 8 heads, 2 layers, and dropout of 50% to regularize the model during training.

```python
max_features = len(unique_words) + 1
num_layers = 2

model = multihead_attention_encoder(num_layers, max_features, input_length=30, model_dim=256,
                                    num_heads=8)
```

```python
model.summary()
```

    Model: "model_1"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    input_2 (InputLayer)         [(None, 30)]              0
    _________________________________________________________________
    encoder_1 (Encoder)          (None, 30, 256)           15391744
    _________________________________________________________________
    global_average_pooling1d_1 ( (None, 256)               0
    _________________________________________________________________
    dense_25 (Dense)             (None, 89)                22873
    =================================================================
    Total params: 15,414,617
    Trainable params: 15,414,617
    Non-trainable params: 0
    _________________________________________________________________

### Training

```python
es = EarlyStopping(monitor='val_accuracy', verbose=1, restore_best_weights=True, patience=10)
model.compile(loss='categorical_crossentropy',
              optimizer=tf.keras.optimizers.Adadelta(learning_rate=0.25, rho=0.95),
              metrics=['accuracy'])
```

```python
model.fit(train_x_word_ids, train_y,
          batch_size=32,
          epochs=500,
          callbacks=[es],
          validation_data=[val_x_word_ids, valid_y])
```

    Train on 96412 samples, validate on 31955 samples
    Epoch 1/500
    96412/96412 [==============================] - 172s 2ms/sample - loss: 1.2374 - accuracy: 0.6300 - val_loss: 1.4554 - val_accuracy: 0.5967
    Epoch 2/500
    96412/96412 [==============================] - 171s 2ms/sample - loss: 1.1979 - accuracy: 0.6393 - val_loss: 1.4509 - val_accuracy: 0.5957
    Epoch 3/500
    96412/96412 [==============================] - 174s 2ms/sample - loss: 1.1654 - accuracy: 0.6467 - val_loss: 1.4665 - val_accuracy: 0.5988
    Epoch 4/500
    96412/96412 [==============================] - 174s 2ms/sample - loss: 1.1385 - accuracy: 0.6535 - val_loss: 1.4546 - val_accuracy: 0.6024
    Epoch 5/500
    96412/96412 [==============================] - 170s 2ms/sample - loss: 1.1073 - accuracy: 0.6608 - val_loss: 1.4940 - val_accuracy: 0.5920
    Epoch 6/500
    96412/96412 [==============================] - 166s 2ms/sample - loss: 1.0813 - accuracy: 0.6671 - val_loss: 1.4530 - val_accuracy: 0.6018
    Epoch 7/500
    96412/96412 [==============================] - 252s 3ms/sample - loss: 1.0563 - accuracy: 0.6737 - val_loss: 1.4635 - val_accuracy: 0.6075
    Epoch 8/500
    96412/96412 [==============================] - 246s 3ms/sample - loss: 1.0332 - accuracy: 0.6790 - val_loss: 1.4664 - val_accuracy: 0.6069
    Epoch 9/500
    96412/96412 [==============================] - 161s 2ms/sample - loss: 1.0095 - accuracy: 0.6865 - val_loss: 1.4863 - val_accuracy: 0.5984
    Epoch 10/500
    96412/96412 [==============================] - 142s 1ms/sample - loss: 0.9917 - accuracy: 0.6914 - val_loss: 1.4900 - val_accuracy: 0.6043
    Epoch 11/500
    96412/96412 [==============================] - 142s 1ms/sample - loss: 0.9699 - accuracy: 0.6966 - val_loss: 1.5019 - val_accuracy: 0.6001
    Epoch 12/500
    96412/96412 [==============================] - 142s 1ms/sample - loss: 0.9513 - accuracy: 0.7018 - val_loss: 1.5119 - val_accuracy: 0.6022
    Epoch 13/500
    96412/96412 [==============================] - 141s 1ms/sample - loss: 0.9330 - accuracy: 0.7068 - val_loss: 1.5185 - val_accuracy: 0.6056
    Epoch 14/500
    96412/96412 [==============================] - 152s 2ms/sample - loss: 0.9159 - accuracy: 0.7119 - val_loss: 1.5220 - val_accuracy: 0.6080
    Epoch 15/500
    96412/96412 [==============================] - 146s 2ms/sample - loss: 0.8931 - accuracy: 0.7187 - val_loss: 1.5592 - val_accuracy: 0.6142
    Epoch 16/500
    96412/96412 [==============================] - 147s 2ms/sample - loss: 0.8777 - accuracy: 0.7227 - val_loss: 1.5678 - val_accuracy: 0.6056
    Epoch 17/500
    96412/96412 [==============================] - 167s 2ms/sample - loss: 0.8655 - accuracy: 0.7253 - val_loss: 1.5708 - val_accuracy: 0.6058
    Epoch 18/500
    96412/96412 [==============================] - 157s 2ms/sample - loss: 0.8490 - accuracy: 0.7305 - val_loss: 1.6009 - val_accuracy: 0.6057
    Epoch 19/500
    96412/96412 [==============================] - 168s 2ms/sample - loss: 0.8299 - accuracy: 0.7364 - val_loss: 1.5860 - val_accuracy: 0.6077
    Epoch 20/500
    96412/96412 [==============================] - 170s 2ms/sample - loss: 0.8204 - accuracy: 0.7382 - val_loss: 1.5907 - val_accuracy: 0.6080
    Epoch 21/500
    96412/96412 [==============================] - 171s 2ms/sample - loss: 0.8049 - accuracy: 0.7436 - val_loss: 1.6179 - val_accuracy: 0.6123
    Epoch 22/500
    96412/96412 [==============================] - 169s 2ms/sample - loss: 0.7890 - accuracy: 0.7467 - val_loss: 1.6243 - val_accuracy: 0.6124
    Epoch 23/500
    96412/96412 [==============================] - 168s 2ms/sample - loss: 0.7788 - accuracy: 0.7523 - val_loss: 1.6785 - val_accuracy: 0.6104
    Epoch 24/500
    96412/96412 [==============================] - 168s 2ms/sample - loss: 0.7664 - accuracy: 0.7541 - val_loss: 1.6495 - val_accuracy: 0.6053
    Epoch 25/500
    96384/96412 [============================>.] - ETA: 0s - loss: 0.7536 - accuracy: 0.7599Restoring model weights from the end of the best epoch.
    96412/96412 [==============================] - 169s 2ms/sample - loss: 0.7536 - accuracy: 0.7599 - val_loss: 1.6839 - val_accuracy: 0.6062
    Epoch 00025: early stopping

    <tensorflow.python.keras.callbacks.History at 0x7f6790888c18>

```python
def classifier_score_top_1(word_ids, y_true, model):

    """ Computes classification accuracy for model.

    Args:
        word_ids: matrix where each row is
        y_true: ground truth labels
        model: pretrained model

    Returns:
        None

    """

    score = 0
    probs = model.predict(word_ids)
    for i in range(word_ids.shape[0]):
        if y_true[i].argmax() == np.argsort(probs[i])[-1]:
            score += 1

    print("Overall Accuracy:", score/word_ids.shape[0])
```

### Validation accuracy

```python
classifier_score_top_1(val_x_word_ids, valid_y, model)
```

    Overall Accuracy: 0.6142387732749178

### Test score

```python
classifier_score_top_1(test_x_word_ids, test_y, model)
```

    Overall Accuracy: 0.6149908478888096

### LR test score

```python
classifier_score_top_1(LR_x_word_ids, LR_y, model)
```

    Overall Accuracy: 0.5317007211538461

### Top-5 accuracy

```python
def classifier_score_top_5(word_ids, y_true, model):
    """ Computes top-5 classification accuracy for model.

    Args:
        word_ids: matrix where each row is
        y_true: true labels
        model: trained model

    Returns:
        None

    """

    score = 0
    probs = model.predict(word_ids)
    for i in range(word_ids.shape[0]):
        if y_true[i].argmax() in np.argsort(probs[i])[-5:]:
            score += 1

    print("Overall Accuracy:", score/word_ids.shape[0])
```

```python
classifier_score_top_5(test_x_word_ids, test_y, model)
```

    Overall Accuracy: 0.911612322774796

```python
classifier_score_top_5(LR_x_word_ids, LR_y, model)
```

    Overall Accuracy: 0.8347355769230769