Copyright (c) 2015, 2016 [Sebastian Raschka](sebastianraschka.com)

2016 [Li-Yi Wei](http://www.liyiwei.org/)

https://github.com/1iyiwei/pyml

[MIT License](https://github.com/1iyiwei/pyml/blob/master/LICENSE.txt)

# Python Machine Learning - Code Examples

# Chapter 11 - Working with Unlabeled Data – Clustering Analysis

Supervised learning
* classification

* regression

Unsupervised learning
* dimensionality reduction

* clustering

Note that the optional watermark extension is a small IPython notebook plugin that I developed to make the code reproducible. You can just skip the following line(s).

```python
%load_ext watermark
%watermark -a '' -u -d -v -p numpy,pandas,matplotlib,scipy,sklearn
```

    last updated: 2016-11-15

    CPython 3.5.2
    IPython 4.2.0

    numpy 1.11.1
    pandas 0.18.1
    matplotlib 1.5.1
    scipy 0.17.1
    sklearn 0.18

*The use of `watermark` is optional. You can install this IPython extension via "`pip install watermark`". For more information, please see: https://github.com/rasbt/watermark.*

### Overview

- [Grouping objects by similarity using k-means](#Grouping-objects-by-similarity-using-k-means)
  - [K-means++](#K-means++)
  - [Hard versus soft clustering](#Hard-versus-soft-clustering)
  - [Using the elbow method to find the optimal number of clusters](#Using-the-elbow-method-to-find-the-optimal-number-of-clusters)
  - [Quantifying the quality of clustering via silhouette plots](#Quantifying-the-quality-of-clustering-via-silhouette-plots)
- [Organizing clusters as a hierarchical tree](#Organizing-clusters-as-a-hierarchical-tree)
  - [Performing hierarchical clustering on a distance matrix](#Performing-hierarchical-clustering-on-a-distance-matrix)
  - [Attaching dendrograms to a heat map](#Attaching-dendrograms-to-a-heat-map)
  - [Applying agglomerative clustering via scikit-learn](#Applying-agglomerative-clustering-via-scikit-learn)
- [Locating regions of high density via DBSCAN](#Locating-regions-of-high-density-via-DBSCAN)
- [Summary](#Summary)

```python
from IPython.display import Image
```

```python
%matplotlib inline
```

# Grouping objects by similarity using k-means

Cluster similar objects (data samples) $\mathbf{X}$ together
* documents, music, video
* users with similar profiles or interests

The number of cluster (k) is a hyper-parameter
* manual specification
* automatic determination

The distance measure $d(x^{(i)}, x^{(j)})$ can be another hyper-parameter
* Euclidean ($L_2$) is a common choice

## K-means clustering example

### Create and visualize the data set

```python
from sklearn.datasets import make_blobs

X, y = make_blobs(n_samples=150,
                  n_features=2,
                  centers=3,
                  cluster_std=0.5,
                  shuffle=True,
                  random_state=0)
```

```python
import matplotlib.pyplot as plt

plt.scatter(X[:, 0], X[:, 1], c='white', marker='o', s=50)
plt.grid()
plt.tight_layout()
#plt.savefig('./figures/spheres.png', dpi=300)
plt.show()
```

```python
import matplotlib.pyplot as plt
import numpy as np

# global plot options for consistency
def plot_options():
    markers = ('s', 'x', 'o', '^', 'v')
    #colors = ('red', 'blue', 'lightgreen', 'gray', 'cyan')
    colors = ['lightgreen', 'orange', 'lightblue', 'gray', 'cyan']
    center_marker = '*'
    center_color = 'red'

    return markers, colors, center_marker, center_color

def plot_clusters(X, y, centers):

    markers, colors, center_marker, center_color = plot_options()

    num_clusters = np.unique(y).shape[0]

    for k in range(num_clusters):
        color = colors[k]
        marker = markers[k]
        plt.scatter(X[y == k, 0], X[y == k, 1],
                    s=50, c=color, marker=marker,
                    label = 'cluster ' + str(k+1))

    plt.scatter(centers[:, 0], centers[:, 1],
                s=250, marker=center_marker, c=center_color,
                label='centroids')
```

```python
# same code from the classifiers
# applicable to any model with predict() method
# here, y is the cluster labels,
# instead of ground truth as in supervised classification
from matplotlib.colors import ListedColormap

def plot_decision_regions(X, y, model, resolution=0.02):

    # setup marker generator and color map

    markers, colors, center_marker, center_color = plot_options()
    cmap = ListedColormap(colors[:len(np.unique(y))])

    # plot the decision surface
    x1_min, x1_max = X[:, 0].min() - 1, X[:, 0].max() + 1
    x2_min, x2_max = X[:, 1].min() - 1, X[:, 1].max() + 1
    xx1, xx2 = np.meshgrid(np.arange(x1_min, x1_max, resolution),
                           np.arange(x2_min, x2_max, resolution))
    Z = model.predict(np.array([xx1.ravel(), xx2.ravel()]).T)
    Z = Z.reshape(xx1.shape)
    plt.contourf(xx1, xx2, Z, alpha=0.4, cmap=cmap)
    plt.xlim(xx1.min(), xx1.max())
    plt.ylim(xx2.min(), xx2.max())
```

### First iteration

Not good clustering

### More iterations

Better clustering results

```python
from sklearn.cluster import KMeans

num_clusters = 3
num_iterations = 3

for iteration in range(num_iterations):
    km = KMeans(n_clusters=num_clusters,
                init='random',
                n_init=1,
                max_iter=iteration+1,
                tol=1e-04,
                random_state=0)
    y_km = km.fit_predict(X)

    plot_decision_regions(X, y_km, km)
    plot_clusters(X, y_km, km.cluster_centers_)
    plt.legend()
    plt.title('# of iteration: ' + str(iteration))
    plt.grid()
    plt.tight_layout()
    plt.show()
```

```python
# estimate inertia, i.e. sum of distances from samples to nearest cluster centers

num_trials = 5
num_iterations = 10

inertia = np.zeros(num_iterations)

for iteration in range(num_iterations):
    cost = 0
    for trial in range(num_trials):
        km = KMeans(n_clusters=num_clusters,
                    init='random',
                    n_init=1,
                    max_iter=iteration+1,
                    tol=1e-04,
                    random_state=0)
        y_km = km.fit_predict(X)
        cost = cost + km.inertia_
    inertia[iteration] = cost/num_trials

#print(inertia)

plt.plot(range(1, len(inertia)+1), inertia, marker='o')
plt.xlabel('iterations')
plt.ylabel('sum of distances of samples to closest centroids')
plt.tight_layout()
plt.show()
```

## K-means clustering algorithm

Input: a set of samples $\mathbf{X}$
* without labels $\mathbf{y}$ as in supervised classification

Distance measure: assume $L_2$ for simplicity and commonality
$$
\begin{align}
d\left(x^{(i)}, x^{(j)}\right)
&= \|x^{(i)} - x^{(j)} \|_2
\\
&= \sqrt{\sum_k \left(x^{(i)}_k - x^{(j)}_k \right)^2}
\end{align}
$$

Steps:
* Initialization: pick (e.g. randomly) $k$ centroids $\mathbf{U} = \{\mu^{(j)}\}, j \in \{1, \cdots, k \}$ from $\mathbf{X}$ as initial cluster centers

* Voronoi: assign each sample $x^{(i)} \in \mathbf{X}$ to the nearest centroid $\mu\left(x^{(i)}\right)$
$$
\begin{align}
\mu\left(x^{(i)}\right) = argmin_{\mu \in \mathbf{U}} d(x^{(i)}, \mu)
\end{align}
$$

* Centroid: move each $\mu^{(j)} \in \mathbf{U}$ to the center of the samples that were assigned to it (i.e. the cluster of $\mu^{(j)}$)
$$
\begin{align}
\mu^{(j)} &= \frac{1}{\sum w_{ij}} \sum w_{ij} x^{(i)}
\\
w_{ij} &=
\begin{cases}
1 \; if \mu\left( x^{(i)}\right) = \mu^{(j)}
\\
0 \; else
\end{cases}
\end{align}
$$

* Repeat the Vornoi and centroid steps until the cluster assignments do not change much or a maximum number of iterations is reached

## Visualizing the clustering process

https://www.youtube.com/watch?v=S0sAnabdCLg

```python
from IPython.display import YouTubeVideo
YouTubeVideo('S0sAnabdCLg')
```

## aka Lloyd's algorithm

Maroon dot: previous centers

Black cross: next centers

Iteration 1:

Iteration 2:

Iteration 3:

Iteration 15:

## Background and intuition

$\mathbf{X} = \{x^{(i)}\}$: data samples

$\mathbf{U} = \{\mu^{(j)} \}$: cluster centers

The goal is to minimize the following objective function:
$$
\begin{align}
E\left(\mathbf{X} , \mathbf{U} \right)
&=
\sum_i \sum_j w_{ij} d^2\left(x^{(i)}, \mu^{(j)}\right)
\end{align}
$$
Subject to the constraint that each sample is assigned to one cluster:
$$
\begin{align}
\sum_j w_{ij} &= 1
\\
w_{ij} &\in \{0,  1\}
\end{align}
$$

Intuitively, we want to find a set of centroids (cluster centers) $\mathbf{U}$ to minimize the total distance of each sample $x^{(i)} \in \mathbf{X}$ with the nearest centroid $\mu^{(j)} \in \mathbf{U}$.

So that the clusters are as tight as possible.

For $L_2$ distance we have
$$
\begin{align}
E\left(\mathbf{X} , \mathbf{U} \right)
&=
\sum_i \sum_j w_{ij} \|x^{(i)} - \mu^{(j)}\|^2
\\
&=
\sum_j \sum_i w_{ij} \|x^{(i)} - \mu^{(j)} \|^2
\end{align}
$$

For the Voronoi step, we try to minimize each inner term of the $\sum_i \sum_j$ order:
$$
\begin{align}
\sum_j w_{ij} \| x^{(i)} - \mu^{(j)} \|^2
\end{align}
$$
Note that the constraints on $w_{ij}$ forces us to choose only one $\mu^{(j)}$.

For the centroid step, we try to minimize the inner term of the $\sum_j \sum_i$ order:
$$
\begin{align}
\sum_i w_{ij} \| x^{(i)} - \mu^{(j)} \|^2
\end{align}
$$
Only samples assign to cluster $j$ matters, i.e. $w_{ij} = 1$, so via calculus we know the optimal solution is to put $\mu_{(j)}$ at the center of all $x^{(i)}$ assigned to it.
$$
\begin{align}
\mu^{(j)} &= \frac{1}{\sum w_{ij}} \sum w_{ij} x^{(i)}
\end{align}
$$

## K-means++ intialization

Random selection
* might not be the best way to initialize the cluster centers
* badly (unluckily) placed centroids can cause convergence issues
* multiple random initialization can help

More systematic approach
* choose initial cluster centers to be as far away from one another as possible
* sequential process, during which the next cluster center is chosen to be the data sample farthest away from all existing centers

This is just for initialization
* the rest, Vornoi and centroid steps, remain the same

## Example differences for initialization

```python
from sklearn.cluster import KMeans

init_options = ['random', 'k-means++']

for init_option in init_options:

    km = KMeans(n_clusters=3,
                init=init_option,
                n_init=1, # just one trial to help visualization
                max_iter=1, # just 1 iteration to help visualize the initial condition
                tol=1e-04,
                random_state=0)
    y_km = km.fit_predict(X)

    plot_decision_regions(X, y_km, km)
    plot_clusters(X, y_km, km.cluster_centers_)
    plt.title(init_option)
    plt.legend()
    plt.grid()
    plt.tight_layout()
    plt.show()
```

## K-means++ initialization algorithm

$\mathbf{U}$: set of centroids, intially empty:
$\mathbf{U} \leftarrow \emptyset$

$\mathbf{U} \leftarrow x^{(i)} \in \mathbf{X}$ via random selection

While $\|\mathbf{U}\| < k$, the target number of clusters
* compute $d^2\left(x^{(i)}, \mathbf{U} \right)$ for each $x^{(i)} \in \mathbf{X}$ to the nearest member in $\mathbf{U}$
* $\mu^{(j)} \leftarrow$ random selection with probability
$\frac{d^2\left(x, \mathbf{U} \right)}{\sum_i d^2\left(x^{(i)}, \mathbf{U} \right)}$
for $x \in \mathbf{X}$
* $\mathbf{U} \leftarrow \mathbf{U} \bigcup \mu^{(j)}$

## Challenges with k-means

The number of clusters k is a hyper-parameter
* not easy to pick
* especially in high dimensions when we cannot visualize easily like in 2D

The clusters:
* no overlap (hard instead of soft decision)
* not hierarchical

We can improve these via other variations of clustering algorithms.

# Hard versus soft clustering

Hard clustering
* each sample is assigned to one cluster

Soft (fuzzy) clustering
* each sample has a probabilistic assignment to all clusters

Main difference: $w_{ij}$, the weight of assigning $x^{(i)}$ to $\mu^{(j)}$
* hard: $w_{ij} \in \{0, 1\}$, binary
* soft: $w_{ij} \in [0, 1]$, continuous

### Objective

$\mathbf{X}$: samples

$\mathbf{U}$: cluster centers

$w_{ij}$: the weight of assigning $x^{(i)}$ to $\mu^{(j)}$

$$
\begin{align}
w_{ij} =
\left[ \sum_{p=1}^{k} \left( \frac{\| x^{(i)} - \mu^{(j)}\|_2}{\| x^{(i)} - \mu^{(p)} \|_2} \right)^{\frac{2}{m-1}} \right]^{-1}
\end{align}
$$

For example, if $k = 3$:
$$
\begin{align}
w_{ij} =
\left[
\left( \frac{\| x^{(i)} - \mu^{(j)}\|_2}{\| x^{(i)} - \mu^{(1)} \|_2} \right)^{\frac{2}{m-1}}
+
\left( \frac{\| x^{(i)} - \mu^{(j)}\|_2}{\| x^{(i)} - \mu^{(2)} \|_2} \right)^{\frac{2}{m-1}}
+
\left( \frac{\| x^{(i)} - \mu^{(j)}\|_2}{\| x^{(i)} - \mu^{(3)} \|_2} \right)^{\frac{2}{m-1}}
\right]^{-1}
\end{align}
$$

$m$: degree of fuzziness
* $m \in [1, \infty)$
* $m = 1$: hard clustering
* $m > 1$: soft clustering, larger $m$ indicates fuzzier membership

$$
\begin{align}
E\left(\mathbf{X} , \mathbf{U} \right)
&=
\sum_i \sum_j w^m_{ij} \|x^{(i)} - \mu^{(j)}\|_2^2
\end{align}
$$

### Constraints
$$
\begin{align}
\sum_j w_{ij} &= 1
\end{align}
$$

Hard clustering - each sample can be assigned to one cluster
$$
\begin{align}
w_{ij} \in \{0,  1\}
\end{align}
$$

Soft clustering - no additional constraint aside from $[0, 1]$ and the sum to 1 above (probability)

### Example

For 3 clusters

#### Hard clustering:
$
w =
\begin{pmatrix}
0 \\
1 \\
0
\end{pmatrix}
$

Read: sample belongs to cluster 2

#### Soft clustering:
$
w =
\begin{pmatrix}
0.10 \\
0.85 \\
0.05
\end{pmatrix}
$

Read: sample is mostly like to belong to cluster 2 with probability $0.85$, but also some chances of belonging to other clusters.

## Fuzzy clustering algorithm

* Initialization: pick (e.g. randomly) $k$ centroids $\mathbf{U} = \{\mu^{(j)}\}, j \in \{1, \cdots, k \}$ from $\mathbf{X}$ as initial cluster centers

* Voronoi:
$$
\begin{align}
w_{ij} =
\left[ \sum_{p=1}^{k} \left( \frac{\| x^{(i)} - \mu^{(j)}\|_2}{\| x^{(i)} - \mu^{(p)} \|_2} \right)^{\frac{2}{m-1}} \right]^{-1}
\end{align}
$$

* Centroid:
$$
\begin{align}
\mu^{(j)} =
\frac{\sum_i w^m_{ij} x^{(i)}}{ \sum_i w^m_{ij}}
\end{align}
$$

* Repeat the Vornoi and centroid steps until the cluster assignments do not change much or a maximum number of iterations is reached

Note that when $m=1$, the above reduces to hard clustering.

What is the effect of $m$ on the Voronoi step?
* $m = 1$
* $m = \infty$
* $m \in (1, \infty)$

For example, if $k = 3$:
$$
\begin{align}
w_{ij} =
\left[
\left( \frac{\| x^{(i)} - \mu^{(j)}\|_2}{\| x^{(i)} - \mu^{(1)} \|_2} \right)^{\frac{2}{m-1}}
+
\left( \frac{\| x^{(i)} - \mu^{(j)}\|_2}{\| x^{(i)} - \mu^{(2)} \|_2} \right)^{\frac{2}{m-1}}
+
\left( \frac{\| x^{(i)} - \mu^{(j)}\|_2}{\| x^{(i)} - \mu^{(3)} \|_2} \right)^{\frac{2}{m-1}}
\right]^{-1}
\end{align}
$$

Let's say
\begin{align}
\| x^{(i)} - \mu^{(1)} \|_2 &= 1
\\
\| x^{(i)} - \mu^{(2)} \|_2 &= 2
\\
\| x^{(i)} - \mu^{(3)} \|_2 &= 3
\end{align}

Then
$$
\begin{align}
w_{i1} &=
\left[
\left( \frac{1}{1} \right)^{\frac{2}{m-1}}
+
\left( \frac{1}{2} \right)^{\frac{2}{m-1}}
+
\left( \frac{1}{3} \right)^{\frac{2}{m-1}}
\right]^{-1}
\\
w_{i2} &=
\left[
\left( \frac{2}{1} \right)^{\frac{2}{m-1}}
+
\left( \frac{2}{2} \right)^{\frac{2}{m-1}}
+
\left( \frac{2}{3} \right)^{\frac{2}{m-1}}
\right]^{-1}
\\
w_{i3} &=
\left[
\left( \frac{3}{1} \right)^{\frac{2}{m-1}}
+
\left( \frac{3}{2} \right)^{\frac{2}{m-1}}
+
\left( \frac{3}{3} \right)^{\frac{2}{m-1}}
\right]^{-1}
\end{align}
$$

When $m = 1$, hard clustering:
$$
\begin{align}
w_{i1} &= 1
\\
w_{i2} &= 0
\\
w_{i3} &= 0
\end{align}
$$

When $m = \infty$, equal fuzzy clustering:
$$
\begin{align}
w_{i1} &= \frac{1}{3}
\\
w_{i2} &= \frac{1}{3}
\\
w_{i3} &= \frac{1}{3}
\end{align}
$$

# Using the elbow method to find the optimal number of clusters

The number of clusters is a hyper-parameter

How to select it?

How to evaluate our model in general?
* This is unsupervised learning, so no ground truth data to compare against

## Solution

Run multiple experiments with different number of clusters

Measure the energy function, plot it against different number of clusters
$$
\begin{align}
E\left(\mathbf{X} , \mathbf{U} \right)
&=
\sum_i \sum_j w^m_{ij} d^2\left(x^{(i)}, \mu^{(j)}\right)
\end{align}
$$

## Example

```python
# the inertia_ class variable stores the energy value
print('Distortion: %.2f' % km.inertia_)
```

    Distortion: 76.30

```python
distortions = []
for i in range(1, 11):
    km = KMeans(n_clusters=i,
                init='k-means++',
                n_init=10,
                max_iter=300,
                random_state=0)
    km.fit(X)
    distortions.append(km.inertia_)
plt.plot(range(1, 11), distortions, marker='o')
plt.xlabel('Number of clusters')
plt.ylabel('Distortion')
plt.tight_layout()
#plt.savefig('./figures/elbow.png', dpi=300)
plt.show()
```

Intuitively, we want to pick the "elbow" part, which achieves best bang for the buck.
* diminishing returns after that elbow/knee point

## Quantifying the quality of clustering  via silhouette plots

Silhouette analysis
* another way to evaluate the quality of clustering

Basic idea
* coherence of each cluster
* separation from other clusters
* high coherence, high separation means good clustering

### Silhouette for clustering

https://en.wikipedia.org/wiki/Silhouette_(clustering)

For each data sample $x^{(i)} \in \mathbf{X}$, we can compute its average distance to a cluster $\mathbf{C} \in \{\mathbf{C}_1, \cdots, \mathbf{C}_k\}$:

$$
\begin{align}
\overline{d}\left(x^{(i)}, \mathbf{C} \right)
&=
\frac{1}{\| \mathbf{C} \|} \sum_{x^{(j)} \in \mathbf{C}} d\left(x^{(i)}, x^{(j)}\right)
\end{align}
$$
, where $d$ is the distance measure (e.g. $L_2$) used for clustering.

We usually skip $x^{(i)}$ comparing against itself, so if $x^{(i)} \in \mathbf{C}$, we have

$$
\begin{align}
\overline{d}\left(x^{(i)}, \mathbf{C} \right)
&=
\frac{1}{\| \mathbf{C} \| - 1} \sum_{x^{(j)} \in \mathbf{C}, j \neq i} d\left(x^{(i)}, x^{(j)}\right)
\end{align}
$$

#### Coherence
$a(i)$: the average distance (dis-similarity) to all other samples within the same cluster:

$$
\begin{align}
a(i) &= \overline{d}\left(x^{(i)}, \mathbf{C} \right)
\\
x^{(i)} &\in \mathbf{C}
\end{align}
$$

#### Separation
$b(i)$: the average distance to the nearest cluster that $x^{(i)}$ does not belong to:

$$
\begin{align}
b(i) &= \min \overline{d}\left(x^{(i)}, \mathbf{C} \right)
\\
\mathbf{C} &\in \{\mathbf{C}_1, \cdots, \mathbf{C}_k \}
\\
x^{(i)} &\not\in \mathbf{C}
\end{align}
$$

#### Silhouette

$s(i)$: the silhouette value of $x^{(i)}$:
$$
\begin{align}
s(i) = \frac{b(i) - a(i)}{\max\left(b(i), a(i)\right)}
\end{align}
$$

Which can be spelled out as:
$$
\begin{align}
s(i) =
\begin{cases}
1 - \frac{a(i)}{b(i)}, & a(i)  b(i)
\end{cases}
\end{align}
$$

So
$
-1 \leq s(i) \leq 1
$
* $s(i)$ is close to 1 if we have $a(i) \ll b(i)$ $\rightarrow$ good clustering
* $s(i)$ is close to $-1$ $\rightarrow$ bad clustering

#### Silhouette profile

Plot silhouette values $s(i)$ for all samples $x^{(i)} \in \mathbf{X}$ to visualize the clustering quality.

### Example

```python
import numpy as np
from matplotlib import cm
from sklearn.metrics import silhouette_samples

# clustering
km = KMeans(n_clusters=3,
            init='k-means++',
            n_init=10,
            max_iter=300,
            tol=1e-04,
            random_state=0)
y_km = km.fit_predict(X)

cluster_labels = np.unique(y_km)
n_clusters = cluster_labels.shape[0]

# the main part is just one functional call
silhouette_vals = silhouette_samples(X, y_km, metric='euclidean')

# visualization
y_ax_lower, y_ax_upper = 0, 0
yticks = []
for i, c in enumerate(cluster_labels):
    c_silhouette_vals = silhouette_vals[y_km == c]
    c_silhouette_vals.sort()
    y_ax_upper += len(c_silhouette_vals)
    color = cm.jet(float(i) / n_clusters)
    plt.barh(range(y_ax_lower, y_ax_upper), c_silhouette_vals, height=1.0,
             edgecolor='none', color=color)

    yticks.append((y_ax_lower + y_ax_upper) / 2.)
    # y_ax_lower += len(c_silhouette_vals)
    y_ax_lower = y_ax_upper # Li-yi: clearer meaning

silhouette_avg = np.mean(silhouette_vals)
plt.axvline(silhouette_avg, color="red", linestyle="--")

plt.yticks(yticks, cluster_labels + 1)
plt.ylabel('Cluster')
plt.xlabel('Silhouette coefficient')

plt.tight_layout()
# plt.savefig('./figures/silhouette.png', dpi=300)
plt.show()
```

We want:
* the average silhouette value to be high
* each cluster to have good silhouette distribution

Comparison to "bad" clustering:

```python
km = KMeans(n_clusters=2,
            init='k-means++',
            n_init=10,
            max_iter=300,
            tol=1e-04,
            random_state=0)
y_km = km.fit_predict(X)

plot_decision_regions(X, y_km, km)
plot_clusters(X, y_km, km.cluster_centers_)
plt.legend()
plt.grid()
plt.tight_layout()
plt.show()
```

```python
cluster_labels = np.unique(y_km)
n_clusters = cluster_labels.shape[0]
silhouette_vals = silhouette_samples(X, y_km, metric='euclidean')
y_ax_lower, y_ax_upper = 0, 0
yticks = []
for i, c in enumerate(cluster_labels):
    c_silhouette_vals = silhouette_vals[y_km == c]
    c_silhouette_vals.sort()
    y_ax_upper += len(c_silhouette_vals)
    color = cm.jet(float(i) / n_clusters)
    plt.barh(range(y_ax_lower, y_ax_upper), c_silhouette_vals, height=1.0,
             edgecolor='none', color=color)

    yticks.append((y_ax_lower + y_ax_upper) / 2.)
    y_ax_lower += len(c_silhouette_vals)

silhouette_avg = np.mean(silhouette_vals)
plt.axvline(silhouette_avg, color="red", linestyle="--")

plt.yticks(yticks, cluster_labels + 1)
plt.ylabel('Cluster')
plt.xlabel('Silhouette coefficient')

plt.tight_layout()
# plt.savefig('./figures/silhouette_bad.png', dpi=300)
plt.show()
```

Notice
* the low average silhouette value
* one cluster has bad silhouette distribution compared to another

# Organizing clusters as a hierarchical tree

Instead of compute $k$ clusters at once, we can compute them gradually in a hierarchy.

We can decide $k$ later after having the complete hierarchy.

## Construction

Two ways to do this:

### Bottom-to-top (agglomerative)
Start with each sample as one cluster, and merge the similar ones until we have one cluster (that contains all samples)

* red $\bigcup$ green $\rightarrow$ yellow
* yellow $\bigcup$ blue $\rightarrow$ cyan

### Top-to-bottom (divisive)
Start with a single cluster (that contains all samples), divide the most diverse one until each sample is in a separate cluster.

* cyan $\rightarrow$ yellow $\bigcup$ blue
* yellow $\rightarrow$ red $\bigcup$ green

## Criteria

How to decide the similarity between two clusters?

### Single linkage

Measure the shortest distance between pairs of samples from the two clusters.

### Complete linkage

Measure the longest distance between pairs of samples from the two clusters.

### Other possibilities

Ward's linkage: MSE of clusters
$$
\begin{align}
E\left(\mathbf{X} , \mathbf{U} \right)
&=
\sum_i \sum_j w^m_{ij} d\left(x^{(i)}, \mu^{(j)}\right)
\end{align}
$$

## Agglomerative clustering

## Example

```python
import pandas as pd
import numpy as np

np.random.seed(123)

variables = ['X', 'Y', 'Z']
labels = ['ID_0', 'ID_1', 'ID_2', 'ID_3', 'ID_4']

X = np.random.random_sample([5, 3])*10
df = pd.DataFrame(X, columns=variables, index=labels)
df
```

      X
      Y
      Z

      ID_0
      6.964692
      2.861393
      2.268515

      ID_1
      5.513148
      7.194690
      4.231065

      ID_2
      9.807642
      6.848297
      4.809319

      ID_3
      3.921175
      3.431780
      7.290497

      ID_4
      4.385722
      0.596779
      3.980443

## Performing hierarchical clustering on a distance matrix

```python
from scipy.spatial.distance import pdist, squareform

row_dist = pd.DataFrame(squareform(pdist(df, metric='euclidean')),
                        columns=labels,
                        index=labels)
row_dist
```

      ID_0
      ID_1
      ID_2
      ID_3
      ID_4

      ID_0
      0.000000
      4.973534
      5.516653
      5.899885
      3.835396

      ID_1
      4.973534
      0.000000
      4.347073
      5.104311
      6.698233

      ID_2
      5.516653
      4.347073
      0.000000
      7.244262
      8.316594

      ID_3
      5.899885
      5.104311
      7.244262
      0.000000
      4.382864

      ID_4
      3.835396
      6.698233
      8.316594
      4.382864
      0.000000

## Linkage usage

Either one of these will do:
* pass a condensed distance matrix (upper triangular) from the `pdist` function
* pass the "original" data array and define the `metric='euclidean'` argument in `linkage`

However, we should not pass the squareform distance matrix, which would yield different distance values although the overall clustering could be the same.

```python
from scipy.cluster.hierarchy import linkage

help(linkage)
```

    Help on function linkage in module scipy.cluster.hierarchy:

    linkage(y, method='single', metric='euclidean')
        Performs hierarchical/agglomerative clustering on the condensed
        distance matrix y.

        y must be a :math:`{n \choose 2}` sized
        vector where n is the number of original observations paired
        in the distance matrix. The behavior of this function is very
        similar to the MATLAB linkage function.

        An :math:`(n-1)` by 4  matrix ``Z`` is returned. At the
        :math:`i`-th iteration, clusters with indices ``Z[i, 0]`` and
        ``Z[i, 1]`` are combined to form cluster :math:`n + i`. A
        cluster with an index less than :math:`n` corresponds to one of
        the :math:`n` original observations. The distance between
        clusters ``Z[i, 0]`` and ``Z[i, 1]`` is given by ``Z[i, 2]``. The
        fourth value ``Z[i, 3]`` represents the number of original
        observations in the newly formed cluster.

        The following linkage methods are used to compute the distance
        :math:`d(s, t)` between two clusters :math:`s` and
        :math:`t`. The algorithm begins with a forest of clusters that
        have yet to be used in the hierarchy being formed. When two
        clusters :math:`s` and :math:`t` from this forest are combined
        into a single cluster :math:`u`, :math:`s` and :math:`t` are
        removed from the forest, and :math:`u` is added to the
        forest. When only one cluster remains in the forest, the algorithm
        stops, and this cluster becomes the root.

        A distance matrix is maintained at each iteration. The ``d[i,j]``
        entry corresponds to the distance between cluster :math:`i` and
        :math:`j` in the original forest.

        At each iteration, the algorithm must update the distance matrix
        to reflect the distance of the newly formed cluster u with the
        remaining clusters in the forest.

        Suppose there are :math:`|u|` original observations
        :math:`u[0], \ldots, u[|u|-1]` in cluster :math:`u` and
        :math:`|v|` original objects :math:`v[0], \ldots, v[|v|-1]` in
        cluster :math:`v`. Recall :math:`s` and :math:`t` are
        combined to form cluster :math:`u`. Let :math:`v` be any
        remaining cluster in the forest that is not :math:`u`.

        The following are methods for calculating the distance between the
        newly formed cluster :math:`u` and each :math:`v`.

          * method='single' assigns

            .. math::
               d(u,v) = \min(dist(u[i],v[j]))

            for all points :math:`i` in cluster :math:`u` and
            :math:`j` in cluster :math:`v`. This is also known as the
            Nearest Point Algorithm.

          * method='complete' assigns

            .. math::
               d(u, v) = \max(dist(u[i],v[j]))

            for all points :math:`i` in cluster u and :math:`j` in
            cluster :math:`v`. This is also known by the Farthest Point
            Algorithm or Voor Hees Algorithm.

          * method='average' assigns

            .. math::
               d(u,v) = \sum_{ij} \frac{d(u[i], v[j])}
                                       {(|u|*|v|)}

            for all points :math:`i` and :math:`j` where :math:`|u|`
            and :math:`|v|` are the cardinalities of clusters :math:`u`
            and :math:`v`, respectively. This is also called the UPGMA
            algorithm.

          * method='weighted' assigns

            .. math::
               d(u,v) = (dist(s,v) + dist(t,v))/2

            where cluster u was formed with cluster s and t and v
            is a remaining cluster in the forest. (also called WPGMA)

          * method='centroid' assigns

            .. math::
               dist(s,t) = ||c_s-c_t||_2

            where :math:`c_s` and :math:`c_t` are the centroids of
            clusters :math:`s` and :math:`t`, respectively. When two
            clusters :math:`s` and :math:`t` are combined into a new
            cluster :math:`u`, the new centroid is computed over all the
            original objects in clusters :math:`s` and :math:`t`. The
            distance then becomes the Euclidean distance between the
            centroid of :math:`u` and the centroid of a remaining cluster
            :math:`v` in the forest. This is also known as the UPGMC
            algorithm.

          * method='median' assigns :math:`d(s,t)` like the ``centroid``
            method. When two clusters :math:`s` and :math:`t` are combined
            into a new cluster :math:`u`, the average of centroids s and t
            give the new centroid :math:`u`. This is also known as the
            WPGMC algorithm.

          * method='ward' uses the Ward variance minimization algorithm.
            The new entry :math:`d(u,v)` is computed as follows,

            .. math::

               d(u,v) = \sqrt{\frac{|v|+|s|}
                                   {T}d(v,s)^2
                            + \frac{|v|+|t|}
                                   {T}d(v,t)^2
                            - \frac{|v|}
                                   {T}d(s,t)^2}

            where :math:`u` is the newly joined cluster consisting of
            clusters :math:`s` and :math:`t`, :math:`v` is an unused
            cluster in the forest, :math:`T=|v|+|s|+|t|`, and
            :math:`|*|` is the cardinality of its argument. This is also
            known as the incremental algorithm.

        Warning: When the minimum distance pair in the forest is chosen, there
        may be two or more pairs with the same minimum distance. This
        implementation may chose a different minimum than the MATLAB
        version.

        Parameters
        ----------
        y : ndarray
            A condensed or redundant distance matrix. A condensed distance matrix
            is a flat array containing the upper triangular of the distance matrix.
            This is the form that ``pdist`` returns. Alternatively, a collection of
            :math:`m` observation vectors in n dimensions may be passed as an
            :math:`m` by :math:`n` array.
        method : str, optional
            The linkage algorithm to use. See the ``Linkage Methods`` section below
            for full descriptions.
        metric : str or function, optional
            The distance metric to use in the case that y is a collection of
            observation vectors; ignored otherwise. See the ``distance.pdist``
            function for a list of valid distance metrics. A custom distance
            function can also be used. See the ``distance.pdist`` function for
            details.

        Returns
        -------
        Z : ndarray
            The hierarchical clustering encoded as a linkage matrix.

```python
# 1. incorrect approach: Squareform distance matrix

from scipy.cluster.hierarchy import linkage

row_clusters = linkage(row_dist, method='complete', metric='euclidean')
pd.DataFrame(row_clusters,
             columns=['row label 1', 'row label 2',
                      'distance', 'no. of items in clust.'],
             index=['cluster %d' % (i + 1)
                    for i in range(row_clusters.shape[0])])
```

      row label 1
      row label 2
      distance
      no. of items in clust.

      cluster 1
      0.0
      4.0
      6.521973
      2.0

      cluster 2
      1.0
      2.0
      6.729603
      2.0

      cluster 3
      3.0
      5.0
      8.539247
      3.0

      cluster 4
      6.0
      7.0
      12.444824
      5.0

```python
# 2. correct approach: Condensed distance matrix

row_clusters = linkage(pdist(df, metric='euclidean'), method='complete')
pd.DataFrame(row_clusters,
             columns=['row label 1', 'row label 2',
                      'distance', 'no. of items in clust.'],
             index=['cluster %d' % (i + 1)
                    for i in range(row_clusters.shape[0])])
```

      row label 1
      row label 2
      distance
      no. of items in clust.

      cluster 1
      0.0
      4.0
      3.835396
      2.0

      cluster 2
      1.0
      2.0
      4.347073
      2.0

      cluster 3
      3.0
      5.0
      5.899885
      3.0

      cluster 4
      6.0
      7.0
      8.316594
      5.0

```python
# 3. correct approach: Input sample matrix

row_clusters = linkage(df.values, method='complete', metric='euclidean')
pd.DataFrame(row_clusters,
             columns=['row label 1', 'row label 2',
                      'distance', 'no. of items in clust.'],
             index=['cluster %d' % (i + 1)
                    for i in range(row_clusters.shape[0])])
```

      row label 1
      row label 2
      distance
      no. of items in clust.

      cluster 1
      0.0
      4.0
      3.835396
      2.0

      cluster 2
      1.0
      2.0
      4.347073
      2.0

      cluster 3
      3.0
      5.0
      5.899885
      3.0

      cluster 4
      6.0
      7.0
      8.316594
      5.0

The ids in each row correspond to a leaf (data sample) or an internal node (cluster).

In this example, id 0 to 4 correspond to the original 5 samples.

Clusters are formed bottom up with id from 5 and increasing up.

The dendrogam can also help visualization.

```python
from scipy.cluster.hierarchy import dendrogram

# make dendrogram black (part 1/2)
# from scipy.cluster.hierarchy import set_link_color_palette
# set_link_color_palette(['black'])

row_dendr = dendrogram(row_clusters,
                       labels=labels,
                       # make dendrogram black (part 2/2)
                       # color_threshold=np.inf
                       )
plt.tight_layout()
plt.ylabel('Euclidean distance')
#plt.savefig('./figures/dendrogram.png', dpi=300,
#            bbox_inches='tight')
plt.show()
```

## Attaching dendrograms to a heat map

```python
# plot row dendrogram
fig = plt.figure(figsize=(8, 8), facecolor='white')
axd = fig.add_axes([0.09, 0.1, 0.2, 0.6])

# note: for matplotlib < v1.5.1, please use orientation='right'
row_dendr = dendrogram(row_clusters, orientation='left')

# reorder data with respect to clustering
df_rowclust = df.ix[row_dendr['leaves'][::-1]]
axd.set_xticks([])
axd.set_yticks([])

# remove axes spines from dendrogram
for i in axd.spines.values():
        i.set_visible(False)

# plot heatmap
axm = fig.add_axes([0.23, 0.1, 0.6, 0.6])  # x-pos, y-pos, width, height
cax = axm.matshow(df_rowclust, interpolation='nearest', cmap='hot_r')
fig.colorbar(cax)
axm.set_xticklabels([''] + list(df_rowclust.columns))
axm.set_yticklabels([''] + list(df_rowclust.index))

# plt.savefig('./figures/heatmap.png', dpi=300)
plt.show()
```

## Applying agglomerative clustering via scikit-learn

As usual, coding in scikit-learn is pretty simple.

```python
from sklearn.cluster import AgglomerativeClustering

ac = AgglomerativeClustering(n_clusters=2, # number of final clusters
                             affinity='euclidean',
                             linkage='complete')
labels = ac.fit_predict(X)
print('Cluster labels: %s' % labels)
print(ac.children_)
```

    Cluster labels: [0 1 1 0 0]
    [[0 4]
     [1 2]
     [3 5]
     [6 7]]

The clustering result is consistent with the dendogram above.

# Locating regions of high density via DBSCAN

Density-based Spatial Clustering of Applications with Noise

A form of clustering different from k-means and hierarhical clustering
* more resilient to noise
* allows general cluster shapes

https://youtu.be/5E097ZLE9Sg?t=1m35s

```python
from IPython.display import YouTubeVideo
YouTubeVideo('5E097ZLE9Sg')
```

## Parameters

* $\epsilon$: neighborhood within radius $\epsilon$ of each sample point
* MinPts: minimum number of points within neighborhood to be considered as "dense" enough

These two define sample density.

## Definitions

### Core point
* at least MinPts of samples within radius $\epsilon$

### Border point
* fewer than MinPts within radius $\epsilon$
* but within $\epsilon$ of a core point

### Noise point
* everything else

## Algorithm

Form a separate cluster for each core point or a connected group of core points.
* Two core points are connected if they are within $\epsilon$ from each other.

Assign each border point to the cluster of its corresponding core point.

Ignore noise points.

## Example

Comparing k-means, hierarchical, and density-based clustering.

```python
from sklearn.datasets import make_moons

X, y = make_moons(n_samples=200, noise=0.05, random_state=0)
plt.scatter(X[:, 0], X[:, 1])
plt.tight_layout()
# plt.savefig('./figures/moons.png', dpi=300)
plt.show()
```

K-means and hierarchical clustering:

```python
f, (ax1, ax2) = plt.subplots(1, 2, figsize=(8, 3))

km = KMeans(n_clusters=2, random_state=0)
y_km = km.fit_predict(X)
ax1.scatter(X[y_km == 0, 0], X[y_km == 0, 1],
            c='lightblue', marker='o', s=40, label='cluster 1')
ax1.scatter(X[y_km == 1, 0], X[y_km == 1, 1],
            c='red', marker='s', s=40, label='cluster 2')
ax1.set_title('K-means clustering')

ac = AgglomerativeClustering(n_clusters=2,
                             affinity='euclidean',
                             linkage='complete')
y_ac = ac.fit_predict(X)
ax2.scatter(X[y_ac == 0, 0], X[y_ac == 0, 1], c='lightblue',
            marker='o', s=40, label='cluster 1')
ax2.scatter(X[y_ac == 1, 0], X[y_ac == 1, 1], c='red',
            marker='s', s=40, label='cluster 2')
ax2.set_title('Agglomerative clustering')

plt.legend()
plt.tight_layout()
#plt.savefig('./figures/kmeans_and_ac.png', dpi=300)
plt.show()
```

Intuitively, the two moons should form two separate clusters.

But geometric distance of k-means and hierarchical clustering might group the wrong parts together.

Density-based clustering considers the topology of the clusters:

```python
from sklearn.cluster import DBSCAN

db = DBSCAN(eps=0.2, min_samples=5, metric='euclidean')
y_db = db.fit_predict(X)
plt.scatter(X[y_db == 0, 0], X[y_db == 0, 1],
            c='lightblue', marker='o', s=40,
            label='cluster 1')
plt.scatter(X[y_db == 1, 0], X[y_db == 1, 1],
            c='red', marker='s', s=40,
            label='cluster 2')
plt.legend()
plt.tight_layout()
#plt.savefig('./figures/moons_dbscan.png', dpi=300)
plt.show()
```

## Disadvantages of density-based clustering

Two hyper-parameters ($\epsilon$ and MinPts) to tune
* versus one $k$ for k-means and hierarhical clustering
* aside from distance measure as another hyper parameter

Does not have a predict() method
* non-parametric, applies to the current dataset only

```python
from sklearn.cluster import DBSCAN

eps_values = [0.1, 0.2, 0.5]

for eps_value in eps_values:
    db = DBSCAN(eps=eps_value, min_samples=5, metric='euclidean')
    y_db = db.fit_predict(X)
    plt.scatter(X[y_db == 0, 0], X[y_db == 0, 1],
                c='lightblue', marker='o', s=40,
                label='cluster 1')
    plt.scatter(X[y_db == 1, 0], X[y_db == 1, 1],
                c='red', marker='s', s=40,
                label='cluster 2')
    plt.legend()
    plt.tight_layout()
    plt.title('eps = ' + str(eps_value))
    plt.show()
```

# Summary

Clustering is a non-supervised learning method.

Three main types of clustering with different flavors and trade-offs:
* k-means
* hierarchical
* density-based

# Application to real dataset

We have been using a synthetic data set (random blobs) so far.

Let's try apply clustering to a read data set: handwritten digits.
* each digit image came with a label for supervised learning
* large datasets can be expensive to label manually

Solution
* cluster first
* manually label a few samples within each cluster
* propagate the labels for the rest

Verification
* use a dataset with labels
* ignore the labels and just cluster
* would the clusters agree with the labels?

# Reading

* PML Chapter 11
* IML Chapter 7.3, 7.7, 7.8


[source](../api/alibi_detect.cd.mmd.rst)

# Maximum Mean Discrepancy

## Overview

The [Maximum Mean Discrepancy (MMD)](http://jmlr.csail.mit.edu/papers/v13/gretton12a.html) detector is a kernel-based method for multivariate 2 sample testing. The MMD is a distance-based measure between 2 distributions *p* and *q* based on the mean embeddings $\mu_{p}$ and $\mu_{q}$ in a reproducing kernel Hilbert space $F$:

\begin{align}
MMD(F, p, q) & = || \mu_{p} - \mu_{q} ||^2_{F} \\
\end{align}

We can compute unbiased estimates of $MMD^2$ from the samples of the 2 distributions after applying the kernel trick. We use by default a [radial basis function kernel](https://en.wikipedia.org/wiki/Radial_basis_function_kernel), but users are free to pass their own kernel of preference to the detector. We obtain a $p$-value via a [permutation test](https://en.wikipedia.org/wiki/Resampling_(statistics)) on the values of $MMD^2$.

For high-dimensional data, we typically want to reduce the dimensionality before computing the permutation test. Following suggestions in [Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift](https://arxiv.org/abs/1810.11953), we incorporate Untrained AutoEncoders (UAE) and black-box shift detection using the classifier's softmax outputs ([BBSDs](https://arxiv.org/abs/1802.03916)) as out-of-the box preprocessing methods and note that [PCA](https://en.wikipedia.org/wiki/Principal_component_analysis) can also be easily implemented using `scikit-learn`. Preprocessing methods which do not rely on the classifier will usually pick up drift in the input data, while BBSDs focuses on label shift.

Detecting input data drift (covariate shift) $\Delta p(x)$ for text data requires a custom preprocessing step. We can pick up changes in the semantics of the input by extracting (contextual) embeddings and detect drift on those. Strictly speaking we are not detecting $\Delta p(x)$ anymore since the whole training procedure (objective function, training data etc) for the (pre)trained embeddings has an impact on the embeddings we extract. The library contains functionality to leverage pre-trained embeddings from [HuggingFace's transformer package](https://github.com/huggingface/transformers) but also allows you to easily use your own embeddings of choice. Both options are illustrated with examples in the [Text drift detection on IMDB movie reviews](../examples/cd_text_imdb.nblink) notebook.

## Usage

### Initialize

Arguments:

* `x_ref`: Data used as reference distribution.

Keyword arguments:

* `backend`: Both **TensorFlow** and **PyTorch** implementations of the MMD detector as well as various preprocessing steps are available. Specify the backend (*tensorflow* or *pytorch*). Defaults to *tensorflow*.

* `p_val`: p-value used for significance of the permutation test.

* `preprocess_x_ref`: Whether to already apply the (optional) preprocessing step to the reference data at initialization and store the preprocessed data. Dependent on the preprocessing step, this can reduce the computation time for the predict step significantly, especially when the reference dataset is large. Defaults to *True*. It is possible that it needs to be set to *False* if the preprocessing step requires statistics from both the reference and test data, such as the mean or standard deviation.

* `update_x_ref`: Reference data can optionally be updated to the last N instances seen by the detector or via [reservoir sampling](https://en.wikipedia.org/wiki/Reservoir_sampling) with size N. For the former, the parameter equals *{'last': N}* while for reservoir sampling *{'reservoir_sampling': N}* is passed.

* `preprocess_fn`: Function to preprocess the data before computing the data drift metrics. Typically a dimensionality reduction technique.

* `kernel`: Kernel used when computing the MMD. Defaults to a Gaussian RBF kernel (`from alibi_detect.utils.pytorch import GaussianRBF` or `from alibi_detect.utils.tensorflow import GaussianRBF` dependent on the backend used).

* `sigma`: Optional bandwidth for the kernel as a `np.ndarray`. We can also average over a number of different bandwidths, e.g. `np.array([.5, 1., 1.5])`.

* `configure_kernel_from_x_ref`: If `sigma` is not specified, the detector can infer it via a heuristic and set `sigma` to the median pairwise distance between 2 samples. If `configure_kernel_from_x_ref` is *True*, we can already set `sigma` at initialization of the detector by inferring it from `x_ref`, speeding up the prediction step. If set to *False*, `sigma` is computed separately for each test batch at prediction time.

* `n_permutations`: Number of permutations used in the permutation test.

* `input_shape`: Optionally pass the shape of the input data.

* `data_type`: can specify data type added to the metadata. E.g. *'tabular'* or *'image'*.

Additional PyTorch keyword arguments:

* `device`: *cuda* or *gpu* to use the GPU and *cpu* for the CPU. If the device is not specified, the detector will try to leverage the GPU if possible and otherwise fall back on CPU.

Initialized drift detector example:

```python
from alibi_detect.cd import MMDDrift

cd = MMDDrift(x_ref, backend='tensorflow', p_val=.05)
```

The same detector in PyTorch:

```python
cd = MMDDrift(x_ref, backend='pytorch', p_val=.05)
```

We can also easily add preprocessing functions for both frameworks. The following example uses a randomly initialized image encoder in PyTorch:

```python
from functools import partial
import torch
import torch.nn as nn
from alibi_detect.cd.pytorch import preprocess_drift

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# define encoder
encoder_net = nn.Sequential(
    nn.Conv2d(3, 64, 4, stride=2, padding=0),
    nn.ReLU(),
    nn.Conv2d(64, 128, 4, stride=2, padding=0),
    nn.ReLU(),
    nn.Conv2d(128, 512, 4, stride=2, padding=0),
    nn.ReLU(),
    nn.Flatten(),
    nn.Linear(2048, 32)
).to(device).eval()

# define preprocessing function
preprocess_fn = partial(preprocess_drift, model=encoder_net, device=device, batch_size=512)

cd = MMDDrift(x_ref, backend='pytorch', p_val=.05, preprocess_fn=preprocess_fn)
```
The same functionality is supported in TensorFlow and the main difference is that you would import from `alibi_detect.cd.tensorflow import preprocess_drift`. Other preprocessing steps such as the output of hidden layers of a model or extracted text embeddings using transformer models can be used in a similar way in both frameworks. TensorFlow example for the hidden layer output:

```python
from alibi_detect.cd.tensorflow import HiddenOutput, preprocess_drift

model = # TensorFlow model; tf.keras.Model or tf.keras.Sequential
preprocess_fn = partial(preprocess_drift, model=HiddenOutput(model, layer=-1), batch_size=128)

cd = MMDDrift(x_ref, backend='tensorflow', p_val=.05, preprocess_fn=preprocess_fn)
```

Check out the [Drift detection on CIFAR10](../examples/cd_mmd_cifar10.nblink) example for more details.

Alibi Detect also includes custom text preprocessing steps in both TensorFlow and PyTorch based on Huggingface's [transformers](https://github.com/huggingface/transformers) package:

```python
import torch
import torch.nn as nn
from transformers import AutoTokenizer
from alibi_detect.cd.pytorch import preprocess_drift
from alibi_detect.models.pytorch import TransformerEmbedding

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

model_name = 'bert-base-cased'
tokenizer = AutoTokenizer.from_pretrained(model_name)

embedding_type = 'hidden_state'
layers = [5, 6, 7]
embed = TransformerEmbedding(model_name, embedding_type, layers)
model = nn.Sequential(embed, nn.Linear(768, 256), nn.ReLU(), nn.Linear(256, enc_dim)).to(device).eval()
preprocess_fn = partial(preprocess_drift, model=model, tokenizer=tokenizer, max_len=512, batch_size=32)

# initialise drift detector
cd = MMDDrift(x_ref, backend='pytorch', p_val=.05, preprocess_fn=preprocess_fn)
```

Again the same functionality is supported in TensorFlow but with `from alibi_detect.cd.tensorflow import preprocess_drift` and `from alibi_detect.models.tensorflow import TransformerEmbedding` imports. Check out the [Text drift detection on IMDB movie reviews](../examples/cd_text_imdb.nblink) example for more information.

### Detect Drift

We detect data drift by simply calling `predict` on a batch of instances `x`. We can return the p-value and the threshold of the permutation test by setting `return_p_val` to *True* and the maximum mean discrepancy metric and threshold by setting `return_distance` to *True*.

The prediction takes the form of a dictionary with `meta` and `data` keys. `meta` contains the detector's metadata while `data` is also a dictionary which contains the actual predictions stored in the following keys:

* `is_drift`: 1 if the sample tested has drifted from the reference data and 0 otherwise.

* `p_val`: contains the p-value if `return_p_val` equals *True*.

* `threshold`: p-value threshold if `return_p_val` equals *True*.

* `distance`: MMD^2 metric between the reference data and the new batch if `return_distance` equals *True*.

* `distance_threshold`: MMD^2 metric value from the permutation test which corresponds to the the p-value threshold.

```python
preds = cd.predict(X, return_p_val=True, return_distance=True)
```

### Saving and loading

The drift detectors can be saved and loaded in the same way as other detectors:

```python
from alibi_detect.utils.saving import save_detector, load_detector

filepath = 'my_path'
save_detector(cd, filepath)
cd = load_detector(filepath)
```

Currently on the **TensorFlow** backend is supported for `save_detector` and `load_detector`. Adding **PyTorch** support is a near term priority.

## Examples

### Graph

[Drift detection on molecular graphs](../examples/cd_mol.nblink)

### Image

[Drift detection on CIFAR10](../examples/cd_mmd_cifar10.nblink)

### Text

[Text drift detection on IMDB movie reviews](../examples/cd_text_imdb.nblink)


# Proyecto - Calculadora
## **Programación**

### *Universidad Central*

### *Elaborado por:*

* Juan Castillo (Interfaz gráfica, Cálculo Vectorial)
* Laura Contreras (Pre-Álgebra, gráficas)
* Carlos Carvajales (Cálculo diferencial)
* Jessica Santos (Álgebra lineal)
* María García (Cálculo Integral)

Querido usuario,

Por favor, lee el manual para más información del funcionamiento de la calculadora.

```python
import matplotlib.pyplot as plt
from ipywidgets import interact, interactive, interact_manual, widgets
import numpy as np
import sympy as spp
from sympy.parsing.sympy_parser import parse_expr
from mpl_toolkits.mplot3d import Axes3D
from sympy.plotting import plot3d,plot, PlotGrid
from IPython.display import display, Markdown

x,y,z,w = spp.symbols('x y z w')

def pre_algebra(Tema, Funcion) :
    try:
        if Tema == 'Operaciones Básicas':
            print(float(parse_expr(Funcion)))

        if Tema == 'Factores y Números primos' :
            Funcion = int(Funcion)
            def fn(segundo_valor,máximo_común_divisor, Numero_primo):
                if máximo_común_divisor == True:
                    def mcd(Funcion, segundo_valor):
                        segundo_valor= parse_expr(segundo_valor)
                        Funcion, segundo_valor=max(int(Funcion), segundo_valor),min(int(Funcion), segundo_valor)
                        while segundo_valor!=0:
                            Funcion, segundo_valor=segundo_valor,Funcion%segundo_valor
                        return Funcion
                    print(mcd(int(Funcion), segundo_valor))
                if Numero_primo == True:
                    def pri(Funcion):
                        phi=0
                        for i in range(1,Funcion):
                            ai=Funcion
                            while i!=0:
                                ai,i=i,ai%i
                            if ai==1:
                                phi+=1
                        if phi== Funcion-1:
                            print("El numero", Funcion, "es primo.")
                        elif phi < Funcion-1:
                            print("El numero", Funcion, " no es primo.")
                    print(pri(Funcion))
            interact(fn, segundo_valor='', máximo_común_divisor= False, Numero_primo= False)

        if Tema == 'Logaritmos, Radicales y Exponenciales' :                          # Tema: Logaritmos, Radicales y Exponenciales
            def l(n, Logaritmos, Logaritmos_Naturales, Radicales, Exponenciales) :
                n = parse_expr(n)
                if Logaritmos == True:
                    try:                                                         # Logaritmos
                        huk = np.log(int(Funcion))/np.log(int(n))
                        print(huk)
                    except:
                        pass
                if Logaritmos_Naturales == True :
                    try:
                        print(np.log(int(Funcion)))
                    except:
                        pass
                if Radicales == True:                                         # Radicales
                    try:
                        n2=(int(Funcion))**(1/n)
                        print(spp.N(n2))
                    except:
                        pass
                if Exponenciales == True:                                     #Exponenciales
                    try:
                        n3=(int(Funcion))**n
                        print(n3)
                    except:
                        pass
            interact(l, n='',  Logaritmos = False, Logaritmos_Naturales = False, Radicales = False, Exponenciales = False)
    except:
        pass

def algebra_lineal(Tema):
    def ing_matrizU(matriz):
        try:
            matriz=parse_expr(matriz)
            m = spp.Matrix(matriz)
            global U
            U = m
            return
        except:
            print("Al parecer escribiste mal la matriz")

    def ing_matrizV(matriz):
        try:
            matriz=parse_expr(matriz)
            m = spp.Matrix(matriz)
            global V
            V = m
            return
        except:
            print("Al parecer escribiste mal la matriz")

    mat1=widgets.Text(description="Matriz",value="[[1,0],[0,1]]")
    mat2=widgets.Text(description="Matriz",value="[[1,0],[0,1]]")

    if Tema == 'Operaciones entre matrices':
        def sumar(n,m):
            try:
                print("Ingrese 2 matrices de igual tamaño para ser sumadas")
                suma=n+m
                return suma
            except:
                print("Las matrices no pueden ser sumadas")
        def resta(s,j):
            try:
                print("Ingrese 2 matrices de igual tamaño para ser restadas")
                restar=s-j
                return restar
            except:
                print("Las matrices no pueden ser restadas")
        def multi(n,m):
            try:
                print("Ingrese 2 matrices: La primera de tamaño m*n y la segunda n*p para ser multiplicadas")
                multi_vec=n*m
                return multi_vec
            except:
                print("No se puede realizar la multiplicacion entre matrices")
        def invert(m):
            try:
                matri_inv=m**-1
                return matri_inv
            except:
                print("Error:La inversa de una matriz solo puede ser calculada para matrices cuadradas")
        def det(m):
            try:
                dt=m.det()
                return dt
            except:
                print("Esta matriz no tiene determinante ya que debe ser cuadrada")

        def f(Operacion,m1,m2):
            mat1=ing_matrizU(m1)
            mat2=ing_matrizV(m2)
            m = U
            r = V
            if Operacion == 'Suma':
                k=sumar(m,r)
                k
            if Operacion == 'Resta':
                k=resta(m,r)
                k
            if Operacion == 'Multiplicación':
                k=multi(m,r)
                k
            if Operacion == 'Inversa':
                k=invert(m)
                l=invert(r)
                def inv(Inversa_1,Inversa_2):
                    if Inversa_1 == True:
                        print("Matriz inversa de la primera matriz",k)
                    if Inversa_2 == True:
                        print("Matriz inversa de la segunda matriz",l)
                    return inv
                interact(inv, Matriz='',  Inversa_1 = False, Inversa_2 = False)

            if Operacion == 'Determinante':
                k=det(m)
                l=det(r)
                def DET(Determinante_1,Determinante_2):
                    if Determinante_1 == True:
                        print("El determinante de la primera matriz es ",k)
                    if Determinante_2 == True:
                        print("El determinante de la segunda matriz es ",l)
                    return DET
                interact(DET, Matriz='',  Determinante_1= False, Determinante_2 = False)
            return k
        interact(f, Operacion =['Suma','Resta','Multiplicación','Inversa','Determinante'],m1=mat1,m2=mat2)

    if Tema == 'Sistema de ecuaciones lineales':
        def solucion(n,m):
            try:
                solucion1=spp.linsolve((n,m),[x,y,z])
                return solucion1
            except:
                print("Ups ocurrio algo, revisa la matriz")
        def g(tema,m1,m2):
            mat1=ing_matrizU(m1)
            mat2=ing_matrizV(m2)
            m = U
            r = V
            if tema == 'Solución por eliminación':
                print("Ingrese las matrices: Tenga en cuenta que la primera matriz es la matriz aumentada y la segunda es el resultado de la matriz aumentada")
                k=solucion(m,r)
                k
            return k
        interact(g, tema ='Solución por eliminación',m1=mat1,m2=mat2)
    if Tema == 'Operaciones entre vectores':
        import numpy as np
        def ing_matrizV(vector):
            try:
                vector=parse_expr(vector)
                vec_1= np.array(vector)
                m=vec_1[np.newaxis, :]
                global V
                V = m
                return
            except:
                print("Al parecer escribiste mal el vector")
        def ing_matrizU(vector):
            try:
                print("Ingresa dos vectores, recuerda que debe ser entre paréntesis y separado por comas: (1,2)")
                vector=parse_expr(vector)
                vec_1= np.array(vector)
                m=vec_1[np.newaxis, :]
                global U
                U = m
            except:
                print("Al parecer escribiste mal el vector")
        mat3=widgets.Text(description="Vector",value="(1,0)")
        mat4=widgets.Text(description="Vector",value="(1,0)")
        def sumar2(n,m):
            try:
                suma2=n+m
                return suma2
            except:
                print("Los vectores no pueden ser sumados")
        def resta2(s,j):
            try:
                restar2=j+-s
                return restar2
            except:
                print("Los vectores no pueden ser restados")
        def h(tema,m1,m2):
            mat3=ing_matrizU(m1)
            mat4=ing_matrizV(m2)
            m = U
            r = V
            if tema == 'Suma entre vectores':
                k=sumar2(r,m)
                print("Este es el resultado de sumar dos vectores:",k)
            if tema == 'Resta entre vectores':
                k=resta2(r,m)
                k
                return k
            if tema == 'Multiplicación':
                def ing_matrizZ(vector):
                    try:
                        vector=parse_expr(vector)
                        vec_1= np.array(vector)
                        m=vec_1[np.newaxis, :]
                        global V
                        V = m
                    except:
                        print("Al parecer escribiste mal el vector")
                def ing_matrizW(vector):
                    try:
                        print("Ingresa un vector y un escalar, recuerda que el vector debe ser entre paréntesis y separado por comas: (1,2)")
                        vector=parse_expr(vector)
                        m=vector
                        global U
                        U = m
                    except:
                        print("Al parecer escribiste mal el escalar")
                mat3=widgets.Text(description="Vector",value="(1,0)")
                mat4=widgets.Text(description="Escalar",value="#")
                def multip(n,m):
                    try:
                        multi_vec=n*m
                        return multi_vec
                    except:
                        print("No se puede realizar la multiplicacion por escalar")
                def i(tema,m1,m2):
                    mat3=ing_matrizZ(m1)
                    mat4=ing_matrizW(m2)
                    m = U
                    r = V
                    if tema == 'Multiplicación por escalar':
                        P=multip(m,r)
                        P
                    return P
                interact(i,tema='Multiplicación por escalar',m1=mat3,m2=mat4)

        interact(h, tema =['Suma entre vectores','Resta entre vectores','Multiplicación'],m1=mat3,m2=mat4)

def calculo_diferencial(Tema, Funcion):
    button1, button2 = widgets.Button(description="Dos Dimensiones"), widgets.Button(description="Tres Dimensiones")
    output1, output2 = widgets.Output(), widgets.Output()
    display(button1,output1); display(button2,output2)
    def DosD(dosd) :                            # 2D
        with output1:
            try:
                plot(Funcion,(x,-20,20), (y,-40,40))
            except:
                print("No es posible gráficar en R2")
    def TresD(tresd) :                           # 3D
        with output2:
            try:
                plot3d(Funcion,(x,-20,20),(y,-40,40), (z,-20,20))
            except:
                print("No es posible gráficar en R3")
    button1.on_click(DosD); button2.on_click(TresD)

    try:
        if Tema ==  'Derivada':
            try:
                def h2( Primera_derivada,Segunda_derivada, Tercera_derivada, Cuarta_derivada, Quinta_derivada):
                    if Primera_derivada == True :
                        try:
                            Primera_derivada = spp.diff(Funcion,x)
                            dom1 = np.arange(-20,20,0.1)
                            ran1 = [Primera_derivada.subs(x,i) for i in dom1]
                            plt.plot(dom1,ran1, 'r', label='Primera derivada')
                            plt.title("Primera derivada"); plt.legend(); plt.grid()
                            print(Primera_derivada)
                        except:
                            pass
                    if Segunda_derivada == True :
                        try:
                            Segunda_derivada = spp.diff(Funcion,x,2)
                            dom2 = np.arange(-20,20,0.1)
                            ran2 = [Segunda_derivada.subs(x,i) for i in dom2]
                            plt.plot(dom2,ran2, 'b', label = "Segunda derivada")
                            plt.title("Segunda derivada"); plt.legend(); plt.grid()
                            print(Segunda_derivada)
                        except:
                            pass
                    if Tercera_derivada == True :
                        try:
                            Tercera_derivada = spp.diff(Funcion,x,3)
                            dom3 = np.arange(-20,20,0.1)
                            ran3 = [Tercera_derivada.subs(x,i) for i in dom3]
                            plt.plot(dom3,ran3, 'g', label = "Tercera derivada")
                            plt.title("Tercera derivada"); plt.legend(); plt.grid()
                            print(Tercera_derivada)
                        except:
                            pass
                    if Cuarta_derivada == True :
                        try:
                            Cuarta_derivada = spp.diff(Funcion,x,4)
                            dom4 = np.arange(-20,20,0.1)
                            ran4 = [Cuarta_derivada.subs(x,i) for i in dom4]
                            plt.plot(dom4,ran4, 'c', label = "Cuarta derivada")
                            plt.title("Cuarta derivada");plt.legend(); plt.grid()
                            print(Cuarta_derivada)
                        except:
                            pass
                    if Quinta_derivada == True :
                        try:
                            Quinta_derivada = spp.diff(Funcion,x,5)
                            dom5 = np.arange(-20,20,0.1)
                            ran5 = [Quinta_derivada.subs(x,i) for i in dom5]
                            plt.plot(dom5,ran5, 'y', label = "Quinta derivada")
                            plt.title("Quinta derivada"); plt.legend(); plt.grid()
                            print(Quinta_derivada)
                        except:
                            pass
                    if Primera_derivada == True and Segunda_derivada == True and Tercera_derivada == True and Cuarta_derivada == True and Quinta_derivada == True :
                        try:
                            plt.plot(dom1, ran1, dom2, ran2, dom3, ran3, dom4, ran4, dom5, ran5)
                            plt.legend(); plt.grid()
                        except:
                            pass
                interact(h2, Primera_derivada = False, Segunda_derivada = False, Tercera_derivada= False, Cuarta_derivada= False, Quinta_derivada= False)
            except:
                pass

        if Tema == 'Límites':
            try:
                def l2(Tendencia):
                    Tendencia = parse_expr(Tendencia)
                    Limite = spp.limit(Funcion, x, Tendencia)
                    print (Limite)
                interact(l2, Tendencia = '')
            except:
                pass
        if Tema == 'Raíces de la función':
            try:
                def NR(valor_inicial):
                    valor_inicial = parse_expr(valor_inicial)
                    derivada = spp.diff(Funcion)
                    for i in range(15) :
                        a = spp.sympify(Funcion).subs(x,valor_inicial)                         #Raíces de funciones
                        b = spp.sympify(derivada).subs(x,valor_inicial)
                        valor_inicial = valor_inicial - spp.N(a/b)
                    return(valor_inicial)
                interact(NR, valor_inicial='')
            except:
                pass
        if Tema == 'Máximos y mínimos':
            try:
                f = Funcion
                dev1 = spp.diff(f)
                dev2 = spp.diff(dev1)
                enx = spp.solve(dev1, x)
                max_or_min = [dev2.subs(x,i) for i in enx]
                def mamin(max_or_min,enx):
                    try:
                        for i in range(len(max_or_min)):
                            if max_or_min[i] >= 0:                                                #Máximos y Mínimos
                                pos = f.subs(x, enx[i])
                                print((enx[i], pos), "es un mínimo")
                            elif max_or_min[i] < 0:
                                neg = f.subs(x, enx[i])
                                print((enx[i], neg), "es un máximo")
                            else:
                                break
                    except Exception:
                        print("La función tiene soluciones complejas. No es posible determinar máximos o mínimos.")
                print(mamin(max_or_min,enx))
            except:
                pass
    except:
        pass

def calculo_integral(Tema, Funcion):
    button1, button2 = widgets.Button(description="Dos Dimensiones"), widgets.Button(description="Tres Dimensiones")
    output1, output2 = widgets.Output(), widgets.Output()
    display(button1,output1); display(button2,output2)
    def DosD(dosd) :                            # 2D
        with output1:
            try:
                plot(Funcion,(x,-20,20), (y,-40,40))
            except:
                print("No es posible gráficar en R2")
    def TresD(tresd) :                           # 3D
        with output2:
            try:
                plot3d(Funcion,(x,-20,20),(y,-40,40), (z,-20,20))
            except:
                print("No es posible gráficar en R3")
    button1.on_click(DosD); button2.on_click(TresD)

    try:
        if Tema == 'Integrales de una variable':
            try:
                def n(Tipo):
                    if Tipo == 'Indefinida':
                        try:
                            def s(dx, dy):
                                if dx == True:
                                    try:
                                        Integral_indefinida = spp.integrate(parse_expr(Funcion), (x))
                                        dom_1 = np.arange(-20,20,0.1)
                                        ran_1 = [Integral_indefinida.subs(x,i) for i in dom_1]
                                        ran_2 = [parse_expr(Funcion).subs(x,i) for i in dom_1]
                                        plt.plot(dom_1,ran_1, 'r', dom_1, ran_2, 'b', label = Integral_indefinida)
                                        plt.title("Integral indefinida"); plt.legend(); plt.grid()
                                        print(spp.integrate(parse_expr(Funcion), (x)))
                                    except:
                                        pass
                                if dy == True:
                                    try:
                                        print(spp.integrate(parse_expr(Funcion), (y)))
                                    except:
                                        pass
                            interact(s, dx = False, dy = False)
                        except:
                            pass
                    if Tipo == 'Definida':
                        try:
                            def s(a, b, dx, dy):
                                if dx == True:
                                    print(spp.integrate(Funcion, (x, a, b)))
                                elif dy == True:
                                    print(spp.integrate(Funcion, (y, a, b)))

                            interact(s, a='', b='', dx = False, dy = False)
                        except:
                            pass
                interact(n, Tipo = ['Indefinida', 'Definida'])
            except:
                pass

        if Tema == 'Sumatoria':
            try:
                def k(a,b):
                    c = 0
                    for i in range(a, b+1):
                        c += parse_expr(Funcion).subs(x, i)
                    return(spp.N(c))
                interact(k, a=widgets.IntText(value=1), b=widgets.IntText(value=1))
            except:
                pass

        if Tema == 'Área bajo una curva':
            try:
                def s(a, b, dx, dy):
                    if dx == True:
                          print(spp.integrate(Funcion, (x, a, b)))
                    if dy == True:
                          print(spp.integrate(Funcion, (x, a, b)))
                interact(s, a='', b='', dx = False, dy = False)
            except:
                pass

        if Tema == 'Volumen bajo una curva':
            try:
                def s(a, b):
                    print(spp.integrate(parse_expr(Funcion)**2, (x, a, b)))
                interact(s, a='', b='')
            except:
                pass

        if Tema == 'Área entre dos curvas':
            try:
                def s(Funcion1, a, b, dx, dy):
                    try:
                        Funcion1 = parse_expr(Funcion1)
                        if dx == True:
                            for i in range(a,b+1):
                                F = Funcion.subs(x,i)
                                F1 = Funcion1.subs(x,i)
                            if F > F1:
                                c = Funcion - Funcion1
                            if F < F1:
                                c = Funcion1 - Funcion
                                print(spp.integrate(c, (x, a, b)))
                        elif dy == True:
                            for i in range(a,b+1):
                                F = Funcion.subs(y,i)
                                F1 = Funcion1.subs(y,i)
                            if F > F1:
                                c = Funcion - Funcion1
                            if F < F1:
                                c = Funcion1 - Funcion
                            print(spp.integrate(c, (y, a, b)))
                    except Exception:
                        print("No es posible calcular el área.")
                interact(s, Funcion1='', a=widgets.IntText(value=1), b=widgets.IntText(value=1), dx = False, dy = False)
            except:
                pass

        if Tema == 'Volumen entre dos curvas':
            try:
                def s(Funcion1, a, b, dx, dy):
                    try:
                        Funcion1 = parse_expr(Funcion1)
                        if dx == True:
                            for i in range(a,b+1):
                                F = Funcion.subs(x,i)
                                F1 = Funcion1.subs(x,i)
                            if F > F1:
                                c = Funcion**2 - Funcion1**2
                            if F < F1:
                                c = Funcion1**2 - Funcion**2
                            print(spp.integrate(c, (x, a, b)))
                        elif dy == True:
                            for i in range(a,b+1):
                                F = Funcion.subs(y,i)
                                F1 = Funcion1.subs(y,i)
                            if F > F1:
                                c = Funcion**2 - Funcion1**2
                            if F < F1:
                                c = Funcion1**2 - Funcion**2
                            print(spp.integrate(c, (y, a, b)))
                    except Exception:
                        print("No es posible calcular el volumen.")
                interact(s, Funcion1='', a=widgets.IntText(value=1), b=widgets.IntText(value=1), dx = False, dy = False)
            except:
                pass
    except:
        pass

def calculo_vectorial(Tema, Funcion):
    button1, button2 = widgets.Button(description="Dos Dimensiones"), widgets.Button(description="Tres Dimensiones")
    output1, output2 = widgets.Output(), widgets.Output()
    display(button1,output1); display(button2,output2)
    def DosD(dosd) :                            # 2D
        with output1:
            try:
                plot(Funcion,(x,-20,20), (y,-40,40))
            except:
                print("No es posible gráficar en R2")
    def TresD(tresd) :                           # 3D
        with output2:
            try:
                plot3d(Funcion,(x,-20,20),(y,-40,40), (z,-20,20))
            except:
                print("No es posible gráficar en R3")
    button1.on_click(DosD); button2.on_click(TresD)

    try:
        if Tema == 'Derivadas parciales':     # Tema: Derivadas Parciales
            try:
                def g(dfx,dfy,dfz) :
                    if dfx == True :
                        try:
                            dfx = spp.diff(Funcion,x)
                            plot3d(dfx,(x,-5,5),(y,-5,5), title='Derivada parcial con respecto a x',xlabel='eje x',ylabel='eje y')
                            print(dfx)
                        except:
                            pass
                    if dfy == True :
                        try:
                            dfy = spp.diff(Funcion,y)
                            plot3d(dfy,(x,-5,5),(y,-5,5), title='Derivada parcial con respecto a y',xlabel='eje x',ylabel='eje y')
                            print(dfy)
                        except:
                            pass
                    if dfz == True :
                        try:
                            dfz = spp.diff(Funcion,z)
                            plot3d(dfz,(z,-5,5),(y,-5,5), title='Derivada parcial con respecto a z',xlabel='eje x',ylabel='eje y')
                            print(dfz)
                        except:
                            pass
                    if dfx== True and dfy ==True and dfz==True:
                        plot3d(dfx,(x,-5,5),(y,-5,5),(z,-5,5)) and plot3d(dfy,(x,-5,5),(y,-5,5),(z,-5,5)) and plot3d(dfz,(z,-5,5),(y,-5,5),(x,-5,5))
                    return
                interact(g, dfx = False, dfy = False, dfz = False)
            except:
                pass

        if Tema == 'Integrales Dobles' :                                # Tema: Integrales Dobles
            try:
                def h(Tipo) :
                    if Tipo == 'Indefinida':                                                 #Indefinida
                        try:
                            intind=spp.integrate(Funcion, (x), (y))
                            print(intind)
                            plot3d(intind, (x,-5,5),(y,-5,5),title='Integral doble indefinida')
                        except:
                              pass
                    if Tipo == 'Definida':                                                # Definida
                        try:
                            def i(a, b, c,d) :
                                print(spp.integrate(Funcion, (x, a, b), (y, c, d)))
                            interact(i, a='', b='', c='', d='')
                        except:
                            pass
                interact(h,Tipo = ['Indefinida','Definida'])
            except:
                pass

        if Tema == 'Integrales Triples' :                                 # Tema: Integrales Triples
            try:
                def h2(Tipo) :
                    if Tipo == 'Indefinida':                                                # Indefinida
                        try:
                            Indefinida = spp.integrate(Funcion, (x), (y), (z))
                            print(Indefinida)
                        except:
                            pass
                    if Tipo == 'Definida':                                                # Definida
                        try:
                            def i2(liminf_x, limsup_x, liminf_y, limsup_y, liminf_z, limsup_z) :
                                print(spp.integrate(Funcion, (x, liminf_x, limsup_x), (y, liminf_y, limsup_y), (z, liminf_z, limsup_z)))
                            interact(i2, liminf_x='', limsup_x='', liminf_y='', limsup_y='', liminf_z='', limsup_z='')
                        except:
                            pass
                interact(h2, Tipo = ['Indefinida','Definida'])
            except:
                pass
    except:
        pass
```

```python
tab_contents = ['Pre-Álgebra','Álgebra lineal', 'Cálculo diferencial', 'Cálculo integral','Cálculo vectorial']
funcion=[interactive(pre_algebra, Tema = ['Operaciones Básicas','Factores y Números primos','Logaritmos, Radicales y Exponenciales'], Funcion=widgets.Text(description="Función: ", value="")),
         interactive(algebra_lineal, Tema = ['Operaciones entre matrices','Operaciones entre vectores','Sistema de ecuaciones lineales']),
         interactive(calculo_diferencial, Tema = ['Derivada', 'Límites', 'Raíces de la función', 'Máximos y mínimos'], Funcion=widgets.Text(description="Función: ", value="")),
         interactive(calculo_integral, Tema = ['Integrales de una variable','Sumatoria', 'Área bajo una curva', 'Volumen bajo una curva', 'Área entre dos curvas', 'Volumen entre dos curvas'], Funcion=widgets.Text(description="Función: ", value="")),
         interactive(calculo_vectorial, Tema = ['Derivadas parciales','Integrales Dobles', 'Integrales Triples'], Funcion=widgets.Text(description="Función: ", value=""))]

tab = widgets.Tab()
tab.children = [widgets.VBox(children = i.children) for i in funcion]
for i in range(len(tab_contents)):
    tab.set_title(i,tab_contents[i])
display(tab)
```

    Tab(children=(VBox(children=(Dropdown(description='Tema', options=('Operaciones Básicas', 'Factores y Números …


# Naive Bayes

author: Jacob Schreiber

contact: jmschreiber91@gmail.com

Naive Bayes refers to a method based off of Bayes' Rule for determining which model fits better. Bayes Rule is the following:

\begin{equation}
P(M|D)P(D) = P(D|M)P(M)
\end{equation}

where M stands for the model and D stands for the data. This equation has several important components. $P(M)$ is known as the prior, because it is your prior belief that a specific model fits the data the best before you see any data. $P(D|M)$ is the likelihood, or the probability of the data under a given model. Lastly, $P(M|D)$ is the posterior, which is the probability of the model being the one which generated the data after observing the data. This equation forms the basis of Bayesian learning, which differs primarily because instead of using the likelihood directly, uses the posterior, which is the likelihood regularized by the prior. In this way we can inject our belief about a situation before observing any data.

This is frequently rewritten in the following way:

\begin{equation}
P(M|D) = \frac{P(D|M)P(M)}{P(D)}
\end{equation}

But what is the probability of the data? It doesn't seem intuitive to calculate this value, and so it's calculated in a bit of a roundabout way:

\begin{equation}
P(D) = \sum\limits_{M \in models} P(D|M)P(M)
\end{equation}

This is known as the partition function and is the computationally intractable part of many Bayesian learning methods because of this sum. Taking the log of this part may turn multiplications into additions, but the overall sum remains and cannot be simplified out.

Naive Bayes is the simplest of the methods which takes advantage of this rule. Basically, you may have several models which you believe generate different types of data, and you want to then go backwards and figure out which model generated a particular sample.

Lets take a look using simple distributions.

```python
from pomegranate import *
import seaborn
%pylab inline
seaborn.set_style('whitegrid')
numpy.set_printoptions(suppress=True)
```

    Populating the interactive namespace from numpy and matplotlib

    WARNING: pylab import has clobbered these variables: ['log', 'random']
    `%matplotlib` prevents importing * from pylab and numpy

```python
X = numpy.concatenate((numpy.random.randn(200) + 5, numpy.random.randn(1000) + 9))
y = numpy.concatenate((numpy.zeros(200), numpy.ones(1000)))
i = numpy.arange(1200)
numpy.random.shuffle(i)
X = X[i]
y = y[i]

plt.figure( figsize=(16, 5))
plt.hist(X[y==0], bins=25, color='m', edgecolor='m')
plt.hist(X[y==1], bins=25, color='c', edgecolor='c')
plt.show()
```

Naive Bayes can act as a supervised version of Gaussian Mixture Models. A GMM could be applied to the unlabelled data similarly to the previous GMM notebook. However, if we have known labels, we don't need to use the EM algorithm. We can just give the components and fit them directly using MLE.

This can be done either by passing in a constructor of a distribution or a list of the actual models. Lets see the constructor method first.

```python
clf = NaiveBayes( NormalDistribution )
clf.fit( X, y )
clf.predict_proba( X[:10] )
```

    array([[ 0.00000109,  0.99999891],
           [ 0.00010142,  0.99989858],
           [ 0.00004986,  0.99995014],
           [ 0.00000052,  0.99999948],
           [ 0.00000392,  0.99999608],
           [ 0.00000002,  0.99999998],
           [ 0.00000127,  0.99999873],
           [ 0.00000004,  0.99999996],
           [ 0.00012191,  0.99987809],
           [ 0.98133653,  0.01866347]])

Using this method we can pass in other arbitrary distributions and fit them. If we want to fit Poisson distributions instead of a Normal, all we have to do is pass in the Poisson constructor.

```python
clf = NaiveBayes( PoissonDistribution )
clf.fit( X, y )
clf.predict_proba( X[:10] )
```

    array([[ 0.03717218,  0.96282782],
           [ 0.06406046,  0.93593954],
           [ 0.05886067,  0.94113933],
           [ 0.0340015 ,  0.9659985 ],
           [ 0.04337448,  0.95662552],
           [ 0.02212743,  0.97787257],
           [ 0.03783927,  0.96216073],
           [ 0.02502925,  0.97497075],
           [ 0.06547776,  0.93452224],
           [ 0.27326096,  0.72673904]])

We can also pass in known distributions as a way to initialize it. We can either pass in a weight vector corresponding to the prior probability of each model, or assume they are equally likely and not pass in anything.

```python
clf = NaiveBayes([NormalDistribution(5, 1), NormalDistribution(9,1)])
clf.predict_proba( X[:10] )
```

    array([[ 0.00001824,  0.99998176],
           [ 0.00097656,  0.99902344],
           [ 0.00052185,  0.99947815],
           [ 0.00000959,  0.99999041],
           [ 0.00005579,  0.99994421],
           [ 0.00000044,  0.99999956],
           [ 0.00002075,  0.99997925],
           [ 0.00000107,  0.99999893],
           [ 0.00114889,  0.99885111],
           [ 0.99271239,  0.00728761]])

We can also fit to the data to refine the model.

```python
clf.fit( X, y )
clf.predict_proba( X[:10] )
```

    array([[ 0.00000109,  0.99999891],
           [ 0.00010142,  0.99989858],
           [ 0.00004986,  0.99995014],
           [ 0.00000052,  0.99999948],
           [ 0.00000392,  0.99999608],
           [ 0.00000002,  0.99999998],
           [ 0.00000127,  0.99999873],
           [ 0.00000004,  0.99999996],
           [ 0.00012191,  0.99987809],
           [ 0.98133653,  0.01866347]])

We're getting back what we had initially. When we fit we're overriding the distributions with the MLE estimates from the data.

However, since Naive Bayes can theoretically use any model which has a fit method and a predict probability method, so can you in pomegranate! Bayesian Networks, hidden Markov models, markov chains, and distributions can all be used. For example, we can compare a 0th order Markov chain to a 1st order Markov chain to see how much better one fits than the other.

```python
a = MarkovChain([DiscreteDistribution({'A': 0.10, 'C': 0.40, 'G': 0.40, 'T': 0.10})])
b = MarkovChain([DiscreteDistribution({'A': 0.10, 'C': 0.40, 'G': 0.40, 'T': 0.10}),
                 ConditionalProbabilityTable([['A', 'A', 0.10],
                                              ['A', 'C', 0.50],
                                              ['A', 'G', 0.30],
                                              ['A', 'T', 0.10],
                                              ['C', 'A', 0.10],
                                              ['C', 'C', 0.40],
                                              ['C', 'T', 0.40],
                                              ['C', 'G', 0.10],
                                              ['G', 'A', 0.05],
                                              ['G', 'C', 0.45],
                                              ['G', 'G', 0.45],
                                              ['G', 'T', 0.05],
                                              ['T', 'A', 0.20],
                                              ['T', 'C', 0.30],
                                              ['T', 'G', 0.30],
                                              ['T', 'T', 0.20]])])

clf = NaiveBayes([a, b])
seqs = numpy.array(map(list, ['ACGATCGAAT', 'GAATTATGC', 'GAAATAAATAC']))

clf.predict_proba(seqs)
```

    array([[ 0.98556304,  0.01443696],
           [ 0.37209302,  0.62790698],
           [ 0.28571429,  0.71428571]])

We can even compare a markov chain to a hidden Markov model. For this, lets look at the CG island detection problem. In essence, there are stretches of DNA which are highly enriched for the nucleotides C and G. However, they do include the nucleotides A and T, and so simply calling all C's and G's part of the islands like a GMM would is not useful. A HMM can model this through its underlying structure.Lets use a markov chain with a preference for staying in the CG states versus a hidden Markov model which is a basic CG-island finder.

```python
d1 = DiscreteDistribution({'A': 0.25, 'C': 0.25, 'G': 0.25, 'T': 0.25})
s1 = State(d1, name="s1")
s2 = State(DiscreteDistribution({'A': 0.10, 'C': 0.40, 'G': 0.40, 'T': 0.10}), name="s2")
s3 = State(d1, name="s3")

c = HiddenMarkovModel('GC Island')
c.add_states(s1, s2, s3)
c.add_transition( c.start, s1, 0.9 )
c.add_transition( c.start, s3, 0.1 )
c.add_transition( s1, s1, 0.80 )
c.add_transition( s1, s2, 0.20 )
c.add_transition( s2, s2, 0.75 )
c.add_transition( s2, s3, 0.25 )
c.add_transition( s3, s3, 0.80 )
c.add_transition( s3, c.end, 0.20 )
c.bake()

plt.figure( figsize=(4, 8) )
c.draw()
```

```python
clf = NaiveBayes([b, c])
seqs = numpy.array(map(list, ['ACGATCGCGAGTGCCATTAT', 'GAATTACTAGATATGC', 'GAAACGGCGAGCATGGCTAAATAC']))

clf.predict_proba(seqs)
```

    array([[ 0.00208704,  0.99791296],
           [ 0.37714684,  0.62285316],
           [ 0.10567194,  0.89432806]])

We can see that the HMM performs better in all three cases, but in one only slightly better. This could be because the markov chain values were initialized without much thought and there is a much better way to do it. Or it could be because HMMs are a much better tool to do this type of prediction.

Naive Bayes models have the same methods as the other models, aka fit, predict, summarize, from_summaries, predict_proba, and predict_log_proba. Lets look at it summarizing batches of 500 sequences for two Markov chains. Obviously this is a very simple example, but it's illustrative of the flexibility of the API. These sequences can be of any length, but we randomly generate a matrix for brevity.

```python
a = MarkovChain([DiscreteDistribution({'A': 0.10, 'C': 0.40, 'G': 0.40, 'T': 0.10})])
b = MarkovChain([DiscreteDistribution({'A': 0.10, 'C': 0.40, 'G': 0.40, 'T': 0.10}),
                 ConditionalProbabilityTable([['A', 'A', 0.10],
                                              ['A', 'C', 0.50],
                                              ['A', 'G', 0.30],
                                              ['A', 'T', 0.10],
                                              ['C', 'A', 0.10],
                                              ['C', 'C', 0.40],
                                              ['C', 'T', 0.40],
                                              ['C', 'G', 0.10],
                                              ['G', 'A', 0.05],
                                              ['G', 'C', 0.45],
                                              ['G', 'G', 0.45],
                                              ['G', 'T', 0.05],
                                              ['T', 'A', 0.20],
                                              ['T', 'C', 0.30],
                                              ['T', 'G', 0.30],
                                              ['T', 'T', 0.20]])])

clf = NaiveBayes([a, b])

dna = 'ACGT'
for i in range(100):
    seqs = numpy.random.randint(0, 4, (500, 50))
    seqs = numpy.array([[dna[i] for i in seq] for seq in seqs])
    y = numpy.concatenate((numpy.ones(100), numpy.zeros(400)))
    clf.summarize(seqs, y)

clf.from_summaries()
```

We can check the underlying distributions to see if they were properly updated.

```python
print "Chain 1 Distribution 1"
print clf.models[0].distributions[0]
print
print "Chain 2 Distribution 1"
print clf.models[1].distributions[0]
print
print "Chain 2 Distribution 2"
print clf.models[1].distributions[1]
print
print "Priors"
print clf.weights
```

    Chain 1 Distribution 1
    {
        "frozen" :false,
        "class" :"Distribution",
        "parameters" :[
            {
                "A" :0.2494635,
                "C" :0.25015850000000001,
                "T" :0.25011800000000001,
                "G" :0.25025999999999998
            }
        ],
        "name" :"DiscreteDistribution"
    }

    Chain 2 Distribution 1
    {
        "frozen" :false,
        "class" :"Distribution",
        "parameters" :[
            {
                "A" :0.24709999999999999,
                "C" :0.2596,
                "T" :0.2414,
                "G" :0.25190000000000001
            }
        ],
        "name" :"DiscreteDistribution"
    }

    Chain 2 Distribution 2
    T	T	0.250791676554
    T	G	0.250047050545
    T	C	0.249793386739
    T	A	0.249367886162
    G	T	0.24863894217
    G	G	0.250973350202
    G	C	0.251324327633
    G	A	0.249063379994
    C	G	0.251796687421
    C	T	0.24950148537
    C	C	0.249932853131
    C	A	0.248768974077
    A	T	0.24965893588
    A	G	0.247371559745
    A	C	0.251251929974
    A	A	0.251717574401

    Priors
    [ 0.8  0.2]

We can see that they've all normalized to basically a uniform distribution, which makes sense because that's the distribution that the data was generated from.