# Unsupervised Learning
Clustering and data mining

# Clustering
Following [*Python Machine Learning 3rd Edition, Chapter 11*](https://github.com/rasbt/python-machine-learning-book-3rd-edition/tree/master/ch11) and [*Hands-on Machine Learning with Scikit-Learn, Keras & TensorFlow, Chapter 9*](https://github.com/ageron/handson-ml2/blob/master/09_unsupervised_learning.ipynb).

[Image source Machine Learning with Python 3rd Edition](https://github.com/rasbt/python-machine-learning-book-3rd-edition)

- We need clustering when we don't have class labels, making this an unsupervised problem.
- When we need to discover a latent or hidden structure. We may have a hypothesis on what this structure might be in logical terms, but in reality most of the algorithms are going to be looking at maximizing distance between centroids.
- Clusters should contain examples that are more similar to each other than in different clusters.

## When might we need to use clustering?
- Determine documents that are similar to each other.
- Group songs when we don't have information on their genre.
- Generation tasks, where should this new sample belong?
- Customer segmentation for marketing campaigns.
- Determine outliers in higher dimension datasets.
- Need to flag high-risk credit card transactions.

## General practices
- Most clustering algorithms use `distance` in some manner. Therefore, it is a best practice to `scale` your data prior to using these algorithms. Since distance will be influenced by absolute magnitudes, it is crucial you don't skip this step (and you shouldn't be skipping it for other modeling tasks either!).
- To visually inspect the clustering, a standard practice is to use dimensionality reduction (e.g., `principle component analysis`) to remove some of the background noise in the data.
- Another standard practice is to use dimensionality reduction to compress the feature space to 2 or 3 dimensions in order to visualize the separation in the clusters - though the preceived distance in 2 dimensions may not be reflect of the absolute or relative difference in the full $d$-dimensional space.
- We'll need a subject matter expert (SME) to review our results and determine the purity of the clusters. If you had some labeled examples you could get a sense of purity using those, but in many cases you'll need some type of feedback loop with reviewers.

## Primary types of Clustering
- Prototype-based (e.g., k-means) clustering.
- Hierarchical (agglomerative) clustering.
- Density-based clustering.
- Graph-based clustering.

# k-Means Clustering
- Type of `prototype-based` clustering - it needs a prototype or model for each cluster that is usually a centroid (mean/center) or medoid (median).
- Identifies spherical shapes well.
- Every example gets assigned a cluster, even if it is a true outlier.
- Biggest downside is you need to tell the algorithm how many clusters there are...
- A bad decision for the number of clusters ($k$) and our performance will be poor.

Demonstrate on the `iris` data:

```python
from sklearn.datasets import load_iris

iris = load_iris()
iris_df = iris.data
iris_label = iris.target

iris_df[:5, :]
```

    array([[5.1, 3.5, 1.4, 0.2],
           [4.9, 3. , 1.4, 0.2],
           [4.7, 3.2, 1.3, 0.2],
           [4.6, 3.1, 1.5, 0.2],
           [5. , 3.6, 1.4, 0.2]])

#### Squeeze to 2-dimensions for visualizing
We'll use PCA to reduce the dataset to 2-dimensions.
- Scale the data.
- Apply PCA to reduce to 2-dimensions.
- Plot the data

```python
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.pipeline import Pipeline

iris_p = Pipeline([('scale', StandardScaler()),
                   ('pca', PCA(n_components=2))
                  ])

iris_2d = iris_p.fit_transform(iris_df)

import matplotlib.pyplot as plt
%matplotlib inline

plt.plot(iris_2d[:,0], iris_2d[:,1], 'ro', alpha=0.5)
plt.title('iris in 2-dimensions', loc='left')
plt.show()
```

> We know there are 3 classes of `iris` but if we didn't have that info, we would probably assume there are 2 based on this plot.

## Basic Process with k-means
- Pick $k$.
- Pick $k$ centroids randomly and assign as the initial centers.
- Assign each example to the nearest cluster, based on `distance`.
- Move the centroids to the center of the examples that were assigned to it.
- Repeat until the centroids stop changing, either based on a tolerance or iteration threshold.

### Distance
Usually this will be `squared Euclidean distance`, where $j$ is the $j$th dimension:

$$
d(x,y)^2 = \sum_{j=1}^{m}(x_j - y_j)^2 = ||x-y||_2^2
$$

### Minimizing within-cluster errors
Going to minimize the within-cluster errors using the `sum of squared errors`, leveraging `Euclidean distance`.

$$
SSE = \sum_{i=1}^{n}{\sum_{j=1}^{k}}w^{(i,j)}||x^{(i)}-\mu^{(j)}||_2^2
$$

$\mu^{j}$ is the $j$th centroid, $i$ is for the $i$-th example, $w^{(i,j)}$ is an indicator for belonging to the cluster:

$$
\begin{equation}
w^{(i,j)}
=
\begin{cases}
1, & \text{if}\ x^{(i)}\in j \\
0, & \text{otherwise}
\end{cases}
\end{equation}
$$

```python
from sklearn.cluster import KMeans

km = KMeans(n_clusters = 2, init='random')
km = km.fit(iris_2d)
iris_cluster = km.predict(iris_2d)

import matplotlib.pyplot as plt
%matplotlib inline

plt.scatter(iris_2d[iris_cluster == 0, 0],
            iris_2d[iris_cluster == 0, 1], color='blue',
            alpha=0.5,
            label='Cluster 0'
           )

plt.scatter(iris_2d[iris_cluster == 1, 0],
            iris_2d[iris_cluster == 1, 1], color='orange',
            alpha=0.5,
            label='Cluster 1'
           )

plt.title('iris in 2-dimensions', loc='left')
plt.legend()

plt.show()
```

- Looked like it did a reasonable job.
- This was easy since there was so much separation between clusters.

### Now what?
- Might be useful to join back to the original data and see if there are commonalities among some of the features.
- Pass it on to a SME to review.

# k-Means++
- Ordinary k-Means randomly sets the initial centroids, which could be very poor.
- Could lead to bad performance and/or slow convergence.
- Similar to cross-validation, we can run k-Means multiple times and pick the best in terms of the `sum of squared errors`.
- This has been proven to be more consistent and have better general performance in literature.

### Basic Process:
- Create a list, $M$, to store the $k$ centroids.
- Randomly choose first centroid, $\mu^{(j)}$, from the examples and add to $M$.
- For examples not in $M$, find the minimum squared distance to the centroids in $M$.
- Randomly select the next centroid, $\mu^{(p)}$ using a weighted sampling distribution and repeat above. The sampling distribution ensures instances farther away from existing centroids are more likely to be new centroids.
- Run normal k-means.

To use `init=k-means++` in the algorithm, which is actually the default.

You can use specify default values for the initial centroids, but that would be probably pretty rare that you would have a guess for those, especially with higher dimension data.

# Another k-Means example
This will use a generated blob dataset.

```python
from sklearn.datasets import make_blobs

X, y = make_blobs(n_samples=2000,
                  n_features=2,
                  centers=10,
                  cluster_std=2,
                  shuffle=True,
                  random_state=0
                 )

plt.scatter(X[:,0], X[:,1], alpha=.25)
plt.show()
```

# What should be set for $k$?
We have two primary tools to determine a reasonable number to set $k$:
- Elbow plot.
- Silhouette plot.

### Elbow plot
- Think of this as a type of grid search for clustering.
- We don't have evaluation metrics, e.g., recall, that we can look to maximize, so we need to use metrics intrinsic to the clusters and data.
- We use the within-cluster SSE (`inertia` or `distortion`) and look to see that reaches a convergence point. inertia_ is available after fitting a model.

```python
from sklearn.cluster import KMeans

distortions = list()

for i in range(1, 15):
    km = KMeans(n_clusters=i, random_state=0)
    km = km.fit(X)
    distortions.append(km.inertia_)

plt.plot(range(1, 15), distortions, marker='o')
plt.xlabel('Number of Clusters')
plt.ylabel('Inertia')
plt.show()
```

> Plot `elbows` or `kinks` at around $6$ or $7$, so we'll use that for $k$.

```python
from sklearn.cluster import KMeans

n = 7

km = KMeans(n_clusters=n, random_state=0)
km = km.fit(X)
pclusters = km.predict(X)

for i in range(n):
    plt.scatter(X[pclusters==i, 0], X[pclusters==i, 1], alpha=0.25)
plt.show()
```

> Elbow method seemed to pick a reasonable number?

## Silhouette Plot
Another method for chosing $k$. It's a measure of how tightly grouped the estimates within clusters.

#### Steps
- Calculate cluster cohesion, $a^{(i)}$, which is the average distance between a sample and all other samples in the cluster. The smaller, the tighter there are together.
- Calcuate the cluster separation, $b^{(i)}$, which is the average distance between each sample in a cluster versus the examples in the nearest cluster.
- Calculate the silhouette, which is the distance between the cluster cohesion and the separation, divided by the greater of the two.

$$
s^{(i)} = \frac{b^{(i)}-a^{(i)}}{max(b^{(i)}, a^{(i)})}
$$

Where $s\in(-1,1)$
- A value of $1$ indicates an ideal cluster, since the clusters are dissimilar from each other.
- A value of $0$ indicates the within and between separation is the same, so the clusters are similar to each other (we don't want this).

```python
from sklearn.metrics import silhouette_score
from sklearn.cluster import KMeans

silhouettes = list()

for i in range(2, 15):
    km = KMeans(n_clusters=i, random_state=0)
    km = km.fit(X)
    s = silhouette_score(X, km.labels_)
    silhouettes.append(s)

plt.plot(range(2,15), silhouettes, marker='o')
plt.xlabel('Number of Clusters')
plt.ylabel('Silhouette Score')
plt.show()
```

> Probably would ignore $2$ since it probably is separating the top and bottom blob. It materially begins to decline after $6$, so we'd likely use that for $k$.

```python
from sklearn.metrics import silhouette_samples

from sklearn.cluster import KMeans

n = 7

km = KMeans(n_clusters=n, random_state=0)
km = km.fit(X)
pclusters = km.predict(X)

silhouette_vals = silhouette_samples(X, km.labels_)
```

```python
import pandas as pd

s_vals = pd.DataFrame(list(zip(silhouette_vals, km.labels_)))
s_vals.columns = ['silhouette', 'cluster']
s_vals = s_vals.sort_values(by=['cluster', 'silhouette'],
                          ascending=[True, False]
                          )
s_vals['idx'] = pd.Series(range(0,2000))
s_vals['cluster'] = s_vals['cluster'].astype(str)

s_vals.groupby('cluster')['silhouette'].plot.density()
plt.legend()
plt.xlim((0,1))
plt.show()
```

> Plot the individual values by cluster to identify potential outliers.

- Data points near $0$ are less like the examples in the cluster.
- Heavy density near $1$ indicates the cluster is grouping similar examples and is dissimilar from other clusters.

You may also see silhouette plots like this:

[Image source: Hands-on Machine Learning with Scikit-Learn, Keras & TensorFlow, Chapter 9](https://github.com/ageron/handson-ml2/blob/master/09_unsupervised_learning.ipynb)

### You can also use clustering for preprocessing
Since this can be used to generate a single data point per example from many, this could be used for dimension reduction in addition to segmentation.

Why it may help?
- Noise reduction.
- Dimensionality reduction.

```python
from sklearn.datasets import load_digits

digits = load_digits()
dX, dy = digits['data'], digits['target']

from sklearn.model_selection import train_test_split
dX_train, dX_test, dy_train, dy_test = train_test_split(dX,
                                                        dy,
                                                        test_size=0.2,
                                                        random_state=100
                                                       )

from sklearn.linear_model import LogisticRegression
log_reg = LogisticRegression(solver='liblinear')
log_reg = log_reg.fit(dX_train, dy_train)

log_reg_score = log_reg.score(dX_test, dy_test)
print(f'Score: {log_reg_score:.2%}')
```

    Score: 95.83%

```python
dX.shape
```

    (1797, 64)

#### Now we'll see using the clusters from k-Means

```python
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

p = Pipeline([('sc', StandardScaler()),
             ('kmeans', KMeans(n_clusters=100)),
              ('logreg', LogisticRegression(solver='liblinear'))])

p = p.fit(dX_train, dy_train)
p_score = p.score(dX_test, dy_test)
print(f'Using k-means preprocesser: {p_score:.2%}')
```

    Using k-means preprocesser: 97.22%

- You could use GridSearchCV to find the best number of clusters.
- We picked $100$ based on literature on this dataset, but you would want to search for the best values.

# Another Application: Semi-Supervised Learning
Let's pretend we only had 50 labeled images for the digits dataset.
- Something like this is somewhat common, especially with text or images.

```python
n_labeled = 50

log_reg = LogisticRegression(solver='liblinear')
log_reg.fit(dX_train[:n_labeled], dy_train[:n_labeled])

small_n_score = log_reg.score(dX_test, dy_test)
print(f'Accuracy with only 50 training examples: {small_n_score:.2%}')
```

    Accuracy with only 50 training examples: 80.00%

> Accuracy much lower than with the full training dataset.

#### What if we create labels from the clusters?
We label other instances that are really close to centroids after running k-Means?

First, we'll need to determine clusters, then determine which labeled instance is closest to the centroid.

[Taken from *Hands on Machine Learning with Scikit-Learn, Keras & TensorFlow pages 253-254*](https://github.com/ageron/handson-ml2/blob/master/09_unsupervised_learning.ipynb)

```python
import numpy as np

k = 50

kmeans = KMeans(n_clusters = k)
X_digits_dist = kmeans.fit_transform(dX_train)

# digit with smallest distance
representative_digit_idx = np.argmin(X_digits_dist, axis=0)
X_representative_digits = dX_train[representative_digit_idx]
```

## These will be images that are nearest to the cluster centers

```python
plt.figure(figsize=(9, 4))
for index, X_representative_digit in enumerate(X_representative_digits):
    plt.subplot(k // 10, 10, index + 1)
    plt.imshow(X_representative_digit.reshape(8, 8), cmap="binary", interpolation="bilinear")
    plt.axis('off')

plt.show()
```

> If we didn't have the labels, we would need to manually look at these and label these manually. Since we have the labels, we'll skip that manual exercise, but be aware it would be necessary.

```python
y_representative_digits = dy_train[representative_digit_idx]

log_reg = LogisticRegression(solver='liblinear')
log_reg = log_reg.fit(X_representative_digits, y_representative_digits)

new_scr = log_reg.score(dX_test, dy_test)
print(f'Accuracy with only 50 representative training examples: {new_scr:.2%}')
```

    Accuracy with only 50 representative training examples: 91.67%

#### Significant improvement in accuracy! Can we do better?
Now we'll iteratively create labels all the instances in the cluster and the closest digit.

```python
y_train_propagated = np.empty(len(dX_train), dtype=np.int32)

for i in range(k):
    y_train_propagated[kmeans.labels_ == i] = y_representative_digits[i]

log_reg = LogisticRegression(solver='liblinear')
log_reg = log_reg.fit(dX_train, y_train_propagated)

new_scr = log_reg.score(dX_test, dy_test)
print(f'Accuracy with propagation: {new_scr:.2%}')
```

    Accuracy with propagation: 91.94%

> Another performance boost!

#### What if we only took the closest examples?
We'll try only propagating to the examples in the closest 20th percentile.

```python
percentile_closest = 20

X_cluster_dist = X_digits_dist[np.arange(len(dX_train)), kmeans.labels_]
for i in range(k):
    in_cluster = (kmeans.labels_ == i)
    cluster_dist = X_cluster_dist[in_cluster]
    cutoff_distance = np.percentile(cluster_dist, percentile_closest)
    above_cutoff = (X_cluster_dist > cutoff_distance)
    X_cluster_dist[in_cluster & above_cutoff] = -1

partially_propagated = (X_cluster_dist != -1)
X_train_partially_propagated = dX_train[partially_propagated]
y_train_partially_propagated = y_train_propagated[partially_propagated]

log_reg = LogisticRegression(solver='liblinear')
log_reg.fit(X_train_partially_propagated, y_train_partially_propagated)

new_scr = log_reg.score(dX_test, dy_test)
print(f'Accuracy with propagation: {new_scr:.2%}')
```

    Accuracy with propagation: 90.56%

# Agglomerative (hierarchical) clustering
Another clustering method builds the clusters as parts of a tree and let's us visualize it via a `dendrogram`.

This can be a useful technique if you expect some type of structure, e.g., you are trying to cluster on recipes. You would expect (hope) the french recipes get grouped together, the Chinese recipes grouped together, ... and then the various variations within those grouped in the terminal nodes.

### Basic Steps
- Compute a distance matrix for the examples (this could be very large).
- Each example starts as its own cluster.
- Merge the two closest clusters based on distance.
- Update the similarity (linkage) matrix.
- Repeat until one cluster remains.

This is a bottom-up mode, since we start with the individual examples and slowly group them into more and more heterogenous clusters until we are back to the entire dataset.

- You can use Euclidean distance or a custom method of your choosing.
- Since this is based on distance; you'll want to standardize your features first!
- Linkage is how the clusters are compared for similarity to one another:

[Image source Machine Learning with Python 3rd Edition Page 368](https://github.com/rasbt/python-machine-learning-book-3rd-edition/tree/master/ch11)

### Hierarchial Clustering with the Wine Data

```python
import pandas as pd

wine = pd.read_csv('data/wine.data', header=None)

wineColumns = ['Type',
'Alcohol',
'Malic acid',
'Ash',
'Alcalinity of ash',
'Magnesium',
'Total phenols',
'Flavanoids',
'Nonflavanoid phenols',
'Proanthocyanins',
'Color intensity',
'Hue',
'OD280/OD315 of diluted wines',
'Proline'
]

wine.columns = wineColumns

wine.head()
```

      Type
      Alcohol
      Malic acid
      Ash
      Alcalinity of ash
      Magnesium
      Total phenols
      Flavanoids
      Nonflavanoid phenols
      Proanthocyanins
      Color intensity
      Hue
      OD280/OD315 of diluted wines
      Proline

      0
      1
      14.23
      1.71
      2.43
      15.6
      127
      2.80
      3.06
      0.28
      2.29
      5.64
      1.04
      3.92
      1065

      1
      1
      13.20
      1.78
      2.14
      11.2
      100
      2.65
      2.76
      0.26
      1.28
      4.38
      1.05
      3.40
      1050

      2
      1
      13.16
      2.36
      2.67
      18.6
      101
      2.80
      3.24
      0.30
      2.81
      5.68
      1.03
      3.17
      1185

      3
      1
      14.37
      1.95
      2.50
      16.8
      113
      3.85
      3.49
      0.24
      2.18
      7.80
      0.86
      3.45
      1480

      4
      1
      13.24
      2.59
      2.87
      21.0
      118
      2.80
      2.69
      0.39
      1.82
      4.32
      1.04
      2.93
      735

```python
wine.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 178 entries, 0 to 177
    Data columns (total 14 columns):
     #   Column                        Non-Null Count  Dtype
    ---  ------                        --------------  -----
     0   Type                          178 non-null    int64
     1   Alcohol                       178 non-null    float64
     2   Malic acid                    178 non-null    float64
     3   Ash                           178 non-null    float64
     4   Alcalinity of ash             178 non-null    float64
     5   Magnesium                     178 non-null    int64
     6   Total phenols                 178 non-null    float64
     7   Flavanoids                    178 non-null    float64
     8   Nonflavanoid phenols          178 non-null    float64
     9   Proanthocyanins               178 non-null    float64
     10  Color intensity               178 non-null    float64
     11  Hue                           178 non-null    float64
     12  OD280/OD315 of diluted wines  178 non-null    float64
     13  Proline                       178 non-null    int64
    dtypes: float64(11), int64(3)
    memory usage: 19.6 KB

#### Scale the data and run the algorithm

```python
from sklearn.cluster import AgglomerativeClustering
from sklearn.preprocessing import StandardScaler

# pretending we don't know the type
wine_X = wine.iloc[:, 1:]

wine_X = StandardScaler().fit_transform(wine_X)

ac = AgglomerativeClustering(n_clusters=5,
                             affinity='euclidean',
                             linkage='complete',
                             compute_distances=True
                            )

labels = ac.fit_predict(wine_X)

print('Clustering completed.')
```

    Clustering completed.

#### Now we can plot the dendrogram
[`plot_dengrogram` taken from scikit-learn example.](https://scikit-learn.org/stable/auto_examples/cluster/plot_agglomerative_dendrogram.html#sphx-glr-auto-examples-cluster-plot-agglomerative-dendrogram-py)

```python
import numpy as np
from scipy.cluster.hierarchy import dendrogram

def plot_dendrogram(model, **kwargs):
    # Create linkage matrix and then plot the dendrogram

    # create the counts of samples under each node
    counts = np.zeros(model.children_.shape[0])
    n_samples = len(model.labels_)
    for i, merge in enumerate(model.children_):
        current_count = 0
        for child_idx in merge:
            if child_idx < n_samples:
                current_count += 1  # leaf node
            else:
                current_count += counts[child_idx - n_samples]
        counts[i] = current_count

    linkage_matrix = np.column_stack([model.children_, model.distances_,
                                      counts]).astype(float)

    # Plot the corresponding dendrogram
    dendrogram(linkage_matrix, **kwargs)

plt.figure(figsize=(12,5))
plot_dendrogram(ac, truncate_mode='lastp')
plt.xlabel('Number of Examples {$Example ID$ or $(Examples)$}')
plt.ylabel('Distance')
plt.title('Dendrogram for Wine Agglomerative Clustering', loc='left')
plt.show()
```

- Can be useful in identifying outliers or seeing how close examples are to one another.
- Still need to review features to determine what the clusters represent.

# Density-based Spatial Clustering of Applications with Noise (DBSCAN)
- Density-based clustering - essentially identifying areas of continuous high density.
- No assumptions about whether the clusters are spherical or not.
- No assumptions about hierarchies.
- Can identity points that are outliers.

### Each Example is Assigned a Label
- Core point: if it's sufficiently close to neighbors defined by a minimum number (`MinPts`) falling within a radius, $\epsilon$. Points within $\epsilon$ are also considered its neighborhood.
- Border point: on the outside border of the cluster boundary, defined by being within the radius ($\epsilon$) of a core point but lacking the minimum number of neighbors.
- Noise point: think of these as outliers that aren't attached to any clusters.

### Basic Process
- Create clusters for each core point or connected group. They are considered connected if they are within $\epsilon$.
- Assign the border points.

[Image source Machine Learning with Python 3rd Edition Page 377](https://github.com/rasbt/python-machine-learning-book-3rd-edition/tree/master/ch11)

### Biggest Advantages
- Identifies noise/outliers.
- No assumptions on structure.

## Example with Half-Moons

```python
from sklearn.datasets import make_moons

X, y = make_moons(n_samples=300, noise=0.05, random_state=0)

plt.scatter(X[:,0], X[:,1])
plt.show()
```

### See how k-Means handles this data

```python
from sklearn.cluster import KMeans

km = KMeans(n_clusters=2)
km = km.fit(X)

pclusters = km.predict(X)

for i in range(n):
    plt.scatter(X[pclusters==i, 0], X[pclusters==i, 1], alpha=0.25)
plt.show()
```

> k-Means assumes spherical clusters, so this isn't going to work...

### Now with DBSCAN
> Note: there isn't a `.predict()` method for DBSCAN since it's meant to identify clusters based on the current dataset. If you needed predictions, you could use the outputs and use with KMeans. See pages 257-258 of *Hands-on Machine Learning with Scikit-Learn, Keras & TensorFlow*.

```python
from sklearn.cluster import DBSCAN

db = DBSCAN(eps=0.2, min_samples=5, metric='euclidean')
pclusters = db.fit_predict(X)
labs = db.labels_
ulabs = np.unique(labs)

for i in ulabs:
    plt.scatter(X[pclusters==i, 0], X[pclusters==i, 1], alpha=0.25)
plt.show()
```

- DBSCAN was able to identity the areas of continously density, and thus, the half moons.
- So for data of arbitrary shapes, this will probably perform better than k-Means or hierarchical clustering.

```python
from sklearn.cluster import DBSCAN

db = DBSCAN(eps=0.05, min_samples=5, metric='euclidean')
pclusters = db.fit_predict(X)
labs = db.labels_
ulabs = np.unique(labs)

plt.figure(figsize=(10,4))
for i in ulabs:
    plt.scatter(X[pclusters==i, 0], X[pclusters==i, 1], alpha=0.25)
plt.legend(ulabs)
plt.show()
```

> Setting the radius too small, and nearly everything could be an outlier ($-1$).

### Cons
- Vulnerable to curse of dimensionality (not unique, not something to be aware of).
- Need to determine $\epsilon$ and the minimum number of points.
- If the density if very uneven, finding reasonable choices for $\epsilon$ and the minimum points could be challenging.

# Additional Clustering Algorithms
- [Balance Iterative Reducing and Clustering using Hierarchies (BIRCH)](https://en.wikipedia.org/wiki/BIRCH): designed for very large datasets. Similar to k-means and traditionally has similar results.
- [Mean-Shift](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.MeanShift.html):  Starts with a circle around each example and computes mean of instances within that circle, then centers so the it is centered on the mean. Shifts to areas of higher density. Circles in similar places are assigned to the same cluster. Very computationally complicated and not suitable for bigger datasets.
- [Affinity](https://en.wikipedia.org/wiki/Affinity_propagation): Voting based.
- [Spectral clustering](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.SpectralClustering.html): Uses similarity matrix and simple embedding and then uses k-means (or another clustering algorithm).
- [Gaussian Mixtures](https://scikit-learn.org/stable/modules/mixture.html)

# Association Rules
- Also referred to as data mining or `market basket analysis`.
- Common algorithm is [`apriori`](https://en.wikipedia.org/wiki/Apriori_algorithm).
- If someone buys `toilet paper`, `water`, and `bread`; they probably also bought `_______`.
- Not the most efficient algorithm in the world.
- $3$ metrics to calculate:
    - $support(m)$ = $\frac{\text{# transactions containing m}}{\text{# transactions}}$

    - $confidence(m\rightarrow n)$ = $\frac{\text{# transactions containing m and n}}{\text{# transactions containing m}}$

    - $lift(m\rightarrow n)$ = $\frac{confidence_{m \rightarrow n}}{support_n}$

This is easily extended to looking at sets of 3, 4, ...

### Steps:
- Set minimum support and confidence.
- Take subsets that exceed minimum support.
- Take rules where confidence exceeds minimum confidence.
- Sort by decreasing lift.

Example with grocery transactions:

```python
import pandas as pd

pd.set_option('display.max_columns', 100)
txn = pd.read_csv('data/grocery.csv', header=None)
txn = txn.reset_index()
txn = txn.rename(columns={'index':'transactionID'})
txn.head()
```

      transactionID
      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31

      0
      0
      citrus fruit
      semi-finished bread
      margarine
      ready soups
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      1
      1
      tropical fruit
      yogurt
      coffee
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      2
      2
      whole milk
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      3
      3
      pip fruit
      yogurt
      cream cheese
      meat spreads
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      4
      4
      other vegetables
      whole milk
      condensed milk
      long life bakery product
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

```python
txn.shape
```

    (9835, 33)

```python
transactions = (
    txn.melt(id_vars='transactionID')
    .dropna()
    .groupby('transactionID')['value'].apply(lambda x: list(set(x)))
).tolist()

for i in range(5):
    print(f'\nTransaction {i}')
    print(*transactions[i])
```

    Transaction 0
    margarine ready soups semi-finished bread citrus fruit

    Transaction 1
    coffee tropical fruit yogurt

    Transaction 2
    whole milk

    Transaction 3
    meat spreads yogurt cream cheese pip fruit

    Transaction 4
    condensed milk other vegetables whole milk long life bakery product

```python
import pandas as pd
from mlxtend.preprocessing import TransactionEncoder

te = TransactionEncoder()
te_ary = te.fit(transactions).transform(transactions)
df = pd.DataFrame(te_ary, columns=te.columns_)
df.head()
```

      Instant food products
      UHT-milk
      abrasive cleaner
      artif. sweetener
      baby cosmetics
      baby food
      bags
      baking powder
      bathroom cleaner
      beef
      berries
      beverages
      bottled beer
      bottled water
      brandy
      brown bread
      butter
      butter milk
      cake bar
      candles
      candy
      canned beer
      canned fish
      canned fruit
      canned vegetables
      cat food
      cereals
      chewing gum
      chicken
      chocolate
      chocolate marshmallow
      citrus fruit
      cleaner
      cling film/bags
      cocoa drinks
      coffee
      condensed milk
      cooking chocolate
      cookware
      cream
      cream cheese
      curd
      curd cheese
      decalcifier
      dental care
      dessert
      detergent
      dish cleaner
      dishes
      dog food
      ...
      ready soups
      red/blush wine
      rice
      roll products
      rolls/buns
      root vegetables
      rubbing alcohol
      rum
      salad dressing
      salt
      salty snack
      sauces
      sausage
      seasonal products
      semi-finished bread
      shopping bags
      skin care
      sliced cheese
      snack products
      soap
      soda
      soft cheese
      softener
      sound storage medium
      soups
      sparkling wine
      specialty bar
      specialty cheese
      specialty chocolate
      specialty fat
      specialty vegetables
      spices
      spread cheese
      sugar
      sweet spreads
      syrup
      tea
      tidbits
      toilet cleaner
      tropical fruit
      turkey
      vinegar
      waffles
      whipped/sour cream
      whisky
      white bread
      white wine
      whole milk
      yogurt
      zwieback

      0
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      True
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      ...
      True
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      True
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False

      1
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      True
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      ...
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      True
      False
      False
      False
      False
      False
      False
      False
      False
      True
      False

      2
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      ...
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      True
      False
      False

      3
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      True
      False
      False
      False
      False
      False
      False
      False
      False
      False
      ...
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      True
      False

      4
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      True
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      ...
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      True
      False
      False

5 rows × 169 columns

```python
from mlxtend.frequent_patterns import apriori

results = apriori(df, min_support=0.02, use_colnames=True)
results['items'] = results['itemsets'].apply(lambda x: len(x))

results.sort_values(by=['items', 'support'], ascending=[False, False]).query('items>1')
```

      support
      itemsets
      items

      120
      0.023183
      (other vegetables, whole milk, root vegetables)
      3

      121
      0.022267
      (other vegetables, yogurt, whole milk)
      3

      91
      0.074835
      (other vegetables, whole milk)
      2

      103
      0.056634
      (whole milk, rolls/buns)
      2

      119
      0.056024
      (yogurt, whole milk)
      2

      ...
      ...
      ...
      ...

      75
      0.020539
      (whole milk, frankfurter)
      2

      60
      0.020437
      (bottled beer, whole milk)
      2

      76
      0.020437
      (frozen vegetables, whole milk)
      2

      96
      0.020437
      (tropical fruit, pip fruit)
      2

      67
      0.020031
      (other vegetables, butter)
      2

63 rows × 3 columns

> More information on [mlxtend and Apriori](http://rasbt.github.io/mlxtend/user_guide/frequent_patterns/apriori/)

# Non-negative Matrix Factorization
- Technique to decompose a matrix into the product of two matrices that approximate the original matrix.
- Can be used for recommendation / collaborative filtering.

We have an original matrix, $V$, and the goal is to find two new matrices, $H$ and $W$, that when multiplied together approximate $V$. This approximated $V$ contains no zeros, and values that were zero and replaced by an estimated floating point value.
- Per the title, these matrices, including $V$, cannot contain negative numbers.
- Ideal for using for counts of items purchased and other events that cannot be negative.

[Image source: Wikipedia](https://en.wikipedia.org/wiki/Non-negative_matrix_factorization#/media/File:NMF.png)

This doesn't have an exact solution, so it is solved with approximation, with the aim of minimizing the distance, $d$, between $V$ and $WH$.

It uses the squared Frobenius norm for distance, which is essentially Euclidean distance for matrices.

$$
d_{Fro}(V,WH)=\frac{1}{2}||V-WH||_{Fro}^2=\frac{1}{2}\sum(V_{ij}-WH_{ij})^2
$$

```python
import pandas as pd
import numpy as np
from sklearn.decomposition import NMF

V = np.array([[12,11,3,4,6,4,4],
              [12,3,6,5,6,4,3],
              [7 ,3,4,0,5, 1,2],
              [7 ,4,3,6,5, 1,2],
              [5 ,0,0,0,2,2,2],
              [3 ,1, 3,0,6,3,2]])

V
```

    array([[12, 11,  3,  4,  6,  4,  4],
           [12,  3,  6,  5,  6,  4,  3],
           [ 7,  3,  4,  0,  5,  1,  2],
           [ 7,  4,  3,  6,  5,  1,  2],
           [ 5,  0,  0,  0,  2,  2,  2],
           [ 3,  1,  3,  0,  6,  3,  2]])

```python
V = pd.DataFrame(V, columns=['Chris', 'Kim', 'Brian', 'Billy', 'Devin', 'John', 'Joe'])
V.index = ['Batman', 'Superman', 'Green Lantern', 'Aquaman', 'Flash', 'Cyborg']

V
```

      Chris
      Kim
      Brian
      Billy
      Devin
      John
      Joe

      Batman
      12
      11
      3
      4
      6
      4
      4

      Superman
      12
      3
      6
      5
      6
      4
      3

      Green Lantern
      7
      3
      4
      0
      5
      1
      2

      Aquaman
      7
      4
      3
      6
      5
      1
      2

      Flash
      5
      0
      0
      0
      2
      2
      2

      Cyborg
      3
      1
      3
      0
      6
      3
      2

```python
from sklearn.decomposition import NMF

nmf = NMF(init=None, max_iter=1000)
nmf.fit(V)

H = pd.DataFrame(nmf.components_, columns=V.columns)
H
```

      Chris
      Kim
      Brian
      Billy
      Devin
      John
      Joe

      0
      1.481381
      0.671551
      0.423763
      0.000098
      0.172562
      0.000000
      0.261832

      1
      1.638245
      0.000000
      0.000000
      0.000000
      0.000000
      0.715297
      0.485652

      2
      0.677868
      0.046757
      0.412258
      5.655834
      1.435203
      0.000000
      0.021804

      3
      0.000000
      2.389431
      0.000000
      0.021620
      1.413645
      1.024519
      0.593397

      4
      0.000000
      0.000000
      6.616534
      0.000000
      7.405330
      3.165260
      0.726275

      5
      0.549050
      0.000000
      0.000000
      0.000000
      2.700429
      0.000000
      0.863808

      6
      1.334195
      0.682190
      1.257843
      3.518748
      0.375986
      0.434316
      0.005638

```python
W = pd.DataFrame(nmf.transform(V), columns=V.columns)
W.index = V.index

W
```

      Chris
      Kim
      Brian
      Billy
      Devin
      John
      Joe

      Batman
      6.102613
      1.447464
      0.616891
      2.839753
      0.000000
      0.000000
      0.127527

      Superman
      4.228374
      3.033561
      0.770046
      0.000000
      0.553173
      0.000000
      0.183117

      Green Lantern
      4.468364
      0.000000
      0.000000
      0.000000
      0.317936
      0.694070
      0.000000

      Aquaman
      3.889098
      0.088770
      0.971149
      0.522732
      0.113336
      0.493160
      0.139625

      Flash
      0.003110
      2.800202
      0.000002
      0.000000
      0.000000
      0.740485
      0.000000

      Cyborg
      0.000000
      1.578600
      0.000000
      0.419434
      0.453809
      0.757708
      0.000000

```python
reconstructed = pd.DataFrame(np.dot(W,H), columns=V.columns)
reconstructed.index = V.index

print('Original\n')
print(V)

print('\nReconstructed\n')
print(np.round(reconstructed,4))
```

    Original

                   Chris  Kim  Brian  Billy  Devin  John  Joe
    Batman            12   11      3      4      6     4    4
    Superman          12    3      6      5      6     4    3
    Green Lantern      7    3      4      0      5     1    2
    Aquaman            7    4      3      6      5     1    2
    Flash              5    0      0      0      2     2    2
    Cyborg             3    1      3      0      6     3    2

    Reconstructed

                     Chris      Kim   Brian   Billy   Devin    John     Joe
    Batman         11.9999  10.9995  3.0008  3.9998  6.0008  4.0001  4.0001
    Superman       11.9999   3.0005  5.9997  5.0000  6.0001  4.0004  3.0000
    Green Lantern   7.0004   3.0007  3.9972  0.0004  4.9998  1.0064  2.0004
    Aquaman         7.0220   4.0014  2.9739  5.9956  5.0274  1.0184  1.9019
    Flash           4.9986   0.0021  0.0013  0.0000  2.0002  2.0030  2.0004
    Cyborg          3.0022   1.0022  3.0026  0.0091  5.9997  2.9953  1.9996

## Using on the grocery data
We are going to restrict the number of people to make this run relatively fast. This is the same format as the above example. Remember this is the `V` matrix.

```python
dfT = df.transpose()
dfT = dfT.iloc[:, 0:300]
dfT[:] = np.where(dfT[:], 1, 0)
dfT.head()
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31
      32
      33
      34
      35
      36
      37
      38
      39
      40
      41
      42
      43
      44
      45
      46
      47
      48
      49
      ...
      250
      251
      252
      253
      254
      255
      256
      257
      258
      259
      260
      261
      262
      263
      264
      265
      266
      267
      268
      269
      270
      271
      272
      273
      274
      275
      276
      277
      278
      279
      280
      281
      282
      283
      284
      285
      286
      287
      288
      289
      290
      291
      292
      293
      294
      295
      296
      297
      298
      299

      Instant food products
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      UHT-milk
      0
      0
      0
      0
      0
      0
      0
      1
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      1
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      1
      0
      0
      0

      abrasive cleaner
      0
      0
      0
      0
      0
      1
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      1
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      1
      0
      0
      0
      0
      0

      artif. sweetener
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      1
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      baby cosmetics
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

5 rows × 300 columns

#### Now we create the `H` matrix:

```python
from sklearn.decomposition import NMF
import datetime

st = datetime.datetime.now()

nmf = NMF(init=None, max_iter=2500)
nmf.fit(dfT)

Ht = pd.DataFrame(nmf.components_, columns=dfT.columns)

en = datetime.datetime.now()
el = en - st

print(f'Elapsed time: {el}')

Ht.head()
```

    Elapsed time: 0:02:02.083216

    /Applications/anaconda3/lib/python3.7/site-packages/sklearn/decomposition/_nmf.py:1641: ConvergenceWarning: Maximum number of iterations 2500 reached. Increase it to improve convergence.
      ConvergenceWarning,

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31
      32
      33
      34
      35
      36
      37
      38
      39
      40
      41
      42
      43
      44
      45
      46
      47
      48
      49
      ...
      250
      251
      252
      253
      254
      255
      256
      257
      258
      259
      260
      261
      262
      263
      264
      265
      266
      267
      268
      269
      270
      271
      272
      273
      274
      275
      276
      277
      278
      279
      280
      281
      282
      283
      284
      285
      286
      287
      288
      289
      290
      291
      292
      293
      294
      295
      296
      297
      298
      299

      0
      0.0
      0.0
      0.00002
      0.0
      0.002198
      0.000006
      0.000000e+00
      0.000000e+00
      0.0
      0.000022
      0.000000
      0.012004
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      1.156554e-34
      0.000000
      0.0
      0.00002
      0.0
      0.000000e+00
      0.0
      0.0
      0.000000
      0.000000
      0.0
      0.0
      0.000000
      0.025311
      0.014763
      0.0
      0.0
      0.0
      0.0
      0.025625
      0.0
      0.000000
      0.004404
      0.000000e+00
      0.0
      0.000000
      0.0
      0.0
      0.000000e+00
      0.0
      0.000000
      ...
      0.0
      0.035928
      0.0
      0.0
      0.000000
      0.000000
      0.0
      0.0
      0.079091
      5.285020e-16
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      0.014111
      0.0
      0.0
      0.005457
      0.0
      0.0
      0.000539
      2.752441e-02
      6.989083e-02
      4.656494e-02
      1.667551e-02
      7.909100e-02
      0.0
      0.0
      0.000053
      0.0
      0.0
      0.002004
      0.001089
      0.0
      0.000000e+00
      0.0
      0.000000
      0.000000
      0.013373
      0.055185
      0.000000
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.000000

      1
      0.0
      0.0
      0.00000
      0.0
      0.000000
      0.000000
      2.668249e-11
      5.402773e-03
      0.0
      0.000000
      0.000000
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000e+00
      0.000000
      0.0
      0.00000
      0.0
      7.973242e-04
      0.0
      0.0
      0.036527
      0.000000
      0.0
      0.0
      0.001406
      0.000000
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.000000
      0.0
      0.068046
      0.000000
      0.000000e+00
      0.0
      0.003181
      0.0
      0.0
      6.253165e-11
      0.0
      0.008652
      ...
      0.0
      0.031180
      0.0
      0.0
      0.000000
      0.000000
      0.0
      0.0
      0.090338
      0.000000e+00
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.041988
      0.000000
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.000000
      0.000000e+00
      1.355903e-02
      1.919380e-02
      1.467939e-02
      2.787107e-02
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.044259
      0.000000
      0.0
      0.000000e+00
      0.0
      0.019703
      0.000000
      0.000000
      0.000000
      0.000000
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.000000

      2
      0.0
      0.0
      0.00000
      0.0
      0.015767
      0.000000
      0.000000e+00
      1.576219e-07
      0.0
      0.000000
      0.000001
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000e+00
      0.000004
      0.0
      0.00000
      0.0
      1.068541e-14
      0.0
      0.0
      0.000000
      0.000004
      0.0
      0.0
      0.008766
      0.048462
      0.015924
      0.0
      0.0
      0.0
      0.0
      0.052345
      0.0
      0.000000
      0.000000
      0.000000e+00
      0.0
      0.000000
      0.0
      0.0
      0.000000e+00
      0.0
      0.033814
      ...
      0.0
      0.000000
      0.0
      0.0
      0.000001
      0.001046
      0.0
      0.0
      0.088102
      0.000000e+00
      0.000003
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      0.000002
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.000000
      8.426635e-02
      6.891331e-02
      3.755027e-09
      0.000000e+00
      6.321974e-02
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.000000
      0.000000
      0.0
      0.000000e+00
      0.0
      0.038972
      0.000000
      0.026377
      0.134186
      0.012480
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.002713

      3
      0.0
      0.0
      0.00000
      0.0
      0.000000
      0.000000
      0.000000e+00
      0.000000e+00
      0.0
      0.000000
      0.000000
      0.581140
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000e+00
      0.000000
      0.0
      0.00000
      0.0
      0.000000e+00
      0.0
      0.0
      0.000000
      0.000000
      0.0
      0.0
      0.000000
      0.000000
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.000000
      0.0
      0.000000
      0.000000
      9.957676e-17
      0.0
      0.000000
      0.0
      0.0
      0.000000e+00
      0.0
      0.000000
      ...
      0.0
      0.000000
      0.0
      0.0
      0.000000
      0.000000
      0.0
      0.0
      0.746334
      0.000000e+00
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      0.000000
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.000000
      1.569832e-09
      9.755250e-17
      0.000000e+00
      4.931461e-09
      1.065942e-09
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.000000
      0.000000
      0.0
      3.690886e-17
      0.0
      0.000000
      0.000000
      0.000000
      0.000000
      0.000000
      0.0
      0.018481
      0.0
      0.0
      0.0
      0.170278

      4
      0.0
      0.0
      0.00000
      0.0
      0.000000
      0.000000
      0.000000e+00
      0.000000e+00
      0.0
      0.000000
      0.000000
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000e+00
      0.000000
      0.0
      0.00000
      0.0
      0.000000e+00
      0.0
      0.0
      0.070052
      0.000000
      0.0
      0.0
      0.000000
      0.000000
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.000000
      0.0
      0.091925
      0.000000
      0.000000e+00
      0.0
      0.000000
      0.0
      0.0
      0.000000e+00
      0.0
      0.000000
      ...
      0.0
      0.043106
      0.0
      0.0
      0.000000
      0.000000
      0.0
      0.0
      0.086847
      0.000000e+00
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.144142
      0.000000
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.000000
      1.656889e-23
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.052576
      0.000000
      0.0
      0.000000e+00
      0.0
      0.000000
      0.019051
      0.000000
      0.000000
      0.068243
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.000000

5 rows × 300 columns

#### Now we create the `W` matrix:

```python
import datetime

st = datetime.datetime.now()
Wt = pd.DataFrame(nmf.transform(dfT), columns=dfT.columns)
Wt.index = dfT.index

en = datetime.datetime.now()
el = en - st

print(f'Elapsed time: {el}')

Wt
```

    Elapsed time: 0:00:00.796923

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31
      32
      33
      34
      35
      36
      37
      38
      39
      40
      41
      42
      43
      44
      45
      46
      47
      48
      49
      ...
      250
      251
      252
      253
      254
      255
      256
      257
      258
      259
      260
      261
      262
      263
      264
      265
      266
      267
      268
      269
      270
      271
      272
      273
      274
      275
      276
      277
      278
      279
      280
      281
      282
      283
      284
      285
      286
      287
      288
      289
      290
      291
      292
      293
      294
      295
      296
      297
      298
      299

      Instant food products
      0.000000
      0.0
      0.000000e+00
      0.000000
      0.000000
      0.000000e+00
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      0.000000e+00
      0.000000
      0.000000
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.000000
      0.000000
      0.000000e+00
      0.000000
      0.000000
      0.000000e+00
      0.0
      0.00000
      0.0
      0.000000e+00
      0.000000
      0.000000
      0.0
      0.000000
      0.000000e+00
      0.0
      0.0
      0.0
      0.000000e+00
      0.0
      0.000000e+00
      0.0
      0.0
      ...
      0.000000e+00
      0.000000
      0.0
      0.000000e+00
      0.000000
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000e+00
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000e+00
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.000000e+00
      0.0
      0.00000
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      0.000000e+00
      0.0
      0.000000
      0.000000
      0.0
      0.0
      0.000000e+00
      0.0
      0.0
      0.0
      0.000000
      0.0
      0.000000e+00
      0.000000

      UHT-milk
      0.000000
      0.0
      1.233575e-07
      0.000000
      0.000000
      1.847258e-07
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      0.000000e+00
      0.000000
      0.000000
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.000000
      0.000000
      0.000000e+00
      0.000000
      0.000000
      0.000000e+00
      0.0
      0.00000
      0.0
      0.000000e+00
      0.000000
      0.000000
      0.0
      0.000000
      2.712566e-07
      0.0
      0.0
      0.0
      0.000000e+00
      0.0
      0.000000e+00
      0.0
      0.0
      ...
      0.000000e+00
      0.000000
      0.0
      0.000000e+00
      0.000000
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000e+00
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000e+00
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.000000e+00
      0.0
      0.00000
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      0.000000e+00
      0.0
      0.000000
      0.000000
      0.0
      0.0
      0.000000e+00
      0.0
      0.0
      0.0
      0.008104
      0.0
      1.566223e-18
      0.000000

      abrasive cleaner
      0.000000
      0.0
      0.000000e+00
      0.000000
      0.000000
      0.000000e+00
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      0.000000e+00
      0.000000
      0.000000
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.000000
      0.000000
      0.000000e+00
      0.000000
      0.000000
      0.000000e+00
      0.0
      0.00000
      0.0
      0.000000e+00
      0.000000
      0.000000
      0.0
      0.000000
      0.000000e+00
      0.0
      0.0
      0.0
      0.000000e+00
      0.0
      0.000000e+00
      0.0
      0.0
      ...
      0.000000e+00
      0.000000
      0.0
      3.567619e-17
      0.000000
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      1.134297e-18
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      9.881313e-324
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.000000e+00
      0.0
      0.00000
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      0.000000e+00
      0.0
      0.000000
      0.000000
      0.0
      0.0
      0.000000e+00
      0.0
      0.0
      0.0
      0.000000
      0.0
      0.000000e+00
      0.774753

      artif. sweetener
      0.000000
      0.0
      0.000000e+00
      0.000000
      0.000000
      0.000000e+00
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      0.000000e+00
      0.000000
      0.000000
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.000000
      0.000000
      0.000000e+00
      0.000000
      0.000000
      0.000000e+00
      0.0
      0.00000
      0.0
      0.000000e+00
      0.000000
      0.000000
      0.0
      0.000000
      0.000000e+00
      0.0
      0.0
      0.0
      0.000000e+00
      0.0
      0.000000e+00
      0.0
      0.0
      ...
      0.000000e+00
      0.000000
      0.0
      0.000000e+00
      0.000000
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000e+00
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      9.881313e-324
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.000000e+00
      0.0
      0.00000
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      0.000000e+00
      0.0
      0.000000
      0.000000
      0.0
      0.0
      0.000000e+00
      0.0
      0.0
      0.0
      0.000000
      0.0
      0.000000e+00
      0.000000

      baby cosmetics
      0.000000
      0.0
      0.000000e+00
      0.000000
      0.000000
      0.000000e+00
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      0.000000e+00
      0.000000
      0.000000
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.000000
      0.000000
      0.000000e+00
      0.000000
      0.000000
      0.000000e+00
      0.0
      0.00000
      0.0
      0.000000e+00
      0.000000
      0.000000
      0.0
      0.000000
      0.000000e+00
      0.0
      0.0
      0.0
      0.000000e+00
      0.0
      0.000000e+00
      0.0
      0.0
      ...
      0.000000e+00
      0.000000
      0.0
      0.000000e+00
      0.000000
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000e+00
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000e+00
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.000000e+00
      0.0
      0.00000
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      0.000000e+00
      0.0
      0.000000
      0.000000
      0.0
      0.0
      0.000000e+00
      0.0
      0.0
      0.0
      0.000000
      0.0
      0.000000e+00
      0.000000

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      white bread
      0.000000
      0.0
      0.000000e+00
      0.000000
      0.000000
      0.000000e+00
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      0.000000e+00
      0.000000
      0.000000
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.000000
      0.000000
      0.000000e+00
      0.000000
      0.000000
      0.000000e+00
      0.0
      0.00000
      0.0
      0.000000e+00
      0.000000
      0.000000
      0.0
      0.000000
      0.000000e+00
      0.0
      0.0
      0.0
      1.616827e-07
      0.0
      0.000000e+00
      0.0
      0.0
      ...
      0.000000e+00
      0.000000
      0.0
      0.000000e+00
      0.000000
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000e+00
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      9.881313e-324
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.000000e+00
      0.0
      0.00000
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      0.000000e+00
      0.0
      0.000000
      0.000000
      0.0
      0.0
      0.000000e+00
      0.0
      0.0
      0.0
      0.000000
      0.0
      0.000000e+00
      0.000000

      white wine
      0.000000
      0.0
      0.000000e+00
      0.000000
      0.000000
      0.000000e+00
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      0.000000e+00
      0.000000
      0.000000
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.000000
      0.000000
      0.000000e+00
      0.000000
      0.000000
      0.000000e+00
      0.0
      0.00000
      0.0
      2.093691e-17
      0.000000
      0.000000
      0.0
      0.000000
      0.000000e+00
      0.0
      0.0
      0.0
      0.000000e+00
      0.0
      0.000000e+00
      0.0
      0.0
      ...
      0.000000e+00
      0.000000
      0.0
      0.000000e+00
      0.000000
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000e+00
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000e+00
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.000000e+00
      0.0
      0.00000
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      2.625385e-13
      0.0
      0.000000
      0.000000
      0.0
      0.0
      0.000000e+00
      0.0
      0.0
      0.0
      0.000000
      0.0
      0.000000e+00
      0.000000

      whole milk
      12.642365
      0.0
      2.520308e-04
      0.000036
      0.000084
      0.000000e+00
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      1.250167
      0.000000e+00
      0.029761
      0.000000
      0.0
      0.0
      1.035107
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000179
      0.0
      0.0
      0.089445
      0.000000
      0.000000e+00
      0.000020
      1.222654
      0.000000e+00
      0.0
      0.90915
      0.0
      0.000000e+00
      0.082572
      0.060907
      0.0
      0.000000
      0.000000e+00
      0.0
      0.0
      0.0
      0.000000e+00
      0.0
      4.182260e-07
      0.0
      0.0
      ...
      1.277955e-01
      0.000024
      0.0
      7.130873e-01
      0.000000
      0.0
      1.180874
      0.0
      0.0
      0.0
      0.0
      0.0
      2.561569e-02
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      3.260833e-322
      0.0
      0.0
      0.122532
      0.0
      0.0
      0.0
      0.000000e+00
      0.0
      0.00059
      0.910999
      0.0
      0.0
      0.0
      0.0
      0.0
      0.051937
      0.000000e+00
      0.0
      0.000065
      0.000000
      0.0
      0.0
      1.861864e-14
      0.0
      0.0
      0.0
      0.000000
      0.0
      0.000000e+00
      0.000333

      yogurt
      0.000000
      0.0
      0.000000e+00
      0.945169
      0.000000
      0.000000e+00
      0.794107
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      0.000000e+00
      0.000793
      0.736841
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      3.064052
      0.0
      0.0
      0.000000
      0.000002
      5.900995e-07
      0.000016
      0.000003
      0.000000e+00
      0.0
      0.00000
      0.0
      0.000000e+00
      0.040057
      0.000000
      0.0
      0.005705
      0.000000e+00
      0.0
      0.0
      0.0
      0.000000e+00
      0.0
      0.000000e+00
      0.0
      0.0
      ...
      2.831448e-07
      0.000000
      0.0
      0.000000e+00
      0.000000
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000e+00
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      6.916919e-323
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.0
      2.777106e-21
      0.0
      0.00000
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      0.000000e+00
      0.0
      0.000000
      0.000000
      0.0
      0.0
      0.000000e+00
      0.0
      0.0
      0.0
      0.000000
      0.0
      0.000000e+00
      0.000002

      zwieback
      0.000000
      0.0
      0.000000e+00
      0.000000
      0.000000
      0.000000e+00
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      4.397701e-18
      0.000000
      0.000000
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.000000
      0.000000
      0.000000e+00
      0.000000
      0.000000
      3.502259e-18
      0.0
      0.00000
      0.0
      0.000000e+00
      0.000000
      0.000000
      0.0
      0.000000
      8.947850e-18
      0.0
      0.0
      0.0
      0.000000e+00
      0.0
      0.000000e+00
      0.0
      0.0
      ...
      0.000000e+00
      0.000000
      0.0
      0.000000e+00
      0.745754
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000e+00
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      9.881313e-324
      0.0
      0.0
      0.000000
      0.0
      0.0
      0.0
      0.000000e+00
      0.0
      0.00000
      0.000000
      0.0
      0.0
      0.0
      0.0
      0.0
      0.000000
      0.000000e+00
      0.0
      0.000000
      0.229419
      0.0
      0.0
      0.000000e+00
      0.0
      0.0
      0.0
      0.000000
      0.0
      0.000000e+00
      0.000000

169 rows × 300 columns

#### And now we multiply `H` and `W` to reconstruct `V`:

```python
reconstructed = pd.DataFrame(np.dot(Wt,Ht), columns=dfT.columns)
reconstructed.index = dfT.index

reconstructed.head()
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31
      32
      33
      34
      35
      36
      37
      38
      39
      40
      41
      42
      43
      44
      45
      46
      47
      48
      49
      ...
      250
      251
      252
      253
      254
      255
      256
      257
      258
      259
      260
      261
      262
      263
      264
      265
      266
      267
      268
      269
      270
      271
      272
      273
      274
      275
      276
      277
      278
      279
      280
      281
      282
      283
      284
      285
      286
      287
      288
      289
      290
      291
      292
      293
      294
      295
      296
      297
      298
      299

      Instant food products
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.0
      0.0
      0.0
      0.0
      0.000000e+00
      0.000000e+00
      0.0
      0.0
      0.0
      1.155055e-50
      0.0
      0.000000e+00
      0.0
      0.0
      0.000000e+00
      2.357664e-50
      0.000000e+00
      2.172140e-17
      4.341257e-18
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      1.002070e-17
      1.581460e-28
      1.451663e-17
      0.000000e+00
      2.158481e-17
      0.0
      0.0
      0.0
      4.537322e-18
      0.000000e+00
      1.592600e-28
      1.052679e-18
      0.000000e+00
      0.0
      2.188574e-29
      0.000000e+00
      0.000000e+00
      9.195452e-29
      0.0
      4.019399e-18
      ...
      0.000000e+00
      8.733560e-29
      0.0
      0.0
      0.000000e+00
      3.334568e-18
      0.000000e+00
      0.0
      3.026346e-28
      2.347839e-17
      6.321676e-18
      0.0
      0.0
      0.0
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.0
      5.496615e-18
      4.597030e-18
      0.000000e+00
      0.000000e+00
      3.131336e-18
      3.910855e-18
      0.000000e+00
      7.024057e-24
      3.795235e-18
      0.000000e+00
      0.0
      0.000000e+00
      0.000000e+00
      0.0
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.0
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.0
      0.000000e+00
      1.515142e-18
      0.000000e+00
      3.108663e-18
      0.000000e+00
      0.000000
      0.0
      0.0
      0.000000e+00

      UHT-milk
      4.248512e-08
      9.137343e-08
      0.000000e+00
      0.000000e+00
      1.919180e-07
      6.627578e-08
      0.0
      1.0
      0.0
      0.0
      8.936388e-09
      8.855335e-08
      0.0
      0.0
      0.0
      1.225381e-07
      0.0
      5.918861e-41
      0.0
      0.0
      4.971322e-13
      8.127113e-38
      0.000000e+00
      4.473542e-07
      1.336826e-20
      6.170872e-09
      0.000000e+00
      1.544722e-32
      4.806625e-13
      4.041319e-39
      6.114673e-09
      1.265120e-08
      5.978116e-09
      2.144908e-09
      0.000000e+00
      0.0
      0.0
      0.0
      6.457192e-09
      4.836768e-08
      0.000000e+00
      6.394794e-08
      1.996394e-07
      0.0
      2.033035e-09
      1.303758e-07
      5.676075e-09
      1.748140e-34
      0.0
      9.999984e-01
      ...
      0.000000e+00
      4.237881e-09
      0.0
      0.0
      1.276853e-13
      1.290572e-10
      0.000000e+00
      0.0
      5.039412e-08
      7.596190e-39
      2.698914e-07
      0.0
      0.0
      0.0
      1.295832e-07
      5.175720e-09
      9.695424e-08
      2.662056e-13
      0.0
      0.000000e+00
      3.034899e-07
      1.265768e-08
      2.638111e-19
      5.951287e-09
      1.039489e-08
      1.986984e-08
      9.954246e-09
      9.552716e-23
      5.521528e-08
      0.0
      9.150437e-37
      4.044662e-09
      0.0
      4.846453e-08
      0.000000e+00
      0.000000e+00
      0.0
      7.203437e-08
      2.756384e-07
      5.087585e-08
      0.0
      1.085422e-08
      2.990573e-08
      1.539447e-09
      2.551649e-09
      4.093835e-08
      0.999998
      0.0
      0.0
      1.822580e-08

      abrasive cleaner
      0.000000e+00
      8.396959e-10
      1.748561e-20
      1.597706e-09
      4.975850e-20
      1.000000e+00
      0.0
      0.0
      0.0
      0.0
      0.000000e+00
      1.097683e-08
      0.0
      0.0
      0.0
      3.279334e-09
      0.0
      0.000000e+00
      0.0
      0.0
      0.000000e+00
      0.000000e+00
      4.804239e-20
      0.000000e+00
      0.000000e+00
      0.000000e+00
      1.448962e-09
      0.000000e+00
      1.043699e-17
      0.000000e+00
      1.622808e-09
      0.000000e+00
      1.814806e-20
      1.000000e+00
      0.000000e+00
      0.0
      0.0
      0.0
      0.000000e+00
      0.000000e+00
      0.000000e+00
      7.837056e-11
      1.326275e-09
      0.0
      1.741627e-09
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.0
      2.451355e-09
      ...
      5.271848e-18
      1.630478e-09
      0.0
      0.0
      0.000000e+00
      1.238118e-09
      0.000000e+00
      0.0
      7.605426e-10
      5.978538e-20
      0.000000e+00
      0.0
      0.0
      0.0
      0.000000e+00
      1.136250e-35
      6.719548e-09
      1.583457e-30
      0.0
      0.000000e+00
      3.992317e-34
      0.000000e+00
      0.000000e+00
      2.837026e-20
      7.017285e-20
      1.651060e-09
      1.327599e-09
      3.351539e-20
      2.767354e-20
      0.0
      0.000000e+00
      1.221729e-09
      0.0
      0.000000e+00
      1.562730e-09
      8.738918e-35
      0.0
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.0
      7.222085e-19
      9.709515e-10
      1.432687e-09
      1.000000e+00
      1.523050e-09
      0.000000
      0.0
      0.0
      1.335061e-09

      artif. sweetener
      0.000000e+00
      0.000000e+00
      1.328591e-37
      0.000000e+00
      1.585889e-29
      6.434943e-12
      0.0
      0.0
      0.0
      0.0
      0.000000e+00
      5.906156e-60
      0.0
      0.0
      0.0
      0.000000e+00
      0.0
      0.000000e+00
      0.0
      0.0
      0.000000e+00
      0.000000e+00
      3.777750e-37
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      3.865155e-31
      1.761336e-17
      0.000000e+00
      0.000000e+00
      0.000000e+00
      3.929816e-59
      1.000000e+00
      1.436883e-33
      0.0
      0.0
      0.0
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      5.019710e-12
      0.0
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.0
      0.000000e+00
      ...
      8.896719e-18
      0.000000e+00
      0.0
      0.0
      0.000000e+00
      0.000000e+00
      5.153540e-31
      0.0
      6.450121e-17
      3.281567e-59
      5.593825e-12
      0.0
      0.0
      0.0
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.0
      0.000000e+00
      4.783973e-12
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.0
      0.000000e+00
      7.946773e-60
      0.0
      0.000000e+00
      0.000000e+00
      5.712692e-38
      0.0
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.0
      0.000000e+00
      0.000000e+00
      0.000000e+00
      1.057928e-21
      0.000000e+00
      0.000000
      0.0
      0.0
      0.000000e+00

      baby cosmetics
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.0
      0.0
      0.0
      0.0
      0.000000e+00
      0.000000e+00
      0.0
      0.0
      0.0
      0.000000e+00
      0.0
      0.000000e+00
      0.0
      0.0
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.0
      0.0
      0.0
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.0
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.0
      0.000000e+00
      ...
      0.000000e+00
      0.000000e+00
      0.0
      0.0
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.0
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.0
      0.0
      0.0
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.0
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.0
      0.000000e+00
      0.000000e+00
      0.0
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.0
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.0
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000e+00
      0.000000
      0.0
      0.0
      0.000000e+00

5 rows × 300 columns

### Let's find individuals that bought frankfurters (hotdogs) and no buns/rolls
You would expect to recommend someone that.

```python
df.query('frankfurter and `rolls/buns` == False', engine='python').head()
```

      Instant food products
      UHT-milk
      abrasive cleaner
      artif. sweetener
      baby cosmetics
      baby food
      bags
      baking powder
      bathroom cleaner
      beef
      berries
      beverages
      bottled beer
      bottled water
      brandy
      brown bread
      butter
      butter milk
      cake bar
      candles
      candy
      canned beer
      canned fish
      canned fruit
      canned vegetables
      cat food
      cereals
      chewing gum
      chicken
      chocolate
      chocolate marshmallow
      citrus fruit
      cleaner
      cling film/bags
      cocoa drinks
      coffee
      condensed milk
      cooking chocolate
      cookware
      cream
      cream cheese
      curd
      curd cheese
      decalcifier
      dental care
      dessert
      detergent
      dish cleaner
      dishes
      dog food
      ...
      ready soups
      red/blush wine
      rice
      roll products
      rolls/buns
      root vegetables
      rubbing alcohol
      rum
      salad dressing
      salt
      salty snack
      sauces
      sausage
      seasonal products
      semi-finished bread
      shopping bags
      skin care
      sliced cheese
      snack products
      soap
      soda
      soft cheese
      softener
      sound storage medium
      soups
      sparkling wine
      specialty bar
      specialty cheese
      specialty chocolate
      specialty fat
      specialty vegetables
      spices
      spread cheese
      sugar
      sweet spreads
      syrup
      tea
      tidbits
      toilet cleaner
      tropical fruit
      turkey
      vinegar
      waffles
      whipped/sour cream
      whisky
      white bread
      white wine
      whole milk
      yogurt
      zwieback

      72
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      True
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      ...
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      True
      False
      False
      True
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      True
      False
      False

      120
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      ...
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      True
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      True
      False
      False
      False
      False
      False
      False
      False

      125
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      True
      False
      False
      False
      False
      False
      False
      False
      True
      False
      False
      False
      False
      False
      False
      False
      False
      True
      False
      False
      False
      False
      False
      False
      False
      False
      False
      ...
      False
      False
      True
      False
      False
      False
      False
      False
      False
      False
      False
      False
      True
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      True
      False
      False
      False
      False
      False
      True
      False
      False
      False
      False
      False
      False
      False
      True
      False
      False

      130
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      True
      False
      False
      False
      False
      ...
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False

      161
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      True
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      ...
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      True
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False
      False

5 rows × 169 columns

```python
person_id = 120

reconstructed[person_id].sort_values(ascending=False).head(15)
```

    long life bakery product    1.042067
    sausage                     1.000017
    frankfurter                 1.000000
    waffles                     0.999967
    pastry                      0.007028
    sliced cheese               0.002700
    beef                        0.001087
    brown bread                 0.000989
    rolls/buns                  0.000681
    napkins                     0.000575
    butter                      0.000293
    hard cheese                 0.000151
    whipped/sour cream          0.000130
    salty snack                 0.000103
    fruit/vegetable juice       0.000092
    Name: 120, dtype: float64

# Readings
- [Stanford k-Means Notes](https://stanford.edu/~cpiech/cs221/handouts/kmeans.html)
- [Dendrogram Gallery](https://www.r-graph-gallery.com/dendrogram.html)
- [DBSCAN Paper](https://www.aaai.org/Papers/KDD/1996/KDD96-037.pdf)
- [Fast Algorithms for Association Rule Mining](http://www.vldb.org/conf/1994/P487.PDF)
- [Non-negative Matrix Factorization Wiki](https://en.wikipedia.org/wiki/Non-negative_matrix_factorization)
- [Non-negative Matrix Factorization Example](https://medium.com/logicai/non-negative-matrix-factorization-for-recommendation-systems-985ca8d5c16c)

```python

```