# The Sparsifier Class

In this notebook we introduce the `Sparsifier` class and demonstrate its basic functionality

```python
import numpy as np
from sparseklearn import Sparsifier
```

```python
%load_ext autoreload
%autoreload 2
```

## Fitting the sparsifier

`fit_sparsifier` will apply the transform $HD$ to $X$, draw the `num_feat_comp` indices we will keep for each row of $HDX$, and then discard the data. As an example, we'll generate a dataset $X \in \mathbb{R}^{10 \times 5}$, where each row is a datapoint and there are 5 features. We'll sparsify $X$ into 3 dimensions, so that we are keeping $RHDX \in \mathbb{R}^{10 \times 3}$.

To use the sparsifier we need to specify the number of features of the original space (`num_feat_full`), the number of samples (`num_samp`), and the number of features we wish to keep in the compressed setting (`num_feat_comp`).

```python
rs = np.random.RandomState(12)
num_samp, num_feat_full = 10, 5
X = rs.rand(num_samp, num_feat_full)
num_feat_comp = 3
sp = Sparsifier(num_feat_full = num_feat_full, num_feat_comp = num_feat_comp, num_samp = num_samp, random_state = rs, transform = None)
sp.fit_sparsifier(X)
```

The `mask` attribute indicates which indices are kept for each datapoint:

```python
sp.mask
```

The `RHDX` attribute contains the data:

```python
sp.RHDX
```

Since we didn't use a transform in the Sparsifier, `RHDX` is just a subset of `X`. Check this for an arbitrary datapoint:

```python
row = 3
(X[row][sp.mask[row]] == sp.RHDX[row]).all()
```

## Operations on sparsified data

The purpose of sparsifying the data is to permit operations on mixtures of sparse datapoints and dense statistics. Having fit the `Sparsifier` object, we can discard $X$ and use built-in methods to compute things like distances and covariances. We can compute the (approximate) pairwise distances between all the datapoints in the sample, for example:

```python
sp.pairwise_distances()
```

We can also compute the pairwise distances between the compressed data and a set of dense datapoints, which might be statistics we wish to estimate.

```python
dense_statistics = np.random.rand(4,num_feat_full)
sp.pairwise_distances(dense_statistics)
```

The `Sparsifier` class contains several other built-in methods to compute related quantities, including:
- `weighted_means`
- `weighted_means_and_variances`
- `pairwise_mahalanobis_distances`
See the docstrings for more information:

```python
help(sp.pairwise_mahalanobis_distances)
```

## The Johnson-Lindenstrauss Lemma

The Johnson-Lindenstrauss Lemma guarantees the existence of a projection $\Omega: \mathbb{R}^P \to \mathbb{R}^Q$ such that pairwise distances on a fixed number of points are preserved within a low tolerance with high probability. The Sparsifier is a random projection that acts like the JL projections. We will see an example of this here. We will first embed low-dimensional data into a higher dimension with some noise, and then see that as we keep more and more components, the error in the pairwise distances shrinks.

```python
from scipy.stats import ortho_group
from sklearn.metrics import pairwise_distances, mean_squared_error
original_latent_dimension = 20
embedded_dimension = 100
num_samp = 100
rs = np.random.RandomState(73)
X = rs.rand(num_samp, original_latent_dimension) # generate the low-dimensional data
X = np.concatenate((X, np.zeros([num_samp,embedded_dimension-original_latent_dimension])), axis=1) # embed into higher dimension
X += rs.normal(scale = 1e-8, size = (num_samp, embedded_dimension)) # add noise
X = X[:, rs.permutation(np.arange(0,embedded_dimension,1))] # shuffle columns
true_distances = pairwise_distances(X)
```

```python
sp = Sparsifier(num_samp = num_samp, num_feat_full = embedded_dimension, num_feat_comp = 10)
sp.fit_sparsifier(X=X)
HDX = sp.apply_HD(X)

num_feat_comp_array = np.arange(1,original_latent_dimension*3)
mse = np.zeros([3, len(num_feat_comp_array)])
for i, num_feat_comp in enumerate(num_feat_comp_array):

    sp1 = Sparsifier(num_samp = num_samp, num_feat_full = embedded_dimension, num_feat_comp = num_feat_comp)
    sp1.fit_sparsifier(HDX=HDX)
    mse[0,i] = mean_squared_error(true_distances, sp1.pairwise_distances())

    sp2 = Sparsifier(num_samp = num_samp, num_feat_full = embedded_dimension, num_feat_comp = num_feat_comp, num_feat_shared = num_feat_comp//10 + 1)
    sp2.fit_sparsifier(HDX=HDX)
    mse[1,i] = mean_squared_error(true_distances, sp2.pairwise_distances())

    sp3 = Sparsifier(num_samp = num_samp, num_feat_full = embedded_dimension, num_feat_comp = num_feat_comp, num_feat_shared = num_feat_comp)
    sp3.fit_sparsifier(HDX=HDX)
    mse[2,i] = mean_squared_error(true_distances, sp3.pairwise_distances())
```

```python
import matplotlib.pyplot as plt
fig,ax = plt.subplots(1,1,figsize=(5,5))
ax.plot(num_feat_comp_array, mse[0], label = 'No shared features', linewidth = 2)
ax.plot(num_feat_comp_array, mse[1], label = '~10% shared features', linewidth = 2)
ax.plot(num_feat_comp_array, mse[2], label = '100% shared features', linewidth = 2)
ax.set_ylabel('MSE', fontsize = 12)
ax.set_xlabel(r'Number of features kept, $Q$', fontsize = 12)
ax.legend()
plt.show()
```

This plot shows the MSE of the pairwise distances as a function of the number of features we keep out of 100 original features. The fact that we resample the projection $\mathbf{R}_i$ for each datapoint is how the algorithms can be one-pass, but as a consequence we lose accuracy compared to the classical case when computing statitics on two compressed points, because we approximate these statitics using information in the indices shared by both points:

\begin{equation}
||\mathbf{x} - \mathbf{y}||^2 \simeq \sum_{j \in \mathbf{R}_x \cap \mathbf{R}_y} (x_j - y_j)^2
\end{equation}

Under high compression it's likely that $\mathbf{R}_x \cap \mathbf{R}_y = \emptyset$, where we use $\mathbf{R}_x \cap \mathbf{R}_y$ to denote the intersection of features preserved under both projections. Keeping the number of features constant, accuracy increases as the number of these forced to be shared between all datapoints is increased. To mitigate this we can force some number features to be shared across all datapoints. We did so in the above using the `num_feat_shared` parameter when we initialized the sparsifier.

In our algorithms we typically do not need to compute such quantities (in fact, to date the only time we do is during `k-means++` initialization, where we sample a new point with probability proportional to its distance from the current set of initial means). Instead we perform computations like $||\mathbf{R}_i^T \mathbf{x}_i - \mathbf{R}_i^T \boldsymbol{\mu}||$ where $\boldsymbol{\mu}$ is dense. In this case we no longer suffer from the problem described above, since we can use every feature preserved by $\mathbf{R}_i$ rather than an intersection with some other projection.

We show an example of this below, where we use the same data as above, but compute the pairwise distances between a few dense points and the rest of the sparsified data.

```python
dense_vectors, preconditioned_dense_vectors = X[:10], HDX[:10]
X_new, HDX_new = X[10:], HDX[10:]
true_distances = pairwise_distances(X_new, Y=dense_vectors)
num_samp = X_new.shape[0]
num_feat_comp_array = np.arange(1,original_latent_dimension*3)
mse = np.zeros([3, len(num_feat_comp_array)])
for i, num_feat_comp in enumerate(num_feat_comp_array):

    sp1 = Sparsifier(num_samp = num_samp, num_feat_full = embedded_dimension, num_feat_comp = num_feat_comp)
    sp1.fit_sparsifier(HDX=HDX_new)
    mse[0,i] = mean_squared_error(true_distances, sp1.pairwise_distances(Y=preconditioned_dense_vectors))

    sp2 = Sparsifier(num_samp = num_samp, num_feat_full = embedded_dimension, num_feat_comp = num_feat_comp, num_feat_shared = num_feat_comp//10 + 1)
    sp2.fit_sparsifier(HDX=HDX_new)
    mse[1,i] = mean_squared_error(true_distances, sp2.pairwise_distances(Y=preconditioned_dense_vectors))

    sp3 = Sparsifier(num_samp = num_samp, num_feat_full = embedded_dimension, num_feat_comp = num_feat_comp, num_feat_shared = num_feat_comp)
    sp3.fit_sparsifier(HDX=HDX_new)
    mse[2,i] = mean_squared_error(true_distances, sp3.pairwise_distances(Y=preconditioned_dense_vectors))

fig,ax = plt.subplots(1,1,figsize=(5,5))
ax.plot(num_feat_comp_array, mse[0], label = 'No shared features', linewidth = 2)
ax.plot(num_feat_comp_array, mse[1], label = '~10% shared features', linewidth = 2)
ax.plot(num_feat_comp_array, mse[2], label = '100% shared features', linewidth = 2)
ax.set_ylabel('MSE', fontsize = 12)
ax.set_xlabel(r'Number of features kept, $Q$', fontsize = 12)
ax.legend()
plt.show()
```

As expected, there is now no advantage to using more shared features.


# Efficient Data Representation by Selecting Prototypes with Importance Weights: An Implementation in PySpark

## Abstract

In this notebook I present a prototype selection algorithm proposed by Gurumoorthy, K. S. et al. ([[1]](#first_ref)) called _ProtoDash_, that is capable not only of selecting prototypes from a source dataset that are representative of a given target dataset, but can also assign non-negative weights to each one, thereby giving a sense of their relative importance in representing this target dataset.

I implement the _ProtoDash_ algorithm using the Apache Spark Python API PySpark and in two experiments demonstrate its performance in application to a well-known large dataset: the MNIST Dataset of Handwritten Digits [[6]](#fifth_ref). In these experiments I first demonstrate that the _ProtoDash_ algorithm is capable of effectively representing an increasingly skewed target dataset based on a completely distinct and randomly sampled source dataset. Furthermore, I show that the algorithm's runtime in practice is largely driven by the computation of the mean inner product between the observations in the target and source datasets, with the actual selection of prototypes taking an increasingly small fraction of time as the dataset sizes increase by orders of magnitude.

## Table of contents
* [0) How can I run this notebook?](#fourth-bullet)
* [1) Introduction](#first0-bullet)
* [2) Problem statement](#first-bullet)
* [3) The ProtoDash algorithm](#second-bullet)
* [4) ProtoDash Time Complexity Analysis](#ss-bullet)
* [5) PySpark implementation of the ProtoDash algorithm](#fifth-bullet)
* [6) Applying the ProtoDash algorithm to the MNIST dataset](#sixth-bullet)
* [7) Conclusion](#ninth-bullet)
* [References](#tenth-bullet)

## 0) How can I run this notebook?

First you will need to have __gsutil__ installed on your computer and ready to run. If you do not, please refer to the link below to get started:

 - https://cloud.google.com/storage/docs/gsutil_install

Now, given a choice of BUCKETNAME, CLUSTERNAME and given a PROJECTNAME, in your computer terminal enter:

    gsutil mb BUCKETNAME

Download the __my-actions.sh__ file from the project repository, then upload it to the created bucket with the terminal command:

    gsutil cp my-actions.sh gs://BUCKETNAME/

Again in a terminal, enter:

    gcloud dataproc clusters create CLUSTERNAME \
    --project PROJECTNAME \
    --bucket BUCKETNAME \
    --initialization-actions gs://dataproc-initialization-actions/jupyter/jupyter.sh,gs://dataproc-initialization-actions/python/pip-install.sh,gs://BUCKETNAME/my-actions.sh \
    --metadata 'PIP_PACKAGES= mnist scipy pandas qpsolvers cvxopt numpy' \
    --master-machine-type n1-standard-4 \
    --master-boot-disk-size 500 \
    --num-workers 5 \
    --worker-machine-type n1-standard-4 \
    --worker-boot-disk-size 500

Once this is done navigate to your GCP portal, create an SSH tunnel and upload this notebook onto the created cluster.

[Back](#oth-bullet)

## 1) Introduction

Given a large and complex dataset, it can be useful to identify a subset of prototypical observations that can best offer a condensed overview of it. To see why consider the case of a machine learning model trained on a massive and noisy real world dataset. For reasons of interpretability it can be crucial to be able to efficiently extract observations from this noisy dataset that are particularly salient, or that essentially best 'describe' it, in order to better understand how the output of the machine learning model is obtained. More generally, the selection of a representative sample can  be useful for summarizing large noisy datasets, reducing these to a smaller, more compact and more informative subset.

The idea of selecting a subset of a large dataset that best offers this distilled view of the data has led to the development of algorithms that can identify a subset $S$ of a set $V$ that maximise a given scoring function $f(S).$ For prototype selection, this function $f(S)$ essentially provides a measure of the relevance of $S$ in accurately describing $V,$ while potentially also penalizing any redundancy in $S,$ to ensure compactness of the selected subset is maintained. Several such algorithms have been proposed, including [[1]](#first_ref), [[3]](#third_ref), [[8]](#eighth_ref).

In this notebook I closely examine one such algorithm called *ProtoDash*, proposed in recent work by [[1]](#first_ref), and implement it using PySpark, the Python API written to support Apache Spark. _ProtoDash_ is worthy of particular interest as it is demonstrated in [[1]](#first_ref) to be capable of selecting both representative samples and outliers that can succintly represent complex data, while assigning its selected prototypes non-negative weights, thus in essence ranking them in relevance. Furthermore, it has a clear computational advantage over the algorithm it replaces, *ProtoGreedy*, built in part on work by [[3]](#third_ref), bringing about a large runtime speedup both theoretically and practically while not compromising on quality of the subset selection.

In this notebook I begin by formulating the problem statement underpinning _ProtoDash_, before presenting the algorithm and discussing its theoretical time complexity. I then show my implementation of the algorithm in PySpark. In an experiment section I wield the implemented algorithm onto a large dataset: the MNIST dataset of Handwritten Digits [[6]](#fifth_ref). I show that my implementation of the algorithm works in identifying prototypical observations from target datasets of varying sizes. Finally, I closely examine how performance of the algorithm is affected by varying dataset size.

[Back](#oth-bullet)

## 2) Problem statement

To understand the problem that _ProtoDash_ aims to solve, suppose we have a dataset $V$ consisting of a number of observations (these could for example correspond to features). Our goal is to find a subset $S \subseteq V$ that can maximize a score function $f(S)$, where $f(S)$ will measure the relevance of $S$ in representing $V$ while favouring compact subsets with minimal redundancy, i.e. favour information-rich and compact subsets.

Denote by $X^{(1)}$ and $X^{(2)}$ the target and source datasets, respectively. We want to find a subset of prototypes $S$ from the source datast $X^{(2)}$ that can best describe the target dataset $X^{(1)}$ by maximising this score function $f(\cdot).$ Now denote by $\mathcal{X}$ the feature space from which observations in $X^{(1)}$ and $X^{(2)}$ are sampled, and consider a kernel function $k: \mathcal{X} \times \mathcal{X} \rightarrow \mathbb{R}$ along with its corresponding reproducing kernel Hilbert space (RKHS) $\mathcal{K}.$ $\mathcal{K}$ has the inner product $k(x_i,x_j) = \langle \phi_{x_i}, \phi_{x_j} \rangle$ in which $\phi_{x}(y) = k(x,y) \in \mathcal{K}$ is a continuous linear function satisfying $\phi_x : h \rightarrow h(x) = \langle \phi_x, h \rangle $ for any function $h \in \mathcal{K}:\mathcal{X} \rightarrow \mathbb{R}.$

The approach proposed by [[1]](#first_ref) is to use the maximum mean discrepancy (MMD) [[2]](#second_ref) as the basis for the scoring function $f(\cdot)$. The MMD measures the distance between two probability distributions $p$ and $q$ by

\begin{equation}
MMD(\mathcal{K},p,q) = \sup_{h \in \mathcal{K}}\langle h, \mathbf{\mu}_p - \mathbf{\mu}_q \rangle,
\end{equation}

where $\mathbf{\mu}_p = \mathbb{E}_{x \sim p}[\phi_x].$ Now since $\mathbf{\mu}_p$ is unknown in the case of a real world dataset, CITE propose to approximate $\mathbf{\mu}_p$ by a weighted combination of $m$ samples in a subset $Z\subseteq X^{(2)}.$ The approximation is then given by $\mathbf{\mu}_p(x) \approx \sum_{j:z_j \in Z} w_j k(z_j,x)$ with $w_j \geq 0$ is a weight of the sample $z_j \in X^{(2)}.$ Therefore the key to minimizing the sample MMD is to choose the most appropriate set of prototypes $Z \subseteq X^{(2)}$ and corresponding non-negative weights $w_j.$

Indexing the elements in $X^{(2)}$ from 1 to $n^{(2)} = |X^{(2)}|,$ and for any $Z \subseteq X^{(2)},$ denote $L_Z \subseteq [n^{(2)}] = \{1,2,\cdots,n^{(2)} \}$ the set containing its indices. We define the function:

\begin{equation}
l(w) = w^T\mathbf{\mu}_p - \frac{1}{2}w^TKw
\end{equation}

where $K_{i,j} = k(y_i,y_j)$ and $\mu_{p,j} = \frac{1}{n^{(1)}}\sum_{x_i \in X^{(1)}}k(x_i,y_j)$ for any $y_j \in X^{(2)}$ is the estimation of the mean $\mu_p$. Our previously described goal of selecting the best subset $Z$ can thus be formulated as follows: We look to find an index set $L_Z$ of cardinality $|L_Z|=m$ with a corresponding weights vector $w$ such that the function $f:2^{[n^{(2)}]} \rightarrow \mathbb{R}^+$ defined by

\begin{equation}
f(L_Z) \equiv \max_{w: supp(w) \in L_Z, w\geq 0} l(w)
\end{equation}

is maximized. Note that here $supp(w) = \{j : w_j>0 \}.$ The maximizer for the set $L_Z$ is denoted by $\zeta^{(L_Z)}.$ This leads us to the presentation of the algorithm, which aims to accomplish this task.

[Back](#oth-bullet)

## 3) The ProtoDash algorithm

The ProtoDash algorithm proposed in [[1]](#first_ref) is shown in the image below.

It can be understood as follows. First, the algorithm takes as inputs a specific number of prototypes to select $m$, a target dataset $X^{(1)}$ and a source dataset $X^{(2)}.$ From the source dataset $X^{(2)}$ the algorithm selects the observation $j^*$ for which the gradient of our previously described objective function $\nabla l(w)_{j^*}$ is the largest from all the set of candidate observations. Having selected this best element $j^*$, the optimal weights vector $\zeta^{L\cup\{j^*\}}$ is computed so as to maximize $f(L\cup{j^*})$. This procedure is iteratively repeated until $|L|=m$ or the lower bound $\epsilon$ on the increase in $f(\cdot)$ is crossed.

The appeal of this algorithm is that it offers a single framework in which to identify both representative observations and outliers since it assigns non-negative weights to all selected prototypes, allowing to rank them in order of importance. Moreover, it also lets us identify representative prototypes for a dataset $A$ from a distinct dataset $B$, with potentially useful applications to settings with covariate shift, for example in the case of massive datasets of real world time series. Finally, the ProtoDash algorithm offers a two orders of magnitude computational speedup over the ProtoGreedy algorithm (an extension of the algorithm proposed in [[3]](#third_ref)) while still being demonstrated to maintain prototype selection quality.

[Back](#oth-bullet)

## 4) ProtoDash Time Complexity Analysis

The breakdown of the ProtoDash algorithm's time complextiy is as follows.
 - In order to run the ProtoDash algorithm we first need to calculate the mean inner product of $X^{(1)}$ with every instance in $X^{(2)}.$ For large sized datasets this operation can be parallized and thus sped up, but its theoretical time complexity nonetheless is $\mathcal{O}(n^{(1)}n^{(2)}),$ where $n^{(i)}$ denotes the number of observations in dataset $X^{(i)}$ for $i=1,2.$

 - For the kernel matrix $K,$ the time complexity of building it is $\mathcal{O}(mn^{(2)})$ as inner products need to be computed within $X^{(2)}$ for each of the $m$ iterations of the algorithm.

 - The selection of the next best observation at iteration $i$ necessitates to evaluate the gradient for each one of $(n^{(2)} - i - 1)$ observations with each evaluation having time complexity $\mathcal{O}(i),$ since it requires calculating the inner product with the previous $i-1$ prototypes. Therefore finding $m$ such prototypes has time complexity $\mathcal{O}(m^2n^{(2)}).$

 - Furthermore, for each iteration $i$ we need to solve a quadratic optimisation problem, achievable in $\mathcal{O}(i^3).$ This step therefore has time complexity $\mathcal{O}(m^4).$

Therefore, combining the time complexity for each step we have that the ProtoDash algorithm has time complexity $\mathcal{O}(n^{(2)}(n^{(1)}+m^2)+m^4).$

[Back](#oth-bullet)

## 5) PySpark implementation of the ProtoDash algorithm

In this section I present my implementation of the ProtoDash algorithm using PySpark.

First, I import relevant modules.

```python
# relevant imports
import time
import mnist
import numpy as np
import pandas as pd
from pyspark.mllib.linalg import DenseVector
import pyspark.sql.functions as F
from pyspark.sql.types import ArrayType, FloatType, StructType, StructField, LongType
from qpsolvers import solve_qp
from cvxopt.solvers import qp
from cvxopt import matrix, spmatrix
import matplotlib.pyplot as plt
from textwrap import wrap

import warnings
warnings.filterwarnings("ignore")
```

My implementation of the algorithm relies on the following set of helper functions.

```python
def create_vec_rdd(X, part=4):
    """
    Function returning a DenseVector RDD from a dataset X.

    Args:
    -X: a dataset with rows corresponding to observations and columns corresponding to features.
    -part: n of partitions.

    Returns:
    the RDD for X
    """
    X_rdd = (sc.parallelize(X, part)
             .map(lambda x: DenseVector(x))
             .zipWithIndex())
    return X_rdd

def mean_inner_product(inp,sigma,n):
    """
    Function computing the gaussian kernel inner product of a vector in Y vs.
    a vector in X, divided by n the total number of observations in X.
    """
    index_1 = inp[0][1]
    inner_product = float(np.exp(inp[0][0].squared_distance(inp[1][0])/(-2*sigma**2))/n)
    return (index_1, inner_product)

def inner_product(inp,sigma):
    """
    Function computing the gaussian kernel inner product of a vector vs.
    another.
    """
    index_1 = inp[0][1]
    index_2 = inp[1][1]
    inner_product = float(np.exp(inp[0][0].squared_distance(inp[1][0])/(-2*sigma**2)))
    return (index_1, [(index_2, inner_product)])

def weighted_sum(inp,w_arr):
    """
    compute the weighted sum of matrix values for a set of indices and weights.
    Note it is fine using a list comprehension here since the number of prototypes m << |X^{(1)}|.
    """
    return float(np.sum(np.array([x[1] for x in inp])*w_arr))

def udf_weighted_sum(w_arr):
    """
    UDF instance of the weighted_sum function.
    """
    return F.udf(lambda l: weighted_sum(l,w_arr))

def merge_lists(x, y):
    """
    merge lists.
    """
    return sorted(x+y, key=lambda tup: tup[0])

# Create UDF corresponding to merge_lists function.
DType = ArrayType(StructType([StructField("_1", LongType()), StructField("_2", FloatType())]))
udf_merge_lists = F.udf(merge_lists, DType)

def optimize(K, u, opt_w0, init_val, max_w=10000):
    """
    Function solving quadratic optimization problem posed by l(w).

    Args:
    -K: inner product matrix
    -u: mean inner product of each prototype
    -opt_w0: initial weights vector
    -init_val: starting run
    -max_w: an upper bound on weight value

    Returns:
    -weights and the objective value
    """
    dim = u.shape[0]
    low_b = np.zeros((dim, 1))
    upper_b = max_w * np.ones((dim, 1))
    x_init = np.append( opt_w0, init_val / K[dim-1, dim-1] )
    G = np.vstack((np.identity(dim), -1*np.identity(dim)))
    h = np.vstack((upper_b, -1*low_b))

    # solve constrained quadratic problem
    soltn = solve_qp(K, -u, G, h, A = None, b = None , solver = 'cvxopt' , initvals = x_init )

    # calculate the objective function value for optimal solution
    x_sol = soltn.reshape( soltn.shape[0], 1 )
    q = - u.reshape( u.shape[0], 1 )
    obj_value = 1/2 * np.matmul(np.matmul(x_sol.T, K), x_sol) + np.matmul(q.T, x_sol)

    return ( soltn, obj_value[0,0] )
```

Below is the ProtoDash algorithm.

```python
def ProtoDashSpark(X,Y,m,sigma,partitions=20,verbose=True):
    """
    PySpark implementation of the ProtoDash algorithm derived in:

    Gurumoorthy, Karthik S., et al. ,
    "Efficient Data Representation by Selecting Prototypes with Importance Weights."
    2019 IEEE International Conference on Data Mining (ICDM). IEEE, 2019.

    Args:
    -X (RDD of indexed DenseVector rows): Target dataset/ the dataset to be represented.
    -Y (RDD of indexed DenseVector rows): Source dataset/ the dataset to select prototypes from.
    -m (integer): total number of prototypes to select.
    -sigma (strictly positive float): gaussian kernel parameter.
    -partitions (integer): number of RDD partitions to compute inner product RDDs with.
    -verbose (boolean): whether or not to print the cumulative number of prototypes selected at each iteration.

    Returns:
    -L (integer list): the set of indices corresponding to selected prototypes.
    -w (float list): the optimal set of weights corresponding to each selected prototype.
    """

    # initialise algorithm starting time
    start_time = time.time()

    # get count of observations in X and Y
    n_Y = Y.count()
    n_X = X.count()

    # build mu DataFrame
    mu_df = (Y.cartesian(X)
              .map(lambda x: mean_inner_product(x,sigma,n_X))
              .reduceByKey(lambda x,y: x+y)
              .toDF(["obs","mu"]))

    # initialise key variables
    L = np.zeros(m, dtype=int)        # set of prototype indices L
    w = np.zeros(m, dtype=float)      # set of optimal prototype weights
    f_eval = np.zeros(m, dtype=float) # set of the f(w) eval. at prototype selection
    n_L = 0                           # count of prototypes selected so far

    # find the index corresponding to the maximum mu value
    max_grad_0 = mu_df.orderBy(F.desc("mu")).limit(1).collect()[0]

    # collect values
    L[n_L] = max_grad_0.obs
    w[n_L] = max_grad_0.mu
    f_eval[n_L] = 1/2 * max_grad_0.mu ** 2
    n_L += 1

    # select the row of Y corresponding to the first chosen index
    Y_row_j0 = Y.filter(lambda x: x[1]==L[:n_L]).collect()[0]

    # take its inner product with all rows of Y to build the starting K dataframe
    K_init_df = (Y.map(lambda x: inner_product((x,Y_row_j0),sigma))
                  .toDF(["obs","K"]))

    # join mu and K dataframes
    join_df = (mu_df.join(K_init_df, "obs")
                    .repartition(partitions))

    # cache join_df as it is reused often
    join_df.cache()

    # compute the new gradient vector
    grad_df = (join_df.withColumn("K_weighted", udf_weighted_sum(w[:n_L])(F.col("K")))
                      .withColumn("grad", F.col("mu") - F.col("K_weighted"))
                      .select("obs","grad"))

    # instantiate list of iteration runtimes
    iteration_times = [time.time() - start_time]

    # begin while loop
    while n_L < m:

        # instantiate iteration start time
        iteration_start_time = time.time()

        # remove the rows that have an index already included in L
        grad_df = grad_df.filter(~grad_df.obs.isin([int(x) for x in L[:n_L]]))

        # find the row that has the maximum value in the filtered gradient vector
        argmax_grad = grad_df.orderBy(F.desc("grad")).limit(1).collect()[0]

        # update L
        L[n_L] = argmax_grad.obs

        # select the row of Y corresponding to the chosen index
        Y_row_j = Y.filter(lambda x: x[1]==L[n_L]).collect()[0]

        # take its inner product with all rows of Y to build new K
        K_int_df = (Y.map(lambda x: inner_product((x,Y_row_j),sigma))
                     .toDF(["obs","new_K_col"]))

        # add new K col to previous K col
        join_df = (join_df.join(K_int_df, "obs")
                          .withColumn("K_merged", udf_merge_lists(F.col("K"), F.col("new_K_col")))
                          .select("obs", "mu", "K_merged")
                          .withColumnRenamed("K_merged", "K"))

        # cache new joined_df
        join_df.cache()

        # increment n_L
        n_L +=1

        # sort L
        L[:n_L] = sorted(L[:n_L])

        if verbose is True and n_L % 5==0:
                print( "Prototypes selected - "+str(n_L) )

        # take max gradient val.
        max_grad = argmax_grad.grad

        # filter join dataframe for given indices in L
        filt_df = (join_df.filter(join_df.obs.isin([int(x) for x in L[:n_L]]))
                          .orderBy(F.col("obs").asc()))

        # take mu vector
        mu_arr = np.array(filt_df.select("mu").collect(), dtype=float)

        # take K matrix
        K_mat = np.array(filt_df.rdd
                                 .map(lambda x: [y[1] for y in x[2]])
                                 .collect(), dtype=float )

        # find optimal weights for the index set L
        opt_res = optimize(K_mat, mu_arr, w[:n_L-1], max_grad)
        (w[:n_L], f_eval[n_L-1]) = opt_res[0], -opt_res[1]

        # compute gradient vector with new optimal weights
        grad_df = (join_df.withColumn("K_weighted", udf_weighted_sum(w[:n_L])(F.col("K")))
                          .withColumn("grad", F.col("mu") - F.col("K_weighted"))
                          .select("obs","grad"))

        # record iteration time in seconds
        iteration_times.append(time.time() - iteration_start_time)

    # tuple of indices and their corresponding weight, sorted by weight in descending order.
    res = sorted([(w[i],L[i]) for i in range(m)], key=lambda tup: -tup[0])

    # measure total algorithm runtime in seconds
    total_time = time.time() - start_time

    # return tuple of index set L and optimal weight set w, set of f_eval and list of runtimes.
    return res, f_eval, [iteration_times, total_time]
```

[Back](#oth-bullet)

## 6) Application of the ProtoDash algorithm on the MNIST dataset

In this section I demonstrate how the ProtoDash algorithm I have implemented in PySpark can be used in practice to efficiently identify prototypes in a large dataset.

To demonstrate this, I will be using the MNIST Dataset of Handwritten digits. This dataset - which can be found in [[6]](#fifth_ref) - consists of a training set of 60,000 labelled examples of handwritten digits and a test set of 10,000 similar labelled examples. Each of the MNIST observations consists of a 28$\times$28 float matrix.

```python
# collect MNIST train/test sets
train_images = np.array(mnist.train_images(), dtype='float')
train_labels = mnist.train_labels()

test_images = np.array(mnist.test_images(), dtype='float')
test_labels = mnist.test_labels()
```

Below is shown the shape of the MNIST dataset.

```python
print( train_images.shape )
print( test_images.shape )
```

    (60000, 28, 28)
    (10000, 28, 28)

Examples of plotted observations in the MNIST dataset are shown below:

```python
# take three MNIST observations
first_image = train_images[0]
second_image = train_images[1]
third_image = train_images[2]

# plot the three observations
fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(8,8))
ax1.imshow(first_image)
ax2.imshow(second_image)
ax3.imshow(third_image)
fig.tight_layout(rect=[0, 0.03, 1, 0.95])
plt.show()
```

## First Experiment: Using ProtoDash to accurately represent increasingly skewed datasets

### Experiment description

In my first experiment, I follow an approach proposed in [[1]](#first_ref) on the MNIST dataset, that seeks to show that given a specific target MNIST dataset $X^{(1)}$ featuring an increasingly over-represented digit and a completely distinct source MNIST dataset $X^{(2)}$ in which no digit is oversampled, the ProtoDash algorithm can effectively identify prototypes in $X^{(2)}$ that capture the increased skewness in $X^{(1)}.$

The design of this experiment is as follows.
Using the MNIST training dataset of 60000 observations, I build multiple target datasets, each consisting of 5420 observations (note that 5420 is the total number of observations of the least frequent digit in MNIST). Each such target set I create is increasingly skewed, for percentages $p=30,50,70,90,100.$ That is, for a given $p$ and a specified digit, I ensure that $p$% of the target dataset consists of this pre-specified digit, with the remaining observations consisting of other randomly selected handwritten digits. That means that I create 5 target datasets, one for each value of $p$, and each with a fraction $p$% consisting of this digit. For example, if $p=30$ and I pre-specified the over-represented digit to be 5, then I create a target dataset in which $30$% of the observations correspond to the digit 5, while the remaining observations are other digits. Once I have finished building these target datasets, I randomly select 1500 observations from the MNIST test set, in order to form my source dataset.

I use this source dataset to sample 50 prototypes for each target dataset, and look to see if the ProtoDash algorithm successfully captures the increasing skewness in the target set.

To undertake this experiment I first implement the function below.

```python
def create_target_set(labels, images, digit, target_n, percentage):
    """
    This function creates a MNIST image dataset in which a specified percentage of the total observations
    correspond to a specific digit, while the remaining observations correspond to other randomly
    chosen digits.

    Args:
    -labels: the digit label for each MNIST image.
    -images: the MNIST image.
    -digit: a digit between 0 and 9.
    -target_n: the number of total observations required in the target dataset.
    -percentage: the percentage of images in the target dataset that correspond to the specified digit.

    Returns:
    -the target images.

    """

    # take integer number of obs. corresponding to digit
    n_dig = int( np.floor( percentage * target_n ) )

    # get indices corresponding to digit
    idx = np.where(labels == digit)[0]

    # reduce indices to specific %
    idx_red = idx[:n_dig]

    # slice images with index and reshape
    target_set_dig = images[idx_red,:]
    target_set_dig = np.reshape( target_set_dig, (target_set_dig.shape[0], 28*28))

    # get remaining indices
    rem = target_n - n_dig
    rem_ind = np.setdiff1d( np.arange(len(labels)), idx_red )[:rem]

    # fill the remaining observations with images corresponding to other digits
    target_set_non_dig = images[rem_ind]
    target_set_non_dig = np.reshape( target_set_non_dig, (target_set_non_dig.shape[0], 28*28))

    # create the dataset
    target_set = np.vstack((target_set_non_dig,target_set_dig))

    # shuffle it
    arr = np.arange(target_n)
    np.random.shuffle(arr)

    return target_set
```

I then fix key ProtoDash parameters.

```python
m = 25           # number of prototypes for ProtoDash to select
part = 6         # number of Pyspark RDD partitions to use
sigma = 50       # gaussian kernel parameter
```

Note that the parameter $\sigma$ can be selected methodically using cross-validation, which is beyond the scope of this project. Further resource on effective cross-validation to choose $\sigma$ can be found in [[5]](#sixth_ref).

Now I fix some important variables for this experiment.

```python
digit = 8            # the digit to be represented at different levels of percentage in the target dataset X_1
n_1 = 5420           # the number of observations in X_1
n_2 = 1500           # the number of observations in X_2
percentages = [.3, .5, .7, .9, 1.] # the percentages of X_1 that will correspond to the chosen digit
```

I then iteratively create the target datasets and run the ProtoDash algorithm to select $m=50$ prototypes from my source dataset to represent them.

```python
# p = 30

# # select the target datasets
# target_set = create_target_set(train_labels, train_images, digit, n_1, p)

# # convert target and source datasets to RDDs
# target_rdd = create_vec_rdd(target_set, part)
# source_rdd = create_vec_rdd(source_set, part)

# # collect the indices of m prototypes along with their ascribed weight
# res, f = ProtoDashSpark(target_rdd, source_rdd, 10, 50, partitions=part,verbose=True)[:2]
```

```python
# res

# plt.imshow( np.reshape(source_set[res[0][1]],(28,28)) )
# plt.show()

# #     ax1[i].imshow( np.reshape( source_set[exp_1_res_list[i][0][1],:], (28, 28)) )
# #     ax1[i].get_xaxis().set_ticks([])
# #     ax1[i].get_yaxis().set_ticks([])

```

```python
# list of experiment results
exp_1_res_list = []

# list of f_eval sequences
exp_1_f_eval_list = []

# set source dataset and labels
source_set = np.reshape( test_images[:n_2], (n_2, 28 * 28) )
source_labels = test_labels[:n_2]

# iterate through percentages
for p in percentages:

    # print percentage
    print (str(p*100)+"%")

    # select the target datasets
    target_set = create_target_set(train_labels, train_images, digit, n_1, p)

    # convert target and source datasets to RDDs
    target_rdd = create_vec_rdd(target_set, part)
    source_rdd = create_vec_rdd(source_set, part)

    # collect the indices of m prototypes along with their ascribed weight
    res, f = ProtoDashSpark(target_rdd, source_rdd, m, sigma, partitions=part,verbose=True)[:2]

    # collect the results
    exp_1_res_list.append(res)
    exp_1_f_eval_list.append(f)
```

    30.0%

### Experiment results

Once this is completed, for each target dataset I plot the top 5 prototypes selected from my source dataset by the ProtoDash algorithm to represent each of my skewed target datasets.

The ranking of ProtoDash prototypes is done in terms of the non-negative weight assigned to each prototype by the ProtoDash algorithm. That is, the prototype with the highest weight is ranked as the observation of the source dataset that is most representative of the target dataset. In the plot below for each skewed target dataset the 5 prototypes are shown in descending weight (i.e. relevance) from top to bottom.

```python
fig, (ax1, ax2, ax3, ax4, ax5) = plt.subplots(5, 5, figsize=(12,10))

for i in range(len(percentages)):
    ax1[i].imshow( np.reshape( source_set[exp_1_res_list[i][0][1],:], (28, 28)) )
    ax1[i].get_xaxis().set_ticks([])
    ax1[i].get_yaxis().set_ticks([])

    ax2[i].imshow( np.reshape( source_set[exp_1_res_list[i][1][1],:], (28, 28)) )
    ax2[i].get_xaxis().set_ticks([])
    ax2[i].get_yaxis().set_ticks([])

    ax3[i].imshow( np.reshape( source_set[exp_1_res_list[i][2][1],:], (28, 28)) )
    ax3[i].get_xaxis().set_ticks([])
    ax3[i].get_yaxis().set_ticks([])

    ax4[i].imshow( np.reshape( source_set[exp_1_res_list[i][3][1],:], (28, 28)) )
    ax4[i].get_xaxis().set_ticks([])
    ax4[i].get_yaxis().set_ticks([])

    ax5[i].imshow( np.reshape( source_set[exp_1_res_list[i][4][1],:], (28, 28)) )
    ax5[i].get_xaxis().set_ticks([])
    ax5[i].get_yaxis().set_ticks([])

ax5[0].set_xlabel("$p$ = 30%", fontsize=20)
ax5[1].set_xlabel("$p$ = 50%", fontsize=20)
ax5[2].set_xlabel("$p$ = 70%", fontsize=20)
ax5[3].set_xlabel("$p$ = 90%", fontsize=20)
ax5[4].set_xlabel("$p$ = 100%", fontsize=20)

fig.suptitle("\n".join(wrap("Top five prototypes selected by ProtoDash, given target dataset $X_1$ with increasing percentage of total observations corresponding to the digit 8", 60)),
            fontsize=20)

plt.show()
```

From a simple visual inspection of this plot we can already observe that the ProtoDash algorithm is highly successful in representing an increasingly skewed dataset based purely on a completely distinct, un-skewed dataset. Indeed, as the percentage of skewness $p$ increases here from 30% to 100%, we can see that the ProtoDash algorithm adapts very quickly by increasingly selecting observations that correspond to the oversampled digit. Furthermore, from a cursory inspection it can be seen that the quality of the top prototype selected by ProtoDash gradually improves with increasing $p.$

In the Pandas DataFrame below I show for each actual percentage $p$ the percentage of $m$ prototypes selected by ProtoDash that correspond to our chosen oversampled digit. Corroborating our previous visual observations we can see that the algorithm increasingly selects our chosen digit from the source dataset as the target dataset becomes increasingly skewed towards it, demonstrating its ability to represent one dataset choosing prototypes from another one with a different distribution of observations.

```python
prototype_p = [np.sum(source_labels[np.array([x[1] for x in exp_1_res_list[i]])]==digit)/m * 100 for i in range(len(percentages))]

pd.DataFrame({"p (%)":np.array(percentages)*100, "% of digit selected by ProtoDash":prototype_p})
```

      p (%)
      % of digit selected by ProtoDash

      0
      30.0
      28.0

      1
      50.0
      48.0

      2
      70.0
      64.0

      3
      90.0
      72.0

      4
      100.0
      80.0

In the figure below, for each skewed target dataset I plot the ProtoDash objective function for increasing prototype dataset size $m.$ For all considered target datasets the objective function is strictly increasing at a decreasing rate, demonstrating that the ProtoDash algorithm satisfies the property of diminishing returns of the scoring function. That is, given a set $C$, two subsets $A \subset B \subset C$ and an observation $j \not \in B,$ we have that $f(A\cup\{j\}) - f(A) \geq f(B\cup\{j\}) - f(B)$ holds. This property is known as _submodularity_, and it can be readily seen from the concavity of the objective function plotted below. Furthermore, we can see that as the target dataset becomes increasingly skewed, the ProtoDash objective function reaches increasingly high values, demonstrating that it becomes increasingly easy for it to identify prototypes that best represent this increasingly uniform target set. This proves the algorithm is indeed responsive to and effectively adapting to the increased skewness in the target data.

```python
fig, ax = plt.subplots(figsize=(8,7))

for i in range(len(percentages)):
    ax.plot( exp_1_f_eval_list[i], 'o-' )

ax.tick_params(axis="both", which="major", labelsize=20)
ax.set_xlabel("Number of prototypes selected $m$", fontsize=20)
ax.set_ylabel("ProtoDash objective func. value ", fontsize=20)
plt.title("ProtoDash objective function value for increasing number of prototypes $m$",fontsize=20)
plt.grid()
plt.legend(["$p =$"+str(int(p*100))+"%" for p in percentages],fontsize=20)
plt.show()
```

[Back](#oth-bullet)

## Second experiment: Testing the speed of ProtoDash on increasingly large target datasets

### Experiment description

In this second experiment I examine the speed of my implementation of the ProtoDash algorithm.

The experiment is designed as follows. By iteratively increasing $n_1$ (i.e. the number of observations in the target dataset $X_1$), I measure the runtime in seconds of the ProtoDash algorithm given the task of selecting $m=10$ representative prototypes from a completely distinct source dataset $X_2$ of size $n_2=1500$ (note this is the source dataset size used throughout [[1]](#first_ref)).

To undertake this experiment I start by fixing basic ProtoDash settings.

```python
m = 10      # number of prototypes for ProtoDash to select
part = 8    # number of Pyspark RDD partitions to use
sigma = 200 # gaussian kernel parameter
```

I choose to test the following sizes for $n_1:$ 10, 50, 100, 500, 1000, 5000, 10000, 50000, to provide a clear idea of how ProtoDash runtime evolves for target datasets with sizes of different orders of magnitude.

```python
# list of X_1 dataset sizes
n_1_list = [10, 50, 100, 500, 1000, 5000, 10000, 50000]

# size of X_2 dataset
n_2 = 1500
```

I then iterate over the values of $n_1$ and collect the ProtoDash runtimes for each one.

```python
# result list
exp_2_res_list = []

# loop through n_1 sizes
for n_1 in n_1_list:

    # print n_1
    print( "n_1 = " + str(n_1) )

    # get datasets
    X_1 = np.reshape( train_images[:n_1], (n_1, 28 * 28 ))
    X_2 = np.reshape( train_images[n_1:n_1+n_2], (n_2, 28 * 28))

    # create RDD datasets
    Y = create_vec_rdd(X_2, part)
    X = create_vec_rdd(X_1, part)

    # apply ProtoDash algorithm
    res = ProtoDashSpark(X,Y,m,sigma,part)

    # collect results
    exp_2_res_list.append(res)
```

    n_1 = 10
    Prototypes selected - 5
    Prototypes selected - 10
    n_1 = 50
    Prototypes selected - 5
    Prototypes selected - 10
    n_1 = 100
    Prototypes selected - 5
    Prototypes selected - 10
    n_1 = 500
    Prototypes selected - 5
    Prototypes selected - 10
    n_1 = 1000
    Prototypes selected - 5
    Prototypes selected - 10
    n_1 = 5000
    Prototypes selected - 5
    Prototypes selected - 10
    n_1 = 10000
    Prototypes selected - 5
    Prototypes selected - 10
    n_1 = 50000
    Prototypes selected - 5
    Prototypes selected - 10

### Experiment results

Below I plot the ProtoDash algorithm runtime in seconds as a function of the total number of observations $n_1$ in target dataset $X^{(1)}.$ It can be seen that for $n_1\geq 1000$ the total ProtoDash runtime appears to increase linearly with $n_1$, which is a result consistent with the previously described theoretical ProtoDash time complexity of $\mathcal{O}(n^{(1)}n^{(2)} + m^4 + m^2n^{(2)}).$ Indeed, given this theoretical time complexity, it can easily be seen that if we keep $m, n^{(2)}$ fixed but gradually increase $n^{(1)}$ until we have that $n^{(1)}>>m,n^{(2)},$ we end up with a time complexity of $\mathcal{O}(n^{(1)})$, i.e. a runtime linear in $n^{(1)}$ as in the plot below.

```python
fig, ax = plt.subplots(figsize=(8,7))
ax.plot( n_1_list, [res_i[2][1] for res_i in exp_2_res_list], 'o-', markersize=10)
ax.set_title("ProtoDash runtime for varying sizes of target dataset $X_1$", fontsize=20)
ax.set_xlabel("Number of observations in target dataset $X_1$", fontsize=20)
ax.set_ylabel("Time (in seconds)", fontsize=20)
ax.tick_params(axis="both", which="major", labelsize=20)
ax.grid()
plt.show()
```

The actual values of the ProtoDash algorithm runtime in seconds for increasing sizes of $n_1$ are given in the pandas DataFrame below.

```python
pd.DataFrame({"$n_1$": n_1_list, " ProtoDash Runtime (s)": [res_i[2][1] for res_i in exp_2_res_list]})
```

      $n_1$
      ProtoDash Runtime (s)

      0
      10
      46.728925

      1
      50
      19.237747

      2
      100
      17.501685

      3
      500
      14.753179

      4
      1000
      15.519634

      5
      5000
      23.136997

      6
      10000
      36.187381

      7
      50000
      125.088802

To show exactly which step of the ProtoDash algorithm takes up the largest fraction of its total runtime as $n_1$ increases, I plot below the ratio of the time taken to compute the mean inner-product vector $\mu$ (which as a reminder has theoretical time complexity $\mathcal{O}(n^{(1)}n^{(2)})$) with the total time taken to run the ProtoDash algorithm. I plot this ratio as a function of increasing $n^{(1)}$.

From the plot we can readily see that as $n^{(1)}$ increases, the computation of the mean inner product vector $\mu$ (with time complexity $\mathcal{O}(n^{(1)}n^{(2)})$) takes up an increasingly large fraction of the total ProtoDash runtime, demonstrating that it is by far the most costly operation undertaken in ProtoDash. As $n^{(1)}$ or $n^{(2)}$ increase asymptotically to infinity this is always true, simply because of the fact that for prototype selection we typically seek a fixed $m << n^{(1)}, n^{(2)},$ so the time complexity as $n^{(1)},n^{(2)}$ increase must become $\mathcal{O}(n^{(1)}n^{(2)}).$

```python
fig, ax = plt.subplots(figsize=(8,7))
ax.plot( n_1_list, [res_i[2][0][0]/res_i[2][1] * 100 for res_i in exp_2_res_list], 'o-', markersize=10)
ax.set_title("Time taken computing $\mu$ divided by total ProtoDash runtime", fontsize=20)
ax.set_xlabel("Number of observations in target dataset $X_1$", fontsize=20)
ax.set_ylabel("%", fontsize=20)
ax.tick_params(axis="x", rotation = 30)
ax.tick_params(axis="both", which="major", labelsize=20)
ax.grid()
plt.show()
```

Now in the figure below I plot the runtime in seconds of the ProtoDash algorithm as a function of the number of prototypes to select ($m$), for different values of $n_1$. Note that in this plot the legend corresponds to the different values of $n_1$.

It is apparent here that once the mean inner product vector $\mu$ has been computed (this is done at the very start of the algorithm, prior to any prototype being selected), the runtimes for the different target dataset sizes are largely comparable, with the curve for each sharing the same slope approximately constant slope for increasing $m$. This demonstrates that once $\mu$ has been computed and thus the most computationally costly step has been completed, selecting any given $m$ will take the same runtime regardless of the target dataset size.

```python
fig, ax = plt.subplots(figsize=(8,7))
for i in range(len(n_1_list)):
    ax.plot( range(1,11), np.cumsum( exp_2_res_list[i][2][0] ), 'o-', markersize=10)
ax.set_title("Time taken selecting $m$ prototypes for varying sizes of dataset $X_1$", fontsize=20)
ax.set_xlabel("Number of prototypes $m$ selected", fontsize=20)
ax.set_ylabel("Time taken (in seconds)", fontsize=20)
ax.legend(n_1_list)
ax.tick_params(axis="x", rotation = 30)
ax.tick_params(axis="both", which="major", labelsize=20)
ax.grid()
plt.show()
```

[Back](#oth-bullet)

## 7) Conclusion

In this notebook I have presented the problem statement underpinning an efficient prototype selection algorithm called $\textit{ProtoDash}$ and implemented the algorithm itself using Apache Spark via the API PySpark. I have shown that my implementation of the algorithm is successfully able to represent a target dataset based on a completely different source dataset, by demonstrating the application of my implemented ProtoDash algorithm on the MNIST dataset. Furthermore, I have examined the performance of my implementation of the $\textit{ProtoDash}$ algorithm in terms of speed, effectively showing that it matches the theoretical time complexity of $\textit{ProtoDash}$ laid out by [[1]](#first_ref).

[Back](#oth-bullet)

## References

[1] Gurumoorthy, Karthik S., et al. "Efficient Data Representation by Selecting Prototypes with Importance Weights." 2019 IEEE International Conference on Data Mining (ICDM). IEEE, 2019.

[2] Gretton, Arthur, et al. "A kernel method for the two-sample-problem." Advances in neural information processing systems. 2007.

[3] Kim, Been, Rajiv Khanna, and Oluwasanmi O. Koyejo. "Examples are not enough, learn to criticize! criticism for interpretability." Advances in neural information processing systems. 2016.

[5] Friedman, J., Hastie, T., & Tibshirani, R. (2001). The elements of statistical learning (Vol. 1, No. 10). New York: Springer series in statistics.

[6] The MNIST Database of Handwritten Digits: http://yann.lecun.com/exdb/mnist/

[7] Koh, P. W., & Liang, P. (2017, August). Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 (pp. 1885-1894). JMLR. org.

[Back](#oth-bullet)


```python
# IMPORT LIBRARIES

import os
import sys
import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
from pandas import set_option
import matplotlib
import matplotlib.pyplot as plt
from matplotlib.font_manager import FontProperties
import seaborn as sns
%matplotlib inline

# from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import MinMaxScaler

# import performance metrics/measures
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, classification_report

# from imblearn.under_sampling import KMeansSMOTE
from imblearn.over_sampling import SMOTE
from imblearn.pipeline import make_pipeline
```

    /usr/local/lib/python3.6/dist-packages/sklearn/externals/six.py:31: FutureWarning: The module is deprecated in version 0.21 and will be removed in version 0.23 since we've dropped support for Python 2.7. Please rely on the official version of six (https://pypi.org/project/six/).
      "(https://pypi.org/project/six/).", FutureWarning)
    /usr/local/lib/python3.6/dist-packages/sklearn/utils/deprecation.py:144: FutureWarning: The sklearn.neighbors.base module is  deprecated in version 0.22 and will be removed in version 0.24. The corresponding classes / functions should instead be imported from sklearn.neighbors. Anything that cannot be imported from sklearn.neighbors is now part of the private API.
      warnings.warn(message, FutureWarning)

```python
# VARIABLES

# created a .csv version of the original dataset (.xls)
INPUT_PATH = "https://raw.githubusercontent.com/robertofranceschi/default-credit-card-prediction/master/dataset/default%20of%20credit%20card%20clients.csv"

# SCALER = 'std'  # else 'minmax'

SHOW_FIGURE = False

# preprocessing
FEATURE_SELECTION = False  # if 'True' Feature Selection will be applied (i.e. 'BILL_AMT2', 'BILL_AMT3', 'BILL_AMT4', 'BILL_AMT5', 'BILL_AMT6' will be removed)
APPLY_PCA = False                # if 'True' Principal Component Analysis will be applied
APPLY_OVERSAMPLING = False       # if 'True' SMOTE Oversampling will be applied

RANDOM_STATE = 42   # fixed in order to have comparable results

# Pandas options
set_option('display.max_colwidth', None)
set_option('precision', 2)
```

```python
# read input
data = pd.read_csv(INPUT_PATH)
data.head()
```

      ID
      LIMIT_BAL
      SEX
      EDUCATION
      MARRIAGE
      AGE
      PAY_0
      PAY_2
      PAY_3
      PAY_4
      PAY_5
      PAY_6
      BILL_AMT1
      BILL_AMT2
      BILL_AMT3
      BILL_AMT4
      BILL_AMT5
      BILL_AMT6
      PAY_AMT1
      PAY_AMT2
      PAY_AMT3
      PAY_AMT4
      PAY_AMT5
      PAY_AMT6
      default.payment.next.month

      0
      1
      20000.0
      2
      2
      1
      24
      2
      2
      -1
      -1
      -2
      -2
      3913.0
      3102.0
      689.0
      0.0
      0.0
      0.0
      0.0
      689.0
      0.0
      0.0
      0.0
      0.0
      1

      1
      2
      120000.0
      2
      2
      2
      26
      -1
      2
      0
      0
      0
      2
      2682.0
      1725.0
      2682.0
      3272.0
      3455.0
      3261.0
      0.0
      1000.0
      1000.0
      1000.0
      0.0
      2000.0
      1

      2
      3
      90000.0
      2
      2
      2
      34
      0
      0
      0
      0
      0
      0
      29239.0
      14027.0
      13559.0
      14331.0
      14948.0
      15549.0
      1518.0
      1500.0
      1000.0
      1000.0
      1000.0
      5000.0
      0

      3
      4
      50000.0
      2
      2
      1
      37
      0
      0
      0
      0
      0
      0
      46990.0
      48233.0
      49291.0
      28314.0
      28959.0
      29547.0
      2000.0
      2019.0
      1200.0
      1100.0
      1069.0
      1000.0
      0

      4
      5
      50000.0
      1
      2
      1
      57
      -1
      0
      -1
      0
      0
      0
      8617.0
      5670.0
      35835.0
      20940.0
      19146.0
      19131.0
      2000.0
      36681.0
      10000.0
      9000.0
      689.0
      679.0
      0

```python
# Dataset balanced?
if SHOW_FIGURE :
    font = FontProperties()
    font.set_family(['Times New Roman', 'serif'])
    font.set_size(14)

    plt.figure(figsize = (6,6))
    sns.countplot('default.payment.next.month', data=data, palette=['steelblue','crimson'])
    # plt.title('Class Distribution Histogram', fontsize=14)
    plt.xticks([0,1],['Not default','Default'],fontproperties=font)
    plt.ylabel('# of samples', fontproperties=font)
    plt.xlabel('')
    # plt.show()
    plt.savefig('Fig - Class Distribution Histogram.png')
```

```python
# Check data types
data.dtypes
```

    ID                              int64
    LIMIT_BAL                     float64
    SEX                             int64
    EDUCATION                       int64
    MARRIAGE                        int64
    AGE                             int64
    PAY_0                           int64
    PAY_2                           int64
    PAY_3                           int64
    PAY_4                           int64
    PAY_5                           int64
    PAY_6                           int64
    BILL_AMT1                     float64
    BILL_AMT2                     float64
    BILL_AMT3                     float64
    BILL_AMT4                     float64
    BILL_AMT5                     float64
    BILL_AMT6                     float64
    PAY_AMT1                      float64
    PAY_AMT2                      float64
    PAY_AMT3                      float64
    PAY_AMT4                      float64
    PAY_AMT5                      float64
    PAY_AMT6                      float64
    default.payment.next.month      int64
    dtype: object

```python
# Check missing values: no as can be also seen with data.info()
data.isnull().values.any()
```

    False

```python
# Summary Statistics
data.describe()
```

      ID
      LIMIT_BAL
      SEX
      EDUCATION
      MARRIAGE
      AGE
      PAY_0
      PAY_2
      PAY_3
      PAY_4
      PAY_5
      PAY_6
      BILL_AMT1
      BILL_AMT2
      BILL_AMT3
      BILL_AMT4
      BILL_AMT5
      BILL_AMT6
      PAY_AMT1
      PAY_AMT2
      PAY_AMT3
      PAY_AMT4
      PAY_AMT5
      PAY_AMT6
      default.payment.next.month

      count
      30000.00
      30000.00
      30000.00
      30000.00
      30000.00
      30000.00
      30000.00
      30000.00
      30000.00
      30000.00
      30000.00
      30000.00
      30000.00
      30000.00
      3.00e+04
      30000.00
      30000.00
      30000.00
      30000.00
      3.00e+04
      30000.00
      30000.00
      30000.00
      30000.00
      30000.00

      mean
      15000.50
      167484.32
      1.60
      1.85
      1.55
      35.49
      -0.02
      -0.13
      -0.17
      -0.22
      -0.27
      -0.29
      51223.33
      49179.08
      4.70e+04
      43262.95
      40311.40
      38871.76
      5663.58
      5.92e+03
      5225.68
      4826.08
      4799.39
      5215.50
      0.22

      std
      8660.40
      129747.66
      0.49
      0.79
      0.52
      9.22
      1.12
      1.20
      1.20
      1.17
      1.13
      1.15
      73635.86
      71173.77
      6.93e+04
      64332.86
      60797.16
      59554.11
      16563.28
      2.30e+04
      17606.96
      15666.16
      15278.31
      17777.47
      0.42

      min
      1.00
      10000.00
      1.00
      0.00
      0.00
      21.00
      -2.00
      -2.00
      -2.00
      -2.00
      -2.00
      -2.00
      -165580.00
      -69777.00
      -1.57e+05
      -170000.00
      -81334.00
      -339603.00
      0.00
      0.00e+00
      0.00
      0.00
      0.00
      0.00
      0.00

      25%
      7500.75
      50000.00
      1.00
      1.00
      1.00
      28.00
      -1.00
      -1.00
      -1.00
      -1.00
      -1.00
      -1.00
      3558.75
      2984.75
      2.67e+03
      2326.75
      1763.00
      1256.00
      1000.00
      8.33e+02
      390.00
      296.00
      252.50
      117.75
      0.00

      50%
      15000.50
      140000.00
      2.00
      2.00
      2.00
      34.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00
      22381.50
      21200.00
      2.01e+04
      19052.00
      18104.50
      17071.00
      2100.00
      2.01e+03
      1800.00
      1500.00
      1500.00
      1500.00
      0.00

      75%
      22500.25
      240000.00
      2.00
      2.00
      2.00
      41.00
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00
      67091.00
      64006.25
      6.02e+04
      54506.00
      50190.50
      49198.25
      5006.00
      5.00e+03
      4505.00
      4013.25
      4031.50
      4000.00
      0.00

      max
      30000.00
      1000000.00
      2.00
      6.00
      3.00
      79.00
      8.00
      8.00
      8.00
      8.00
      8.00
      8.00
      964511.00
      983931.00
      1.66e+06
      891586.00
      927171.00
      961664.00
      873552.00
      1.68e+06
      896040.00
      621000.00
      426529.00
      528666.00
      1.00

```python
# rename variable 'PAY_0' to 'PAY_1'
data.rename(columns={"PAY_0": "PAY_1"}, inplace=True)
# rename target variable: 'default.payment.next.month' to 'Default'
data.rename(columns={"default.payment.next.month": "Default"}, inplace=True)
# drop first attribute "ID"
data.drop('ID', axis = 1, inplace =True)
# Class label to category
data["Default"] = data["Default"].astype('category')
```

```python
data['MARRIAGE'].value_counts()
```

    2    15964
    1    13659
    3      323
    0       54
    Name: MARRIAGE, dtype: int64

```python
data['EDUCATION'].value_counts()
```

    2    14030
    1    10585
    3     4917
    5      280
    4      123
    6       51
    0       14
    Name: EDUCATION, dtype: int64

```python
# category '0' undocumented is deleted
data = data.drop(data[data['MARRIAGE']==0].index)
# we could also group the 0 category with 3:others
# data['MARRIAGE']=np.where(data['MARRIAGE'] == 0, 3, data['MARRIAGE'])

# categories 0, 5 and 6 are unknown and are deleted
data = data.drop(data[data['EDUCATION']==0].index)
data = data.drop(data[data['EDUCATION']==5].index)
data = data.drop(data[data['EDUCATION']==6].index)
# we could also group the categories together
# data['EDUCATION']=np.where(data['EDUCATION'] == 5, 4, data['EDUCATION'])
# data['EDUCATION']=np.where(data['EDUCATION'] == 6, 4, data['EDUCATION'])
# data['EDUCATION']=np.where(data['EDUCATION'] == 0, 4, data['EDUCATION'])
```

```python
if SHOW_FIGURE :
    # 1=graduate school, 2=university, 3=high school 4=others
    data['EDUCATION'].value_counts().plot(kind='bar', figsize=(10,6))
    # plt.title("Number of cars by make")
    plt.xticks([0,1,2,3],['University','Graduate\nSchool', 'High\nSchool', 'Others'],fontproperties=font,rotation=0)
    # plt.xlabel('Education level', fontproperties=font)
    plt.ylabel('# of clients', fontproperties=font)
    # plt.show()
    plt.savefig('Fig - Education Level barplot.png')
```

```python
if SHOW_FIGURE :
    # 1=married, 2=single, 3=others
    data['MARRIAGE'].value_counts().plot(kind='bar', figsize=(10,6))
    # plt.title("Number of cars by make")
    plt.xticks([0,1,2],['Single','Married', 'Others'],fontproperties=font,rotation=0)
    # plt.xlabel('Marital Status', fontproperties=font)
    plt.ylabel('# of clients', fontproperties=font)
    # plt.show()
    plt.savefig('Fig - Marital Status.png')
```

```python
# Payment delay description
data[['PAY_1', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6']].describe()
```

      PAY_1
      PAY_2
      PAY_3
      PAY_4
      PAY_5
      PAY_6

      count
      29601.00
      29601.00
      29601.00
      29601.00
      29601.00
      29601.00

      mean
      -0.01
      -0.13
      -0.16
      -0.22
      -0.26
      -0.29

      std
      1.12
      1.20
      1.20
      1.17
      1.14
      1.15

      min
      -2.00
      -2.00
      -2.00
      -2.00
      -2.00
      -2.00

      25%
      -1.00
      -1.00
      -1.00
      -1.00
      -1.00
      -1.00

      50%
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      75%
      0.00
      0.00
      0.00
      0.00
      0.00
      0.00

      max
      8.00
      8.00
      8.00
      8.00
      8.00
      8.00

```python
# REPAYMENT STATUS = -1=pay duly, 1=payment delay for one month, 2=payment delay for two months, ... 8=payment delay for eight months, 9=payment delay for nine months and above
# for att in ['PAY_1', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6']:
#   print(f"# {att} -------")
#   print(data[att].value_counts())
```

```python
# since PAY_n can take as values only -1,1,2,3,4,5,6,7,8,9
for att in ['PAY_1', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6']:
  # categories -2,-1 are grouped into a single class -1: pay duty
  filter = (data[att] == -2) | (data[att] == -1)
  data.loc[filter, att] = -1
  # print(data[att].unique())
  # moreover the category 0 is undocumented
  # so each category >= 0 has been updated by adding 1
  data[att] = data[att].astype('int64')
  filter = (data[att] >= 0)
  data.loc[filter, att] = data.loc[filter, att] + 1
  # print(data[att].unique())
  # print("##############################################")
```

```python
# Payment delay description
data[['PAY_1', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6']].describe()
```

      PAY_1
      PAY_2
      PAY_3
      PAY_4
      PAY_5
      PAY_6

      count
      29601.00
      29601.00
      29601.00
      29601.00
      29601.00
      29601.00

      mean
      0.79
      0.67
      0.64
      0.59
      0.55
      0.52

      std
      1.34
      1.40
      1.39
      1.35
      1.31
      1.33

      min
      -1.00
      -1.00
      -1.00
      -1.00
      -1.00
      -1.00

      25%
      -1.00
      -1.00
      -1.00
      -1.00
      -1.00
      -1.00

      50%
      1.00
      1.00
      1.00
      1.00
      1.00
      1.00

      75%
      1.00
      1.00
      1.00
      1.00
      1.00
      1.00

      max
      9.00
      9.00
      9.00
      9.00
      9.00
      9.00

```python
# # REPAYMENT STATUS = -1=pay duly, 1=payment delay for one month, 2=payment delay for two months, ... 8=payment delay for eight months, 9=payment delay for nine months and above
# for att in ['PAY_1', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6']:
#   print(f"# {att} -------")
#   print(data[att].value_counts())
```

```python
if SHOW_FIGURE :
    # Creating a new dataframe with categorical variables
    subset = data[['SEX', 'EDUCATION', 'MARRIAGE', 'PAY_1', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6', 'Default']]

    f, axes = plt.subplots(3, 3, figsize=(20, 15), facecolor='white')
    f.suptitle('FREQUENCY OF CATEGORICAL VARIABLES (BY TARGET)')
    ax1 = sns.countplot(x="SEX", hue="Default", data=subset, palette="Blues", ax=axes[0,0])
    ax2 = sns.countplot(x="EDUCATION", hue="Default", data=subset, palette="Blues",ax=axes[0,1])
    ax3 = sns.countplot(x="MARRIAGE", hue="Default", data=subset, palette="Blues",ax=axes[0,2])
    ax4 = sns.countplot(x="PAY_1", hue="Default", data=subset, palette="Blues", ax=axes[1,0])
    ax5 = sns.countplot(x="PAY_2", hue="Default", data=subset, palette="Blues", ax=axes[1,1])
    ax6 = sns.countplot(x="PAY_3", hue="Default", data=subset, palette="Blues", ax=axes[1,2])
    ax7 = sns.countplot(x="PAY_4", hue="Default", data=subset, palette="Blues", ax=axes[2,0])
    ax8 = sns.countplot(x="PAY_5", hue="Default", data=subset, palette="Blues", ax=axes[2,1])
    ax9 = sns.countplot(x="PAY_6", hue="Default", data=subset, palette="Blues", ax=axes[2,2]);
```

```python
# Bill Statement description
data[['BILL_AMT1', 'BILL_AMT2', 'BILL_AMT3', 'BILL_AMT4', 'BILL_AMT5', 'BILL_AMT6']].describe()
```

      BILL_AMT1
      BILL_AMT2
      BILL_AMT3
      BILL_AMT4
      BILL_AMT5
      BILL_AMT6

      count
      29601.00
      29601.00
      2.96e+04
      29601.00
      29601.00
      29601.00

      mean
      50957.43
      48942.19
      4.68e+04
      43122.55
      40235.55
      38858.45

      std
      73370.24
      70923.99
      6.91e+04
      64196.38
      60699.34
      59519.89

      min
      -165580.00
      -69777.00
      -1.57e+05
      -170000.00
      -81334.00
      -339603.00

      25%
      3528.00
      2970.00
      2.65e+03
      2329.00
      1780.00
      1278.00

      50%
      22259.00
      21050.00
      2.00e+04
      19005.00
      18091.00
      17118.00

      75%
      66623.00
      63497.00
      5.98e+04
      54271.00
      50072.00
      49121.00

      max
      964511.00
      983931.00
      1.66e+06
      891586.00
      927171.00
      961664.00

```python
# Previous Payment Description
data[['PAY_AMT1', 'PAY_AMT2', 'PAY_AMT3', 'PAY_AMT4', 'PAY_AMT5', 'PAY_AMT6']].describe()
```

      PAY_AMT1
      PAY_AMT2
      PAY_AMT3
      PAY_AMT4
      PAY_AMT5
      PAY_AMT6

      count
      29601.00
      2.96e+04
      29601.00
      29601.00
      29601.00
      29601.00

      mean
      5649.56
      5.89e+03
      5198.42
      4828.66
      4795.03
      5181.33

      std
      16568.26
      2.31e+04
      17580.91
      15711.06
      15244.22
      17657.26

      min
      0.00
      0.00e+00
      0.00
      0.00
      0.00
      0.00

      25%
      1000.00
      8.25e+02
      390.00
      298.00
      259.00
      138.00

      50%
      2100.00
      2.01e+03
      1800.00
      1500.00
      1500.00
      1500.00

      75%
      5005.00
      5.00e+03
      4500.00
      4014.00
      4042.00
      4000.00

      max
      873552.00
      1.68e+06
      896040.00
      621000.00
      426529.00
      528666.00

```python
data.LIMIT_BAL.describe()
```

    count      29601.00
    mean      167550.54
    std       129944.02
    min        10000.00
    25%        50000.00
    50%       140000.00
    75%       240000.00
    max      1000000.00
    Name: LIMIT_BAL, dtype: float64

```python
# plt.figure(figsize = (14,6))
# plt.title('Amount of credit limit - Density Plot')
# sns.set_color_codes("pastel")
# sns.distplot(data['LIMIT_BAL'],kde=True,bins=200, color="blue")
# plt.show()
```

```python
# data['LIMIT_BAL'].value_counts().head(10)
```

```python
# x1 = list(data[data['Default'] == 1]['LIMIT_BAL'])
# x2 = list(data[data['Default'] == 0]['LIMIT_BAL'])

# plt.figure(figsize=(12,4))
# sns.set_context('notebook', font_scale=1.2)
# #sns.set_color_codes("pastel")
# plt.hist([x1, x2], bins = 40, color=['steelblue', 'lightblue'])
# # plt.xlim([0,600000])
# plt.legend(['Yes', 'No'], title = 'Default', loc='upper right', facecolor='white')
# plt.xlabel('Limit Balance (NT dollar)')
# plt.ylabel('Frequency')
# plt.title('LIMIT BALANCE HISTOGRAM BY TYPE OF CREDIT CARD', SIZE=15)
# plt.box(False)
# plt.savefig('ImageName', format='png', dpi=200, transparent=True);
```

```python
# class_1 = data.loc[data['Default'] == 1]["LIMIT_BAL"]
# class_0 = data.loc[data['Default'] == 0]["LIMIT_BAL"]
# plt.figure(figsize = (14,6))
# # plt.title('Default amount of credit limit  - grouped by Payment Next Month (Density Plot)')
# sns.set_color_codes("pastel")
# sns.distplot(class_1,kde=True,bins=200, color="red")
# sns.distplot(class_0,kde=True,bins=200, color="green")
# plt.savefig('Fig - Density plot LIMIT_BAL grouped by label.png')
```

```python
if SHOW_FIGURE :
    # HISTOGRAMS PLOTS FOR FEATURES IMPORTANCE
    fig, axes= plt.subplots(6,4, figsize=(12,12))
    non_def = data[data.Default==0]
    default = data[data.Default==1]
    ax = axes.ravel()

    for i,label in enumerate(data.columns[:-1]):
        _,bins = np.histogram(data[label],bins=100)
        ax[i].hist(non_def[label],bins=bins,color='r',alpha=.5)# red color for malignant class
        ax[i].hist(default[label],bins=bins,color='g',alpha=0.3)# alpha is for transparency in the overlapped region
        ax[i].set_title(data.columns[i],fontsize=9)
        ax[i].axes.get_xaxis().set_visible(True) # the x-axis co-ordinates are not so useful, as we just want to look how well separated the histograms are
        ax[i].set_yticks(())

    ax[0].legend(['Non Default','Default'],loc='best',fontsize=8)
    plt.tight_layout() # let's make good plots
    #plt.show()
```

```python
# Correlation matrix
sns.set(style="white", font_scale=1)
numeric = ['LIMIT_BAL','AGE','BILL_AMT1','BILL_AMT2','BILL_AMT3', 'BILL_AMT4', 'BILL_AMT5', 'BILL_AMT6', 'PAY_AMT1','PAY_AMT2', 'PAY_AMT3', 'PAY_AMT4', 'PAY_AMT5', 'PAY_AMT6']
corr = data[numeric].corr() # .corr is used to find corelation

mask = np.triu(np.ones_like(corr, dtype=np.bool))
f, ax = plt.subplots(figsize=(20, 13))
cmap = sns.diverging_palette(220, 10, as_cmap=True)
ax=sns.heatmap(corr, mask=mask, vmax=1, vmin=-1, center=0,
            square=True, linewidths=.5, cmap=cmap, cbar_kws={"shrink": .5}, annot=True, annot_kws={"size": 10})
cbar=ax.collections[0].colorbar
cbar.set_ticks([-1, -0.50, 0, 0.50, 1])
# plt.xticks(fontsize=6)
# plt.yticks(fontsize=10)
# plt.show()
plt.savefig('Fig - Correlation matrix by means of the Pearson’s coefficient for all feature pairs.png')
```

```python
#distribution correlated features -- scatter interaction
import matplotlib.patches as mpatches

data_np=data.to_numpy()
target=data.Default

# variables to
BILL_AMT1 = data['BILL_AMT1'].to_numpy()
BILL_AMT2 = data['BILL_AMT2'].to_numpy()
BILL_AMT3 = data['BILL_AMT3'].to_numpy()
BILL_AMT4 = data['BILL_AMT4'].to_numpy()
BILL_AMT5 = data['BILL_AMT5'].to_numpy()
BILL_AMT6 = data['BILL_AMT6'].to_numpy()
AGE = data['AGE'].to_numpy()
LIMIT_BAL = data['LIMIT_BAL'].to_numpy()
PAY_AMT1 = data['PAY_AMT1'].to_numpy()

fig, ax = plt.subplots(1,3, figsize= (15,6))

labels=["Non Default","Default"]
pop_a = mpatches.Patch(color='steelblue', label='Non Default')
pop_b = mpatches.Patch(color='crimson', label='Default')
colors=['crimson', 'steelblue']

ax[0].scatter(BILL_AMT1, BILL_AMT2, c=target, cmap=matplotlib.colors.ListedColormap(colors), label=labels, alpha=0.5)
ax[0].grid()
ax[0].set_xlabel('BILL_AMT1')
ax[0].set_ylabel('BILL_AMT2')
ax[0].legend(handles= [pop_a,pop_b])

ax[1].scatter(BILL_AMT2, BILL_AMT3, c=target, cmap=matplotlib.colors.ListedColormap(colors), alpha=0.5)
ax[1].grid()
ax[1].set_xlabel('BILL_AMT2')
ax[1].set_ylabel('BILL_AMT3')
ax[1].legend(handles= [pop_a,pop_b])

ax[2].scatter(BILL_AMT4,BILL_AMT5, c=target, cmap=matplotlib.colors.ListedColormap(colors), alpha=0.5)
ax[2].grid()
ax[2].set_xlabel('BILL_AMT4')
ax[2].set_ylabel('BILL_AMT5')
ax[2].legend(handles= [pop_a,pop_b])

plt.tight_layout()# let's make good plots
plt.show()
```

```python
#distribution un-correlated features -- scatter interaction

fig, ax = plt.subplots(1,3, figsize= (15,6))

labels=["Non Default","Default"]
pop_a = mpatches.Patch(color='steelblue', label='Non Default')
pop_b = mpatches.Patch(color='crimson', label='Default')
colors=['crimson', 'steelblue']

ax[0].scatter(AGE, LIMIT_BAL, c=target, cmap=matplotlib.colors.ListedColormap(colors), label=labels, alpha=0.5)
ax[0].grid()
ax[0].set_xlabel('AGE')
ax[0].set_ylabel('LIMIT_BAL')
ax[0].legend(handles= [pop_a,pop_b])

ax[1].scatter(AGE, BILL_AMT1, c=target, cmap=matplotlib.colors.ListedColormap(colors), alpha=0.5)
ax[1].grid()
ax[1].set_xlabel('AGE')
ax[1].set_ylabel('BILL_AMT1')
ax[1].legend(handles= [pop_a,pop_b])

ax[2].scatter(PAY_AMT1,BILL_AMT1, c=target, cmap=matplotlib.colors.ListedColormap(colors), alpha=0.5)
ax[2].grid()
ax[2].set_xlabel('PAY_AMT1')
ax[2].set_ylabel('BILL_AMT1')
ax[2].legend(handles= [pop_a,pop_b])

plt.tight_layout()# let's make good plots
plt.show()
```

```python
scaler = MinMaxScaler()
data['LIMIT_BAL'] = scaler.fit_transform(data['LIMIT_BAL'].values.reshape(-1, 1))
data['AGE'] = scaler.fit_transform(data['AGE'].values.reshape(-1, 1))

for i in range(1,7):
    scaler = MinMaxScaler()
    data['PAY_' + str(i)] = scaler.fit_transform(data['PAY_' + str(i)].values.reshape(-1, 1))

for i in range(1,7):
    scaler = MinMaxScaler()
    data['BILL_AMT' + str(i)] = scaler.fit_transform(data['BILL_AMT' + str(i)].values.reshape(-1, 1))

for i in range(1,7):
    scaler = MinMaxScaler()
    data['PAY_AMT' + str(i)] = scaler.fit_transform(data['PAY_AMT' + str(i)].values.reshape(-1, 1))

# BOXPLOT
cols = ['LIMIT_BAL','AGE','BILL_AMT1','BILL_AMT2','BILL_AMT3', 'BILL_AMT4', 'BILL_AMT5', 'BILL_AMT6', 'PAY_AMT1','PAY_AMT2', 'PAY_AMT3', 'PAY_AMT4', 'PAY_AMT5', 'PAY_AMT6']

fig=plt.figure(1, figsize=(25,15))
ax=fig.add_subplot(111)
sns.boxplot(data=data[cols])
plt.xticks(np.arange(0,14), labels=cols, rotation=25, fontsize=18)
plt.yticks(fontsize=18)
# plt.title('Boxplot', fontsize= 35)

plt.savefig('Fig - Boxplot')
```

#### Standardization

Another possible scaling technique is referred to as **standardization**, which transforms each value $x$ of feature $X$ as follows, independently for each column:

\begin{equation}
x^{\prime} =\frac{x-\mu _{X}}{\sigma _{X}}
\end{equation}

where $\mu_{X}$ is the sample mean and $\sigma_{X}$ is the sample standard deviation of feature $X$. This way, we force each attribute to have a new empirical mean value of 0 and variance 1.

```python
# Standard Scaler
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaled_data = scaler.fit_transform(data)

# BOXPLOT
cols = ['LIMIT_BAL','AGE','BILL_AMT1','BILL_AMT2','BILL_AMT3', 'BILL_AMT4', 'BILL_AMT5', 'BILL_AMT6', 'PAY_AMT1','PAY_AMT2', 'PAY_AMT3', 'PAY_AMT4', 'PAY_AMT5', 'PAY_AMT6']

fig=plt.figure(1, figsize=(25,15))
ax=fig.add_subplot(111)
sns.boxplot(data=scaled_data[:,:-10])
plt.xticks(np.arange(0,14), labels=cols, rotation=25, fontsize=18)
plt.yticks(fontsize=18)
# plt.title('Boxplot', fontsize= 35)

plt.savefig('Fig - Boxplot Std Scaler')
```

## 3. Data Preprocessing

```python
# Set 'category' type to categorical attributes
for att in ['SEX', 'EDUCATION', 'MARRIAGE']:
  data[att] = data[att].astype('category')

# one-hot encoding
data=pd.concat([pd.get_dummies(data['SEX'], prefix='SEX'),
                pd.get_dummies(data['EDUCATION'], prefix='EDUCATION'),
                pd.get_dummies(data['MARRIAGE'], prefix='MARRIAGE'),
                data],axis=1)
# drop original columns
data.drop(['EDUCATION'],axis=1, inplace=True)
data.drop(['SEX'],axis=1, inplace=True)
data.drop(['MARRIAGE'],axis=1, inplace=True)
# print samples
data.head()
```

      SEX_1
      SEX_2
      EDUCATION_1
      EDUCATION_2
      EDUCATION_3
      EDUCATION_4
      MARRIAGE_1
      MARRIAGE_2
      MARRIAGE_3
      LIMIT_BAL
      AGE
      PAY_1
      PAY_2
      PAY_3
      PAY_4
      PAY_5
      PAY_6
      BILL_AMT1
      BILL_AMT2
      BILL_AMT3
      BILL_AMT4
      BILL_AMT5
      BILL_AMT6
      PAY_AMT1
      PAY_AMT2
      PAY_AMT3
      PAY_AMT4
      PAY_AMT5
      PAY_AMT6
      Default

      0
      0
      1
      0
      1
      0
      0
      1
      0
      0
      0.01
      0.05
      0.4
      0.4
      0.0
      0.0
      0.0
      0.0
      0.15
      0.07
      0.09
      0.16
      0.08
      0.26
      0.00e+00
      4.09e-04
      0.00e+00
      0.00e+00
      0.00e+00
      0.00e+00
      1

      1
      0
      1
      0
      1
      0
      0
      0
      1
      0
      0.11
      0.09
      0.0
      0.4
      0.2
      0.2
      0.2
      0.4
      0.15
      0.07
      0.09
      0.16
      0.08
      0.26
      0.00e+00
      5.94e-04
      1.12e-03
      1.61e-03
      0.00e+00
      3.78e-03
      1

      2
      0
      1
      0
      1
      0
      0
      0
      1
      0
      0.08
      0.22
      0.2
      0.2
      0.2
      0.2
      0.2
      0.2
      0.17
      0.08
      0.09
      0.17
      0.10
      0.27
      1.74e-03
      8.91e-04
      1.12e-03
      1.61e-03
      2.34e-03
      9.46e-03
      0

      3
      0
      1
      0
      1
      0
      0
      1
      0
      0
      0.04
      0.28
      0.2
      0.2
      0.2
      0.2
      0.2
      0.2
      0.19
      0.11
      0.11
      0.19
      0.11
      0.28
      2.29e-03
      1.20e-03
      1.34e-03
      1.77e-03
      2.51e-03
      1.89e-03
      0

      4
      1
      0
      0
      1
      0
      0
      1
      0
      0
      0.04
      0.62
      0.0
      0.2
      0.0
      0.2
      0.2
      0.2
      0.15
      0.07
      0.11
      0.18
      0.10
      0.28
      2.29e-03
      2.18e-02
      1.12e-02
      1.45e-02
      1.62e-03
      1.28e-03
      0

```python
# Separating features and target
y = data.Default     # target default=1 or non-default=0
X = data.drop('Default', axis = 1, inplace = False)
```

```python
# Check data set dimension
print(X.shape)
print("Number of samples:", X.shape[0])
print("Number of attributes:", X.shape[1])
```

    (29601, 29)
    Number of samples: 29601
    Number of attributes: 29

```python
from sklearn.model_selection import train_test_split

X_train_val, X_test, y_train_val, y_test = train_test_split(X, y, test_size=0.25, random_state=RANDOM_STATE, stratify=y)
```

```python
# Check dimensions
print(np.shape(X_train_val))
print(np.shape(X_test))

print("Training set:")
print(f" + Non-defaulters (y=0): {len(y_train_val[y_train_val==0])}")
print(f" + Defaulters (y=1):\t {len(y_train_val[y_train_val==1])}")

print("Test set:")
print(f" + Non-defaulters (y=0): {len(y_test[y_test==0])}")
print(f" + Defaulters (y=1):\t {len(y_test[y_test==1])}")
```

    (22200, 29)
    (7401, 29)
    Training set:
     + Non-defaulters (y=0): 17246
     + Defaulters (y=1):	 4954
    Test set:
     + Non-defaulters (y=0): 5750
     + Defaulters (y=1):	 1651

```python
if FEATURE_SELECTION :
    # remove features with correlation coefficent >= 0.93
    upper = corr.where(np.triu(np.ones(corr.shape), k=1).astype(np.bool))
    to_drop = [column for column in upper.columns if any(upper[column] >= 0.92)]
    print(to_drop) #['BILL_AMT2', 'BILL_AMT3', 'BILL_AMT4', 'BILL_AMT5', 'BILL_AMT6']
    data= data.drop(data[to_drop], axis = 1)
```

```python
from sklearn.decomposition import PCA

print(f"Actual number of components: {len(X_train_val.columns.values)}")

n_pc = len(X_train_val.columns.values) # with n_components = 11 -> 0.98693009 variance explained

pca = PCA(n_components=n_pc)
pca.fit(X_train_val)

# print(np.cumsum(pca.explained_variance_ratio_))
# print(pca.explained_variance_ratio_)

fig=plt.figure(1, figsize=(10,5))
plt.grid(b=True, which='major', axis='both')
plt.plot(pca.explained_variance_ratio_, marker='o', color='#20c8b8')
plt.plot(np.cumsum(pca.explained_variance_ratio_), marker='o', color='#e8a000')
# plt.xticks([n for n in range(0,n_pc)], [ "PC"+str(n) for n in range(1,n_pc+1)])
plt.legend(["Variance explained by a single component", "Cumulative variance explained"])
plt.xlabel('Number of principal components')
plt.ylabel('Variance Explained')
plt.savefig("PCA");
```

```python
n_pc = 12
pca = PCA(n_components=n_pc)
pca.fit(X_train_val)
#   X_train_val.index = pd.RangeIndex(start=0, stop=len(X_train_val), step=1)
X_12d_train_val = pd.DataFrame(pca.transform(X_train_val))
X_12d_test = pd.DataFrame(pca.transform(X_test))
X_12d_train_val.columns = ['PC' + str(i) for i in range(1, n_pc+1) ]
X_12d_test.columns = ['PC' + str(i) for i in range(1, n_pc+1) ]
X_12d_train_val.head()
```

      PC1
      PC2
      PC3
      PC4
      PC5
      PC6
      PC7
      PC8
      PC9
      PC10
      PC11
      PC12

      0
      1.15
      0.53
      0.42
      -0.20
      0.05
      -0.06
      -0.03
      8.00e-03
      -1.02e-01
      -0.12
      0.13
      -9.53e-03

      1
      0.17
      -0.67
      -0.82
      -0.04
      0.04
      0.37
      -0.10
      4.24e-02
      -4.01e-03
      -0.07
      -0.01
      -8.76e-03

      2
      -0.87
      -0.29
      0.56
      0.85
      -0.28
      -0.02
      -0.27
      1.47e-01
      2.03e-01
      -0.08
      -0.11
      -2.49e-02

      3
      -0.52
      -0.38
      0.93
      -0.31
      0.71
      -0.10
      0.09
      -7.04e-02
      2.04e-02
      -0.09
      -0.05
      3.32e-03

      4
      -1.07
      -0.25
      -0.30
      -0.23
      0.24
      0.39
      -0.03
      -9.61e-03
      3.43e-02
      -0.20
      0.11
      -1.48e-04

```python
APPLY_PCA = False

if APPLY_PCA:
    X_train_val = X_12d_train_val
    X_test = X_12d_test
    print(X_12d_train_val.shape)
    print(X_12d_test.shape)

print(f"PCA APPLIED: {APPLY_PCA}")
print(X_train_val.shape)
print(X_test.shape)
```

    (22200, 12)
    (7401, 12)
    PCA APPLIED: True
    (22200, 12)
    (7401, 12)

```python
import imblearn
from imblearn.over_sampling import SMOTE
from imblearn.under_sampling import ClusterCentroids
from imblearn.pipeline import make_pipeline

import sklearn.ensemble
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.model_selection import StratifiedKFold
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import GridSearchCV
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC, LinearSVC
from sklearn.metrics import f1_score, make_scorer, precision_score, recall_score, accuracy_score, plot_confusion_matrix
from sklearn.svm import LinearSVC, SVC
from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import LogisticRegression

# PARAMETER CLASSIFICATION
ALGORITHM = 'LogisticRegression' # 'RandomForest', 'LogisticRegression', 'LDA', 'KNN'
APPLY_OVERSAMPLING = False
APPLY_UNDERSAMPLING = True

if APPLY_OVERSAMPLING:
    APPLY_UNDERSAMPLING = False

if APPLY_UNDERSAMPLING:
    APPLY_OVERSAMPLING = False
```

```python
# TRAIN AND VALIDATION
# Stratified K-Fold Cross Validation

# Pipeline
if ALGORITHM == 'SVM' :
    # Support Vector Machine (SVM) classifier
    # best config found {'C': 1, 'gamma': 0.01, 'kernel': 'poly'} # 0.7963817277250114
    parameter_grid = {
      'C': [100, 0.1, 1, 10],
      'kernel': ['rbf', 'poly'],
      'gamma': [0.0001, 0.001, 0.01]
    }
    clf_name = 'svc__'
    classifier = SVC() # or SVC

elif ALGORITHM == 'RandomForest':
    # Random Forest classifier
    # {'randomforestclassifier__criterion': 'entropy', 'randomforestclassifier__max_features': 'sqrt', 'randomforestclassifier__n_estimators': 100, 'randomforestclassifier__oob_score': True}
    parameter_grid = {
      "criterion":["gini", "entropy"],
      "max_features":[None, "sqrt"],
      "oob_score":[True],
      "n_estimators":[10, 50, 100, 200]
    }
    clf_name = 'randomforestclassifier__'
    classifier = RandomForestClassifier()

elif ALGORITHM == 'LogisticRegression':
    # Logistic Regression classifier
    parameter_grid = {
        "C":[0.0001, 0.001, 0.01, 0.1, 1, 10]
    }
    clf_name = 'logisticregression__'
    classifier = LogisticRegression()

elif ALGORITHM == 'KNN':
    # K-Nearest Neighbors classifier
    parameter_grid = {
        "n_neighbors":[500, 800, 1500, 2500, 3500, 4500]
    }
    clf_name = 'kneighborsclassifier__'
    classifier = KNeighborsClassifier()
else :
    raise RuntimeError("Choose a correct classifier.");

new_params = {clf_name + key: parameter_grid[key] for key in parameter_grid}

kf = StratifiedKFold(n_splits=5, random_state=RANDOM_STATE)

if APPLY_OVERSAMPLING:
    # apply oversampling on training dataset
    imba_pipeline = make_pipeline(SMOTE(random_state=RANDOM_STATE), classifier)
elif APPLY_UNDERSAMPLING:
    # TO DO
    imba_pipeline = make_pipeline(ClusterCentroids(random_state=RANDOM_STATE), classifier)
else :
    # DO NOT apply oversampling on training dataset, just the classifier
    imba_pipeline = make_pipeline(classifier)

SCORE = 'precision' # 'accuracy', 'precision', 'recall', 'f1'
grid_imba = GridSearchCV(imba_pipeline, param_grid=new_params, cv=kf, scoring=SCORE, return_train_score=True)

grid_imba.fit(X_train_val, y_train_val)

print(f"Best configuration found for {classifier}:")
print(grid_imba.best_params_)
print(f"Val {SCORE}:   {grid_imba.best_score_}")
```

    /usr/local/lib/python3.6/dist-packages/sklearn/model_selection/_split.py:296: FutureWarning: Setting a random_state has no effect since shuffle is False. This will raise an error in 0.24. You should leave random_state to its default (None), or set shuffle=True.
      FutureWarning
    /usr/local/lib/python3.6/dist-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function safe_indexing is deprecated; safe_indexing is deprecated in version 0.22 and will be removed in version 0.24.
      warnings.warn(msg, category=FutureWarning)
    /usr/local/lib/python3.6/dist-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function safe_indexing is deprecated; safe_indexing is deprecated in version 0.22 and will be removed in version 0.24.
      warnings.warn(msg, category=FutureWarning)
    /usr/local/lib/python3.6/dist-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function safe_indexing is deprecated; safe_indexing is deprecated in version 0.22 and will be removed in version 0.24.
      warnings.warn(msg, category=FutureWarning)
    /usr/local/lib/python3.6/dist-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function safe_indexing is deprecated; safe_indexing is deprecated in version 0.22 and will be removed in version 0.24.
      warnings.warn(msg, category=FutureWarning)
    /usr/local/lib/python3.6/dist-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function safe_indexing is deprecated; safe_indexing is deprecated in version 0.22 and will be removed in version 0.24.
      warnings.warn(msg, category=FutureWarning)
    /usr/local/lib/python3.6/dist-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function safe_indexing is deprecated; safe_indexing is deprecated in version 0.22 and will be removed in version 0.24.
      warnings.warn(msg, category=FutureWarning)
    /usr/local/lib/python3.6/dist-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function safe_indexing is deprecated; safe_indexing is deprecated in version 0.22 and will be removed in version 0.24.
      warnings.warn(msg, category=FutureWarning)
    /usr/local/lib/python3.6/dist-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function safe_indexing is deprecated; safe_indexing is deprecated in version 0.22 and will be removed in version 0.24.
      warnings.warn(msg, category=FutureWarning)
    /usr/local/lib/python3.6/dist-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function safe_indexing is deprecated; safe_indexing is deprecated in version 0.22 and will be removed in version 0.24.
      warnings.warn(msg, category=FutureWarning)
    /usr/local/lib/python3.6/dist-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function safe_indexing is deprecated; safe_indexing is deprecated in version 0.22 and will be removed in version 0.24.
      warnings.warn(msg, category=FutureWarning)
    /usr/local/lib/python3.6/dist-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function safe_indexing is deprecated; safe_indexing is deprecated in version 0.22 and will be removed in version 0.24.
      warnings.warn(msg, category=FutureWarning)
    /usr/local/lib/python3.6/dist-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function safe_indexing is deprecated; safe_indexing is deprecated in version 0.22 and will be removed in version 0.24.
      warnings.warn(msg, category=FutureWarning)
    /usr/local/lib/python3.6/dist-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function safe_indexing is deprecated; safe_indexing is deprecated in version 0.22 and will be removed in version 0.24.
      warnings.warn(msg, category=FutureWarning)
    /usr/local/lib/python3.6/dist-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function safe_indexing is deprecated; safe_indexing is deprecated in version 0.22 and will be removed in version 0.24.
      warnings.warn(msg, category=FutureWarning)
    /usr/local/lib/python3.6/dist-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function safe_indexing is deprecated; safe_indexing is deprecated in version 0.22 and will be removed in version 0.24.
      warnings.warn(msg, category=FutureWarning)
    /usr/local/lib/python3.6/dist-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function safe_indexing is deprecated; safe_indexing is deprecated in version 0.22 and will be removed in version 0.24.
      warnings.warn(msg, category=FutureWarning)
    /usr/local/lib/python3.6/dist-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function safe_indexing is deprecated; safe_indexing is deprecated in version 0.22 and will be removed in version 0.24.
      warnings.warn(msg, category=FutureWarning)

```python
# TEST
# evaluation on test data with bestparams

# Oversampling
smote = SMOTE()
X_train_val, y_train_val = smote.fit_resample(X_train_val, y_train_val)

# retrain with best params
clf = classifier
clf.fit(X_train_val, y_train_val)
y_pred = clf.predict(X_test)

accuracy=accuracy_score(y_test, y_pred)
f1=f1_score(y_test, y_pred)
precision=precision_score(y_test, y_pred)
recall=recall_score(y_test, y_pred)

print(accuracy, f1, precision, recall)
print(f"")

# print(classification_report())

disp = plot_confusion_matrix(clf, X_test, y_test,
                                 #display_labels=class_names,
                                 cmap=plt.cm.Blues,
                                 #display_labels=data.target_names,
                                 normalize='true')

disp.ax_.set_title(f'normalized confusion matrix {clf_name}')

```