# Music Machine Learning - Support Vector Machines (SVM)

### Author: Philippe Esling (esling@ircam.fr)

In this course we will cover
1. A recap on the [motivation](#motivation) behind Support Vector Machines (SVM)
2. A simple use case for [understanding and fitting SVMs](#fitting) with the scikit library
3. Developing our own [implementation](#implem) of linear SVMs
4. An introuction to [kernels](#improve) in SVMs and how to implement them

## Support Vector Machines (SVM)

In this tutorial, we will cover a more advanced classification algorithm through the use of *Support Vector Machines* (SVMs). The goal is to gain an intuition of how SVMs works and how to use *Gaussian kernel* with SVMs to find out the decision boundary. The implementation proposed here follows a simplified version of the *Sequential Minimal Optimization* (SMO) algorithm for training support vector machines, which is detailed in the following [paper](http://cs229.stanford.edu/materials/smo.pdf). The complete SMO algorithm can then be quite easily implemented following the [full paper](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/tr-98-14.pdf).

Once again, to simplify your work, we provide the following set of functions that you should find in the `data` folder, and in the `helper_plot` file

  |**File**|*Explanation*|
  |-------:|:------------|
  |`svm_linear.mat`|Example data for a quasi-linear problem|
  |`svm_gaussian.mat`|Example data for a non-linear but well-defined problem|
  |`plot_data`|Allows to plot the set of data points|
  |`visualize_boundary`|Plots a non-linear decision boundary from a SVM model|
  |`plot_svc_decision_function`|Plots a linear decision boundary|

```python
import numpy as np
from scipy import io as sio
from matplotlib import pyplot as plt
from helper_plot import plot_data, visualize_boundary, visualize_boundary_linear, hdr_plot_style, plot_svc_decision_function
```

## Motivation

In the previous course on neural networks, we learned how to train a simple neuron. We saw that this simple model could be interpreted as defining a line, separating the space into two different regions. This allows to consider problems of *discriminative classification*: rather than modeling each class, we simply find a line that divides the classes from each other. As an example of these _linearly separable_ problems, consider the simple case of a classification task, in which the two classes of points are well separated:

```python
from sklearn.datasets.samples_generator import make_blobs
hdr_plot_style()
X, y = make_blobs(n_samples=50, centers=2, random_state=0, cluster_std=0.65)
plt.figure(figsize=(10, 8)); plt.grid(True)
plt.scatter(X[:, 0], X[:, 1], c=y, s=80, edgecolor='w', cmap='winter');
plt.title("Linear classification example");
```

As we have seen previously, a single neuron classifier would attempt to draw a straight line separating the two sets of data, and thereby create a model for classification, by optimizing the parameters $\theta=\{\mathbf{w}, b\}$ of

$$
\begin{equation}
y=\sigma\left(\sum_{i = 1}^{n}w_{i}.x_{i} + b\right)
\label{eq1}
\end{equation}
$$

For two dimensional data like that shown here, this is a task we could do by hand. But the major problem that we can directly see, is that there is an infinite number of lines that can perfectly discriminate between the two classes, as exemplified in the next example.

```python
xfit = np.linspace(-1, 3.5)
plt.figure(figsize=(10, 8)); plt.grid(True)
plt.scatter(X[:, 0], X[:, 1], c=y, s=80, edgecolor='w', cmap='winter')
for m, b in [(1, 0.65), (0.5, 1.6), (-0.2, 2.9)]:
    plt.plot(xfit, m * xfit + b, '-w')
plt.xlim(-1, 3.5);
```

Note that these solutions are all valid, and will mostly depend on the (random) initialization of our parameters. Although these all perfectly discriminate between our samples, we can clearly see that they are more or less "optimal" to the problem at hand. So our question today is how could we formalize this problem, and if we could find a formulation to adress and solve this issue.

### Notion of _margin_

The intuition behind comparing different solutions is that rather than estimating only the line between the classes, we can also look at the *margin* of some width defined around those line. In our simple case, it would look like this

```python
xfit = np.linspace(-1, 3.5)
plt.figure(figsize=(10, 8)); plt.grid(True)
plt.scatter(X[:, 0], X[:, 1], c=y, s=80, edgecolor='w', cmap='winter')
for m, b, d in [(1, 0.65, 0.25), (0.5, 1.6, 0.55), (-0.2, 2.9, 0.15)]:
    yfit = m * xfit + b
    plt.plot(xfit, yfit, '-w')
    plt.fill_between(xfit, yfit - d, yfit + d, edgecolor='w', color='#AAAAAA', alpha=0.6)
plt.xlim(-1, 3.5);
```

In *Support Vector Machines (SVMs)*, the goal is to find that optimal model, in which our separating line maximizes this margin. Hence, SVMs are part of a class of models called *maximum margin* estimator.

## Fitting a SVM

More formally the goal of SVMs is that given a set of input vectors $ \mathbf{x} $ separated in two classes (noted $ \mathbf{x}^{+} $ and $ \mathbf{x}^{-} $), we would like to find the *optimal boundary* (separating line) between the two classes. To do so, we want to find the separation that gives the *maximal margin* between the two sets. Hence, we can first define the following problem

$$
\begin{equation}
\begin{cases}
\mathbf{w}\cdot\mathbf{x}^{+}+\mathbf{b}\geq1\\
\mathbf{w}\cdot\mathbf{x}^{-}+\mathbf{b}\leq-1
\end{cases}
\label{eq1}
\end{equation}
$$

By setting the variable $ y_{i}={+}/{-}1 $, we can rewrite this problem as

$$
\begin{equation}
y_{i}\left(x_{i}\cdot\mathbf{w}+\mathbf{b}\right)-1\geq0
\label{eq2}
\end{equation}
$$

The idea is that for some of the input vectors (called *support vectors*), we will have an equality in equation (1), where $ \mathbf{w}\cdot\mathbf{x}^{+}+\mathbf{b}=1 $ and $ \mathbf{w}\cdot\mathbf{x}^{-}+\mathbf{b}=-1 $. By substracting those two equation, we see that we are trying to find a solution to

$$
\begin{equation}
\frac{\mathbf{w}}{\left\Vert \mathbf{w}\right\Vert }\cdot\left(x^{+}-x^{-}\right)=\frac{2}{\left\Vert \mathbf{w}\right\Vert }=Width
\label{eq3}
\end{equation}
$$

Therefore, solving the set of equations amounts to find the maximal margin between the support vectors (as they *support* classification).

### Fitting the SVM with scikit-learn

First, we can see the result of fitting this approach to our original data. To do so, we will use the SVM model found in `scikit-learn` to train an SVM model on this data. To do so, we can use the following code

```python
from sklearn.svm import SVC # "Support vector classifier"
model = SVC(kernel='linear')
model.fit(X, y)
```

    SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
      decision_function_shape='ovr', degree=3, gamma='auto', kernel='linear',
      max_iter=-1, probability=False, random_state=None, shrinking=True,
      tol=0.001, verbose=False)

To visualize the results, we will use our quick convenience function that plots the SVM decision boundaries

```python
plt.figure(figsize=(10, 8)); plt.grid(True)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor='w', cmap='winter')
plot_svc_decision_function(model);
```

This separating line is the one that maximizes the margin between our two sets of points.
Notice that some training points are right on the margin. These points are the pivotal elements of this fit, and are known as the *support vectors*, and give the algorithm its name. In `scikit-learn`, we can directly obtain the values of these points, stored in the ``support_vectors_`` attribute of the classifier:

```python
model.support_vectors_
```

    array([[0.39920934, 3.01626962],
           [2.45161058, 3.35844964],
           [2.06209253, 2.05847948],
           [3.02263145, 2.12999162]])

The major advantage of SVM is that for the fit, only the position of the support vectors matter; any points further from the margin which are on the correct side do not modify the fit. Technically, this is because these points do not contribute to the loss function used to fit the model, so their position and number do not matter so long as they do not cross the margin. We can see this, for example, if we plot the model learned from the first 50 or 150 points of this dataset

```python
# Plotting SVM trained on subsampled dataset
def plot_svm_subsample(N=10, ax=None):
    X, y = make_blobs(n_samples=200, centers=2, random_state=0, cluster_std=0.60)
    X = X[:N]
    y = y[:N]
    model = SVC(kernel='linear', C=1E10)
    model.fit(X, y)
    ax = ax or plt.gca()
    ax.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor = 'w', cmap='winter')
    ax.set_xlim(-1, 4)
    ax.set_ylim(-1, 6)
    plot_svc_decision_function(model, ax)

fig, ax = plt.subplots(1, 2, figsize=(16, 6))
fig.subplots_adjust(left=0.0625, right=0.95, wspace=0.1)
for axi, N in zip(ax, [60, 150]):
    plot_svm_subsample(N, axi)
    axi.set_title('N = {0}'.format(N))
```

In the left panel, we see the model and the support vectors for 60 training points.
In the right panel, we have doubled the number of training points, but the model has not changed: the three support vectors from the left panel are still the support vectors from the right panel.
This insensitivity to the exact behavior of distant points is one of the strengths of the SVM model.

## Implementing our own linear SVM

Here, we are going to try to implement our own linear SVM. We recall that, given a set of input vectors $ \mathbf{x} $ separated in two classes ($ \mathbf{x}^{+} $ and $ \mathbf{x}^{-} $, we would like to find the *optimal boundary* (separating line) between the two classes. This means that we are trying to find a solution to

$$
\begin{equation}
\frac{\mathbf{w}}{\left\Vert \mathbf{w}\right\Vert }\cdot\left(x^{+}-x^{-}\right)=\frac{2}{\left\Vert \mathbf{w}\right\Vert }=Width
\label{eq3}
\end{equation}
$$

Therefore, solving the set of equations amounts to find the maximal margin between the support vectors (as they *support* classification). As seen in the slides, we know that this problem can also be expressed as

$$
\begin{equation}
f\left(x\right)=\sum_{i=1}^{m}\alpha_{i}y_{i}\left\langle \phi\left(x_{i}\right),\phi\left(x\right)\right\rangle +b
\end{equation}
$$

Where the function $\phi(.)$ will be used later to introduce _kernels_, but for now, we will consider that $\phi(x)=x$, which reduces our problem to

$$
\begin{equation}
f\left(x\right)=\sum_{i=1}^{m}\alpha_{i}y_{i}\left\langle x_{i},x \right\rangle +b
\end{equation}
$$

we can introduce a kernel $K\left(x_{i},x\right)=\phi\left(x_{i}\right)\cdot\phi\left(x\right)$ in the equation in order to solve non-linear optimization problems. We can use this formulation to perform the *predictions* of the algorithm and in the first part of this tutorial, we will only consider the *linear* kernel $K\left(x_{i},x\right)=x_{i}\cdot x$.

Now, if we look at the way the SVM model that we trained is implemented in `scikit-learn`, we have the following variables

```python
print(vars(model))
```

    {'decision_function_shape': 'ovr', '_impl': 'c_svc', 'kernel': 'linear', 'degree': 3, 'gamma': 'auto', 'coef0': 0.0, 'tol': 0.001, 'C': 1.0, 'nu': 0.0, 'epsilon': 0.0, 'shrinking': True, 'probability': False, 'cache_size': 200, 'class_weight': None, 'verbose': False, 'max_iter': -1, 'random_state': None, '_sparse': False, 'class_weight_': array([1., 1.]), 'classes_': array([0, 1]), '_gamma': 0.5, 'support_': array([18, 25,  7, 46], dtype=int32), 'support_vectors_': array([[0.39920934, 3.01626962],
           [2.45161058, 3.35844964],
           [2.06209253, 2.05847948],
           [3.02263145, 2.12999162]]), 'n_support_': array([2, 2], dtype=int32), 'dual_coef_': array([[-0.24571733, -0.99437155,  1.        ,  0.24008889]]), 'intercept_': array([3.45657164]), 'probA_': array([], dtype=float64), 'probB_': array([], dtype=float64), 'fit_status_': 0, 'shape_fit_': (50, 2), '_intercept_': array([-3.45657164]), '_dual_coef_': array([[ 0.24571733,  0.99437155, -1.        , -0.24008889]])}

For the moment, our variables of interest are
- `support_vectors_` : _Support vectors_ (dataset points that support classification)
- `dual_coef_` : The $\alpha_{i} * y_{i}$ (coefficients from the dual formulation) for the _support_vectors_
- `n_support_` : Number of support vectors for each class (ordered)
- `_intercept_` : Value of $b$ in our formulation (bias)

In order to begin our own implementation of the SVM, we are going to start by computing the _decision function_ of the model. We are going to slightly change the model structure, to better reflect the mathematical formulation than `scikit-learn`. Hence, our model structure and all of its corresponding parameters will be defined as

``` Python
model = {};
model["x"];            # Values of _support vectors_ in input data
model["y"];            # Classes labels of _support vectors_
model["n"];            # Number of support for each class
model["b"];            # Value of bias
model["alphas"];       # Value of alphas
model["kernel"];       # Type of kernel
model["w"];            # Weights of vectors in primal formulation
```

In the following, we show how to shift from the `scikit-learn` version to ours

```python
#%% Creating a fake (random) model
our_model = {};
# Values of _support vectors_ in input data
our_model["x"] = model.support_vectors_
# Corresponding classes of the support vectors
our_model["y"] = np.concatenate([[0] * model.n_support_[0], [1] * model.n_support_[1]])
# Number of vectors for each class
our_model["n"] = model.n_support_
# Value of threshold
our_model["b"] = model._intercept_
# Values of alphas
our_model["alphas"] = model._dual_coef_[0]
print(our_model)
```

    {'x': array([[0.39920934, 3.01626962],
           [2.45161058, 3.35844964],
           [2.06209253, 2.05847948],
           [3.02263145, 2.12999162]]), 'y': array([0, 0, 1, 1]), 'n': array([2, 2], dtype=int32), 'b': array([-3.45657164]), 'alphas': array([ 0.24571733,  0.99437155, -1.        , -0.24008889])}

Computing the _predictions_ of the SVM is performed by calling `model.decision_function(X)` in `scikit_learn`. We are going to implement this behavior ourselves by coding our own `decision_function` function

``` Python
def decision_function(model, X)
    """
    % Returns a vector of predictions using a trained SVM model.
    % model   : svm model returned from svmTrain.
    % X       : m x n matrix where each example is a row.
    """
    return pred
```

For the time being we are using a _linear kernel_, but we might later use a different type of kernel, so we will directly use a `kernel` function which computes our kernel between the support vectors `x_s` and a dataset `X`.

***

**Exercice**
1. Implement the linear kernel function
1. Complete the decision function based on the dual formulation

***

```python
def kernel(x, X, type='linear'):

    ######################
    # YOUR CODE GOES HERE
    ######################

    if (type == 'linear'):
        k = ...
    return k

# This function should replicates the scikit-learn one
def decision_function(model, X):

    ######################
    # YOUR CODE GOES HERE
    ######################

    return ...

# Evaluate the decision function
ax = plt.figure(figsize=(10,8)).gca()
ax.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor = 'w', cmap='winter')
ax.set_xlim(-1, 4); ax.set_ylim(-1, 6)
xlim = ax.get_xlim(); ylim = ax.get_ylim()
# Create grid to evaluate model
x = np.linspace(xlim[0], xlim[1], 30)
yp = np.linspace(ylim[0], ylim[1], 30)
Y, Xp = np.meshgrid(yp, x)
xy = np.vstack([Xp.ravel(), Y.ravel()]).T
P = decision_function(our_model, xy.transpose())
P = P.reshape((30, 30))
# plot decision boundary and margins
ax.contour(Xp, Y, P, colors='w',levels=[-1, 0, 1], alpha=0.9,linestyles=['--', '-', '--']);
ax.scatter(our_model["x"][:, 0], our_model["x"][:, 1],s=300, linewidth=2, edgecolor='w', facecolors='none');
```

## Learning phase

In order to perform learning, we will try to maximize the *dual formulation* problem expressed as

$$
\begin{equation}
w\left(\alpha\right)=\sum_{i}\alpha_{i}-\frac{1}{2}\sum_{i}\sum_{j}y_{i}y_{j}\alpha_{i}\alpha_{j}\left\langle \phi\left(x_{i}\right),\phi\left(x_{j}\right)\right\rangle
\end{equation}
$$

with $0\leq \alpha_{i} \leq C$ and $\sum_{i=1}^{m}{\alpha_{i} y_{i}} = 0$. $C$ is a chosen *regularization hyper-parameter*. To ease your work, we will implement only a simplified version of the SMO algorithm which is mainly focused on the optimization aspects. The SMO algorithm selects two $\alpha$ parameters, $\alpha_{i}$ and $\alpha_{j}$ and optimizes the objective value jointly for both these $\alpha_{i,j}$. Finally it adjusts the $b$ parameter based on the new $\alpha_{i,j}$. This process is repeated until the $\alpha_{i,j}$ converge.

### Selecting $\alpha$

The full SMO algorithm use heuristics to choose which $\alpha_{i}$ and $\alpha_{j}$ to optimize (which greatly increase the convergence speed. However, for this simplified version, we will simply iterate on all $\alpha_{i},i=1,\cdots ,m$ and select a random $\alpha_{j}$, to perform the joint optimization of $\alpha_{i}$ and $\alpha_{j}$

### Optimizing $\alpha_{i}$ and $\alpha_{j}$

To jointly optimize the value, we first find bounds $L$ and $H$ such that $L\leq \alpha_{j} \leq H$ holds to satisfy the constraint of the original problem $0\leq \alpha_{i} \leq C$

$$
\begin{array}{ccc}
if\mbox{ }y_{i}\neq y_{j}, & L=max\left(0,\alpha_{j}-\alpha_{i}\right), & H=min\left(C,C+\alpha_{j}-\alpha_{i}\right)\\
if\mbox{ }y_{i}=y_{j}, & L=max\left(0,\alpha_{i}+\alpha_{j}-C\right), & H=min\left(C,\alpha_{i}+\alpha_{j}\right)
\end{array}
$$

Now we want to find $\alpha_{j}$ so as to maximize the objective function. It can be shown (as in the original paper) that the optimal $\alpha_{j}$ is given by

$$
\alpha_{j}=\alpha_{j}-\frac{y_{j}\left(E_{i}-E_{j}\right)}{\eta}
$$

where

$$
\begin{array}{ccc}
E_{k} & = & f\left(x_{k}\right)-y_{k}\\
\eta & = & 2\left\langle x_{i},x_{j}\right\rangle -\left\langle x_{i},x_{i}\right\rangle -\left\langle x_{j},x_{j}\right\rangle
\end{array}
$$

$E_{k}$ defines the error between the prediction of the SVM on the example $k$ and its desired label. When computing $\eta$, we can also replace the inner product by a kernel function $K$. Finally, we clip the value of $\alpha_{j}$ to lie within the selected bounds.

$$
\alpha_{j}=\begin{cases}
H & if\mbox{ }\alpha_{j}>H\\
\alpha_{j} & if\mbox{ }L\leq\alpha_{j}\leq H\\
L & if\mbox{ }\alpha_{j}<L
\end{cases}
$$

Now that we have solved $\alpha_{j}$, we can update the value for $\alpha_{i}$ through

$$
\alpha_{i}^{t}=\alpha_{i}^{t-1}+y_{i}y_{j}\left(\alpha_{j}^{(t-1)}-\alpha_{j}\right)
$$

### Computing threshold $b$

Finally, we can compute the threshold $b$ to satisfy the original constraints by selecting

$$
b=\begin{cases}
b_{1} & if\mbox{ }0<\alpha_{i}<C\\
b_{2} & if\mbox{ }0<\alpha_{j}<C\\
\left(b_{1}+b_{2}\right)/2 & otherwise
\end{cases}
$$

where

$$
b_{1}=b-E_{i}-y_{i}\left(\alpha_{i}^{(t-1)}-\alpha_{i}\right)\left\langle x_{i},x_{i}\right\rangle -y_{j}\left(\alpha_{j}^{(t-1)}-\alpha_{j}\right)\left\langle x_{i},x_{j}\right\rangle
$$
$$
b_{2}=b-E_{i}-y_{i}\left(\alpha_{i}^{(t-1)}-\alpha_{i}\right)\left\langle x_{i},x_{j}\right\rangle -y_{j}\left(\alpha_{j}^{(t-1)}-\alpha_{j}\right)\left\langle x_{j},x_{j}\right\rangle
$$

For the first part of this tutorial, we will compute the main iterations of the algorithm (minimization of the objective function), while relying on a *linear* kernel. This implies that we will only be able to perform linear discrimination. However, remember that the formulation of the SVMs provide an *optimal* and (gloriously) *convex* answer to this problem.

```Python
def svm_train(X, Y, C, kernelFunction, tol, maxIter):
    """
    % Trains an SVM classifier using a simplified SMO algorithm.
    % X       : m x n matrix of m training examples (with n-dimensional features).
    % Y       : column vector of class identifiers
    % C       : standard SVM regularization parameter
    % tol     : tolerance value used for determining equality of floating point numbers.
    % maxIter : number of iterations over the dataset before the algorithm stops.
    """
    return model
```

***

**Exercise**

  1. Update the `svm_train` code to perform the training of a SVM with linear kernel.
  2. Run your `svm_train` function on our linear dataset to obtain similar results as `scikit-learn`
  3. Update the framework to display the decision boundary at each iteration of the algorithm.

***

```python
def svm_train(X, Y_b, C, kernel_type, tol=1e-3, max_iter=5):
    """
    SVMTRAIN Trains an SVM classifier using a simplified version of the SMO
    # X       : m x n matrix of m training examples (with n-dimensional features).
    # Y       : column vector of class identifiers (0 or 1)
    # C       : standard SVM regularization parameter
    # tol     : tolerance value used for determining equality of floating point numbers.
    # maxIter : number of iterations over the dataset before the algorithm stops.
    """
    # Data parameters
    m, n = X.shape
    # Variables
    alphas = np.zeros((m, 1))
    b, passes, eta, L, H = 0, 0, 0, 0, 0
    E = np.zeros((m, 1))
    Y = Y_b.copy()
    Y[Y == 0] = -1

    # Pre-compute the kernel matrix (in practice, the optimized packages will _not_ do this)
    K = np.array(kernel(X, X.transpose(), kernel_type))

    # Train
    print('Training ...')
    while passes < max_iter:
        # Check that some alphas changed
        num_changed_alphas = 0;
        # Iterative over all alpha_i
        for i in range(m):

            ######################
            # YOUR CODE GOES HERE
            # => Perform one pass of the SMO algorithm for each alpha_i
            ######################

        if (num_changed_alphas == 0):
            passes = passes + 1
        else:
            passes = 0
    print(' Done !');
    # Save the model
    idx = (alphas > 0).transpose()[0];
    model = {}
    model["x"] = X[idx, :]
    model["y"] = Y[idx]
    model["b"] = b
    model["alphas"] = alphas[idx, 0] * Y[idx]
    model["kernel"] = kernel
    return model

# Training Linear SVM
print('Training Linear SVM ...')
# Parameters of the training
C = 1                  # C regularization value
tolerance = 1e-5       # Tolerance value for thresholds
nb_iterations = 50      # Number of iterations
# Perform the training (try to fiddle the previous values to see effect).
model = svm_train(X, y, C, 'linear', tolerance, nb_iterations);
# Evaluate the decision function
ax = plt.figure(figsize=(10, 8)).gca()
ax.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor = 'w', cmap='winter')
ax.set_xlim(-1, 4); ax.set_ylim(-1, 6)
xlim = ax.get_xlim(); ylim = ax.get_ylim()
# Create grid to evaluate model
x = np.linspace(xlim[0], xlim[1], 30)
yp = np.linspace(ylim[0], ylim[1], 30)
Y, Xp = np.meshgrid(yp, x)
xy = np.vstack([Xp.ravel(), Y.ravel()]).T
P = decision_function(model, xy.transpose())
P = P.reshape((30, 30))
# plot decision boundary and margins
ax.contour(Xp, Y, P, colors='w',levels=[-1, 0, 1], alpha=0.9,linestyles=['--', '-', '--'])
ax.scatter(model["x"][:, 0], model["x"][:, 1],s=300, linewidth=2, edgecolor='w', facecolors='none');
plt.title('Our trained SVM :)');
```

# Beyond linear problems with _kernels_

For the time being, our SVM is only able to solve _linear_ problems. However, the SVM approach becomes truly powerful is when it is combined with *kernels*. As seen in the slides, the major idea is to project our (non-linearly separable) data into a higher-dimensional space, where it becomes linearly separable (allowing to fit for nonlinear relationships with a linear classifier).

To better understand this idea, we can look at the same non linearly separable data that we discussed in the slides. Here, we try to fit a _linear_ SVM to this non-linear problem.

```python
from sklearn.datasets.samples_generator import make_circles
X, y = make_circles(100, factor=.1, noise=.1)

clf = SVC(kernel='linear').fit(X, y)
plt.figure(figsize=(10, 8)); plt.grid(True)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor='w', cmap='winter')
plot_svc_decision_function(clf, plot_support=False);
```

It is clear that no linear discrimination will *ever* be able to separate this data. As discussed, we need to think about how we might project the data into a higher dimension such that a linear separator *would* be sufficient. For example, one simple projection we could use would be to compute a *radial basis function* centered on the middle group of points.

```python
r = np.exp(-(X ** 2).sum(1))
```

We can visualize this extra data dimension using a three-dimensional plot

```python
from mpl_toolkits import mplot3d

def plot_3D(elev=30, azim=30, X=X, y=y):
    plt.figure(figsize=(10, 8));
    ax = plt.subplot(projection='3d')
    ax.scatter3D(X[:, 0], X[:, 1], r, c=y, s=50, edgecolor='w', cmap='winter')
    ax.view_init(elev=elev, azim=azim)
    ax.set_xlabel('x')
    ax.set_ylabel('y')
    ax.set_zlabel('z')
plot_3D(elev=30, azim=-50, X=X, y=y);
```

We can see that with this additional dimension, the data becomes trivially linearly separable, by drawing a separating plane. Here we had to choose and carefully tune our projection. In general, the need to make such a choice is a problem, as we would like to somehow automatically find the best basis functions to use. One strategy to this end is to compute a basis function centered at *every* point in the dataset, and let the SVM algorithm find the result. This type of basis function transformation is known as a *kernel transformation*, as it is based on a similarity relationship (or kernel) between **each pair of points**.

To see this idea in action first, we will (as previously) use `scikit-learn`, where we can apply _kernelized SVM_ simply by changing our _linear kernel_ to an _RBF (radial basis function)_ kernel, using the ``kernel`` model hyperparameter:

```python
model = SVC(kernel='rbf', C=1E6)
model.fit(X, y)
```

    SVC(C=1000000.0, cache_size=200, class_weight=None, coef0=0.0,
      decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf',
      max_iter=-1, probability=False, random_state=None, shrinking=True,
      tol=0.001, verbose=False)

Once fitted, we can see that our SVM is able to find the largest margin between these two sets of data

```python
plt.figure(figsize=(10, 8)); plt.grid(True)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor='w', cmap='winter')
plot_svc_decision_function(model)
plt.scatter(model.support_vectors_[:, 0], model.support_vectors_[:, 1], s=300, lw=1, facecolors='none');
```

Using this kernelized support vector machine, we learn a suitable nonlinear decision boundary.
This kernel transformation strategy is used often in machine learning to turn fast linear methods into fast nonlinear methods, especially for models in which the kernel trick can be used.

## Implementing the gaussian kernel

As we have seen previously, we know that the complete SVM optimization problem can also be expressed as

$$
\begin{equation}
f\left(x\right)=\sum_{i=1}^{m}\alpha_{i}y_{i}\left\langle \phi\left(x_{i}\right),\phi\left(x\right)\right\rangle +b
\end{equation}
$$

where we can introduce the kernel $K\left(x_{i},x\right)=\phi\left(x_{i}\right)\cdot\phi\left(x\right)$ in the equation in order to solve non-linear optimization problems. We can use this formulation to perform the *predictions* of the algorithm. Consequently, the complete *dual formulation* problem expressed as

$$
\begin{equation}
w\left(\alpha\right)=\sum_{i}\alpha_{i}-\frac{1}{2}\sum_{i}\sum_{j}y_{i}y_{j}\alpha_{i}\alpha_{j}\left\langle \phi\left(x_{i}\right),\phi\left(x_{j}\right)\right\rangle
\end{equation}
$$

with $0\leq \alpha_{i} \leq C$ and $\sum_{i=1}^{m}{\alpha_{i} y_{i}} = 0$. The Gaussian kernel (also called *Radial Basis Function (RBF)* kernel) allows to perform the optimization of non-linear classification problems. It is defined as

$$
K\left(x,y\right)=exp\left(\frac{-\left\Vert x-y\right\Vert ^{2}}{\left(2\sigma^{2}\right)}\right)
$$

As seen in the slides, the underlying idea is that instead of trying to solve a non-linear separation problem in a given space, we might first transform the space into a target space in which the problem is linearly separable. Therefore, we transform the space through a kernel (which is directly expressed in the minimization problem) and solve the linear separation problem in the target space. Given the starter code and the previous exercise, you should be able to directly plug this kernel inside the `svmTrain` and `svmPredict` functions.

***

**Exercise**

  1. Update the `kernel` function code to be able to rely on a Gaussian kernel.
  2. Check that you can run the `decision_function` code to rely on this Gaussian kernel.
  2. Run the `svm_train` function on our new (non-linear) dataset to obtain similar results as `scikit-learn`
  3. Once again, display the decision boundary at each iteration of the algorithm.

***

```python
def kernel(x, X, type='linear'):

    ######################
    # YOUR CODE GOES HERE
    ######################

    if (type == 'linear'):
        k = ...
    elif kernel == 'gaussian':
        k = ...
    return k

plt.title('Traning Dataset')
# Parameters of the training
C = 1                  # C regularization value
tolerance = 1e-3       # Tolerance value for thresholds
nb_iterations = 20     # Number of iterations
sigma = 0.1            # Variance of Gaussian kernel
# Train the SVM with a Gaussian (RBF) kernel
model = svm_train(X, y, C, 'gaussian', tolerance, nb_iterations)
# Evaluate the decision function
ax = plt.figure(figsize=(10, 8)).gca()
ax.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor = 'w', cmap='winter')
ax.set_xlim(-1, 4); ax.set_ylim(-1, 6)
xlim = ax.get_xlim(); ylim = ax.get_ylim()
# Create grid to evaluate model
x = np.linspace(xlim[0], xlim[1], 30)
yp = np.linspace(ylim[0], ylim[1], 30)
Y, Xp = np.meshgrid(yp, x)
xy = np.vstack([Xp.ravel(), Y.ravel()]).T
P = decision_function(model, xy.transpose())
P = P.reshape((30, 30))
# plot decision boundary and margins
ax.contour(Xp, Y, P, colors='w',levels=[-1, 0, 1], alpha=0.9,linestyles=['--', '-', '--'])
ax.scatter(model["x"][:, 0], model["x"][:, 1],s=300, linewidth=2, edgecolor='w', facecolors='none');
plt.title('Our trained SVM :)');
```

## Supplementary kernels

Now that you have mastered the concept of kernels and updated your code to include the Gaussian kernel, you can augment your classifier and optimizaton problem to include several different kernels as those proposed afterwards.

#### Polynomial kernel.

Intuitively, the polynomial kernel looks not only at the given features of input samples to determine their similarity, but also combinations of these.

$$
K\left(x,y\right)=\left(x^{T}y+1\right)^{d}
$$

#### Sigmoid (tanh) kernel.

This kernel takes two parameters: $a$ and $r$. For $a > 0$, we can view $a$ as a scaling parameter of the input data, and $r$ as a shifting parameter that controls the threshold of mapping. For $a < 0$, the dot-product of the input data is not only scaled but reversed.

$$
K\left(x,y\right)=tanh\left(kx^{T}y+\Theta\right)
$$

***

**Exercise**

  1. Update the `kernel` function code to include these supplementary kernels.
  2. Test both the `decision_function` and `svm_train` codes for these new kernels.

***

```python

```

## Audio application

As seen in the previous tutorial, we have considered only a *binary classification* problem (where elements can only belong to one of two classes). This simplifies the work as we simply need to separate the space between two types of points. However, in real-world problems, we usually want to solve *multi-class* problems with any given number of classes. For this tutorial, we will rely on a simple trick which is to consider that a $n$-class problem can simply be restated as $n$ different binary classification problems. The underlying idea is that each class defines a binary classifier which tells us if a given element is part of this class, or belongs to *any of the other* classes.

***

**Exercise**

  1. Change the training loop to define $n$separate binary problems
  2. For each problem, train an SVM to perform classification.
  3. Evaluate the global classification accuracy of your system.

***

```python

```

```python
from helper_data import import_dataset, compute_transforms, compute_features
class_path = 'data/classification'
# 0.1 - Import the classification dataset
data_struct = import_dataset(class_path, 'classification')
# 0.2 - Pre-process the audio to obtain spectral transforms
data_struct = compute_transforms(data_struct)
# 0.3 - Compute a set of temporal and spectral features
data_struct = compute_features(data_struct)
```


# JEST internal workshop → Intro to ML (Crash Course)

## This is an short version (without explanations) of the workshop, to access the full version of the document click [here](https://github.com/tamagusko/workshop-intro-ml/blob/main/introML.ipynb).

## Topics

1. Data Collection
2. Exploratory data analysis
    2.1 Data visualization
3. Preprocessing
    3.1 Missing Data, Encoding, Standardizing, Feature Engineering
    3.2 Split data in train/test
4. Model
    4.1  Decision Tree
    4.2. Random Forest
    4.3. KNN (k-nearest neighbors)
    4.4. XGBoost (eXtreme Gradient Boosting)
    4.5. Neural Network (Multilayer perceptron)
5. Comparison of models (Metrics)
6. Improving the model
7. Saving and loading models
8. How to identify overfitting?
9. Automating tasks (Choosing the best model)
---

---

# Practical example: Predicting Pulsar Star

More details in: [Predicting Pulsar Star](https://www.kaggle.com/colearninglounge/predicting-pulsar-starintermediate)

## Proposed problem

Classify stars between Pulsar (1) and non-Pulsar (0).

Note: The data were collected and previously treated.

## 1. Data Collection

### Dataset - [pulsar_stars.csv](data/pulsar_stars.csv)

**rows**: 16,259 spurious examples (RFI/noise), and 1,639 real pulsar examples. Total of 17,898 rows.

**columns**: 9 features, namely:
  1. Mean of the integrated profile.
  2. Standard deviation of the integrated profile.
  3. Excess kurtosis of the integrated profile.
  4. Skewness of the integrated profile.
  5. Mean of the DM-SNR curve.
  6. Standard deviation of the DM-SNR curve.
  7. Excess kurtosis of the DM-SNR curve.
  8. Skewness of the DM-SNR curve.
  9. Class (1=Pulsar, 0=non-Pulsar)

So, the features are columns 1 to 8, and the label is the column 9.

```python
# Data manipulation
import pandas as pd
# Data visualization
import seaborn as sns
import matplotlib.pyplot as plt
```

More details in [A Complete Pandas Guide](https://towardsdatascience.com/a-complete-pandas-guide-2dc53c77a002)

```python
url = 'https://raw.githubusercontent.com/spingegod/ProjetoTI_part1/master/ProjetoTI/pulsar_stars.csv'
```

```python
df = pd.read_csv(url)
```

```python
df
```

      Mean of the integrated profile
      Standard deviation of the integrated profile
      Excess kurtosis of the integrated profile
      Skewness of the integrated profile
      Mean of the DM-SNR curve
      Standard deviation of the DM-SNR curve
      Excess kurtosis of the DM-SNR curve
      Skewness of the DM-SNR curve
      target_class

      0
      140.562500
      55.683782
      -0.234571
      -0.699648
      3.199833
      19.110426
      7.975532
      74.242225
      0

      1
      102.507812
      58.882430
      0.465318
      -0.515088
      1.677258
      14.860146
      10.576487
      127.393580
      0

      2
      103.015625
      39.341649
      0.323328
      1.051164
      3.121237
      21.744669
      7.735822
      63.171909
      0

      3
      136.750000
      57.178449
      -0.068415
      -0.636238
      3.642977
      20.959280
      6.896499
      53.593661
      0

      4
      88.726562
      40.672225
      0.600866
      1.123492
      1.178930
      11.468720
      14.269573
      252.567306
      0

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      17893
      136.429688
      59.847421
      -0.187846
      -0.738123
      1.296823
      12.166062
      15.450260
      285.931022
      0

      17894
      122.554688
      49.485605
      0.127978
      0.323061
      16.409699
      44.626893
      2.945244
      8.297092
      0

      17895
      119.335938
      59.935939
      0.159363
      -0.743025
      21.430602
      58.872000
      2.499517
      4.595173
      0

      17896
      114.507812
      53.902400
      0.201161
      -0.024789
      1.946488
      13.381731
      10.007967
      134.238910
      0

      17897
      57.062500
      85.797340
      1.406391
      0.089520
      188.306020
      64.712562
      -1.597527
      1.429475
      0

17898 rows × 9 columns

## 2. Exploratory data analysis

## 2.1 Data visualization

```python
df.shape
```

    (17898, 9)

```python
df.tail()
```

      Mean of the integrated profile
      Standard deviation of the integrated profile
      Excess kurtosis of the integrated profile
      Skewness of the integrated profile
      Mean of the DM-SNR curve
      Standard deviation of the DM-SNR curve
      Excess kurtosis of the DM-SNR curve
      Skewness of the DM-SNR curve
      target_class

      17893
      136.429688
      59.847421
      -0.187846
      -0.738123
      1.296823
      12.166062
      15.450260
      285.931022
      0

      17894
      122.554688
      49.485605
      0.127978
      0.323061
      16.409699
      44.626893
      2.945244
      8.297092
      0

      17895
      119.335938
      59.935939
      0.159363
      -0.743025
      21.430602
      58.872000
      2.499517
      4.595173
      0

      17896
      114.507812
      53.902400
      0.201161
      -0.024789
      1.946488
      13.381731
      10.007967
      134.238910
      0

      17897
      57.062500
      85.797340
      1.406391
      0.089520
      188.306020
      64.712562
      -1.597527
      1.429475
      0

### View data type

```python
df.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 17898 entries, 0 to 17897
    Data columns (total 9 columns):
     #   Column                                         Non-Null Count  Dtype
    ---  ------                                         --------------  -----
     0    Mean of the integrated profile                17898 non-null  float64
     1    Standard deviation of the integrated profile  17898 non-null  float64
     2    Excess kurtosis of the integrated profile     17898 non-null  float64
     3    Skewness of the integrated profile            17898 non-null  float64
     4    Mean of the DM-SNR curve                      17898 non-null  float64
     5    Standard deviation of the DM-SNR curve        17898 non-null  float64
     6    Excess kurtosis of the DM-SNR curve           17898 non-null  float64
     7    Skewness of the DM-SNR curve                  17898 non-null  float64
     8   target_class                                   17898 non-null  int64
    dtypes: float64(8), int64(1)
    memory usage: 1.2 MB

### Data overview

```python
df.describe().T
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      Mean of the integrated profile
      17898.0
      111.079968
      25.652935
      5.812500
      100.929688
      115.078125
      127.085938
      192.617188

      Standard deviation of the integrated profile
      17898.0
      46.549532
      6.843189
      24.772042
      42.376018
      46.947479
      51.023202
      98.778911

      Excess kurtosis of the integrated profile
      17898.0
      0.477857
      1.064040
      -1.876011
      0.027098
      0.223240
      0.473325
      8.069522

      Skewness of the integrated profile
      17898.0
      1.770279
      6.167913
      -1.791886
      -0.188572
      0.198710
      0.927783
      68.101622

      Mean of the DM-SNR curve
      17898.0
      12.614400
      29.472897
      0.213211
      1.923077
      2.801839
      5.464256
      223.392140

      Standard deviation of the DM-SNR curve
      17898.0
      26.326515
      19.470572
      7.370432
      14.437332
      18.461316
      28.428104
      110.642211

      Excess kurtosis of the DM-SNR curve
      17898.0
      8.303556
      4.506092
      -3.139270
      5.781506
      8.433515
      10.702959
      34.539844

      Skewness of the DM-SNR curve
      17898.0
      104.857709
      106.514540
      -1.976976
      34.960504
      83.064556
      139.309331
      1191.000837

      target_class
      17898.0
      0.091574
      0.288432
      0.000000
      0.000000
      0.000000
      0.000000
      1.000000

### NA values?

```python
df.isna().any()
```

     Mean of the integrated profile                  False
     Standard deviation of the integrated profile    False
     Excess kurtosis of the integrated profile       False
     Skewness of the integrated profile              False
     Mean of the DM-SNR curve                        False
     Standard deviation of the DM-SNR curve          False
     Excess kurtosis of the DM-SNR curve             False
     Skewness of the DM-SNR curve                    False
    target_class                                     False
    dtype: bool

### Duplicate rows?

```python
print("Duplicate rows: ", df.duplicated().sum())
```

    Duplicate rows:  0

### Target (label) class

```python
df.target_class.value_counts()
```

    0    16259
    1     1639
    Name: target_class, dtype: int64

```python
df.target_class.value_counts().plot.pie(autopct='%.2f%%')
```

### Correlation matrix

```python
correlation = df.corr()
correlation.style.background_gradient(cmap='coolwarm').set_precision(2)
```

        	         Mean of the integrated profile	         Standard deviation of the integrated profile	         Excess kurtosis of the integrated profile	         Skewness of the integrated profile	         Mean of the DM-SNR curve	         Standard deviation of the DM-SNR curve	         Excess kurtosis of the DM-SNR curve	         Skewness of the DM-SNR curve	        target_class

                         Mean of the integrated profile
                        1.00
                        0.55
                        -0.87
                        -0.74
                        -0.30
                        -0.31
                        0.23
                        0.14
                        -0.67

                         Standard deviation of the integrated profile
                        0.55
                        1.00
                        -0.52
                        -0.54
                        0.01
                        -0.05
                        0.03
                        0.03
                        -0.36

                         Excess kurtosis of the integrated profile
                        -0.87
                        -0.52
                        1.00
                        0.95
                        0.41
                        0.43
                        -0.34
                        -0.21
                        0.79

                         Skewness of the integrated profile
                        -0.74
                        -0.54
                        0.95
                        1.00
                        0.41
                        0.42
                        -0.33
                        -0.20
                        0.71

                         Mean of the DM-SNR curve
                        -0.30
                        0.01
                        0.41
                        0.41
                        1.00
                        0.80
                        -0.62
                        -0.35
                        0.40

                         Standard deviation of the DM-SNR curve
                        -0.31
                        -0.05
                        0.43
                        0.42
                        0.80
                        1.00
                        -0.81
                        -0.58
                        0.49

                         Excess kurtosis of the DM-SNR curve
                        0.23
                        0.03
                        -0.34
                        -0.33
                        -0.62
                        -0.81
                        1.00
                        0.92
                        -0.39

                         Skewness of the DM-SNR curve
                        0.14
                        0.03
                        -0.21
                        -0.20
                        -0.35
                        -0.58
                        0.92
                        1.00
                        -0.26

                        target_class
                        -0.67
                        -0.36
                        0.79
                        0.71
                        0.40
                        0.49
                        -0.39
                        -0.26
                        1.00

### Data visualization ()

```javascript
%%javascript
// Avoid scroll bar on very large images
IPython.OutputArea.prototype._should_scroll = function(lines) {
    return false;
}
```

    <IPython.core.display.Javascript object>

```python
# Plots the histogram of all columns (excluding target_class).
for column in df.columns[:-1]:
    plt.figure()
    sns.histplot(x=df[column], hue=df.target_class)
```

## 3 Preprocessing

### 3.1 Missing Data, Encoding, Standardizing, Feature Engineering

Look in: [freecodecamp: Preprocessing](https://www.youtube.com/watch?v=0B5eIE_1vpU&t=2076s) (not necessary in this dataset)

### 3.2 Split data in train/test

```python
X = df.drop(['target_class'], axis=1)
y = df.target_class
```

```python
# Prepare data to train models
from sklearn.model_selection import train_test_split

# 50/50 (common: 70/30 or 80/20)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, train_size=0.5, random_state=42, stratify=y)

X_train.shape, X_test.shape, y_train.shape, y_test.shape
```

    ((8949, 8), (8949, 8), (8949,), (8949,))

## 4. Models

## How to use Scikit-Learn?

By [Andre Ye](https://www.linkedin.com/in/andre-ye-501746150/), in KDnuggets:[The Ultimate Scikit-Learn Machine Learning Cheatsheet](https://www.kdnuggets.com/2021/01/ultimate-scikit-learn-machine-learning-cheatsheet.html).

```python
# Import
from sklearn.BRANCH import MODEL_NAME
from sklearn.metrics import METRIC_NAME

# Build
MODEL = MODEL_NAME()

# Train
MODEL.fit(X_train, y_train)

# Predict
y_pred_MODEL = MODEL.predict(X_test)

# Eval
METRIC = METRIC_NAME((y_test, y_pred_MODEL))
```

```python
# Metrics
from sklearn.metrics import accuracy_score
from sklearn.metrics import precision_score
from sklearn.metrics import matthews_corrcoef
from sklearn.metrics import confusion_matrix
from sklearn.metrics import roc_curve
from sklearn.metrics import auc
from sklearn.metrics import confusion_matrix
```

*Confusion Matrix:*
\begin{pmatrix}
True Negative & False Positive\\
False Negative & True Positive
\end{pmatrix}

\begin{equation}
Accuracy = \frac{TP+TN}{(TP+TN+FP+FN)} \\
Precision = \frac{TP}{(TP+FP)} \\
MCC = \frac{TP*TN-FP*FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}} \\
\end{equation}

*where:*

*TP = True Positive
TN = True Negative
FP = False Positive
FN = False Negative*

```python
# Simple functions to run metrics and print results

def evaluationModels(y_test, y_pred):
    accuracy = accuracy_score(y_test, y_pred)
    precision = precision_score(y_test, y_pred)
    # Best metric for binary (0/1) classification
    mcc = matthews_corrcoef(y_test, y_pred)
    fp = confusion_matrix(y_test, y_pred)[0][1]
    return accuracy, precision, mcc, fp

def printMetrics(model, accuracy, precision, mcc, fp):
    modelName = model.__class__.__name__
    bar = "-" * 25
    print(f"{bar}\n {modelName}\n{bar}")
    print(f'- Accuracy: {accuracy:0.3f}')
    print(f'- Precision: {precision:0.3f}')
    print(f'- MCC: {mcc:0.3f}')
    print(f'- False Positives: {fp}')
```

### 4.1 Decision Tree

See the parameters in: [sklearn.tree.DecisionTreeClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html)

More details in: [Decision Trees](https://scikit-learn.org/stable/modules/tree.html)

How to tune the parameters: [InDepth: Parameter tuning for Decision Tree](https://medium.com/@mohtedibf/indepth-parameter-tuning-for-decision-tree-6753118a03c3)

```python
from sklearn.tree import DecisionTreeClassifier

DT = DecisionTreeClassifier(max_depth=3, random_state=42)
```

```python
# Training
DT.fit(X_train, y_train)
```

    DecisionTreeClassifier(max_depth=3, random_state=42)

```python
# Training parameters
DT.get_params()
```

    {'ccp_alpha': 0.0,
     'class_weight': None,
     'criterion': 'gini',
     'max_depth': 3,
     'max_features': None,
     'max_leaf_nodes': None,
     'min_impurity_decrease': 0.0,
     'min_impurity_split': None,
     'min_samples_leaf': 1,
     'min_samples_split': 2,
     'min_weight_fraction_leaf': 0.0,
     'presort': 'deprecated',
     'random_state': 42,
     'splitter': 'best'}

```python
# Predict
y_pred_DT = DT.predict(X_test)
```

```python
# Evaluation
accuracy, precision, mcc, fp = metricsDT = evaluationModels(y_test, y_pred_DT)
printMetrics(DT, accuracy, precision, mcc, fp)
```

    -------------------------
     DecisionTreeClassifier
    -------------------------
    - Accuracy: 0.978
    - Precision: 0.893
    - MCC: 0.868
    - False Positives: 85

MCC varies beetwen -1 (perfect negative correlation) and 1 (perfect positive correlation).
OBS: 0 (random flip)

```python
confusion_matrix(y_test, y_pred_DT) # (TN FP \\ FN & TP)
```

    array([[8045,   85],
           [ 109,  710]])

### Trick (worst possible threshold)

```python
# Bad practice (don't do it)
import warnings
warnings.filterwarnings('ignore')
```

```python
threshold = [0] * len(X_test)  # All results equal to 0

accuracy, precision, mcc, fp = metricsThreshold = evaluationModels(y_test, threshold)
printMetrics(threshold, accuracy, precision, mcc, fp)  # The name is not defined
```

    -------------------------
     list
    -------------------------
    - Accuracy: 0.908
    - Precision: 0.000
    - MCC: 0.000
    - False Positives: 0

```python
# Feature Importance

featureImportanceDT = pd.DataFrame({'Feature': list(X_train.columns),
                                  'Importance': DT.feature_importances_}).\
    sort_values('Importance', ascending=False)

featureImportanceDT
```

      Feature
      Importance

      2
      Excess kurtosis of the integrated profile
      0.972950

      5
      Standard deviation of the DM-SNR curve
      0.019431

      0
      Mean of the integrated profile
      0.006398

      1
      Standard deviation of the integrated profile
      0.001221

      3
      Skewness of the integrated profile
      0.000000

      4
      Mean of the DM-SNR curve
      0.000000

      6
      Excess kurtosis of the DM-SNR curve
      0.000000

      7
      Skewness of the DM-SNR curve
      0.000000

### 4.2 Random Forest

See the parameters in: [sklearn.ensemble.RandomForestClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html)

More details in: [Random Forests](https://scikit-learn.org/stable/modules/ensemble.html#random-forests)

How to tune the parameters: [Random Forest: Hyperparameters and how to fine-tune them](https://towardsdatascience.com/random-forest-hyperparameters-and-how-to-fine-tune-them-17aee785ee0d)

```python
from sklearn.ensemble import RandomForestClassifier

# Building (need to tune for better results)
RF = RandomForestClassifier(max_depth=4, random_state=42)

# Training
RF.fit(X_train, y_train)
```

    RandomForestClassifier(max_depth=4, random_state=42)

```python
# Training parameters
RF.get_params()
```

    {'bootstrap': True,
     'ccp_alpha': 0.0,
     'class_weight': None,
     'criterion': 'gini',
     'max_depth': 4,
     'max_features': 'auto',
     'max_leaf_nodes': None,
     'max_samples': None,
     'min_impurity_decrease': 0.0,
     'min_impurity_split': None,
     'min_samples_leaf': 1,
     'min_samples_split': 2,
     'min_weight_fraction_leaf': 0.0,
     'n_estimators': 100,
     'n_jobs': None,
     'oob_score': False,
     'random_state': 42,
     'verbose': 0,
     'warm_start': False}

```python
# Predict
y_pred_RF = RF.predict(X_test)
```

```python
# Evaluating
accuracy, precision, mcc, fp = metricsRF = evaluationModels(y_test, y_pred_RF)
printMetrics(RF, accuracy, precision, mcc, fp)
```

    -------------------------
     RandomForestClassifier
    -------------------------
    - Accuracy: 0.979
    - Precision: 0.929
    - MCC: 0.870
    - False Positives: 52

```python
# Feature Importance
featureImportanceRF = pd.DataFrame({'feature': list(X_train.columns),
                                  'importance': RF.feature_importances_}).\
    sort_values('importance', ascending=False)

featureImportanceRF
```

      feature
      importance

      2
      Excess kurtosis of the integrated profile
      0.410166

      3
      Skewness of the integrated profile
      0.215174

      0
      Mean of the integrated profile
      0.176768

      4
      Mean of the DM-SNR curve
      0.059261

      5
      Standard deviation of the DM-SNR curve
      0.054170

      1
      Standard deviation of the integrated profile
      0.033804

      7
      Skewness of the DM-SNR curve
      0.030396

      6
      Excess kurtosis of the DM-SNR curve
      0.020261

### 4.3 K Nearest Neighbors (KNN)

See the parameters in: [sklearn.neighbors.KNeighborsClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html)

More details in: [Nearest Neighbors](https://scikit-learn.org/stable/modules/neighbors.html)

```python
from sklearn.neighbors import KNeighborsClassifier

# Building
KNN = KNeighborsClassifier(metric='euclidean')

# Training
KNN.fit(X_train, y_train)
KNN.get_params()
```

    {'algorithm': 'auto',
     'leaf_size': 30,
     'metric': 'euclidean',
     'metric_params': None,
     'n_jobs': None,
     'n_neighbors': 5,
     'p': 2,
     'weights': 'uniform'}

```python
# Predict
y_pred_KNN = KNN.predict(X_test)
```

```python
# Evaluating
accuracy, precision, mcc, fp = metricsKNN = evaluationModels(y_test, y_pred_KNN)
printMetrics(KNN, accuracy, precision, mcc, fp)
```

    -------------------------
     KNeighborsClassifier
    -------------------------
    - Accuracy: 0.972
    - Precision: 0.892
    - MCC: 0.827
    - False Positives: 79

### 4.4 XGBoost (Ensemble model)

See the parameters in: [XGBoost Parameters](https://xgboost.readthedocs.io/en/latest/parameter.html)

More details in: [Getting Started with XGBoost in scikit-learn](https://towardsdatascience.com/getting-started-with-xgboost-in-scikit-learn-f69f5f470a97)

* XGBoost is regularized (avoid overfit).
* XGBoost Have hyperparameters to handle imbalanced data and null values.
* Combine some of the best practices and algorithms in a single model.
* Fast and easy to use.
* It is my go-to algorithm for dealing with structured data.

```python
import xgboost as xgb

# Building (need to tune for better results)
XGB = xgb.XGBClassifier(max_depth=4, random_state=42)

# Training
XGB.fit(X_train, y_train)
```

    XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
                  colsample_bynode=1, colsample_bytree=1, gamma=0, gpu_id=-1,
                  importance_type='gain', interaction_constraints='',
                  learning_rate=0.300000012, max_delta_step=0, max_depth=4,
                  min_child_weight=1, missing=nan, monotone_constraints='()',
                  n_estimators=100, n_jobs=0, num_parallel_tree=1, random_state=42,
                  reg_alpha=0, reg_lambda=1, scale_pos_weight=1, subsample=1,
                  tree_method='exact', validate_parameters=1, verbosity=None)

```python
# Training parameters
XGB.get_params()
```

    {'objective': 'binary:logistic',
     'base_score': 0.5,
     'booster': 'gbtree',
     'colsample_bylevel': 1,
     'colsample_bynode': 1,
     'colsample_bytree': 1,
     'gamma': 0,
     'gpu_id': -1,
     'importance_type': 'gain',
     'interaction_constraints': '',
     'learning_rate': 0.300000012,
     'max_delta_step': 0,
     'max_depth': 4,
     'min_child_weight': 1,
     'missing': nan,
     'monotone_constraints': '()',
     'n_estimators': 100,
     'n_jobs': 0,
     'num_parallel_tree': 1,
     'random_state': 42,
     'reg_alpha': 0,
     'reg_lambda': 1,
     'scale_pos_weight': 1,
     'subsample': 1,
     'tree_method': 'exact',
     'validate_parameters': 1,
     'verbosity': None}

```python
# Predict
y_pred_XGB = XGB.predict(X_test)
```

```python
# Evaluating
accuracy, precision, mcc, fp = metricsXGB = evaluationModels(y_test, y_pred_XGB)
printMetrics(XGB, accuracy, precision, mcc, fp)
```

    -------------------------
     XGBClassifier
    -------------------------
    - Accuracy: 0.979
    - Precision: 0.912
    - MCC: 0.868
    - False Positives: 67

```python
# Feature Importance
featureImportanceXGB = pd.DataFrame({'feature': list(X_train.columns),
                                  'importance': XGB.feature_importances_}).\
    sort_values('importance', ascending=False)

featureImportanceXGB
```

      feature
      importance

      2
      Excess kurtosis of the integrated profile
      0.705079

      5
      Standard deviation of the DM-SNR curve
      0.066338

      1
      Standard deviation of the integrated profile
      0.043162

      6
      Excess kurtosis of the DM-SNR curve
      0.040389

      0
      Mean of the integrated profile
      0.038011

      3
      Skewness of the integrated profile
      0.037221

      4
      Mean of the DM-SNR curve
      0.035053

      7
      Skewness of the DM-SNR curve
      0.034746

### 4.5 Neural Network: Multi-Layer Perceptron (MLP) - Most simple NN model

See the parameters in: [sklearn.neural_network.MLPClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html)

More details in: [Multi-layer Perceptron (MLP)](https://scikit-learn.org/stable/modules/neural_networks_supervised.html)

```python
from sklearn.neural_network import MLPClassifier

MLP = MLPClassifier(alpha=1, max_iter=1000)

# Training
MLP.fit(X_train, y_train)
```

    MLPClassifier(alpha=1, max_iter=1000)

```python
# Training parameters
MLP.get_params()
```

    {'activation': 'relu',
     'alpha': 1,
     'batch_size': 'auto',
     'beta_1': 0.9,
     'beta_2': 0.999,
     'early_stopping': False,
     'epsilon': 1e-08,
     'hidden_layer_sizes': (100,),
     'learning_rate': 'constant',
     'learning_rate_init': 0.001,
     'max_fun': 15000,
     'max_iter': 1000,
     'momentum': 0.9,
     'n_iter_no_change': 10,
     'nesterovs_momentum': True,
     'power_t': 0.5,
     'random_state': None,
     'shuffle': True,
     'solver': 'adam',
     'tol': 0.0001,
     'validation_fraction': 0.1,
     'verbose': False,
     'warm_start': False}

```python
# Predict
y_pred_MLP = MLP.predict(X_test)
```

```python
# Evaluating
accuracy, precision, mcc, fp = metricsMLP = evaluationModels(y_test, y_pred_MLP)
printMetrics(MLP, accuracy, precision, mcc, fp)
```

    -------------------------
     MLPClassifier
    -------------------------
    - Accuracy: 0.977
    - Precision: 0.937
    - MCC: 0.855
    - False Positives: 44

## 5. Comparison of models

```python
results_accuracy = {
    'Threshold': metricsThreshold[0],
    'DT': metricsDT[0],
    'RF': metricsRF[0],
    'KNN': metricsKNN[0],
    'MLP': metricsMLP[0],
    'XGB': metricsXGB[0],

}
results_precision = {
    'Threshold': metricsThreshold[1],
    'DT': metricsDT[1],
    'RF': metricsRF[1],
    'KNN': metricsKNN[1],
    'MLP': metricsMLP[1],
    'XGB': metricsXGB[1],
}
results_mcc = {
    'Threshold': metricsThreshold[2],
    'DT': metricsDT[2],
    'RF': metricsRF[2],
    'KNN': metricsKNN[2],
    'MLP': metricsMLP[2],
    'XGB': metricsXGB[2],
}

results_fp = {
    'Threshold': metricsThreshold[3],
    'DT': metricsDT[3],
    'RF': metricsRF[3],
    'KNN': metricsKNN[3],
    'MLP': metricsMLP[3],
    'XGB': metricsXGB[3],
}

acc_df = pd.DataFrame.from_dict(results_accuracy, orient='index', columns=['Accuracy'])
precision_df = pd.DataFrame.from_dict(results_precision, orient='index', columns=['Precision'])
mcc_df = pd.DataFrame.from_dict(results_mcc, orient='index', columns=['MCC'])
fp_df = pd.DataFrame.from_dict(results_fp, orient='index', columns=['FP'])
results = pd.concat([acc_df, precision_df, mcc_df, fp_df], axis=1)
results
```

      Accuracy
      Precision
      MCC
      FP

      Threshold
      0.908481
      0.000000
      0.000000
      0

      DT
      0.978322
      0.893082
      0.868007
      85

      RF
      0.979104
      0.929348
      0.869823
      52

      KNN
      0.972399
      0.891781
      0.827126
      79

      MLP
      0.976981
      0.937233
      0.855058
      44

      XGB
      0.978657
      0.912073
      0.868180
      67

## 6. Improving the model

What can we do better?

- Be careful in this improvement phase that we often make the model worse.

```python
from sklearn.model_selection import GridSearchCV
# Try possible combinations of the parameters and find the best ones.

from sklearn.model_selection import RandomizedSearchCV
# Random search for the best parameters (slow and computationally intensive).

from sklearn.ensemble import StackingClassifier
# Combine algorithms and make a hybrid model.
```

More details in [Machine Learning: GridSearchCV & RandomizedSearchCV](https://towardsdatascience.com/machine-learning-gridsearchcv-randomizedsearchcv-d36b89231b10) and [Stacking made easy with Sklearn](https://towardsdatascience.com/stacking-made-easy-with-sklearn-e27a0793c92b).

## 7. Save Model (doesn't work in the colab env)

```python
import pickle
import datetime

current_date: str = datetime.date.today().strftime("%Y-%m-%d")

# save models
filename_DT = f'models/{current_date}_Pulsar_DT.mlmodel'
pickle.dump(DT, open(filename_DT, 'wb'))

filename_RF = f'models/{current_date}_Pulsar_RF.mlmodel'
pickle.dump(RF, open(filename_RF, 'wb'))

filename_KNN = f'models/{current_date}_Pulsar_KNN.mlmodel'
pickle.dump(KNN, open(filename_KNN, 'wb'))

filename_MLP = f'models/{current_date}_Pulsar_MLP.mlmodel'
pickle.dump(MLP, open(filename_MLP, 'wb'))

filename_XGB = f'models/{current_date}_Pulsar_XGB.mlmodel'
pickle.dump(XGB, open(filename_XGB, 'wb'))
```

## 7.1 Load model (doesn't work in the colab env)

```python
loaded_model = pickle.load(open(filename_RF, 'rb'))

# In real life you will use data from the production pipeline.
result = loaded_model.score(X_test, y_test)

print(f'Random Forest: {result:0.3f}')
```

Random Forest: 0.979

## 8. How to identify overfitting?

```python
maxDepths = 15

depths = [i for i in range(1, maxDepths+1)]  # +1 to include run 20
train_results, test_results = list(), list()

for i in depths:
    MODEL = RandomForestClassifier(max_depth=i, random_state=42)
    MODEL.fit(X_train, y_train)
    method = MODEL.__class__.__name__
    # train
    y_pred_train = MODEL.predict(X_train)
    train_accuracy = accuracy_score(y_train, y_pred_train)
    train_results.append(train_accuracy)
    # test
    y_pred_test = MODEL.predict(X_test)
    test_accuracy = accuracy_score(y_test, y_pred_test)
    test_results.append(test_accuracy)

# Results train vs test
plt.figure(figsize=(14, 5))
plt.plot(depths, train_results, '-o', label='Train')
plt.plot(depths, test_results, '-o', label='Test')
plt.legend(loc="best")
plt.title(f"{method} accuracy for training and testing a maximum of {maxDepths} depths")
plt.show()

```

## 9. Automating boring tasks (Choosing the best model)

```python
# importing more models

from sklearn.ensemble import AdaBoostClassifier, GradientBoostingClassifier
from sklearn.naive_bayes import GaussianNB
```

```python
results_cols = ["Classifier", "Accuracy", "Precision", "MCC", "False Positives"]
results = pd.DataFrame(columns = results_cols)

# Models
classifiers = [
    DecisionTreeClassifier(max_depth=3, random_state=42),
    RandomForestClassifier(max_depth=4, random_state=42),
    KNeighborsClassifier(metric='euclidean'),
    xgb.XGBClassifier(max_depth=4, random_state=42),
    MLPClassifier(alpha=1, max_iter=1000),
    AdaBoostClassifier(random_state=42),
    GradientBoostingClassifier(max_depth=4, random_state=42),
    GaussianNB()
]
```

```python
for clf in classifiers:
    clf.fit(X_train, y_train)
    method = clf.__class__.__name__
    y_pred = clf.predict(X_test)
    accuracy, precision, mcc, fp = evaluationModels(y_test, y_pred)
    printMetrics(clf, accuracy, precision, mcc, fp)

    temp = pd.DataFrame([[method, accuracy * 100, precision * 100, mcc * 100, fp]], columns = results_cols)
    results = results.append(temp)
```

    -------------------------
     DecisionTreeClassifier
    -------------------------
    - Accuracy: 0.978
    - Precision: 0.893
    - MCC: 0.868
    - False Positives: 85
    -------------------------
     RandomForestClassifier
    -------------------------
    - Accuracy: 0.979
    - Precision: 0.929
    - MCC: 0.870
    - False Positives: 52
    -------------------------
     KNeighborsClassifier
    -------------------------
    - Accuracy: 0.972
    - Precision: 0.892
    - MCC: 0.827
    - False Positives: 79
    -------------------------
     XGBClassifier
    -------------------------
    - Accuracy: 0.979
    - Precision: 0.912
    - MCC: 0.868
    - False Positives: 67
    -------------------------
     MLPClassifier
    -------------------------
    - Accuracy: 0.975
    - Precision: 0.936
    - MCC: 0.844
    - False Positives: 44
    -------------------------
     AdaBoostClassifier
    -------------------------
    - Accuracy: 0.978
    - Precision: 0.928
    - MCC: 0.863
    - False Positives: 52
    -------------------------
     GradientBoostingClassifier
    -------------------------
    - Accuracy: 0.977
    - Precision: 0.890
    - MCC: 0.861
    - False Positives: 87
    -------------------------
     GaussianNB
    -------------------------
    - Accuracy: 0.943
    - Precision: 0.637
    - MCC: 0.713
    - False Positives: 403

```python
for metrics in results_cols[1:]:
    plt.figure(figsize = (10,6))
    sns.barplot(x = metrics, y = 'Classifier', data = results)
    plt.xlabel(f'{metrics} %')
    plt.title(f'{metrics} of models')
    plt.show()
```

```python
results.sort_values('MCC', ascending=False)
```

      Classifier
      Accuracy
      Precision
      MCC
      False Positives

      0
      RandomForestClassifier
      97.910381
      92.934783
      86.982279
      52

      0
      XGBClassifier
      97.865683
      91.207349
      86.818047
      67

      0
      DecisionTreeClassifier
      97.832160
      89.308176
      86.800703
      85

      0
      AdaBoostClassifier
      97.809811
      92.847318
      86.311296
      52

      0
      GradientBoostingClassifier
      97.720416
      88.973384
      86.079695
      87

      0
      MLPClassifier
      97.541625
      93.595342
      84.447965
      44

      0
      KNeighborsClassifier
      97.239915
      89.178082
      82.712574
      79

      0
      GaussianNB
      94.256341
      63.726373
      71.257658
      403

```python

```


# Process scheduling

## Problem definition

A set of n jobs must be processed in a machine that can handle one job at a time.
Task j needs pj hours to be processed.
A directed and acyclic graph G = (V, E), with V = {1, . . . , n}, establishes a partial
order for job processing in the machine. That is, if there exists a path δi,j from i to
j in G, then job i must be processed before job j.
Given nonnegative weights wj , j = 1, . . . , n, in which order should we process the
jobs in order to minimize the weighted sum of the start processing time of all jobs,
while respecting the precedence order? For the modeling task that follows, sj is the
instant that job j starts to be processed.
Tasks:

(a) Formulate the problem in mixed-integer linear programming using discrete and
continuous variables.

(b) Model the problem in AMPL and solve the instance given below, in which
V = {1, . . . , 12}. Present the results.

| job (j)  | length (pj)| weight (wj) | Arcs (j,i)    |
|----|----|----|---------------|
| 1  | 3  | 5  | (1,3)         |
| 2  | 2  | 3  |               |
| 3  | 6  | 7  | (3,12), (3,7) |
| 4  | 2  | 6  |               |
| 5  | 5  | 1  |               |
| 6  | 4  | 2  | (6,7)         |
| 7  | 4  | 8  |               |
| 8  | 3  | 4  | (8,6)         |
| 9  | 10 | 7  |               |
| 10 | 1  | 1  | (10,12)       |
| 11 | 8  | 6  |               |
| 12 | 7  | 2  |               |

### Model

\begin{equation}
\min  \sum_{j =1}^{N} s_j \times w_j \\
\mathrm{s.t.} : N \in \mathbb{Z}^{+} \\
s_{j} \ge (s_i+p_i).b_{i,j} \\
s_{i} \ge (s_j+p_j).\hat{b}_{i,j} \\
s_j \in \mathbb{R}_{\ge 0} \space \forall j \\
p_j \in \mathbb{R}_{+} \\
w_j \in \mathbb{R}_{+} \\
b_{i,j} \in \{0,1\} \space \forall i,j \\
\hat{b}_{i,j} \in \{0,1\} \space \forall i,j \\
b_{i,j}=1-\hat{b}_{i,j}\space \forall i,j \\
b_{j,j}=0\space \forall j \\
arc_{i,j} \in \{0,1\} \space \forall i,j \\
b_{i,j} \ge arc_{i,j}
\end{equation}

### Modeling (instance)

```python
import gurobipy as gp
from gurobipy import GRB, Model
import pandas as pd
import numpy as np
```

#### Create supporting (instance) data

```python
N = 12; # 'size' of the problem, here it's both the number of processes and number of slots

# Create a list with the well indexes as given by the problem (prevents python's 0-indexing)
V = [item for item in range(1, N+1)]

# supporting data:
w = {j:[5,3,7,6,1,2,8,4,7,1,6,2][j-1] for j in V} # these are the given process 'weights' (importances)
p = {j:[3,2,6,2,5,4,4,3,10,1,8,7][j-1] for j in V} # these are the given process time lenghts

"""
'dependance table'
     p1 p2 p3 p4
p1  [0  1  0  0] <- p1 must come before p2
p2  [0  0  0  0]
p3  [1  0  0  0] <- p3 must come before p1
p4  [0  0  0  0]
"""
D = np.zeros((N,N))
arcs = ((1,3),(3,12),(3,7),(6,7),(8,6),(10,12)) # these arcs are in order (i,j), where i depends on j
for arc in arcs:
    x=int(arc[0]) - 1 # minus one is because of python 0 indexing
    y=int(arc[1]) - 1 # minus one is because of python 0 indexing
    D[x][y] = 1
print('dependance table\n',D)
```

    dependance table
     [[0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 1.]
     [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]
     [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]

#### Model creation

```python

m = Model("process_scheduling")

# Decision variables:
# 1)represents the starting time of each process
'''
s: [1, 3, 4, ...] process 1 starts at time=1, process 2 starts at time=3...
'''
s = m.addVars(V, vtype=GRB.CONTINUOUS, name="s")

# 2)decision varibles. If B[i,j]=1, then process i comes before process j
"""
'comes before table'
     p1 p2 p3 p4
p1  [0  1  1  1] p1 comes before p2, p3 and p4 - first
p2  [0  0  0  0] p2 comes before no one-last
p3  [0  1  0  1] p3 comes before p2 and p4 - second
p4  [0  1  0  0] p4 - third
p1->p3->p4->p2
"""

B=m.addVars(V, V, vtype=GRB.BINARY, name='B')
NB=m.addVars(V, V, vtype=GRB.BINARY, name='NB') # negation of B

m.update()
```

    Set parameter Username
    Academic license - for non-commercial use only - expires 2022-05-20

#### Constraints

```python
#Constraints over the binary tables
for i in V:
    for j in V:
        m.addConstr(B[i,j]==1-NB[i,j], name=f"BNB{i}_{j}") # if job i comes before job j, then job j cannot come before job i

m.update()
```

```python
# Constraints related to the
# 1)non-overlaping usages of resources
for j in V:
    for i in (set(V)-set([j])):
        m.addConstr(s[j]>=(s[i]+p[i])*B[i,j])
        m.addConstr(s[i]>=(s[j]+p[j])*NB[i,j])

# 2)given process precedence requirements
for i in V:
    for j in V:
        Di = i - 1
        Dj = j - 1
        if D[Di][Dj] == 1:
            m.addConstr(B[i,j]==1,name=f'arc({i},{j})')

# 3) a job cannot happen before itself
for i in V:
    m.addConstr(B[i,i]==0)

m.update()
```

#### Solution

```python
m.setObjective(sum([s[j]*w[j] for j in V]), GRB.MINIMIZE)

m.update()
```

```python
m.optimize()
```

    Gurobi Optimizer version 9.5.1 build v9.5.1rc2 (win64)
    Thread count: 2 physical cores, 4 logical processors, using up to 4 threads
    Optimize a model with 162 rows, 300 columns and 306 nonzeros
    Model fingerprint: 0x7ca0adbe
    Model has 264 quadratic constraints
    Variable types: 12 continuous, 288 integer (288 binary)
    Coefficient statistics:
      Matrix range     [1e+00, 1e+00]
      QMatrix range    [1e+00, 1e+00]
      QLMatrix range   [1e+00, 1e+01]
      Objective range  [1e+00, 8e+00]
      Bounds range     [1e+00, 1e+00]
      RHS range        [1e+00, 1e+00]
    Presolve removed 156 rows and 162 columns
    Presolve time: 0.00s
    Presolved: 762 rows, 894 columns, 2154 nonzeros
    Presolved model has 504 SOS constraint(s)
    Variable types: 516 continuous, 378 integer (378 binary)

    Root relaxation: objective 2.478783e+02, 506 iterations, 0.01 seconds (0.00 work units)

        Nodes    |    Current Node    |     Objective Bounds      |     Work
     Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time

         0     0  247.87830    0  244          -  247.87830      -     -    0s
         0     0  274.41538    0  228          -  274.41538      -     -    0s
         0     0  274.41538    0  228          -  274.41538      -     -    0s
         0     0  282.19077    0  224          -  282.19077      -     -    0s
         0     0  282.19077    0  193          -  282.19077      -     -    0s
         0     0  284.40157    0  161          -  284.40157      -     -    0s
         0     0  284.40157    0  164          -  284.40157      -     -    0s
         0     0  285.50287    0  197          -  285.50287      -     -    0s
         0     0  286.81818    0  198          -  286.81818      -     -    0s
         0     0  286.84706    0  211          -  286.84706      -     -    0s
         0     0  289.66241    0  210          -  289.66241      -     -    0s
         0     0  290.34697    0  206          -  290.34697      -     -    0s
         0     0  290.60709    0  232          -  290.60709      -     -    0s
         0     0  290.86322    0  228          -  290.86322      -     -    0s
         0     0  290.86322    0  193          -  290.86322      -     -    0s
         0     2  290.86322    0  190          -  290.86322      -     -    0s
    *  257   113              70    1129.0000000  374.20000  66.9%   4.7    0s
    H  293   146                    1120.0000000  374.20000  66.6%   4.6    0s
    H  324   145                     902.0000000  374.20000  58.5%   5.0    0s
    *  329   145              78     898.0000000  374.20000  58.3%   5.0    0s
    H  353   158                     869.0000000  374.20000  56.9%   5.1    0s
    H 1204   509                     868.0000000  423.00000  51.3%   5.1    0s
    H 1652   625                     865.0000000  427.00000  50.6%   8.2    1s
    * 2373   666              54     861.0000000  466.20000  45.9%   9.8    2s
    H 2398   633                     860.0000000  466.20000  45.8%   9.8    2s
    H 3600   840                     859.0000000  509.03597  40.7%   9.8    3s
    H 4532  1115                     856.0000000  529.66447  38.1%   9.9    3s
      6164  1489  829.00000   42    7  856.00000  555.20955  35.1%  10.0    5s
     17490  2545  684.00000   41   13  856.00000  656.00000  23.4%   9.7   10s
     30835  1776 infeasible   50       856.00000  748.00000  12.6%   9.4   15s

    Cutting planes:
      Gomory: 3
      MIR: 27

    Explored 37097 nodes (340886 simplex iterations) in 16.78 seconds (7.12 work units)
    Thread count was 4 (of 4 available processors)

    Solution count 10: 856 859 860 ... 1120

    Optimal solution found (tolerance 1.00e-04)
    Best objective 8.560000001483e+02, best bound 8.560000001483e+02, gap 0.0000%

```python
for j in V:
    start=round(s[j].x)
    end = start+p[j]
    print(f'Process {j} starts at {start} and ends at {end}')
```

    Process 1 starts at 2 and ends at 5
    Process 2 starts at 5 and ends at 7
    Process 3 starts at 10 and ends at 16
    Process 4 starts at 0 and ends at 2
    Process 5 starts at 50 and ends at 55
    Process 6 starts at 16 and ends at 20
    Process 7 starts at 20 and ends at 24
    Process 8 starts at 7 and ends at 10
    Process 9 starts at 33 and ends at 43
    Process 10 starts at 24 and ends at 25
    Process 11 starts at 25 and ends at 33
    Process 12 starts at 43 and ends at 50


CRP dynamically rebalances Collateral to ensure the ayToken minted (i.e. the loan) remains solvent, especially in an adverse market environment (i.e. the value of the loan does not exceed the value of Collateral). This dynamic rebalancing, together with a careful choice of the key parameters (including LTV and volatility assumption) allows ALEX to eliminate the liquidation needs. Any residual gap risk (which CRP cannot address entirely) is addressed through maintaining a strong reserve fund.
When a Borrower mints ayToken by providing appropriate Collateral, the Collateral is converted into a basket of Collateral and Token, with the weights determined by CRP. CRP determines the weights based on the prevailing LTV and uses the following formula:

$$ w_{Token} = N(d_1) $$
$$ w_{Collateral} = (1 - w_{Token}) $$

\begin{equation}
d_1 = \frac{1}{\sigma \sqrt{(t)}} [ln (\frac{LTV_t}{LTV_0}) + t \cdot (APY_{Token} - APY_{Collateral} + \frac{\sigma^2}{2})]
\label{eq:CRP} \tag{1}
\end{equation}

We conduct a number of simulations to help understand the loan values and default risk (i.e LTV >1) in various market environments.  For simplicity, fees and APY are not considered in this round of simulation. We set initial weights to be 50/50 and loan life time = 91 days.

### Key parameters
In this simualtion, we consider the following key parameters:

    (1) Initial LTV: range from 0.7 to 0.95
    (2) Black scholes volatility: $\sigma$ range from 0.1 - 0.5

### Market enviroment
To simulate the different markets, we assume the token price move at a linear annual growth rate (r) plus random walk with volatility $\eta$.  We allow the growth rate to vary from -2 to 2 (corresponding to 25% or 200% of the initial price). The token price's drop won't pressure the LTV(t), but will affect the total pool value.

    (3) Growth rate: $r$ range from $[-2, 2]$
    (4) Growth rate volatility: $\eta$ range from 0.1 - 0.5

### Measures

For each setup [1-4], based on the token price change, we rebalance Collateral and token according to $\eqref{eq:CRP}$.  The following metrics are measures during the whole loan life term:

a. LTV at time t
b. Token price at t
c. Collateral weight at t
d. Impermanent loss of the pool LP.
    $$IL = \frac{PoolValue_{usd}}{HoldValue_{usd}}$$

For example the Token price moves up with low volatility.

#### Simulation 2:
For each selected combination (1) to (4), run the process 100 times and count the number of events that LVT(t) > 1 at any time point.  Report the frequency by LTV0 and growth rate for a given volatility setup.

For example this is the simulation results when growth rate volatility is high, and black sholes volatility is under-estimated.
The size of button indicate the chance of default when LVT is hight, even the growth rate is low.

URLhttps://medium.com/balancer-protocol/80-20-balancer-pools-ad7fed816c8d

```python
# plot of liquity

import scipy
import matplotlib.pyplot as plt
import matplotlib.animation
from matplotlib.widgets import Slider
import seaborn as sns
import numpy as np
import random
import pandas as pd
from ipywidgets import *
from scipy.stats import norm

```

```python
def gen_data(t, BTC, Collateral, LTV0, bs_vol, Growth_rate, Real_vol):
    for i in range(0, len(t)):
        if i == 0:
            btc_usd = BTC
            borrow_btc_usd = Collateral*LTV0
            ltv = LTV0
            collateral_usd = Collateral
            w_btc = norm.cdf((np.log(btc_usd)-np.log(BTC)+ t[i]*(0.5*bs_vol**2))/bs_vol/np.sqrt(t[i]))
            btc_locked = collateral_usd/btc_usd* w_btc
            usd_locked = collateral_usd*(1-w_btc)
            loss = 1
            out = np.array([t[i], btc_usd, borrow_btc_usd, ltv, btc_locked, usd_locked, collateral_usd, w_btc, loss])
            out = out.reshape(1,9)
        else:
            ti = t[i-1]-t[i]
            rw = random.random()
            btc_usd_previous = btc_usd
            btc_usd = btc_usd_previous*np.exp(ti * (Growth_rate - 0.5*Real_vol**2) +  Real_vol *np.sqrt(ti)*norm.ppf(rw))
            borrow_btc_usd = borrow_btc_usd/btc_usd_previous*btc_usd
            w_btc = norm.cdf((np.log(btc_usd)-np.log(BTC)+ t[i]*(0.5*bs_vol**2))/bs_vol/np.sqrt(t[i]))
            collateral_usd = btc_locked * btc_usd + usd_locked
            ltv = borrow_btc_usd/collateral_usd
            btc_locked = collateral_usd/btc_usd*w_btc
            usd_locked = collateral_usd * (1-w_btc)
            holdvalue = Collateral #/2 +  Collateral/2/BTC*btc_usd
            loss = collateral_usd/holdvalue
            _out = np.array([t[i], btc_usd, borrow_btc_usd, ltv, btc_locked, usd_locked, collateral_usd, w_btc, loss])

            out = np.append(out, _out.reshape(1,9), axis =0)
    output = pd.DataFrame(data=out)
    output.columns = ['t', 'btc_usd', 'borrow_btc_usd', 'ltv', 'btc_locked', 'usd_locked', 'collateral_usd', 'w_btc', 'imper_loss']
    output['pool_value_ratio'] = output['collateral_usd']/Collateral
    return output
```

```python
# Simulation 1
#set initial parameters
Growth_rate = 2
Real_vol = 0.4
Collateral = 10000
LTV0 = 0.95
bs_vol = 0.2
BTC = 50
t = np.linspace(91,1,91)/365

```

```python
%matplotlib notebook

import matplotlib.pyplot as plt
#fig = plt.figure(figsize=(6,6))
fig, axs = plt.subplots(2,2,figsize=(8,10))
# Create main axis
fig.subplots_adjust(bottom=0.2, top=0.80)
# Create axes for sliders
ax_growth = fig.add_axes([0.3, 0.82, 0.4, 0.02])
ax_growth.spines['top'].set_visible(True)
ax_growth.spines['right'].set_visible(True)

ax_real_vol = fig.add_axes([0.3, 0.87, 0.4, 0.02])
ax_real_vol.spines['top'].set_visible(True)
ax_real_vol.spines['right'].set_visible(True)

ax_ltv = fig.add_axes([0.3, 0.92, 0.4, 0.02])
ax_ltv.spines['top'].set_visible(True)
ax_ltv.spines['right'].set_visible(True)

ax_bs_vol = fig.add_axes([0.3, 0.97, 0.4, 0.02])
ax_bs_vol.spines['top'].set_visible(True)
ax_bs_vol.spines['right'].set_visible(True)

# Create sliders
s_real_vol= Slider(ax=ax_real_vol, label='Real Vol', valmin=0.1, valmax=1.0, valinit=0.2, valfmt=' %1.2f ', facecolor='#cc7000')
s_growth = Slider(ax=ax_growth, label='Growth', valmin=-2, valmax=2, valinit=2, valfmt=' %1.2f', facecolor='#cc7000')
s_bs_vol= Slider(ax=ax_bs_vol, label='BS Vol', valmin=0.1, valmax=1.0, valinit=0.2, valfmt=' %1.2f ', facecolor='#cc7000')
s_ltv = Slider(ax=ax_ltv, label='LTV0', valmin=0.7, valmax=0.95, valinit=0.9, valfmt=' %1.2f', facecolor='#cc7000')

# Plot default data
t =  np.linspace(91,1,91)/365
t
real_vol_0 = 0.2
growth_0 = 2
ltv0 = 0.9
bs_vol0 = 0.2
_data = gen_data(t, BTC, Collateral, ltv0, bs_vol0, growth_0, real_vol_0)
x = _data.index.values
y1 = _data['ltv']
y2 = _data['btc_usd']/BTC
y3 = 1-_data['w_btc']
y4 = _data['imper_loss']
y5 = _data['w_btc']

f_d1, = axs[0,0].plot(x, y1, linewidth=2.5)
f_d2, = axs[0,1].plot(x, y2, linewidth=2.5)
f_d3, = axs[1,0].plot(x, y3, linewidth=2.5)
f_d4, = axs[1,1].plot(x, y4, linewidth=2.5)
f_d5, = axs[1,0].plot(x, y5, linewidth=2.5)
axs[0,0].set_ylim(0.7,1.05)
axs[0,0].axhline(y = 1, color = 'r', linestyle = '--')
axs[0,1].set_ylim(0.5,2)
axs[1,0].set_ylim(-0.01,1.01)
axs[1,1].set_ylim(0.7,2)
axs[0,0].set_ylabel('Loan-to-Value')
axs[0,1].set_ylabel('USDC in BTC')
axs[1,0].set_ylabel('Weights (USDC: Blue, BTC: Orange)')
axs[1,1].set_ylabel('Pool value in BTC as % of initial collateral value')

axs[0,0].set_xlabel('Day')
axs[0,1].set_xlabel('Day')
axs[1,0].set_xlabel('Day')
axs[1,1].set_xlabel('Day')

# Update values
def update(val):
    _real_vol = s_real_vol.val
    _growth = s_growth.val
    _bs_vol = s_bs_vol.val
    _ltv = s_ltv.val

    _data = gen_data(t, BTC, Collateral, _ltv, _bs_vol, _growth, _real_vol)
    x = _data.index.values
    _y1 = _data['ltv']
    _y2 = _data['btc_usd']/BTC
    _y3 = 1-_data['w_btc']
    _y4 = _data['imper_loss']
    _y5 = _data['w_btc']

    f_d1.set_data(x,_y1)
    f_d2.set_data(x,_y2)
    f_d3.set_data(x,_y3)
    f_d4.set_data(x,_y4)
    f_d5.set_data(x,_y5)
    fig.canvas.draw_idle()

s_real_vol.on_changed(update)
s_growth.on_changed(update)
s_ltv.on_changed(update)
s_bs_vol.on_changed(update)

# Set axis labels
# ax1.set_xlabel('Day')
# ax1.set_ylabel('LTV')

plt.show()

```

    <IPython.core.display.Javascript object>

```python
# import pickle as pl
# #pl.dump(fig, file = ('./ALEX/CRP.pickle', 'w'))
# with open('./ALEX/CRP.pickle', 'wb') as f:
#     pl.dump(fig,f)

```

```python

```

    0.9611258693651961

```python
# import sys
# # !{sys.executable} -m pip install mpld3
# import mpld3
# mpld3.show(fig)
```

## simulation 2

```python
# simulation 2
def ltv_simulation(s, LTV0, bs_vol, Growth_rate, Real_vol):
    freq = 0
    for k in range(0,s):
        _out = gen_data(t, BTC, Collateral, LTV0, bs_vol, Growth_rate, Real_vol)
        #print(max(_out['ltv']))
        freq+= (max(_out['ltv'])>=1)
    return freq/s
#ltv_simulation(100, LTV0, bs_vol, Growth_rate, Real_vol)
```

```python
# Generate whole simulation output table for different combinations of parameters

xyz = []
i =0
for real_vol in np.linspace(0.1, 1.0, 11):
     for growth in np.linspace(0.3, 3, 10):
         for ltv in np.linspace(0.7, 0.95, 11):
             for bs_vol in np.linspace(0.1, 0.6, 10):
                 if i%100 == 0:
                     print(i)
                 z = ltv_simulation(50, ltv, bs_vol, growth, real_vol)
                 xyz.append([real_vol, growth, ltv, bs_vol, z])
                 i+=1
_data = pd.DataFrame(np.array(xyz))
_data.columns = ['real_vol', 'growth', 'ltv', 'bs_vol', 'default_risk']
_data.to_csv('simulation2_out.csv')

```

    0
    100
    200
    300
    400
    500
    600
    700
    800
    900
    1000
    1100
    1200
    1300
    1400
    1500
    1600
    1700
    1800
    1900
    2000
    2100
    2200
    2300
    2400
    2500
    2600
    2700
    2800
    2900
    3000
    3100
    3200
    3300
    3400
    3500
    3600
    3700
    3800
    3900
    4000
    4100
    4200
    4300
    4400
    4500
    4600
    4700
    4800
    4900
    5000
    5100
    5200
    5300
    5400
    5500
    5600
    5700
    5800
    5900
    6000
    6100
    6200
    6300
    6400
    6500
    6600
    6700
    6800
    6900
    7000
    7100
    7200
    7300
    7400
    7500
    7600
    7700
    7800
    7900
    8000
    8100
    8200
    8300
    8400
    8500
    8600
    8700
    8800
    8900
    9000
    9100
    9200
    9300
    9400
    9500
    9600
    9700
    9800
    9900
    10000
    10100
    10200
    10300
    10400
    10500
    10600
    10700
    10800
    10900
    11000
    11100
    11200
    11300
    11400
    11500
    11600
    11700
    11800
    11900
    12000

```python
df_plot = pd.read_csv('simulation2_out.csv')
df_plot.drop('Unnamed: 0', 1, inplace=True)
df_plot.describe()
```

      real_vol
      growth
      ltv
      bs_vol
      default_risk

      count
      12100.000000
      12100.00000
      12100.00000
      12100.000000
      12100.000000

      mean
      0.550000
      1.65000
      0.82500
      0.350000
      0.181221

      std
      0.284617
      0.86172
      0.07906
      0.159578
      0.311253

      min
      0.100000
      0.30000
      0.70000
      0.100000
      0.000000

      25%
      0.280000
      0.90000
      0.75000
      0.211111
      0.000000

      50%
      0.550000
      1.65000
      0.82500
      0.350000
      0.000000

      75%
      0.820000
      2.40000
      0.90000
      0.488889
      0.225000

      max
      1.000000
      3.00000
      0.95000
      0.600000
      1.000000

```python
def get_simulation(LTV0, bs_vol):
    xyz = []
    for real_vol in np.linspace(0.1, 0.5, 5):
        for growth in np.linspace(1, 3, 4):
            z = ltv_simulation(10, LTV0, bs_vol, growth, real_vol)
            xyz.append([real_vol, growth, z])
    _data = pd.DataFrame(np.array(xyz))
    _data.columns = ['x', 'y', 'size']
    return _data

```

```python
%matplotlib inline
from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets
import matplotlib.pyplot as plt, random

def series(real_vol, bs_vol):
    _x, _y = 'growth', 'ltv'
    _df = df_plot[(df_plot.real_vol <real_vol+1e-4) &(df_plot.real_vol >real_vol-1e-4) \
                  & (df_plot.bs_vol< bs_vol+1e-4) & (df_plot.bs_vol>bs_vol-1e-4)]
    x, y, z = _df[_x], _df[_y], _df['default_risk']*100
    sc = plt.scatter(x,y, s=z)
    #plt.legend(*sc.legend_elements("sizes", num = 10))
    plt.xlabel('growth (of borrowed token price)')
    plt.ylabel('ltv')
    plt.xlim(0,4)
    return()
plt.rcParams['figure.figsize'] = (10,6)
fig2 = interact(series, real_vol=(0.1, 0.5, 0.04), bs_vol = (0.1, 0.5, 0.04))
```

    interactive(children=(FloatSlider(value=0.30000000000000004, description='real_vol', max=0.5, min=0.1, step=0.…

```python
#import sys
#!{sys.executable} -m pip install celluloid
```

    Collecting celluloid
      Downloading celluloid-0.2.0-py3-none-any.whl (5.4 kB)
    Requirement already satisfied: matplotlib in /Users/chanahn/python3/lib/python3.7/site-packages (from celluloid) (3.2.2)
    Requirement already satisfied: python-dateutil>=2.1 in /Users/chanahn/python3/lib/python3.7/site-packages (from matplotlib->celluloid) (2.8.1)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /Users/chanahn/python3/lib/python3.7/site-packages (from matplotlib->celluloid) (2.4.7)
    Requirement already satisfied: numpy>=1.11 in /Users/chanahn/python3/lib/python3.7/site-packages (from matplotlib->celluloid) (1.19.2)
    Requirement already satisfied: cycler>=0.10 in /Users/chanahn/python3/lib/python3.7/site-packages (from matplotlib->celluloid) (0.10.0)
    Requirement already satisfied: kiwisolver>=1.0.1 in /Users/chanahn/python3/lib/python3.7/site-packages (from matplotlib->celluloid) (1.2.0)
    Requirement already satisfied: six in /Users/chanahn/python3/lib/python3.7/site-packages (from cycler>=0.10->matplotlib->celluloid) (1.15.0)
    Installing collected packages: celluloid
    Successfully installed celluloid-0.2.0
    WARNING: You are using pip version 21.0.1; however, version 21.1.3 is available.
    You should consider upgrading via the '/Users/chanahn/python3/bin/python -m pip install --upgrade pip' command.

```python

import matplotlib.pyplot as plt
from matplotlib import cm
import numpy as np
from celluloid import Camera
import matplotlib.animation

camera = Camera(plt.figure())
_x, _y = 'growth', 'ltv'
i,j = 0,0
for real_vol in np.linspace(0.1, 0.5, 11):
    i += 1
    for bs_vol in np.linspace(0.1, 0.5, 11):
        j += 1
        _df = df_plot[(df_plot.real_vol <real_vol+1e-4) &(df_plot.real_vol >real_vol-1e-4) \
                      & (df_plot.bs_vol< bs_vol+1e-4) & (df_plot.bs_vol>bs_vol-1e-4)]
        x, y, z = _df[_x], _df[_y], _df['default_risk']*100
        sc = plt.scatter(x,y, s=z)
        plt.legend(*sc.legend_elements("sizes", num = 10))
        plt.xlabel('growth (of borrowed token price)')
        plt.ylabel('ltv')
        plt.title('Observed freq. of default (LTV>=1)')
        plt.xlim(0,4)
        plt.text(0.5, 0.8, "real_vol={:.2%}, bs_vol= {:.2%}".format(real_vol, bs_vol))
        if i%3 ==0 & j%3 ==0:
            camera.snap()
camera.snap()
anim = camera.animate()
writergif = matplotlib.animation.PillowWriter(fps=3)
anim.save('default_simulation.gif', writer=writergif)
# numpoints = 10
# points = np.random.random((2, numpoints))
# colors = cm.rainbow(np.linspace(0, 1, numpoints))
# camera = Camera(plt.figure())
# for _ in range(100):
#     points += 0.1 * (np.random.random((2, numpoints)) - .5)
#     plt.scatter(*points, c=colors, s=100)
#     camera.snap()

```

```python

```