## SurfinPy

#### Tutorial 5 - Chemical Potential vs Chemical Potential Including Vibration Properties

In the Tutorial 2 we showed how experimental data could be used to determine the temperature dependent free energy term for gaseous species and then plot a phase diagram that represents 298 K.  In this example we will expand this methodology to calculate the vibrational properties for solid phases (<i>i.e.</i> zero point energy, vibrational entropy) and include these values in the generation of the phase diagrams.  This allows for a more accurate calculation of phase diagrams without the need to include experimental corrections for solid phases.

##### Methodoloy

As with tutorial 2 for gaseous species, the standard free energy varies significantly with temperature, and as DFT simulations are designed for condensed phase systems, we use experimental data to determine the temperature dependent free energy term for gaseous species obtained from the NIST database.  In addition we also calculate the vibrational properties for the solid phases modifying the free energy (G) for solid phases to be;

\begin{align}
\Delta G_f = U_0 + U_{\text{ZPE}} -T\Delta S_{\text{vib}}
\end{align}

$U_0$ is the calculated internal energy from a DFT calculation, $U_{\text{ZPE}}$ is the zero point energy and $S_{\text{vib}}$ is the vibrational entropy.

\begin{align}
	U_{\text{ZPE}} = \sum_i^{3n} \frac{R \theta_i}{2}
\end{align}

\begin{align}
S_{\text{vib}} = \frac{U_{\text{vib}} - A_{\text{vib}}}{T}
\end{align}

where $A_{\text{vib}}$ and $U_{\text{vib}}$ is the vibrational Helmholtz free energy and vibrational internal energy and defined as;

\begin{align}
U_{\text{vib}} =  \sum_i^{3n} \frac{R\theta}{e^{(\theta/T)}-1}
\end{align}

\begin{align}
	A_{\text{vib}} = \sum_i^{3n} RT \ln{(1-e^{-\theta_i/T})}
\end{align}

3n is the total number of vibrational modes, n is the number of species and $\theta_i$ is the characteristic vibrational temperature (frequency of the vibrational mode in Kelvin).

\begin{equation}
	\theta_i = \frac{h\nu_i}{k_B}
	\label{eq:theta}
\end{equation}

##### code

```python
import matplotlib.pyplot as plt
from surfinpy import bulk_mu_vs_mu as bmvm
from surfinpy import utils as ut
from surfinpy import data
```

```python
temperature_range = [298, 299]
```

SurfinPy has the function to calculate the zero point energy and the vibrational entropy when entropy and zpe = True.

```python
bulk = data.ReferenceDataSet(cation = 1, anion = 1, energy = -92.0, funits = 10, file = 'bulk_vib.yaml', entropy=True, zpe=True, temp_range=temperature_range)
```

    /opt/anaconda3/lib/python3.7/site-packages/surfinpy-1.0.3-py3.7.egg/surfinpy/utils.py:182: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
      vib_prop = yaml.load(file)

In addition to entropy and zpe keyword you must provide the a file containing the vibrational modes and number of formula units used in taht calculations.  You must create the yaml file using the following format

```
    F-Units : number
    Frequencies :
    - mode1
    - mode2

```

Vibrational modes can be calculated via a density functional pertibation calculation or via the phonopy code.

```python

MgO = data.DataSet(cation = 10, x = 0, y = 0, energy = -92,
                   label = "Periclase", entropy = True, zpe=True, file = 'MgO_vib.yaml',
                   funits = 10, temp_range=temperature_range)

Art = data.DataSet(cation = 10, x = 5, y = 20, energy = -468,
                   label = "Artinite", entropy = True, zpe=True, file = 'Art_vib.yaml',
                   funits = 5, temp_range=temperature_range)

Bru = data.DataSet(cation = 10, x = 0, y = 10, energy = -228,
                   label = "Brucite", entropy = True, zpe=True, file = 'Bru_vib.yaml',
                   funits =  10, temp_range=temperature_range)

Nes = data.DataSet(cation = 10, x = 10, y = 30, energy = -706,
                   label = "Nesquehonite", entropy = True, zpe=True, file = 'Nes_vib.yaml',
                   funits = 10, temp_range=temperature_range)

Mag = data.DataSet(cation = 10, x = 10, y = 0, energy = -310,
                   label = "Magnesite", entropy = True, zpe=True,  file = 'Mag_vib.yaml',
                   funits =  10, temp_range=temperature_range)

Lan = data.DataSet(cation = 10, x = 10, y = 50, energy = -972,
                   label = "Lansfordite", entropy = True, zpe=True, file = 'Lan_vib.yaml',
                   funits =  10, temp_range=temperature_range)

Hyd = data.DataSet(cation = 10, x = 8, y = 10, energy = -398,
                   label = "Hydromagnesite", entropy = True, zpe=True, file = 'Hyd_vib.yaml',
                   funits =  2, temp_range=temperature_range)
```

```python
data = [MgO, Art, Bru, Nes,  Mag, Lan, Hyd]
```

```python
x_energy=-20.53412969
y_energy=-12.83725889
```

```python
CO2_exp = ut.fit_nist("CO2.txt")[298]
Water_exp = ut.fit_nist("H2O.txt")[298]

CO2_corrected = x_energy + CO2_exp
Water_corrected = y_energy + Water_exp

deltaX = {'Range': [ -1, 0.6],  'Label': 'CO_2'}
deltaY = {'Range': [ -1, 0.6], 'Label': 'H_2O'}
```

```python
temp_298 = bmvm.calculate(data, bulk, deltaX, deltaY, CO2_corrected, Water_corrected)
ax = temp_298.plot_mu_p(temperature=298, colourmap="RdBu", figsize=(6, 4))

```

```python

```


```python
from IPython.core.display import HTML
from IPython.display import Image
HTML("""
<style>
.output_png {
    display: table-cell;
    text-align: center;
    vertical-align: middle;
}
</style>
""")
```

<style>
.output_png {
    display: table-cell;
    text-align: center;
    vertical-align: middle;
}
</style>

# *Circuitos Elétricos I*

## Semana 2 - Circuitos divisores

### Carregamento de um circuito de alimentação

Quando uma fonte de tensão fornece energia a um dado elemento de circuito, diz-se que a fonte encontra-se *carregada*, ou que o elemento conectado à fonte é uma *carga* que está sendo alimentada pela fonte. De maneira geral, o carregamento de um circuito é o processo de introdução de elementos que extrairão corrente do circuito. Quanto maior for a  magnitude da corrente drenada, maior será o efeito de carregamento.

Da relação do divisor de tensão, temos que:

$$ \begin{align} v_x &= \frac{R_x}{R_{eq}}v_s = \frac{R_x}{R_x+R_y}v_s\\
                 v_y &= \frac{R_y}{R_{eq}}v_s = \frac{R_y}{R_x+R_y}v_s
    \end{align}$$

Ao conectar uma carga $R_L$ ao resistor $R_y$, temos que a tensão $v_y$ será dada por:

$$\begin{align} v_y &= \frac{(R_y||R_L)}{R_{eq}}v_s = \frac{(R_y||R_L)}{R_x+(R_y||R_L)}v_s\\ \\
                v_y &= \frac{\frac{R_yR_L}{R_y+R_L}}{R_x+\frac{R_yR_L}{R_y+R_L}}v_s = \frac{R_yR_L}{R_x(R_y+R_L)+R_yR_L}v_s = \frac{R_y}{R_x(\frac{R_y}{R_L}+1)+R_y}v_s
\end{align}$$

Note que, se $R_L \gg R_y$, a tensão $v_y$ com a carga conectada aproxima-se da tensão obtida com os terminais em aberto.

### Visualizando o comportamento do circuito alimentador em função da carga conectada

```python
import numpy as np
import matplotlib.pyplot as plt

# parâmetros do circuito
Vs = 15
Rx = 100
Ry = 200

# cálculo das tensões com o divisor de tensão carregado
# com uma carga RL
R = np.arange(0,4000,25)
Vy = np.zeros(R.shape)
Vx = np.zeros(R.shape)
iL = np.zeros(R.shape)

for k, RL in enumerate(R):
    if RL == 0:
        Vy[k] = 0
        iL[k] = Vs/Rx
    else:
        Vy[k] = ( Ry/( Rx * (Ry/RL + 1) + Ry ) )*Vs
        iL[k] = Vy[k]/RL

    Vx[k] = Vs-Vy[k]

plt.figure(figsize=(12,5))
plt.plot(R, Vy,'--', label='Tensão $v_y$ sobre $R_L$||$R_y$', linewidth=2)
plt.plot(R, Vx,'--' , label='Tensão $v_x$ sobre $R_x$', linewidth=2)
plt.hlines((Ry/(Rx+Ry))*Vs, R.min(0),R.max(0),'k', label = 'Tensão $v_y$ (sem carregamento)')
plt.hlines((Rx/(Rx+Ry))*Vs, R.min(0),R.max(0),'r', label = 'Tensão $v_x$ (sem carregamento)')

plt.legend()
plt.xlabel('$R_L$ [Ω]')
plt.ylabel('[V]')
plt.grid()

plt.figure(figsize=(12,4))
plt.plot(R, iL,'--', label='$i_L(R_L)$', linewidth=2)

plt.legend()
plt.xlabel('$R_L$ [Ω]')
plt.ylabel('$i_L$ [A]')
plt.grid();

Image("figures/J3C0.png", width=300)
```

**Pergunta**: como os gráficos acima podem ser utilizados para explicar os riscos de conectar diversos equipamentos elétricos numa mesma tomada?

## Exercícios

### Exercício 1:

a. No circuito abaixo, $V_s=$120 V, determine o valor das tensões nos terminais $a$, $b$ e $c$ indicados, com relação ao potencial de referência.

b. Determine o valor das tensões nos terminais indicados considerando que cada um deles alimenta uma carga de 20 $\Omega$.

### Exercício 2:

Determine o valor de $R_1$, $R_2$ e $R_3$ no circuito divisor de tensão ilustrado na figura seguinte. O circuito pode ser projetado utilizando resistores de 2 W?

Jupyter notebooks/Circuitos Eletricos I - Semana 2.2.ipynb


Table of Contents

1  Alternating Least Squares with Weighted Regularization

1.1  Matrix Factorization

2  Reference

```python
# code for loading the format for the notebook
import os

# path : store the current path to convert back to it later
path = os.getcwd()
os.chdir(os.path.join('..', 'notebook_format'))
from formats import load_style
load_style(css_style = 'custom2.css', plot_style = False)
```

```python
os.chdir(path)

# 1. magic for inline plot
# 2. magic to print version
# 3. magic so that the notebook will reload external python modules
# 4. magic to enable retina (high resolution) plots
# https://gist.github.com/minrk/3301035
%matplotlib inline
%load_ext watermark
%load_ext autoreload
%autoreload 2
%config InlineBackend.figure_format = 'retina'

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from subprocess import call
from sklearn.metrics import mean_squared_error

%watermark -a 'Ethen' -d -t -v -p numpy,pandas,matplotlib,sklearn
```

    Ethen 2018-04-07 14:27:09

    CPython 3.6.4
    IPython 6.2.1

    numpy 1.14.2
    pandas 0.22.0
    matplotlib 2.2.2
    sklearn 0.19.1

# Alternating Least Squares with Weighted Regularization

Recommendation system is a popular topic in recent years, what is does (or its goal) is to seek to predict the "rating" or "preference" that a user would give to an item. Given the rating, the business will then recommend preferable new items to the user for further purchases. The type of recommendation system that we will be discussing is known as **collaborative filtering**, where the features of the user (e.g. age, gender) or the item (e.g. perishable or not) itself do not play any role in the algorithm. Instead, we rely solely on the ratings that a user gave to the item.

We start by loading some sample data to make this a bit more concrete. For this introduction, we'll be using the [MovieLens dataset](http://grouplens.org/datasets/movielens/). The website has datasets of various sizes, but we just start with the smallest one [MovieLens 100K Dataset](http://files.grouplens.org/datasets/movielens/ml-100k.zip). This dataset consists of 100,000 movie ratings by users (on a 1-5 scale). The main data set `u.data` file consists of four columns (tab-delimited), including user-id (starting at 1), item-id (starting at 1), rating, and timestamp (we won't be using this field).

```python
# download the dataset if it isn't in the same folder
file_dir = 'ml-100k'
file_path = os.path.join(file_dir, 'u.data')
if not os.path.isdir(file_dir):
    call(['curl', '-O', 'http://files.grouplens.org/datasets/movielens/ml-100k.zip'])
    call(['unzip', 'ml-100k.zip'])

names = ['user_id', 'item_id', 'rating', 'timestamp']
df = pd.read_csv(file_path, sep = '\t', names = names)
print(df.shape)
df.head()
```

    (100000, 4)

      user_id
      item_id
      rating
      timestamp

      0
      196
      242
      3
      881250949

      1
      186
      302
      3
      891717742

      2
      22
      377
      1
      878887116

      3
      244
      51
      2
      880606923

      4
      166
      346
      1
      886397596

As we can see, the only data that we have is how each user interacted or rated each item. Given this information, collaborative filtering will start by constructing a user-item matrix with each distinct user being the row, item being the column and the value for each cell will simply be the rating that the user gave to the item. Apart from building the matrix, we will also print out some other information to help us understand our data a bit better.

```python
# create the rating matrix r_{ui}, remember to
# subract the user and item id by 1 since
# the indices starts from 0
n_users = df['user_id'].unique().shape[0]
n_items = df['item_id'].unique().shape[0]
ratings = np.zeros((n_users, n_items))
for row in df.itertuples(index = False):
    ratings[row.user_id - 1, row.item_id - 1] = row.rating

# compute the non-zero elements in the rating matrix
matrix_size = np.prod(ratings.shape)
interaction = np.flatnonzero(ratings).shape[0]
sparsity = 100 * (interaction / matrix_size)

print('dimension: ', ratings.shape)
print('sparsity: {:.1f}%'.format(sparsity))
ratings
```

    dimension:  (943, 1682)
    sparsity: 6.3%

    array([[5., 3., 4., ..., 0., 0., 0.],
           [4., 0., 0., ..., 0., 0., 0.],
           [0., 0., 0., ..., 0., 0., 0.],
           ...,
           [5., 0., 0., ..., 0., 0., 0.],
           [0., 0., 0., ..., 0., 0., 0.],
           [0., 5., 0., ..., 0., 0., 0.]])

```python
# change default figure and font size
plt.rcParams['figure.figsize'] = 8, 6
plt.rcParams['font.size'] = 12

plt.hist(np.sum(ratings != 0, axis = 1), histtype = 'stepfilled', bins = 30,
         alpha = 0.85, label = '# of ratings', color = '#7A68A6', normed = True)
plt.axvline(x = 10, color = 'black', linestyle = '--')
plt.legend(loc = "upper right")
plt.show()
```

From the information above, we know there are 943 unique users, 1682 unique items. Within the rating matrix, only 6.3% of the cells have a value, although we filled in empty ratings as 0, we should not assume these values to truly be zero. More appropriately, they are just missing entries. This kind of sparsity is extremely common in recommendation system, where people seldom give ratings to things that they have purchased. One thing to note is that if the sparsity of the matrix is below 1% (rule of thumb), then the dataset might be too sparse to perform any sort of modeling.

Next the histogram tells us that every user has given at least more than 10 ratings, we will use this to perform the train/test split of the data for testing the algorithm that we'll build later.

One tricky thing about splitting the data into training and testing is that: In supervise machine learning we normally build the trainining and testing holdout set by randomly splitting the rows. In those cases, this idea works, because we have a model with features/target that we are trying to fit a function to. For recommender systems with collaborative filtering (no features), this just won't work anymore, because all of the items/users need to be available when the model is first built. So what we do instead is mask a random sample of the user/item ratings to validate and compare how well the recommender system did in predicting the ratings of those masked values. In our case, given we already know each user has given more than 10 ratings, what we'll do is for every user, we remove 10 of the item ratings and and assign them to the test set. The testing matrix is printed below, as hopefully, you can see that some of the values are indeed different from the original rating matrix.

```python
def create_train_test(ratings):
    """
    split into training and test sets,
    remove 10 ratings from each user
    and assign them to the test set
    """
    test = np.zeros(ratings.shape)
    train = ratings.copy()
    for user in range(ratings.shape[0]):
        test_index = np.random.choice(
            np.flatnonzero(ratings[user]), size = 10, replace = False)

        train[user, test_index] = 0.0
        test[user, test_index] = ratings[user, test_index]

    # assert that training and testing set are truly disjoint
    assert np.all(train * test == 0)
    return train, test

train, test = create_train_test(ratings)
del ratings
train
```

    array([[5., 3., 4., ..., 0., 0., 0.],
           [4., 0., 0., ..., 0., 0., 0.],
           [0., 0., 0., ..., 0., 0., 0.],
           ...,
           [0., 0., 0., ..., 0., 0., 0.],
           [0., 0., 0., ..., 0., 0., 0.],
           [0., 5., 0., ..., 0., 0., 0.]])

## Matrix Factorization

Now that the data preprocessing part has been taken care of, let's get to the more exciting part, the algorithm. The algorithm that we'll introduce today is Matrix Factorization.

Recall that we had a user-item matrix, $R$ where nonzero elements of the matrix are ratings that a user has given an item. What Matrix Factorization does is it decomposes a large matrix into products of matrices, namely, $R = U \times V$. See the picture below taken from a quick [google search](https://www.google.com/search?q=matrix+factorization&source=lnms&tbm=isch&sa=X&ved=0ahUKEwiD8Ku02_vQAhXGsVQKHdpSAW4Q_AUICCgB&biw=1433&bih=732#imgrc=aFtUeC0wtJCkgM%3A) for a better understanding:

Matrix factorization assumes that:

- Each user can be described by $d$ features. For example, feature 1 might be a referring to how much each user likes disney movies.
- Each item, movie in this case, can be described by an analogous set of $d$ features. To correspond to the above example, feature 1 for the movie might be a number that says how close the movie is to a disney movie.

After learning the two smaller matrices, all we have to do is to perform a matrix multiplication of the two matrices and the end result will be a our approximation for the rating the user would give that item (movie).

The cool thing about this is that, we do not know what these features are nor do we have to determine them beforehand, which is why these features are often refer to as latent features. Next, we also don't know how many latent features are optimal for the task at hand. Thus, we can use random search or grid search or other fancy techniques to perform hyperparameter tuning and determine the best number for $d$.

Given all those information, the next question is: how do we learn the user matrix, $U$, and item matrix, $V$? Well, like a lot of machine learning algorithm, by minimizing a loss function.

We start by denoting our $d$ feature user into math by letting a user $u$ take the form of a $1 \times d$-dimensional vector $\textbf{x}_{u}$. These for often times referred to as latent vectors or low-dimensional embeddings. Similarly, an item *i* can be represented by a $1 \times d$-dimensional vector $\textbf{y}_{i}$. And the rating that we predict user $u$ will give for item $i$ is just the dot product of the two vectors

$$
\begin{align}
\hat r_{ui} &= \textbf{x}_{u} \textbf{y}_{i}^{T} = \sum\limits_{d} x_{ud}y_{di}
\end{align}
$$

Where $\hat r_{ui}$ represents our prediction for the true rating $r_{ui}$. Next, we will choose a objective function to minimize the square of the difference between all ratings in our dataset ($S$) and our predictions. This produces a objective function of the form:

$$
\begin{align}
L &= \sum\limits_{u,i \in S}( r_{ui} - \textbf{x}_{u} \textbf{y}_{i}^{T} )^{2} + \lambda \big( \sum\limits_{u} \left\Vert \textbf{x}_{u} \right\Vert^{2} + \sum\limits_{i} \left\Vert \textbf{y}_{i} \right\Vert^{2} \big)
\end{align}
$$

Note that we've added on two $L_{2}$ regularization terms, with $\lambda$ controlling the strength at the end to prevent overfitting of the user and item vectors. $\lambda$, is another hyperparameter that we'll have to search for to determine the best value. The concept of regularization can be a topic of itself, and if you're confused by this, consider checking out [this documentation](http://nbviewer.jupyter.org/github/ethen8181/machine-learning/blob/master/regularization/regularization.ipynb) that covers it a bit more.

Now that we formalize our objective function, we'll introduce the **Alternating Least Squares with Weighted Regularization (ALS-WR)** method for optimizing it. The way it works is we start by treating one set of latent vectors as constant. For this example, we'll pick the item vectors, $\textbf{y}_{i}$. We then take the derivative of the loss function with respect to the other set of vectors, the user vectors, $\textbf{x}_{u}$ and solve for the non-constant vectors (the user vectors).

$$
\begin{align}
\frac{\partial L}{\partial \textbf{x}_{u}}
&= - 2 \sum\limits_{i}(r_{ui} - \textbf{x}_{u} \textbf{y}_{i}^{T} ) \textbf{y}_{i} + 2 \lambda \textbf{x}_{u} = 0 \\
&= -(\textbf{r}_{u} - \textbf{x}_{u} Y^{T} )Y + \lambda \textbf{x}_{u} = 0 \\
&= \textbf{x}_{u} (Y^{T}Y + \lambda I) = \textbf{r}_{u}Y \\
&= \textbf{x}_{u} = \textbf{r}_{u}Y (Y^{T}Y + \lambda I)^{-1}
\end{align}
$$

To clarify it a bit, let us assume that we have $m$ users and $n$ items, so our ratings matrix is $m \times n$.

- The row vector $\textbf{r}_{u}$ represents users *u*'s row from the ratings matrix with all the ratings for all the items (so it has dimension $1 \times n$)
-  We introduce the symbol $Y$, with dimensions $n \times d$, to represent all item row vectors vertically stacked on each other
- Lastly, $I$ is the identity matrix which has dimension $d \times d$ to ensure the matrix multiplication's dimensionality will be correct when we add the $\lambda$

Now comes the alternating part: With these newly updated user vectors in hand, in the next round, we hold them as constant, and take the derivative of the loss function with respect to the previously constant vectors (the item vectors). As the derivation for the item vectors is quite similar, we will simply list out the end formula:

$$
\begin{align}
\frac{\partial L}{\partial \textbf{y}_{i}}
&= \textbf{y}_{i} = \textbf{r}_{i}X (X^{T}X + \lambda I)^{-1}
\end{align}
$$

Then we alternate back and forth and carry out this two-step process until convergence. The reason we alternate is, optimizing user latent vectors, $U$, and item latent vectors $V$ simultaneously is hard to solve. If we fix $U$ or $V$ and tackle one problem at a time, we potentially turn it into a easier sub-problem. Just to summarize it, ALS works by:

1. Initialize the user latent vectors, $U$, and item latent vectors $V$ with randomly
2. Fix $U$ and solve for $V$
3. Fix $V$ and solve for $U$
4. Repeat step 2 and 3 until convergence

Now that we have our equations, let's program this thing up! We'll set the model to use 20 factors and a regularization value of 0.01 (chosen at random) and train it for 100 iterations and compute the mean square error on the train and test set to assess algorithm convergence.

```python
class ExplicitMF:
    """
    Train a matrix factorization model using Alternating Least Squares
    to predict empty entries in a matrix

    Parameters
    ----------
    n_iters : int
        number of iterations to train the algorithm

    n_factors : int
        number of latent factors to use in matrix
        factorization model, some machine-learning libraries
        denote this as rank

    reg : float
        regularization term for item/user latent factors,
        since lambda is a keyword in python we use reg instead
    """

    def __init__(self, n_iters, n_factors, reg):
        self.reg = reg
        self.n_iters = n_iters
        self.n_factors = n_factors

    def fit(self, train, test):
        """
        pass in training and testing at the same time to record
        model convergence, assuming both dataset is in the form
        of User x Item matrix with cells as ratings
        """
        self.n_user, self.n_item = train.shape
        self.user_factors = np.random.random((self.n_user, self.n_factors))
        self.item_factors = np.random.random((self.n_item, self.n_factors))

        # record the training and testing mse for every iteration
        # to show convergence later (usually, not worth it for production)
        self.test_mse_record  = []
        self.train_mse_record = []
        for _ in range(self.n_iters):
            self.user_factors = self._als_step(train, self.user_factors, self.item_factors)
            self.item_factors = self._als_step(train.T, self.item_factors, self.user_factors)
            predictions = self.predict()
            test_mse = self.compute_mse(test, predictions)
            train_mse = self.compute_mse(train, predictions)
            self.test_mse_record.append(test_mse)
            self.train_mse_record.append(train_mse)

        return self

    def _als_step(self, ratings, solve_vecs, fixed_vecs):
        """
        when updating the user matrix,
        the item matrix is the fixed vector and vice versa
        """
        A = fixed_vecs.T.dot(fixed_vecs) + np.eye(self.n_factors) * self.reg
        b = ratings.dot(fixed_vecs)
        A_inv = np.linalg.inv(A)
        solve_vecs = b.dot(A_inv)
        return solve_vecs

    def predict(self):
        """predict ratings for every user and item"""
        pred = self.user_factors.dot(self.item_factors.T)
        return pred

    @staticmethod
    def compute_mse(y_true, y_pred):
        """ignore zero terms prior to comparing the mse"""
        mask = np.nonzero(y_true)
        mse = mean_squared_error(y_true[mask], y_pred[mask])
        return mse
```

```python
def plot_learning_curve(model):
    """visualize the training/testing loss"""
    linewidth = 3
    plt.plot(model.test_mse_record, label = 'Test', linewidth = linewidth)
    plt.plot(model.train_mse_record, label = 'Train', linewidth = linewidth)
    plt.xlabel('iterations')
    plt.ylabel('MSE')
    plt.legend(loc = 'best')
```

```python
als = ExplicitMF(n_iters = 100, n_factors = 40, reg = 0.01)
als.fit(train, test)
plot_learning_curve(als)
```

Notice that ALS converges after a few sweeps, which is one of the main reason for its popularity. Fast, thus scalable to bigger dataset.

# Reference

- [Blog: Intro to Recommender Systems: Collaborative Filtering](http://blog.ethanrosenthal.com/2015/11/02/intro-to-collaborative-filtering/)
- [Blog: Explicit Matrix Factorization: ALS, SGD, and All That Jazz](http://blog.ethanrosenthal.com/2016/01/09/explicit-matrix-factorization-sgd-als/)
- [Quora: What is the Alternating Least Squares method in recommendation systems?](https://www.quora.com/What-is-the-Alternating-Least-Squares-method-in-recommendation-systems)


# Count models

Counting is child's play, as easy as 1, 2, 3, right?

As usual, the answer's not so simple: some things are easy to count and some things aren't, making some numbers easier to predict than others. What do we do?

1. **Count what's available** as carefully as possible.
2. Build an appropriate **probability model** to predict likely outcomes.

To explore this, we'll look at 4 real examples:
* cases of chronic medical conditions,
* car crashes in Tennessee,
* births in the United States, and
* coughing in Spain.

We'll see how these relate to 3 fundamental **count models**:
* binomial models,
* Poisson models, and
* negative binomial models.

If time permits, we'll also talk about using the **Kolmogorov-Smirnov test** to compare observed and simulated samples. When we meet again, we'll see how to build better predictive models, namely **regression models for counts** that incorporate predictor variables.

All of this (and more!) is in Chapters 5 and 7 of my Manning book, [Regression: A friendly guide](https://www.manning.com/books/regression-a-friendly-guide); these 2 chapters will be added to the MEAP soon. If you just can't wait, this notebook and the relevant CSVs are available _now!_ in my [regression repo on github](https://github.com/mbrudd/regression) — please clone and submit comments!

## Imports and settings

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from scipy.stats import binom
from scipy.stats import poisson
from scipy.stats import nbinom
from scipy.stats import kstest
import warnings
warnings.filterwarnings('ignore')
sns.set_theme()
plt.rcParams['figure.figsize'] = [15,6]
```

## Binomial models

### Chronic conditions by age

Here's a counting problem: given a group of people, how many have a chronic medical condition? Let's see what the 2009 [National Ambulatory Medical Care Survey (NAMCS)](https://www.cdc.gov/nchs/ahcd/index.htm) says:

```python
chronic = pd.read_csv("chronic.csv")
cohorts = pd.read_csv("chronic_cohorts.csv")
cohorts
```

      Age
      Total
      Sick
      Percentage

      0
      0
      796
      21
      0.026382

      1
      1
      505
      32
      0.063366

      2
      2
      320
      29
      0.090625

      3
      3
      258
      37
      0.143411

      4
      4
      299
      30
      0.100334

      ...
      ...
      ...
      ...
      ...

      96
      96
      13
      12
      0.923077

      97
      97
      11
      10
      0.909091

      98
      98
      6
      5
      0.833333

      99
      99
      2
      1
      0.500000

      100
      100
      6
      5
      0.833333

101 rows × 4 columns

In this dataset, _Sick_ means having at least one of the following conditions:

* arthritis,
* asthma,
* COPD,
* cancer,
* depression,
* diabetes,
* hyperlipidemia,
* hypertension,
* obesity,
* osteoporosis,
* cerebrovascular disease,
* chronic renal failure,
* congestive heart failure, or
* ischemic heart disease.

This is quite a list! The snippet shown corroborates what's expected: older people are more likely to have at least one of these conditions. For example, 30.4% of 25-year-olds have one and 81.43% of the 65-year-olds do:

```python
cohorts[ (cohorts["Age"]==25) | (cohorts["Age"]==65) ]
```

      Age
      Total
      Sick
      Percentage

      25
      25
      273
      83
      0.304029

      65
      65
      463
      377
      0.814255

These percentages are **empirical probabilities**. Based on them, what are the chances of having _exactly_ (a) 83 sick people out of 273 25-year-olds and (b) 377 sick people out of 463 65-year-olds? We need

### The binomial distribution

If you try something $n$ times with a chance $p$ of **success** on each **trial**, the number $Y$ of successes is a **random variable**: you don't know what the exact outcome will be until it happens, but you know the probability of each possible outcome. In fact, the probability of getting exactly $k$ successes is

$$P( \, Y = k \, ) ~ = ~ \binom{n}{k} \, p^{k} \, (1-p)^{n-k} \ , \quad \text{where} \quad \binom{n}{k} = \frac{ n! }{ k! (n-k)! } \quad .  $$

This is the **probability mass function (PMF)** for the **binomial distribution** $B(n,p)$. For a **binomial random variable** $Y \sim B(n,p)$,

$$\operatorname{E}\left( \, Y \, \right) ~ = ~ np \quad \text{and} \quad \operatorname{Var}( \, Y \, ) ~ = ~ np(1-p) \ .$$

> **Use the `binom` module from `scipy` (`import`ed above) to work with binomial distributions.**

25-year-olds in the NAMCS sample are like a **binomial experiment** with $n=273$ and $p=.304029$ ; the chance of observing exactly 83 successes (perverse lingo, I know!) is just over 5% :

```python
p_25 = cohorts["Percentage"][25]
binom.pmf(83,273,p_25)
```

    0.05243020970380286

65-year-olds are like a binomial experiment with 𝑛=463 and 𝑝=.814255 ; the chance of observing exactly 377 successes is just under 5% :

```python
p_65 = cohorts["Percentage"][65]
binom.pmf(377,463,p_65)
```

    0.047625769242453334

Don't be alarmed at these low probabilities — these are actually the most likely outcomes:

```python
fig, (ax1, ax2) = plt.subplots(1, 2)
fig.suptitle('Binomial models of chronic condition incidence')
x = np.arange(binom.ppf(0.01, 273, p_25),
              binom.ppf(0.99, 273, p_25))
ax1.plot(x, binom.pmf(x, 273, p_25), 'bo', ms=8)
ax1.vlines(x, 0, binom.pmf(x, 273, p_25), colors='b', lw=3, alpha=0.5)
ax1.set_xlabel('25-year-olds')
x = np.arange(binom.ppf(0.01, 463, p_65),
              binom.ppf(0.99, 463, p_65))
ax2.plot(x, binom.pmf(x, 463, p_65), 'bo', ms=8)
ax2.vlines(x, 0, binom.pmf(x, 463, p_65), colors='b', lw=3, alpha=0.5)
ax2.set_xlabel('65-year-olds');
```

It's no accident that these look like normal distributions — if $n$ is large and/or $p$ is close to $0.5$, $B(n,p)$ is approximately normal with mean $np$ and variance $np(1-p)$ :

$$ B(n,p) ~ \approx ~ N( np, np(1-p) ) \quad \text{for} \quad n \gg 1 \quad \text{or} \quad p \approx 0.5 \ .$$

Turn this around for some quick and dirty calculations. For example, $B(463,.814) \approx N( 377, 70)$, so there will usually be between $377 - 2\sqrt{70} \approx 360 $ and $377 + 2\sqrt{70} \approx 394$ successes, as you can easily check above!

Things are different if $n$ is small and/or $p$ is far from $0.5$. No worries, though — `binom.pmf` works fine either way.

### Binomial logistic regression

Instead of working separately with each age group — i.e., each **covariate class** — we should really construct a **binomial logistic regression model** to consolidate the age group percentages efficiently.

```python
sns.regplot(data=chronic,x="Age",
            y="Condition",
            logistic=True,
            scatter=False,
            ci=None,
            line_kws={"lw":"4"})
plt.xlim(-2,102)
plt.ylabel("Probability")
plt.plot( cohorts["Age"], cohorts["Percentage"],'.k');
```

This particular logistic model, plotted in blue, is **simple** — it involves only one predictor :

$$\log{ \left( \text{Odds of a condition} \right) } ~ = ~ -2.04 + .052*\text{Age} \ . $$

This model is really a family of binomial distributions, one for each covariate class; each probability is directly related to `Age`.

## Poisson models

### Monthly car crashes

Here's a different counting problem: how many traffic accidents are there each month where you live? For my home state of Tennessee, the [Department of Transportation provides the relevant data](https://www.tn.gov/safety/stats/crashdata.html):

```python
crashes = pd.read_csv("TDOT.csv")
crashes
```

      County
      Year
      January
      February
      March
      April
      May
      June
      July
      August
      September
      October
      November
      December

      0
      Anderson
      2010
      138
      113
      169
      169
      155
      152
      165
      176
      178
      169
      179
      152

      1
      Bedford
      2010
      71
      72
      77
      70
      87
      99
      90
      91
      86
      118
      105
      95

      2
      Benton
      2010
      21
      23
      27
      32
      34
      29
      16
      43
      31
      50
      29
      36

      3
      Bledsoe
      2010
      5
      1
      3
      3
      5
      1
      7
      4
      3
      6
      4
      3

      4
      Blount
      2010
      126
      159
      171
      145
      153
      139
      201
      264
      338
      267
      316
      269

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      1040
      Wayne
      2020
      23
      15
      19
      27
      18
      22
      20
      24
      23
      18
      22
      19

      1041
      Weakley
      2020
      39
      35
      40
      19
      28
      43
      37
      47
      54
      44
      33
      21

      1042
      White
      2020
      30
      31
      33
      21
      50
      32
      36
      46
      34
      29
      42
      43

      1043
      Williamson
      2020
      394
      424
      270
      170
      226
      304
      312
      310
      293
      405
      368
      377

      1044
      Wilson
      2020
      282
      268
      239
      143
      267
      258
      285
      295
      273
      306
      335
      299

1045 rows × 14 columns

```python
crashes["County"].nunique()
```

    95

Of these 95 counties, let's check out Meigs County (home of the ghost town [Cute, Tennessee](https://en.wikipedia.org/wiki/Cute,_Tennessee)) :

```python
meigs = crashes[crashes["County"]=="Meigs"]
meigs = meigs.drop("County",1)
meigs = meigs.melt( id_vars="Year", var_name="Month", value_name="Crashes" )
meigs
```

      Year
      Month
      Crashes

      0
      2010
      January
      11

      1
      2011
      January
      13

      2
      2012
      January
      12

      3
      2013
      January
      10

      4
      2014
      January
      16

      ...
      ...
      ...
      ...

      127
      2016
      December
      16

      128
      2017
      December
      18

      129
      2018
      December
      9

      130
      2019
      December
      11

      131
      2020
      December
      17

132 rows × 3 columns

```python
sns.histplot( data=meigs, x="Crashes", discrete=True)
plt.title("Monthly crashes in Meigs County, TN");
```

### The Poisson distribution

What's the relevant probability distribution? Let's see...

* There are a bunch of encounters between cars every month — lots of opportunities for accidents to occur.
* Most encounters don't result in an accident (thank goodness!), but accidents occur at a roughly constant rate per month.

This is a binomial experiment with a large number $n$ of trials, a small chance $p$ of "success" (an accident), and a roughly constant expected number $\lambda = np$ of successes per month, so that $\displaystyle{p = \frac{\lambda}{n}} \ll 1$. Letting $Y$ denote the number of accidents per month,

$$ \begin{align}
P( Y = k ) & ~ = ~ \frac{ n! }{ k! (n-k)! } \left( \frac{\lambda}{n} \right)^{k} \left( 1 - \frac{\lambda}{n} \right)^{(n-k)} \\
& ~ = ~ \frac{ n! }{ k! (n-k)! } \left( \frac{\lambda}{n} \right)^{k} \left( 1 - \frac{\lambda}{n} \right)^{n} \left( 1 - \frac{\lambda}{n} \right)^{-k} \\
& ~ \approx ~ \frac{ n! }{ k! (n-k)! } \left( \frac{\lambda}{n} \right)^{k} e^{-\lambda} \\
& ~ = ~ \frac{n (n-1) (n-2) \cdots (n-k+1) }{ n \cdot n \cdot n \cdots n } \ e^{-\lambda} \, \frac{\lambda^k}{ k! } \\
& ~ \approx ~ e^{-\lambda} \, \frac{\lambda^k}{ k! } \ .
\end{align} $$

Lo and behold, that's it! This is the PMF for the **Poisson distribution** with **rate** $\lambda$, denoted $\operatorname{Pois}(\lambda)$. The mean and variance of a **Poisson random variable** $Y \sim \operatorname{Pois}(\lambda)$ are _equal_:

$$\operatorname{E}\left( \, Y \, \right) ~ = ~ \operatorname{Var}( \, Y \, ) ~ = ~ \lambda \ .$$

> **Use the `poisson` module from `scipy` (`import`ed above) to work with Poisson distributions.**

### Simulating monthly crashes

Now you'll see why I picked Meigs County:

```python
meigs.agg( Mean = ("Crashes","mean"), Variance = ("Crashes","var") )
```

      Crashes

      Mean
      13.712121

      Variance
      15.198936

How well does the relevant Poisson distribution model crashes in Meigs County?

```python
meigs["Simulation"] = poisson.rvs(meigs["Crashes"].mean(),
                                  size=meigs.shape[0] )
fig, axs = plt.subplots(2, sharex=True)
sns.histplot( data=meigs, x="Crashes", discrete=True, ax=axs[0])
axs[0].set_title("Observed crashes per month in Meigs County, TN")
sns.histplot(data=meigs, x="Simulation", discrete=True, ax=axs[1])
axs[1].set_title("Simulated crashes per month in Meigs County, TN");
```

### Monthly births

What happens if we count _births_ every month instead of car crashes? Let's look at recent data from the [National Vital Statistics System](https://www.cdc.gov/nchs/nvss/births.htm), downloaded directly from [CDC Wonder](https://wonder.cdc.gov/natality-expanded-current.html):

```python
births = pd.read_csv("births.csv")
births
```

      county
      year
      month
      births

      0
      Baldwin County, AL
      2016
      January
      63

      1
      Baldwin County, AL
      2016
      February
      67

      2
      Baldwin County, AL
      2016
      March
      69

      3
      Baldwin County, AL
      2016
      April
      63

      4
      Baldwin County, AL
      2016
      May
      51

      ...
      ...
      ...
      ...
      ...

      27727
      Winnebago County, WI
      2019
      August
      66

      27728
      Winnebago County, WI
      2019
      September
      45

      27729
      Winnebago County, WI
      2019
      October
      41

      27730
      Winnebago County, WI
      2019
      November
      51

      27731
      Winnebago County, WI
      2019
      December
      54

27732 rows × 4 columns

```python
births_by_county = births.groupby("county",
                                  as_index=False).agg(Mean = ("births", "mean"),
                                                      Variance = ("births", "var"),
                                                      Max = ("births","max") )
births_by_county["Ratio"] = births_by_county["Variance"] / births_by_county["Mean"]
births_by_county = births_by_county[ births_by_county["Max"] < 100 ]
births_by_county = births_by_county.sort_values(by="Ratio", ascending=False)
births_by_county[ abs( births_by_county["Ratio"] - 1 ) < .05 ]
```

      county
      Mean
      Variance
      Max
      Ratio

      572
      Yellowstone County, MT
      68.666667
      71.929078
      87
      1.047511

      293
      Litchfield County, CT
      45.125000
      47.260638
      60
      1.047327

      20
      Ascension Parish, LA
      51.812500
      54.155585
      69
      1.045222

      81
      Catawba County, NC
      68.666667
      71.418440
      87
      1.040074

      48
      Boone County, KY
      51.687500
      53.070479
      72
      1.026757

      426
      Richland County, OH
      36.333333
      36.950355
      49
      1.016982

      422
      Randolph County, NC
      49.333333
      49.631206
      65
      1.006038

      536
      Washington County, MD
      57.104167
      57.414450
      73
      1.005434

      46
      Blount County, TN
      32.250000
      32.319149
      43
      1.002144

      486
      St. Lawrence County, NY
      27.416667
      27.226950
      45
      0.993080

Oddly enough, the mean and variance for births per month are closest for a county in Tennessee! How well do births in Blount County agree with $\operatorname{Pois}(32.25,32.32)$?

```python
blount = births[ births[ "county"] == "Blount County, TN" ]
fig, axs = plt.subplots(2, sharex=True)
sns.histplot( data=blount, x="births", discrete=True, ax=axs[0])
axs[0].set_title("Observed births per month in Blount County, TN")
pois_rv = pd.DataFrame({"Simulation" : poisson.rvs(np.mean(blount["births"]),
                                                   size=blount.shape[0])})
sns.histplot(data=pois_rv, x="Simulation", discrete=True, ax=axs[1])
axs[1].set_title("Simulated births per month in Blount County, TN");
```

### Simple Poisson regression: a preview

A simple Poisson regression model fits a family of Poisson RVs to observations. Each covariate class — one for each value $x$ of the predictor — has a mean rate $\lambda$ given (approximately!) by

$$ \log{ \left( \lambda \right) } ~ = ~ a + bx \ . $$

We'll discuss this in detail in the next Twitch session!

## Negative binomial models

### Overdispersed births

The Poisson distribution can clearly be a reasonable model for car crashes or births per month in a given county — ***if*** the mean and the variance are ***equal***! This is definitely not always the case:

```python
births_by_county
```

      county
      Mean
      Variance
      Max
      Ratio

      253
      Kanawha County, WV
      28.305556
      155.989683
      52
      5.510921

      235
      Jackson County, MI
      58.645833
      157.850621
      91
      2.691591

      423
      Rankin County, MS
      54.145833
      131.063387
      80
      2.420563

      342
      Monroe County, IN
      48.041667
      115.317376
      72
      2.400362

      441
      Saline County, AR
      53.541667
      127.955674
      79
      2.389834

      ...
      ...
      ...
      ...
      ...
      ...

      469
      Sheboygan County, WI
      39.833333
      36.141844
      55
      0.907327

      556
      Whitfield County, GA
      47.708333
      42.593972
      65
      0.892799

      226
      Houston County, GA
      70.270833
      61.903812
      89
      0.880932

      312
      Marathon County, WI
      50.479167
      42.084663
      66
      0.833704

      549
      Wayne County, OH
      46.145833
      36.893174
      58
      0.799491

226 rows × 5 columns

```python
kanawha = births[ births["county"] == "Kanawha County, WV"]
kanawha.shape
```

    (36, 4)

```python
fig, axs = plt.subplots(2, sharex=True)
sns.histplot(data=kanawha,
             x="births",
             discrete=True, ax=axs[0])
axs[0].set_title("Observed births per month in Kanawha County, WV")
pois = pd.DataFrame({"Sample" : poisson.rvs(np.mean(kanawha["births"]),
                                            size=kanawha.shape[0])})
sns.histplot(data=pois, x="Sample",
             discrete=True, ax=axs[1])
axs[1].set_title("Poisson-simulated births per month in Kanawha County, WV");
```

Monthly births in Kanawha County, WV are definitely _not_ Poisson distributed! This is **overdispersion**: the variance is much larger than the mean. Overdispersion and **underdispersion** (variance smaller than the mean) are common in count modeling; in either case, Poisson models are _not_ appropriate — they only apply to counts that are **equidispersed**.

### Overdispersed car crashes

***All*** of the counties in the `crashes` dataset exhibit overdispersion, with Meigs County the closest to being equidispersed. At the other extreme, monthly crashes in nearby Hamilton County have a variance that is more than 20 times their mean:

```python
hamilton = crashes[ crashes["County"] == "Hamilton" ]
hamilton = hamilton.drop("County",1)
hamilton = hamilton.melt(id_vars="Year",
                         var_name="Month",
                         value_name="Crashes" )
hamilton.agg(Mean = ("Crashes","mean"),
             Variance = ("Crashes","var") )
```

      Crashes

      Mean
      1047.060606

      Variance
      22476.347444

```python
fig, axs = plt.subplots(2, sharex=True)
sns.histplot( data=hamilton, x="Crashes", discrete=True, ax=axs[0])
axs[0].set_title("Observed crashes per month in Hamilton County, TN")
pois = pd.DataFrame({"Sample" : poisson.rvs(np.mean(hamilton["Crashes"]),
                                            size=hamilton.shape[0])})
sns.histplot(data=pois, x="Sample", discrete=True, ax=axs[1])
axs[1].set_title("Poisson-simulated crashes per month in Hamilton County, TN");
```

Terrible!! Never use a Poisson model when there's significant overdispersion!

### Coughs per hour

Here are 2 common features of the `crashes` and `births` datasets:

* counts are recorded _per month_, and
* _there are no zero counts_ for any month.

Of course, there's nothing special about working with months, and there can be plenty of zeroes in some datasets. When monitoring a person's coughs, for example, it's natural to record coughs _per hour_, yielding lots of zeroes. Here are coughs recorded per hour by the [Hyfe cough monitoring app](https://www.hyfeapp.com/) for a person in Spain:

```python
primera = pd.read_csv("primera.csv")
primera["datetime"] = pd.to_datetime( primera["datetime"] )
primera
```

      datetime
      coughs

      0
      2020-11-06 01:00:00
      0

      1
      2020-11-06 02:00:00
      0

      2
      2020-11-06 03:00:00
      0

      3
      2020-11-06 04:00:00
      0

      4
      2020-11-06 05:00:00
      0

      ...
      ...
      ...

      7076
      2021-08-27 22:00:00
      0

      7077
      2021-08-27 23:00:00
      0

      7078
      2021-08-28 00:00:00
      0

      7079
      2021-08-28 01:00:00
      0

      7080
      2021-08-28 02:00:00
      0

7081 rows × 2 columns

Despite what's shown for these 10 hours, this person coughed quite a bit from time to time:

```python
sns.lineplot(data=primera, x="datetime", y="coughs");
```

Even so, there were way more hours with 0 or just a few coughs:

```python
sns.histplot(data=primera, x="coughs", discrete=True);
```

There's clearly some overdispersion here, as we can easily check:

```python
primera.agg(Mean = ("coughs","mean"),
            Variance = ("coughs","var") )
```

      coughs

      Mean
      0.99195

      Variance
      12.03341

This is pretty typical for coughing — here's another Hyfe user from Spain:

```python
segunda = pd.read_csv("segunda.csv")
segunda["datetime"] = pd.to_datetime( segunda["datetime"] )
segunda
```

      datetime
      coughs

      0
      2020-11-14 01:00:00
      0

      1
      2020-11-14 02:00:00
      0

      2
      2020-11-14 03:00:00
      0

      3
      2020-11-14 04:00:00
      0

      4
      2020-11-14 05:00:00
      0

      ...
      ...
      ...

      2012
      2021-02-05 21:00:00
      0

      2013
      2021-02-05 22:00:00
      0

      2014
      2021-02-05 23:00:00
      0

      2015
      2021-02-06 00:00:00
      0

      2016
      2021-02-06 01:00:00
      0

2017 rows × 2 columns

```python
sns.lineplot(data=segunda, x="datetime", y="coughs");
```

```python
sns.histplot(data=segunda, x="coughs", discrete=True);
```

```python
segunda.agg(Mean = ("coughs","mean"),
            Variance = ("coughs","var"))
```

      coughs

      Mean
      0.411502

      Variance
      2.487328

### The negative binomial distribution

The Poisson distribution doesn't apply to overdispersed counts, so what does? The go-to option in many cases is the **negative binomial distribution**, whose PMF has various forms. The form implemented by `scipy.nbinom` is

$$ P( Y = k ) = \binom{ k+n-1 }{ n-1 } p^{n} \left( 1 - p \right)^{k} \ , \quad k = 0, 1, 2, \ldots \ , $$

reminiscent of the binomial distribution; this is the probability of needing $k+n$ trials to observe $n$ successes when the chance of success per trial is $p$. The mean and variance of $Y \sim \operatorname{NB}(n,p))$ are

$$\operatorname{E}\left( \, Y \, \right) ~ = ~ \frac{n}{p} \quad \text{and} \quad \operatorname{Var}( \, Y \, ) ~ = ~ \frac{n(1-p)}{p^2} \ .$$

For count models, different parameterizations are common and more practical. Let $\mu$ denote the average of $Y \sim \operatorname{NB}(n,p))$, let $\sigma^2$ denote its variance, and define the **dispersion parameter** $\alpha$ by

$$ \sigma^2 ~ = ~ \mu + \alpha \mu^2 \quad \iff \quad \alpha ~ = ~ \frac{\sigma^2 - \mu}{\mu^2} \ . $$

Given the values of 2 of these parameters, relate them to $n$ and $p$:

$$ p ~ = ~ \frac{\mu}{\mu + \sigma^2} ~ = ~ \frac{1}{2 + \alpha \mu} \quad \text{and} \quad n ~ = ~ \frac{\mu^2}{\mu + \sigma^2} ~ = ~ \frac{\mu}{2 + \alpha \mu} \ .$$

You can then translate between the versions — $\operatorname{NB}(n,p)$, $\operatorname{NB}(\mu,\sigma^2)$ and $\operatorname{NB}(\mu,\alpha)$ — as needed. Beware that other people use $\displaystyle{\frac{1}{\alpha}}$ as the dispersion parameter, and other relationships between it and the variance are possible. And most importantly, notice that _two parameters are needed!!_

> **Use the `nbinom` module from `scipy` (`import`ed above) to work with negative binomial distributions.**

### Simulating monthly births

```python
mu = kanawha["births"].mean()
var = kanawha["births"].var()
n_kanawha = mu**2 / (mu + var )
p_kanawha = mu / ( mu + var )
fig, axs = plt.subplots(2, sharex=True)
sns.histplot( data=kanawha, x="births", discrete=True, ax=axs[0])
axs[0].set_title("Observed births per month in Kanawha County, WV")
NB = pd.DataFrame({"Simulation" : nbinom.rvs(n_kanawha, p_kanawha,
                                             size=kanawha.shape[0])})
sns.histplot(data=NB, x="Simulation", discrete=True, ax=axs[1])
axs[1].set_title("NB-simulated births per month in Kanawha County, WV");
```

### Simulating monthly crashes

```python
mu = hamilton["Crashes"].mean()
var = hamilton["Crashes"].var()
n = mu**2 / (mu + var )
p = mu / ( mu + var )
fig, axs = plt.subplots(2, sharex=True)
sns.histplot( data=hamilton, x="Crashes", discrete=True, ax=axs[0])
axs[0].set_title("Observed crashes per month in Hamilton County, TN")
NB = pd.DataFrame({"Simulation" : nbinom.rvs(n, p,
                                             size=hamilton.shape[0])})
sns.histplot(data=NB, x="Simulation", discrete=True, ax=axs[1])
axs[1].set_title("NB-simulated crashes per month in Hamilton County, TN");
```

### Simulating coughs per hour

```python
mu = primera["coughs"].mean()
var = primera["coughs"].var()
n = mu**2 / (mu + var )
p = mu / ( mu + var )
primera["simulation"] = nbinom.rvs(n, p, size=primera.shape[0])
fig, axs = plt.subplots(2, sharex=True)
sns.lineplot( data=primera, x="datetime", y="coughs", ax=axs[0])
axs[0].set_title("Recorded coughs per hour")
sns.lineplot(data=primera, x="datetime", y="simulation", ax=axs[1])
axs[1].set_title("NB-simulated coughs per hour");
```

```python
fig, axs = plt.subplots(2, sharex=True)
sns.histplot( data=primera, x="coughs", discrete=True, ax=axs[0])
axs[0].set_title("Recorded coughs per hour")
sns.histplot(data=primera, x="simulation", discrete=True, ax=axs[1])
axs[1].set_title("NB-simulated coughs per hour");
```

```python
mu = segunda["coughs"].mean()
var = segunda["coughs"].var()
n = mu**2 / (mu + var )
p = mu / ( mu + var )
segunda["simulation"] = nbinom.rvs(n, p, size=segunda.shape[0])
fig, axs = plt.subplots(2, sharex=True)
sns.lineplot( data=segunda, x="datetime", y="coughs", ax=axs[0])
axs[0].set_title("Recorded coughs per hour")
sns.lineplot(data=segunda, x="datetime", y="simulation", ax=axs[1])
axs[1].set_title("NB-simulated coughs per hour");
```

```python
fig, axs = plt.subplots(2, sharex=True)
sns.histplot( data=segunda, x="coughs", discrete=True, ax=axs[0])
axs[0].set_title("Recorded coughs per hour")
sns.histplot(data=segunda, x="simulation", discrete=True, ax=axs[1])
axs[1].set_title("NB-simulated coughs per hour");
```

### Simple negative binomial regression: a preview

A simple negative binomial regression model fits a family of NB RVs to observations. Each covariate class — one for each value $x$ of the predictor — has a mean rate $\mu$ given (approximately!) by

$$ \log{ \left( \mu \right) } ~ = ~ a + bx $$

_AND_ a dispersion parameter $\alpha$ determined by the fitting process. More on this next time!

## The Kolmogorov-Smirnov test

### Cumulative distribution functions

The PMF of a discrete RV $Y$ — binomial, Poisson, or negative binomial — determines $P( Y = k )$ for any nonnegative integer $k$; an alternative characterization of $Y$ is its **cumulative distribution function (CDF)**, which gives $P( Y \leq k )$ for any such $k$. Using recorded or simulated counts instead of a theoretical distribution yields **empirical cumulative distribution functions (ECDF)**, and comparing them is the basis of the **Kolmogorov-Smirnov test** — as with any other hypothesis test, you should always complement this one with a visual and a good dose of common sense!

```python
sns.ecdfplot(data=meigs[["Crashes","Simulation"]]);
```

```python
kstest(meigs["Crashes"],meigs["Simulation"])
```

    KstestResult(statistic=0.09848484848484848, pvalue=0.5457001750199713)

The null hypothesis here is that _both of these samples are drawn from the same distribution_ — the $p$-value is large, so we have no reason to reject this hypothesis for the crashes in Meigs County. Based on this data, monthly crashes there do seem to be Poisson distributed!

```python
ks_kanawha = pd.DataFrame({"births":kanawha["births"],
                           "simulation":nbinom.rvs( n_kanawha, p_kanawha, size=kanawha.shape[0])})
sns.ecdfplot(data=ks_kanawha);
```

```python
kstest(ks_kanawha["births"], ks_kanawha["simulation"])
```

    KstestResult(statistic=0.25, pvalue=0.2122866675915156)

```python
sns.ecdfplot(data=segunda[["coughs","simulation"]]);
```

```python
kstest(segunda["coughs"],segunda["simulation"])
```

    KstestResult(statistic=0.04858701041150223, pvalue=0.017093207893396013)

```python

```


# Music Machine Learning - Support Vector Machines (SVM)

### Author: Philippe Esling (esling@ircam.fr)

In this course we will cover
1. A recap on the [motivation](#motivation) behind Support Vector Machines (SVM)
2. A simple use case for [understanding and fitting SVMs](#fitting) with the scikit library
3. Developing our own [implementation](#implem) of linear SVMs
4. An introuction to [kernels](#improve) in SVMs and how to implement them

## Support Vector Machines (SVM)

In this tutorial, we will cover a more advanced classification algorithm through the use of *Support Vector Machines* (SVMs). The goal is to gain an intuition of how SVMs works and how to use *Gaussian kernel* with SVMs to find out the decision boundary. The implementation proposed here follows a simplified version of the *Sequential Minimal Optimization* (SMO) algorithm for training support vector machines, which is detailed in the following [paper](http://cs229.stanford.edu/materials/smo.pdf). The complete SMO algorithm can then be quite easily implemented following the [full paper](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/tr-98-14.pdf).

Once again, to simplify your work, we provide the following set of functions that you should find in the `data` folder, and in the `helper_plot` file

  |**File**|*Explanation*|
  |-------:|:------------|
  |`svm_linear.mat`|Example data for a quasi-linear problem|
  |`svm_gaussian.mat`|Example data for a non-linear but well-defined problem|
  |`plot_data`|Allows to plot the set of data points|
  |`visualize_boundary`|Plots a non-linear decision boundary from a SVM model|
  |`plot_svc_decision_function`|Plots a linear decision boundary|

```python
import numpy as np
from scipy import io as sio
from matplotlib import pyplot as plt
from helper_plot import plot_data, visualize_boundary, visualize_boundary_linear, hdr_plot_style, plot_svc_decision_function
import numpy.random as rand
```

## Motivation

In the previous course on neural networks, we learned how to train a simple neuron. We saw that this simple model could be interpreted as defining a line, separating the space into two different regions. This allows to consider problems of *discriminative classification*: rather than modeling each class, we simply find a line that divides the classes from each other. As an example of these _linearly separable_ problems, consider the simple case of a classification task, in which the two classes of points are well separated:

```python
from sklearn.datasets import make_blobs
hdr_plot_style()
X, y = make_blobs(n_samples=50, centers=2, random_state=0, cluster_std=0.65)
plt.figure(figsize=(10, 8)); plt.grid(True)
plt.scatter(X[:, 0], X[:, 1], c=y, s=80, edgecolor='w', cmap='winter');
plt.title("Linear classification example");
```

As we have seen previously, a single neuron classifier would attempt to draw a straight line separating the two sets of data, and thereby create a model for classification, by optimizing the parameters $\theta=\{\mathbf{w}, b\}$ of

$$
\begin{equation}
y=\sigma\left(\sum_{i = 1}^{n}w_{i}.x_{i} + b\right)
\end{equation}
$$

For two dimensional data like that shown here, this is a task we could do by hand. But the major problem that we can directly see, is that there is an infinite number of lines that can perfectly discriminate between the two classes, as exemplified in the next example.

```python
xfit = np.linspace(-1, 3.5)
plt.figure(figsize=(10, 8)); plt.grid(True)
plt.scatter(X[:, 0], X[:, 1], c=y, s=80, edgecolor='w', cmap='winter')
for m, b in [(1, 0.65), (0.5, 1.6), (-0.2, 2.9)]:
    plt.plot(xfit, m * xfit + b, '-w')
plt.xlim(-1, 3.5);
```

Note that these solutions are all valid, and will mostly depend on the (random) initialization of our parameters. Although these all perfectly discriminate between our samples, we can clearly see that they are more or less "optimal" to the problem at hand. So our question today is how could we formalize this problem, and if we could find a formulation to adress and solve this issue.

### Notion of _margin_

The intuition behind comparing different solutions is that rather than estimating only the line between the classes, we can also look at the *margin* of some width defined around those line. In our simple case, it would look like this

```python
xfit = np.linspace(-1, 3.5)
plt.figure(figsize=(10, 8)); plt.grid(True)
plt.scatter(X[:, 0], X[:, 1], c=y, s=80, edgecolor='w', cmap='winter')
for m, b, d in [(1, 0.65, 0.25), (0.5, 1.6, 0.55), (-0.2, 2.9, 0.15)]:
    yfit = m * xfit + b
    plt.plot(xfit, yfit, '-w')
    plt.fill_between(xfit, yfit - d, yfit + d, edgecolor='w', color='#AAAAAA', alpha=0.6)
plt.xlim(-1, 3.5);
```

In *Support Vector Machines (SVMs)*, the goal is to find that optimal model, in which our separating line maximizes this margin. Hence, SVMs are part of a class of models called *maximum margin* estimator.

## Fitting a SVM

More formally the goal of SVMs is that given a set of input vectors $ \mathbf{x} $ separated in two classes (noted $ \mathbf{x}^{+} $ and $ \mathbf{x}^{-} $), we would like to find the *optimal boundary* (separating line) between the two classes. To do so, we want to find the separation that gives the *maximal margin* between the two sets. Hence, we can first define the following problem

$$
\begin{equation}
\begin{cases}
\mathbf{w}\cdot\mathbf{x}^{+}+\mathbf{b}\geq1\\
\mathbf{w}\cdot\mathbf{x}^{-}+\mathbf{b}\leq-1
\end{cases}
\end{equation}
$$

By setting the variable $ y_{i}={+}/{-}1 $, we can rewrite this problem as

$$
\begin{equation}
y_{i}\left(x_{i}\cdot\mathbf{w}+\mathbf{b}\right)-1\geq0
\end{equation}
$$

The idea is that for some of the input vectors (called *support vectors*), we will have an equality in equation (1), where $ \mathbf{w}\cdot\mathbf{x}^{+}+\mathbf{b}=1 $ and $ \mathbf{w}\cdot\mathbf{x}^{-}+\mathbf{b}=-1 $. By substracting those two equation, we see that we are trying to find a solution to

$$
\begin{equation}
\frac{\mathbf{w}}{\left\Vert \mathbf{w}\right\Vert }\cdot\left(x^{+}-x^{-}\right)=\frac{2}{\left\Vert \mathbf{w}\right\Vert }=Width
\end{equation}
$$

Therefore, solving the set of equations amounts to find the maximal margin between the support vectors (as they *support* classification).

### Fitting the SVM with scikit-learn

First, we can see the result of fitting this approach to our original data. To do so, we will use the SVM model found in `scikit-learn` to train an SVM model on this data. To do so, we can use the following code

```python
from sklearn.svm import SVC # "Support vector classifier"
model = SVC(kernel='linear')
model.fit(X, y)
```

To visualize the results, we will use our quick convenience function that plots the SVM decision boundaries

```python
plt.figure(figsize=(10, 8)); plt.grid(True)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor='w', cmap='winter')
plot_svc_decision_function(model);
```

This separating line is the one that maximizes the margin between our two sets of points.
Notice that some training points are right on the margin. These points are the pivotal elements of this fit, and are known as the *support vectors*, and give the algorithm its name. In `scikit-learn`, we can directly obtain the values of these points, stored in the ``support_vectors_`` attribute of the classifier:

```python
model.support_vectors_
```

The major advantage of SVM is that for the fit, only the position of the support vectors matter; any points further from the margin which are on the correct side do not modify the fit. Technically, this is because these points do not contribute to the loss function used to fit the model, so their position and number do not matter so long as they do not cross the margin. We can see this, for example, if we plot the model learned from the first 50 or 150 points of this dataset

```python
# Plotting SVM trained on subsampled dataset
def plot_svm_subsample(N=10, ax=None):
    X, y = make_blobs(n_samples=200, centers=2, random_state=0, cluster_std=0.60)
    X = X[:N]
    y = y[:N]
    model = SVC(kernel='linear', C=1E10)
    model.fit(X, y)
    ax = ax or plt.gca()
    ax.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor = 'w', cmap='winter')
    ax.set_xlim(-1, 4)
    ax.set_ylim(-1, 6)
    plot_svc_decision_function(model, ax)

fig, ax = plt.subplots(1, 2, figsize=(16, 6))
fig.subplots_adjust(left=0.0625, right=0.95, wspace=0.1)
for axi, N in zip(ax, [60, 150]):
    plot_svm_subsample(N, axi)
    axi.set_title('N = {0}'.format(N))
```

In the left panel, we see the model and the support vectors for 60 training points.
In the right panel, we have doubled the number of training points, but the model has not changed: the three support vectors from the left panel are still the support vectors from the right panel.
This insensitivity to the exact behavior of distant points is one of the strengths of the SVM model.

## Implementing our own linear SVM

Here, we are going to try to implement our own linear SVM. We recall that, given a set of input vectors $ \mathbf{x} $ separated in two classes ($ \mathbf{x}^{+} $ and $ \mathbf{x}^{-} $, we would like to find the *optimal boundary* (separating line) between the two classes. This means that we are trying to find a solution to

$$
\begin{equation}
\frac{\mathbf{w}}{\left\Vert \mathbf{w}\right\Vert }\cdot\left(x^{+}-x^{-}\right)=\frac{2}{\left\Vert \mathbf{w}\right\Vert }=Width
\end{equation}
$$

Therefore, solving the set of equations amounts to find the maximal margin between the support vectors (as they *support* classification). As seen in the slides, we know that this problem can also be expressed as

$$
\begin{equation}
f\left(x\right)=\sum_{i=1}^{m}\alpha_{i}y_{i}\left\langle \phi\left(x_{i}\right),\phi\left(x\right)\right\rangle +b
\end{equation}
$$

Where the function $\phi(.)$ will be used later to introduce _kernels_, but for now, we will consider that $\phi(x)=x$, which reduces our problem to

$$
\begin{equation}
f\left(x\right)=\sum_{i=1}^{m}\alpha_{i}y_{i}\left\langle x_{i},x \right\rangle +b
\end{equation}
$$

we can introduce a kernel $K\left(x_{i},x\right)=\phi\left(x_{i}\right)\cdot\phi\left(x\right)$ in the equation in order to solve non-linear optimization problems. We can use this formulation to perform the *predictions* of the algorithm and in the first part of this tutorial, we will only consider the *linear* kernel $K\left(x_{i},x\right)=x_{i}\cdot x$.

Now, if we look at the way the SVM model that we trained is implemented in `scikit-learn`, we have the following variables

```python
print(vars(model))
```

For the moment, our variables of interest are
- `support_vectors_` : _Support vectors_ (dataset points that support classification)
- `dual_coef_` : The $\alpha_{i} * y_{i}$ (coefficients from the dual formulation) for the _support_vectors_
- `n_support_` : Number of support vectors for each class (ordered)
- `_intercept_` : Value of $b$ in our formulation (bias)

In order to begin our own implementation of the SVM, we are going to start by computing the _decision function_ of the model. We are going to slightly change the model structure, to better reflect the mathematical formulation than `scikit-learn`. Hence, our model structure and all of its corresponding parameters will be defined as

``` Python
model = {};
model["x"];            # Values of _support vectors_ in input data
model["y"];            # Classes labels of _support vectors_
model["n"];            # Number of support for each class
model["b"];            # Value of bias
model["alphas"];       # Value of alphas
model["kernel"];       # Type of kernel
model["w"];            # Weights of vectors in primal formulation
```

In the following, we show how to shift from the `scikit-learn` version to ours

```python
#%% Creating a fake (random) model
our_model = {}
# Values of _support vectors_ in input data
our_model["x"] = model.support_vectors_
# Corresponding classes of the support vectors
our_model["y"] = np.concatenate([[0] * model.n_support_[0], [1] * model.n_support_[1]])
# Number of vectors for each class
our_model["n"] = model.n_support_
# Value of threshold
our_model["b"] = model._intercept_
# Values of alphas
our_model["dual_coef"] = model._dual_coef_[0]
our_model["kernel_type"] = "linear"
print(our_model)
```

Computing the _predictions_ of the SVM is performed by calling `model.decision_function(X)` in `scikit_learn`. We are going to implement this behavior ourselves by coding our own `decision_function` function

``` Python
def decision_function(model, X)
    """
    % Returns a vector of predictions using a trained SVM model.
    % model   : svm model returned from svmTrain.
    % X       : m x n matrix where each example is a row.
    """
    return pred
```

For the time being we are using a _linear kernel_, but we might later use a different type of kernel, so we will directly use a `kernel` function which computes our kernel between the support vectors `x_s` and a dataset `X`.

***

**Exercice**
1. Implement the linear kernel function
1. Complete the decision function based on the dual formulation

***

```python
def kernel(x_1, xs, type='linear'):
    ######################
    # YOUR CODE GOES HERE
    ######################
    if (type == 'linear'):
        ks = np.matmul(x_1, xs)
    return ks

# This function should replicates the scikit-learn one
def decision_function(model, X):
    ######################
    # YOUR CODE GOES HERE
    ######################
    x_1 = model["x"]
    ks = kernel(x_1, X, model["kernel_type"])
    bs = np.broadcast_to(model["b"], shape=(ks.shape[1]))
    dual_coefs = model["dual_coef"][:, np.newaxis].transpose()
    m = np.matmul(dual_coefs, ks)
    res = m + bs
    return res

# Evaluate the decision function
ax = plt.figure(figsize=(10,8)).gca()
# ax.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor = 'w', cmap='winter')
ax.set_xlim(-1, 4); ax.set_ylim(-1, 6)
xlim = ax.get_xlim(); ylim = ax.get_ylim()
# Create grid to evaluate model
x = np.linspace(xlim[0], xlim[1], 30)
yp = np.linspace(ylim[0], ylim[1], 30)
Y, Xp = np.meshgrid(yp, x)
xy = np.vstack([Xp.ravel(), Y.ravel()]).T
P = decision_function(our_model, xy.transpose())
P = P.reshape((30, 30))
# plot decision boundary and margins
ax.contour(Xp, Y, P, colors='w',levels=[-1, 0, 1], alpha=0.9,linestyles=['--', '-', '--']);
ax.scatter(our_model["x"][:, 0], our_model["x"][:, 1],s=300, linewidth=2, edgecolor='w', facecolors='none');
```

## Learning phase

In order to perform learning, we will try to maximize the *dual formulation* problem expressed as

$$
\begin{equation}
w\left(\alpha\right)=\sum_{i}\alpha_{i}-\frac{1}{2}\sum_{i}\sum_{j}y_{i}y_{j}\alpha_{i}\alpha_{j}\left\langle \phi\left(x_{i}\right),\phi\left(x_{j}\right)\right\rangle
\end{equation}
$$

with $0\leq \alpha_{i} \leq C$ and $\sum_{i=1}^{m}{\alpha_{i} y_{i}} = 0$. $C$ is a chosen *regularization hyper-parameter*. To ease your work, we will implement only a simplified version of the SMO algorithm which is mainly focused on the optimization aspects. The SMO algorithm selects two $\alpha$ parameters, $\alpha_{i}$ and $\alpha_{j}$ and optimizes the objective value jointly for both these $\alpha_{i,j}$. Finally it adjusts the $b$ parameter based on the new $\alpha_{i,j}$. This process is repeated until the $\alpha_{i,j}$ converge.

### Selecting $\alpha$

The full SMO algorithm use heuristics to choose which $\alpha_{i}$ and $\alpha_{j}$ to optimize (which greatly increase the convergence speed. However, for this simplified version, we will simply iterate on all $\alpha_{i},i=1,\cdots ,m$ and select a random $\alpha_{j}$, to perform the joint optimization of $\alpha_{i}$ and $\alpha_{j}$

### Optimizing $\alpha_{i}$ and $\alpha_{j}$

To jointly optimize the value, we first find bounds $L$ and $H$ such that $L\leq \alpha_{j} \leq H$ holds to satisfy the constraint of the original problem $0\leq \alpha_{i} \leq C$

$$
\begin{array}{ccc}
if y_{i}\neq y_{j}, & L=max\left(0,\alpha_{j}-\alpha_{i}\right), & H=min\left(C,C+\alpha_{j}-\alpha_{i}\right)\\
if y_{i}=y_{j}, & L=max\left(0,\alpha_{i}+\alpha_{j}-C\right), & H=min\left(C,\alpha_{i}+\alpha_{j}\right)
\end{array}
$$

Now we want to find $\alpha_{j}$ so as to maximize the objective function. It can be shown (as in the original paper) that the optimal $\alpha_{j}$ is given by

$$
\alpha_{j}=\alpha_{j}-\frac{y_{j}\left(E_{i}-E_{j}\right)}{\eta}
$$

where

$$
\begin{array}{ccc}
E_{k} & = & f\left(x_{k}\right)-y_{k}\\
\eta & = & 2\left\langle x_{i},x_{j}\right\rangle -\left\langle x_{i},x_{i}\right\rangle -\left\langle x_{j},x_{j}\right\rangle
\end{array}
$$

$E_{k}$ defines the error between the prediction of the SVM on the example $k$ and its desired label. When computing $\eta$, we can also replace the inner product by a kernel function $K$. Finally, we clip the value of $\alpha_{j}$ to lie within the selected bounds.

$$
\alpha_{j}=\begin{cases}
H & if \alpha_{j}>H\\
\alpha_{j} & if L\leq\alpha_{j}\leq H\\
L & if \alpha_{j}<L
\end{cases}
$$

Now that we have solved $\alpha_{j}$, we can update the value for $\alpha_{i}$ through

$$
\alpha_{i}^{t}=\alpha_{i}^{t-1}+y_{i}y_{j}\left(\alpha_{j}^{(t-1)}-\alpha_{j}\right)
$$

### Computing threshold $b$

Finally, we can compute the threshold $b$ to satisfy the original constraints by selecting

$$
b=\begin{cases}
b_{1} & if 0<\alpha_{i}<C\\
b_{2} & if 0<\alpha_{j}<C\\
\left(b_{1}+b_{2}\right)/2 & otherwise
\end{cases}
$$

where

$$
b_{1}=b-E_{i}-y_{i}\left(\alpha_{i}^{(t-1)}-\alpha_{i}\right)\left\langle x_{i},x_{i}\right\rangle -y_{j}\left(\alpha_{j}^{(t-1)}-\alpha_{j}\right)\left\langle x_{i},x_{j}\right\rangle
$$
$$
b_{2}=b-E_{i}-y_{i}\left(\alpha_{i}^{(t-1)}-\alpha_{i}\right)\left\langle x_{i},x_{j}\right\rangle -y_{j}\left(\alpha_{j}^{(t-1)}-\alpha_{j}\right)\left\langle x_{j},x_{j}\right\rangle
$$

For the first part of this tutorial, we will compute the main iterations of the algorithm (minimization of the objective function), while relying on a *linear* kernel. This implies that we will only be able to perform linear discrimination. However, remember that the formulation of the SVMs provide an *optimal* and (gloriously) *convex* answer to this problem.

```Python
def svm_train(X, Y, C, kernelFunction, tol, maxIter):
    """
    % Trains an SVM classifier using a simplified SMO algorithm.
    % X       : m x n matrix of m training examples (with n-dimensional features).
    % Y       : column vector of class identifiers
    % C       : standard SVM regularization parameter
    % tol     : tolerance value used for determining equality of floating point numbers.
    % maxIter : number of iterations over the dataset before the algorithm stops.
    """
    return model
```

***

**Exercise**

  1. Update the `svm_train` code to perform the training of a SVM with linear kernel.
  2. Run your `svm_train` function on our linear dataset to obtain similar results as `scikit-learn`
  3. Update the framework to display the decision boundary at each iteration of the algorithm.

***

```python
def svm_train(X, Y_b, C, kernel_type, tol=1e-3, max_iter=5):
    """
    SVMTRAIN Trains an SVM classifier using a simplified version of the SMO
    # X       : m x n matrix of m training examples (with n-dimensional features).
    # Y       : column vector of class identifiers (0 or 1)
    # C       : standard SVM regularization parameter
    # tol     : tolerance value used for determining equality of floating point numbers.
    # maxIter : number of iterations over the dataset before the algorithm stops.
    """
    # Data parameters
    m, n = X.shape
    # Variables
    alphas = np.zeros((m, 1))
    b, passes, eta, L, H = 0, 0, 0, 0, 0
    E = np.zeros((m, 1))
    Y = Y_b.copy()
    Y[Y == 0] = -1

    #%% Creating a fake (random) model
    model = {}
    # Values of _support vectors_ in input data
    model["x"] = X
    # Corresponding classes of the support vectors
    model["y"] = Y
    # Number of vectors for each class
    model["n"] = n
    # Value of threshold
    model["b"] = b
    # Values of alphas
    model["alphas"] = alphas
    model["dual_coef"] = alphas * Y
    model["kernel_type"] = "linear"

    # Pre-compute the kernel matrix (in practice, the optimized packages will _not_ do this)
    K = np.array(kernel(X, X.transpose(), kernel_type))
    # Train
    print('Training ...')
    while passes < max_iter:
        # Check that some alphas changed
        num_changed_alphas = 0
        # Iterative over all alpha_i
        for i in range(m):
            ######################
            # YOUR CODE GOES HERE
            # => Perform one pass of the SMO algorithm for each alpha_i
            ######################
            j = rand.randint(m-1)
            print(i, j)
            if j == i:
                j = m-1
            # m_v = lambda x: np.array(x)[:, np.newaxis]
            make_vector_el = lambda x: x.reshape((*x.shape, 1))
            x_i, x_j = model["x"][i], model["x"][j]
            y_i, y_j = model["y"][i], model["y"][j]
            alpha_i, alpha_j = model["alphas"][i], model["alphas"][j]
            l = h = 0
            if y_i == y_j:
                l = max(0, alpha_j - alpha_i)
                h = C + min(0, alpha_j - alpha_i)
            else:
                l = max(0, alpha_i + alpha_j - C)
                h = min(C, alpha_i + alpha_j)
            e_i = np.sum(model["alphas"].transpose() * model["y"].transpose() * K[:, i]) - y_i
            e_j = np.sum(model["alphas"].transpose() * model["y"].transpose() * K[:, j]) - y_i
            eta = 2*K[i, j] - K[i, i] - K[j, j]

            # compute new alpha_j
            alpha_j_previous = alpha_j
            alpha_i_previous = alpha_i
            b_previous = b

            alpha_j = max(L, min(H, alpha_j - y_j * (e_i - e_j) / eta))
            alpha_i = alpha_i + y_i * y_j * (alpha_j_previous - alpha_j)

            b_1 = b_previous - e_i - y_i * (alpha_i_previous - alpha_i) * K[i, i] - y_j * (alpha_j_previous - alpha_j) * K[i, j]
            b_2 = b_previous - e_i - y_i * (alpha_i_previous - alpha_i) * K[i, i] - y_j * (alpha_j_previous - alpha_j) * K[j, j]
            b = b_1 if 0 < alpha_i  < C else b_2 if 0 < alpha_j < C else (b_1 + b_2) * 0.5

            model["alphas"][i], model["alphas"][j] = alpha_i, alpha_j
            model["b"] = b

            count_same = lambda a, b: 1 if np.linalg.norm(a - b) < tol else 0

            num_changed_alphas += count_same(alpha_i, alpha_i_previous) + count_same(alpha_j, alpha_j_previous) + count_same(b_previous, b)

        if (num_changed_alphas == 0):
            passes = passes + 1
        else:
            passes = 0
            print(model["alphas"])
    print(' Done !');
    # Save the model
    idx = (alphas > 0).transpose()[0]
    model = {}
    model["x"] = X[idx, :]
    model["y"] = Y[idx]
    model["b"] = b
    model["dual_coef"] = alphas[idx, 0] * Y[idx]
    model["kernel"] = kernel
    model["kernel_type"] = kernel_type
    return model

# Training Linear SVM
print('Training Linear SVM ...')
# Parameters of the training
C = 1                  # C regularization value
tolerance = 1e-5       # Tolerance value for thresholds
nb_iterations = 50      # Number of iterations
# Perform the training (try to fiddle the previous values to see effect).
model = svm_train(X, y, C, 'linear', tolerance, nb_iterations);
# Evaluate the decision function
ax = plt.figure(figsize=(10, 8)).gca()
ax.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor = 'w', cmap='winter')
ax.set_xlim(-1, 4); ax.set_ylim(-1, 6)
xlim = ax.get_xlim(); ylim = ax.get_ylim()
# Create grid to evaluate model
x = np.linspace(xlim[0], xlim[1], 30)
yp = np.linspace(ylim[0], ylim[1], 30)
Y, Xp = np.meshgrid(yp, x)
xy = np.vstack([Xp.ravel(), Y.ravel()]).T
P = decision_function(model, xy.transpose())
P = P.reshape((30, 30))
# plot decision boundary and margins
ax.contour(Xp, Y, P, colors='w',levels=[-1, 0, 1], alpha=0.9,linestyles=['--', '-', '--'])
ax.scatter(model["x"][:, 0], model["x"][:, 1],s=300, linewidth=2, edgecolor='w', facecolors='none');
plt.title('Our trained SVM :)');
```

```python

```

# Beyond linear problems with _kernels_

For the time being, our SVM is only able to solve _linear_ problems. However, the SVM approach becomes truly powerful is when it is combined with *kernels*. As seen in the slides, the major idea is to project our (non-linearly separable) data into a higher-dimensional space, where it becomes linearly separable (allowing to fit for nonlinear relationships with a linear classifier).

To better understand this idea, we can look at the same non linearly separable data that we discussed in the slides. Here, we try to fit a _linear_ SVM to this non-linear problem.

```python
from sklearn.datasets.samples_generator import make_circles
X, y = make_circles(100, factor=.1, noise=.1)

clf = SVC(kernel='linear').fit(X, y)
plt.figure(figsize=(10, 8)); plt.grid(True)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor='w', cmap='winter')
plot_svc_decision_function(clf, plot_support=False);
```

It is clear that no linear discrimination will *ever* be able to separate this data. As discussed, we need to think about how we might project the data into a higher dimension such that a linear separator *would* be sufficient. For example, one simple projection we could use would be to compute a *radial basis function* centered on the middle group of points.

```python
r = np.exp(-(X ** 2).sum(1))
```

We can visualize this extra data dimension using a three-dimensional plot

```python
from mpl_toolkits import mplot3d

def plot_3D(elev=30, azim=30, X=X, y=y):
    plt.figure(figsize=(10, 8));
    ax = plt.subplot(projection='3d')
    ax.scatter3D(X[:, 0], X[:, 1], r, c=y, s=50, edgecolor='w', cmap='winter')
    ax.view_init(elev=elev, azim=azim)
    ax.set_xlabel('x')
    ax.set_ylabel('y')
    ax.set_zlabel('z')
plot_3D(elev=30, azim=-50, X=X, y=y);
```

We can see that with this additional dimension, the data becomes trivially linearly separable, by drawing a separating plane. Here we had to choose and carefully tune our projection. In general, the need to make such a choice is a problem, as we would like to somehow automatically find the best basis functions to use. One strategy to this end is to compute a basis function centered at *every* point in the dataset, and let the SVM algorithm find the result. This type of basis function transformation is known as a *kernel transformation*, as it is based on a similarity relationship (or kernel) between **each pair of points**.

To see this idea in action first, we will (as previously) use `scikit-learn`, where we can apply _kernelized SVM_ simply by changing our _linear kernel_ to an _RBF (radial basis function)_ kernel, using the ``kernel`` model hyperparameter:

```python
model = SVC(kernel='rbf', C=1E6)
model.fit(X, y)
```

Once fitted, we can see that our SVM is able to find the largest margin between these two sets of data

```python
plt.figure(figsize=(10, 8)); plt.grid(True)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor='w', cmap='winter')
plot_svc_decision_function(model)
plt.scatter(model.support_vectors_[:, 0], model.support_vectors_[:, 1], s=300, lw=1, facecolors='none');
```

Using this kernelized support vector machine, we learn a suitable nonlinear decision boundary.
This kernel transformation strategy is used often in machine learning to turn fast linear methods into fast nonlinear methods, especially for models in which the kernel trick can be used.

## Implementing the gaussian kernel

As we have seen previously, we know that the complete SVM optimization problem can also be expressed as

$$
\begin{equation}
f\left(x\right)=\sum_{i=1}^{m}\alpha_{i}y_{i}\left\langle \phi\left(x_{i}\right),\phi\left(x\right)\right\rangle +b
\end{equation}
$$

where we can introduce the kernel $K\left(x_{i},x\right)=\phi\left(x_{i}\right)\cdot\phi\left(x\right)$ in the equation in order to solve non-linear optimization problems. We can use this formulation to perform the *predictions* of the algorithm. Consequently, the complete *dual formulation* problem expressed as

$$
\begin{equation}
w\left(\alpha\right)=\sum_{i}\alpha_{i}-\frac{1}{2}\sum_{i}\sum_{j}y_{i}y_{j}\alpha_{i}\alpha_{j}\left\langle \phi\left(x_{i}\right),\phi\left(x_{j}\right)\right\rangle
\end{equation}
$$

with $0\leq \alpha_{i} \leq C$ and $\sum_{i=1}^{m}{\alpha_{i} y_{i}} = 0$. The Gaussian kernel (also called *Radial Basis Function (RBF)* kernel) allows to perform the optimization of non-linear classification problems. It is defined as

$$
K\left(x,y\right)=exp\left(\frac{-\left\Vert x-y\right\Vert ^{2}}{\left(2\sigma^{2}\right)}\right)
$$

As seen in the slides, the underlying idea is that instead of trying to solve a non-linear separation problem in a given space, we might first transform the space into a target space in which the problem is linearly separable. Therefore, we transform the space through a kernel (which is directly expressed in the minimization problem) and solve the linear separation problem in the target space. Given the starter code and the previous exercise, you should be able to directly plug this kernel inside the `svmTrain` and `svmPredict` functions.

***

**Exercise**

  1. Update the `kernel` function code to be able to rely on a Gaussian kernel.
  2. Check that you can run the `decision_function` code to rely on this Gaussian kernel.
  2. Run the `svm_train` function on our new (non-linear) dataset to obtain similar results as `scikit-learn`
  3. Once again, display the decision boundary at each iteration of the algorithm.

***

```python
def kernel(x, X, type='linear'):

    ######################
    # YOUR CODE GOES HERE
    ######################

    if (type == 'linear'):
        k = ...
    elif kernel == 'gaussian':
        k = ...
    return k

plt.title('Traning Dataset')
# Parameters of the training
C = 1                  # C regularization value
tolerance = 1e-3       # Tolerance value for thresholds
nb_iterations = 20     # Number of iterations
sigma = 0.1            # Variance of Gaussian kernel
# Train the SVM with a Gaussian (RBF) kernel
model = svm_train(X, y, C, 'gaussian', tolerance, nb_iterations)
# Evaluate the decision function
ax = plt.figure(figsize=(10, 8)).gca()
ax.scatter(X[:, 0], X[:, 1], c=y, s=50, edgecolor = 'w', cmap='winter')
ax.set_xlim(-1, 4); ax.set_ylim(-1, 6)
xlim = ax.get_xlim(); ylim = ax.get_ylim()
# Create grid to evaluate model
x = np.linspace(xlim[0], xlim[1], 30)
yp = np.linspace(ylim[0], ylim[1], 30)
Y, Xp = np.meshgrid(yp, x)
xy = np.vstack([Xp.ravel(), Y.ravel()]).T
P = decision_function(model, xy.transpose())
P = P.reshape((30, 30))
# plot decision boundary and margins
ax.contour(Xp, Y, P, colors='w',levels=[-1, 0, 1], alpha=0.9,linestyles=['--', '-', '--'])
ax.scatter(model["x"][:, 0], model["x"][:, 1],s=300, linewidth=2, edgecolor='w', facecolors='none');
plt.title('Our trained SVM :)');
```

## Supplementary kernels

Now that you have mastered the concept of kernels and updated your code to include the Gaussian kernel, you can augment your classifier and optimizaton problem to include several different kernels as those proposed afterwards.

#### Polynomial kernel.

Intuitively, the polynomial kernel looks not only at the given features of input samples to determine their similarity, but also combinations of these.

$$
K\left(x,y\right)=\left(x^{T}y+1\right)^{d}
$$

#### Sigmoid (tanh) kernel.

This kernel takes two parameters: $a$ and $r$. For $a > 0$, we can view $a$ as a scaling parameter of the input data, and $r$ as a shifting parameter that controls the threshold of mapping. For $a < 0$, the dot-product of the input data is not only scaled but reversed.

$$
K\left(x,y\right)=tanh\left(kx^{T}y+\Theta\right)
$$

***

**Exercise**

  1. Update the `kernel` function code to include these supplementary kernels.
  2. Test both the `decision_function` and `svm_train` codes for these new kernels.

***

```python

```

## Audio application

As seen in the previous tutorial, we have considered only a *binary classification* problem (where elements can only belong to one of two classes). This simplifies the work as we simply need to separate the space between two types of points. However, in real-world problems, we usually want to solve *multi-class* problems with any given number of classes. For this tutorial, we will rely on a simple trick which is to consider that a $n$-class problem can simply be restated as $n$ different binary classification problems. The underlying idea is that each class defines a binary classifier which tells us if a given element is part of this class, or belongs to *any of the other* classes.

***

**Exercise**

  1. Change the training loop to define $n$separate binary problems
  2. For each problem, train an SVM to perform classification.
  3. Evaluate the global classification accuracy of your system.

***

```python

```

```python
from helper_data import import_dataset, compute_transforms, compute_features
class_path = 'data/classification'
# 0.1 - Import the classification dataset
data_struct = import_dataset(class_path, 'classification')
# 0.2 - Pre-process the audio to obtain spectral transforms
data_struct = compute_transforms(data_struct)
# 0.3 - Compute a set of temporal and spectral features
data_struct = compute_features(data_struct)
```