# Support Vector Machines

    Authors: Jesús Cid Sueiro (jcid@tsc.uc3m.es)
             Jerónimo Arenas García (jarenas@tsc.uc3m.es)

This notebook is a compilation of material taken from several sources:

- The [sklearn documentation](href = http://scikit-learn.org/stable/modules/svm.html>)
- A [notebook](https://github.com/jakevdp/sklearn_pycon2015/blob/master/notebooks/03.1-Classification-SVMs.ipynb) by [Jake Vanderplas](https://github.com/jakevdp>)
- [Wikipedia](https://en.wikipedia.org/wiki/Support_vector_machine)

    Notebook version: 1.0 (Oct 28, 2015)
                      1.1 (Oct 27, 2016)
                      2.0 (Nov 2, 2017)
                      2.1 (Oct 20, 2018)
                      2.2 (Oct 20, 2019)

    Changes:
    v.1.0 - First version
    v.1.1 - Typo correction and illustrative figures for linear SVM
    v.2.0 - Compatibility with Python 3 (backcompatible with Python 2.7)
    v.2.1 - Minor corrections on the notation
    v.2.2 - Minor equation errors. Reformatted hyperlinks. Restoring broken visualization of images in some Jupyter versions.
    v.2.3 - Notation revision

```python
from __future__ import print_function

# To visualize plots in the notebook
%matplotlib inline

# Imported libraries
#import csv
#import random
#import matplotlib
import matplotlib.pyplot as plt
from mpl_toolkits import mplot3d
#import pylab

import numpy as np
#from sklearn.preprocessing import PolynomialFeatures
from sklearn import svm
from sklearn.datasets.samples_generator import make_blobs
from sklearn.datasets.samples_generator import make_circles

from ipywidgets import interact
```

## 1. Introduction

  [Source: [sklearn documentation](http://scikit-learn.org/stable/modules/svm.html)  ]

Support vector machines (SVMs) are a set of supervised learning methods used for classification, regression and outliers detection.

The advantages of support vector machines are:

- Effective in high dimensional spaces.
- Still effective in cases where number of dimensions is greater than the number of samples.
- Uses a subset of training points in the decision function (called support vectors), so it is also memory efficient.
- Versatile: different Kernel functions can be specified for the decision function.

The disadvantages of support vector machines include:

- SVMs do not directly provide probability estimates.

## 2. Motivating Support Vector Machines

  [Source: A [notebook](https://github.com/jakevdp/sklearn_pycon2015/blob/master/notebooks/03.1-Classification-SVMs.ipynb) by [Jake Vanderplas](https://github.com/jakevdp>)]

Support Vector Machines (SVMs) are a kind of ***discriminative*** classifiers: that is, they draw a boundary between clusters of data without making any explicit assumption about the probability model underlying the data generation process.

Let's show a quick example of support vector classification. First we need to create a dataset:

```python
X, y = make_blobs(n_samples=50, centers=2, random_state=0, cluster_std=0.60)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='copper')
plt.xlabel("$x_0$", fontsize=14)
plt.ylabel("$x_1$", fontsize=14)
plt.axis('equal')
plt.show()
```

A discriminative classifier attempts to draw a line between the two sets of data. Immediately we see an inconvenience: such problem is ill-posed! For example, we could come up with several possibilities which perfectly discriminate between the classes in this example:

```python
xfit = np.linspace(-1, 3.5)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='copper')

for m, b in [(1, 0.65), (0.5, 1.6), (-0.2, 2.9)]:
    plt.plot(xfit, m * xfit + b, '-k')

plt.xlim(-1, 3.5);
plt.xlabel("$x_0$", fontsize=14)
plt.ylabel("$x_1$", fontsize=14)
plt.axis('equal')
plt.show()
```

These are three very different separators which perfectly discriminate between these samples. Depending on which you choose, a new data point will be classified almost entirely differently! How can we improve on this?

Support Vector Machines (SVM) select the boundary decision maximizing the ***margin***. The margin of a classifier is defined as twice the maximum signed distance between the decision boundary and the training data. By *signed* we mean that the distance to misclassified samples is counted negatively. Thus, if the classification problem is "separable" (i.e. if there exist a decision boundary with zero errors in the training set), the SVM will choose the zero-error decision boundary that is "as far as possible" from the training data.

In summary, what an SVM does is to not only draw a line, but consider the "sample free" region about the line. Here's an example of what it might look like:

```python
xfit = np.linspace(-1, 3.5)
plt.scatter(X[:,0], X[:,1], c=y, s=50, cmap='copper')

for m, b, d in [(1, 0.65, 0.33), (0.5, 1.6, 0.55), (-0.2, 2.9, 0.2)]:
    yfit = m * xfit + b
    plt.plot(xfit, yfit, '-k')
    plt.fill_between(xfit, yfit-d, yfit+d, edgecolor='none',
                     color='#AAAAAA', alpha=0.4)

plt.xlim(-1, 3.5)
plt.xlabel("$x_0$", fontsize=14)
plt.ylabel("$x_1$", fontsize=14)
plt.axis('equal')
plt.show()
```

Notice here that if we want to maximize this width, the middle fit is clearly the best. This is the intuition of the SVM, which optimizes a linear discriminant model in conjunction with a margin representing the perpendicular distance between the datasets.

## 3. Linear SVM

  [Source: adapted from [Wikipedia](https://en.wikipedia.org/wiki/Support_vector_machine)]

In order to present the SVM in a formal way, consider a training dataset $\mathcal{D} = \left\{ (\mathbf{x}_k, y_k) \mid \mathbf{x}_k\in \Re^M,\, y_k \in \{-1,1\}, k=0,\ldots, {K-1}\right\}$, where the binary symmetric label $y_k\in \{-1,1\}$ indicates the class to which the point $\mathbf{x}_k$ belongs. Each $\mathbf{x}_k$ is an $M$-dimensional real vector. We want to find the maximum-margin hyperplane that divides the points having $y_k=1$ from those having $y_k=-1$.

Any hyperplane can be written as the set of points $\mathbf{x}$ satisfying

$$
\mathbf{w}^\intercal \mathbf{x} - b=0,
$$

where ${\mathbf{w}}$ denotes the (not necessarily normalized) normal vector to the hyperplane. The parameter $\tfrac{b}{\|\mathbf{w}\|}$ determines the offset of the hyperplane from the origin along the normal vector ${\mathbf{w}}$.

If the training data are linearly separable, we can select two parallel hyperplanes in a way that they separate the data and there are no points between them, and then try to maximize their distance. The region bounded by them is called "the margin". These hyperplanes can be described by the equations

$$\mathbf{w}^\intercal \mathbf{x} - b=1$$
and
$$\mathbf{w}^\intercal \mathbf{x} - b=-1.$$

Note that the two equations above can represent any two parallel hyperplanes in $\Re^M$. Essentially, the direction of vector $\mathbf{w}$ determines the orientation of the hyperplanes, whereas parameter $b$ and the norm of $\mathbf{w}$ can be used to select their exact location.

To compute the distance between the hyperplanes, we can obtain the projection of vector ${\mathbf x}_1 - {\mathbf x}_2$, where ${\mathbf x}_1$ and ${\mathbf x}_2$ are points from each of the hyperplanes, onto a unitary vector orthonormal to the hyperplanes:

$$\text{Distance between hyperplanes} = \left[\frac{\mathbf{w}}{\|\mathbf{w}\|}\right]^\top ({\mathbf x}_1 - {\mathbf x}_2) = \frac{\mathbf{w}^\top {\mathbf x}_1 - \mathbf{w}^\top {\mathbf x}_2}{\|\mathbf{w}\|} = \frac{2}{\|\mathbf{w}\|}.$$

Therefore, to maximize the distance between the planes we want to minimize $\|\mathbf{w}\|$.

As we also have to prevent data points from falling into the margin, we add the following constraints: for each $k$ either

\begin{align}
\mathbf{w}^\top \mathbf{x}_k - b &\ge +1, \qquad\text{ if  }\;\;y_k=1, \qquad \text{or} \\
\mathbf{w}^\top \mathbf{x}_k - b &\le -1, \qquad\text{ if  }\;\;y_k=-1.
\end{align}

This can be rewritten as:

$$
y_k(\mathbf{w}^\top \mathbf{x}_k - b) \ge 1, \quad \text{ for all } 0 \le k \le K-1.
$$

We can put this together to get the optimization problem:

$$
(\mathbf{w}^*,b^*) = \arg\min_{(\mathbf{w},b)} \|\mathbf{w}\| \\
\text{subject to: }
y_k(\mathbf{w}^\top \mathbf{x}_k - b) \ge 1, \, \text{ for any } k = 0, \dots, {K-1}
$$

This optimization problem is difficult to solve because it depends on $\|\mathbf{w}\|$, the norm of $\mathbf{w}$, which involves a square root. Fortunately it is possible to alter the minimization objective $\|\mathbf{w}\|$ by substituting it with $\tfrac{1}{2}\|\mathbf{w}\|^2$ (the factor of $\frac{1}{2}$ being used for mathematical convenience) without changing the solution (the minimum of the original and the modified equation have the same $\mathbf{w}$ and $b$):

$$
(\mathbf{w}^*,b^*) = \arg\min_{(\mathbf{w},b)} \frac{1}{2}\|\mathbf{w}\|^2 \\
\text{subject to: }
y_k(\mathbf{w}^\top \mathbf{x}_k - b) \ge 1, \, \text{ for any } k = 0, \dots, {K-1}
$$

This is a particular case of a *quadratic programming* problem.

### 3.1. Primal form

The optimization problem stated in the preceding section can be solved by means of a generalization of the Lagrange method of multipliers for inequality constraints, using the so called Karush–Kuhn–Tucker (KKT) multipliers $\boldsymbol{\alpha}$. According to it, the constrained problem can be expressed as

$$(\mathbf{w}^*,b^*, \boldsymbol{\alpha}^*) = \arg\min_{\mathbf{w},b } \max_{\boldsymbol{\alpha}\geq 0 } \left\{ \frac{1}{2}\|\mathbf{w}\|^2 - \sum_{k=0}^{K-1}{\alpha_k\left[y_k(\mathbf{w}^\top \mathbf{x}_k - b)-1\right]} \right\}
$$

that is, we look for a *saddle point*.

A key result in convex optimization theory is that, for the kind of optimization problems discussed here (see [here](http://www.onmyphd.com/?p=kkt.karush.kuhn.tucker&ckattempt=1), for instance), the *max* and *min* operators are interchangeable, so that
$$
(\mathbf{w}^*,b^*, \boldsymbol{\alpha}^*) = \arg\max_{\boldsymbol{\alpha}\geq 0 } \min_{\mathbf{w},b } \left\{ \frac{1}{2}\|\mathbf{w}\|^2 - \sum_{k=0}^{K-1}{\alpha_k\left[y_k(\mathbf{w}^\top \mathbf{x}_k - b)-1\right]} \right\}
$$

Note that the inner minimization problem is now quadratic in $\mathbf{w}$ and, thus, the minimum can be found by differentiation:
$$
\mathbf{w}^* = \sum_{k=0}^{K-1}{\alpha_k y_k\mathbf{x}_k}.
$$

#### 3.1.1. Support Vectors

In view of the optimization problem, we can check that all the points which can be separated as $y_k(\mathbf{w}^\top \mathbf{x}_k - b) - 1 > 0 $ do not matter since we must set the corresponding $\alpha_k$ to zero. Therefore, only a few $\alpha_k$ will be greater than zero. The corresponding $\mathbf{x}_k$ are known as `support vectors`.

It can be seen that the optimum parameter vector $\mathbf{w}^\ast$ can be expressed in terms of the support vectors only:

$$
\mathbf{w}^* = \sum_{k\in {\cal{S}}_{SV}}{\alpha_k y_k\mathbf{x}_k}.
$$

where ${\cal{S}}_{SV}$ is the set of indexes associated to support vectors.

#### 3.1.2. The computation of $b$

Support vectors lie on the margin and satisfy $y_k(\mathbf{w}^\top \mathbf{x}_k - b) = 1$. From this condition, we can obtain the value of $b$, since for any support vector:

$$\mathbf{w}^\top\mathbf{x}_k - b = \frac{1}{y_k} = y_k \iff b = \mathbf{w}^\top\mathbf{x}_k - y_k
$$

This estimate of $b$, the centerpoint of the division, depends only on a single pair $y_k$ and $x_k$.  We may get a more robust estimate of the center by averaging over all of the $N_{SV}$ support vectors, if we believe the population mean is a good estimate of the midpoint, so in practice, $b$ is often computed as:

$$
b = \frac{1}{N_{SV}} \sum_{\mathbf{x}_k\in {\cal{S}}_{SV}}{(\mathbf{w}^\top\mathbf{x}_k - y_k)}
$$

### 3.2. Dual form

Writing the classification rule in its unconstrained dual form reveals that the *maximum-margin hyperplane* and therefore the classification task is only a function of the *support vectors*, the subset of the training data that lie on the margin.

Using the fact that $\|\mathbf{w}\|^2 = \mathbf{w}^\top \mathbf{w}$ and substituting $\mathbf{w} = \sum_{k=0}^{K-1}{\alpha_k y_k\mathbf{x}_k}$, we obtain

\begin{align}
(b^*, \boldsymbol{\alpha}^*)
   &= \arg\max_{\boldsymbol{\alpha}\geq 0 }
   \min_b \left\{
   \sum_{k=0}^{K-1}\alpha_k -
   \frac{1}{2}
   \sum_{k=0}^{K-1} \sum_{j=0}^{K-1} {\alpha_k \alpha_j y_k y_j \mathbf{x}_k^\top\mathbf{x}_j}
   + b \sum_{k=0}^{K-1}\alpha_k y_k
\right\}
\end{align}

Note that, if $\sum_{k=0}^{K-1}\alpha_k y_k \neq 0$ the optimal value  of $b$ is $+\infty$ of $-\infty$, and

\begin{align}
\min_b \left\{
\sum_{k=0}^{K-1}\alpha_k -
\frac{1}{2}
\sum_{k=0}^{K-1} \sum_{j=0}^{K-1} {\alpha_k \alpha_j y_k y_j \mathbf{x}_k^\top\mathbf{x}_j}
+ b \sum_{k=0}^{K-1}\alpha_k y_k
\right\} = -\infty.
\end{align}

Therefore, any $\boldsymbol{\alpha}$ satifying $\sum_{k=0}^{K-1}\alpha_k y_k \neq 0$ is suboptimal, so that the optimal multipliers must satisfy the condition  $\sum_{k=0}^{K-1}\alpha_k y_k = 0$.

Summarizing, the dual formulation of the optimization problem is

$$
\boldsymbol{\alpha}^* = \arg\max_{\boldsymbol{\alpha}\geq 0} \sum_{k=0}^{K-1} \alpha_k -
       \frac12 \sum_{k,j} \alpha_k \alpha_j y_k y_j\;\kappa(\mathbf{x}_k, \mathbf{x}_j)  \\
\text{subject to: } \qquad \sum_{k=0}^{K-1} \alpha_k y_k = 0.
$$

where the *kernel* $\kappa(\cdot)$ is defined by $\kappa(\mathbf{x}_k,\mathbf{x}_j)=\mathbf{x}_k^\top\mathbf{x}_j$.

Many implementations of the SVM use this dual formulation. They proceed in three steps:

1. Solve the dual problem to obtain $\boldsymbol{\alpha}^*$. Usually, only a small number of $\alpha_k^*$ are nonzero. The corresponding values of ${\bf x}_k$ are called the *support vectors*.

2. Compute $\mathbf{w}^* = \sum_{k=0}^{K-1} \alpha_k^* y_k\mathbf{x}_k$

3. Compute $b^* = \frac{1}{N_{SV}} \sum_{\alpha_k^*\neq 0}{(\mathbf{w}^{*\top}\mathbf{x}_k - y_k)}
$

## 4. Fitting a Support Vector Machine

  [Source: A [notebook](https://github.com/jakevdp/sklearn_pycon2015/blob/master/notebooks/03.1-Classification-SVMs.ipynb) by [Jake Vanderplas](https://github.com/jakevdp>)]

Now we'll fit a Support Vector Machine Classifier to these points.

```python
clf = svm.SVC(kernel='linear')
clf.fit(X, y)
```

    SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
      decision_function_shape='ovr', degree=3, gamma='auto', kernel='linear',
      max_iter=-1, probability=False, random_state=None, shrinking=True,
      tol=0.001, verbose=False)

To better visualize what's happening here, let's create a quick convenience function that will plot SVM decision boundaries for us:

```python
def plot_svc_decision_function(clf, ax=None):
    """Plot the decision function for a 2D SVC"""
    if ax is None:
        ax = plt.gca()
    x = np.linspace(plt.xlim()[0], plt.xlim()[1], 30)
    y = np.linspace(plt.ylim()[0], plt.ylim()[1], 30)
    Y, X = np.meshgrid(y, x)
    P = np.zeros_like(X)
    for i, xi in enumerate(x):
        for j, yj in enumerate(y):
            P[i, j] = clf.decision_function(np.array([xi, yj]).reshape(1,-1))
    # plot the margins
    ax.contour(X, Y, P, colors='k',
               levels=[-1, 0, 1], alpha=0.5,
               linestyles=['--', '-', '--'])

plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='copper')
plot_svc_decision_function(clf);
```

Notice that the dashed lines touch a couple of the points: these points are the *support vectors*. In scikit-learn, these are stored in the ``support_vectors_`` attribute of the classifier:

```python
plt.scatter(clf.support_vectors_[:, 0], clf.support_vectors_[:, 1],
            s=200, marker='s');
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='copper')
plot_svc_decision_function(clf)

```

Let's use IPython's interact functionality to explore how the distribution of points affects the support vectors and the discriminative fit.

```python
def plot_svm(N=10):
    X, y = make_blobs(n_samples=200, centers=2,
                      random_state=0, cluster_std=0.60)
    X = X[:N]
    y = y[:N]
    clf = svm.SVC(kernel='linear')
    clf.fit(X, y)
    plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='copper')
    plt.xlim(-1, 4)
    plt.ylim(-1, 6)
    plot_svc_decision_function(clf, plt.gca())
    plt.scatter(clf.support_vectors_[:, 0], clf.support_vectors_[:, 1],
                s=200, facecolors='none')

interact(plot_svm, N=[10, 200], kernel='linear')
```

    interactive(children=(Dropdown(description='N', options=(10, 200), value=10), Output()), _dom_classes=('widget…

    <function __main__.plot_svm(N=10)>

Notice the unique thing about SVM is that only the support vectors matter: that is, if you moved any of the other points without letting them cross the decision boundaries, they would have no effect on the classification results!

## 5. Non-separable problems.

  [Source: adapted from [Wikipedia](https://en.wikipedia.org/wiki/Support_vector_machine)]

In 1995, Corinna Cortes and Vladimir N. Vapnik suggested a modified maximum margin idea that allows for mislabeled examples. If there exists no hyperplane that can split the `positive` and `negative` samples, the `Soft Margin`
 method will choose a hyperplane that splits the examples as cleanly as possible, while still maximizing the distance to the nearest cleanly split examples. The method introduces non-negative slack variables, $\xi_k$, which measure the degree of misclassification of the data $\mathbf{x}_k$

$$y_k(\mathbf{w}^\top\mathbf{x}_k - b) \ge 1 - \xi_k \quad k=0,\ldots, K-1.
$$

The objective function is then increased by a function which penalizes non-zero $\xi_k$, and the optimization becomes a trade off between a large margin and a small error penalty. If the penalty function is linear, the optimization problem becomes:

$$(\mathbf{w}^*,\mathbf{\xi}^*, b^*) = \arg\min_{\mathbf{w},\mathbf{\xi}, b } \left\{\frac{1}{2} \|\mathbf{w}\|^2 + C \sum_{k=0}^{K-1} \xi_k \right\} \\
\text{subject to: } \quad y_k(\mathbf{w}^\intercal\mathbf{x}_k - b) \ge 1 - \xi_k, \quad \xi_k \ge 0, \quad  k=0,\ldots, K-1.
$$

This constraint along with the objective of minimizing $\|\mathbf{w}\|$ can be solved using KKT multipliers as done above. One then has to solve the following problem:

$$
\arg\min_{\mathbf{w}, \mathbf{\xi}, b } \max_{\boldsymbol{\alpha}, \boldsymbol{\beta} }
\left\{ \frac{1}{2}\|\mathbf{w}\|^2
+ C \sum_{k=0}^{K-1} \xi_k
- \sum_{k=0}^{K-1} {\alpha_k\left[y_k(\mathbf{w}^\top \mathbf{x}_k - b) -1 + \xi_k\right]}
- \sum_{k=0}^{K-1} \beta_k \xi_k \right \}\\
\text{subject to: } \quad
\alpha_k, \beta_k \ge 0.
$$

A similar analysis to that in the separable case can be applied to show that the dual formulation of the optimization problem is

$$
        \boldsymbol{\alpha}^* = \arg\max_{0 \leq \alpha_k \leq C} \sum_{k=0}^{K-1} \alpha_k -
       \frac12 \sum_{k,j} \alpha_k \alpha_j y_k y_j \;\kappa(\mathbf{x}_k, \mathbf{x}_j)  \\
\text{subject to: } \qquad \sum_{k=0}^{K-1} \alpha_k y_k = 0.
$$
Note that the only difference with the separable case is given by the constraints $\alpha_k \leq C$.

## 6. Nonlinear classification

  [Source: adapted from [Wikipedia](https://en.wikipedia.org/wiki/Support_vector_machine)]

The original optimal hyperplane algorithm proposed by Vapnik in 1963 was a linear classifier. However, in 1992, Bernhard E. Boser, Isabelle M. Guyon and Vladimir N. Vapnik suggested a way to create nonlinear classifiers by applying the *kernel trick* to maximum-margin hyperplanes. The resulting algorithm is formally similar, except that every dot product is replaced by a nonlinear kernel function. This allows the algorithm to fit the maximum-margin hyperplane in a transformed feature space. The transformation may be nonlinear and the transformed space high dimensional; thus though the classifier is a hyperplane in the high-dimensional feature space, it may be nonlinear in the original input space.

The kernel is related to the transform $\phi(\mathbf{x})$ by the equation $\kappa(\mathbf{x}, \mathbf{x}') = \phi(\mathbf{x})^\top \phi(\mathbf{x}')$. However, note that we do not need to explicitly compute $\phi(\mathbf{x})$, as long as we can express all necessary calculations in terms of the kernel function only, as it is the case for the optimization problem in the dual case.

The predictions of the SVM classifier can also be expressed in terms of kernels only, so that we never need to explicitely compute $\phi(\mathbf{x})$.

$$\begin{align}\hat y({\mathbf{x}}) & = {\mathbf {w^\ast}}^\top \phi(\mathbf{x}) - b^\ast \\ \\
& = \left[\sum_{k \in {\cal{S}}_{SV}} \alpha_k^* y_k \phi(\mathbf{x}_k)\right]^\top {\phi(\mathbf{x})} - b^\ast  \\ \\
& = - b^\ast + \sum_{k \in {\cal{S}}_{SV}} \alpha_k^* y_k \; \kappa(\mathbf{x}_k, {\mathbf{x}})
\end{align}$$

Some common kernels include:

* **Gaussian**: $\kappa(\mathbf{x},\mathbf{x}')=\exp(-\gamma \|\mathbf{x} - \mathbf{x}'\|^2)$, for $\gamma > 0$. Sometimes parametrized using $\gamma=\dfrac{1}{2 \sigma^2}$. This is by far the most widely used kernel.
* Polynomial (homogeneous): $\kappa(\mathbf{x},\mathbf{x}')=(\mathbf{x}^\top \mathbf{x}')^d$
* Polynomial (inhomogeneous): $\kappa(\mathbf{x},\mathbf{x}') = (\mathbf{x}^\top \mathbf{x}' + 1)^d$
* Hyperbolic tangent: $\kappa(\mathbf{x},\mathbf{x}') = \tanh(\gamma \mathbf{x}^\top \mathbf{x}'+c)$, for some (not every) $\gamma > 0$ and $c < 0$.

### 6.1. Example.

  [Source: A [notebook](https://github.com/jakevdp/sklearn_pycon2015/blob/master/notebooks/03.1-Classification-SVMs.ipynb) by [Jake Vanderplas](https://github.com/jakevdp>)]

Where SVM gets incredibly exciting is when it is used in conjunction with *kernels*.
To motivate the need for kernels, let's look at some data which is not linearly separable:

```python
X, y = make_circles(100, factor=.1, noise=.1)

clf = svm.SVC(kernel='linear').fit(X, y)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='copper')
plot_svc_decision_function(clf);
```

Clearly, no linear discrimination will ever separate these data.
One way we can adjust this is to apply a **kernel**, which is some functional transformation of the input data.

For example, one simple model we could use is a **radial basis function**

```python
r = np.exp(-(X[:, 0] ** 2 + X[:, 1] ** 2))
```

If we plot this along with our data, we can see the effect of it:

```python
def plot_3D(elev=30, azim=30):
    ax = plt.subplot(projection='3d')
    ax.scatter3D(X[:, 0], X[:, 1], r, c=y, s=50, cmap='spring')
    ax.view_init(elev=elev, azim=azim)
    ax.set_xlabel('x')
    ax.set_ylabel('y')
    ax.set_zlabel('r')

interact(plot_3D, elev=[-90, 90], azip=(-180, 180));
```

    interactive(children=(Dropdown(description='elev', options=(-90, 90), value=-90), IntSlider(value=30, descript…

We can see that with this additional dimension, the data becomes trivially linearly separable!
This is a relatively simple kernel; SVM has a more sophisticated version of this kernel built-in to the process. This is accomplished by using the Gaussian kernel (``kernel='rbf'``), short for *radial basis function*:

```python
clf = svm.SVC(kernel='rbf', C=10)
clf.fit(X, y)

plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='copper')
plot_svc_decision_function(clf)
plt.scatter(clf.support_vectors_[:, 0], clf.support_vectors_[:, 1],
            s=200, facecolors='none');
```

Here there are effectively $K$ basis functions: one centered at each point! Through a clever mathematical trick, this computation proceeds very efficiently using the "Kernel Trick", without actually constructing the matrix of kernel evaluations.

**Exercise**: Apply the linear SVM and the SVM with Gaussian kernel to the discrimination of classes `Versicolor` and `Virginica` in the Iris Dataset, using atributes $x_0$ and $x_1$ only. Plot the corresponding decision boundaries and the support vectors.

## 7. Hyperparameters

Note that the SVM formulation has several free parameters (hyperparameters) that must be selected out of the optimization problem:

- The free parameter $C$ used to solve non-separable problems.
- The kernel parameters (for instance, parameter $\gamma$ for the Gaussian kernel).

These parameters are usually adjusted using a cross-validation procedure.


# NBA Team Ratings, Pt. 2

In NBA Team Ratings, Pt. 1, I built a simple Bayesian Hierarchical Model achieved all our goals, i.e. applying shrinkage, taking matchups and home-court into account. That said, it left me wanting a bit more.
Specifically, that model ignores an immense amount of data, only looking at points per possession.
What if, instead of treating points per possession as a continuous variable, we estimated the probability of a team scoring 0, 1, 2, 3, or 4 points for each possession?
This would both give us useful insights into the way teams succeeded or failed, and also potentially model some of the non-linear dynamics of team matchups.

Starting off, our model of each game will be a multinomial distribution: given $ n $ trials of an event with probabilities $ p_1, p_2, \cdots, p_m $ for each potential outcome, it models the count of each outcome.
In our case, $ n $ is the number of possessions and the $ p_i $'s are the probabilities of scoring $ i $ points:

\begin{align}
\overrightarrow{x_i} &= [x_{i,0}, x_{i,1}, \cdots, x_{i,4}] \thicksim \text{Multi} \left( \text{poss}_i, \vec{p}_i \right) \notag \\
% \text{where } \overrightarrow{x_i} &= [x_{i,0}, x_{i,1}, \cdots, x_{i,4}] \notag \\
\text{where } \overrightarrow{x_i} &\equiv \text{number of possessions with 0, 1, ..., 4 points scored in the }i^\text{th} \text{ game} \notag \\
\text{poss}_i &\equiv \text{possessions in the }i^\text{th} \text{ game} \notag \\
\vec{p}_i &= [p_{i,0}, p_{i, 1}, \cdots, p_{i,4}] \notag \\
&\equiv \text{probability of 0, 1, ..., 4 points scored in a possession} \notag
\end{align}

Where things get a bit complex is in calculating $ \vec{p}_i $ for each matchup.
Say we know the average NBA team scores zero points in a possession 33\% of the time.
What percentage of possessions do we expect to have zero points when an offense that averages 50\% plays a defense that averages 25\%?
One way to combine these is through an extension of the "log5" method popularized by Bill James, by multiplying normalized odds.

First off, odds express the relative probability of an event occuring, i.e. odds are the likelihood of an event occurring, divided by the likelihood of it not occurring: $ \text{"odds"} = \frac{p}{1 - p} $.
In our NBA example, the odds of the league average team scoring no points are $ \frac{0.33}{1 - 0.33} = 0.5 $, while the offense has odds of $ \frac{0.5}{1 - 0.5} = 1 $, and the defense is at $ \frac{0.25}{1 - 0.25} = 0.33 $.
Next, we normalize each team's odds in relation to league average: the offense has $ \frac{1}{0.5} = 2 $, while the defense has $ \frac{0.33}{0.5} = 0.67 $, i.e. the offense's odds of scoring zero points are twice that of the average team, while the defense's odds are two thirds.
Finally, we multiply the normalized odds together to get a combined set of odds.

\begin{align}
\text{Given } \mathbb{P}(y) &= p_0, \text{ } \mathbb{P}(y|x_i) = p_i \notag \\
\mathbb{P}(y|x_i, x_2, \cdots, x_n) &= p \notag \\
\text{odds}_i &= \frac{p_i}{1 - p_i} \notag \\
\text{odds} &= \text{odds}_0 \prod_{i=1}^{n} \frac{\text{odds}_i}{\text{odds}_0} \notag
\end{align}

Here, each $x_i$ represents a factor, e.g. that the Phoenix Suns are on offense, that the Philadelphia 76'ers are on defense, or that the home team is on offense.
To get a combined probability out the other end, we simply reverse our original transformation of probability to odds, as follows:

\begin{align}
p &= \frac{\text{odds}}{1 + \text{odds}} \notag
\end{align}

Now that we have a way of combining probabilities to predict observations in a model, we need to define the parameters of our model.
\begin{align}
\overrightarrow{\text{off}}_i &\equiv \text{probability the ith team will score 0, 1, ..., 4 points on offense} \notag \\
\overrightarrow{\text{def}}_i &\equiv \text{probability the ith team will score 0, 1, ..., 4 points on defense} \notag \\
\overrightarrow{\text{home}}_i &\equiv \text{probability the home team will score 0, 1, ..., 4 points on offense} \notag \\
&\text{where } \overrightarrow{\text{off}}_i \text{, } \overrightarrow{\text{def}}_i \text{, and } \overrightarrow{\text{home}} \text{ are of shape } [p_0, p_1, \cdots, p_4] \notag
\end{align}

Defining priors for these model parameters requires introducing a new distribution, the Dirichlet, or multivariate beta distribution.
This distribution is commonly used in Bayesian statistics because it is the conjugate prior of the categorical, and more relevant to us, multinomial distributions.
It is a family of continuous, multivariate probability distributions parameterized by a concentration vector $\overrightarrow{\alpha}$.
We'll use it to define the $ \overrightarrow{\text{off}}_i $, $ \overrightarrow{\text{def}}_i $, and $ \overrightarrow{\text{home}} $ parameters:
\begin{align}
\overrightarrow{\text{off}}_i &\thicksim \text{Dir}(\overrightarrow{\alpha}_{\text{off}}) \notag \\
\overrightarrow{\text{def}}_i &\thicksim \text{Dir}(\overrightarrow{\alpha}_{\text{def}}) \notag \\
\overrightarrow{\text{home}} &\thicksim \text{Dir}(\overrightarrow{\alpha}_{\text{home}}) \notag
\end{align}

The mean of the Dirichlet distribution is defined by an L1-normalized $ \overrightarrow{\alpha} $:
\begin{align}
\operatorname{\mathbb{E}} \left(\text{Dir}(\overrightarrow{\alpha})\right) &= \frac{\overrightarrow{\alpha}}{\alpha_0} \notag \\
\text{where } \alpha_0 &\equiv \sum_i{\alpha_i} \notag
\end{align}

Since the variance is conveniently proportional to $ \frac{1}{\alpha_0^2} $, this allows us to parametrize $ \overrightarrow{\alpha}_{\text{off}} $, $ \overrightarrow{\alpha}_{\text{def}} $, and $ \overrightarrow{\alpha}_{\text{home}} $ as simply proportional to our prior for their probabilities.
For $ \overrightarrow{\text{off}}_i $ and $ \overrightarrow{\text{def}}_i $, our prior is simply league average, but for $ \overrightarrow{\text{home}} $, I used a prior relative to league average based on the improved performance of home teams in previous seasons.
I also chose to fix $ \alpha_{0,\text{home}} $ based on some experimentation, but used hyperpriors for the $ \alpha_{0,\text{off}}$ and $\alpha_{0,\text{def}}$ hyperparameters, using the familiar Gamma distribution:
\begin{align}
\alpha_{0,\text{off}} &\thicksim \text{Gamma}(1{,}000, 300) \notag \\
\alpha_{0,\text{def}} &\thicksim \text{Gamma}(2{,}000, 500) \notag \\
\alpha_{0,\text{home}} &\equiv 100{,}000 \notag
\end{align}

Running this new model on the 2019 NBA Regular Season yields similar results despite a completely different model formulation:

```python
import pymc3 as pm
from matplotlib import pyplot as plt
import pandas as pd
import numpy as np

from pynba import possessions_from_file, team_id_to_abb, plot_ratings, use_blackontrans_style

league = "nba"
year = 2019
season_type = "Regular Season"
```

```python
possessions = possessions_from_file(league, year, season_type)
```

```python
possessions["points_scored_corrected"] = possessions["points_scored"].map(lambda val: min(val, 4))
grouped = possessions.groupby(by=["off_team_id", "def_team_id", "home_team_id"], as_index=False)
matchups = grouped.agg(
    {
        "league": "first",
        "year": "first",
        "season_type": "first",
        "date": "first",
        "points_scored_corrected": "sum",
        "possession_num": "count",
    }
)
results = np.array([
    [(subgroup["points_scored_corrected"] == ind).sum() for ind in range(5)]
    for _, subgroup in grouped
])
```

```python
n_teams = matchups["off_team_id"].unique().shape[0]
team_id_to_team_ind = {
    team_id: team_ind
    for team_ind, team_id in enumerate(matchups["off_team_id"].unique())
}
team_id_to_team_abb = team_id_to_abb(league, year)
team_abb_to_team_id = {
    team_abb: team_id
    for team_id, team_abb in team_id_to_team_abb.items()
}
team_ind_to_team_abb = {
    team_ind: team_id_to_team_abb[team_id]
    for team_id, team_ind in team_id_to_team_ind.items()
}
mu = results.sum(0) / results.sum()
off_index = (
    matchups["off_team_id"].map(team_id_to_team_ind).to_numpy()
)
def_index = (
    matchups["def_team_id"].map(team_id_to_team_ind).to_numpy()
)
home_index = (
    (matchups["off_team_id"] != matchups["home_team_id"])
    .to_numpy()
    .astype(int)
)

with pm.Model() as model:
    alpha0_off = pm.Gamma("alpha0_off", mu=1000, sigma=300)
    alpha0_def = pm.Gamma("alpha0_def", mu=2000, sigma=500)
    alpha_off = mu * alpha0_off
    alpha_def = mu * alpha0_def
    alpha_home = (mu + np.array([-0.006, 0, 0.004, 0.002, 0])) * 100000
    teams_off = pm.Dirichlet("team_off", alpha_off, shape=(n_teams, 5))
    teams_def = pm.Dirichlet("team_def", alpha_def, shape=(n_teams, 5))
    hc_off = pm.Dirichlet("hc_off", alpha_home)
    hc_def = 2 * mu - hc_off
    offense = teams_off[off_index]
    defense = teams_def[def_index]
    home_court = pm.math.stack([hc_off, hc_def])[home_index]
    mu0 = np.tile((mu / (1 - mu)), (off_index.shape[0], 1))
    p0 = offense / (1 - offense) * defense / (1 - defense) * home_court / (1 - home_court) / mu0 ** 2
    p = p0 / (p0 + 1)
    pm.Multinomial("game_result", n=matchups["possession_num"].to_numpy().reshape(-1, 1), p=p, observed=results)
```

```python
with model:
    trace = pm.sample(
        draws=10000,
        init="adapt_diag",
        chains=4,
        random_seed=42,
        return_inferencedata=False,
    )
```

    2021-07-10 21:21:39 UTC | /home/jupyter/venv/lib/python3.8/site-packages/pymc3/sampling.py:sample @ 495 | INFO | Auto-assigning NUTS sampler...
    2021-07-10 21:21:39 UTC | /home/jupyter/venv/lib/python3.8/site-packages/pymc3/sampling.py:init_nuts @ 2081 | INFO | Initializing NUTS using adapt_diag...
    2021-07-10 21:23:15 UTC | /home/jupyter/venv/lib/python3.8/site-packages/pymc3/sampling.py:sample @ 556 | INFO | Multiprocess sampling (4 chains in 4 jobs)
    2021-07-10 21:23:15 UTC | /home/jupyter/venv/lib/python3.8/site-packages/pymc3/sampling.py:_print_step_hierarchy @ 232 | INFO | NUTS: [hc_off, team_def, team_off, alpha0_def, alpha0_off]

  <progress value='44000' class='' max='44000' style='width:300px; height:20px; vertical-align: middle;'></progress>
  100.00% [44000/44000 44:17<00:00 Sampling 4 chains, 0 divergences]

    2021-07-10 22:07:34 UTC | /home/jupyter/venv/lib/python3.8/site-packages/pymc3/sampling.py:sample @ 628 | INFO | Sampling 4 chains for 1_000 tune and 10_000 draw iterations (4_000 + 40_000 draws total) took 2658 seconds.

```python
off_probs = trace["team_off"].mean(0)
off_ratings = ((off_probs - mu).dot(np.arange(5).reshape(-1, 1)) * 100).reshape(-1)
def_probs = trace["team_def"].mean(0)
def_ratings = ((mu - def_probs).dot(np.arange(5).reshape(-1, 1)) * 100).reshape(-1)
```

```python
results_df = pd.DataFrame({
    "team": [team_ind_to_team_abb[ind] for ind in range(n_teams)],
    "net_scoring_above_average": off_ratings + def_ratings,
    "off_scoring_above_average": off_ratings,
    "def_scoring_above_average": def_ratings,
    "league": league,
    "year": year,
    "season_type": season_type,
})

results_df.sort_values("net_scoring_above_average", ascending=False)
```

      team
      net_scoring_above_average
      off_scoring_above_average
      def_scoring_above_average
      league
      year
      season_type

      12
      MIL
      6.514540
      3.146588
      3.367953
      nba
      2019
      Regular Season

      7
      GSW
      5.594340
      4.637548
      0.956792
      nba
      2019
      Regular Season

      24
      TOR
      4.441016
      2.327319
      2.113697
      nba
      2019
      Regular Season

      8
      HOU
      4.162117
      4.397545
      -0.235429
      nba
      2019
      Regular Season

      25
      UTA
      3.990294
      0.373729
      3.616566
      nba
      2019
      Regular Season

      20
      POR
      3.693816
      3.550877
      0.142939
      nba
      2019
      Regular Season

      6
      DEN
      3.569358
      2.111179
      1.458179
      nba
      2019
      Regular Season

      1
      BOS
      3.028115
      1.238901
      1.789215
      nba
      2019
      Regular Season

      23
      OKC
      2.663322
      -0.084723
      2.748045
      nba
      2019
      Regular Season

      17
      IND
      2.035635
      -0.513273
      2.548908
      nba
      2019
      Regular Season

      18
      PHI
      1.834752
      1.550283
      0.284469
      nba
      2019
      Regular Season

      22
      SAS
      1.592312
      2.181690
      -0.589378
      nba
      2019
      Regular Season

      9
      LAC
      1.000700
      1.548013
      -0.547313
      nba
      2019
      Regular Season

      16
      ORL
      0.168599
      -1.424785
      1.593384
      nba
      2019
      Regular Season

      14
      BKN
      -0.243152
      -0.789682
      0.546530
      nba
      2019
      Regular Season

      21
      SAC
      -0.707013
      -0.003210
      -0.703804
      nba
      2019
      Regular Season

      29
      CHA
      -0.756276
      1.147352
      -1.903628
      nba
      2019
      Regular Season

      11
      MIA
      -0.780095
      -2.693527
      1.913432
      nba
      2019
      Regular Season

      28
      DET
      -0.780359
      -1.442858
      0.662499
      nba
      2019
      Regular Season

      3
      NOP
      -0.836531
      0.932292
      -1.768823
      nba
      2019
      Regular Season

      5
      DAL
      -0.945833
      -0.863206
      -0.082627
      nba
      2019
      Regular Season

      13
      MIN
      -0.973287
      0.872847
      -1.846134
      nba
      2019
      Regular Season

      10
      LAL
      -1.033815
      -2.046979
      1.013165
      nba
      2019
      Regular Season

      27
      WAS
      -1.943052
      0.585976
      -2.529028
      nba
      2019
      Regular Season

      26
      MEM
      -1.991411
      -3.615139
      1.623728
      nba
      2019
      Regular Season

      0
      ATL
      -4.306885
      -1.707600
      -2.599285
      nba
      2019
      Regular Season

      19
      PHX
      -6.543554
      -3.640806
      -2.902748
      nba
      2019
      Regular Season

      4
      CHI
      -6.706195
      -4.482405
      -2.223790
      nba
      2019
      Regular Season

      15
      NYK
      -7.208345
      -4.755111
      -2.453234
      nba
      2019
      Regular Season

      2
      CLE
      -7.741308
      -2.322653
      -5.418655
      nba
      2019
      Regular Season

```python
use_blackontrans_style()

fig = plt.figure(figsize=(8, 8))
ax = fig.add_subplot(1, 1, 1)
points_mu = possessions["points_scored_corrected"].sum() / possessions.shape[0] * 100
for team, off_aa, def_aa in zip(results_df["team"], results_df["off_scoring_above_average"], results_df["def_scoring_above_average"]):
    team_id = team_abb_to_team_id[team]
    filt = matchups["off_team_id"] == team_id
    raw_off = matchups.loc[filt, "points_scored_corrected"].sum() / matchups.loc[filt, "possession_num"].sum() * 100 - points_mu
    filt = matchups["def_team_id"] == team_id
    raw_def = points_mu - matchups.loc[filt, "points_scored_corrected"].sum() / matchups.loc[filt, "possession_num"].sum() * 100
    ax.annotate("", xy=(off_aa, def_aa), xytext=(raw_off, raw_def), arrowprops={"arrowstyle": "->", "color": "white"})
plot_ratings(results_df, ax)

```


```javascript
%%javascript
MathJax.Hub.Config({
    TeX: { equationNumbers: { autoNumber: "AMS" } }
});
```

    <IPython.core.display.Javascript object>

# 03. Differential Privacy

[Differential privacy](https://www.microsoft.com/en-us/research/publication/differential-privacy/) is a popular mechanism to quantitatively assess the privacy loss of a given probabilistic query schema or data transformation method. The fundamental equation of differntial privacy is given as

$$
\begin{equation}
    \mathrm{Pr}[\mathcal{K}(D_1) \in S] \le exp(\epsilon) \times \mathrm{Pr}[\mathcal{K}(D_2) \in S] \label{eq:dp}
\end{equation}
$$

Here, $\mathrm{Pr}[\mathcal{K}(D_1) \in S]$ is the probability of a
randomized function $\mathcal{K}$ to yield one of values in the $S$ when evaluating
it on a given dataset $D_1$. The right side is identical to the left except
that the function is now evaluated on a dataset $D_2$ that differs from $D_1$
in at most one element. And finally, $\epsilon$ is a parameter that describes
how much information is leaked (or generated) by the function.

Sounds pretty abstract, so let's work out a simple example: Let's assume we want to build a differentially private dataset from the adult data that we've looked at in part 02. The goal here is to protect an adversary from gaining too much information about the sensitive attribute (income > 50k or not) when adding that person's data to the dataset. With differential privacy, we look at the state of the dataset before and after a person was added and quantify the privacy loss as given by equation ($\ref{eq:dp}$). The scheme we're evaluating here is a so-called **randomized response method**, which works as follows:

* With probability $1-p$, we add a person's true attribute value to the database.
* With probability $p$ we choose a random boolean (0/1) value from a distribution returning $0$ with probability $k$ and $1$ with probability $1-k$ and add that value to the database instead.

Using this scheme, an attacker cannot know with certainty if the real attribute value of the person or a random one was added to the database. This protects the privacy of the person but of course it also adds noise to the database, making it more difficult to use for legitimate users as well.

In practice, we therefore always need to weigh privacy against utility when employing differential privacy. In this notebook, we will calculate the $\epsilon$ and other relevant parameters for our scheme above and see how we can use this differentially private data to make predictions about the income distribution of the people in our dataset.

#### Calculating $\epsilon$

In our differentially private scheme, the probability of adding the true attribute value to the database is $1-p$. The probability of adding a random value is therefore $p$ and the probability of that value being $0$ is $k$. So how can we relate this to eq. (\ref{eq:dp})? Well, we can set $D_1$ and $D_2$ as the versions of our database **before** and **after** adding the person's data to it. Let's say that before adding the person's data there are $n$ $1$'s in the database. We can then use a query $\mathcal{K}$ that returns the number of 1's in the database and choose our result set as $S = \{n\}$. Before adding the person to the database, $\mathcal{K}(D_1)=n$ with certainty, hence $\mathrm{Pr}(\mathcal{K}(D_1))=1$. After adding the person's data, the probability that the query result is still $n$ can be calculated as follows, depending on the person's attribute value:

* If a persons's attribute value is $0$, the probability that $\mathcal{K}$ is unchanged after adding the data to the database is given as $1-p+p\cdot k$.
* If a person's attribute value is $1$, the probability that $\mathcal{K}$ is unchanged after adding the data to the database is given as $p\cdot k$.

We therefore have the two equations

$$
\begin{eqnarray}
\mathrm{Pr}[\mathcal{K}(D_1) \in S | x_i=1] & = & 1 \le \exp{\epsilon}\cdot \mathrm{Pr}[\mathcal{K}(D_2) \in S | x_i=1] = \exp{\epsilon}\cdot p \cdot k \\
\mathrm{Pr}[\mathcal{K}(D_1) \in S | x_i=0] & = & 1 \le \exp{\epsilon}\cdot \mathrm{Pr}[\mathcal{K}(D_2) \in S | x_i=0] = \exp{\epsilon}\cdot (1-p+p \cdot k) \\
\end{eqnarray}
$$

This yields

$$
\begin{eqnarray}
 \epsilon & \ge & -\ln{\left(p \cdot k\right)} \\
 \epsilon & \ge & -\ln{\left(1-p+p\cdot k\right)} \\
\end{eqnarray}
$$

Since we're interested in an upper bound for $\epsilon$ and since $-\ln{\left(1-p+p\cdot k\right)} \le -\ln{p\cdot k}$, we obtain

$$
\begin{equation}
\epsilon = -\ln{\left(p\cdot k\right)}
\end{equation}
$$

## Exercise

**Write a function that returns the value of epsilon for a given $p$ and $k$.**

```python
# %load "../solutions/differential-privacy/epsilon.py"

def epsilon(p, k):
    """
    :param p: The probability of returning a random value instead of the true one
    :param k: The probability of returning 1 when generating a random value
    :returns: The epsilon for the given values of p, k
    """
```

## Exercise

** Plot $\epsilon$ for various values of $p$ and $k$.**

## Exercise: Different Scheme

Let's assume we propose the following anonymization scheme for our dataset:

* With probability $1-p$, we add a person's true attribute value to the database
* With probability $p$, we do not add anything to the database

Can you calculate the $\epsilon$ of this scheme? Which scheme do you prefer, and why? Does this scheme always provide "plausible deniability"?

```python
%load "../solutions/differential-privacy/different-scheme.md"
```

## What does this tell us?

Calculating the $\epsilon$ is great, but what does it actually tell us about the privacy loss or risk for our use case? Let's assume an adversary want to learn about the real value of a person's attribute. If she knows the model used for generating the data, she could then use Bayesian reasoning to calculate the probability of a person's attribute being $1$ given the observed difference in the database, which we call $\Delta$. Using Bayes theorem we can calculate this as (for $\Delta = 1$ here)

$$
\begin{equation}
    P(x_i=1 | \Delta = 1) = P(\Delta = 1| x_i = 1)\cdot \frac{P(x_i=1)}{P(\Delta=0)}
\end{equation}
$$

For our scheme, we know that

$$
\begin{equation}
    P(\Delta = 1 | x_i = \mathrm{1}) = (1-p) + p\cdot(1-k) = 1-pk
\end{equation}
$$

and

$$
\begin{equation}
    P(\Delta = 1) = (1-p)\cdot P(x_i = \mathrm{1}) + p\cdot(1-k)
\end{equation}
$$

so we obtain

$$
\begin{equation}
    P(x_i=1 | \Delta = 1) = \frac{(1-pk)\cdot P(x_i = \mathrm{1})}{(1-p)\cdot P(x_i = \mathrm{1})+p\cdot(1-k)}
\end{equation}
$$

Let's see how this relates to $\epsilon$!

## Exercise

**Write a function that calculates thecondition probability as given in eq. (4).**

```python
# %load "../solutions/differential-privacy/conditional-prob.py"

def p_cond(p, k, p_1):
    """
    :param   p: The probability of returning a random value instead of the true one
    :param   k: The probability of returning 1 when generating a random value
    :param p_1: The probability of a person to have an attribute value x_i=1
    """
```

## Exercise

** Choose a given k (e.g. 0.5) as well as a value for P(x_i=yes) and plot the conditional probability from eq. (4) as a function of p.**

# Implementing It

Now that we have a feeling for our scheme we can implement it! For that, we load the "adult census" data from the k-anonymity case study again.

```python
import pandas as pd

names = (
    'age',
    'workclass', #Private, Self-emp-not-inc, Self-emp-inc, Federal-gov, Local-gov, State-gov, Without-pay, Never-worked.
    'fnlwgt', # "weight" of that person in the dataset (i.e. how many people does that person represent) -> https://www.kansascityfed.org/research/datamuseum/cps/coreinfo/keyconcepts/weights
    'education',
    'education-num',
    'marital-status',
    'occupation',
    'relationship',
    'race',
    'sex',
    'capital-gain',
    'capital-loss',
    'hours-per-week',
    'native-country',
    'income',
)
categorical = set((
    'workclass',
    'education',
    'marital-status',
    'occupation',
    'relationship',
    'sex',
    'native-country',
    'race',
    'income',
))
df = pd.read_csv("../data/k-anonymity/adult.all.txt", sep=", ", header=None, names=names, index_col=False, engine='python');# We load the data using Pandas
```

## Exercise

**Implement a function that processes a new value according to the differentially private scheme discussed above.**

```python
# %load "../solutions/differential-privacy/process-value.py"

import random

def process_value(value, p, k):
    """
    :param value: The value to apply the differentially private scheme to.
    :param     p: The probability of returning a random value instead of the true one
    :param     k: The probability of returning 1 when generating a random value
    :    returns: A new, differentially private value
    """
```

## Exercise

**Now apply this method to the "income" column of the adult dataset to obtain a differentially private dataset.**

```python
# %load "../solutions/differential-privacy/apply.py"

import numpy as np

p = 0.8
k = 0.4

df['income_binary'] = np.where(df['income'] == '<=50k', 0, 1)
df['income_dp'] = 0

# ...
```

# Working With Differentially Private Data

After collecting the differentially private data, we want of course to make use of it! For example, we might want to estimate the probability of a person having an income > 50T\$ based on the data we've collected, which we assume is [Bernoulli distributed](https://en.wikipedia.org/wiki/Bernoulli_distribution) with a probability $p_{1}$. Now, when adding up the data from $n$ persons, the resulting value is [binomially distributed](https://en.wikipedia.org/wiki/Bernoulli_distribution). The mean of this distribution is given as $\mathrm{E}_{1} = n\cdot p_{1}$ and the variance as $\mathrm{Var}_1 = n\cdot p_1 \cdot (1-p_1)$. A consistent and unbiased estimator of $\mathrm{E}_1$ is $\hat{\mathrm{E}}_{1} = \sum_i x_{1}^i$, which then gives an estimate for $p_{1}$ of $\hat{p}_{1} = \hat{\mathrm{E}}_1/n$.

Now, if we apply the differential privacy mechanism to our dataset, the probability of obtaining a $1$ will change to $p_{1,dp} = (1-p)\cdot p_{1}+p\cdot(1-k)$. Therefore, an unbiased and consistent estimator of $p_1$ based on $p_{1,dp}$ is given as
$$
\begin{equation}
\hat{p}_1 = \frac{\hat{p}_{1,dp}-p\cdot(1-k)}{1-p}
\end{equation}
$$

As before, $\hat{p}_{1,dp}=\sum_i x_{1,dp}^i/n$. Note that this naive estimator can return a negative probability, which can be avoided by using a more suitable method like a maximum likelihood estimator.

## Exercise

**Write an estimator for $\hat{p}_1$ based on a differentially private dataset with parameters $p$ and $k$..**

```python
# %load "../solutions/differential-privacy/p-1.py"

def p_1_estimator(p_1dp, p, k):
    """
    :param p_1dp: The empirical probability of x_i=1 of our DP dataset.
    :param     p: The p value of our DP scheme.
    :param     k: The k value of our DP scheme.
    :    returns: An estimate of p_1 of our DP dataset.
    """
```

## Exercise

**Apply the estimator to the differentially private dataset created above to generate an estimate of $p_1$.**

```python
# %load "../solutions/differential-privacy/estimate-p1.py"

p_1_hat = ...
```

## Exercise

**Write a function that estimates the variance of $\hat{p}_{1}$ and calculate its value for the case above.**

Hint: The variance of $\hat{p}_1$ can be estimated as $$\hat{\mathrm{Var}}_1 = \frac{\hat{\mathrm{Var}}_{1,dp}}{(1-p)^2} = \frac{\hat{p}_{1,dp}\cdot(1-\hat{p}_{1,dp})}{(1-p)^2\cdot n}$$

```python
# %load "../solutions/differential-privacy/estimate-var.py"

def var_1_estimator(p_1dp, n, p, k):
    """
    :param p_1dp: The empirical probability of x_i=1 of our DP dataset.
    :param     n: The number of samples in our dataset.
    :param     p: The p value of our DP scheme.
    :param     k: The k value of our DP scheme.
    :    returns: An estimate of the variance of our DP dataset.
    """

var_1_hat = var_1_estimator(p_1dp, len(df), p, k)
var_1_hat
```

    0.00012764584739878977

## Exercise

** Repeat the data generation process $N$ (e.g. 500) times. For each resulting dataset, estimate $\hat{p}_1$ and store the value in a list, so that we can plot it later.**

```python
# %load "../solutions/differential-privacy/repeat-dp.py"

p_1_hats = []

for j in range(500):

    # ...

    p_1_hats.append(p_1_hat)

p_1_hats = np.array(p_1_hats)
```

```python
# We then compare these estimates to the expected distribution (via central limit theorem: a normal distribution
# with expectation p_1 and variance var_1_hat)

import matplotlib.pylab as pl

pl.hist(p_1_hats, density=True)

gauss = lambda x, mu, var: 1/np.sqrt(2*np.pi*var)*np.exp(-(x-mu)**2/(2*var))

p_1_hat = p_1_hats.mean()

x = np.linspace(0.1, 0.3, 1000)

pl.plot(x, gauss(x, p_1_hat, var_1_hat));
```

# Summary

That's it! As you can see, applying a differential privacy mechanism to your data is not so difficult, you have to take into account the added noise though, which will make your estimates less precise.

notebooks/03 - Differential Privacy.ipynb


```python
from math import pi
import numpy as np
import sympy as sy
from sympy.utilities.lambdify import lambdify
```

```python
import matplotlib.pyplot as plt
```

```python
phi = sy.Symbol("phi")
_lambda = sy.Symbol("Lambda")
delta = sy.Symbol("delta")

R_x = sy.rot_axis1(phi).T
R_z = sy.rot_axis3(-_lambda).T
R_y = sy.rot_axis2(delta).T
```

```python
vector = sy.Matrix([0,1,0])

final_vector = R_y * R_z * R_x * vector

planform_vector = sy.Matrix(final_vector)
planform_vector[-1] = 0
```

```python
print(sy.latex(planform_vector, mul_symbol="\cdot"))
```

    \left[\begin{matrix}\sin{\left(\Lambda \right)}\cdot\cos{\left(\delta \right)}\cdot\cos{\left(\phi \right)} + \sin{\left(\delta \right)}\cdot\sin{\left(\phi \right)}\\\cos{\left(\Lambda \right)}\cdot\cos{\left(\phi \right)}\\0\end{matrix}\right]

```python
planform_vector
```

$\displaystyle \left[\begin{matrix}\sin{\left(\Lambda \right)} \cos{\left(\delta \right)} \cos{\left(\phi \right)} + \sin{\left(\delta \right)} \sin{\left(\phi \right)}\\\cos{\left(\Lambda \right)} \cos{\left(\phi \right)}\\0\end{matrix}\right]$

```python
sy.simplify((final_vector.T * u_planform_vector)[0])
```

$\displaystyle \frac{\left(\sin{\left(\Lambda \right)} \cos{\left(\delta \right)} \cos{\left(\phi \right)} + \sin{\left(\delta \right)} \sin{\left(\phi \right)}\right) \left(\sin{\left(\phi \right)} + \cos{\left(\Lambda \right)} \cos{\left(\phi \right)}\right) \sin{\left(\Lambda \right)} + \cos^{2}{\left(\Lambda \right)} \cos^{2}{\left(\phi \right)}}{\cos{\left(\phi \right)}}$

```python
sy.simplify((planform_vector.T / sy.cos(phi) * final_vector)[0])
```

$\displaystyle \frac{\left(\sin{\left(\Lambda \right)} \cos{\left(\delta \right)} \cos{\left(\phi \right)} + \sin{\left(\delta \right)} \sin{\left(\phi \right)}\right)^{2}}{\cos{\left(\phi \right)}} + \cos^{2}{\left(\Lambda \right)} \cos{\left(\phi \right)}$

```python
print(sy.python(final_vector, ))
```

    Lambda = Symbol('Lambda')
    delta = Symbol('delta')
    phi = Symbol('phi')
    e = MutableDenseMatrix([[sin(Lambda)*cos(delta)*cos(phi) + sin(delta)*sin(phi)], [cos(Lambda)*cos(phi)], [-sin(Lambda)*sin(delta)*cos(phi) + sin(phi)*cos(delta)]])

```python
final_vector
```

$\displaystyle \left[\begin{matrix}\sin{\left(\Lambda \right)} \cos{\left(\delta \right)} \cos{\left(\phi \right)} + \sin{\left(\delta \right)} \sin{\left(\phi \right)}\\\cos{\left(\Lambda \right)} \cos{\left(\phi \right)}\\- \sin{\left(\Lambda \right)} \sin{\left(\delta \right)} \cos{\left(\phi \right)} + \sin{\left(\phi \right)} \cos{\left(\delta \right)}\end{matrix}\right]$

```python
sy.simplify(final_vector.subs(delta, 0)[-1] / final_vector.subs(delta, 0)[0])
```

$\displaystyle \frac{\tan{\left(\phi \right)}}{\sin{\left(\Lambda \right)}}$

```python
sy.tan(pi/2)
```

$\displaystyle 1.63312393531954 \cdot 10^{16}$

```python
final_vector[0]
```

$\displaystyle \sin{\left(\Lambda \right)} \cos{\left(\delta \right)} \cos{\left(\phi \right)} + \sin{\left(\delta \right)} \sin{\left(\phi \right)}$

```python
numerical_vector = lambdify((phi, _lambda, delta), final_vector)
```

```python
fig = plt.figure(figsize=(10,10))
axes = fig.add_subplot(111, projection='3d')

vec = numerical_vector(pi/3, pi/10, pi/10)

# Winglet LE
axes.plot([0, vec[0]], [0, vec[1]], [0, vec[2]], lw = 2.0)

# Plot projections
options_proj = dict(lw = 1.0, linestyle = '--')

axes.plot([0, vec[0]], [0, vec[1]], [0, 0], **options_proj)
axes.plot([0, 0], [0, vec[1]], [0, vec[2]], **options_proj)
axes.plot([0, vec[0]], [1, 1], [0, vec[2]], **options_proj)

axes.set_xlabel("x")
axes.set_ylabel("y")
axes.set_zlabel("z")

axes.set_xlim(xmin = 0.0, xmax = 1.0)
axes.set_ylim(ymin = 0.0, ymax = 1.0)
axes.set_zlim(zmin = 0.0, zmax = 1.0)
axes.set_title("Winglet LE and projections")
```

```python
print(sy.latex(vector))
```

    \left[\begin{matrix}0\\1\\0\end{matrix}\right]

```python
fig = plt.figure(figsize=(10,10))
axes = fig.add_subplot(121, projection='3d')

vec_1 = numerical_vector(pi/4, pi/4)
vec_2 = numerical_vector(pi/4, pi/3)

# Winglet LE
axes.plot([0, vec_1[0]], [0, vec_1[1]], [0, vec_1[2]], lw = 2.0, c = 'b')
axes.plot([0, vec_2[0]], [0, vec_2[1]], [0, vec_2[2]], lw = 2.0, c="r")

# Plot projections
options_proj = dict(lw = 1.0, linestyle = '--')

# Out of plane projection
axes.plot([0,0], [0, vec_1[1]], [0, vec_1[2]], **options_proj, c = 'b')
axes.plot([0,0], [0, vec_2[1]], [0, vec_2[2]], **options_proj, c = "r")

# In plane projection
axes.plot([0, vec_1[0]], [0, vec_1[1]], [0,0], **options_proj, c = 'b')
axes.plot([0, vec_2[0]], [0, vec_2[1]], [0,0], **options_proj, c = "r")

axes.set_xlabel("x")
axes.set_ylabel("y")
axes.set_zlabel("z")

axes.set_xlim(xmin = 0.0, xmax = 1.0)
axes.set_ylim(ymin = 0.0, ymax = 1.0)
axes.set_zlim(zmin = 0.0, zmax = 1.0)
axes.set_title("Sweep angle influence in the zy-projection")

axes = fig.add_subplot(122, projection='3d')
vec_1 = numerical_vector(pi/4, pi/4)
vec_2 = numerical_vector(pi/3, pi/4)

# Winglet LE
axes.plot([0, vec_1[0]], [0, vec_1[1]], [0, vec_1[2]], lw = 2.0, c = 'b')
axes.plot([0, vec_2[0]], [0, vec_2[1]], [0, vec_2[2]], lw = 2.0, c="r")

# Plot projections
options_proj = dict(lw = 1.0, linestyle = '--')

# Out of plane projection
axes.plot([0,0], [0, vec_1[1]], [0, vec_1[2]], **options_proj, c = 'b')
axes.plot([0,0], [0, vec_2[1]], [0, vec_2[2]], **options_proj, c = "r")

# In plane projection
axes.plot([0, vec_1[0]], [0, vec_1[1]], [0,0], **options_proj, c = 'b')
axes.plot([0, vec_2[0]], [0, vec_2[1]], [0,0], **options_proj, c = "r")

axes.set_xlabel("x")
axes.set_ylabel("y")
axes.set_zlabel("z")

axes.set_xlim(xmin = 0.0, xmax = 1.0)
axes.set_ylim(ymin = 0.0, ymax = 1.0)
axes.set_zlim(zmin = 0.0, zmax = 1.0)
axes.set_title("Cant angle influence in the xy-projection")

plt.savefig("sweep_cant_angle_projections.png", dpi = 300, bbox_inches="tight", transparent=True)
```